Skip to content

feat: add HTTP and NATS output sinks#18

Merged
mostafa merged 1 commit into
mainfrom
feat/http-nats-output-sinks
Apr 25, 2026
Merged

feat: add HTTP and NATS output sinks#18
mostafa merged 1 commit into
mainfrom
feat/http-nats-output-sinks

Conversation

@mostafa
Copy link
Copy Markdown
Member

@mostafa mostafa commented Apr 25, 2026

Summary

  • Add HttpSink -- POST batched NDJSON lines to a remote HTTP endpoint using reqwest. Lines are batched by count (batch_size, default 100) or time (batch_timeout_ms, default 500ms) and sent as a single NDJSON body. Retries on 5xx/timeout with exponential backoff (max_retries, default 3). Custom headers configurable in YAML (e.g. Authorization).
  • Add NatsSink -- publish each NDJSON line to a NATS subject via core publish (fire-and-forget) using async-nats. Behind the nats feature flag to keep the dependency optional. Subject parsed from the URL path (nats://host:port/subject) or overridden via global.output.nats.subject.
  • Both sinks are selectable via URL-scheme detection on the existing --output CLI flag: --output http://... for HTTP, --output nats://... for NATS, plain paths remain file output, absent remains stdout.
  • Both implement the existing sync EventSink trait using a tokio channel + spawned async worker, and slot into the existing BackpressureSink wrapper without changes.
  • New config section global.output with http and nats sub-blocks for optional tuning.
  • Updated README with new output types, quick-start examples, and config reference entries.

Files changed

File What
Cargo.toml async-nats optional dep, nats feature flag
src/config/global.rs OutputConfig, HttpOutputConfig, NatsOutputConfig
src/output.rs HttpSink, NatsSink, parse_nats_url(), tests
src/main.rs Sink construction with URL-scheme detection
README.md Documentation for new output sinks

Test plan

  • Unit tests for parse_nats_url (4 cases: with subject, nested subject, no subject, trailing slash)
  • Integration test for HttpSink basic POST (wiremock mock server, verifies batch delivery)
  • Integration test for HttpSink retry on 503 (wiremock, verifies 3 attempts before success)
  • cargo clippy clean (default features and --features nats)
  • cargo test passes (all 139 tests)

Extend Helr's output beyond stdout/file with two new EventSink
implementations, selectable via URL scheme on the existing --output flag:

- HttpSink: POST batched NDJSON to an HTTP endpoint (reqwest). Batches
  by count or timeout, retries on 5xx/timeout with exponential backoff.
  Configurable via global.output.http (batch_size, headers, retries).

- NatsSink: publish each NDJSON line to a NATS subject via core publish
  (async-nats, behind --features nats). Subject parsed from URL or
  overridden via global.output.nats.subject.

Both use a tokio channel + spawned async worker to bridge the sync
EventSink trait, and slot into the existing BackpressureSink wrapper
without changes.
@mostafa mostafa merged commit 3561ccc into main Apr 25, 2026
6 checks passed
@mostafa mostafa deleted the feat/http-nats-output-sinks branch April 25, 2026 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant