Conversation
Add structured extract syntax supporting three languages:
- jq (default, string shorthand): `extract: ".emails[]"`
- JSONPath (via jsonpath-rust): `extract: { expr: "$.emails[*]", type: jsonpath }`
- CEL (via cel-rust): `extract: { expr: "data.items.filter(x, x > 3)", type: cel }`
Changes:
- Add ExtractExpr enum (Jq/JsonPath/Cel) to rsigma-eval sources
- Update parser to handle both string (jq shorthand) and mapping forms
- Add File source extract field (previously only Http/Command/Nats had it)
- Implement JSONPath via jsonpath-rust (RFC 9535 compliant)
- Implement CEL via cel crate with json<->cel value conversion
- Update all resolvers to pass typed ExtractExpr
- Add integration tests for all three languages + error cases
- Add parsing tests for structured extract syntax
- Add TTL support to SourceCache (with_ttl, with_sqlite_and_ttl)
- Entries older than TTL are not returned from get()
- Add evict_expired() method for periodic cleanup
- Add DELETE /api/v1/sources/cache/{source_id} endpoint
- Store InstrumentedResolver in AppState for cache access
- Expose cache() method on InstrumentedResolver
- Add tests for TTL expiration and eviction
Prevent cycles in include expansion by rejecting included content that itself contains Include directives. This provides a hard safety guarantee against infinite recursion when processing dynamic pipeline includes from potentially untrusted sources. - Add MAX_INCLUDE_DEPTH constant (1) - Scan parsed included transformations for nested Include directives - Return descriptive error identifying the offending source - Add unit test for nested include rejection
Subscribe to `rsigma.control.resolve` when daemon-nats is enabled and dynamic sources exist. Publishing to this subject triggers re-resolution: - Empty payload: re-resolve all sources - Payload with source ID: re-resolve that specific source This enables external orchestrators to proactively push cache invalidation signals without polling the HTTP API.
SIGHUP now triggers both rule reload AND dynamic source re-resolution. This allows operators to force-refresh all sources with a single signal, complementing the existing interval/watch/API/NATS control mechanisms.
Add --allow-remote-include flag to the daemon subcommand that permits include directives to reference HTTP and NATS sources. By default, includes are restricted to local sources (file/command) for security.
Optional (required: false) sources that fail at startup now get automatically retried 5 seconds later via the refresh scheduler. This avoids blocking daemon startup on transient failures while ensuring optional data becomes available once the source recovers.
Add resolve_all_with_state() that updates PipelineState source tracking (mark_source_resolved/mark_source_failed) as sources are resolved. This enables pipeline state consumers to observe resolution progress and status per source, complementing the existing PipelineState API.
Records the Unix timestamp of the last successful resolution for each dynamic source. This enables monitoring dashboards and alerting on stale sources (e.g. source not resolved in the last N minutes).
When --resolve-sources is passed, the validate command also resolves dynamic pipeline sources (file/command/HTTP) and expands templates. This catches source connectivity issues during CI/pre-deploy validation without requiring the full daemon.
When dynamic sources are configured, the status endpoint now includes a dynamic_sources object with total sources tracked, cumulative resolves, errors, and cache hits. Omitted entirely when no dynamic sources are in use.
When --dry-run is passed, the resolve command lists all matching dynamic sources with their type, required status, and refresh policy without performing actual resolution. Useful for verifying pipeline configuration before connecting to external sources.
- Use tokio::task::block_in_place when resolving dynamic sources in load_rules to prevent panic when called from async context (SIGHUP, file watcher hot-reload) - Handle unused Result from CEL context.add_variable (MSRV CI fix)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
jsonpath-rust) and CEL (cel) extract languages with structured extract syntax ({ expr: "...", type: jsonpath|cel|jq }) alongside the existing JQ shorthandDELETE /api/v1/sources/cache/{source_id}invalidation endpointrsigma.control.resolve) for remote re-resolution triggers--allow-remote-includeCLI flag exposed on daemonresolve_all_with_state()for PipelineState source trackingrsigma_source_last_resolved_timestampgauge metricrsigma validate --resolve-sourcesflag for CI validation/api/v1/statusextended withdynamic_sourcessummaryrsigma resolve --dry-runflagTest plan
cargo test --workspace)cargo clippy --workspace --all-targets --all-features -- -D warningscleanrsigma resolve --dry-run -p pipeline.ymlrsigma validate --resolve-sources -p pipeline.yml rules/--allow-remote-include, verify SIGHUP triggers both reload + resolve