Skip to content

feat(dynalogs): structured logs per-run / per-node with secret redaction (#300) - #333

Merged
codeforstartups merged 1 commit into
codeforstartups:developmentfrom
shivamm-gupta:issue-300-structured-run-logs
Oct 7, 2026
Merged

codeforstartups merged 1 commit into
codeforstartups:developmentfrom
shivamm-gupta:issue-300-structured-run-logs

Conversation

@shivamm-gupta

Copy link
Copy Markdown
Collaborator

Description

Closes #300

Implements structured, queryable logs at run and node granularity under the dynalogs pillar, building on dynavec's opt-in structured logging and secret redaction (#200).

Changes

  1. Structured Models & Serialization:

    • Introduced \RunLogRecord\ representing agent/graph run and node execution events with ISO UTC timestamps, correlation identifiers (
      un_id,
      ode_id, \span_id), severity level, messages, contextual metadata dictionaries, and exception tracebacks.
    • Provides lossless dictionary and single-line JSON serialization with automatic secret redaction.
  2. Context Propagation & Automatic Correlation:

    • Added
      un_context(run_id=...)\ and
      ode_context(node_id=..., span_id=...)\ context managers leveraging \contextvars\ for async and threaded context isolation.
    • Any log call executed within active contexts automatically inherits and stamps active correlation identifiers without requiring manual propagation.
  3. Secret Redaction Enhancements:

    • Extended recursive secret sanitization across authorization headers, Bearer tokens, private keys, API keys, AWS credentials, and embedded exceptions.
    • Guarded AI metrics (\ okens, \ otal_tokens, \prompt_tokens, etc.) so execution token counts remain numeric and unredacted.
  4. Pluggable Sinks:

    • \LogSink\ interface with \StdoutSink\ (streaming formatted JSON to stdout/streams), \InMemoryLogSink\ (filtering/inspecting buffered records in memory), and \DynamoDBLogSink\ (persisting partitioned log records into DynamoDB single-table schema).
  5. Ergonomic Logging API:

    • Added \RunLogger\ with level filtering, sink dispatching, and standard methods (\debug, \info, \warning, \error, \exception).
    • Exposed \get_run_logger(), \set_default_run_logger(), and \log_run_event()\ helper.

Verification

  • Added 10 comprehensive unit tests in \ ests/test_run_logging.py\ covering context propagation, auto-generated IDs, secret redaction, min-level filtering, exception tracebacks, \StdoutSink, \InMemoryLogSink, and buffered/unbuffered \DynamoDBLogSink.
  • All 15 logging tests passed offline with 0 AWS dependencies (\pytest tests/test_run_logging.py tests/test_logging.py).
  • Formatted and verified clean with
    uff check\ and
    uff format.

…ction (codeforstartups#300)

- Add RunLogRecord model with lossless serialization and recursive secret redaction
- Add run_context and node_context managers with contextvars for automatic correlation
- Support pluggable sinks: StdoutSink, InMemoryLogSink, and DynamoDBLogSink
- Enhance secret redaction for API tokens, bearer tokens, private keys, and exception traces while preserving AI token count metrics
- Add comprehensive unit test suite in tests/test_run_logging.py
@codeforstartups

Copy link
Copy Markdown
Owner

Reviewed — strong dynalogs foundation (closes #300). 🎯

RunLogger with pluggable sinks (Stdout / InMemory / DynamoDBLogSink), run_context/node_context contextvar-based correlation (run_id + node_id + span_id), and genuinely careful secret redaction: redacts by key name (secret/password/token/key_id/access_key/bearer/…) and value patterns (AWS secrets, Bearer …, key=value), recursively across dicts/lists/sets — while deliberately preserving token-count metrics (tokens, *_tokens, token_count) so LLM usage numbers survive. That nuance is exactly right.

All 4 CI checks green incl. strict mypy. This is the first dynalogs pillar piece to land — nicely done @shivamm-gupta! Natural next steps on top of this: #299 (span tree) and #301 (run history). Merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Structured logs per-run / per-node with secret redaction

2 participants