Merge pull request #265 from NVIDIA/release/0.4

Forward-merge release/0.4 into main

Author: GPUtester

docs/observability-plugin/atif.mdx

@@ -174,8 +174,9 @@ local→remote transition keeps object names stable. A trailing `/` is added to
 
 If an upload fails for a given destination, that destination is recorded as
 unhealthy and skipped on later trajectories. The other destinations continue
-to receive writes. All recorded sink failures are joined into the dispatcher's
-last-error result on plugin teardown.
+to receive writes. Fatal dispatcher failures, such as trajectory serialization
+failures, stop later ATIF observation and are reported during plugin teardown;
+per-destination sink failures are isolated to the failed sink.
 
 ## Expected Output
 

docs/observability-plugin/configuration.mdx

@@ -125,6 +125,39 @@ Include only the sections you want to configure. In layered `plugins.toml`
 files, omission inherits lower-precedence values; write `enabled = false` to
 disable an inherited section.
 
+## Failure Behavior
+
+NeMo Relay treats exporter configuration and exporter delivery failures
+differently. Configuration and activation failures are fail-closed for the
+observability setup: validation or initialization returns an error before the
+new plugin configuration becomes active. Runtime delivery failures are
+fail-open for application work: the tool, LLM, or agent run continues while the
+affected exporter records, logs, or reports the delivery problem.
+
+| Failure | Behavior |
+|---|---|
+| Invalid `plugins.toml`, duplicate component kinds, malformed component shapes, unsupported values, unavailable exporter features, ATOF file-open failures, invalid ATOF endpoint config, unavailable ATOF streaming support, or ATOF endpoint worker startup failures | Validation or initialization fails. If a previous plugin configuration was active, NeMo Relay attempts to restore it after a failed replacement. |
+| ATOF event serialization or file write/flush failure after activation | Application work continues. The exporter stores the failure, stops accepting later events for that file, and returns the stored error from `force_flush()` or `shutdown()`. |
+| ATOF streaming endpoint connection or send failure after activation | File output and other already-started endpoints continue. Endpoint failures are logged with the endpoint index; endpoint flush and close timeouts are logged instead of blocking shutdown indefinitely. |
+| ATIF local file write, HTTP storage, or S3-compatible storage failure | Application work continues. The failed sink is recorded as unhealthy and skipped for later trajectories. Other configured sinks continue to receive writes. |
+| ATIF dispatcher serialization or subscriber-management failure | The ATIF dispatcher records a fatal exporter error and stops observing later ATIF events. Other observability sections continue to run. |
+| OpenTelemetry or OpenInference construction failure | Plugin initialization fails before the subscriber is registered. |
+| OpenTelemetry or OpenInference export failure after registration | Application work continues. The OTLP exporter reports failures through its runtime logging and flush or shutdown path. |
+
+Missing or delayed telemetry is represented as absence of exporter output, not
+as synthetic success or failure events. NeMo Relay does not backfill events for
+subscribers that register late. If the plugin is cleared while an agent scope
+is still open, the ATIF dispatcher writes the partial trajectory it has already
+observed.
+
+Use `nemo-relay doctor` to validate local ATOF and ATIF output directories,
+OTLP HTTP endpoints, and ATOF streaming endpoints. Validate ATIF remote storage
+destinations, including HTTP storage and S3-compatible storage, with exporter
+logs and backend-side checks because `nemo-relay doctor` does not probe
+`atif.storage`. For local artifact paths, verify that the running process can
+create the output directory and that teardown calls `plugin.clear()` or
+`clear_plugin_configuration()` before the process exits.
+
 ## Per-Language Plugin Configuration
 
 <Tabs>

integrations/coding-agents/README.md

@@ -163,6 +163,14 @@ enabled = true
 endpoint = "http://127.0.0.1:4318/v1/traces"
 ```
 
+During setup or launch, invalid shared TOML, malformed plugin config, unsupported exporter settings,
+or unavailable exporter features will fail closed. The
+wrapper does not start the coding agent against a configuration that cannot be
+parsed, validated, or activated. Once the gateway and agent are running,
+exporter delivery failures follow the observability plugin policy: application
+work continues while the failing ATOF, ATIF, OpenTelemetry, or OpenInference
+destination records, logs, or reports the failure.
+
 ## Hook Forwarding
 
 The transparent wrapper hooks call `nemo-relay hook-forward <agent>` with the
@@ -174,11 +182,15 @@ Claude Code and Codex plugin hooks call `nemo-relay plugin-shim hook <agent>`.
 The plugin shim ensures the local sidecar is reachable, then forwards the hook
 payload to the plugin sidecar endpoint.
 
-Since hook forwarding fails open by default, observability outages do not block the
-coding agent. For wrapper-generated `hook-forward` commands, add
+Since hook forwarding fails open by default, gateway or sidecar outages do not
+block the coding agent. The hook command exits successfully after logging the
+forwarding problem, so the host agent can continue even though that hook
+payload may be missing from telemetry. For wrapper-generated `hook-forward`
+commands, add
 `--fail-closed` when policy requires hook delivery to block the agent. For
 plugin shim hooks, set `NEMO_RELAY_FAIL_CLOSED=1` in the hook execution
-environment.
+environment. In that mode, forwarding failures return a non-zero hook command
+status to the host.
 
 Useful wrapper options: