[zhenhuaw-me]

Description

When LLM() or trtllm-serve fails to start, the cause usually lives in an executor worker and users only see RuntimeError: Executor worker returned error, or MPI worker exited unexpectedly when a non-leader rank fails. This PR makes startup failures self-explanatory and reports their category and restart loops in usage telemetry. Nothing changes on the success path and no defaults change.

• Root cause in the final error. Workers classify their init failure where the traceback is still available and send the classification with the rank and the failing stage, taken from the existing _StartupTimer / _WarmupTimer phases. Non-leader ranks now raise a string-only error, so the proxy sees it through the MPI future, or through the trtllm-llmapi-launch forwarder. Example: Executor worker returned error on rank 1 during executor_creation/model: [resource] RuntimeError: CUDA Runtime API error: <cudaError_t.cudaErrorMemoryAllocation: 2>.
• Coarse categories. Six categories say what went wrong: config_input, resource, unsupported, model_load, runtime_init (bringing up the runtime failed: worker death during init, distributed setup, kernel compilation) and other (not attributable; internal-looking errors such as an AttributeError raised inside TRT-LLM or an unknown CUDA error code land here with a "please report it" hint). The categories, their exception classes and the standard hints live in the private, stdlib-only tensorrt_llm/_errors.py, which is not tied to startup and is not exported. The hint key doubles as the error's reason in code; nothing is reported.
• Raise sites declare their category. Each category has one class that derives only from its builtin: ConfigInputError and UnsupportedError are ValueErrors; ResourceError, ModelLoadError and RuntimeInitError are RuntimeErrors, so callers can always catch them with except ValueError / except RuntimeError. They share code through a private mixin that is not an exception type, so there is no new base to catch. The category travels with the type; messages carry no marker. Any other exception class can declare a category with a category class attribute. Raise sites use the class even where that changes the builtin of an existing raise. About 50 startup raise sites are converted, among them: worker death detected by the proxy, a rank that sees no GPU (was a ZeroDivisionError), conflicting checkpoint quantization configs, missing weight files, unknown architectures, MoE backend resolution, KV cache budget, SM gates, model-loader config validators, YAML files whose root is not a mapping, and the trtllm-bench memory estimate. QuantAlgo raises UnsupportedError for a name it does not know, e.g. a checkpoint quantized with a newer format; it is still a ValueError with the same message, and pydantic still reports an invalid config value as a validation error.
• No catch-and-re-raise for classification. Errors that TRT-LLM does not raise pass through unchanged, and the classifier recognizes them by type and raising module: a FileNotFoundError raised in CLI code (trtllm-serve, including disaggregated -c, trtllm-bench, trtllm-eval) is config_input with the config hint, while missing weights stay model_load. The only text marker is in the worker error that trtllm-llmapi-launch forwards as text, which carries the worker's category as [<category>] ....
• Classification and the hint line, at startup boundaries only. tensorrt_llm/_startup_errors.py classifies a failure (declared category first, then exception type, errno and CUDA error code, then a fixed list of message phrases and the raising module) and logs one ERROR line with an actionable hint at the LLM() boundary, in the proxy before an abort that can end the process (MPI_Abort), and at the CLI boundary. Once an LLM has initialized in the process, the CLI boundary no longer classifies: a later crash keeps just its traceback, while another LLM() that fails to start is still classified at its own boundary.
• Cheap preflight checks. tensorrt_llm/llmapi/startup_checks.py runs before workers load weights and fails on:
  • no visible GPU, reporting CUDA_VISIBLE_DEVICES, the NVML GPU count and driver CUDA version, and the likely causes (multi-GPU LLM(), trtllm-serve, and every worker rank);
  • fewer visible GPUs than TP×PP×CP when a multi-GPU LLM spawns its own MPI pool on this node, in the existing multi-GPU check;
  • a GPU below the lowest architecture listed for the weight quantization format in the docs' hardware support matrix, except that FP8 block scales, FP8 per-channel per-token and Marlin NVFP4 are allowed on SM89+, where their GEMMs run (KV cache formats are not checked: their support depends on the attention kernel);
  • an unregistered model architecture, resolved through the same provider lookup the worker uses (is_model_arch_known in modeling_utils.py, which trtllm-serve's diffusion routing now uses as well).
• Behavior changes worth a look.
  • An unregistered architecture is now rejected in the main process with ValueError instead of inside the worker (test_modeling_out_of_tree updated).
  • Some converted raise sites change builtin; no code in the repo catches the old one. RuntimeError → ValueError: too few MPI ranks for the executor (ConfigInputError), and the SM gates of DeepSeek-V4, MiniMax-M3 MSA, NVFP4 cold-page quantization, TriAttention and the DeepGEMM path of the Cutlass MoE (UnsupportedError). ValueError → RuntimeError: a pre-quantized checkpoint without quant_algo (ModelLoadError), a KV cache budget exhausted by the DeepSeek-V4 indexer workspace or by the draft's fixed cost, and trtllm-serve without a visible GPU (ResourceError). TypeError → ValueError: a trtllm-bench extra_llm_api_options file whose root is not a mapping (ConfigInputError).
  • get_model_architecture raises ConfigInputError (Gemma4 needs a newer transformers) and UnsupportedError (unknown architecture) instead of RuntimeError; its two fallbacks, in inputs/registry.py and modeling_dflash.py, catch these two classes as well, so they fall back exactly as before.
  • The final Executor worker returned error ... line carries the worker's category before the root cause.
  • The GPU-count check now counts every rank of an LLM-spawned pool on this node. It practically never fired before, because it compared against a per-node share derived from gpus_per_node, which defaults to the local GPU count.
  • Hints are local log lines; telemetry reports only the category, see below.
• Usage telemetry. Today the exit report carries no cause, and every process gets a new random session ID, so neither the cause of a failed start nor restart loops can be seen. The event schema goes to 0.8, which is promoted to production; events from a build with this PR are stored there with the new fields:
  • errorCategory on the exit report: the category above for a session that ended during startup with an exception (or an LLM API exit after a failed LLM()), recorded at the LLM() failure handler and at the CLI boundary; "" otherwise, and for child failures seen by a supervisor.
  • priorStartupFailures and startupStateStatus on the exit and initial reports: how many times in a row this launch had failed to start, bucketed (0, 1, 2, 3, 10, 30, 100, 300, 1000), from a small local file per launch (~/.cache/tensorrt_llm/usage/, or TRTLLM_USAGE_STATE_DIR). The file name hashes inputs that stay the same when a launch restarts: the TRT-LLM version, command line, GPU assignment, launcher rank and process role (not the Slurm job ID or the host name, which can change with every submission), so copies of one launch on hosts that share a home directory share one count; none of these is sent. A failed start adds one, an initialized LLM or serving resets it, signals leave it. Only the reporting process touches the file, at most once, never when telemetry is disabled; an opt-out that arrives while the process runs (e.g. a later TelemetryConfig(disabled=True)) stops any file access that has not started. I/O errors only set the status to unavailable.
  • The file I/O is bounded on both paths: it runs on a short-lived daemon thread, like the existing 50 ms NVML lookup, including any wait for another thread's I/O on the file, and both the exit report and the background reporter wait for it at most 100 ms (the exit report within its 0.5 s bound). If it has not finished, e.g. on a stalled file system, the report goes out with priorStartupFailures=0 and startupStateStatus="unavailable", so a stalled file system holds up neither shutdown nor the initial report and heartbeats; the exit report keeps its errorCategory, and a write that finishes later is still atomic.
  • VisualGen sessions report neither field. The usage notice, the README telemetry section and the schema README describe the fields and the file; _errors.py notes that the category names are a data contract.

Test Coverage

• New CPU tests: tests/unittest/others/test_startup_errors.py (categories, reasons, the category classes, a missing file opened by CLI code versus elsewhere, an unknown QuantAlgo, the CLI boundary skipping failures after an LLM initialized), tests/unittest/llmapi/test_startup_checks.py and tests/unittest/inputs/test_input_processor_fallback.py (an unregistered or Gemma4-without-transformers architecture still gets the DefaultInputProcessor), all in l0_cpu.
• tests/unittest/executor/test_proxy_fast_death.py covers the worker classification carried to the proxy, non-leader errors, and the hint logged before the abort.
• Updated: test_modeling_out_of_tree.py for the new failure point, and four KV cache tests for the new builtin of their raise site (test_indexer_workspace_reserve, test_kv_cache_budget_split, test_quantization_for_cold_page, test_kv_cache_compression_manager).
• Telemetry: new tests/unittest/usage/test_startup_state.py (buckets, launch key, file rules, corrupt and unusable state, pruning) and new cases in test_reporter.py (exit rules, restart loop, success reset, partial report, VisualGen, opt-outs touching no file, also a late opt-out in the reporter and on the exit path, a blocked state file and a held state lock within the exit bound, and the initial report not waiting for them), test_cli_telemetry.py (CLI usage errors, SystemExit causes, precedence of the LLM() category, a missing --config file in each CLI, an explicit VisualGen trtllm-serve that fails an option check in serve() or in Click's own parsing reporting no startup fields), test_schema.py (enums match the classifier and the buckets), test_llm_telemetry.py and test_e2e_capture.py. TestStagingEndpoint (--run-staging only) adds an exit report with every errorCategory and startupStateStatus value and a partial initial report.
• Staging, with 0.8 enabled on stage: events built with this PR's schema.build_gxt_payload (exit reports with every errorCategory and startupStateStatus value and priorStartupFailures up to 100, a heartbeat, a partial initial report sent with its exit report) were accepted by gfestage, and each one is in PDX-UAT:stage-events-trtllm with the values sent. The stage gateway does not validate against 0.8 yet; the rollout status is in the PR conversation.
• On GB200, after rebasing onto main a545268cf3 (one import conflict in inputs/registry.py with [TRTLLM-16725][refactor] BREAKING: Use a string enum for checkpoint formats and simplify checkpoint I/O helpers #20703) and building the C++ from source: test_startup_checks 43 passed (FP8 block scales and per-channel per-token accepted on SM89 and rejected on SM80, an externally declared architecture accepted); CPU-stage emulation of test_startup_checks, test_lazy_model_zoo, test_startup_errors and usage: 718 passed; test_llm_args, test_lazy_model_zoo, test_serve_diffusion_routing and test_startup_checks: 648 passed; GPU stage (test_modeling_out_of_tree, test_llm_telemetry, test_proxy_fast_death): 79 passed.
• Earlier, on GB200, on this branch rebased onto main a7e88dee4a (since rebased onto aea80a0ec8 without conflicts, which picks up main's fix for the disaggregated context-first BF16 tests and its waiver of the MBART tokenizer test), with the C++ built from source (the newest precompiled wheel is 7 commits behind, with C++ changes since):
  • test_startup_checks (41 passed) and test_cli_telemetry (72 passed) cover the architecture check (model_kwargs["architectures"] when set, no check with a custom checkpoint_loader) and the VisualGen flag detection before Click parses the options. The unregistered-architecture launch gives the same category, hint and final line as before, and the same checkpoint with model_kwargs={"architectures": ["LlamaForCausalLM"]} starts.
  • main's test_vision_gpu_normalize_reaches_input_processor drives _build_model with a stub args, so it now also stubs the two startup checks _build_model runs; test_llm_args in the CPU-stage emulation: 463 passed.
  • CPU-stage emulation (-m cpu_only, no visible GPU), including tests/unittest/inputs, test_hf_quant_config, quantization/test_mode and api_stability: 1768 passed, 1 skipped, among them the classification of main's "not included in this TensorRT-LLM build" architecture error as unsupported; the new timing-dependent telemetry tests passed 30 of 30 repeats. GPU stage (test_proxy_fast_death, test_llm_telemetry, test_modeling_out_of_tree): 79 passed. test_llm_telemetry and test_disagg_telemetry_launcher with GPUs: 52 passed. Telemetry capture tests (test_e2e_capture, test_transport): 15 passed, 5 staging-only skipped; the staging cases pass with each POST validated against the JSON schema locally instead of sent.
  • Tests of the converted modules (test_llm_args, test_postprocessor_hook, test_indexer_workspace_reserve, test_kv_cache_compression_manager): 578 passed; tests/unittest/inputs with the get_model_architecture callers (test_lilicorr, the Gemma4 wrapper test): 383 passed, 1 skipped; config loaders (test_disagg_utils, test_disagg_telemetry_launcher, test_image_generation_eval_cli): 69 passed; main's test_cuda_architecture_check: 5 passed; main's validate_fp4_mla_config tests: 6 passed (its FP4 MLA checks raise UnsupportedError, and ConfigInputError for use_kv_cache_manager_v2=False); main's modeling_v2 routing tests, which resolve through AutoModelForCausalLM: 336 passed, 70 skipped.
  • 17 failing trtllm-serve / LLM() launches: HF 401 and offline, config typos, a missing --config file for trtllm-serve and for trtllm-serve disaggregated -c, no visible GPU, TP > GPUs, port in use (caught by main's multi-frontend port reservation), rank-1 OOM through the MPI pool and through trtllm-llmapi-launch, unregistered architecture, truncated weights, killed worker, and a failing custom module. All give the expected category, hint and final line, unchanged from before the rebase.
  • Telemetry, captured locally (nothing sent): the exit reports of 16 of the 17 launches above carry their category, 0 and new; the killed-worker launch sends no report because the process ends in MPI_Abort, as before this PR. A trtllm-serve loop on a busy port reports 0, 1, 2, 3, 3; once the port is free, the initial report carries 3 and the count resets. An LLM API loop on truncated weights behaves the same way. Two instances on different GPUs keep separate counts, and the same launch on two simulated hosts sharing a state directory continues one count. Opt-out creates no file, an unusable state directory reports unavailable, and VisualGen reports "" and creates no file, also when --enable_visual_gen fails an early option check, in serve() or in Click's parsing (--port invalid), twice in a row (the same launch without the flag counts 0, then 1). A state file that blocks on open (a FIFO) still lets failing and successful starts send their reports with unavailable. trtllm-serve starts 8 frontends by default ([TRTLLM-14510][feat] BREAKING: serve: default num_serve_frontends to 8 with single-frontend fallback #19379), and each attached frontend reports as its own session under its own launch key (TLLM_EXECUTOR_FRONTEND_ID). A successful default start therefore sends 8 initial and exit reports and keeps 8 state files: the launcher's reports carry the count and the 7 frontends' report 0. Failed starts end before the frontends spawn and are counted once; with --num_serve_frontends 1 the values equal those before the rebase. The state file costs about 3 to 9 ms on an NFS home depending on the node, on either path (0.7 ms on local disk for the exit path); when the I/O blocks, report_exit returns after 101 ms and the reporter moves on after 100 ms.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Dev Engineer Review

• The change adds six startup-error categories, typed exceptions, stage tracking, and classification across exception chains. Worker failures now carry category, rank, stage, and root-cause details through proxy initialization.
• Startup checks validate GPU visibility and count, quantization compute capability, and model-architecture registration. Several existing raise sites now use different exception types. Callers that catch specific types may need compatibility checks.
• Telemetry schema changes from 0.7 to 0.8. Reports add startup-error categories and prior-failure state. A local launch-specific counter uses bounded state access; VisualGen sessions omit startup-state reporting.
• Downstream consumers must accept schema 0.8 before deployment. The supplied information does not establish production promotion or current-head check results. Review findings and severity counts are not available.

QA Engineer Review

• Tests add coverage for startup classification, worker error propagation, startup preflight checks, telemetry schema and reporting, and persistent failure state. Other tests update expected exception types and telemetry payloads.
• tests/integration/test_lists/test-db/l0_cpu.yml adds unittest/llmapi/test_startup_checks.py. Other changed tests with list references appear in l0_a10.yml, l0_cpu.yml, or l0_h100.yml. No changed test was found in the manual-QA lists. The remaining changed tests have no literal list reference in the inspected lists.
• Coverage verdict: needs follow-up. Several changed tests have no inspected CI-list entry. Current-head test results and production telemetry verification are not available.

Per-File QA Perspective

Documentation and source files

• README.md — Verify that the documented categories, local failure counts, storage paths, and opt-out behavior match the implementation.
• tensorrt_llm/_errors.py — Verify exception inheritance and category, reason, and hint defaults. Check compatibility for callers that catch prior exception types.
• tensorrt_llm/_startup.py — Verify failed timers and nested phases record the intended stage, including for BaseException.
• tensorrt_llm/_startup_errors.py — Verify exception-chain classification, worker metadata propagation, summary limits, and preservation of original errors if classification or logging fails.
• tensorrt_llm/_torch/attention/backends/sparse/deepseek_v4/module.py — Verify below-SM90 devices raise UnsupportedError and callers handle it.
• tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_availability.py — Verify unsupported GPU architectures raise UnsupportedError.
• tensorrt_llm/_torch/custom_ops/torch_custom_ops.py — Verify only the DeepGEMM-on-non-Hopper failure is recategorized and its cause remains chained.
• tensorrt_llm/_torch/model_config.py — Verify conflicting KV-cache quantization settings raise ModelLoadError.
• tensorrt_llm/_torch/models/checkpoints/hf/weight_loader.py — Verify missing usable checkpoint files raise ModelLoadError.
• tensorrt_llm/_torch/models/checkpoints/mistral/config_loader.py — Verify unknown quantization formats raise UnsupportedError.
• tensorrt_llm/_torch/models/modeling_auto.py — Verify unknown model architectures raise UnsupportedError with the intended messages.
• tensorrt_llm/_torch/models/modeling_dflash.py — Verify typed exceptions trigger the intended architecture fallback.
• tensorrt_llm/_torch/models/modeling_utils.py — Verify error types for missing architectures, unsupported Gemma4 architectures, and other unknown architectures.
• tensorrt_llm/_torch/moe/fused_moe/moe_resolution.py — Verify unavailable or disallowed degraded MoE implementations raise UnsupportedError.
• tensorrt_llm/_torch/pyexecutor/_util.py — Verify budget failures raise ResourceError and unsupported-device paths raise UnsupportedError.
• tensorrt_llm/_torch/pyexecutor/model_loader.py — Verify affected configuration failures use the intended exception classes.
• tensorrt_llm/_torch/pyexecutor/warmup_timer.py — Verify warmup failures record their phase before re-raising, including non-Exception failures.
• tensorrt_llm/bench/benchmark/utils/general.py — Verify non-mapping YAML values raise ConfigInputError.
• tensorrt_llm/bench/benchmark/visual_gen.py — Verify VisualGen is marked before telemetry records startup state.
• tensorrt_llm/bench/tuning/heuristics.py — Verify insufficient engine or KV-cache capacity raises ResourceError with actionable guidance.
• tensorrt_llm/commands/_telemetry.py — Verify startup categories are reported only before successful LLM initialization and terminal outcomes retain prior behavior.
• tensorrt_llm/commands/eval.py — Verify invalid YAML roots raise ConfigInputError and None remains accepted.
• tensorrt_llm/commands/serve.py — Verify GPU detection, VisualGen selection, and YAML and gRPC validation paths.
• tensorrt_llm/executor/base_worker.py — Verify no-visible-GPU detection runs before device-ID calculation and valid device selection remains unchanged.
• tensorrt_llm/executor/executor.py — Verify insufficient MPI ranks produce parallel-size guidance and valid launches remain unchanged.
• tensorrt_llm/executor/postprocessor_hook.py — Verify hook import, construction, and callability failures raise ConfigInputError.
• tensorrt_llm/executor/proxy.py — Verify classified worker failures reach ordered teardown with rank and root-cause details. Check that unclassified worker deaths retain their existing path.
• tensorrt_llm/executor/worker.py — Verify worker classification and leader status reporting preserve errors across ranks and process boundaries.
• tensorrt_llm/inputs/registry.py — Verify typed architecture errors still select DefaultInputProcessor where intended.
• tensorrt_llm/llmapi/llm.py — Verify initialization-failure reporting, success marking, GPU preflight, and architecture and quantization checks.
• tensorrt_llm/llmapi/llm_args.py — Verify invalid YAML roots raise ConfigInputError.
• tensorrt_llm/llmapi/llm_utils.py — Verify quantization conflicts and unsupported formats use the intended exception types and hints.
• tensorrt_llm/llmapi/startup_checks.py — Verify GPU-count requirements for local and distributed launches, architecture registration, and quantization checks, including skip conditions.
• tensorrt_llm/models/quant_config_utils.py — Verify conflicting FP8 KV-cache configuration raises ConfigInputError.
• tensorrt_llm/quantization/mode.py — Verify unknown enum values raise UnsupportedError with supported-format guidance.
• tensorrt_llm/usage/__init__.py — Verify record_startup_error_category is exposed through the package API.
• tensorrt_llm/usage/schema.py — Verify schema 0.8 models enforce the new field types and defaults.
• tensorrt_llm/usage/schemas/README.md — Verify documented fields, state lifecycle, environment variable, and VisualGen behavior match the implementation.
• tensorrt_llm/usage/schemas/trtllm_usage_event_schema.json — Verify the JSON schema requires and constrains the new fields. Confirm downstream consumers accept schema 0.8.
• tensorrt_llm/usage/startup_state.py — Verify launch-key stability, path precedence, expiry, file permissions, atomic writes, pruning, and storage-error fallbacks.
• tensorrt_llm/usage/usage_lib.py — Verify category precedence, startup-state actions, timeouts, opt-out behavior, and VisualGen exclusions in initial and terminal reports.

Test and test-list files

• tests/integration/test_lists/test-db/l0_cpu.yml — Adds unittest/llmapi/test_startup_checks.py. Verify that the entry resolves to the intended test.
• tests/unittest/_torch/kv_cache_compression/test_quantization_for_cold_page.py — Updates the expected exception for unsupported cold-page hardware. No inspected CI or manual-QA list reference was found.
• tests/unittest/_torch/modeling/test_modeling_out_of_tree.py — Changes the unknown-architecture expectation and removes cause inspection. No inspected list reference was found.
• tests/unittest/_torch/pyexecutor/kv_cache/test_kv_cache_budget_split.py — Updates expected budget-failure exception types. No inspected list reference was found.
• tests/unittest/_torch/pyexecutor/kv_cache/test_kv_cache_compression_manager.py — Updates the TriAttention compatibility exception expectation. Referenced in test-db/l0_a10.yml.
• tests/unittest/_torch/pyexecutor/test_indexer_workspace_reserve.py — Updates the expected workspace-budget exception type. No inspected list reference was found.
• tests/unittest/executor/test_proxy_fast_death.py — Adds coverage for classified non-leader failures and propagation of rank, stage, root cause, and classification. Referenced in test-db/l0_a10.yml.
• tests/unittest/inputs/test_input_processor_fallback.py — Adds fallback coverage for unregistered architectures and typed errors. No inspected list reference was found.
• tests/unittest/llmapi/test_llm_args.py — Updates startup-check mocks while testing vision_gpu_normalize forwarding. Referenced in test-db/l0_cpu.yml and test-db/l0_a10.yml.
• tests/unittest/llmapi/test_llm_telemetry.py — Tests initialization-failure categories and category clearing after successful initialization. Referenced in test-db/l0_cpu.yml, test-db/l0_a10.yml, and test-db/l0_h100.yml.
• tests/unittest/llmapi/test_startup_checks.py — Adds CPU-only coverage for GPU visibility and count, quantization capability, and architecture registration. Referenced in test-db/l0_cpu.yml.
• tests/unittest/others/test_startup_errors.py — Adds classification, exception-chain, metadata, worker-wrapping, logging, and CLI-boundary coverage. No inspected list reference was found.
• tests/unittest/usage/conftest.py — Adds per-test startup-state directory isolation. Verify that the fixture does not interfere with tests that set an explicit directory. No inspected list reference was found.
• tests/unittest/usage/test_cli_telemetry.py — Updates schema expectations and adds CLI category, cause, exit, and restart-count coverage. No inspected list reference was found.
• tests/unittest/usage/test_e2e_capture.py — Updates end-to-end payload expectations for schema 0.8 and startup-state fields. Referenced in test-db/l0_a10.yml and test-db/l0_h100.yml.
• tests/unittest/usage/test_reporter.py — Adds coverage for failure-count actions, category reporting, VisualGen and opt-out exclusions, and bounded state access. No inspected list reference was found.
• tests/unittest/usage/test_schema.py — Adds model and JSON-schema consistency checks for category, count, and status values. No inspected list reference was found.
• tests/unittest/usage/test_startup_state.py — Adds coverage for bucketing, launch keys, storage selection, state actions, expiry, permissions, failures, and pruning. No inspected list reference was found.
• tests/unittest/usage/test_transport.py — Adds opt-in staging submissions for startup exit and partial initial reports. Referenced in test-db/l0_a10.yml and test-db/l0_h100.yml; source material does not establish current-head staging or gateway results.

[Mgluhovskoi]

Could you upload schema 0.8 to SMS, enable it in staging, and test representative events? Please verify the new fields in Kibana Discover using the PDX-UAT:stage-events-trtllm data view, and check for parser/gateway errors before promoting the schema to production. After promotion, confirm production events arrive correctly too using PDX:events-trtllm.

[zhenhuaw-me]

/bot run

[tensorrt-cicd]

PR_Github #82924 [ run ] completed with state FAILURE. Commit: 472ae0b
/LLM/main/L0_MergeRequest_PR pipeline #68486 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[zhenhuaw-me]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #83064 [ run ] triggered by Bot. Commit: 201b460 Link to invocation

[tensorrt-cicd]

PR_Github #83064 [ run ] completed with state SUCCESS. Commit: 201b460
/LLM/main/L0_MergeRequest_PR pipeline #68618 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[zhenhuaw-me]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #83404 [ run ] triggered by Bot. Commit: 710b431 Link to invocation

[tensorrt-cicd]

PR_Github #83404 [ run ] completed with state SUCCESS. Commit: 710b431
/LLM/main/L0_MergeRequest_PR pipeline #68936 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[zhenhuaw-me]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #83459 [ run ] triggered by Bot. Commit: 6630b10 Link to invocation

[tensorrt-cicd]

PR_Github #83459 [ run ] completed with state SUCCESS. Commit: 6630b10
/LLM/main/L0_MergeRequest_PR pipeline #68986 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[zhenhuaw-me]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #83604 [ run ] triggered by Bot. Commit: 6630b10 Link to invocation

[tensorrt-cicd]

PR_Github #83604 [ run ] completed with state SUCCESS. Commit: 6630b10
/LLM/main/L0_MergeRequest_PR pipeline #69114 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[zhenhuaw-me]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #83651 [ run ] triggered by Bot. Commit: 6630b10 Link to invocation

[tensorrt-cicd]

PR_Github #83651 [ run ] completed with state SUCCESS. Commit: 6630b10
/LLM/main/L0_MergeRequest_PR pipeline #69162 completed with status: 'UNSTABLE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[zhenhuaw-me]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #83679 [ run ] triggered by Bot. Commit: 6630b10 Link to invocation

[tensorrt-cicd]

PR_Github #83679 [ run ] completed with state SUCCESS. Commit: 6630b10
/LLM/main/L0_MergeRequest_PR pipeline #69188 completed with status: 'SUCCESS'

CI Report

Link to invocation