safeoutputs tools/list can return 0 tools right after gateway reports healthy, with no retry — large dispatch-workflow configs make this likelier

[blozano-tt]

Summary

Mount MCP servers as CLIs (mount_mcp_as_cli.cjs) can hard-fail a run with
Error: safeoutputs tools/list returned 0 tools even though the MCP gateway's own
/health endpoint reported both backends "running" moments earlier, and even though
gateway.startupTimeout was correctly propagated. This looks like a race between the
gateway's binary "process is up" health signal and the safeoutputs backend's own
(apparently much slower) internal tool-schema construction — and a large
dispatch-workflow config (many allowed workflows, each with its own rich
workflow_dispatch input JSON schema) is a concrete, reproducible way to trigger it.

Evidence from a real failing run

tenstorrent/tt-metal, workflow "Silencer", run
https://github.com/tenstorrent/tt-metal/actions/runs/30729757440 (job agent,
step "Mount MCP servers as CLIs"):

[info] Health response body: {"status":"healthy","specVersion":"1.15.0","gatewayVersion":"v0.4.7","servers":{"github":{"status":"running","uptime":3},"safeoutputs":{"status":"running","uptime":3}}}
[info] Gateway is ready!
...
Mounting MCP server 'github' (host url: http://127.0.0.1:8080/mcp/github, ...)...
  Found 28 tool(s)
  ✓ Mounted as: /home/runner/work/_temp/gh-aw/mcp-cli/bin/github
Mounting MCP server 'safeoutputs' (host url: http://127.0.0.1:8080/mcp/safeoutputs, ...)...
Error: safeoutputs tools/list returned 0 tools. Failing fast — the live MCP gateway has no tools registered. Check the MCP gateway startup logs for ECONNRESET errors or delayed backend registration.
    at recoverSafeOutputsToolsIfNeeded (.../actions/mount_mcp_as_cli.cjs:118:9)
    at safeoutputs (.../actions/mount_mcp_as_cli.cjs:130:47)
##[error]Unhandled error: Error: safeoutputs tools/list returned 0 tools...

Timestamps: the gateway's own health check succeeded on the first attempt (0s
elapsed), gateway reported ready, github's tools/list returned 28 tools
successfully — and 190ms later, safeoutputs's tools/list on the same
already-healthy gateway returned an empty list. recoverSafeOutputsToolsIfNeeded
(the fallback mentioned in the error) had nothing to recover from — this was a fresh
runner with no prior cached tools.json, so the whole step failed fast and
unrecoverably.

This is the workflow's dispatch_workflow safe-output config in the run above (for
context on config size — 44 entries in workflows, several of which have
non-trivial workflow_dispatch input schemas of their own, e.g. enum lists for
target hardware/model choices that get embedded as tool input schema):

"dispatch_workflow": {
  "allowed_refs": ["silencer/*"],
  "max": 1,
  "workflow_files": { /* 44 entries */ },
  "workflows": [ /* 44 entries */ ]
}

The generated safeoutputs tool list itself (per the agent's own investigation
inside the run, reading the cached tools.json written earlier in the job) is 54
tools once fully built — larger than in workflows without dispatch-workflow or
with a small workflows allowlist.

What I think is happening

The gateway's /health endpoint apparently reports a backend "status":"running"
as soon as the backend process has started and accepted a connection — not once
it has finished building and registering its own tool schema internally. For a
backend with a small, fixed tool set this distinction doesn't matter in practice
(schema construction is instant). For safeoutputs with a large
dispatch-workflow config, building 44 workflows' worth of tool definitions
(including each target workflow's own workflow_dispatch input schema) appears to
take long enough that tools/list, queried by mount_mcp_as_cli.cjs immediately
after the overall gateway reports healthy, can still see zero registered tools.

I understand from investigating that gateway.startupTimeout was recently made
to always be emitted (sourced from tools.startup-timeout or the 120s default)
specifically to stop the gateway's own internal wait logic from being clamped to
a hardcoded 30s. That fix doesn't cover this gap: the gateway had already reported
healthy well inside that timeout (in ~5s), so startupTimeout never came into play
here — the failure is mount_mcp_as_cli.cjs's own single, unretried tools/list
call against a backend whose "running" status doesn't (yet) imply "finished
registering tools."

Suggested fix

In mount_mcp_as_cli.cjs, when tools/list returns empty for a backend the
gateway itself reports as "running", retry with a short backoff (a handful of
attempts over a few seconds) before falling back to recoverSafeOutputsToolsIfNeeded
and then failing fast — rather than treating the very first empty response as
final. This would need to be bounded (so a genuinely broken backend still fails
fast eventually), but a single immediate check with no retry seems too eager given
that backend readiness and gateway-level health are evidently not synchronized for
safeoutputs under a large config.

Workaround we're using in the meantime

Re-running the workflow — this looks like a timing race, not a deterministic
config error, so a retry has a reasonable chance of landing on the other side of
the schema-build window. Flagging here rather than assuming it'll resolve itself,
since a large dispatch-workflow config seems to make the window meaningfully
wider than the common case this code path was presumably tuned against.