Follow-up to #44249: threat-detection still no-ops on `arc-dind` pull step omits `build-tools`, so the sysroot-stage fails (`No such image`)

[github-antoine-brechon]

Hi @pelikhan , follow-up to #44249 . #44445 correctly propagated RunnerConfig to the detection job's execution step, resolving the spawn /usr/local/bin/copilot ENOENT. But the detection job builds a second, separate minimal WorkflowData in the AWF image pull step which still omits RunnerConfig. As a result the pull step doesn't fetch the build-tools image that the now-arc-dind sysroot-stage requires, so the firewall stack fails to start and detection still never runs. The two failure modes swapped (copilot ENOENT → No such image: build-tools); the net effect threat detection silently no-ops on arc-dind while reporting green is unchanged.

Summary

#44445 made the detection job arc-dind by propagating RunnerConfig into the WorkflowData that buildDetectionEngineExecutionStep constructs. Being arc-dind, the detection engine now runs the AWF sysroot-stage, which needs ghcr.io/github/gh-aw-firewall/build-tools:<tag>. But buildPullAWFContainersStep builds its own minimal WorkflowData for the pre-pull step and does not set RunnerConfig, so collectDockerImages (which only adds build-tools under isArcDindTopology) omits it. The detection job pre-pulls only squid, agent, api-proxy; then docker compose up -d --pull never fails with No such image: …/build-tools, the firewall never starts, and the detection engine never executes. As in #44249, continue-on-error: true swallows the failure and the job is green.

Environment

• gh-aw v0.82.7 (AWF v0.27.27, mcpg v0.4.1)
• Engine: GitHub Copilot CLI (engine: copilot)
• Runner: actions-runner-controller gha-runner-scale-set with a privileged dind sidecar (split filesystem), runner: topology: arc-dind
• safe-outputs.threat-detection enabled, inline detection engine (default; gh-aw-detection feature off)

Symptom

detection job is green, but the engine never runs:

[INFO] ARC/DinD: sysroot-stage will use image ghcr.io/github/gh-aw-firewall/build-tools:0.27.27@sha256:…
[INFO] Starting containers...
 Container awf-sysroot-stage  Error response from daemon: No such image: ghcr.io/github/gh-aw-firewall/build-tools:0.27.27@sha256:…
[ERROR] Failed to start containers: Command failed with exit code 1: docker compose up -d --pull never
##[error]Process completed with exit code 1.
…
📋 continue-on-error: true
📄 No lines containing THREAT_DETECTION_RESULT found in 83 lines
##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log.

The detection job's pre-pull step fetched only three images:

download_docker_images.sh \
  …/gh-aw-firewall/agent:0.27.27@… \
  …/gh-aw-firewall/api-proxy:0.27.27@… \
  …/gh-aw-firewall/squid:0.27.27@…
# no build-tools

The agent job (full WorkflowData, arc-dind) pre-pulls build-tools and its firewall starts fine the divergence is detection-only.

Root cause (source pkg/workflow)

Two separate minimal WorkflowData values are built for the detection job, and only one was fixed by #44445:

1. Execution step fixed by #44445. buildDetectionEngineExecutionStep (threat_detection_inline_engine.go) now sets RunnerConfig: data.RunnerConfig, so the engine runs arc-dind (chroot + daemon-visible copilot).

2. Pull step still broken. buildPullAWFContainersStep (threat_detection_external.go) builds its own detectionData and omits RunnerConfig:

   detectionData := &WorkflowData{
       Tools:         map[string]any{},
       AI:            engineSetting,
       SandboxConfig: &SandboxConfig{Agent: &AgentSandboxConfig{Type: SandboxTypeAWF}},
       ActionCache:   data.ActionCache,
       Features:      data.Features,
       // RunnerConfig NOT set  ← bug
   }
   images := collectDockerImages(detectionData.Tools, detectionData, c.actionMode)

collectDockerImages (docker.go) only adds build-tools under arc-dind:

// Add build-tools sysroot image for ARC/DinD topology.
if isArcDindTopology(workflowData) {
    buildToolsImage := constants.DefaultFirewallRegistry + "/build-tools:" + awfImageTag
    …
}

With RunnerConfig nil, isArcDindTopology(detectionData) is false, so build-tools is skipped from the pull list while the execution step (correctly arc-dind after #44445) then demands it via the sysroot-stage. The two views of the same job disagree.

Reproduction

1. Compile an engine: copilot workflow with runner: topology: arc-dind + safe-outputs.threat-detection (inline engine).
2. Run on an ARC + dind-sidecar runner. Agent succeeds; detection is green but its log shows No such image: …/build-tools, docker compose up -d --pull never exit 1, and No THREAT_DETECTION_RESULT found.

Proposed fix + implementation plan

1. pkg/workflow/threat_detection_external.go, buildPullAWFContainersStep propagate the topology into the pre-pull detectionData, mirroring fix: propagate RunnerConfig to detection job to fix arc-dind Copilot ENOENT #44445:

   detectionData := &WorkflowData{
       …
       RunnerConfig: data.RunnerConfig, // include build-tools sysroot image on arc-dind
   }

   Then collectDockerImages adds build-tools and docker compose up -d --pull never finds it.

2. Prevent recurrence (recommended). The detection job now constructs minimal WorkflowData in at least two places (buildDetectionEngineExecutionStep, buildPullAWFContainersStep) that must stay in sync this omission has now bitten twice. Extract a single buildDetectionWorkflowData(data) helper that both call, so topology (and future fields like ActionCache/Features) can't diverge.

3. Update the stale comment in buildPullAWFContainersStep ("three containers (squid, agent, api-proxy)") arc-dind adds a fourth (build-tools).

4. Tests (pkg/workflow/threat_detection_test.go): assert that with runner: topology: arc-dind, the detection pull step's image list includes …/build-tools:<tag>; and a regression assert that non-arc-dind detection still pulls only the three firewall images.

5. Surface the failure (carried over from Threat-detection job silently no-ops on arc-dind runners: Copilot binary not staged into the chroot (spawn /usr/local/bin/copilot ENOENT) #44249). A firewall-bring-up failure under continue-on-error: true still yields a green job and only a swallowed No THREAT_DETECTION_RESULT. Emitting a warning/annotation when the detection engine step exits non-zero would have made both this and Threat-detection job silently no-ops on arc-dind runners: Copilot binary not staged into the chroot (spawn /usr/local/bin/copilot ENOENT) #44249 visible instead of silently green.

6. Run make agent-finish (build, test, recompile, format, lint).