Skip to content

Fix: Use init container to populate /github/workflow directories #28

@rubionic

Description

@rubionic

Problem

When using cached-privileged-kubernetes mode, actions that require access to $GITHUB_EVENT_PATH (like Docker Buildx) fail because /github/workflow/event.json does not exist.

Error:

ERROR: failed to read GITHUB_EVENT_PATH "/github/workflow/event.json": open /github/workflow/event.json: no such file or directory

Root Cause

The GitHub Actions runner-container-hooks package has a bug where the prepare script (which copies GitHub workspace directories from /__w/_temp/ to /github/) only executes when userMountVolumes are defined.

Code location: packages/k8s/src/hooks/prepare-job.ts lines 101-104

let prepareScript: { containerPath: string; runnerPath: string } | undefined
if (args.container?.userMountVolumes?.length) {
    prepareScript = prepareJobScript(args.container.userMountVolumes || [])
}

The prepare script is responsible for:

cp -R /__w/_temp/_github_home /github/home
cp -R /__w/_temp/_github_workflow /github/workflow

Without this script running, the /github volume remains empty even though:

  • ✅ The /github EmptyDir volume is created by the k8s-novolume hooks
  • ✅ The /__w/_temp/_github_workflow/ directory exists with content
  • ❌ Content is never copied to /github/workflow/

Previous Attempt: Dummy Volume (#27)

PR #27 attempted to work around this by adding a dummy user mount volume to trigger the userMountVolumes.length check. However, this approach did not solve the problem - the error still occurs in the latest workflow run.

Proposed Solution: Init Container

Add an init container to the hook extension template that explicitly copies the GitHub workspace directories before the main container starts.

Implementation

In pkg/templates/templates/overlay.yaml, add an init container to the pod spec:

# After defining the main container spec
initContainers = [
  {
    "name": "prepare-github-workspace",
    "image": "${RUNNER_IMAGE}",  # Use the same runner image
    "command": ["/bin/sh", "-c"],
    "args": [
      """
      set -e
      echo "Preparing GitHub workspace directories..."
      cp -R /__w/_temp/_github_home /github/home 2>/dev/null || echo "No _github_home to copy"
      cp -R /__w/_temp/_github_workflow /github/workflow 2>/dev/null || echo "No _github_workflow to copy"
      echo "GitHub workspace preparation complete"
      """
    ],
    "volumeMounts": [
      {"name": "work", "mountPath": "/__w"},
      {"name": "github", "mountPath": "/github"}
    ]
  }
]

Why This Works

  1. Runs before the main container - Init containers execute sequentially before the main container starts
  2. Has access to both volumes - Can read from /__w/_temp/ and write to /github/
  3. Independent of hook logic - Doesn't rely on the buggy conditional prepare script
  4. Explicit and testable - Clear responsibility and easy to verify
  5. Graceful degradation - Uses || true to handle cases where source directories don't exist

Benefits Over Dummy Volume Approach

Aspect Dummy Volume (#27) Init Container (This Issue)
Reliability ❌ Didn't work ✅ Explicit copy operation
Clarity ⚠️ Exploits side effect ✅ Clear intent
Independence ❌ Relies on hook internals ✅ Self-contained solution
Debugging ❌ Hidden in hook execution ✅ Visible in init container logs
Maintenance ⚠️ Fragile workaround ✅ Standard Kubernetes pattern

Testing Plan

  1. Deploy updated deskrun with init container configuration
  2. Trigger workflow in instant-cf Optimize CI build performance using deskrun pattern with host nix/docker cache instant-cf#25
  3. Verify init container logs show successful copy:
    kubectl logs -n arc-systems <pod-name> -c prepare-github-workspace
  4. Check main container has /github/workflow/event.json:
    kubectl exec -n arc-systems <pod-name> -- ls -la /github/workflow/
  5. Confirm Docker Buildx step succeeds

Implementation Checklist

  • Add init container definition to overlay.yaml
  • Ensure volume mounts include both work and github volumes
  • Use proper image reference (RUNNER_IMAGE variable)
  • Add error handling for missing source directories
  • Update tests to include init container in expected output
  • Document the workaround and its purpose
  • Remove dummy volume from Fix: Add dummy volume to force k8s-novolume hook prepare script execution #27 once init container is verified

Related Issues

Future Work

Once this workaround is implemented and verified:

  1. File upstream bug report to actions/runner-container-hooks
  2. Monitor for upstream fix
  3. Remove init container workaround when fixed version is available

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions