feat(build): add glibc-static supervisor libc variant (#2682)

The supervisor binary runs inside sandbox images whose libc and glibc
version are unknown at build time, so it must be statically linked. Add
SUPERVISOR_LIBC to select between the default musl variant and a new
glibc-static variant that builds the GNU target with +crt-static.

glibc-static has no cross-compile path: zig cc accepts -static for
*-linux-gnu targets and emits a dynamically linked binary anyway. The
staging script therefore refuses a cross-arch request for that variant
rather than silently degrading linkage, and requires a native
per-architecture build.

Add verify-static-binary.sh, run after every supervisor build in both the
staging script and CI so linkage cannot regress unnoticed for either
variant. It inspects via readelf (or greadelf/llvm-readelf) and fails closed
rather than trusting the tool's exit status: every inspection must produce no
diagnostics, the input must be an executable ELF (ET_EXEC, or ET_DYN with
DF_1_PIE) whose PT_LOAD segments all lie within the file, whose dynamic table
agrees with PT_DYNAMIC, and which carries no PT_INTERP and no DT_NEEDED. That
rejects a dynamically linked, truncated, corrupt, non-ELF, or shared-object
input that naive parsing would misread as static. Hosts without any inspector
(e.g. macOS, which ships no binutils) skip with a warning; Linux, including
CI, requires one and fails closed.

No image or release workflow builds the glibc-static variant, so add a
dedicated supervisor-static-validate workflow that builds it on both
architectures and runs the verifier. rust-native-build.yml uses self-hosted
runners, which reject pull_request-triggered jobs, so it validates in the merge
queue and on pushes to main that touch the build inputs, plus a nightly
schedule, so the GNU + crt-static build branch cannot regress unnoticed.

The default is unchanged, so image, release, and CI behavior is identical.
Selecting glibc-static statically links LGPL glibc into a redistributed
binary, which is why it is opt-in.

Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
Co-authored-by: Mrunal Patel <mrunalp@gmail.com>

Author: EmilienM

.github/workflows/rust-native-build.yml

@@ -5,10 +5,14 @@ name: Rust Image Binary Build (openshell-gateway / openshell-sandbox / openshell
 
 # Build Rust binaries per Linux architecture before the Docker image build
 # consumes them as prebuilt artifacts. Gateway images use GNU-linked binaries
-# for the NVIDIA distroless C/C++ runtime; supervisor and cli images use musl/static
+# for the NVIDIA distroless C/C++ runtime; supervisor and cli images use static
 # binaries so the final image can remain scratch. Gateway GNU binaries are
 # built with an explicit glibc 2.28 floor so image, package, and tarball
 # artifacts share the same host portability contract.
+#
+# The supervisor libc is selectable via the `supervisor-libc` input (musl or
+# glibc-static). Both variants are fully static and are verified as such,
+# because the supervisor is executed from inside arbitrary sandbox images.
 
 on:
   workflow_call:
@@ -21,6 +25,11 @@ on:
         description: "Linux architecture to build (amd64 or arm64)"
         required: true
         type: string
+      supervisor-libc:
+        description: "libc variant for the sandbox component (musl or glibc-static)"
+        required: false
+        type: string
+        default: "musl"
       cargo-version:
         description: "Pre-computed cargo version (skips internal git-based computation)"
         required: false
@@ -76,10 +85,12 @@ jobs:
       COMPONENT: ${{ inputs.component }}
       ARCH: ${{ inputs.arch }}
       FEATURES: ${{ inputs.features }}
+      SUPERVISOR_LIBC: ${{ inputs['supervisor-libc'] }}
       # Partition the GHA sccache cache per (component, arch). Without this,
       # concurrent jobs collide on the same cache key and later-starting
-      # writers hit 409 Conflict.
-      SCCACHE_GHA_VERSION: ${{ inputs.component }}-${{ inputs.arch }}
+      # writers hit 409 Conflict. The sandbox component also partitions per
+      # libc variant so musl and glibc-static builds do not evict each other.
+      SCCACHE_GHA_VERSION: ${{ inputs.component }}-${{ inputs.arch }}${{ inputs.component == 'sandbox' && format('-{0}', inputs['supervisor-libc']) || '' }}
     container:
       image: ghcr.io/nvidia/openshell/ci:latest
       credentials:
@@ -132,9 +143,28 @@ jobs:
               ;;
           esac
 
+          # The sandbox binary must stay fully static. musl gets there via the
+          # musl target; glibc-static uses the GNU target with +crt-static and
+          # relies on this job running natively on the target architecture,
+          # because zig cannot statically link glibc.
+          static_libc=musl
+          if [[ "$COMPONENT" == "sandbox" ]]; then
+            case "$SUPERVISOR_LIBC" in
+              musl) static_libc=musl ;;
+              glibc-static) static_libc=gnu ;;
+              *)
+                echo "unsupported supervisor-libc: $SUPERVISOR_LIBC (expected musl or glibc-static)" >&2
+                exit 1
+                ;;
+            esac
+          fi
+
           case "$ARCH" in
             amd64)
-              if [[ "$COMPONENT" == "sandbox" || "$COMPONENT" == "cli" ]]; then
+              if [[ "$COMPONENT" == "sandbox" && "$static_libc" == "gnu" ]]; then
+                target=x86_64-unknown-linux-gnu
+                zig_target=
+              elif [[ "$COMPONENT" == "sandbox" || "$COMPONENT" == "cli" ]]; then
                 target=x86_64-unknown-linux-musl
                 zig_target=x86_64-linux-musl
               else
@@ -143,7 +173,10 @@ jobs:
               fi
               ;;
             arm64)
-              if [[ "$COMPONENT" == "sandbox" || "$COMPONENT" == "cli" ]]; then
+              if [[ "$COMPONENT" == "sandbox" && "$static_libc" == "gnu" ]]; then
+                target=aarch64-unknown-linux-gnu
+                zig_target=
+              elif [[ "$COMPONENT" == "sandbox" || "$COMPONENT" == "cli" ]]; then
                 target=aarch64-unknown-linux-musl
                 zig_target=aarch64-linux-musl
               else
@@ -167,7 +200,7 @@ jobs:
       - name: Cache Rust target and registry
         uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2
         with:
-          shared-key: rust-native-${{ inputs.component }}-${{ inputs.arch }}-zig-wrapper-${{ hashFiles('tasks/scripts/setup-zig-cc-wrapper.sh') }}
+          shared-key: rust-native-${{ inputs.component }}-${{ inputs.arch }}${{ inputs.component == 'sandbox' && format('-{0}', inputs['supervisor-libc']) || '' }}-zig-wrapper-${{ hashFiles('tasks/scripts/setup-zig-cc-wrapper.sh') }}
           cache-directories: .cache/sccache
           cache-targets: "true"
 
@@ -239,6 +272,11 @@ jobs:
             cargo_cmd=(cargo zigbuild)
             build_target="${{ steps.target.outputs.zig_target }}"
             args+=(--features bundled-z3)
+          elif [[ "${{ inputs.component }}" == "sandbox" && "$SUPERVISOR_LIBC" == "glibc-static" ]]; then
+            # Static glibc requires the native toolchain's libc.a (build-essential
+            # in the CI image); cargo-zigbuild is not usable here because zig
+            # accepts -static for *-linux-gnu and links dynamically anyway.
+            export RUSTFLAGS="${RUSTFLAGS:-} -C target-feature=+crt-static"
           fi
           args+=(
             --release
@@ -276,6 +314,13 @@ jobs:
           BIN="target/${{ steps.target.outputs.target }}/release/${{ steps.target.outputs.binary }}"
           tasks/scripts/verify-glibc-symbols.sh 2.28 "$BIN"
 
+      - name: Verify static linkage
+        if: inputs.component == 'sandbox'
+        run: |
+          set -euo pipefail
+          BIN="target/${{ steps.target.outputs.target }}/release/${{ steps.target.outputs.binary }}"
+          tasks/scripts/verify-static-binary.sh "$BIN"
+
       - name: Stage binary for prebuilt layout
         run: |
           set -euo pipefail

.github/workflows/supervisor-static-validate.yml

@@ -0,0 +1,70 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+name: Supervisor Static Linkage Validation
+
+# The glibc-static supervisor variant (SUPERVISOR_LIBC=glibc-static) has no
+# other CI caller: docker-build.yml builds the default musl variant, so the
+# GNU + crt-static build branch and its native-only, per-arch requirements are
+# never exercised by image or release CI. Build the variant here on both
+# architectures so it cannot regress unnoticed. rust-native-build.yml runs
+# verify-static-binary.sh for the sandbox component, which fails the job on any
+# dynamic linkage.
+#
+# Linkage can change from a new/updated dependency or a source change, not just
+# from the build scripts, so the push path filters cover the workspace manifests
+# and crate sources in addition to the build tooling. A nightly schedule is the
+# unfiltered backstop for anything the filters miss.
+#
+# rust-native-build.yml runs on NVIDIA self-hosted runners, which reject jobs
+# triggered by `pull_request`. This workflow therefore follows the repo's
+# self-hosted convention (see branch-checks.yml / branch-e2e.yml): validate in
+# the merge queue (pre-merge), on push to main (post-merge), nightly, and on
+# demand — never on `pull_request`.
+
+on:
+  merge_group:
+    types: [checks_requested]
+  push:
+    branches: [main]
+    paths:
+      - "Cargo.toml"
+      - "Cargo.lock"
+      - "crates/**"
+      - "rust-toolchain.toml"
+      - "mise.toml"
+      - "mise.lock"
+      - ".cargo/config.toml"
+      - "tasks/scripts/stage-prebuilt-binaries.sh"
+      - "tasks/scripts/verify-static-binary.sh"
+      - ".github/workflows/rust-native-build.yml"
+      - ".github/workflows/supervisor-static-validate.yml"
+  schedule:
+    # Nightly (04:17 UTC) unfiltered run so a linkage regression cannot slip
+    # through the path filters unnoticed. Schedules run only on the default branch.
+    - cron: "17 4 * * *"
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+  packages: read
+
+jobs:
+  glibc-static:
+    name: glibc-static supervisor (${{ matrix.arch }})
+    strategy:
+      fail-fast: false
+      matrix:
+        arch: [amd64, arm64]
+    uses: ./.github/workflows/rust-native-build.yml
+    with:
+      component: sandbox
+      arch: ${{ matrix.arch }}
+      supervisor-libc: glibc-static
+      artifact-name: supervisor-glibc-static-${{ matrix.arch }}
+      retention-days: 1
+    secrets: inherit

architecture/build.md

@@ -68,6 +68,31 @@ The gateway bundles z3 into the release binary so Linux packages, standalone
 tarballs, and gateway images do not depend on distro-specific z3 shared-library
 SONAMEs.
 
+The supervisor is the one binary whose libc is selectable, because it is the one
+binary executed inside a userland OpenShell does not control. `SUPERVISOR_LIBC`
+chooses between `musl` (default) and `glibc-static`. Both produce a fully static
+binary; the choice does not change the runtime layout or the supervisor image base.
+Static linkage is a hard requirement rather than a preference, so both variants
+are verified by `tasks/scripts/verify-static-binary.sh`, which fails the build on
+any `PT_INTERP` or `DT_NEEDED` entry.
+
+The two variants differ only in build-time constraints:
+
+| | `musl` (default) | `glibc-static` |
+|---|---|---|
+| Cross-compiles | yes, via `cargo zigbuild` | no — must build natively per architecture |
+| Host requirement | zig + cargo-zigbuild | glibc static libraries (`glibc-static` on Fedora/RHEL, `libc6-dev` on Debian/Ubuntu) |
+| libc license | MIT | LGPL-2.1-or-later, statically linked |
+
+`cargo zigbuild` cannot produce the `glibc-static` variant: `zig cc` accepts
+`-static` for `*-linux-gnu` targets and emits a dynamically linked binary
+anyway. The staging script therefore refuses to cross-compile that variant
+instead of silently degrading linkage.
+
+Selecting `glibc-static` statically links LGPL glibc into a redistributed
+binary, which carries relinking obligations that musl (MIT) does not. Treat the
+default as the shipping configuration unless that has been reviewed.
+
 ## Container Builds
 
 The Docker image pipeline is a two-step flow: build the Rust binary natively
@@ -91,9 +116,11 @@ package-managed VM support does not raise the package runtime requirement.
 Gateway staging and release workflows set up the Zig C/C++ wrapper before
 bundled Z3 builds and verify the maximum referenced `GLIBC_*` symbol version
 before publishing or copying artifacts.
-Supervisor binaries remain static musl and use `cargo zigbuild` when available,
-including native CPU architectures, so C dependencies are compiled for the musl
-target instead of the host GNU libc target. Local Docker image tasks infer the
+Supervisor binaries are static in every configuration. The default `musl`
+variant uses `cargo zigbuild` when available, including native CPU
+architectures, so C dependencies are compiled for the musl target instead of the
+host GNU libc target. The `glibc-static` variant uses plain `cargo build` with
+`+crt-static` and requires a native per-architecture build. Local Docker image tasks infer the
 target architecture from `DOCKER_PLATFORM` when set. Otherwise, they require
 valid container engine host metadata and fail when the engine query is
 unavailable or reports an unsupported architecture, avoiding host-kernel
@@ -114,11 +141,15 @@ Runtime layout:
   as a release artifact. Linux GNU VM driver binaries must not reference
   `GLIBC_*` symbols newer than `GLIBC_2.28`; release workflows verify this
   before publishing artifacts.
-- **Supervisor**: Alpine base with `nftables`, static musl binary at
-  `/openshell-sandbox`. Static linkage keeps the binary usable when the image
-  is mounted/extracted into sandbox environments (Docker extraction, Podman
-  image volumes, Kubernetes init-container copy-self), while `nftables` supports
-  Kubernetes supervisor sidecar egress enforcement.
+- **Supervisor**: Alpine base with `nftables`, static binary at
+  `/openshell-sandbox` (musl by default; see `SUPERVISOR_LIBC` above). Static
+  linkage keeps the binary usable when the image is mounted/extracted into
+  sandbox environments (Docker extraction, Podman image volumes, Kubernetes
+  init-container copy-self), whose libc and glibc version are not known at build
+  time, while `nftables` supports Kubernetes supervisor sidecar egress
+  enforcement. The VM driver bundles its own supervisor build
+  (`tasks/scripts/vm/build-supervisor-bundle.sh`) and does not read
+  `SUPERVISOR_LIBC`.
 
 Gateway image builds bake the corresponding supervisor image tag into the
 gateway binary so Docker sandboxes do not depend on `:latest` by default.

deploy/docker/Dockerfile.supervisor

@@ -15,9 +15,12 @@
 #
 # Use tasks/scripts/docker-build-image.sh supervisor (or `mise run build:docker:supervisor`)
 # to stage the binary and build the image in one step. CI builds the binary
-# per-architecture via the `rust-native-build.yml` workflow (with the musl
-# target) and uploads it as an artifact, which is downloaded into the same
-# staging directory before the image build job runs.
+# per-architecture via the `rust-native-build.yml` workflow and uploads it as an
+# artifact, which is downloaded into the same staging directory before the image
+# build job runs.
+#
+# The binary is static under either supported libc variant (`SUPERVISOR_LIBC`:
+# musl by default, or glibc-static), so this Alpine base runs it unchanged.
 
 FROM alpine:3.22 AS supervisor