test(execd): close OSEP-0018 R-q/R-s/R-l(b)/R-m/R-u e2e gaps + ebpf smoke - #1562
Merged
Pangjiping merged 34 commits intoAug 19, 2026
Conversation
… R-q, R-s, R-j/R-c) R-q: isolation.custom.toml ([seccomp] deny chmod family + keep_capabilities CAP_NET_RAW) + TestHardeningCustomPolicyE2E — denied syscall EACCES in /command, CapEff/CapBnd=0x2000 via ambient raise, docker phase 3 (ceiling keeps NET_RAW) + k8s ConfigMap second key. R-s: TestHardeningDriftE2E — hardened TOML with execd_run_as_init=false reports init_mode=none + layers degraded with EXECD_INIT guidance while /command still runs the floor; docker phase 4. R-d: marked declined (server-config-driven surface, Python covers the shared path). R-j/R-c: scripts/execd-ebpf-smoke.sh — bare-container smoke running /execd-ebpf with CAP_BPF/CAP_PERFMON/CAP_SYSLOG + tracefs mount, generates exec/connect/privilege events via docker exec and asserts the JSONL audit file; prechecks BTF/cgroup-v2/tracefs so it doubles as the 5.10 empirical validation. Also: _hardening_report cache is per-sandbox (multiple classes share one k8s pytest invocation).
…M contract Decision: no out-of-band stop endpoint or credential channel. In-namespace SIGTERM keeps stopping the sandbox (same as pre-OSEP), kill -9 1 stays inert via the PID 1 shield (the property R2 is really about), and the K8s Restart recycle keeps DefaultRestartCommand=[kill,1]. External runtime SIGTERM still forwards for graceful shutdown. R2 downgraded to a documented non-goal; test comment synced.
R-l(b): tests/python/tests/test_execd_k8s_restart_recycle_e2e.py — Pool whose pod template runs execd as PID 1 (bootstrap.sh + EXECD_INIT=1 + EXECD=/execd + keepalive) with the Restart recycle strategy; releasing the BatchSandbox pod-execs kill 1, execd forwards SIGTERM and exits, kubelet restarts the container (restartCount increases), pod survives and execd is PID 1 again. Verifies the restart_default.go 'contract compatible' comment e2e. Wired into scripts/python-k8s-execd-init-e2e.sh (exports E2E_NAMESPACE/EXECD_IMG + runs the new file). R-m: TestHardeningDefaultOffE2E (hardening e2e, docker phase 5) — plain server, no isolation TOML, init off: endpoint reports init_mode=none, signal_shield=false, every layer disabled, workload unaffected (ceiling caps, Seccomp=0, NoNewPrivs=0). write_server_config gained an inject_toml switch. Plus test_sustained_fork_heavy_mix_keeps_process_table_ bounded: ~30s of interleaved /command churn + background sleepers, bounded zombie-free process table throughout. R-u: test_runtime_stop_forwards_sigterm_and_propagates_exit_code (init e2e, docker bridge): entrypoint traps TERM (marker + exit 7), container located via opensandbox.io/id label, docker stop -> sandbox Failed with 'exited with code 7' and marker present (docker cp from the stopped container).
Pangjiping
requested review from
Generalwin,
Spground,
fengcone,
hittyt,
jwx0925,
kevinlynx and
ninan-nn
as code owners
August 18, 2026 11:46
github-actions
Bot
added
component/execd
documentation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 938defb975
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
docker cp runs on the host where the variable is not expanded, so the literal $OPENSANDBOX_ID path never matched the file the container-side shell created. Use a fixed marker path (per-container layer, no collision).
…18 R-j/R-c) execd-test.yml smoke job gains a self-hosted matrix leg; the eBPF smoke step (scripts/execd-ebpf-smoke.sh --build, linux-only) runs on both ubuntu-latest (stock CI kernel) and self-hosted (the 5.10 empirical validation of the inline-filename exec-hook fallback). Host prerequisites (BTF, cgroup v2, tracefs) are prechecked by the script; container gets CAP_BPF/CAP_PERFMON/CAP_SYSLOG. detect-changes execd matcher now covers scripts/execd-ebpf-smoke.sh.
Keep a single OSEP-0018 mention in each file header as a topic marker; remove the per-test / per-step R-* tag noise from comments, docstrings, skip reasons and workflow step names.
Pangjiping
force-pushed
the
feat/execd-hardening-custom-policy-drift-e2e
branch
from
August 18, 2026 12:14
560ab6d to
94fe1bb
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 94fe1bb563
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
On 5.10 kernels the on_commit_creds kprobe fails CO-RE loading
('bad CO-RE relocation: invalid func unknown#...'), and LoadAndAssign
atomically loaded all three programs, so exec/connect hooks died with it
and the observer reported degraded with zero events. Load each program
individually (shared events ringbuf via MapReplacements), skip hooks the
kernel rejects, and keep auditing with the rest; the capabilities
endpoint stays active and lists the missing hooks in the message. Smoke
assertions unchanged (exec+connect events still mandatory).
github-actions
Bot
added
size/XL
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 94018117d3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The container (root) creates the JSONL audit file with root-only perms, so the non-root host runner got Permission denied when asserting on it. Pre- create the file with 0666 (lumberjack appends in place). Also exec an external /bin/sleep so sched_process_exec actually fires (shell builtins don't exec).
When the capabilities message reports 'hooks not active' (e.g. the commit_creds kprobe failing on a given kernel), dump the container's execd log so the exact load/attach error is captured in CI — previously the smoke only printed container logs on hard failure, so the ubuntu-latest leg's privilege-hook error was invisible (issue opensandbox-group#1563).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f2890dc74c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The python3 report probe runs under set -e, so a non-zero exit (2 for a partial hook degrade) aborted the smoke before the container-log dump and the audit assertions. Wrap it in set +e/set -e so the exit code is captured and the smoke continues to assert exec+connect events.
The cilium verifier error truncates at 4 omitted lines, hiding the exact CO-RE relocation that poisons on_commit_creds; dump the full container log so the fix can target the real field.
Root cause of the privilege-hook CO-RE load failure on x86_64 (issue opensandbox-group#1563): the committed audit_bpfel.o embeds pt_regs relocations from an arm64 vmlinux.h (BPF_KPROBE expands to user_pt_regs[0:0:0] = regs[0], which x86_64 targets do not have), so the kernel poisons the relocation. Add 'make generate-ebpf' (dump the BUILD HOST's own vmlinux.h + bpf2go -target amd64 -target arm64) and a CI freshness check that regenerates on the ubuntu-latest runner and fails if the committed bytecode drifts. The correct x86_64 bytecode will be produced by this CI step and committed separately.
…eration The Dockerfile ebpf-builder stage cross-compiles in a container without access to the host BTF, so the per-arch vmlinux.h must be committed (one per TARGETARCH) rather than dumped at image build time. This step dumps the runner's own vmlinux.h (x86_64 on ubuntu-latest) as an artifact so it can be committed, and keeps the bytecode freshness check.
The ubuntu runner's kernel (6.17) is newer than linux-tools-generic's build, so the meta-package fails to provide bpftool. The standalone 'bpftool' package tracks the runner kernel.
Drop the in-CI bytecode regeneration (the ebpf-builder Dockerfile stage will regenerate per TARGETARCH from committed prog/vmlinux-<arch>.h); this step only fetches the runner's x86_64 vmlinux.h as an artifact so it can be committed once.
Root cause (issue opensandbox-group#1563): the committed audit_bpfel.o embedded pt_regs relocations from an arm64 vmlinux.h, so BPF_KPROBE's user_pt_regs[0:0:0] (=regs[0]) poisoned on x86_64. The bytecode must be compiled per architecture. - Dockerfile ebpf-builder: ARG TARGETARCH selects prog/vmlinux-<arch>.h, bpf2go regenerates the bytecode for that arch, then go build -tags ebpf. This works for the buildx --platform linux/amd64,linux/arm64 publish flow and for local docker builds — no host kernel BTF needed. - Makefile: build-ebpf -> generate-ebpf ARCH=$(go env GOARCH) using the committed per-arch vmlinux.h; local builds regenerate their own arch. - prog/vmlinux-arm64.h: committed (dumped from an arm64 host). - prog/vmlinux-x86_64.h: fetched via a temporary CI step, to be committed once the artifact is downloaded. - Removed the CI dump/freshness steps (bytecode is a build artifact now).
github-actions
Bot
added
size/XXL
The Dockerfile ebpf-builder now regenerates the CO-RE bytecode per TARGETARCH from committed prog/vmlinux-<arch>.h; both headers are in place. Remove the one-shot CI artifact step that fetched the x86_64 dump.
WORKDIR is /build/components/execd, so the headers live under pkg/ebpf/prog/ and the source at pkg/ebpf/prog/audit.bpf.c — the case statement and bpf2go invocation must use those paths (the Makefile cds into pkg/ebpf first, which is why its relative paths were fine).
bpf2go needs bpf_helpers.h/bpf_tracing.h/bpf_core_read.h etc. The Dockerfile previously reached into the module cache ($GOMODCACHE/.../btf/testdata), which is fragile inside the builder stage. Vendor the cilium/ebpf v0.16.0 headers (bpf/ subdir) so both the Dockerfile and make generate-ebpf use -Iprog only.
bpf2go -target amd64|arm64 emits audit_x86_bpfel.go / audit_arm64_bpfel.go (per-arch, build-tagged); the old generic audit_bpfel.go/audit_bpfeb.go stayed in the tree and redeclared loadAudit & friends. Delete them — the Dockerfile builder regenerates the per-arch files at image build time.
The generated audit_x86_bpfel.go etc. are build artifacts (no longer committed), so the ebpf-tagged test needs them generated first. Install clang on the runner and run make generate-ebpf ARCH=$(go env GOARCH) before go test -tags ebpf.
VMLINUX was set inside the recipe shell but expanded by make (empty); use $$VMLINUX and chain the recipe with semicolons. Drop the cd into pkg/ebpf — use full relative paths from the repo root for both cflags and the source file.
The two committed vmlinux.h dumps (~300k lines) are replaced by prog/audit_types.h (~150 lines) declaring only the kernel members the audit programs touch. CO-RE resolves every accessor by member NAME against the target kernel BTF at load time, so the compile-time layout does not need to match any kernel — this makes the bpf2go build hermetic, architecture-independent and removes the per-arch vmlinux.h requirement from the Dockerfile/Makefile (issue opensandbox-group#1563). vendored libbpf headers stay in prog/bpf/.
Buildx injects TARGETARCH only into stages that declare it; without the ARG the -target flag was empty and bpf2go failed with 'unsupported target'.
The vendored libbpf helper declarations reference __wsum/__s32/__s64/ struct __sk_buff, previously supplied by vmlinux.h — declare them in audit_types.h.
bpf_tracing.h's non-__VMLINUX_H__ path reads PT_REGS_PARM1 as (x)->rdi on x86-64 — the vmlinux.h-era definition used di, which the minimal header inherited. Also declare uint32_t/uint64_t/uint16_t and BPF_MAP_TYPE_RINGBUF that vmlinux.h used to provide.
… k8s matcher - audit.go: report the eBPF layer as 'degraded' (spec contract: configured but a prerequisite is missing) when some hooks fail to load/attach, while keeping the working hooks auditing; previously partial coverage reported 'active', which SDK clients could mistake for full auditing - smoke: 'degraded' with 'hooks not active' continues (exec+connect still asserted), 'unsupported' fails; privilege events are now mandatory — su reliably fires commit_creds, so zero means the hook is broken - detect-changes: route test_execd_k8s_restart_recycle_e2e.py to the kubernetes-mini-e2e matcher (it only runs under the k8s nightly)
R-c: kernel-5.10 eBPF validation landed via the execd-ebpf smoke on both CI legs; documents the root cause found (arm64 pt_regs relocations in the committed bytecode, issue opensandbox-group#1563) and the per-TARGETARCH bytecode fix. R-j: JSONL audit e2e implemented (smoke asserts exec/connect/privilege events; privilege=0 is a hard failure). Status header now lists only R-e, R-g and R-f as remaining.
… detail The Implementation Status section now leads with the three remaining items (R-e/R-g/R-f), summarizes completed phases and declined items in one paragraph, and condenses the open-question resolutions to a sentence. The Remaining work table stays as the per-item record.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the remaining pure-test gaps of OSEP-0018 (execd as sandbox init) and adds an empirical eBPF smoke. No runtime/server behavior changes; R-a (trusted stop channel) is formally declined and R-d (cross-language SDK e2e) is cancelled.
Changes
[seccomp] deny+keep_capabilitiese2e: newcomponents/execd/configs/isolation.custom.toml(deny = ["chmod","fchmodat","fchmodat2"]replacing the built-in denylist +keep_capabilities=["CAP_NET_RAW"]);TestHardeningCustomPolicyE2Easserts the denied syscall fails with EACCES in/command, the workload showsCapEff/CapBnd=0x2000(ambient raise survives execve), and the endpoint reports the overrides active. Docker phase 3 (ceiling keeps NET_RAW) + k8s ConfigMap second key.TestHardeningDriftE2E(docker phase 4) — hardened TOML withexecd_run_as_init=falsereportsinit_mode:none+ layersdegradedwithEXECD_INITguidance while execd-spawned/commandstill runs the floor.tests/test_execd_k8s_restart_recycle_e2e.py(k8s nightly) — Pool pod template runs execd as PID 1 (bootstrap.sh+EXECD_INIT=1+EXECD=/execd+ keepalive) with Restart recycle; releasing the BatchSandbox pod-execskill 1, execd forwards SIGTERM and exits, kubelet restarts the container (restartCount increases), pod survives with execd PID 1 again — verifies therestart_default.go"contract compatible" comment e2e.TestHardeningDefaultOffE2E(docker phase 5, plain server:init_mode:none, all layersdisabled, workload unaffected);test_sustained_fork_heavy_mix_keeps_process_table_bounded(~30s interleaved churn + sleepers, bounded zombie-free table).test_runtime_stop_forwards_sigterm_and_propagates_exit_code(docker bridge) — entrypoint traps TERM (marker + exit 7),docker stopthe container via theopensandbox.io/idlabel, sandbox endsFailedwith "exited with code 7" and the marker is recovered viadocker cp.scripts/execd-ebpf-smoke.sh— runs theexecd-ebpfvariant in a bare container (CAP_BPF+CAP_PERFMON+CAP_SYSLOG, tracefs mount), prechecks BTF/cgroup-v2/tracefs, generates exec/connect/privilege events viadocker execin the same cgroup, and asserts the JSONL audit file. Doubles as the kernel-5.10 empirical validation._hardening_reportcache is per-sandbox (multiple classes share one k8s pytest invocation).Verification
bash -npasses on the scripts.python-execd-hardening-e2e(self-hosted) now runs 5 phases; k8s nightly runs the new recycle test. eBPF smoke is standalone (run on a 5.10 node to validate R-c).Closes #1563