Skip to content

Commit 7174983

Browse files
moezdilmoezdil
authored
fix(sandbox): add mechanistic smoke test for L4 deny and document the L4/L7 split (#1412)
* fix(sandbox): add mechanistic smoke test for L4 deny and document the L4/L7 split The old smoke script exercised an L7 PUT which hung because the denial aggregator is only wired to L4 CONNECT denies, not L7 enforcement. Add mechanistic-smoke.sh which triggers an L4 deny, waits for the aggregator to flush, and asserts a pending chunk appears under openshell rule get --status pending. Document the intentional L4-only scope of the mechanistic mapper in architecture/sandbox.md. Fixes #1333 Signed-off-by: mesutoezdil <mesudozdil@gmail.com> * refactor(smoke): remove redundant variable inits and merge double step call Signed-off-by: mesutoezdil <mesudozdil@gmail.com> * fix(smoke): wire mechanistic smoke into mise and guard TMP_DIR - Initialize TMP_DIR before trap to prevent unbound variable on early exit - Add e2e:mechanistic-smoke mise task with gateway setup - Document mechanistic smoke in policy-advisor README * test(proxy): verify L4 deny enqueues a DenialEvent Signed-off-by: mesutoezdil <mesudozdil@gmail.com> * fix(proxy): remove unnecessary path qualifications in L4 denial smoke test --------- Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
1 parent 9e5aee4 commit 7174983

5 files changed

Lines changed: 197 additions & 0 deletions

File tree

‎architecture/sandbox.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,21 @@ Sandbox logs are emitted locally and can also be pushed back to the gateway.
8989
Security-relevant sandbox behavior uses OCSF structured events; internal
9090
diagnostics use ordinary tracing.
9191

92+
## Policy Proposals
93+
94+
When an L4 CONNECT is denied, the proxy emits a `DenialEvent`. The denial
95+
aggregator batches these events and flushes summaries to the gateway every 10
96+
seconds (configurable via `OPENSHELL_DENIAL_FLUSH_INTERVAL_SECS`). The gateway
97+
runs them through the mechanistic mapper, which generates a pending
98+
`NetworkPolicyRule` proposal visible under `openshell rule get --status pending`.
99+
100+
L7 denials (HTTP 403 from method/path rules) are intentionally excluded from
101+
mechanistic mapping. L4 denials carry only `host:port`, which a deterministic mapper can handle.
102+
L7 denials carry method, path, query, and body context. The agent loop reads
103+
the structured 403 and authors the narrowest rule. Mechanistically mapping L7
104+
would either over-broaden rules or require path-templating logic that rots
105+
quickly.
106+
92107
## Failure Behavior
93108

94109
- If gateway config polling fails, the sandbox keeps its last-known-good policy.

‎crates/openshell-sandbox/src/proxy.rs‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6649,4 +6649,40 @@ network_policies:
66496649
}
66506650
}
66516651
}
6652+
6653+
#[test]
6654+
fn test_emit_denial_enqueues_denial_event() {
6655+
let (tx, mut rx) = mpsc::unbounded_channel::<DenialEvent>();
6656+
let decision = ConnectDecision {
6657+
action: NetworkAction::Deny {
6658+
reason: "no matching policy".into(),
6659+
},
6660+
generation: 0,
6661+
binary: Some(PathBuf::from("/usr/bin/curl")),
6662+
binary_pid: Some(1234),
6663+
ancestors: vec![],
6664+
cmdline_paths: vec![],
6665+
};
6666+
6667+
emit_denial(
6668+
&Some(tx),
6669+
"blocked.invalid",
6670+
443,
6671+
"/usr/bin/curl",
6672+
&decision,
6673+
"no matching policy",
6674+
"connect",
6675+
);
6676+
6677+
let event = rx
6678+
.try_recv()
6679+
.expect("DenialEvent should be enqueued after L4 deny");
6680+
assert_eq!(event.host, "blocked.invalid");
6681+
assert_eq!(event.port, 443);
6682+
assert_eq!(event.binary, "/usr/bin/curl");
6683+
assert_eq!(event.denial_stage, "connect");
6684+
assert_eq!(event.deny_reason, "no matching policy");
6685+
assert!(event.l7_method.is_none());
6686+
assert!(event.l7_path.is_none());
6687+
}
66526688
}

‎e2e/policy-advisor/README.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,3 +52,18 @@ bash e2e/policy-advisor/test.sh
5252
Requires Docker, `agent_policy_proposals_enabled=true`, and a GitHub token with
5353
contents write on the repository. The test auto-resolves the token from
5454
`DEMO_GITHUB_TOKEN`, `GITHUB_TOKEN`, `GH_TOKEN`, or `gh auth token`.
55+
56+
## Mechanistic smoke
57+
58+
Lightweight regression for the L4 CONNECT deny → mechanistic chunk pipeline.
59+
No GitHub token or LLM required.
60+
61+
```bash
62+
mise run e2e:mechanistic-smoke
63+
```
64+
65+
Or manually against a running gateway with `agent_policy_proposals_enabled=true`:
66+
67+
```bash
68+
OPENSHELL_BIN=target/debug/openshell bash e2e/policy-advisor/mechanistic-smoke.sh
69+
```

‎e2e/policy-advisor/mechanistic-smoke.sh‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
#!/usr/bin/env bash
2+
3+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
4+
# SPDX-License-Identifier: Apache-2.0
5+
6+
# Regression smoke for the mechanistic policy mapper.
7+
#
8+
# Triggers an L4 CONNECT deny from inside a sandbox, waits for the denial
9+
# aggregator to flush, and asserts that a pending mechanistic chunk appears
10+
# under `openshell rule get --status pending`.
11+
#
12+
# This is deliberately L4-only. L7 denials (method/path 403s) are the agent
13+
# loop's job; the mechanistic mapper only covers L4 CONNECT denials. See #1333.
14+
#
15+
# Prereqs: a running gateway with agent_policy_proposals_enabled=true.
16+
17+
set -euo pipefail
18+
19+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
20+
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
21+
22+
if [[ -z "${OPENSHELL_BIN:-}" ]]; then
23+
if [[ -x "${REPO_ROOT}/target/debug/openshell" ]]; then
24+
OPENSHELL_BIN="${REPO_ROOT}/target/debug/openshell"
25+
else
26+
OPENSHELL_BIN="openshell"
27+
fi
28+
fi
29+
30+
RUN_ID="${RUN_ID:-$(date +%Y%m%d-%H%M%S)}"
31+
SANDBOX="${SANDBOX:-mechanistic-smoke-${RUN_ID}}"
32+
KEEP_SANDBOX="${KEEP_SANDBOX:-0}"
33+
# Allow override so CI can set a shorter interval via OPENSHELL_DENIAL_FLUSH_INTERVAL_SECS.
34+
FLUSH_WAIT="${FLUSH_WAIT:-15}"
35+
36+
BOLD='\033[1m'
37+
CYAN='\033[36m'
38+
GREEN='\033[32m'
39+
RED='\033[31m'
40+
RESET='\033[0m'
41+
42+
step() { printf "\n${BOLD}${CYAN}==> %s${RESET}\n\n" "$1"; }
43+
ok() { printf " ${GREEN}✓${RESET} %s\n" "$*"; }
44+
fail() { printf "\n${RED}FAIL:${RESET} %s\n" "$*" >&2; exit 1; }
45+
46+
TMP_DIR=""
47+
48+
cleanup() {
49+
if [[ "$KEEP_SANDBOX" != "1" ]]; then
50+
"$OPENSHELL_BIN" sandbox delete "$SANDBOX" >/dev/null 2>&1 || true
51+
fi
52+
[[ -z "$TMP_DIR" ]] || rm -rf "$TMP_DIR"
53+
}
54+
trap cleanup EXIT
55+
56+
preflight() {
57+
step "Preflight"
58+
local raw_settings
59+
if ! raw_settings="$("$OPENSHELL_BIN" settings get --global --json 2>&1)"; then
60+
fail "cannot reach gateway: ${raw_settings}"
61+
fi
62+
local enabled
63+
enabled="$(printf '%s' "$raw_settings" \
64+
| jq -r '.settings.agent_policy_proposals_enabled // "<unset>"')"
65+
[[ "$enabled" == "true" ]] \
66+
|| fail "set agent_policy_proposals_enabled=true first:
67+
$OPENSHELL_BIN settings set --global --key agent_policy_proposals_enabled --value true --yes"
68+
ok "agent_policy_proposals_enabled=true"
69+
}
70+
71+
create_sandbox() {
72+
step "Creating sandbox '${SANDBOX}' (no network policy)"
73+
TMP_DIR="$(mktemp -d)"
74+
SSH_CONFIG="${TMP_DIR}/ssh_config"
75+
76+
"$OPENSHELL_BIN" sandbox delete "$SANDBOX" >/dev/null 2>&1 || true
77+
"$OPENSHELL_BIN" sandbox create \
78+
--name "$SANDBOX" \
79+
--no-auto-providers \
80+
--no-tty \
81+
--keep \
82+
-- bash -lc "echo sandbox ready" \
83+
| sed 's/^/ /'
84+
85+
"$OPENSHELL_BIN" sandbox ssh-config "$SANDBOX" > "$SSH_CONFIG"
86+
SSH_HOST="$(awk '/^Host / { print $2; exit }' "$SSH_CONFIG")"
87+
[[ -n "$SSH_HOST" ]] || fail "could not parse SSH host"
88+
89+
for _i in $(seq 1 30); do
90+
ssh -F "$SSH_CONFIG" "$SSH_HOST" true >/dev/null 2>&1 && { ok "SSH up"; return; }
91+
sleep 2
92+
done
93+
fail "SSH timed out"
94+
}
95+
96+
trigger_l4_deny() {
97+
step "Triggering L4 CONNECT deny from inside sandbox"
98+
# blocked.invalid is guaranteed unroutable and not in any policy.
99+
ssh -F "$SSH_CONFIG" "$SSH_HOST" \
100+
"curl -sf --max-time 5 https://blocked.invalid/ || true" >/dev/null 2>&1 || true
101+
ok "curl attempted (deny expected)"
102+
}
103+
104+
assert_pending_chunk() {
105+
step "Waiting ${FLUSH_WAIT}s then checking for pending chunk"
106+
sleep "$FLUSH_WAIT"
107+
local output
108+
output="$("$OPENSHELL_BIN" rule get "$SANDBOX" --status pending 2>&1)"
109+
printf '%s\n' "$output" | sed 's/^/ /'
110+
printf '%s\n' "$output" | grep -qi "blocked.invalid" \
111+
|| fail "no pending chunk for blocked.invalid"
112+
ok "pending mechanistic chunk present for blocked.invalid"
113+
}
114+
115+
main() {
116+
command -v jq >/dev/null || fail "jq is required"
117+
preflight
118+
create_sandbox
119+
trigger_l4_deny
120+
assert_pending_chunk
121+
step "Smoke pass"
122+
}
123+
124+
main "$@"

‎tasks/test.toml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,13 @@ run = "e2e/rust/e2e-vm.sh"
8383
description = "Run smoke e2e against a standalone gateway with the Docker compute driver"
8484
run = "e2e/rust/e2e-docker.sh"
8585

86+
["e2e:mechanistic-smoke"]
87+
description = "Run mechanistic L4 smoke against a Docker-backed gateway"
88+
run = [
89+
"cargo build -p openshell-cli --features openshell-core/dev-settings",
90+
"e2e/with-docker-gateway.sh bash -lc 'target/debug/openshell settings set --global --key agent_policy_proposals_enabled --value true --yes && OPENSHELL_BIN=$PWD/target/debug/openshell bash e2e/policy-advisor/mechanistic-smoke.sh'",
91+
]
92+
8693
["e2e:docker:gpu"]
8794
description = "Run GPU e2e against a standalone gateway with the Docker compute driver"
8895
env = { OPENSHELL_E2E_DOCKER_GPU = "1", OPENSHELL_E2E_DOCKER_TEST = "gpu_device_selection", OPENSHELL_E2E_DOCKER_FEATURES = "e2e-docker-gpu" }

0 commit comments

Comments
 (0)