Skip to content

execd.md

Latest commit

 

History

History
257 lines (197 loc) · 10 KB

execd.md

File metadata and controls

257 lines (197 loc) · 10 KB
Raw
title execd
description The in-sandbox execution daemon providing HTTP APIs for code execution, shell commands, filesystem operations, PTY sessions, and metrics.

execd - OpenSandbox Execution Daemon

execd is the runtime daemon used inside OpenSandbox sandboxes.

It is built on Gin and exposes HTTP APIs for code execution, shell commands, filesystem operations, PTY sessions, and metrics.

Quick Start

1) Build

cd components/execd
make build

On Linux, make build uses the native C compiler and static libc to produce bin/opensandbox-session-gate. The published execd image already installs this helper. If you run execd from a source build and need isolated sessions, install it at the fixed trusted runtime path first:

make build-session-gate
sudo make install-session-gate
# /opt/opensandbox/opensandbox-session-gate (mode 0555)

Compilation runs before privilege escalation; the install target only copies the built helper. Keep /opt/opensandbox and the helper root-owned and not group- or world-writable. Other execd APIs still work without it, but isolated-session capability probing and creation fail closed.

2) Start Jupyter Server

./tests/jupyter.sh

3) Run execd

./bin/execd \
  --jupyter-host=http://127.0.0.1:54321 \
  --jupyter-token=your-jupyter-token \
  --port=44772

4) Verify

curl -v http://localhost:44772/ping

API

  • OpenAPI spec: execd-api.yaml
  • Common capability groups:
    • Code execution (/code, SSE stream)
    • Session and command execution (/session, /command)
    • Filesystem operations (/files, /directories)
    • Isolated sessions (/v1/isolated/session, bubblewrap namespaces)
    • PTY over WebSocket (/pty)
    • Local metrics endpoints (/metrics, /metrics/watch)

Shell-backed sessions use Bash when it is available and fall back to sh on minimal images that do not include Bash. This applies to PTY sessions, the Bash session API (which keeps its existing name for compatibility), and isolated sessions. Commands submitted to a fallback session must use syntax supported by that image's sh implementation.

PTY WebSocket access

The first WebSocket attached to /pty/{session_id}/ws is the exclusive read/write holder. A second read/write connection receives 409 Conflict unless it uses ?takeover=1 to replace that holder.

After the read/write holder has started the shell, any number of read-only clients can attach with:

ws://localhost:44772/pty/{session_id}/ws?mode=viewer&since=0

Viewer connections receive the retained replay followed by live output, but they never acquire or evict the read/write holder. The JSON connected frame sets role to viewer (holders receive role: "holder") so clients can use the appropriate binary-frame decoder. Holders receive 0x01 stdout and, in pipe mode, 0x02 stderr frames. Viewers receive 0x03 replay frames for both retained and live output: an 8-byte big-endian offset followed by raw bytes.

Binary stdin and JSON stdin, signal, or resize frames are rejected with a READ_ONLY error; ping remains available. The server closes a viewer after five rejected mutating frames to bound error traffic from a misbehaving client. WebSocket backpressure from a slow or disconnected viewer does not block the interactive holder's live output pipe. In pipe mode, viewer output uses the combined replay stream because replay does not preserve stdout/stderr channel boundaries.

A viewer can attach only while the shell is running. When the shell exits, viewers receive the exit frame and close. If a holder later starts the same session again, its bounded replay buffer is retained, so a viewer reconnecting with since=0 can receive retained output from the preceding shell lifetime.

Isolated Sessions

Isolated sessions run a shell inside a per-execution bubblewrap (bwrap) namespace, created via POST /v1/isolated/session. Bash is preferred, with sh used as a fallback. Beyond the workspace, callers can expose additional host paths into the namespace.

UID modes and capabilities

The optional uid_mode request field selects how identity is established:

  • setpriv (the default) uses the container's existing user namespace and drops to the requested UID/GID with setpriv.
  • userns creates a new user namespace and maps the requested UID/GID inside it, which can work in environments where the capabilities required by setpriv mode are unavailable.

At startup, execd probes both modes independently. GET /v1/isolated/capabilities reports setpriv_available and userns_available; the overall available field is true when either mode is usable. Creating a session returns 503 NOT_SUPPORTED only when the selected mode is unavailable. The probes exercise the same identity path used at runtime: the public setpriv_available flag covers execd's default UID/GID path (so a root session that keeps UID/GID 0 does not require the setpriv binary), while userns applies the UID/GID mapping and the setuid-aware --disable-userns policy. A setpriv request that selects IDs different from execd's own is checked against a separate startup identity-switch probe and returns 503 NOT_SUPPORTED before session side effects when that switch is not available.

For a private-network Session (share_net: false), execd fixes the authenticated network namespace and its owning user namespace before the native workload gate is released. The two namespace bind mounts use an execd-owned, unpredictable directory below /run/execd/namespaces and stay owned by the Session until synchronous teardown. This applies to both UID modes; shared-network Sessions do not create namespace pins.

Bind mounts

Two request fields control extra host paths:

  • extra_writable: a list of paths bind-mounted read-write at the same path inside the namespace (source == destination).
  • binds: explicit source → dest mappings, each optionally read-only.
    • source (required): host path to bind. It must already exist and is resolved (symlinks followed) before use.
    • dest: mount destination inside the namespace; defaults to source when omitted. It must be an existing mount point — bwrap cannot create a destination under the read-only root, so create the directory first.
    • readonly (default false): mount read-only (--ro-bind) when true, read-write (--bind) otherwise.

Example:

{
  "workspace": { "path": "/workspace", "mode": "rw" },
  "binds": [
    { "source": "/data/in",  "dest": "/mnt/in", "readonly": true },
    { "source": "/data/out", "dest": "/mnt/out" }
  ]
}

Writable allowlist

The source path of every extra_writable entry and every binds entry must fall within the allowed_writable allowlist (see the isolation config file below). The allowlist is enforced against the fully symlink-resolved real path, so a symlink cannot redirect a bind outside the allowlist. An empty allowlist rejects all extra_writable/binds requests.

The built-in default allowlist is /workspace, /mnt, /media, /data (subpaths included). Set allowed_writable in the isolation config to override it.

Configuration

CLI Flags

Flag Default Description
--jupyter-host "" Jupyter server URL reachable by execd.
--jupyter-token "" Jupyter token for HTTP/WebSocket auth.
--port 44772 HTTP listen port.
--log-level 6 Log level (0=Emergency, 7=Debug).
--access-token "" Optional shared API access token.
--graceful-shutdown-timeout 1s SSE tail-drain wait window before closing.
--jupyter-idle-poll-interval 100ms Poll interval after Jupyter reports idle.
--isolation-config "" Path to the isolation TOML config (see below).

Environment Variables

Variable Description
JUPYTER_HOST Same as --jupyter-host (overridden by explicit flag).
JUPYTER_TOKEN Same as --jupyter-token (overridden by explicit flag).
EXECD_ACCESS_TOKEN Same as --access-token (overridden by explicit flag).
EXECD_API_GRACE_SHUTDOWN Same as --graceful-shutdown-timeout.
EXECD_JUPYTER_IDLE_POLL_INTERVAL Same as --jupyter-idle-poll-interval.
EXECD_ISOLATION_CONFIG Same as --isolation-config.
EXECD_CLONE3_COMPAT Linux clone3 compatibility switch (see below).
EXECD_LOG_FILE Optional log output file path; default is stdout.
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT Preferred OTLP metrics endpoint.
OTEL_EXPORTER_OTLP_ENDPOINT Fallback OTLP endpoint when metrics-specific endpoint is unset.
OPENSANDBOX_ID Optional sandbox_id metric/resource attribute.
OPENSANDBOX_EXECD_METRICS_EXTRA_ATTRS Optional extra metric attrs (k=v,k2=v2).

Isolation Config File

Isolated sessions read an optional TOML file given by --isolation-config (or EXECD_ISOLATION_CONFIG). All fields are optional; omitted fields use built-in defaults.

# Parent directory for per-session overlay upper directories.
upper_root = "/var/lib/execd/isolation"

# Host paths callers may request via extra_writable / binds.
# Enforced against the fully symlink-resolved real path; subpaths are allowed.
# Default: ["/workspace", "/mnt", "/media", "/data"]. Empty = reject all.
allowed_writable = ["/workspace", "/mnt", "/media", "/data"]

Observability

OpenTelemetry Metrics

OTLP metrics export is enabled when either endpoint is set:

  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
  • OTEL_EXPORTER_OTLP_ENDPOINT

Local Metrics Endpoints

  • GET /metrics: point-in-time host metrics snapshot
  • GET /metrics/watch: SSE stream (1s cadence)

Linux clone3 Compatibility

Some sandbox environments fail on clone3(2). Set EXECD_CLONE3_COMPAT in sandbox env to force fallback behavior:

  • 1 / true / yes / on: enable seccomp fallback
  • reexec: enable fallback and re-exec binary

License

execd is part of OpenSandbox. See the LICENSE.