Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions components/execd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ English | [中文](README_zh.md)
- [Architecture](#architecture)
- [Getting Started](#getting-started)
- [Configuration](#configuration)
- [Linux `clone3` compatibility inside sandboxes](#linux-clone3-compatibility-inside-sandboxes)
- [API Reference](#api-reference)
- [Supported Languages](#supported-languages)
- [Development](#development)
Expand Down Expand Up @@ -82,6 +83,7 @@ English | [中文](README_zh.md)
| `pkg/jupyter/execute/` | Execution result types and stream parsers |
| `pkg/jupyter/session/` | Session management and lifecycle |
| `pkg/util/` | Utilities (safe goroutine helpers, glob helpers) |
| `pkg/clone3compat/` | Optional seccomp shim for `clone3` → `ENOSYS` on Linux |
| `tests/` | Test scripts and tools |

## Getting Started
Expand Down Expand Up @@ -169,6 +171,43 @@ export JUPYTER_TOKEN=your-token

Environment variables override defaults but are superseded by explicit CLI flags.

| Variable | Description |
|----------|-------------|
| `EXECD_CLONE3_COMPAT` | Linux only. See [below](#linux-clone3-compatibility-inside-sandboxes). |

### Linux `clone3` compatibility inside sandboxes

Some sandbox images ship **glibc ≥ 2.34**, which prefers the `clone3(2)` syscall. On **older container engines** (for example Docker **before 20.10.10** and matching containerd CRI versions), or wherever `clone3` is not handled correctly, **process creation can fail**. That affects anything that forks or execs: **Go `os/exec`** inside execd, shell commands, package managers (`apt`, `dnf`), and language runtimes that start subprocesses.

Typical symptoms are errors mentioning **`clone3`**, **`Function not implemented`**, or **`Operation not permitted`** when running commands or code **inside the sandbox**—not necessarily when starting the OpenSandbox server on the host.

#### Enabling the built-in workaround

execd can install a **seccomp** rule so `clone3` returns **`ENOSYS`**, forcing libc and the Go runtime to fall back to `clone(2)`—the same idea as [AkihiroSuda/clone3-workaround](https://github.com/AkihiroSuda/clone3-workaround). Implementation: [`pkg/clone3compat/`](pkg/clone3compat/).

Set the variable **in the sandbox container environment** (where execd is PID 1 or otherwise started):

| Value | Behavior |
|-------|----------|
| `1`, `true`, `yes`, `on` | Load the filter at the beginning of `main` (after Go runtime `init`). |
| `reexec` | Load the filter, then `exec` the same binary again so subsequent `init` runs already under seccomp (closest to wrapping with the external workaround binary). |

Unset, empty, or `0` / `false` / `off` / `no` disables the feature.

#### SDK

Set `EXECD_CLONE3_COMPAT` in `Sandbox.create` `env` so execd sees it when the sandbox starts:

```python
sandbox = await Sandbox.create(
"opensandbox/code-interpreter:v1.0.2",
entrypoint=["/opt/opensandbox/code-interpreter.sh"],
env={"EXECD_CLONE3_COMPAT": "1"},
)
```

The sandbox must allow **seccomp** and **`PR_SET_NO_NEW_PRIVS`** for the filter to load. Upgrading the host container engine/kernel is the long-term fix when possible.

## API Reference

[API Spec](../../specs/execd-api.yaml).
Expand Down
41 changes: 41 additions & 0 deletions components/execd/README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
- [架构设计](#架构设计)
- [快速开始](#快速开始)
- [配置说明](#配置说明)
- [沙箱内的 Linux clone3 兼容](#沙箱内的-linux-clone3-兼容)
- [API 参考](#api-参考)
- [支持的语言](#支持的语言)
- [开发指南](#开发指南)
Expand Down Expand Up @@ -80,6 +81,7 @@
| `pkg/jupyter/execute/` | 执行结果类型与流解析器 |
| `pkg/jupyter/session/` | 会话管理与生命周期 |
| `pkg/util/` | 通用工具(安全 goroutine、glob 辅助) |
| `pkg/clone3compat/` | Linux 下可选 seccomp,使 `clone3` 返回 `ENOSYS` |
| `tests/` | 测试脚本和工具 |

## 快速开始
Expand Down Expand Up @@ -167,6 +169,45 @@ export JUPYTER_TOKEN=your-token

环境变量优先于默认值,但会被显式的 CLI 标志覆盖。

| 变量 | 说明 |
|------|------|
| `EXECD_CLONE3_COMPAT` | 仅 Linux。详见 [下文](#沙箱内的-linux-clone3-兼容)。 |

### 沙箱内的 Linux clone3 兼容

部分沙箱镜像使用 **glibc ≥ 2.34**,会优先使用 **`clone3(2)`** 系统调用。在**较旧的容器引擎**上(例如 Docker **低于 20.10.10** 及对应 containerd CRI 版本),或运行时对 `clone3` 支持不完整时,**创建子进程可能失败**。

受影响场景包括:execd 内部的 **Go `os/exec`**、shell 命令、`apt`/`dnf` 等包管理器,以及会拉子进程的语言运行时。

常见报错里会出现 **`clone3`**、**`Function not implemented`** 或 **`Operation not permitted`**,且多发生在**沙箱内**执行命令或代码时,而不一定是宿主机上启动 OpenSandbox 服务时。

#### 启用内置规避

execd 可通过 **seccomp** 让 `clone3` 返回 **`ENOSYS`**,迫使 libc 与 Go 运行时回退到 **`clone(2)`**,思路与 [AkihiroSuda/clone3-workaround](https://github.com/AkihiroSuda/clone3-workaround) 一致。实现见 [`pkg/clone3compat/`](pkg/clone3compat/)。

在**沙箱容器环境变量**中设置(execd 作为容器入口或主进程时):

| 取值 | 行为 |
|------|------|
| `1`、`true`、`yes`、`on` | 在 `main` 开头加载过滤器(Go `init` 之后)。 |
| `reexec` | 加载过滤器后对同二进制再 `exec` 一次,使后续 `init` 已在 seccomp 下运行(最接近外挂 workaround 二进制的方式)。 |

未设置、为空,或为 `0` / `false` / `off` / `no` 时关闭。

#### SDK

在 `Sandbox.create` 的 `env` 中设置 `EXECD_CLONE3_COMPAT`,使 execd 在沙箱启动时即可读到:

```python
sandbox = await Sandbox.create(
"opensandbox/code-interpreter:v1.0.2",
entrypoint=["/opt/opensandbox/code-interpreter.sh"],
env={"EXECD_CLONE3_COMPAT": "1"},
)
```

沙箱需允许加载 seccomp 过滤器(**seccomp**、**`PR_SET_NO_NEW_PRIVS`**)。长期仍建议在宿主机上升级容器引擎/内核,使 `clone3` 全链路可用。

## API 参考

[API Spec](../../specs/execd-api.yaml)。
Expand Down
3 changes: 2 additions & 1 deletion components/execd/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ go 1.24.0
require (
github.com/alibaba/opensandbox/internal v0.0.0
github.com/bmatcuk/doublestar/v4 v4.9.1
github.com/elastic/go-seccomp-bpf v1.6.0
github.com/gin-gonic/gin v1.10.0
github.com/go-playground/validator/v10 v10.28.0
github.com/go-sql-driver/mysql v1.8.1
Expand All @@ -13,6 +14,7 @@ require (
github.com/shirou/gopsutil v3.21.11+incompatible
github.com/stretchr/testify v1.10.0
go.uber.org/automaxprocs v1.6.0
golang.org/x/sys v0.38.0
k8s.io/apimachinery v0.34.2
k8s.io/client-go v0.34.2
)
Expand Down Expand Up @@ -54,7 +56,6 @@ require (
golang.org/x/arch v0.8.0 // indirect
golang.org/x/crypto v0.45.0 // indirect
golang.org/x/net v0.47.0 // indirect
golang.org/x/sys v0.38.0 // indirect
golang.org/x/text v0.31.0 // indirect
google.golang.org/protobuf v1.36.5 // indirect
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
Expand Down
2 changes: 2 additions & 0 deletions components/execd/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ3
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/elastic/go-seccomp-bpf v1.6.0 h1:NYduiYxRJ0ZkIyQVwlSskcqPPSg6ynu5pK0/d7SQATs=
github.com/elastic/go-seccomp-bpf v1.6.0/go.mod h1:5tFsTvH4NtWGfpjsOQD53H8HdVQ+zSZFRUDSGevC0Kc=
github.com/fxamacker/cbor/v2 v2.9.0 h1:NpKPmjDBgUfBms6tr6JZkTHtfFGcMKsw3eGcmD/sapM=
github.com/fxamacker/cbor/v2 v2.9.0/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
github.com/gabriel-vasile/mimetype v1.4.10 h1:zyueNbySn/z8mJZHLt6IPw0KoZsiQNszIpU+bX4+ZK0=
Expand Down
6 changes: 6 additions & 0 deletions components/execd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (

_ "go.uber.org/automaxprocs/maxprocs"

"github.com/alibaba/opensandbox/execd/pkg/clone3compat"
"github.com/alibaba/opensandbox/execd/pkg/flag"
"github.com/alibaba/opensandbox/execd/pkg/log"
_ "github.com/alibaba/opensandbox/execd/pkg/util/safego"
Expand All @@ -31,11 +32,16 @@ import (

// main initializes and starts the execd server.
func main() {
clone3Compat := clone3compat.MaybeApply()

version.EchoVersion("OpenSandbox Execd")

flag.InitFlags()

log.Init(flag.ServerLogLevel)
if clone3Compat {
log.Warn("execd running with clone3 compatibility (seccomp returns ENOSYS for clone3)")
}

controller.InitCodeRunner()
engine := web.NewRouter(flag.ServerAccessToken)
Expand Down
27 changes: 27 additions & 0 deletions components/execd/pkg/clone3compat/compat.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
// Copyright 2026 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package clone3compat installs an optional seccomp rule so clone3(2) returns ENOSYS and
// libc / the Go runtime fall back to clone(2), following the same idea as
// https://github.com/AkihiroSuda/clone3-workaround .
//
// Enable on Linux via environment when starting execd:
//
// EXECD_CLONE3_COMPAT=1 — install the filter at process start (after Go runtime init).
// EXECD_CLONE3_COMPAT=reexec — install the filter then re-exec the same binary so all
// package init code runs with the filter already active
// (closest to wrapping with the external clone3-workaround binary).
//
// Disabled when unset or empty, or set to 0, false, off, no.
package clone3compat
103 changes: 103 additions & 0 deletions components/execd/pkg/clone3compat/compat_linux.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
// Copyright 2026 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//go:build linux

package clone3compat

import (
"errors"
"fmt"
"os"
"strings"
"syscall"

seccomp "github.com/elastic/go-seccomp-bpf"
"golang.org/x/sys/unix"
)

const (
envCompat = "EXECD_CLONE3_COMPAT"
envApplied = "_EXECD_CLONE3_COMPAT_APPLIED"
)

// MaybeApply optionally installs a seccomp rule so clone3 returns ENOSYS, matching the
// behavior of https://github.com/AkihiroSuda/clone3-workaround .
// It returns true if this process is running with that compatibility active (including
// a post-reexec process that inherited the seccomp filter).
func MaybeApply() bool {
mode := strings.ToLower(strings.TrimSpace(os.Getenv(envCompat)))
switch mode {
case "", "0", "false", "off", "no":
return false
case "1", "true", "yes", "on":
if err := loadClone3EnosysFilter(); err != nil {
_, _ = fmt.Fprintf(os.Stderr, "execd: %v\n", err)
os.Exit(1)
}
return true
case "reexec":
if os.Getenv(envApplied) == "1" {
return true
}
if err := loadClone3EnosysFilter(); err != nil {
_, _ = fmt.Fprintf(os.Stderr, "execd: %v\n", err)
os.Exit(1)
}
if err := os.Setenv(envApplied, "1"); err != nil {
_, _ = fmt.Fprintf(os.Stderr, "execd: clone3 compat: set %s: %v\n", envApplied, err)
os.Exit(1)
}
exe, err := os.Readlink("/proc/self/exe")
if err != nil {
_, _ = fmt.Fprintf(os.Stderr, "execd: clone3 compat: readlink /proc/self/exe: %v\n", err)
os.Exit(1)
}
exe = strings.TrimSuffix(exe, " (deleted)")
if err := unix.Exec(exe, os.Args, os.Environ()); err != nil {
_, _ = fmt.Fprintf(os.Stderr, "execd: clone3 compat: exec: %v\n", err)
os.Exit(1)
}
panic("unreachable") // Exec replaces this process.
default:
_, _ = fmt.Fprintf(os.Stderr, "execd: invalid %s=%q (use 1, true, or reexec)\n", envCompat, os.Getenv(envCompat))
os.Exit(1)
}

return false
}

func loadClone3EnosysFilter() error {
if !seccomp.Supported() {
return errors.New("clone3 compat: seccomp is not available on this kernel")
}
f := seccomp.Filter{
NoNewPrivs: true,
Flag: seccomp.FilterFlagTSync,
Policy: seccomp.Policy{
DefaultAction: seccomp.ActionAllow,
Syscalls: []seccomp.SyscallGroup{
{
Names: []string{"clone3"},
// Not plain ActionErrno: assembler defaults errno to EPERM.
Action: seccomp.ActionErrno | seccomp.Action(syscall.ENOSYS),
},
},
},
}
if err := seccomp.LoadFilter(f); err != nil {
return fmt.Errorf("clone3 compat: %w", err)
}
return nil
}
20 changes: 20 additions & 0 deletions components/execd/pkg/clone3compat/compat_stub.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright 2026 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//go:build !linux

package clone3compat

// MaybeApply is a no-op on non-Linux platforms.
func MaybeApply() bool { return false }
13 changes: 13 additions & 0 deletions sandboxes/code-interpreter/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,19 @@

FROM opensandbox/code-interpreter-base:latest

ARG TARGETARCH
# Prebuilt binary from https://github.com/AkihiroSuda/clone3-workaround/releases/tag/v1.0.0 (amd64 only).
# The binary is linked against libseccomp (runtime package libseccomp2 on Ubuntu).
RUN set -eux; \
if [ "${TARGETARCH}" = "amd64" ]; then \
apt-get update && apt-get install -y --no-install-recommends libseccomp2 && rm -rf /var/lib/apt/lists/*; \
curl -fsSL -o /usr/local/bin/clone3-workaround \
https://github.com/AkihiroSuda/clone3-workaround/releases/download/v1.0.0/clone3-workaround.x86_64; \
chmod 755 /usr/local/bin/clone3-workaround; \
else \
echo "Skipping clone3-workaround: upstream provides amd64/x86_64 binary only (TARGETARCH=${TARGETARCH})"; \
fi

# Install Python kernels
RUN set -euo pipefail \
&& echo "Setting up ipykernel for Python 3.10, 3.11, 3.12, 3.13, 3.14" \
Expand Down
5 changes: 5 additions & 0 deletions sandboxes/code-interpreter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ provide an out-of-the-box multi-language code execution environment.
- **Version Switching**: Easy runtime version switching without rebuilding
- **Jupyter Integration**: Built-in Jupyter Notebook with multi-language kernels
- **Multi-Architecture**: Supports both amd64 and arm64 architectures
- **clone3-workaround (amd64)**: The image installs [AkihiroSuda/clone3-workaround](https://github.com/AkihiroSuda/clone3-workaround) v1.0.0 as `/usr/local/bin/clone3-workaround` on **linux/amd64** only (upstream ships no arm64 binary), plus **`libseccomp2`** because the upstream binary is dynamically linked to `libseccomp`. Use it to wrap commands on very old Docker/containerd hosts, e.g. `clone3-workaround apt-get update`.
- **Production Ready**: Optimized for containerized execution environments

## Supported Languages & Versions
Expand Down Expand Up @@ -58,6 +59,10 @@ docker run -it --rm \
opensandbox/code-interpreter:latest
```

### `EXECD_CLONE3_COMPAT` (clone3-workaround)

If you set `EXECD_CLONE3_COMPAT` to `1`, `true`, `yes`, `on`, or `reexec` (same semantics as [execd](../../components/execd/README.md#linux-clone3-compatibility-inside-sandboxes)), the entrypoint script **re-executes itself** under `/usr/local/bin/clone3-workaround` before Jupyter and kernel setup. That binary is included on **linux/amd64** only; on **arm64** builds the script prints a warning and continues without wrapping. After a successful wrap, the script **unsets** `EXECD_CLONE3_COMPAT` in the running process tree. Use `0`, `false`, `off`, `no`, or leave unset to disable.

## Version Switching

The image includes a built-in version switching script `/opt/opensandbox/code-interpreter-env.sh`. You need to use the
Expand Down
5 changes: 5 additions & 0 deletions sandboxes/code-interpreter/README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
- **版本切换**:无需重新构建,支持运行时快速切换版本
- **Jupyter 集成**:内置 Jupyter Notebook 并支持多语言内核
- **多架构支持**:同时支持 amd64 和 arm64 架构
- **clone3-workaround(仅 amd64)**:在 **linux/amd64** 镜像中安装 [AkihiroSuda/clone3-workaround](https://github.com/AkihiroSuda/clone3-workaround) v1.0.0 至 `/usr/local/bin/clone3-workaround`(上游无 arm64 预编译包),并安装 **`libseccomp2`**(上游二进制动态链接 `libseccomp`)。在极旧 Docker/containerd 宿主机上可用其包裹命令,例如 `clone3-workaround apt-get update`。
- **生产就绪**:针对容器化执行环境进行了优化

## 支持的语言与版本
Expand Down Expand Up @@ -57,6 +58,10 @@ docker run -it --rm \
sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:latest
```

### `EXECD_CLONE3_COMPAT`(clone3-workaround)

若将 `EXECD_CLONE3_COMPAT` 设为 `1`、`true`、`yes`、`on` 或 `reexec`(与 [execd](../../components/execd/README_zh.md#沙箱内的-linux-clone3-兼容) 一致),入口脚本会在启动 Jupyter/内核前用 **`/usr/local/bin/clone3-workaround` 重新 `exec` 自身**。**linux/amd64** 镜像内含该二进制;**arm64** 构建会打印警告并跳过包装。包装成功后脚本会在当前进程树中 **`unset` `EXECD_CLONE3_COMPAT`**。设为 `0`、`false`、`off`、`no` 或不设置则关闭此逻辑。

## 如何切换版本

镜像内置了一个环境切换脚本 `/opt/opensandbox/code-interpreter-env.sh`,你需要使用 `source` 命令加载它来修改当前 Shell
Expand Down
Loading
Loading