Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/real-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ jobs:
execd_image = "opensandbox/execd:local"
[egress]
image = "opensandbox/egress:local"
mode = "dns"
[docker]
network_mode = "bridge"
[storage]
Expand Down Expand Up @@ -144,6 +145,7 @@ jobs:
execd_image = "opensandbox/execd:local"
[egress]
image = "opensandbox/egress:local"
mode = "dns+nft"
[docker]
network_mode = "bridge"
[storage]
Expand Down
4 changes: 4 additions & 0 deletions kubernetes/charts/opensandbox-server/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,8 @@ configToml: |
informer_watch_timeout_seconds = 60
workload_provider = "batchsandbox"
batchsandbox_template_file = "/etc/opensandbox/example.batchsandbox-template.yaml"

[egress]
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.0.3"
mode = "dns+nft"

105 changes: 71 additions & 34 deletions server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,38 +255,74 @@ EOF
- Informer settings are **beta** and enabled by default to reduce API calls; set `informer_enabled = false` to turn off.
- Resync and watch timeouts control how often the cache refreshes; tune for your cluster API limits.

### Egress sidecar for `networkPolicy`
### Egress configuration

- **Required when using `networkPolicy`**: Configure the sidecar image. The `egress.image` setting is mandatory when requests include `networkPolicy`:
```toml
[runtime]
type = "docker"
execd_image = "opensandbox/execd:v1.0.7"

[egress]
image = "opensandbox/egress:v1.0.3"
```
- Supported only in Docker bridge mode; requests with `networkPolicy` are rejected when `network_mode=host` or when `egress.image` is not configured.
- Main container shares the sidecar netns and explicitly drops `NET_ADMIN`; the sidecar keeps `NET_ADMIN` to manage iptables.
- IPv6 is disabled in the shared namespace when the egress sidecar is injected to keep policy enforcement consistent.
- Sidecar image is pulled before start; delete/expire/failure paths attempt to clean up the sidecar as well.
- Request example (`CreateSandboxRequest` with `networkPolicy`):
```json
{
"image": {"uri": "python:3.11-slim"},
"entrypoint": ["python", "-m", "http.server", "8000"],
"timeout": 3600,
"resourceLimits": {"cpu": "500m", "memory": "512Mi"},
"networkPolicy": {
"defaultAction": "deny",
"egress": [
{"action": "allow", "target": "pypi.org"},
{"action": "allow", "target": "*.python.org"}
]
}
}
```
- When `networkPolicy` is empty or omitted, no sidecar is injected (allow-all at start).
The **`[egress]`** block configures the **egress sidecar** image and enforcement mode. The server only starts this sidecar when a sandbox is created **with** a `networkPolicy` (outbound allow/deny rules). If the create request omits `networkPolicy`, no egress sidecar is added and outbound traffic is not restricted by this mechanism.

#### Keys

| Key | Type | Default | Required | Description |
|-----|------|---------|----------|-------------|
| `image` | string | — | **Yes** whenever `networkPolicy` is used in a create request | OCI image containing the egress binary. Pulled before the sidecar starts. |
| `mode` | `dns` or `dns+nft` | `dns` | No | How the sidecar enforces policy. Written to the sidecar as `OPENSANDBOX_EGRESS_MODE` (see below). |

#### `mode` values

- **`dns`**: DNS-based enforcement via the in-sidecar DNS proxy. No nftables layer-2 rules from this path. **CIDR and static IP targets in the policy are not enforced** (use domain-style rules only if you rely on `dns` mode).
- **`dns+nft`**: Same DNS path, plus nftables where available (see the [egress component README](../components/egress/README.md) for capabilities and fallbacks). **CIDR and static IP allow/deny rules are supported** via nftables when the table is applied successfully.

#### Per-request `networkPolicy`

- Rules are defined on **`CreateSandboxRequest.networkPolicy`** (default action and ordered egress rules: hostnames / patterns, and IP or CIDR entries when using **`dns+nft`**).
- The serialized policy is passed into the sidecar as **`OPENSANDBOX_EGRESS_RULES`** (JSON).
- An auth token may be attached for the egress HTTP API; see runtime behavior below.

#### Docker runtime

- **`egress.image` must be set** in config when clients send `networkPolicy`; otherwise the request is rejected.
- Outbound policy requires **`docker.network_mode = "bridge"`**. Requests with `networkPolicy` are rejected for `network_mode=host` or for user-defined Docker networks that are incompatible with the sidecar attachment model.
- The main sandbox container shares the sidecar’s network namespace, **drops `NET_ADMIN`**, and relies on the sidecar for policy; the sidecar **keeps `NET_ADMIN`**.
- **IPv6** is disabled in the shared namespace so allow/deny behavior stays consistent.

#### Kubernetes runtime

- When `networkPolicy` is present, the workload pod includes an **egress** sidecar built from `egress.image`, in addition to the main sandbox container.
- **`egress.image`** is required in the same way as for Docker.

#### Operational notes

- The sidecar image is pulled (or validated) before start; delete, expiry, and failure paths attempt to remove the sidecar.
- For deeper behavior (DNS proxy, nftables, limits), refer to the **egress** component documentation under `components/egress/`.

#### Example (`~/.sandbox.toml`)

```toml
[runtime]
type = "docker"
execd_image = "opensandbox/execd:v1.0.7"

[egress]
image = "opensandbox/egress:v1.0.3"
mode = "dns"
```

#### Example create request with `networkPolicy`

```json
{
"image": {"uri": "python:3.11-slim"},
"entrypoint": ["python", "-m", "http.server", "8000"],
"timeout": 3600,
"resourceLimits": {"cpu": "500m", "memory": "512Mi"},
"networkPolicy": {
"defaultAction": "deny",
"egress": [
{"action": "allow", "target": "pypi.org"},
{"action": "allow", "target": "*.python.org"}
]
}
}
```

### Run the server

Expand Down Expand Up @@ -501,9 +537,10 @@ curl -X DELETE \

### Egress configuration

| Key | Type | Required | Description |
|---------------|--------|----------|--------------------------------|
| `egress.image` | string | **Required when using `networkPolicy`** | Container image with egress binary. Must be configured when `networkPolicy` is provided in sandbox creation requests. |
| Key | Type | Default | Required if using `networkPolicy` | Description |
|-----|------|---------|-----------------------------------|-------------|
| `egress.image` | string | — | Yes | Egress sidecar image (OCI reference). |
| `egress.mode` | `dns` \| `dns+nft` | `dns` | No | `OPENSANDBOX_EGRESS_MODE`. CIDR/IP rules need `dns+nft`; `dns` is domain-oriented only. |

### Docker configuration

Expand Down
62 changes: 49 additions & 13 deletions server/README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,23 +231,59 @@ mode = "direct" # Docker 运行时仅支持 direct(直连,无 L7 网关)
- Informer 配置为 **Beta**,默认开启以减少 API 压力;若需关闭设置 `informer_enabled = false`。
- resync / watch 超时用于控制缓存刷新频率,可根据集群 API 限流调优。

### Egress sidecar 配置与使用
### Egress 配置(`[egress]` 配置块)

**`[egress]`** 用于配置 **egress 侧车** 的镜像与执行模式。仅当创建沙箱的请求中带有 **`networkPolicy`**(出站允许/拒绝规则)时,服务器才会注入该侧车;若请求未带 `networkPolicy`,不会添加 egress 侧车,也不会通过该机制限制出站流量。

#### 配置项

| 键 | 类型 | 默认值 | 何时必填 | 说明 |
|----|------|--------|----------|------|
| `image` | string | — | 任意一次创建请求携带 `networkPolicy` 时 **必填** | 包含 egress 可执行文件的容器镜像;侧车启动前会拉取或校验镜像。 |
| `mode` | `dns` 或 `dns+nft` | `dns` | 否 | 侧车如何执行策略,写入环境变量 `OPENSANDBOX_EGRESS_MODE`(见下)。 |

#### `mode` 取值

- **`dns`**:通过侧车内 DNS 代理做基于域名的策略;不依赖本路径下的 nftables 二层规则。**策略中的 CIDR、静态 IP 类目标不会被强制执行**(若只用 `dns` 模式,请使用域名类规则)。
- **`dns+nft`**:在 `dns` 的基础上启用 nftables(能力与回退行为见 [egress 组件说明](../components/egress/README.md))。**支持 CIDR 与静态 IP 的放行/拒绝规则**(nftables 表成功下发时生效)。

#### 请求体中的 `networkPolicy`

- 规则在 **`CreateSandboxRequest.networkPolicy`** 中声明(默认动作与有序的 egress 规则:域名/通配符;在使用 **`dns+nft`** 时还可包含 IP 或 CIDR 条目)。
- 序列化后的策略以 JSON 形式注入侧车环境变量 **`OPENSANDBOX_EGRESS_RULES`**。
- 可能同时下发用于 egress HTTP API 的鉴权信息(与运行时行为一致)。

#### Docker 运行时

- 客户端传入 `networkPolicy` 时,配置中必须设置 **`egress.image`**,否则请求会被拒绝。
- 出站策略要求 **`docker.network_mode = "bridge"`**;`network_mode=host` 或与侧车挂载模型不兼容的用户自定义网络下,携带 `networkPolicy` 的请求会被拒绝。
- 主沙箱容器与侧车 **共享网络命名空间**,主容器 **drop `NET_ADMIN`**,由侧车保留 **`NET_ADMIN`** 完成策略相关操作。
- 共享 netns 内会 **禁用 IPv6**,以保证放行/拒绝行为一致。

#### Kubernetes 运行时

- 当请求带有 `networkPolicy` 时,工作负载 Pod 中除主容器外,还会增加基于 **`egress.image`** 的 **egress** 侧车。
- **`egress.image`** 的必填规则与 Docker 相同。

#### 运维说明

- 侧车镜像在启动前拉取或校验;删除、过期、失败等路径会尽量清理侧车。
- DNS 代理、nftables、能力边界等详见仓库内 **`components/egress/`** 文档。

#### 配置示例(`~/.sandbox.toml`)

- **使用 `networkPolicy` 时必需**:配置 sidecar 镜像。当请求携带 `networkPolicy` 时,`egress.image` 配置项是必需的:
```toml
[runtime]
type = "docker"
execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.7"
execd_image = "opensandbox/execd:v1.0.7"

[egress]
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.0.3"
image = "opensandbox/egress:v1.0.3"
mode = "dns"
```

- 仅支持 Docker bridge 模式(`network_mode=host` 时会拒绝携带 `networkPolicy` 的请求,或当 `egress.image` 未配置时也会拒绝)。
- 主容器共享 sidecar 网络命名空间,主容器会显式 drop `NET_ADMIN`,sidecar 保留 `NET_ADMIN` 完成 iptables。
- 注入 sidecar 时会在共享 netns 内默认禁用 IPv6,以保持策略生效一致性。
- 侧车镜像会在启动前自动拉取;删除/过期/失败时会尝试同步清理 sidecar。
- 请求体示例(`CreateSandboxRequest` 中携带 `networkPolicy`):
#### 带 `networkPolicy` 的创建请求示例

```json
{
"image": {"uri": "python:3.11-slim"},
Expand All @@ -263,7 +299,6 @@ image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.0
}
}
```
- `networkPolicy` 为空/缺省时不注入 sidecar,默认 allow-all。

### 启动服务

Expand Down Expand Up @@ -477,9 +512,10 @@ curl -X DELETE \

### Egress 配置

| 键 | 类型 | 必需 | 描述 |
|---------------|--------|----|--------------------------------|
| `egress.image` | string | **使用 `networkPolicy` 时必需** | 包含 egress 二进制文件的容器镜像。当创建沙箱的请求中包含 `networkPolicy` 时,必须配置此项。 |
| 键 | 类型 | 默认值 | 使用 `networkPolicy` 时是否必填 | 说明 |
|----|------|--------|--------------------------------|------|
| `egress.image` | string | — | 是 | Egress 侧车镜像(OCI 引用)。 |
| `egress.mode` | `dns` \| `dns+nft` | `dns` | 否 | `OPENSANDBOX_EGRESS_MODE`。CIDR/IP 类规则需 `dns+nft`;`dns` 仅面向域名类策略。 |

### Docker 配置

Expand Down
7 changes: 7 additions & 0 deletions server/example.config.k8s.toml
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,10 @@ batchsandbox_template_file = "~/batchsandbox-template.yaml"
[ingress]
# Ingress exposure mode: direct (default) or gateway
mode = "direct"

[egress]
# Egress configuration
# -----------------------------------------------------------------
image = "opensandbox/egress:v1.0.3"
# Enforcement: "dns" (DNS proxy only) or "dns+nft" (nftables + DNS).
mode = "dns"
7 changes: 7 additions & 0 deletions server/example.config.k8s.zh.toml
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,10 @@ batchsandbox_template_file = "~/batchsandbox-template.yaml"
[ingress]
# Ingress exposure mode: direct (default) or gateway
mode = "direct"

[egress]
# Egress configuration
# -----------------------------------------------------------------
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.0.3"
# Enforcement: "dns" (DNS proxy only) or "dns+nft" (nftables + DNS).
mode = "dns"
2 changes: 2 additions & 0 deletions server/example.config.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ execd_image = "opensandbox/execd:v1.0.7"
# Egress configuration
# -----------------------------------------------------------------
image = "opensandbox/egress:v1.0.3"
# Enforcement: "dns" (DNS proxy only) or "dns+nft" (nftables + DNS).
mode = "dns"

[storage]
# Volume and storage configuration
Expand Down
2 changes: 2 additions & 0 deletions server/example.config.zh.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd
# Egress configuration
# -----------------------------------------------------------------
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.0.3"
# Enforcement: "dns" (DNS proxy only) or "dns+nft" (nftables + DNS).
mode = "dns"

[storage]
# 卷存储配置
Expand Down
12 changes: 12 additions & 0 deletions server/src/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,9 @@
GATEWAY_ROUTE_MODE_HEADER = "header"
GATEWAY_ROUTE_MODE_URI = "uri"

EGRESS_MODE_DNS = "dns"
EGRESS_MODE_DNS_NFT = "dns+nft"


def _is_valid_ip(host: str) -> bool:
try:
Expand Down Expand Up @@ -350,6 +353,13 @@ class EgressConfig(BaseModel):
description="Container image for the egress sidecar (used when network policy is requested).",
min_length=1,
)
mode: Literal[
EGRESS_MODE_DNS,
EGRESS_MODE_DNS_NFT,
] = Field(
default=EGRESS_MODE_DNS,
description="Egress enforcement passed to the sidecar as OPENSANDBOX_EGRESS_MODE (dns or dns+nft).",
)


class RuntimeConfig(BaseModel):
Expand Down Expand Up @@ -617,6 +627,8 @@ def get_config_path() -> Path:
"StorageConfig",
"KubernetesRuntimeConfig",
"EgressConfig",
"EGRESS_MODE_DNS",
"EGRESS_MODE_DNS_NFT",
"SecureRuntimeConfig",
"DEFAULT_CONFIG_PATH",
"CONFIG_ENV_VAR",
Expand Down
11 changes: 11 additions & 0 deletions server/src/services/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,14 @@
OPEN_SANDBOX_EGRESS_AUTH_HEADER = "OPENSANDBOX-EGRESS-AUTH"
SANDBOX_EGRESS_AUTH_TOKEN_METADATA_KEY = "opensandbox.io/egress-auth-token"

# Environment variable name for passing network policy to egress sidecar
EGRESS_RULES_ENV = "OPENSANDBOX_EGRESS_RULES"
# Must match components/egress/pkg/constants/configuration.go EnvEgressMode
EGRESS_MODE_ENV = "OPENSANDBOX_EGRESS_MODE"
# Must match components/egress/pkg/constants/configuration.go EnvEgressToken
OPENSANDBOX_EGRESS_TOKEN = "OPENSANDBOX_EGRESS_TOKEN"


class SandboxErrorCodes:
"""Canonical error codes for sandbox service operations."""

Expand Down Expand Up @@ -101,5 +109,8 @@ class SandboxErrorCodes:
"OPEN_SANDBOX_INGRESS_HEADER",
"OPEN_SANDBOX_EGRESS_AUTH_HEADER",
"SANDBOX_EGRESS_AUTH_TOKEN_METADATA_KEY",
"EGRESS_RULES_ENV",
"EGRESS_MODE_ENV",
"OPENSANDBOX_EGRESS_TOKEN",
"SandboxErrorCodes",
]
10 changes: 7 additions & 3 deletions server/src/services/docker.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@
)
from src.config import AppConfig, get_config
from src.services.constants import (
EGRESS_MODE_ENV,
EGRESS_RULES_ENV,
OPENSANDBOX_EGRESS_TOKEN,
SANDBOX_EGRESS_AUTH_TOKEN_METADATA_KEY,
SANDBOX_EMBEDDING_PROXY_PORT_LABEL,
SANDBOX_EXPIRES_AT_LABEL,
Expand Down Expand Up @@ -92,7 +95,6 @@
ensure_valid_host_path,
ensure_volumes_valid,
)

logger = logging.getLogger(__name__)


Expand All @@ -110,7 +112,6 @@ def _running_inside_docker_container() -> bool:
HOST_NETWORK_MODE = "host"
BRIDGE_NETWORK_MODE = "bridge"
PENDING_FAILURE_TTL_SECONDS = int(os.environ.get("PENDING_FAILURE_TTL", "3600"))
EGRESS_RULES_ENV = "OPENSANDBOX_EGRESS_RULES"
EGRESS_SIDECAR_LABEL = "opensandbox.io/egress-sidecar-for"


Expand Down Expand Up @@ -1989,9 +1990,12 @@ def _start_egress_sidecar(
self._ensure_image_available(egress_image, None, sandbox_id)

policy_payload = json.dumps(network_policy.model_dump(by_alias=True, exclude_none=True))
assert self.app_config.egress is not None # validated by ensure_egress_configured with networkPolicy
egress_mode = self.app_config.egress.mode
sidecar_env = [
f"{EGRESS_RULES_ENV}={policy_payload}",
f"OPENSANDBOX_EGRESS_TOKEN={egress_token}",
f"{EGRESS_MODE_ENV}={egress_mode}",
f"{OPENSANDBOX_EGRESS_TOKEN}={egress_token}",
]

sidecar_host_config_kwargs: dict[str, Any] = {
Expand Down
Loading
Loading