feat: 5505 Harden Python custom logic block sandbox & extend libraries

* feat: 5504 Add scikit-learn, xarray, geopandas to Python custom logic block
Extend supported Python libraries per issue #5504.

Added libraries:
- scikit-learn: machine learning (classification, regression, clustering)
- xarray: labeled multi-dimensional arrays for climate/environmental data
- geopandas: geospatial DataFrames with geometry and spatial operations

Already available (no install needed):
- calendar, datetime, collections, math, copy: Python built-ins
- dateutil, six: transitive dependencies of pandas

Not available in Pyodide (WASM):
- rasterio: depends on GDAL (C/C++ library not compiled to WASM)
- rioxarray: depends on rasterio
Workaround: pre-process raster data outside the block (e.g. convert GeoTIFF to CSV/JSON) and pass as input documents.

* feat: 5505 Harden Python custom logic block sandbox
- Replace js module with restricted stub (blocks from js import fetch and all JS bridge access, survives re-import attempts)
- Replace pyodide.http with restricted stub (prevents re-import)
- Block all os.exec*/os.spawn*/os.system/os.popen functions
- Block subprocess.run/call/check_call/check_output/Popen (module remains importable for library compatibility)
- Install sys.meta_path import hook to prevent bypassing module restrictions via __import__ or importlib
- Remove unnecessary libraries: duckdb, sqlalchemy, bokeh, altair, cartopy, seaborn (matplotlib remains as transitive dep of networkx)

* feat: experimental Docker container isolation for Python custom logic block
Add Docker-based sandbox for Python code execution in custom logic blocks. Set PYTHON_SANDBOX_MODE=docker to enable (default is Pyodide worker for backward compatibility).

Container security (no resource limits — matching develop):
- --network=none, --cap-drop=ALL, --security-opt=no-new-privileges
- --read-only, --user=1001:1001 (non-root)
- --name=python-sandbox-<uuid> (named for cleanup)
- --log-driver=none, --pull=never
- --tmpfs /tmp:rw,noexec,nosuid,size=64m
- Image name validation (regex)

Defense-in-depth Python sandbox (both paths):
- js/pyodide.http stubs + import hook
- builtins.__import__ guarded via closure (hides _original_import)
- os.system/exec*/spawn*/popen, subprocess.run/call/Popen blocked
- os.environ cleared, importlib.reload blocked
- ctypes/cffi/_posixsubprocess import blocked
- processLine checks settled before firing callbacks

Pyodide worker improvements:
- Timeout (PYTHON_SANDBOX_TIMEOUT_MS, default 120s)
- worker.on('exit') rejects on non-zero exit code
- safeResolve/safeReject prevent double settlement
- disposeTables() called on all exit/error paths

Docker worker: promise-only errors, settled guards in all callback paths, processLine helper, stdin error handling, done(final=true) tracking, package load failure reporting, non-blocking cleanup.

Bug fixes: debug field (data.message->data.result), command injection, disposeTables in error paths, __globals__ bypass, pint removed.

* docs: update Python implementation guide with new libraries and sandbox security

Update supported libraries list: add scikit-learn, xarray, geopandas.
Document removed libraries (duckdb, sqlalchemy, pint, bokeh, altair, cartopy, seaborn) with reasons. Add sandbox security section covering blocked operations and execution modes (Pyodide default, Docker experimental). Document built-in modules and transitive dependencies.

* feat: replace Pyodide/WASM with CPython in Docker sandbox, harden Pyodide worker

Docker sandbox: replace Node.js + Pyodide (WASM) with native CPython 3.12. Same JSON stdin/stdout protocol — zero changes in host-side Docker worker. Benefits: <1s startup (was 30-60s), ~300MB memory (was 2-4GB), native speed, rasterio/rioxarray now available.

Pyodide worker hardening:
- Block socket networking functions (socket.socket, create_connection, getaddrinfo, gethostbyname, etc.)
- Update import hook to PEP 451 API (find_spec)
- Extract shared package list to python-packages.json

Both paths:
- Accumulate pendingDone promises via array + Promise.all (fixes race where multiple done() calls could lose in-flight work)
- Smart JSON serializer for numpy/pandas/datetime types in CPython
- Fix DockerCallbacks.onDone type to Promise<void> | void
- Remove unused traceback import

* docs: comprehensive Python implementation guide with Docker mode and security

Update python-implementation-in-guardian.md with:
- Library versions for all installed packages
- Docker-only libraries (rasterio, rioxarray)
- Full Docker mode documentation (setup, benefits, security flags)
- Execution modes comparison (Pyodide vs Docker)
- Sandbox security details for both modes
- Vulnerability comparison table
- Configuration reference

* fix: prevent unhandled promise rejection when done() throws
Add .catch(safeReject) to pendingDones promises so that errors from done() (e.g. invalid output schema) are caught instead of becoming unhandled promise rejections that crash the process.

In develop, these errors are caught by the worker message handler's try/catch → reject(). With pendingDones pattern, the promise could reject after the exit handler already resolved, causing a crash.

* feat: add python-sandbox config to all docker-compose files, fix Pyodide warmup permissions
- Add commented-out python-sandbox service and docker.sock volume to docker-compose.yml, docker-compose-production.yml, docker-compose-production-build.yml, docker-compose-quickstart.yml
- Comment out python-sandbox in docker-compose-build.yml for consistency (default mode is Pyodide, docker.sock should not be mounted by default)
- Fix EACCES permission denied in Pyodide warmup: chown pyodide dir to node user in policy-service Dockerfile
- Update python-implementation-in-guardian.md with all compose files

* docs: clarify python-sandbox is an image build, not a service
---------
Signed-off-by: nikolay-zezin <Nikolay.Zezin@waveaccess.global>

Author: nikolay-zezin

docker-compose-build.yml

@@ -211,10 +211,19 @@ services:
     volumes:
       - ./policy-service/tls:/usr/local/app/tls:ro
       - ./policy-service/configs:/usr/local/app/configs:ro
+      # Uncomment for PYTHON_SANDBOX_MODE=docker:
+      # - /var/run/docker.sock:/var/run/docker.sock:ro
     <<: *service-template
     expose:
       - '5006'
 
+  # Uncomment for PYTHON_SANDBOX_MODE=docker:
+  # python-sandbox:
+  #   build:
+  #     context: ./policy-service/docker/python-sandbox
+  #     dockerfile: Dockerfile
+  #   image: guardian/python-sandbox:latest
+
   prometheus:
     image: prom/prometheus:v2.44.0
     volumes:

docker-compose-production-build.yml

@@ -194,10 +194,19 @@ services:
     volumes:
       - ./policy-service/tls:/usr/local/app/tls:ro
       - ./policy-service/configs:/usr/local/app/configs:ro
+      # Uncomment for PYTHON_SANDBOX_MODE=docker:
+      # - /var/run/docker.sock:/var/run/docker.sock:ro
     <<: *service-template
     expose:
       - '5006'
 
+  # Uncomment for PYTHON_SANDBOX_MODE=docker:
+  # python-sandbox:
+  #   build:
+  #     context: ./policy-service/docker/python-sandbox
+  #     dockerfile: Dockerfile
+  #   image: guardian/python-sandbox:latest
+
   prometheus:
     image: prom/prometheus:v2.44.0
     volumes:

docker-compose-production.yml

@@ -177,10 +177,19 @@ services:
     volumes:
       - ./policy-service/tls:/usr/local/app/tls:ro
       - ./policy-service/configs:/usr/local/app/configs:ro
+      # Uncomment for PYTHON_SANDBOX_MODE=docker:
+      # - /var/run/docker.sock:/var/run/docker.sock:ro
     <<: *service-template
     expose:
       - '5006'
 
+  # Uncomment for PYTHON_SANDBOX_MODE=docker:
+  # python-sandbox:
+  #   build:
+  #     context: ./policy-service/docker/python-sandbox
+  #     dockerfile: Dockerfile
+  #   image: guardian/python-sandbox:latest
+
   prometheus:
     image: prom/prometheus:v2.44.0
     volumes:

docker-compose-quickstart.yml

@@ -133,10 +133,19 @@ services:
     volumes:
       - ./policy-service/tls:/usr/local/app/tls:ro
       - ./policy-service/configs:/usr/local/app/configs:ro
+      # Uncomment for PYTHON_SANDBOX_MODE=docker:
+      # - /var/run/docker.sock:/var/run/docker.sock:ro
     <<: *service-template
     expose:
       - '5006'
 
+  # Uncomment for PYTHON_SANDBOX_MODE=docker:
+  # python-sandbox:
+  #   build:
+  #     context: ./policy-service/docker/python-sandbox
+  #     dockerfile: Dockerfile
+  #   image: guardian/python-sandbox:latest
+
   queue-service:
     image: gcr.io/hedera-registry/queue-service:${GUARDIAN_VERSION:-latest}
     depends_on:

docker-compose.yml

@@ -198,10 +198,19 @@ services:
     volumes:
       - ./policy-service/tls:/usr/local/app/tls:ro
       - ./policy-service/configs:/usr/local/app/configs:ro
+      # Uncomment for PYTHON_SANDBOX_MODE=docker:
+      # - /var/run/docker.sock:/var/run/docker.sock:ro
     <<: *service-template
     expose:
       - '5006'
 
+  # Uncomment for PYTHON_SANDBOX_MODE=docker:
+  # python-sandbox:
+  #   build:
+  #     context: ./policy-service/docker/python-sandbox
+  #     dockerfile: Dockerfile
+  #   image: guardian/python-sandbox:latest
+
   prometheus:
     image: prom/prometheus:v2.44.0
     volumes:

docs/guardian/standard-registry/policies/python-implementation-in-guardian.md

@@ -25,7 +25,7 @@ A new dropdown setting has been added to the Custom Logic block in the Policy Ed
 
 #### Use Case
 
-Choose "Python" when you want to leverage Python’s expressive syntax and advanced computation libraries for policy logic.
+Choose "Python" when you want to leverage Python's expressive syntax and advanced computation libraries for policy logic.
 
 ### 2. Python Scripting Support
 
@@ -69,23 +69,172 @@ This field helps track the Guardian system version that was used to generate or
 * Python execution is subject to the limitations and security constraints defined in Guardian's runtime.
 {% endhint %}
 
-### 4. Supported Python Libraries and its Versions
-
-| Library Name | Version |
-| :----------: | :-----: |
-|     numpy    |  1.26.4 |
-|     scipy    |  1.12.0 |
-|     sympy    |   1.12  |
-|    pandas    |  2.2.0  |
-|     pint     |  0.25.1 |
-|    duckdb    |  1.0.0  |
-|  sqlalchemy  |  2.0.29 |
-|    cftime    |  1.6.3  |
-|  matplotlib  |  3.5.2  |
-|    seaborn   |  0.13.2 |
-|     bokeh    |  3.4.1  |
-|    altair    |  5.3.0  |
-|    cartopy   |  0.23.0 |
-|    astropy   |  6.0.1  |
-|  statsmodels |  0.14.2 |
-|   networkx   |   3.3   |
+### 4. Supported Python Libraries
+
+#### Installed Libraries
+
+| Library Name | Import Name | Version |
+| :----------: | :---------: | :-----: |
+| numpy | `numpy` | 1.26.4 |
+| scipy | `scipy` | 1.12.0 |
+| sympy | `sympy` | 1.12 |
+| pandas | `pandas` | 2.2.0 |
+| pint | `pint` | 0.25.3 |
+| cftime | `cftime` | 1.6.3 |
+| astropy | `astropy` | 6.0.1 |
+| statsmodels | `statsmodels` | 0.14.2 |
+| networkx | `networkx` | 3.3 |
+| scikit-learn | `sklearn` | 1.4.2 |
+| xarray | `xarray` | 2024.3.0 |
+| geopandas | `geopandas` | 0.14.3 |
+
+{% hint style="info" %}
+Library versions listed are for the default Pyodide mode. Docker mode may have newer versions as it uses native CPython with pip.
+{% endhint %}
+
+#### Docker-Only Libraries
+
+These libraries require native C/C++ dependencies (GDAL) and are only available in Docker mode:
+
+| Library Name | Import Name | Purpose |
+| :----------: | :---------: | :------ |
+| rasterio | `rasterio` | Read/write raster geospatial data (GeoTIFF, satellite imagery) |
+| rioxarray | `rioxarray` | Bridge between xarray and rasterio — CRS management, reprojection |
+
+#### Python Built-in Modules (always available)
+
+| Module | Purpose |
+| :----: | :------ |
+| `calendar` | Calendar rendering, weekday calculations |
+| `datetime` | Date/time types and arithmetic |
+| `collections` | OrderedDict, Counter, defaultdict, namedtuple |
+| `math` | Basic math functions (sin, log, sqrt, pi) |
+| `copy` | Deep/shallow copy of objects |
+
+#### Available as Transitive Dependencies (no explicit install needed)
+
+| Library | Import Name | Purpose | Installed via |
+| :-----: | :---------: | :------ | :------------ |
+| python-dateutil | `dateutil` | Smart date parsing, relative deltas | pandas |
+| six | `six` | Python 2/3 compatibility | pandas → python-dateutil |
+| matplotlib | `matplotlib` | Data visualization | networkx (transitive) |
+
+#### Removed Libraries (Issue #5505)
+
+The following libraries were removed as part of sandbox hardening. They are unnecessary for computation — their data processing features are covered by pandas, and they were designed to work with external resources (databases, networks, web servers) that are not available in the sandbox.
+
+| Library | Reason for Removal |
+| :-----: | :----------------- |
+| duckdb | SQL database engine; covered by pandas |
+| sqlalchemy | SQL toolkit/ORM; covered by pandas |
+| bokeh | Visualization; unnecessary for computation |
+| altair | Visualization; unnecessary for computation |
+| cartopy | Map visualization; unnecessary for computation |
+| seaborn | Visualization; unnecessary for computation |
+
+### 5. Execution Modes
+
+Guardian supports two execution modes for Python custom logic blocks, controlled by the `PYTHON_SANDBOX_MODE` environment variable.
+
+#### Pyodide Mode (default)
+
+The default mode runs Python code using Pyodide (CPython compiled to WebAssembly) inside a Node.js Worker Thread.
+
+* **No additional infrastructure required** — works out of the box
+* **Startup:** packages are pre-cached at policy-service startup for faster execution
+* **Limitation:** some C-extension packages (rasterio, rioxarray) are unavailable in WASM
+
+**Configuration:** No env var needed (default), or explicitly set `PYTHON_SANDBOX_MODE=pyodide`
+
+#### Docker Mode (experimental)
+
+Runs Python code in an ephemeral Docker container using native CPython 3.12. Provides OS-level isolation.
+
+**Container security flags:**
+
+| Flag | Purpose |
+| :--- | :------ |
+| `--network=none` | All network access blocked |
+| `--cap-drop=ALL` | No Linux capabilities |
+| `--security-opt=no-new-privileges` | Prevent privilege escalation |
+| `--read-only` | Read-only root filesystem |
+| `--user=1001:1001` | Non-root execution |
+| `--log-driver=none` | No container log storage |
+| `--pull=never` | Never pull untrusted images |
+| `--tmpfs /tmp` | Writable scratch space (noexec, destroyed on exit) |
+
+**Setup:**
+
+1. Build the sandbox image:
+```bash
+docker buildx build -t guardian/python-sandbox:latest policy-service/docker/python-sandbox
+```
+Or via docker-compose:
+```bash
+docker compose -f docker-compose-build.yml build python-sandbox
+```
+
+2. Set the environment variable in policy-service configuration:
+```
+PYTHON_SANDBOX_MODE=docker
+```
+
+3. Ensure the policy-service container has Docker socket access. For docker-compose deployments, uncomment the Docker socket volume mount and the `python-sandbox` image build definition in the relevant compose file:
+   - `docker-compose-build.yml`, `docker-compose.yml`, `docker-compose-production.yml`, `docker-compose-production-build.yml`, `docker-compose-quickstart.yml` — uncomment the Docker socket volume and `python-sandbox` image build
+
+{% hint style="warning" %}
+Docker mode requires the Docker daemon to be available. The policy-service needs access to the Docker socket to spawn sandbox containers. For production deployments, consider using a Docker API proxy to restrict operations to sandbox container management only.
+{% endhint %}
+
+### 6. Sandbox Security
+
+Python code in custom logic blocks runs in a sandboxed environment. The following restrictions are enforced:
+
+#### Pyodide Mode Restrictions
+
+| Restriction | Details |
+| :---------- | :----- |
+| JavaScript bridge (`from js import ...`) | Blocked via module stub + import hook |
+| `pyodide.http` network access | Blocked via module stub + import hook |
+| `os.system`, `os.popen`, `os.exec*`, `os.spawn*` | All replaced with blocked function |
+| `subprocess.run`, `subprocess.Popen` | All execution functions replaced |
+| `socket.socket`, `socket.connect` | All networking functions replaced |
+| `os.environ` (secrets) | Cleared on startup (only HOME/PATH kept) |
+| `importlib.reload` | Blocked to prevent undoing patches |
+| `builtins.__import__` | Guarded via closure to prevent bypass |
+| Execution timeout | Configurable via `PYTHON_SANDBOX_TIMEOUT_MS` (default 120s) |
+
+#### Docker Mode Restrictions
+
+All restrictions above are provided by Docker container isolation:
+
+* **Network:** `--network=none` blocks all connections (verified: HTTP requests fail)
+* **File system:** `--read-only` + no host mounts — container sees only its own minimal filesystem
+* **Processes:** commands run inside isolated container only, destroyed after execution
+* **Environment:** `os.environ` cleared before user code runs
+* **Resources:** container destroyed with `--rm` after each execution
+
+#### Vulnerability Comparison
+
+| Attack Vector | Pyodide Mode | Docker Mode |
+| :------------ | :----------- | :---------- |
+| Network requests | Blocked (Python-level) | Blocked (OS-level `--network=none`) |
+| Host filesystem access | Blocked (WASM virtual FS) | Blocked (`--read-only`, no mounts) |
+| Process execution | Blocked (functions replaced) | Runs inside isolated container |
+| `os.environ` secrets | Cleared | Cleared + container has own env |
+| `ctypes` C function calls | Not blocked (needed by pandas, harmless in WASM) | Runs inside isolated container |
+| Python introspection bypass | Possible (known limitation) | Irrelevant — container is isolated |
+| Memory/CPU exhaustion | Timeout only | Timeout + container destroyed |
+
+{% hint style="info" %}
+* **Pyodide mode** is suitable when users are trusted or semi-trusted. It blocks common attack vectors but is vulnerable to sophisticated Python introspection attacks.
+* **Docker mode** is suitable for untrusted code. OS-level isolation makes Python-level bypasses irrelevant — the container has no network, no host access, and is destroyed after execution.
+{% endhint %}
+
+### 7. Configuration Reference
+
+| Environment Variable | Default | Description |
+| :------------------- | :------ | :---------- |
+| `PYTHON_SANDBOX_MODE` | `pyodide` | Execution mode: `pyodide` (default) or `docker` |
+| `PYTHON_SANDBOX_TIMEOUT_MS` | `120000` | Execution timeout in milliseconds (both modes) |
+| `PYTHON_SANDBOX_IMAGE` | `guardian/python-sandbox:latest` | Docker sandbox image name (Docker mode only) |

policy-service/Dockerfile

@@ -54,6 +54,9 @@ COPY --link --from=deps /usr/local/app/node_modules node_modules/
 COPY --link --from=deps /usr/local/app/package.json ./
 COPY --link --from=build /usr/local/app/dist dist/
 
+# Allow node user to write Pyodide package cache (warmup downloads wheels at startup)
+RUN chown -R node:node node_modules/pyodide/ 2>/dev/null || true
+
 # Change the user to node
 USER node
 

policy-service/docker/python-sandbox/.dockerignore

@@ -0,0 +1,6 @@
+node_modules
+npm-debug.log
+.git
+.gitignore
+README.md
+*.md

policy-service/docker/python-sandbox/Dockerfile

@@ -0,0 +1,21 @@
+FROM python:3.12.10-slim
+
+WORKDIR /sandbox
+
+# Install system dependencies for geospatial libraries
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    gdal-bin libgdal-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Python packages (pinned to major.minor for reproducibility)
+COPY requirements.txt ./
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy entrypoint
+COPY entrypoint.py ./
+
+# Create non-root user and fix permissions
+RUN adduser --disabled-password --uid 1001 sandbox && chown -R sandbox:sandbox /sandbox
+USER sandbox
+
+ENTRYPOINT ["python3", "entrypoint.py"]