Initial Implementation Product Requirements Doc

[markurtz]

Rustarium Initial Product Requirements Document

• Document Version: 1.0
• Document Status: Draft
• Date: 2026-05-15

Executive Summary

Problem Statement

Executing untrusted agentic code, benchmarking LLM frameworks, and distributing High-Performance Computing (HPC) workloads typically requires managing disparate, heavy-weight solutions such as custom system daemons, complex cluster managers, or nested container environments. Developers and researchers lack a unified, lightweight framework that delivers strict execution isolation, minimal startup latency, and seamless integration into standard Python workflows without imposing significant infrastructure overhead.

Product Value Proposition

Rustarium is a high-performance, telemetrized process orchestrator and sandbox designed primarily for Python, and architected to seamlessly extend to WebAssembly (WASM) and other runtimes in the future. It provides a simple, zero-friction framework—accessible via a standard pip install—for submitting arbitrary computational tasks and mapping them to secure, ephemeral sandboxes or networked cluster nodes. By leveraging Rust's native performance, standardized interfaces for compute execution, deep telemetry, and direct OS/Kernel primitives, Rustarium offers a transparent, auditable, and highly reliable platform for process orchestration and distributed workload management.

Strategic Alignment

Rustarium targets the rapidly converging needs of Agentic workflows, HPC workloads, and evaluation/benchmarking scenarios. By enforcing a dichotomous security profile (strict isolation vs. trusted, high performance) and a robust local-first leader/follower orchestration model, the platform aligns with the strategic vision of delivering an elite, intuitive developer experience. This positions Rustarium as a foundational orchestration layer focused entirely on high-performance, sandboxed local execution for the MVP.

User Stories

• Agens Maintainer (Agentic OS)
  • Need: A lightweight process sandboxing framework accessible via a zero-dependency pip install.
  • Goal: Execute user-supplied code with absolute filesystem and network isolation, minimal process startup latency, and no requirement for external system daemons or root access.
• GuideLLM Maintainer (LLM Benchmarking)
  • Need: A framework to replace custom orchestration logic for evaluating LLM runtime performance characteristics.
  • Goal: Simulate real-world traffic/request patterns, evaluate LLM engines, and gather detailed telemetry/performance statistics.
  • Requirements: A Python API, long-lived/synchronized Python processes, iterable workloads, and generic object passing via IPC.
• Speculators Maintainer (Training Framework) [Phase 2]
  • Need: Replace custom multi-node communication logic that handles transferring large tensor payloads from teacher to student nodes.
  • Goal: Enable easy multi-node communication setup and high-performance IPC.
  • Requirements: A Python API, minimal overhead for passing large Python objects/iterables between nodes, and maximum configurability.
• Security Architect / Researcher
  • Need: Flexible, tiered security boundaries for isolated execution.
  • Goal: Enforce rigid, unprivileged isolation barriers by default, while allowing escalation to hardware-enforced resource ceilings (via cgroups/sudo) or delegation to host runtimes (Docker) when maximum containment is configured.
• System Administrator
  • Need: Comprehensive visibility into process execution across the system.
  • Goal: Gather detailed telemetry and performance statistics for all running processes to monitor system health and identify potential issues.

Requirements

Functional Requirements

The functional requirements are organized around the core pillars of the Rustarium platform, ensuring a robust, scalable, and secure orchestration experience for developers and system administrators.

1. Developer Interfaces (API & CLI)

The platform must expose intuitive interfaces that abstract away the complexity of distributed process orchestration. The ultimate goal is an elite developer experience: a simple pip install that immediately unlocks performant, Docker/VM-level process orchestration.

• Python SDK & "Pip Install" North Star: A fully-typed, easy-to-use Python API bundled as a pip installable package containing the Rust orchestration engine. It must provide deep integration within existing Python codebases with zero-friction installation.
• Job-Based Submission Model: Workloads must be treated as scoped job submissions. Users define an iterable source of data/workloads, coupled with process and workload-specific configurations.
• Declarative CLI & Environment Diagnostics: A command-line interface for rapid job submission, status monitoring, and configuration validation. Crucially, the CLI must include a rustarium doctor or rustarium setup utility to gracefully diagnose and guide users through system-level environmental requirements (e.g., cgroups v2 permissions, Docker socket access) during FTUX.
• Real-time Progress Updates: Standardized, asynchronous status updates must be streamed from running processes back to the caller for ongoing job submissions.

2. Workload & Process Orchestration

Rustarium must orchestrate process lifecycles and workload scheduling efficiently, utilizing native Rust performance while exposing high-level Python abstractions.

• Process Lifecycle Management: Full support for startup, shutdown, health-checking, and monitoring of long-lived, iterative, and synchronized processes.
  • Python Version Matrix: The orchestrator must support arbitrary Python interpreters found on the host system, seamlessly handling virtual environments (venv, conda) inside the Tier 1 sandbox.
  • The Poison Pill / Hard Throttling: For Agentic workflows executing untrusted code (e.g., infinite loops), the Rust layer must enforce hard CPU-time and Wall-time throttling, terminating unresponsive workers with a SIGKILL when ceilings are breached.
• Leader/Follower Topology (Local-First): The MVP architecture is strictly local-first. The specific parent Python/Rust process thread executing the API/SDK acts as the Leader. Followers are exclusively local, isolated subprocesses launched by the Leader. All multi-node mechanisms, including distributed consensus and generic control planes, are deferred to Phase 2.
• Data Pipeline Efficiency: The core engineering focus for execution must maximize the efficiency of the data pipeline—specifically how the Leader pushes workloads down to local followers and collects streaming telemetry/updates back over IPC.
• Advanced Scheduling Strategies: Out-of-the-box support for multiple workload scenarios:
  • Synchronous/Concurrent: Target active workload count with a configurable number of processes.
  • Constant Rate: Workloads evenly distributed across processes over time.
  • Burst / Throughput-Optimized: Instantaneous spin-up of maximum processes to process workloads as fast as possible.
  • Poisson Distribution: Randomized arrival rates simulating real-world traffic.
  • Priority-Based: Queue execution prioritizing critical workloads.
• Flexible Constraints & Stop Criteria: Job termination configurable by constraints such as maximum error rates, execution time limits, completion counts, resource consumption, or process uptime.
• Load Balancing & Custom Hooks: Built-in strategies for distributing workloads efficiently across processes, along with extensible hooks allowing developers to inject custom scheduling, orchestration, or load-balancing logic.

3. Inter-Process Communication (IPC)

The platform must seamlessly handle data transfer between the orchestrator and worker processes with minimal serialization overhead.

• Transparent IPC: Consumers interact purely with the job API; the underlying IPC mechanisms (Unix Domain Sockets) between the Rust daemon and worker processes are fully abstracted.
• Serialization Pipeline (MVP): For the MVP, all data transport relies strictly on serialized payloads over high-speed local Unix Domain Sockets (UDS). True zero-copy shared memory transport is deferred to Phase 2.
• Asynchronous Support: Native compatibility with Python async/await syntax for all communication interfaces.
• Auditable Channels: All IPC traffic to and from processes must be fully auditable, inspectable, and secure.

4. Security, Isolation & Sandboxing

Workers must run in secure, isolated environments to prevent accidental or malicious state changes from escaping to the host system.

• Multi-Tiered Sandboxing Architecture (Linux & macOS): The orchestration engine implements a tiered security abstraction. Windows support is deferred to Phase 2.
  • By default, it operates entirely in user space (Tier 1: Rootless). On Linux, this leverages unprivileged namespaces. On macOS, Tier 1 provides user-space isolation utilizing standard POSIX file permissions, basic chroot-equivalent environment boundaries, and virtual path remapping.
  • When executed with elevated system privileges on Linux (Tier 2: Elevated), the engine must automatically engage platform-native resource controllers—specifically cgroups v2—to guarantee absolute hardware ceilings. (Note: macOS does not support Tier 2).
  • When explicitly configured to do so, the system gracefully delegates heavy containment barriers to a local host-level container system (Tier 3: Delegated), interacting directly with the Docker Desktop daemon via local UNIX domain sockets.
• Security Profiles: Jobs must be explicitly declared under a security profile. The MVP focuses entirely on the SECURE_ISOLATED profile (Agentic workflows) offering fully restricted IPC and locked-down host access to prevent sandbox escapes. The HIGH_PERFORMANCE_TRUSTED profile (zero-copy memory mapping) is deferred to Phase 2.
• CWD Path Uniformity: The MVP sandbox filesystem prioritizes path emulation. The executing code within the sandbox must see and interact with paths as if it were running natively in the host’s Current Working Directory (CWD) from which the process was launched. Remapping host paths into a localized sandbox structure under the hood is acceptable, provided this layer is abstract and invisible to the running worker code.
• Emulated Write-Isolation (Scratchpad Strategy): The system must not rely on kernel-level OverlayFS. Instead, implement a lightweight user-space emulated sandbox filesystem. The engine exposes external host file locations as read-only streams. Any file modification, creation, or deletion performed by the worker process is transparently intercepted and routed to an isolated, temporary scratchpad directory. Changes remain isolated and are never written back to the host system until an explicit caller-initiated commit trigger is executed.
• Resource Profiling: Workloads can define specific resource profiles, requesting passthrough access to explicit files or resource patterns when necessary.

5. Observability (Telemetry, Metrics & Tracing)

Rustarium must provide deep, standardized visibility into both system health and workload performance, integrating seamlessly with modern observability stacks.

• Distributed Tracing: Span-based tracing across the entire stack using OpenTelemetry standard patterns.
• Categorized Telemetry Streams: Data collection is strictly separated into three logical clusters:
  • Active Logging: Debugging, troubleshooting, and progress tracking for events.
  • Workload Statistics: Metrics around running workloads (e.g., latency percentiles, error rates, tokens-per-second, overhead tracking).
  • System Health Metrics: Resource utilization across processes and nodes (CPU, memory, disk).
• Third-Party Integration: Metrics must be formatted to standard specifications (e.g., Prometheus) for easy ingestion into tools like Grafana or Datadog.
• Custom Instrumentation: Developers must have the programmatic and configuration-based ability to define and emit custom metrics, logs, spans, and events.
• Inline Data Masking Pipeline: To prevent PII or sensitive payload leakage, the Rust orchestrator must expose an inline, high-performance data masking pipeline at the IPC boundary. The JobConfig schema includes a mask_patterns: list[str] parameter. The Rust daemon intercepts stdout/stderr streams and applies these string/regex masking rules on-the-fly before the logs are pushed into the host process's logging framework.

6. Container Runtime Integrations

To support complex and diverse environments, the framework must interface smoothly with standard container runtimes.

• Docker Integration: Native support for the Docker API to spin up sandboxed containers for running processes.
• Complex Orchestrations: Out-of-the-box compatibility with Docker Compose for multi-container deployments.
• Runtime Agnosticism: The orchestration layer must be abstracted to easily support alternative runtimes like Podman, containerd, and standard Kubernetes CRI in the future.
• Cross-Platform API Client Abstraction: The runtime integration layer relies purely on UNIX domain sockets (.sock endpoints) for Linux and macOS environments. Windows platform abstraction (Named Pipes) is deferred to Phase 2.

7. Reliability, Fault Tolerance & Configuration Management

The platform must be resilient to failures and easily configurable across diverse deployments.

• Automatic Recovery: Built-in mechanisms to detect worker process crashes, automatically restart failed processes based on backoff policies, and re-queue dropped workloads.
• State Persistence: Optional state persistence for the orchestrator to resume job execution and retain telemetry in the event of a daemon restart or failure.
• Declarative Configuration: Support for defining cluster and sandbox configurations via standard formats (YAML, TOML, JSON) and environment variables, ensuring easy CI/CD and Kubernetes integration.

Non-Functional Requirements (NFRs)

To guarantee Rustarium meets the needs of High-Performance Computing (HPC) and fast agentic workflows, the system must adhere to strict performance and resource constraints:

• Startup Latency: Cold-start overhead introduced by the Rust orchestrator layer must remain < 10ms for Tier 1 (Rootless) operations on Linux and macOS. When initializing Tier 3 (Delegated Docker containers), the overhead must not exceed < 50ms beyond the baseline performance of the underlying host container system. All worker processes will launch cleanly on-demand to guarantee an untainted state machine; persistent pre-warmed process pools are explicitly out-of-scope for safety.
• Scalability Limits: A single orchestrator node must comfortably manage and monitor 1,000+ concurrent worker processes without degradation in telemetry collection or scheduling loops.
• Daemon Footprint: The resident memory footprint of the Rustarium daemon should not exceed < 50MB, and baseline CPU utilization (when idle) should be < 1%.
• IPC Throughput: Zero/Low-overhead object passing should introduce sub-millisecond serialization latency and support gigabyte-scale data transfers (e.g., large ML tensors) with zero-copy mechanisms where possible.

Technical Specs

APIs & SDK Interfaces

The Python SDK serves as the primary entry point for developers. It leverages PyO3 to wrap the underlying Rust/Tokio orchestration engine, prioritizing a streamlined, high-performance developer experience.

1. Core Client Design (RustariumClient & AsyncRustariumClient)

The client manages the local orchestrator and the pool of worker processes.

• Decoupled Synchronous & Asynchronous Clients: To fix IDE tooling friction, the SDK enforces strict structural separation by providing two completely independent classes:
  • AsyncRustariumClient: The .submit() method returns a native asynchronous generator (AsyncIterable).
  • RustariumClient: Exposes a fully synchronous API. To minimize code duplication, the synchronous client internally wraps, drives, and blocks on the core async execution pathway.
• GIL Insulation & Thread Isolation: The core Rust orchestration daemon and its underlying Tokio event loop must run on native OS threads completely independent of the main Python execution thread. When passing a synchronous iterable into RustariumClient.submit(), the SDK must offload iteration to a separate background worker thread (e.g., loop.run_in_executor()) to prevent blocking Python I/O from disrupting the orchestration loop.
• Static Cluster Configuration: Cluster and process limits are defined entirely upon initialization (e.g., maximum system resources, maximum number of concurrent processes). The client abstracts process management away from the developer—dynamically scaling, matching workloads to available processes, and handling worker lifecycles automatically based on the submitted workloads.

2. Job Submission & Workload Context

Workloads are submitted via a typed interface that gracefully bridges Python and Rust concurrency models, emphasizing contextual defaults and specific overrides.

• Signature: client.submit(workloads: Iterable[Any], default_config: JobConfig | None = None)
• Contextual Workloads: The submission accepts an iterable. This iterable can contain plain Python objects or specific Workload objects.
  • Plain Objects: Inherit the default_config settings for execution (or fail if no default was provided).
  • Workload Objects: Explicitly define their own requirements, constraints, and sandboxing definitions.
• Configuration Validation: All settings, constraints, and workloads rely on Pydantic v2 for validation in the Python space. The validated payload is handed off to the Rust layer cleanly without duplicating validation overhead.

3. Standardized Update Flow & Result Streaming

The client does not return distinct streams for logs or metrics. Instead, it returns a unified iterable (asynchronous in AsyncRustariumClient) that yields comprehensive updates.

• The Update Stream: async for update in client.submit(...):
• Update Payload: Each yielded update object contains:
  • job_context: Overall job-level metadata and aggregated metrics.
  • rustarium_context: System-level orchestration health and state.
  • workload_status: Status transitions for the specific workload (e.g., executing, completed, failed).
  • result: The actual execution output (if completed).
  • metrics: Telemetry, latency, and resource statistics specifically tied to that workload's execution.
• Invisible Logging: Logs are entirely abstracted. The engine synchronizes all stdout/stderr and internal logs back to the main process, mimicking a standard single-process setup (compatible with loguru/standard logging). Logs are automatically routed to the main process's configured handlers.

4. Job Constraints & Custom Logic

Job constraints are treated as first-class citizens, controlling precisely when a job should naturally complete or forcibly terminate.

• Built-in Constraints: Support for standard hard limits, such as max_errors, max_error_rate, max_workloads, and max_execution_time. By default, a job runs until the provided iterable is exhausted.
• Custom Constraint Logic: Developers can inject custom Python logic/callbacks that are evaluated during the update loop to determine if the job should dynamically complete, pause, or error out based on specific programmatic conditions.
• Metrics Collection: The Rust engine automatically collects fine-grained statistics (process utilization, throughput, latency percentiles) and attaches them to the standard update flow for monitoring and constraint evaluation.

5. Error Topology

Native Rust panics and underlying orchestration errors are caught and surfaced to Python via a typed exception hierarchy, grouped logically for the caller to handle easily.

• RustariumError (Base Class)
  • ConfigurationError: Invalid constraints, unparseable objects, or bad environment state.
  • OrchestrationError: General engine failures, communication timeouts, or scheduling issues.
  • WorkerError: Process crashes, user-code exceptions, or application-level panics.
  • SecurityError: Isolation violations such as attempting a SandboxEscape, or breaching memory/resource ceilings.

Workload & Configuration Definitions

To ensure a clean boundary between the Python SDK and the Rust orchestration engine, all configuration rules must be clearly defined. The system must support general object passthrough while allowing users to apply specific configuration parameters.

• Schema Validation:
  • Python Validation: The system utilizes Pydantic v2 on the Python layer to handle configuration validation, applying sensible defaults and managing object instantiation.
  • Rust Deserialization: The Rust backend trusts the validation performed by the Python layer. It utilizes efficient serialization (e.g., serde with bincode) mapping to its internal types without needing to strictly constrain the arbitrary payload data.

1. JobConfig Schema

The JobConfig acts as the overarching policy for a submitted batch of workloads. It defines the environment, resources, and rules for the execution.

• Execution Settings: min_workers, max_workers
  • allow_process_reuse: Boolean flag (default: False). If set to True, the orchestrator uses the Execution Match-Key to safely cache and reuse idle workers. If False, workers are forcibly destroyed and spawned on a per-submission basis for maximum safety.
• Runtime Environment:
  • Support for defining Python versions, explicitly managing or creating virtual environments (venv/conda), and verifying that specific dependencies are installed and available to the process before execution.
  • env_vars: Allow/deny lists for environment variables.
  • mask_patterns: A list[str] of sensitive strings or regex patterns that must be scrubbed from standard output before entering host logging pipelines.
• Filesystem & Network Access:
  • Access Paradigm: Sandboxes enforce CWD Path Uniformity, mimicking host paths precisely but utilizing the Emulated Write-Isolation (Scratchpad) layer to intercept changes.
  • filesystem_access: A boolean toggle (if true, the sandbox inherits the runner's access; if false, it only has access to its local sandbox). Alternatively, an explicit list of allowed/disallowed paths, globs, or regexes specifying read, write, or execute permissions.
  • network_access: A boolean toggle for overall network availability. Must also support explicit lists of IP addresses, namespaces, domains, path-level addresses, globs, or regexes to strictly control ingress and egress routing.
• Security & Isolation Enums: Defines the dichotomous security profile.
  • SecurityProfile.SECURE_ISOLATED: Strict lockdown, disables shared memory (shm), severs network access by default, and enforces strict filesystem scoping.
  • SecurityProfile.HIGH_PERFORMANCE_TRUSTED: (Phase 2) Relaxed isolation allowing zero-copy memory mapping for massive tensor transfers.

2. Workload Object Schema

A Workload represents a single unit of execution. The system supports general passthrough where arbitrary Python objects (without any constraints on their type or implementation) can be submitted.

• Payload: The arbitrary Python object or data.
• Wrapper Class: Submitted generic objects are automatically wrapped in a Workload class with default settings. Users can explicitly wrap their objects in this workload wrapper class to define workload-specific constraint overrides (e.g., higher memory limits for a single specific payload).

3. Scheduling Strategy Payloads

Concrete sub-configurations are required depending on the chosen distribution model:

• Concurrent: target_workloads (int) - The active number of workloads that are supposed to be running and submitted at any one point in time (not strictly the number of processes) (able to run down to 1 for synchronous execution).
• Constant Rate: tasks_per_second (float) - Ensures an even distribution of tasks over time.
• Burst: No specific payload needed; maximizes utilization instantly.
• Poisson Distribution: arrival_rate_lambda (float) - The average rate of incoming tasks to simulate natural, randomized traffic.

4. Constraints & Hard Limits (The Poison Pill)

To protect the host system from runaway code, strict limits must be enforceable per workload.

• Thresholds:
  • cpu_time_limit_ms: Maximum CPU time allowed.
  • wall_time_limit_ms: Maximum real-world elapsed time.
  • max_memory_bytes: Peak memory ceiling.
  • max_errors / max_error_rate: Job-level constraints that halt execution if too many workloads fail.
• Violation Policies:
  • ViolationPolicy.GRACEFUL: Issues a SIGTERM and waits for a predefined grace period before killing.
  • ViolationPolicy.SIGKILL: The "Poison Pill" — immediate, uncatchable process termination.
• Enforcement Fallbacks: If specific hard constraints (like strict memory limits) are requested but cannot be enforced by the underlying tier/OS (e.g., Tier 1 on macOS lacking native cgroups), the system must explicitly fail with a clear, actionable error directing the user to a tier that supports it.

Logging, Telemetry & Performance

To guarantee observability and maintain maximum execution speed, Rustarium's telemetry and logging subsystems are designed around a high-performance "push-to-main" architecture. Data sanitization and masking are explicitly out-of-scope for the orchestrator, delegating these responsibilities to external log collectors or aggregators.

1. Centralized Application Logging

Worker processes are automatically injected with a unified logging setup upon startup.

• IPC Log Routing: Instead of writing to local files or complex external sinks, the worker logging setup intercepts and pushes all log events back to the main API/SDK process via high-speed IPC.
• Main Process Aggregation: The main process synchronizes and aggregates logs from all Rust orchestrator threads and Python worker processes.
• User-Supplied Logger Integration: Aggregated logs are pushed into a single, centralized logger implementation supplied or configured by the user (e.g., standard Python logging, loguru). This ensures proper formatting for OpenTelemetry and allows developers to leverage existing ecosystem tooling.

2. System & Application Metrics

Standard system health and application execution metrics follow the same unified pipeline as application logs.

• Unified Pipeline: Metrics tracking process health (CPU, memory footprints) and internal application states are routed through the high-performance IPC channel to the main process.
• Aggregated Output: Metrics are exposed via the user's configured telemetry framework (e.g., Prometheus) directly from the centralized main process.

3. Custom Performance & User Statistics

To support massively data-intensive benchmarking and evaluation frameworks (like GuideLLM), the platform provides an entirely separate, highly flexible interface dedicated specifically to generic timing and value collection.

• Unified Timings Interface: A standalone, low-overhead interface exposed universally across the main process, the Rust orchestrator, and the running worker processes.
• Flexible Collection & Aggregation: Developers can report arbitrary performance timings, latency markers, or custom user metric values from anywhere in the stack. These raw metrics are collected centrally in the main process, enabling developers to define diverse aggregation and statistical calculation strategies (e.g., custom percentiles, throughput tracking) natively in Python prior to final export.

Sandboxing & Capability Matrix

The Sandboxing & Capability Matrix defines how Rustarium enforces security boundaries and resource limits across different operating systems. It maps high-level job configuration settings (e.g., network_access, filesystem_access, SecurityProfile) to the specific, underlying native kernel primitives available on the host platform.

1. Cross-Platform Primitives Matrix (MVP)

The orchestrator utilizes a multi-tiered approach, automatically selecting the appropriate primitives based on the host OS and the requested Isolation Tier:

Isolation Tier	Linux Implementation	macOS Implementation
Tier 1 (Rootless)	Unprivileged User Namespaces, pivot_root	POSIX Permissions / Virtual Path Remapping
Tier 2 (Elevated)	cgroups v2, seccomp-bpf	N/A (Use Tier 3)
Tier 3 (Delegated)	Docker/Podman Engine via local Unix Socket	Docker Desktop via Unix Socket

• Graceful Degradation: If a specific constraint (e.g., strict memory limits) is requested but unsupported by the host OS at the current tier (such as Tier 1 on macOS), the engine will not silently ignore the constraint. Instead, it explicitly fails the job submission with a clear error, prompting the user to elevate to a supported tier (e.g., Tier 3 Docker).

2. Tier 1 Sandboxing Architecture & Memory Isolation

Tier 1 is the default execution mode, designed to provide the highest level of isolation possible without requiring root access or external daemon installations (like Docker).

• Strict Subprocess Forking: Untrusted code must never run inside the host Python process heap or the main Rust orchestrator thread via direct evaluation (e.g., PyO3). Instead, every worker is launched as a fully isolated, dedicated subprocess.
• The UDS Boundary: Communication between the trusted Rust orchestrator and the untrusted Python sandbox is restricted entirely to Inter-Process Communication (IPC). This relies exclusively on Unix Domain Sockets (UDS) for Linux and macOS. (Windows Named Pipes are deferred to Phase 2).
• Memory Isolation Guarantee: By enforcing this strict process boundary and UDS communication model, Rustarium guarantees that memory corruption, malicious pointer manipulation, or segfaults occurring within the untrusted worker code cannot compromise the host orchestrator process.

3. Capability Mapping to Security Profiles

The user's declared SecurityProfile determines which kernel primitives are engaged during worker instantiation:

• SECURE_ISOLATED (Agentic Profile)
  • Network: Creates a disconnected network namespace (Linux) or enforces network drop rules (macOS/Windows) to prevent data exfiltration.
  • Filesystem: Uses pivot_root/chroot (Linux) to jail the process. Access is strictly limited to an ephemeral tmpfs and explicitly configured volume mounts.
  • IPC/Shared Memory: Disables all shared memory (/dev/shm) access. Cross-process data transfer relies entirely on serialized payloads over UDS.
• HIGH_PERFORMANCE_TRUSTED (HPC Profile) [Phase 2]
  • Network: Inherits the host's networking stack for maximum throughput.
  • Filesystem: Broad host filesystem access is permitted, constrained only by the user running the orchestrator.
  • IPC/Shared Memory: Enables zero-copy memory mapping. Worker processes can directly map large tensor structures into shared memory, bypassing serialization overhead entirely for massive performance gains.

Lifecycle State Machines

The orchestration engine relies on strict, deterministic state machines to govern both individual worker processes and the overarching job submissions. This ensures predictable resource scaling, precise telemetry tracking, and robust failure recovery.

1. Worker State Machine

The Worker State Machine manages the lifecycle of an individual isolated process. Workers are launched purely on-demand to guarantee state purity. Idle workers may only be reused if their footprint matches the deterministic Execution Match-Key (a hash of python_env_path, requirements, code_source, and JobConfig) and allow_process_reuse is set to True.

• Valid States:

  • Pending: The worker has been requested by the scheduler, but the host OS primitives (namespaces, cgroups, or Docker socket) are still initializing.
  • Ready: The worker is fully initialized, connected via UDS to the Leader, and awaiting a workload payload.
  • Executing: The worker has received a workload and is actively processing the user-defined logic. Telemetry and logs are actively streaming.
  • Throttled: A temporary state triggered by resource ceiling proximity. The worker pauses workload ingestion or CPU cycles until limits normalize.
  • Terminated: The worker has been cleanly shut down (either by natural job completion, failure to match new execution keys, or explicit termination via SIGTERM).
  • Failed: The worker exited unexpectedly, breached a hard isolation limit (Sandbox Escape attempt), or was killed by the Poison Pill (SIGKILL) due to exceeding limits.

• Transition Rules:

  • Pending -> Ready: Normal startup sequence.
  • Ready -> Executing: Workload is assigned and dispatched via IPC.
  • Executing -> Ready: Workload completes successfully, and allow_process_reuse is True.
  • Executing <-> Throttled: Triggered by internal orchestration metrics to prevent host starvation.
  • Executing -> Failed: Worker crashes, times out (wall_time_limit_ms breached), or violates security constraints.
  • Failed -> Pending: If the orchestrator's automatic recovery policy permits restarting the worker.
  • Any State -> Terminated: The orchestrator initiates a teardown sequence (or process reuse is disabled).

2. Job State Machine

The Job State Machine governs the collective execution of the batch of workloads submitted by the user. It aggregates the states of its associated workers to provide a high-level status to the Python SDK.

• Valid States:

  • Submitted: The Python SDK has validated the payload and dispatched the configuration to the Rust daemon, but scheduling has not yet begun.
  • Running: The job is actively processing workloads. At least one worker is in the Executing state, and the source iterable is not yet exhausted.
  • Draining: The source iterable has been completely consumed. The orchestrator is waiting for the final active workers to transition from Executing to Ready or Failed.
  • Completed: All workloads have been processed. This state is reached even if some workloads failed, provided the total failures remain strictly under the configured max_errors or max_error_rate limits defined in the JobConfig.
  • Aborted: The job was forcibly halted. This occurs if a user explicitly cancels the job, if the max_errors threshold is breached, or if the Leader node experiences a critical failure.

• Transition Rules & Partial Failure Policies:

  • Submitted -> Running: The scheduler begins assigning workloads to Ready workers.
  • Running -> Draining: The end of the workload iterable is reached.
  • Draining -> Completed: All active workers finish processing. The system aggregates final telemetry.
  • Running / Draining -> Aborted: A workload failure causes the total error count to breach the max_errors ceiling. The orchestrator immediately broadcasts a termination signal to all remaining workers processing this job.
  • Any State -> Aborted: User-initiated cancellation via the SDK.

Testing

Test Scopes & Levels

• Test Scopes Matrix:
  • Unit Tests (test:unit): Isolated function-level validation. cargo test for Rust internals (e.g., cgroup parser, namespace instantiation) and pytest tests/unit for Python SDK components (e.g., configuration validation, workload serialization).
  • Integration Tests (test:integration): Component boundary validation. Ensures the Python RustariumClient can successfully initiate the Rust daemon, establish the IPC channel (UDS), and pass basic payloads back and forth.
  • End-to-End Tests (test:e2e): Full lifecycle validation. Simulates real-world workloads from job submission to final telemetry collection, involving actual process forking, isolation enforcement, and workload execution.
• Test Markers (Pytest):
  • smoke: Quick tests to check basic functionality.
  • sanity: Detailed tests to ensure major functions work correctly.
  • regression: Tests to ensure that new changes do not break existing functionality.
• Execution Levels (CI/CD Gates):
  • Development (PRs): Runs Unit (smoke, sanity) and Integration (smoke) to provide fast feedback during active development.
  • Main (Merges): Runs Unit (smoke, sanity), Integration (smoke, sanity), and E2E (smoke) to validate core behaviors upon merging.
  • Nightly: Runs a broader matrix including Unit (smoke, sanity, regression), Integration (smoke, sanity), and E2E (smoke, sanity) across the full span of supported Python versions.
  • Release: Comprehensive suite executing all scopes (Unit, Integration, E2E) across all markers (smoke, sanity, regression) to guarantee immutable build stability prior to tagging.
• Rust vs. Python Separation: Isolation of low-level Rust unit tests (cargo test for cgroup parsing and UDS throughput) from Python-level integration tests (pytest for SDK ergonomics and asynchronous generators).
• Environment-Mocking Strategy: Define how CI runners will mock elevated permissions (Sudo/Tier 2) and external runtime sockets (Docker/Tier 3) to prevent environmental drift from breaking baseline tests.

Functional & Compatibility Testing

• Core Functionality Matrix:
  • Payload Serialization: Verify generic Python objects, deeply nested iterables, and specific Workload wrappers survive the round-trip from Leader to Follower unchanged.
  • Constraint Enforcement: Assert that max_execution_time cleanly triggers ViolationPolicy.GRACEFUL and subsequent SIGTERM, while memory limit breaches trigger the Poison Pill (SIGKILL).
• Cross-Platform Parity Testing:
  • Linux: Assert Tier 1 (Namespaces) and Tier 2 (cgroups v2) behavior. Verify pivot_root appropriately restricts file reads.
  • macOS: Assert Tier 1 local scoping functions correctly, and explicitly verify the system gracefully degrades/rejects strict memory constraints with a clear error unless Tier 3 is engaged.
• Delegated Runtime Integration (Tier 3):
  • Verify the orchestrator can successfully interact with local Docker and Podman daemon sockets.
  • Ensure custom container image pulls, lifecycle state mapping (starting, running, stopping), and volume mounts pass through correctly via the abstracted API client.
• Interpreter Matrix: Specify an automated matrix testing routine across all supported Python runtimes (CPython 3.9 through 3.12, PyPy) to verify virtualenv/conda execution path resolution within the Tier 1 sandbox.
• Agnostic Loop Validation: Ensure communication loops cleanly manage UNIX domain sockets (.sock endpoints) across standard and Tier 3 Docker environments.

Performance & Scalability Validation

• Throughput & Scaling Benchmarks:
  • IPC Serialization Throughput: Measure the raw Mbps throughput for passing highly complex JSON-like objects over UDS, ensuring serialization overhead does not bottleneck the orchestrator.
  • Maximum Scheduling Rate: Benchmark the central scheduler's ability to dispatch 10,000 trivial noop workloads per second across 100 worker processes without inducing lag in the unified update stream.
  • Telemetry Accuracy: Inject known quantities of structured logs and custom user metrics from 50 concurrent workers; assert that the main process aggregates 100% of the events without dropping data under load.
• Latency Threshold Tests: Build strict regression tests ensuring cold-start overhead remains beneath the <10ms (Tier 1) and <50ms (Tier 3) limits.
• High-Volume Concurrent Stressors: Outline scale testing scripts that instantly spin up 1,000 idle and active worker processes on a single node to assert the Rust daemon’s resident memory footprint remains bound below <50MB.
• Zero-Copy Verification: Construct a test verifying that when the High-Performance/Trusted profile is active, large objects (e.g., >1GB NumPy arrays) are transferred without invoking a single deep-copy allocation.

Security, Resilience & Negative Testing

• Data Privacy & Exception Isolation:

  • Ensure that error stack traces and exceptions originating from untrusted workers do not inadvertently leak portions of other workloads' memory or sensitive orchestrator state.

• Penetration & Escape Verification: Design automated tests that attempt standard container escape tactics (e.g., executing unauthorized chroot sequences, trying to open network connections under SECURE_ISOLATED, writing outside explicit volume mounts) and assert that they are cleanly intercepted.

• Resilience & Recovery Validation:

  • Resource Exhaustion Simulation: Force a worker environment into an OOM (Out Of Memory) condition. Verify the daemon cleanly catches the kernel kill signal, marks the workload as failed, logs the SecurityError, and spins up a replacement worker without halting the broader job.
  • Daemon Crash Recovery: Kill the main Python SDK process mid-job. Verify that the Rust orchestrator cleanly orphans or terminates the associated workers to prevent lingering zombie processes on the host.
  • Constraint Tolerance: Submit a job where 40% of the workloads are designed to randomly throw exceptions. Assert that if the max_error_rate constraint is set to 50%, the job ultimately reaches Completed state and yields the partial valid results.

• Chaos Engineering Harness: Outline tests that simulate sudden, unannounced worker process terminations (SIGKILL) and verification of backoff recovery loops.

Documentation

The project documentation is built on MkDocs Material and automatically deployed via GitHub Pages (.github/workflows/_docs.yml). The structure is split across Getting Started, Guides, Examples, Reference, and Community sections to ensure clear navigation for users at all experience levels.

Getting Started & Installation

• Zero-Friction Guide: Document the immediate pip install rustarium flow, showcasing a 5-line "Hello World" sandboxed execution snippet directly on the landing page (docs/index.md). Maintain a comprehensive installation matrix for pip, uv, hatch, and Docker deployments in docs/getting-started/installation.md.
• The Diagnostic Tool Spec (rustarium doctor): Detail the terminal output design and FTUX workflow for the rustarium doctor utility. This must act as a distinct subcommand under the primary rustarium CLI entry point. Document its automated checks for OS support, cgroups v2 availability, Docker socket permissions, and Python virtual environment state. This guide will be located in docs/getting-started/quickstart.md, replacing the template bootstrap instructions.

Workflow Guides & Architectural Concepts

Guides focus on conceptual underpinnings and comprehensive walkthroughs of core features (architecture, security, custom scheduling), decoupled from specific single-file runnable examples.

• Sandboxing Architecture Deep Dive: Document the multi-tiered isolation strategy (Namespaces, cgroups v2, Docker) to clarify how the dichotomous security profiles interact with host OS primitives, mapped out in docs/guides/architecture.md.
• HPC/Tensor Transfer Blueprint: Provide comprehensive optimization guides showcasing shared memory setups for massive multi-node training tasks using the HIGH_PERFORMANCE_TRUSTED profile, located in docs/guides/hpc-tensors.md.
• Plugin Interface Guide: Create abstract base class documentation showing developers how to implement custom scheduling algorithms in Python and register them dynamically with the core Rust engine. This should be added to docs/guides/custom-scheduling.md.

Core Use Case Examples

Examples provide complete, runnable code files demonstrating core functionality and specific use cases across all major capabilities of the orchestrator.

• Agentic Cookbook: Provide full, runnable code recipes demonstrating how to construct an unprivileged Agent code execution block that processes untrusted inputs securely under the SECURE_ISOLATED profile. This will live under docs/examples/agentic-execution.md.
• LLM Benchmarking Suite: Walk through a runnable script setting up simulated Poisson-distributed traffic patterns to stress-test an inference engine endpoint using the Python SDK. This will be provided in docs/examples/llm-benchmarking.md.

API & SDK Reference

• Tooling Pipeline: Utilizing MkDocs Material with the mkdocstrings plugin to pull directly from Python source files in src/. The generation is automated via mkdocs-gen-files running docs/scripts/gen_ref_pages.py, creating a dynamic literate navigation structure for the API reference at docs/reference/api/.
• Rust/Python Type Stubs: Ensure that inline type stubs (.pyi files) exported by PyO3 are correctly parsed by the documentation generator so that the Python auto-generated docs match the underlying Rust signatures perfectly. All public interfaces must include comprehensive Sphinx-style docstrings.

Roadmap

MVP (Must Have): Local-First Secure Sandbox

• Distribution: Pip-installable Python package bundling the Rust orchestration engine.
• Architecture: Leader/Follower topology (Local API/SDK process driving worker lifecycles).
• Security: Strict MVP Isolation enforcing Filesystem, Network, and Environment variable sandboxing (Tier 1 Linux Namespaces only).
• Execution: Local execution via unprivileged user namespaces or the host Docker socket (Tier 3).
• Scheduling: Basic FIFO and Concurrent Scheduling strategies with Local IPC.

Phase 2 (Should Have): Multi-Node & High Performance

• Networking: Multi-node TCP/gRPC data distribution and telemetry collection over the network.
• Architecture Expansion: Cross-platform support expanding Tier 1 & Tier 2 to macOS and Windows targets.
• Security Scaling: Tier 2 hardware-enforced isolation via cgroups v2 integration.
• Scheduling: Priority Queues and Dynamic Backpressure models.

Future (Could Have): Extensibility & Simulation

• Extended Runtimes: Native execution of WASM binaries (e.g., Wasmtime integration).
• Enhanced Security: MicroVM sandboxing (Firecracker, gVisor) for untrusted enterprise workloads.
• Advanced Simulation: Poisson and Burst traffic distribution models.