Gymnasium api by drogovozDP · Pull Request #1350 · kubernetes-sigs/agent-sandbox

drogovozDP · 2026-08-07T18:38:47Z

This is an example PR. The goal: highlight the potential changes. This example PR provides:

clients/integrations/gymnasium: a gymnasium integration itself.
example/agent-sandbox-gymnasium: a small tutorial on how to use SandboxEnv.

netlify · 2026-08-07T18:38:53Z

✅ Deploy Preview for agent-sandbox ready!

Name	Link
🔨 Latest commit	`0888c09`
🔍 Latest deploy log	https://app.netlify.com/projects/agent-sandbox/deploys/6a76263b7f384100072cd39b
😎 Deploy Preview	https://deploy-preview-1350--agent-sandbox.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

kubernetes-prow · 2026-08-07T18:38:54Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: drogovozDP
Once this PR has been reviewed and has the lgtm label, please assign igooch for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

clients/OWNERS
examples/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2026-08-07T18:38:58Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (3)

needs-ok-to-test
do-not-merge/work-in-progress
cncf-cla: no

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7d17abe2-f395-4e27-ae86-75fb784594d9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

kubernetes-prow · 2026-08-07T18:38:58Z

Hi @drogovozDP. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copilot

Pull request overview

This PR adds a Gymnasium-style RL environment interface for Agent Sandbox (Python) and provides an end-to-end example (FastAPI sandbox runtime + Kubernetes manifests + a Jupyter notebook) to demonstrate training/fine-tuning workflows against remote sandbox pods.

Changes:

Adds SandboxEnv + reward/termination function abstractions to the Python SDK (k8s_agent_sandbox.sandbox_env) with an rl optional extra for Gymnasium.
Introduces a standalone integration package under clients/integrations/gymnasium (with unit tests) that exposes similar Gymnasium APIs.
Adds a tutorial example under examples/agent-sandbox-gymnasium/ including build artifacts, manifests, and a training notebook.

Reviewed changes

Copilot reviewed 29 out of 30 changed files in this pull request and generated 31 comments.

Show a summary per file

File	Description
examples/agent-sandbox-gymnasium/source/main.py	Adds a FastAPI runtime that executes commands inside the sandbox container.
examples/agent-sandbox-gymnasium/source/Dockerfile	Builds the example runtime image.
examples/agent-sandbox-gymnasium/source/cloudbuild.yaml	Cloud Build recipe for publishing the example runtime image.
examples/agent-sandbox-gymnasium/sandbox/template.yaml	SandboxTemplate for running the example runtime image.
examples/agent-sandbox-gymnasium/sandbox/warmpool.yaml	SandboxWarmPool for pre-warming sandboxes for RL workloads.
examples/agent-sandbox-gymnasium/README.md	Tutorial instructions for deploying and running the example.
examples/agent-sandbox-gymnasium/jupyter/rl_training.ipynb	Demonstrates using the env and a training loop from Jupyter.
examples/agent-sandbox-gymnasium/jupyter/jupyter.yaml	Deploys a GPU Jupyter instance to run the tutorial notebook.
examples/agent-sandbox-gymnasium/jupyter/jupyter-rbac.yaml	RBAC allowing the Jupyter pod to manage SandboxClaims/Sandboxes.
clients/python/agentic-sandbox-client/pyproject.toml	Adds an `rl` extra dependency for Gymnasium usage.
clients/python/agentic-sandbox-client/k8s_agent_sandbox/sandbox_env/gymnasium_env.py	SDK-native Gymnasium env implementation (connection modes + lifecycle).
clients/python/agentic-sandbox-client/k8s_agent_sandbox/sandbox_env/reward_fn.py	Reward function interface for RL tasks.
clients/python/agentic-sandbox-client/k8s_agent_sandbox/sandbox_env/reward_fns.py	Concrete reward implementations (sparse, step penalty, etc.).
clients/python/agentic-sandbox-client/k8s_agent_sandbox/sandbox_env/termination_fn.py	Termination function interface for episode termination.
clients/python/agentic-sandbox-client/k8s_agent_sandbox/sandbox_env/termination_fns.py	Concrete termination function(s).
clients/python/agentic-sandbox-client/k8s_agent_sandbox/sandbox_env/init.py	Re-exports for sandbox_env module.
clients/integrations/gymnasium/pyproject.toml	Defines the standalone `k8s-agent-sandbox-gymnasium` integration package.
clients/integrations/gymnasium/k8s_agent_sandbox_gymnasium/gymnasium_env.py	Standalone integration Gymnasium env implementation.
clients/integrations/gymnasium/k8s_agent_sandbox_gymnasium/reward_fn.py	Reward function interface for the integration package.
clients/integrations/gymnasium/k8s_agent_sandbox_gymnasium/reward_fns.py	Reward implementations for the integration package.
clients/integrations/gymnasium/k8s_agent_sandbox_gymnasium/termination_fn.py	Termination function interface for the integration package.
clients/integrations/gymnasium/k8s_agent_sandbox_gymnasium/termination_fns.py	Termination implementations for the integration package.
clients/integrations/gymnasium/k8s_agent_sandbox_gymnasium/init.py	Re-exports for the integration package.
clients/integrations/gymnasium/tests/unit/test_gymnasium_env.py	Unit tests for the integration env.
clients/integrations/gymnasium/tests/unit/test_reward_fn.py	Unit tests for RewardFn.
clients/integrations/gymnasium/tests/unit/test_reward_fns.py	Unit tests for reward implementations.
clients/integrations/gymnasium/tests/unit/test_termination_fn.py	Unit tests for TerminationFn.
clients/integrations/gymnasium/tests/unit/test_termination_fns.py	Unit tests for termination implementations.
clients/integrations/gymnasium/tests/unit/init.py	Test package marker.

+        result = subprocess.run(
+            req.command,
+            shell=True,
+            capture_output=True,
+            text=True,
+            executable="/bin/bash",
+            timeout=60
+        )


+gcloud container clusters create-auto sandbox-rl-cluster --location=us-east1
+```
+
+Follow the instructions from oficial guide to [install Agent Sandbox CRDs and Router](https://github.com/kubernetes-sigs/agent-sandbox#installation).


+  - apiGroups: ["extensions.agents.x-k8s.io"]
+    resources: ["sandboxclaims", "sandboxes"]
+    verbs: ["get", "list", "watch", "create", "delete"]
+  - apiGroups: ["agents.x-k8s.io"]
+    resources: ["sandboxes", "sandboxclaims"]
+    verbs: ["get", "list", "watch", "create", "delete"]


+      - name: jupyter
+        image: tensorflow/tensorflow:latest-gpu-jupyter


+   },
+   "outputs": [],
+   "source": [
+    "!pip install \"git+https://github.com/volatilemolotov/agent-sandbox.git@gymnasium-api#subdirectory=clients/integrations/gymnasium\" --upgrade"


@@ -0,0 +1 @@
+# Unit tests


+    def __init__(
+        self,
+        reward_fn: RewardFn,
+        termination_fn: TerminationFn,
+        warmpool: str = "simple-sandbox-warmpool",
+        namespace: str = "default",
+        client: SandboxClient = None,
+        max_episode_steps: int = 20,
+        max_obs_length: int = 4096,
+    ):
+        super().__init__()
+
+        if not isinstance(reward_fn, RewardFn):
+            raise TypeError(f"reward_fn must be a RewardFn instance, got {type(reward_fn)}")
+
+        self.reward_fn         = reward_fn
+        self.termination_fn    = termination_fn
+        self.warmpool          = warmpool
+        self.namespace         = namespace
+        self.max_episode_steps = max_episode_steps
+        self.max_obs_length    = max_obs_length
+
+        self._client        = client
+        self._sandbox       = None


+class SparseTaskTermination(TerminationFn):
+    """
+    Returns True if obs["reward"] == 1 and False otherwise
+
+    Example:
+        termination_fn = SparseTaskTermination()
+    """


+        "The 'gymnasium' library is required to use 'k8s_agent_sandbox.sandbox_env'. "
+        "Please install the 'rl' extra: `pip install 'k8s-agent-sandbox[rl]'` "
+        "or install gymnasium directly: `pip install gymnasium`."


+[project]
+name = "k8s-agent-sandbox-gymnasium"
+description = "Gymnasium-compatible RL environments backed by kubernetes-sigs/agent-sandbox"
+readme = "README.md"
+requires-python = ">=3.10"


drogovozDP added 26 commits June 23, 2026 16:06

init commit

4a1433c

update example

dd25276

update

e747e19

update notebook

2abb013

prepare the exmaple

52d2c36

remove output from rl_training.ipynb

a9c49b4

refactor files

13286da

remove manifests

9bb2ce8

update gymnasium dependency

43e42e6

fix docstring

9c6b33d

fix info block

cd82fe0

add info[env_error]

953cef8

update termination logic

816f325

add shell=True

40adc27

parametrized image path

2b44c5c

update the readme

ae93e31

fix typos in termination

d3a7b92

update termination

c17e4e6

update infra

1fd44c7

change hello_world.py to hello_world.sh

fb0264c

add timeout

ffde5e8

remove public repo

47e2bae

create new integration

1d10b36

update

fa78583

update

da833c4

update integration structure

0888c09

github-project-automation Bot added this to Agent Sandbox Aug 7, 2026

kubernetes-prow Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 7, 2026

github-project-automation Bot moved this to Backlog in Agent Sandbox Aug 7, 2026

kubernetes-prow Bot requested review from barney-s and justinsb August 7, 2026 18:38

kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 7, 2026

kubernetes-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 7, 2026

janetkuo requested a lite review from Copilot August 7, 2026 19:00

Copilot started reviewing on behalf of janetkuo August 7, 2026 19:00 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

janetkuo added the action-required: resolve-copilot-comments label Aug 7, 2026

Navigation Menu

Uh oh!

Gymnasium api - #1350

Gymnasium api#1350
drogovozDP wants to merge 26 commits into
kubernetes-sigs:mainfrom
volatilemolotov:gymnasium-api

drogovozDP commented Aug 7, 2026 •

edited

Loading

Uh oh!

netlify Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

kubernetes-prow Bot commented Aug 7, 2026

Uh oh!

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading

Review skipped

Uh oh!

kubernetes-prow Bot commented Aug 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		- name: jupyter
		image: tensorflow/tensorflow:latest-gpu-jupyter

Uh oh!

Conversation

drogovozDP commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for agent-sandbox ready!

Uh oh!

kubernetes-prow Bot commented Aug 7, 2026

Uh oh!

coderabbitai Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

kubernetes-prow Bot commented Aug 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

drogovozDP commented Aug 7, 2026 •

edited

Loading

netlify Bot commented Aug 7, 2026 •

edited

Loading

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading