Skip to content

feat(sm-integration): Add SageMaker training as a backend. - #115

Open
Linbo-Liu wants to merge 3 commits into
awslabs:mainfrom
Linbo-Liu:sm-integration
Open

feat(sm-integration): Add SageMaker training as a backend.#115
Linbo-Liu wants to merge 3 commits into
awslabs:mainfrom
Linbo-Liu:sm-integration

Conversation

@Linbo-Liu

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

What

Adds an experimental SageMaker Training Sessions backend: a GRPO loop that trains an AgentCore Runtime (ACR)-deployed agent with no local GPU cluster. SageMaker hosts the policy weights, the sampler, and the optimizer behind an SDK, so the RL loop itself is a plain single-process asyncio program that can run on a laptop or a small EC2 box. Token capture reuses the in-repo rollout gateway, exactly as the experimental verl backend does.

How it fits together

The only new engine seam is SageMakerSdkBackend (rollout_gateway/sampling_backends/sagemaker_sdk.py) — a gateway SamplingBackend over the SageMaker SamplingClient that maps token_ids -> token_ids + logprobs. Like TinkerSdkBackend, it does not render: the gateway owns tokenization, which keeps loss-masking well-defined and matches the existing placement rule (independently reachable hosted SDK → sampling_backends/).

The training loop

backends/experimental/sagemaker/ is driven by a single YAML config (config.py / config.yaml.example):

File Role
train_grpo.py GRPO loop: rollout → advantages → forward_backwardoptim_step → rebind sampler
rollout.py One ACR rollout: create gateway session, invoke the agent, await the S3 result, drain trajectories
datum.py TraceRecord + advantage → SageMaker training datum
config.py / config.yaml.example Typed config dataclass and a template to copy
prepare_datasets/prepare_gsm8k.py GSM8K → payload-column Parquet

Tests

Tested on GSM8k. Stable training for 200+ steps. Test accuracy increases from ~70% -> ~90% with gpt-oss-20b.

Docs

New docs/site/src/content/docs/guides/sagemaker-backend-setup.md (installation, config reference, what the loop does step by step, current limits), wired into the sidebar and linked from index.mdx and guides/overview.mdx alongside slime / rllm / verl. A short SETUP.md sits next to the code for readers who get there from the source tree.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@Linbo-Liu Linbo-Liu changed the title Sm integration feat(sm-integration): Add SageMaker training as a backend. Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant