feat(sm-integration): Add SageMaker training as a backend. - #115
Open
Linbo-Liu wants to merge 3 commits into
Open
feat(sm-integration): Add SageMaker training as a backend.#115Linbo-Liu wants to merge 3 commits into
Linbo-Liu wants to merge 3 commits into
Conversation
…gentcore cli for ACR deployment; fix response_schema and chat_template for gpt-oss models in Rollout Gateway.
Linbo-Liu
force-pushed
the
sm-integration
branch
from
August 26, 2026 00:24
581f81f to
46b0c28
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
What
Adds an experimental SageMaker Training Sessions backend: a GRPO loop that trains an AgentCore Runtime (ACR)-deployed agent with no local GPU cluster. SageMaker hosts the policy weights, the sampler, and the optimizer behind an SDK, so the RL loop itself is a plain single-process asyncio program that can run on a laptop or a small EC2 box. Token capture reuses the in-repo rollout gateway, exactly as the experimental verl backend does.
How it fits together
The only new engine seam is
SageMakerSdkBackend(rollout_gateway/sampling_backends/sagemaker_sdk.py) — a gatewaySamplingBackendover the SageMakerSamplingClientthat mapstoken_ids -> token_ids + logprobs. LikeTinkerSdkBackend, it does not render: the gateway owns tokenization, which keeps loss-masking well-defined and matches the existing placement rule (independently reachable hosted SDK →sampling_backends/).The training loop
backends/experimental/sagemaker/is driven by a single YAML config (config.py/config.yaml.example):train_grpo.pyforward_backward→optim_step→ rebind samplerrollout.pydatum.pyTraceRecord+ advantage → SageMaker training datumconfig.py/config.yaml.exampleprepare_datasets/prepare_gsm8k.pyTests
Tested on GSM8k. Stable training for 200+ steps. Test accuracy increases from ~70% -> ~90% with
gpt-oss-20b.Docs
New
docs/site/src/content/docs/guides/sagemaker-backend-setup.md(installation, config reference, what the loop does step by step, current limits), wired into the sidebar and linked fromindex.mdxandguides/overview.mdxalongside slime / rllm / verl. A shortSETUP.mdsits next to the code for readers who get there from the source tree.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.