Skip to content

fix setting ft state dicts when ft checkpointing is disabled #1915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 30, 2025
Merged

fix setting ft state dicts when ft checkpointing is disabled #1915

merged 1 commit into from
Oct 30, 2025

Conversation

@tushar00jain
Copy link
Contributor

@tushar00jain tushar00jain commented Oct 17, 2025 •
edited
Loading

Summary:

  • when ft dataloader checkpointing is disabled, we also don't set the ft state
  • make it so that when ft checkpointing is disabled, we still set the state dict so that model, optimizer etc. can be recovered from a different replica

Stack created with Sapling. Best reviewed with ReviewStack.

This was referenced Oct 17, 2025
Draft
Draft
Draft
@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 17, 2025
fegin
fegin reviewed Oct 17, 2025
View reviewed changes
@tushar00jain tushar00jain marked this pull request as ready for review October 17, 2025 21:43
tianyu-l
tianyu-l reviewed Oct 17, 2025
View reviewed changes
@tushar00jain tushar00jain force-pushed the pr1915 branch 2 times, most recently from d3b5640 to 267f201 Compare October 24, 2025 19:01
tianyu-l
tianyu-l reviewed Oct 24, 2025
View reviewed changes
fegin
fegin requested changes Oct 27, 2025
View reviewed changes
Copy link
Contributor

@fegin fegin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ft_inner_manager is confusing. I would suggest that you just make it a boolean flag.

@tushar00jain
fix setting ft state dicts when ft checkpointing is disabled
22a1a9a 
Summary:
- when ft dataloader checkpointing is disabled, we also don't set the ft state
- make it so that when ft checkpointing is disabled, we still set the state dict so that model, optimizer etc. can be recovered from a different replica
@tushar00jain tushar00jain merged commit e150caa into pytorch:main Oct 30, 2025
8 of 13 checks passed
@tushar00jain tushar00jain deleted the pr1915 branch October 30, 2025 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tianyu-l tianyu-l tianyu-l left review comments

@fegin fegin fegin approved these changes

@wwwjn wwwjn Awaiting requested review from wwwjn wwwjn is a code owner

@wconstab wconstab Awaiting requested review from wconstab wconstab is a code owner

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tushar00jain @fegin @tianyu-l