Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds mixing loader for FSL datasets #70

Merged
merged 61 commits into from
Nov 1, 2024
Merged

Adds mixing loader for FSL datasets #70

merged 61 commits into from
Nov 1, 2024

Conversation

undfined
Copy link
Collaborator

@undfined undfined commented Oct 19, 2024 •
edited
Loading

This change set introduces a new mechanism for building a NumpyFSLDataset from a predetermined ratio of tokens per source. The UX is to supply a SourceMixtureDatasetConfig class to NumpyFSLDatasetMixtureConfig that at data loading time counts tokens for all the sources/files provided, validates the intended ratios can be met from the source files, and builds the mixture of paths and number of instances to retain from each file that is then used to construct a typical NumpyFSLDataset with some modifications:

Lots of tests were added for the new mixture class and some downstream tests in numpy_dataset as well.

@undfined
WIP: Generate a mixture dataset
a8b5ec6
@undfined
WIP: Adds dry run
637fee9
@undfined
Test cleanup
346135c
@undfined
WIP: Make it fast
53def38
@undfined
WIP: Simple benchmark
8649cc8
@undfined
WIP: Refactor
e3d7011
@undfined
Launch script
efe766b
@undfined
temp changes to test
5dff40c
@undfined
deps for now
2703538
@undfined
Try with session
3ee3278
@undfined
Try internal client
cd1c6d2
@undfined
Try boto3
9895b23
@undfined
Fixes
3c15f52
@undfined
?
0c9355b
@undfined
Cleanup + session stuff
abb362a
@undfined
Use environ
82a1af9
@undfined
JUST use env vars please boto
d0a80ba
@undfined
No unions of containers
e621f8e
@undfined
prepare first
0689c42
@undfined
Loader handles prepare
8ab2e99
@undfined
Try recording torch exceptions
dcfda67
@undfined
Don't need overrides
23a0806
@undfined
Figure out why config/creds are missing
8cfa282
@undfined
fmt
fd1a508
@undfined
Env not ready yet
01a40ea
@undfined
print beaker user
8bde2b3
@undfined
uncomment eval file
ae208f6
@undfined
replicate CommonComponents setup
ce9d06f
@undfined
Some class init stuff
d1eb4df
@undfined
Some more config logging
dbce279
@undfined undfined changed the title Adds Mixing loader Adds mixing loader for FSL datasets Oct 25, 2024
@undfined undfined requested a review from epwalsh October 25, 2024 22:54
epwalsh
epwalsh reviewed Oct 25, 2024
View reviewed changes
Copy link
Member

@epwalsh epwalsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a complete review yet but here are some high-level comments

src/olmo_core/data/source_mixture.py Outdated Show resolved Hide resolved
src/olmo_core/data/source_mixture.py Outdated Show resolved Hide resolved
src/olmo_core/data/source_mixture.py Outdated Show resolved Hide resolved
src/olmo_core/data/source_mixture.py Outdated Show resolved Hide resolved
src/olmo_core/data/source_mixture.py Outdated Show resolved Hide resolved
pyproject.toml Outdated Show resolved Hide resolved
src/examples/train_with_mixture.py Outdated Show resolved Hide resolved
src/olmo_core/data/types.py Outdated Show resolved Hide resolved
src/olmo_core/data/numpy_dataset.py Outdated Show resolved Hide resolved
src/olmo_core/data/numpy_dataset.py Outdated Show resolved Hide resolved
epwalsh
epwalsh reviewed Oct 29, 2024
View reviewed changes
Copy link
Member

@epwalsh epwalsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really good! A couple more minor comments

src/olmo_core/data/numpy_dataset.py Outdated Show resolved Hide resolved
src/olmo_core/data/numpy_dataset.py Outdated Show resolved Hide resolved
epwalsh
epwalsh reviewed Oct 30, 2024
View reviewed changes
Copy link
Member

@epwalsh epwalsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more comments

src/olmo_core/data/utils.py Show resolved Hide resolved
src/olmo_core/data/source_mixture.py Outdated Show resolved Hide resolved
src/olmo_core/data/source_mixture.py Outdated Show resolved Hide resolved
src/olmo_core/data/source_mixture.py Outdated Show resolved Hide resolved
CHANGELOG.md Outdated Show resolved Hide resolved
src/olmo_core/data/numpy_dataset.py Outdated Show resolved Hide resolved
src/olmo_core/data/numpy_dataset.py Outdated Show resolved Hide resolved
epwalsh
epwalsh approved these changes Oct 30, 2024
View reviewed changes
Copy link
Member

@epwalsh epwalsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@undfined undfined merged commit 4928f82 into main Nov 1, 2024
14 checks passed
@undfined undfined deleted the undfined/mixing-loader branch November 1, 2024 18:22
@undfined undfined restored the undfined/mixing-loader branch November 1, 2024 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@epwalsh epwalsh epwalsh approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@undfined @epwalsh