Availability of DCLM data mixes used in figure 3

[IanMagnusson]

Thank you for this fascinating work!

@benbogin, @taidnguyen, and I are interested in working with some of the variant data mixes that you produced in your work. In particular we wonder if it might be possible to get access to the data mixes behind the diamond shaped points in figure 3 of your paper (copied here). Which data mixes do these points correspond to?

Also if there are other variant data mixes than these that you are already interested in releasing, we would be very much interested to know. A publicly released set of data mixes that ablate specific data decisions would be a very valuable research artifact for the community that I'm sure lots of people would make use of.

[Mivg]

Hi @IanMagnusson
Those datasets are merely the result of different filtering pipelines, and not inherently different data sources, and thus both hold little information for the general public and require extensive storage to host if release.
however, we can try to release more sets of weights for fasttext classifiers that are responsible to some of these datasets, and then you can easily and efficiently reproduce them locally upon need.
Would that work?

[IanMagnusson]

Got it. Yup I understand the variants here differ in processing not in sources. I should have said "corpus" rather than "data mix" to clarify. But as your paper shows such differences are potentially more impactful than tinkering with including different sources, I think the community might be interested in a standard set of pretraining corpora with controlled variations in processing. If having folks reproduce them locally is all that is feasible, we totally understand.

Perhaps to start with I'm curious which are the variants included in this figure? I think it's some subset of all the ablations discussed in the paper right? Thanks!

[jeffreywpli]

Hi @IanMagnusson sorry for delay. So the variants in that plot are as follows:

• loosening the fasttext filter threshold to be top 20% instead of top 10%
• a couple variants of the final bff deduplication changing the min_ngram hyperparmater and amount of sharding
• a version with minhash+substring deduplication instead of bff

Also, we can share some of the other fasttext filters we tried we tried (though are not captured in this plot specifically) if you are looking for other variants to include in your study.

[IanMagnusson]

Thanks so much for the clarification and providing the classifiers! We'll definitely look into using these in our project!