How does Nemo Curator sort petabytes of Common Crawl data for quality bucketing?

[XevWright]

Hey everyone! 😊

I've been diving into something interesting about the NemotronCC pipeline (nemo curator) and got a bit stuck—hoping some brilliant mind here can help me out! 🌟

So here's the deal: when creating quality buckets (0-19), the process requires sorting ALL documents by their quality score first before splitting them into buckets. That makes total sense for fair distribution... Common Crawl is MASSIVE. We're talking petabytes of data that definitely can't fit on one machine, so processing has to be distributed across many servers. But sorting normally requires comparing items globally, which gets insanely complicated when your data is spread across thousands of machines!

I've been thinking:

✨ How do they actually pull off this global sort at such an enormous scale?

✨ Is there some clever trick to avoid a full sort?

✨ How do they handle the shuffle phase without killing the network?

✨ Does Nemotron use some special distributed algorithm I don't know about?

I'm genuinely curious about the real engineering magic behind this! If anyone has insights into how large-scale sorting works for web-scale datasets (or specifically for NemotronCC), I'd be forever grateful! 🙏

Also totally okay with paper recommendations, engineering blog links, or just rough ideas!

Thanks so much in advance,

[ayushdg]

Thanks for reaching out. As you've pointed out it's extremely compute intensive to sort at the CC scale. The goal of the bucketing process is to create bins based on the score distribution such that ~5% of documents fall in each bucket. This can be approximated by a couple of approaches.

1. Sample a distribution of the dataset randomly and use that sample to guesstimate the rough split for buckets.
2. Some distributed frameworks support algorithms (such as t-digest) for computing approximate distributed quantiles which would also work in this case.

Side note: We're working on releasing some of the scripts that might help with nemotron-cc. It's a work in progress here NVIDIA-NeMo/Nemotron#81.

[XevWright]

Thanks for the in-depth explanation! That clears up a lot of my confusion.