Brief analysis of different tokenizers

[bigwolfeman]

Starcoder2 and Phi-3 are strong standouts.
This was conducted on a my local setup using the current training script to 8m tokens. Run to run variance I have been seeing is about 1.5s

[bigwolfeman]

[vukrosic]

This is interesting idea, however those 8M tokens could accidentally fit a certain tokenizer while the full 1B might be more suitable for a different optimizer. This would need more research but it's a good start. Thank you for the contribution.

[bigwolfeman]

I will run each out to 1b tokens tonight. It takes awhile to retokenize the data for each run.

[bigwolfeman]

strong results. The load on my gpu was a little higher during some of this run, the time differences are not very reliable as a result. The Phi and Mistral should be lower than they are. I have a strong preference toward Starcoder2. They have chunked numbers up in a way that dramatically improves math ability of the model from other testing I have done in the past.

This is run out in the same fashion as before to 110m tokens.

[vukrosic]

Seems interesting, we will use it later, not sure when we will get into this but it's good that it's here

[vukrosic]

Tokenizers are important to research as well

[bigwolfeman]

If you need it, I can retokenize the entire dataset. I have a lot of CPU to throw at it.