[Question] Tokenizer is not counted in submission size

[DouglasOrr]

The submission size calculation doesn't count persisting the tokenizer itself - is this right? Two challenges to this: first, the size includes everything necessary (modulo generic Python requirements) to fully specify and inference the model, so it seems a shame to lose this property by omitting the tokenizer. Second, not counting this allows large-vocabulary models an artificial advantage, since they get the strings "for free".

(However, I appreciate this may be a pragmatic decision, as although the fineweb_1024_bpe.vocab file is small, fineweb_1024_bpe.model is large, e.g. 150 kB compressed & is presumably larger than it needs to be. I presume the model isn't uniquely reconstructable from the vocab(?))

[kyleliang919]

Not sure about your first point but to your second point you would end up paying for the large vocab size with large embedding table so it's not free advantage.

[DouglasOrr]

True. I can certainly think of ways that could make use of an arbitrarily large set of strings via this side-channel to gain an unfair advantage, but these would all hopefully fail against "Submissions that edit the tokenizer will be examined much more carefully, since bugs may unjustly improve your score."

[0hq]

This is a good point. I can imagine some ridiculous N-gram tokenizers perhaps that let you sneak in lots of bits into training. For now, let's just assume things like this are 'not in the spirit of the competition' and can be evaluated on a case-by-case basis.

[pszemraj]

hey - was thinking about this 'independently' and in the process of checking/investigating found this thread. Am I understanding this correctly that tokenizer files/code "don't count", putting training an optimized 1024 vocab (or more, if embeddings size cost is 'worth') tokenmonster optimized jointly across the train set -> eval set text (both known) is tenable?? You could probably even have a whole optimization branch of back calculating what the merges and so on should have been given the model size and other constraints...

imo that is both:

1. against the spirit of the competition (unless you really want people to optimize vocabs and/or you are fairly certain that yours is close to optimal already)
2. nonsensical in terms of "16mb artifact contains the things needed for my model" because.. any tokenizers other than the default (or some spicy charLM/ ord() etc) are clearly core files needed to use it

I think that having any tokenizer files/data other than the default count towards the total 16mb size is one way to mitigate/discourage this