each sequence starts with a new document?

[fan1dy]

I read in your paper that"when packing multiple documents into one sequence, we ensure each sequence starts with a new document rather than in the middle of one." Have you done any ablation studies on this? What is the impact on the downstream performance when just packing the documents in the standard fashion? Thanks!

[gaotianyu1350]

Hi,

We found in a preliminary experiment that it hurts the performance a bit if we don't do this. Our guess is that when we don't make sure each sequence starts with the beginning of a document, models won't learn to utilize the metadata at the beginning as well. Hope that makes sense!