Allow tokenization for gzipped json files in datapreprocess#285

taidnguyen wants to merge 1 commit intomlfoundations:mainmlfoundations/open_lm:mainfrom

taidnguyen:maintaidnguyen/open_lm:mainCopy head branch name to clipboard