Fix file splitting for sharding. Assignees: @kushalarora Labels: enhancement #53
Description
Current only 1167/1179 batches are processed as shards are unbalanced due to use of random for splitting. Count all the lines and then split equally. Maybe, split the number of tokens are also equal.