How is the tokenizer supposed to be used?

[mark-hahn]

I am developing an app that could probably use jscpd. I want to find out the most efficient way to use it. I see that there is a tokenizer available. What use would there be for a tokenizer? Would the search go faster with tokens pre-generated? If so, is this because the tokens are shorter than the original text when doing the Rabin-Karp search?