Add a benchmarking suite

There's been a couple of PRs lately that have been opened in hopes of speeding things up. We should probably add a good set of benchmarks that can be re-used by everyone trying to make an optimization change so as to make changes like these easier to evaluate.

Ideally, we can also yank some of Ruffs benchmarks to also see how much a change affects it does heavily consume this library.