Non-magical workflows

[mrocklin]

When I look at the headline demonstration of benchtoolz in the README my gut reaction is that it's very magical. This turns me off a tiny bit. I wonder if there is a good non-magical workflow that we could also think about when developing.

[eriknw]

Thanks for sharing your reaction. I dislike magical things too, but, with benchmarking, I dislike the alternatives even more. If it's any consolation, the "magic" aims to be simple and straightforward.

Is your criticism of the presentation of the workflow, or the workflow itself? What other workflow would you prefer?

[eriknw]

Also, which magical parts gave you pause? benchtoolz is still largely in a proof-of-concept phase (but I think the proof-of-concept has been demonstrated), and I expect major changes to occur. So, feel free to think big!

[mrocklin]

Looking around and parsing function names is what seemed magical to me.

A more explicit version might be something like the following:

def bench_zeroes(zeros=[zeros_imul, zeros_mul, zeros_repeat, zeros_slow],
                 data={'empty': 0, 'small': 10, 'large': 10000}):
    zeros(data)

But often I actually don't want to do a comparison of many different implementations. Rather I want to just maintain a set of benchmarks and see which ones increase or decrease as I change commits. In the common case I actually just want to run a bit of functionality and time it in a reasonable manner and report that time.