Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate benchmarks to criterion #14

Open
ordian opened this issue Sep 3, 2019 · 3 comments
Open

migrate benchmarks to criterion #14

ordian opened this issue Sep 3, 2019 · 3 comments

Comments

@ordian
Copy link

ordian commented Sep 3, 2019

https://github.com/bheisler/criterion.rs compiles on rust stable, meaning you can cargo check --benches in your CI to catch accidentally broken benches. You also get charts and statistics for free.

@oconnor663
Copy link
Owner

I've used Criterion in a few places before. The reason I haven't used it here is that it seems to take a very long time to run by default, while the test::Bencher framework seems to give me numbers relatively quickly. At least in the context of this project, the results from test::Bencher seem to be pretty reliable, and it's been nice to be able to do really quick benchmarks while I play with inlining flags and things like that.

Maybe you can suggest some non-default parameters for Criterion that would make it run faster? But I don't know enough about the crate to know what to try.

@ordian
Copy link
Author

ordian commented Sep 4, 2019

I guess something like this should help:

criterion_group! {
	name = bench_multiprocess;
	config = dont_take_an_eternity_to_run();
	targets = blake2b_hash_many,
		hash_sneves_blake2sp,
		openssl_sha1,
}
criterion_main!(bench_multiprocess);

fn dont_take_an_eternity_to_run() -> Criterion {
	Criterion::default().nresamples(1_000)
		.without_plots()
		.sample_size(10)
}

For more options see https://bheisler.github.io/criterion.rs/criterion/struct.Criterion.html

@oconnor663
Copy link
Owner

I'd be happy to take a pull request for this as long as we could keep the total benchmarking time under, say, 200% of what it is now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants