Dynamic `OperationsPerInvoke`

[summivox]

I load a list of strings externally as data for my benchmarks. My goal is to measure average time for me to process one string. Of course the data is loaded before benchmarking, but since it happens at runtime, I cannot supply OperationsPerInvoke attribute param to BenchmarkAttribute, which requires compile-time constant. Is there a way around?

[redknightlois]

@AndreyAkinshin Any idea when this is gonna be implemented? Cause we really need it. If you didnt started already I can tell @jbayardo to implement it and send the PR.

[AndreyAkinshin]

Hi @redknightlois. It seems easy to implement but I'm not sure about API. Do you have any ideas how it should look? Also, I want to add an ability to specify dynamic Params (a method which returns values for Params field/property instead of a constant array). It would be cool to use the same approach for the both cases.

[summivox]

For OperationsPerInvoke: My preference is for the benchmark method to return an int showing how many iterations it actually run.

[redknightlois]

I use returning values to avoid dead code elimination issues, so the result approach could not be used generically. Having an Attribute that you can bind to a property could be more enough. Also providing a delegate to the Job configuration could work too.

[summivox]

@redknightlois while I also used return value as an attempt to prevent dead code elimination, I feel there could be a more explicit way (e.g. Benchmark.Sink(value)).

Using return value has the following advantages: it's still specified in the benchmark method, does not require further reflection, can be easily extended to async (Task<int>).

[redknightlois]

If it can be done, it is actually a very good idea to have a Benchmark.Sink(value) or Benchmark.PreventElimination(value)

Probably it could be done with something like this:

[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization | MethodImplOptions.PreserveSig)]
public void PreventElimination<T>(T value)
{ }

We should look into the assembler to be sure, but sounds like a viable thing.

[summivox]

@AndreyAkinshin : Just realized that in order for this to be backwards-compatible, the benchmark method should still be tagged with an attribute (e.g. ReturnOperationsPerInvoke).

Fiddling with code now.

[AndreyAkinshin]

I think, we shouldn't use the returned value as OperationPerInvoke. It looks like a hack, it force you to design the method body in a special way, it can not be generalized for other benchmark attributes.

it is actually a very good idea to have a Benchmark.Sink(value) or Benchmark.PreventElimination(value)

@redknightlois, we will have such method in v0.10.0, I'm working on it right now (it's the last thing in the engine which I want to finish for this release). Unfortunately, it's not easy to implement it right way. Your PreventElimination<T> will not work well and could spoil some benchmark results. It will be possible to use my new Sink method inside a benchmark, but I'm not sure that it's a good idea in general case: it's still easy to design a wrong microbenchmark.

So, I suggest to continue to use returning values to avoid dead code elimination, and come up with another way to specify OperationPerInvoke.

[dazinator]

Any further news on this?

[AndreyAkinshin]

@dazinator, I definitely want to implement it in the nearest future, but I still don't know how the API should look like.