No more Benchmark runs will be launched as NO measurements were obtained from the previous run! #40
Description
Using the exact dependencies (NUnit 3.7.1, Benchmark.NET 0.10.8 and CodeJam.PerfTests.NUnit 0.1.4-beta) and the wiki example:
[Category("PerfTests: NUnit examples")]
[CompetitionAnnotateSources] // Opt-in feature: source annotations.
public class SimplePerfTest
{
private const int Count = 200;
// Perf test runner method.
[Test]
public void RunSimplePerfTest() => Competition.Run(this);
// Baseline competition member.
// All relative metrics will be compared with metrics of the baseline method.
[CompetitionBaseline]
public void Baseline() => Thread.SpinWait(Count);
// Competition member #1. Should take ~3x more time to run.
[CompetitionBenchmark]
public void SlowerX3() => Thread.SpinWait(3 * Count);
// Competition member #2. Should take ~5x more time to run.
[CompetitionBenchmark]
public void SlowerX5() => Thread.SpinWait(5 * Count);
// Competition member #3. Should take ~7x more time to run.
[CompetitionBenchmark]
public void SlowerX7() => Thread.SpinWait(7 * Count);
}
I get the following:
Test completed with errors, details below.
Errors:
* Run #1: No result reports for benchmarks: Baseline, SlowerX3, SlowerX5, SlowerX7.
at CodeJam.PerfTests.Running.Core.NUnitCompetitionRunner.ReportExecutionErrors(String messages, CompetitionState competitionState) in C:\Users\igors\Source\Repos\CodeJam\PerfTests\src-NUnit\Running.Core\NUnitCompetitionRunner.cs:line 81
at CodeJam.PerfTests.Running.Core.CompetitionRunnerBase.RunCore(Type benchmarkType, ICompetitionConfig competitionConfig, CompetitionFeatures competitionFeatures) in C:\Users\igors\Source\Repos\CodeJam\PerfTests\src\[L5_AllTogether]\Running.Core\CompetitionRunnerBase.cs:line 233
BenchmarkDotNet=v0.10.8, OS=Windows 10 Redstone 1 (10.0.14393)
Processor=Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), ProcessorCount=8
Frequency=3515621 Hz, Resolution=284.4448 ns, Timer=TSC
[Host] : Clr 4.0.30319.42000, 64bit RyuJIT-v4.7.2558.0
Job=Competition Force=False Toolchain=InProcessToolchain
InvocationCount=256 LaunchCount=1 RunStrategy=Throughput
TargetCount=256 UnrollFactor=16 WarmupCount=128
AdjustMetrics=True
Method | Mean | StdDev | Scaled | Scaled-StdDev | GcAllocations |
--------- |-----:|-------:|-------:|--------------:|--------------:|
Baseline | NA | NA | 1.00 | 0.00 | ? |
SlowerX3 | NA | NA | ? | ? | ? |
SlowerX5 | NA | NA | ? | ? | ? |
SlowerX7 | NA | NA | ? | ? | ? |
Benchmarks with issues:
SimplePerfTest.Baseline: Competition(Force=False, Toolchain=InProcessToolchain, InvocationCount=256, LaunchCount=1, RunStrategy=Throughput, TargetCount=256, UnrollFactor=16, WarmupCount=128)
SimplePerfTest.SlowerX3: Competition(Force=False, Toolchain=InProcessToolchain, InvocationCount=256, LaunchCount=1, RunStrategy=Throughput, TargetCount=256, UnrollFactor=16, WarmupCount=128)
SimplePerfTest.SlowerX5: Competition(Force=False, Toolchain=InProcessToolchain, InvocationCount=256, LaunchCount=1, RunStrategy=Throughput, TargetCount=256, UnrollFactor=16, WarmupCount=128)
SimplePerfTest.SlowerX7: Competition(Force=False, Toolchain=InProcessToolchain, InvocationCount=256, LaunchCount=1, RunStrategy=Throughput, TargetCount=256, UnrollFactor=16, WarmupCount=128)
============= SimplePerfTest =============
// ? Squid.Strategies.Tests.Collect2.SimplePerfTest, Squid.Strategies.Tests
---- Run 1, total runs (expected): 1 -----
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!
// Exception: System.InvalidOperationException: Sequence contains no elements
at System.Linq.Enumerable.Last[TSource](IEnumerable`1 source)
at BenchmarkDotNet.Running.BenchmarkRunnerCore.Execute(ILogger logger, Benchmark benchmark, IToolchain toolchain, BuildResult buildResult, IConfig config, IResolver resolver, GcStats& gcStats)
at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!
// Exception: System.InvalidOperationException: Sequence contains no elements
at System.Linq.Enumerable.Last[TSource](IEnumerable`1 source)
at BenchmarkDotNet.Running.BenchmarkRunnerCore.Execute(ILogger logger, Benchmark benchmark, IToolchain toolchain, BuildResult buildResult, IConfig config, IResolver resolver, GcStats& gcStats)
at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!
// Exception: System.InvalidOperationException: Sequence contains no elements
at System.Linq.Enumerable.Last[TSource](IEnumerable`1 source)
at BenchmarkDotNet.Running.BenchmarkRunnerCore.Execute(ILogger logger, Benchmark benchmark, IToolchain toolchain, BuildResult buildResult, IConfig config, IResolver resolver, GcStats& gcStats)
at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!
// Exception: System.InvalidOperationException: Sequence contains no elements
at System.Linq.Enumerable.Last[TSource](IEnumerable`1 source)
at BenchmarkDotNet.Running.BenchmarkRunnerCore.Execute(ILogger logger, Benchmark benchmark, IToolchain toolchain, BuildResult buildResult, IConfig config, IResolver resolver, GcStats& gcStats)
at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)
There are no any results runs
There are no any results runs
There are no any results runs
There are no any results runs
Benchmarks with issues:
SimplePerfTest.Baseline: Competition(Force=False, Toolchain=InProcessToolchain, InvocationCount=256, LaunchCount=1, RunStrategy=Throughput, TargetCount=256, UnrollFactor=16, WarmupCount=128)
SimplePerfTest.SlowerX3: Competition(Force=False, Toolchain=InProcessToolchain, InvocationCount=256, LaunchCount=1, RunStrategy=Throughput, TargetCount=256, UnrollFactor=16, WarmupCount=128)
SimplePerfTest.SlowerX5: Competition(Force=False, Toolchain=InProcessToolchain, InvocationCount=256, LaunchCount=1, RunStrategy=Throughput, TargetCount=256, UnrollFactor=16, WarmupCount=128)
SimplePerfTest.SlowerX7: Competition(Force=False, Toolchain=InProcessToolchain, InvocationCount=256, LaunchCount=1, RunStrategy=Throughput, TargetCount=256, UnrollFactor=16, WarmupCount=128)
// ! #1.1 01.442s, ExecutionError@Analyser: No result reports for benchmarks: Baseline, SlowerX3, SlowerX5, SlowerX7.
// ! Hint: Ensure that benchmarks were run successfully and did not throw any exceptions.
// ! Breaking competition execution. High severity error occured.