Add Roslyn analyzers to detect incorrect usage of BenchmarkDotNet

[silkfire]

This PR introduces an extensive set of analyzers that warns the user of incorrect usage of BenchmarkDotNet. This is something that has been asked since 2017 but has yet to be included as of this date. BDN has a set of validators that use reflection to detect errors but they are only triggered after the benchmark code has been compiled and is about to run.

I had the idea to implement this in 2022 but the testing framework back then wasn't trivial to use so I gave up in the end. Today, the Roslyn analyzer testing is completely testing framework-agnostic, making things considerably easier. It's also trivial to add multiple source files, references and framework assemblies in order to test your analyzer precisely the way you want.
All unit tests are implemented using xUnit v2.

With these analyzers, developers can detect errors early and solve them immediately. The descriptions are very clear and succinct, guiding the user and explaining the reasoning behind the specific rule.

Here's a list of currently implemented analyzers. There are still some remaining but I believe this is a good start and covers most common usage errors. The rest is up for grabs and can be added along the way.

Rule ID	Category	Severity	Title	Description	Comment
BDN1000	Usage	Error	Benchmark class (or any of its ancestors) has no annotated method(s)	The referenced benchmark class (or any of its inherited classes) must have at least one method annotated with the [Benchmark] attribute	Triggers when invoking BenchmarkRunner.Run<BenchmarkClass>() and the benchmark class BenchmarkClass (or any of its inherited classes) has no public methods marked with the [Benchmark] attribute
BDN1001	Usage	Error	Benchmark classes must be public	A benchmark class referenced in the BenchmarkRunner.Run method must be public
BDN1002	Usage	Error	Benchmark classes must be unsealed	A benchmark class referenced in the BenchmarkRunner.Run method must be unsealed
BDN1003	Usage	Error	Benchmark classes must be non-abstract	A benchmark class referenced in the BenchmarkRunner.Run method must be non-abstract
BDN1004	Usage	Error	Generic benchmark classes must be annotated with at least one [GenericTypeArguments] attribute	A generic benchmark class referenced in the BenchmarkRunner.Run method must be annotated with at least one [GenericTypeArguments] attribute
BDN1100	Usage	Error	Benchmark classes annotated with a [GenericTypeArguments] attribute must be non-abstract	A benchmark class annotated with a [GenericTypeArguments] attribute must be non-abstract
BDN1101	Usage	Error	Benchmark classes annotated with a [GenericTypeArguments] attribute must be generic	A benchmark class annotated with a [GenericTypeArguments] attribute must be generic, having between one to three type parameters
BDN1102	Usage	Error	Number of type arguments passed to a [GenericTypeArguments] attribute must match the number of type parameters on the targeted benchmark class	The number of type arguments passed to a [GenericTypeArguments] attribute must match the number of type parameters on the targeted benchmark class
BDN1103	Usage	Error	Benchmark methods must be public	A method annotated with the [Benchmark] attribute must be public
BDN1104	Usage	Error	Benchmark methods must be non-generic	A method annotated with the [Benchmark] attribute must be non-generic
BDN1105	Usage	Error	Benchmark classes must be non-static	A benchmark class must be an instance class
BDN1106	Usage	Error	Single null argument to the [BenchmarkCategory] attribute results in unintended null array
BDN1107	Usage	Error	Only one benchmark method can be baseline per class
BDN1108	Usage	Warning	Only one benchmark method can be baseline per class and category
BDN1200	Usage	Error	Only one parameter attribute can be applied to a field	Parameter attributes are mutually exclusive; only one of the attributes [Params], [ParamsSource] or [ParamsAllValues] can be applied to a field at any one time
BDN1201	Usage	Error	Only one parameter attribute can be applied to a property	Parameter attributes are mutually exclusive; only one of the attributes [Params], [ParamsSource] or [ParamsAllValues] can be applied to a property at any one time
BDN1202	Usage	Error	Fields annotated with a parameter attribute must be public	A field annotated with a parameter attribute must be public
BDN1203	Usage	Error	Properties annotated with a parameter attribute must be public	A property annotated with a parameter attribute must be public
BDN1204	Usage	Error	Fields annotated with a parameter attribute cannot be read-only	Parameter attributes are not valid on fields with a readonly modifier
BDN1205	Usage	Error	Parameter attributes are not valid on constant field declarations	Parameter attributes are not valid on constant field declarations
BDN1206	Usage	Error	Properties annotated with a parameter attribute cannot have an init-only setter	A property annotated with a parameter attribute must have a public, assignable setter i.e. { set; }
BDN1207	Usage	Error	Properties annotated with a parameter attribute must have a public setter	A property annotated with a parameter attribute must have a public setter; make sure that the access modifier of the setter is empty and that the property is not an auto-property or an expression-bodied property.
BDN1300	Usage	Error	The [Params] attribute must include at least one value	The [Params] attribute requires at least one value. No values were provided, or an empty array was specified.
BDN1301	Usage	Error	Type of all value(s) passed to the [Params] attribute must match the type of (or be implicitly convertible to) the annotated field or property	The type of each value provided to the [Params] attribute must match the type of (or be implicitly convertible to) the field or property it is applied to
BDN1302	Usage	Warning	Unnecessary single value passed to [Params] attribute	Providing a single value to the [Params] attribute is unnecessary. This attribute is only useful when provided two or more values.
BDN1303	Usage	Error	The [ParamsAllValues] attribute cannot be applied to fields or properties of enum types marked with [Flags]	The [ParamsAllValues] attribute cannot be applied to a field or property of an enum type marked with the [Flags] attribute. Use this attribute only with non-flags enum types, as [Flags] enums support bitwise combinations that cannot be exhaustively enumerated.
BDN1304	Usage	Error	The [ParamsAllValues] attribute is only valid on fields or properties of enum or bool type and nullable type for another allowed type	The [ParamsAllValues] attribute can only be applied to a field or property of enum or bool type (or nullable of these types)
BDN1400	Usage	Error	[Arguments] attribute can only be used on methods annotated with the [Benchmark] attribute	The [Arguments] attribute can only be used on methods annotated with the [Benchmark] attribute
BDN1401	Usage	Error	Benchmark methods without [Arguments] attribute(s) cannot declare parameters	This method declares one or more parameters but is not annotated with any [Arguments] attributes. To ensure correct argument binding, methods with parameters must explicitly be annotated with one or more [Arguments] attributes. Either add the [Arguments] attribute(s) or remove the parameters.
BDN1402	Usage	Error	Number of values passed to an [Arguments] attribute must match the number of parameters declared in the targeted benchmark method	The number of values passed to an [Arguments] attribute must match the number of parameters declared in the targeted benchmark method
BDN1403	Usage	Error	Values passed to an [Arguments] attribute must match exactly the parameters declared in the targeted benchmark method in both type (or be implicitly convertible to) and order	The values passed to an [Arguments] attribute must match the parameters declared in the targeted benchmark method in both type (or be implicitly convertible to) and order

TODO

• Check that the argument to [ArgumentsSource] points to a valid method
• Check that the argument to [ParamsSource] points to a valid method

See #2666 for discussion as well as #389.

[silkfire]

@dotnet-policy-service agree

[silkfire]

I'm not sure whether the analyzers should be automatically enabled with the base BenchmarkDotNet package or be opt-in via its own NuGet package, what do you think?

[timcassell]

They should be enabled by default.

[silkfire]

So maybe the VSIX package project can be removed then as the analyzer can be referenced through an analyzer project reference.

[timcassell]

So maybe the VSIX package project can be removed then as the analyzer can be referenced through an analyzer project reference.

I actually think the analyzer should be included directly into the annotations package. Otherwise, it was found that a separate analyzer package pulls in too many unnecessary dependencies. It's a bit complicated to set up the build to do it, though, so I can do it separately after this is merged if you want. [Edit] Or I can push to your branch after your changes are complete.

[silkfire]

I'm getting Referenced assembly 'BenchmarkDotNet.Analyzers, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' does not have a strong name. when trying to compile the test project.

[timcassell]

You need to import common.props in the analyzer project, too.

[silkfire]

Solved it. I also needed to add the public key of the assembly to the InternalsVisibleTo attribute.

[silkfire]

So maybe the VSIX package project can be removed then as the analyzer can be referenced through an analyzer project reference.

I actually think the analyzer should be included directly into the annotations package. Otherwise, it was found that a separate analyzer package pulls in too many unnecessary dependencies. It's a bit complicated to set up the build to do it, though, so I can do it separately after this is merged if you want. [Edit] Or I can push to your branch after your changes are complete.

Do I just reference the analyzer project from the annotations project? Or do I need to do something special for the analyzers to activate for the user? Mind that we of course don't want the analyzers to activate for the annotations project, but transitively for the user.

[timcassell]

Do I just reference the analyzer project from the annotations project?

Yes that's sufficient for now.

[timcassell]

Also you should move the analyzers test project to under the tests/ directory (and move the analyzers project up 1 level).

[silkfire]

Is Baseline = true unique per benchmark category or per benchmark class?

[timcassell]

Is Baseline = true unique per benchmark category or per benchmark class?

It's per category.

[timcassell]

Please also test these cases:

private const int x = 100;

[Params(x)]
public int num;

[Arguments(x)]
public void Benchmark(int i) { }

[Params(DifferentType.SomeConst)]
public int num;

[Arguments(DifferentType.SomeConst)]
public void Benchmark(int i) { }

[silkfire]

Please also test these cases:

private const int x = 100;

[Params(x)]
public int num;

[Arguments(x)]
public void Benchmark(int i) { }

[Params(DifferentType.SomeConst)]
public int num;

[Arguments(DifferentType.SomeConst)]
public void Benchmark(int i) { }

I had that in the pipeline too.

[silkfire]

I'm not sure what's causing the build errors to be honest, as I haven't touched the original BenchmarkDotNet project. I'm trying to run the tests locally but am getting the error Referenced assembly 'BenchmarkDotNet.IntegrationTests.FSharp, Version=0.14.1.0, Culture=neutral, PublicKeyToken=null' does not have a strong name.

[timcassell]

You can check the logs https://github.com/dotnet/BenchmarkDotNet/actions/runs/18837492926

[silkfire]

I see a lot of build errors are due to error CS1591: Missing XML comment for publicly visible type or member which shouldn't be relevant for an analyzer. I'll disable them on the project-level.

[silkfire]

All workflows are green except the test-macos (x64) job. Is it an environment issue or something on my part?

[timcassell]

Flaky tests, unrelated to these changes.

Is there any more work you want to do here, or it's good to go?

[silkfire]

I feel very confident in releasing this first iteration of the analyzers. They cover most common scenarios that were requested in the discussion and I or other contributors can always build further on top of this. Thanks for your patience and guidance during the review process!

As you mentioned among other things what is remaining is to adjust the pipeline. I noticed that the Analyzers project is built in debug mode:

BenchmarkDotNet.Analyzers -> D:\a\BenchmarkDotNet\BenchmarkDotNet\src\BenchmarkDotNet.Analyzers\bin\Debug\netstandard2.0\BenchmarkDotNet.Analyzers.dll

[timcassell]

Alright, I'll work on that and push to your branch when I get some time.

[AndreyAkinshin]

@silkfire thank you very much for working on this!

I'm not really sure how the combination of categories is supposed to work (or if it's intended?)

The categories were introduced in the early days of BenchmarkDotNet (2016-2017). No prior design work was done: it was an experimental feature that evolved gradually based on user feedback. If some behavior feels unnatural or some corner cases don't make much sense, feel free to suggest changes.

[silkfire]

Are we okay with the naming of the project, as it might be conflated with the BenchmarkDotNet.Analysers namespace?

[timcassell]

Are we okay with the naming of the project, as it might be conflated with the BenchmarkDotNet.Analysers namespace?

It's fine. It won't be visible to users anyway.

[AndreyAkinshin]

Are we okay with the naming of the project, as it might be conflated with the BenchmarkDotNet.Analysers namespace?

Fine for me too. I have thoughts on reworking the current Analysers and maybe even dropping this concept from the public API. Currently, they are used mostly for statistical post-checks (moving to perfolizer and pragmastat) and error aggregation (moving to core logic). With the new result data format, we won't need an extension point here since users will have other ways to implement their post-checks. It's a huge piece of work with no ETA yet, but long-term we can resolve this name clash.

I don't see any issues with using this name for the new package now. On the compilation level, it won't lead to any conflicts since the new package uses American spelling (Analyzers) while the existing namespace uses British spelling (Analysers).