Unit tests for command/option parsing correctness in CLI app?

[ptr727]

Looking for advice and example code on how to add unit tests for CLI parsing correctness?

Some way to provide a string, run parse on it, then test for success/failure and value presence, seems not too difficult, but is there a way to express the results in e.g. xUnit [Theory] section?

[zivkan]

What I do is have a method that creates the parser data structures and takes in Func<????, Task<int>> delegates that each command sets as its action. My Program.Main passes in the product code that executes the commands, while test code passes in a delegate that validates that all the function arguments have the expected values.

For example, something like:

[Theory]
[InlineData("quiet", LogLevel.Quiet)]
[InlineData("detailed", LogLevel.Detailed)]
public async Task ValidateFooBarVerbosity(string input, LogLevel expected)
{
    CliCommand parser = FooBarCommand.Create(ValidateArguments);
    _ = await parser.Parse($"foo -v {input}").InvokeAsync();
   
    async Task ValidateArguments(FooBarArguments args)
    {
        args.Verbosity.Should().Be(expected);
    }
}

Writing tests for error conditions is a bit different, instead of parser.Parse(...).InvokeAsync(), it's just var result = parser.Parser(...);, and there's some properties on the parse results data structure that you can validate that it caught an error.

This way I can write many exhaustive tests for the CLI parsing, without it actually executing the product code, so it's nice and fast to execute and doesn't have side effects.

[ptr727]

Can you share examples on how to parse the values in ParseResult, I've just a bit of time navigating the structure in the debugger, I can see the command, but I can't figure out how to to get access to global and command specific parameters?

Re no built-in binding support in beta 5, what does that mean?

[jonsequitur]

By calling ParseResult.GetValue (renamed in beta5 from GetValueForArgument / GetValueForOption), you can check the parsed value of any option or argument.

You can see lots of usage examples in these tests:

https://github.com/dotnet/command-line-api/blob/060374e56c1b2e741b6525ca8417006efb54fbd7/src/System.CommandLine.Tests/GetValueByNameParserTests.cs

https://github.com/dotnet/command-line-api/blob/e8282e765515d32c96ea6adb2ec83653ed1379a5/src/System.CommandLine.Tests/Binding/TypeConversionTests.cs

Re no built-in binding support in beta 5, what does that mean?

This refers to the functionality in earlier versions that allowed System.CommandLine to create an instance of a class with properties like Abc and Xyz bound by convention from options like --abc and --xyz.

[ptr727]

This refers to the functionality in earlier versions that allowed System.CommandLine to create an instance of a class with properties like Abc and Xyz bound by convention from options like --abc and --xyz.

Hmm, are you talking about System.CommandLine.NamingConventionBinder I use this, it is an easy way to have a single class with properties representative of all possible options, then I just pass the class around. To my knowledge this was the only way to do it for multiple options, at least when the I thing 8 limit was created. What will the alternative / migration path be?

For parsing, looks like Beta4 does not have a GetValueForOption variant that takes just the option string, and I need to recreate the Option object for every parameter, too much work to either recreate the option or to expose all options are reusable functions. Using a local function and Invoke is easier. I'll re-evaluate when Beta5 makes it to nuget.

[jonsequitur]

Hmm, are you talking about System.CommandLine.NamingConventionBinder

Yes.

I'll re-evaluate when Beta5 makes it to nuget.

It's available on the build feed here if you'd like the evaluate sooner:

https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet-libraries/NuGet/System.CommandLine/versions

[ptr727]

Yes.

What are my options for migration away from System.CommandLine.NamingConventionBinder?

Any docs for b4 to b5 migration available, I see lots of errors that need correcting in my code.