SubLevelCommandNameShouldBeSentToTelemetry test is flaky on macOS

[MichaelSimons]

Summary

The test TelemetryFilterTests.SubLevelCommandNameShouldBeSentToTelemetry in dotnet.Tests.dll.14 fails intermittently on the macOS arm64 (osx.15.arm64.open) CI leg. It passes on Windows and Linux consistently.

Failing Test

Microsoft.DotNet.Tests.TelemetryTests.TelemetryFilterTests.SubLevelCommandNameShouldBeSentToTelemetry

Error: The test parses ["new", "console"] and expects a telemetry entry with argument=Hash("CONSOLE"), but instead gets argument=Hash("") (empty string) with subcommand=Hash("CONSOLE").

Root Cause

InstantiateCommand.ExecuteAsync (src/Cli/Microsoft.TemplateEngine.Cli/Commands/create/InstantiateCommand.cs line 250) dynamically adds template names (e.g., "console") as subcommands to the NewCommand in the static Parser.RootCommand tree:

args.NewOrInstantiateCommand.Subcommands.Add(templateCommandToRun);

This mutation persists for the lifetime of the process. In production this is not an issue because each dotnet CLI invocation is a separate process. However, in the test runner:

1. If another test in the same process triggers template instantiation (e.g., dotnet new console) before SubLevelCommandNameShouldBeSentToTelemetry runs, "console" becomes a registered subcommand of the new command.
2. When Parser.Parse(["new", "console"]) is then called, System.CommandLine classifies "console" as TokenType.Command (matched subcommand) instead of TokenType.Argument (the ShortNameArgument).
3. The AllowListToSendVerbSecondVerbFirstArgument telemetry rule reads the token types to determine secondVerb vs firstArgument, resulting in subcommand="CONSOLE" and argument="".

The intermittent nature is due to non-deterministic xUnit test ordering — the test passes when it runs before any template instantiation pollutes the static parser, and fails when it runs after.

Evidence