Updating a test's status that has already been reported results in a new test counter

[thomhurst]

This may not be considered a bug, so close if it won't be fixed.

Describe the bug

If a test that has previously been reported, is reported again with a new status, it is counted as another test, and doesn't just update the existing one.

Steps To Reproduce

Test A > Report Success
Test A > Report Failure (maybe a hook that ran after it failed, and you need to attach that failure to something?)

// See https://aka.ms/new-console-template for more information

using Microsoft.Testing.Platform.Builder;
using Microsoft.Testing.Platform.Capabilities;
using Microsoft.Testing.Platform.Capabilities.TestFramework;
using Microsoft.Testing.Platform.Extensions.Messages;
using Microsoft.Testing.Platform.Extensions.TestFramework;
using Microsoft.Testing.Platform.TestHost;

var builder = await TestApplication.CreateBuilderAsync(args);
builder.RegisterTestFramework(_ => new TestFrameworkCapabilities(), (_, _) => new DummyAdapter());
var app = await builder.BuildAsync();
return await app.RunAsync();

internal class DummyAdapter : ITestFramework, IDataProducer
{
    public string Uid => nameof(DummyAdapter);

    public string Version => string.Empty;

    public string DisplayName => string.Empty;

    public string Description => string.Empty;

    public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)];

    public Task<CloseTestSessionResult> CloseTestSessionAsync(CloseTestSessionContext context) => Task.FromResult(new CloseTestSessionResult { IsSuccess = true });

    public Task<CreateTestSessionResult> CreateTestSessionAsync(CreateTestSessionContext context) => Task.FromResult(new CreateTestSessionResult { IsSuccess = true });

    public Task ExecuteRequestAsync(ExecuteRequestContext context)
    {
        context.MessageBus.PublishAsync(this, new TestNodeUpdateMessage(new SessionUid("1"), new TestNode
        {
            Uid = "2",
            DisplayName = "Blah",
            Properties = new PropertyBag(PassedTestNodeStateProperty.CachedInstance)
        }));
        
        context.MessageBus.PublishAsync(this, new TestNodeUpdateMessage(new SessionUid("1"), new TestNode
        {
            Uid = "2",
            DisplayName = "Blah",
            Properties = new PropertyBag(new TimeoutTestNodeStateProperty(new Exception()))
        }));
     
        context.Complete();
        
        return Task.CompletedTask;
    }

    public Task<bool> IsEnabledAsync() => Task.FromResult(true);
}

internal class Capabilities : ITestFrameworkCapabilities
{
    IReadOnlyCollection<ITestFrameworkCapability> ICapabilities<ITestFrameworkCapability>.Capabilities
        => Array.Empty<ITestFrameworkCapability>();
}

Expected behavior

Previously reported status with same test Uid is overwritten with new status

Actual behavior

Updated status is counted as a new test

Test run summary: Failed!
total: 2
failed: 1
succeeded: 1
skipped: 0
duration: 71ms

[nohwnd]

The current behavior seems correct, the result of the test is the final state, overwriting it would likely force us to rewrite some or lot of the code to work on list of events, rather than aggregating the momentary state. We would also have hard time knowing when the final state arrived, or if there will be more updates in the future.

To me the test should report one single final result when it is done running and done tearing down. If more errors happen later, additional error should be reported to the scope that contains it, or to the whole run.

[Evangelink]

The behavior is also provided to somehow match VSTest behavior when a test is not expanded/unfolded and so multiple results are possible for the same "test case".