Run all tests in Visual Studio runs Explicit tests

[rprouse]

It doesn't run them when running from the command line though, so VS must be passing in a filter that includes them?

[akazakov8591]

I have the same problem with the VS solution that I created in VS 2015. It's interesting that the other solution behaves as expected - ignores the tests with the Explicit attribute. The only difference between these two solutions is that the one that behaves correctly was originally created using VS 2012. Don't know if it makes any difference, but currently I have no clue what's causing this problem in my new solution.

[rprouse]

@akazakov8591 is the VS 2012 solution .NET Core? This issue is for the .NET Core adapter. Are you using it or the NUnit 3 Test Adapter?

[CharliePoole]

For the NUnit 3 adapter, see nunit/nunit3-vs-adapter#47

[akazakov8591]

I'm running unit tests using ReSharper that is integrated with VS 2015. NUnit is istalled from NuGet package manager and the version that is used in the project where Explicit attribute does not work is 3.2.1. The version of NUnit that is used in the solution where Explicit works is 2.6.4. So, as @CharliePoole said, it's clearly a problem with the latest version of NUnit adapter. I wonder if it's going to be fixed.

[CharliePoole]

@akazakov8591 If you are running with ReSharper then you are not running with the NUnit 3 VS Adapter and our fix won't help you. :-(

Since ReSharper is proprietary, I have no idea how it runs NUnit tests. However, if they use the method of selecting test names and passing them to NUnit in a name filter, then they will have the same problem that we have in the adapter. You may want to file an issue with JetBrains on this.

[akazakov8591]

Thank you, @CharliePoole. ReSharper NUnit works with NUnit v.2.6.4 based on ReSharper NUnit help. I had to uninstall NUnit 3.2.1 package and install v.2.6.4. Explicit tests work for me as expected now.

[patrickjamesbarry]

When running all tests in Visual Studio 2015 Update 3, "dotnet-test-nunit": "3.4.0-beta-2", and dotnet version (1.0.0-preview2-003121), the test that I have marked as Ignored is causing the tests to fail. Is this the same issue this ticket represents (if not, i'll create a new one)? When running the tests that are NOT ignored, they succeed. Also, all test run as expected on command line.

Output:

No handler for message 'TestExecution.TestResult' when at state 'TestExecutionSentTestRunnerProcessStartInfo'
An exception occurred while invoking executor 'executor://dotnetunittestexecutor/v1': Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
An existing connection was forcibly closed by the remote host

[rprouse]

@patrickjamesbarry just to confirm, running Ignored tests causes the crash in the output, not Explicit tests? Probably a different bug, so I will need to look at it.

[CharliePoole]

@patrickjamesbarry Not the same issue. Explicit is a test state that is known by NUnit but unknown to Visual Studio so it requires special handling. Ignore should be well understood and translates directly to a VS state. Also, your output message is pretty specific.

@rprouse Are you still having this problem with your "adapter" ? It's long ago resolved for the NUnit 3 VS adapter. Where can I peek at your code that is equivalent to that adapter? I'm wondering if we can pull some things together across the NUnit 2, NUnit 3 and dotnet adapter code bases.

[rprouse]

@CharliePoole the test list is sent to the adapter and received in SetupSinks in https://github.com/nunit/dotnet-test-nunit/blob/master/src/dotnet-test-nunit/TestRunner.cs. They are then turned into filters in the CreateTestFilter method.

[CharliePoole]

Are you getting the list of tests to run from VS? Once you have a list of tests from VS, you are going to run explicit tests. There is no way around it because you can't determine whether a given test was included because it was selected or because a parent was selected.

I may simply not understand how this integrates with VS as well as being used on the command line. Could be a whole different thing than what I'm used to in the adapter, where VS calls specific interface members.

[rprouse]

@CharliePoole the list of tests is passed to the adapter via IPC from VS and I access it in the SetupSinks method, so as you say, I am not sure what I can do about it. I need to walk through that code in the debugger, but unfortunately, I haven't been able to run it in the debugger in Visual Studio mode.

On the command line, we get the test list via the command line using our standard --where so that works as expected.

As for understanding the difference between the command line and VS, if you look at SetupSinks, if a port is specified on the command line, I open I socket and create network (remote) sinks. That means we are running under a program like VS. If there is no port, I setup console sinks that just write to Console.Out. You will notice there that a test list is only received via the sinks if there is a port specified, ie only under VS.

[CharliePoole]

@rprouse As you know, the adapter is called directly from VS and has a separate entry point used when the user selects "run all" or when vstest console is used. At that entry, we just get a list of assemblies and we are able to leave handling of Explicit tests to NUnit itself. OTOH, when called at the entry point that gives a list of tests to run, it's impossible to correctly handle Explicit tests because we don't have enough information.

If there is no true "run all" entry point called via IPC then I think you are out of luck. For that reason, it seems as if the VS folks should implement such an interface.

[BrainCrumbz]

Hello @rprouse , @CharliePoole
It's very likely you are aware of it, but just in case:

while hunting for issues in VS test integration, to help figuring out what's going on inside VS Test Adapter (the one at the far left in these diagrams), one can also set an environment variable to get more logs in VS Output Window.

The variable name is DOTNET_TEST_TRACE and should be set to 1. This of course holds at least for preview2 tooling, haven't checked with preview3 and further.

Just found that while browsing into dotnet-test source code here.

HTH

[filmor]

@CharliePoole How about defaulting to not run Explicit tests as long as there are non-explicit tests included in the list?

[CharliePoole]

@filmor Not sure I understand....

Let's say that VS tells us to run a mix of tests, some Explicit, some not. We don't have that info readily available, but it could be looked up for each test, since we have the entire tree of tests discovered by NUnit.

I think you are saying that we should ignore what VS tells us to run and just run the non-explicit tests.

But that call from VS could have originated in a number of different ways:

1. User clicked run all
2. User selected by class, trait, etc.
3. User specifically selected those tests by name
4. Some combination of 2 and 3.

The user should expect us to ignore explicit tests in cases 1 and 2 but to run them in case 3. In case 4, we should run only those selected by name. Unfortunately, due to the design of the interface, we don't have the information needed to make the right decision. So if we did what (I think) you are suggesting, cases 3 and 4 would be broken. We would have to tell users that explicit tests are only run if they are the only tests selected.

Even if we did this, it would still not work 100% as expected because the ExplicitAttribute may also be used on a test fixture, Technically, those test cases under the fixture are not explicit. Attributes are not "inherited" by test cases. Since the VS interface only deals with test cases, those would appear to us a non-explicit and wouldn't behave as expected.

Basically, the issue here is twofold:

1. That there is not a separate API for running all tests. At least that's what I think I understand from the discussion above. In the standard vs adapter (not dotnet-test-nunit) a separate entry point is called when the user selects Run All. This makes it possible for us to deal with case 1 correctly, thereby minimizing the problem to a large degree. @rprouse Am I correct in understanding from your comments above that the run all entry point in dotnet-test-nunit is not called?

2. The overall architecture of the API makes some assumptions about how a test framework works, without providing a way for a particular adapter to indicate that it works differently. Those assumptions are pretty much based on the design of first-generation test frameworks like MSTest and very early versions of NUnit and mbUnit. In this particular case, the assumption is "We know what should be run if the user clicks RunAll" Other examples are "Test names are unique" and "We can deduce the class of a test from its name." 😞

Bear in mind that my comments are based on writing and working with the original adapters - nunit-vs-adapter and nunit3-vs-adapter. I'm relying on @rprouse to pull me back if I have misunderstood anything about how dotnet-test-adapter may differ in operation, particularly wrt run all tests.

@rprouse I"m wondering if there is anything for you to actually fix here, or if we are just stuck with the mismatch.

[rprouse]

@CharliePoole there is not a separate entry point, but there might be differences in the filter that is passed to us that we could use. That said, I'm not sure if we will be doing another release. I can't even build it now that VS 2017 is on my machine and it will stop working as soon as people update their .NET Core tooling or install VS 2017.