Build 2015 (vNext) fails when a test is ignored using [Ignore("No reason given")]

[jessehouwing]

This is with the latest NUnit 2.6 version and the latest NUnit runner (2.0), both fetched from Nuget.

All projects are .NET 4.0

[OsirisTerje]

I'll set up a repro

[OsirisTerje]

What is happening here is that the test(s) with Ignore are in fact being skipped, but the Ignore statement including a reason text causes this message to be output in addition. This happens only on the TFS Build vNext, not on the old Xaml build and not on standalone VS.
I can't see that the adapter is adding this message explicitly either, so it might come from NUnit itself.
@CharliePoole : Does NUnit itself add the reason text to some outputs ?

[OsirisTerje]

This should be a warning text in vNext, as it is in Visual Studio. The issue has been reported to the MS PG, and we'll close it here, as we can't do anything.

[CharliePoole]

@OsirisTerje How about something like vs-followup as a label (or other words that make sense to you) That would eliminate strangeness (to me) of having both a closed: and a status: on the same issue.

[jessehouwing]

In this case it may make sense to link it to an issue in the
vso-agent-tasks project...
On Dec 21, 2015 20:00, "CharliePoole" [email protected] wrote:

@OsirisTerje https://github.com/OsirisTerje How about something like
vs-followup as a label (or other words that make sense to you) That would
eliminate strangeness (to me) of having both a closed: and a status: on the
same issue.

—
Reply to this email directly or view it on GitHub
#92 (comment)
.

[prawalagarwal]

Whatever is shown on the console in VNext builds is the output coming out from the vstest exe (the same executable which runs test for VS IDE too), which in turn is showing the output from nunit adapter. Here the ignore message comes out in the stderr stream from the nunit adapter and hence the red color. Running from CLI also prints the message but CLI is not color coding the stderr. It just color codes exit message if the exit code is non 0 from the adapter.

VS IDE handles the stderr stream and shows it as a warning message.

[jessehouwing]

Since this is informational text, should it not be pushed to the StdOut instead of StdErr?

[prawalagarwal]

Yes it should be pushed to stdout. Will be interesting to find out why nunit adapter was implemented this way.

[OsirisTerje]

The test adapter just converts whatever comes from NUnit itself, the test outcome from an Ignored test is TestOutcome.Skipped. Then the message is added to the Result.ErrorMessage property, there is no other message properties. The convertion to stderr and stdout is not in the adapter. There is a collection of TestResultMessages which is not being set. The purpose of this collection is not known to us. Is this something that should be used, and if so, where does this information end up in the test explorer ?

[OsirisTerje]

The relevant code is here: https://github.com/nunit/nunit-vs-adapter/blob/master/src/NUnitTestAdapter/TestConverter.cs

[OsirisTerje]

I see that @cklutz has a fork where he has used the message collection for that purpose, separating between stdout and stderr. I assume that is the way to go. It would be nice if he could add a PR to get that code back into the project.
Using the collection separates between the types, but what is then the ErrorMessage property used for ?

[CharliePoole]

The adapter does not write to stderr or stdout. A quick look at the code will show that and it's obvious that it can't since it's not a console program. What the adapter does is provide TestResult objects to the controlling program - vstest in this case - which then decides what to do with it.

I think what @prawalagarwal actually means is that messages can be provided as a part of the result output using categories, two of which use StdErr and StdOut in their names. AFAIK this is undocumented stuff - at least it was undocumented when I wrote the adapter.

That's why we originally marked this as not fixable by us. It looks as if @cklutz has leveraged the messages collection of the TestResult to provide a better result. We should do the same, after investigating where the various messages show up in all three of the environments we use.

[Note that the adapter was written for use in the IDE. Features that emphasize TFS or the VS console runner were tacked on later.]

@OsirisTerje Personally, I would keep this issue closed. If we want to completely change how we handle messages, it's a new feature and we should track it separately.

[CharliePoole]

@cklutz I sent you an invitation to become a contributor to NUnit. It looks like you have already done several things in your fork, which are outstanding issues in this adapter and/or the v3 adapter. I hope you'll join us.

[cklutz]

@CharliePoole Thanks! I did. Although no promises about my (current) availability ;-)

[OsirisTerje]

@cklutz Any chance for you to move the relevant code up to a branch here - so we can have a go at it ? If you can get it up, I can start do the relevant testing on it. Would also be great to have your email, mine is up in my profile.
@CharliePoole : I'll create a feature issue to track that work, but would like to keep this bug open until fixed. I'll also remove the MS followup since it seems we can handle it.

[CharliePoole]

@cklutz If you are willing to take this issue on, I'll assign it to you. You said you don't have much time, but I notice you already have the code in your fork. We could copy it, but I think it will be better done if you do it yourself.

[CharliePoole]

@cklutz Whoops! I posted on the wrong issue - reposted the comment on #97!