Skip to content

report.md

Latest commit

 

History

History
267 lines (182 loc) · 9.62 KB

report.md

File metadata and controls

267 lines (182 loc) · 9.62 KB
Raw

Reporting test results

Test discovery, execution results in a test run can be controlled with test loggers. This document will cover details on installation, usage and authoring of test loggers.

Test loggers

A test logger is a test platform extension to control reporting of test results. It can perform tasks when a test run message, individual test results or the test run completion events are reported by the test platform.

You can author a test logger to print messages on the console, generate result files of a specific reporting format, or even report results to various CI/CD services. Default inputs to a test logger can be provided in the command line.

Please refer to this section for instructions on creating a test logger and todo if you're interested in the architecture of a test logger.

Available test loggers

Scenario NuGet Package Source Repository
Local, CI, CD Inbuilt Trx Logger
Local, CI, CD Inbuilt Console Logger
Local, CI, CD Inbuilt Html Logger
Migration only, temporary Inbuilt Test Id Report Logger
Local, CI, CD XunitXml.TestLogger Xunit Logger
Local, CI, CD NunitXml.TestLogger Nunit Logger
Local, CI, CD JunitXml.TestLogger Junit Logger
AppVeyor AppVeyor.TestLogger AppVeyor Logger
Azure Pipelines AzurePipelines.TestLogger Azure Pipelines Logger
GitHub Actions GitHubActionsTestLogger GitHub Actions Test Logger
TeamCity TeamCity.VSTest.TestAdapter TeamCity Logger

Want to add your logger? Please send a PR with changes in this doc.

Acquisition

A test logger should be made available as a NuGet package (preferred), or as a zip file (for e.g. loggers for C++ etc.).

If it's a NuGet package, the test logger assemblies should get copied to the build output directory. When looking for a test logger, vstest will look for them in the same directory as the test assemblies. In most cases this means that test projects reference either the project (same codebase) or NuGet package with the logger.

If the test logger is made available as a zip file, it should be extracted to one of the following locations:

  1. the Extensions folder along side vstest.console.exe. E.g. in case of dotnet-cli, the path could be /sdk/<version>/Extensions directory.
  2. any well known location on the filesystem

Note

New in 15.1

In case of #2, user can specify the full path to the location using /TestAdapterPath:<path> command line switch. Test platform will locate extensions from the provided directory.

Naming

Test platform will look for assemblies named *.testlogger.dll when it's trying to load test loggers.

Note

For the 15.0 version, the test loggers are also discovered from *.testadapter.dll

Create a test logger

Go through the following steps to create your own logger

  1. Add a nuget reference of package Microsoft.TestPlatform.ObjectModel.
  2. Implement ITestLoggerWithParameters (or ITestLogger, if your logger is not expecting any parameters). Logger Example
  3. Name your logger assembly *.testlogger.dll. Detailed

Enable a test logger

A test logger must be explicitly enabled using the command line. E.g.

 vstest.console test_project.dll /logger:mylogger

Where mylogger is the LoggerUri or FriendlyName of the logger.

Configure reporting

Additional arguments to a logger can also be passed in the command line. E.g.

 vstest.console test_project.dll /logger:mylogger;Setting=Value

Where mylogger is the LoggerUri or FriendlyName of the logger. Setting is the name of the additional argument and Value is its value.

It is up to the logger implementation to support additional arguments.

Syntax of default loggers

1) Console logger

Console logger is the default logger and it is used to output the test results to a terminal.

Syntax

For dotnet test or dotnet vstest (note: dotnet vstest is superseded by dotnet test, which can run assemblies directly):

--logger:console[;verbosity=<Defaults to "minimal">]

For vstest.console.exe:

/logger:console[;verbosity=<Defaults to "normal">]

Argument verbosity defines the verbosity level of the console logger. Allowed values for verbosity are quiet, minimal, normal and detailed.

Example

vstest.console.exe Tests.dll /logger:"console;verbosity=normal"

If you are using dotnet test, then use the following command:

dotnet test Tests.csproj --logger:"console;verbosity=normal"

or you can also use argument -v | --verbosity of dotnet test:

dotnet test Tests.csproj -v normal

2) Trx logger

Trx logger is used to log test results into a Visual Studio Test Results File (TRX).

Syntax

/logger:trx [;LogFileName=<Defaults to unique file name>]

Where LogFileName can be absolute or relative path. If the path is relative, it will be relative to the TestResults directory, created under current working directory.

Examples

Suppose the current working directory is c:\tempDirectory.

vstest.console.exe Tests.dll /logger:trx

trx file will get generated in location c:\tempDirectory\TestResults.

vstest.console.exe Tests.dll /logger:"trx;LogFileName=relativeDir\logFile.txt"

trx file will be `c:\tempDirectory\TestResults\relativeDir\logFile.txt`.

```shell
vstest.console.exe Tests.dll /logger:"trx;LogFileName=c:\temp\logFile.txt"

trx file will be c:\temp\logFile.txt.

3) Html logger

Html logger is used to log test results into a HTML file.

Syntax

/logger:html [;LogFileName=<Defaults to unique file name>]

Where "LogFileName" can be absolute or relative path. If path is relative, it will be relative to "TestResults" directory, created under current working directory.

Examples

Suppose the current working directory is c:\tempDirectory.

vstest.console.exe Tests.dll /logger:html

HTML file will get generated in location c:\tempDirectory\TestResults.

vstest.console.exe Tests.dll /logger:"html;LogFileName=relativeDir\logFile.html"

HTML file will be c:\tempDirectory\TestResults\relativeDir\logFile.html.

vstest.console.exe Tests.dll /logger:"html;LogFileName=c:\temp\logFile.html"

HTML file will be c:\temp\logFile.html.

4) Test id report logger

Warning

This logger is temporary and will be removed together with the SHA1 test id implementation it reports on. It exists only to support migrating stored test case ids across the change of id hashing algorithm. See Test id report logger for the full documentation, including the columns, a worked example, and the important case of adapter assigned ids.

The test id report logger writes one row per test carrying both the SHA1 derived and the xxHash128 derived test id, so that ids stored before the algorithm changes can be mapped onto the ids that replace them.

Syntax

/logger:testids [;LogFileName=<Defaults to TestIds.csv, qualified by target framework>]

Where LogFileName can be absolute or relative path. If the path is relative, it will be relative to the TestResults directory, created under current working directory. An explicitly named report is overwritten if it exists. When LogFileName is not given the default name is qualified by the target framework and an existing report is not overwritten - the next free TestIds_net8.0(1).csv is taken

  • so that a multi targeted project, or several projects of a solution sharing a results directory, do not overwrite each other's mappings.

Example

vstest.console.exe Tests.dll /logger:testids

The report will be c:\tempDirectory\TestResults\TestIds_net8.0.csv for a test assembly targeting net8.0. The path is printed at the end of the run.

Related links

TODO: link to author a test logger