Using Istanbul for instrumentation (instead of V8)

[shirblc]

Hi!

I've recently moved my project from Karma to web-test-runner. I'm having trouble with coverage reports (wildly inaccurate reports are produced), though, so I decided to try the solution in #1715. However, that specific solution doesn't seem to work for my use case (TypeScript with decorators), and the one in the docs doesn't seem to produce anything different.

I've experimented a little bit and discovered that seeing nativeInstrumentation to false has no impact - web-test-runner seems to be using native instrumentation regardless of whether or not it's set to true (the first commit shows how it runs coverage regardless of the setting; the second commit shows the plugin having zero impact). Trying the code in the docs with another browser (both webkit and Firefox) produces no code coverage, which shows it's not being run at all (makes sense considering the above). Is there something I'm missing about how to use web-test-runner with custom instrumentation, or is there something broken?

This repo shows a very basic reproduction of both of those issues (just run npm install and npm run test). I've verified that the Istanbul plugin is being called using some logging, but for some reason no coverage report is being outputted, which seems like a web-test-runner issue. I'd be happy to do some more digging and see if I can find where it's going wrong, but if everything's behaving as intended and I misunderstood or the code in the docs is just outdated, it'd be great to hear.

Thank you!

[rokob]

I believe this is related to test-runner-playwright which could be fixed via #2779. If you set nativeInstrumentation to true and have the istanbul plugin enabled then you should get coverage from istanbul reported. Obviously setting nativeInstrumentation to true is not what you want to do but it could be a temporary workaround until that lands.

Note that using the test-runner-chrome which uses puppeteer will have this work, it is just the playwright runner that is broken.

[shirblc]

I've tried the repo above with the Chrome runner, Puppeteer runner and Playwright runner. All of them show the exact same behaviour.

When nativeInstrumentation is set to false without an alternative, according to the Playwright Launcher's code, it should raise an error. It doesn't. It just returns the coverage reported by V8. This, as well as the fact coverage reports generated by V8 are identical to the ones supposedly generated by the Babel plugin, leads me to believe it doesn't work for Chrome and Puppeteer either - it just returns the native coverage.

If you've got an example showing otherwise though I'd love to see it, maybe my setup is wrong. Or if you see something wrong with how I did it, I'd love to hear that either - I just want to get this resolved.

[rokob]

The key in your sample repo in web-test-runner.config.mjs is CoverageConfig it should be coverageConfig with a lowercase c at the beginning. You are getting the default config which has nativeInstrumentation turned on.

[shirblc]

I don't know how I missed that... thank you so much!

In that case, I think you're right and this should be fixed by #2779.