Implement automatic test suite skipping facility into the browser test harness

[juj]

Implement automatic test suite skipping facility into the browser test harness, so that users do not need to manually maintain skip lists of features. (or in present state, have to maintain fewer of them).

Paves the way toward test harness working for users out of the box, improving First-Time User Experience, rather than have a large bulk of tests fail for users by default.

[sbc100]

Very nice!

I do worry a little bit about any mechanism that automatically skips test though.

Having to spell out EMTEST_LACKS_GRAPHICS_HARDWARE explicitly in the CI configuration means that the we don't need to trust the auto-disable feature.

This is why we have EMTEST_SKIP_NODE_CANARY explicitly rather than trying to automatically figure out at runtime if we should skip them.

I think I've seen cases where we were mistakenly always skipping certain tests due to a bug in the auto-skip magic, and its went unnoticed.

Imagine this scenario where we have an auto-skip mechanism:

1.I setup a CI bot with idea that it will run all the tests under node.
2. I install node on the bot
3. I run the tests and see that they all pass
4. Someone adds some new tests that depend on a more recent version of node.
5. An auto-skip mechanism means I never actually run them and bot keeps passing

I would much rather have my the bot start failing at (4) so can explictly choose to either:

1. Update node.
2. Set some environment variable such as SKIP_NODE24 or SKIP_MEMORY64 to opt out explicitly.

Another example is that we don't automatically skip d8 tests when d8 is not found. We expect folks to explicitly opt out with EMTEST_SKIP_V8.

Maybe your are not proposing that we go that far, but JIC I thought I would mention the rationale for the status quo.

[sbc100]

Can we not use firefox --version for this?

[juj]

firefox --version seems to be a linux thing. It doesn't exist on Windows.

[juj]

There are multiple use cases that motivate this work:

1. Whenever we have people come in to raise a bug that something is not working for them, the most common reply from the team is to say "do the tests work for you?" to get to a baseline understanding of what could be at fault.

Then if the user goes and points EMTEST_BROWSER=my_browser and runs test/runner browser, then depending on their browser, they could get some hundred failures in the browser suite. Most of these failures will be just some random JavaScript syntax errors, that do not give a clue to the reader that the failures would be due to a missing feature in browser, leading the user to think that the project would be poorly developed/maintained/tested/whatever.

Our EMTEST_SKIP_* and EMTEST_LACKS_* are not discoverable by users, and even if they were, there is no hint/connection to the user to figure out which of these random JS syntax errors would be addressed by which env. vars, unless they try out applying any and all of them for their particular browser, or familiarize themselves with the gory internals of the harness and the details of the failing tests. Which takes a lot of effort.

2. When we get new contributors raising PRs, like is the case now at fix: added missing options to EmscriptenAudioWorkletNodeCreateOptions closes #23982 #25497 (comment) for example.. the answer to give in practice (if we want to be sure they get a green run) is "ehh.. it's complicated", whereas it should just be as simple as "Point EMTEST_BROWSER to your browser and run test/runner browser".

3. When you are an Emscripten developer raising a bug report up to a browser database, like for example in https://bugzilla.mozilla.org/show_bug.cgi?id=1992558#c2 , it is risky to state "to repro, run test/runner browser", since that will likely provide that barrage of tests failing in JS syntax errors. This makes it more difficult to report complex harness-related issues to upstream browser vendors. (I have two harness-related bugs I would like to report to upstream Safari, but it is hard to make a quality repro case here that wouldn't give them red herrings)

4. When you are a downstream user of Emscripten, who is shipping code on a particular set of hardware * OS * browser * browser_min_version combinations, and you need to ensure that all these combinations will actually work like you advertise to your customers.. you'll need to setup a CI to verify that it does. But that leads you to have to discover all the intricate details of every single test, to manually manage the skips for every combination. That takes a lot of work.

And when you do all that work, there is currently no model to contribute that information back to the project, so that others running the harness could benefit from out of the box. Other people running the suite will need to independently do that same work. Until now by this PR, that is.

1.I setup a CI bot with idea that it will run all the tests under node.
2. I install node on the bot
3. I run the tests and see that they all pass
4. Someone adds some new tests that depend on a more recent version of node.
5. An auto-skip mechanism means I never actually run them and bot keeps passing

Yeah, I agree this kind of accident could happen. Here I would stress that the fact that all existing functionality will still be green, i.e. the Emscripten project will not have been regressed (at least as far as test coverage is concerned), since if there is a new feature that requires newer Node, then all the tests that are running against older Node will still be ok, verifying that the project runs successfully on the older Node. So nothing has regressed(?) - maybe just the newly added test coverage is not being incorporated correctly.

It is definitely a burden on the project that it also needs to keep up with new Node versions. But the risk of missing a new feature long-term by "auto-magic" seems low.. it is not like the Emscripten project could be oblivious to the latest feature developments in its domain (like Wasm Exnref exceptions), since the Emscripten devs are also contributing to the development of those feature specifications.

Maybe we can add a disable feature to these auto-detection flags. For example:

• EMTEST_LACKS_OFFSCREEN_CANVAS=1 : force-skip all OffscreenCanvas tests
• EMTEST_LACKS_OFFSCREEN_CANVAS not present: auto-detect the browser
• EMTEST_LACKS_OFFSCREEN_CANVAS=0: force-don't-skip OffscreenCanvas tests

That way Emscripten CI can opt-out from the auto-disables and be able to run all the tests?

[juj]

I updated the PR to enable setting EMTEST_LACKS_*=0 to enable force-running tests. This way we could have CircleCI config auto-run all tests, so there wouldn't exist any accidents with tests getting skipped?

[sbc100]

How about is the versions checking this was all behind EMTEST_AUTOSKIP=1 which you could enable in your CI?

Specifically I think I would want to turn off EMTEST_AUTOSKIP in the main CI and instead be explict.

Funnily enough we just had an example of where skipping tests caused us to loose coverage. In this case were explictly skipping using EMTEST_SKIP_WASM64 and EMTEST_SKIP_NODE_CANARY, but all of the wasm64 tests in test_other.py were not being run until #25531 landed. I suppose this is unrelated really because were explicitly skipping but its the kind of problem I'd like to minimize if possible.