test_runner: add 'test:summary' event

[cjihrig]

This commit adds a new 'test:summary' event to the test runner's reporting interface. This new event serves two purposes:

• In the future, the test runner internals will no longer need to change the process exit code. This may be important to run() users. Unfortunately, this is a breaking change, so it needs to be changed in a major version.
• The reporting interface now has a single event that can identify passing or failing test runs.

Refs: #53867
Refs: #54812

(Still need to add docs)

[nodejs-github-bot]

Review requested:

• @nodejs/test_runner

[codecov]

Codecov Report

Attention: Patch coverage is 88.23529% with 4 lines in your changes missing coverage. Please review.

Project coverage is 88.02%. Comparing base (e35299a) to head (81e000d).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/test_runner/harness.js	72.72%	3 Missing ⚠️
lib/internal/test_runner/test.js	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #54851      +/-   ##
==========================================
- Coverage   88.03%   88.02%   -0.01%     
==========================================
  Files         652      652              
  Lines      183797   183822      +25     
  Branches    35863    35868       +5     
==========================================
+ Hits       161804   161813       +9     
- Misses      15242    15259      +17     
+ Partials     6751     6750       -1     

Files with missing lines	Coverage Δ
lib/internal/main/test_runner.js	100.00% <100.00%> (ø)
lib/internal/test_runner/tests_stream.js	91.61% <100.00%> (+0.53%)	⬆️
lib/internal/test_runner/utils.js	55.88% <100.00%> (+0.14%)	⬆️
lib/internal/test_runner/test.js	96.93% <87.50%> (-0.07%)	⬇️
lib/internal/test_runner/harness.js	92.65% <72.72%> (-0.72%)	⬇️

... and 22 files with indirect coverage changes

[RedYetiDev]

(Still need to add docs)

I've added the wip label, as you stated that you still need to add the documentation for this PR

[atlowChemi]

Unfortunately, this is a breaking change, so it needs to be changed in a major version.

Added semver-major PRs that contain breaking changes and should be released in the next major version.

[cjihrig]

Unfortunately, this is a breaking change, so it needs to be changed in a major version.

The original implementation that changed the process exit code was semver major. I removed that change, so this is only semver minor now

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/62438/

[mcollina]

Can we make it non-breaking by detecting if there is a summary listener and if not reverting to the old behavior? I think having this as a major would significantly impact the ability to backport certain fixes to v20 and v22.

[cjihrig]

Can we make it non-breaking by detecting if there is a summary listener and if not reverting to the old behavior?

This change is no longer breaking. Do you mean the setting of the exit code? Yes, we could do that - I think I'd prefer to do that in a separate PR though.

[mcollina]

For whatever reason I didn't see the last couple of messages, nevermind!

[mcollina]

lgtm

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/62605/

[cjihrig]

Rebased to pick up the latest changes and renamed all to tests.

[RedYetiDev]

Can you remove the logic for #53937's handling of the exit code for thresholds, as it's no longer needed with this, right?

Notably the documentation about exit codes here, here, and here.

[cjihrig]

No. run() will continue to set the process exit code even though it's not needed with this event. Changing run() to not change the exit code is a breaking change that we shouldn't make until all supported release lines have access to this event.

[cjihrig]

This needs a re-approval in order to land. The CI is good.

[RedYetiDev]

No. run() will continue to set the process exit code even though it's not needed with this event. Changing run() to not change the exit code is a breaking change that we shouldn't make until all supported release lines have access to this event.

Oh okay, I thought that that didn't happen with this change. My bad. I'll probably open a followup (breaking?) PR to change that after this.