compiletest output on message mismatch is hard to interpret

[RalfJung]

I find the new compiletest output on message mismatches quite hard to interpret. As an example:

---- [ui] tests/ui/pattern/usefulness/unions.rs stdout ----

error: 2 unexpected diagnostics reported, 2 expected diagnostics not reported
--- reported in JSON output but not expected in test file ---
tests/ui/pattern/usefulness/unions.rs:22:15: ERROR: non-exhaustive patterns: `U8AsBool { n: 0 }` and `U8AsBool { b: false }` not covered [E0004]
  expected with message non-exhaustive patterns: `U8AsBool { n: 0_u8 }` and `U8AsBool { b: false }` not covered
tests/ui/pattern/usefulness/unions.rs:28:15: ERROR: non-exhaustive patterns: `(U8AsBool { n: 0 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0 }, false)` and 1 more not covered [E0004]
  expected with message non-exhaustive patterns: `(U8AsBool { n: 0_u8 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0_u8 }, false)` and 1 more not covered
---
--- expected in test file but not reported in JSON output ---
tests/ui/pattern/usefulness/unions.rs:22:13: ERROR: non-exhaustive patterns: `U8AsBool { n: 0_u8 }` and `U8AsBool { b: false }` not covered
  reported with message non-exhaustive patterns: `U8AsBool { n: 0 }` and `U8AsBool { b: false }` not covered [E0004]
  reported with kind NOTE with message patterns `U8AsBool { n: 0 }` and `U8AsBool { b: false }` not covered
  reported with kind NOTE with message the matched value is of type `U8AsBool`
tests/ui/pattern/usefulness/unions.rs:28:13: ERROR: non-exhaustive patterns: `(U8AsBool { n: 0_u8 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0_u8 }, false)` and 1 more not covered
  reported with message non-exhaustive patterns: `(U8AsBool { n: 0 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0 }, false)` and 1 more not covered [E0004]
  reported with kind NOTE with message patterns `(U8AsBool { n: 0 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0 }, false)` and 1 more not covered
  reported with kind NOTE with message the matched value is of type `(U8AsBool, bool)`
---

Screenshot (with colors):

Several things are confusing here:

• This error has two parts, "reported in JSON output but not expected in test file" and "expected in test file but not reported in JSON output". However, the headers separating those parts are hard to spot since they are indented with these ---. That makes the tests/ui/pattern/usefulness/unions.rs:22:15 look like more of a header than the actual header. The colors actually make this worse since the black filename stands out much more than the green title.
• For the "reported in JSON output but not expected in test file", there are then lines saying "expected with message". So the error wasn't expected but also it was expected? I think what this means to say is that another message was expected on the same line but I had to read the .rs file to understand that.
• Similarly for the "expected in test file but not reported in JSON output", it then says "reported with kind" there. So something didn't get reported but also did get reported.
• Showing all the messages reported on that line is too much and drowns the relevant output.

As an example of output I find much easier to interpret, here's an output mismatch printed by ui_test:

error: `constructing valid value` not found in diagnostics on line 3
 --> tests/fail/intrinsics/zero_fn_ptr.rs:3:62
  |
3 |     let _ = unsafe { std::mem::zeroed::<fn()>() }; //~ERROR: constructing valid value
  |                                                              ^^^^^^^^^^^^^^^^^^^^^^^^ expected because of this pattern
  |

error: there were 1 unmatched diagnostics
 --> tests/fail/intrinsics/zero_fn_ptr.rs:3:22
  |
3 |     let _ = unsafe { std::mem::zeroed::<fn()>() }; //~ERROR: constructing valid value
  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^ Error: Undefined Behavior: constructing invalid value: encountered a null function pointer
  |

Maybe showing the full span is overdoing it, but the structure is at least much more clear.

Here's a possible incremental improvement of the current output that's already a lot easier to read for me:

error: 2 unexpected diagnostics reported
tests/ui/pattern/usefulness/unions.rs:22:15: ERROR: non-exhaustive patterns: `U8AsBool { n: 0 }` and `U8AsBool { b: false }` not covered [E0004]
  a different message was expected here: non-exhaustive patterns: `U8AsBool { n: 0_u8 }` and `U8AsBool { b: false }` not covered
tests/ui/pattern/usefulness/unions.rs:28:15: ERROR: non-exhaustive patterns: `(U8AsBool { n: 0 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0 }, false)` and 1 more not covered [E0004]
  a different message was expected here: non-exhaustive patterns: `(U8AsBool { n: 0_u8 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0_u8 }, false)` and 1 more not covered

error: 2 expected diagnostics not reported
tests/ui/pattern/usefulness/unions.rs:22:13: ERROR: non-exhaustive patterns: `U8AsBool { n: 0_u8 }` and `U8AsBool { b: false }` not covered
  a different message was reported here: non-exhaustive patterns: `U8AsBool { n: 0 }` and `U8AsBool { b: false }` not covered [E0004]
tests/ui/pattern/usefulness/unions.rs:28:13: ERROR: non-exhaustive patterns: `(U8AsBool { n: 0_u8 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0_u8 }, false)` and 1 more not covered
  a different message was reported here: non-exhaustive patterns: `(U8AsBool { n: 0 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0 }, false)` and 1 more not covered [E0004]

This got introduced in #140622.
Cc @petrochenkov @jieyouxu

[petrochenkov]

This error has two parts, "reported in JSON output but not expected in test file" and "expected in test file but not reported in JSON output". However, the headers separating those parts are hard to spot since they are indented with these ---. That makes the tests/ui/pattern/usefulness/unions.rs:22:15 look like more of a header than the actual header. The colors actually make this worse since the black filename stands out much more than the green title.

I was optimizing for vertical space in case of many failing tests, but adding a couple of newlines probably won't hurt.

The "in JSON output but not ... in test file" wording was specifically added (not by me, earlier) because people got confused about what "expected" and "reported" mean in this context.

In this part the "possible incremental improvement" looks ok to me, if the wording for error: is extended with "in json output"/"in test file".

For the "reported in JSON output but not expected in test file", there are then lines saying "expected with message". So the error wasn't expected but also it was expected? I think what this means to say is that another message was expected on the same line but I had to read the .rs file to understand that.
Similarly for the "expected in test file but not reported in JSON output", it then says "reported with kind" there. So something didn't get reported but also did get reported.

Well, in this case I assumed it was clear that things are expected/reported with a different kind/message.
The word "different" can be added to the wording, I guess, but it makes the message longer and shifts to printed error to the right.

Showing all the messages reported on that line is too much and drowns the relevant output.

Not all messages are reported, some filtering is already done.
I tried to report more to avoid missing something useful, but we can probably sacrifice the suggestions in which both kind and message are different, yeah (if there better suggestions are available).

[petrochenkov]

I'll try to make these tweaks today/tomorrow.

[RalfJung]

I tried to report more to avoid missing something useful, but we can probably sacrifice the suggestions in which both kind and message are different, yeah (if there better suggestions are available).

Yeah, and I would also generally say we shouldn't suggest more than 1 thing. Most of the space should be used for what's wrong, not potentially-incorrect guesses. Even then a small difference in the message will already produce 4 lines of output to sift through.

Well, in this case I assumed it was clear that things are expected/reported with a different kind/message.
The word "different" can be added to the wording, I guess, but it makes the message longer and shifts to printed error to the right.

If the difference is small it can be hard to even see that it is showing a different message.