[llvm-lit] Show test output when combining -v or -a with -q

[hnrklssn]

Lit has various flags to control output, that interact in non-obvious ways when combined: --quiet --succinct --verbose --show-all. Their order doesn't matter, they are completely separate flags.

At the moment, --quiet completely overrides --verbose and --show-all, without any sort of warning. I think it would be more useful if they could be combined such that --quiet still silences lit's note and warning diagnostics, but --verbose and --show-all still can show the output of test cases.

[jh7370]

I'm thinking that we might want a more refined "last one wins" style approach, because I think it makes as little sense for -a/-v to produce test output when -q is enabled as the current behaviour. However, regardless, I think this probably needs an RFC for better visibility: the change has the potential to break people's existing scripts etc.

[hnrklssn]

To be clear, I specifically want to be able to combine the test output from -a/-v with the diagnostic silencing of -q, not for -a/-v to simply override -q. But fair enough on the RFC part.

[RoboTux]

I would agree with James, lit's doc has the following on --quiet:

Suppress any output except for test failures.

Therefore I would expect --quiet, --succinct, --verbose and --show-all to be mutually exclusive. Last one wins would be compatible with that. I think a separate --mask-warnings (YMMV for naming) which is implied to by --quiet would be better IMO. Where is the RFC by the way? I couldn't find it when searching discourse.

[hnrklssn]

Where is the RFC by the way? I couldn't find it when searching discourse.

I have not had time to put one together. Taking the RFC route, we might as well think over the overall output flag system. I'm thinking that it would be less confusing if they were truly mutually exclusive, which would be ensured if we introduced a --output-format= flag with a number of different options. Then --quiet, --succinct, --verbose and --show-all could be remapped to aliases for --output-format=quiet etc, and only 1 could take effect at a time.

That said I've got a bit on my plate now (I made this patch during a hackathon), so I don't know when I'll have the bandwidth to both write up a well thought through RFC and refactor this. If you're keen to see this a bit sooner I'm happy to pass the baton

[RoboTux]

I agree with --output-format and remapping existing options to it. And sorry I don't have the bandwidth either right now.