Skip to content

Commit 348460a

Browse files
committed
Auto merge of #143002 - Enselic:tests-ui-run-fail-exit-vs-signal, r=petrochenkov
tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) And introduce two new directives for ui tests: * `run-crash` * `run-fail-or-crash` Normally a `run-fail` ui test like tests that panic shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal/crash however. Introduce and use `run-crash` for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process exit code. Example exit code for crash on Windows: `0xc000001d` (`STATUS_ILLEGAL_INSTRUCTION`). Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127". Some tests behave differently on different targets: * Targets without unwind support will abort (crash) instead of exit with failure code 101 after panicking. As a special case, allow crashes for `run-fail` tests for such targets. * Different sanitizer implementations handle detected memory problems differently. Some abort (crash) the process while others exit with failure code 1. Introduce and use `run-fail-or-crash` for such tests. This adds further (cc rust-lang/rust#142304, rust-lang/rust#142886) protection against the regression in rust-lang/rust#123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See rust-lang/rust#143002 (comment). - [x] Handle targets without unwind support - [x] Add `run-fail-or-crash` for some sanitizer tests - [x] remote-test-client. See rust-lang/rust#143448 ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: aarch64-apple try-job: x86_64-msvc-1 try-job: x86_64-gnu try-job: dist-i586-gnu-i586-i686-musl try-job: test-various try-job: armhf-gnu
2 parents 78450eb + 73a513c commit 348460a

File tree

2 files changed

+19
-7
lines changed

2 files changed

+19
-7
lines changed

src/tests/directives.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,10 @@ expectations](ui.md#controlling-passfail-expectations).
7575
| `check-fail` | Building (no codegen) should fail | `ui`, `crashes` | N/A |
7676
| `build-pass` | Building should pass | `ui`, `crashes`, `codegen`, `incremental` | N/A |
7777
| `build-fail` | Building should fail | `ui`, `crashes` | N/A |
78-
| `run-pass` | Running the test binary should pass | `ui`, `crashes`, `incremental` | N/A |
79-
| `run-fail` | Running the test binary should fail | `ui`, `crashes` | N/A |
78+
| `run-pass` | Program must exit with code `0` | `ui`, `crashes`, `incremental` | N/A |
79+
| `run-fail` | Program must exit with code `1..=127` | `ui`, `crashes` | N/A |
80+
| `run-crash` | Program must crash | `ui` | N/A |
81+
| `run-fail-or-crash` | Program must `run-fail` or `run-crash` | `ui` | N/A |
8082
| `ignore-pass` | Ignore `--pass` flag | `ui`, `crashes`, `codegen`, `incremental` | N/A |
8183
| `dont-check-failure-status` | Don't check exact failure status (i.e. `1`) | `ui`, `incremental` | N/A |
8284
| `failure-status` | Check | `ui`, `crashes` | Any `u16` |

src/tests/ui.md

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -448,7 +448,7 @@ even run the resulting program. Just add one of the following
448448
- `//@ build-pass` — compilation and linking should succeed but do
449449
not run the resulting binary.
450450
- `//@ run-pass` — compilation should succeed and running the resulting
451-
binary should also succeed.
451+
binary should make it exit with code 0 which indicates success.
452452
- Fail directives:
453453
- `//@ check-fail` — compilation should fail (the codegen phase is skipped).
454454
This is the default for UI tests.
@@ -457,10 +457,20 @@ even run the resulting program. Just add one of the following
457457
- First time is to ensure that the compile succeeds without the codegen phase
458458
- Second time is to ensure that the full compile fails
459459
- `//@ run-fail` — compilation should succeed, but running the resulting
460-
binary should fail.
461-
462-
For `run-pass` and `run-fail` tests, by default the output of the program itself
463-
is not checked.
460+
binary should make it exit with a code in the range `1..=127` which
461+
indicates regular failure. On targets without unwind support, crashes
462+
are also accepted.
463+
- `//@ run-crash` — compilation should succeed, but running the resulting
464+
binary should fail with a crash. Crashing is defined as "not exiting with
465+
a code in the range `0..=127`". Example on Linux: Termination by `SIGABRT`
466+
or `SIGSEGV`. Example on Windows: Exiting with the code for
467+
`STATUS_ILLEGAL_INSTRUCTION` (`0xC000001D`).
468+
- `//@ run-fail-or-crash` — compilation should succeed, but running the
469+
resulting binary should either `run-fail` or `run-crash`. Useful if a test
470+
crashes on some targets but just fails on others.
471+
472+
For `run-pass`. `run-fail`, `run-crash` and `run-fail-or-crash` tests, by
473+
default the output of the program itself is not checked.
464474

465475
If you want to check the output of running the program, include the
466476
`check-run-results` directive. This will check for a `.run.stderr` and

0 commit comments

Comments
 (0)