Skip to content

Commit d3fea80

Browse files
Start a small playbook on how to deal with failing webtests. (#2962)
This is a starting point on how to deal with failures in the web-test checks. They are supposed to warn for legitimate issues but if they fail, they can be overridden. The web-tests will be separated into its own job to better differentiate if issues in building the English translation or web-testing the English translation are happening
1 parent d1899de commit d3fea80

File tree

1 file changed

+26
-1
lines changed

1 file changed

+26
-1
lines changed

tests/README.md

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,32 @@ when `npm start` or `npm test` is executed.
1717
> **Tip:** Use `cargo xtask web-tests` to run the tests in this directory from
1818
> anywhere in the repository.
1919
20-
For local testing and quick iterations it is possible to use `mdbook serve`
20+
For local testing and quick iterations it is possible to use `cargo xtask serve`
2121
which creates a small HTTP server on port 3000 by default. There is a special
2222
config that is invoked with `npm run test-mdbook` that uses
2323
`http://localhost:3000`
24+
25+
## Deal with failing tests
26+
27+
When you see tests failing they should already indicate what checks broke for
28+
specific pages.
29+
30+
### Legitimate warnings
31+
32+
You might e.g. need to reduce the length of an overlong page (or get an
33+
exemption) or updating some mdbook infrastructure had a breaking change that
34+
breaks functionality. These issues need to be fixed before this change is
35+
merged.
36+
37+
### Broken test environment
38+
39+
Sometimes tests can also fail in the CI environment with errors like this.
40+
41+
```
42+
ERROR webdriver: WebDriverError: tab crashed
43+
```
44+
45+
If you see messages like these, it may indicate an issue with the web-tests that
46+
is not caused by your changes. Please file a bug to report this. As a temporary
47+
workaround, if all other checks pass and you are confident your changes are
48+
correct, you may override the web-test requirement to merge your pull request.

0 commit comments

Comments
 (0)