bootstrap: retry `cargo` invocations if stderr contains a known pattern

[marcoieni]

Why

Our CI auto builds sometimes fail for known reasons that are not related to the PRs we are trying to merge.

Most of the times, these errors are hard to understand and fix (or can't be fixed at all), decreasing the success rate of the auto builds for several weeks or months.
The impact is that we loose days of parallel compute time and hours of maintainers time that need to analyze the error message and reschedule the PRs in the merge queue.

Feature

We want to list the the stderr of the known issues we are aware of in the config.toml file that bootstrap uses. These patterns can be expressed as regex.
We want bootstrap to retry cargo invocations up to two times if stderr matches one of the listed patterns.

This would help reduce the failure rate of our CI because it would significantly reduce the percentage of jobs failing due to spurious errors.

The error messages need to be precise enough to avoid retrying cargo invocations over genuine problems.

Known error patterns can be found here. Not all of them can be listed.

As a start, we could just have 1 stderr string in the list (this one doesn't need to be a regex):

• ranlib.exe: could not create temporary file whilst writing archive: no more archived files which is discussed in CI: spurious error building LLVM on mingw: ranlib.exe: could not create temporary file whilst writing archive: no more archived files #108227

Questions

• Is config.toml the right place to put the known stderr patterns? In Zulip, Jieyou proposed introducing another file: retry-patterns.toml. I'll leave it to the bootstrap team to decide.
• Which format do we use to write the stderr patterns in the config.toml file? For example, it can be an array of strings. It could also be an "object" if we want to customize how many times to retry per error message. I'll leave it to the boostrap team to decide.
• how do we make sure these patterns are present in the config.toml used for CI? I'm not familiar with how the config.toml for the CI is generated.

Zulip links

• idea proposed here
• agreement reached here

[jieyouxu]

Is config.toml the right place to put the known stderr patterns? In Zulip, Jieyou proposed introducing another file: retry-patterns.toml. I'll leave it to the bootstrap team to decide.

FTR config.toml is usually for local user config (CI sometimes writes their own config.toml) and that is specifically .gitignore'd, but I haven't thought too hard about the pros and cons of including it in the same config file.

[jieyouxu]

Notably, if we decide to pursue this approach, we need to be precise enough in the error patterns to not retry genuine failures.

[marcoieni]

Ok. Let us know when the bootstrap team is able to answer the questions listed, so that somebody can proceed with the implementation.
I could have some capacity to work on this in the week of 6 Jan for example 👍

[jieyouxu]

cc @rust-lang/bootstrap: what do we think about this?

[Kobzol]

This sounds reasonable, it would definitely be useful.

Note that implementing this directly in bootstrap might be tricky. We sometimes don't capture stderr of cargo (in fact I think that in most situations we don't capture it?), but rather redirect it to stderr of the bootstrap. Furthermore, we would probably need to do this on a per-Cargo invocation, and we invoke Cargo on many places. So to avoid doing these restarts on a lot of source locations inside bootstrap, which would be quite crazy, it would be better to somehow thread it through this, which would ideally do the restart automatically inside, without the rest of bootstrap having to think about it.

I wouldn't be opposed against doing this outside of bootstrap, tbh, by just wrapping the top-level run build CI step in a loop that would do rm -rf build && run.sh and then restart the loop N times if the generated stderr contains the invalid substrings. More specifically, I'm not sure if I agree with this comment, given the state of bootstrap's codebase 😆 But this has to be tried out.

Anyway, if we do it inside bootstrap, we can either put it into config.toml or add a CLI flag, that's probably not super important. We can modify config.toml on CI using the --set argument, we already do that on several places.

Btw, it looks like the linked issue is not a spurious failure, but rather a known issue (with a known fix). A better candidate might be the MSVC locking failures.

[onur-ozkan]

This could be useful only for CI, right? I think this should be done outside of the bootstrap using a wrapper script that handles this magic. It doesn't seem necessary to include it in the bootstrap as it's unrelated to its core purpose.

Also, how do we plan to handle error reports? If we silently skip these errors, we might forget to fix them which could increase build time for certain runners.

[clubby789]

What if we just have the existing failure/RLA messages with a note that the step is being retried?

[onur-ozkan]

What if we just have the existing failure/RLA messages with a note that the step is being retried?

Yeah, that should be enough.

[marcoieni]

If we do this outside of bootstrap can we retry at the same level of granularity?

E.g. CI, could retry ./x.py --stage 2 test, but that would take a lot of time.

Instead, bootstrap knows the cargo command that failed, so it can retry only that command, which is faster, right?