Skip to content

Spurious failure with stalled macOS builds (specifically the check i686-apple-darwin job) #44221

Closed
@kennytm

Description

@kennytm
Member

First seen on 2017 Aug 31st in:

Symptom:

The job just stopped in the middle with no output in 30 minutes without any reason. End of the log looks like:

[00:40:25] �[m�[m�[32m�[1m   Compiling�[m rustc_llvm v0.0.0 (file:///Users/travis/build/rust-lang/rust/src/librustc_llvm)
[00:40:31] �[m�[m�[32m�[1m   Compiling�[m flate2 v0.2.19
[00:40:34] �[m�[m�[32m�[1m   Compiling�[m rustc_errors v0.0.0 (file:///Users/travis/build/rust-lang/rust/src/librustc_errors)



No output has been received in the last 30m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

The build has been terminated

Status of Travis and MacStadium are both green on 2017 Aug 31st. Not sure if unreported upstream issue or something else.

Activity

ssbarnea

ssbarnea commented on Sep 1, 2017

@ssbarnea

Well, I was hit by the same issue using brew upgrade .... failed on rust 1.20 ... seems like a recurring problem with rust.

alexcrichton

alexcrichton commented on Sep 1, 2017

@alexcrichton
Member

I talked with Travis yesterday and got:

both your mac pros are healthy, once has two VMs and the other has three
both our networking and SAN stuff is hitting some limits, so it may be one of those, unfortunately.

which makes me think it's probably upstream issues and we don't really have a way to work around :(

kennytm

kennytm commented on Sep 5, 2017

@kennytm
MemberAuthor

Not sure if Travis CI is aware of this, but the problem is very likely in the connection to the log collector or RabbitMQ, not the VMs themselves (yeah not something we can fix).

I'm currently monitoring https://travis-ci.org/rust-lang/rust/jobs/272020017, and logs are being written out if you "Follow log" at the web interface, but opening it again the logs will be all gone.

screenshot_2017-09-05 20 30 20_gfnuzu-fs8


🤔 The failing jobs are always the check jobs, not the dist jobs. The difference between them are xcode8.2 vs xcode7.3. I wonder if this can be fixed by upgrading the check jobs to xcode8.3 or xcode9. (I don't think we want to downgrade to xcode7.3.)

added 3 commits that reference this issue on Sep 5, 2017

Auto merge of #44338 - kennytm:xcode9, r=<try>

Auto merge of #44338 - kennytm:xcode9, r=alexcrichton

kennytm

kennytm commented on Sep 6, 2017

@kennytm
MemberAuthor

Incident notice: https://www.traviscistatus.com/incidents/2f0443bbphld

We’re investigating an increased rate of internal restarts of macOS builds, resulting in longer boot times for both public and private repositories. This has resulted in an increased backlog for macOS builds at travis-ci.org

Sep 6, 09:00 UTC


Edit: Upstream acknowledged the logging issue:

In addition to longer boot times, users are experiencing an increase in errored builds due to log timeouts when running macOS builds. We are investigating networking issues and will update again as soon as we know more.

Sep 06, 2017 - 10:45 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @alexcrichton@ssbarnea@kennytm

        Issue actions

          Spurious failure with stalled macOS builds (specifically the `check i686-apple-darwin` job) · Issue #44221 · rust-lang/rust