Running Python tests with Pytest fails with: Error expanding output globs: Failed to scan directory /private/...

[ruslan-roboto]

Describe the bug
I run all integration tests for my application using pants --integ test --force ::. This used to succeed, but now it fails with an IntrinsicError related to "expanding output globs".

Here's a (sanitized) example of the failure. The test file name is different every time:

11:52:53.85 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

IntrinsicError: Failed to execute: Process {
    argv: [
        "./pytest_runner.pex_pex_shim.sh",
        "--color=yes",
        "--junit-xml=packages.myapp.test.myapp_integ_tests.integ.test_interesting_condition.py.integ_tests.xml",
        "-o",
        "junit_family=xunit2",
        "--cov-report=",
        "--cov-config=build-support/.coveragerc",
        "--cov=packages/myapp/src",
        "--cov=packages/myapp_integ/src",
        "--cov=packages/myapp_integ/test",
        "packages/myapp_integ/test/myapp_integ_tests/integ/test_interesting_condition.py",
    ],
    env: {
        "AWS_PROFILE": "<redacted>",
        "PEX_EXTRA_SYS_PATH": "packages/myapp/src:packages/myapp_integ/src:packages/myapp_integ/test",
    },
    working_directory: None,
    input_digests: InputDigests {
        complete: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<deeec633893aa0d8bf3190c13044ad29bf7f83b8b1ee6d1ce574a29340ff5045>,
                size_bytes: 831,
            },
            tree: "Some(..)",
        },
        nailgun: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>,
                size_bytes: 0,
            },
            tree: "Some(..)",
        },
        inputs: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<deeec633893aa0d8bf3190c13044ad29bf7f83b8b1ee6d1ce574a29340ff5045>,
                size_bytes: 831,
            },
            tree: "Some(..)",
        },
        immutable_inputs: {},
        use_nailgun: {},
    },
    output_files: {
        RelativePath(
            ".coverage",
        ),
        RelativePath(
            "packages.myapp_integ.test.myapp_integ_tests.integ.test_interesting_condition.py.integ_tests.xml",
        ),
    },
    output_directories: {
        RelativePath(
            "extra-output",
        ),
    },
    timeout: None,
    execution_slot_variable: None,
    concurrency_available: 0,
    description: "Run Pytest for packages/myapp_integ/test/myapp_integ_tests/integ/test_interesting_condition.py:integ_tests",
    level: Debug,
    append_only_caches: {
        CacheName(
            "pex_root",
        ): RelativePath(
            ".cache/pex_root",
        ),
        CacheName(
            "python_build_standalone",
        ): RelativePath(
            ".python-build-standalone",
        ),
    },
    jdk_home: None,
    cache_scope: PerSession,
    execution_environment: ProcessExecutionEnvironment {
        name: None,
        platform: Macos_arm64,
        strategy: Local,
    },
    remote_cache_speculation_delay: 0ns,
    attempt: 0,
}

Error expanding output globs: Failed to scan directory "/private/var/folders/7c/3ny04jh95k9fmjxtksxqyh2m0000gn/T/pants-sandbox-tK9sl1/": No such file or directory (os error 2)

Pants version
2.23.0

OS
MacOS Sonoma 14.7.4

Additional info

• I tried deleting .pants.d/ and re-running - no effect
• Checked whether I was running out of disk space - got plenty left.
• Restarted my Mac since I hadn't done so in months - no effect

[huonw]

Sorry for the issue and thanks for filing a bug.

This used to succeed

Do you remember anything that changed between when it worked and now? For instance, upgrading Pants versions?

[ruslan-roboto]

@huonw we updated from Pants 2.21 to Pants 2.23 about a month ago. I'll see if the issue persists on Pants 2.24. I can't think of anything else about our integration test setup that has materially changed. Thanks for taking a look!

[huonw]

Please let us know the results of your investigation! If it still reproduces, the next steps will be:

• best case: make a reduced reproducer than you can share
• if not, at least share as much of your pants.toml and the relevant BUILD files as you can