Skip to content

[DEBUG] macos GHA workflow #7149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 49 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
4636bb7
[DEBUG] macos GHA workflow
pmeier Jan 30, 2023
90de366
try fix pytorch install
pmeier Jan 30, 2023
30f087f
add more requirments
pmeier Jan 30, 2023
094057f
try using name activation for Linux build as well
pmeier Jan 30, 2023
5434d8f
try install setuptools explicitly
pmeier Jan 30, 2023
7e3cac7
try explicitly installing clang
pmeier Jan 30, 2023
59544c9
fix multiline
pmeier Jan 30, 2023
8810582
try search for gcc on all platforms
pmeier Jan 30, 2023
9b83493
fix windows runner
pmeier Jan 30, 2023
971db92
also search for clang
pmeier Jan 30, 2023
186f4e7
reinstate mac
pmeier Jan 30, 2023
a1024ff
try default on linux as well
pmeier Jan 30, 2023
585e19e
set +-u
pmeier Jan 30, 2023
c25e375
never re-enable
pmeier Jan 30, 2023
82a6226
try remove compilers once again
pmeier Jan 31, 2023
a599497
fix matrix variable
pmeier Jan 31, 2023
2607d81
add CONDA_BUILD env var debug
pmeier Jan 31, 2023
c6537dc
unset CONDA_BUILD
pmeier Jan 31, 2023
8f42b39
debug certifi
pmeier Jan 31, 2023
a6c62e3
try matrix over jobs
pmeier Jan 31, 2023
dded7ef
preinstall dependencies
pmeier Jan 31, 2023
21374ef
Revert "try matrix over jobs"
pmeier Jan 31, 2023
c258895
remove Windows workflow
pmeier Feb 1, 2023
ff45d05
try unify linux and mac
pmeier Feb 1, 2023
7a60180
use proper linux runner
pmeier Feb 1, 2023
29c1414
consolidate CPU and GPU workflows
pmeier Feb 1, 2023
a7213d0
increase timeout on macos
pmeier Feb 1, 2023
21bb958
fix indent
pmeier Feb 1, 2023
d08fa2f
fix timeout for linux as well
pmeier Feb 1, 2023
fe4fd86
add explicit exit from script
pmeier Feb 2, 2023
f670df7
increase timeout for mac workflows
pmeier Feb 2, 2023
b46ecbb
align timeout for all workflows
pmeier Feb 2, 2023
349288f
expand macos runners
pmeier Feb 2, 2023
637a6e0
Merge branch 'main' into am-i-going-crazy
pmeier Feb 2, 2023
fe27b6e
use strict mode
pmeier Feb 2, 2023
044041c
Merge branch 'main' into am-i-going-crazy
pmeier Feb 2, 2023
95d95ac
try using conda env from path
pmeier Feb 2, 2023
7fd032b
Revert "try using conda env from path"
pmeier Feb 2, 2023
0b52f38
try conda init
pmeier Feb 2, 2023
1010f82
source correct bash config
pmeier Feb 2, 2023
eb26cb4
try allowing unset variables during activation
pmeier Feb 2, 2023
46111b4
more debug
pmeier Feb 2, 2023
71c8add
add sleep for SSH access
pmeier Feb 2, 2023
2fcceea
try invoking hook manually
pmeier Feb 2, 2023
bd94cc8
remove set -e for now
pmeier Feb 2, 2023
e39510b
cleanup
pmeier Feb 2, 2023
90199a8
remove obsolete m1 workflow
pmeier Feb 2, 2023
afe9abf
try renable windows workflows
pmeier Feb 3, 2023
9c84c48
make bash mode more strict
pmeier Feb 3, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions .github/script_unittest.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
#!/usr/bin/env bash

set -euxo pipefail

echo '::group::Prepare conda'
CONDA_PATH=$(which conda)
eval "$(${CONDA_PATH} shell.bash hook)"
# The `setuptools` package installed through `conda` includes a patch that errors if something is installed
# through `setuptools` while the `CONDA_BUILD` environment variable is set.
# https://github.com/AnacondaRecipes/setuptools-feedstock/blob/f5d8d256810ce28fc0cf34170bc34e06d3754041/recipe/patches/0002-disable-downloads-inside-conda-build.patch
# (Although we are not using the `-c conda-forge` channel, the patch is equivalent but not public for
# `setuptools` from the `-c defaults` channel)
# Since we aren't using `conda build` here, we unset it to avoid installation problems later
# TODO: investigate where `CONDA_BUILD` is set and maybe fix unset it there
unset CONDA_BUILD
echo '::endgroup::'

echo '::group::Set PyTorch conda channel'
# TODO: Can we maybe have this as environment variable in the job template? For example, `IS_RELEASE`.
if [[ (${GITHUB_EVENT_NAME} = 'pull_request' && (${GITHUB_BASE_REF} = 'release'*)) || (${GITHUB_REF} = 'refs/heads/release'*) ]]; then
POSTFIX=test
else
POSTFIX=nightly
fi
PYTORCH_CHANNEL=pytorch-"${POSTFIX}"
echo "${PYTORCH_CHANNEL}"
echo '::endgroup::'

echo '::group::Set PyTorch GPU mutex'
case $GPU_ARCH_TYPE in
cpu)
PYTORCH_MUTEX=cpuonly
;;
cuda)
PYTORCH_MUTEX="pytorch-cuda=${GPU_ARCH_VERSION}"
;;
*)
echo "Unknown GPU_ARCH_TYPE=${GPU_ARCH_TYPE}"
exit 1
;;
esac
echo "${PYTORCH_MUTEX}"
echo '::endgroup::'

echo '::group::Create build environment'
conda create \
--name ci \
--quiet --yes \
python="${PYTHON_VERSION}" pip \
setuptools ninja \
libpng jpeg \
Pillow numpy requests
conda activate ci
pip install 'av<10'
echo '::endgroup::'

echo '::group::Install PyTorch'
conda install \
--quiet --yes \
-c "${PYTORCH_CHANNEL}" \
-c nvidia \
pytorch \
"${PYTORCH_MUTEX}"

if [[ $GPU_ARCH_TYPE = 'cuda' ]]; then
python3 -c "import torch; exit(not torch.cuda.is_available())"
fi
echo '::endgroup::'

echo '::group::Install TorchVision'
python setup.py develop
echo '::endgroup::'

echo '::group::Collect PyTorch environment information'
python -m torch.utils.collect_env
echo '::endgroup::'

echo '::group::Install testing utilities'
pip install --progress-bar=off pytest pytest-mock pytest-cov
echo '::endgroup::'

echo '::group::Run tests'
pytest --durations=25
echo '::endgroup::'
57 changes: 0 additions & 57 deletions .github/workflows/test-linux-cpu.yml

This file was deleted.

61 changes: 0 additions & 61 deletions .github/workflows/test-linux-gpu.yml

This file was deleted.

38 changes: 38 additions & 0 deletions .github/workflows/test-linux.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Unit-tests on Linux

on:
pull_request:
push:
branches:
- nightly
- main
- release/*
workflow_dispatch:

jobs:
tests:
strategy:
matrix:
python-version:
- "3.8"
- "3.9"
- "3.10"
runner: ["linux.12xlarge"]
gpu-arch-type: ["cpu"]
include:
- python-version: 3.8
runner: linux.g5.4xlarge.nvidia.gpu
gpu-arch-type: cuda
gpu-arch-version: "11.7"
fail-fast: false
uses: pytorch/test-infra/.github/workflows/linux_job.yml@main
with:
repository: pytorch/vision
runner: ${{ matrix.runner }}
timeout: 120
script: |
export PYTHON_VERSION=${{ matrix.python-version }}
export GPU_ARCH_TYPE=${{ matrix.gpu-arch-type }}
export GPU_ARCH_VERSION=${{ matrix.gpu-arch-version }}

./.github/script_unittest.sh
50 changes: 0 additions & 50 deletions .github/workflows/test-m1.yml

This file was deleted.

34 changes: 34 additions & 0 deletions .github/workflows/test-macos.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Unit-tests on macOS

on:
pull_request:
push:
branches:
- nightly
- main
- release/*
workflow_dispatch:

jobs:
tests:
strategy:
matrix:
python-version:
- "3.8"
- "3.9"
- "3.10"
runner:
- macos-12
- macos-m1-12
gpu-arch-type: ["cpu"]
fail-fast: false
uses: pytorch/test-infra/.github/workflows/macos_job.yml@main
with:
repository: pytorch/vision
timeout: 120
runner: ${{ matrix.runner }}
script: |
export PYTHON_VERSION=${{ matrix.python-version }}
export GPU_ARCH_TYPE=${{ matrix.gpu-arch-type }}

./.github/script_unittest.sh
38 changes: 38 additions & 0 deletions .github/workflows/test-windows.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Unit-tests on Windows

on:
pull_request:
push:
branches:
- nightly
- main
- release/*
workflow_dispatch:

jobs:
tests:
strategy:
matrix:
python-version:
- "3.8"
- "3.9"
- "3.10"
runner: ["windows.4xlarge"]
gpu-arch-type: ["cpu"]
include:
- python-version: 3.8
runner: windows.8xlarge.nvidia.gpu
gpu-arch-type: cuda
gpu-arch-version: "11.7"
fail-fast: false
uses: pytorch/test-infra/.github/workflows/windows_job.yml@main
with:
repository: pytorch/vision
runner: ${{ matrix.runner }}
timeout: 120
script: |
export PYTHON_VERSION=${{ matrix.python-version }}
export GPU_ARCH_TYPE=${{ matrix.gpu-arch-type }}
export GPU_ARCH_VERSION=${{ matrix.gpu-arch-version }}

./.github/script_unittest.sh