Skip to content

nemo-ci-cpu #386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ko3n1g wants to merge 8 commits into from
Closed

nemo-ci-cpu #386

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d4cde6d
nemo-ci-cpu
ko3n1g Aug 13, 2025
7359361
fix name
ko3n1g Aug 14, 2025
b3b493c
test
ko3n1g Aug 14, 2025
89a2209
revert
ko3n1g Aug 14, 2025
4e1f980
fix
ko3n1g Aug 14, 2025
bcb536f
test
ko3n1g Aug 14, 2025
f5c53cd
fix
ko3n1g Aug 14, 2025
9e1f743
Update cicd-main.yml
ko3n1g Aug 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
249 changes: 113 additions & 136 deletions .github/actions/test-template/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,141 +50,118 @@ inputs:
runs:
using: "composite"
steps:
# - name: Install curl, sudo
# shell: bash
# run: |
# apt-get update
# apt-get install -y curl sudo uuid-runtime systemd

# - name: Install Docker with custom script
# shell: bash
# run: |
# # Download and modify the Docker install script
# curl -fsSL https://get.docker.com -o get-docker.sh

# # Install only CLI components (remove daemon installation)
# export DEBIAN_FRONTEND=noninteractive
# apt-get update -qq
# apt-get install -y ca-certificates curl gnupg lsb-release

# install -m 0755 -d /etc/apt/keyrings
# curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" -o /etc/apt/keyrings/docker.asc
# chmod a+r /etc/apt/keyrings/docker.asc

# echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu jammy stable" > /etc/apt/sources.list.d/docker.list

# apt-get update -qq
# # Install CLI only, skip daemon and user management
# apt-get install -y docker-ce-cli docker-buildx-plugin docker-compose-plugin

# docker --version

# - name: Install Azure CLI
# if: ${{ inputs.has-azure-credentials == 'true' }}
# shell: bash
# run: |
# echo "::group::Install Azure CLI"
# # Create systemd override for proper dependencies
# curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
# echo "::endgroup::"

# - name: Azure Login
# if: ${{ inputs.has-azure-credentials == 'true' }}
# uses: azure/login@v2
# with:
# client-id: ${{ inputs.azure-client-id }}
# tenant-id: ${{ inputs.azure-tenant-id }}
# subscription-id: ${{ inputs.azure-subscription-id }}

# - name: Azure ACR Login
# if: ${{ inputs.has-azure-credentials == 'true' }}
# shell: bash
# run: |
# az acr login --name nemoci

# - name: Configure local file cache
# shell: bash
# if: ${{ inputs.has-azure-credentials == 'true' && inputs.is_unit_test == 'false' }}
# env:
# MNT_PATH: ${{ steps.azure-fileshare.outputs.MNT_PATH }}
# run: |
# echo "::group::Configure local artifact cache"

# set -x -e -u -o pipefail

# sudo apt update
# sudo apt install -y cachefilesd

# # Create systemd override for proper dependencies
# sudo mkdir -p /etc/systemd/system/cachefilesd.service.d
# printf "[Unit]\nAfter=network.target local-fs.target\nWants=local-fs.target\n\n[Service]\nRestart=on-failure\nRestartSec=2\n" | sudo tee /etc/systemd/system/cachefilesd.service.d/override.conf > /dev/null
# sudo systemctl daemon-reload

# # Enable cachefilesd in config file
# sudo sed -i 's/^#RUN=yes/RUN=yes/' /etc/default/cachefilesd
# sudo cat /etc/default/cachefilesd

# # Enable and restart the service
# sudo systemctl enable cachefilesd
# sudo systemctl restart cachefilesd
# sudo systemctl status cachefilesd
# echo "::endgroup::"

# - name: Azure Fileshare
# if: ${{ inputs.has-azure-credentials == 'true' && inputs.is_unit_test == 'false' }}
# shell: bash
# id: azure-fileshare
# run: |
# echo "::group::Mount SMB drive"
# sudo apt update
# sudo apt install -y cifs-utils

# RESOURCE_GROUP_NAME="azure-gpu-vm-runner_group"
# STORAGE_ACCOUNT_NAME="nemocistorageaccount2"
# FILE_SHARE_NAME="fileshare"

# MNT_ROOT="/media"
# MNT_PATH="$MNT_ROOT/$STORAGE_ACCOUNT_NAME/$FILE_SHARE_NAME"

# echo "MNT_PATH=$MNT_PATH" | tee -a "$GITHUB_OUTPUT"

# sudo mkdir -p $MNT_PATH

# # Create a folder to store the credentials for this storage account and
# # any other that you might set up.
# CREDENTIAL_ROOT="/etc/smbcredentials"
# sudo mkdir -p "/etc/smbcredentials"

# # Get the storage account key for the indicated storage account.
# # You must be logged in with az login and your user identity must have
# # permissions to list the storage account keys for this command to work.
# STORAGE_ACCOUNT_KEY=$(az storage account keys list \
# --resource-group $RESOURCE_GROUP_NAME \
# --account-name $STORAGE_ACCOUNT_NAME \
# --query "[0].value" --output tsv | tr -d '"')

# # Create the credential file for this individual storage account
# SMB_CREDENTIAL_FILE="$CREDENTIAL_ROOT/$STORAGE_ACCOUNT_NAME.cred"
# if [ ! -f $SMB_CREDENTIAL_FILE ]; then
# echo "username=$STORAGE_ACCOUNT_NAME" | sudo tee $SMB_CREDENTIAL_FILE > /dev/null
# echo "password=$STORAGE_ACCOUNT_KEY" | sudo tee -a $SMB_CREDENTIAL_FILE > /dev/null
# else
# echo "The credential file $SMB_CREDENTIAL_FILE already exists, and was not modified."
# fi

# # Change permissions on the credential file so only root can read or modify the password file.
# sudo chmod 600 $SMB_CREDENTIAL_FILE

# # This command assumes you have logged in with az login
# HTTP_ENDPOINT=$(az storage account show --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --query "primaryEndpoints.file" --output tsv | tr -d '"')
# SMB_PATH=$(echo $HTTP_ENDPOINT | cut -c7-${#HTTP_ENDPOINT})$FILE_SHARE_NAME

# STORAGE_ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query "[0].value" --output tsv | tr -d '"')

# sudo mount -t cifs $SMB_PATH $MNT_PATH -o credentials=$SMB_CREDENTIAL_FILE,serverino,nosharesock,actimeo=30,mfsymlinks,fsc,cache=strict

# ls -al $MNT_PATH/TestData
# echo "::endgroup::"
- name: Install curl, sudo
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y curl uuid-runtime

- name: Install Azure CLI
if: ${{ inputs.has-azure-credentials == 'true' }}
shell: bash
run: |
echo "::group::Install Azure CLI"
# Create systemd override for proper dependencies
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
echo "::endgroup::"

- name: Azure Login
if: ${{ inputs.has-azure-credentials == 'true' }}
uses: azure/login@v2
with:
client-id: ${{ inputs.azure-client-id }}
tenant-id: ${{ inputs.azure-tenant-id }}
subscription-id: ${{ inputs.azure-subscription-id }}

- name: Azure ACR Login
if: ${{ inputs.has-azure-credentials == 'true' }}
shell: bash
run: |
az acr login --name nemoci

- name: Configure local file cache
shell: bash
if: ${{ inputs.has-azure-credentials == 'true' && inputs.is_unit_test == 'false' }}
env:
MNT_PATH: ${{ steps.azure-fileshare.outputs.MNT_PATH }}
run: |
echo "::group::Configure local artifact cache"

set -x -e -u -o pipefail

sudo apt update
sudo apt install -y cachefilesd

# Create systemd override for proper dependencies
sudo mkdir -p /etc/systemd/system/cachefilesd.service.d
printf "[Unit]\nAfter=network.target local-fs.target\nWants=local-fs.target\n\n[Service]\nRestart=on-failure\nRestartSec=2\n" | sudo tee /etc/systemd/system/cachefilesd.service.d/override.conf > /dev/null
sudo systemctl daemon-reload

# Enable cachefilesd in config file
sudo sed -i 's/^#RUN=yes/RUN=yes/' /etc/default/cachefilesd
sudo cat /etc/default/cachefilesd

# Enable and restart the service
sudo systemctl enable cachefilesd
sudo systemctl restart cachefilesd
sudo systemctl status cachefilesd
echo "::endgroup::"

- name: Azure Fileshare
if: ${{ inputs.has-azure-credentials == 'true' && inputs.is_unit_test == 'false' }}
shell: bash
id: azure-fileshare
run: |
echo "::group::Mount SMB drive"
sudo apt update
sudo apt install -y cifs-utils

RESOURCE_GROUP_NAME="azure-gpu-vm-runner_group"
STORAGE_ACCOUNT_NAME="nemocistorageaccount2"
FILE_SHARE_NAME="fileshare"

MNT_ROOT="/media"
MNT_PATH="$MNT_ROOT/$STORAGE_ACCOUNT_NAME/$FILE_SHARE_NAME"

echo "MNT_PATH=$MNT_PATH" | tee -a "$GITHUB_OUTPUT"

sudo mkdir -p $MNT_PATH

# Create a folder to store the credentials for this storage account and
# any other that you might set up.
CREDENTIAL_ROOT="/etc/smbcredentials"
sudo mkdir -p "/etc/smbcredentials"

# Get the storage account key for the indicated storage account.
# You must be logged in with az login and your user identity must have
# permissions to list the storage account keys for this command to work.
STORAGE_ACCOUNT_KEY=$(az storage account keys list \
--resource-group $RESOURCE_GROUP_NAME \
--account-name $STORAGE_ACCOUNT_NAME \
--query "[0].value" --output tsv | tr -d '"')

# Create the credential file for this individual storage account
SMB_CREDENTIAL_FILE="$CREDENTIAL_ROOT/$STORAGE_ACCOUNT_NAME.cred"
if [ ! -f $SMB_CREDENTIAL_FILE ]; then
echo "username=$STORAGE_ACCOUNT_NAME" | sudo tee $SMB_CREDENTIAL_FILE > /dev/null
echo "password=$STORAGE_ACCOUNT_KEY" | sudo tee -a $SMB_CREDENTIAL_FILE > /dev/null
else
echo "The credential file $SMB_CREDENTIAL_FILE already exists, and was not modified."
fi

# Change permissions on the credential file so only root can read or modify the password file.
sudo chmod 600 $SMB_CREDENTIAL_FILE

# This command assumes you have logged in with az login
HTTP_ENDPOINT=$(az storage account show --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --query "primaryEndpoints.file" --output tsv | tr -d '"')
SMB_PATH=$(echo $HTTP_ENDPOINT | cut -c7-${#HTTP_ENDPOINT})$FILE_SHARE_NAME

STORAGE_ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query "[0].value" --output tsv | tr -d '"')

sudo mount -t cifs $SMB_PATH $MNT_PATH -o credentials=$SMB_CREDENTIAL_FILE,serverino,nosharesock,actimeo=30,mfsymlinks,fsc,cache=strict

ls -al $MNT_PATH/TestData
echo "::endgroup::"

- name: Checkout repository
uses: actions/checkout@v2
Expand Down Expand Up @@ -331,7 +308,7 @@ runs:
coverage.xml
.coverage
include-hidden-files: true

- name: Container shutdown
if: always()
shell: bash
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/cicd-main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ permissions:

jobs:
pre-flight:
runs-on: ubuntu-latest
runs-on: nemo-ci-aws-cpu
outputs:
is_ci_workload: ${{ steps.is_ci_workload.outputs.main }}
no_fail_fast: ${{ steps.no_fail_fast.outputs.main }}
Expand Down Expand Up @@ -91,7 +91,7 @@ jobs:
)
&& !cancelled()
needs: [pre-flight, cicd-wait-in-queue]
runs-on: self-hosted-nemo
runs-on: nemo-ci-aws-gpu-x8
name: Launch_Unit_Tests
environment: nemo-ci
steps:
Expand Down Expand Up @@ -119,7 +119,7 @@ jobs:
- script: L2_Launch_converter
timeout: 60
needs: [cicd-unit-tests]
runs-on: self-hosted-nemo
runs-on: nemo-ci-aws-gpu-x2
if: |
(
success()
Expand Down
Loading