Skip to content

Commit 4cf3f9a

Browse files
ludfjigludfjig
authored
Test MSHV 6.18 weekly and benchmark it daily (#1941)
Tests MSHV 6.18 weekly and benchmarks it daily on Azure Linux 3 with the released MSHV 6.18 kernel * Adds `WeeklyMshv618.yml` job to run tests and examples * Adds a `benchmarks-mshv-6-18` job to `DailyBenchmarks.yml` * Adds an optional `mshv_kernel` input to `dep_build_test.yml`, `dep_run_examples.yml`, and `dep_benchmarks.yml`. When it is set, mshv3 jobs run on `hld-azlinux3-mshv-<mshv_kernel>-<vendor>`. Existing callers do not pass it and keep their pools. * When `mshv_kernel` is set, benchmark artifacts are named `benchmarks_Linux_mshv3-<mshv_kernel>_<vendor>`. This avoids a name collision with the 6.6 artifacts in the same daily run. `bench-report` reads the label as its own hypervisor, so 6.6 results never match a 6.18 baseline. The new pools `hld-azlinux3-mshv-6.18-amd` and `hld-azlinux3-mshv-6.18-intel` hold up max 3 VMs each to save azure quota. This is suffient since we don't run per pr passed runs: https://github.com/hyperlight-dev/hyperlight/actions/runs/38781317655?pr=1941 --------- Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
1 parent 83849ab commit 4cf3f9a

7 files changed

Lines changed: 127 additions & 10 deletions

File tree

‎.github/workflows/DailyBenchmarks.yml‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,11 +38,27 @@ jobs:
3838
cpu_vendor: ${{ matrix.cpu_vendor }}
3939
arch: ${{ matrix.arch }}
4040

41+
# mshv3 on the MSHV 6.18 kernel pools. A separate job, so a failure here
42+
# cannot cancel the matrix above.
43+
benchmarks-mshv-6-18:
44+
needs: [build-guests]
45+
strategy:
46+
fail-fast: false
47+
matrix:
48+
cpu_vendor: [amd, intel]
49+
uses: ./.github/workflows/dep_benchmarks.yml
50+
secrets: inherit
51+
with:
52+
hypervisor: mshv3
53+
mshv_kernel: "6.18"
54+
cpu_vendor: ${{ matrix.cpu_vendor }}
55+
arch: X64
56+
4157
# File a GitHub issue if any job fails.
4258
notify-failure:
4359
runs-on: ubuntu-latest
44-
needs: [build-guests, benchmarks]
45-
if: always() && (needs.build-guests.result == 'failure' || needs.benchmarks.result == 'failure')
60+
needs: [build-guests, benchmarks, benchmarks-mshv-6-18]
61+
if: always() && (needs.build-guests.result == 'failure' || needs.benchmarks.result == 'failure' || needs.benchmarks-mshv-6-18.result == 'failure')
4662
permissions:
4763
issues: write
4864
steps:

‎.github/workflows/WeeklyMshv618.yml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json
2+
3+
name: Weekly MSHV 6.18
4+
5+
# Runs the mshv3 tests and examples on the Azure Linux 3 pools with the MSHV
6+
# 6.18 kernel. PRs test only the default mshv3 pools. DailyBenchmarks.yml
7+
# benchmarks these pools.
8+
9+
on:
10+
schedule:
11+
- cron: '0 3 * * 1' # Runs at 03:00 UTC every Monday
12+
workflow_dispatch: # Allow manual triggering
13+
14+
permissions:
15+
contents: read
16+
17+
jobs:
18+
build-guests:
19+
strategy:
20+
fail-fast: false
21+
matrix:
22+
config: [debug, release]
23+
uses: ./.github/workflows/dep_build_guests.yml
24+
secrets: inherit
25+
with:
26+
arch: X64
27+
config: ${{ matrix.config }}
28+
29+
build-test:
30+
needs: build-guests
31+
permissions:
32+
# checkout in the called workflow
33+
contents: read
34+
# pull goldens from GHCR in the called workflow
35+
packages: read
36+
strategy:
37+
fail-fast: false
38+
matrix:
39+
cpu_vendor: [amd, intel]
40+
config: [debug, release]
41+
uses: ./.github/workflows/dep_build_test.yml
42+
secrets: inherit
43+
with:
44+
hypervisor: mshv3
45+
mshv_kernel: "6.18"
46+
cpu_vendor: ${{ matrix.cpu_vendor }}
47+
arch: X64
48+
config: ${{ matrix.config }}
49+
50+
run-examples:
51+
needs: build-guests
52+
strategy:
53+
fail-fast: false
54+
matrix:
55+
cpu_vendor: [amd, intel]
56+
config: [debug, release]
57+
uses: ./.github/workflows/dep_run_examples.yml
58+
secrets: inherit
59+
with:
60+
hypervisor: mshv3
61+
mshv_kernel: "6.18"
62+
cpu_vendor: ${{ matrix.cpu_vendor }}
63+
arch: X64
64+
config: ${{ matrix.config }}
65+
66+
# The first label is the de-duplication key used by notify-ci-failure.sh, so
67+
# these failures share one release-blocking issue. That label must exist in
68+
# the repo or `gh issue create` fails.
69+
notify-failure:
70+
runs-on: ubuntu-latest
71+
needs: [build-guests, build-test, run-examples]
72+
if: always() && github.event_name != 'pull_request' && (needs.build-guests.result == 'failure' || needs.build-test.result == 'failure' || needs.run-examples.result == 'failure')
73+
permissions:
74+
# Job-level permissions replace the workflow-level grant, so checkout needs contents:read here.
75+
contents: read
76+
issues: write
77+
steps:
78+
- name: Checkout code
79+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
80+
81+
- name: Notify Weekly MSHV 6.18 Failure
82+
run: ./dev/notify-ci-failure.sh --title="Weekly MSHV 6.18 Failure - ${{ github.run_number }}" --labels="area/ci-periodics-mshv-6.18,area/ci-periodics,area/testing,lifecycle/needs-review,release-blocker"
83+
env:
84+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/dep_benchmarks.yml‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,10 @@
55
# Artifact upload:
66
# Benchmark results are uploaded as workflow artifacts named
77
# benchmarks_<OS>_<hypervisor>_<cpu_vendor>, kept for as long as GitHub
8-
# allows so that a later run can still be compared against them. Reading
9-
# them is left to whoever wants a report, so that a comparison covers every
10-
# configuration at once.
8+
# allows so that a later run can still be compared against them. A set
9+
# mshv_kernel joins the hypervisor, as in benchmarks_Linux_mshv3-6.18_amd.
10+
# Reading them is left to whoever wants a report, so that a comparison
11+
# covers every configuration at once.
1112

1213
name: Run Benchmarks
1314

@@ -31,6 +32,11 @@ on:
3132
description: CPU architecture for the build, X64 or arm64 (passed from caller matrix)
3233
required: true
3334
type: string
35+
mshv_kernel:
36+
description: MSHV kernel of the Azure Linux 3 pool, for example 6.18. Empty selects the default mshv3 pool.
37+
required: false
38+
type: string
39+
default: ""
3440
baseline_ref:
3541
description: >-
3642
Commit to measure these results against. When set, its benchmarks run
@@ -57,7 +63,7 @@ jobs:
5763
runs-on: ${{ fromJson(inputs.arch == 'X64'
5864
&& format('["self-hosted", "{0}", "X64", "1ES.Pool=hld-{1}-{2}", "JobId=run-benchmarks-{3}-{4}-{5}"]',
5965
inputs.hypervisor == 'hyperv-ws2025' && 'Windows' || 'Linux',
60-
inputs.hypervisor == 'hyperv-ws2025' && 'win2025' || inputs.hypervisor == 'mshv3' && 'azlinux3-mshv' || inputs.hypervisor,
66+
inputs.hypervisor == 'hyperv-ws2025' && 'win2025' || inputs.hypervisor == 'mshv3' && format('azlinux3-mshv{0}', inputs.mshv_kernel != '' && format('-{0}', inputs.mshv_kernel) || '') || inputs.hypervisor,
6167
inputs.cpu_vendor,
6268
github.run_id,
6369
github.run_number,
@@ -131,7 +137,7 @@ jobs:
131137
132138
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
133139
with:
134-
name: benchmarks_${{ runner.os }}_${{ inputs.hypervisor }}_${{ inputs.cpu_vendor }}
140+
name: benchmarks_${{ runner.os }}_${{ inputs.hypervisor }}${{ inputs.mshv_kernel != '' && format('-{0}', inputs.mshv_kernel) || '' }}_${{ inputs.cpu_vendor }}
135141
path: ./target/criterion/
136142
if-no-files-found: error
137143
# The longest GitHub keeps an artifact, so that a run stays readable

‎.github/workflows/dep_build_test.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,11 @@ on:
2626
description: CPU architecture for the build (passed from caller matrix)
2727
required: true
2828
type: string
29+
mshv_kernel:
30+
description: MSHV kernel of the Azure Linux 3 pool, for example 6.18. Empty selects the default mshv3 pool.
31+
required: false
32+
type: string
33+
default: ""
2934
full_aarch64:
3035
description: >-
3136
When "true", also run the single-driver tests on aarch64. PRs leave
@@ -61,7 +66,7 @@ jobs:
6166
runs-on: ${{ fromJson(inputs.arch == 'X64'
6267
&& format('["self-hosted", "{0}", "X64", "1ES.Pool=hld-{1}-{2}", "JobId=build-and-test-{3}-{4}-{5}-{6}"]',
6368
inputs.hypervisor == 'hyperv-ws2025' && 'Windows' || 'Linux',
64-
inputs.hypervisor == 'hyperv-ws2025' && 'win2025' || inputs.hypervisor == 'mshv3' && 'azlinux3-mshv' || inputs.hypervisor,
69+
inputs.hypervisor == 'hyperv-ws2025' && 'win2025' || inputs.hypervisor == 'mshv3' && format('azlinux3-mshv{0}', inputs.mshv_kernel != '' && format('-{0}', inputs.mshv_kernel) || '') || inputs.hypervisor,
6570
inputs.cpu_vendor,
6671
inputs.config,
6772
github.run_id,

‎.github/workflows/dep_run_examples.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,11 @@ on:
2626
description: CPU architecture for the build (passed from caller matrix)
2727
required: true
2828
type: string
29+
mshv_kernel:
30+
description: MSHV kernel of the Azure Linux 3 pool, for example 6.18. Empty selects the default mshv3 pool.
31+
required: false
32+
type: string
33+
default: ""
2934

3035
env:
3136
CARGO_TERM_COLOR: always
@@ -45,7 +50,7 @@ jobs:
4550
runs-on: ${{ fromJson(inputs.arch == 'X64'
4651
&& format('["self-hosted", "{0}", "X64", "1ES.Pool=hld-{1}-{2}", "JobId=run-examples-{3}-{4}-{5}-{6}"]',
4752
inputs.hypervisor == 'hyperv-ws2025' && 'Windows' || 'Linux',
48-
inputs.hypervisor == 'hyperv-ws2025' && 'win2025' || inputs.hypervisor == 'mshv3' && 'azlinux3-mshv' || inputs.hypervisor,
53+
inputs.hypervisor == 'hyperv-ws2025' && 'win2025' || inputs.hypervisor == 'mshv3' && format('azlinux3-mshv{0}', inputs.mshv_kernel != '' && format('-{0}', inputs.mshv_kernel) || '') || inputs.hypervisor,
4954
inputs.cpu_vendor,
5055
inputs.config,
5156
github.run_id,

‎docs/benchmarking-hyperlight.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Hyperlight uses the [Criterion](https://bheisler.github.io/criterion.rs/book/ind
55
## When Benchmarks are run
66

77
1. Daily (scheduled)
8-
- Benchmarks run daily via `DailyBenchmarks.yml`. Results are stored as workflow artifacts with 90-day retention, and are what pull requests are compared against.
8+
- Benchmarks run daily via `DailyBenchmarks.yml`. Results are stored as workflow artifacts with 90-day retention, and are what pull requests are compared against. The daily run also benchmarks mshv3 on the MSHV 6.18 kernel pools, as `benchmarks_Linux_mshv3-6.18_<cpu_vendor>`.
99

1010
```
1111
sandboxes/create_sandbox

‎docs/github-labels.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ In addition to **kind/*** labels, we use optional **area/*** labels to specify t
6363
- **area/API** - Related to the API or public interface.
6464
- **area/ci-periodics** - Applied to issues that track failures from scheduled (periodic) CI jobs. These issues are opened automatically by the CI failure notifier (`dev/notify-ci-failure.sh`).
6565
- **area/ci-periodics-aarch64** - De-duplication label for failures from the daily aarch64 workflow (`.github/workflows/DailyArm64.yml`). It keeps those failures on their own dedicated, release-blocking issue instead of sharing the general `area/ci-periodics` issue.
66+
- **area/ci-periodics-mshv-6.18** - De-duplication label for failures from the weekly MSHV 6.18 workflow (`.github/workflows/WeeklyMshv618.yml`). It keeps those failures on their own release-blocking issue.
6667
- **area/dependencies** - Concerns dependencies or related components. This label is different from **kind/dependencies**, which should only used for PRs.
6768
- **area/documentation** - Related to documentation updates or improvements.
6869
- **area/infrastructure** - Concerns infrastructure rather than core functionality.

0 commit comments

Comments
 (0)