Skip to content

Run test template tests with Arcade #51020

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 3, 2025
+41 −61
Merged

Run test template tests with Arcade #51020

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
571fd37
Run test template tests with Arcade
Youssef1313 Sep 26, 2025
ac67de5
Update sdk-build.yml
Youssef1313 Sep 26, 2025
7e865a2
Only test templates
Youssef1313 Sep 26, 2025
537aa47
Added job for TestTemplatesCG to run only on official builds. Cleaned…
MiYanni Sep 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .vsts-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ extends:
oneESCompat:
templateFolderName: templates-official
publishTaskPrefix: 1ES.
populateInternalRuntimeVariables: true
runtimeSourceProperties: /p:DotNetRuntimeSourceFeed=https://ci.dot.net/internal /p:DotNetRuntimeSourceFeedKey=$(dotnetbuilds-internal-container-read-token-base64)
locBranch: release/10.0.1xx
# WORKAROUND: BinSkim requires the folder exist prior to scanning.
Expand Down Expand Up @@ -141,6 +142,14 @@ extends:
_SignType: real
dependsOn: Official_windows_x64
downloadManifestMsiPackages: true
### TestTemplatesCG ###
# Note: This job is only used to allow the test templates to be built locally on the agent as opposed to Helix.
# The tests acquire the templates' PackageReferences from NuGet, which allows them to be scanned by CG (component governance).
# CG is only ran internally, so this job makes sense to only run alongside of the official jobs.
- categoryName: TestTemplatesCG
testProjects: $(Build.SourcesDirectory)/test/dotnet-new.IntegrationTests/dotnet-new.IntegrationTests.csproj
testRunnerAdditionalArguments: -class Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewTestTemplatesTests
publishXunitResults: true
Comment on lines +145 to +152
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the primary change. Let me know if this doesn't make sense. @Youssef1313

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes sense.


############### LINUX ###############
- template: /eng/pipelines/templates/jobs/sdk-job-matrix.yml@self
Expand All @@ -153,6 +162,7 @@ extends:
oneESCompat:
templateFolderName: templates-official
publishTaskPrefix: 1ES.
populateInternalRuntimeVariables: true
runtimeSourceProperties: /p:DotNetRuntimeSourceFeed=https://ci.dot.net/internal /p:DotNetRuntimeSourceFeedKey=$(dotnetbuilds-internal-container-read-token-base64)
${{ if and(eq(parameters.runTestBuild, false), ne(variables['Build.Reason'], 'PullRequest')) }}:
timeoutInMinutes: 90
Expand Down Expand Up @@ -235,6 +245,7 @@ extends:
oneESCompat:
templateFolderName: templates-official
publishTaskPrefix: 1ES.
populateInternalRuntimeVariables: true
runtimeSourceProperties: /p:DotNetRuntimeSourceFeed=https://ci.dot.net/internal /p:DotNetRuntimeSourceFeedKey=$(dotnetbuilds-internal-container-read-token-base64)
${{ if and(eq(parameters.runTestBuild, false), ne(variables['Build.Reason'], 'PullRequest')) }}:
timeoutInMinutes: 90
Expand All @@ -260,6 +271,8 @@ extends:
vmImage: macOS-latest
os: macOS
helixTargetQueue: osx.13.arm64
populateInternalRuntimeVariables: true
runtimeSourceProperties: /p:DotNetRuntimeSourceFeed=https://ci.dot.net/internal /p:DotNetRuntimeSourceFeedKey=$(dotnetbuilds-internal-container-read-token-base64)
Copy link
Member

@MiYanni MiYanni Sep 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fairly certain this is necessary but was missing. This job is only ran when the checkbox for enableArm64Job is used in a manual run. So, probably no one noticed.

macOSJobParameterSets:
- categoryName: TestBuild
targetArchitecture: arm64
Expand All @@ -270,7 +283,9 @@ extends:
- template: /eng/dotnet-format/dotnet-format-integration.yml@self
parameters:
oneESCompat:
templateFolderName: templates-official
publishTaskPrefix: 1ES.
populateInternalRuntimeVariables: true
runtimeSourceProperties: /p:DotNetRuntimeSourceFeed=https://ci.dot.net/internal /p:DotNetRuntimeSourceFeedKey=$(dotnetbuilds-internal-container-read-token-base64)

############### PUBLISH STAGE ###############
Expand Down
46 changes: 0 additions & 46 deletions build/RunTestTemplateTests.ps1
View file
Open in desktop

This file was deleted.

18 changes: 14 additions & 4 deletions eng/dotnet-format/dotnet-format-integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ parameters:
- name: oneESCompat
type: object
default:
templateFolderName: templates
publishTaskPrefix: ''

- name: TestArguments
Expand Down Expand Up @@ -63,6 +64,9 @@ parameters:
- name: runtimeSourceProperties
type: string
default: ''
- name: populateInternalRuntimeVariables
type: boolean
default: false

jobs:
- job: Formatting_Check
Expand All @@ -77,8 +81,11 @@ jobs:
os: windows
timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
steps:
- template: /eng/common/templates/steps/enable-internal-runtimes.yml
- template: /eng/common/templates/steps/enable-internal-sources.yml
- ${{ if eq(parameters.populateInternalRuntimeVariables, true) }}:
- template: /eng/common/${{ parameters.oneESCompat.templateFolderName }}/steps/enable-internal-sources.yml
parameters:
legacyCredential: $(dn-bot-dnceng-artifact-feeds-rw)
- template: /eng/common/${{ parameters.oneESCompat.templateFolderName }}/steps/enable-internal-runtimes.yml
- script: .\restore.cmd ${{ parameters.runtimeSourceProperties }}
displayName: 🟣 Restore dependencies
- script: |
Expand Down Expand Up @@ -106,8 +113,11 @@ jobs:
os: windows
timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
steps:
- template: /eng/common/templates/steps/enable-internal-runtimes.yml
- template: /eng/common/templates/steps/enable-internal-sources.yml
- ${{ if eq(parameters.populateInternalRuntimeVariables, true) }}:
- template: /eng/common/${{ parameters.oneESCompat.templateFolderName }}/steps/enable-internal-sources.yml
parameters:
legacyCredential: $(dn-bot-dnceng-artifact-feeds-rw)
- template: /eng/common/${{ parameters.oneESCompat.templateFolderName }}/steps/enable-internal-runtimes.yml
- script: eng\dotnet-format\integration-test.cmd -repo '${{ testArgs._repo }}' -branchName '${{ testArgs._branchName }}' -sha '${{ testArgs._sha }}' -targetSolution '${{ testArgs._targetSolution }}' -useParentSdk ${{ testArgs._useParentSdk }} -testPath '$(Agent.TempDirectory)\temp' -stage 'prepare' -runtimeSourceProperties '${{ parameters.runtimeSourceProperties }}'
displayName: 🟣 Prepare ${{ testArgs._repoName }} for formatting

Expand Down
22 changes: 12 additions & 10 deletions eng/pipelines/templates/jobs/sdk-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@ parameters:
testProjects: $(Build.SourcesDirectory)/test/UnitTests.proj
publishRetryConfig: false
publishXunitResults: false
testRunnerAdditionalArguments: ''
enableSbom: true
populateInternalRuntimeVariables: false
timeoutInMinutes: 150
### ENV VARS ###
testFullMSBuild: false
Expand Down Expand Up @@ -71,11 +73,11 @@ jobs:
targetPath: $(Build.SourcesDirectory)/eng/BuildConfiguration
artifactName: BuildConfiguration

# Populate internal runtime variables.
- template: /eng/common/templates/steps/enable-internal-sources.yml
parameters:
legacyCredential: $(dn-bot-dnceng-artifact-feeds-rw)
- template: /eng/common/templates/steps/enable-internal-runtimes.yml
- ${{ if eq(parameters.populateInternalRuntimeVariables, true) }}:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also didn't need to run on every build. It only applies to internal builds. So, added the parameter to control that.

- template: /eng/common/${{ parameters.oneESCompat.templateFolderName }}/steps/enable-internal-sources.yml
parameters:
legacyCredential: $(dn-bot-dnceng-artifact-feeds-rw)
- template: /eng/common/${{ parameters.oneESCompat.templateFolderName }}/steps/enable-internal-runtimes.yml

- ${{ if eq(parameters.downloadManifestMsiPackages, true) }}:
- task: DownloadBuildArtifacts@1
Expand All @@ -84,7 +86,7 @@ jobs:
downloadPath: $(Build.SourcesDirectory)/artifacts/downloaded-manifest-msi-packages
itemPattern: '**/*Manifest-*.Msi.*.nupkg'
checkDownloadedFiles: true
displayName: Download Manifest msi packages
displayName: 🟣 Download Manifest MSI Packages

############### BUILDING ###############
- ${{ if eq(parameters.pool.os, 'windows') }}:
Expand All @@ -105,10 +107,6 @@ jobs:
BuildConfig: $(buildConfiguration)
TestFullMSBuild: ${{ parameters.testFullMSBuild }}

- ${{ if eq(parameters.targetArchitecture, 'x64') }}:
- powershell: build/RunTestTemplateTests.ps1
displayName: 🟣 Run Test Templates Tests

- ${{ else }}:
- script: |
source $(Build.SourcesDirectory)/eng/common/native/init-os-and-arch.sh
Expand Down Expand Up @@ -141,12 +139,14 @@ jobs:
- script: $(Build.SourcesDirectory)/artifacts/bin/redist/$(buildConfiguration)/dotnet/dotnet workload install wasm-tools --skip-manifest-update
workingDirectory: $(Build.SourcesDirectory)/artifacts/bin
displayName: 🟣 Install wasm-tools Workload

# For the /p:Projects syntax for PowerShell, see: https://github.com/dotnet/msbuild/issues/471#issuecomment-1146466335
- ${{ if eq(parameters.pool.os, 'windows') }}:
- powershell: eng/common/build.ps1
-restore -test -ci -prepareMachine -nativeToolsOnMachine
-configuration $(buildConfiguration)
/p:Projects=\`"${{ replace(parameters.testProjects, ';', '`;') }}\`"
/p:TestRunnerAdditionalArguments="${{ parameters.testRunnerAdditionalArguments }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed to add a way to pass this value. Since the arguments do not include ; and ,, but do contain spaces, the double doubles around the value should be sufficient.

/p:TargetArchitecture=${{ parameters.targetArchitecture }}
${{ parameters.runtimeSourceProperties }}
/p:CustomHelixTargetQueue=${{ parameters.helixTargetQueue }}
Expand All @@ -159,6 +159,7 @@ jobs:
HelixAccessToken: $(HelixApiAccessToken)
RunAoTTests: ${{ parameters.runAoTTests }}
TestFullMSBuild: ${{ parameters.testFullMSBuild }}

- ${{ else }}:
# For the /p:Projects syntax for Bash, see: https://github.com/dotnet/msbuild/issues/471#issuecomment-1690189034
# The /p:CustomHelixTargetQueue syntax is: <queue-name>@<container-url>
Expand All @@ -167,6 +168,7 @@ jobs:
-restore -test -ci -prepareMachine
-configuration $(buildConfiguration)
'/p:Projects="${{ parameters.testProjects }}"'
/p:TestRunnerAdditionalArguments="${{ parameters.testRunnerAdditionalArguments }}"
/p:TargetArchitecture=${{ parameters.targetArchitecture }}
/p:TargetRid=${{ parameters.runtimeIdentifier }}
${{ parameters.osProperties }}
Expand Down
1 change: 0 additions & 1 deletion eng/pipelines/templates/jobs/sdk-job-matrix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ parameters:
container: azureLinux30Amd64
helixTargetContainer: $(helixTargetContainerPrefix)ubuntu-24.04-helix-amd64
osProperties: /p:OSName=linux /p:BuildSdkDeb=true
runTests: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was not necessary as runTests defaults to true.

# Helix is hanging on this job using the container. See: https://github.com/dotnet/dnceng/issues/6000
disableJob: true
- categoryName: TemplateEngine
Expand Down
Loading