Run test template tests with Arcade #51020

Youssef1313 · 2025-09-26T08:24:46Z

This provides better error reporting for test failures which is also unified with everything else in CI. See also #50946 (comment)

eng/pipelines/templates/jobs/sdk-build.yml

MiYanni

So, while this is a cleaner solution to the existing operation, I'm still confused about the existing operation. This change means that all builds, public PR builds and internal CI builds, will run this secondary build. Isn't the point to get CG to scan the acquired dependencies from the templates? If so, CG doesn't run externally on PRs.

You linked to where the jobs are defined by default for the PR run in the sdk-job-matrix.yml. But there is a similar section in .vsts-ci.yml:

sdk/.vsts-ci.yml

Lines 111 to 143 in f684b50

    
           ${{ if and(eq(parameters.runTestBuild, false), ne(variables['Build.Reason'], 'PullRequest')) }}: 
        
             timeoutInMinutes: 180 
        
             windowsJobParameterSets: 
        
             ### OFFICIAL ### 
        
             - categoryName: Official 
        
               publishArgument: $(_publishArgument) 
        
               signArgument: $(_signArgument) 
        
               officialBuildProperties: $(_officialBuildProperties) /p:BuildWorkloads=true 
        
               enableDefaultArtifacts: true 
        
               runTests: false 
        
               publishRetryConfig: true 
        
               variables: 
        
                 _SignType: real 
        
             - categoryName: Official 
        
               targetArchitecture: x86 
        
               publishArgument: $(_publishArgument) 
        
               signArgument: $(_signArgument) 
        
               officialBuildProperties: $(_officialBuildProperties) 
        
               runTests: false 
        
               variables: 
        
                 _SignType: real 
        
               dependsOn: Official_windows_x64 
        
               downloadManifestMsiPackages: true 
        
             - categoryName: Official 
        
               targetArchitecture: arm64 
        
               publishArgument: $(_publishArgument) 
        
               signArgument: $(_signArgument) 
        
               officialBuildProperties: $(_officialBuildProperties) 
        
               runTests: false 
        
               variables: 
        
                 _SignType: real 
        
               dependsOn: Official_windows_x64 
        
               downloadManifestMsiPackages: true

Just make a new job in that section that only runs a build with the desired configuration.

Actually, I'm just gonna make a change to your PR so that we can get stuff moving.

… up invocation of populateInternalRuntimeVariables. Fixed Arcade template folder name paths. Minor formatting changes.

MiYanni · 2025-09-26T22:45:38Z

eng/pipelines/templates/jobs/sdk-job-matrix.yml

    container: azureLinux30Amd64
    helixTargetContainer: $(helixTargetContainerPrefix)ubuntu-24.04-helix-amd64
    osProperties: /p:OSName=linux /p:BuildSdkDeb=true
-    runTests: true


This was not necessary as runTests defaults to true.

MiYanni · 2025-09-26T22:46:27Z

eng/pipelines/templates/jobs/sdk-build.yml

            -restore -test -ci -prepareMachine -nativeToolsOnMachine
            -configuration $(buildConfiguration)
            /p:Projects=\`"${{ replace(parameters.testProjects, ';', '`;') }}\`"
+            /p:TestRunnerAdditionalArguments="${{ parameters.testRunnerAdditionalArguments }}"


Needed to add a way to pass this value. Since the arguments do not include ; and ,, but do contain spaces, the double doubles around the value should be sufficient.

MiYanni · 2025-09-26T22:47:12Z

.vsts-ci.yml

+            ### TestTemplatesCG ###
+            # Note: This job is only used to allow the test templates to be built locally on the agent as opposed to Helix.
+            # The tests acquire the templates' PackageReferences from NuGet, which allows them to be scanned by CG (component governance).
+            # CG is only ran internally, so this job makes sense to only run alongside of the official jobs.
+            - categoryName: TestTemplatesCG
+              testProjects: $(Build.SourcesDirectory)/test/dotnet-new.IntegrationTests/dotnet-new.IntegrationTests.csproj
+              testRunnerAdditionalArguments: -class Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewTestTemplatesTests
+              publishXunitResults: true


This is the primary change. Let me know if this doesn't make sense. @Youssef1313

It makes sense.

MiYanni · 2025-09-27T00:04:39Z

eng/pipelines/templates/jobs/sdk-build.yml

-      parameters:
-        legacyCredential: $(dn-bot-dnceng-artifact-feeds-rw)
-    - template: /eng/common/templates/steps/enable-internal-runtimes.yml
+    - ${{ if eq(parameters.populateInternalRuntimeVariables, true) }}:


This also didn't need to run on every build. It only applies to internal builds. So, added the parameter to control that.

MiYanni · 2025-09-27T00:06:29Z

.vsts-ci.yml

              os: macOS
            helixTargetQueue: osx.13.arm64
+            populateInternalRuntimeVariables: true
+            runtimeSourceProperties: /p:DotNetRuntimeSourceFeed=https://ci.dot.net/internal /p:DotNetRuntimeSourceFeedKey=$(dotnetbuilds-internal-container-read-token-base64)


I'm fairly certain this is necessary but was missing. This job is only ran when the checkbox for enableArm64Job is used in a manual run. So, probably no one noticed.

Youssef1313 · 2025-09-30T19:20:57Z

@MiYanni Are we able to get this one merged soon please?

MiYanni

Approving, but I'd be comfortable if someone else also reviewed, since I ended up doing a bunch of changes.

MiYanni · 2025-10-01T00:39:57Z

@agocke We're getting an AOT test failure:

[xUnit.net 00:02:35.43]     Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishAnAotApp.NativeAot_hw_fails_with_unsupported_host_rid(targetFramework: "net10.0") [FAIL]
...
[xUnit.net 00:02:35.43]         Generating native code
[xUnit.net 00:02:35.43]       gcc : error : unrecognized command-line option ‘--target=aarch64-linux-gnu’

agocke · 2025-10-01T07:46:30Z

Does it fail without the changes in this PR?

Youssef1313 · 2025-10-01T08:12:04Z

@agocke The failure is unrelated to the changes as it was seen before. #50901 (comment)

I restarted the job as it's likely flakiness and not consistent failure.

Youssef1313 · 2025-10-02T09:59:06Z

@MiYanni It's green now. Can we get it in ASAP to try to understand why the merge of release/10.0.1xx to main is failing?

Ping @dotnet/dotnet-cli for review.

Youssef1313 · 2025-10-02T10:27:16Z

This is also blocking the flow of changes in 10.0.1xx that went post-RC2 to VMR.

Run test template tests with Arcade

571fd37

Youssef1313 force-pushed the dev/ygerges/test-template-tests branch from 0b13233 to 571fd37 Compare September 26, 2025 08:25

Update sdk-build.yml

ac67de5

Youssef1313 commented Sep 26, 2025

View reviewed changes

eng/pipelines/templates/jobs/sdk-build.yml Outdated Show resolved Hide resolved

Only test templates

7e865a2

Youssef1313 marked this pull request as ready for review September 26, 2025 10:43

Youssef1313 requested review from MiYanni, mariam-abdulla and marcpopMSFT September 26, 2025 10:43

Youssef1313 mentioned this pull request Sep 26, 2025

[automated] Merge branch 'release/10.0.1xx' => 'main' #50946

Open

MiYanni requested changes Sep 26, 2025

View reviewed changes

build-analysis bot mentioned this pull request Sep 26, 2025

GenerateBundle Task fails writing to singlefilehost.exe #50784

Open

Added job for TestTemplatesCG to run only on official builds. Cleaned…

537aa47

… up invocation of populateInternalRuntimeVariables. Fixed Arcade template folder name paths. Minor formatting changes.

MiYanni reviewed Sep 26, 2025

View reviewed changes

MiYanni reviewed Sep 27, 2025

View reviewed changes

MiYanni approved these changes Sep 30, 2025

View reviewed changes

MiYanni mentioned this pull request Oct 1, 2025

Set dogfood script working directory #50895

Open

MiYanni requested a review from a team October 1, 2025 00:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Run test template tests with Arcade #51020

Run test template tests with Arcade #51020

Youssef1313 commented Sep 26, 2025 •

edited

Loading

Uh oh!

Uh oh!

MiYanni left a comment

Uh oh!

MiYanni Sep 26, 2025

Uh oh!

MiYanni Sep 26, 2025

Uh oh!

MiYanni Sep 26, 2025

Uh oh!

Youssef1313 Sep 29, 2025

Uh oh!

MiYanni Sep 27, 2025

Uh oh!

MiYanni Sep 27, 2025 •

edited

Loading

Uh oh!

Youssef1313 commented Sep 30, 2025

Uh oh!

MiYanni left a comment

Uh oh!

MiYanni commented Oct 1, 2025

Uh oh!

agocke commented Oct 1, 2025

Uh oh!

Youssef1313 commented Oct 1, 2025 •

edited

Loading

Uh oh!

Youssef1313 commented Oct 2, 2025

Uh oh!

Youssef1313 commented Oct 2, 2025

Uh oh!

Uh oh!

	${{ if and(eq(parameters.runTestBuild, false), ne(variables['Build.Reason'], 'PullRequest')) }}:
	timeoutInMinutes: 180
	windowsJobParameterSets:
	### OFFICIAL ###
	- categoryName: Official
	publishArgument: $(_publishArgument)
	signArgument: $(_signArgument)
	officialBuildProperties: $(_officialBuildProperties) /p:BuildWorkloads=true
	enableDefaultArtifacts: true
	runTests: false
	publishRetryConfig: true
	variables:
	_SignType: real
	- categoryName: Official
	targetArchitecture: x86
	publishArgument: $(_publishArgument)
	signArgument: $(_signArgument)
	officialBuildProperties: $(_officialBuildProperties)
	runTests: false
	variables:
	_SignType: real
	dependsOn: Official_windows_x64
	downloadManifestMsiPackages: true
	- categoryName: Official
	targetArchitecture: arm64
	publishArgument: $(_publishArgument)
	signArgument: $(_signArgument)
	officialBuildProperties: $(_officialBuildProperties)
	runTests: false
	variables:
	_SignType: real
	dependsOn: Official_windows_x64
	downloadManifestMsiPackages: true

Run test template tests with Arcade #51020

Are you sure you want to change the base?

Run test template tests with Arcade #51020

Conversation

Youssef1313 commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

MiYanni left a comment

Choose a reason for hiding this comment

Uh oh!

MiYanni Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

MiYanni Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

MiYanni Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

Youssef1313 Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

MiYanni Sep 27, 2025

Choose a reason for hiding this comment

Uh oh!

MiYanni Sep 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Youssef1313 commented Sep 30, 2025

Uh oh!

MiYanni left a comment

Choose a reason for hiding this comment

Uh oh!

MiYanni commented Oct 1, 2025

Uh oh!

agocke commented Oct 1, 2025

Uh oh!

Youssef1313 commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Youssef1313 commented Oct 2, 2025

Uh oh!

Youssef1313 commented Oct 2, 2025

Uh oh!

Uh oh!

Youssef1313 commented Sep 26, 2025 •

edited

Loading

MiYanni Sep 27, 2025 •

edited

Loading

Youssef1313 commented Oct 1, 2025 •

edited

Loading