chore: [Backport] Small yamato tweaks (#3724)

* chore: [Backport] Small yamato tweaks

* Backport of PR-3725

---------

Co-authored-by: michal-chrobot <[email protected]>

Author: EmandM

.yamato/README.md

@@ -9,7 +9,6 @@ CI related files are present inside .yamato/ folder and we can distinguish speci
 
 ### Helper jobs
 - `.yamato/package-pack.yml` responsible for generating package artifacts (.tgz) required for testing and publishing.
-- `.yamato/project-pack.yml` responsible for generating package artifacts (.tgz) required for testing and publishing. This packs all packages of a given project.
 - `.yamato/_run-all.yml` responsible for grouping tests into groups for easier management (for example "all console tests").
 - `.yamato/_triggers.yml` responsible for defining triggers (PR, nightly, weekly etc.) and defining which tests to run.
 - `disable-burst-if-requested.py` responsible for helping to disable burst if needed.
@@ -77,4 +76,4 @@ Currently, the CI implementation supports the following platforms:
 
 ## Design Considerations
 In theory, we could manually write jobs for every configuration. However, this approach would be more error-prone, especially when modifications or fixes are needed, as it would require keeping track of all configurations.
-The downside of our current approach is that it can sometimes impact readability due to the use of nested if and for statements.
+The downside of our current approach is that it can sometimes impact readability due to the use of nested if and for statements.

.yamato/_run-all.yml

@@ -17,8 +17,6 @@ run_quick_checks:
   dependencies:
     - .yamato/package-pack.yml#package_pack_-_ngo_ubuntu
     - .yamato/project-standards.yml#standards_ubuntu_testproject_6000.2
-    # Run API validation to early-detect all new APIs that would force us to release new minor version of the package. Note that for this to work the package version in package.json must correspond to "actual package state" which means that it should be higher than last released version
-    - .yamato/vetting-test.yml#vetting_test
 
 # Runs all package tests
 run_all_package_tests:

.yamato/_triggers.yml

@@ -41,7 +41,7 @@
     # It's important to ensure that all dependencies exist (this can be verified in Yamato) since a modification in parameters may result in a given job not being generated, and thus we will not be able to run such erroneous job.
 
 
-#-----------------------------------------------------------------------------------  
+#-----------------------------------------------------------------------------------
 
 # After some experimenting with CI setups we discovered that even though sometimes we don't need CI to run (no reason to run package tests if only Documentation is changed) there are some checks that devs may not realize but changes in seemingly unrelated files will cause their failures
 # This trigger was created to ensure that ALL PRs run this minimal check even when we don't need to run full tests
@@ -69,6 +69,9 @@ pr_code_changes_checks:
   name: Code changes PR checks
   # Run the following tests on a selection of different desktop platforms
   dependencies:
+    # Run API validation to early-detect all new APIs that would force us to release new minor version of the package. Note that for this to work the package version in package.json must correspond to "actual package state" which means that it should be higher than last released version
+    - .yamato/vetting-test.yml#vetting_test
+
     # Run package EditMode and Playmode package tests on 6000.2 (latest supported editor) and an older supported editor (2022.3) (2022.3 will soon be a minimum supported editor)
     - .yamato/package-tests.yml#package_test_-_ngo_6000.2_mac
     - .yamato/package-tests.yml#package_test_-_ngo_2022.3_win
@@ -104,9 +107,9 @@ pr_code_changes_checks:
         "**/*.md"
       ]
     cancel_old_ci: true
-      
-      
-      
+
+
+
 
 
 
@@ -122,6 +125,8 @@ develop_nightly:
         frequency: daily
         rerun: always
   dependencies:
+    # Run API validation to early-detect all new APIs that would force us to release new minor version of the package. Note that for this to work the package version in package.json must correspond to "actual package state" which means that it should be higher than last released version
+    - .yamato/vetting-test.yml#vetting_test
     # Run project standards to verify package/default project
     - .yamato/project-standards.yml#standards_ubuntu_testproject_6000.2
     - .yamato/project-standards.yml#standards_ubuntu_testproject_2022.3
@@ -164,6 +169,8 @@ develop_weekly_trunk:
         frequency: weekly
         rerun: always
   dependencies:
+    # Run API validation to early-detect all new APIs that would force us to release new minor version of the package. Note that for this to work the package version in package.json must correspond to "actual package state" which means that it should be higher than last released version
+    - .yamato/vetting-test.yml#vetting_test
     # Run project standards to verify package/default project
     - .yamato/_run-all.yml#run_all_projects_standards
     # Run package EditMode and Playmode tests on desktop platforms

.yamato/code-coverage.yml

@@ -8,22 +8,22 @@
   # Line and branch coverage statistics
   # Generated HTML reports for coverage visualization
   # Additional metrics for coverage analysis
-  
+
 # CONFIGURATION STRUCTURE--------------------------------------------------------------
   # Jobs are generated using nested loops through:
   # 1. For default platform only (Ubuntu) since coverage would not vary between platforms (no need for checks on more platforms)
   # 2. For default editor version (6000.2) since coverage would not vary between editors (no need for checks on more editors)
-  
+
 #TECHNICAL CONSIDERATIONS---------------------------------------------------------------
   # In theory this job also runs package tests, but we don't want to use it as default since is heavier (because of added coverage analysis) and coverage is not changing that often
   # Requires Unity Editor installation
   # Burst compilation is disabled to ensure accurate coverage measurement
   # In order to properly use -coverage-results-path parameter we need to start it with $PWD (which means the absolute path). Otherwise, coverage results will not be visible
-  
+
 # QUALITY CONSIDERATIONS--------------------------------------------------------------------
   # To see where this job is included (in trigger job definitions) look into _triggers.yml file
-  
-  
+
+
 {% for platform in test_platforms.default -%}
 {% for editor in validation_editors.default -%}
 code_coverage_{{ platform.name }}_{{ editor }}:
@@ -38,12 +38,12 @@ code_coverage_{{ platform.name }}_{{ editor }}:
   commands:
     - unity-downloader-cli --fast --wait -u {{ editor }} -c Editor {% if platform.name == "mac" %} --arch arm64 {% endif %} # For macOS we use ARM64 models
     - upm-pvp create-test-project test-project --packages "upm-ci~/packages/*.tgz" --unity .Editor
-    - UnifiedTestRunner --suite=editor --suite=playmode --editor-location=.Editor --testproject=test-project --enable-code-coverage coverage-upload-options="reportsDir:$PWD/test-results/CoverageResults;name:NGOv1_{{ platform.name }}_{{ editor }};flags:NGOv1_{{ platform.name }}_{{ editor }};verbose" --coverage-results-path=$PWD/test-results/CoverageResults --coverage-options="generateHtmlReport;generateAdditionalMetrics;assemblyFilters:+Unity.Netcode.Editor,+Unity.Netcode.Runtime" --extra-editor-arg=--burst-disable-compilation --timeout=1800 --reruncount=1 --clean-library-on-rerun --artifacts-path=test-results
+    - UnifiedTestRunner --suite=editor --suite=playmode --editor-location=.Editor --testproject=test-project --enable-code-coverage coverage-upload-options="reportsDir:$PWD/test-results/CoverageResults;name:NGOv1_{{ platform.name }}_{{ editor }};flags:NGOv1_{{ platform.name }}_{{ editor }};verbose" --coverage-results-path=$PWD/test-results/CoverageResults --coverage-options="generateHtmlReport;generateAdditionalMetrics;assemblyFilters:+Unity.Netcode.Editor,+Unity.Netcode.Runtime" --extra-editor-arg=--burst-disable-compilation --timeout={ test_timeout }} --rerun-strategy=Test --retry={{ num_test_retries }} --clean-library-on-rerun --artifacts-path=test-results
   artifacts:
     logs:
       paths:
         - "test-results/**/*"
   dependencies:
     - .yamato/package-pack.yml#package_pack_-_ngo_ubuntu
 {% endfor -%}
-{% endfor -%}
+{% endfor -%}

.yamato/console-standalone-test.yml

@@ -4,22 +4,22 @@
 # DESCRIPTION--------------------------------------------------------------------------
   # This job is responsible for Console platform test validation.
   # Those tests cover both PlayMode and EditMode tests from package test assemblies.
-  
+
 # CONFIGURATION STRUCTURE--------------------------------------------------------------
   # Jobs are generated using nested loops (separate build phase and run phase). Worth noting that run phase uses the build as dependency:
     # 1. For all console platform (Switch, ps4, ps5, xbox360, xboxOne)
     # 2. For all supported Unity Editor versions (for NGOv1.X this means 2022.3+ editors)
     # 3. For the default project.
-  
+
 # TECHNICAL CONSIDERATIONS---------------------------------------------------------------
   # For console devices a split is required into two phases:
     # 1. Build Phase: Creates standalone players for console platforms
     # 2. Run Phase: Executes runtime tests on actual console devices
   # The Run phase uses build job as dependency
-  
+
   # Note: More of a Unity specific but test assemblies need to be included in the build phase command
   # Note: All builds can be made on x64 machines since those are compatible with ARM64 target devices
-  
+
 # PLATFORM SPECIFICS-----------------------------------------------------------------
   # Common Requirements:
     # All consoles require IL2CPP scripting backend
@@ -29,13 +29,13 @@
     # Switch: ARM64 architecture only
     # Other Consoles: x64 architecture
     # Each console requires specific SDK paths and tools
-  
+
 # QUALITY THOUGHTS--------------------------------------------------------------------
   # TODO: consider adding all projects that have tests
   # To see where this job is included (in trigger job definitions) look into _triggers.yml file
-  
-  
-  
+
+
+
 # BUILD PHASE CONFIGURATION------------------------------------------------------------------------------------
 {% for project in projects.default -%}
 {% for platform in test_platforms.console_build -%}
@@ -51,7 +51,7 @@ console_standalone_build_{{ project.name }}_{{ platform.name }}_{{ editor }}:
 {% endif %}
   commands:
     - unity-downloader-cli --fast --wait -u {{ editor }} -c Editor -c il2cpp -c {{ platform.name }}
-    - UnifiedTestRunner --testproject={{ project.path }} --architecture={% if platform.name == "switch" %}arm64{% else %}x64{% endif %} --scripting-backend=il2cpp --suite=playmode --platform={{ platform.standalone }} --editor-location=.Editor --artifacts-path=artifacts --player-save-path=build/players --testfilter="Unity.Netcode.RuntimeTests.*" --extra-editor-arg=-batchmode --extra-editor-arg=-nographics --reruncount=1 --clean-library-on-rerun --build-only --timeout=1800
+    - UnifiedTestRunner --testproject={{ project.path }} --architecture={% if platform.name == "switch" %}arm64{% else %}x64{% endif %} --scripting-backend=il2cpp --suite=playmode --platform={{ platform.standalone }} --editor-location=.Editor --artifacts-path=artifacts --player-save-path=build/players --testfilter="Unity.Netcode.RuntimeTests.*" --extra-editor-arg=-batchmode --extra-editor-arg=-nographics --reruncount=1 --clean-library-on-rerun --build-only --timeout={{ test_timeout}}
   variables:
 # PS4 related
     SCE_ORBIS_SDK_DIR: 'C:\Users\bokken\SCE\ps4_sdk_12_00'
@@ -72,9 +72,9 @@ console_standalone_build_{{ project.name }}_{{ platform.name }}_{{ editor }}:
 {% endfor -%}
 {% endfor -%}
 {% endfor -%}
-  
-  
-  
+
+
+
 # RUN PHASE CONFIGURATION------------------------------------------------------------------------------------
 {% for project in projects.default -%}
 {% for platform in test_platforms.console_test -%}
@@ -109,4 +109,4 @@ console_standalone_test_{{ project.name }}_{{ platform.name }}_{{ editor }}:
     - .yamato/console-standalone-test.yml#console_standalone_build_{{ project.name }}_{{ platform.name }}_{{ editor }}
 {% endfor -%}
 {% endfor -%}
-{% endfor -%}
+{% endfor -%}

.yamato/desktop-standalone-tests.yml

@@ -4,30 +4,30 @@
 # DESCRIPTION--------------------------------------------------------------------------
   # This job is responsible for Desktop platform test validation.
   # Those tests cover both PlayMode and EditMode tests from package test assemblies.
-  
+
 # CONFIGURATION STRUCTURE--------------------------------------------------------------
   # Jobs are generated using nested loops (separate build phase and run phase). Worth noting that run phase uses the build as dependency:
     # 1. For all desktop platform (Windows, macOS, Ubuntu)
     # 2. For all supported Unity Editor versions (for NGOv1.X this means 2022.3+ editors)
     # 3. For the default project.
     # 4. For all scripting backends (mono, il2cpp)
-  
+
 # TECHNICAL CONSIDERATIONS---------------------------------------------------------------
   # For desktop devices a split is into two phases is not required but we use it for consistency with setup of others standalone platforms:
     # 1. Build Phase: Creates standalone players for desktop platforms
     # 2. Run Phase: Executes runtime tests on actual desktop devices
   # The Run phase uses build job as dependency
-  
+
   # Note: More of a Unity specific but test assemblies need to be included in the build phase command
   # Note: All builds can be made on x64 machines since those are compatible with ARM64 target devices
-  
+
 # QUALITY THOUGHTS--------------------------------------------------------------------
   # TODO: consider adding all projects that have tests
   # To see where this job is included (in trigger job definitions) look into _triggers.yml file
-  
+
 #-----------------------------------------------------------------------------------
-  
-  
+
+
 # BUILD PHASE CONFIGURATION------------------------------------------------------------------------------------
 {% for project in projects.default -%}
 {% for platform in test_platforms.desktop -%}
@@ -53,15 +53,16 @@ desktop_standalone_build_{{ project.name }}_{{ platform.name }}_{{ backend }}_{{
       paths:
         - "artifacts/**/*"
   dependencies:
-    - .yamato/project-pack.yml#project_pack_-_{{ project.name }}_{{ platform.name }}
+    - .yamato/_run-all.yml#run_quick_checks # initial checks to perform fast validation of common errors
+    - .yamato/package-pack.yml#package_pack_-_ngo_{{ platform.name }}
 {% endfor -%}
 {% endfor -%}
 {% endfor -%}
 {% endfor -%}
-  
-  
-  
-  
+
+
+
+
 # RUN PHASE CONFIGURATION------------------------------------------------------------------------------------
 {% for project in projects.default -%}
 {% for platform in test_platforms.desktop -%}
@@ -78,7 +79,7 @@ desktop_standalone_test_{{ project.name }}_{{ platform.name }}_{{ backend }}_{{
 {% endif %}
   commands:
     - unity-downloader-cli --fast --wait -u {{ editor }} -c Editor {% if backend == "il2cpp" %} -c il2cpp {% endif %}
-    - UnifiedTestRunner --suite=playmode --player-load-path=build/players --artifacts-path=test-results --testproject={{ project.path }} --editor-location=.Editor --playergraphicsapi=Null --fail-on-assert --reruncount=1 --clean-library-on-rerun --timeout=1800
+    - UnifiedTestRunner --suite=playmode --player-load-path=build/players --artifacts-path=test-results --testproject={{ project.path }} --editor-location=.Editor --playergraphicsapi=Null --fail-on-assert --rerun-strategy=Test --retry={{ num_test_retries }} --clean-library-on-rerun --timeout={{ test_timeout }}
   artifacts:
     logs:
       paths: