Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions docs/workflow/building/coreclr/cross-building.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
* [Cross-Building using Docker](#cross-building-using-docker)
* [Cross-Compiling for ARM32 and ARM64 with Docker](#cross-compiling-for-arm32-and-arm64-with-docker)
* [Cross-Compiling for FreeBSD with Docker](#cross-compiling-for-freebsd-with-docker)
* [Building CoreCLR with Bootstrapping](#building-coreclr-with-bootstrapping)
* [Building Tests with Bootstrapping](#building-tests-with-bootstrapping)

This guide will go more in-depth on how to do cross-building across multiple operating systems and architectures. It's worth mentioning this is not an any-to-any scenario. Only the combinations explained here are possible/supported. If/When any other combinations get supported/discovered, this document will get updated accordingly.

Expand Down Expand Up @@ -183,3 +185,100 @@ To build the bootstrap subset of the runtime repo, you can build the `bootstrap`
For simplicity, a `--bootstrap` option is also provided. This option will build the `bootstrap` subset, clean up the artifacts directory, and then build the runtime repo with the `--use-bootstrap` option. This is useful for building the runtime repo with the live NativeAOT version without having to run two separate commands.

The `--bootstrap` option is automatically specified when building the runtime repo for .NET Source Build, as the vast majority of Source Build scenarios use non-portable RIDs.

### Building Tests with Bootstrapping

For community-supported platforms where Microsoft does not publish targeting/runtime/apphost packs (e.g. FreeBSD, illumos, Haiku, linux-riscv64, linux-loongarch64, linux-ppc64le), the test builds need to consume the locally-built bootstrap artifacts instead of trying to download packs from NuGet. Both CoreCLR runtime tests and libraries tests support this via the `--use-bootstrap` flag.

The steps below use FreeBSD ARM64 as an example. The same flow applies to any other community-supported target by swapping `--os`/`--arch` and the docker image.

#### Building in the Container

Start the cross-build container:

```bash
docker run --rm -it \
-v $(pwd):/runtime \
-w /runtime \
-e ROOTFS_DIR=/crossrootfs/arm64 \
mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-net11.0-cross-freebsd-14-arm64 \
bash
```

Inside the container, set the target and build the product, then the tests:

```bash
os=freebsd
arch=arm64

# Initial product build (also produces the bootstrap subset).
./build.sh clr+libs --cross --arch $arch --os $os --bootstrap

# Subsequent rebuilds (e.g. after editing code) reuse the existing bootstrap.
./build.sh clr+libs --cross --arch $arch --os $os --use-bootstrap

# CoreCLR runtime tests.
src/tests/build.sh -cross -$arch -$os -p:LibrariesConfiguration=Debug --use-bootstrap

# Libraries tests (produces zipped per-library test archives under artifacts/helix/tests/).
./build.sh libs.tests --cross --arch $arch --os $os --use-bootstrap -p:ArchiveTests=true
```

Without `--use-bootstrap`, restore would fail with `NU1102` errors because the apphost/runtime/targeting packs for `freebsd-arm64` are not published to NuGet feeds.

#### Running CoreCLR Runtime Tests on the Target

From the host machine, pack and upload `artifacts/tests/coreclr/<os>.<arch>.Debug` to the target:

```bash
tar -czf coreclr-tests-freebsd-arm64-Debug.tar.gz artifacts/tests/coreclr/freebsd.arm64.Debug
scp coreclr-tests-freebsd-arm64-Debug.tar.gz $TargetMachine:/tmp
```

On the target machine, extract and run a test. For example, the interpreter tests:

```bash
ssh $TargetMachine

mkdir coreclr-tests && cd $_
tar -xzf /tmp/coreclr-tests-freebsd-arm64-Debug.tar.gz

CORE_ROOT=$(pwd)/artifacts/tests/coreclr/freebsd.arm64.Debug/Tests/Core_Root \
DOTNET_TieredCompilation=0 \
artifacts/tests/coreclr/freebsd.arm64.Debug/JIT/Interpreter/InterpreterTester/InterpreterTester.sh
```

Exit code `100` means pass; any other value means fail.

#### Running Libraries Tests on the Target

Libraries tests need the test host (`artifacts/bin/testhost/...`) plus the per-library test archive. From the host machine:

```bash
# Pack and upload the test host.
tar -czf testhost_net11.0-freebsd-Debug-arm64.tar.gz artifacts/bin/testhost/net11.0-freebsd-Debug-arm64
scp testhost_net11.0-freebsd-Debug-arm64.tar.gz $TargetMachine:/tmp

# Copy a specific test's archive (paths and names vary; this example uses System.Text.RegularExpressions).
scp artifacts/helix/tests/freebsd.AnyCPU.Debug/System.Text.RegularExpressions.Unit.Tests.zip $TargetMachine:/tmp
```

On the target machine:

```bash
ssh $TargetMachine

mkdir testhost
tar -xzf /tmp/testhost_net11.0-freebsd-Debug-arm64.tar.gz -C testhost

mkdir regex-tests && cd $_
unzip /tmp/System.Text.RegularExpressions.Unit.Tests.zip

./RunTests.sh --runtime-path ~/testhost/artifacts/bin/testhost/net11.0-freebsd-Debug-arm64
```

#### Notes

* `--use-bootstrap` works on any platform; on community-supported platforms it is the only way to build the tests because the targeting/runtime/apphost packs are not published to NuGet feeds.
* If restore still fails after passing the flag, double-check that the `bootstrap` subset built successfully and produced `artifacts/bin/bootstrap/`.
* Building/packing an individual library test project (e.g. `./dotnet.sh build src/libraries/System.Formats.Nrbf/tests -p:UseBootstrap=true -p:ArchiveTests=true ...`) currently produces an unusable archive (missing `xunit.console.runtimeconfig.json` and similar). Until that is fixed, build the whole `libs.tests` subset as shown above.
2 changes: 1 addition & 1 deletion eng/targetingpacks.targets
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@
ExcludedRuntimeIdentifiers="android"
AppHostPackNamePattern="$(LocalFrameworkOverrideName).Host.**RID**"
AppHostPackVersion="$(ProductVersion)"
AppHostRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86;linux-ppc64le;linux-riscv64;linux-musl-riscv64;linux-loongarch64;linux-musl-loongarch64"
AppHostRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86;freebsd-x64;freebsd-arm64;openbsd-x64;openbsd-arm64;illumos-x64;solaris-x64;haiku-x64;linux-ppc64le;linux-riscv64;linux-musl-riscv64;linux-loongarch64;linux-musl-loongarch64"
TargetFramework="$(NetCoreAppCurrent)"
Condition="'$(UseLocalAppHostPack)' == 'true' and '@(KnownAppHostPack->AnyHaveMetadataValue('TargetFramework', '$(NetCoreAppCurrent)'))' != 'true'" />
</ItemGroup>
Expand Down
4 changes: 4 additions & 0 deletions src/libraries/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@

<Import Project="..\..\Directory.Build.props" />

<PropertyGroup Condition="'$(UseBootstrap)' == 'true' and ('$(IsTestProject)' == 'true' or '$(IsTestSupportProject)' == 'true' or '$(IsTrimmingTestProject)' == 'true')">
<UseBootstrapLayout>true</UseBootstrapLayout>
</PropertyGroup>

<PropertyGroup>
<BeforeTargetFrameworkInferenceTargets>$(RepositoryEngineeringDir)BeforeTargetFrameworkInference.targets</BeforeTargetFrameworkInferenceTargets>
<GeneratePlatformNotSupportedAssemblyHeaderFile>$(RepositoryEngineeringDir)LicenseHeader.txt</GeneratePlatformNotSupportedAssemblyHeaderFile>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -124,14 +124,8 @@
GetUnixCrossBuildArgumentsForDNNE"
BeforeTargets="DnneBuildNativeExports">
<PropertyGroup>
<DnneNetHostDir>$([System.IO.Path]::GetDirectoryName('$(AppHostSourcePath)'))</DnneNetHostDir>
</PropertyGroup>
</Target>

<Target Name="SetRidGraphPath"
BeforeTargets="ProcessFrameworkReferences">
<PropertyGroup>
<RuntimeIdentifierGraphPath Condition="'$(UseBootstrap)' == 'true'">$([MSBuild]::NormalizePath('$(BootstrapRidGraphDir)', 'runtime.json'))</RuntimeIdentifierGraphPath>
<DnneNetHostDir Condition="'$(UseBootstrap)' == 'true'">$(DotNetHostBinDir.TrimEnd('/\'))</DnneNetHostDir>
<DnneNetHostDir Condition="'$(UseBootstrap)' != 'true'">$([System.IO.Path]::GetDirectoryName('$(AppHostSourcePath)'))</DnneNetHostDir>
</PropertyGroup>
</Target>
</Project>
2 changes: 1 addition & 1 deletion src/libraries/tests.proj
Original file line number Diff line number Diff line change
Expand Up @@ -820,7 +820,7 @@
<ProjectReference
Include="$(RepoRoot)src\tests\Common\XUnitLogChecker\XUnitLogChecker.csproj"
Targets="Publish"
AdditionalProperties="%(AdditionalProperties);_IsPublishing=true;Configuration=Release;PublishDir=$(XUnitLogCheckerLibrariesOutDir);ROOTFS_DIR=$(ROOTFS_DIR);RuntimeIdentifier=$(TargetRid)" />
AdditionalProperties="%(AdditionalProperties);_IsPublishing=true;Configuration=Release;CoreCLRConfiguration=$(CoreCLRConfiguration);LibrariesConfiguration=$(LibrariesConfiguration);TasksConfiguration=$(TasksConfiguration);ToolsConfiguration=$(ToolsConfiguration);PublishDir=$(XUnitLogCheckerLibrariesOutDir);ROOTFS_DIR=$(ROOTFS_DIR);RuntimeIdentifier=$(TargetRid);UseBootstrap=$(UseBootstrap)" />
</ItemGroup>

<Target Name="GenerateMergedCoverageReport"
Expand Down
2 changes: 2 additions & 0 deletions src/tests/Common/Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@

<Import Project="$(RepoRoot)eng/liveBuilds.targets" />
<Import Project="$(RepositoryEngineeringDir)toolAot.targets" />
<Import Project="$(RepositoryEngineeringDir)targetingpacks.targets" Condition="'$(UseBootstrap)' == 'true'" />
<Import Project="$(RepoRoot)eng/liveILLink.targets" Condition="'$(UseBootstrap)' == 'true'" />

<PropertyGroup>
<GCStressDependsOnCoreDisTools>false</GCStressDependsOnCoreDisTools>
Expand Down
4 changes: 4 additions & 0 deletions src/tests/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@
<Import Project="$(MSBuildThisFileDirectory)\Common\dir.sdkbuild.props" Condition="'$(UsingMicrosoftNETSdk)' == 'true'" />
<Import Project="$(MSBuildThisFileDirectory)\Common\dir.common.props" Condition="'$(UsingMicrosoftNETSdk)' != 'true'" />

<PropertyGroup Condition="'$(UseBootstrap)' == 'true'">
<UseBootstrapLayout>true</UseBootstrapLayout>
</PropertyGroup>

<PropertyGroup>
<RunningOnUnix Condition="('$(RunningOnUnix)' == '') And ('$(MSBuildRuntimeType)' == 'Core') And ('$(OS)'!='Windows_NT')">true</RunningOnUnix>
</PropertyGroup>
Expand Down
1 change: 1 addition & 0 deletions src/tests/Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -517,6 +517,7 @@
<Import Project="$(RepoRoot)/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.props" />
<Import Project="$(RepoRoot)eng/liveBuilds.targets" />
<Import Project="$(RepositoryEngineeringDir)toolAot.targets" />
<Import Project="$(RepositoryEngineeringDir)targetingpacks.targets" Condition="'$(UseBootstrap)' == 'true'" />
<Import Project="$(MSBuildProjectFullPath).targets" Condition="Exists('$(MSBuildProjectFullPath).targets')"/>
<Import Project="$(RepoRoot)/src/tests/Common/mergedrunner.targets" Condition="'$(IsMergedTestRunnerAssembly)' == 'true'" />

Expand Down
6 changes: 4 additions & 2 deletions src/tests/build.proj
Original file line number Diff line number Diff line change
Expand Up @@ -173,8 +173,9 @@

<PropertyGroup>
<_RestoreProperties>MSBuildRestoreSessionId=$([System.Guid]::NewGuid());TargetOS=$(TargetOS);TargetArchitecture=$(TargetArchitecture);Configuration=$(Configuration);CrossBuild=$(CrossBuild);SetTFMForRestore=true;RuntimeIdentifier=$(RuntimeIdentifier)</_RestoreProperties>
<_RestoreProperties Condition="'$(UseLocalAppHostPack)' == 'true' or '$(TargetOS)' == 'browser'">$(_RestoreProperties);EnableAppHostPackDownload=false;EnableTargetingPackDownload=false;EnableRuntimePackDownload=false</_RestoreProperties>
<_RestoreProperties Condition="'$(TargetOS)' == 'browser'">$(_RestoreProperties);EnableAppHostPackDownload=false;EnableTargetingPackDownload=false;EnableRuntimePackDownload=false</_RestoreProperties>
<_RestoreProperties Condition="'$(TargetsAppleMobile)' == 'true'">$(_RestoreProperties);RestoreUseStaticGraphEvaluation=false</_RestoreProperties>
<_RestoreProperties Condition="'$(UseBootstrap)' == 'true'">$(_RestoreProperties);UseBootstrap=true</_RestoreProperties>
</PropertyGroup>

<!-- Use a dedicated traversal project to restore all test dependencies in one graph invocation. -->
Expand Down Expand Up @@ -231,6 +232,7 @@
<GroupBuildCmd>$(GroupBuildCmd) "/p:PackageOS=$(PackageOS)"</GroupBuildCmd>
<GroupBuildCmd>$(GroupBuildCmd) "/p:RuntimeFlavor=$(RuntimeFlavor)"</GroupBuildCmd>
<GroupBuildCmd>$(GroupBuildCmd) "/p:RuntimeVariant=$(RuntimeVariant)"</GroupBuildCmd>
<GroupBuildCmd Condition="'$(UseBootstrap)' == 'true'">$(GroupBuildCmd) "/p:UseBootstrap=true"</GroupBuildCmd>
<GroupBuildCmd Condition="'$(ServerGarbageCollection)' != ''">$(GroupBuildCmd) "/p:ServerGarbageCollection=$(ServerGarbageCollection)"</GroupBuildCmd>
<GroupBuildCmd>$(GroupBuildCmd) "/p:CLRTestBuildAllTargets=$(CLRTestBuildAllTargets)"</GroupBuildCmd>
<GroupBuildCmd>$(GroupBuildCmd) "/p:UseCodeFlowEnforcement=$(UseCodeFlowEnforcement)"</GroupBuildCmd>
Expand Down Expand Up @@ -272,7 +274,7 @@
<MSBuild
Projects="Common\XUnitLogChecker\XUnitLogChecker.csproj"
Targets="Publish"
Properties="_IsPublishing=true;Configuration=Release;PublishDir=$(XunitTestBinBase)/XUnitLogChecker;RuntimeIdentifier=$(RuntimeIdentifier)"
Properties="_IsPublishing=true;Configuration=Release;CoreCLRConfiguration=$(CoreCLRConfiguration);LibrariesConfiguration=$(LibrariesConfiguration);TasksConfiguration=$(TasksConfiguration);ToolsConfiguration=$(ToolsConfiguration);PublishDir=$(XunitTestBinBase)/XUnitLogChecker;RuntimeIdentifier=$(RuntimeIdentifier);UseBootstrap=$(UseBootstrap)"
Condition="$(IsXUnitLogCheckerSupported)" />
</Target>

Expand Down
5 changes: 5 additions & 0 deletions src/tests/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@ usage_list+=("-nativeaot - Builds the tests for Native AOT compilation.")
usage_list+=("-priority1 - Include priority=1 tests in the build.")
usage_list+=("-perfmap - Emit perfmap symbol files when compiling the framework assemblies using Crossgen2.")
usage_list+=("-allTargets - Build managed tests for all target platforms (including test projects in which CLRTestTargetUnsupported resolves to true).")
usage_list+=("-use-bootstrap - Use artifacts produced by the bootstrap subset for local targeting, runtime, and apphost packs.")
usage_list+=("")
usage_list+=("-runtests - Run tests after building them.")
usage_list+=("-mono, -excludemonofailures - Build the tests for the Mono runtime honoring mono-specific issues.")
Expand Down Expand Up @@ -225,6 +226,10 @@ handle_arguments_local() {
__UnprocessedBuildArgs+=("/p:CLRTestBuildAllTargets=allTargets")
;;

use-bootstrap|-use-bootstrap)
__UnprocessedBuildArgs+=("/p:UseBootstrap=true")
;;

rebuild|-rebuild)
__RebuildTests=1
;;
Expand Down