Full multi-arch support

[aszady]

PR #151 introduced initial support for resolving dependencies across multiple architectures for resolve or lockfile commands.
However, including the same package for multiple architectures within a single resolution context remains problematic. Since this is a common requirement in cross-compilation use cases, I suggest extending multi-arch resolution to fully support such scenarios.

Looking forward to your feedback!

Proposed changes

• Packages need to be identified by some combination of name and arch, instead of just by name. The bazeldnf code will distinguish between foo.x86_64 and foo.i686 as separate packages.

• The alias repository needs to provide access to multiple architectures of a package. This behaviour should be customizable with a client-provided macro, as it's hard to derive a one-size-fits-all solution. The default implementation could be, for example:

  • @bazeldnf_rpms//foo:foo - points to the primary/default architecture
  • @bazeldnf_rpms//foo:foo.x86_64 - points to the x86_64 version
  • @bazeldnf_rpms//foo:foo.i686 - points to the i686 version

• SAT loaders / constraint updates:

  1. Group packages by (name, arch) instead of just name for the purposes of "best" version, so that the choice is fully independent between distinct architectures of the same package. This also gives possibility for foo.i686 to participate in dependency resolution even if foo.x86_64 is the best for foo.

  2. Relax provider uniqueness constraints – multiple packages may provide the same capability (e.g. if they have different architectures). In a multi-arch setting this is essential, as packages are often required just by their name or by library name, which is shared between multiple architectures. This also more closely matches the behaviour of dnf, where we could have e.g. multiple providers of webserver. (Related: SAT solver: Relax unique package provider clause for a package requires #142).

  3. Because of the above, we need to add all providers as dependencies in the lock file – if the requirement is satisfied by more than one package. Let's just list them all and let the client make a choice. This can be perceived as matching the system environment more closely, where any package has access to the full filesystem anyway, which includes all possible requirement alternatives to its requirement.

• Lockfile format extensions:

  • id field containing a unique, opaque identifier for each package. Useful for constructing repository names and mapping dependencies within the lockfile. So far, only the package name was used for these purposes.
  • arch field containing the package architecture, which can be used later in the alias repository for custom logic.

Backwards Compatibility

Existing single-architecture workflows will continue to work unchanged. Old lockfiles will be supported. With default settings, target references in the alias repository will remain compatible for single-architecture cases.

[manuelnaranjo]

Why not having multiple lock files one per arch and select one rpmtree or the other based on platforms on the user side and not on the rules side? Each lock file would create a different proxy repo.
Even in cross compilation scenarios you will get binaries that run on a particular platform (Bazel call this the host platform) and produces binaries for a certain platform (target platform in Bazel terms).
Bazeldnf can't know that cross-gcc-armb64 will run on amd64 Linux and output binaries for arm64. The user encapsulating the tools would be the one knowing this.
IMO attempting for multiplatform resolution is a recipe to make things harder than what they need to be, when the solution can be implemented in the client side with the current state of affairs.

[manuelnaranjo]

@kellyma2 @rmohr maybe you folks want to chime in?

[kellyma2]

The use case for this is for i386/x86_64 multiarch scenarios where there's a possiblity that some packages may have dependencies on the other arch. I don't think that having multiple lockfiles will enable this. For example, in our situation we have a 64-bit GCC but we're using it to produce i386 binaries and it needs to use 32-bit libc, etc to this end.

[manuelnaranjo]

Ohh so not cross compilation but dual arch setups? I thought i686 support was finally ditched from the kernel and some other projects, but I think something similar may apply to arm64/32

[kellyma2]

Correct, for dual arch. I don't know if this is common for arm64/32, but definitely an option for x86.

Distro support for pure 32-bit is largely being vapourized, but i think there's still reasonable support for 32-bit on 64-bit systems (at least for x86)

[aszady]

The particular example that won't work for multiple lock files approach is the following:

0. exec is x86_64 (compatible with i686); target is i686.
1. There's foo.i686 build tool, distributed only for i686; it depends on bar (architecture not specified, e.g. just a binary to run).
2. There's bar.x86_64 build tool, distributed only for x86_64.
3. They share some common transitive dependency – usually it's often sth like glibc – foo.i686 requiring effectively glibc.i686 and bar.x86_64 requiring glibc.x86_64.

Without (3), the work in #151 is enough.
But with (3), we are forced to have a single package in multiple architectures installed – with no possibility to split that into two resolution contexts (multiple locks).