Build an aggregated repository containing all direct-from-maven bundles as well as all Orbit bundles

[merks]

Oomph has been building a repository that is computed by composing the direct-from-maven dependencies of all the target platforms of all the projects participating in SimRel that define such dependencies, updated to their latest available version from Maven central:

https://download.eclipse.org/oomph/simrel-maven

The details are documented here:

https://github.com/merks/simrel-maven#readme

Now Oomph is building a repository that is the aggregation of these direct-from-maven dependencies along with the contents of an Orbit repository:

https://download.eclipse.org/oomph/simrel-orbit/

The aggregation is currently defined like this:

The reason for the inclusion of org.apache.commons.httpclient from https://download.eclipse.org/tools/orbit/downloads/drops/R20201118194144/repository is because otherwise the aggregation fails like this:

Cannot complete the install because one or more required items could not be found. Software being installed: validationSet_orbit 1.0.0 Missing requirement: org.apache.xmlrpc.client 3.1.3.v20160817-1930 requires 'java.package; org.apache.commons.httpclient 3.0.1' but it could not be found Cannot satisfy dependency: mappedRepo_download.eclipse.org_tools_orbit_downloads_latest-R 1.0.0 depends on: org.eclipse.equinox.p2.iu; org.eclipse.orbit.releng.recipes.feature.aggregation.feature.group 0.0.0 Cannot satisfy dependency: org.eclipse.orbit.releng.recipes.feature.aggregation.feature.group 1.0.0.v20230210-1433 depends on: org.eclipse.equinox.p2.iu; org.apache.xmlrpc.client [3.1.3.v20160817-1930,3.1.3.v20160817-1930] Cannot satisfy dependency: validationSet_orbit 1.0.0 depends on: org.eclipse.equinox.p2.iu; mappedRepo_download.eclipse.org_tools_orbit_downloads_latest-R [1.0.0,1.0.0]
Missing requirement: Apache XML-RPC Client Library 3.1.3.v20160817-1930 (org.apache.xmlrpc.client 3.1.3.v20160817-1930) requires 'java.package; org.apache.commons.httpclient 3.0.1' but it could not be found

JavaPackage(org.apache.commons.httpclient 3.0.1) is required by:
ValidationSet(orbit)
Contribution(orbit)
MappedRepository(https://download.eclipse.org/tools/orbit/downloads/latest-R)
InstallableUnit(org.eclipse.orbit.releng.recipes.feature.aggregation.feature.group 1.0.0.v20230210-1433)
InstallableUnit(org.apache.xmlrpc.client 3.1.3.v20160817-1930)

Of course we could just make that repository a validation repository, but I don't think it makes sense to build a repository where some of the IUs can't actually be installed because of missing requirements. (The direct-from-maven repository is separately tested for such requirement completeness.)

The https://download.eclipse.org/eclipse/updates/4.27/R-4.27-202303020300 repository is added as validation repository because some IUs explicitly require org.eclipse.osgi but they would likely be better of with package imports of the org.osgi packages they are actually using, e.g.,

Cannot complete the install because one or more required items could not be found. Software being installed: validationSet_orbit 1.0.0 Missing requirement: javax.xml 1.3.4.v201005080400 requires 'osgi.bundle; org.eclipse.osgi 0.0.0' but it could not be found Missing requirement: javax.xml 1.4.1.v20220503-2331 requires 'osgi.bundle; org.eclipse.osgi 0.0.0' but it could not be found Cannot satisfy dependency: jakarta.xml.bind 2.3.3.v20221203-1659 depends on: java.package; javax.xml.datatype 1.3.0 Cannot satisfy dependency: mappedRepo_download.eclipse.org_tools_orbit_downloads_latest-R 1.0.0 depends on: org.eclipse.equinox.p2.iu; org.eclipse.orbit.releng.recipes.feature.aggregation.feature.group 0.0.0 Cannot satisfy dependency: org.apache.santuario.xmlsec 2.2.0.v20221112-0806 depends on: java.package; javax.xml.bind [2.3.0,3.0.0) Cannot satisfy dependency: org.eclipse.orbit.releng.recipes.feature.aggregation.feature.group 1.0.0.v20230210-1433 depends on: org.eclipse.equinox.p2.iu; org.apache.santuario.xmlsec [2.2.0.v20221112-0806,2.2.0.v20221112-0806] Cannot satisfy dependency: validationSet_orbit 1.0.0 depends on: org.eclipse.equinox.p2.iu; mappedRepo_download.eclipse.org_tools_orbit_downloads_latest-R [1.0.0,1.0.0]
Missing requirement: JAXP XML 1.3.4.v201005080400 (javax.xml 1.3.4.v201005080400) requires 'osgi.bundle; org.eclipse.osgi 0.0.0' but it could not be found
Bundle(org.eclipse.osgi) is required by:
ValidationSet(orbit)
Contribution(orbit)
MappedRepository(https://download.eclipse.org/tools/orbit/downloads/latest-R)
InstallableUnit(org.eclipse.orbit.releng.recipes.feature.aggregation.feature.group 1.0.0.v20230210-1433)
InstallableUnit(org.apache.santuario.xmlsec 2.2.0.v20221112-0806)
InstallableUnit(jakarta.xml.bind 2.3.3.v20221203-1659)
InstallableUnit(javax.xml 1.3.4.v201005080400)
Missing requirement: XML Commons External Components XML APIs 1.4.1.v20220503-2331 (javax.xml 1.4.1.v20220503-2331) requires 'osgi.bundle; org.eclipse.osgi 0.0.0' but it could not be found

Bundle(org.eclipse.osgi) is required by:
ValidationSet(orbit)
Contribution(orbit)
MappedRepository(https://download.eclipse.org/tools/orbit/downloads/latest-R)
InstallableUnit(org.eclipse.orbit.releng.recipes.feature.aggregation.feature.group 1.0.0.v20230210-1433)
InstallableUnit(org.apache.santuario.xmlsec 2.2.0.v20221112-0806)
InstallableUnit(jakarta.xml.bind 2.3.3.v20221203-1659)
InstallableUnit(javax.xml 1.4.1.v20220503-2331)

The build generates reports and we can see that there are a great many duplicates:

We should consider whether it makes sense for Orbit to continue to provide bundles for which there are also direct-from-maven versions because of course an important SimRel goal is to reduce or eliminate the number of duplicates. We saw during the last release cycle all the problems around gson versions, so duplicates can and do cause problems even for bundles that are not singletons.

[merks]

@jonahgraham

Perhaps we can find time soon to chat about the best approach to move forward on this prototype work which is maintained here:

https://git.eclipse.org/c/oomph/org.eclipse.oomph.incubator.git/tree/orbit

[jonahgraham]

I'll send you an email to set up a time.

[merks]

@jonahgraham

In follow up to our discussions, you are correct that adding the older orbit repo to the aggregation pulls in a number of additional things, not only the one specified IU (though that prevents pulling in all the content). I split the one contribution into three so that I can easily see what's coming from the older orbit repo:

I will see which of those I can hunt down from Maven. Here are some that I already found:

• https://repo1.maven.org/maven2/jakarta/xml/bind/jakarta.xml.bind-api/
• https://repo1.maven.org/maven2/jakarta/activation/jakarta.activation-api/

[merks]

@jonahgraham

It seems a lost cause to try to eliminate these. The dependencies on them typically are very strict, or they are not available at Maven at all. Moreover, almost all of these are ending up in SimRel so someone is still getting them from older Orbit repositories. Even the platform is doing that:

https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/6e88b23d32af67acbe0cc2e7d73c224c83de1e48/eclipse.platform.releng.prereqs.sdk/eclipse-sdk-prereqs.target#L51-L80

So I think it would be good to include these in the aggregated result. Are you okay with that?

[jonahgraham]

Are you okay with that?

Yes I am.

[cdietrich]

@merks will there be a release specific stable url for this like we have for orbit.
e.g. https://download.eclipse.org/oomph/simrel-maven/2023-06
or is the plan to include this repo in the orbit composite repo https://download.eclipse.org/tools/orbit/downloads/2023-06

[merks]

@cdietrich

Yes, these are published using the same JustJ-provided infrastructure as is used by EMF. So when there is a release, the most recent milestone is mirrored to the final release location. So just as EMF has these:

• https://download.eclipse.org/modeling/emf/emf/builds/release/2.33.0
• https://download.eclipse.org/modeling/emf/emf/builds/release/latest

Eventually these two will exist:

• https://download.eclipse.org/oomph/simrel-orbit/release/4.28.0
• https://download.eclipse.org/oomph/simrel-orbit/release/latest

Exactly the same applies for these repositories:

https://download.eclipse.org/oomph/simrel-maven/

Of course the release repositories are permanently retained.

I follow the Platform's version numbering. The correspondence is well documented:

https://wiki.eclipse.org/Simultaneous_Release

[cdietrich]

@merks the point is about orbit:
they exist with M1 or even before
so i dont have for to wait for the release to e.g.
consume them in targets and in our wizard

same for https://download.eclipse.org/releases/2023-06/

[merks]

For background, is the note that I posted:

As most of you know, Orbit provides regular builds scheduled to coordinate with the SimRel schedule. For 2023-06 M1, Jonah produced this build:

https://download.eclipse.org/tools/orbit/downloads/drops/S20230403203110/index.html

and this specific p2 repository:

https://download.eclipse.org/tools/orbit/downloads/drops/S20230403203110/repository

This repository composes the latest such stable build:

https://download.eclipse.org/tools/orbit/downloads/latest-S/

It is currently planned to reduce what's available via Orbit. In particular, we plan to stop adding new versions of libraries to Orbit for those libraries available at Maven Central with proper OSGi metadata. For such libraries one can use the m2e location in your project's *.target to fetch those libraries (and ensure that you PGP sign them), or use what's described below.

For SimRel I have analyzed those contributing projects that are pulling libraries directly from Maven and have implemented dependabot-like tools to help keep those projects up-to-date with the latest version as described here:

https://github.com/merks/simrel-maven/blob/main/README.md

I've been helping the Platform directly with this activity; WWD and tm4e ought to have a look at the reports generated for them...

In any case, starting for 2023-06 M1, I will produce regular milestone builds (and eventually release/recommend builds) of all the direct-from-maven libraries being contributed by SimRel projects, updated to their latest available minor version:

https://download.eclipse.org/oomph/simrel-maven/milestone/S202304041507

This will always reference the latest such milestone:

https://download.eclipse.org/oomph/simrel-maven/milestone/latest

In fact, there are a few additional libraries added because the build infrastructure tests that each library's transitive requirements are satisfied by other libraries in the repository. Going forward, we plan to add libraries to this repository to replace the older version in Orbit and also removing the library from Orbit. Of course these libraries are all PGP signed.

To complement the Orbit repository and the direct-from-maven repository, I will also be producing regular milestone (and eventually release/recommend builds) of the aggregation of those two repositories, e.g., this for M1:

https://download.eclipse.org/oomph/simrel-orbit/milestone/S202304041533

This repository composes the latest such stable build:

https://download.eclipse.org/oomph/simrel-orbit/milestone/latest

While building that repository (using the CBI p2 aggregator), I noticed that a significant number of the bundles in the latest Orbit stable build depend on bundles that are only available in older Orbit builds. So the aggregation also pulls in any transitively-required bundles from the final older (CVS-driven) Orbit recommended (release) repository. As such, the simrel-orbit repository is transitively complete with respect to requirements.

The simrel-orbit repository is intended to provide one-stop-shopping to cover all your Orbit requirements and to help coordinate the versions of all your direct-from-maven requirements. Please try to use the latest versions available here, and look at your reports to avoid duplication and to avoid problems like the highly-problematic multiple-gsons problem we had during the previous release cycle.

If you have questions, please read the details in the following issue and ask the questions on that issue to avoid spamming this mailing list:

#3

(Sorry, Oomph is not yet migrated to Github, so I keep this in my account repo for now.)

[merks]

@merks the point is about orbit: they exist with M1 or even before so i dont have for to wait for the release to e.g. consume them in targets and in our wizard

I'm really not sure the exact point. Yes, Orbit provides a recommend repository for the 2023-03 release but not yet for the 2023-06 release. Orbit can do that because it's gone through a release cycle, but these new repositories have not gone through a release cycle.

As for consumption in your targets, you ought to be using https://download.eclipse.org/tools/orbit/downloads/latest-S/ so that you coordinate on the versions for the current release cycle, not on the versions used during the previous release cycle. So it concerns me that you appear to only use releases and furthermore that you expect that approach to be self evident.

In any case, https://download.eclipse.org/tools/orbit/downloads/latest-S contains exactly the same bundles as https://download.eclipse.org/tools/orbit/downloads/latest-R currently, so https://download.eclipse.org/oomph/simrel-orbit/milestone/latest also contains exactly the same bundles as you are already using. An this repository will always exist and will always point at the version you ought to be using in combination with your SimRel contributions...

[cdietrich]

@merks there is https://download.eclipse.org/tools/orbit/downloads/2023-06 already available as is https://download.eclipse.org/releases/2023-06/

the point is e.g. for wizard we need a stable url accross releases
https://download.eclipse.org/tools/orbit/downloads/latest-S/
is not

[ruspl-afed]

First, thank you for your effort @merks

I have a question.
How can we notify a SimRel participant project regarding the 3rd party library changes in upstream to react in time?

For example Platform, CDT, Passage and may be others use com.sun.jna* and it would be great if they use the same version, but the process of synchronization is unclear. SLF4J could be an even more intriguing example.

[merks]

@cdietrich

Which wizard?

I could provide another location that composes what's there now, but I'd have to develop some automated way to maintaining that and it's not entirely clearly to me why/where you need a stable URL in advance.

@ruspl-afed

Yes, folks can be notified but one has to figure out if that's some Github issue, or some bugzilla and then hope there is really someone listening on the other end. Or we can assume that folks are not entirely passive and read the cross-projects notifications and act accordingly. There are even reports generated for these repos:

https://download.eclipse.org/oomph/simrel-orbit/milestone/S202304041533/archive/download.eclipse.org/oomph/simrel-orbit/milestone/S202304041533/index.html

We can see that com.sun.jna is very popular:

In the end, I think the process is very clear. Everyone use the latest version.