How to contribute artifacts directly?

[HannesWell]

For eclipse-xtext/xtext#2056 I would like to add com.google.inject:guice:5.1.0 to the simrel-orbit, but no other project that uses maven-targets contributes it.

Can add it directly here? My first guess was to add it to the original.target in the suplement folder, but I'm not sure about that.
Can I just create a PR in this repo to add it?

[merks]

Some investigation is required...

It fails like this:

[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.eclipse.oomph.maven.site raw:1.0.0.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):1.0.0-SNAPSHOT
[ERROR]   Missing requirement: com.google.inject 5.1.0 requires 'java.package; com.google.common.base [30.1.0,31.0.0)' but it could not be found
[ERROR]   Cannot satisfy dependency: org.eclipse.oomph.maven.all.feature.group 4.28.0.qualifier depends on: org.eclipse.equinox.p2.iu; com.google.inject [5.1.0,5.1.0]
[ERROR]   Cannot satisfy dependency: org.eclipse.oomph.maven.site raw:1.0.0.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):1.0.0-SNAPSHOT depends on: org.eclipse.equinox.p2.iu; org.eclipse.oomph.maven.all.feature.group 0.0.0

I'd expect that to come from this which is present.

        <dependency>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>
          <version>31.1-jre</version>
          <type>jar</type>
        </dependency>

I'll investigate tomorrow.

The stuff still lives git.eclipse.org...

https://git.eclipse.org/c/oomph/org.eclipse.oomph.incubator.git/plain/OomphIncubator.setup

[HannesWell]

Guice 5.1.0 indeed still requires Guava 30.1, this is a longer standing issue that did not get much attention from the guice devs yet :/

• Update Google Guava to 31.1-jre google/guice#1642

There was even the suggestion to remove the need for guava from guice, but that was also not answered:

• Clarification - Guice 4.1 compatibility with guava-23.6-jre.jar google/guice#1222 (comment)

[merks]

Yes, it needs this old thing:

        <dependency>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>
          <version>30.1-jre</version>
          <type>jar</type>
        </dependency>

Then it misses this:

[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.eclipse.oomph.maven.site raw:1.0.0.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):1.0.0-SNAPSHOT
[ERROR]   Missing requirement: com.google.inject 5.1.0 requires 'java.package; com.google.errorprone.annotations 0.0.0' but it could not be found
[ERROR]   Cannot satisfy dependency: org.eclipse.oomph.maven.all.feature.group 4.28.0.qualifier depends on: org.eclipse.equinox.p2.iu; com.google.inject [5.1.0,5.1.0]

And then I don't think there is a maven artifact that is also an OSGi library for this...

What a mess....

[HannesWell]

I hope that is available in https://mvnrepository.com/artifact/com.google.errorprone/error_prone_annotations

But yes that's all not so nice...
It also Looks like that the google-devs are not eager to Update their own deps.

Maybe you could also try to include compile dependencies in the Maven Location automatically? Then it should be simpler to set up and since only the required deps are in included in the repo we cannot avoid it anyway.

[merks]

These are not OSGi bundles:

https://repo1.maven.org/maven2/com/google/errorprone/error_prone_annotation/2.9.0/
https://repo1.maven.org/maven2/com/google/errorprone/error_prone_annotations/2.9.0/

In general, I want to avoid using implicit dependencies because I want to know explicitly what is being pulled in and what version is being pulled in so that I can check those versions and report on them.

I could generate bundle information for those, but that's no longer a direct-from-maven dependency where we simply reuse the jar (maybe even keep the PGP signature). It's pretty much what Orbit does with its BND recipes to repackage the jars.

It seems to me that if we're not careful, everyone will happily repackage such things slightly differently, pulling in transitive dependencies without actually knowing what those are, producing multiple different artifacts with the same artifact key. That appears to be what this is doing:

https://github.com/eclipse/tm4e/blob/84c448f4d75b314f2fda8ae4b29ae07667a8e1de/target-platform/tm4e-target.target#L35-L54

There's also an aopalliance dependency, which is not an OSGi bundle, but that a least is available in Orbit.

It seems the m2e resolution is only considering the explicit pom dependencies:

https://repo1.maven.org/maven2/com/google/inject/guice/5.1.0/guice-5.1.0.pom

So it doesn't care that the package requirements of the OSGi bundle don't resolve:

Import-Package: com.google.common.base;version="[30.1,31)",com.google.
 common.cache;version="[30.1,31)",com.google.common.collect;version="[
 30.1,31)",com.google.common.primitives;version="[30.1,31)",com.google
 .errorprone.annotations,javax.annotation;version="[3.0,4)",javax.inje
 ct,org.aopalliance.intercept

That's going to be a problem when you try to actually use the bundle though isn't it?

[HannesWell]

These are not OSGi bundles:

https://repo1.maven.org/maven2/com/google/errorprone/error_prone_annotation/2.9.0/ https://repo1.maven.org/maven2/com/google/errorprone/error_prone_annotations/2.9.0/

Oh man, that's really a mess.

In general, I want to avoid using implicit dependencies because I want to know explicitly what is being pulled in and what version is being pulled in so that I can check those versions and report on them.

Yes, that's understandable. But from my experience not every Maven dependency pulled into the TP by a Maven-TargetLocation is eventually required in an OSGi runtime. Especially Annotations are sometimes not needed. So altough they might be in the TP, they don't end up in a product or repo since P2 then only considers OSGi metadata and not maven-dependencies.

I could generate bundle information for those, but that's no longer a direct-from-maven dependency where we simply reuse the jar (maybe even keep the PGP signature). It's pretty much what Orbit does with its BND recipes to repackage the jars.

I agree that its bad, but I'm a bit hopeless that it would work without.

As a side note, what do you think about migrating the BND-recipes in Orbit to use maven targets with explict BND-instructions that generated the OSGi metadata? That would also be an opportunity to sort out what's already available as OSGi-compatible jar on Maven-Central.

It seems to me that if we're not careful, everyone will happily repackage such things slightly differently, pulling in transitive dependencies without actually knowing what those are, producing multiple different artifacts with the same artifact key. That appears to be what this is doing:

https://github.com/eclipse/tm4e/blob/84c448f4d75b314f2fda8ae4b29ae07667a8e1de/target-platform/tm4e-target.target#L35-L54

Yes, in such cases we will probably need a central repo forever that manages how such dependencies are OSGi-ified.

It seems the m2e resolution is only considering the explicit pom dependencies:

https://repo1.maven.org/maven2/com/google/inject/guice/5.1.0/guice-5.1.0.pom

Yes the target-location can only pull in all (transitive) Maven dependencies if configured accordingly. Since that Target-location only knows about maven, that's the only thing it can do.

So it doesn't care that the package requirements of the OSGi bundle don't resolve:

Import-Package: com.google.common.base;version="[30.1,31)",com.google.
 common.cache;version="[30.1,31)",com.google.common.collect;version="[
 30.1,31)",com.google.common.primitives;version="[30.1,31)",com.google
 .errorprone.annotations,javax.annotation;version="[3.0,4)",javax.inje
 ct,org.aopalliance.intercept

That's going to be a problem when you try to actually use the bundle though isn't it?

That's right and that's indeed likely a problem. Funny enough that the Maven dependency to errorprone.annotations only comes transitively through guava, which does not require it at runtime (at least according to its OSGi metadata).

I looked a bit into the guice code and wonder if the com.google.errorprone.annotations.Keep annotation really needs to have a runtime retention policy from its doc I would say that SOURCE source should be sufficient for the described purpose of static code analysis.
The com.google.errorprone.annotations.CheckReturnValue annotation could have better reasons to be kept since callers of annotation methods/classes should be checked with that.
It's a mess.

[merks]

I think it's very interesting to consider using the m2e target location for simple BND recipes and I'd like to explore that further, time permitting. But that time is not available until after next week at the earliest...

[HannesWell]

I think it's very interesting to consider using the m2e target location for simple BND recipes and I'd like to explore that further, time permitting. But that time is not available until after next week at the earliest...

I just created an issue to discuss that in Orbit: eclipse-orbit/orbit#8

[merks]

I got the first thing, error_prone_annotations, working:

		  <dependencies>
			  <dependency>
				  <groupId>com.google.errorprone</groupId>
				  <artifactId>error_prone_annotations</artifactId>
				  <version>2.18.0</version>
				  <type>jar</type>
			  </dependency>
			  <dependency>
				  <groupId>com.google.guava</groupId>
				  <artifactId>guava-base</artifactId>
				  <version>r30</version>
				  <type>jar</type>
			  </dependency>
		  </dependencies>
		  <instructions><![CDATA[
Bundle-Name:           Bundle derived from maven artifact ${mvnGroupId}:${mvnArtifactId}:${mvnVersion}
version:               ${version_cleanup;${mvnVersion}}
Bundle-SymbolicName:   ${mvnGroupId}.${mvnArtifactId}
Bundle-Version:        ${version}
Import-Package:        *;resolution:=optional
Export-Package:        *;version="${version}";-noimport:=true
DynamicImport-Package: *
]]></instructions>

But the guava-base dependency doesn't work. Maybe because it's kind of a weird version number? The error is

Could not resolve artifact com.google.guava:guava-base:jar:r30

That definitely exists though:

https://repo1.maven.org/maven2/com/google/guava/guava-base/r03/

I also wonder about the details of the BND instructions. E.g., does it make sense to consider all package imports optional? I have my doubts about that. In general, will each artifact need specialized instructions? I kind of suspect so because r30 is a pretty strange version number.

The generated manifest has quite some "noise" in it such that repeated builds will not produce the same manifest:

Manifest-Version: 1.0
Automatic-Module-Name: com.google.errorprone.annotations
Bnd-LastModified: 1681659866842
Build-Jdk-Spec: 11
Bundle-ManifestVersion: 2
Bundle-Name: Bundle derived from maven artifact com.google.errorprone:
 error_prone_annotations:2.18.0
Bundle-SymbolicName: com.google.errorprone.error_prone_annotations
Bundle-Version: 2.18.0
Created-By: 17.0.4.1 (Eclipse Adoptium)
DynamicImport-Package: *
Export-Package: com.google.errorprone.annotations;version="2.18.0";use
 s:="javax.lang.model.element",com.google.errorprone.annotations.concu
 rrent;version="2.18.0"
Import-Package: javax.lang.model.element;resolution:=optional
Originally-Created-By: Maven JAR Plugin 3.2.2
Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.8))"
Tool: Bnd-6.4.0.202211291949

That's not so nice...

[HannesWell]

But the guava-base dependency doesn't work. Maybe because it's kind of a weird version number? The error is

Could not resolve artifact com.google.guava:guava-base:jar:r30

That definitely exists though:

https://repo1.maven.org/maven2/com/google/guava/guava-base/r03/

That's right that this exists, but it seems to be an artifact from 2010: https://mvnrepository.com/artifact/com.google.guava/guava-base/r03
And I don't think we want that. 😅 The usual guava works fine.

I also wonder about the details of the BND instructions. E.g., does it make sense to consider all package imports optional? I have my doubts about that. In general, will each artifact need specialized instructions? I kind of suspect so because r30 is a pretty strange version number.

No we probably don't want to make all imports optional. This is just chosen as default to make it as flexible as possible and working as many cases as possible (e.g. often compile-time only deps are also listed) assuming that the required dependencies are actually specified in the pom. But since in general the BND-instructions are just a fallback it is totally fine to modify them to a more reasonable value for real world use.

It of course depends on the exact metadata, but for what I have encountered so far e.g. with using wildcards smartly one set of instructions was sufficient.

The generated manifest has quite some "noise" in it such that repeated builds will not produce the same manifest:

You can use the -noextraheaders instruction to remove them. If that is not sufficient you can explicitly remove headers using the -removeheaders instruction (and list the headers to remove afterwards.

The BND-lib references and incides can often be helpful:

• https://bnd.bndtools.org/chapters/800-headers.html
• https://bnd.bndtools.org/chapters/820-instructions.html
• https://bnd.bndtools.org/chapters/825-instructions-ref.html
• https://bnd.bndtools.org/chapters/850-macros.html
• https://bnd.bndtools.org/chapters/855-macros-ref.html

To sum this up, I would use a location like the following. I tried to use the latest version possible.
E.g. for guava 30.x I used 30.1.1 which can also replace the 30.x version from orbit even for those bundles that require guava<31.
:

		<location includeDependencyDepth="none" includeDependencyScopes="" includeSource="true" label="Google" missingManifest="generate" type="Maven">
			<dependencies>
				<dependency>
					<groupId>com.google.errorprone</groupId>
					<artifactId>error_prone_annotations</artifactId>
					<version>2.18.0</version>
					<type>jar</type>
				</dependency>
				<dependency>
					<groupId>com.google.guava</groupId>
					<artifactId>failureaccess</artifactId>
					<version>1.0.1</version>
					<type>jar</type>
				</dependency>
				<dependency>
					<groupId>com.google.guava</groupId>
					<artifactId>guava</artifactId>
					<version>30.1.1-jre</version>
					<type>jar</type>
				</dependency>
				<dependency>
					<groupId>com.google.inject</groupId>
					<artifactId>guice</artifactId>
					<version>5.1.0</version>
					<type>jar</type>
				</dependency>
				<dependency>
					<groupId>javax.inject</groupId>
					<artifactId>javax.inject</artifactId>
					<version>1</version>
					<type>jar</type>
				</dependency>
			</dependencies>
			<instructions><![CDATA[
Bundle-Name:           Bundle derived from maven artifact ${mvnGroupId}:${mvnArtifactId}:${mvnVersion}
version:               ${version_cleanup;${mvnVersion}}
Bundle-SymbolicName:   eclipse.orbit.wrapped.${mvnGroupId}.${mvnArtifactId}
Bundle-Version:        ${version}
Import-Package:        *
Export-Package:        *;version="${version}";-noimport:=true
-noextraheaders: true
]]></instructions>
			<exclude>com.google.code.findbugs:jsr305:3.0.1</exclude>
			<exclude>com.google.code.findbugs:jsr305:3.0.2</exclude>
			<exclude>com.google.j2objc:j2objc-annotations:1.3</exclude>
			<exclude>org.checkerframework:checker-qual:3.5.0</exclude>
		</location>

[HannesWell]

Btw. I noticed that changed instructions are not always taken into account by m2e. I'll look into that tomorrow.

[merks]

That's helpful information! I forgot it just required an older version of guava for the "base" package, which is already a very annoying problem when trying to eliminate duplicates, and especially so for guava which tends to be visible in APIs...

Going further, this does seem to be a bit of a rat hole with this failure trying what you showed:

[ERROR] Failed to resolve target definition file:/D:/Users/merks/oomph-incubator-1.0/git/org.eclipse.oomph.incubator/maven/tp/Maven.target: Artifact MavenArtifactFacade [wrappedArtifact=javax.inject:javax.inject:jar:1] of location MavenDependencyRoots = [GroupId = com.google.errorprone, ArtifactId = error_prone_annotations, Version = 2.18.0, ArtifactType = jar, IncludeDependencyScope = , GroupId = com.google.guava, ArtifactId = guava, Version = 30.1-jre, ArtifactType = jar, IncludeDependencyScope = , GroupId = javax.inject, ArtifactId = javax.inject, Version = 1, ArtifactType = jar, IncludeDependencyScope = ], IncludeDependencyScope = [compile], MissingManifestStrategy = GENERATE, IncludeSource = true is not a valid jar file

With this failure without that dependency:

[ERROR] Missing requirement: com.google.inject 5.1.0 requires 'java.package; javax.annotation [3.0.0,4.0.0)' but it could not be found

Note the package version range above which makes me wonder does this ancient really provide that package version?

https://repo1.maven.org/maven2/javax/inject/javax.inject/1/

Given it's not an OSGi bundle, we can't really say can we.

I will investigate further.

[HannesWell]

With this failure without that dependency:

[ERROR] Missing requirement: com.google.inject 5.1.0 requires 'java.package; javax.annotation [3.0.0,4.0.0)' but it could not be found

Regarding this failure, the version range is probably coming due the presence of javax.annotation.Nullable in the code, which is provided by com.google.code.findbugs:jsr305:3.0.1.

The good news is that in the 5.1.1-SNAPSHOT, since google/guice#1173 (which was just merged tonight) version range from the javax.annotation has no version range anymore.
I assume the intention was to remove it completely, but at least it can now be served by the javax.annotation bundle (which in fact does not contain the Nullable annotation), but that seems not necessary at runtime anyways.

Furthermore with this change Import-Package header values were changed from version 5.1.0

Import-Package:
 com.google.common.base;version="[30.1,31)",
 com.google.common.cache;version="[30.1,31)",
 com.google.common.collect;version="[30.1,31)",
 com.google.common.primitives;version="[30.1,31)",
 com.google.errorprone.annotations,
 javax.annotation;version="[3.0,4)",
 javax.inject,
 org.aopalliance.intercept

to

Import-Package:
 com.google.common.base;version="31.0",
 com.google.common.cache;version="31.0",
 com.google.common.collect;version="31.0",
 com.google.common.primitives;version="31.0",
 com.google.errorprone.annotations,
 javax.annotation,
 javax.inject,
 org.aopalliance.intercept,
 sun.misc

The missing upper bound is not great in SemVer terms but at least allows to use Guice with a new major version of Guava, which comes relatively often but often only breaks not relevant parts.

[merks]

This issue suggests that this "findbugs" thing also leads to problems:

eclipse-platform/eclipse.platform.ui#727 (comment)

@HannesWell

Even just updating what I have I hit this new problem:

[ERROR] Missing requirement: ch.qos.logback.classic 1.4.7 requires 'osgi.serviceloader; (osgi.serviceloader=ch.qos.logback.classic.spi.Configurator)' but it could not be found
[ERROR] Cannot satisfy dependency: org.eclipse.oomph.maven.all.feature.group 4.28.0.qualifier depends on: org.eclipse.equinox.p2.iu; ch.qos.logback.classic [1.4.7,1.4.7]

Looking in the jar, I get the sense that this jar itself should be providing this capability:

Is there something wrong with the new 1.4.7 version?