JAVA 21 - GWT 2.12.0 - Unable to get known Configuration java.util.ServiceConfigurationError

[ibrahimali123]

My Project is running locally correctly, but it gives me this error in the Google app engine logs staging and is not working. I don't know what this error means.

Unable to get known Configuration java.util.ServiceConfigurationError: at org.eclipse.jetty.util.ServiceLoaderSpliterator$ServiceProvider.get(ServiceLoaderSpliterator.java:101) at org.eclipse.jetty.ee8.webapp.Configurations.lambda$getKnown$0(Configurations.java:77) at org.eclipse.jetty.util.ServiceLoaderSpliterator.tryAdvance(ServiceLoaderSpliterator.java:46) at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:332) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at org.eclipse.jetty.ee8.webapp.Configurations.getKnown(Configurations.java:75) at org.eclipse.jetty.ee8.webapp.WebAppContext.preConfigure(WebAppContext.java:447) at org.eclipse.jetty.ee8.webapp.WebAppContext.doStart(WebAppContext.java:500) at com.google.apphosting.runtime.jetty.ee8.AppEngineWebAppContext.doStart(AppEngineWebAppContext.java:237) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.Handler$Abstract.doStart(Handler.java:491) at org.eclipse.jetty.server.handler.ContextHandler.lambda$doStart$0(ContextHandler.java:754) at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.call(ContextHandler.java:1452) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:754) at org.eclipse.jetty.ee8.nested.ContextHandler$CoreContextHandler.doStart(ContextHandler.java:2269) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.addManaged(ContainerLifeCycle.java:482) at org.eclipse.jetty.server.handler.HotSwapHandler.setHandler(HotSwapHandler.java:74) at com.google.apphosting.runtime.jetty.AppVersionHandler.ensureHandler(AppVersionHandler.java:97) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:235) at com.google.apphosting.runtime.RequestRunner.dispatchServletRequest(RequestRunner.java:349) at com.google.apphosting.runtime.RequestRunner.dispatchRequest(RequestRunner.java:234) at com.google.apphosting.runtime.RequestRunner.run(RequestRunner.java:200) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:273) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.util.ServiceConfigurationError: org.eclipse.jetty.ee8.webapp.Configuration: org.eclipse.jetty.ee8.webapp.WebAppConfiguration not a subtype at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593) at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1244) at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273) at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309) at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393) at org.eclipse.jetty.util.ServiceLoaderSpliterator.tryAdvance(ServiceLoaderSpliterator.java:37) ... 25 more

I am using gwt-dev 2.12.0 (the latest one) and using jetty-ee8

Here is my appengine-web.xml:
<property name="appengine.use.EE8" value="true"/>

This is a sample from the dependency tree:
+- org.gwtproject:gwt-dev:jar:2.11.0:compile | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile | +- com.google.code.gson:gson:jar:2.6.2:compile | +- org.ow2.asm:asm:jar:9.2:compile | +- org.ow2.asm:asm-util:jar:9.2:compile | | +- org.ow2.asm:asm-tree:jar:9.2:compile | | \- org.ow2.asm:asm-analysis:jar:9.2:compile | +- org.ow2.asm:asm-commons:jar:9.2:compile [INFO] | +- colt:colt:jar:1.2.0:compile [INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile [INFO] | +- commons-io:commons-io:jar:2.4:compile [INFO] | +- com.ibm.icu:icu4j:jar:63.1:compile [INFO] | +- tapestry:tapestry:jar:4.0.2:compile [INFO] | +- org.eclipse.jetty:jetty-webapp:jar:9.4.44.v20210927:compile [INFO] | | +- org.eclipse.jetty:jetty-xml:jar:9.4.44.v20210927:compile [INFO] | | \- org.eclipse.jetty:jetty-servlet:jar:9.4.44.v20210927:compile [INFO] | | +- org.eclipse.jetty:jetty-security:jar:9.4.44.v20210927:compile [INFO] | | | \- org.eclipse.jetty:jetty-server:jar:9.4.44.v20210927:compile [INFO] | | \- org.eclipse.jetty:jetty-util-ajax:jar:9.4.44.v20210927:compile [INFO] | +- org.eclipse.jetty:jetty-servlets:jar:9.4.44.v20210927:compile [INFO] | | +- org.eclipse.jetty:jetty-continuation:jar:9.4.44.v20210927:compile [INFO] | | +- org.eclipse.jetty:jetty-http:jar:9.4.44.v20210927:compile [INFO] | | +- org.eclipse.jetty:jetty-util:jar:9.4.44.v20210927:compile [INFO] | | \- org.eclipse.jetty:jetty-io:jar:9.4.44.v20210927:compile [INFO] | +- org.eclipse.jetty:jetty-annotations:jar:9.4.44.v20210927:compile [INFO] | | +- org.eclipse.jetty:jetty-plus:jar:9.4.44.v20210927:compile [INFO] | | | \- org.eclipse.jetty:jetty-jndi:jar:9.4.44.v20210927:compile [INFO] | | \- javax.annotation:javax.annotation-api:jar:1.3.2:compile [INFO] | \- org.eclipse.jetty:apache-jsp:jar:9.4.44.v20210927:compile [INFO] | +- org.eclipse.jetty.toolchain:jetty-schemas:jar:3.1.2:compile [INFO] | \- org.mortbay.jasper:apache-jsp:jar:8.5.70:compile [INFO] | +- org.mortbay.jasper:apache-el:jar:8.5.70:compile [INFO] | \- org.eclipse.jdt:ecj:jar:3.19.0:compile

[lachlan-roberts]

You are including gwt-dev in your webapp which includes a bunch of Jetty Jars.
This should be scope provided so it does not bring these jars into WEB-INF/lib.

Using the flag appengine.use.EE8 means you are running with Jetty 12, so those Jetty 9.4 jars are not used by the server anyway.

[ibrahimali123]

@lachlan-roberts thanks for your comment.

I have changed this
<dependency> <groupId>org.gwtproject</groupId> <artifactId>gwt-dev</artifactId> </dependency>
to be:
<dependency> <groupId>org.gwtproject</groupId> <artifactId>gwt-dev</artifactId> <scope>provided</scope> </dependency>

It shows the same error on the staging environment, but it works correctly when run locally. I also tried excluding these, but the following error is shown when running GWT Super Dev locally.
<dependency> <groupId>org.gwtproject</groupId> <artifactId>gwt-dev</artifactId> <exclusions> <exclusion> <groupId>org.eclipse.jetty</groupId> <artifactId>jetty-webapp</artifactId> </exclusion> <exclusion> <groupId>org.eclipse.jetty</groupId> <artifactId>jetty-servlets</artifactId> </exclusion> <exclusion> <groupId>org.eclipse.jetty</groupId> <artifactId>jetty-annotations</artifactId> </exclusion> <exclusion> <groupId>org.eclipse.jetty</groupId> <artifactId>apache-jsp</artifactId> </exclusion> </exclusions> </dependency>

This is the error:
java.lang.NoClassDefFoundError: org/eclipse/jetty/server/Connector at com.google.gwt.dev.codeserver.SourceHandler.<clinit>(SourceHandler.java:63) at com.google.gwt.dev.codeserver.Recompiler.loadModule(Recompiler.java:500) at com.google.gwt.dev.codeserver.Recompiler.initWithoutPrecompile(Recompiler.java:204) at com.google.gwt.dev.codeserver.Outbox.maybePrecompile(Outbox.java:89) at com.google.gwt.dev.codeserver.Outbox.<init>(Outbox.java:61) at com.google.gwt.dev.codeserver.CodeServer.makeOutboxTable(CodeServer.java:192) at com.google.gwt.dev.codeserver.CodeServer.start(CodeServer.java:151) at com.google.gwt.dev.codeserver.CodeServer.main(CodeServer.java:104) at com.google.gwt.dev.codeserver.CodeServer.main(CodeServer.java:55) Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.server.Connector at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ... 9 more

[lachlan-roberts]

I don't think excluding the Jetty dependencies is the right thing to do.
This will be causing the java.lang.NoClassDefFoundError because its expecting to find the Jetty 9.4 classes but they have been excluded.
Just keep gwt-dev entirely as a provided dependency.

But now I think that might actually be a separate issue to the original stacktrace.

Caused by: java.util.ServiceConfigurationError: org.eclipse.jetty.ee8.webapp.Configuration: org.eclipse.jetty.ee8.webapp.WebAppConfiguration not a subtype

Can please you do a Jetty Server Dump and post the results here.
See https://github.com/GoogleCloudPlatform/appengine-java-standard/blob/main/debugging.md#jetty-server-dump

[ibrahimali123]

@lachlan-roberts
jetty_dump.txt
The dump logs are in this file.
should I use app.yaml for Java 21 and the app engine? because I am using appengine-web.xml and I don't know if it is the causer

[ludoch]

Hi, If your source code uses import com.google.appengine.api.* (i.e if you are using the AppEngine APIs), you need to use appengine-web.xml.
If not, then you can move to app.yaml configuration, but there, you also need to change your pacakging in pom.xml (not war but jar) and provide your own Web Server.

A few questions from me:
1/ are you upgrading an existing project, and was it working before?
2/ could you share the pom.xml so I better understand the deps?

With our bundled Jetty server supporting GAE APIs. we do use a strongly isolated class loader for the application code, so we do not expose Jetty internal classes (or a very small set, that we put in the runtime-shared-*.jar )....
If it was working for a non Java21 app, since this content changed a bit for Jetty12, we might have to extend.a bit this shared jar, so answering question 1 is important.
Thanks!

[ludoch]

Ho, I see you are not including the GAE api jars in your web-inf/lib (dump file), so you can probably swich to app.yaml configuration.
So I see a few spring jars files, you probably are using SpringBoot and you could move to the SpringBoot far jar model?

If you want to keep appengine-web.xml, you can refer to this working SpringBoot war code sample using latest Springboot:

https://github.com/ludoch/appengine-spring-boot

and study the dependencies in the pom.xml file.

[ludoch]

I looked at the transitive dependencies from GWT to Jetty, and it does not see very correct, as it mixes Jetty9 and Jetty12 jars and might be using way to much Jetty dependencies without real specified needs. Those extra jars conflict with our bundled Jetty runtime.

for references, these are the deps I see coming from gwt-dev:

 <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-alpn-client</artifactId>
        <version>12.0.12</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-annotations</artifactId>
        <version>9.4.44.v20210927</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-client</artifactId>
        <version>12.0.12</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-continuation</artifactId>
        <version>9.4.44.v20210927</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-http</artifactId>
        <version>12.0.12</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-io</artifactId>
        <version>12.0.12</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-plus</artifactId>
        <version>12.0.12</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty.toolchain</groupId>
        <artifactId>jetty-schemas</artifactId>
        <version>3.1.2</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-security</artifactId>
        <version>12.0.12</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-server</artifactId>
        <version>12.0.12</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-servlet</artifactId>
        <version>9.4.44.v20210927</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-servlets</artifactId>
        <version>9.4.44.v20210927</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-util</artifactId>
        <version>12.0.12</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-util-ajax</artifactId>
        <version>12.0.12</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-webapp</artifactId>
        <version>9.4.44.v20210927</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-xml</artifactId>
        <version>12.0.12</version>
    </dependency>

[ibrahimali123]

@ludoch Yes, I am upgrading to Java 21 with appengine.use.EE8 and the project was working before on java 17 without any problem. And you are right there are conflict jars versions of the jetty in gwt-dev as your last comment and I think the problem coming from that.

[ludoch]

So gwt-dev is not really ready for Jetty12 used in our java21 runtime...
Maybe file a file on https://github.com/gwtproject/gwt or fork it to see if the change would be easy?
Maybe @lachlan-roberts from the Jetty team can help,

[lachlan-roberts]

I think it is unrelated to the gwt-dev dependency, in the jetty-dump.txt there was no gwt-dev in WEB-INF/lib and the ServiceConfigurationError was still seen.

I tried adding most of these jars in my WEB-INF/lib and I could not see any ServiceConfigurationError.
And I got a lot more info in my dump than what was in the provided jetty-dump.txt.

@ibrahimali123 could you please provide a simplified project to reproduce this?

[maigovannon]

Hi @ibrahimali123 Could you please share your GCP project ID? We can take a look at the logs from our end?

[ludoch]

Hi @ibrahimali123

FYI, I cannot repro with a sample from the gwt download zip (DynaTableRf which has a pom and adding appengine extra files and config with updating pom to latest everything)

The sample pom.xml has

    <dependency>
      <groupId>org.gwtproject</groupId>
      <artifactId>gwt-dev</artifactId>
      <!-- "provided" so that we don't deploy -->
      <scope>provided</scope>
    </dependency>

What would you need a -dev in prod? See the scope for the sample.

Could you add a simple repro app source with maven pom so we can study the issue more?

[ludoch]

Hi,
We could not reproduce this issue, so we would really appreciate if you can help us, as we want to understand if it is a runtime issue or an app configuration issue.

Anyway you can share more information regarding your pom.xml, your dependencies and possibly the source code handler that triggers this exception?
Thank you so much!