Ozone 1.3.0 + Spark 3.5.2 : Unable to create OzoneClient object

[kmizumar]

Hello experts, I need your help.

Our spark driver fails to create an OzoneClient object with the following stack, after showing tons of INFO RetryInvocationHandler: java.lang.IllegalStateException, while invoking $Proxy36.submitRequest over nodeId=om-06,nodeAddress=<REDACTED>:9862 after 499 failover attempts. Trying to failover immediately.

24/08/26 06:24:50 ERROR OMFailoverProxyProviderBase: Failed to connect to OMs: [om-05, om-06, om-30]. Attempted 500 failovers.
24/08/26 06:24:50 ERROR OzoneClientFactory: Couldn't create RpcClient protocol exception: 
java.lang.IllegalStateException
    at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:494)
    at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:119)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:162)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
    at jdk.proxy3/jdk.proxy3.$Proxy36.submitRequest(Unknown Source)
    at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:289)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1454)
    at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:239)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:148)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:67)
    at <REDACTED>.tools.Main$.main(Main.scala:74)
    at <REDACTED>.tools.Main.main(Main.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:569)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" java.io.IOException: Couldn't create RpcClient protocol
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:148)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:67)
    at <REDACTED>.tools.Main$.main(Main.scala:74)
    at <REDACTED>.tools.Main.main(Main.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:569)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalStateException
    at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:494)
    at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:119)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:162)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
    at jdk.proxy3/jdk.proxy3.$Proxy36.submitRequest(Unknown Source)
    at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:289)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1454)
    at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:239)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
    ... 16 more
24/08/26 06:24:50 INFO ShutdownHookManager: Shutdown hook called
24/08/26 06:24:50 INFO ShutdownHookManager: Deleting directory /tmp/spark-b16b5c9f-bbe2-4256-8609-047787f0bf9c
24/08/26 06:24:50 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system...
24/08/26 06:24:50 INFO MetricsSystemImpl: s3a-file-system metrics system stopped.
24/08/26 06:24:50 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete.

We're doing nothing special in creating the object and confirmed our core-site.xml and ozone-site.xml work fine with another simple Ozone Java API client.

    val ozoneClient = OzoneClientFactory.getRpcClient()

I suspect this is a kind of dependency problem.
Could anyone please shed light on this? How can we investigate the root cause of this issue?

$ cat build.sbt 
ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.14"

lazy val scopeVersion = "4.1.0"
lazy val sparkVersion = "3.5.2"
//lazy val ozoneVersion = "1.4.0"
lazy val ozoneVersion = "1.3.0"

lazy val root = (project in file("."))
  .settings(
    name := "cais-tools",
    idePackagePrefix := Some("<REDACTED>.tools"),
    assemblySettings,
    assembly / mainClass := Some("<REDACTED>.tools.Main"),
    libraryDependencies ++= Seq(
      "com.github.scopt" %% "scopt" % scopeVersion,
      "org.apache.spark" %% "spark-core" % sparkVersion % Provided,
      "org.apache.spark" %% "spark-sql" % sparkVersion % Provided,
      "org.apache.spark" %% "spark-kubernetes" % sparkVersion % Provided,
      "org.apache.spark" %% "spark-hadoop-cloud" % sparkVersion % Provided,
      "org.apache.ozone" % "ozone-client" % ozoneVersion,
    )
  )

lazy val assemblySettings = Seq(
  assembly / assemblyJarName := s"${name.value}-${version.value}.jar",
  assembly / assemblyMergeStrategy := {
    case PathList("META-INF", _*) => MergeStrategy.discard
    case x => MergeStrategy.first
  }
)

---

#6549 looks similar, but I'm not confident.

[yazynin]

Why you use ozone 1.3.0? this installation worked before?

[kmizumar]

Here are a few reasons why we are using Ozone 1.3.0 as a client:

• The target Ozone cluster is running on code based on version 1.3.0.
• We have achieved the expected operations without issues when using the Ozone Java API for operations from a Java client.

We encounter this problem while trying to combine Spark to perform large-scale parallel execution of individually executable data processing tasks.

[siddhantsangwan]

Not an expert in this area, but I can help you in ensuring the basics are correct.
Have you configured these correctly?

ozone.om.service.ids = omservice
ozone.om.address.omservice.om0 = <om-node1-host>:9862
ozone.om.address.omservice.om1 = <om-node2-host>:9862
ozone.om.address.omservice.om2 = <om-node3-host>:9862
ozone.om.nodes.omservice = om0,om1,om2

<property>
    <name>fs.ofs.impl</name
    <value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value>
</property>

Following from this issue, I think Spark may have to use this jar:

share/ozone/lib/ozone-filesystem-hadoop3-client-1.3.0.jar

Check if you have this jar in the classpath.

[kmizumar]

Check if you have this jar in the classpath.

Interesting, I added the following line to my libraryDependencies

      "org.apache.ozone" % "ozone-filesystem-hadoop3-client" % ozoneVersion,

then the call stack changed as follows:

24/08/28 13:23:08 INFO OzoneClientFactory: Creating OzoneClient with default configuration.
Exception in thread "main" java.lang.NoSuchMethodError: 'org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMResponse org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolPB.submitRequest(com.google.protobuf.RpcController, org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest)'
    at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:289)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1454)
    at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:239)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:148)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:67)
    at <REDACTED>.tools.Main$.main(Main.scala:74)
    at <REDACTED>.tools.Main.main(Main.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:569)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
24/08/28 13:23:08 INFO ShutdownHookManager: Shutdown hook called
24/08/28 13:23:08 INFO ShutdownHookManager: Deleting directory /tmp/spark-c77ccce7-0c89-4d7c-8026-279632539459
24/08/28 13:23:08 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system...
24/08/28 13:23:08 INFO MetricsSystemImpl: s3a-file-system metrics system stopped.
24/08/28 13:23:08 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
Stream closed EOF for user-kmizumar/cais-tools-verify-8d869d91992718c4-driver (spark-kubernetes-driver)

Is this a packaging issue?

[siddhantsangwan]

@kmizumar What's the latest on this, were you able to resolve the problem? @GeorgeJahad will you be able to help here?

[GeorgeJahad]

Sorry, I'm not working on Ozone right now and don't have time to look. If it is the same problem as the one I was working on, then this is the key comment:
https://issues.apache.org/jira/browse/HDDS-6926?focusedCommentId=17754164&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17754164

In particular:

This is the ozone jar file that should be used with Spark and any other tool that uses the hadoop-client jar:
hadoop-ozone/ozonefs-hadoop3-client/target/ozone-filesystem-hadoop3-client-1.3.0-SNAPSHOT.jar

It should be used with spark instead of this jar file:
ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar