Pods httpfs-0 and s3g-0 Keep Restarting Due to Liveness Probe Failures in Ozone 2.0.0 Kubernetes Deployment

[roach231428]

Hello, I am testing Apache Ozone 2.0.0 using the official Kubernetes example, but encountering continuous restarts in the httpfs and s3g pods due to failing liveness probes.

---

Observations

Pod: httpfs-0

• Continually restarts due to failed liveness probes.

• Probing the WebHDFS endpoint results in:

  {"RemoteException":{"message":"User: hadoop is not allowed to impersonate hadoop","exception":"AuthorizationException","javaClassName":"org.apache.hadoop.security.authorize.AuthorizationException"}}

• Indicates a missing or misconfigured proxy user configuration.

• ConfigMap includes:

  HTTPFS-SITE.XML_httpfs.proxyuser.hadoop.groups: '*'
  HTTPFS-SITE.XML_httpfs.proxyuser.hadoop.hosts: '*'

  but the file /opt/hadoop/etc/config/core-site.xml is not mounted in the container, possibly due to a volume or path issue.

• Thus, httpfs fails to authorize the proxy user hadoop, leading to a 500 response and liveness failure.

Pod: s3g-0

• Liveness probe sends GET /health to port 9878.

• Response is HTTP 403 Forbidden with the following message:

  <Error>
    <Code>InvalidRequest</Code>
    <Message>Error creating s3 auth info. The request may not be signed using AWS V4 signing algorithm, or might be invalid</Message>
  </Error>

• However, logs show that a SIGTERM is received right after startup due to health check failure.

• This is likely a misconfiguration in the liveness probe — it assumes an open HTTP endpoint, but S3G requires signed requests.

---

Environment

• Kubernetes version: v1.33.1 (single control plane + one worker)
• Apache Ozone version: 2.0.0 (apache/ozone:2.0.0)
• Deployment method: kubectl apply -k ozone-2.0.0/kubernetes/examples/ozone
• Official guide followed: https://ozone.apache.org/docs/edge/start/kubernetes.html
• Modifications: datanode replica count set to 1 for testing

---

Key Issues

• Authorization: The default config does not allow the hadoop user to impersonate hadoop, which breaks services relying on proxyuser functionality.

• Liveness Probe Behavior:

  • httpfs: probe to /webhdfs/v1/?op=LISTSTATUS&user.name=hadoop fails with impersonation error.
  • s3g: /health endpoint fails due to unauthenticated access or misinterpreted proxy call.

---

What I Need Help With

1. Fixing impersonation:

   • How should proxyuser configuration and mount paths be set to allow hadoop to impersonate itself in Kubernetes?

2. Alternative health checks:

   • Is there a recommended way to adjust the liveness probe for s3g to bypass the proxy or use a no-auth endpoint?

3. Working example:

   • Is there a known good configuration of Ozone 2.0.0 on Kubernetes with minimal changes that supports single-node testing?

Thanks!

[roach231428]

Switching the liveness probe from httpGet to tcpSocket appears to resolve the issue on s3g. That said, it's unclear whether a TCP-level check is sufficient for validating application health in this context.

[roach231428]

I believe I’ve found a solution for this issue.

In s3g-statefulset.yaml, we need to update the livenessProbe by replacing the httpGet check with a tcpSocket check, like this:

...
        livenessProbe:
          tcpSocket:
            port: 9870
...

Additionally, to resolve the impersonation issue in httpfs, we should add the following core-site.xml settings in config-configmap.yaml to ensure they take effect:

  CORE-SITE.XML_hadoop.proxyuser.hadoop.groups: '*'
  CORE-SITE.XML_hadoop.proxyuser.hadoop.hosts: '*'

[kerneltime]

please feel free to post a PR with the changes.

[roach231428]

please feel free to post a PR with the changes.

Sure, the pull request has been submitted here: #8775.