Within EKS to offload a topic to S3 got AuthorizationException

[y00273676]

Describe the bug

As deploying Pulsar inside AWS EKS (Kubernetes hosted by AWS), while trying to use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to assume the AWS credential in order to offload ledgers into S3 Bucket, I've encountered exceptions like:

Error in offload
null

Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.jclouds.rest.AuthorizationException: The AWS Access Key Id you provided does not exist in our records.
command terminated with exit code 1

Login the eks pod

root@pulsar-broker-3:/pulsar# echo $AWS_SECRET_ACCESS_KEY
****************URQQ 
root@pulsar-broker-3:/pulsar# echo $AWS_SECRET_ACCESS_KEY
****************I+Gz

root@pulsar-broker-3:/pulsar# aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************URQQ              env
secret_key     ****************I+Gz              env
    region           cn-northwest-1              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']

So the AWS_SECRET_ACCESS_KEY and AWS_SECRET_ACCESS_KEY is valid

To Reproduce
Steps to reproduce the behavior:

precondition： the aws s3 bucket was created

1. Deploy Pulsar Broker within a EKS pod, if that is too costy, you can inject environement variables and broker.conf like this:

   broker.conf:
   managedLedgerOffloadDriver: "aws-s3"
   s3ManagedLedgerOffloadBucket: "xxxxxxx"
   s3ManagedLedgerOffloadRegion: "cn-northwest-1"
   
   env:
   - name: "AWS_ACCESS_KEY_ID"
     value: "****************URQQ  "
   - name: "AWS_SECRET_ACCESS_KEY"
     value: "****************I+Gz"
   

2. Set the. offload threshold manually

   kubectl -n pulsar exec -ti toolset-0 -c pulsar-toolset -- ./bin/pulsar-admin --auth-params "token:xxxxxxxxxx" --auth-plugin "org.apache.pulsar.client.impl.auth.AuthenticationToken" --admin-url http://pulsar-broker.pulsar:8080  topics offload --size-threshold 1K yourTopic
   
   

3. Send pulsar msg to you topic

4. Get offload status

   kubectl -n pulsar exec -ti toolset-0 -c pulsar-toolset -- ./bin/pulsar-admin --auth-params "token:xxxxxxxx" --auth-plugin "org.apache.pulsar.client.impl.auth.AuthenticationToken" --admin-url http://pulsar-broker.pulsar:8080  topics offload-status -w yourTopic
   

[zymap]

@y00273676 Can you use the same credential to access AWS S3 successfully by AWS CLI? Looks like the secret key missing? You may check if it exists in the AWS

[y00273676]

@y00273676 Can you use the same credential to access AWS S3 successfully by AWS CLI? Looks like the secret key missing? You may check if it exists in the AWS

The secret key is not missing, I can upload file to s3 and get key list by AWS CLI

[zymap]

How did you deploy pulsar to EKS? Do you have any configuration files? Do you deploy the cluster with helm charts?

[codelipenghui]

The issue had no activity for 30 days, mark with Stale label.