Support assuming an intermediate role when using the AWS-MSK-IAM SASL mechanism

[yolken]

Description

This change updates the AWS-MSK-IAM SASL mechanism to optionally support assuming an intermediate role to get the MSK credentials. See #64 for more background and motivation.

The intermediate role ARN can be specified in the cluster config (by setting assumeRole in the SASL section) or, for subcommands that don't require a cluster config, on the command-line (by setting the --sasl-assume-role flag).

Testing

TBD

[erikdw]

-- hopefully @roman-ef can test #64 .

[ghost]

Yes, we're looking into this. Hopefully we'll get back to you by the end of this week. Sorry it's taking so long

[vijaya314]

Hi @erikdw @yolken, I have done some tests using build from this branch and it appears that assume-role does not seem to work as expected as I get SASL Authentication Failed error.

AWS Profile used in tests

A) main-profile: one which should be able to assume role intended.
B) role-to-be-assumed-profile: The one having access to perform operations on cluster.

Cluster Configs:

A) cluster-config-with-assume-role.yaml: Cluster config which has assumeRole defined under sasl section.

sasl:
    enabled: true
    mechanism: AWS-MSK-IAM
    assumeRole: arn:aws:iam::XXXXXXXXXXX:role/role-to-be-assumed-ew1

B) cluster-config-with-NO-assume-role.yaml: Cluster config which has NO assumeRole in sasl section.

Tests

Test 1: ❌ Using AWS Profile = A, Custer Config = A, command get topics

➜  deploy git:(spike/topicctl-assume-role-change) ✗ AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=main-profile topicctl --debug get topics --cluster-config cluster-config-with-assume-role.yaml
[2022-06-14 22:27:29] DEBUG No ZK addresses provided, using broker admin client
[2022-06-14 22:27:29] DEBUG Connecting to cluster on address b-1.xxxxxxx.yyyyy.zz.kafka.eu-ZZZZ-1.amazonaws.com:9098 with TLS enabled=true, SASL enabled=true
[2022-06-14 22:27:29] DEBUG Getting supported API versions
[2022-06-14 22:27:31] ERROR 58: SASL Authentication Failed: SASL Authentication failed: [e5f1f15c-f814-4e6e-9d2f-f88a1be7bd92]: Internal error

(NOTE: Test # 2 is regression test and it works as expected)
Test 2: ✅ Using AWS Profile = B, Custer Config = B, command get topics (expected OK)

➜  deploy git:(spike/topicctl-assume-role-change) ✗ AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=role-to-assumed-profile topicctl get topics --cluster-config cluster-config-with-NO-assume-role.yaml
[2022-06-14 22:27:14]  INFO Checking cluster ID against version in cluster
[2022-06-14 22:27:15]  INFO Topics:
----------------------------------------------+------------+-------------+-----------+------------
                     NAME                     | PARTITIONS | REPLICATION | RETENTION |   RACKS
                                              |            |             |   MINS    | (MIN,MAX)
----------------------------------------------+------------+-------------+-----------+------------
  __amazon_msk_canary                         | 3          | 3           | 1440      | (3,3)
  __consumer_offsets                          | 50         | 3           |           | (3,3)
  xxxx.topic-1                                | 3          | 3           |           | (3,3)
  xxxx.topic-2                                | 3          | 3           |           | (3,3)
----------------------------------------------+------------+-------------+-----------+------------

Test 3: ❌ Using AWS Profile = A, Custer Config = B but with --sasl-assume-rule command line option, command get topics

➜  deploy git:(spike/topicctl-assume-role-change) ✗ AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=main-profile topicctl --debug --sasl-assume-role 'arn:aws:iam::XXXXXXXXXXX:role/role-to-be-assumed-ew1' get topics --cluster-config cluster-config-with-NO-assume-role.yaml
[2022-06-14 22:27:47] DEBUG No ZK addresses provided, using broker admin client
[2022-06-14 22:27:47] DEBUG Connecting to cluster on address b-1.xxxxxxx.yyyyy.zz.kafka.eu-ZZZZ-1.amazonaws.com:9098 with TLS enabled=true, SASL enabled=true
[2022-06-14 22:27:47] DEBUG Getting supported API versions
[2022-06-14 22:27:48] ERROR 58: SASL Authentication Failed: SASL Authentication failed: [4351bc1c-b503-4a3c-96a9-952351f8cce5]: Internal error

Test 4: ❌ Using AWS Profile = A, Custer Config = B but with --sasl-assume-rule command line option, command check

➜  deploy git:(spike/topicctl-assume-role-change) ✗ AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=main-profile topicctl --debug --sasl-assume-role 'arn:aws:iam::XXXXXXXXXXX:role/role-to-be-assumed-ew1' check --cluster-config cluster-config-with-NO-assume-role.yaml ./kafka/topics/topic.yaml
[2022-06-14 22:28:00] ERROR unknown flag: --sasl-assume-role

Test 5: ❌ Using AWS Profile = A, Custer Config = B but with --sasl-assume-rule command line option, command apply

➜  deploy git:(spike/topicctl-assume-role-change) ✗ AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=main-profile topicctl --debug --sasl-assume-role 'arn:aws:iam::XXXXXXXXXXX:role/role-to-be-assumed-ew1' apply --cluster-config cluster-config-with-NO-assume-role.yaml ./kafka/topics/topic.yaml
[2022-06-14 22:46:08] ERROR unknown flag: --sasl-assume-role

Based on Test 4/5, it appears --sasl-assume-rule is not recognised command option for check and apply

Please let us know if you need more information around tests.

NB: I have XX/YY/ZZ the org specific stuff.

[yolken]

@vijaya314, got it, so you'd like to be able to set the assume role on the command line even if using a config? I just made some updates to support this, let me know if they work for you.

[vijaya314]

@vijaya314, got it, so you'd like to be able to set the assume role on the command line even if using a config? I just made some updates to support this, let me know if they work for you.

Hi @yolken. I wanted assume role to work either via config Or via command line. But I performed tests using both methods just in case either of method works.

Today, I have pulled recent changes and run the same tests again.

• I no longer get unknown flag error for check and apply command using --sasl-assume-rule command line option.
• But assume role does not seem to work (using either of config or command line options). Now I am getting same error 58: SASL Authentication Failed: SASL Authentication failed: [d67801d5-fde3-42d0-901d-db161e6ee48a]: Internal error for all Test # 1, Test # 3, Test # 4, Test # 5.

Let me know if you need more details.

Thank you.

[ghotiphud]

@vijaya314 @yolken I think the issue you've run into with the tests is similar to #110. Which seems to be an issue with how the AWS Session is created. Would you be able to confirm if passing the cluster region into the session.NewSession() call fixes this issue?