shovel: Support SASL EXTERNAL and/or SASL disabled

[carlsmedstad]

Is your feature request related to a problem? Please describe.

We're trying to instantiate a Dynamic Shovel propagating messages from a third-party AMQP 1.0 broker to a queue in our cluster (deployed using cluster-operator). The ambition is to replace a Python component built with python-qpid-proton.

We're creating the shovel using the API:

curl -X PUT -u "<USERNAME>:<PASSWORD>" \
http://<CLUSTER_URL>:15672/api/parameters/shovel/<VHOST_NAME>/<SHOVEL_NAME> \
-d '{
  "value": {
    "src-protocol": "amqp10",
    "src-uri": "amqps://<SOURCE_HOST>:<PORT>?certfile=/path/to/client.crt&keyfile=/path/to/client.key",
    "src-address": "queue://<SOURCE_QUEUE_NAME>",
    "dest-protocol": "amqp091",
    "dest-uri": "amqp://<USERNAME>:<PASSWORD>@<DESTINATION_HOST>/<VHOST>",
    "dest-queue": "regulations"
  }
}'

The shovel fails to connect to the third-party broker due to failed SASL authentication:

14:55:11.580462+00:00 [error] <0.1083252.0> ** State machine <0.1083252.0> terminating
14:55:11.580462+00:00 [error] <0.1083252.0> ** Last event = {cast,
14:55:11.580462+00:00 [error] <0.1083252.0>                     {'v1_0.sasl_mechanisms',
14:55:11.580462+00:00 [error] <0.1083252.0>                         {array,symbol,
14:55:11.580462+00:00 [error] <0.1083252.0>                             [{symbol,<<"EXTERNAL">>},{symbol,<<"PLAIN">>}]}}}
14:55:11.580462+00:00 [error] <0.1083252.0> ** When server state  = {sasl_hdr_rcvds,
14:55:11.580462+00:00 [error] <0.1083252.0>                          {state,0,<0.1083251.0>,
14:55:11.580462+00:00 [error] <0.1083252.0>                           #Ref<0.651287666.108265474.143866>,<0.1083259.0>,
14:55:11.580462+00:00 [error] <0.1083252.0>                           [{<0.1083247.0>,
14:55:11.580462+00:00 [error] <0.1083252.0>                             [alias|
14:55:11.580462+00:00 [error] <0.1083252.0>                              #Ref<0.0.21215107.651287666.108331010.143865>]}],
14:55:11.580462+00:00 [error] <0.1083252.0>                           <0.1083250.0>,
14:55:11.580462+00:00 [error] <0.1083252.0>                           {ssl,
14:55:11.580462+00:00 [error] <0.1083252.0>                            {sslsocket,
14:55:11.580462+00:00 [error] <0.1083252.0>                             {gen_tcp,#Port<0.13298>,tls_connection,undefined},
14:55:11.580462+00:00 [error] <0.1083252.0>                             [<0.1083258.0>,<0.1083257.0>]}},
14:55:11.580462+00:00 [error] <0.1083252.0>                           undefined,undefined,
14:55:11.580462+00:00 [error] <0.1083252.0>                           #{notify => <0.1083247.0>,port => 5671,
14:55:11.580462+00:00 [error] <0.1083252.0>                             address => "<SRC_HOST>",
14:55:11.580462+00:00 [error] <0.1083252.0>                             sasl => anon,
14:55:11.580462+00:00 [error] <0.1083252.0>                             hostname => <<"<SRC_HOST>">>,
14:55:11.580462+00:00 [error] <0.1083252.0>                             properties =>
14:55:11.580462+00:00 [error] <0.1083252.0>                              #{<<"ignore-maintenance">> => {boolean,true}},
14:55:11.580462+00:00 [error] <0.1083252.0>                             tls_opts =>
14:55:11.580462+00:00 [error] <0.1083252.0>                              {secure_port,
14:55:11.580462+00:00 [error] <0.1083252.0>                               [{certfile,
14:55:11.580462+00:00 [error] <0.1083252.0>                                 "/etc/rabbitmq/shovel-tls/client.crt"},
14:55:11.580462+00:00 [error] <0.1083252.0>                                {keyfile,"/etc/rabbitmq/shovel-tls/client.key"},
14:55:11.580462+00:00 [error] <0.1083252.0>                                {verify,verify_peer}]},
14:55:11.580462+00:00 [error] <0.1083252.0>                             notify_when_opened => <0.1083247.0>,
14:55:11.580462+00:00 [error] <0.1083252.0>                             notify_when_closed => <0.1083247.0>,
14:55:11.580462+00:00 [error] <0.1083252.0>                             transfer_limit_margin => 0,
14:55:11.580462+00:00 [error] <0.1083252.0>                             max_frame_size => 1048576}}}
14:55:11.580462+00:00 [error] <0.1083252.0> ** Reason for termination = exit:{sasl_not_supported,anon}
14:55:11.580462+00:00 [error] <0.1083252.0> ** Callback modules = [amqp10_client_connection]
14:55:11.580462+00:00 [error] <0.1083252.0> ** Callback mode = state_functions

We have confirmed that our old Python component can connect to the broker using both:

• SASL EXTERNAL
• SASL disabled

However, we're unable to configure the shovel to do either of the above:

• The above error occurs also when sasl=none is provided to the shovel (in the URI). So it seems it's not possible to disable SASL authentication.
• While the amqp10_client_connection module seems to support SASL EXTERNAL, the parse_uri function in amqp10_client.erl does not. Hence, it seems like SASL EXTERNAL is currently not supported in the shovel plugin.

Appreciate any input on how to resolve this, thanks!

Describe the solution you'd like

The shovel plugin supports:

• SASL EXTERNAL
• SASL disabled

Describe alternatives you've considered

No response

Additional context

No response

[michaelklishin]

@carlsmedstad your conclusion is incorrect. Shovels use regular AMQP 0-9-1 and AMQP 1.0 clients, so anything that is supported for those clients is supported by shovels.

[michaelklishin]

What AMQP 1.0 shovels support is a different story. Those clients libraries are nowhere near as mature as the AMQP 0-9-1 one which has been around for over a decade.

[carlsmedstad]

Thank you for clarifying that. I think what we're asking for then is for the AMQP 1.0 client library to support sasl=external provided in the URI.

We are by no means familiar with Erlang or the RabbitMQ code base (yet), but from what we've been able to gather this would be possible with a fairly small code change:

diff --git a/deps/amqp10_client/src/amqp10_client.erl b/deps/amqp10_client/src/amqp10_client.erl
index 1c4674d995..22efe1c635 100644
--- a/deps/amqp10_client/src/amqp10_client.erl
+++ b/deps/amqp10_client/src/amqp10_client.erl
@@ -449,6 +449,7 @@ parse_result(Map) ->
     Query  = maps:from_list(uri_string:dissect_query(Query0)),
     Sasl = case Query of
                #{"sasl" := "anon"} -> anon;
+               #{"sasl" := "external"} -> external;
                #{"sasl" := "plain"} when UserInfo =:= undefined orelse length(UserInfo) =:= 0 ->
                    throw(plain_sasl_missing_userinfo);
                _ ->

If we're on the right track here, would you like us to open a new issue?