emqx edge docker bridge connected lost after some time

[moluzhui]

BUG REPORT

Expected behavior

emqx edge bridge always keep connect

Actual behavior

The emqx edge container bridge seems be lost after taking over for some time, because it takes effect after restart edge container

Test code

When I type docker logs -f edge-bridge-container, it appears as follows

emqx v3.2.4 is started successfully!
2019-12-07 11:14:47.999 [error] [Cube] connected failed, Error: {error,{unauthorized_client,undefined}} 
2019-12-07 11:15:18.751 [error] [Cube] Connecting remote cube server failed, Reason : {unauthorized_client,
                                                       undefined}
2019-12-07 11:15:18.752 [error] [Cube] connected failed, Error: {error,{unauthorized_client,undefined}} 
2019-12-07 11:15:49.360 [error] [Cube] Connecting remote cube server failed, Reason : {unauthorized_client,
                                                       undefined}
2019-12-07 11:15:49.361 [error] [Cube] connected failed, Error: {error,{unauthorized_client,undefined}} 
2019-12-07 11:16:20.419 [error] [Cube] Connecting remote cube server failed, Reason : {unauthorized_client,
                                                       undefined}
2019-12-07 11:16:20.419 [error] [Cube] connected failed, Error: {error,{unauthorized_client,undefined}} 
2019-12-07 11:16:37.752 [error] [Cube] Connecting remote cube server failed, Reason : {unauthorized_client,
                                                       undefined}
2019-12-07 11:16:37.752 [error] [Cube] connected failed, Error: {error,{unauthorized_client,undefined}} 
['2019-12-07T11:16:42Z']:emqx start
2019-12-07 11:17:08.452 [error] [Cube] Connecting remote cube server failed, Reason : {unauthorized_client,
                                                       undefined}
2019-12-07 11:17:08.452 [error] [Cube] connected failed, Error: {error,{unauthorized_client,undefined}} 
2019-12-07 11:17:39.291 [error] [Cube] Connecting remote cube server failed, Reason : {unauthorized_client,
                                                       undefined}
2019-12-07 11:17:39.291 [error] [Cube] connected failed, Error: {error,{unauthorized_client,undefined}} 
2019-12-07 11:18:09.953 [error] [Cube] Connecting remote cube server failed, Reason : {unauthorized_client,
                                                       undefined}

The cube service is turned on by default, and I don't seem to find the off configuration in the emqx-cube. Conf file , and there seems to be no impact on the bridge plugin.

What caused the bridge to break?

some time, an error is as follows

bridge.mqtt.edgeai.mountpoint=/
emqx v3.2.4 is started successfully!
2019-12-07 11:38:35.148 [error] [Cube] Connecting remote cube server failed, Reason : {unauthorized_client,
                                                       undefined}
2019-12-07 11:38:35.149 [error] [Cube] connected failed, Error: {error,{unauthorized_client,undefined}} 
['2019-12-07T11:38:40Z']:emqx start
2019-12-07 11:39:04.692 [error] ** State machine <0.1313.0> terminating
** Last event = {info,{timeout,#Ref<0.4258508646.869007361.30858>,ack}}
** When server state  = {connected,
                         {state,undefined,<0.1312.0>,
                          #{disconnected => #Fun<emqx_bridge_mqtt.6.102392321>,
                            puback => #Fun<emqx_bridge_mqtt.4.102392321>,
                            publish => #Fun<emqx_bridge_mqtt.5.102392321>},
                          "192.168.79.200",1883,[],#Port<0.16>,[],60000,true,
                          <<"bridge_test">>,true,<<"user">>,<<"passwd">>,4,
                          <<"MQTT">>,60000,#Ref<0.4258508646.869007361.30856>,
                          false,false,false,
                          {mqtt_msg,0,false,false,undefined,undefined,
                           undefined,undefined},
                          #{},
                          [{call,
                            {subscribe,2},
                            {<0.1300.0>,#Ref<0.4258508646.869007361.30857>},
                            {subscribe,#{},
                             [{<<"test/+/edge_broker_id/#">>,
                               #{nl => 0,qos => 1,rap => 0,rh => 0}}]},
                            {1575,718714,691642}}],
                          #{<<"test/+/edge_broker_id/#">> =>
                             #{nl => 0,qos => 1,rap => 0,rh => 0}},
                          infinity,
                          {emqx_inflight,0,{0,nil}},
                          #{},true,30000,#Ref<0.4258508646.869007361.30858>,
                          20000000,undefined,0,3,
                          {none,#{max_size => 268435455,version => 4}}}}
** Reason for termination = error:badarg
** Callback mode = state_functions
** Stacktrace =
**  [{emqx_client,'-timeout_calls/3-fun-0-',4,[]},
     {lists,foldl,3,[]},
     {emqx_client,connected,3,[]},
     {gen_statem,call_state_function,5,[]},
     {gen_statem,loop_event_state_function,6,[]},
     {proc_lib,init_p_do_apply,3,[]}]

2019-12-07 11:39:04.694 [error]   crasher:
    initial call: emqx_client:init/1
    pid: <0.1313.0>
    registered_name: []
    exception error: bad argument
      in function  emqx_client:'-timeout_calls/3-fun-0-'/4 
      in call from lists:foldl/3 
      in call from emqx_client:connected/3 
      in call from gen_statem:call_state_function/5 
      in call from gen_statem:loop_event_state_function/6 
    ancestors: [emqx_bridge_worker_test,emqx_bridge_mqtt_sup,<0.1298.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1312.0>,<0.1300.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 6772
    stack_size: 27
    reductions: 26202
  neighbours:
    neighbour:
      pid: <0.1312.0>
      registered_name: []
      initial_call: {erlang,apply,2}
      current_function: {emqx_bridge_mqtt,ack_collector,4}
      ancestors: []
      message_queue_len: 0
      links: [<0.1300.0>,<0.1313.0>]
      trap_exit: false
      status: waiting
      heap_size: 233
      stack_size: 5
      reductions: 763
      current_stacktrace: [{emqx_bridge_mqtt,ack_collector,4,[]}]
2019-12-07 11:39:04.696 [error] ** State machine emqx_bridge_worker_test terminating
** Last event = {state_timeout,do_connect}
** When server state  = {standing_by,
                         #{batch_bytes_limit => 1048576000,
                           batch_count_limit => 32,
                           bridge_handler => undefined,
                           connect_fun => #Fun<emqx_bridge_worker.5.75989401>,
                           connect_module => emqx_bridge_mqtt,
                           connection => undefined,
                           forwards => [<<"test/+/cloud_broker_id/#">>],
                           if_record_metrics => true,inflight => [],
                           max_inflight_batches => 32,mountpoint => <<"/">>,
                           reconnect_delay_ms => 30000,
                           replayq =>
                            #{committer => <0.1302.0>,
                              config =>
                               #{dir => "data/emqx_test_bridge/",
                                 seg_bytes => 10485760},
                              head_segno => 1,
                              in_mem => {[],[]},
                              marshaller =>
                               #Fun<emqx_bridge_worker.3.75989401>,
                              sizer => fun emqx_bridge_msg:estimate_size/1,
                              stats => #{bytes => 0,count => 0},
                              w_cur =>
                               #{bytes => 0,count => 0,
                                 fd =>
                                  {file_descriptor,raw_file_io_delayed,
                                   #{buffer =>
                                      #Ref<0.4258508646.869138433.30448>,
                                     delay_size => 65536,owner => <0.1300.0>,
                                     pid => <0.1301.0>}},
                                 segno => 1}},
                           start_type => auto,
                           subscriptions =>
                            [{<<"test/+/edge_broker_id/#">>,1},
                             {<<"mapa_init_params">>,1}]}}
** Reason for termination = exit:{{badarg,
                                   [{emqx_client,'-timeout_calls/3-fun-0-',4,
                                     []},
                                    {lists,foldl,3,[]},
                                    {emqx_client,connected,3,[]},
                                    {gen_statem,call_state_function,5,[]},
                                    {gen_statem,loop_event_state_function,6,
                                     []},
                                    {proc_lib,init_p_do_apply,3,[]}]},
                                  {gen_statem,call,
                                   [<0.1313.0>,
                                    {subscribe,#{},
                                     [{<<"test/+/edge_broker_id/#">>,
                                       #{nl => 0,qos => 1,rap => 0,rh => 0}}]},
                                    infinity]}}
** Callback mode = [state_functions,state_enter]
** Stacktrace =
**  [{gen,do_call,4,[]},
     {gen_statem,call_dirty,4,[]},
     {emqx_bridge_mqtt,'-subscribe_remote_topics/2-fun-0-',2,[]},
     {lists,foreach,2,[]},
     {emqx_bridge_mqtt,start,1,[]},
     {emqx_bridge_connect,start,2,[]},
     {emqx_bridge_worker,do_connect,3,[]},
     {gen_statem,call_state_function,5,[]}]

2019-12-07 11:39:04.697 [error]   crasher:
    initial call: emqx_bridge_worker:init/1
    pid: <0.1300.0>
    registered_name: emqx_bridge_worker_test
    exception exit: {{badarg,
                         [{emqx_client,'-timeout_calls/3-fun-0-',4,[]},
                          {lists,foldl,3,[]},
                          {emqx_client,connected,3,[]},
                          {gen_statem,call_state_function,5,[]},
                          {gen_statem,loop_event_state_function,6,[]},
                          {proc_lib,init_p_do_apply,3,[]}]},
                     {gen_statem,call,
                         [<0.1313.0>,
                          {subscribe,#{},
                              [{<<"test/+/edge_broker_id/#">>,
                                #{nl => 0,qos => 1,rap => 0,rh => 0}}]},
                          infinity]}}
      in function  gen:do_call/4 
      in call from gen_statem:call_dirty/4 
      in call from emqx_bridge_mqtt:'-subscribe_remote_topics/2-fun-0-'/2 
      in call from lists:foreach/2 
      in call from emqx_bridge_mqtt:start/1 
      in call from emqx_bridge_connect:start/2 
      in call from emqx_bridge_worker:do_connect/3 
      in call from gen_statem:call_state_function/5 
    ancestors: [emqx_bridge_mqtt_sup,<0.1298.0>]
    message_queue_len: 3
    messages: [{disconnected,#Ref<0.4258508646.869007361.30516>,badarg},
                  {'EXIT',<0.1313.0>,
                      {badarg,
                          [{emqx_client,'-timeout_calls/3-fun-0-',4,[]},
                           {lists,foldl,3,[]},
                           {emqx_client,connected,3,[]},
                           {gen_statem,call_state_function,5,[]},
                           {gen_statem,loop_event_state_function,6,[]},
                           {proc_lib,init_p_do_apply,3,[]}]}},
                  {'EXIT',<0.1312.0>,
                      {badarg,
                          [{emqx_client,'-timeout_calls/3-fun-0-',4,[]},
                           {lists,foldl,3,[]},
                           {emqx_client,connected,3,[]},
                           {gen_statem,call_state_function,5,[]},
                           {gen_statem,loop_event_state_function,6,[]},
                           {proc_lib,init_p_do_apply,3,[]}]}}]
    links: [<0.1299.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 10958
    stack_size: 27
    reductions: 39728
  neighbours:

2019-12-07 11:39:04.699 [error]     supervisor: {local,emqx_bridge_mqtt_sup}
    errorContext: child_terminated
    reason: {{badarg,[{emqx_client,'-timeout_calls/3-fun-0-',4,[]},
                      {lists,foldl,3,[]},
                      {emqx_client,connected,3,[]},
                      {gen_statem,call_state_function,5,[]},
                      {gen_statem,loop_event_state_function,6,[]},
                      {proc_lib,init_p_do_apply,3,[]}]},
             {gen_statem,call,
                         [<0.1313.0>,
                          {subscribe,#{},
                                     [{<<"test/+/edge_broker_id/#">>,
                                       #{nl => 0,qos => 1,rap => 0,rh => 0}}]},
                          infinity]}}
    offender: [{pid,<0.1300.0>},
               {id,test},
               {mfargs,
                   {emqx_bridge_worker,start_link,
                       [test,
                        [{address,"192.168.79.200:1883"},
                         {bridge_mode,true},
                         {clean_start,true},
                         {client_id,"bridge_test"},
                         {connect_module,emqx_bridge_mqtt},
                         {forwards,["test/+/cloud_broker_id/#"]},
                         {keepalive,60000},
                         {max_inflight_batches,32},
                         {mountpoint,"/"},
                         {password,"passwd"},
                         {proto_ver,v4},
                         {queue,
                             #{batch_bytes_limit => 1048576000,
                               batch_count_limit => 32,
                               replayq_dir => "data/emqx_test_bridge/",
                               replayq_seg_bytes => 10485760}},
                         {reconnect_delay_ms,30000},
                         {retry_interval,20000},
                         {ssl,false},
                         {ssl_opts,
                             [{versions,['tlsv1.2','tlsv1.1',tlsv1]},
                              {ciphers,
                                  [{psk,aes_128_cbc,sha},
                                   {psk,aes_256_cbc,sha},
                                   {psk,'3des_ede_cbc',sha},
                                   {psk,rc4_128,sha}]},
                              {user_lookup_fun,{fun emqx_psk:lookup/3,<<>>}},
                              {keyfile,"etc/certs/client-key.pem"},
                              {ciphers,
                                  ["ECDHE-ECDSA-AES256-GCM-SHA384",
                                   "ECDHE-RSA-AES256-GCM-SHA384"]},
                              {certfile,"etc/certs/client-cert.pem"},
                              {cacertfile,"etc/certs/cacert.pem"}]},
                         {start_type,auto},
                         {subscriptions,
                             [{"test/+/edge_broker_id/#",1},
                              {"mapa_init_params",1}]},
                         {username,"user"}]]}},
               {restart_type,permanent},
               {shutdown,5000},
               {child_type,worker}]
2019-12-07 11:39:05.705 [error] [Cube] Connecting remote cube server failed, Reason : {unauthorized_client,
                                                       undefined}

EMQ version

emqx-edge:v3.2.4

Docker version

Docker version 18.09.3, build 774a1f4