Boot fails when importing definitions at boot

[PaperTsar]

Describe the bug

The issue arises when a stream queue is defined in definitions.json and loaded through rabbitmq.conf. The first time the container is started with docker compose it boots fine, the definitions are read without issues. However, after a shutdown and restart, the container fails to boot. The error from the container is highlighted here:

 2025-10-21 20:00:56.603505+00:00 [error] <0.467.0> BOOT FAILED
 2025-10-21 20:00:56.603505+00:00 [error] <0.467.0> ===========
 2025-10-21 20:00:56.603505+00:00 [error] <0.467.0> Error during startup: {error,<<"{resources_missing,\n    [{absent,\n         {amqqueue,\n             {resource,<<\"/\">>,queue,<<\"logs\">>},\n             true,false,none,\n             [{<<\"x-queue-type\">>,longstr,<<\"stream\">>}],\n             <0.622.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<\"/\">>,\n             #{user => <<\"rmq-internal\">>},\n             rabbit_stream_queue,\n             #{name => \"__logs_1761075697741821717\",\n               nodes => [rabbit@rabbitmq],\n               reference => {resource,<<\"/\">>,queue,<<\"logs\">>},\n               leader_pid => <0.622.0>,leader_node => rabbit@rabbitmq,\n               replica_nodes => [],epoch => 12,retention => [],\n               event_formatter =>\n                   {rabbit_stream_queue,format_osiris_event,\n                       [{resource,<<\"/\">>,queue,<<\"logs\">>}]}}},\n         nodedown}]}">>}
 2025-10-21 20:00:56.603505+00:00 [error] <0.467.0> 

The issue does not happen if the definitions.json file is not loaded on the second restart. Queues and exchanges remain intact. It also does not happen if the queue is not a stream.

Reproduction steps

1. docker compose up -d
2. docker compose down
3. docker compose up

Expected behavior

No crash.

Additional context

docker-compose.yaml

services:
  rabbitmq:
    image: rabbitmq:4.1.4
    hostname: rabbitmq
    ports:
      - "5672:5672"   
      - "15672:15672" 
      - "5552:5552"
    volumes:
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro
      - ./enabled_plugins:/etc/rabbitmq/enabled_plugins:ro
      - ./definitions.json:/etc/rabbitmq/custom-definitions.json:ro
      - rabbitmq-lib:/var/lib/rabbitmq/
      - rabbitmq-log:/var/log/rabbitmq/
volumes:
  rabbitmq-lib:
  rabbitmq-log:

rabbitmq.conf

# AMQP + Management
listeners.tcp.1 = 0.0.0.0:5672

# Streams binary protocol listener
stream.listeners.tcp.1 = 0.0.0.0:5552
stream.advertised_host = rabbitmq
stream.advertised_port = 5552

# Import definitions at boot (recommended modern way)
definitions.import_backend = local_filesystem
definitions.local.path = /etc/rabbitmq/custom-definitions.json

definitions.json

{
  "server_version": "4.1.4",
  "users": [
    {
      "name": "guest",
      "tags": [
        "administrator"
      ],
      "password_hash": "oh8bQvgqCDtuNqq8omsAeXhYFRNS9CBA3VkW8ZSnPRgWCreh"
    }
  ],
  "vhosts": [
    {
      "name": "/",
      "tags": [],
      "description": "Default virtual host",
      "default_queue_type": null,
      "metadata": {
        "tags": [],
        "description": "Default virtual host",
        "default_queue_type": "classic"
      }
    }
  ],
  "permissions": [
    {"user": "guest", "vhost": "/", "configure": ".*","read": ".*", "write": ".*" }
  ],
  "parameters": [],
  "policies": [],
  "queues": [
    {"name": "logs", "vhost": "/", "durable": true, "auto_delete": false, "arguments": { "x-queue-type": "stream" }}
  ],
  "exchanges": [
    {"name": "logs", "vhost": "/", "type": "topic", "durable": true, "auto_delete": false, "arguments": {}},
  ],
  "bindings": [
    {"vhost": "/", "source": "logs", "destination": "logs", "destination_type": "queue", "routing_key": "#", "arguments": {}, "properties_key": null }
  ]
}

enabled_plugins

[rabbitmq_management,rabbitmq_stream].

Docker compose up output:

Attaching to rabbitmq-1
 Application syslog exited with reason: stopped
 2025-10-21 20:00:54.940798+00:00 [notice] <0.209.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
 2025-10-21 20:00:54.941283+00:00 [notice] <0.209.0> Logging: configured log handlers are now ACTIVE
 2025-10-21 20:00:54.949333+00:00 [info] <0.209.0> ra: starting system coordination
 2025-10-21 20:00:54.949430+00:00 [info] <0.209.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/coordination/rabbit@rabbitmq
 2025-10-21 20:00:54.957479+00:00 [info] <0.216.0> ra_coordination_log_ets: in system coordination initialising. Mem table opts: [set,{write_concurrency,auto},public,{compressed,false}]
 2025-10-21 20:00:55.004996+00:00 [info] <0.222.0> ra system 'coordination' running pre init for 2 registered servers
 2025-10-21 20:00:55.025293+00:00 [info] <0.223.0> ra: meta data store initialised for system coordination. 2 record(s) recovered
 2025-10-21 20:00:55.040198+00:00 [notice] <0.228.0> WAL: ra_coordination_log_wal init, mem-tables table name: ra_coordination_log_open_mem_tables
 2025-10-21 20:00:55.072835+00:00 [info] <0.209.0> ra: starting system quorum_queues
 2025-10-21 20:00:55.072906+00:00 [info] <0.209.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/quorum/rabbit@rabbitmq
 2025-10-21 20:00:55.073152+00:00 [info] <0.233.0> ra_log_ets: in system quorum_queues initialising. Mem table opts: [set,{write_concurrency,auto},public,{compressed,true}]
 2025-10-21 20:00:55.074013+00:00 [info] <0.237.0> ra system 'quorum_queues' running pre init for 0 registered servers
 2025-10-21 20:00:55.074796+00:00 [info] <0.238.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
 2025-10-21 20:00:55.075039+00:00 [notice] <0.243.0> WAL: ra_log_wal init, mem-tables table name: ra_log_open_mem_tables
 2025-10-21 20:00:55.078250+00:00 [info] <0.245.0> ra_system_recover: ra system 'quorum_queues' server recovery strategy rabbit_quorum_queue:system_recover
 2025-10-21 20:00:55.078357+00:00 [info] <0.245.0> [rabbit_quorum_queue:system_recover/1] rabbit not booted, skipping queue recovery
 2025-10-21 20:00:55.078535+00:00 [info] <0.209.0> ra: starting system coordination
 2025-10-21 20:00:55.078560+00:00 [info] <0.209.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/coordination/rabbit@rabbitmq
 2025-10-21 20:00:55.589456+00:00 [notice] <0.247.0> RabbitMQ metadata store: candidate -> leader in term: 10 machine version: 1, last applied 10
 2025-10-21 20:00:55.802483+00:00 [info] <0.209.0> 
 2025-10-21 20:00:55.802483+00:00 [info] <0.209.0>  Starting RabbitMQ 4.1.4 on Erlang 27.3.4.3 [jit]
 2025-10-21 20:00:55.802483+00:00 [info] <0.209.0>  Copyright (c) 2007-2025 Broadcom Inc and/or its subsidiaries
 2025-10-21 20:00:55.802483+00:00 [info] <0.209.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
 
   ##  ##      RabbitMQ 4.1.4
   ##  ##
   ##########  Copyright (c) 2007-2025 Broadcom Inc and/or its subsidiaries
   ######  ##
   ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com
 
   Erlang:      27.3.4.3 [jit]
   TLS Library: OpenSSL - OpenSSL 3.3.5 30 Sep 2025
   Release series support status: see https://www.rabbitmq.com/release-information
 
   Doc guides:  https://www.rabbitmq.com/docs
   Support:     https://www.rabbitmq.com/docs/contact
   Tutorials:   https://www.rabbitmq.com/tutorials
   Monitoring:  https://www.rabbitmq.com/docs/monitoring
   Upgrading:   https://www.rabbitmq.com/docs/upgrade
 
   Logs: <stdout>
 
   Config file(s): /etc/rabbitmq/rabbitmq.conf
                   /etc/rabbitmq/conf.d/10-defaults.conf
                   /etc/rabbitmq/conf.d/20-management_agent.disable_metrics_collector.conf
 
   Starting broker...2025-10-21 20:00:55.804255+00:00 [info] <0.209.0> 
 2025-10-21 20:00:55.804255+00:00 [info] <0.209.0>  node           : rabbit@rabbitmq
 2025-10-21 20:00:55.804255+00:00 [info] <0.209.0>  home dir       : /var/lib/rabbitmq
 2025-10-21 20:00:55.804255+00:00 [info] <0.209.0>  config file(s) : /etc/rabbitmq/rabbitmq.conf
 2025-10-21 20:00:55.804255+00:00 [info] <0.209.0>                 : /etc/rabbitmq/conf.d/10-defaults.conf
 2025-10-21 20:00:55.804255+00:00 [info] <0.209.0>                 : /etc/rabbitmq/conf.d/20-management_agent.disable_metrics_collector.conf
 2025-10-21 20:00:55.804255+00:00 [info] <0.209.0>  cookie hash    : EcuomObPtG8nJsBjUZInyQ==
 2025-10-21 20:00:55.804255+00:00 [info] <0.209.0>  log(s)         : <stdout>
 2025-10-21 20:00:55.804255+00:00 [info] <0.209.0>  data dir       : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq
 2025-10-21 20:00:56.212163+00:00 [info] <0.209.0> Running boot step pre_boot defined by app rabbit
 2025-10-21 20:00:56.212273+00:00 [info] <0.209.0> Running boot step rabbit_global_counters defined by app rabbit
 2025-10-21 20:00:56.212598+00:00 [info] <0.209.0> Running boot step rabbit_osiris_metrics defined by app rabbit
 2025-10-21 20:00:56.212672+00:00 [info] <0.209.0> Running boot step rabbit_core_metrics defined by app rabbit
 2025-10-21 20:00:56.213346+00:00 [info] <0.209.0> Running boot step rabbit_alarm defined by app rabbit
 2025-10-21 20:00:56.222919+00:00 [info] <0.332.0> Memory high watermark set to 9372 MiB (9828180787 bytes) of 15621 MiB (16380301312 bytes) total
 2025-10-21 20:00:56.225609+00:00 [info] <0.334.0> Enabling free disk space monitoring (disk free space: 115364556800, total memory: 16380301312)
 2025-10-21 20:00:56.225683+00:00 [info] <0.334.0> Disk free limit set to 50MB
 2025-10-21 20:00:56.227882+00:00 [info] <0.209.0> Running boot step code_server_cache defined by app rabbit
 2025-10-21 20:00:56.228010+00:00 [info] <0.209.0> Running boot step file_handle_cache defined by app rabbit
 2025-10-21 20:00:56.233678+00:00 [info] <0.337.0> Limiting to approx 1073741719 file handles (966367545 sockets)
 2025-10-21 20:00:56.233873+00:00 [info] <0.338.0> FHC read buffering: OFF
 2025-10-21 20:00:56.233909+00:00 [info] <0.338.0> FHC write buffering: ON
 2025-10-21 20:00:56.234378+00:00 [info] <0.209.0> Running boot step worker_pool defined by app rabbit
 2025-10-21 20:00:56.234484+00:00 [info] <0.287.0> Will use 7 processes for default worker pool
 2025-10-21 20:00:56.234518+00:00 [info] <0.287.0> Starting worker pool 'worker_pool' with 7 processes in it
 2025-10-21 20:00:56.234883+00:00 [info] <0.209.0> Running boot step rabbit_registry defined by app rabbit
 2025-10-21 20:00:56.235076+00:00 [info] <0.209.0> Running boot step database defined by app rabbit
 2025-10-21 20:00:56.235510+00:00 [info] <0.209.0> Peer discovery: configured backend: rabbit_peer_discovery_classic_config
 2025-10-21 20:00:56.237229+00:00 [info] <0.209.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
 2025-10-21 20:00:56.241228+00:00 [info] <0.209.0> Successfully synced tables from a peer
 2025-10-21 20:00:56.249979+00:00 [info] <0.209.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
 2025-10-21 20:00:56.250150+00:00 [info] <0.209.0> Successfully synced tables from a peer
 2025-10-21 20:00:56.250303+00:00 [info] <0.209.0> Running boot step tracking_metadata_store defined by app rabbit
 2025-10-21 20:00:56.250687+00:00 [info] <0.360.0> Setting up a table for connection tracking on this node: tracked_connection
 2025-10-21 20:00:56.250832+00:00 [info] <0.360.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
 2025-10-21 20:00:56.250884+00:00 [info] <0.360.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
 2025-10-21 20:00:56.250918+00:00 [info] <0.360.0> Setting up a table for channel tracking on this node: tracked_channel
 2025-10-21 20:00:56.251059+00:00 [info] <0.360.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
 2025-10-21 20:00:56.251195+00:00 [info] <0.209.0> Running boot step networking_metadata_store defined by app rabbit
 2025-10-21 20:00:56.251287+00:00 [info] <0.209.0> Running boot step feature_flags defined by app rabbit
 2025-10-21 20:00:56.251468+00:00 [info] <0.209.0> Running boot step auth_backend_plugins_check defined by app rabbit
 2025-10-21 20:00:56.251524+00:00 [info] <0.209.0> Running boot step codec_correctness_check defined by app rabbit
 2025-10-21 20:00:56.251575+00:00 [info] <0.209.0> Running boot step external_infrastructure defined by app rabbit
 2025-10-21 20:00:56.251597+00:00 [info] <0.209.0> Running boot step rabbit_event defined by app rabbit
 2025-10-21 20:00:56.251729+00:00 [info] <0.209.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
 2025-10-21 20:00:56.251806+00:00 [info] <0.209.0> Running boot step rabbit_auth_mechanism_anonymous defined by app rabbit
 2025-10-21 20:00:56.251895+00:00 [info] <0.209.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
 2025-10-21 20:00:56.251951+00:00 [info] <0.209.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
 2025-10-21 20:00:56.251984+00:00 [info] <0.209.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
 2025-10-21 20:00:56.252037+00:00 [info] <0.209.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
 2025-10-21 20:00:56.252073+00:00 [info] <0.209.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
 2025-10-21 20:00:56.252111+00:00 [info] <0.209.0> Running boot step rabbit_exchange_type_local_random defined by app rabbit
 2025-10-21 20:00:56.252147+00:00 [info] <0.209.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
 2025-10-21 20:00:56.252263+00:00 [info] <0.209.0> Running boot step rabbit_priority_queue defined by app rabbit
 2025-10-21 20:00:56.252305+00:00 [info] <0.209.0> Priority queues enabled, real BQ is rabbit_variable_queue
 2025-10-21 20:00:56.252347+00:00 [info] <0.209.0> Running boot step kernel_ready defined by app rabbit
 2025-10-21 20:00:56.252366+00:00 [info] <0.209.0> Running boot step pg_local_amqp_connection defined by app rabbit
 2025-10-21 20:00:56.258394+00:00 [info] <0.209.0> Running boot step pg_local_amqp_session defined by app rabbit
 2025-10-21 20:00:56.258530+00:00 [info] <0.209.0> Running boot step rabbit_sysmon_minder defined by app rabbit
 2025-10-21 20:00:56.258629+00:00 [info] <0.209.0> Running boot step rabbit_epmd_monitor defined by app rabbit
 2025-10-21 20:00:56.259609+00:00 [info] <0.369.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
 2025-10-21 20:00:56.259754+00:00 [info] <0.209.0> Running boot step guid_generator defined by app rabbit
 2025-10-21 20:00:56.265438+00:00 [info] <0.209.0> Running boot step rabbit_node_monitor defined by app rabbit
 2025-10-21 20:00:56.265742+00:00 [info] <0.373.0> Starting rabbit_node_monitor (in ignore mode)
 2025-10-21 20:00:56.265841+00:00 [info] <0.209.0> Running boot step delegate_sup defined by app rabbit
 2025-10-21 20:00:56.266258+00:00 [info] <0.209.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
 2025-10-21 20:00:56.266356+00:00 [info] <0.209.0> Running boot step core_initialized defined by app rabbit
 2025-10-21 20:00:56.266380+00:00 [info] <0.209.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
 2025-10-21 20:00:56.266537+00:00 [info] <0.209.0> Running boot step rabbit_classic_queue defined by app rabbit
 2025-10-21 20:00:56.266648+00:00 [info] <0.209.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
 2025-10-21 20:00:56.266688+00:00 [info] <0.209.0> Running boot step rabbit_definitions_hashing defined by app rabbit
 2025-10-21 20:00:56.266721+00:00 [info] <0.209.0> Running boot step rabbit_exchange_parameters defined by app rabbit
 2025-10-21 20:00:56.278887+00:00 [info] <0.209.0> Running boot step rabbit_policies defined by app rabbit
 2025-10-21 20:00:56.279432+00:00 [info] <0.209.0> Running boot step rabbit_policy defined by app rabbit
 2025-10-21 20:00:56.279569+00:00 [info] <0.209.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
 2025-10-21 20:00:56.279644+00:00 [info] <0.209.0> Running boot step rabbit_quorum_queue defined by app rabbit
 2025-10-21 20:00:56.279838+00:00 [info] <0.209.0> Running boot step rabbit_stream_coordinator defined by app rabbit
 2025-10-21 20:00:56.282237+00:00 [info] <0.209.0> Running boot step rabbit_vhost_limit defined by app rabbit
 2025-10-21 20:00:56.282435+00:00 [info] <0.209.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
 2025-10-21 20:00:56.282498+00:00 [info] <0.209.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
 2025-10-21 20:00:56.282447+00:00 [warning] <0.395.0> rabbit_stream_coordinator failed to update stream:
 2025-10-21 20:00:56.282447+00:00 [warning] <0.395.0> function_clause
 2025-10-21 20:00:56.282447+00:00 [warning] <0.395.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 12,system_time => 1761075720760,reply_mode => noreply,machine_version => 5,...},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[115|...]},{line,1388}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
 2025-10-21 20:00:56.282543+00:00 [info] <0.209.0> Management plugin: using rates mode 'basic'
 2025-10-21 20:00:56.282648+00:00 [warning] <0.395.0> rabbit_stream_coordinator failed to update stream:
 2025-10-21 20:00:56.282648+00:00 [warning] <0.395.0> function_clause
 2025-10-21 20:00:56.282648+00:00 [warning] <0.395.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 25,system_time => 1761075846362,reply_mode => noreply,machine_version => 5,...},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[115|...]},{line,1388}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
 2025-10-21 20:00:56.282747+00:00 [info] <0.209.0> Running boot step recovery defined by app rabbit
 2025-10-21 20:00:56.303429+00:00 [info] <0.407.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
 2025-10-21 20:00:56.313876+00:00 [info] <0.407.0> Starting message stores for vhost '/'
 2025-10-21 20:00:56.315892+00:00 [info] <0.407.0> Started message store of type transient for vhost '/'
 2025-10-21 20:00:56.317769+00:00 [info] <0.407.0> Started message store of type persistent for vhost '/'
 2025-10-21 20:00:56.317932+00:00 [info] <0.407.0> Recovering 0 queues of type rabbit_classic_queue took 13ms
 2025-10-21 20:00:56.317965+00:00 [info] <0.407.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
 2025-10-21 20:00:56.318005+00:00 [info] <0.407.0> Recovering 1 queues of type rabbit_stream_queue took 0ms
 2025-10-21 20:00:56.319931+00:00 [info] <0.209.0> Running boot step prevent_startup_if_node_was_reset defined by app rabbit
 2025-10-21 20:00:56.320009+00:00 [info] <0.209.0> Running boot step empty_db_check defined by app rabbit
 2025-10-21 20:00:56.320066+00:00 [info] <0.209.0> Will not seed default virtual host and user: have definitions to load...
 2025-10-21 20:00:56.320110+00:00 [info] <0.209.0> Running boot step rabbit_observer_cli defined by app rabbit
 2025-10-21 20:00:56.320169+00:00 [info] <0.209.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
 2025-10-21 20:00:56.320272+00:00 [info] <0.209.0> Running boot step background_gc defined by app rabbit
 2025-10-21 20:00:56.320389+00:00 [info] <0.209.0> Running boot step routing_ready defined by app rabbit
 2025-10-21 20:00:56.320444+00:00 [info] <0.209.0> Running boot step pre_flight defined by app rabbit
 2025-10-21 20:00:56.320479+00:00 [info] <0.209.0> Running boot step notify_cluster defined by app rabbit
 2025-10-21 20:00:56.320507+00:00 [info] <0.209.0> Running boot step networking defined by app rabbit
 2025-10-21 20:00:56.320638+00:00 [info] <0.209.0> Running boot step rabbit_quorum_queue_periodic_membership_reconciliation defined by app rabbit
 2025-10-21 20:00:56.320755+00:00 [info] <0.209.0> Running boot step definition_import_worker_pool defined by app rabbit
 2025-10-21 20:00:56.320800+00:00 [info] <0.287.0> Starting worker pool 'definition_import_pool' with 7 processes in it
 2025-10-21 20:00:56.321043+00:00 [info] <0.209.0> Running boot step cluster_name defined by app rabbit
 2025-10-21 20:00:56.321107+00:00 [info] <0.209.0> Running boot step virtual_host_reconciliation defined by app rabbit
 2025-10-21 20:00:56.321258+00:00 [info] <0.209.0> Running boot step direct_client defined by app rabbit
 2025-10-21 20:00:56.321352+00:00 [info] <0.209.0> Running boot step cluster_tags defined by app rabbit
 2025-10-21 20:00:56.324106+00:00 [info] <0.209.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
 2025-10-21 20:00:56.324283+00:00 [info] <0.467.0> Resetting node maintenance status
 2025-10-21 20:00:56.487685+00:00 [warning] <0.492.0> Metrics collection disabled in management agent, management only interface started
 2025-10-21 20:00:56.527895+00:00 [info] <0.492.0> Management plugin: HTTP (non-TLS) listener started on port 15672
 2025-10-21 20:00:56.528057+00:00 [info] <0.520.0> Statistics database started.
 2025-10-21 20:00:56.528115+00:00 [info] <0.519.0> Starting worker pool 'management_worker_pool' with 3 processes in it
 2025-10-21 20:00:56.528899+00:00 [info] <0.548.0> started Stream TCP listener on 0.0.0.0:5552
 2025-10-21 20:00:56.529809+00:00 [info] <0.467.0> Applying definitions from file at '/etc/rabbitmq/custom-definitions.json'
 2025-10-21 20:00:56.529888+00:00 [info] <0.467.0> Asked to import definitions. Acting user: rmq-internal
 2025-10-21 20:00:56.534586+00:00 [info] <0.467.0> Importing concurrently 1 users...
 2025-10-21 20:00:56.537944+00:00 [info] <0.467.0> Importing concurrently 1 vhosts...
 2025-10-21 20:00:56.543284+00:00 [info] <0.467.0> Importing concurrently 1 permissions...
 2025-10-21 20:00:56.546263+00:00 [info] <0.457.0> Successfully set permissions for user 'guest' in virtual host '/' to '.*', '.*', '.*'
 2025-10-21 20:00:56.546425+00:00 [info] <0.467.0> Importing concurrently 2 exchanges...
 2025-10-21 20:00:56.546912+00:00 [info] <0.467.0> Importing concurrently 1 queues...
 2025-10-21 20:00:56.562766+00:00 [info] <0.467.0> Importing concurrently 1 bindings...
 2025-10-21 20:00:56.603505+00:00 [error] <0.467.0> 
 2025-10-21 20:00:56.603505+00:00 [error] <0.467.0> BOOT FAILED
 2025-10-21 20:00:56.603505+00:00 [error] <0.467.0> ===========
 2025-10-21 20:00:56.603505+00:00 [error] <0.467.0> Error during startup: {error,<<"{resources_missing,\n    [{absent,\n         {amqqueue,\n             {resource,<<\"/\">>,queue,<<\"logs\">>},\n             true,false,none,\n             [{<<\"x-queue-type\">>,longstr,<<\"stream\">>}],\n             <0.622.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<\"/\">>,\n             #{user => <<\"rmq-internal\">>},\n             rabbit_stream_queue,\n             #{name => \"__logs_1761075697741821717\",\n               nodes => [rabbit@rabbitmq],\n               reference => {resource,<<\"/\">>,queue,<<\"logs\">>},\n               leader_pid => <0.622.0>,leader_node => rabbit@rabbitmq,\n               replica_nodes => [],epoch => 12,retention => [],\n               event_formatter =>\n                   {rabbit_stream_queue,format_osiris_event,\n                       [{resource,<<\"/\">>,queue,<<\"logs\">>}]}}},\n         nodedown}]}">>}
 2025-10-21 20:00:56.603505+00:00 [error] <0.467.0> 
 
 BOOT FAILED
 ===========
 Error during startup: {error,<<"{resources_missing,\n    [{absent,\n         {amqqueue,\n             {resource,<<\"/\">>,queue,<<\"logs\">>},\n             true,false,none,\n             [{<<\"x-queue-type\">>,longstr,<<\"stream\">>}],\n             <0.622.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<\"/\">>,\n             #{user => <<\"rmq-internal\">>},\n             rabbit_stream_queue,\n             #{name => \"__logs_1761075697741821717\",\n               nodes => [rabbit@rabbitmq],\n               reference => {resource,<<\"/\">>,queue,<<\"logs\">>},\n               leader_pid => <0.622.0>,leader_node => rabbit@rabbitmq,\n               replica_nodes => [],epoch => 12,retention => [],\n               event_formatter =>\n                   {rabbit_stream_queue,format_osiris_event,\n                       [{resource,<<\"/\">>,queue,<<\"logs\">>}]}}},\n         nodedown}]}">>}
 
 2025-10-21 20:00:56.648738+00:00 [notice] <0.395.0> rabbit_stream_coordinator: candidate -> leader in term: 8 machine version: 5, last applied 31
 2025-10-21 20:00:56.656718+00:00 [warning] <0.395.0> rabbit_stream_coordinator failed to update stream:
 2025-10-21 20:00:56.656718+00:00 [warning] <0.395.0> function_clause
 2025-10-21 20:00:56.656718+00:00 [warning] <0.395.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 46,system_time => 1761076314434,reply_mode => noreply,machine_version => 5,...},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[115|...]},{line,1388}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_log_segment,fold0,8,[...]},{ra_log_reader,'-segment_fold/5-fun-1-',4,...},{ra_log_reader,'-segment_fold/5-lists^foldl/2-1-',...},{ra_log_reader,...}]
 2025-10-21 20:00:56.656959+00:00 [warning] <0.395.0> rabbit_stream_coordinator failed to update stream:
 2025-10-21 20:00:56.656959+00:00 [warning] <0.395.0> function_clause
 2025-10-21 20:00:56.656959+00:00 [warning] <0.395.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 61,system_time => 1761076584002,reply_mode => noreply,machine_version => 5,...},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[115|...]},{line,1388}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_log_segment,fold0,8,[...]},{ra_log_reader,'-segment_fold/5-fun-1-',4,...},{ra_log_reader,'-segment_fold/5-lists^foldl/2-1-',...},{ra_log_reader,...}]
 2025-10-21 20:00:56.657147+00:00 [warning] <0.395.0> rabbit_stream_coordinator failed to update stream:
 2025-10-21 20:00:56.657147+00:00 [warning] <0.395.0> function_clause
 2025-10-21 20:00:56.657147+00:00 [warning] <0.395.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 83,system_time => 1761076654902,reply_mode => noreply,machine_version => 5,...},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[115|...]},{line,1388}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_log_segment,fold0,8,[...]},{ra_log_reader,'-segment_fold/5-fun-1-',4,...},{ra_log_reader,'-segment_fold/5-lists^foldl/2-1-',...},{ra_log_reader,...}]
 2025-10-21 20:00:56.679905+00:00 [info] <0.594.0> Stream: __logs_1761075697741821717 will use /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/stream/__logs_1761075697741821717 for osiris log data directory
 2025-10-21 20:00:56.680060+00:00 [info] <0.570.0> rabbit_stream_coordinator: started writer __logs_1761075697741821717 on rabbit@rabbitmq in 6
 2025-10-21 20:00:56.680061+00:00 [warning] <0.595.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 13 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680082+00:00 [warning] <0.578.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 9 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680051+00:00 [warning] <0.575.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 8 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680166+00:00 [warning] <0.573.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 7 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680173+00:00 [warning] <0.583.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 11 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680229+00:00 [warning] <0.580.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 10 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680181+00:00 [warning] <0.585.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 12 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.682693+00:00 [warning] <0.395.0> rabbit_stream_coordinator failed to update stream:
 2025-10-21 20:00:56.682693+00:00 [warning] <0.395.0> function_clause
 2025-10-21 20:00:56.682693+00:00 [warning] <0.395.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 111,system_time => 1761076856680,reply_mode => noreply,machine_version => 5,...},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[115|...]},{line,1388}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_log,fold,5,[...]},{ra_server,apply_to,5,...},{ra_server,handle_leader,...},{ra_server_proc,...}]
 2025-10-21 20:00:56.687627+00:00 [info] <0.594.0> osiris_writer:init/1: name: __logs_1761075697741821717 last offset: -1 committed chunk id: -1 epoch: 6
 2025-10-21 20:00:56.687733+00:00 [info] <0.594.0> osiris_writer:terminate/2: name __logs_1761075697741821717 reason: shutdown
 2025-10-21 20:00:56.690846+00:00 [info] <0.598.0> Stream: __logs_1761075697741821717 will use /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/stream/__logs_1761075697741821717 for osiris log data directory
 2025-10-21 20:00:56.691031+00:00 [info] <0.597.0> rabbit_stream_coordinator: started writer __logs_1761075697741821717 on rabbit@rabbitmq in 14
 2025-10-21 20:00:56.693039+00:00 [info] <0.598.0> osiris_writer:init/1: name: __logs_1761075697741821717 last offset: -1 committed chunk id: -1 epoch: 14
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0> Stopping RabbitMQ applications and their dependencies in the following order:
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     rabbitmq_management
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     rabbitmq_management_agent
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     rabbitmq_web_dispatch
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     rabbitmq_stream
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     rabbit
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     oauth2_client
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     rabbitmq_prelaunch
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     amqp_client
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     rabbit_common
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     jose
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     rabbitmq_stream_common
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     khepri
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     ra
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     cowboy
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     os_mon
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     sysmon_handler
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     osiris
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0>     mnesia
 2025-10-21 20:00:57.657464+00:00 [info] <0.467.0> 
 2025-10-21 20:00:57.657597+00:00 [info] <0.467.0> Stopping application 'rabbitmq_management'
 2025-10-21 20:00:57.659604+00:00 [warning] <0.483.0> HTTP listener registry could not find context rabbitmq_management_tls
 2025-10-21 20:00:57.660763+00:00 [notice] <0.45.0> Application rabbitmq_management exited with reason: stopped
 2025-10-21 20:00:57.660869+00:00 [info] <0.467.0> Stopping application 'rabbitmq_management_agent'
 2025-10-21 20:00:57.662091+00:00 [notice] <0.45.0> Application rabbitmq_management_agent exited with reason: stopped
 2025-10-21 20:00:57.662205+00:00 [info] <0.467.0> Stopping application 'rabbitmq_web_dispatch'
 2025-10-21 20:00:57.663505+00:00 [notice] <0.45.0> Application rabbitmq_web_dispatch exited with reason: stopped
 2025-10-21 20:00:57.663630+00:00 [info] <0.467.0> Stopping application 'rabbitmq_stream'
 2025-10-21 20:00:57.663743+00:00 [info] <0.548.0> stopped Stream TCP listener on 0.0.0.0:5552
 2025-10-21 20:00:57.665533+00:00 [notice] <0.45.0> Application rabbitmq_stream exited with reason: stopped
 2025-10-21 20:00:57.665677+00:00 [info] <0.467.0> Stopping application 'rabbit'
 2025-10-21 20:00:57.666446+00:00 [info] <0.407.0> Virtual host '/' is stopping
 2025-10-21 20:00:57.666564+00:00 [info] <0.604.0> Closing all connections in vhost '/' on node 'rabbit@rabbitmq' because the vhost is stopping
 2025-10-21 20:00:57.666590+00:00 [info] <0.421.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
 2025-10-21 20:00:57.672124+00:00 [info] <0.421.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
 2025-10-21 20:00:57.672346+00:00 [info] <0.417.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
 2025-10-21 20:00:57.676351+00:00 [info] <0.417.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
 2025-10-21 20:00:57.680664+00:00 [notice] <0.45.0> Application rabbit exited with reason: stopped
 {exit,{5104,{error,<<123,114,101,115,111,117,114,99,101,1152025-10-21 20:00:57.835239+00:00 [notice] <0.45.0> Application rabbitmq_prelaunch exited with reason: stopped
 ,95,109,105,115,115,105,110,103,44,10,32,32,32,32,91,123,97,98,115,101,110,116,44,10,32,32,32,32,32,32,32,32,32,123,97,109,113,113,117,101,117,101,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,123,114,101,115,111,117,114,99,101,44,60,60,34,47,34,62,62,44,113,117,101,117,101,44,60,60,34,108,111,103,115,34,62,62,125,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,116,114,117,101,44,102,97,108,115,101,44,110,111,110,101,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,91,123,60,60,34,120,45,113,117,101,117,101,45,116,121,112,101,34,62,62,44,108,111,110,103,115,116,114,44,60,60,34,115,116,114,101,97,109,34,62,62,125,93,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,60,48,46,54,50,50,46,48,62,44,91,93,44,91,93,44,91,93,44,117,110,100,101,102,105,110,101,100,44,117,110,100,101,102,105,110,101,100,44,91,93,44,91,93,44,108,105,118,101,44,48,44,91,93,44,60,60,34,47,34,62,62,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,35,123,117,115,101,114,32,61,62,32,60,60,34,114,109,113,45,105,110,116,101,114,110,97,108,34,62,62,125,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,114,97,98,98,105,116,95,115,116,114,101,97,109,95,113,117,101,117,101,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,35,123,110,97,109,101,32,61,62,32,34,95,95,108,111,103,115,95,49,55,54,49,48,55,53,54,57,55,55,52,49,56,50,49,55,49,55,34,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,110,111,100,101,115,32,61,62,32,91,114,97,98,98,105,116,64,114,97,98,98,105,116,109,113,93,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,114,101,102,101,114,101,110,99,101,32,61,62,32,123,114,101,115,111,117,114,99,101,44,60,60,34,47,34,62,62,44,113,117,101,117,101,44,60,60,34,108,111,103,115,34,62,62,125,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,108,101,97,100,101,114,95,112,105,100,32,61,62,32,60,48,46,54,50,50,46,48,62,44,108,101,97,100,101,114,95,110,111,100,101,32,61,62,32,114,97,98,98,105,116,64,114,97,98,98,105,116,109,113,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,114,101,112,108,105,99,97,95,110,111,100,101,115,32,61,62,32,91,93,44,101,112,111,99,104,32,61,62,32,49,50,44,114,101,116,101,110,116,105,111,110,32,61,62,32,91,93,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,101,118,101,110,116,95,102,111,114,109,97,116,116,101,114,32,61,62,10,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,123,114,97,98,98,105,116,95,115,116,114,101,97,109,95,113,117,101,117,101,44,102,111,114,109,97,116,95,111,115,105,114,105,115,95,101,118,101,110,116,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,91,123,114,101,115,111,117,114,99,101,44,60,60,34,47,34,62,62,44,113,117,101,117,101,44,60,60,34,108,111,103,115,34,62,62,125,93,125,125,125,44,10,32,32,32,32,32,32,32,32,32,110,111,100,101,100,111,119,110,125,93,125>>}},[{rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,462}]},{init,start_it,1,[]},{init,start_em,1,[]},{init,do_boot,3,[]}]}

 Runtime terminating during boot ({5104,{error,<<123,114,101,115,111,117,114,99,101,115,95,109,105,115,115,105,110,103,44,10,32,32,32,32,91,123,97,98,115,101,110,116,44,10,32,32,32,32,32,32,32,32,32,123,97,109,113,113,117,101,117,101,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,123,114,101,115,111,117,114,99,101,44,60,60,34,47,34,62,62,44,113,117,101,117,101,44,60,60,34,108,111,103,115,34,62,62,125,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,116,114,117,101,44,102,97,108,115,101,44,110,111,110,101,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,91,123,60,60,34,120,45,113,117,101,117,101,45,116,121,112,101,34,62,62,44,108,111,110,103,115,116,114,44,60,60,34,115,116,114,101,97,109,34,62,62,125,93,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,60,48,46,54,50,50,46,48,62,44,91,93,44,91,93,44,91,93,44,117,110,100,101,102,105,110,101,100,44,117,110,100,101,102,105,110,101,100,44,91,93,44,91,93,44,108,105,118,101,44,48,44,91,93,44,60,60,34,47,34,62,62,44,10,32,32,32,32,32,32,32,32,32,32,32,32,32,35,123,117,115,101,114,3
 Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

[michaelklishin]

This is not a bug.

Definition imports can involve a non-trivial web of inter-dependencies between different resources. Perhaps the most famous scenario is a shovel that needs a source and a destination defined (up until #11793, and to a certain extent, 3.8.6).

They may or may not be defined in the definitions, or before the definitions. Definition import has been moved to the latest possible moment more than once in RabbitMQ's history, and it's never enough.

The solution is to untangle the core of your topology (virtual hosts, queues, streams, bindings, users, permissions) and everything that depends on that topology. The core can be imported at node boot, everything else has to be imported at a later point.

[michaelklishin]

According to the logs, a stream is imported from definitions at boot time, and something forces the stream to be updated, which causes a failure:

Relevant log section

 2025-10-21 20:00:56.648738+00:00 [notice] <0.395.0> rabbit_stream_coordinator: candidate -> leader in term: 8 machine version: 5, last applied 31
 2025-10-21 20:00:56.656718+00:00 [warning] <0.395.0> rabbit_stream_coordinator failed to update stream:
 2025-10-21 20:00:56.656718+00:00 [warning] <0.395.0> function_clause
 2025-10-21 20:00:56.656718+00:00 [warning] <0.395.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 46,system_time => 1761076314434,reply_mode => noreply,machine_version => 5,...},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[115|...]},{line,1388}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_log_segment,fold0,8,[...]},{ra_log_reader,'-segment_fold/5-fun-1-',4,...},{ra_log_reader,'-segment_fold/5-lists^foldl/2-1-',...},{ra_log_reader,...}]
 2025-10-21 20:00:56.656959+00:00 [warning] <0.395.0> rabbit_stream_coordinator failed to update stream:
 2025-10-21 20:00:56.656959+00:00 [warning] <0.395.0> function_clause
 2025-10-21 20:00:56.656959+00:00 [warning] <0.395.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 61,system_time => 1761076584002,reply_mode => noreply,machine_version => 5,...},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[115|...]},{line,1388}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_log_segment,fold0,8,[...]},{ra_log_reader,'-segment_fold/5-fun-1-',4,...},{ra_log_reader,'-segment_fold/5-lists^foldl/2-1-',...},{ra_log_reader,...}]
 2025-10-21 20:00:56.657147+00:00 [warning] <0.395.0> rabbit_stream_coordinator failed to update stream:
 2025-10-21 20:00:56.657147+00:00 [warning] <0.395.0> function_clause
 2025-10-21 20:00:56.657147+00:00 [warning] <0.395.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 83,system_time => 1761076654902,reply_mode => noreply,machine_version => 5,...},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[115|...]},{line,1388}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_log_segment,fold0,8,[...]},{ra_log_reader,'-segment_fold/5-fun-1-',4,...},{ra_log_reader,'-segment_fold/5-lists^foldl/2-1-',...},{ra_log_reader,...}]
 2025-10-21 20:00:56.679905+00:00 [info] <0.594.0> Stream: __logs_1761075697741821717 will use /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/stream/__logs_1761075697741821717 for osiris log data directory
 2025-10-21 20:00:56.680060+00:00 [info] <0.570.0> rabbit_stream_coordinator: started writer __logs_1761075697741821717 on rabbit@rabbitmq in 6
 2025-10-21 20:00:56.680061+00:00 [warning] <0.595.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 13 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680082+00:00 [warning] <0.578.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 9 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680051+00:00 [warning] <0.575.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 8 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680166+00:00 [warning] <0.573.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 7 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680173+00:00 [warning] <0.583.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 11 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680229+00:00 [warning] <0.580.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 10 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.680181+00:00 [warning] <0.585.0> rabbit_stream_coordinator: failed to start writer __logs_1761075697741821717 on rabbit@rabbitmq in 12 Error: {error,{already_started,<0.594.0>}}
 2025-10-21 20:00:56.682693+00:00 [warning] <0.395.0> rabbit_stream_coordinator failed to update stream:
 2025-10-21 20:00:56.682693+00:00 [warning] <0.395.0> function_clause

[acogoluegnes]

I reproduced the issue as long as I'm on MacOS and using the volumes. The problem goes away on MacOS without the volumes. I don't hit any problem on Linux. Can you confirm you are using MacOS?

[michaelklishin]

@PaperTsar making sure that you have seen ^^^ because only you can answer this follow-up question