Stop the world

[silentsokolov]

Hey all

We have been using version 8.1 of the product for a year, but decided to upgrade (qw-airmail-20250522-hotfix) in order to fix issue #5819. After the upgrade, we started noticing strange behavior let’s call it a “stop the world”. During this time, the cluster stops accepting and indexing documents. According to the metrics, it doesn’t seem to be doing anything at all

From issue #4925, I understand that this might be related to merge splits, but according to the metrics, the number of merges doesn’t increase, memory usage stays the same, it’s just a complete “stop the world” situation. At the same time, the cluster is healthy, all nodes are fine - they just don’t do anything

Could you please explain what might be causing this and how we can avoid it?

Settings

          indexer:
            split_store_max_num_bytes: 100G
            split_store_max_num_splits: 5000
            cpu_capacity: 12
            merge_concurrency: 8

          ingest_api:
            max_queue_memory_usage: 32GB
            max_queue_disk_usage: 48GB

The indexes (5) have default policy merge

[fmassot]

Hi @silentsokolov, thanks for the report. I saw your message on discord, let me dig into that.