Recommend a full cluster restart for upgrade

Author: rdettai

docs/operating/upgrades.md

@@ -1,5 +1,5 @@
 ---
-title: Version 0.7 upgrade
+title: Version upgrade
 sidebar_position: 4
 ---
 
@@ -23,4 +23,22 @@ No migration is done if `otel-traces-v0_7` already exists. If you want `service_
 
 Quickwit 0.9 introduces a new ingestion service to to power the ingest and bulk APIs (v2). The new ingest is enabled and used by default, even though the legacy one (v1) remains enabled to finish indexing residual data in the legacy write ahead logs. Note that `ingest_api.max_queue_disk_usage` is enforced on both ingest versions separately, which means that the cumulated disk usage might be up to twice this limit.
 
-The control plane should be upgraded first in order to enable the new ingest source (v2) on all existing indexes. Ingested data into previously existing indexes on upgraded indexer nodes will not be picked by the indexing pipelines until the control plane is upgraded. Because the indexing plan is computed differently in 0.9, all pipelines will be restarted when upgrading the control plane. If possible, we recommend avoiding rolling upgrades for indexers. Instead, scale down the number of indexers to zero first, then upgrade the control plane and finally scale the upgraded indexers back up. Finally, if you intend to enable compression for the ingest service (`ingest_api.grpc_compression_algorithm`), you must do so in two steps: first, upgrade the indexer nodes with compression disabled, then update the node configuration to enable compression, and finally restart the indexer nodes.
+When upgrading to 0.9, we recommend to perform a full cluster restart.
+
+<!--
+Reasons:
+- Ingested data into previously existing indexes on upgraded indexer nodes will not be picked by the indexing pipelines until the control plane is upgraded.
+- The indexing plan is computed differently in 0.9, all pipelines will be restarted when upgrading the control plane.
+- If you intend to enable compression for the ingest service (`ingest_api.grpc_compression_algorithm`), you must do so in two steps: first, upgrade the indexer nodes with compression disabled, then update the node configuration to enable compression, and finally restart the indexer nodes.
+- Obscure bug raised in https://github.com/quickwit-oss/quickwit/issues/5787#issuecomment-2979470315
+-->
+
+Shutdown order:
+1) indexers, searchers and janitor
+2) control plane
+3) metastores
+
+Start up order:
+1) metastores
+2) control plane
+3) indexers, searchers and janitor