ECK: Document recovery from failed volume upsize (#3694)

In elastic/cloud-on-k8s#4467 it's noted that
some users are dealing with volume expansion failure issues, and
documenting how to recover from this situation would be helpful. This is
the attempt to update that documentation.

---------

Signed-off-by: Michael Montgomery <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Peter Brachwitz <[email protected]>

Author: 3 people

troubleshoot/deployments/cloud-on-k8s/common-problems.md

@@ -306,3 +306,14 @@ Check that the role mapping was deleted:
 GET /_security/role_mapping/<roleName>
 {}
 ```
+
+
+## Volume expansion failed [k8s-common-problems-volume-failed-expansion]
+
+If you attempted an expansion of an {{es}} data volume via its [volume claim template](/deploy-manage/deploy/cloud-on-k8s/volume-claim-templates.md#k8s-volume-claim-templates-update), you may have encountered scenarios where the operation failed. For example older versions of the Azure Disk CSI driver did not allow volume expansion without shutting down the Virtual Machine to which the volume was attached. If you try to adjust the volume claim template back to the original size you will encounter an error:
+
+```
+Failed to apply spec change: handle volume expansion: decreasing storage size is not supported: an attempt was made to decrease storage size for claim elasticsearch-data
+```
+
+In this scenario the best course of action is to rename the existing `nodeSet` to a new name while simultaneously updating the volume claim template to the original size. This operation will bring a new `StatefulSet` online while moving all existing data to the new volumes, and will delete the old `StatefulSet` and its volumes once the operation is complete.