HDDS-14351. Add preStop and graceful termination for OM and SCM

[shuan1026]

What changes were proposed in this pull request?

Follow-up to PR #20 discussion: add a preStop hook / longer graceful termination so OM/SCM do not stop while still holding Ratis leadership.

• Add lifecycle.preStop on OM/SCM StatefulSets (replicas > 1): if this pod is leader, run ozone admin om|scm transfer -r before shutdown.
• Add terminationGracePeriodSeconds for HA OM/SCM (default path used 150s) so a slow transfer + short sleep still finishes before SIGKILL.
• The preStop script only does the leader transfer; it does not send kill -TERM 1 as in the PR #20 discussion sample. kubelet sends SIGTERM to the container automatically once preStop returns, and the image's PID 1 (dumb-init) forwards it to the OM/SCM process, so an explicit kill would just duplicate that and adds a dependency on the entrypoint's PID 1.
• Complements the existing pre-upgrade om-leader-transfer-job (scale-down + persistence); this covers ordinary pod delete / roll / drain.
• Set publishNotReadyAddresses: true on the OM/SCM headless Services. During termination a terminating endpoint always has ready: false, so by default it is dropped from the headless Service DNS (Kubernetes docs). The transferee then cannot resolve the outgoing leader's address to confirm the handoff, and transfer -r blocks until TIMED_OUT(60s). Publishing not-ready addresses keeps the terminating pod resolvable so the transfer completes quickly (see discussion).

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14351

How was this patch tested?

• Green CI run, and tested manually with leader transfer and basic ozone commands.

• kind, OM HA (om.replicas=3): over repeated leader pod-delete cycles, preStop transfers leadership and quorum consistently recovers to a single LEADER.
• kind, SCM HA (scm.replicas=3): same repeated leader pod-delete cycles, with the same result.
• Compared with main (both OM and SCM): no preStop / default 30s grace; deleting the leader does not attempt transfer (failover is only Raft election after SIGTERM).

Manually Test:

kind create cluster --name ozone-helm-prestop

# OM HA (default)
helm upgrade --install ozone charts/ozone --wait --timeout 15m
kubectl exec ozone-om-0 -c om -- ozone admin om roles -id cluster1
kubectl delete pod ozone-om-1                       # delete current LEADER
kubectl get events --sort-by='.lastTimestamp' | grep -iE 'preStop|Killing'
kubectl exec ozone-om-0 -c om -- ozone admin om roles -id cluster1
# expect: single LEADER; transfer done in preStop (no TIMED_OUT)

# SCM HA
helm upgrade --install ozone charts/ozone --set scm.replicas=3 --wait --timeout 20m
kubectl exec ozone-scm-0 -c scm -- ozone admin scm roles -id cluster1
kubectl delete pod ozone-scm-1                       # delete current LEADER
kubectl get events --sort-by='.lastTimestamp' | grep -iE 'preStop|Killing'
kubectl exec ozone-scm-0 -c scm -- ozone admin scm roles -id cluster1
# expect: single LEADER; transfer done in preStop

[shuan1026]

PTAL @adoroszlai @chungen0126 Thank you!