HDDS-15456. Add SCM DB lookup and identify orphan(wrt SCM) and deleted-but-present containers

[sreejasahithi]

What changes were proposed in this pull request?

This PR extends ozone debug datanode container analyze with optional SCM metadata lookup via --scm-db option.

When a scm.db path is provided, the command classifies each container ID found during the on-disk DataNode scan HDDS-15455 against SCM metadata and reports:

• Orphan containers (wrt SCM): present on disk but not in SCM
• Deleted but present containers: present on disk but marked DELETED in SCM

Containers in SCM with any other lifecycle state are skipped. Without --scm-db, the command prints a hint that SCM lookup is required for orphan and deleted-but-present detection and displays only the duplicate container information.

What is the link to the Apache JIRA

HDDS-15456

How was this patch tested?

Tested in docker cluster by manually creating the inconsistencies
Sample outputs:

bash-5.1$ ozone debug datanode container analyze --scm-db /tmp/scm-offline
Warning: could not determine the SCM cluster ID from the VERSION file next to /tmp/scm-offline. Cluster ID comparison with DataNode volume cluster ID was skipped. Verify --scm-db is from the same cluster as this DataNode.
Number of orphan containers(wrt SCM) on this DataNode: 2
Container 19 (2 occurrences):
  path=/data/hdds0/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/19
  status=MISSING_METADATA size=61440 bytes

  path=/data/hdds2/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/19
  status=VALID size=62006 bytes

Container 20 (1 occurrence):
  path=/data/hdds2/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/20
  status=VALID size=62006 bytes

Number of containers marked DELETED in SCM but present on disk on this DataNode: 0
Number of containers with duplicate container directories on this DataNode: 2
Container 14 (2 occurrences):
  path=/data/hdds4/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/14
  status=VALID size=62006 bytes

  path=/data/hdds5/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/14
  status=VALID size=62006 bytes

Container 19 (2 occurrences):
  path=/data/hdds0/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/19
  status=MISSING_METADATA size=61440 bytes

  path=/data/hdds2/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/19
  status=VALID size=62006 bytes

bash-5.1$

bash-5.1$ ozone debug datanode container analyze --scm-db tmp/scm-offline-deleted
Warning: could not determine the SCM cluster ID from the VERSION file next to /tmp/scm-offline-deleted. Cluster ID comparison with DataNode volume cluster ID was skipped. Verify --scm-db is from the same cluster as this DataNode.
Number of orphan containers(wrt SCM) on this DataNode: 2
Container 19 (2 occurrences):
  path=/data/hdds0/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/19
  status=MISSING_METADATA size=61440 bytes

  path=/data/hdds2/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/19
  status=VALID size=62006 bytes

Container 20 (1 occurrence):
  path=/data/hdds2/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/20
  status=VALID size=62006 bytes

Number of containers marked DELETED in SCM but present on disk on this DataNode: 2
Container 7 (1 occurrence):
  path=/data/hdds4/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/7
  status=VALID size=62003 bytes

Container 14 (1 occurrence):
  path=/data/hdds4/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/14
  status=VALID size=62006 bytes

Number of containers with duplicate container directories on this DataNode: 1
Container 19 (2 occurrences):
  path=/data/hdds0/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/19
  status=MISSING_METADATA size=61440 bytes

  path=/data/hdds2/hdds/CID-fd35e799-8edf-4d5d-bda8-2899ca3d4593/current/containerDir0/19
  status=VALID size=62006 bytes

[sreejasahithi]

@devmadhuu could you please review this PR.

[errose28]

cc @rnblough

[devmadhuu]

Thanks @sreejasahithi for the patch. Largely looks good. Some review comments, Pls check.

[devmadhuu]

@sreejasahithi PR changes largely looks good, few comments in code inline and some nits below for better tool reliability:

1. Can do a simple sanity check for clusterID between datanode VERSION filesand scm DB clusterId. This will help user to avoid any accidental use of some other cluster SCM DB and avoid large number of invalid orphan container reporting.
2. We have changed from --count to --length option now , so if any existing code may break , may need to rethink from backward compatibility point of view.

[sreejasahithi]

@sreejasahithi PR changes largely looks good, few comments in code inline and some nits below for better tool reliability:

1. Can do a simple sanity check for clusterID between datanode VERSION filesand scm DB clusterId. This will help user to avoid any accidental use of some other cluster SCM DB and avoid large number of invalid orphan container reporting.
2. We have changed from --count to --length option now , so if any existing code may break , may need to rethink from backward compatibility point of view.

Thanks for the suggestions. I added a cluster ID check between the DataNode VERSION file and the SCM DB cluster ID on mismatch we print a warning and skip orphan/deleted reporting.
For --count, I kept it as a hidden deprecated option for backward compatibility, --length is the new default via ListLimitOptions.

[devmadhuu]

Thanks @sreejasahithi for improving the patch. Largely looks good to me. Just one nit. Please check.

[devmadhuu]

What if scmDb != null but path not correct, so both above if and else if will skip and code just continues to printDuplicates and return null, i.e. exits successfully.