Index status / Report unreadable cluster health responses

[juanluisrp]

Index status / Report unreadable cluster health responses

Follow up of #9477.

Problem

EsServerStatusChecker reads the cluster health through the typed Elasticsearch client:

HealthResponse response = client.cluster().health();

The typed client only decodes the health response of the server version it is built for. The client bundled since 4.4.10 (8.19.13) has unassigned_primary_shards as a required property of the health response, and that property was only added to _cluster/health in Elasticsearch 8.16. Against an older server the call fails with:

node: http://host:9200/, status: 200, [es/cluster.health] Failed to decode response

The request succeeded (status: 200), only the decoding failed, but the single catch (Exception e) in the checker turns everything into:

uninitialized
Unable to revive connection to http://host:9200. Error is ...

So the catalogue permanently displays a connection banner while the server is reachable. The message sends the administrator looking at the network, the firewall and CORS instead of at the version of the index server.

Reading the cluster health (first commit)

• EsRestClient.getServerStatus() falls back to a low level GET /_cluster/health when the typed client can not decode the response, and reads the status property from the raw JSON. The reported state then matches the state of the cluster. The fallback uses the same RestClient instance that backs the typed transport, so the es.username / es.password credentials and the https settings are the ones already configured.
• The decoding error is reported once, since the check runs every five seconds by default (es.index.checker.interval), and the message names the client version and points at the compatibility of the index server. It is logged as an error because the default log configuration only logs errors for geonetwork.index.
• If the fallback fails too, EsServerStatusChecker reports the decoding failure separately from a connection failure, again pointing at the version of the index server. The original error is kept as a suppressed exception.

This does not add support for unsupported Elasticsearch versions, it only makes the index status and the log say what is actually wrong.

Reporting the version of the index server (second commit)

The version of the server is only visible in the site information page, so nothing tells an administrator that the server is not the one this GeoNetwork is built for. EsServerStatusChecker now compares the major version of the server with the version of the client once the server can be reached, and reports a difference once:

ERROR [geonetwork.index] - Index server at http://localhost:9200 is Elasticsearch 7.17.15 but this GeoNetwork
version is built with the Elasticsearch client 8.19.13. Only Elasticsearch 8.x is supported, check the
installation guide. Running another version leads to errors which are not always reported as a version issue.

The version is read again on the next run when the server can not be reached, so a catalogue started before its index server still reports it. Nothing is logged when the versions match, and the state of the index is not changed by this check.

Tests

EsServerStatusCheckerTest covers the status branches (the cluster status is reported as is, a connection failure is still reported as uninitialized with the "Unable to revive connection" message, a response the client can not decode is reported with a message pointing at the server version) and the version check (reported for another major version, silent for the version of the client and for a version which can not be parsed, read once, read again when the server can not be reached).

The index module had no test sources yet, junit is inherited from the root pom so no dependency change was needed.

Verified against Elasticsearch 7.17.15

Before, the home page shows the banner of #9477 and /api/site/index/status returns:

{"id":"index","state":{"id":"uninitialized", ...},
 "message":"Unable to revive connection to http://localhost:9200. Error is node: http://localhost:9200/, status: 200, [es/cluster.health] Failed to decode response"}

After, no banner and the real state of the cluster:

{"id":"index","state":{"id":"yellow", ...},"message":"Index status is yellow. Check index server log."}

with a single line in the log:

ERROR [geonetwork.index] - Failed to decode the cluster health response returned by http://localhost:9200 using the
Elasticsearch client 8.19.13. Check that the index server version is compatible with this GeoNetwork version.
Reading the cluster status using the low level client. Error is node: http://localhost:9200/, status: 200,
[es/cluster.health] Failed to decode response.

Searching still fails on that server, with the error of the incompatibility that is really there (No mapping found for [createDate] in order to sort on), instead of a connection error on the home page.

The version check was verified the same way: against 7.17.15 the message above is logged once at startup, against 8.19.13 nothing is logged.

Checked with and without credentials on a 7.17.15 server with xpack.security.enabled=true: the fallback authenticates like any other call, and a missing password is still reported as before, missing authentication credentials for REST request [/_cluster/health].

Checklist

• mvn -pl index test passes (8 tests)
• No new checkstyle violations in the module