4.5: Qualify sorted-search and pressure behavior on Elasticsearch 9.5.5 #198

Parent

eosrio/hyperion-tools#28

What to deliver

A finite Hyperion-specific correctness and performance report for the ES9.5.5 candidate, with exact server/client/transport settings and preserved retained-history behavior.

Acceptance criteria

• Pin the test server image/digest, client and transport; inspect actual mappings/index creation versions and upgrade prerequisites. Test8.19.1client+9.5.5server only as an explicit transitional pair, and evaluate any9.xclient separately.
• Run actual sorted action-query shapes across populated/empty/timeout shards and both sort directions; verify ordering, partial/timeout signaling and upstream fixed regression cases.
• Exercise repeated failed/cancelled searches and late replies while measuring heap/breakers and post-load recovery. Include bulk pressure, item failures/acknowledgment, replicas and durable document identity.
• Collect throughput, latency distributions, CPU/heap/GC, pressure/rejections, merges/refresh/storage and real receiving-node evidence under read, write and mixed workloads.
• Classify conditional features precisely: the non-indexed/columnar keyword fix does not establish a general keyword speedup; PIT/cross-cluster and security exposure are separate applicable cases.
• Publish compatibility/upgrade findings through the release matrix. Preserve the architecture's usable-history/no mandatory full Hyperion history reindex requirement; do not automatically upgrade a live cluster.

Blocked by

None for candidate fixture/regression work. Supported release/upgrade selection remains with the release matrix; multi-ingest-node scaling requires verified routing.

Primary research

https://github.com/eosrio/hyperion-tools/blob/46f12c97374d0b89102ec14c0d661d245f69fb1e/docs/research/elasticsearch-9.5.5.md

Release context

Part of the Hyperion 4.5 component profiling program, requested on 2026-10-18. The source-backed program and first runtime evidence are in https://github.com/eosrio/hyperion-tools/blob/46f12c97374d0b89102ec14c0d661d245f69fb1e/docs/performance/hyperion-4.5-profiling.md . Existing reliability work remains in #187; retained-history upgrade and release selection remain in #190 and eosrio/hyperion-tools#20. No automatic server upgrade or runtime-support claim follows from a benchmark.

Version audit - 2026-10-18 12:03 UTC

Current tracked harness defaults differ from the 9.5.5 qualification candidate:

• History API release/4.5 at e821fdf: client 8.19.1, locked transport 8.10.1, E2E server default 9.3.1 (ES_VERSION / harness override).
• Hyperion Tools main at 0f3865925a681e60f0a3b1eab8ef90c940a35720: benchmark Elasticsearch/Kibana default 9.4.2 (ELASTIC_VERSION override).
• Both Compose defaults use version tags, not immutable image digests. Specify the candidate and digest explicitly in this issue's runs; a default-harness pass does not qualify 9.5.5. Version labels do not establish live server versions or security exposure.

The follow-up review and immutable source links are in eosrio/hyperion-tools#30 . No new actionable official release/advisory change was found; no new performance/compatibility measurements or configuration changes were made.

Execution checkpoint - 2026-10-18

Operator authorized updating both harness defaults and starting this qualification:

• API E2E Compose/ChainManager defaults: test: start Elasticsearch 9.5.5 compatibility qualification #200 . Tools benchmark Elasticsearch/Kibana defaults and example: chore: select Elasticsearch 9.5.5 for benchmark harness hyperion-tools#31 . Explicit version overrides remain supported; existing historical measurements retain their old server version.
• First executed slice: https://github.com/eosrio/hyperion-history-api/blob/298db4241d3ff3b43582d60fd0f797fe52921675/docs/performance/elasticsearch-9.5.5-initial.md . Exact source/image/runtime/transport hashes, raw reports/profiles and portable evidence checksums are linked there.
• Three unprofiled and two profiled single-node synthetic runs passed. Each acknowledged 2,500 exact documents, checked 674 searches/nine bulks through the real query helpers and completed ownership-checked cleanup. Node 24.19 default/HttpConnection and Bun 1.4.3 HttpConnection exercised client8.19.1/transport8.10.1 against ES9.5.5.
• Two failed draft allocation attempts remain preserved. The disposable Docker fixture needed absolute20/10/5GiB disk thresholds because percentage watermarks prevented shard allocation with96GiB free. No other cluster was changed.
• Initial timing finding: bulk server took=200ms sometimes exceeded client wall time below100ms on both runtimes. The clock-control checkpoint below explains the reproduced pattern; original raw values remain intact. Profiled later runs were faster than earlier controls, so collector overhead/runtime ranking is not established.

This issue remains OPEN. Broad acceptance checkboxes above are intentionally unchanged: timeout/failed/late-shard and cancellation recovery, replicas/pressure/durability, full mappings/API/plugin paths, retained-index creation/upgrade evidence and sustained controlled profiling are still required. No live production upgrade or compatibility-matrix expansion follows from this slice.

Clock-control and routing checkpoint - 2026-10-18

Source-backed report and immutable checksummed evidence: https://github.com/eosrio/hyperion-history-api/blob/a6398097cd861393b89d5fffb1ac383339ce3bf2/docs/performance/elasticsearch-9.5.5-clock.md . Raw HTTP probes on Windows Node 24.19.0 and Java 27 inside the Linux ES container each ran 100 one-document bulks per stage. Default clock interval: 3/100 and 4/100 overshoots; explicit 0 ms: zero each; restored 200 ms: 9/100 and 8/100. All 600 items succeeded. The exact server-build source uses a cached clock by default. This resolves the timing-source investigation for the reproduced pattern, without establishing a performance regression, runtime ranking or precise network-overhead estimate. Original values/warnings remain unchanged. The setting was changed only by recreating an owned disposable node; final container/volume cleanup was verified. No production tuning or automatic upgrade follows.

Ingest routing prerequisite #199 is implemented in PR #201 at 95bae91 (open, merge pending): real receiving-server identity checks, six regressions, typecheck/build and 134 unit tests pass; Node 22/24 exact-head CI is green. Sustained multi-node profiling remains separate.

Keep this issue OPEN with broad acceptance unchanged. The next correctness slice is sorted searches spanning populated, empty and failed/timed-out/late shards plus cancellation and post-failure breaker/heap recovery. Replicas/pressure/durability, full templates/API/plugins, retained-history upgrade evidence and sustained profiling also remain outstanding.

Failure, cancellation and recovery checkpoint - 2026-10-19

Implemented and independently reviewed the next finite slice:

• Qualification harness/evidence: test: qualify Elasticsearch timeout and cancellation recovery #203 . Immutable measured record: https://github.com/eosrio/hyperion-history-api/blob/2f76badff3fed51d09c303c6c46a1eb44051bd6d/docs/performance/elasticsearch-9.5.5-failures.md .
• Three final executions (Node 24.19 HTTP, Node default Undici, Bun 1.4.3 automatic HTTP) each passed 87 cases and verified 1,000 acknowledged documents plus exact four-index cleanup. Sorted precollection and collection deadlines, partial/strict shard failures, HTTP aborts, explicit tagged-task cancellation and bounded script completion after abort are covered. Five fault waves per case; all exposed task/context/current-work/request-breaker counters recovered.
• Peak sampled request-breaker charge was only 2,104 bytes, with zero trips. JVM heap/GC observations vary and do not establish leak freedom or pressure recovery. Recovery gates include observation/polling overhead and are not runtime performance comparisons.
• A separate offline check executed the exact ES 9.5.5 production ArraySearchPhaseResults and QueryPhaseResultConsumer classes: 5,429 assertions, deterministic late-after-close cases and 200 concurrent rounds per consumer. It validates reference and supplied accounting-breaker bookkeeping, not running-server heap or distributed transport ordering.
• Attempts to prove successful REST shard delivery after phase-consumer closure did not establish that order. The failed drafts and corrected source interpretation remain preserved; no green integration claim follows from them. Running partial-merge closure and that transport ordering remain open.
• The actual v1/v2 route audit discovered partial successes entering the cache and abandoned work continuing. Separate PR fix: reject incomplete action searches and cancel abandoned requests #202 rejects incomplete results with 503, maps transport timeouts to 504, preserves ES error status and propagates actual disconnect cancellation without breaking healthy POSTs. Its 18 route tests and exact-head Node 22/24 CI passed; this runtime fix is independent of the qualification PR.
• Research record follow-up: docs: record Elasticsearch failure and cancellation qualification hyperion-tools#33 . API test: start Elasticsearch 9.5.5 compatibility qualification #200/fix: route Elasticsearch writes to configured ingest nodes #201 and Tools Wrong data types formating #32 are confirmed merged. No live cluster, dependency pin or production setting changed. Final disposable container/volume removal and zero residual indices/keys/search tasks were independently verified; manifests/checksums retain all raw evidence.

Keep #198 OPEN with broad acceptance checkboxes unchanged. Next material coverage remains replicas and bulk-pressure/acknowledgment durability, large memory-pressure recovery, full API/plugin mappings, retained-index upgrade evidence, sustained profiling and distributed late-result ordering. The Hyperion 4.5 usable-history/no mandatory full history reindex constraint remains in force.

Review correction and merge checkpoint - 2026-10-19

PR #203 is merged into release/4.5 as bbf2188. Both findings in review 5566805267 were confirmed and fixed in 6096bcb: collection deadlines now require a started delay and a slow-script query timeout, while server task-cancellation counters are distinct from query timeouts. Both review threads are resolved.

Three fresh Node 24 HTTP / Node default Undici / Bun 1.4.3 automatic HTTP runs each passed 87 cases using the stricter guards. Across these reruns, 30 collection deadlines, 15 HTTP aborts with actual server task cancellation and 15 explicit task cancellations passed. Full branch tests passed 150 tests / 463 assertions; whole-project and runner typechecks, TS7 build and exact-head Node 22/24 plus native-fixture CI passed. The original evidence is unchanged and its native paths were independently rechecked from raw events. Fresh owned indices, keys, tasks, container and volume were all cleaned up.

• Measured record and review correction
• Additional checksummed evidence
• Passing PR CI

API #202 and Tools #33 are also confirmed merged. Keep #198 OPEN and the broad acceptance criteria unchanged: the sampled request charge remains only 2,104 bytes, and sustained pressure, replicas/durable bulk acknowledgment, full mappings/API/plugin paths, retained-history upgrade and distributed late-result ordering remain separate qualification work. No live cluster or dependency pins changed.

Replica, bulk-pressure and durable acknowledgment checkpoint - 2026-10-19

• Qualification PR: test: qualify Elasticsearch replicas and durable bulk acknowledgments #205 . Immutable measured record and checksummed raw evidence: https://github.com/eosrio/hyperion-history-api/blob/79bdf06976509845e027436f46833f87dcfbcac0/docs/performance/elasticsearch-9.5.5-replicas.md . Research follow-through: docs: record Elasticsearch replica and acknowledgment evidence hyperion-tools#34 .
• Three final Node 24.19 HTTP/default and Bun 1.4.3 automatic HTTP runs passed against digest-pinned ES 9.5.5, client 8.19.1 and transport 8.10.1, with two data nodes, a master witness and a dedicated RabbitMQ broker. They verified 4,569 persisted documents. Native coordinating-pressure 429 rejection was replayed with identical IDs/sources in bounded eight-document batches; matching node rejection counters increased and reservations/write work returned to baseline.
• Every pressure and broker dataset was read with exact IDs/sources/counts from both physical data-node copies before the crash and after recovery: 24 checks across the final runs. Primary SIGKILL promoted the surviving replica; restarting the original volume recovered 120 uncommitted translog operations per run with no preceding Lucene flush. Explicit request durability was used. Active-shard admission 503, valid degraded single-copy receipts and designated two-copy receipts are distinguished.
• Actual built ElasticRoutes plus real AMQP deliveries reproduced batch settlement failures: shared-channel acknowledgments could settle unrelated or unsuccessful work. Separate runtime PR fix: acknowledge only completed Elasticsearch deliveries #204 corrects per-delivery settlement, ordered response validation, success accounting/progress and recovery when the current delivery channel cannot settle. Its 31 focused tests include an actual child exit request; the fixture does not qualify whole worker/master respawn or backpressure.
• Local branch validation:211 unit tests / 1,824 assertions, application and qualification typechecks, TS7 build, four mocked Docker cleanup failure scenarios. All 69 archive members and measured source hashes independently verified. Final postflight found zero probe indices/tasks/queues/broker connections/current pressure or search/write work; four containers/four volumes/network removed with zero labelled residue. Exact-head API Node 22/24 and native-fixture CI passed: https://github.com/eosrio/hyperion-history-api/actions/runs/39227216146 . Tools research CI also passed: https://github.com/eosrio/hyperion-tools/actions/runs/39227257200 .

Keep #198 OPEN and broad acceptance criteria unchanged. This bounded admission/process-crash evidence does not establish sustained throughput, large heap/circuit-breaker recovery, broker/host power-loss durability, full worker/master integration, full production mappings/plugins or retained-history upgrade compatibility. The coordinating rejection does not reproduce the translated-update primary-expansion fault fixed by Elastic #158235; retain that exact regression as separate work. No dependency upgrade, live cluster change or automatic deployment was performed, and no full Hyperion history reindex is introduced.

Sustained load and full-worker checkpoint - 2026-10-19

The preceding three PRs are confirmed merged: API #204 (c00cf96), API #205 (a086180) and Tools #34 (60680bf10630a618b773f889b737673950310c4c).

The next finite slice is API #207, with reader prerequisite #206 and research record eosrio/hyperion-tools#35. Immutable report: https://github.com/eosrio/hyperion-history-api/blob/32bc4f8407e4dac2c9b40750ff9e9eca21b01a8e/docs/performance/elasticsearch-9.5.5-workers.md . Checksummed raw evidence: https://github.com/eosrio/hyperion-history-api/tree/32bc4f8407e4dac2c9b40750ff9e9eca21b01a8e/docs/performance/evidence/es955-workers-20261019 .

• Two fresh Node 24.19 / automatic Undici runs each passed 100 s write, 100 s read and 100 s mixed windows, plus warmup and three controlled sink-stall/recovery waves. Client8.19.1/transport8.10.1/amqplib2.0.1 and immutable ES9.5.5 image remain pinned.
• The actual production master and all twelve workers ran against owned Spring/Redis/MongoDB/RabbitMQ/three-node ES fixtures. Real SHIP traffic decoded block headers; dense synthetic actions separately entered the real action queue. No historical action/ABI, contract/state projection or served HTTP API acceptance follows.
• Control verified 48,260 actions and profiling verified 47,700: all 95,960 logical IDs/sources were independently checked on each data node. Both final source digests match within each run. Publisher confirms, one/two-copy item receipts and final copy verification remain distinct; some warmup receipts had only one successful copy.
• All six waves observed actual historical-reader pause, a pending next range with no request method invocation before release, continuing live-reader progress and dispatch of that exact pending range after resume. The proxy gates bulk request forwarding before ES dispatch; this is delayed sink completion, not native indexing-pressure saturation or a hard queue cap.
• Thirteen real CPU profiles cover master and every worker through recovery, with per-PID CPU/RSS/heap/event-loop and backend CPU/heap/GC/pressure/breaker/refresh/merge/storage data retained. The control also runs observer/proxy instrumentation. One sequential pair cannot establish collector overhead, a runtime ranking or production capacity. Native pressure/write rejections, breaker trips and merge throttling were zero; exposed accounting recovered, without proving large-memory recovery or leak freedom.
• Local validation passed 222 unit tests, app/runner Bun typechecks, TS7 build, 11 mocked fixture lifecycle cases and 87 real observer/profile/shutdown assertions. Independent evidence review verified 410 regular archive members, every reported source hash, all output digests, phase/resource summaries and cleanup identities. Failed pilots/setup/review controls remain preserved and excluded.
• Both final postflights independently verified all observed processes gone, zero broker connections/messages/unacked work and idle ES tasks/accounting. Each exact-owned seven-container/seven-volume/network fixture was removed with zero residue. No dependency upgrade, live cluster change or deployment occurred.

#206 fixes actual publisher drain/current-channel identity and pending unsent cargo continuation after replacement. The full path also exposed Windows file-URL import and log-filename prerequisites. Remaining negative/unknown confirms and historical/repair/live replay contracts are now tracked in #208 under #187, linked to #195/#196. Fresh Bun1.4.3 probes still fail the pinned uWebSockets native import on this Windows host; Node24 loads both required native modules. This is no full-worker Bun support claim.

Keep #198 OPEN and leave the broad acceptance checkboxes unchanged. Representative historical/contract/API paths, longer replicated load and overhead controls, large-memory pressure, broker/worker-loss replay, distributed late-result ordering and retained-index creation/upgrade evidence remain outstanding. Preserve existing history without a mandatory full history reindex.

Worker qualification review correction - 2026-10-19

All four findings in PR #207 review 5570640946 were confirmed and corrected in d235393: abnormal/pre-shutdown child exits now fail, readiness is subscribed before asynchronous queue initialization, profile runs over 600 steady seconds reject before resource acquisition, and AMQP connection/channel errors follow the report/cleanup path. All four threads are resolved. Validation passed 226 unit tests, 302 real observer assertions, app/runner typechecks and TS7 build; independent Spec and Standards reviews found no remaining issues. AMQP error probes use installed models and exact runner acquisition statements without a real broker handshake.

A fresh bounded 60-second steady-phase profile passed three sink-stall/recovery waves and verified all 17,440 exact action sources independently on both data nodes, all 13 post-recovery CPU profiles, master/child actual OS exits zero/no signal, three complete idle postflight samples and removal of all seven owned containers, seven volumes and network. This supplemental run is excluded from the original two five-minute totals of 95,960 actions. The original archive is unchanged; its own/child exit and explicit profile-flush records were independently audited. Original reports lacked the parent-observed master OS exit field, now captured by the stricter runner.

Correction report · checksummed supplemental evidence. Keep #198 OPEN with the broad acceptance criteria unchanged; no live settings, dependencies or deployment changed.

Final correction CI passed at d1ec03f: Node 22, Node 24 and native fixture checks. CI run. Independent supplemental evidence audit also found zero issues across all 86 archive members and 19 measured source hashes.