Narrow a transaction's ordered index reads for strings from U+D800 and TEXTSTRING keys by porunov · Pull Request #5085 · JanusGraph/janusgraph

porunov · 2026-10-15T16:33:30Z

An ordered graph query which a mixed index sorts, in a transaction whose new or changed elements fill the query's limit, reads from the index only the elements whose value of the first order key doesn't come after that of the last of them within the limit (#5023). For a string, it took that range only below U+D800 and only for a key mapped as STRING. Since #5067, memory orders strings by code point, as the indexes do, and since #5069, the ranges of a TEXTSTRING key compare its whole strings on every index backend. This PR takes the range for any string without a surrogate out of its pair, and for a TEXTSTRING key as well, without a string or text analyzer either way. After a transaction renamed 20,000 of 50,000 vertices to names from U+E000, order().by("name", desc).limit(10) runs one Elasticsearch search in 90 ms, instead of 155 in 1.2 s.

It also fixes failures of #5023's ranges from long values:

JanusGraph's Lucene backend, OpenSearch 2 and Elasticsearch 6 to 8 can't build a range from a lower bound of more than 1,000 UTF-8 bytes. So a descending query whose last new value was longer failed with "Could not call index", on master for a STRING key below U+D800.
Solr takes a query in a URL of at most 8 KB, as JanusGraph sends its queries to Solr as GET requests. So a query in either direction failed on Solr whose value had about 8 KB, or 2.7 KB of characters of three UTF-8 bytes.

A descending bound is now cut to its longest prefix of whole characters within 1,000 bytes, and no ascending range is taken from a value of more than 1,000 bytes.

A surrogate without its pair isn't valid UTF-16, and no index holds it as it is: Lucene, Solr, OpenSearch and Elasticsearch up to 8 hold U+FFFD, and Elasticsearch 9.5 holds a question mark, while memory places it after U+FFFF. So a descending range of a string with a character from U+FFFD up, such as an emoji, starts at U+FFFD in place of that character and what follows it. Where the index holds U+FFFD, a descending query then returns what it returns without the range, also where the index holds such strings. On Elasticsearch 9.5, a descending range from nearly any string may leave such a stored string out, as master's ranges of strings below U+D800 already may.

Changes

StandardJanusGraphTx.isRangedAsOrdered takes a range for a string without a surrogate out of its pair (hasUnpairedSurrogate, new), in place of one below U+D800 (isBelowTheSurrogates). Memory and the indexes compare valid strings alike: by code point, the order of their UTF-8 bytes.
isComparedWhole takes a field mapped as TEXTSTRING as well as STRING. A TEXTSTRING key's ranges and its order read the field of its whole strings on each backend:
- Lucene: the dual field <key>_____s. The keyword analyzer makes its one term the whole string, and its sort doc values hold the string as it is. Its ranges read it since Compare the whole strings of a TEXTSTRING key in ranges and not-equals #5069.
- Elasticsearch and OpenSearch: <key>__STRING, a keyword field. Its ranges read it since Compare the whole strings of a TEXTSTRING key in ranges and not-equals #5069.
- Solr: with dynamic fields (dyn-fields, the default), <key>_s, the field of a STRING key of the same name, or <mapped name>_s. The Solr schema defines it: the schema JanusGraph ships makes *_s a StrField, which keeps the string whole. Its ranges always read it; Compare the whole strings of a TEXTSTRING key in ranges and not-equals #5069 changed only its not-equal.
- The order reads that field on all three.
- A string analyzer may give that field other terms than the whole string, so a key with one still takes no range. A text analyzer shapes the key's words rather than that field, but a key with one is left out as well, out of caution, as before.
rangeBound (new) gives the bound of the range, or none:
- The value itself in an ascending order, and for a value which isn't a string.
- A cut at U+FFFD. In a descending order, a string with a character from U+FFFD up is cut before the first such character, and U+FFFD follows.
  - Storage keeps a string of up to 16,000 UTF-16 units as it was written, with its surrogate without a pair.
  - An index which holds U+FFFD compares such a stored string with a valid value as memory does, with one exception. The stored string has the surrogate at the first unit in which the two differ, and the value has a character from U+FFFD up there. Then memory may place the stored string above the value, while the index places it at or below. An exhaustive check of all strings of up to three of 13 chosen units, and the new randomized unit test, find no other case.
  - An ascending range, at most the value, holds such a string, which comes after the value anyway.
  - A descending range from the value itself could leave it out, though it comes before the value. The index holds it with the value's units up to the first character from U+FFFD up, and U+FFFD or more in its place, so it never comes below the cut bound.
  - Elasticsearch 9.5 holds such a surrogate as a question mark, below the letters. A bound which held such strings there would start at a question mark in place of the value's first character from a question mark up, and so read nearly the whole index. rangeBound doesn't try.
- At most 1,000 bytes.
  - JanusGraph's Lucene backend, OpenSearch 2 and Elasticsearch 6 to 8 build a range from a lower bound up to the greatest string by following its bytes one by one. They fail beyond a depth of 1,000 bytes with IllegalArgumentException: input automaton is too large: 1001 (Lucene 9's Operations.isFinite).
  - Solr builds string ranges without that limit (SolrRangeQuery), but JanusGraph sends its queries to Solr as GET requests, and Solr 9.10 takes at most 8 KB of request line and headers. ClientUtils.escapeQueryChars and the URL make at most six characters of a byte, so a bound of 1,000 bytes takes at most 6,000.
  - So a descending bound is cut further, to its longest prefix of whole characters of at most 1,000 UTF-8 bytes. A prefix is a lower bound still, so the range holds all that the range from the whole value would.
  - An upper bound can't be cut, so no ascending range is taken from a value of more than 1,000 UTF-8 bytes. That also avoids the one failure of an upper bound on Lucene 9, which follows its bytes only while they are NUL: 1,000 NUL characters followed by more.
QueryExecutor.withoutResultsAfter: its javadoc says that the narrowed sub-query may keep some results after the value, those up to a bound after it.
Docs: the "Queries See the Transaction's Changes" section of the transactions page says that direct index queries (graph.indexQuery() or a transaction's indexQuery()) read the index alone. They leave out the elements which the transaction removed, but its other changes don't change what they find.
- IndexQueryBuilder.execute reads the index, and leaves out only the ids which turn into a removed element or into none.
- executeTotals counts what the index holds.
Changelog entry, after the one of Find the vertices a transaction changed for its graph queries by key and value #5023.

Tests

StandardJanusGraphTxOrderRangeTest (janusgraph-test, 11 tests, 6 before):
- takesARangeForAStringOfValidUtf16 (new): U+D7FF, U+E000, U+FFFD, U+FFFF, U+1F600, U+10000, U+10FFFF and strings of them, on STRING and TEXTSTRING fields, both ways.
- takesNoRangeForAStringWithASurrogateOutOfItsPair: a lone high or low surrogate, at the end or before another character, a reversed pair, and a high surrogate before a pair.
- takesARangeForAStringOnlyOnAFieldWhichComparesItWhole: STRING and TEXTSTRING, neither with a string or text analyzer, and not TEXT or the default mapping.
- cutsTheDescendingBoundOfAStringBeforeItsFirstCharacterFromUFFFDUp (new): the bounds of both directions.
- cutsTheDescendingBoundOfALongStringToItsFirstThousandUtf8Bytes (new): 1,001 w, 400 U+4E2D and 400 U+E000, a two-byte character which would end at byte 1,001, and the U+FFFD in place of an emoji at either side of the limit.
- takesNoAscendingRangeFromAStringOfMoreThanAThousandUtf8Bytes (new): 1,000 w, 333 U+4E2D and 996 a followed by an emoji take an ascending range whole; 1,001 w, 334 U+4E2D, 999 a followed by a two-byte character, and 997 a followed by an emoji take none.
- theRangeHoldsEveryStoredStringWhichMemoryOrdersUpToTheValue (new): 100,000 random pairs of a value and a stored string, of characters from both sides of the surrogates and of surrogates on their own. Wherever OrderList places the stored string at or before the value, a model of an index which holds U+FFFD places it within the range from the bound: UTF-8 bytes, with U+FFFD for a surrogate without its pair.
JanusGraphIndexTest, on every index backend. The mixed indexes are named mixed, which the Solr test container creates.
- testTransactionReadsTheIndexOnlyUpToItsLastNewValueWithinTheLimit: its value which an index may compare otherwise is now a lone U+D800, as U+E000 takes the range (see the next test), and an emoji's descending range starts at U+FFFD. The results are as before.
- testTransactionReadsTheIndexUpToAStringBeyondTheSurrogatesInCodePointOrder (new): stored U+FFFF and U+1F601, new U+E000 and U+1F600.
  - The ascending range up to U+1F600 holds U+FFFF, which comes below it by code point and above it by UTF-16 unit.
  - The descending range from U+E000 holds both emoji.
  - The descending range of U+1F600 starts at U+FFFD.
- testTransactionRangeKeepsTheResultsOfADescendingOrderWithAStoredUnpairedSurrogate (new): stored a, a lone U+DBFF, and a lone U+DC00 followed by b. With U+FFFE, and with U+1F600, added in a transaction, the descending range of limit(1) starts at U+FFFD, and limit(1) returns what the query without a limit returns first.
  - On Lucene that is U+DC00 b, which memory places before both new strings.
  - On Elasticsearch 9.5 it is the new string, as the index holds the stored surrogates as question marks, below a.
  - The test holds whichever of the two a backend does.
- testTransactionRangesFromALastNewValueOfMoreThanAThousandBytes (new): a STRING key and a TEXTSTRING key, a stored value of 1,000 w followed by x, and a stored a. The transaction adds a value of 1,000 w, of 1,001 w, of 400 U+4E2D or of 400 U+E000.
  - Both ranges from 1,000 w take it whole.
  - From the longer values, the descending range starts at their first 1,000 bytes, and no ascending range is taken.
  - limit(1) in either direction returns what the query without a limit returns first.
- testTransactionReadsTheIndexOfADualMappedKeyOnlyUpToItsLastNewValueWithinTheLimit (new, where TEXTSTRING is supported): titles of several words in mixed case. The ranges up to Long Zed and zz top 2, and from zz top, are taken and give the right titles. A range of the text field's lower-cased words would find no stored title up to Long Zed, and leave out ~Tilde Title from zz top.
- testTransactionReadsTheIndexWholeWhereItMayCompareOtherwise: its TEXTSTRING case moves to the new test above. A new value of a lone U+D800 takes no range either way, and the results are right.
- testTransactionQueriesAgreeWithAScanOfTheirElements: the strings now include U+E000, U+FFFD, U+FFFF and emoji, and a new TEXTSTRING order key holds words in mixed case, also from both sides of the surrogates. The scan orders strings by code point. It runs 20 seeds, as before.
Mutation checks, against StandardJanusGraphTxOrderRangeTest and the Lucene index tests:
- Master's code (its guard, neither cut, and ascending ranges of any length): five unit tests fail, and five Lucene tests on the missing range. Among them, the long-value test fails at its first TEXTSTRING query.
- No cut at 1,000 bytes: the cut unit test fails, and on Lucene the long-value test fails with "Could not call index", as master's descending range from 1,001 w does.
- A cut at 1,001 bytes, or at 1,000 UTF-16 units rather than bytes: the cut and the ascending unit tests fail, and on Lucene the long-value test fails with "Could not call index".
- Ascending ranges of any length: the ascending unit test fails, and on Lucene the long-value test finds an ascending range from 1,001 w. On Solr, a range from 8 KB fails, as above.
- No cut at U+FFFD:
  - three unit tests fail;
  - the random pairs find a stored lone U+DE00 followed by U+D7FF outside the descending range from U+10000 followed by a;
  - on Lucene, three index tests fail on the profile. Without that check, limit(1) of the stored-surrogate test returns U+FFFE where the query without a limit returns U+DC00 b first.
- A cut at U+FFFD to U+FFFF but not from U+10000: the same three unit tests fail. On Lucene, the stored-surrogate test returns U+1F600 where the query without a limit returns U+DC00 b first.
- Master's guard with this PR's cuts: two unit tests fail, as U+E000 and TEXTSTRING take no range, and five Lucene tests on the missing range.
- TEXTSTRING left out: two unit tests fail, and two Lucene tests on the TEXTSTRING key's missing range: the dual-mapped test and the long-value test.
- A high surrogate taken as paired whatever follows it:
  - takesNoRange... fails on U+D800;
  - the random pairs find a stored U+FFFF U+FFFD U+DBFF outside the ascending range up to U+DBFF U+FFFD U+FFFF, whose U+DBFF has no pair;
  - on Lucene, ...WhereItMayCompareOtherwise and ...OnlyUpToItsLastNewValueWithinTheLimit take a range from the lone U+D800.
- Analyzers not checked: takesARangeForAStringOnlyOnAFieldWhichComparesItWhole fails.
The randomized check with 400 seeds (locally, not committed), on Lucene and on Elasticsearch 9.5:
- 8,000 checks on each, all of which agree with the scan.
- 227 of the queries took a string range, 99 of them from a string beyond U+D800:
  - 63 of the STRING key: 36 cut descending, 5 descending uncut and 22 ascending;
  - 36 of the TEXTSTRING key: 28 cut descending and 8 ascending.

Verification

Locally, Java 11, on master 3285cbf18, with checkstyle (no violations):

janusgraph-driver (130), janusgraph-core (228), janusgraph-inmemory (650, 85 skipped), janusgraph-test (1,107), janusgraph-berkeleyje (349, 9 skipped) and janusgraph-lucene (397, 148 skipped; BerkeleyLuceneTest 108) pass.
BerkeleyElasticsearchTest (121, 1 skipped) passes against Elasticsearch 9.5.4 in a container.
janusgraph-solr compiles, with its tests. Solr is left to CI, as its test container doesn't start on this machine. SolrJ sends documents in javabin (BinaryRequestWriter is the default of HttpSolrClient and CloudLegacySolrClient), whose encoder writes U+FFFD for a surrogate without its pair, as Lucene's BytesRef does.
How the backends hold a surrogate without its pair, probed with curl: a keyword field of Elasticsearch 9.5.4 holds it as a question mark, and a keyword field of Elasticsearch 6.6, 7.17 and 8.15, and of OpenSearch 2.19 and 3.9, as U+FFFD. JanusGraph's Jackson writes every surrogate as a \u escape.
The limits on long bounds, probed likewise:
- Elasticsearch 6.6, 7.17 and 8.15, and OpenSearch 2.19, fail on a gte range from 1,001 bytes, and build an lte range up to 1,001 bytes.
- Elasticsearch 9.5.4 and OpenSearch 3.9 build both.
- Lucene 9.12.3 builds gte up to 1,000 bytes, for characters of one to four bytes, and lte up to 40,000 bytes, but not up to 1,000 NUL characters followed by more.
- Solr 9.10.1 builds a name_s range from 7,000 ASCII characters in either direction. A GET request with one from 8,200 fails with 414 URI Too Long, and one from 900 U+4E2D with 431 Request Header Fields Too Large; a POST takes one from 20,000.

Measurements

Elasticsearch 9.5.4, a single node in Docker, inmemory storage. A mixed index holds kind, name (STRING) and title (TEXTSTRING) of 50,000 vertices of kind k, named n00000 to n49999 and titled Title 00000 to Title 49999. A transaction renames, or retitles, the 20,000 vertices with the highest names, and g.V().has("kind", "k").order().by(key, desc).limit(10) runs in it.

How it was measured:

Each query has a transaction of its own, so that none reads another's results from the transaction's cache.
Searches are the query_total of the graph's indexes.
The times are medians of 9 queries, after 2 to warm up, in two passes of reversed order (master, branch, branch, master), with only the core jar differing.
The branch was an earlier version of this PR, without the limit of 1,000 bytes, which these short values don't reach.

New values	master	branch
names: U+E000, a space and the number	1,221 ms and 1,222 ms, 155 searches	94 ms and 89 ms, 1 search
names: U+1F600, a space and the number	1,184 ms and 1,210 ms, 155 searches	89 ms and 88 ms, 1 search
titles (`TEXTSTRING`): `zz`, a space and the number	1,205 ms and 1,214 ms, 155 searches	89 ms and 88 ms, 1 search
names: `zz`, a space and the number (control)	88 ms and 86 ms, 1 search	88 ms and 85 ms, 1 search

Master: the profiles show no range, and twelve executions of the index query, under limits doubling from 15 to 30,720, as the index has the 20,000 changed vertices first.
Branch: the profiles show one execution, with name >= U+E000 49990, name >= U+FFFD (cut from U+1F600 49990) and title >= zz 49990.
Results: both sides return the same ten vertices in every case.
One search: the control, which takes one search on both sides, takes 85 to 88 ms, as long as the branch's queries.

Notes

Behaviour: where the range is newly taken, a query returns what it returned before, and reads less of the index, as long as the strings which the index holds are valid UTF-16. A query whose last new value has more than 1,000 UTF-8 bytes no longer fails: a descending one on JanusGraph's Lucene backend, OpenSearch 2 and Elasticsearch 6 to 8, and one in either direction on Solr beyond its 8 KB. An ascending one from such a value reads the index whole again. A last new value with a surrogate without its pair, and a key with a string or text analyzer, still read the index whole too.
Stored strings with a surrogate without its pair: the index places them by what it holds, while memory places the transaction's new elements by its own order, as since Order the strings of graph queries by code point, as the indexes do #5067. For the range:
- Where the index holds U+FFFD, the cut keeps a descending range from leaving out such a string.
- On Elasticsearch 9.5, a descending range from nearly any string may leave such a string out, now also from strings beyond U+D800. Master's ranges of strings below U+D800 already do so: with a stored lone U+DC00 followed by b, and z added in a transaction, master's order().by("name", desc).limit(1) takes the range name >= z and returns z, while the query without a limit returns U+DC00 b first. On Lucene, master returns U+DC00 b.
- Where the index holds U+FFFD, an ascending range may leave out a stored string of more than 16,000 UTF-16 units which storage holds with a question mark in place of such a surrogate, as it does where it compresses the string from its UTF-8 bytes. Master's ranges below U+D800 already do so: with a stored lone U+D800 followed by 16,000 x, and a added in a transaction, master's order().by("name", asc).limit(1) on Lucene takes name <= a and returns a, while the query without a limit returns the stored string, read back as ?xx…, first. Keep the unpaired surrogates of strings longer than 16,000 characters #5098 makes storage keep such surrogates in long strings. With it, this case remains only for strings written before it.
Solr custom schemas: a schema which makes the field of a key's whole strings analyzed breaks the range on Solr, for a TEXTSTRING key as for a STRING key since Find the vertices a transaction changed for its graph queries by key and value #5023, and the order of either. The test schema does so for full_text_s, which belongs to a key with English string and text analyzers, so it takes no range. The changelog says so too.
Left out:
- A TEXTSTRING key with a text analyzer only could take the range too, as that analyzer doesn't shape the field of its whole strings on Lucene or Elasticsearch. It is left out as before.
- Querying Solr by POST would take bounds and other values longer than 8 KB. That is a change of its own.
Cost: checking the value for a surrogate without its pair, cutting a descending bound, and measuring an ascending one walk the value's characters once per narrowed query.

🤖 Generated with Claude Code

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open findings

codePoints().toArray() allocates two arrays per string compare, which can become a significant… · New
The descending-path scan walks the full string and may allocate a very large intermediate… · New
This test performs 100,000 iterations and repeatedly encodes strings to UTF-8 bytes… · New
The comment says “The values in their UTF-16 units”, but the implementation operates on… · New

What changed in this PR

This PR improves how JanusGraph narrows mixed-index reads for ordered traversals inside a transaction when in-tx mutations fill the query limit, expanding support to more valid UTF-16 strings (including beyond U+D800) and TEXTSTRING mappings, while also preventing backend failures from overly long string range bounds.

Changes:

Extend ordered-range narrowing eligibility from STRING to TEXTSTRING (when whole-string comparison is safe) and from “below U+D800” to any valid UTF-16 without unpaired surrogates.
Introduce rangeBound(...) to safely derive range bounds (cuts descending bounds at U+FFFD and enforces a 1000 UTF-8 byte limit; disables ascending bounds beyond the limit).
Add/expand unit + backend tests and update docs/changelog to describe the new behavior and limitations.

File	Description
janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/StandardJanusGraphTx.java	Core logic changes: broaden ranged-string eligibility and introduce bounded/safe range bound derivation.
janusgraph-core/src/main/java/org/janusgraph/graphdb/query/QueryExecutor.java	Javadoc clarifies that narrowing may use a bound beyond the provided value in some cases.
janusgraph-test/src/test/java/org/janusgraph/graphdb/transaction/StandardJanusGraphTxOrderRangeTest.java	New/expanded unit tests for valid UTF-16 strings, surrogate handling, and bound-cutting behavior.
janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphIndexTest.java	New/updated integration tests validating range narrowing across backends and new TEXTSTRING behavior.
docs/changelog.md + docs/basics/transactions.md	Document behavior changes, limitations, and direct index query semantics.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open findings

This comparator allocates two new int[] arrays per string comparison (codePoints().toArray()),… · New
This randomized test does 100k iterations and, in many cases, performs UTF-8 encoding work… · New
ByteBuffer.array() is not guaranteed to be supported for all ByteBuffer implementations (it can… · New
rangeBound(value, descending) is computed before checking indexSerializer.supports(...). Since… · New
The statement “a byte which a URL escapes takes three characters” appears to understate worst-case… · New

4 resolved since last review

🧠 Review effort: Lite

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

This comparator allocates two int arrays per string comparison (codePoints().toArray()), which… · New
The javadoc contradicts itself: it says 'a string without a surrogate out of its pair' (which is… · New
This comment claims the descending bound becomes 'prefix + U+FFFD' whenever there is a character >=… · New

5 resolved since last review

🧠 Review effort: Lite

…d TEXTSTRING keys Fixes JanusGraph#5084 An ordered graph query in a transaction whose new elements fill its limit reads from the mixed index only the elements whose value of the first order key doesn't come after that of the last of them within the limit, where every index compares that value as memory orders it. For a string, StandardJanusGraphTx.isRangedAsOrdered took the range only below U+D800 and for a key mapped as STRING. Since JanusGraph#5067, memory orders strings by code point, as Elasticsearch, OpenSearch, Lucene and Solr do, and since JanusGraph#5069, the ranges of a TEXTSTRING key compare its whole strings in the field which the index sorts it by, on every index backend. So a query whose last new value had a character from U+E000 or from U+10000, or whose key was mapped as TEXTSTRING, read the index past every element the transaction had moved: after renaming 20,000 of 50,000 vertices to names from U+E000, order().by("name", desc).limit(10) ran 155 Elasticsearch searches in 1.2 s, where one search in 90 ms does. The range is now taken for any string without a surrogate out of its pair, on a field mapped as STRING or TEXTSTRING without a string or text analyzer. A surrogate without its pair isn't valid UTF-16, and no index holds it as it is: Lucene, Solr, OpenSearch and Elasticsearch up to 8 hold U+FFFD, Elasticsearch 9.5 a question mark, while memory places it after U+FFFF in a stored string which storage keeps as written, as it keeps one of up to 16,000 UTF-16 units. Where the index holds U+FFFD, such a stored string may come below a valid value while memory places it above, where the value has a character from U+FFFD up at the first unit in which they differ. An ascending range holds such a string anyway, as it comes after the value. A descending range from such a value now starts at the value's units before that character followed by U+FFFD, which the index places no such string below (rangeBound). JanusGraph's Lucene backend, OpenSearch 2 and Elasticsearch 6 to 8 can't build a range from a lower bound of more than 1,000 UTF-8 bytes ("input automaton is too large"), and Solr takes a query in a URL of at most 8 KB, as JanusGraph sends its queries to Solr as GET requests. So a descending query whose last new value was longer failed on the former with "Could not call index", and on Solr a query in either direction whose value had about 8 KB, also on master for a STRING key below U+D800. A descending bound is now cut to its longest prefix of whole characters within 1,000 bytes, a lower bound still, and no ascending range is taken from a value of more than 1,000 bytes, as an upper bound can't be cut. Two cases are left as on master, where ranges of strings below U+D800 show them too. Elasticsearch 9.5 places such a surrogate below the letters, so its descending range from nearly any string may leave out a stored string with one. And where the index holds U+FFFD, an ascending range may leave out a stored string of more than 16,000 units which storage holds with a question mark in place of such a surrogate, as it does where it compresses the string from its UTF-8 bytes. The transactions page now also says that direct index queries read the index alone: they leave out the elements which the transaction removed, but its other changes don't change what they find. Signed-off-by: Oleksandr Porunov <alexandr.porunov@gmail.com> Co-authored-by: Oleksandr Porunov <alexandr.porunov@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open findings

codePoints().toArray() allocates an int[] on every comparison, which can be a significant cost… · New
This helper assumes the returned ByteBuffer is array-backed (bytes.array()), which is not… · New
In theRangeHoldsEveryStoredStringWhichMemoryOrdersUpToTheValue this method can be called a large… · New
The doc states the derived descending bound is the prefix followed by U+FFFD, but the subsequent… · New

3 resolved since last review

🧠 Review effort: Lite

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

This comparator allocates two int[] arrays on every comparison (codePoints().toArray()), which… · New
ByteBuffer.array() is not guaranteed to be supported for all ByteBuffer implementations… · New

4 resolved since last review

_{Review effort: Lite}

+                final int[] first = ((String) a).codePoints().toArray();
+                final int[] second = ((String) b).codePoints().toArray();
+                for (int i = 0; i < Math.min(first.length, second.length); i++) {
+                    if (first[i] != second[i]) {
+                        return Integer.compare(first[i], second[i]);
+                    }
+                }
+                return Integer.compare(first.length, second.length);


+    private static byte[] indexed(String string) throws CharacterCodingException {
+        final ByteBuffer bytes = StandardCharsets.UTF_8.newEncoder().onMalformedInput(CodingErrorAction.REPLACE)
+            .replaceWith(new byte[]{(byte) 0xEF, (byte) 0xBF, (byte) 0xBD}).encode(CharBuffer.wrap(string));
+        return Arrays.copyOf(bytes.array(), bytes.limit());


porunov requested a balanced review from Copilot October 15, 2026 16:33

Copilot AI reviewed Oct 15, 2026

View reviewed changes

Copilot stopped reviewing on behalf of porunov due to an error October 15, 2026 17:54
The job was not acquired by a runner even after multiple attempts.

porunov mentioned this pull request Oct 15, 2026

Keep the unpaired surrogates of strings longer than 16,000 characters #5098

Open

porunov force-pushed the tx-query-narrowing-more-strings branch from 4bd1a48 to 60eb395 Compare October 15, 2026 19:35

porunov requested a balanced review from Copilot October 15, 2026 19:37

Copilot AI reviewed Oct 15, 2026

View reviewed changes

porunov force-pushed the tx-query-narrowing-more-strings branch from 60eb395 to dade9c4 Compare October 15, 2026 19:59

Copilot stopped reviewing on behalf of porunov due to an error October 15, 2026 21:35
The job was not acquired by a runner even after multiple attempts.

porunov requested a balanced review from Copilot October 16, 2026 11:18

Copilot AI reviewed Oct 16, 2026

View reviewed changes

porunov force-pushed the tx-query-narrowing-more-strings branch from dade9c4 to 99cbbdb Compare October 16, 2026 13:20

porunov requested a balanced review from Copilot October 16, 2026 13:21

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Copilot stopped reviewing on behalf of porunov due to an error October 16, 2026 16:45
The job was not acquired by a runner even after multiple attempts.

porunov requested a balanced review from Copilot October 17, 2026 14:59

Copilot AI reviewed Oct 17, 2026

View reviewed changes

Copilot stopped reviewing on behalf of porunov due to an error October 17, 2026 18:11
The job was not acquired by a runner even after multiple attempts.

Conversation

porunov commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Verification

Measurements

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

porunov commented Oct 15, 2026 •

edited

Loading