Keep the unpaired surrogates of strings longer than 16,000 characters by porunov · Pull Request #5098 · JanusGraph/janusgraph

porunov · 2026-10-15T19:22:27Z

StringSerializer compresses a string longer than 16,000 UTF-16 units (LONG_COMPRESSION_THRESHOLD) with GZIP, from String.getBytes(StandardCharsets.UTF_8). UTF-8 has no encoding for an unpaired surrogate, a unit from U+D800 to U+DFFF which isn't half of a surrogate pair, and getBytes writes a question mark in its place. So such a string came back from storage with question marks in place of its unpaired surrogates, while a shorter string keeps them: StringSerializer encodes it itself, each unit as one to three bytes. Strings which differ only there were stored alike: the values of a SET property, which a SET keeps in the keys of its columns, became one, and the key of a composite index entry held the question marks too, so a unique index took such strings for duplicates. The same held for the long elements of a String[] property and the long text of a JSON property, which the same serializer writes. This PR compresses the WTF-8 bytes of a string instead, which differ from its UTF-8 bytes only where it has an unpaired surrogate, and decodes them back, under the same compressor id. So every string without unpaired surrogates is written byte for byte as before, strings written before read as before, and earlier versions still read the strings written now.

Changes

StringSerializer, the GZIP compressor:
- encodeWtf8 (new) gives the bytes to compress.
  - A string without surrogates takes getBytes(StandardCharsets.UTF_8), as before, once a loop has looked for the first surrogate.
  - A string with a surrogate, paired or not, is encoded by a loop: each character and surrogate pair as UTF-8 encodes it, which gives getBytes's bytes, and an unpaired surrogate as the three bytes which UTF-8 would give its code point, ED A0 80 to ED BF BF. That is WTF-8, which the WTF-8 specification defines.
- decodeWtf8 (new) decodes the decompressed bytes with new String(bytes, StandardCharsets.UTF_8), as before.
  - Java's UTF-8 decoder reads the three bytes of a surrogate as U+FFFD, so the bytes are searched for them only when the decoded string holds U+FFFD.
  - Such three bytes are ED, then A0 to BF, then a continuation byte. UTF-8 has ED only before 80 to 9F. A counted loop searches the bytes for ED.
  - Each of them becomes its surrogate, and the bytes between them are decoded with new String as well.
- The compressor id stays 1, GZIP.
Reading the stored bytes: the stream which GZIPInputStream reads the stored bytes from now overrides read(byte[], int, int), through which GZIPInputStream fills its buffer. InputStream's own implementation of it calls read() for each byte. The override checks its arguments as InputStream's does. See the notes for what it saves.
decompress closes its GZIPInputStream, which frees the native memory of its Inflater at once, where master left that to the garbage collector. Closing it closes the stream it reads the stored bytes from, which does nothing.
JanusGraphAssert.assertSameChars (new): asserts that two strings are equal, and where they aren't, tells the first UTF-16 unit which differs. Surefire cuts a message off at an unpaired surrogate: assertEquals on these strings reported expected: < and nothing more.
Changelog entry.

Tests

SerializerTest (janusgraph-test), five new tests:
- testLongStringsKeepUnpairedSurrogates: strings of 16,009 to 16,012 units, of ASCII, Latin-1, Cyrillic and CJK characters and surrogate pairs, with unpaired surrogates at the start, in the middle and at the end: high and low ones, the first and last of each, two in a row, next to a pair, and next to a question mark and U+FFFD. They read back as written, as do short strings with the same surrogates, a string of 16,000 units ending in one (not compressed) and one of 16,001, and 16,001 unpaired low surrogates.
- testRandomLongStringsKeepUnpairedSurrogates: 100 random strings of 16,001 to 17,001 units, of units from every range, unpaired surrogates included, and supplementary characters, read back as written. Their bytes, decoded as earlier versions decode them, give the strings with U+FFFD in place of each unpaired surrogate.
- testUnpairedSurrogatesOfLongStringsAreWrittenAsTheirCodePoints: the bytes which a string with unpaired U+DC00, U+D800 and U+DFFF and a pair is compressed from are ED B0 80, ED A0 80, ED BF BF and F0 9F 98 80 around the UTF-8 of the rest, and the compressor id is 1.
- testLongStringsWithoutUnpairedSurrogatesAreWrittenAsBefore: six long strings without unpaired surrogates (ASCII, mixed scripts with pairs, U+FFFD and question marks, CJK, only pairs, random) are written byte for byte as master wrote them: the length of the compressed bytes and the compressor id, then getBytes(StandardCharsets.UTF_8) compressed with GZIPOutputStream.
- testLongStringsWrittenBeforeAreReadAsBefore: bytes written as master writes them read back as before, with question marks in place of the unpaired surrogates, and U+FFFD and the question marks of the string kept.
JanusGraphTest.testLongStringsKeepUnpairedSurrogates (every storage backend), with a unique composite index on the key:
- A vertex's string of 16,005 units, with an unpaired low surrogate at the start, an unpaired high one next to a pair in the middle and one at the end, reads back as written in a new transaction.
- The index finds the vertex by the string, and not by the string with question marks in place of its unpaired surrogates.
- A second vertex whose string differs only in the last unpaired surrogate, U+DC00 in place of U+DBFF, is added, and the index finds each vertex by its string.
- A vertex keeps the three values of a SET key which differ only in their last unit: U+D800, U+DC00 or a question mark.

On master's code:

JanusGraphTest.testLongStringsKeepUnpairedSurrogates fails on inmemory: "Expected 16005 chars but was 16005, which first differ at 0: expected U+DC00 but was U+003F". Past that check, master finds the vertex by the string with question marks too, its unique index rejects the second vertex with a SchemaViolationException, and the vertex keeps one value of the SET (on BerkeleyJE; inmemory keeps three copies of the one ending in a question mark): the same steps show these in programs run on master.
In SerializerTest, testLongStringsKeepUnpairedSurrogates and testRandomLongStringsKeepUnpairedSurrogates fail the same way. testUnpairedSurrogatesOfLongStringsAreWrittenAsTheirCodePoints fails as the bytes are 24,019 rather than 24,025: three unpaired surrogates take one byte each instead of three.
The two tests of what stays the same, testLongStringsWithoutUnpairedSurrogatesAreWrittenAsBefore and testLongStringsWrittenBeforeAreReadAsBefore, pass on master.

Mutation checks, each run on the unit tests and on the graph test on inmemory:

Master's encoder with the new decoder fails the same tests as master.
Master's decoder with the new encoder: the strings read back with U+FFFD in place of their unpaired surrogates, which fails the graph test and two unit tests.
A decoder which takes A0 to AF alone as the second byte, so high surrogates only: the low surrogates come back as U+FFFD, which fails the graph test and two unit tests.
Surrogate pairs written as two sequences of three bytes (CESU-8): the strings still read back as written, but testLongStringsWithoutUnpairedSurrogatesAreWrittenAsBefore finds other bytes for the strings with pairs, testUnpairedSurrogatesOfLongStringsAreWrittenAsTheirCodePoints finds 26,695 bytes, and earlier versions would read each pair as two U+FFFD, which fails testRandomLongStringsKeepUnpairedSurrogates.
The new read(byte[], int, int) not counting the bytes it read, or reading one byte too many: four of the new tests and testStringCompression fail with ArrayIndexOutOfBoundsException as the stream reads past the stored bytes, and testSerializationMixture reads the next value from the wrong place. The first also fails the graph test.
An encoder which takes getBytes when the first surrogate is half of a pair: a string with a pair before an unpaired surrogate loses that one, which fails testLongStringsKeepUnpairedSurrogates and testRandomLongStringsKeepUnpairedSurrogates.
A decoder which doesn't search the last three bytes: an unpaired surrogate at the end comes back as U+FFFD, which fails the graph test and two unit tests.
Equivalence: with every long string encoded by the loop, without getBytes, every test passes, testLongStringsWithoutUnpairedSurrogatesAreWrittenAsBefore included: the loop gives getBytes's bytes for those six strings, as it follows the JDK's UTF-8 encoder branch for branch, apart from unpaired surrogates. A string with a pair but no unpaired surrogate takes that loop. Without the check for U+FFFD before the search, every test passes too.

Locally, Java 11, with checkstyle, on master dd4f0f845: janusgraph-driver (130), janusgraph-core (228), janusgraph-inmemory (654, 85 skipped) and janusgraph-test (1,107) pass, and so does BerkeleyGraphTest (226, 7 skipped). CI runs the graph test on the other storage backends.

Notes

Compatibility:
- Valid strings are written byte for byte as before (testLongStringsWithoutUnpairedSurrogatesAreWrittenAsBefore).
- Strings written before read as before, with their question marks. Java's UTF-8 encoder never writes the bytes of a surrogate: it writes a pair as four bytes and an unpaired surrogate as a question mark. Every version since Titan compressed long strings so (4b0231d, 2014).
- Earlier versions read a string written now with U+FFFD in place of each unpaired surrogate. That is what new String(bytes, StandardCharsets.UTF_8) makes of the three bytes, on JDK 11, 17, 21 and 26. A new compressor id would have made such strings unreadable to them: "Unknown compressor type for id".
Composite indexes:
- Fixed: a unique index took two long strings which differ only in an unpaired surrogate, or in an unpaired surrogate against a question mark, for duplicates, and threw a SchemaViolationException on the second. Now they have keys of their own.
- Entries written before hold the form with question marks, which is also what the strings now read back as. A lookup by that form finds them, while a lookup by a string with unpaired surrogates finds only the entries written since, and a unique index doesn't take the two for duplicates. A reindex doesn't change that, as it reads the values from storage.
While earlier versions write to the same graph: an earlier version reads such a string written now with U+FFFD in place of each unpaired surrogate, and encodes it so again. So it can't remove the string where its bytes are part of a key:
- A value of a SET property stays, as the earlier version deletes the column of the form it read, which doesn't exist.
- A composite index entry stays: the index still finds the element by the string, and a unique index keeps the string taken.
- On BerkeleyJE, with this branch writing a vertex, master removing its values and this branch reading on, the vertex still has the long value of its SET, the unique index finds it by a value it no longer has, and another vertex with that value is rejected with a SchemaViolationException.
- With ConsistencyModifier.LOCK on such a key or index, on a storage backend whose locks check the expected values (CQL, HBase), the earlier version's commit should fail instead with "Expected value mismatch": by code reading, not run.
- Upgrade every instance before such strings are written, or afterwards remove the leftover values with this version and replace the composite indexes concerned with new ones: REINDEX adds entries but never removes any, and REMOVE_STALE_ENTRIES removes only those of elements which no longer exist.
Mixed indexes get a string from the transaction as it was written. A reindex, and the CDC worker, read it from storage: so far with question marks in place of its unpaired surrogates, now as written. Each index keeps an unpaired surrogate its own way; Lucene, for one, turns it into U+FFFD.
Order: where JanusGraph sorts strings itself, it now places such a string by its unpaired surrogates, as it places a shorter one, rather than by question marks.
Narrow a transaction's ordered index reads for strings from U+D800 and TEXTSTRING keys #5085 adds its own REPLACEMENT_CHARACTER and a check for unpaired surrogates to StandardJanusGraphTx. Whichever of the two merges second can share the constant, and a helper where one fits both.

Cost, measured with JMH: StringSerializer writes and reads a string of 100,000 UTF-16 units, with master's janusgraph-core jar against this branch's on one classpath. Two passes of reversed order ran 3 forks a side each, on Java 11 and an Apple M5 Pro; the table gives the medians of the 6 fork means a side, and an exact Mann-Whitney p over them.

String (bytes written)	Write: master	Branch	Change	Read: master	Branch	Change
Words of a vocabulary of 500 (29 KB)	1,147 µs	1,145 µs	−0.2% (p 0.82)	102 µs	98.2 µs	−3.3% (p 0.002)
Cyrillic words (81 KB)	11,256 µs	11,246 µs	−0.1% (p 0.82)	410 µs	412 µs	+0.5% (p 0.94)
Cyrillic words with an emoji at unit 100 (81 KB)	11,264 µs	11,305 µs	+0.4% (p 0.31)	403 µs	414 µs	+2.9% (p 0.24)
Random CJK characters (219 KB)	6,753 µs	6,803 µs	+0.7% (p 0.13)	903 µs	894 µs	−1.0% (p 0.041)
Random CJK characters with an emoji at unit 100 (219 KB)	6,849 µs	6,764 µs	−1.2% (p 0.041)	909 µs	890 µs	−2.1% (p 0.093)
Words of random letters, 5% of them an emoji (64 KB)	1,594 µs	1,590 µs	−0.2% (p 0.82)	252 µs	254 µs	+0.6% (p 0.39)
The vocabulary's words with one U+FFFD (29 KB)	1,185 µs	1,195 µs	+0.9% (p 0.70)	165 µs	194 µs	+17.2% (p 0.002)
Words of random letters with 10 unpaired surrogates (63 KB)	1,421 µs	1,412 µs	−0.6% (p 0.94)	224 µs	280 µs	+25% (p 0.002)
The vocabulary's words with an unpaired surrogate in every tenth unit (36 KB on master, 62 KB here)	1,326 µs	1,914 µs	+44% (p 0.002)	125 µs	424 µs	+240% (p 0.002)

Strings without unpaired surrogates are written and read as fast as on master, within the noise, or faster, except for a string which holds U+FFFD: its bytes are searched for surrogates on each read.
- A loop looks for the first surrogate. It cost nothing measurable for the Latin-1 words, and about 10 µs for 100,000 units of Cyrillic or CJK text in a separate run.
- A string with a surrogate pair, like the ones with an emoji, is then encoded by the loop which encodes unpaired surrogates too, as fast as getBytes.
- The first version of this PR checked the whole string for unpaired surrogates before taking getBytes, which made the writes of the text with 5% emoji 3.0% slower (p 0.002) in the same benchmark.
Strings with unpaired surrogates are kept, where master lost them, on the slower paths.
- Their reads decode the bytes twice and allocate more: 1.71 MB rather than 0.41 MB for the string with 10 of them.
- A counted loop searches the bytes for ED. Searching them byte by byte in one loop made the reads of the string with 10 of them 62% slower than master's in the same benchmark, rather than 25%.
- With many of them, the bytes compress less well, as they are the surrogates' bytes rather than question marks.
Every other read gains from the change below, which makes up for the check for U+FFFD.
Searching the decompressed bytes for surrogates on every read, rather than only when the decoded string holds U+FFFD, took the decompression of the Latin-1 words from 103 µs to 115 µs in a rougher run of one fork, while the check for U+FFFD left it at 102 µs. On the other strings both were within the noise.

Reading the stored bytes: the override of read(byte[], int, int) alone, measured the same way in an earlier run as the branch with and without it, made every read faster and left writes as they were:
- reads 2.2% to 9.0% faster for each of the seven strings, p 0.002 to 0.041: for example 108 µs to 103 µs for the vocabulary's words, 460 µs to 419 µs for the Cyrillic ones and 960 µs to 940 µs for the CJK characters;
- writes within 2.1%, p 0.39 or more.

🤖 Generated with Claude Code

Fixes JanusGraph#5097 StringSerializer compresses a string longer than 16,000 UTF-16 units with GZIP, from String.getBytes(UTF_8). UTF-8 has no encoding for an unpaired surrogate, a unit from U+D800 to U+DFFF which isn't half of a surrogate pair, and getBytes writes a question mark in its place. So such a string came back from storage with question marks in place of its unpaired surrogates, while a shorter string, which StringSerializer encodes unit by unit, keeps them. Strings which differ only there were stored alike: the values of a SET property became one, and a unique composite index took such strings for duplicates. The bytes to compress are now WTF-8: UTF-8, except that an unpaired surrogate takes the three bytes which UTF-8 would give its code point. A string without surrogates is still encoded by getBytes, byte for byte as before; one with a surrogate is encoded by a loop, which gives getBytes's bytes for every other character and every pair. Decompression decodes those three bytes back into the surrogate. Java's UTF-8 decoder reads them as U+FFFD, so the bytes are searched for them only when the decoded string holds U+FFFD. The compressor id stays GZIP. Strings written before read as before, since Java's encoder never writes the bytes of a surrogate. An earlier version reads a string written now with U+FFFD in place of each unpaired surrogate, where a new compressor id would have made the string unreadable to it. The stream which decompression reads the stored bytes through now fills GZIPInputStream's buffer in one call of read(byte[], int, int), where InputStream's implementation of it called read() for each byte. Decompression also closes the GZIPInputStream, which frees the native memory of its Inflater at once. Signed-off-by: Oleksandr Porunov <alexandr.porunov@gmail.com> Co-authored-by: Oleksandr Porunov <alexandr.porunov@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

read() increments bytesRead even when returning -1, which can push bytesRead beyond… · New
This allocation can overflow (negative array size) if text.length() is large enough (`>… · New
Asserting full StaticBuffer equality for byte-for-byte compatibility is good, but if… · New

What changed in this PR

This PR fixes loss of unpaired UTF-16 surrogates when JanusGraph compresses long strings (>16,000 UTF-16 units) by switching the compressed representation from Java UTF-8 bytes (which replace unpaired surrogates with ?) to WTF-8 bytes and decoding them back on read, preserving backward compatibility.

Changes:

Update StringSerializer GZIP compression to write WTF-8 bytes for long strings and decode them back on decompress; also improve the InputStream.read(byte[], int, int) path and close GZIPInputStream.
Add targeted tests to validate surrogate preservation, backward compatibility, and byte-level stability for valid strings.
Add assertSameChars helper for reliable assertion messages when strings contain unpaired surrogates; document behavior in changelog.

File	Description
janusgraph-core/src/main/java/org/janusgraph/graphdb/database/serialize/attribute/StringSerializer.java	Switch long-string compression payload to WTF-8 and decode on read; optimize GZIP read path and ensure stream closure.
janusgraph-test/src/test/java/org/janusgraph/graphdb/serializer/SerializerTest.java	Add comprehensive unit tests for long strings with/unpaired surrogates and backward compatibility byte formats.
janusgraph-backend-testutils/src/main/java/org/janusgraph/testutil/JanusGraphAssert.java	Add string assertion helper that reports the first differing UTF-16 unit safely.
janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphTest.java	Add integration test ensuring composite unique index + SET cardinality behave correctly with long strings containing unpaired surrogates.
docs/changelog.md	Document behavior change and operational caveats for mixed-version clusters and indexed keys.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open findings

Objects.checkFromIndexSize is only available starting in Java 9. If this module is intended to… · New
expected is dereferenced without a null check (expected.equals(actual)). If assertSameChars… · New

3 resolved since last review

🧠 Review effort: Lite

… keys Fixes JanusGraph#5109 LuceneIndex removes a value of a key of several values from a document by comparing the string which convertToStringValue gives for the value with the stored values of the field, as Lucene reads them back. For two kinds of values that string wasn't the stored one: - A float is stored as the double which it widens to, whose decimal string differs from the float's for most floats: "0.1" against "0.10000000149011612" for 0.1f. The zeros, NaN, the infinities and floats such as 1.5f have the same string either way. - A string is stored in UTF-8, which has no encoding for an unpaired surrogate, so Lucene writes U+FFFD in its place and reads that back. Such a value stayed in the document when it was removed, and eq, the ranges and the text predicates kept finding the element by it, also once the element held no value of the key. So did the value which an edge held before it was set to another one. And a value given again to a SET key whose document held it was held twice, as the addition didn't find the entry which it replaces. convertToStringValue now gives a number as buildStoreField stores it, a whole number as a long and a decimal as a double, and a string as it reads back from UTF-8. What Lucene writes doesn't change. A string longer than 16,000 UTF-16 units is removed only where storage gives it back with its unpaired surrogates, which JanusGraph#5098 makes it do: master reads it back with question marks in their place. Values which Lucene stores alike, such as strings which differ only in an unpaired surrogate, or in one and U+FFFD, now always share one entry where a SET key holds both, as two instants of one millisecond did: the addition of the second replaces the entry of the first, as it compares equal now, so removing either leaves the other unfound. Values which an earlier version left in a document stay there until the document is written whole again: a reindex writes the document of each element which holds a value of the index, and leaves the others. Signed-off-by: Oleksandr Porunov <alexandr.porunov@gmail.com> Co-authored-by: Oleksandr Porunov <alexandr.porunov@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

assertSameChars will throw NullPointerException when expected is null due to… · New
The heading says '16,000 characters' but the behavior is based on UTF-16 units (as the paragraph… · New

2 resolved since last review

_{Review effort: Lite}

+    public static void assertSameChars(String expected, String actual) {
+        assertNotNull(actual);
+        assertTrue(expected.equals(actual), () -> {
+            int i = 0;
+            while (i < expected.length() && i < actual.length() && expected.charAt(i) == actual.charAt(i)) i++;
+            return String.format("Expected %d chars but was %d, which first differ at %d: expected %s but was %s",
+                expected.length(), actual.length(), i, unitAt(expected, i), unitAt(actual, i));
+        });
+    }


 `I` written by a JVM with a Turkish or Azerbaijani default locale. A mixed index on Elasticsearch whose store name holds
 an `I` or a dotted capital `İ`, and whose index such a JVM created, needs one too.

+##### Strings longer than 16,000 characters keep their unpaired surrogates


porunov requested a balanced review from Copilot October 15, 2026 19:22

porunov mentioned this pull request Oct 15, 2026

Narrow a transaction's ordered index reads for strings from U+D800 and TEXTSTRING keys #5085

Open

Copilot AI reviewed Oct 15, 2026

View reviewed changes

Copilot stopped reviewing on behalf of porunov due to an error October 15, 2026 19:48
The job was not acquired by a runner even after multiple attempts.

This was referenced Oct 16, 2026

Let Lucene remove floats and lone-surrogate strings from LIST and SET keys #5110

Open

Work around Lucene miswriting long strings with unpaired surrogates #5112

Open

porunov requested a balanced review from Copilot October 16, 2026 11:19

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Comment thread ...core/src/main/java/org/janusgraph/graphdb/database/serialize/attribute/StringSerializer.java

Comment thread janusgraph-backend-testutils/src/main/java/org/janusgraph/testutil/JanusGraphAssert.java

porunov requested a balanced review from Copilot October 17, 2026 15:04

Copilot AI reviewed Oct 17, 2026

View reviewed changes

Copilot stopped reviewing on behalf of porunov due to an error October 17, 2026 18:15
Workflow completed but conclusion is not available. This may indicate rate limiting or API issues.

Navigation Menu

Repository navigation

Keep the unpaired surrogates of strings longer than 16,000 characters - #5098

Keep the unpaired surrogates of strings longer than 16,000 characters#5098
porunov wants to merge 1 commit into
JanusGraph:masterfrom
porunov:long-strings-keep-unpaired-surrogates

porunov commented Oct 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

porunov commented Oct 15, 2026

Changes

Tests

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants