From 8befebefb41b686dedb51f298d6775b0888fe5fb Mon Sep 17 00:00:00 2001 From: Oleksandr Porunov Date: Fri, 16 Oct 2026 00:53:08 +0100 Subject: [PATCH] Let Lucene remove floats and lone-surrogate strings from LIST and SET keys Fixes #5109 LuceneIndex removes a value of a key of several values from a document by comparing the string which convertToStringValue gives for the value with the stored values of the field, as Lucene reads them back. For two kinds of values that string wasn't the stored one: - A float is stored as the double which it widens to, whose decimal string differs from the float's for most floats: "0.1" against "0.10000000149011612" for 0.1f. The zeros, NaN, the infinities and floats such as 1.5f have the same string either way. - A string is stored in UTF-8, which has no encoding for an unpaired surrogate, so Lucene writes U+FFFD in its place and reads that back. Such a value stayed in the document when it was removed, and eq, the ranges and the text predicates kept finding the element by it, also once the element held no value of the key. So did the value which an edge held before it was set to another one. And a value given again to a SET key whose document held it was held twice, as the addition didn't find the entry which it replaces. convertToStringValue now gives a number as buildStoreField stores it, a whole number as a long and a decimal as a double, and a string as it reads back from UTF-8. What Lucene writes doesn't change. A string longer than 16,000 UTF-16 units is removed only where storage gives it back with its unpaired surrogates, which #5098 makes it do: master reads it back with question marks in their place. Values which Lucene stores alike, such as strings which differ only in an unpaired surrogate, or in one and U+FFFD, now always share one entry where a SET key holds both, as two instants of one millisecond did: the addition of the second replaces the entry of the first, as it compares equal now, so removing either leaves the other unfound. Values which an earlier version left in a document stay there until the document is written whole again: a reindex writes the document of each element which holds a value of the index, and leaves the others. Signed-off-by: Oleksandr Porunov Co-authored-by: Oleksandr Porunov Co-authored-by: Claude Opus 5.5 --- docs/changelog.md | 27 ++ .../graphdb/JanusGraphIndexTest.java | 112 +++++++ .../diskstorage/lucene/LuceneIndex.java | 10 +- .../lucene/LuceneIndexValueRemovalTest.java | 273 ++++++++++++++++++ 4 files changed, 420 insertions(+), 2 deletions(-) create mode 100644 janusgraph-lucene/src/test/java/org/janusgraph/diskstorage/lucene/LuceneIndexValueRemovalTest.java diff --git a/docs/changelog.md b/docs/changelog.md index dbf767e6de..af62a6f6cc 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -422,6 +422,33 @@ stores and orders above positive infinity, also finds what it does without the i NaN, `lt` and `gt` of NaN nothing, and `neq` of any other value finds NaN as well. A raw range reaches NaN by a NaN bound, as in `score:[5 TO NaN]`. +##### Lucene no longer keeps the floats, and the strings with unpaired surrogates, removed from `LIST` and `SET` keys + +On a Lucene mixed index, most floats (`Float`) removed from a key of `LIST` or `SET` cardinality stayed in the +element's document, as did the float which such a key of an edge held before it was set to another one, and `eq` and +the ranges kept finding the element by them: after `g.V(v).properties("weights").drop()`, `g.V().has("weights", 0.1f)` +and `g.V().has("weights", gt(0f))` still found `v`. Lucene stores a float as the double which it widens to, and +compared the removed float with the stored values by its own decimal string, which differs from the double's for most +floats: `0.1` against `0.10000000149011612`. Floats whose strings are the same, such as `1.5f` or `-2.5f`, were +removed. A string with an unpaired surrogate stayed likewise, whatever its mapping, as Lucene stores U+FFFD in place of +the surrogate. Lucene now compares a removed value with the stored ones in the form in which it stores them. What it +writes doesn't change. A string longer than 16,000 UTF-16 units is removed too where storage gives it back with its +unpaired surrogates, as it does from 1.2.0 on (see "Strings longer than 16,000 UTF-16 units keep their unpaired +surrogates"); one which an earlier version stored comes back with a question mark in place of each unpaired surrogate, +and stays in the index when it's removed. + +In exchange, a `SET` key mapped as `STRING` or `TEXTSTRING` which holds two strings that Lucene stores alike, which +differ only in an unpaired surrogate, or in one and U+FFFD, holds one entry for both, whichever came first, so once one +of them is removed, the other isn't found either. Two instants of one millisecond share one entry in the same way, as +before. + +What an earlier version left in an index stays there: the removed values, and a second copy of a value which a `SET` +key held when the value was set again, so that removing that value now takes out one of the two. A reindex writes the +document of each element which holds a value of the index anew, without them, but leaves the document of an element +which holds none as it is. To clear those too, stop the instances, delete the directory of each mixed index concerned, +`index.[X].directory/`, and reindex those indexes. They stay enabled meanwhile, so until the reindex has +finished, their queries miss the elements which it hasn't written yet. + ##### Ranges and not-equals of a `TEXTSTRING` key compare its strings whole A key mapped as `TEXTSTRING` holds each string whole in a field of its own, which the index sorts the key by, and its diff --git a/janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphIndexTest.java b/janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphIndexTest.java index 5f1e1dcd45..9d0de5888b 100644 --- a/janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphIndexTest.java +++ b/janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphIndexTest.java @@ -809,6 +809,118 @@ public void testListDeleteAddInOneTransaction() { assertCount(0, tx.query().has("alias", Text.CONTAINS, "Marko").vertices()); } + //A float removed from a key of several values is no longer found, also one whose decimal string changes when it is + //widened to a double, as Lucene stores it: 0.1f as 0.10000000149011612 + @ParameterizedTest + @ValueSource(strings = {"LIST", "SET"}) + public void testRemovedFloatsOfAKeyOfSeveralValuesAreNotFound(Cardinality cardinality) { + if (!indexFeatures.supportsCardinality(cardinality)) return; + + final PropertyKey weights = mgmt.makePropertyKey("weights").dataType(Float.class).cardinality(cardinality) + .make(); + final PropertyKey kind = makeKey("kind", String.class); + makeLabel("knows"); + mgmt.buildIndex("mixed", Vertex.class).addKey(weights).buildMixedIndex(INDEX); + mgmt.buildIndex("edges", Edge.class).addKey(weights).addKey(kind, getStringMapping()).buildMixedIndex(INDEX); + finishSchema(); + + //Each float but 1.5f has a decimal string other than the double's which it widens to + final float[] removed = {0.1f, -0.3f, 3.14f, Float.MIN_VALUE, Float.MAX_VALUE, 1.5f}; + final JanusGraphVertex some = tx.addVertex(); + final JanusGraphVertex none = tx.addVertex(); + for (final float value : removed) { + some.property(cardinality.convert(), "weights", value); + none.property(cardinality.convert(), "weights", value); + } + some.property(cardinality.convert(), "weights", -2.5f); + //The float of an edge is replaced, and that of another edge removed, which keeps its document by its kind + final Object replaced = some.addEdge("knows", none, "weights", 0.1f).id(); + final Object cleared = some.addEdge("knows", none, "weights", 0.1f, "kind", "k").id(); + clopen(); + + GraphTraversalSource g = tx.traversal(); + for (final float value : removed) { + g.V(some.id(), none.id()).properties("weights").hasValue(value).drop().iterate(); + } + g.E(replaced).property("weights", 3.14f).iterate(); + g.E(cleared).properties("weights").drop().iterate(); + clopen(option(FORCE_INDEX_USAGE), true); + + g = tx.traversal(); + for (final float value : removed) { + assertFalse(g.V().has("weights", value).hasNext(), Float.toString(value)); + assertEquals(0L, g.V().has("weights", value).count().next(), Float.toString(value)); + } + assertEquals(Collections.singletonList(some.id()), g.V().has("weights", -2.5f).id().toList()); + //A range of every finite float doesn't find the vertex whose floats were all removed + assertEquals(Collections.singletonList(some.id()), g.V().has("weights", P.gte(-Float.MAX_VALUE)).id().toList()); + assertEquals(1L, g.V().has("weights", P.gte(-Float.MAX_VALUE)).count().next()); + assertFalse(g.E().has("weights", 0.1f).hasNext()); + assertEquals(0L, g.E().has("weights", 0.1f).count().next()); + assertEquals(Collections.singletonList(replaced), g.E().has("weights", 3.14f).id().toList()); + assertEquals(Collections.singletonList(replaced), g.E().has("weights", P.gte(-Float.MAX_VALUE)).id().toList()); + assertEquals(1L, g.E().has("weights", P.gte(-Float.MAX_VALUE)).count().next()); + assertEquals(Collections.singletonList(cleared), g.E().has("kind", "k").id().toList()); + } + + //A removed string of a key of several values is no longer found, also one with an unpaired surrogate, which UTF-8 + //has no encoding for: Lucene stores U+FFFD in its place + @ParameterizedTest + @ValueSource(strings = {"LIST", "SET"}) + public void testRemovedStringsWithUnpairedSurrogatesOfAKeyOfSeveralValuesAreNotFound(Cardinality cardinality) { + if (!indexFeatures.supportsCardinality(cardinality)) return; + + final List keys = new ArrayList<>(); + for (final Object[] key : new Object[][]{{"defaultString", Mapping.DEFAULT}, {"text", Mapping.TEXT}, + {"string", Mapping.STRING}, {"textString", Mapping.TEXTSTRING}}) { + if (key[1] == Mapping.DEFAULT || indexFeatures.supportsStringMapping((Mapping) key[1])) { + keys.add(key); + } + } + final JanusGraphManagement.IndexBuilder index = mgmt.buildIndex("mixed", Vertex.class); + for (final Object[] key : keys) { + index.addKey(mgmt.makePropertyKey((String) key[0]).dataType(String.class).cardinality(cardinality).make(), + ((Mapping) key[1]).asParameter()); + } + index.buildMixedIndex(INDEX); + finishSchema(); + + final String removed = "Ab\uD800c Removed"; + final JanusGraphVertex some = tx.addVertex(); + final JanusGraphVertex none = tx.addVertex(); + for (final Object[] key : keys) { + some.property(cardinality.convert(), (String) key[0], removed); + some.property(cardinality.convert(), (String) key[0], "Kept"); + none.property(cardinality.convert(), (String) key[0], removed); + } + clopen(option(FORCE_INDEX_USAGE), true); + assertStringsFound(keys, "removed", "Ab", Sets.newHashSet(some.id(), none.id())); + + for (final Object[] key : keys) { + tx.traversal().V(some.id(), none.id()).properties((String) key[0]).hasValue(removed).drop().iterate(); + } + clopen(option(FORCE_INDEX_USAGE), true); + assertStringsFound(keys, "removed", "Ab", Collections.emptySet()); + assertStringsFound(keys, "kept", "Kept", Collections.singleton(some.id())); + } + + //The vertices found by a word, on a key mapped to the words of its strings, and by a prefix, on a key mapped to its + //whole strings: eq of a string with an unpaired surrogate finds nothing on Elasticsearch and Solr + private void assertStringsFound(List keys, String word, String prefix, Set expected) { + final GraphTraversalSource g = tx.traversal(); + for (final Object[] key : keys) { + final String name = (String) key[0]; + if (key[1] != Mapping.STRING) { + assertEquals(expected, g.V().has(name, Text.textContains(word)).id().toSet(), name); + assertEquals((long) expected.size(), g.V().has(name, Text.textContains(word)).count().next(), name); + } + if (key[1] == Mapping.STRING || key[1] == Mapping.TEXTSTRING) { + assertEquals(expected, g.V().has(name, Text.textPrefix(prefix)).id().toSet(), name); + assertEquals((long) expected.size(), g.V().has(name, Text.textPrefix(prefix)).count().next(), name); + } + } + } + @Test public void testIndexing() throws InterruptedException { diff --git a/janusgraph-lucene/src/main/java/org/janusgraph/diskstorage/lucene/LuceneIndex.java b/janusgraph-lucene/src/main/java/org/janusgraph/diskstorage/lucene/LuceneIndex.java index 91d1ccffdd..d21e5f0004 100644 --- a/janusgraph-lucene/src/main/java/org/janusgraph/diskstorage/lucene/LuceneIndex.java +++ b/janusgraph-lucene/src/main/java/org/janusgraph/diskstorage/lucene/LuceneIndex.java @@ -74,6 +74,7 @@ import org.apache.lucene.store.FSDirectory; import org.apache.lucene.store.LockObtainFailedException; import org.apache.lucene.util.BytesRef; +import org.apache.lucene.util.UnicodeUtil; import org.janusgraph.core.Cardinality; import org.janusgraph.core.attribute.Cmp; import org.janusgraph.core.attribute.Geo; @@ -543,7 +544,10 @@ private static boolean storesLowerCase(Mapping mapping, Object value) { private String convertToStringValue(final Mapping mapping, Object value) { String converted; if (value instanceof Number) { - converted = value.toString(); + //As buildStoreField stores it: a float as the double which it widens to, whose decimal string differs from + //the float's for most floats, such as 0.10000000149011612 for 0.1f + converted = AttributeUtils.isWholeNumber((Number) value) ? String.valueOf(((Number) value).longValue()) + : String.valueOf(((Number) value).doubleValue()); } else if (AttributeUtils.isString(value)) { converted = storesLowerCase(mapping, value) ? lowerCaseCodePoints((String) value) : (String) value; } else if (value instanceof Date) { @@ -557,7 +561,9 @@ private String convertToStringValue(final Mapping mapping, Object value) { } else { throw new IllegalArgumentException("Unsupported type: " + value); } - return converted; + //Lucene stores a string in UTF-8, with U+FFFD in place of each unpaired surrogate + return AttributeUtils.isString(value) && !UnicodeUtil.validUTF16String(converted) + ? new BytesRef(converted).utf8ToString() : converted; } // NOTE: new SET/LIST store fields must be sync with convertToStringValue diff --git a/janusgraph-lucene/src/test/java/org/janusgraph/diskstorage/lucene/LuceneIndexValueRemovalTest.java b/janusgraph-lucene/src/test/java/org/janusgraph/diskstorage/lucene/LuceneIndexValueRemovalTest.java new file mode 100644 index 0000000000..2c37592d67 --- /dev/null +++ b/janusgraph-lucene/src/test/java/org/janusgraph/diskstorage/lucene/LuceneIndexValueRemovalTest.java @@ -0,0 +1,273 @@ +// Copyright 2026 JanusGraph Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.janusgraph.diskstorage.lucene; + +import org.apache.lucene.index.DirectoryReader; +import org.apache.lucene.index.Term; +import org.apache.lucene.search.IndexSearcher; +import org.apache.lucene.search.TermQuery; +import org.apache.lucene.search.TopDocs; +import org.apache.lucene.store.FSDirectory; +import org.janusgraph.core.Cardinality; +import org.janusgraph.core.JanusGraphElement; +import org.janusgraph.core.attribute.Cmp; +import org.janusgraph.core.attribute.Text; +import org.janusgraph.core.schema.Mapping; +import org.janusgraph.diskstorage.BackendException; +import org.janusgraph.diskstorage.indexing.IndexProviderTest; +import org.janusgraph.diskstorage.indexing.IndexQuery; +import org.janusgraph.diskstorage.indexing.IndexTransaction; +import org.janusgraph.diskstorage.indexing.KeyInformation; +import org.janusgraph.diskstorage.util.StandardBaseTransactionConfig; +import org.janusgraph.diskstorage.util.time.TimestampProviders; +import org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration; +import org.janusgraph.graphdb.query.condition.PredicateCondition; +import org.janusgraph.graphdb.tinkerpop.optimize.step.Aggregation; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import org.junit.jupiter.params.provider.ValueSource; + +import java.io.File; +import java.io.IOException; +import java.time.Duration; +import java.time.Instant; +import java.util.Arrays; +import java.util.Collections; +import java.util.Date; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.UUID; +import java.util.stream.Collectors; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +/** + * A value removed from a key of several values leaves its Lucene document, as does a value of a SET key which an + * addition of it replaces, also where the document stores the value in another form than the value's own string. A + * float is stored as the double which it widens to, whose decimal string differs from the float's for most floats, and + * a string in UTF-8, with U+FFFD in place of each unpaired surrogate. + */ +public class LuceneIndexValueRemovalTest { + + private static final String STORE = "vertex"; + private static final String KEY = "weights"; + + //Each of them but 1.5f, the zeros, NaN and the infinities has a decimal string other than the double's which it + //widens to, such as 0.10000000149011612 for 0.1f + private static final float[] FLOATS = {0.1f, -0.3f, 3.14f, Float.MIN_VALUE, Float.MIN_NORMAL, Float.MAX_VALUE, + 1.5f, 0.0f, -0.0f, Float.NaN, Float.POSITIVE_INFINITY, Float.NEGATIVE_INFINITY}; + + //The name, data type and mapping of a key of each data type, the value which its document keeps, the one which is + //removed, and the kept value as Lucene reads it back + private static final Object[][] KEYS = { + {"byte", Byte.class, Mapping.DEFAULT, (byte) 7, Byte.MIN_VALUE, "7"}, + {"short", Short.class, Mapping.DEFAULT, (short) 7, Short.MIN_VALUE, "7"}, + {"int", Integer.class, Mapping.DEFAULT, 7, Integer.MIN_VALUE, "7"}, + {"long", Long.class, Mapping.DEFAULT, Long.MAX_VALUE, Long.MIN_VALUE, "9223372036854775807"}, + {"float", Float.class, Mapping.DEFAULT, -0.0f, 0.1f, "-0.0"}, + {"double", Double.class, Mapping.DEFAULT, -0.0, 0.0, "-0.0"}, + {"date", Date.class, Mapping.DEFAULT, new Date(7), new Date(-1), "7"}, + {"instant", Instant.class, Mapping.DEFAULT, Instant.ofEpochSecond(0, 7_000_001), Instant.ofEpochMilli(-1), "7"}, + {"boolean", Boolean.class, Mapping.DEFAULT, false, true, "0"}, + {"uuid", UUID.class, Mapping.DEFAULT, new UUID(7, 7), new UUID(1, 1), new UUID(7, 7).toString()}, + {"defaultString", String.class, Mapping.DEFAULT, "Kept Words", "Removed Words", "kept words"}, + {"text", String.class, Mapping.TEXT, "Kept Words", "Removed Words", "kept words"}, + {"string", String.class, Mapping.STRING, "Kept Words", "Removed Words", "Kept Words"}, + {"textString", String.class, Mapping.TEXTSTRING, "Kept Words", "Removed Words", "kept words"}}; + + private LuceneIndex index; + + @BeforeEach + public void setUp() throws BackendException { + index = new LuceneIndex(LuceneIndexTest.getLocalLuceneTestConfig()); + index.clearStorage(); + index.close(); + index = new LuceneIndex(LuceneIndexTest.getLocalLuceneTestConfig()); + } + + @AfterEach + public void tearDown() throws BackendException { + index.close(); + } + + private IndexTransaction openTx(Map keys) throws BackendException { + return new IndexTransaction(index, IndexProviderTest.getIndexRetriever(keys), + StandardBaseTransactionConfig.of(TimestampProviders.MILLI), Duration.ofMillis(2000L)); + } + + private IndexTransaction openTx(KeyInformation key) throws BackendException { + return openTx(Collections.singletonMap(KEY, key)); + } + + //One document keeps 2.5f, and the other one none of its floats + @ParameterizedTest + @ValueSource(strings = {"LIST", "SET"}) + public void testRemovedFloatsLeaveTheDocument(Cardinality cardinality) throws Exception { + final KeyInformation key = IndexProviderTest.of(Float.class, cardinality); + IndexTransaction tx = openTx(key); + for (final float value : FLOATS) { + tx.add(STORE, "some", KEY, value, true); + tx.add(STORE, "none", KEY, value, true); + } + tx.add(STORE, "some", KEY, 2.5f, true); + tx.commit(); + tx = openTx(key); + for (final float value : FLOATS) { + tx.delete(STORE, "some", KEY, value, false); + tx.delete(STORE, "none", KEY, value, false); + } + tx.commit(); + + assertEquals(Collections.singletonList("2.5"), storedValues("some", KEY)); + assertEquals(Collections.emptyList(), storedValues("none", KEY)); + tx = openTx(key); + for (final float value : FLOATS) { + final PredicateCondition equal = PredicateCondition.of(KEY, Cmp.EQUAL, value); + assertEquals(Collections.emptySet(), query(tx, equal), Float.toString(value)); + assertEquals(0L, count(tx, equal), Float.toString(value)); + } + assertEquals(Collections.singleton("some"), query(tx, PredicateCondition.of(KEY, Cmp.EQUAL, 2.5f))); + tx.rollback(); + } + + //A SET holds a value once, which an addition of it replaces, and a LIST as often as it was added, of which a + //removal takes out one + @ParameterizedTest + @ValueSource(strings = {"LIST", "SET"}) + public void testAFloatAddedAgainIsHeldOnceByASetAndTwiceByAList(Cardinality cardinality) throws Exception { + final KeyInformation key = IndexProviderTest.of(Float.class, cardinality); + final String stored = String.valueOf((double) 0.1f); + IndexTransaction tx = openTx(key); + tx.add(STORE, "doc", KEY, 0.1f, true); + tx.commit(); + tx = openTx(key); + tx.add(STORE, "doc", KEY, 0.1f, false); + tx.commit(); + final List held = cardinality == Cardinality.SET ? Collections.singletonList(stored) + : Arrays.asList(stored, stored); + assertEquals(held, storedValues("doc", KEY)); + + tx = openTx(key); + tx.delete(STORE, "doc", KEY, 0.1f, false); + tx.commit(); + assertEquals(held.subList(1, held.size()), storedValues("doc", KEY)); + tx = openTx(key); + assertEquals(cardinality == Cardinality.SET ? Collections.emptySet() : Collections.singleton("doc"), + query(tx, PredicateCondition.of(KEY, Cmp.EQUAL, 0.1f))); + tx.rollback(); + } + + //Strings with an unpaired surrogate, and one with a pair, which UTF-8 holds as it is + @ParameterizedTest + @CsvSource({"LIST, DEFAULT", "LIST, TEXT", "LIST, STRING", "LIST, TEXTSTRING", + "SET, DEFAULT", "SET, TEXT", "SET, STRING", "SET, TEXTSTRING"}) + public void testRemovedStringsWithUnpairedSurrogatesLeaveTheDocument(Cardinality cardinality, Mapping mapping) + throws Exception { + final KeyInformation key = IndexProviderTest.of(String.class, cardinality, mapping.asParameter()); + final String[] removed = {"Ab\uD800c", "\uDC00", "x\uD83D\uDE00y"}; + IndexTransaction tx = openTx(key); + for (final String value : removed) { + tx.add(STORE, "doc", KEY, value, true); + } + tx.add(STORE, "doc", KEY, "Kept", true); + tx.commit(); + assertStringsFound(key, mapping, removed, Collections.singleton("doc")); + tx = openTx(key); + for (final String value : removed) { + tx.delete(STORE, "doc", KEY, value, false); + } + tx.commit(); + + assertEquals(Collections.singletonList(mapping == Mapping.STRING ? "Kept" : "kept"), storedValues("doc", KEY)); + final Optional wholeStrings = LuceneIndex.getDualFieldName(KEY, key); + if (wholeStrings.isPresent()) { + assertEquals(Collections.singletonList("Kept"), storedValues("doc", wholeStrings.get())); + } + assertStringsFound(key, mapping, removed, Collections.emptySet()); + } + + //The documents found by eq of each of the strings, on a key of whole strings, and by a word of the first one, on a + //key of words + private void assertStringsFound(KeyInformation key, Mapping mapping, String[] values, Set expected) + throws BackendException { + final IndexTransaction tx = openTx(key); + if (mapping == Mapping.STRING || mapping == Mapping.TEXTSTRING) { + for (final String value : values) { + assertEquals(expected, query(tx, PredicateCondition.of(KEY, Cmp.EQUAL, value)), value); + } + } + if (mapping != Mapping.STRING) { + assertEquals(expected, query(tx, PredicateCondition.of(KEY, Text.CONTAINS, "ab"))); + } + tx.rollback(); + } + + //Each key keeps the value which wasn't removed, which it holds before the removed one, so that a comparison which + //took the two for one value, as one of -0.0 and 0.0 by their numbers would, removes the kept value instead + @ParameterizedTest + @ValueSource(strings = {"LIST", "SET"}) + public void testRemovedValuesOfEachDataTypeLeaveTheDocument(Cardinality cardinality) throws Exception { + final Map keys = new HashMap<>(); + for (final Object[] key : KEYS) { + keys.put((String) key[0], + IndexProviderTest.of((Class) key[1], cardinality, ((Mapping) key[2]).asParameter())); + } + IndexTransaction tx = openTx(keys); + for (final Object[] key : KEYS) { + tx.add(STORE, "doc", (String) key[0], key[3], true); + tx.add(STORE, "doc", (String) key[0], key[4], true); + } + tx.commit(); + tx = openTx(keys); + for (final Object[] key : KEYS) { + tx.delete(STORE, "doc", (String) key[0], key[4], false); + } + tx.commit(); + + for (final Object[] key : KEYS) { + assertEquals(Collections.singletonList(key[5]), storedValues("doc", (String) key[0]), (String) key[0]); + } + assertEquals(Collections.singletonList("Kept Words"), + storedValues("doc", LuceneIndex.getDualFieldName("textString", keys.get("textString")).get())); + } + + //The values of the field which the document holds, as Lucene reads them back from the directory of the store + private static List storedValues(String document, String field) throws IOException { + final String indexDirectory = LuceneIndexTest.getLocalLuceneTestConfig() + .get(GraphDatabaseConfiguration.INDEX_DIRECTORY); + try (FSDirectory directory = FSDirectory.open(new File(indexDirectory, STORE).toPath()); + DirectoryReader reader = DirectoryReader.open(directory)) { + final IndexSearcher searcher = new IndexSearcher(reader); + final TopDocs hits = searcher.search(new TermQuery(new Term(LuceneIndex.DOCID, document)), 2); + assertEquals(1, hits.scoreDocs.length, document); + return Arrays.asList(searcher.storedFields().document(hits.scoreDocs[0].doc).getValues(field)); + } + } + + private static Set query(IndexTransaction tx, PredicateCondition condition) + throws BackendException { + return tx.queryStream(new IndexQuery(STORE, condition)).collect(Collectors.toSet()); + } + + private static long count(IndexTransaction tx, PredicateCondition condition) + throws BackendException { + return tx.queryAggregation(new IndexQuery(STORE, condition), Aggregation.COUNT).longValue(); + } +}