Leave out removed vertices before sorting a query in memory

[porunov]

Fixes #5099

A graph query ordered by a key which its index doesn't sort it by, such as one which a composite index answers, or a mixed index which doesn't hold the order key, failed with IllegalStateException: Vertex with id N was removed in a transaction which had removed one of the vertices it matched. QueryProcessor's PreSortingIterator sorted what the index returned, reading the order key of each element, before getFilterIterator left out the elements which the transaction had deleted. The query now leaves out the removed elements before it sorts, and the other deleted ones as before.

Changes

• QueryProcessor.PreSortingIterator leaves out the elements which the transaction removed (isRemoved()), whose order values can't be read, before it sorts what the backend returned. Its check against fetching MAX_SORT_ITERATION elements still counts every element which the backend returned.
  • getFilterIterator still leaves out the other elements which the transaction deleted, such as those which it changed so that they no longer match, as the sorted elements are reached, as for an unsorted query: only the elements which the caller reads are checked, and a change which it makes while it reads them is seen.
• Changelog: a section before Graph queries read on past the edges and properties which the transaction removed.

Tests

• JanusGraphTest.testQueriesSortedInMemorySkipTheElementsWhichTheTransactionRemoved (every storage backend; composite indexes of vertices and of edges by kind): in a transaction which drops a vertex, and with it its edge, and another edge, g.V().has('kind', 'p').order().by('name'), the same descending with a limit, graph.query().has('kind', 'p').orderBy('name', asc).vertices(), the same orders of edges, and an order of a vertex's edges by name in local(), return the vertices and edges which are left, in order.
• JanusGraphIndexTest.testQueriesSortedInMemorySkipTheElementsWhichTheTransactionRemoved (every index backend; mixed indexes named mixed and edges, which the Solr test container creates, holding kind but not name): the same.
• JanusGraphTest.testQueriesSortedInMemoryLeaveOutTheElementsRemovedWhileTheyAreRead (every storage backend; a composite index of vertices by kind): a vertex which the transaction removes, or whose kind it changes, while it reads graph.query().has('kind', 'p').orderBy('name', asc).vertices() is left out when it is reached, and so is an edge which it removes while it reads v.query().labels('road').orderBy('name', asc).edges() of a vertex whose properties it changed, as for unordered queries.
• QueryProcessorTest.leavesOutTheRemovedElementsBeforeSortingWhatTheBackendReturns (no backend): of what a mocked executor returns, a removed element, whose order value can't be read, is left out before the others are sorted.

On master dd4f0f845, both testQueriesSortedInMemorySkipTheElementsWhichTheTransactionRemoved tests fail at their first query with IllegalStateException: Vertex with id N was removed., on inmemory, Lucene, Elasticsearch and Solr, and so does the QueryProcessorTest case. The orders of edges, whose removed ids turn into no element, returned the edges which are left already. testQueriesSortedInMemoryLeaveOutTheElementsRemovedWhileTheyAreRead passes on master, which checks the sorted elements as they are reached too.

Mutations:

• Master's PreSortingIterator, which sorts the removed elements with the others, fails both testQueriesSortedInMemorySkipTheElementsWhichTheTransactionRemoved tests and the QueryProcessorTest case, as on master.
• Checking every deletion before the sort (executor.isDeleted on each element which the backend returns) instead of as the elements are reached, as a first version of this change did, fails each of the three parts of testQueriesSortedInMemoryLeaveOutTheElementsRemovedWhileTheyAreRead: the removed vertex is reached and its name can't be read, and the changed vertex and the removed edge are returned.

Locally, Java 11, with checkstyle (no violations):

• janusgraph-driver (130), janusgraph-core (229), janusgraph-inmemory (656, 85 skipped), janusgraph-test (1,102), janusgraph-berkeleyje (352, 9 skipped) and janusgraph-lucene (394, 148 skipped) pass.
• The TinkerPop process suites of inmemory (-Dtest.skip.tp=false, InMemoryJanusGraphProcessTest and InMemoryMultiQueryJanusGraphProcessTest, 1,860 tests, 80 skipped) pass.
• BerkeleyElasticsearchTest (118, 1 skipped) passes against Elasticsearch 9.5.4 in a container.
• On Solr, JanusGraphIndexTest.testQueriesSortedInMemorySkipTheElementsWhichTheTransactionRemoved passes on BerkeleyJE against an in-JVM MiniSolrCloudCluster (Solr 9.10.1), as BerkeleySolrTest runs it against its container, and fails there with master's QueryProcessor with the IllegalStateException. The Solr test container is left to CI.

Notes

• With Turn the ids of an index query into elements in one pass #5075, which changes how SubqueryIterator turns the ids of an index query into elements, this PR merges without conflicts; it changes only what QueryProcessor does with the elements.
• A query which the index sorts never sorted the elements in memory, and wasn't affected.
• Cost: only isRemoved(), which reads nothing, is checked before the sort, so the query does no more work than on master. Checking every deletion there instead costs a scan of the vertex's added relations of a constrained label for each element which the backend returns, where the check as the elements are reached takes only those which the caller reads: on inmemory, v.query().direction(OUT).labels('child').orderBy('name', asc).limit(10).edges() on a vertex with 20,000 edges of a ONE2MANY label child, in a transaction which added 2,000 more, took 1.8 to 2.0 s that way, against 22 to 33 ms on master and 23 to 32 ms with this change (0.21 to 0.23 s, against 36 to 49 and 31 to 36 ms, with 200 added edges; the median of 7 timed runs after 3 untimed ones, in each of two runs).

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

The focused fix preserves existing limits and lazy filtering, includes targeted regression coverage, and has no identified blocking issues.

0 open findings

What changed in this PR

Fixes #5099 by excluding removed elements before JanusGraph sorts query results in memory, avoiding reads of removed vertices’ properties.

Changes:

• Preserves backend fetch limits and lazy deletion checks.
• Adds unit and backend regression tests for removals before sorting and during iteration.
• Documents the fix in the changelog.

File	Description
janusgraph-core/​src/​test/​java/​org/​janusgraph/​graphdb/​query/​QueryProcessorTest.java	Tests filtering removed elements before sorting.
janusgraph-core/​src/​main/​java/​org/​janusgraph/​graphdb/​query/​QueryProcessor.java	Skips removed elements while counting all fetched results.
janusgraph-backend-testutils/​src/​main/​java/​org/​janusgraph/​graphdb/​JanusGraphTest.java	Covers composite indexes and mutations during iteration.
janusgraph-backend-testutils/​src/​main/​java/​org/​janusgraph/​graphdb/​JanusGraphIndexTest.java	Covers sorting with mixed indexes.
docs/​changelog.md	Explains the corrected query behavior.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.