Skip to content

Order by a key of several values as the indexes do - #5102

Open
porunov wants to merge 1 commit into
JanusGraph:masterfrom
porunov:order-by-keys-of-several-values
Open

Order by a key of several values as the indexes do#5102
porunov wants to merge 1 commit into
JanusGraph:masterfrom
porunov:order-by-keys-of-several-values

Conversation

@porunov

@porunov porunov commented Oct 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Fixes #5101

A graph query ordered by a key of several values, LIST or SET, failed with IllegalStateException: Multiple properties exist for the provided key wherever JanusGraph compared an element with several values of the key itself, as OrderList read the key with valueOrNull, which takes a single value:

  • where it merged the vertices which a transaction added or changed with the results of a mixed index, as a vertex with two names which the transaction made match the query;
  • where it sorted the results in memory, as those of a composite index, or of g.V().order().by('names'), which no index answers, on a graph whose mixed index backends all sort by such a key;
  • where it merged the queries of the branches of an or(), such as g.V().or(has('uid', 'u1'), has('uid', 'u2')).order().by('names') on such a graph.

On Lucene, which keeps doc values only for keys of one value, a query of the API ordered by such a key, which a Lucene index holding the key answered, failed with JanusGraphException: Could not call index, caused by IllegalStateException: unexpected docvalues type NONE for field 'names' (expected=SORTED); so did a traversal's order of edges by a key declared LIST, which a traversal folds in whatever the cardinality of the key, as an edge holds one value of a key at most.

JanusGraph now compares such an element by the least of its values in an ascending order and by the greatest in a descending one, which is how Elasticsearch and OpenSearch sort a field of several values by default (JanusGraph sets no sort mode), and Solr a multi-valued field, so that the elements which JanusGraph places come where the index puts the others. An element without a value of the key still comes last. An index whose backend can't sort by such a key, as Lucene's, no longer takes the order, and JanusGraph sorts the results itself.

Changes

  • OrderList.OrderEntry.value(element) (new) is the value by which the order compares an element: the element's value of the key, or, for a key of LIST or SET cardinality, the first of its values in the order, compared as any two values of the key are (compareOrderValues: strings by code point), so the least for an ascending order and the greatest for a descending one; null where it has none. OrderEntry.compare, OrderList.values() and the new OrderList.value(element, position) read the values through it. Whether a key holds several values is decided once, when the order is built.
  • QueryProcessor reads the value of the first order key of the last new element within the limit through the order as well, which it read with valueOrNull and failed on. The range which Find the vertices a transaction changed for its graph queries by key and value #5023 then gives the index query still applies to keys of one value only.
  • IndexSelectionUtil.indexCoversOrder(MixedIndexType, OrderList, IndexSerializer) (new) holds an index to sort by the order only where its backend can sort by each of its keys of several values (IndexFeatures.supportsOrderingListProperty), which Lucene's can't. IndexSerializer.getQuery and the brute force and approximate index selection strategies use it; the old form is deprecated.
  • GraphCentricQueryBuilder holds the indexes which a strategy selects to sort the results only where the first of them was given the query's order, the same keys in the same directions (sortedOnlyWhereTheIndexSorts and takesTheOrder, new). A custom strategy which decides by the deprecated two-argument form holds an index which wasn't given the order sorted, and one could give the index only part of the order; their results, in the index's own order or with the ties of that part in it, would have gone through as sorted. JanusGraph sorts them now.
  • StandardJanusGraphTx no longer leaves the vertices whose values of a key of several values the transaction changed where an index which sorts by the key places them, by their values before the transaction (hasOrderKeysOfOneValue, removed): as for a key of one value, the query takes them among its new vertices and places them by the values they hold now, which OrderList can now compare.
  • Docs: the Ordering section of the index performance page says how an element of several values of an order key is ordered, by an index or by JanusGraph, which traversals JanusGraph folds such an order into, and which it leaves to TinkerPop.
  • Changelog: a section before Graph queries in a transaction find the vertices it changed, whatever the order of their conditions, with notes for custom index backends and index selection strategies. Two sentences of that section of Find the vertices a transaction changed for its graph queries by key and value #5023, which said that it placed only the vertices of order keys of one value, now hold for keys of several values too.

Tests

  • QueryTest.testOrderListComparesAKeyOfSeveralValuesByItsLeastOrGreatestValue (inmemory, the comparison itself): an OrderList by a LIST key of strings and by a SET key of integers, ascending and descending, compares a vertex of several values by its least or greatest one, picked by code point (U+FFFF before U+1F600, which String.compareTo puts first), places a vertex without a value last, gives the same order through values() and compareValues(), counts a value which the vertex gains, and orders the ties of a key of several values by a key of one value.
  • JanusGraphIndexTest.testOrderByAKeyOfSeveralValuesComparesItsLeastOrGreatestValue (every index backend; the mixed index is named mixed, which the Solr test container creates). Vertices with a LIST key names and a SET key scores of several values, one with one value, and one with none: graph.query() orders by names and by scores, of what the mixed index returns and of what a composite index returns, ascending and descending, by the least and the greatest value, the vertex without a value last; so do traversals and an or() of branches which a composite index answers where every mixed index backend sorts by such a key. Then a transaction makes a vertex with two names match the query, and gives another one a further name: the queries, a limited one included, and the traversals place them by the values they hold now.
  • JanusGraphIndexTest.testOrderingListProperty: where every mixed index backend sorts by a key of several values, g.V().order().by('name2'), which no index answers, is sorted by the least value ascending and by the greatest descending, as the index sorts it, rather than failing; elsewhere TinkerPop still fails on it.
  • JanusGraphIndexTest.testResultsWhichAStrategyHoldsSortedByAnIndexWithoutTheOrderAreSortedInMemory (every index backend): with query.index-select-strategy set to a strategy which decides by the deprecated indexCoversOrder(MixedIndexType, OrderList) (DeprecatedOrderCoverageIndexSelectionStrategy, new, in the test utilities), a query ordered by a LIST key, ascending and descending, which the strategy holds the mixed index to sort, comes sorted on Lucene too.
  • JanusGraphIndexTest.testResultsWhichAStrategyHoldsSortedByAnIndexGivenPartOfTheOrderAreSortedInMemory (every index backend): with a strategy which gives the mixed index only the first key of the order (PartialOrderIndexSelectionStrategy, new, in the test utilities), a query ordered by name and then by age, ascending and descending, comes sorted by both, where the index sorts the ties of name in an order of its own, which can't be both that of their ages and its reverse.
  • JanusGraphIndexTest.testOrderOfEdgesByAKeyDeclaredToHoldSeveralValues (every index backend): edges of a key declared LIST in the mixed index edges are ordered by it, ascending and descending, by a traversal, and by graph.query(), which places the edge without a value last in both directions.
  • JanusGraphIndexTest.testTransactionOrdersVerticesByTheValuesOfAKeyOfSeveralValuesWhichTheyHoldNow, Find the vertices a transaction changed for its graph queries by key and value #5023's testTransactionKeepsTheIndexOrderOfAKeyOfSeveralValues renamed: the values which the transaction gives the vertices turn the index's order around, ascending and descending, also next to a key of one value which it changed, where the test checked only which vertices came.

On master dd4f0f845, with these tests:

  • On Elasticsearch, testOrderByAKeyOfSeveralValuesComparesItsLeastOrGreatestValue fails with the IllegalStateException at the query of the composite index, testOrderingListProperty with it at the order which no index answers, and the renamed test of Find the vertices a transaction changed for its graph queries by key and value #5023 gets the index's order instead of the transaction's. A probe on master fails the same way for the or() of branches, and for each query in the transaction which made a vertex of two names match; the orders which Elasticsearch sorts agree with the least and the greatest value.
  • On Lucene, testOrderByAKeyOfSeveralValuesComparesItsLeastOrGreatestValue fails with Could not call index, caused by unexpected docvalues type NONE for field 'names' (expected=SORTED).
  • QueryTest's test fails with the IllegalStateException.
  • On Lucene, the order of edges and the query under the strategy of the deprecated form fail with Could not call index.

Mutations, each reverting one part of the change, are all caught:

  • An element of several values read by valueOrNull, as before: the unit test and the index test on Lucene fail with the IllegalStateException.
  • The value picked from the other end, the greatest ascending and the least descending: the unit test and the index test.
  • The values compared by their natural order when one is picked, strings by UTF-16 unit: the unit test.
  • An index which can't sort by a key of several values given the order: the index test on Lucene fails with Could not call index.
  • A strategy's sortedness trusted where the index wasn't given the order: the query under the strategy of the deprecated form returns [a, d, b, c] on Lucene, its index's own order.
  • A strategy's sortedness trusted where the index was given any order, as in a first version of this change: under the strategy which gives the index only part of the order, the query ordered by name and then by age descending returns [1, 2, 3, 0] on Lucene, the ties of name in the index's order.
  • The bound read by valueOrNull, as before: on Elasticsearch, the index test's limited query in the transaction fails with the IllegalStateException.
  • The vertices whose key of several values the transaction changed left where the index has them, as before: on Elasticsearch, the index test and the renamed test of Find the vertices a transaction changed for its graph queries by key and value #5023 get the index's order.

Locally, Java 11, with checkstyle (no violations), before the review's changes below:

  • janusgraph-driver (130), janusgraph-core (228), janusgraph-inmemory (652, 85 skipped), janusgraph-test (1,103), janusgraph-berkeleyje (350, 9 skipped) and janusgraph-lucene (396, 148 skipped) pass.
  • The TinkerPop process suites of inmemory (-Dtest.skip.tp=false, InMemoryJanusGraphProcessTest and InMemoryMultiQueryJanusGraphProcessTest, 1,860 tests, 80 skipped) pass.
  • BerkeleyElasticsearchTest (120, 1 skipped) passes against Elasticsearch 9.5.4 in a container.
  • The five index tests of keys of several values pass on Solr 9.10.1 in the JVM (MiniSolrCloudCluster, as the Solr test container doesn't start here). Solr sorts a multi-valued field by its least value ascending and its greatest descending, and the schemas JanusGraph ships and tests with put the documents without a value last (sortMissingLast). Solr's CI runs them in the container.
  • With them (the first index's order compared with the query's, the test of a strategy which gives part of the order, and the descending order of edges): QueryTest and JanusGraphStepStrategyTest (91) and the six index tests of keys of several values pass on Lucene, and the six on Elasticsearch 9.5.4 in a container; on Solr in the JVM, the two strategy tests and the order of edges pass.

Notes

  • This changes what queries return. A query which failed now returns its elements, ordered by the least or the greatest value as the index orders them. A vertex whose values of such a key the transaction changed now comes where its values place it, rather than where the index placed it by its values before the transaction. A query of the API which a Lucene index answers, ordered by such a key, is sorted by JanusGraph, reading all its matches first, as any query whose order no index covers is.
  • Traversals fold an order().by() of vertices by a key of several values only where every mixed index backend can sort by it, as before; elsewhere TinkerPop sorts, and fails on a vertex with several values of the key. An order of edges they fold whatever the cardinality of the key, as before, as an edge holds one value of a key at most.
  • With Leave out the elements which lack the order key, as TinkerPop does #5108, which leaves out the elements without a value of an order key, this one merges without conflicts. That change keeps the vertices without a value of an order which has a key of several values, as JanusGraph couldn't compare such a key in memory; it is to be rebased after this PR, and then lifts that exception and changes the comments and docs which give that reason.
  • Cost in a transaction. A query which an index sorts by a key of several values now checks every vertex whose values of the key the transaction changed, to place it, as Find the vertices a transaction changed for its graph queries by key and value #5023 does for keys of one value. On Elasticsearch 9.5.4, with 5,000 vertices whose LIST value the transaction extended, g.V().has('kind', 'k').order().by('several', desc).limit(10) took 234 ms the first time and then 42 and 37 ms, with the right vertices; master took 30, 8 and 6 ms and returned the wrong ones (z04656, z04081, z04196, where the values which the transaction gave put z04999, z04998, z04997 first), and the same query by a key of one value, which takes Find the vertices a transaction changed for its graph queries by key and value #5023's bound, 59, 39 and 34 ms.
  • Find the vertices a transaction changed for its graph queries by key and value #5023's bound, which reads an index only up to the last new value within a query's limit, still applies to keys of one value only (withoutResultsAfter). With the least or greatest value it could apply to keys of several values too, as a range on such a field holds where any value is in it: the least value at most the bound ascending, the greatest at least it descending. Left as a follow-up.
  • Custom index backends: graph.query() now gives an order by a key of several values only to an index whose backend declares IndexFeatures.supportsOrderingListProperty, where master gave it to any index holding the key. A backend which can sort by such a key but doesn't declare it now gets every match read and sorted by JanusGraph, which fails with a QueryException at 1,000,000 matches; the changelog says so.
  • Leave out removed vertices before sorting a query in memory #5100 should merge first. On Lucene, an order of vertices by such a key, which this PR now sorts in memory, goes through QueryProcessor's PreSortingIterator, which reads the order key of each vertex the index returns, a vertex which the transaction removed included: on this PR alone, after tx.getVertex(c).remove(), tx.query().has('gender', 'female').orderBy('names', asc).vertices() fails with Vertex with id N was removed, where master failed with Could not call index, so it's no regression for the backends which JanusGraph ships. With Leave out removed vertices before sorting a query in memory #5100, which leaves out the removed vertices before it sorts, it returns [a, b, d]. The changelog's "JanusGraph now sorts the results of such a query itself" holds in a transaction which removed such a vertex once Leave out removed vertices before sorting a query in memory #5100 is in.
  • With the PR which orders UUIDs as their string forms, which changes compareOrderValues, the value of several which an element is ordered by is picked in that order too; the two merge without conflicts.

🤖 Generated with Claude Code

@porunov @claude
Order by a key of several values as the indexes do
c9c2791 
Fixes JanusGraph#5101

A graph query ordered by a key of several values, LIST or SET, failed
with an IllegalStateException, "Multiple properties exist for the
provided key", wherever JanusGraph compared an element with several
values of the key itself: OrderList read the key with valueOrNull, which
takes a single value. That was so where it sorted the results in memory,
as those of a composite index; where it merged the vertices which a
transaction added or changed with those of a mixed index, as a vertex
which the transaction made match the query; and where it merged the
queries of the branches of an or().

OrderList now compares such an element by the least of its values for an
ascending order and by the greatest for a descending one, compared as
any two values of the key are, strings by code point. Elasticsearch,
OpenSearch and Solr sort such a key so by default, so the elements which
JanusGraph places come where the index puts the others. An element
without a value of the key still comes last.

Lucene keeps no doc values for a key of several values, so a query which
a Lucene index answered, ordered by such a key, failed with "Could not
call index", as did a traversal's order of edges, which it folds in
whatever the cardinality of the key. An index whose backend can't sort
by such a key no longer takes the order, and JanusGraph sorts the
results itself; IndexSelectionUtil.indexCoversOrder takes the
IndexSerializer for that, and its old form is deprecated.
GraphCentricQueryBuilder holds the indexes which a strategy selects to
sort the results only where the first of them was given the query's
order, each of its keys in its direction, so that results which a custom
strategy holds sorted by the old form, or by an index given only part of
the order, are sorted too.

A vertex whose values of such a key a transaction changed kept the place
which an index that sorts by the key gave it, as JanusGraph couldn't
compare it; it now comes where the values it holds place it, as for a
key of one value.

Signed-off-by: Oleksandr Porunov <alexandr.porunov@gmail.com>
Co-authored-by: Oleksandr Porunov <alexandr.porunov@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@porunov
porunov requested a balanced review from Copilot October 15, 2026 22:40
Copilot AI reviewed Oct 15, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-backend ordering changes and the still-open #5100 prerequisite require human integration and merge-sequencing verification.

0 open findings

What changed in this PR

Fixes #5101 by aligning JanusGraph’s in-memory ordering of multi-valued properties with mixed-index ordering, extending transaction-ordering corrections from #5023.

Changes:

  • Uses the minimum value ascending and maximum descending for LIST and SET keys.
  • Falls back to in-memory sorting when an index cannot provide the complete requested order.
  • Adds regression tests and documents ordering semantics and compatibility implications.
File Description
janusgraph-test/​src/​test/​java/​org/​janusgraph/​graphdb/​query/​QueryTest.java Tests multi-value comparisons and Unicode ordering.
janusgraph-core/​src/​main/​java/​org/​janusgraph/​graphdb/​transaction/​StandardJanusGraphTx.java Repositions vertices with changed multi-valued order keys.
janusgraph-core/​src/​main/​java/​org/​janusgraph/​graphdb/​query/​QueryProcessor.java Reads limit-bound values through the order definition.
janusgraph-core/​src/​main/​java/​org/​janusgraph/​graphdb/​query/​index/​IndexSelectionUtil.java Checks backend support for multi-valued ordering.
janusgraph-core/​src/​main/​java/​org/​janusgraph/​graphdb/​query/​index/​BruteForceIndexSelectionStrategy.java Uses capability-aware order coverage.
janusgraph-core/​src/​main/​java/​org/​janusgraph/​graphdb/​query/​index/​ApproximateIndexSelectionStrategy.java Accounts for ordering capabilities during selection.
janusgraph-core/​src/​main/​java/​org/​janusgraph/​graphdb/​query/​graph/​GraphCentricQueryBuilder.java Verifies the index receives the complete requested order.
janusgraph-core/​src/​main/​java/​org/​janusgraph/​graphdb/​internal/​OrderList.java Selects minimum or maximum property values for comparison.
janusgraph-core/​src/​main/​java/​org/​janusgraph/​graphdb/​database/​IndexSerializer.java Omits unsupported backend orders.
janusgraph-backend-testutils/​src/​main/​java/​org/​janusgraph/​graphdb/​PartialOrderIndexSelectionStrategy.java Supplies partial ordering for regression tests.
janusgraph-backend-testutils/​src/​main/​java/​org/​janusgraph/​graphdb/​JanusGraphIndexTest.java Tests backend ordering, transaction changes, and fallbacks.
janusgraph-backend-testutils/​src/​main/​java/​org/​janusgraph/​graphdb/​DeprecatedOrderCoverageIndexSelectionStrategy.java Exercises legacy order-coverage behavior.
docs/​schema/​index-management/​index-performance.md Explains multi-valued ordering and traversal limitations.
docs/​changelog.md Documents behavior changes and custom-backend compatibility.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@porunov porunov mentioned this pull request Oct 16, 2026
Draft

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

An order by a LIST or SET key fails with "Multiple properties exist for the provided key" where JanusGraph compares the elements itself

2 participants

@porunov