From 64be35233759ec577d95cfea113283757d815b1c Mon Sep 17 00:00:00 2001 From: Kangwei Zhu Date: Tue, 14 Oct 2025 09:57:08 -0400 Subject: [PATCH] Deflake tests in GraphSONTypedCompatibilityTest, GraphSONUntypedCompatibilityTest and GraphBinaryCompatibilityTest We observed several tests in GraphSONTypedCompatibilityTest that exhibited flaky behavior when executed with NonDex. Specifically speaking, we can reproduce them by using the following commands. ``` mvn clean install -DskipTests -Drat.skip=true ``` * Test shouldReadWriteEdge[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteEdge[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWriteEdge[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteEdge[expect(v3)]" -Drat.skip=true ``` * Test shouldReadWritePath[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWritePath[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWritePath[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWritePath[expect(v3)]" -Drat.skip=true ``` * Test shouldReadWriteProperty[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteProperty[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWriteProperty[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteProperty[expect(v3)]" -Drat.skip=true ``` * Test shouldReadWriteTraverser[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteTraverser[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWriteTraverser[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteTraverser[expect(v3)]" -Drat.skip=true ``` * Test shouldReadWriteVertexProperty[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertexProperty[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWriteVertexProperty[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertexProperty[expect(v3)]" -Drat.skip=true ``` * Test shouldReadWriteVertex[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertex[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWriteVertex[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertex[expect(v3)]" -Drat.skip=true ``` And the error should be something like this: ``` [ERROR] Failures: [ERROR] GraphSONTypedCompatibilityTest>AbstractTypedCompatibilityTest.shouldReadWriteEdge:322 expected:10]> but was:10]> [ERROR] GraphSONTypedCompatibilityTest>AbstractTypedCompatibilityTest.shouldReadWriteEdge:322 expected:10]> but was:10]> ``` Upon our investigation, the root cause is the use of: ``` protected Map vertices = new ConcurrentHashMap<>(); protected Map edges = new ConcurrentHashMap<>(); ``` in TinkerGraph.java, which does not guarantee a deterministic order. The simplest fix would be to replace ConcurrentHashMap with LinkedHashMap, as we did in a previous PR. We've confirmed that this change could remove the flakiness of these tests. However, we are concerned that such a change might introduce unintended side effects in the code under test. Another possible fix would be to deterministically select a fixed id, but that approach would make the test become sensitive to future implementation changes. Thus, we decided to use this sorting-based approach to deflake this test. This fix also deflakes the entire GraphSONUntypedCompatibilityTest and GraphBinaryCompatibilityTest, which used to fail under NonDex in the 3.7-dev branch, this could be verified by: ``` mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUntypedCompatibilityTest" -Drat.skip=true ``` ``` mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphbinary.GraphBinaryCompatibilityTest" -Drat.skip=true ``` Co-authored-by: Siddhi Jhunjhunwala --- .../tinkerpop/gremlin/structure/io/Model.java | 29 ++++++++++++++----- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/gremlin-util/src/test/java/org/apache/tinkerpop/gremlin/structure/io/Model.java b/gremlin-util/src/test/java/org/apache/tinkerpop/gremlin/structure/io/Model.java index 19969f2760a..dd2a5b13097 100644 --- a/gremlin-util/src/test/java/org/apache/tinkerpop/gremlin/structure/io/Model.java +++ b/gremlin-util/src/test/java/org/apache/tinkerpop/gremlin/structure/io/Model.java @@ -40,6 +40,7 @@ import org.apache.tinkerpop.gremlin.structure.VertexProperty; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerFactory; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils; import org.apache.tinkerpop.gremlin.util.message.RequestMessage; import org.apache.tinkerpop.gremlin.util.message.ResponseMessage; @@ -115,12 +116,24 @@ private Model() { addCoreEntry(new java.sql.Timestamp(1481750076295L), "Timestamp", ""); addCoreEntry(UUID.fromString("41d2e28a-20a4-4ab0-b379-d810dede3786"), "UUID"); - addGraphStructureEntry(graph.edges().next(), "Edge", ""); - addGraphStructureEntry(g.V().out().out().path().next(), "Path", ""); - addGraphStructureEntry(graph.edges().next().properties().next(), "Property", ""); + addGraphStructureEntry(IteratorUtils.list(graph.edges()).stream() + .sorted((e1, e2) -> Integer.compare((Integer)e1.id(), (Integer)e2.id())) + .iterator().next(), "Edge", ""); + addGraphStructureEntry(g.V().order().by(T.id).out().out().path().next(), "Path", ""); + addGraphStructureEntry(IteratorUtils.list(IteratorUtils.list(graph.edges()).stream() + .sorted((e1, e2) -> Integer.compare((Integer)e1.id(), (Integer)e2.id())) + .iterator().next().properties()).stream() + .sorted((p1, p2) -> p1.key().compareTo(p2.key())) + .iterator().next(), "Property", ""); addGraphStructureEntry(graph, "TinkerGraph", "`TinkerGraph` has a custom serializer that is registered as part of the `TinkerIoRegistry`."); - addGraphStructureEntry(graph.vertices().next(), "Vertex", ""); - addGraphStructureEntry(graph.vertices().next().properties().next(), "VertexProperty", ""); + addGraphStructureEntry(IteratorUtils.list(graph.vertices()).stream() + .sorted((v1, v2) -> Integer.compare((Integer)v1.id(), (Integer)v2.id())) + .iterator().next(), "Vertex", ""); + addGraphStructureEntry(IteratorUtils.list(IteratorUtils.list(graph.vertices()).stream() + .sorted((v1, v2) -> Integer.compare((Integer)v1.id(), (Integer)v2.id())) + .iterator().next().properties()).stream() + .sorted((p1, p2) -> Long.compare((Long)p1.id(), (Long)p2.id())) + .iterator().next(), "VertexProperty", ""); addGraphProcessEntry(SackFunctions.Barrier.normSack, "Barrier", ""); addGraphProcessEntry(new Bytecode.Binding("x", 1), "Binding", "A \"Binding\" refers to a `Bytecode.Binding`."); @@ -153,7 +166,7 @@ private Model() { // TextP was only added at 3.4.0 and is not supported with untyped GraphSON of any sort addGraphProcessEntry(TextP.containing("ark"), "TextP", ""); addGraphProcessEntry(createStaticTraversalMetrics(), "TraversalMetrics", ""); - addGraphProcessEntry(g.V().hasLabel("person").asAdmin().nextTraverser(), "Traverser", ""); + addGraphProcessEntry(g.V().hasLabel("person").order().by(T.id).asAdmin().nextTraverser(), "Traverser", ""); final Map requestBindings = new HashMap<>(); requestBindings.put("x", 1); @@ -192,7 +205,9 @@ private Model() { addResponseMessageEntry(responseMessage, "Authentication Challenge", "When authentication is enabled, an initial request to the server will result in an authentication challenge. The typical response message will appear as follows, but handling it could be different depending on the SASL implementation (e.g. multiple challenges maybe requested in some cases, but not in the default provided by Gremlin Server)."); responseMessage = ResponseMessage.build(UUID.fromString("41d2e28a-20a4-4ab0-b379-d810dede3786")). code(org.apache.tinkerpop.gremlin.util.message.ResponseStatusCode.SUCCESS). - result(Collections.singletonList(graph.vertices().next())).create(); + result(Collections.singletonList(IteratorUtils.list(graph.vertices()).stream() + .sorted((v1, v2) -> Integer.compare((Integer)v1.id(), (Integer)v2.id())) + .iterator().next())).create(); addResponseMessageEntry(responseMessage, "Standard Result", "The following `ResponseMessage` is a typical example of the typical successful response Gremlin Server will return when returning results from a script."); addExtendedEntry(new BigDecimal(new BigInteger("123456789987654321123456789987654321")), "BigDecimal", "");