Skip to content

Commit 64be352

Browse files
Deflake tests in GraphSONTypedCompatibilityTest, GraphSONUntypedCompatibilityTest and GraphBinaryCompatibilityTest
We observed several tests in GraphSONTypedCompatibilityTest that exhibited flaky behavior when executed with NonDex. Specifically speaking, we can reproduce them by using the following commands. ``` mvn clean install -DskipTests -Drat.skip=true ``` * Test shouldReadWriteEdge[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteEdge[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWriteEdge[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteEdge[expect(v3)]" -Drat.skip=true ``` * Test shouldReadWritePath[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWritePath[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWritePath[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWritePath[expect(v3)]" -Drat.skip=true ``` * Test shouldReadWriteProperty[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteProperty[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWriteProperty[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteProperty[expect(v3)]" -Drat.skip=true ``` * Test shouldReadWriteTraverser[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteTraverser[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWriteTraverser[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteTraverser[expect(v3)]" -Drat.skip=true ``` * Test shouldReadWriteVertexProperty[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertexProperty[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWriteVertexProperty[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertexProperty[expect(v3)]" -Drat.skip=true ``` * Test shouldReadWriteVertex[expect(v2)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertex[expect(v2)]" -Drat.skip=true ``` * Test shouldReadWriteVertex[expect(v3)] ```bash mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertex[expect(v3)]" -Drat.skip=true ``` And the error should be something like this: ``` [ERROR] Failures: [ERROR] GraphSONTypedCompatibilityTest>AbstractTypedCompatibilityTest.shouldReadWriteEdge:322 expected:<e[17][7-develops->10]> but was:<e[13][1-develops->10]> [ERROR] GraphSONTypedCompatibilityTest>AbstractTypedCompatibilityTest.shouldReadWriteEdge:322 expected:<e[17][7-develops->10]> but was:<e[13][1-develops->10]> ``` Upon our investigation, the root cause is the use of: ``` protected Map<Object, Vertex> vertices = new ConcurrentHashMap<>(); protected Map<Object, Edge> edges = new ConcurrentHashMap<>(); ``` in TinkerGraph.java, which does not guarantee a deterministic order. The simplest fix would be to replace ConcurrentHashMap with LinkedHashMap, as we did in a previous PR. We've confirmed that this change could remove the flakiness of these tests. However, we are concerned that such a change might introduce unintended side effects in the code under test. Another possible fix would be to deterministically select a fixed id, but that approach would make the test become sensitive to future implementation changes. Thus, we decided to use this sorting-based approach to deflake this test. This fix also deflakes the entire GraphSONUntypedCompatibilityTest and GraphBinaryCompatibilityTest, which used to fail under NonDex in the 3.7-dev branch, this could be verified by: ``` mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUntypedCompatibilityTest" -Drat.skip=true ``` ``` mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphbinary.GraphBinaryCompatibilityTest" -Drat.skip=true ``` Co-authored-by: Siddhi Jhunjhunwala <[email protected]>
1 parent a795d38 commit 64be352

File tree

1 file changed

+22
-7
lines changed
  • gremlin-util/src/test/java/org/apache/tinkerpop/gremlin/structure/io

1 file changed

+22
-7
lines changed

gremlin-util/src/test/java/org/apache/tinkerpop/gremlin/structure/io/Model.java

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
import org.apache.tinkerpop.gremlin.structure.VertexProperty;
4141
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerFactory;
4242
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
43+
import org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils;
4344
import org.apache.tinkerpop.gremlin.util.message.RequestMessage;
4445
import org.apache.tinkerpop.gremlin.util.message.ResponseMessage;
4546

@@ -115,12 +116,24 @@ private Model() {
115116
addCoreEntry(new java.sql.Timestamp(1481750076295L), "Timestamp", "");
116117
addCoreEntry(UUID.fromString("41d2e28a-20a4-4ab0-b379-d810dede3786"), "UUID");
117118

118-
addGraphStructureEntry(graph.edges().next(), "Edge", "");
119-
addGraphStructureEntry(g.V().out().out().path().next(), "Path", "");
120-
addGraphStructureEntry(graph.edges().next().properties().next(), "Property", "");
119+
addGraphStructureEntry(IteratorUtils.list(graph.edges()).stream()
120+
.sorted((e1, e2) -> Integer.compare((Integer)e1.id(), (Integer)e2.id()))
121+
.iterator().next(), "Edge", "");
122+
addGraphStructureEntry(g.V().order().by(T.id).out().out().path().next(), "Path", "");
123+
addGraphStructureEntry(IteratorUtils.list(IteratorUtils.list(graph.edges()).stream()
124+
.sorted((e1, e2) -> Integer.compare((Integer)e1.id(), (Integer)e2.id()))
125+
.iterator().next().properties()).stream()
126+
.sorted((p1, p2) -> p1.key().compareTo(p2.key()))
127+
.iterator().next(), "Property", "");
121128
addGraphStructureEntry(graph, "TinkerGraph", "`TinkerGraph` has a custom serializer that is registered as part of the `TinkerIoRegistry`.");
122-
addGraphStructureEntry(graph.vertices().next(), "Vertex", "");
123-
addGraphStructureEntry(graph.vertices().next().properties().next(), "VertexProperty", "");
129+
addGraphStructureEntry(IteratorUtils.list(graph.vertices()).stream()
130+
.sorted((v1, v2) -> Integer.compare((Integer)v1.id(), (Integer)v2.id()))
131+
.iterator().next(), "Vertex", "");
132+
addGraphStructureEntry(IteratorUtils.list(IteratorUtils.list(graph.vertices()).stream()
133+
.sorted((v1, v2) -> Integer.compare((Integer)v1.id(), (Integer)v2.id()))
134+
.iterator().next().properties()).stream()
135+
.sorted((p1, p2) -> Long.compare((Long)p1.id(), (Long)p2.id()))
136+
.iterator().next(), "VertexProperty", "");
124137

125138
addGraphProcessEntry(SackFunctions.Barrier.normSack, "Barrier", "");
126139
addGraphProcessEntry(new Bytecode.Binding("x", 1), "Binding", "A \"Binding\" refers to a `Bytecode.Binding`.");
@@ -153,7 +166,7 @@ private Model() {
153166
// TextP was only added at 3.4.0 and is not supported with untyped GraphSON of any sort
154167
addGraphProcessEntry(TextP.containing("ark"), "TextP", "");
155168
addGraphProcessEntry(createStaticTraversalMetrics(), "TraversalMetrics", "");
156-
addGraphProcessEntry(g.V().hasLabel("person").asAdmin().nextTraverser(), "Traverser", "");
169+
addGraphProcessEntry(g.V().hasLabel("person").order().by(T.id).asAdmin().nextTraverser(), "Traverser", "");
157170

158171
final Map<String,Object> requestBindings = new HashMap<>();
159172
requestBindings.put("x", 1);
@@ -192,7 +205,9 @@ private Model() {
192205
addResponseMessageEntry(responseMessage, "Authentication Challenge", "When authentication is enabled, an initial request to the server will result in an authentication challenge. The typical response message will appear as follows, but handling it could be different depending on the SASL implementation (e.g. multiple challenges maybe requested in some cases, but not in the default provided by Gremlin Server).");
193206
responseMessage = ResponseMessage.build(UUID.fromString("41d2e28a-20a4-4ab0-b379-d810dede3786")).
194207
code(org.apache.tinkerpop.gremlin.util.message.ResponseStatusCode.SUCCESS).
195-
result(Collections.singletonList(graph.vertices().next())).create();
208+
result(Collections.singletonList(IteratorUtils.list(graph.vertices()).stream()
209+
.sorted((v1, v2) -> Integer.compare((Integer)v1.id(), (Integer)v2.id()))
210+
.iterator().next())).create();
196211
addResponseMessageEntry(responseMessage, "Standard Result", "The following `ResponseMessage` is a typical example of the typical successful response Gremlin Server will return when returning results from a script.");
197212

198213
addExtendedEntry(new BigDecimal(new BigInteger("123456789987654321123456789987654321")), "BigDecimal", "");

0 commit comments

Comments
 (0)