Enabled Neo4jGraphQLSubscriptionsCDCEngine produces approx. 112 erroneous CDC queries per minute

[andreloeffelmann]

Describe the bug
We have the Neo4jGraphQLSubscriptionsCDCEngine active with flag onlyGraphQLEvents set to true.
No other configurations set.

This setup leads to CDC queries (CALL db.cdc.query...) within the database producing the errors 52N37 and 50N42.
The errors/queries occur around 112 times / minute.
The generated queries all look the same according to the Aura Ops Manager:

CALL db.cdc.query(******, [{ select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }])

Expected behavior
An enabled subscription engine should not produce faulty CDC queries

Screenshots

System

• Aura Virtual Dedicated Cloud
• Version: @neo4j/[email protected]

[angrykoala]

Hi @andreloeffelmann

I've double checked and the query seems to be correct. I've noticed that those two error codes are sent when attempting to query CDC events on a database where the CDC is not enabled. Is CDC enable in FULL mode for that database?
The instructions to enable CDC for an Aura database: https://neo4j.com/docs/cdc/current/get-started/aura/

[andreloeffelmann]

Hi @angrykoala, yes, CDC is enabled in FULL mode.

[angrykoala]

Ok, I'm gonna need some more info then to debug this one. The number of errors seem to roughly match the CDC polling for subscriptions (~1 second, with 2 errors per failed query)

• Do subscriptions work without the onlyGraphQLEvents flag?
• Is the GraphQL server logging any errors?
• Is there a GraphQL schema that you could share to replicate this issue?

Thanks

[andreloeffelmann]

• Do subscriptions work without the onlyGraphQLEvents flag?
  • yes, in theory. But this is not feasible for us due to Neo4jGraphQLSubscriptionsCDCEngine with enabled CDC@Aura kills the neo4j-graphql server whenever massive data changes happen within a short period of time #5787. I cannot tell whether the errors do not occur without the flag since disabling it is not possible (see the referenced ticket).
• Is the GraphQL server logging any errors?
  • we indeed have some error logs within the GraphQL server. But they do occur less frequently than the errors in the database
  • the following error was logged 4 times within the last 6 hours: gqlStatusDescription: 'error: general processing exception - unexpected error. Failed to connect to server. Please ensure that your database is listening on the correct host and port and that you have compatible encryption settings both on Neo4j server and driver. Note that the default encryption setting has changed in Neo4j 4.0. Caused by: read ECONNRESET'
  • this error was logged 2 times within the last 6 hours: gqlStatusDescription: "error: general processing exception - unexpected error. Failed to connect to server. Please ensure that your database is listening on the correct host and port and that you have compatible encryption settings both on Neo4j server and driver. Note that the default encryption setting has changed in Neo4j 4.0. Caused by: Connection lost. Server didn't respond in 60000ms"
  • no other errors have been logged
• Is there a GraphQL schema that you could share to replicate this issue?
  • this is not easily possible. it's kind of internal stuff. if absolutely required we could try to find a way how to sanitize it somehow

[angrykoala]

Yeah, those errors seem to be unrelated to this particular issue. Are you able to try a smaller schema to test if that still fails? I'm suspecting that this issue may be cause by the txMetadata filter having problems to filter through so many CDC events but I'd like to rule out it being a malformed query by the graphql library

[andreloeffelmann]

Hey @angrykoala , so it turns out that we have had an old server with an old version of neo4j-graphql (v.6.2.0) running in parallel to our current one which no one was aware of anymore 🙄
Switching that server of solved the issue - no more failed queries.
Sorry for the inconvenience!