Skip to content

Ask the database for distinct edge connections instead of scanning edges - #2280

Draft
kmcginnes wants to merge 1 commit into
edge-stack/4-sample-edge-connections-per-typefrom
edge-stack/5-edge-connection-discovery-strategy
Draft

kmcginnes wants to merge 1 commit into
edge-stack/4-sample-edge-connections-per-typefrom
edge-stack/5-edge-connection-discovery-strategy

Conversation

@kmcginnes

@kmcginnes kmcginnes commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The sampled query from the previous layer can miss a rare edge connection. This adds an exact strategy and picks between them automatically.

  • Complete runs groupCount() keyed by a named project() over every edge, which returns one entry per distinct combination rather than one per edge. Sampled is the per-type branching from the previous layer.
  • A pure planDiscovery picks between them from the edge type count and the edge total. The total is now part of the query key, so an answer from a smaller graph isn't reused. Requests are bounded by edges read and by characters of query text, and sampled batches by the worst case of every branch reaching its limit.
  • The 50,000-edge scan budget is the largest volume that never failed in testing, not a prediction. When a complete scan turns out too large, it's abandoned and redone as sampled. Each request carries its own query id and abort signal, so the database stops working too.
  • When nothing cheaper is left, discovery throws EdgeConnectionDiscoveryError, which keeps the original error as the cause and says what to change:
    • Fetch timeout: "Raise the Fetch Timeout in this connection's advanced options, or clear it, since this request may simply need longer than that allows. Until then, the Schema view shows node types without the edge connections between them."
    • Database timeout or memory limit: "Raise the query timeout in the database configuration, such as the DB cluster parameter group for Neptune, or use an instance with more memory. Until then, the Schema view shows node types without the edge connections between them."
  • Neptune's MemoryLimitExceededException gets its own display message. The ADR records every rejected query shape and what it actually returned.

How to read

  1. docs/adr/20260918-ask-the-database-for-distinct-edge-connections.md, including every rejected shape. Start here.
  2. edgeConnectionsTemplate.ts, the two shapes.
  3. discoveryPlan.ts, the decision, pure and testable without a database.
  4. index.ts, running the plan, degrading and cancelling.
  5. discoveryError.ts, what the user is told.

Validation

  • Planner tests cover budgets, label-length chunking, the branch cap, and the unknown-total rule.
  • Discovery tests cover degrading from complete to sampled on fetch timeouts, database timeouts and memory limits, a user cancel propagating untouched, and abandoning in-flight requests.
  • The complete shape is native on every Neptune version tested and on TinkerPop 3.6.2. On Neptune 1.3.5.0, discovery took 53 requests and 119s, against 520 requests and 516s for one request per edge type.
  • pnpm checks and pnpm test clean on this layer.

Related Issues

Check List

  • I confirm that my contribution is made under the terms of the Apache 2.0 license.
  • I have verified pnpm checks passes with no errors.
  • I have verified pnpm test passes with no failures.
  • I have covered new added functionality with unit tests if necessary.
  • I have updated documentation if necessary.

@kmcginnes
kmcginnes added this pull request to stack #2283 September 25, 2026 22:25
kmcginnes added a commit that referenced this pull request Sep 25, 2026
## Description

- `mapWithConcurrency` rejected on the first failure, but its other
lanes kept looping and sending requests nobody was waiting for. The
first rejection now stops the pool from starting new work. Callbacks
already running aren't cancelled.
- This also affects openCypher and SPARQL schema and edge connection
fetches, which used to drain the whole queue after a failure.
- Later in the stack, edge connection discovery relies on this, so
abandoning an attempt actually stops it.

## Validation

- New test: after one callback rejects, no further callbacks start.
- `pnpm checks` and `pnpm test` clean on this layer.

## Related Issues

- Part of #2141 (split out of #2244)
- Layer 1 of 6. Stack, bottom first: #2281 → #2277 → #2278 → #2279 →
#2280 → #2282

### Check List

- [x] I confirm that my contribution is made under the terms of the
Apache 2.0 license.
- [x] I have verified `pnpm checks` passes with no errors.
- [x] I have verified `pnpm test` passes with no failures.
- [x] I have covered new added functionality with unit tests if
necessary.
- [x] I have updated documentation if necessary.
kmcginnes added a commit that referenced this pull request Sep 25, 2026
## Description

- `mapWithConcurrency` rejected on the first failure, but its other
lanes kept looping and sending requests nobody was waiting for. The
first rejection now stops the pool from starting new work. Callbacks
already running aren't cancelled.
- This also affects openCypher and SPARQL schema and edge connection
fetches, which used to drain the whole queue after a failure.
- Later in the stack, edge connection discovery relies on this, so
abandoning an attempt actually stops it.

## Validation

- New test: after one callback rejects, no further callbacks start.
- `pnpm checks` and `pnpm test` clean on this layer.

## Related Issues

- Part of #2141 (split out of #2244)
- Layer 1 of 6. Stack, bottom first: #2281 → #2277 → #2278 → #2279 →
#2280 → #2282

### Check List

- [x] I confirm that my contribution is made under the terms of the
Apache 2.0 license.
- [x] I have verified `pnpm checks` passes with no errors.
- [x] I have verified `pnpm test` passes with no failures.
- [x] I have covered new added functionality with unit tests if
necessary.
- [x] I have updated documentation if necessary.
@kmcginnes
kmcginnes force-pushed the edge-stack/5-edge-connection-discovery-strategy branch 2 times, most recently from 5e92375 to e9373f6 Compare September 25, 2026 23:16
@kmcginnes
kmcginnes force-pushed the edge-stack/5-edge-connection-discovery-strategy branch from e9373f6 to 265cdac Compare September 25, 2026 23:22
@kmcginnes
kmcginnes force-pushed the edge-stack/5-edge-connection-discovery-strategy branch from 265cdac to aef270b Compare September 25, 2026 23:40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant