Skip to content

Look for existing instances in all spaces when saving - #142

Merged
apdavison merged 1 commit into
HumanBrainProject:masterfrom
apdavison:fix-cross-space-duplicates
Sep 15, 2026
Merged

apdavison merged 1 commit into
HumanBrainProject:masterfrom
apdavison:fix-cross-space-duplicates

Conversation

@apdavison

Copy link
Copy Markdown
Member

Fixes #136

Currently, save() restricts its existence query to the space it is about to write to, so a locally-constructed object whose counterpart already lives in a different space is not recognised, and a duplicate is created. This happens most often with recursive=True, where new child objects inherit the parent's target space.

The restriction was intended as an optimization, but benchmarking against the pre-production KG showed no measurable benefit: Person queries took the same time whether or not they were restricted to a space, and File queries timed out after ~50 s even when restricted to a space containing no files.

With the changes in this PR, save() now runs a single existence query across all spaces, which also returns the space of each matching instance.

  1. If one of the matches is in the target space, that instance is used, and matches in other spaces are not treated as duplicates.
  2. If the only match is in another space, it is used and updated in the space where it lives, with a warning, rather than a duplicate being created in the requested space.
  3. If there are multiple matches but none of them are in the target space, an exception is raised unless ignore_duplicates=True. This resolves the long-standing TODO about existing objects in a different space.

To let save() choose between matches, the query part of exists() is split into private methods (_exists_without_query(), _query_matching_instances(), _check_for_duplicates(), _use_matching_instance()), shared by exists() and save(). The signature and behaviour of exists(), including in_spaces, are unchanged. When exists() binds to a match found by the query, the object's space is set to the space of that instance.

Existence queries request EXISTENCE_QUERY_SIZE (2) results for efficiency, since it is rare to have more than one duplicate. If the response reports a larger total, all matches are retrieved in a second query, so that an instance in the target space is not missed.

Related bug fix: In exists(), a ConnectionError other than RemoteDisconnected was swallowed, leaving the result undefined and raising a NameError. The ConnectionError is now re-raised.

Tests: MockKGClient now returns the space of matching instances under the name used by the query API. New tests cover children and top-level objects found in another space, preferring the target space, duplicates in the target space and in other spaces, retrieving all matches, and connection errors.

save() restricted its existence query to the space it was about to write to, so a locally-constructed object whose counterpart already lives in a different space was not recognised, and a duplicate was created. This happened most often with recursive=True, where new child objects inherit the parent's target space.

The restriction was intended as an optimization, but benchmarking against the pre-production KG showed no measurable benefit: Person queries took the same time whether or not they were restricted to a space, and File queries timed out after ~50 s even when restricted to a space containing no files.

save() now runs a single existence query across all spaces, which also returns the space of each matching instance. If one of the matches is in the target space, that instance is used, and matches in other spaces are not treated as duplicates. If the only match is in another space, it is used and updated in the space where it lives, with a warning, rather than a duplicate being created in the requested space. Multiple matches, none of them in the target space, raise an exception unless ignore_duplicates=True. This resolves the long-standing TODO about existing objects in a different space.

To let save() choose between matches, the query part of exists() is split into private methods (_exists_without_query, _query_matching_instances, _check_for_duplicates, _use_matching_instance), shared by exists() and save(). The signature and behaviour of exists(), including in_spaces, are unchanged. When exists() binds to a match found by the query, the object's space is set to the space of that instance.

Existence queries request EXISTENCE_QUERY_SIZE (2) results. If the response reports a larger total, all matches are retrieved in a second query, so that an instance in the target space is not missed.

In exists(), a ConnectionError other than RemoteDisconnected was swallowed, leaving the result undefined and raising a NameError. It is now re-raised.

MockKGClient now returns the space of matching instances under the name used by the query API. New tests cover children and top-level objects found in another space, preferring the target space, duplicates in the target space and in other spaces, retrieving all matches, and connection errors.
@apdavison apdavison added this to the 0.15 milestone Sep 14, 2026
@apdavison
apdavison merged commit 633b02e into HumanBrainProject:master Sep 15, 2026
6 checks passed
@apdavison
apdavison deleted the fix-cross-space-duplicates branch September 15, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

save(recursive=True) creates duplicates of children that already exist in another space

1 participant