Skip to content

BE-866: atlas: Use configured paths as generation roots - #9778

Open
indietyp wants to merge 1 commit into
bm/be-867-align-atlas-fitting-s3-configuration-and-constraintsfrom
bm/be-866-derive-the-generations-path-from-the-atlas-s3-bucket
Open

indietyp wants to merge 1 commit into
bm/be-867-align-atlas-fitting-s3-configuration-and-constraintsfrom
bm/be-866-derive-the-generations-path-from-the-atlas-s3-bucket

Conversation

@indietyp

@indietyp indietyp commented Sep 18, 2026

Copy link
Copy Markdown
Member

🌟 What is the purpose of this PR?

  • Append current, previous, repository/ and active/ directly to configured upload/download paths, without inserting generations/.
  • Leave the local GenerationRoot layout unchanged.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

  • require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

  • do not affect the execution graph

⚠️ Breaking changes and known limitations

Breaking: for an existing generations/ layout, update both upload and download settings: s3://bucket/atlass3://bucket/atlas/generations. The objects do not move.

❓ How to test this?

yarn workspace @rust/hash-graph-atlas test:unit file::generation

Stack created with GitHub Stacks CLIGive Feedback 💬

@vercel

vercel Bot commented Sep 18, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hash Ready Ready Preview Sep 18, 2026 6:41pm UTC
3 Skipped Deployments
Project Deployment Actions Updated
hashdotdesign-tokens Ignored Ignored Preview Sep 18, 2026 6:41pm UTC
petrinaut Skipped Skipped Sep 18, 2026 6:41pm UTC
petrinaut-docs Skipped Skipped Sep 18, 2026 6:41pm UTC

Request Review

@vercel
vercel Bot temporarily deployed to Preview – petrinaut September 18, 2026 18:31 Inactive
@vercel
vercel Bot temporarily deployed to Preview – petrinaut-docs September 18, 2026 18:31 Inactive
@indietyp
indietyp added this pull request to stack #9781 September 18, 2026 18:31
@github-actions github-actions Bot added area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > backend Owned by the @backend team labels Sep 18, 2026
@indietyp
indietyp deployed to pull-request September 18, 2026 18:32 — with GitHub Actions Active
@codecov

codecov Bot commented Sep 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 18.18182% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.45%. Comparing base (fcd6775) to head (2d6fc7b).

Files with missing lines Patch % Lines
...tlas/src/file/generation/test_utils/integration.rs 0.00% 18 Missing ⚠️
Additional details and impacted files
@@                                       Coverage Diff                                       @@
##           bm/be-867-align-atlas-fitting-s3-configuration-and-constraints    #9778   +/-   ##
===============================================================================================
  Coverage                                                           66.44%   66.45%           
===============================================================================================
  Files                                                                1849     1849           
  Lines                                                              195370   195357   -13     
  Branches                                                             7975     7975           
===============================================================================================
  Hits                                                               129823   129823           
+ Misses                                                              64024    64011   -13     
  Partials                                                             1523     1523           
Flag Coverage Δ
apps.hash-graph 12.27% <ø> (ø)
rust.hash-graph-atlas 80.20% <18.18%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@indietyp indietyp self-assigned this Sep 18, 2026
@indietyp indietyp changed the title chore: simplify remote generation namespace structure BE-866: atlas: Use configured paths as generation roots Sep 18, 2026
@indietyp
indietyp marked this pull request as ready for review September 18, 2026 18:59
@cursor

cursor Bot commented Sep 18, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Breaking operator contract for S3/upload/download path configuration; misconfigured roots would fail to find or publish generations until paths are updated.

Overview
Atlas remote upload/download paths now treat the configured --upload / --download prefix as the generations root itself: pointers and objects live at current, previous, repository/{id}/, and active/{id}/ directly under that prefix. The code no longer inserts a generations/ directory layer (RemoteRoot, download/upload helpers, and S3 integration tests were updated accordingly).

CLI help and the atlas README describe this layout. Local --root on-disk layout is unchanged.

Breaking for deployments: if upload/download previously pointed at the parent of a generations/ folder (e.g. s3://bucket/atlas), point them at the folder that actually holds those objects (e.g. s3://bucket/atlas/generations) without moving data.

Reviewed by Cursor Bugbot for commit 2d6fc7b. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark results

@rust/hash-graph-benches – Integrations

policy_resolution_large

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 2002 $$19.0 \mathrm{ms} \pm 147 \mathrm{μs}\left({\color{gray}3.04 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$2.12 \mathrm{ms} \pm 10.7 \mathrm{μs}\left({\color{gray}-1.824 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 1002 $$8.61 \mathrm{ms} \pm 73.0 \mathrm{μs}\left({\color{gray}-1.880 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: high, policies: 3314 $$26.7 \mathrm{ms} \pm 369 \mathrm{μs}\left({\color{gray}-2.457 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: low, policies: 1 $$8.93 \mathrm{ms} \pm 192 \mathrm{μs}\left({\color{lightgreen}-5.175 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: medium, policies: 1527 $$15.1 \mathrm{ms} \pm 144 \mathrm{μs}\left({\color{gray}-1.821 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 2078 $$19.8 \mathrm{ms} \pm 150 \mathrm{μs}\left({\color{gray}0.959 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$2.44 \mathrm{ms} \pm 18.2 \mathrm{μs}\left({\color{gray}0.967 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 1033 $$9.22 \mathrm{ms} \pm 72.7 \mathrm{μs}\left({\color{gray}0.329 \mathrm{\%}}\right) $$ Flame Graph

policy_resolution_medium

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 102 $$2.38 \mathrm{ms} \pm 16.8 \mathrm{μs}\left({\color{lightgreen}-6.555 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$1.88 \mathrm{ms} \pm 9.68 \mathrm{μs}\left({\color{gray}-0.205 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 52 $$2.11 \mathrm{ms} \pm 11.0 \mathrm{μs}\left({\color{gray}-0.616 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: high, policies: 269 $$3.38 \mathrm{ms} \pm 28.7 \mathrm{μs}\left({\color{gray}1.67 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: low, policies: 1 $$2.31 \mathrm{ms} \pm 15.3 \mathrm{μs}\left({\color{gray}4.05 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: medium, policies: 108 $$2.58 \mathrm{ms} \pm 15.7 \mathrm{μs}\left({\color{gray}-4.207 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 133 $$2.88 \mathrm{ms} \pm 28.2 \mathrm{μs}\left({\color{gray}1.55 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$2.10 \mathrm{ms} \pm 12.3 \mathrm{μs}\left({\color{gray}-0.804 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 63 $$2.55 \mathrm{ms} \pm 13.9 \mathrm{μs}\left({\color{gray}-0.644 \mathrm{\%}}\right) $$ Flame Graph

policy_resolution_none

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 2 $$1.69 \mathrm{ms} \pm 8.09 \mathrm{μs}\left({\color{gray}-2.189 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$1.70 \mathrm{ms} \pm 11.5 \mathrm{μs}\left({\color{gray}1.34 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 2 $$1.80 \mathrm{ms} \pm 14.5 \mathrm{μs}\left({\color{gray}2.19 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 8 $$1.89 \mathrm{ms} \pm 10.4 \mathrm{μs}\left({\color{gray}0.866 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$1.76 \mathrm{ms} \pm 8.85 \mathrm{μs}\left({\color{gray}-3.905 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 3 $$1.96 \mathrm{ms} \pm 10.0 \mathrm{μs}\left({\color{gray}-0.631 \mathrm{\%}}\right) $$ Flame Graph

policy_resolution_small

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 52 $$1.99 \mathrm{ms} \pm 49.9 \mathrm{μs}\left({\color{gray}1.65 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$1.79 \mathrm{ms} \pm 11.5 \mathrm{μs}\left({\color{gray}1.61 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 26 $$1.83 \mathrm{ms} \pm 9.72 \mathrm{μs}\left({\color{gray}-4.501 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: high, policies: 94 $$2.14 \mathrm{ms} \pm 9.27 \mathrm{μs}\left({\color{gray}-1.686 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: low, policies: 1 $$1.91 \mathrm{ms} \pm 10.1 \mathrm{μs}\left({\color{gray}-2.683 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: medium, policies: 27 $$2.09 \mathrm{ms} \pm 14.6 \mathrm{μs}\left({\color{gray}-1.042 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 66 $$2.14 \mathrm{ms} \pm 18.2 \mathrm{μs}\left({\color{gray}-0.898 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$1.95 \mathrm{ms} \pm 13.5 \mathrm{μs}\left({\color{gray}1.47 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 29 $$2.02 \mathrm{ms} \pm 12.0 \mathrm{μs}\left({\color{gray}-3.037 \mathrm{\%}}\right) $$ Flame Graph

read_scaling_complete

Function Value Mean Flame graphs
entity_by_id;one_depth 1 entities $$21.8 \mathrm{ms} \pm 108 \mathrm{μs}\left({\color{gray}0.224 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 10 entities $$46.3 \mathrm{ms} \pm 229 \mathrm{μs}\left({\color{gray}-2.208 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 25 entities $$23.5 \mathrm{ms} \pm 116 \mathrm{μs}\left({\color{gray}-0.131 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 5 entities $$27.2 \mathrm{ms} \pm 129 \mathrm{μs}\left({\color{gray}-0.704 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 50 entities $$28.1 \mathrm{ms} \pm 181 \mathrm{μs}\left({\color{gray}-1.731 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 1 entities $$23.2 \mathrm{ms} \pm 160 \mathrm{μs}\left({\color{gray}-3.206 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 10 entities $$271 \mathrm{ms} \pm 1.37 \mathrm{ms}\left({\color{gray}-0.723 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 25 entities $$57.4 \mathrm{ms} \pm 564 \mathrm{μs}\left({\color{gray}0.266 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 5 entities $$56.7 \mathrm{ms} \pm 418 \mathrm{μs}\left({\color{gray}-3.044 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 50 entities $$170 \mathrm{ms} \pm 964 \mathrm{μs}\left({\color{gray}1.79 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 1 entities $$7.30 \mathrm{ms} \pm 127 \mathrm{μs}\left({\color{gray}0.761 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 10 entities $$7.48 \mathrm{ms} \pm 42.3 \mathrm{μs}\left({\color{gray}2.64 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 25 entities $$7.40 \mathrm{ms} \pm 53.3 \mathrm{μs}\left({\color{gray}1.59 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 5 entities $$7.32 \mathrm{ms} \pm 28.6 \mathrm{μs}\left({\color{gray}0.713 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 50 entities $$7.62 \mathrm{ms} \pm 42.7 \mathrm{μs}\left({\color{gray}2.39 \mathrm{\%}}\right) $$ Flame Graph

read_scaling_linkless

Function Value Mean Flame graphs
entity_by_id 1 entities $$7.28 \mathrm{ms} \pm 36.5 \mathrm{μs}\left({\color{gray}1.61 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10 entities $$7.16 \mathrm{ms} \pm 33.3 \mathrm{μs}\left({\color{gray}-1.390 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 100 entities $$7.24 \mathrm{ms} \pm 51.9 \mathrm{μs}\left({\color{gray}0.221 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 1000 entities $$7.43 \mathrm{ms} \pm 40.0 \mathrm{μs}\left({\color{gray}0.434 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10000 entities $$7.65 \mathrm{ms} \pm 50.6 \mathrm{μs}\left({\color{gray}1.62 \mathrm{\%}}\right) $$ Flame Graph

representative_read_entity

Function Value Mean Flame graphs
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1 $$7.56 \mathrm{ms} \pm 41.8 \mathrm{μs}\left({\color{gray}-1.515 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1 $$7.57 \mathrm{ms} \pm 125 \mathrm{μs}\left({\color{gray}2.41 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1 $$7.56 \mathrm{ms} \pm 43.5 \mathrm{μs}\left({\color{gray}1.00 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1 $$7.42 \mathrm{ms} \pm 39.0 \mathrm{μs}\left({\color{gray}-1.540 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2 $$7.58 \mathrm{ms} \pm 50.5 \mathrm{μs}\left({\color{gray}3.37 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1 $$7.70 \mathrm{ms} \pm 47.1 \mathrm{μs}\left({\color{gray}-0.205 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1 $$7.91 \mathrm{ms} \pm 267 \mathrm{μs}\left({\color{red}7.50 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1 $$7.41 \mathrm{ms} \pm 32.3 \mathrm{μs}\left({\color{gray}0.426 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1 $$7.71 \mathrm{ms} \pm 45.5 \mathrm{μs}\left({\color{gray}1.56 \mathrm{\%}}\right) $$ Flame Graph

representative_read_entity_type

Function Value Mean Flame graphs
get_entity_type_by_id Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba $$5.50 \mathrm{ms} \pm 30.0 \mathrm{μs}\left({\color{gray}0.788 \mathrm{\%}}\right) $$ Flame Graph

representative_read_multiple_entities

Function Value Mean Flame graphs
entity_by_property traversal_paths=0 0 $$40.3 \mathrm{ms} \pm 350 \mathrm{μs}\left({\color{red}5.49 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=255 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true $$78.7 \mathrm{ms} \pm 576 \mathrm{μs}\left({\color{gray}0.230 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false $$43.1 \mathrm{ms} \pm 351 \mathrm{μs}\left({\color{gray}-1.646 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true $$53.1 \mathrm{ms} \pm 472 \mathrm{μs}\left({\color{red}6.13 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true $$60.7 \mathrm{ms} \pm 710 \mathrm{μs}\left({\color{gray}3.77 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true $$63.7 \mathrm{ms} \pm 337 \mathrm{μs}\left({\color{gray}2.29 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=0 0 $$29.7 \mathrm{ms} \pm 190 \mathrm{μs}\left({\color{gray}0.340 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=255 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true $$49.4 \mathrm{ms} \pm 278 \mathrm{μs}\left({\color{gray}-2.728 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false $$34.0 \mathrm{ms} \pm 261 \mathrm{μs}\left({\color{gray}-0.154 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true $$41.3 \mathrm{ms} \pm 308 \mathrm{μs}\left({\color{gray}4.80 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true $$40.7 \mathrm{ms} \pm 186 \mathrm{μs}\left({\color{gray}-2.306 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true $$42.7 \mathrm{ms} \pm 259 \mathrm{μs}\left({\color{gray}1.91 \mathrm{\%}}\right) $$

scenarios

Function Value Mean Flame graphs
full_test query-limited $$69.5 \mathrm{ms} \pm 348 \mathrm{μs}\left({\color{lightgreen}-7.182 \mathrm{\%}}\right) $$ Flame Graph
full_test query-unlimited $$77.9 \mathrm{ms} \pm 448 \mathrm{μs}\left({\color{lightgreen}-5.933 \mathrm{\%}}\right) $$ Flame Graph
linked_queries query-limited $$14.7 \mathrm{ms} \pm 255 \mathrm{μs}\left({\color{lightgreen}-5.385 \mathrm{\%}}\right) $$ Flame Graph
linked_queries query-unlimited $$343 \mathrm{ms} \pm 2.02 \mathrm{ms}\left({\color{gray}1.89 \mathrm{\%}}\right) $$ Flame Graph

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

1 participant