refactor: group flat root modules into scan/ sync/ viz/ subpackages (#625)#641
Open
nlebovits wants to merge 2 commits into
Open
refactor: group flat root modules into scan/ sync/ viz/ subpackages (#625)#641nlebovits wants to merge 2 commits into
nlebovits wants to merge 2 commits into
Conversation
…625) Part of #618. Relocate 20 flat portolan_cli/ root modules into three responsibility-scoped subpackages, shrinking the flat root from 64 to 44 .py files: - sync/ <- push, pull, upload, upload_progress, download, checksums; the sync/clone orchestrator lives in sync/core.py - scan/ <- scan_classify->classify, scan_detect->detect, scan_infer->infer, scan_fix->fix, scan_output->output, scan_progress->progress, check; the scan orchestrator lives in scan/core.py - viz/ <- thumbnail, thumbnail_style, style, pmtiles, pmtiles_links The two orchestrators (formerly scan.py / sync.py) live in <pkg>/core.py with import-free __init__.py. This is deliberate: making the orchestrator the package __init__ would force every leaf-submodule import to eagerly run it, reintroducing the collection_id<->scan and add<->sync import cycles. Public symbols are now imported from portolan_cli.scan.core / portolan_cli.sync.core. All moves are pure git renames (history preserved); the repo uses absolute imports exclusively, so relocating files left their own imports valid and only references TO the moved modules were rewritten. Import-linter contracts lock the new boundaries: - viz-is-a-leaf: viz imports neither scan nor sync (transitive). - scan-below-sync: scan never directly imports sync/viz (direct-only, since the check command legitimately fans out through shared root modules). The validation-seam ignore_imports comments are updated to the new paths. CLI entry point (portolan_cli:cli) unchanged. Verified locally: lint-imports (7/7), mypy --strict (151 files), ruff, and targeted scan/sync/push/pull/viz/validation suites (336 passed) green.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
✅ Files skipped from review due to trivial changes (2)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughThe change groups scan, sync, and visualization functionality into dedicated subpackages. New checksum, classification, style, and thumbnail-style modules are added, while existing production imports, CLI wiring, backend integrations, validation rules, documentation examples, tests, and mocks are updated to the new paths. Import-linter contracts document the package boundaries and prevent prohibited direct imports. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@portolan_cli/sync/checksums.py`:
- Around line 43-91: The compute_dir_checksum function currently hashes only
file metadata, so same-size edits with preserved mtimes are missed. Update it to
incorporate each contained file’s actual bytes into the SHA-256 checksum while
retaining deterministic relative-path handling and recursive traversal; ensure
unreadable files are handled consistently rather than silently producing a valid
content checksum.
In `@tests/integration/test_nested_catalog_clone.py`:
- Around line 90-93: Update all three relevant test call sites to patch
portolan_cli.sync.core.pull rather than portolan_cli.sync.pull, matching the
binding used by portolan_cli.sync.core. Keep the existing mock behavior
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0a61b91a-407f-417d-b0f5-d92bfab74b32
📒 Files selected for processing (120)
portolan_cli/add.pyportolan_cli/backends/iceberg/backend.pyportolan_cli/backends/json_file.pyportolan_cli/cli.pyportolan_cli/collection_id.pyportolan_cli/convert.pyportolan_cli/discovery.pyportolan_cli/extract/common/orchestrator_base.pyportolan_cli/metadata/fix.pyportolan_cli/metadata/scan.pyportolan_cli/preparation.pyportolan_cli/query.pyportolan_cli/scan/__init__.pyportolan_cli/scan/check.pyportolan_cli/scan/classify.pyportolan_cli/scan/core.pyportolan_cli/scan/detect.pyportolan_cli/scan/fix.pyportolan_cli/scan/infer.pyportolan_cli/scan/output.pyportolan_cli/scan/progress.pyportolan_cli/status.pyportolan_cli/sync/__init__.pyportolan_cli/sync/checksums.pyportolan_cli/sync/core.pyportolan_cli/sync/download.pyportolan_cli/sync/pull.pyportolan_cli/sync/push.pyportolan_cli/sync/upload.pyportolan_cli/sync/upload_progress.pyportolan_cli/validation/rules.pyportolan_cli/viz/__init__.pyportolan_cli/viz/pmtiles.pyportolan_cli/viz/pmtiles_links.pyportolan_cli/viz/style.pyportolan_cli/viz/thumbnail.pyportolan_cli/viz/thumbnail_style.pypyproject.tomltests/benchmark/test_scan_performance.pytests/iceberg/unit/test_on_post_add.pytests/iceberg/unit/test_pull.pytests/integration/test_check_conversion_config.pytests/integration/test_clone_ergonomics_integration.pytests/integration/test_cog_reoptimize.pytests/integration/test_concurrency_enforcement.pytests/integration/test_download_integration.pytests/integration/test_nested_catalog_clone.pytests/integration/test_partition_paths.pytests/integration/test_pmtiles_integration.pytests/integration/test_pull_integration.pytests/integration/test_pull_parallel_integration.pytests/integration/test_push_integration.pytests/integration/test_push_parallel_integration.pytests/integration/test_s3_moto.pytests/integration/test_scan_unrecognized_files.pytests/integration/test_sync_integration.pytests/integration/test_thumbnail_workflow.pytests/integration/test_upload_integration.pytests/spec_compliance/test_extensions_doc_parity.pytests/unit/extract/common/test_orchestrator_base.pytests/unit/test_add.pytests/unit/test_asset_role_media.pytests/unit/test_check_remove_legacy.pytests/unit/test_check_workflow.pytests/unit/test_cli_check.pytests/unit/test_cli_profile_default.pytests/unit/test_cli_push_chunk_concurrency.pytests/unit/test_cli_push_max_connections.pytests/unit/test_cli_push_workers.pytests/unit/test_clone_ergonomics.pytests/unit/test_collection_id.pytests/unit/test_concurrency_defaults.pytests/unit/test_concurrency_flag_issue_323.pytests/unit/test_discover_nested_collections.pytests/unit/test_download.pytests/unit/test_filegdb_add_support.pytests/unit/test_hive_partition_collection_id.pytests/unit/test_issue_252_catalog_json_roundtrip.pytests/unit/test_json_geojson_detection.pytests/unit/test_nested_catalog_inference.pytests/unit/test_partitioning.pytests/unit/test_partitioning_arbitrary.pytests/unit/test_pmtiles.pytests/unit/test_pull.pytests/unit/test_pull_async.pytests/unit/test_pull_parallel.pytests/unit/test_pull_restore.pytests/unit/test_push.pytests/unit/test_push_async.pytests/unit/test_push_catalog_wide.pytests/unit/test_push_default_remote.pytests/unit/test_push_intermediate_catalogs.pytests/unit/test_push_metadata_sync.pytests/unit/test_push_root_files.pytests/unit/test_push_verbose.pytests/unit/test_scan.pytests/unit/test_scan_classify.pytests/unit/test_scan_coverage.pytests/unit/test_scan_detect.pytests/unit/test_scan_edge_cases.pytests/unit/test_scan_filegdb_discovery.pytests/unit/test_scan_fix.pytests/unit/test_scan_fix_coverage.pytests/unit/test_scan_fix_normalization.pytests/unit/test_scan_infer.pytests/unit/test_scan_infer_coverage.pytests/unit/test_scan_nested.pytests/unit/test_scan_output.pytests/unit/test_scan_output_batching.pytests/unit/test_scan_progress.pytests/unit/test_scan_structure.pytests/unit/test_scan_unrecognized_files_property.pytests/unit/test_style.pytests/unit/test_sync.pytests/unit/test_tabular_support.pytests/unit/test_thumbnail.pytests/unit/test_thumbnail_style.pytests/unit/test_upload.pytests/unit/test_upload_progress.pytests/unit/test_validation_rules.py
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@portolan_cli/sync/checksums.py`:
- Around line 43-91: The compute_dir_checksum function currently hashes only
file metadata, so same-size edits with preserved mtimes are missed. Update it to
incorporate each contained file’s actual bytes into the SHA-256 checksum while
retaining deterministic relative-path handling and recursive traversal; ensure
unreadable files are handled consistently rather than silently producing a valid
content checksum.
In `@tests/integration/test_nested_catalog_clone.py`:
- Around line 90-93: Update all three relevant test call sites to patch
portolan_cli.sync.core.pull rather than portolan_cli.sync.pull, matching the
binding used by portolan_cli.sync.core. Keep the existing mock behavior
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0a61b91a-407f-417d-b0f5-d92bfab74b32
📒 Files selected for processing (120)
portolan_cli/add.pyportolan_cli/backends/iceberg/backend.pyportolan_cli/backends/json_file.pyportolan_cli/cli.pyportolan_cli/collection_id.pyportolan_cli/convert.pyportolan_cli/discovery.pyportolan_cli/extract/common/orchestrator_base.pyportolan_cli/metadata/fix.pyportolan_cli/metadata/scan.pyportolan_cli/preparation.pyportolan_cli/query.pyportolan_cli/scan/__init__.pyportolan_cli/scan/check.pyportolan_cli/scan/classify.pyportolan_cli/scan/core.pyportolan_cli/scan/detect.pyportolan_cli/scan/fix.pyportolan_cli/scan/infer.pyportolan_cli/scan/output.pyportolan_cli/scan/progress.pyportolan_cli/status.pyportolan_cli/sync/__init__.pyportolan_cli/sync/checksums.pyportolan_cli/sync/core.pyportolan_cli/sync/download.pyportolan_cli/sync/pull.pyportolan_cli/sync/push.pyportolan_cli/sync/upload.pyportolan_cli/sync/upload_progress.pyportolan_cli/validation/rules.pyportolan_cli/viz/__init__.pyportolan_cli/viz/pmtiles.pyportolan_cli/viz/pmtiles_links.pyportolan_cli/viz/style.pyportolan_cli/viz/thumbnail.pyportolan_cli/viz/thumbnail_style.pypyproject.tomltests/benchmark/test_scan_performance.pytests/iceberg/unit/test_on_post_add.pytests/iceberg/unit/test_pull.pytests/integration/test_check_conversion_config.pytests/integration/test_clone_ergonomics_integration.pytests/integration/test_cog_reoptimize.pytests/integration/test_concurrency_enforcement.pytests/integration/test_download_integration.pytests/integration/test_nested_catalog_clone.pytests/integration/test_partition_paths.pytests/integration/test_pmtiles_integration.pytests/integration/test_pull_integration.pytests/integration/test_pull_parallel_integration.pytests/integration/test_push_integration.pytests/integration/test_push_parallel_integration.pytests/integration/test_s3_moto.pytests/integration/test_scan_unrecognized_files.pytests/integration/test_sync_integration.pytests/integration/test_thumbnail_workflow.pytests/integration/test_upload_integration.pytests/spec_compliance/test_extensions_doc_parity.pytests/unit/extract/common/test_orchestrator_base.pytests/unit/test_add.pytests/unit/test_asset_role_media.pytests/unit/test_check_remove_legacy.pytests/unit/test_check_workflow.pytests/unit/test_cli_check.pytests/unit/test_cli_profile_default.pytests/unit/test_cli_push_chunk_concurrency.pytests/unit/test_cli_push_max_connections.pytests/unit/test_cli_push_workers.pytests/unit/test_clone_ergonomics.pytests/unit/test_collection_id.pytests/unit/test_concurrency_defaults.pytests/unit/test_concurrency_flag_issue_323.pytests/unit/test_discover_nested_collections.pytests/unit/test_download.pytests/unit/test_filegdb_add_support.pytests/unit/test_hive_partition_collection_id.pytests/unit/test_issue_252_catalog_json_roundtrip.pytests/unit/test_json_geojson_detection.pytests/unit/test_nested_catalog_inference.pytests/unit/test_partitioning.pytests/unit/test_partitioning_arbitrary.pytests/unit/test_pmtiles.pytests/unit/test_pull.pytests/unit/test_pull_async.pytests/unit/test_pull_parallel.pytests/unit/test_pull_restore.pytests/unit/test_push.pytests/unit/test_push_async.pytests/unit/test_push_catalog_wide.pytests/unit/test_push_default_remote.pytests/unit/test_push_intermediate_catalogs.pytests/unit/test_push_metadata_sync.pytests/unit/test_push_root_files.pytests/unit/test_push_verbose.pytests/unit/test_scan.pytests/unit/test_scan_classify.pytests/unit/test_scan_coverage.pytests/unit/test_scan_detect.pytests/unit/test_scan_edge_cases.pytests/unit/test_scan_filegdb_discovery.pytests/unit/test_scan_fix.pytests/unit/test_scan_fix_coverage.pytests/unit/test_scan_fix_normalization.pytests/unit/test_scan_infer.pytests/unit/test_scan_infer_coverage.pytests/unit/test_scan_nested.pytests/unit/test_scan_output.pytests/unit/test_scan_output_batching.pytests/unit/test_scan_progress.pytests/unit/test_scan_structure.pytests/unit/test_scan_unrecognized_files_property.pytests/unit/test_style.pytests/unit/test_sync.pytests/unit/test_tabular_support.pytests/unit/test_thumbnail.pytests/unit/test_thumbnail_style.pytests/unit/test_upload.pytests/unit/test_upload_progress.pytests/unit/test_validation_rules.py
🛑 Comments failed to post (2)
portolan_cli/sync/checksums.py (1)
43-91: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail printf '\n## File outline\n' ast-grep outline portolan_cli/sync/checksums.py --view expanded || true printf '\n## Relevant file section\n' nl -ba portolan_cli/sync/checksums.py | sed -n '1,220p' printf '\n## Usages of compute_dir_checksum and is_current\n' rg -n "compute_dir_checksum|is_current\\(" portolan_cli -SRepository: portolan-sdi/portolan-cli
Length of output: 457
🏁 Script executed:
#!/bin/bash set -euo pipefail printf '\n## checksums.py with line numbers\n' awk '{printf "%4d %s\n", NR, $0}' portolan_cli/sync/checksums.py | sed -n '1,220p' printf '\n## usages\n' rg -n "compute_dir_checksum|is_current\\(" portolan_cli -S printf '\n## nearby sync logic\n' rg -n "checksum|currentness|mtime|fingerprint" portolan_cli/sync -SRepository: portolan-sdi/portolan-cli
Length of output: 9425
🏁 Script executed:
#!/bin/bash set -euo pipefail printf '\n## query.py is_current section\n' awk '{printf "%4d %s\n", NR, $0}' portolan_cli/query.py | sed -n '160,290p' printf '\n## preparation.py usage section\n' awk '{printf "%4d %s\n", NR, $0}' portolan_cli/preparation.py | sed -n '170,230p'Repository: portolan-sdi/portolan-cli
Length of output: 8388
Directory checksum can miss content changes
compute_dir_checksum()is used asasset.sha256and byis_current(), but it only hashes path/size/mtime. A same-size edit with preserved mtimes will be treated as unchanged, so modified FileGDB directories can be skipped. Hash file contents here, or rename this to a cache-only fingerprint.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@portolan_cli/sync/checksums.py` around lines 43 - 91, The compute_dir_checksum function currently hashes only file metadata, so same-size edits with preserved mtimes are missed. Update it to incorporate each contained file’s actual bytes into the SHA-256 checksum while retaining deterministic relative-path handling and recursive traversal; ensure unreadable files are handled consistently rather than silently producing a valid content checksum.tests/integration/test_nested_catalog_clone.py (1)
90-93: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash # Check how portolan_cli.sync.core imports the pull function rg -n 'import.*pull' portolan_cli/sync/core.pyRepository: portolan-sdi/portolan-cli
Length of output: 229
🏁 Script executed:
#!/bin/bash set -euo pipefail printf '\n== core import ==\n' sed -n '1,60p' portolan_cli/sync/core.py printf '\n== test patch sites ==\n' rg -n 'patch\("portolan_cli\.sync\.pull|patch\("portolan_cli\.sync\.core\.pull|patch\("portolan_cli\.sync\.pull\.pull' tests/integration/test_nested_catalog_clone.py printf '\n== surrounding test context ==\n' sed -n '80,240p' tests/integration/test_nested_catalog_clone.pyRepository: portolan-sdi/portolan-cli
Length of output: 7999
Patch
portolan_cli.sync.core.pullin these testsportolan_cli.sync.coreimportspulldirectly, sopatch("portolan_cli.sync.pull")won’t replace the binding used here. Update the three call sites to patchportolan_cli.sync.core.pullinstead.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integration/test_nested_catalog_clone.py` around lines 90 - 93, Update all three relevant test call sites to patch portolan_cli.sync.core.pull rather than portolan_cli.sync.pull, matching the binding used by portolan_cli.sync.core. Keep the existing mock behavior unchanged.
Follow-up to the scan/sync/viz grouping. Two categories of test reference broke because the sync orchestrator now lives in portolan_cli.sync.core: - 32 sites patched "portolan_cli.sync.pull" — the orchestrator's bound `pull` (clone() calls it). After the move that path resolves to the `pull` *submodule* instead of the name bound in the orchestrator, so the patch no longer intercepted the call. Redirect to "portolan_cli.sync.core.pull". `pull` was the only submodule name the orchestrator also re-binds, so it's the only collision the mechanical rewrite couldn't disambiguate. - test_extension_registry used `from portolan_cli import scan_classify` (a module-object import, not a dotted path), which the dotted-path rewrite didn't catch. Point it at `from portolan_cli.scan import classify`. Verified: full unit suite (5046 passed) and all scan/check/pull/push/sync/clone suites green; the two remaining local integration timeouts are pre-existing slow tests (77s symlink add, loguru-queue duckdb glob) tripping an aggressive local --timeout, and pass in isolation — not refactor regressions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #625. Part of the #618 epic — the sequenced-last, mechanical namespace flatten (all prior tickets #619–#624 have landed).
What
Relocates 20 flat
portolan_cli/root modules into three responsibility-scoped subpackages. Flat root shrinks 64 → 44.pyfiles.sync/push,pull,upload,upload_progress,download,checksums+core(clone/sync orchestration)scan/classify,detect,infer,fix,output,progress,check+core(scan orchestrator)viz/thumbnail,thumbnail_style,style,pmtiles,pmtiles_linksInside
scan/andsync/the redundantscan_/upload_prefixes are dropped to match the existingmetadata/,extract/convention (metadata/scan.py, notmetadata/metadata_scan.py).Why
core.pyinstead of a fat__init__.pyThe orchestrators (formerly
scan.py/sync.py) live in<pkg>/core.pywith an import-free__init__.py. Putting the orchestrator in__init__.pylooked tempting (zero public-API churn) but is wrong here: importing any leaf submodule (e.g.scan.detect) forces Python to run the parent__init__first, eagerly loading the 1440-line orchestrator — which reintroduces thecollection_id ⇄ scanandadd ⇄ syncimport cycles. Verified: importing a leaf submodule first no longer triggers the orchestrator.Public symbols now come from
portolan_cli.scan.core/portolan_cli.sync.core.Mechanics
git mvrenames — history preserved.\b-bounded regex, sosync.pushsubmodule stays distinct from thesync.push_asyncorchestrator symbol).portolan_cli:clientry point unchanged.Import-linter boundaries (new)
viz-is-a-leaf—vizimports neitherscannorsync(transitive; viz truly is a leaf).scan-below-sync—scannever directly importssync/viz(allow_indirect_imports, because thecheckcommand legitimately fans out through shared root modules likecatalog/metadata/convert, which themselves use sync/viz).The
validation-seam-for-reisdocumentary comments are updated to the new paths (scan.classify,viz.pmtiles_links).Verification (local)
lint-imports— 7/7 kept, 0 brokenmypy --strict— clean (151 files)ruff check+ruff format— cleanportolan --helpworksFull suite runs in CI.
Acceptance criteria
lint-importsgreen; CLI entry point unchanged. (Full test suite → CI.)🤖 Generated with Claude Code
Summary by CodeRabbit
collection.jsonasset/manifest registration and COG render enrichment.