Skip to content

Parallel checksum calculation for file invalidation #366

@nikblanchet

Description

@nikblanchet

Summary

File invalidation checksums are currently computed sequentially in FileTracker. Parallelize checksum computation for significant performance improvement.

Current Behavior

  • FileTracker.createSnapshot() computes checksums sequentially
  • 100-500 files: ~200ms
  • 1000+ files: ~400ms

Proposed Enhancement

Use Promise.all() / asyncio.gather() to compute checksums in parallel.

Expected Impact

  • 4x speedup: 200ms → 50ms (100-500 files)
  • 4x speedup: 400ms → 100ms (1000+ files)

Implementation

TypeScript (cli/src/utils/file-tracker.ts:34):

const snapshotPromises = filepaths.map(async (filepath) => {
  // Compute checksum...
});
const results = await Promise.all(snapshotPromises);

Python (analyzer/src/utils/file_tracker.py:52):

async def create_snapshot_async(filepaths: list[str]) -> dict[str, FileSnapshot]:
    tasks = [compute_checksum(fp) for fp in filepaths]
    results = await asyncio.gather(*tasks)

Effort

~1 hour

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort-smallSmall effort: <2 hoursenhancementNew feature or requestimpact-highHigh impact on users or systemperformancePerformance optimization issuespost-mvpPost-MVP feature, not needed for initial release

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions