Commit 0f5b7af
Rewrite extractor with mmap + rayon + memchr for 5x speedup
- Memory-mapped I/O (memmap2) — zero-copy file access, no BufReader allocations
- Work-stealing parallelism (rayon) — replaces manual thread::scope chunking
- SIMD byte search (memchr) — accelerated divider and newline detection
- Byte-based progress — uses file size metadata instead of line counting,
eliminating the slow pre-count pass
- Single extract delegate — extract() now calls extract_multi internally,
removing ~50 lines of duplicated logic
- Multi-file scan: 22.8s → 4.3s on the same 1 GB / 16.6M line workload
- Net reduction: 118 lines removed while adding 3 dependency crates
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>1 parent fd9a2dd commit 0f5b7af
4 files changed
Lines changed: 287 additions & 405 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
0 commit comments