Skip to content

Commit 0f5b7af

Browse files
sinescodeclaude
andcommitted
Rewrite extractor with mmap + rayon + memchr for 5x speedup
- Memory-mapped I/O (memmap2) — zero-copy file access, no BufReader allocations - Work-stealing parallelism (rayon) — replaces manual thread::scope chunking - SIMD byte search (memchr) — accelerated divider and newline detection - Byte-based progress — uses file size metadata instead of line counting, eliminating the slow pre-count pass - Single extract delegate — extract() now calls extract_multi internally, removing ~50 lines of duplicated logic - Multi-file scan: 22.8s → 4.3s on the same 1 GB / 16.6M line workload - Net reduction: 118 lines removed while adding 3 dependency crates Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent fd9a2dd commit 0f5b7af

4 files changed

Lines changed: 287 additions & 405 deletions

File tree

Cargo.lock

Lines changed: 70 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,12 @@
11
[package]
22
name = "ulpExtractor"
3-
version = "0.3.0"
3+
version = "0.3.1"
44
edition = "2021"
55

66
[dependencies]
77
clap = { version = "4", features = ["derive"] }
88
indicatif = "0.17"
99
console = "0.15"
10+
memmap2 = "0.9"
11+
rayon = "1.10"
12+
memchr = "2"

0 commit comments

Comments
 (0)