Spike/flatbuffer mmap search#31
Merged
Merged
Conversation
Replace the Nuke build with Fallout (Fallout.Common), moving config from .nuke/ to .fallout/ and updating the bootstrap scripts and build project. Adopt the .slnx solution format and refresh project package references, including pinning Microsoft.CodeAnalysis.*.Workspaces to 4.14.0 in cdeBenchmarks to resolve an NU1608 transitive version conflict. Update CLAUDE.md to document the Fallout build system.
…Phase 0)
Adds a standalone retained-footprint probe (cdeMemProbe) with a deterministic
synthetic-catalog generator, plus BenchmarkDotNet load and search suites in
cdeBenchmarks. Commits the locked baseline every later phase is measured against.
- cdeMemProbe: loads a .cde, settles the GC, reports retained managed heap /
working set / bytes-per-entry (BDN MemoryDiagnoser cannot measure retained heap).
- SyntheticCatalog: shared, seeded generator (~50:1 file:dir) used by probe + benches.
- CatalogLoadBenchmarks / SearchBenchmarks: load wall-clock and find throughput
across {substring,regex} x {name,path}.
- baseline/: footprint (~212 B/entry; 2.0 GB heap @ 10M) and search timings.
Baseline already confirms two plan hypotheses: hashed vs no-hash footprint differ by
only ~5 B/entry (Hash16 is inline regardless), and path search allocates ~973 MB/query.
S3 - route the CLI find path (FindService) through the synchronous FindOptions.Find instead of the work-stealing FindAsync. The async path ran every entry through an async Task<bool> state machine plus Task.Yield/Task.Delay every 500/2000 entries. Measured on a 1M-entry catalog: name search 1746ms -> 36ms (48x), path search 1755ms -> 236ms (7.4x), regex name 1817ms -> 60ms (30x). cdeWin already used the sync path; only the CLI was on the slow one. R1a - FileEntryCount/DirEntryCount changed long -> uint on DirEntry/RootEntry/ ICommonEntry. They were already (uint)-truncated at assignment, so semantics are unchanged. In-memory only, no catalog format impact. Confirmed 211.94 -> 203.94 B/entry. Tests: cdeLibTest 126 passed. Format-affecting work (path-alloc S1, file/dir split, hash side-table) is deferred to Phases 2-3.
…hase 2) Reimplement EntryHelper.MakeFullPath to build the full path in one pass up the ParentCommonEntry chain instead of recursively calling parent.FullPath at every level (which allocated a fresh full-path string per ancestor). Add FullPathContains that matches a substring over a pooled Span<char> with no result-string allocation, wired into the find substring path matcher. Measured on a 1M-entry catalog (cumulative with Phase 1): - substring path search: 1755ms/978MB -> 102ms/53MB (17x faster, 94.6% less alloc) - regex path search: 1828ms/1349MB -> 136ms/265MB All path-building callers (find, dupes, GUI, dump) get the cheaper single-pass build. cdeLibTest 126 passed, including EntryHelper tests that assert exact path strings. Note: dropping ParentCommonEntry (originally Phase 2) folds into Phase 3 - .FullPath is only called on directories or via PairDirEntry, so only files can shed the pointer, which needs the file/dir type split.
…at change) Move DirEntry's directory-only fields (Children + the two summary counts) into a lazily-allocated private ExtraData object, allocated only for directories. A file - the vast majority of entries - now carries one null 8-byte _extra reference instead of an always-null Children ref plus two count fields. Hash stays INLINE on purpose: moving it off-entry forces every hashed file to allocate an ExtraData whose header exceeds the 16-byte Hash16 (measured +16 B/entry regression for hashed catalogs). Keeping it inline means no catalog ever regresses. No .cde format change: Children keeps Key 3 via the property, counts are IgnoreMember. Round-trip is byte-identical to baseline (36,668,199 bytes). Measured 1M no-hash: 203.94 -> 198.85 B/entry; hashed has no regression. No find CPU regression (name 36.7ms, path 95.0ms). cdeLibTest 126 + cdeWinTest 32 passed. The full polymorphic split (~3x more saving) needs an abstract union root, IList type changes across lib/GUI/web, ~40 construction sites, a format bump, and GUI runtime validation - left as an explicit opt-in follow-up (see baseline/phase-results.md).
Add a --shared-names generator mode to isolate the per-file name-string cost, and document the finding: names cost ~70 B/entry (198.85 normal vs 129.23 with a shared name on a 1M fixture) - the single biggest remaining memory lever. Two options assessed with estimates + risks: UTF-8 byte[] per name (~15-20 B/entry, hot-path byte-matching rewrite) and a per-RootEntry name pool (~40 B/entry, largest win, very invasive). Both rewrite the hottest comparison path and bump the .cde format. See src/cdeBenchmarks/baseline/name-storage-investigation.md.
Cancellation was only checked inside the throttled progress-reporting block
("only check for cancel on progress reports"). Since the adaptive-frequency change
(7d46506/a22539b2), progress reports fire only every ~50k entries on large catalogs,
so an active search - especially a slow regex - ignored Worker.CancellationPending
for tens of thousands of entries and felt like it never cancelled.
Check the cheap CancellationPending volatile read every 4096 entries, independently
of progress reporting, in both the sync (cdeWin / CLI) and legacy async find paths.
Responsive even on a slow regex; negligible overhead on a fast full scan.
cdeLibTest 126 passed; full solution builds clean.
…siveness) Raw search throughput strongly favours the synchronous path at every catalog count (benchmarked: sync vs legacy async = 44x @ 1 catalog, 30x @ 10, 31x @ 100). The async path only ever "felt faster" because it streamed progress/results on a 100ms timer, while the sync path reported every ~50k entries - so on a long search results appeared in large infrequent chunks. Give the sync path the same time-based streaming: behind the cheap 4096-entry housekeeping gate, report progress at most every ~100ms (tick-threshold compare, no multiply/overflow). Fast AND smooth. Sync speed unchanged (33-55ms for 1M entries across 1/10/100 catalogs). Adds MultiCatalogSearchBenchmarks documenting the sync-vs-async comparison across catalog counts.
Find parallelizes across catalogs, so VisitorFunc runs on multiple threads. The result list used a plain List<T>.Add, which is not thread-safe - searching many catalogs at once could drop results or throw as concurrent adds raced on the backing array. Guard Add with a lock, and hand the UI an immutable snapshot taken under that lock on each progress tick instead of the live list the worker threads are still mutating (the virtual ListView indexes into it on the UI thread). After Find() returns all threads are joined, so the final result uses the list directly with no extra copy. cdeWin builds clean; cdeWinTest 32 passed.
Prototype a SoA catalog model (parallel value-type arrays; tree shape via int firstChild/nextSibling/parent indices) and a tree->SoA converter, with a cdeMemProbe --soa mode to measure its retained footprint. Measured (1M/10M fixtures): structural cost drops from 129.23 to 42.23 B/entry (-67%) - SoA holds the same catalog structure in ~1/3 the memory (10M: 421 MB vs ~1.29 GB). It is the only option that removes the per-entry object header. Search and full-path reconstruction validated on the store. Prototype only (lives in cdeMemProbe, production model untouched). A real migration is a core-model rewrite (serialization, search, dupes, hashing, GUI adapter); see src/cdeBenchmarks/baseline/soa-prototype.md for the proven numbers and phasing.
…migration P1) First phase of the struct-of-arrays migration, additive - the production pointer tree is untouched. - cdeLib/Entities/Soa/EntryStore.cs: SoA catalog model (parallel arrays; int firstChild/nextSibling/parent links; lazy Hash[] only when hashed) plus a robust iterative tree->store converter (counts the real tree, not summary fields) and allocation-free full-path reconstruction by walking Parent[]. - cdeLib/Entities/Soa/EntryStoreSearch.cs: index-based find (substring/regex, name/path, file/folder filter) - a cache-friendly flat-array scan. - cdeLibTest/Soa/EntryStoreTests.cs: 12 tests proving store search results are IDENTICAL to the pointer-tree FindOptions across every query shape, plus full-path equivalence for every entry. - cdeMemProbe now uses the cdeLib EntryStore (prototype copy removed). Footprint reconfirmed: 42.23 B/entry structural vs 129.23 for the tree (-67%). cdeLibTest 138 passed.
…nEntry> (migration P2) Re-type the interface Children from IList<DirEntry> to a covariant, read-only IReadOnlyList<ICommonEntry> so a struct-of-arrays backing (EntryStore via a coming EntryRef adapter) can satisfy ICommonEntry without materialising DirEntry objects. The concrete tree classes keep their mutable IList<DirEntry> Children (used by build/sort/MessagePack) and expose the abstract view via explicit interface implementation - a List<DirEntry> satisfies IReadOnlyList<ICommonEntry> at runtime through interface covariance, so almost all consumers (Count, foreach, indexing, LINQ) compile unchanged. Only three internal tree ops that mutated/keyed children via the interface needed updates: SortAllChildrenByPath (cast to concrete for Sort) and the two copy-hash lookups (Dictionary keyed by ICommonEntry). Behaviour preserved: cdeLibTest 138 + cdeWinTest 32 passed; full solution builds clean. This unblocks the EntryRef adapter (next) that lets the GUI/dupes run on SoA.
…ation P3) EntryRef presents one EntryStore entry (by index) as an ICommonEntry: read members map onto the store arrays; build/mutate members throw (a store is produced wholesale by the loader, not edited entry-by-entry). This is the payoff of the Children abstraction: the existing tree-oriented, ICommonEntry-based code now runs on the struct-of-arrays model unchanged. Tests prove EntryHelper.TraverseTreePair over an EntryRef root yields the same full paths as the tree, children/counts/GetListFromRoot/FullPath match, and mutators throw. (Gate child access on having children, not the directory flag - the root, like RootEntry, isn't flagged a directory but has children.) cdeLibTest 143 + cdeWinTest 32 passed; full solution builds clean. Intended for occasional access (display, result rows); bulk traversal still uses index-based EntryStoreSearch to avoid per-entry wrapper allocations.
…ation P4) Realizes the SoA memory win in an actual command. FindService (CLI find) now converts each loaded catalog to an EntryStore, releases the pointer tree, and searches the store via the index-based EntryStoreSearch. Verified end-to-end: `cde find` returns correct results incl. deep nested full paths. Critical fix found by measurement: the converter must REUSE the tree''s interned name strings, not rebuild them. DirEntry now holds its extension in a named _ext field and exposes NamePart/ExtPart; EntryStore stores names split (Name[]+Ext[]) pointing at the SAME interned objects, rejoining via FullName(i) and appending parts directly in path building (no full-name allocation). Without this the store allocated 1M fresh name strings on top of the pool-retained originals and footprint went UP. Measured on a real 1M-entry .cde (cdeMemProbe --store, synchronous load so the tree is fully released): tree 162.2 -> store 104.9 bytes/entry, -35%. (Earlier store numbers were polluted by an async state machine retaining the tree.) cdeLibTest 143 + cdeWinTest 32 passed; full solution builds clean.
The cdeWin catalog list and search-result rows render root-level metadata (volume, default/actual .cde name, drive hint, avail/total space, scan dates, description, counts, size). EntryStore now captures all of it at conversion so a store is a complete, self-describing catalog - the prerequisite for holding catalogs as stores in the GUI. cdeLibTest 18 SoA tests passed (incl. metadata round-trip).
…ing foundation) Add EntryStoreFindOptions (pattern + name/path + file/folder + size/date/hour/ not-older-than ranges) and an EntryStoreSearch.Find overload that evaluates them directly against the store arrays - matching the filter set the cdeWin GUI search applies (which the CLI find did not need). Tested. This is the search foundation the GUI store-wiring requires; the presenter/form rewiring itself is the remaining step.
…on P5) cdeWin now loads each catalog, converts it to an EntryStore, releases the pointer tree, and holds the catalog root as an EntryRef - so the GUI runs on the struct-of- arrays model (~1/3 the memory) for the many-catalog interactive workflow. - CatalogListViewHelper retyped IListViewHelper<RootEntry> -> <ICommonEntry>; catalog list render + RootCompare read metadata from EntryRef.Store. - Search runs over the stores via the filter-capable EntryStoreSearch (size/date/hour parity) with cancel + 100ms streaming; results are EntryRef-based PairDirEntry. - Navigation (SetNewDirectoryRoot/BuildRootNode/Tag) and reload moved to ICommonEntry; catalog identity reads EntryRef.Store (falls back to GetRootEntry for tree pairs). - Added cancel/progress hooks to EntryStoreSearch; presenter tests updated for EntryRef catalog roots. cdeLibTest 145 + cdeWinTest 32 passed; full solution builds clean. NOTE: validated via build + unit tests only - the live WinForms UI was not run here and needs manual verification (catalog list, search, navigation, reload).
Prototype a struct-of-arrays catalog file read zero-copy over a memory map, so 'load' becomes mmap and search streams over the mapping. - ColumnarFormat: hand-rolled SoA .cdex writer (primitive columns + UTF-8 name blob), built from an EntryStore. - ColumnarReader: mmap reader exposing columns as ReadOnlySpan over the mapping; zero-alloc UTF-8 byte-scan name/path search. - cdeMemProbe --migrate (one-way MessagePack .cde -> .cdex) and --flat (mmap + search + measure) modes. Result on a real 649k-entry catalog: managed heap 82MB(tree)/44MB(store) -> 0.07MB; open 1123ms -> 3ms; full name scan allocates 40 bytes. Costs: +36% disk, working set ~= touched (reclaimable) pages, ASCII-only case fold + int offsets to address before production. See src/cdeBenchmarks/baseline/flatbuffer-mmap-spike.md.
… migrate Move the spike's columnar/mmap format into production cdeLib and add a one-way migration command. - cdeLib/Entities/Columnar: ColumnarFormat (writer), ColumnarCatalogReader (zero-copy mmap reader exposing columns as ReadOnlySpan over the mapping, parses catalog metadata once), Utf8Matcher (ordinal case-insensitive, zero-alloc ASCII fast path + decoded fallback for non-ASCII names). - Hardened vs spike: 64-bit name-blob offsets (no 2GB cap), Unicode-correct case folding, format magic+version gate, file/folder filtering. - 'cde migrate [file]' converts MessagePack .cde -> .cdex (current dir + one level down, or a single file). - cdeMemProbe now uses the cdeLib implementation (spike copy removed). - ColumnarCatalogTests: round-trip + search parity vs EntryStoreSearch (7 tests). cdeLib AllowUnsafeBlocks enabled for the mmap pointer acquisition.
…e C) The find CLI now prefers the columnar format: if any .cdex catalogs exist in the current dir (or one level down) they are searched over their memory maps with no managed catalog load; otherwise it falls back to loading the .cde trees. - ColumnarCatalogReader.Find: unified pattern + name/path + file/folder search matching EntryStoreSearch semantics exactly (index 0/root skipped; substring byte-scan, regex via IgnoreCase|Singleline|Compiled decode). - ICatalogRepository.GetColumnarFileList discovers .cdex (current dir + one down), mirroring GetCacheFileList. - FindService.FindColumnar searches a set of open readers. - Program.RunFind prefers .cdex, disposes readers after. - Tests: regex parity vs EntryStoreSearch (10 columnar tests total). Verified end-to-end: find + greppath return correct nested paths from a .cdex with no .cde present.
…age D foundation) Introduce IEntrySource as the common read-only, index-addressed catalog surface over both the in-memory EntryStore and the zero-copy ColumnarCatalogReader, so EntryRef (and the GUI) work over either backing without caring whether the catalog is on the heap or in a memory map. - IEntrySource: per-entry accessors + full-filter search + catalog metadata. - EntryStore implements it (metadata fields -> properties; thin wrappers; instance Find delegates to EntryStoreSearch). - ColumnarCatalogReader implements it: child/sibling/hash column accessors, AppendFullPath, and a full-filter Find (size/date/hour + cancel/scan) matching EntryStoreSearch semantics exactly over the mapping. - EntryRef now backs onto IEntrySource (was EntryStore); exposes Source. - Tests: full-filter size-range parity + EntryRef navigating over a reader (12 columnar tests). 155 lib tests green. Pure cdeLib refactor; GUI rewiring to hold readers follows next.
…py load (stage D) cdeWin now memory-maps columnar .cdex catalogs instead of loading .cde trees into the managed heap, for the 100-catalog low-memory case. - Catalogs held as IEntrySource via EntryRef: a ColumnarCatalogReader (mmap) when .cdex exist, else an EntryStore built from the .cde tree. - LoadCatalogService.GetColumnarFiles discovers .cdex (current dir + config path); presenter prefers it and only falls back to .cde load when none are found. - Search runs via source.Find (full size/date/hour filter parity); result pairs are EntryRefs over the source. - mmap sources disposed on reload and on form close (mappings released). - StoreOf/StoreOfPair -> SourceOf/SourceOfPair (IEntrySource); SameRoot compares sources. - Test: with a real migrated .cdex present, the presenter loads via mmap, does NOT call the .cde loader, and releases the mapping on close (33 presenter tests pass). Validated via build + unit tests only; the live WinForms UI was not run here and needs manual verification (catalog list, search, navigation, reload, exit).
hash and dupes now consume (and hash produces) the .cdex format instead of .cde. The tree-based hashing engine (Duplication) is reused unchanged: each .cdex is reconstructed into a mutable tree, hashed, and written back as a fresh .cdex; dupes reconstructs trees and reuses the existing detection. - CatalogTreeBuilder.FromSource/FromColumnarFiles: rebuild a mutable RootEntry tree from an IEntrySource (inverse of EntryStore.Build), copying modified-ticks/flags/hash verbatim for exact round-trip. - IEntrySource.ModifiedTicksOf exposes raw stored ticks for reconstruction. - HashCatalogCommandHandler: source .cdex (GetColumnarFileList), ApplyHash, write each back via ColumnarFormat.Write. No .cdex found -> warn + no-op. - FindDuplicateCommandHandler: source .cdex, reuse FindDuplicates. - Tests: tree<->columnar round-trip incl. hashes; DuplicationTest exercise migrated to scan -> migrate -> hash(.cdex) -> dupes flow. 158 lib tests. Verified end-to-end: scan -> migrate -> rm .cde -> hash -> dupes reports the correct duplicate files from .cdex only.
scan now produces the zero-copy .cdex format instead of .cde, and reuses hashes from an existing .cdex on re-scan (reconstructed into a tree). With this, .cdex is the only format any command writes; 'cde migrate' becomes a one-time upgrade tool for legacy .cde catalogs rather than a routine step. - Both scan handlers (cde/ScanProgress active CLI override + cdeLib default) reuse hashes from .cdex (CatalogTreeBuilder.FromSource) and save via ColumnarFormat.Write(EntryStore.Build(re)). - Fix a latent bug surfaced by this: TraverseTreesCopyHash copied Hash + IsPartialHash but never set IsHashDone, so a reused hash was silently dropped on the next save (affected both DirEntry and RootEntry copy paths, for .cde too). Now sets IsHashDone on copy. - Tests: reconstruct-from-.cdex -> TraverseTreesCopyHash carries the hash (159 lib tests); DuplicationTest simplified now scan emits .cdex directly. Verified end-to-end: scan -> .cdex (no .cde); hash; dupes finds dupes; re-scan WITHOUT re-hash -> dupes still finds them (hash reuse persists).
Mirror the scan progress UX for `cde hash`. Hashing work emits progress and status events that a Spectre.Console status spinner renders live, instead of progress only going to the log. - Duplication: add opt-in ProgressEvent/StatusMessageEvent callbacks; fall back to the existing logger when unset (dupes path/tests unchanged). - cde/HashProgress: override handler runs work on a background task and drives HashProgressConsole; consumers update the display state. - AppContainerBuilder: register the override and exclude the cdeLib hash handler from message-bus auto-declaration (same pattern as scan). Fix percentage exceeding 100%: FilesProcessed is cumulative across the partial and full hash passes, but the denominator was only the partial-pass set size. Each phase now reports its own 0..100% progress (full phase uses a captured baseline), with a display-side clamp as a safety net.
- Document the 'cde migrate' command (one-way .cde -> .cdex conversion) - Update stale .Net7 reference to .NET 10 in Readme.md - developer.md: build now uses Fallout (replaced Nuke); add build details
The project does not use MediatR; CQRS/messaging is via SlimMessageBus (IRequestHandler/IMessageBus). Update serialization docs to reflect the current columnar .cdex format (zero-copy mmap) as primary, with the legacy .cde tree format using MessagePack (default), FlatSharp, or protobuf-net.
Directory junctions and symbolic links (reparse points) carry the Directory attribute, so the scan previously descended into them. This could cause scan cycles (e.g. a junction pointing at an ancestor) and duplicated catalog content. By default the scan now records reparse points but does not follow them. Pass --follow-junctions to restore the previous descend-into behaviour. The flag is threaded CLI -> CreateCacheCommand -> RootEntry.RecurseTree.
Document that PowerShell here-strings (@'...'@) and Bash here-docs (<<'EOF') are not interchangeable across the two shell tools, after a stray @ leaked into a git commit message from using the wrong syntax.
- Drop the redundant explicit handler RegisterType<>().AsSelf() block; SlimMessageBus AutoDeclareFrom already self-registers handlers and builder.Populate surfaces them into Autofac. - Extract focused helpers (ConfigureMessageBus, RegisterCoreServices, WarnMissingConfig) and convert BuildContainer to TryBuildContainer. - Replace the hardcoded cdeLib-override exclusion list with a convention: skip any cdeLib request handler whose request type the cde assembly also handles (IConsumer pub/sub events stay additive). Self-maintaining. - Move cdeLib Logger->ILogger registration into CdelibModule where it belongs. Note: RegisterCoreServices takes IConfigurationRoot (not IConfiguration) so the instance registers as IConfigurationRoot, which Configuration depends on.
Program was a god class: composition root, CLI parser, and every command's implementation, pulling dependencies ad-hoc via Resolve<T>() (service locator). - Add CdeApp: command implementations with IFindService/ICatalogRepository/ IMessageBus injected via constructor. Removes all 8 scattered Resolve<T>() calls; Program now resolves one CdeApp and dispatches parsed verbs to it (382 -> ~110 lines, single responsibility). - Drop the Task.Run().Wait() sync-over-async wrappers around MessageBus.Send; call GetAwaiter().GetResult() directly (no threadpool hop). - FindPopulous: remove the redundant second Where filter and the CompareDirEntries comparator; single Where + OrderByDescending. - Extract MigrateOne, HandleReplCommand, PrintReplHelp from long methods. Static Program.CreateCache/HashCatalog shims retained (delegate to CdeApp) so cdeLibTest/DuplicationTest keeps driving scans through Program.
Main is now async Task<int> and dispatches the parsed verb via pattern matching. Bus-backed commands (CreateCache/Hash/Dupes/Update) return Task and are awaited directly, so the production path no longer blocks a thread on GetAwaiter().GetResult(). Synchronous interactive/inspection commands are adapted to a completed task via RunSync. The verb switch also replaces the long WithParsed fluent chain. The static Program.CreateCache/HashCatalog shims (used only by DuplicationTest) block on the async methods; that blocking is now confined to test-support code, not the CLI entry path. Test remains unchanged and green.
…ionCancellation The Ctrl-C break used a global mutable static (Hack.BreakConsoleFlag) checked across scan traversal, the two-phase hashing, and the CLI inspection loops. - Add OperationCancellation: an injectable, resettable cancellation signal that exposes a real CancellationToken (registered SingleInstance in CdelibModule). Reset() supports the existing "press break again per phase" semantics that a one-shot CancellationToken cannot. - RootEntry.PopulateRoot/RecurseTree/TryEnumerateDirectory take an optional CancellationToken and check token.IsCancellationRequested (Finder/test callers use the default and are unchanged). - Scan handlers (cde + cdeLib) inject the signal, pass its token into the scan, and check it cooperatively after PopulateRoot — preserving the graceful "incomplete scan will not be saved" behaviour (no OperationCanceledException). - Duplication injects the signal; the two hash phases check IsCancellationRequested and Reset() between them. Program's Ctrl-C handler calls Cancel(); CdeApp uses it for the REPL per-prompt reset and the inspection-loop breaks. - Delete Hack.cs. Tests updated to supply the new dependency.
…ions MigrateOptions is co-located with the other verb option classes in CommandLineOptions.cs; add it to the existing MA0048 suppression list like its siblings. Also remove the suppression for UpgradeOptions, which no longer exists (renamed to UpdateOptions, now in its own file).
Extract the .cdex hash-reuse block into ReuseHashesFromExistingCatalog and the final summary calculation/logging into ReportScanSummary. MainLoop drops under the 60-line limit and reads as a clear sequence of steps. No behaviour change.
Apply a consistent code-style/modernization pass across the solution: prefer const over readonly where applicable, pattern matching (is/or) over comparison chains, and blank-line/formatting normalization. No behaviour change; solution builds and the cdeLibTest suite (159 passed, 7 skipped) is green.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR modernizes CDE’s catalog storage and runtime by introducing a zero-copy, memory-mapped columnar catalog format (.cdex) and wiring core commands plus the WinForms GUI to operate on the new index-addressed data model (SoA / IEntrySource + EntryRef). It also bundles cancellation/DI refactors, updated progress reporting, benchmark/probe harnesses, and a build-system migration to Fallout.
Changes:
- Add a columnar on-disk
.cdexformat and supporting SoA abstractions (IEntrySource,EntryStore,EntryRef) to enable mmap-backed, zero-copy reads. - Update CLI/handlers (scan/hash/dupes/find/migrate) and cdeWin presenter/tests to prefer
.cdexand exercise mmap cleanup behavior. - Introduce injectable cooperative cancellation (
OperationCancellation), new progress event plumbing, updated benchmarks/probes, and Fallout-based build updates.
Reviewed changes
Copilot reviewed 114 out of 116 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/cdeWinTest/TestCDEWinPresenterBase.cs | Update presenter test base to use ICommonEntry and SoA EntryRef roots. |
| src/cdeWinTest/TestCDEWinPresenter_OptimiseRegexPattern.cs | Tighten presenter test fixture fields/overrides. |
| src/cdeWinTest/cdeWinTest.csproj | Bump test SDK/NUnit packages. |
| src/cdeWinTest/CDEWinFormPresenterTest.cs | Adapt tests to ICommonEntry catalog list + add mmap .cdex load test. |
| src/cdeWin/UpDownHelper.cs | Comment typo fix. |
| src/cdeWin/StringExtension.cs | Switch cache lock type to Lock. |
| src/cdeWin/StrictKeyEqualityComparer.cs | Move StrictKeyEqualityComparer into its own file. |
| src/cdeWin/SplitContainerExtensions.cs | Make helper method private. |
| src/cdeWin/Program.cs | Restrict Version/ProductName visibility. |
| src/cdeWin/LoaderForm.cs | Minor condition simplification. |
| src/cdeWin/LoadCatalogService.cs | Add .cdex discovery API for GUI; keep .cde loading path. |
| src/cdeWin/ListViewHelper.cs | Extract interface, tighten visibility, comment cleanup. |
| src/cdeWin/ListViewExtensions.cs | Comment casing cleanup. |
| src/cdeWin/KeyEqualityComparer.cs | Make base comparer ctor protected; remove nested strict comparer. |
| src/cdeWin/IListViewHelper.cs | New extracted interface for ListView helper. |
| src/cdeWin/ICDEWinForm.cs | Catalog list now typed as ICommonEntry. |
| src/cdeWin/ContextMenuHelper.cs | Use field auto-property accessor and remove backing field. |
| src/cdeWin/CDEWinForm.cs | Wire catalog list to ICommonEntry, minor UI code cleanup. |
| src/cdeWin/cdeWin.csproj | Dependency bumps. |
| src/cdeMemProbe/SyntheticCatalog.cs | New deterministic synthetic catalog generator for perf/memory measurement. |
| src/cdeMemProbe/cdeMemProbe.csproj | New probe project targeting net10 with server GC + unsafe enabled. |
| src/cdeLibTest/TimePartialParameterTest.cs | Minor const usage cleanup in tests. |
| src/cdeLibTest/Soa/EntryStoreTests.cs | New tests validating SoA store correctness vs tree find. |
| src/cdeLibTest/Soa/EntryRefTests.cs | New tests validating EntryRef adapter behavior. |
| src/cdeLibTest/RootEntryTest.cs | Minor modern pattern usage and dead-test removal. |
| src/cdeLibTest/Performance/PerformanceTreeTraversal.cs | Convert repeat counts to const. |
| src/cdeLibTest/Infrastructure/Hashing/Crc32.cs | Refactor hash loop via LINQ aggregate. |
| src/cdeLibTest/Infrastructure/DuplicationTests.cs | Update duplication ctor usage with OperationCancellation. |
| src/cdeLibTest/Infrastructure/DuplicationPerfTest.cs | Minor formatting cleanup. |
| src/cdeLibTest/IdeaStructNode.cs | Use C# primary declaration for empty class. |
| src/cdeLibTest/EntryHelperTest.cs | Add regression test for nested FullPath building. |
| src/cdeLibTest/DuplicationTest.cs | Update to .cdex flow and new duplication ctor signature. |
| src/cdeLibTest/DirEntryEnumeratorTest.cs | Use collection expressions. |
| src/cdeLibTest/CommonEntryTest_TraverseTreesCopyHash.cs | Minor cleanup in hash-copy test. |
| src/cdeLibTest/cdeLibTest.csproj | Bump test packages. |
| src/cdeLib/TimePartialParameter.cs | Refactor to use field in getters (currently broken; see comments). |
| src/cdeLib/OperationCancellation.cs | New injectable, resettable Ctrl-C cancellation primitive. |
| src/cdeLib/Module/CdelibModule.cs | Register logger + OperationCancellation in DI. |
| src/cdeLib/Infrastructure/ObjectPool.cs | Improve exception name + collection expression for pooled list. |
| src/cdeLib/Infrastructure/Hashing/MurMurHash3.cs | Minor arithmetic simplification. |
| src/cdeLib/Infrastructure/Hashing/HashHelper.cs | Switch to async stream read in hashing loop. |
| src/cdeLib/Infrastructure/FileSystemAdapter.cs | Minor string literal cleanup. |
| src/cdeLib/Infrastructure/Config/AppConfigurationSection.cs | Remove redundant this.. |
| src/cdeLib/Hashing/HashStatusMessageEvent.cs | New hash status event record. |
| src/cdeLib/Hashing/HashProgressEvent.cs | New hash progress event record. |
| src/cdeLib/Hashing/HashCompletedEvent.cs | New hash completion event record. |
| src/cdeLib/Hashing/HashCatalogCommandHandler.cs | Hash .cdex catalogs by rebuilding trees and rewriting .cdex. |
| src/cdeLib/Hack.cs | Remove global break flag (replaced by OperationCancellation). |
| src/cdeLib/FindService.cs | Route CLI find to SoA store; add .cdex mmap find path. |
| src/cdeLib/FindOptions.cs | Improve cancellation/progress throttling + allocation-free substring path matching. |
| src/cdeLib/Extensions/ListExtensions.cs | Use switch for optimized list/array sort paths. |
| src/cdeLib/Entities/Soa/EntryStoreSearch.cs | New index-based search over SoA store (has per-entry path alloc issue; see comments). |
| src/cdeLib/Entities/Soa/EntryStoreFindOptions.cs | New search options for store-based filtering. |
| src/cdeLib/Entities/Soa/EntryStore.cs | New SoA catalog representation + tree-to-store converter. |
| src/cdeLib/Entities/Soa/EntryRef.cs | New ICommonEntry adapter over IEntrySource (store or mmap). |
| src/cdeLib/Entities/RootEntry.cs | Add follow-junctions + cancellation token scanning; update child typing/covariant view. |
| src/cdeLib/Entities/IEntrySource.cs | New common, index-addressed read-only catalog surface. |
| src/cdeLib/Entities/ICommonEntry.cs | Change Children to IReadOnlyList<ICommonEntry>; counts to uint. |
| src/cdeLib/Entities/EntryHelper.cs | Add allocation-free FullPath substring matcher + refactor path building. |
| src/cdeLib/Entities/DirEntry.cs | Split extension, move dir-only state to lazy side-object, covariant children view. |
| src/cdeLib/Entities/Columnar/Utf8Matcher.cs | New UTF-8 ordinal-ignore-case matcher optimized for ASCII. |
| src/cdeLib/Entities/Columnar/ColumnarFormat.cs | New .cdex columnar writer (has 2GB buffering/cast pitfalls; see comments). |
| src/cdeLib/Entities/CatalogTreeBuilder.cs | New tree reconstruction from IEntrySource (store/mmap). |
| src/cdeLib/Duplicates/FindDuplicateCommandHandler.cs | Update dupes to operate over .cdex catalogs. |
| src/cdeLib/Duplicates/DuplicationStatistics.cs | Expression-bodied computed property. |
| src/cdeLib/Duplicates/Duplication.cs | Add cancellation + progress callbacks; correct phase progress reporting. |
| src/cdeLib/DateTimePartialParameter.cs | Small parsing refactors and const format string. |
| src/cdeLib/cdeLib.csproj | Enable unsafe blocks; dependency bumps. |
| src/cdeLib/Catalog/ICatalogRepository.cs | Add .cdex discovery API. |
| src/cdeLib/Catalog/CreateCacheCommandHandler.cs | Scan writes .cdex and reuses hashes from existing .cdex. |
| src/cdeLib/Catalog/CreateCacheCommand.cs | Add FollowJunctions option. |
| src/cdeLib/Catalog/CatalogRepository.cs | Add .cdex file discovery; minor refactors. |
| src/cdeBenchmarks/SearchBenchmarks.cs | New search throughput benchmark suite. |
| src/cdeBenchmarks/PoolingBenchmarks.cs | Minor loop cleanup + verbatim string. |
| src/cdeBenchmarks/MultiCatalogSearchBenchmarks.cs | New benchmark for multi-catalog sync vs legacy async. |
| src/cdeBenchmarks/Hash16Benchmarks.cs | Modernize array/list initializations and loops. |
| src/cdeBenchmarks/cdeBenchmarks.csproj | Pin Roslyn Workspaces versions; reference cdeMemProbe for shared fixture. |
| src/cdeBenchmarks/CatalogLoadBenchmarks.cs | New load benchmark measuring allocations/timing. |
| src/cdeBenchmarks/CatalogFixture.cs | New shared fixture helper for benchmarks. |
| src/cdeBenchmarks/baseline/soa-prototype.md | Baseline documentation for SoA prototype. |
| src/cdeBenchmarks/baseline/search-baseline.md | Baseline search benchmark results + analysis. |
| src/cdeBenchmarks/baseline/README.md | Baseline harness documentation and repro commands. |
| src/cdeBenchmarks/baseline/phase-results.md | Phase delta tracking document. |
| src/cdeBenchmarks/baseline/name-storage-investigation.md | Investigation notes on name storage footprint options. |
| src/cdeBenchmarks/baseline/footprint-baseline.csv | Committed baseline footprint data. |
| src/cdeBenchmarks/baseline/flatbuffer-mmap-spike.md | Spike results and repro steps for mmap columnar approach. |
| src/cde/ScanProgress/CreateCacheCommandHandler.cs | CLI scan writes .cdex, reuses hashes, uses cancellation token. |
| src/cde/HashProgress/HashStatusMessageHandler.cs | New handler to display hash status messages in Spectre UI. |
| src/cde/HashProgress/HashProgressNotificationHandler.cs | New handler to update Spectre progress state. |
| src/cde/HashProgress/HashProgressConsole.cs | New Spectre progress UI (has thread-safety + busy-spin issues; see comments). |
| src/cde/HashProgress/HashCompletedEventHandler.cs | New completion handler for hash UI. |
| src/cde/HashProgress/HashCatalogCommandHandler.cs | CLI-level hash handler publishing progress/status events. |
| src/cde/GlobalSuppressions.cs | Add suppression for new migrate verb; remove old upgrade suppression. |
| src/cde/CommandLine/CommandLineOptions.cs | Add --follow-junctions and new migrate verb. |
| src/cde/CdeApp.cs | New DI-hosted CLI implementation entry point + .cde→.cdex migrate logic. |
| src/cde/cde.csproj | Dependency bumps (Autofac, Humanizer, analyzers, Spectre). |
| src/cde/AppContainerBuilder.cs | DI/container build refactor + auto handler override filtering. |
| src/cde.slnx | New solution definition file. |
| src/cde.sln.DotSettings | Add cdex to dictionary. |
| Readme.md | Document .NET 10, scan follow-junctions, and migrate command. |
| developer.md | Document Fallout build usage and scripts. |
| claude.md | Update architecture/docs for SlimMessageBus + .cdex primary format + Fallout. |
| build/Configuration.cs | Switch build tooling namespace to Fallout. |
| build/Build.cs | Port build definition from Nuke to Fallout. |
| build/_build.csproj | Switch build properties/packages to Fallout; bump NuGet.Frameworks. |
| build.sh | Update temp directory path to .fallout. |
| build.ps1 | Update temp directory path to .fallout. |
| .gitignore | Ignore BenchmarkDotNet artifacts. |
| .fallout/parameters.json | New Fallout parameters file. |
| .fallout/build.schema.json | New Fallout schema (rename NukeBuild→FalloutBuild). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
13
to
20
| public int Hour | ||
| { | ||
| get | ||
| { | ||
| ThrowExceptionIfSet(); | ||
| return _hour; | ||
| return field; | ||
| } | ||
| } |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…d compile - TimePartialParameter: add private set to Hour/Minute/Second so the constructor can assign while keeping the API read-only (fixes compile against the get-only field-keyword properties). - EntryStore: add TryWriteFullPath(Span<char>) — an allocation-free mirror of AppendFullPath that writes the path into a caller buffer. - EntryStoreSearch: substring path searches now build the path into a rented char buffer and run Span<char>.Contains instead of allocating a full-path string per entry; only regex mode materialises a string. Applied to both Find overloads. - HashProgressConsole: use ConcurrentQueue<string> for the cross-thread message queue (handlers enqueue while the UI loop dequeues), and add a 50ms sleep to the progress loop so it stops busy-spinning a core.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Zero-copy memory-mapped columnar catalog (
.cdex) + perf/runtime modernizationTL;DR
Replaces the materialized MessagePack
.cdedirectory tree with a struct-of-arrayscolumnar format (
.cdex) that is read zero-copy over a memory map. "Loading" acatalog becomes mmap-ing it: no managed object graph is built, and the working set is
only the file pages a query touches (reclaimable OS page cache, not GC heap). All five
commands —
scan,find,hash,dupes, and thecdeWinGUI — run on.cdex.Measured impact
On a real 649k-entry catalog (
flatbuffer-mmap-spike.md):.cde).cdex)Cost: ~+36% on-disk size; working set ≈ touched (reclaimable) pages.
Headline feature — the
.cdexarcperf(spike)SoA prototype — parallel value-type arrays; tree shape viafirstChild/nextSibling/parentint indices. Proved −67% structural footprint.perf(soa)productionEntryStore+ index search (P1–P5) — additive; 12equivalence tests prove SoA search results are identical to the pointer-tree
FindOptionsacross every query shape. CLIfindandcdeWinwired onto it.spike(mmap)zero-copy.cdex— hand-rolled SoA writer (primitive columns +UTF-8 name blob) and an mmap reader exposing columns as
ReadOnlySpanover themapping; zero-alloc UTF-8 byte-scan search.
feat(columnar)promotion tocdeLib— hardened vs the spike: 64-bit name-bloboffsets (no 2 GB cap), Unicode-correct case folding (
Utf8Matcher), magic+versiongate. Adds
cde migrate(MessagePack.cde→.cdex).IEntrySource— common read-only, index-addressed surface over both the in-memoryEntryStoreand the mmapColumnarCatalogReader;EntryRefand the GUI work overeither backing transparently.
.cdex—find(zero-copy over mmap, falls back to.cde),hash/dupes(rebuild a mutable tree per.cdex, reuse the unchanged detectionengine), and
scanwrites.cdexdirectly and reuses hashes from an existing.cdexon re-scan.Notable bug fixed along the way
TraverseTreesCopyHashcopiedHash+IsPartialHashbut never setIsHashDone, so areused hash was silently dropped on the next save (affected
.cdetoo). Now fixed.Other themes bundled in this branch
find at 8 B/entry (non-breaking), allocation-free single-pass path search, lean files
via an
ExtraDataside-object.CdeApp; make the CLI pathfully async (drop sync-over-async); replace the global
Hack.BreakConsoleFlagwith aninjectable
OperationCancellation; simplifyAppContainerBuilderregistration.scan: add--follow-junctions(default off).either backing transparently.
.cdex—find(zero-copy over mmap, falls back to.cde),hash/dupes(rebuild a mutable tree per.cdex, reuse the unchanged detectionengine), and
scanwrites.cdexdirectly and reuses hashes from an existing.cdexon re-scan.Notable bug fixed along the way
TraverseTreesCopyHashcopiedHash+IsPartialHashbut never setIsHashDone, so areused hash was silently dropped on the next save (affected
.cdetoo). Now fixed.Other themes bundled in this branch
find at 8 B/entry (non-breaking), allocation-free single-pass path search, lean files
via an
ExtraDataside-object.CdeApp; make the CLI pathfully async (drop sync-over-async); replace the global
Hack.BreakConsoleFlagwith aninjectable
OperationCancellation; simplifyAppContainerBuilderregistration.scan: add--follow-junctions(default off).bounds, cross-instance delegate corruption).
New source (selected)
cdeLib/Entities/Columnar/{ColumnarFormat,ColumnarCatalogReader,Utf8Matcher}.cscdeLib/Entities/IEntrySource.cscdeLib/Entities/Soa/{EntryStore,EntryStoreSearch,EntryRef,EntryStoreFindOptions}.cscde/CdeApp.cs,cdeLib/OperationCancellation.cscdeLibTest/Columnar/ColumnarCatalogTests.cs,cdeLibTest/Soa/{EntryStoreTests,EntryRefTests}.cscdeBenchmarks/baseline/{soa-prototype,flatbuffer-mmap-spike}.mdCompatibility / migration
.cdexis now the only format any command writes.cde migrate [file]upgradeslegacy
.cde(current dir + one level down, or a single file) — a one-time step.findstill reads legacy.cdewhen no.cdexis present.cdeLibenablesAllowUnsafeBlocksfor the mmap pointer acquisition.Testing
cdeLibtests green, including round-trip + search-parity (substring/regex,name/path, file/folder, size/date/hour filters) between
.cdex,EntryStore, and thelegacy pointer tree.
cdeWinpresenter tests confirm the GUI loads via mmap (does not call the.cdeloader when
.cdexexist) and releases mappings on reload/close.scan → .cdex(no.cde) →hash→dupes; re-scan withoutre-hash still finds dupes (hash reuse persists).
navigation / reload / exit recommended before merge.