akspa0 · akspa0 · Sep 21, 2025 · Sep 21, 2025 · Sep 21, 2025 · Sep 21, 2025
diff --git a/.gitattributes b/.gitattributes
@@ -0,0 +1,23 @@
+# Ensure shell scripts and source files stay LF for Linux execution
+*.sh text eol=lf
+*.bash text eol=lf
+scripts/linux/** text eol=lf
+*.py text eol=lf
+*.md text eol=lf
+*.yml text eol=lf
+*.yaml text eol=lf
+*.toml text eol=lf
+*.json text eol=lf
+
+# Treat KryoFlux streams and other binaries as binary
+*.raw binary
+*.bin binary
+*.png binary
+*.jpg binary
+*.jpeg binary
+*.gif binary
+*.bmp binary
+*.webp binary
+
+# Optional: avoid diffing large binaries explicitly (binary macro already sets this)
+# *.raw -diff
diff --git a/README.md b/README.md
diff --git a/docs/Kryoflux STREAM format reference.md b/docs/Kryoflux STREAM format reference.md
diff --git a/docs/cheatsheet.md b/docs/cheatsheet.md
@@ -1,43 +1,43 @@
 # FloppyAI Cheatsheet
 
-Quick-reference for the most common workflows and flags. Run everything from the repository root using the main script. Module syntax is also supported as an alternate.
+Quick-reference for the most common workflows and flags. Run everything from the `FloppyAI/` directory using the main script. Module syntax is also supported as an alternate.
 
 - Show help
   ```bash
-  python FloppyAI/src/main.py --help
+  python src/main.py --help
   ```
 
 ## One‑liners
 
 - Analyze a single .raw file
   ```bash
-  python FloppyAI/src/main.py analyze path\to\file.raw --output-dir .\test_outputs\single
+  python src/main.py analyze path\to\file.raw --output-dir .\test_outputs\single
   ```
 
 - Analyze a full disk directory (PC 3.5" HD, MFM overlays)
   ```bash
-  python FloppyAI/src/main.py analyze_disk path\to\disk_dir \
+  python src/main.py analyze_disk path\to\disk_dir \
     --media-type 35HD --format-overlay --overlay-mode mfm --angular-bins 720 \
     --output-dir .\test_outputs\disk_run
   ```
 
 - Analyze a Mac GCR disk (400K/800K, zoned)
   ```bash
-  python FloppyAI/src/main.py analyze_disk path\to\mac_disk \
+  python src/main.py analyze_disk path\to\mac_disk \
     --media-type 35DD --format-overlay --overlay-mode gcr \
     --gcr-candidates "12,10,8,9,11,13" --angular-bins 900 \
     --output-dir .\test_outputs\mac_run
   ```
 
 - Compare two reads (after each has a surface_map.json)
   ```bash
-  python FloppyAI/src/main.py compare_reads .\test_outputs\win95_0 .\test_outputs\win95_1 \
+  python src/main.py compare_reads .\test_outputs\win95_0 .\test_outputs\win95_1 \
     --output-dir .\test_outputs\diff_win95
   ```
 
 - Build a corpus, auto‑generating per‑disk maps first
   ```bash
-  python FloppyAI/src/main.py analyze_corpus .\stream_dumps \
+  python src/main.py analyze_corpus .\stream_dumps \
     --generate-missing --media-type 35HD \
     --format-overlay --overlay-mode mfm --angular-bins 720 \
     --output-dir .\test_outputs\corpus
@@ -85,23 +85,23 @@ Quick-reference for the most common workflows and flags. Run everything from the
 - PC disk (1.44MB) two‑read comparison
   ```bash
   # Read A
-  python FloppyAI/src/main.py analyze_disk FloppyAI\stream_dumps\1.44\win95boot\0 \
+  python src/main.py analyze_disk .\stream_dumps\1.44\win95boot\0 \
     --media-type 35HD --format-overlay --overlay-mode mfm --angular-bins 720 \
     --output-dir .\test_outputs\win95_0
 
   # Read B
-  python FloppyAI/src/main.py analyze_disk FloppyAI\stream_dumps\1.44\win95boot\1 \
+  python src/main.py analyze_disk .\stream_dumps\1.44\win95boot\1 \
     --media-type 35HD --format-overlay --overlay-mode mfm --angular-bins 720 \
     --output-dir .\test_outputs\win95_1
 
   # Compare
-  python FloppyAI/src/main.py compare_reads .\test_outputs\win95_0 .\test_outputs\win95_1 \
+  python src/main.py compare_reads .\test_outputs\win95_0 .\test_outputs\win95_1 \
     --output-dir .\test_outputs\diff_win95
   ```
 
 - Mac corpus (generate + aggregate)
   ```bash
-  python FloppyAI/src/main.py analyze_corpus .\stream_dumps\mac \
+  python src/main.py analyze_corpus .\stream_dumps\mac \
     --generate-missing --media-type 35DD \
     --format-overlay --overlay-mode gcr --gcr-candidates "12,10,8,9,11,13" \
     --angular-bins 900 --output-dir .\test_outputs\corpus_mac
@@ -147,4 +147,4 @@ Quick-reference for the most common workflows and flags. Run everything from the
   - For Mac disks, ensure `--overlay-mode gcr` and pass good `--gcr-candidates`.
 
 - Package not found when using `-m`:
-  - Run from the repository root (the folder that contains `FloppyAI/`).
+  - Run from the `FloppyAI/` directory and use `python -m src.main ...`.
diff --git a/docs/decoder_port_plan.md b/docs/decoder_port_plan.md
@@ -0,0 +1,126 @@
+# Decoder Port Plan (Greaseweazle‑inspired)
+
+This plan describes how we will re‑implement, in our own code, a compact flux‑to‑sectors decoder pipeline inspired by Greaseweazle, tailored for our KryoFlux C2/OOB streams and analysis workflow.
+
+## Goals
+- Provide optional, format‑aware decoding from raw flux to sectors for verification and experiments.
+- Start with IBM MFM (PC 3.5"/5.25"), expand later to FM and Apple GCR.
+- Keep decoding separate and optional; our primary focus remains flux‑first analysis.
+- Produce small, useful artifacts (JSON summaries, light visuals) without per‑track PNG spam.
+
+## Non‑Goals (for now)
+- Full filesystem extraction or preservation imaging beyond sector maps.
+- Cross‑machine orchestration changes (Linux DTC workflow remains as‑is).
+- Copying or vendoring Greaseweazle code: we re‑implement the necessary logic.
+
+## Inputs & Outputs
+- Input: Our KryoFlux C2/OOB `.raw` files (`trackNN.S.raw`) with multiple revolutions.
+- Outputs:
+  - `decode_summary.json` per track: sectors (CHRN), CRC status, data size, best‑rev selection, PLL/jitter stats.
+  - Optional lightweight visual marker PNG per track (sync/IDAM/DAM positions), disabled by default.
+  - Integration hooks to append decode metrics into corpus/analysis summaries.
+
+## Status Update — STREAM Write (2025‑09‑22)
+
+- HxC reads our generated STREAM files and shows rough flux for Side 0; Side 1 appears sparse/garbled depending on the pattern.
+- DTC refuses to open the generated files for writing, reporting:
+  - `Image name:` / `Can't open image file:`
+- We strictly used STREAM mode and CLI ordering (`-f<prefix>` first, then `-i0 -d -s -e -g -w`) and tried multiple base variants (`-ftrack`, `-ftrack.raw`, absolute `-f$(pwd)/track`, `-f$(pwd)/track.raw`). None were accepted by DTC on the test host.
+
+Planned actions for alignment (tomorrow):
+- Gather known‑good reference STREAMs via DTC capture for a few tracks/sides (e.g., 00.0, 00.1).
+- Implement `kfx_probe.py` to dump OOB blocks (type/len/offsets) and opcode histograms; compare reference vs generated.
+- Ensure OOB type 2 (Disk Index) includes a 16‑bit LE timer payload; verify initial index presence and header mode expectations (ASCII preamble vs OOB‑first).
+- Adjust `src/stream_export.py` to mirror the reference files byte‑for‑byte per rev 1.1 and empirical DTC behavior.
+
+See also:
+- `docs/kryoflux_stream_notes.md` — concise summary of the STREAM protocol rev 1.1 we are following.
+- `docs/stream_write_investigation.md` — commands tried, variants matrix, and findings log.
+
+## Module Layout (new)
+
+```text
+FloppyAI/src/decoding/
+  c2_adapter.py    # Convert C2/OOB -> per-rev flux transition times (+index boundaries)
+  pll.py           # VFO/PLL for bitcell recovery (phase/period tracking, jitter stats)
+  mfm.py           # MFM framer: sync detect (A1), IDAM & DAM parse, CRC16-IBM
+  fm.py            # (Phase 3) FM decoder
+  gcr.py           # (Phase 3) Apple GCR (400K/800K), zoned RPM awareness
+  track_decode.py  # Orchestrate per-rev decode; choose best rev per sector; summarize
+
+FloppyAI/tools/
+  flux2sectors.py  # CLI: .raw -> decode_summary.json (and optional markers)
+```
+
+## Data Flow
+
+```mermaid
+flowchart LR
+    KF[KryoFlux C2/OOB .raw] --> AD[c2_adapter]
+    AD --> FLUX[Per-rev flux times]
+    FLUX --> PLL[pll: VFO/bitcell]
+    PLL --> MFM[mfm: framer & CRC]
+    MFM --> TD[track_decode: best-of per track]
+    TD --> OUT[decode_summary.json]
+```
+
+## Algorithm Highlights
+- **C2 Adapter (`c2_adapter.py`)**
+  - Reuse our existing C2/OOB parsing (from `FluxAnalyzer.parse`) to extract transitions and index events.
+  - Convert ticks to seconds with `sck_hz`; preserve per‑rev segmentation.
+
+- **PLL/VFO (`pll.py`)**
+  - Nominal bitcell from profile/RPM (e.g., ~4 µs MFM @ 300 RPM zones).
+  - For each transition: compute phase error; gently update period (alpha/beta). Jitter stats (RMS) per rev.
+  - Emit a clocked stream (clock/data) suitable for MFM framing.
+
+- **MFM Framer (`mfm.py`)**
+  - Detect sync (A1 with missing clock bit) sequences to align.
+  - Parse IDAM (FE) → CHRN, size; DAM/DE/FB → data and CRC16‑IBM checks.
+  - Robust re‑sync after noise; tolerate weak bits between frames.
+
+- **Track Assembly (`track_decode.py`)**
+  - Decode each revolution independently; pick best rev per sector (CRC‑OK first; else best score).
+  - Summarize coverage, per‑rev success ratio, jitter/lock stats.
+
+## CLI & Integrations
+- **`tools/flux2sectors.py`**
+  - `--input track00.0.raw` → `decode_summary.json` (and optional markers).
+  - Options: `--format mfm` (default), `--rpm`, `--profile`, `--rev-limit`.
+- **`tools/analyze_captures.py --decode-mfm` (Phase 2)**
+  - Append per‑file decode metrics (sectors_ok/total, jitter_rms_ns, best_rev_ok_ratio).
+- **`tools/compare_corpus.py` (Phase 2)**
+  - Include jitter deltas and “best rev” changes across passes for stability analysis.
+
+## Phases & Deliverables
+- **Phase 1 (MFM MVP)**
+  - Implement: `c2_adapter.py`, minimal `pll.py`, `mfm.py`, `track_decode.py`, `tools/flux2sectors.py`.
+  - Validate on known MFM captures; confirm plausible sector maps and CRC‑OK counts.
+- **Phase 2 (Integration)**
+  - Add `--decode-mfm` to `analyze_captures.py` and corpus comparison metrics (jitter, best rev).
+- **Phase 3 (Formats)**
+  - Add FM decoder; add Apple GCR decoder with zoned RPM awareness.
+  - Optional: simple IMG writer for PC MFM (flat sector order) for quick mount tests.
+
+## Testing Strategy
+- Unit‑level tests for PLL (phase/jitter), MFM CRC, sync detection.
+- Golden‑file tests: small known MFM track(s), verify sector map.
+- End‑to‑end smoke: `.raw -> flux2sectors.py -> decode_summary.json`, inspect counts and CRC.
+
+## Success Criteria
+- Phase 1: For clean MFM tracks, majority of sectors decode with CRC‑OK in at least one rev.
+- Phase 2: Decode metrics appear in analysis summaries and correlate with stability findings.
+- Phase 3: FM and GCR paths operational on representative media.
+
+## Risks & Mitigations
+- **RPM/bitcell mismatch**: Use profile defaults; allow CLI overrides; add auto‑hint from index timing.
+- **Weak media**: Best‑rev selection masks some issues; still report jitter/lock stats.
+- **Performance**: Offer `--rev-limit` and skip visuals by default; vectorize hot loops as needed.
+
+## Licensing Note
+We implement our own decoder logic, informed by public knowledge and experimentation. We do not copy or vendor Greaseweazle code; any similarities are conceptual. This keeps our core licensing unconstrained while enabling experiments.
+
+## Next Steps
+1. Implement Phase 1 modules and `tools/flux2sectors.py`.
+2. Validate on a known MFM dataset.
+3. Integrate optional decode into `analyze_captures.py` and `compare_corpus.py`.
diff --git a/docs/dtc_stream_rebuild_plan.md b/docs/dtc_stream_rebuild_plan.md
@@ -0,0 +1,62 @@
+# DTC Stream Rebuild Plan (CAPSImg-Grounded)
+
+## Goals
+- Produce KryoFlux STREAM files that DTC accepts without error.
+- Eliminate guesswork by matching DTC’s own output byte-for-byte for OOB structure and payload formatting.
+- Keep HxC-compatible path intact; add a strict DTC path for hardware workflows.
+
+## References (local)
+- CAPSImg sources: `lib/capsimg_source_windows/`
+  - Detection logic: `CAPSImg/DiskImageFactory.cpp::IsKFStream` (OOB-first; Info type at offset 0; looks for "KryoFlux" in Info payload)
+  - Stream classes: `CStreamImage`, `CStreamCueImage` (we do not require cue files)
+- Real DTC captures: use your recent working set under `FloppyAI/stream_dumps/...`
+- Our writers:
+  - HxC/default path: `src/stream_export.py` (kept unchanged)
+  - DTC-style path (baseline): `src/stream_export_dtc.py`
+
+## Constraints and Decisions
+- DTC path: OOB-first streams (no ASCII preamble), per CAPS detection logic.
+- First OOB must be Info (0x0D, 0x04, sizeLE, payload) and early payload must contain the string "KryoFlux".
+- We match DTC’s header content and ordering exactly:
+  - KFInfo #1: `host_date=YYYY.MM.DD, host_time=HH:MM:SS, hc=0`
+  - KFInfo #2: `name=KryoFlux DiskSystem, version=V, date=Mon DD YYYY, time=HH:MM:SS, hwid=1, hwrv=1, hs=1, sck=S, ick=I`
+  - Initial `StreamInfo` (SP=0, ms=0), periodic `StreamInfo` blocks with SP equal to ISB bytes at insertion, and ms matching DTC cadence
+  - Index (0x02) only at rev boundaries; `SampleCounter` equals last emitted flux ticks; `IndexCounter` derived from sck->seconds->ick using DTC’s rounding
+  - At least one flux cell after the final Index; then `StreamEnd` (SP, RC=0); then EOF sentinel (0x0D with size 0x0D0D => 4 bytes 0x0D)
+
+## Work Plan
+- Step 1. Implement OOB diff tooling
+  - File: `tools/stream_oob_diff.py`
+  - Parse OOB blocks from two streams and print a structured diff: types, positions, sizes, and decoded payloads for Info/StreamInfo/Index/StreamEnd/EOF.
+  - Use this to drive byte-accurate convergence against a real capture.
+
+- Step 2. Create strict DTC writer module
+  - File: `src/stream_export_dtc_strict.py`
+  - Emit OOB-first blocks matching DTC format; port text formatting (spacing, date formats, float precision for sck/ick) based on real capture.
+  - Maintain structural fixes: `Index.SampleCounter = last_flux_ticks`; add a small dummy flux after final Index; `StreamEnd` then EOF.
+
+- Step 3. Wire generator with strict writer option
+  - File: `tools/patterns_to_stream_set.py`
+  - Add `--writer strict-dtc` that imports and invokes `write_kryoflux_stream_dtc_strict`.
+  - Force header mode to OOB-first for this path.
+
+- Step 4. Validate & Iterate
+  - Use `tools/stream_oob_diff.py` to compare `strict-dtc` output with a real DTC capture (same track/side), adjust until OOB diff is clean.
+  - Run `tools/kfx_probe.py --validate` to confirm structural integrity.
+  - On Linux DTC host, run `scripts/linux/dtc_write_read_set.sh --image-dir <dir> --drive 0 --tracks <range> --sides <list>`; inspect the script’s log for the exact dtc command and output.
+
+## Acceptance Criteria
+- OOB diff shows identical OOB sequence (type, order, sizes, payload text/fields) between our output and a real DTC capture for the same reference case.
+- `kfx_probe.py --validate` reports OK.
+- `dtc_write_read_set.sh` successfully writes `strict-dtc` streams on the Linux host.
+
+## Risks & Mitigations
+- DTC build variations: mitigate by basing on your exact real capture and dtc version.
+- Float precision in sck/ick: tune formatting to match reference text exactly.
+- Periodic `StreamInfo` cadence: mirror reference SP and ms pattern.
+
+## Next Actions
+- Implement `tools/stream_oob_diff.py`.
+- Implement `src/stream_export_dtc_strict.py`.
+- Add `--writer strict-dtc` to `tools/patterns_to_stream_set.py`.
+- Run OOB diff against your provided reference and iterate until identical.