Skip to content

refactor: split cmd/server/ monolith files (store.go 7389L, routes.go 2714L, db.go 2346L) — contributor onboarding cliff #1382

@Kpa-clawbot

Description

@Kpa-clawbot

Background

This is the second time a serious contributor has bounced off the code shape. The "we're so unfriendly to contributors people are talking about a from-scratch rewrite" failure mode is a maintainer-level problem, not a style preference.

Concrete current state

cmd/server/store.go      7389 lines  ← in-memory store, every feature
cmd/server/routes.go     2714 lines  ← all HTTP handlers
cmd/server/db.go         2346 lines  ← SQLite reads + schema reads
cmd/server/types.go       991 lines  ← all wire/protocol types
cmd/ingestor/main.go     1223 lines  ← ingestor entry + handlers
cmd/ingestor/db.go        991 lines  ← ingestor inserts
total                  15,654 lines  across 6 files

For comparison, the entire cmd/server/ package would fit comfortably in 30-40 focused files of <500 lines each — typical Go service shape.

What's wrong with the current shape

  1. Every recent PR (5-6 this week alone — bug(paths-through-node): 1-byte prefix collision resolves arbitrarily — attributes packets to wrong node #1352, fix(#1356): WCAG 2.2 AA map a11y — cluster bubbles, role pills, multi-byte labels #1357, bug(channels): channel view shows stale messages — latest packets hidden by dedupe keeping first-observed timestamp + insertion order #1366, fix(#1366): channels view shows latest message time — backend emits LatestSeen, not FirstSeen #1368, fix(#1370): revert ingestor envelope-timestamp path — server ingest time for packet/observation storage (counters #1233) #1372, bug(channels): /api/channels emits ghost 'unknown' bucket for encrypted-no-key packets — shows up alongside real channels after PSK add #1373) touches the same 2-3 monolith files. Merge conflicts on master are constant. Worktree-based parallel work is harder than it should be.
  2. Cognitive load to make a one-line change: contributors must scroll a 7400-line file to find the right function, then keep ambient state about what else lives in that file in their head while editing.
  3. No package boundaries inside the service — every function in cmd/server/store.go is in package main. There's no compiler-enforced "channels code can't reach into multibyte internals" — everything is one big lump of state.
  4. AI-coding-friendly only by accident: large-context models tolerate this; humans don't. Filtering for "humans can contribute" is the higher bar.
  5. Specific reviewer feedback: external contributors have publicly said it's why they're not contributing / are considering forks.

Proposed direction (high-level — design pass before any code)

Split cmd/server/store.go and cmd/server/db.go along feature axes. Suggested first cut (subject to design):

  • cmd/server/store/store.go — core PacketStore struct + Load/Save lifecycle (~300 lines)
  • cmd/server/store/channels.goGetChannels, GetChannelMessages, channel-list aggregation (~600 lines)
  • cmd/server/store/multibyte.gocomputeMultiByteCapability, hash-size analytics (~500 lines)
  • cmd/server/store/paths.gonodeInResolvedPath, paths-through-node, path-hop index (~800 lines)
  • cmd/server/store/analytics.goGetAnalyticsChannels, payload-mix, RF analytics (~700 lines)
  • cmd/server/store/topology.go — neighbor-graph build, topology, edges (~600 lines)
  • cmd/server/store/observers.go — observer registry, region resolution (~400 lines)
  • cmd/server/store/disambig.goresolveWithContext, buildPrefixMap, fallback resolution (~500 lines)
  • cmd/server/store/dedup.go — broadcastTxs, dedupe key logic, cache invalidation (~400 lines)
  • residual catch-all for genuinely shared helpers (~500 lines)

Same axis for routes.gohandlers/<domain>.go (channels, analytics, nodes, packets, live, multibyte, paths, …).

For db.go (DB-backed sibling), mirror the same split.

Optional: introduce a internal/store/ package to make the splits compiler-enforced (cross-domain calls become explicit imports).

Acceptance criteria (design phase only)

This issue is a DESIGN issue — file PR(s) only after the design lands here in comments.

  • A design comment lands on this issue with: (a) the proposed file split (names + responsibilities), (b) the proposed package boundary (single package main vs internal/store/), (c) the migration ordering (which files split first, what's a single PR vs a series), (d) the merge-conflict mitigation plan (announce + freeze window for in-flight branches).
  • Expert review on the design (Munger for architecture, Torvalds for the "is this actually simpler" check) BEFORE implementation.
  • Implementation PR series cleared with the project, in order:
    • PR 1: tooling only — add a make refactor-precheck (or equivalent) script that asserts git mv-based moves preserve go test ./... -count=1 byte-equality on test output. No code moves yet.
    • PR 2: extract one domain (suggest: channels.go) using only git mv of function blocks. CI green proof. Zero behavior change.
    • PR 3+: subsequent domain extractions, one per PR, each behind a CI green gate.
  • Each refactor PR MUST be pure — no behavior changes piggybacked. Reviewed strictly under AGENTS.md's "pure refactor" exemption: existing tests byte-unchanged + still green.
  • After refactor: file count goes from ~6 monoliths to ~30-40 focused files; no single file in cmd/server/ exceeds 1000 lines.

Out of scope

  • Rewriting the PacketStore API surface.
  • Switching to a different language / framework.
  • Splitting the binary into multiple processes.
  • Touching cmd/ingestor/ in the first refactor wave (file separately if needed).

Risks

  • Massive merge conflicts for in-flight branches. Mitigation: pick a freeze window, announce, do the first split fast.
  • Loss of git blame lineage. Mitigation: every move uses git mv (preserves blame across renames) — never copy+delete.
  • Reviewer fatigue: 10+ refactor PRs is a lot. Mitigation: make each PR a single domain, tag clearly, batch reviews.

Refs

  • External contributor feedback relayed via Discord on 2026-05-22 (translated above).
  • cmd/server/store.go 7389 lines (master HEAD).
  • cmd/server/routes.go 2714 lines.
  • AGENTS.md "Pure refactor" exemption rule — defines the test-byte-unchanged gate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions