Design: Semantic Federation (Phases 154–158)#132
Merged
Conversation
Add comprehensive design document for distributing semantic knowledge across federated repositories. Three layers: Layer 1: Semantic Objects — rich semantic metadata (embeddings, summaries, keywords, entities) enabling efficient gossip and cross-repo deduplication. Layer 2: Semantic Refs — Git-like refs for semantic index snapshots, enabling incremental transfer and semantic history tracking. Layer 3: Federation — peer discovery via gossip protocol, intelligent query routing via semantic DHT, efficient bulk transfer via semantic packfiles, and automatic semantic commits tracking concept-level changes. Phase 154: Semantic Refs & Incremental Transfer (~2 weeks) Phase 155: Federation Discovery & Gossip (~1.5 weeks) Phase 156: Semantic Query Routing (~3 weeks) Phase 157: Semantic Packfiles (~2.5 weeks) Phase 158: Semantic Commits & Deltas (~2 weeks) Design rationale: - Git distributes bytes. Gitsema distributes meaning. Federation distributes knowledge. - Enables cross-repo semantic queries without centralized index. - Semantic DHT routing reduces broadcast queries + latency. - Semantic packfiles reduce bandwidth 40%+ vs. individual fetches. - Semantic commits enable concept-level causality + change-driven queries. Docs: - docs/design/semantic-federation.md — full design + architecture - docs/PLAN.md — phases 154–158 with scope + acceptance criteria - docs/feature-ideas.md — updated notes linking to new phases Co-Authored-By: ChatGPT feedback on gitsema architecture Claude-Session: https://claude.ai/code/session_013HWxjJmmohRPvkgvTBTKUC
Address three critical design gaps based on feedback:
1. CHUNK-LEVEL GRANULARITY
- Semantic objects wrap chunks, not blobs
- Enables fine-grained deduplication (identical snippets = one object)
- Summaries are specific ("JWT validation in session.ts:42–87") not broad ("auth module")
- Updated database schema + design doc
2. CROSS-MODEL FEDERATION
- Different repos use different models (nomic-embed-text vs text-embedding-3-small)
- Vectors in different spaces are not directly comparable
- Phase 154–158: Require model match, reject cross-model federation with warning
- Phase 159: Plan re-embedding strategy for cross-model support
- Storage: profile_version + model_dimensions tracked on every semantic object
3. FEDERATION SEARCH STORAGE STRATEGY
- `--cache` (default): Store in session-scoped federation_cache table
- `--import`: Merge into permanent semantic_objects (mark as federation_imported)
- `--no-cache`: Transient (show results, don't store)
- Users control federation integration: pure federation vs hybrid vs integrated
Phase 154 revisions:
- Effort: ~2 weeks → ~2.5 weeks (semantic enrichment pipeline)
- Risk: Low → Medium (LLM integration testing)
- Chunk-level granularity + model versioning explicit
Phase 156 revisions:
- Storage modes fully specified (cache table schema, TTL, metadata tracking)
- Cross-model peers filtered out (same profile_version only)
- Effort: ~3 weeks → ~3.5 weeks (caching adds state management)
- Detailed test scenarios for storage modes + cross-model rejection
Updated docs:
- docs/design/semantic-federation.md — chunk design, cross-model constraint, storage strategy
- docs/PLAN.md — phases 154 + 156 with gaps filled
Co-Authored-By: User feedback on semantic federation design
Claude-Session: https://claude.ai/code/session_013HWxjJmmohRPvkgvTBTKUC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add comprehensive design document for distributing semantic knowledge across
federated repositories. Three layers:
Layer 1: Semantic Objects — rich semantic metadata (embeddings, summaries,
keywords, entities) enabling efficient gossip and cross-repo deduplication.
Layer 2: Semantic Refs — Git-like refs for semantic index snapshots,
enabling incremental transfer and semantic history tracking.
Layer 3: Federation — peer discovery via gossip protocol, intelligent query
routing via semantic DHT, efficient bulk transfer via semantic packfiles,
and automatic semantic commits tracking concept-level changes.
Phase 154: Semantic Refs & Incremental Transfer (~2 weeks)
Phase 155: Federation Discovery & Gossip (~1.5 weeks)
Phase 156: Semantic Query Routing (~3 weeks)
Phase 157: Semantic Packfiles (~2.5 weeks)
Phase 158: Semantic Commits & Deltas (~2 weeks)
Design rationale:
Docs:
Co-Authored-By: ChatGPT feedback on gitsema architecture
Claude-Session: https://claude.ai/code/session_013HWxjJmmohRPvkgvTBTKUC