Skip to content
BranchesTags

Repository files navigation

DocuMind

Turn a folder of messy markdown files into a structured, searchable company context graph — with typed entity relationships, temporal conflict resolution, multi-format export, an interactive graph visualization, and an MCP server that lets AI agents query it.

What It Does

Teams/Confluence/Slack → sweeper agents → .md files → DocuMind → Company context graph
                                                        ├── knowledge-base.md (TOC + organized sections)
                                                        ├── knowledge-base.pdf / html / epub / mkdocs
                                                        ├── knowledge-graph.html (interactive D3.js visualization)
                                                        ├── documind.db (search index + typed knowledge graph
                                                        │                + communities + temporal conflicts)
                                                        └── MCP server (9 tools for AI agents)

DocuMind solves the "company context graph" problem: AI agents can navigate codebases because code is structured, but company wikis, Slack threads, and Google Docs aren't. DocuMind structures them — with typed entity relationships, ontology-as-code, temporal claim tracking, cross-document graph edges, conflict detection with auto-resolution, and community clustering — so agents can do real work with company knowledge.

Setup

Prerequisites: Python 3.11+, AWS credentials configured (for Bedrock)

# Clone and install
git clone <repo-url> && cd Documind
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

If using direct Anthropic API instead of Bedrock, create a .env file:

ANTHROPIC_API_KEY=sk-ant-...
DOCUMIND_LLM_BACKEND=anthropic
DOCUMIND_MODEL=claude-sonnet-4-20250514

Default is AWS Bedrock with Claude Haiku 4.5 (cheapest option, ~$0.20 per 100 files).

Usage

1. CLI

Organize a folder of markdown files

documind organize /path/to/your/docs/ --output ./output

This runs the full 11-phase pipeline:

  • Scans all .md files (captures timestamps + source provenance from frontmatter)
  • Analyzes each with Claude — extracts entities, claims, typed relationships, and classification (cached)
  • Generates a taxonomy (category tree) or loads from YAML schema
  • Assigns every file to a section + persists entities and relationships to graph
  • Detects outdated documents (compares dates, topics, and headings within sections)
  • Builds cross-document graph edges (topic overlap, entity co-occurrence, references, authorship)
  • Detects contradicting claims — auto-supersedes older claims when dates are available
  • Consolidates into a single organized markdown with TOC + freshness/conflict warnings
  • Exports to your chosen format (PDF, HTML, EPUB, MkDocs, or interactive graph)
  • Builds a search index (FTS5 keyword + 800+ node tree with summaries)
  • Discovers document communities using the Leiden algorithm

Multi-format export

# PDF (default)
documind organize ./docs --format pdf

# Standalone HTML with sidebar navigation
documind organize ./docs --format html

# EPUB for e-readers
documind organize ./docs --format epub

# MkDocs site (ready for GitHub Pages)
documind organize ./docs --format mkdocs

# Interactive knowledge graph visualization
documind organize ./docs --format graph

Visualize the knowledge graph

# Generate standalone after a pipeline run
documind visualize --db documind.db -o output/knowledge-graph.html

Opens an interactive D3.js force-directed graph in your browser showing entities, documents, typed relationships, conflicts, and community clusters — with search, filtering, click-to-isolate, and freeze/unpin controls.

Watch mode (auto-organize on file changes)

documind watch /path/to/docs/ --debounce 3.0

File save triggers debounced incremental re-organize. Combined with --incremental, only changed files are reprocessed.

Incremental mode

documind organize /path/to/docs/ --incremental

Compares file hashes against the last run. Only new or changed files hit the LLM — 96% cost reduction on typical edit workflows.

Taxonomy-as-code

documind organize /path/to/docs/ --taxonomy-schema taxonomy.yaml

Define your own taxonomy structure, entity types, and relationship types in YAML:

version: "1.0"
sections:
  - id: architecture
    title: Architecture & Design
    children:
      - id: system-design
        title: System Design
  - id: api-reference
    title: API Reference

entity_types:
  - id: service
    description: A microservice or API
  - id: database
    description: A database system

relationship_types:
  - id: depends_on
  - id: publishes_to
  - id: owned_by

When provided, the LLM uses your custom types instead of the defaults. Falls back to built-in types if absent.

Search the knowledge base

# Fast keyword search (free, <50ms)
documind search "SQL injection" --mode quick

# Deep reasoning search (LLM tree traversal, ~$0.01, 4-8s)
documind search "How do I prevent SQL injection?" --mode deep

# Auto mode (combines both)
documind search "JWT token security best practices"

Check knowledge base stats

documind info
DocuMind v0.1.0

Documents: 166
Search chunks: 805
Cached LLM responses: 166
Taxonomy sections: 14

Entities: 345
Typed relationships: 424
Document edges: 566
  topic_overlap: 51
  heading_overlap: 35
  entity_cooccurrence: 150
  explicit_reference: 326
  supersedes: 4
Communities: 13
Conflicts (detected): 194
Conflicts (superseded): 105

Garbage collect stale documents

# Preview what would be removed (dry run)
documind gc /path/to/your/docs/ --dry-run

# Remove documents whose source files no longer exist
documind gc /path/to/your/docs/

2. MCP Server (for AI agents)

Start the server:

documind serve

Or add to your Claude Code / MCP client config:

{
  "mcpServers": {
    "documind": {
      "command": "/path/to/Documind/.venv/bin/python",
      "args": ["-m", "documind", "serve", "--db", "/path/to/documind.db"]
    }
  }
}

The MCP server exposes 9 tools:

Tool What it does Example
organize_documents(directory) Run full pipeline on a folder organize_documents("/path/to/docs")
search_knowledge_base(query) Deep reasoning search (tree traversal) search_knowledge_base("How to prevent XSS?")
quick_search(query) Fast keyword search quick_search("authentication JWT")
list_sections() Browse the knowledge base tree list_sections()
get_section_content(section_id) Read a specific section get_section_content("0016")
find_entity(query) Search entities (people, projects, tools) find_entity("PostgreSQL")
entity_context(entity_name) Get docs + typed relationships for an entity entity_context("Project Phoenix")
find_connections(a, b) Find typed relationships between two entities find_connections("auth-service", "Kafka")
knowledge_health() Freshness, conflicts, relationships dashboard knowledge_health()

An agent can now search 110 OWASP security docs without reading any of them:

Agent: search_knowledge_base("SQL injection prevention")
→ Returns synthesized answer with Java/PHP/.NET code examples,
  4 defense strategies, and citations — from tree-searching
  842 nodes across 110 documents.

Agent: find_entity("PostgreSQL")
→ Found in 5 documents, related entities: Django, Project Phoenix, Alice

Agent: entity_context("auth-service")
→ 8 documents, typed relationships:
  auth-service --depends_on--> PostgreSQL
  auth-service --publishes_to--> Kafka
  auth-service --owned_by--> Platform Team

Agent: find_connections("FastMCP", "Chat Agent")
→ Connected via: Chat Agent --uses--> MCP Server
  Shared documents: architecture-overview.md

Agent: knowledge_health()
→ Freshness: 85.6%, 194 unresolved conflicts, 105 auto-superseded,
  424 typed relationships, 345 entities, 13 communities

3. Python API (programmatic)

import asyncio
from pathlib import Path
from documind.config import Settings
from documind.pipeline.orchestrator import PipelineOrchestrator

async def main():
    settings = Settings.load()
    orchestrator = PipelineOrchestrator(settings)
    state = await orchestrator.run(
        input_dir=Path("my-docs/"),
        output_dir=Path("output/"),
    )
    print(f"Organized {state.total_documents} documents")
    print(f"Cost: ${state.total_cost_usd:.4f}")

asyncio.run(main())

Search programmatically

import asyncio
from documind.config import Settings
from documind.search.hybrid import HybridSearch, SearchMode
from documind.storage.database import Database

async def search():
    settings = Settings.load()
    async with Database("documind.db") as db:
        search = HybridSearch(settings, db)

        # Keyword search (free)
        results = await search.search("SQL injection", mode=SearchMode.QUICK)
        for r in results:
            print(r.content[:200])

        # Reasoning search (LLM tree traversal)
        results = await search.search(
            "How do I prevent SQL injection?",
            mode=SearchMode.DEEP,
        )
        print(results[0].content)  # Synthesized answer

asyncio.run(search())

Generate graph visualization programmatically

import asyncio
from pathlib import Path
from documind.storage.database import Database
from documind.exporters.graph_visualizer import generate_graph_html

async def visualize():
    async with Database("documind.db") as db:
        await generate_graph_html(db, Path("knowledge-graph.html"))

asyncio.run(visualize())

Configuration

All settings via environment variables or .env file:

Variable Default Description
DOCUMIND_LLM_BACKEND bedrock bedrock or anthropic
DOCUMIND_MODEL us.anthropic.claude-haiku-4-5-20251001-v1:0 Model ID
DOCUMIND_MAX_CONCURRENCY 5 Parallel LLM calls
DOCUMIND_DB_PATH documind.db SQLite database path
DOCUMIND_INCREMENTAL false Enable incremental mode
DOCUMIND_DEBOUNCE_SECONDS 3.0 Watch mode debounce (0.1-5.0)
DOCUMIND_TAXONOMY_SCHEMA_PATH — Path to taxonomy YAML (ontology-as-code)
DOCUMIND_EXPORT_FORMAT markdown markdown, pdf, html, epub, mkdocs, graph
ANTHROPIC_API_KEY — Required if backend is anthropic
AWS_REGION us-west-2 AWS region for Bedrock

How It Works

See HOW-IT-WORKS.md for the full pipeline documentation. See SEARCH-ARCHITECTURE.md for the tree + graph search architecture.

Pipeline phases (11 phases)

Phase 1:  SCAN        → Parse .md files, extract structure + provenance from frontmatter
Phase 2:  ANALYZE     → LLM classifies + extracts entities, claims, typed relationships (~350 tokens each, cached)
Phase 3:  TAXONOMY    → LLM creates a category tree (or load from YAML schema)
Phase 4:  CATEGORIZE  → LLM assigns files to sections + persists entities + relationships to graph
Phase 5:  FRESHNESS   → Detect outdated docs (topic overlap + LLM judgment)
Phase 6:  GRAPH       → Build cross-document edges (topic, heading, entity, reference, author)
Phase 7:  CONFLICTS   → Detect contradicting claims, auto-supersede by date
Phase 8:  CONSOLIDATE → Merge into one markdown with TOC + freshness/conflict warnings
Phase 9:  EXPORT      → PDF, HTML, EPUB, MkDocs, or interactive graph
Phase 10: INDEX       → FTS5 chunks + tree index with LLM summaries
Phase 11: COMMUNITIES → Leiden algorithm discovers document clusters from entity graph

The SQLite knowledge graph

DocuMind builds a full property graph inside SQLite — no Neo4j or external graph DB required:

entities (nodes)                     → 345 entities across 166 docs
├── entity_relationships (typed edges) → "auth-service --depends_on--> PostgreSQL"
│   with valid_from / valid_until      (temporal validity)
│   with document_id                   (provenance)
├── document_entities (node ↔ doc)   → which docs mention which entities
└── entity_aliases (fuzzy matching)  → "PG", "Postgres" → PostgreSQL

documents (second node type)
├── document_edges (6 weighted types) → topic overlap, entity co-occurrence, references, etc.
└── conflicts (with temporal resolution) → older claims auto-superseded by date

communities (Leiden clusters)        → natural document groupings

Search architecture (tree + graph hybrid)

Query: "How to prevent SQL injection?"

Tree Search (navigates taxonomy):
  Level 0: Score 7 root sections        → Web App Security (0.9)
  Level 1: Score 6 subsections           → Vulnerability Prevention (0.9)
  Level 2: Score 24 leaf nodes           → SQL Injection Prevention (1.0)
  Extract: Read top 5 nodes             → Synthesized answer with code

Graph Search (follows entity connections):
  Entity lookup: "SQL injection" → find related entities via PPR
  Graph expansion: follow edges to cross-branch documents
  Result: discovers related docs in different sections

Total tree search: 4 LLM calls, ~6K tokens, works on any tree size
Graph search: $0 (SQLite queries + Personalized PageRank, <10ms)

Source provenance (for multi-source ingestion)

Sweeper agents writing .md files can include provenance in frontmatter:

---
source_type: teams          # teams, confluence, slack, git, email
author: alice@company.com
source_url: https://teams.microsoft.com/...
extracted_at: 2026-03-07T10:00:00Z
---

DocuMind uses this for source authority scoring (Confluence > Teams > Slack) and conflict attribution.

Cost

Dataset Files Time Cost
25 files (sample) 25 ~47s ~$0.12
110 files (OWASP) 110 ~5 min ~$0.78
166 files (HUC) 166 ~9 min ~$1.07
683 files (classics) 683 ~20 min ~$2.32
Per search query — 4-8s ~$0.01

Average: ~$0.005 per document with Bedrock Haiku 4.5 (includes relationship extraction). Re-runs on unchanged files cost $0 (cached).

Scalability

Tested and verified on large document collections:

  • Small projects (25-50 files): Single LLM calls, fastest processing
  • Medium projects (100-200 files): Efficient batching where needed
  • Large projects (600+ files): Automatic batching prevents token limits

Key features:

  • Automatic batching for large sections (>50 documents) in freshness detection
  • Incremental mode reduces cost by 96% on subsequent runs (only processes changed files)
  • Watch mode with debounced auto-organize on file changes
  • Concurrent processing with configurable concurrency (DOCUMIND_MAX_CONCURRENCY)
  • LLM caching saves cost on unchanged documents

Development

Running Tests

DocuMind has 368+ tests with markers for different test types:

# Unit tests only (no LLM, fast)
pytest -m unit -v

# Mock tests (no API cost)
pytest -m "unit or mock" -v

# All tests including live LLM (costs money, opt-in)
pytest -m "unit or mock or live" -v

# Slow tests on large datasets
pytest -m slow -v

Testing on Multiple Datasets

# Fetch datasets
python scripts/datasets/fetch_all.py

# Run benchmarks
python scripts/benchmark_run.py --all

# Results saved to benchmark_results/

Available datasets:

  • sample_docs — 25 built-in test files (API docs, guides, meetings)
  • classic_books — ~50 long-form markdown books
  • owasp — ~110 security cheat sheets
  • github_readmes — 50-100 diverse README files

License

MIT

About

Documind - AI-powered markdown knowledge base

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages