diff --git a/README.md b/README.md index bad859d..94ea41f 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ open-index ui --brain my-brain ``` Prefer containers, or need a brain several agents share? โ†’ -**[`docs/deployment.md`](./docs/deployment.md)** (`docker compose --profile sqlite up`). +**[`docs/deployment.mdx`](./docs/deployment.mdx)** (`docker compose --profile sqlite up`). ### Commands @@ -200,7 +200,7 @@ Then `open-index index` (loads file-backed entities) and `open-index validate`. 4. **Agent write-back** โ€” a Stop hook that records learnings via `put_entity` (the "continuously improving" loop). -See [Entity Management](./entity-management.md) for guidance on cadence and decay. +See [Entity Management](./docs/guides/entity-management.mdx) for guidance on cadence and decay. ### 4. Explore @@ -244,7 +244,7 @@ building legal, marketing, support, or other specialized agents on Open Index. ## Using the brain from a cloud agent (production) -๐Ÿ“– **[Full deployment guide โ†’ `docs/deployment.md`](./docs/deployment.md)** โ€” local, +๐Ÿ“– **[Full deployment guide โ†’ `docs/deployment.mdx`](./docs/deployment.mdx)** โ€” local, remote (with and without Docker), TLS/proxying, and exactly what to paste into Claude Code, Claude Desktop, or Cursor. @@ -324,7 +324,7 @@ cluster (or `brain.db`), so give it a persistent home and a backup. Rule of thum # Controlling search -๐Ÿ“– **[Full configuration reference โ†’ `docs/configuration.md`](./docs/configuration.md)** โ€” +๐Ÿ“– **[Full configuration reference โ†’ `docs/guides/search-configuration.mdx`](./docs/guides/search-configuration.mdx)** โ€” decision tables for `storage: file | index`, SQLite vs OpenSearch, and every search knob. **Schema** (per field): data `type` (string/number/boolean/timestamp), `processing` @@ -391,7 +391,7 @@ are missing. 3. Add tests under `tests/` for behaviour changes, and run `pytest`. 4. If you touched a brain in `examples/`, run `open-index validate --brain examples/` so schemas and entities stay consistent. -5. Update the README / `entity-management.md` when you change user-facing behaviour. +5. Update the README / the docs under `docs/` when you change user-facing behaviour. 6. Open a PR describing *what* changed and *why*, and link the issue. CI runs the test suite on Python 3.10 and 3.13 and validates every brain in `examples/`. diff --git a/cookbooks.md b/cookbooks.md deleted file mode 100644 index cddb2b9..0000000 --- a/cookbooks.md +++ /dev/null @@ -1,6 +0,0 @@ -Example use-cases: - -1. Production Architecture -2. Business Process & Org Structure Mapping -3. Sales lifecycle Brain -4. Real-time infrastructure representation \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index a813f58..5ae6a88 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -25,7 +25,7 @@ x-brain-service: &brain-service # are written back on put_entity. # # The container runs as uid 10001, so this directory must be writable by - # it โ€” otherwise indexing fails on a bind mount. See docs/deployment.md. + # it โ€” otherwise indexing fails on a bind mount. See docs/deployment.mdx. - ${BRAIN_DIR:-./brain}:/brain # The local embedding model (~90MB) is downloaded on first use. fastembed # otherwise caches it under a temp directory, which a container throws away diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..577f5e8 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,35 @@ +# Open Index docs + +The documentation site, built with [Mintlify](https://mintlify.com). Content lives +in `.mdx` files; navigation and theme are configured in [`docs.json`](./docs.json). + +## Preview locally + +```bash +npm i -g mint # install the Mintlify CLI once +cd docs +mint dev # serve at http://localhost:3000 +``` + +## Check for broken links + +```bash +cd docs +mint broken-links +``` + +## Structure + +| Path | Contents | +|---|---| +| `index.mdx` | Landing page | +| `quickstart.mdx` | Install + first brain | +| `concepts.mdx` | The four primitives | +| `use-cases.mdx` | Example domains | +| `guides/` | Creating a brain, populating entities, entity management, search config | +| `agents/` | MCP context layer, connectors | +| `deployment.mdx` | Local / remote / Docker deployment | +| `reference/cli.mdx` | Every `open-index` command | + +Deploy by connecting this repo in the Mintlify dashboard (root directory `docs`); +pushes to the default branch publish automatically. diff --git a/docs/agents/connectors.mdx b/docs/agents/connectors.mdx new file mode 100644 index 0000000..a58fda1 --- /dev/null +++ b/docs/agents/connectors.mdx @@ -0,0 +1,62 @@ +--- +title: Connectors +description: Pull entities from someone else's MCP server into your brain on a schedule. +--- + +Everything in [Open Index as a context layer](/agents/mcp) is about exposing **your +brain** over MCP. A *connector* is the reverse: a script in +`/connectors/*.py` that calls **someone else's** MCP server and turns the +results into entities. + +```python +from open_index.connectors import Connector, EntitySpec + + +class LinearConnector(Connector): + name = "linear-issues" + + # Where the source MCP server lives. ${ENV} refs are resolved at run time, + # so the URL and token stay out of git. + mcp_url = "${LINEAR_MCP_URL}" + mcp_auth_headers = {"Authorization": "Bearer ${LINEAR_TOKEN}"} + + schedule = "daily" # manual | hourly | daily | weekly | 6h | 30m | 1w + + def extract_issues(self): + for item in self.paginate("list_issues", result_key="issues"): + yield EntitySpec( + doc_type="issue", + id=f"issue:{item['id']}", + name=item["title"], + fields={"status": item["state"]}, + related_to=[(f"product:{item['project']}", "belongs to product")], + ) +``` + +## Where do you get `mcp_url`? + +From whoever runs that server: + +- A hosted vendor MCP server โ€” from their docs (e.g. `https://mcp.vendor.com/mcp`). +- Another Open Index brain โ€” its `open-index serve` URL, i.e. `http://host:8080/mcp`. +- A local stdio-only MCP server โ€” **not supported here.** Connectors speak + streamable HTTP/SSE over `httpx` only. It needs an HTTP endpoint. + + +The URL must be the full endpoint path โ€” the same one an agent would use. + + +## Running connectors + +```bash +export LINEAR_MCP_URL=https://mcp.linear.app/mcp +export LINEAR_TOKEN=... + +open-index list-connectors # what's discovered, and its URL +open-index ingest linear-issues # run it now +open-index run # run everything whose schedule is due +open-index run --loop 3600 # or wire `open-index run` into cron/CI +``` + +`open-index run` tracks last-run times in a gitignored `.open_index_state.json`, so +it decides *whether* a connector is due โ€” your cron drives the clock. diff --git a/docs/agents/mcp.mdx b/docs/agents/mcp.mdx new file mode 100644 index 0000000..852b955 --- /dev/null +++ b/docs/agents/mcp.mdx @@ -0,0 +1,75 @@ +--- +title: Open Index as your agent's context layer +description: Expose a brain over MCP so agents can read domain context and keep it current. +--- + +`open-index mcp` runs an MCP server (stdio) exposing the brain to any MCP client โ€” +**read and write by default**. + +## The tools + +- The server publishes dynamic, brain-specific instructions as part of the agent + prompt so supporting hosts can navigate the domain before the first tool call. + + + + Refresh the domain guide after the index or schema changes โ€” reports doc_types, + fields, and the relationship vocabulary of *your* brain. + + + Retrieve domain context by query, optionally scoped to doc_types. + + + Fetch a single entity by its `:` id. + + + Add or update entities โ€” validated, honoring the storage policy. `put_entities` + writes a whole batch in one call with a shared `provenance` block. + + + Define a new concept from the agent side. + + + +## Read-only mode + +Use `open-index mcp --read-only` (or `open-index serve --read-only`) to opt out when +an agent should retrieve domain context but never maintain it. + +```bash +open-index mcp --brain ./my-brain # read + write (default) +open-index mcp --brain ./my-brain --read-only # retrieval only +``` + +## Local context-fetch analytics + +CLI and MCP searches, entity fetches, and navigation-guide reads are recorded in the +user's local state directory (`~/.local/state/open-index/`), outside the brain +checkout. The Analytics tab shows fetch counts by client/operation, frequently +fetched queries or entity IDs, latency, failures, zero-result searches, and recent +activity. + + +This file stays local and is never sent to Open Index's creators. + + +## Claude Code conveniences + +`open-index init` includes optional Claude Code conveniences: `.mcp.json`, a +`CLAUDE.md` describing durable editing workflows (not runtime navigation), and an +**`edit-brain` skill**. They are one client integration, not a requirement for +building legal, marketing, support, or other specialized agents on Open Index. + +## Portable agent setup skill + +`skills/setup-open-index/SKILL.md` follows the portable Agent Skills `SKILL.md` +format used by agent runtimes including OpenClaw, Hermes, and Claude Code. Give or +install this skill in the selected runtime when the agent should set up Open Index +itself. It covers installation, domain-brain initialization, generic MCP wiring, +default read/write verification, the `--read-only` opt-out, and production +guardrails. + +## Connecting a client + +For the exact config block per client (Claude Code, Claude Desktop, Cursor) and the +remote HTTP path, see [Deployment โ†’ Connecting your agent](/deployment#5-connecting-your-agent). diff --git a/docs/concepts.mdx b/docs/concepts.mdx new file mode 100644 index 0000000..5a29f16 --- /dev/null +++ b/docs/concepts.mdx @@ -0,0 +1,98 @@ +--- +title: Core concepts +description: The four primitives a brain is built from, and how entities link into a graph. +--- + +A brain is built from four primitives. Everything else in Open Index is a way to +create, store, search, or maintain them. + + + + A concept you want to track and maintain โ€” `service`, `customer`, `issue`. + + + The fields stored for a given doc_type, plus how each is searched and weighted. + + + One instance of a doc_type, stored per its schema and linked to others. + + + An optional source you extract entities from, e.g. an MCP server. + + + +## doc_type + +A **doc_type** is an empirical representation of the kind of object you want to +store in your brain. Sample doc_types by domain: + +- **infra** โ€” `service`, `datastore`, `dashboard`, `runbook`, `alert` +- **sales** โ€” `lead`, `deal`, `account` +- **lending** โ€” `loan`, `borrower`, `application` +- **personal** โ€” `goal`, `project`, `person`, `area`, `note` + +Each doc_type is one YAML file in `doc_types/`. It declares a schema, an optional +display config, and an optional relationship vocabulary. + +## doc_schema + +The **schema** lists the fields a doc_type stores. Per field you set the data +`type`, how it's searched (`syntactic`, `semantic`, or `none`), and a ranking +`boost`. See [Search configuration](/guides/search-configuration) for the full +field reference. + +## entity + +An **entity** is one instance of a doc_type. Its `id` must be `:` +(e.g. `service:checkout`). + +Every entity carries the reserved **`related_to`** field โ€” the correlation field +present on all entities. It defines the graph edges: each edge is a `target` plus a +free-text `relationship_edge_meaning`. + +```json +{ + "doc_type": "service", + "id": "service:checkout", + "name": "Checkout", + "related_to": [ + { "target": "datastore:postgres-main", "relationship_edge_meaning": "writes to" }, + { "target": "dashboard:checkout-latency", "relationship_edge_meaning": "is monitored by" } + ] +} +``` + +This is how you say "this ticket is about that service" without any graph database. +The explorer's **Map** tab renders these edges directly. + +## connector + +A **connector** is an optional ingestion script in `/connectors/*.py` that +pulls from someone else's MCP server on a schedule and turns the results into +entities. See [Connectors](/agents/connectors). + +## Where entities live: `storage: file | index` + +Each doc_type declares its source of truth, so curated and machine-generated data +don't fight over git. + + + + JSON files under `entities//` are the source of truth โ€” git-tracked + and PR-reviewable. Right for curated, human- or agent-authored entities. + `open-index index` reconciles these from disk on each run. + + + The search DB owns these entities; they are **not** written to files. Right for + connector-pulled, high-volume, or temporal data (hundreds of services, memories, + alerts) that would otherwise churn the repo. + + + + +`open-index index` reconciles **file**-backed types from disk and leaves +**index**-backed entities untouched. Match how a type is written to how it's +declared, or the next `index` run will wipe DB-written entities on a `file` type. + + +See [Search configuration](/guides/search-configuration) for the decision table. diff --git a/docs/deployment.md b/docs/deployment.mdx similarity index 68% rename from docs/deployment.md rename to docs/deployment.mdx index 6991448..9c6b9d4 100644 --- a/docs/deployment.md +++ b/docs/deployment.mdx @@ -1,4 +1,7 @@ -# Deploying a brain and connecting your agent +--- +title: Deployment +description: Deploy a brain and connect your agent โ€” local, remote, Docker, TLS, and the exact config to paste. +--- This guide answers three questions, in order: @@ -6,10 +9,11 @@ This guide answers three questions, in order: 2. How do I run it โ€” [local](#2-local-brain-stdio), [remote without Docker](#3-remote-brain-without-docker), [remote with Docker](#4-remote-brain-with-docker) 3. [How do I get the MCP details into my agent?](#5-connecting-your-agent) โ€” the part that is easy to get wrong -There is a second, unrelated use of "MCP" in open-index โ€” *connectors*, which pull -data **from** someone else's MCP server into your brain. That's [section 6](#6-the-other-direction-connectors-that-pull-from-an-mcp-server). - ---- + +There is a second, unrelated use of "MCP" in Open Index โ€” *connectors*, which pull +data **from** someone else's MCP server into your brain. That's +[Connectors](/agents/connectors). + ## 1. Which setup do I want? @@ -37,17 +41,17 @@ Two independent choices. Pick one from each column. **Rule of thumb:** local + SQLite to start. Move to remote + OpenSearch when a *second writer* appears โ€” that's the line SQLite can't cross, not entity count. -> **OpenSearch is only set up for you in the Docker path.** Running it outside -> Docker means operating a cluster yourself; open-index will happily connect to -> one ([config below](#opensearch-without-docker)), but this repo only ships a -> ready-made cluster in `docker-compose.yml`. + +**OpenSearch is only set up for you in the Docker path.** Running it outside Docker +means operating a cluster yourself; Open Index will happily connect to one +([config below](#opensearch-without-docker)), but this repo only ships a ready-made +cluster in `docker-compose.yml`. + Switching backends does **not** require editing `brain.yaml` โ€” set `OPEN_INDEX_SEARCH_BACKEND=sqlite|opensearch` in the environment and it wins over the file. That's what the compose profiles do. ---- - ## 2. Local brain (stdio) Nothing to host. The agent starts the server itself over stdio. @@ -58,12 +62,11 @@ open-index init my-brain open-index index --brain my-brain ``` -`open-index init` already writes a `.mcp.json` into the brain directory, so if -you open Claude Code **in that folder**, it connects automatically โ€” no further -setup. +`open-index init` already writes a `.mcp.json` into the brain directory, so if you +open Claude Code **in that folder**, it connects automatically โ€” no further setup. -To connect from somewhere else (a different repo, Claude Desktop, Cursor), get -the config block: +To connect from somewhere else (a different repo, Claude Desktop, Cursor), get the +config block: ```bash open-index mcp-config --brain ./my-brain @@ -80,16 +83,15 @@ open-index mcp-config --brain ./my-brain } ``` -The path is absolutized deliberately: your agent's working directory is usually -not the brain directory, and a relative `--brain .` quietly opens the wrong place -(or an empty one). See [section 5](#5-connecting-your-agent) for where this block goes. - ---- +The path is absolutized deliberately: your agent's working directory is usually not +the brain directory, and a relative `--brain .` quietly opens the wrong place (or an +empty one). See [Connecting your agent](#5-connecting-your-agent) for where this +block goes. ## 3. Remote brain, without Docker -Use this when you have a VM and don't want containers. The brain becomes an HTTP -MCP endpoint that any agent can register by URL. +Use this when you have a VM and don't want containers. The brain becomes an HTTP MCP +endpoint that any agent can register by URL. ### Install and run @@ -116,14 +118,16 @@ open-index ยท brain 'acme' ยท read+write ยท search backend: sqlite agent config: open-index mcp-config --url http://10.0.1.42:8080 --token $OPEN_INDEX_TOKEN ``` -> **`0.0.0.0` is not an address.** It's the *bind* address โ€” "listen on every -> interface". It is never what you paste into an agent. That's why the banner -> prints the reachable addresses separately. + +**`0.0.0.0` is not an address.** It's the *bind* address โ€” "listen on every +interface". It is never what you paste into an agent. That's why the banner prints +the reachable addresses separately. + ### The token is not optional -`serve` exposes `put_entity` and `create_doc_type`. Without a token, anyone who -can reach the port can rewrite your brain. Either set `OPEN_INDEX_TOKEN`, or pass +`serve` exposes `put_entity` and `create_doc_type`. Without a token, anyone who can +reach the port can rewrite your brain. Either set `OPEN_INDEX_TOKEN`, or pass `--read-only` to drop the write tools: ```bash @@ -156,9 +160,9 @@ Restart=on-failure WantedBy=multi-user.target ``` -`ExecStartPre` matters: file-backed entities live in git, not in the index. After -a `git pull` or a fresh machine, the index is empty until `open-index index` runs -and the brain answers every query with nothing. +`ExecStartPre` matters: file-backed entities live in git, not in the index. After a +`git pull` or a fresh machine, the index is empty until `open-index index` runs and +the brain answers every query with nothing. ### TLS / behind a proxy @@ -185,8 +189,8 @@ open-index serve --brain /srv/acme-brain --public-url https://brain.acme.com/mcp ### OpenSearch without Docker -Point `brain.yaml` at your existing cluster. Secrets stay as `${ENV}` refs, -resolved when the connection is opened: +Point `brain.yaml` at your existing cluster. Secrets stay as `${ENV}` refs, resolved +when the connection is opened: ```yaml search: @@ -206,8 +210,6 @@ export OPEN_INDEX_SEARCH_BACKEND=opensearch export OPEN_INDEX_OPENSEARCH_HOSTS=https://opensearch.internal:9200 ``` ---- - ## 4. Remote brain, with Docker The shortest path to a shared brain, and the only path where OpenSearch is set up @@ -218,17 +220,20 @@ cp .env.example .env # Set OPEN_INDEX_TOKEN (openssl rand -hex 32) and BRAIN_DIR (path to your brain). ``` -**SQLite** โ€” no external services: - -```bash -docker compose --profile sqlite up --build -``` - -**OpenSearch** โ€” adds a single-node cluster with a persistent volume: - -```bash -docker compose --profile opensearch up --build -``` + + + No external services: + ```bash + docker compose --profile sqlite up --build + ``` + + + Adds a single-node cluster with a persistent volume: + ```bash + docker compose --profile opensearch up --build + ``` + + Both serve `http://localhost:8080/mcp`. The only difference is `OPEN_INDEX_SEARCH_BACKEND`; your `brain.yaml` is identical either way, so you can @@ -247,29 +252,27 @@ and entities stay in your git repo. On start the entrypoint: 1. fails fast with a clear message if `/brain/brain.yaml` isn't there, 2. waits for OpenSearch to report healthy when that backend is selected, -3. runs `open-index index` so file-backed entities are loaded (skip with - `OPEN_INDEX_SKIP_INDEX=1`), +3. runs `open-index index` so file-backed entities are loaded (skip with `OPEN_INDEX_SKIP_INDEX=1`), 4. execs `open-index serve`. ### Permissions on the mounted brain directory The container runs as uid **10001** (not root), so a bind-mounted brain directory -owned by your user is not writable by it โ€” indexing fails on the first write. -Give the container ownership and keep group access for yourself: +owned by your user is not writable by it โ€” indexing fails on the first write. Give +the container ownership and keep group access for yourself: ```bash sudo chown -R 10001:"$(id -g)" /path/to/my-brain chmod -R g+rwX /path/to/my-brain ``` -You can still read and edit the files; writes from inside the container land as -uid 10001 with your group. +You can still read and edit the files; writes from inside the container land as uid +10001 with your group. ### Many brains from one process -`serve --brain ` runs one brain. For more than a handful, `--brains -` serves every brain under a directory from a single process, each at -`//mcp`: +`serve --brain ` runs one brain. For more than a handful, `--brains ` +serves every brain under a directory from a single process, each at `//mcp`: ```bash open-index serve --brains /srv/brains --port 8080 @@ -277,10 +280,10 @@ open-index serve --brains /srv/brains --port 8080 # /srv/brains/sales/ โ†’ /sales/mcp ``` -This matters because of what is *not* duplicated. A process per brain re-loads -the Python runtime and a ~250MB resident embedding model each time, so a modest -host tops out at a handful. In one process the model is loaded once and a brain -costs only its config and doc_types. +This matters because of what is *not* duplicated. A process per brain re-loads the +Python runtime and a ~250MB resident embedding model each time, so a modest host +tops out at a handful. In one process the model is loaded once and a brain costs +only its config and doc_types. Measured on the bundled example brain, SQLite-backed: @@ -293,18 +296,19 @@ That is **1.8MB of marginal cost per brain**, and 200 mount in ~4 seconds. Each brain keeps its own storage, its own read/write policy and its own token โ€” `OPEN_INDEX_TOKEN_` gates one brain (`OPEN_INDEX_TOKEN_SALES_EU` for -`sales-eu/`), and `--token` covers any without one. Nothing is shared between -brains except the process and the model. +`sales-eu/`), and `--token` covers any without one. Nothing is shared between brains +except the process and the model. -Two extras come with it: `GET /` lists every brain with its URL, entity count -and doc_types, and `GET /healthz` is an unauthenticated probe for a load -balancer. +Two extras come with it: `GET /` lists every brain with its URL, entity count and +doc_types, and `GET /healthz` is an unauthenticated probe for a load balancer. -> **Prefer SQLite here.** With OpenSearch, every brain is a separate cluster -> index and therefore a shard; the working guidance is ~20 shards per GB of -> heap, so hundreds of brains would hit that ceiling long before RAM. SQLite -> gives each brain its own file and no shard cost at all. Use OpenSearch for the -> few brains that genuinely need concurrent writers or >10k entities. + +**Prefer SQLite here.** With OpenSearch, every brain is a separate cluster index and +therefore a shard; the working guidance is ~20 shards per GB of heap, so hundreds of +brains would hit that ceiling long before RAM. SQLite gives each brain its own file +and no shard cost at all. Use OpenSearch for the few brains that genuinely need +concurrent writers or >10k entities. + ### Running one-off commands @@ -330,16 +334,14 @@ docker run -p 8080:8080 \ - **Persistence.** SQLite: `brain.db` is inside your mounted brain dir โ€” back that up. OpenSearch: the `opensearch-data` named volume. Index-backed entities - (`storage: index`) exist **only** there; they are not in git and are not - recreated by `open-index index`. Back it up or be able to re-ingest. + (`storage: index`) exist **only** there; they are not in git and are not recreated + by `open-index index`. Back it up or be able to re-ingest. - **The OpenSearch cluster here has security disabled** (`DISABLE_SECURITY_PLUGIN=true`) - and binds to loopback. That's fine for a single host where only the brain - container talks to it; enable the security plugin and set - `search.username`/`password` before putting it on a shared network. -- **Behind a proxy**, set `OPEN_INDEX_PUBLIC_URL` in `.env` so the printed - connection details are the ones agents can actually use. - ---- + and binds to loopback. That's fine for a single host where only the brain container + talks to it; enable the security plugin and set `search.username`/`password` before + putting it on a shared network. +- **Behind a proxy**, set `OPEN_INDEX_PUBLIC_URL` in `.env` so the printed connection + details are the ones agents can actually use. ## 5. Connecting your agent @@ -387,9 +389,11 @@ A remote block looks like this: } ``` -> Committing a token into `.mcp.json` puts it in git history. For a shared repo, -> prefer a per-developer user-scoped entry, or a read-only endpoint with no token -> for the committed config. + +Committing a token into `.mcp.json` puts it in git history. For a shared repo, prefer +a per-developer user-scoped entry, or a read-only endpoint with no token for the +committed config. + ### Check it works before blaming the agent @@ -403,8 +407,8 @@ curl -i -X POST https://brain.acme.com/mcp \ -d '{"jsonrpc":"2.0","id":"1","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}' ``` -Once connected, the agent should call `navigation_guidelines()` first โ€” it -reports the doc_types, fields, and relationship vocabulary of *your* brain. +Once connected, the agent should call `navigation_guidelines()` first โ€” it reports +the doc_types, fields, and relationship vocabulary of *your* brain. ### Troubleshooting @@ -418,58 +422,3 @@ reports the doc_types, fields, and relationship vocabulary of *your* brain. | Writes fail with "unknown doc_type" | Create the doc_type first (`create_doc_type`), or check `navigation_guidelines()` for existing ones. | | Local stdio server opens an empty brain | Relative `--brain .` resolved against the agent's cwd. Use an absolute path โ€” `mcp-config` emits one. | | Hangs/timeouts behind nginx | Response buffering is on. Set `proxy_buffering off`. | - ---- - -## 6. The other direction: connectors that pull *from* an MCP server - -Everything above is about exposing **your brain** over MCP. A *connector* is the -reverse: a script in `/connectors/*.py` that calls **someone else's** MCP -server and turns the results into entities. - -```python -from open_index.connectors import Connector, EntitySpec - - -class LinearConnector(Connector): - name = "linear-issues" - - # Where the source MCP server lives. ${ENV} refs are resolved at run time, - # so the URL and token stay out of git. - mcp_url = "${LINEAR_MCP_URL}" - mcp_auth_headers = {"Authorization": "Bearer ${LINEAR_TOKEN}"} - - schedule = "daily" # manual | hourly | daily | weekly | 6h | 30m | 1w - - def extract_issues(self): - for item in self.paginate("list_issues", result_key="issues"): - yield EntitySpec( - doc_type="issue", - id=f"issue:{item['id']}", - name=item["title"], - fields={"status": item["state"]}, - related_to=[(f"product:{item['project']}", "belongs to product")], - ) -``` - -**Where do you get `mcp_url`?** From whoever runs that server: - -- A hosted vendor MCP server โ€” from their docs (e.g. `https://mcp.vendor.com/mcp`). -- Another open-index brain โ€” its `open-index serve` URL, i.e. `http://host:8080/mcp`. -- A local stdio-only MCP server โ€” **not supported here.** Connectors speak - streamable HTTP/SSE over `httpx` only. It needs an HTTP endpoint. - -The URL must be the full endpoint path, the same one an agent would use. - -```bash -export LINEAR_MCP_URL=https://mcp.linear.app/mcp -export LINEAR_TOKEN=... - -open-index list-connectors # what's discovered, and its URL -open-index ingest linear-issues # run it now -open-index run # run everything whose schedule is due -open-index run --loop 3600 # or wire `open-index run` into cron/CI -``` - -`open-index run` tracks last-run times in a gitignored `.open_index_state.json`, -so it decides *whether* a connector is due โ€” your cron drives the clock. diff --git a/docs/docs.json b/docs/docs.json new file mode 100644 index 0000000..046e417 --- /dev/null +++ b/docs/docs.json @@ -0,0 +1,75 @@ +{ + "$schema": "https://mintlify.com/docs.json", + "theme": "mint", + "name": "Open Index", + "colors": { + "primary": "#7c3aed", + "light": "#a78bfa", + "dark": "#6d28d9" + }, + "favicon": "/favicon.svg", + "navigation": { + "groups": [ + { + "group": "Get Started", + "pages": [ + "index", + "quickstart", + "concepts", + "use-cases" + ] + }, + { + "group": "Build a Brain", + "pages": [ + "guides/creating-a-brain", + "guides/populating-entities", + "guides/entity-management", + "guides/search-configuration" + ] + }, + { + "group": "Agents & Connectors", + "pages": [ + "agents/mcp", + "agents/connectors" + ] + }, + { + "group": "Deploy", + "pages": [ + "deployment" + ] + }, + { + "group": "Reference", + "pages": [ + "reference/cli" + ] + } + ] + }, + "logo": { + "light": "/logo/light.svg", + "dark": "/logo/dark.svg" + }, + "navbar": { + "links": [ + { + "label": "Discord", + "href": "https://discord.gg/AQ3tusPtZn" + } + ], + "primary": { + "type": "button", + "label": "GitHub", + "href": "https://github.com/DrDroidLab/open-index" + } + }, + "footer": { + "socials": { + "github": "https://github.com/DrDroidLab/open-index", + "discord": "https://discord.gg/AQ3tusPtZn" + } + } +} diff --git a/docs/favicon.svg b/docs/favicon.svg new file mode 100644 index 0000000..e7e6a10 --- /dev/null +++ b/docs/favicon.svg @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/docs/guides/creating-a-brain.mdx b/docs/guides/creating-a-brain.mdx new file mode 100644 index 0000000..149dd4d --- /dev/null +++ b/docs/guides/creating-a-brain.mdx @@ -0,0 +1,108 @@ +--- +title: Creating a brain +description: Scaffold a brain, define doc_types, and author entities step by step. +--- + +`open-index init ` scaffolds a brain directory; then you author two kinds of +file โ€” **doc_types** (schemas) and **entities** (instances). + +``` +my-brain/ + brain.yaml # name + storage/search backend + doc_types/*.yaml # one schema per doc_type + entities/**/*.json # entities, with related_to edges + connectors/*.py # optional ingestion scripts +``` + +## 1. Define a doc_type + +A doc_type is a concept plus its schema โ€” one YAML file in `doc_types/`: + +```yaml doc_types/service.yaml +doc_type: service +description: A deployed service. +storage: file # file = git source of truth ยท index = DB-owned (default) +display: + label_field: name + color: "#7c3aed" +schema: + fields: + - { name: name, type: string, search: syntactic, boost: 6 } # weighted 6ร— in ranking + - { name: description, type: text, search: semantic } + - { name: owner, type: string, search: syntactic } +relationships: # the correlations this type uses โ€” optional but recommended + - { name: "writes to", target_doc_type: datastore } + - { name: "is monitored by", target_doc_type: dashboard } +``` + +- **`boost`** sets per-field search weight โ€” a hit in a `boost: 6` title outranks a + `boost: 1` description hit 6-to-1. Optional; defaults to 1. +- **`relationships`** declares the edge vocabulary so correlations are discoverable + (shown in the UI + navigation guide) and lightly validated (right target type). + Optional โ€” entities may still use undeclared meanings. + +Create one with `open-index add-doc-type service` (writes a stub you edit), or ask +your agent. + +## 2. Add entities + +An entity is one instance. For `storage: file` types, write one JSON per entity +under `entities//`: + +```json entities/service/checkout.json +{ + "doc_type": "service", + "id": "service:checkout", + "name": "Checkout", + "owner": "payments-team", + "related_to": [ + { "target": "datastore:postgres-main", "relationship_edge_meaning": "writes to" }, + { "target": "dashboard:checkout-latency", "relationship_edge_meaning": "is monitored by" } + ] +} +``` + +- `id` must be `:`. +- **`related_to`** is the reserved correlation field present on **every** entity โ€” it + defines the graph edges (`target` + `relationship_edge_meaning`). This is how you + say "this ticket is about that service" without any graph database. + +Then load and validate: + +```bash +open-index index # loads file-backed entities +open-index validate # validates brain.yaml, schemas, and every entity file +``` + +## 3. Explore + +`open-index ui` opens a read-only explorer. The sidebar always shows every doc_type +with its count and storage policy, so the structure is visible without navigating +anywhere. Four tabs: + + + + Search + browse + drill into an entity's relationships. + + + Auto-anchored on the most-connected entities โ€” click any node to expand it. + + + What context CLI/MCP/UI clients fetched, and how often. Zero-result searches + show what to model next. + + + Connectors and their schedules. + + + +## Next + + + + Manual, bulk import, connectors, and agent write-back. + + + Field search kinds, boosts, and semantic weighting. + + diff --git a/docs/guides/entity-management.mdx b/docs/guides/entity-management.mdx new file mode 100644 index 0000000..de57cd3 --- /dev/null +++ b/docs/guides/entity-management.mdx @@ -0,0 +1,57 @@ +--- +title: Entity management +description: How often to create entities, how many, and how to keep a brain from filling with noise. +--- + +A doc_type is the type of object you store; an entity is an actual instance of it. +Once you can create entities, the question becomes *how many* and *how often* โ€” +because a brain's quality depends as much on what you leave out as what you put in. + +## Recommendation on cadence + +Typically, create doc_types whose new instances are added at a **human pace** (once +a day, once a week) rather than a real-time pace. + + + + Deployments change at a non-real-time pace โ€” a great doc_type to track. + + + Pods are ephemeral. An entity per pod churns the brain with no lasting value. + + + +The same logic scales the `customer` doc_type: an entity per customer is a poor fit +for a B2C company, but reasonable for B2B enterprise sales where the customer count +is in the thousands. + +## Why not create too many entities + + +Over-populating a brain has two costs: + +1. **Bloat and noise** โ€” more entities means more low-signal hits diluting search. +2. **Stale-data overhead** โ€” removing entities that no longer matter gets + challenging over time. + + +## Exception: valuable temporal data + +Some temporal data is worth keeping centrally โ€” for example, **alerts and +deployments** in a brain used for AI troubleshooting. + +In those cases, model the data so it **decays exponentially with time**. That way, +flushing old data doesn't hurt the brain's quality โ€” the recent, relevant slice +always dominates. + +## How entities get created + +- Manual entry / upload from the UI. +- Talking to an agent connected to the brain via CLI / MCP. +- A webhook or API trigger from a script on your end. +- A recurring cron defined in the brain configuration. + +For each MCP server you add as a source, you define what data to extract and which +doc_type to store it in, then run it once to generate the entities โ€” optionally on a +schedule. See [Populating entities](/guides/populating-entities) and +[Connectors](/agents/connectors). diff --git a/docs/guides/populating-entities.mdx b/docs/guides/populating-entities.mdx new file mode 100644 index 0000000..c19b22f --- /dev/null +++ b/docs/guides/populating-entities.mdx @@ -0,0 +1,80 @@ +--- +title: Populating entities +description: Four ways to add entities to a brain, all landing in one validated store. +--- + +There are four ways to populate a brain, and they all write through the same +validated store that honors each doc_type's storage policy. + + + + Write JSON directly, or open Claude Code in the folder and let it call + `put_entity` / `create_doc_type` over MCP. + + + Import a file directly, or let an agent write a batch in one call with `put_entities`. + + + `connectors/*.py` pull from an MCP server on a `schedule`; run with + `open-index ingest ` or `open-index run`. + + + A Stop hook that records learnings via `put_entity` โ€” the "continuously + improving" loop. + + + +## Bulk import + +```bash +open-index import issues.csv --doc-type issue --asserted-by import:jira +open-index import export.jsonl --dry-run # validate first, write nothing +``` + +JSON arrays, JSONL, and CSV all work: + +- Bare slugs are qualified (`checkout` โ†’ `product:checkout`). +- CSV scalars are coerced to their declared types. +- A `related_to` column takes `target|meaning` pairs separated by `;`. +- A bad row is reported and skipped โ€” the rest still land. +- `--asserted-by` / `--confidence` attribute the whole batch once instead of per row. + +## Ways to create entities + + + + Create a JSON object representing the entity you want to store, and add it from + the explorer. + + + Talk to an agent connected to the brain; it validates and writes for you. + + + A script on your end pushes entities into the brain. + + + A connector defined in the brain configuration runs on a schedule. + + + +## Distilling from unstructured documents + +To extract or distill information from unstructured documents into entities: + +- **Git repo knowledge bases** โ€” connect an agent to the brain via an MCP server and + prompt it to extract information from the repo. +- **Commercial tools** (Notion, Confluence, โ€ฆ) โ€” connect an agent to both the brain + (via MCP) and the source tool (via MCP server / CLI). +- **Scripted** โ€” use a script that connects to the brain via an API and extracts + information from documents. + + +For distilled knowledge from an existing knowledge base, the recommendation is to +connect that source's MCP server to a Claude Code instance also connected to the brain. + + +## Next + +Read [Entity management](/guides/entity-management) for guidance on cadence, scale, +and decay โ€” the difference between a brain that stays sharp and one that fills with +noise. diff --git a/docs/configuration.md b/docs/guides/search-configuration.mdx similarity index 83% rename from docs/configuration.md rename to docs/guides/search-configuration.mdx index 9d5eaf3..b409241 100644 --- a/docs/configuration.md +++ b/docs/guides/search-configuration.mdx @@ -1,4 +1,7 @@ -# Configuring storage and search +--- +title: Search configuration +description: Where entities live, which engine stores them, and how search behaves. +--- Three decisions, in the order you'll hit them: @@ -9,8 +12,6 @@ Three decisions, in the order you'll hit them: The first is per **doc_type** and is the one people get wrong. The other two are per **brain**. ---- - ## 1. Where entities live: `file` vs `index` Set on each doc_type, in `doc_types/.yaml`: @@ -33,19 +34,19 @@ reviewable: services, runbooks, products, policies. Curated knowledge belongs in git. **Pick `index` when** the data is generated, high-volume, or temporal: alerts, -deployments, memories, anything a connector pulls on a schedule. Hundreds of -rows churning through git helps nobody. +deployments, memories, anything a connector pulls on a schedule. Hundreds of rows +churning through git helps nobody. -> **The trap:** `open-index index` reconciles file-backed types *from disk*. If a -> doc_type is `file` and you wrote entities to it only via the DB, the next -> `index` run deletes them. Conversely, JSON files for an `index` type are -> ignored. Match how a type is written to how it's declared. + +**The trap:** `open-index index` reconciles file-backed types *from disk*. If a +doc_type is `file` and you wrote entities to it only via the DB, the next `index` +run deletes them. Conversely, JSON files for an `index` type are ignored. Match how +a type is written to how it's declared. + -Changing the policy later is fine, but move the data with it โ€” switch to `file` -and the existing DB rows will be wiped on the next `index` unless you export them -to JSON first. - ---- +Changing the policy later is fine, but move the data with it โ€” switch to `file` and +the existing DB rows will be wiped on the next `index` unless you export them to +JSON first. ## 2. Which engine: SQLite vs OpenSearch @@ -56,9 +57,11 @@ search: backend: sqlite # or: opensearch ``` -There is no `storage.backend`. `storage:` only sets the SQLite file path. -(Older brains that set `storage.backend` still load; it never did anything, and -now logs a warning.) + +There is no `storage.backend`. `storage:` only sets the SQLite file path. (Older +brains that set `storage.backend` still load; it never did anything, and now logs a +warning.) + | | SQLite (default) | OpenSearch | |---|---|---| @@ -94,7 +97,7 @@ search: verify_certs: true ``` -Running it in containers โ†’ [`deployment.md`](./deployment.md). +Running it in containers โ†’ [Deployment](/deployment). ### Environment overrides @@ -105,10 +108,8 @@ Running it in containers โ†’ [`deployment.md`](./deployment.md). | `OPEN_INDEX_OPENSEARCH_INDEX` | `search.index` | | `OPEN_INDEX_DB_PATH` | `storage.path` | -Empty values don't override, so `docker compose` passing an unset variable -through as `""` leaves the file's value alone. - ---- +Empty values don't override, so `docker compose` passing an unset variable through +as `""` leaves the file's value alone. ## 3. Tuning search @@ -130,16 +131,16 @@ schema: | `boost` | number > 0, default 1 | Ranking weight | | `required` | `true` / `false` | Reject entities missing it | -- **`syntactic`** โ€” keyword and prefix matching. Right for names, owners, - statuses, anything you'd filter on. +- **`syntactic`** โ€” keyword and prefix matching. Right for names, owners, statuses, + anything you'd filter on. - **`semantic`** โ€” the field is embedded and matched by meaning, so "checkout is slow" finds an entity that says "latency spike at payment". Right for prose. -- **`none`** โ€” stored but never searched. Right for opaque ids and blobs that - would only add noise. +- **`none`** โ€” stored but never searched. Right for opaque ids and blobs that would + only add noise. -**`boost` is a genuine multiplier.** A match in a `boost: 6` field outranks one -in a `boost: 1` field 6-to-1 โ€” not approximately. The usual shape is a high boost -on `name`, a moderate one on a summary, and 1 everywhere else. +**`boost` is a genuine multiplier.** A match in a `boost: 6` field outranks one in a +`boost: 1` field 6-to-1 โ€” not approximately. The usual shape is a high boost on +`name`, a moderate one on a summary, and 1 everywhere else. ### Per brain @@ -150,9 +151,9 @@ search: ``` `semantic_weight` blends the two scores. The default `0.3` keeps keyword matches -dominant and lets semantic similarity rescue queries that use different words -than the text. Raise it if your users describe things rather than name them; set -`0` to turn vectors off at query time. +dominant and lets semantic similarity rescue queries that use different words than +the text. Raise it if your users describe things rather than name them; set `0` to +turn vectors off at query time. ### Embeddings @@ -172,8 +173,9 @@ export OPEN_INDEX_EMBEDDING_MODEL=text-embedding-3-small export OPEN_INDEX_EMBEDDING_DIM=1536 ``` -**Re-embed after any change that invalidates existing vectors** โ€” enabling -semantic search on a populated brain, switching models, or changing dimensions: + +**Re-embed after any change that invalidates existing vectors** โ€” enabling semantic +search on a populated brain, switching models, or changing dimensions: ```bash open-index index --reembed @@ -181,8 +183,7 @@ open-index index --reembed Changing dimensions on OpenSearch also needs the index recreated, since the `knn_vector` mapping is fixed at creation. - ---- + ## Worked examples @@ -194,8 +195,8 @@ storage: { path: ./brain.db } search: { backend: sqlite, semantic_weight: 0.3 } ``` -**Shared team brain with connectors** โ€” OpenSearch (several writers), curated -types in git, pulled types DB-only. +**Shared team brain with connectors** โ€” OpenSearch (several writers), curated types +in git, pulled types DB-only. ```yaml search: @@ -204,6 +205,7 @@ search: username: "${OPENSEARCH_USER}" password: "${OPENSEARCH_PASSWORD}" ``` + ```yaml # doc_types/runbook.yaml โ†’ curated, reviewed in PRs storage: file @@ -217,8 +219,6 @@ storage: index search: { backend: sqlite, semantic_weight: 0 } ``` ---- - ## Troubleshooting | Symptom | Cause | diff --git a/docs/index.mdx b/docs/index.mdx new file mode 100644 index 0000000..b948efb --- /dev/null +++ b/docs/index.mdx @@ -0,0 +1,60 @@ +--- +title: Open Index +description: Build domain-specific, structured context that agents can actually operate on โ€” and keep it correct as things change. +--- + +**Open Index** is a tool for building domain-specific, accurate, structured data +that agents can actually operate on โ€” and for keeping that data correct as things +change. + +You use Open Index to build a **brain**: a searchable, continuously-improving +context graph of your domain. A brain is **domain-agnostic** โ€” model a support org +(`product โ†’ "has common issue" โ†’ issue`), a sales pipeline (`customer โ†’ order`), +your infrastructure (`service โ†’ runbook`), or anything else. You define the +concepts; Open Index stores them, searches them, and draws the map. + + + + Install, spin up the bundled example brain, and open the explorer in a few minutes. + + + doc_type, doc_schema, entity, connector โ€” the four primitives a brain is built from. + + + Define doc_types, author entities, and load them into the search index. + + + Expose the brain over MCP so agents can read context and keep it current. + + + +## What a brain is built from + +A brain is built from four primitives: + +- **doc_type** โ€” a concept you want to track and maintain (e.g. `service`, `customer`, `issue`). +- **doc_schema** โ€” the fields stored for a given doc_type. +- **entity** โ€” one instance of a doc_type, stored per its schema. Every entity can + link to others via `related_to` (the target) + `relationship_edge_meaning` (free-text edge semantics). +- **connector** โ€” an optional source you extract entities from (e.g. an MCP server). + +## A context layer for domain-specialized agents + +Open Index is designed to sit behind agents specialized for a domain โ€” legal, +marketing, customer support, sales, infrastructure, or a domain of your own. The +MCP server gives those agents structured context and a validated way to keep that +context current: + +- **agent prompt** โ€” dynamic domain navigation is published through MCP server + instructions so supporting hosts can inject it before the first turn. +- **read** โ€” `navigation_guidelines()` refreshes those instructions; + `search_brain()` and `get_entity()` retrieve domain context. +- **write** โ€” `put_entity()` (add/update an entity), `create_doc_type()` (define a concept). + +Read and write is the default MCP mode, so a domain agent can both use knowledge +and maintain it. Add `--read-only` when the agent should consume context without +mutating it. + + + Ask questions, share the brains you're modelling, or discuss an idea before you build it. + diff --git a/docs/logo/dark.svg b/docs/logo/dark.svg new file mode 100644 index 0000000..8b68e2b --- /dev/null +++ b/docs/logo/dark.svg @@ -0,0 +1,10 @@ + + + + + + + + + Open Index + diff --git a/docs/logo/light.svg b/docs/logo/light.svg new file mode 100644 index 0000000..4e4518b --- /dev/null +++ b/docs/logo/light.svg @@ -0,0 +1,10 @@ + + + + + + + + + Open Index + diff --git a/docs/quickstart.mdx b/docs/quickstart.mdx new file mode 100644 index 0000000..9a8d3a4 --- /dev/null +++ b/docs/quickstart.mdx @@ -0,0 +1,90 @@ +--- +title: Quickstart +description: Install Open Index, run the bundled example brain, and open the explorer. +--- + +## Install + +```bash +pip install -e '.[all]' # core + UI (Streamlit) + MCP server +``` + +The `[all]` extra pulls in the Streamlit explorer and the MCP server. For a +narrower install, pick the extras you need โ€” `[ui]`, `[mcp]`, `[serve]`, +`[semantic]`, `[opensearch]`. + +## Try the bundled example + +The repo ships a **support brain** (products, issues, segments, comments) you can +run immediately. + +```bash +open-index index --brain examples/support-brain +open-index ui --brain examples/support-brain # open the Map tab, pick an anchor +``` + + +Three runnable examples ship in `examples/`: `support-brain`, `infra-brain`, and +`personal-brain`. Each is a complete brain you can index, search, and explore. + + +## Start your own brain + + + + ```bash + open-index init my-brain + ``` + This writes `brain.yaml`, a `doc_types/` folder, an `entities/` folder, and + optional Claude Code conveniences (`.mcp.json`, `CLAUDE.md`, an editing skill). + + + ```bash + open-index add-doc-type customer --brain my-brain + ``` + Edit the generated `doc_types/customer.yaml` to declare the fields you want to store. + + + Author one JSON per entity under `my-brain/entities/**/*.json`, or import in bulk: + ```bash + open-index import customers.csv --doc-type customer --brain my-brain + ``` + + + ```bash + open-index index --brain my-brain + open-index ui --brain my-brain + ``` + + + +## A brain on disk + +``` +my-brain/ + brain.yaml # name + storage/search backend + doc_types/*.yaml # one schema per doc_type (fields, boosts, display color) + entities/**/*.json # entities, with related_to edges + connectors/*.py # optional ingestion scripts (MCP โ†’ entities) +``` + +Storage defaults to **SQLite + FTS5** (zero external services). The backend sits +behind a pluggable interface with two implementations: SQLite (default, local/dev) +and **OpenSearch** (select with `search.backend: opensearch`). + +## Next steps + + + + Understand doc_types, entities, and the relationship graph. + + + Define schemas and author entities step by step. + + + Wire the brain into Claude Code, Claude Desktop, or Cursor over MCP. + + + Run the brain remotely for a team, cloud agents, or CI. + + diff --git a/docs/reference/cli.mdx b/docs/reference/cli.mdx new file mode 100644 index 0000000..d110faf --- /dev/null +++ b/docs/reference/cli.mdx @@ -0,0 +1,52 @@ +--- +title: CLI reference +description: Every open-index command and what it does. +--- + +All commands accept `--brain ` to point at a brain other than the current +directory. + +## Building a brain + +| Command | What it does | +|---|---| +| `open-index init [dir]` | Scaffold a new brain directory. | +| `open-index add-doc-type ` | Add a doc_type schema stub under `doc_types/`. | +| `open-index add-entity ` | Validate + store an entity JSON file. | +| `open-index import ` | Bulk-import entities from JSON / JSONL / CSV. | +| `open-index index` | (Re)load `entities/**/*.json` into the search index. | +| `open-index validate` | Validate `brain.yaml`, schemas, and every entity file (use in CI). | + + +`open-index index` accepts `--reembed` to rebuild semantic vectors โ€” run it after +enabling semantic search on a populated brain, switching embedding models, or +changing dimensions. + + +## Connectors + +| Command | What it does | +|---|---| +| `open-index ingest ` | Run a connector now to pull entities from an MCP server. | +| `open-index run [--force] [--loop N]` | Run every connector whose `schedule` is due (wire into cron/CI). | +| `open-index list-connectors` | List discovered connectors and their source URLs. | + +## Searching & exploring + +| Command | What it does | +|---|---| +| `open-index search [-t doc_type]` | Search from the terminal. | +| `open-index ui` | Launch the Streamlit explorer (Explore / **Map** / Analytics / Jobs). | + +## Serving over MCP + +| Command | What it does | +|---|---| +| `open-index mcp [--read-only]` | Run the MCP context layer over stdio. **Read+write by default**; `--read-only` opts out of writes. | +| `open-index serve [--port --token --read-only]` | Serve the MCP context layer over **HTTP** for remote agents (bearer-token auth). | +| `open-index serve --brains ` | Serve **every** brain under a directory from one process, each at `//mcp`. | +| `open-index mcp-config [--url --token]` | Print the MCP connection block to paste into your agent. | + +See [Deployment](/deployment) for the full serving guide, and +[Open Index as a context layer](/agents/mcp) for the MCP tools those commands +expose. diff --git a/docs/use-cases.mdx b/docs/use-cases.mdx new file mode 100644 index 0000000..0687688 --- /dev/null +++ b/docs/use-cases.mdx @@ -0,0 +1,28 @@ +--- +title: Use cases +description: Example domains people model as a brain. +--- + +A brain is domain-agnostic โ€” you define the concepts and the edges between them. +Some example use cases: + + + + Model services, datastores, dashboards, runbooks, and alerts, with the edges + between them, as a live map of your infrastructure. + + + Map teams, roles, ownership, and the processes that connect them. + + + Track leads, deals, and accounts through the pipeline as linked entities. + + + Keep a representation of infrastructure current with connector-pulled, + time-decaying data. + + + +The three runnable examples in `examples/` โ€” `support-brain`, `infra-brain`, and +`personal-brain` โ€” are complete brains you can index and explore to see these shapes +in practice. diff --git a/entity-management.md b/entity-management.md deleted file mode 100644 index 0b75615..0000000 --- a/entity-management.md +++ /dev/null @@ -1,48 +0,0 @@ -### Entity-management - -A doc_type is an empirical representation of the type of objects you want to store in your brain. An entity is the actual instance of that doc_type that you're storing within the brain. - -The different ways you can create entities are: -1. Through manual entry/upload from UI -Just create a simple JSON object that representing the entity you want to store in the brain. - -2. By talking to an agent that's connected to the brain via it's CLI / MCP server. - - -3. Through a webhook/API trigger from a script on your end -4. Through a recurring cron defined in the brain configuration - - -Note: If you'd like to extract/distill information from unstructured documents into entities in the brain, there are two recommended approaches: -- Github Repo based knowledge bases: Connect an agent to the brain via an MCP server and prompt it to extract information from the repo -- Commercial tools based knowledge bases (e.g. Notion, Confluence, etc.): Connect an agent to the brain via an MCP server and to the respective tool via MCP server/CLI -- Use a script that connects to the brain via an API and extracts information from documents - - - -Now, you add MCP servers to the brain. -For each of these MCP servers, you define what is the data you extract from a given MCP server and what doc_type you want to store it in. -Then you run it for the first time to generate the entities for that doc_type. You can also define each of these entities to be generated at a given cron. - -If you want to manually add entities to a doc_type, you can do so from the UI, API or using the MCP Server. Each entity to be added needs to be a valid JSON. - -Distilled knowledge: -- If you want the brain to have knowledge distilled from your existing knowledge base, the recommendation is to connect that MCP server (which has the data) to a claude code instance - - -Recommendation on entity creation: -Typically, we recommend creating doc_types that have new instances (entities) added at a human pace (e.g. once a day, once a week, etc.) rather than real-time pace. - -For example: -1. Kubernetes deployments is a great doc_type to have in the brain but creating an entity for each pod is not recommended. Deployments change at non real-time pace where as pods are ephemeral objects. -2. An entity for each customer for a B2C company would not be recommended but could be done for B2B Enterprise Sales business where the entity count for the doc_type "customer" will be in thousands. - -The reason for this is that if you create too many entities: -1. It can lead to information bloat and noise in the brain. -2. It can create overhead of removing the stale entities, which can get challenging over time. - -Exceptions: -In some cases, temporal data is valuable and could be relevant for the organisation to keep centrally in the brain: -- Alerts & deployments in a brain that's being used for AI troubleshooting. - -In such scenarios, it is recommended to have data that is exponentially decaying w.r.t time. That ways, flushing data will not hurt your brain's quality. \ No newline at end of file diff --git a/examples/cloud-brain/tools_server.py b/examples/cloud-brain/tools_server.py index 3f8c464..cd52ac3 100644 --- a/examples/cloud-brain/tools_server.py +++ b/examples/cloud-brain/tools_server.py @@ -1,7 +1,7 @@ """A REAL MCP server (JSON-RPC 2.0 over HTTP) that serves a Kubernetes-style inventory of *stable* assets โ€” namespaces, nodes, services, deployments (200 total). Ephemeral pods are deliberately NOT served: they churn and aren't -knowledge worth keeping (see entity-management.md). No SDK, no external deps. +knowledge worth keeping (see docs/guides/entity-management.mdx). No SDK, no external deps. python examples/cloud-brain/tools_server.py 9920 # then, in another shell: CLOUD_MCP_URL=http://127.0.0.1:9920 open-index ingest k8s --brain examples/cloud-brain diff --git a/open_index/ui/app.py b/open_index/ui/app.py index 8fef0c7..8c859ba 100644 --- a/open_index/ui/app.py +++ b/open_index/ui/app.py @@ -590,7 +590,7 @@ def render_jobs(brain: Brain) -> None: if not found: st.info("No connectors yet.") st.caption("Add `connectors/*.py` to pull entities from an MCP server โ€” " - "see docs/deployment.md.") + "see docs/agents/connectors.mdx.") return import inspect