Ontology schema and connections, graph in Find, and cold-boot resilience - #72
Merged
Conversation
Two gaps that turned out to be the same gap: nothing said what the graph was supposed to contain, so nothing could check it and nothing could search it. A relation was a single free-text verb. That is only half a relation: a graph is read from whichever node you are standing on, and an edge traversed from its target printed the verb pointing the wrong way, so `person --officer_of--> org` read, from the organisation, as though the organisation were the officer. intel/ontology_schema.py now declares every relation with both of its names plus the kinds its endpoints are expected to be, and KNOWN_RELS is derived from it rather than being a second hand-kept list beside it. That drift was already real: thirteen relations the code mints every day (`mentions`, `resolves_to`, `has_subdomain`, `indicates_threat` and friends) had fallen out of the frozenset that claimed to list them. A test now scans the backend for rel literals and fails if one is undeclared. The same module declares the properties each kind carries, seeded only from props the code demonstrably writes. Validation warns and never rejects: the registry has to be able to hold an edge an analyst invents, and routes/extract.py mints links whose verb comes out of a language model. Object and link writes answer with a `warnings` array describing what was stored; nothing gates the write. Search: until now the only way to reach a stored object was to already know its exact canonical id, because `get` is id-exact and `list_by_kind` filters one props field. /api/search/objects looks like the answer and is not, since it searches the live observation store rather than the graph. There is now an FTS5 index over each object's id, kind, property names and flattened property values, maintained from Python in upsert/assert_props/delete because a SQLite trigger would only ever see the raw JSON blob and index its braces. Deployments whose rows predate the index backfill once, lazily, on first search. Queries are reduced to quoted word tokens before they reach MATCH, so a callsign with a hyphen or a stray quote returns no rows instead of raising. Backend 2255 -> 2308.
The graph canvas printed the stored verb on every edge, so an edge expanded out of its target read backwards: from an organisation, the `officer_of` edge to a person said the organisation was the officer. Now the edge label comes from the relation's declared pair, chosen by which end the analyst expanded from, and it reads as words rather than as snake_case. The situation link list gets the same treatment, always forwards, because a situation owns its outgoing edges. The schema is fetched once per session and cached in a module, since it is static per deployment. If it never arrives the label falls back to the verb with its underscores opened up, which is also what the backend does for a relation an analyst or a model invented, so nothing here can leave the canvas unreadable. Backend baseline 2255 -> 2308, web 741.
KML was write-only and a GeoJSON the operator dropped on the console was
parsed in the browser and never reached the backend, so the one file
format an OSINT analyst is most likely to be handed could not become a
dataset. Foundry now reads GeoJSON, KML and KMZ the same way it reads a
CSV: one row per feature, the feature's own properties keeping their
names, plus the coordinate pair and the original geometry as a string
cell.
Deriving lat/lon rather than only storing the shape is what makes this
worth doing: the lat/lon sniffer already in foundry/geo.py then finds
the columns unaided, so an uploaded map file comes straight back out of
/api/foundry/datasets/{id}/geo and a binding can mint ontology objects
from it with no extra wiring. A feature with extent gets the centre of
its bounding box, which is a place to put a pin and is honest about not
being a centroid. A file that states its own lat/lon columns keeps them.
Stdlib only: json, xml.etree, zipfile. KML is matched on local element
names because the wild carries 2.2, 2.1 and no namespace at all side by
side. Shapefile and LAS/LAZ are still absent and stay absent until
something asks for them, since both need a real dependency.
Backend 2308 -> 2331.
…rces
Everything this platform ingested was a pull it initiated: a poller, a
socket it dialled, a broker it subscribed to, an operator uploading a
file. Two consequences. Nothing could push to it, and every source had
to be one somebody wrote code for.
Inbound push is one route. POST /api/ingest/{dataset_id} appends a JSON
object or array to a dataset through the SAME append + auto-sync pair an
upload uses, so a pushed row reaches the ontology through whatever
binding is already configured, obeys the same caps and shows up in the
same version history. It is also the only route with no session
dependency, since a sender has no session, so the token is the whole
gate: generated server-side, stored only as a sha256, compared with
compare_digest, never logged or echoed after the response that mints it,
and the body is capped before it is parsed rather than after. An unknown
dataset and an unarmed one answer with the identical 404, so the route
cannot be used to find out which dataset ids exist.
Connections are the other direction. An MQTT topic, a Kafka topic or a
read-only query against the operator's own database, batched into a
dataset. MQTT needed no dependency: the MQTT 3.1.1 codec already existed
for one hard-coded broker, so app/mqtt_client.py is that codec with the
broker taken out, and the AIS guard test still runs against it through
the old names, which is what makes the move provably lossless. Kafka and
SQL are optional extras, import-guarded like titiler: absent, the kind
reports itself unavailable and the app still boots keyless.
A SQL connection stores the NAME of an environment variable holding the
DSN and refuses anything that looks like a connection string. The row is
returned by the list route and lives in foundry.db, and a password there
is a leak with several copies; driver errors are scrubbed of the DSN
before they land in last_error.
The route-coverage guard earned its keep here: it failed on /api/ingest
and the honest fix was not an exemption but the missing surface, so
Foundry gained a Connections view that arms and rotates a push endpoint
and shows the curl line once.
Kafka against a real cluster and SQL against a real database are
untested; there is no broker or server on this box. Configuration,
validation, availability reporting and supervision are covered.
Backend 2331 -> 2371.
Three gaps in the analyst surface, none of which needed a new app. The human-in-the-loop approval queue was a module dict. "The agent re-proposes on its next run" only holds if the agent runs again, so an operator who left three proposals open overnight and restarted the backend came back to none, with no record that anything had been waiting. It is a SQLite table now, beside the audit log it belongs with. Expiry is enforced on read as well as on prune, because after a restart nothing has pruned yet and a proposal that aged out while the process was down must not come back looking live. Approval takes the row before dispatching, so a double click cannot run the same write-back twice. The Brief tab produced documents: self-contained HTML, and PPTX through /api/report/pptx. What it could not do was present. There is now a full-screen deck driven by the same derivation the panel renders, so the two cannot disagree, with arrow and space navigation and a landscape print stylesheet that lays out every slide rather than whichever one was on screen. No presentation library: a deck is one slide at a time and a print rule. Explorer gained the stored graph as a second source. It only ever searched the live store, which answers "what is being emitted right now" — a promoted object that the feed has since forgotten was unreachable unless you knew its exact canonical id. Its facets come from the returned kinds and its columns from the declared schema, so the properties on show are the ones the kind is known to carry. Inbox stays where it is. The plan called for promoting it to a first-class app; shell/panels.ts records it as deliberately re-homed to the title bar during the 27-to-7 consolidation, and it is live there with an unread count. A second address for one surface is the exact thing that consolidation removed, so this is a decision left standing rather than a piece of work skipped. Backend 2371 -> 2377, web 741 -> 750.
Two follow-ups on the ontology wave. Find searches the live store: what is being emitted right now, near a point. That silently excludes everything somebody already decided mattered, because a promoted object outlives the feed that produced it. An "In the graph" group now sits under the radius results, keyed off the same box and shaped like the organisation card already beside it. It renders nothing at all when the query is too short, when the graph has no match, or when the backend does not answer, so it can never push the results the operator asked for down the panel. Explorer's Export CSV and Save search were still wired to the live result set while the ontology source was showing. Export would have downloaded rows that were not on screen and Save search would have saved a live-store subscription from an ontology view; both now belong to the live source only. Web 750 -> 758.
The MQTT connection was the one part of the connections work with no evidence behind the bytes: configuration, availability and supervision were covered, and the wire codec was covered as pure functions, but nothing ever opened a connection. CONNECT, waiting for CONNACK, SUBSCRIBE, decoding a PUBLISH and answering a PINGREQ were all assumed. The broker in the test is forty lines of asyncio speaking MQTT 3.1.1 back. A public broker would have made this a network probe that fails on a machine with no egress, and a mock of our own client would have proved nothing about the encoding. It covers the happy path, a broker that refuses the connection (which must raise rather than sit in the read loop, or the runner's backoff never fires), an unreachable host, the default port, and three malformed urls. The last test runs the connection runner end to end with nothing mocked but the broker: a published message becomes a dataset row and then an ontology object through a binding. Checked by pointing it at a dead port first, where it fails. Kafka and SQL still have no equivalent; both need a client this build does not install and infrastructure this box does not have. Backend 2377 -> 2386.
The SQL connection's configuration boundary was covered — a connection string can never be stored where an environment-variable name belongs — but nothing ever opened a database, so the parts that only exist at run time were assumed: that the DSN resolves from the environment, that the query executes, that .mappings() produces the row shape the batcher wants, and that a driver error can be scrubbed of the connection string before it lands on a row the list route returns. SQLAlchemy drives SQLite, so all of that runs with no server. The test pulls from a real database into a dataset and out the other side as an ontology object through a binding, and a second one points the query at a missing table and asserts the recorded error names the table but not the DSN. Checked by pointing the query at an empty result first, where it fails. sqlalchemy joins the dev dependencies for that reason. It stays out of the runtime dependencies — a deployment that does not use SQL sources installs nothing — but without it in dev the test skips and the documented backend baseline moves by four between machines. Kafka gets no equivalent and is now recorded as such: its protocol is too large to answer from a test the way MQTT's was, and there is no broker here. Backend 2386 -> 2390.
Not the wire. Kafka's protocol is a consumer-group handshake across half a dozen request types, and a fake broker good enough to satisfy aiokafka would be likelier to encode our own misunderstanding of it than to catch one. MQTT earned an in-test broker because its protocol is four packet types; this one does not, and apps/api/CLAUDE.md now says so rather than leaving a reader to assume parity. Two things needed no broker and were assumed until now. The import guard's positive branch: every other availability assertion covers the ABSENT case, so a probe that always answered "unavailable" would have satisfied all of them. And the failure an operator will actually meet — a mistyped hostname — which has to land on the connection row and be retried rather than ending the supervisor's task. Pointing the runner at a closed port reaches both. aiokafka joins the dev dependencies for the same reason sqlalchemy did: without it the file skips and the documented baseline moves between machines. It stays out of the runtime dependencies. Backend 2390 -> 2393.
The floating badge clipped under full-surface apps and had to dodge the console's text band with route-specific offsets. On the console globe the marker now lives in the GlobeOverlays bottom-right status cluster with the other map readouts; only the 2D route, which has no status footer, keeps a floating chip.
Through the Vite proxy a not-yet-accepting backend answers 500, not a connection refusal, and the fixed 15x2s retry ceiling lost the cold-boot race (measured: accept ~35s after page load, ceiling gave up at ~29s) — the page stranded on 'config error' until a manual reload. Retry 5xx/network failures forever (they are exactly what a backend boot self-heals), bound each attempt with a 4s abort so a queued connection cannot hang the loop, and fail fast only on 4xx. Tests cover the timeout, the 30-consecutive-502 cold boot, and the 4xx fast-fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this branch does
Ontology substrate. The ontology gains a declared, searchable schema; links can be read from the end you are standing on; a map file can be read as a dataset; and operators can push data in or point the platform at their own sources (MQTT, SQL, Kafka), each proven against a real socket/engine where the environment allows.
Graph in the console. The approval queue stays, briefs are presented, and the link graph is browsable — offered directly in Find, with the live-only controls no longer acting on the graph view.
Cold-boot resilience (fixes the map panel stranding on "config error"). Through the Vite dev proxy a not-yet-accepting backend answers 500, not a refusal, and the boot config fetch's fixed 15×2 s retry ceiling lost the race against a real cold start (measured: accept ~35 s after page load, ceiling gave up at ~29 s) — the page stranded until a manual reload. The fetch now retries 5xx/network failures indefinitely with a 4 s per-attempt abort, and fails fast only on 4xx. Unit tests cover the timeout, a 30-consecutive-502 cold boot, and the 4xx fast-fail.
Predicted-motion marker. Moved from a floating badge (which clipped under full-surface apps) into the globe's bottom-right status cluster; the 2D route keeps a small floating chip.
Verification
bash scripts/verify.shALL GREEN: 2393 api tests passed + 2 skipped, 762 web tests, typecheck and lint clean.