Skip to content

offline profile 6: narratives and collective statements can run on a local Ollama model, with provenance shown on the report - #2945

Open
colinmegill wants to merge 3 commits into
edgefrom
offline-6-ollama-narrative-statement
Open

colinmegill wants to merge 3 commits into
edgefrom
offline-6-ollama-narrative-statement

Conversation

@colinmegill

@colinmegill colinmegill commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Item 13 of the offline profile. With LLM_PROVIDER=ollama, the narrative report and the collective statement now run on the local Ollama model, the same switch and settings that topic naming already uses. The report says when a local model wrote a section or a statement.

The provider switch

Narrative report (delphi, 801_narrative_report_batch.py)

  • With LLM_PROVIDER=ollama, 801 builds the same requests it would batch (same prompts, same section order, max_tokens 8000). It sends them to OLLAMA_HOST/OLLAMA_MODEL as non-streaming chat calls in JSON format.
  • Each section is stored as it completes, so completion order is kept, under the section name 803 would derive from the batch custom_id (e.g. batch_re_global_groups, which the report looks up). The row shape is the one 803 writes.
  • No batch is submitted and no AWAITING_NARRATIVE_BATCH checker row is created. The job completes when 801 exits.
  • Bounded.
    • OLLAMA_NARRATIVE_CONCURRENCY sets how many requests run at once (default 1).
    • OLLAMA_REQUEST_TIMEOUT_SECONDS limits one request (default 600).
    • OLLAMA_NARRATIVE_JOB_TIMEOUT_SECONDS is the job's budget (default 3000, under the poller's 3600). Once it is spent, the remaining sections are not sent.
    • OLLAMA_NUM_CTX sets the context window (default 16384). Ollama's own default window would silently cut a prompt that carries 100 comments.
    • Compose forwards all four.
  • Truthful failure. If a section times out, gets an HTTP error, comes back malformed (not a JSON object with paragraphs) or fails to write, 801 exits 1. The poller then marks the job FAILED, and 801 writes error_message on the row ("k of n sections failed …").
    • Some sections succeeded: each failed section is stored as the existing error-shaped JSON, so the report shows the failure rather than an empty slot.
    • Every section failed: 801 writes no rows at all. The report shows the newest job that has rows, so an earlier good run stays visible instead of being replaced by a run made only of errors.
  • The poller passes OLLAMA_MODEL to 801 for ollama, instead of requiring ANTHROPIC_MODEL.
  • OllamaProvider now binds an ollama.Client(host=…) to its endpoint. The old ollama.client.api_base = … assignment raised AttributeError for every non-default endpoint whenever the ollama package was installed, and it is installed in the image. As a result, LLM_PROVIDER=ollama topic naming could not reach http://ollama:11434 and fell back to keyword labels.

Collective statement (server, collectiveStatement.ts)

  • With LLM_PROVIDER=ollama, the route sends the same system and user prompt to OLLAMA_HOST/OLLAMA_MODEL. It uses the same JSON parse and the same fallback shape downstream.
  • A failed call answers 500 and stores nothing.
  • The local request timeout is capped at 240 s, below nginx's 300 s proxy_read_timeout. A slow local model therefore returns that clean 500, rather than a 504 followed by a stored statement. The cap is documented in docs/configuration.md.
  • New config keys: llmProvider, ollamaHost, ollamaModel, ollamaNumCtx and ollamaRequestTimeoutSeconds. The server reads them from its env file.
  • When the statement fires is unchanged: this PR only adds a provider.

Provenance

  • Narrative sections. Each Delphi_NarrativeReports item written by the local path carries provider: "ollama" and the model. They are top-level attributes, and they are also inside the item's existing metadata map, which GET /delphi/reports already returns. The job's Delphi_JobQueue item gets provider and model attributes.
  • Statements. Each Delphi_CollectiveStatement item records provider and model. The POST response also returns created_at, model and provider; the report already read those fields.
  • Shown. A small LocalModelNote renders "Generated by a local model (); quality differs from the hosted model." when provider is ollama. It appears in six places:
    • under a statement: topic page, statement modal, statements report;
    • under a narrative section: comments report, topic report, topic page.

Unchanged with LLM_PROVIDER=anthropic

No behaviour change with LLM_PROVIDER=anthropic.

  • The narrative still goes through the Anthropic Batch API and the 803 checker. The batch custom_id code moved into a helper whose output is identical, and a test pins it.
  • The statement still uses claude-opus-4-8 with the same parameters.

The only differences on the hosted path are additive fields: the statement item and the POST response now also name the provider, and the POST response also returns created_at and model.

No schema change (provenance stored in the existing JSON payloads — confirm no new column; if a column would be needed, store it inside the existing item/JSON and say so). Confirmed: no Postgres table or column is touched. Provenance lives as attributes of the existing DynamoDB items (schemaless) and inside the narrative item's existing metadata map. No new table or index. No Lambda.

Tests

No live model calls; all fixtures are generated.

  • delphi (tests/test_narrative_ollama.py, 25 tests). A fake Ollama HTTP server runs in-process. The cases cover:

    • the happy path: section names, row shape, provider recorded on sections and job, no checker row, same order, one request in flight;
    • the concurrency knob;
    • malformed output, request timeout, the job budget and a failed write, each ending in a failed job with exit 1;
    • a run in which every section fails, which writes no rows;
    • the strict call's HTTP and empty-reply errors;
    • dispatch in main, and the poller's model choice;
    • an unchanged batch custom_id;
    • the topic-naming client reaching a non-default endpoint.

    tests/test_compose_math_env.py (LLM keys) and tests/topic_naming pass.

  • server (__tests__/unit/collectiveStatementProvider.test.ts, 9 tests). They cover:

    • provider resolution;
    • the 240 s timeout cap;
    • the Ollama call's body and its errors;
    • the hosted path, unchanged and recorded;
    • the Ollama path's stored provenance;
    • the malformed-output fallback;
    • a failed call storing nothing.

    tsc --noEmit is clean.

  • client-report. LocalModelNote.test.jsx passes, including a case that renders the note as a span inside an existing paragraph (the topic page does this, because a <p> cannot hold a <p>). The existing CommentsReport suite also passes.

Follow-ups

  • F2. The older server-side narrative has no Ollama option. server/src/routes/reportNarrative.ts, behind /api/v3/reportNarrative, uses Anthropic, Gemini or OpenAI.
  • F5. The source census is not regenerated. docs/configuration-env-reference.md does not list the new keys yet: the server's LLM_PROVIDER and OLLAMA_*, and Delphi's OLLAMA_NARRATIVE_*, OLLAMA_NUM_CTX and OLLAMA_REQUEST_TIMEOUT_SECONDS.
  • F6. One setting switches all three uses. LLM_PROVIDER=ollama switches topic names, narratives and statements together. A per-use override (e.g. keep statements hosted) is left for a later PR.

Follow-ups for the offline overlay (#2938)

  • docs/offline.md is not on edge yet, so its checklist is not edited here. Once offline profile 4: the offline compose overlay, the image bundle script and the install guide (stacked on #2929, #2937, #2936) #2938 lands, item 7 should read: "Topic tier: the admin starts a Delphi run. The local model names the topics, or keyword labels do if the model is absent. The narrative report and the collective statement come from the local model, and the report marks them 'Generated by a local model ()'." Drop the "Narrative report and collective statement" line from its known gaps.
  • The server reads OLLAMA_HOST from its env file. example.offline.env sets LLM_PROVIDER and OLLAMA_MODEL but not OLLAMA_HOST, so the server would default to localhost:11434. It needs OLLAMA_HOST=http://ollama:11434.

🤖 Generated with Claude Code

https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s

colinmegill and others added 2 commits October 3, 2026 04:32
With LLM_PROVIDER=ollama, 801 sends the same prompts it would batch, in
the same section order, to OLLAMA_HOST/OLLAMA_MODEL and stores each
section as it completes, under the section names 803 would use. No batch
is submitted and no checker job is created: the job completes when 801
exits, and a failed, timed-out or malformed section fails the job (exit
1, error_message on the row) with an error placeholder in that section.

Provenance: each NarrativeReports item carries provider "ollama" and the
model, both at the top level and inside its existing metadata map (which
the reports route already returns); the job row records provider and
model. No new table or column.

Bounded: one request at a time by default (OLLAMA_NARRATIVE_CONCURRENCY),
a per-request timeout (OLLAMA_REQUEST_TIMEOUT_SECONDS, 600), a per-job
budget (OLLAMA_NARRATIVE_JOB_TIMEOUT_SECONDS, 3000) and a context window
(OLLAMA_NUM_CTX, 16384). Compose forwards all four.

The poller passes OLLAMA_MODEL to 801 for ollama instead of requiring
ANTHROPIC_MODEL. The Ollama client is now bound to its endpoint with
ollama.Client(host=...): the old module-attribute assignment raised
AttributeError for any non-default endpoint whenever the ollama package
was installed, which it is in the image.

LLM_PROVIDER=anthropic is unchanged: the batch custom_id code moved into
a helper with the same output, and the batch and 803 paths are untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…a model, and the report says when a local model wrote a section

collectiveStatement.ts gains the provider switch Delphi uses: with
LLM_PROVIDER=ollama it posts the same system and user prompt to
OLLAMA_HOST/OLLAMA_MODEL (non-streaming, JSON format, bounded by
OLLAMA_REQUEST_TIMEOUT_SECONDS, context OLLAMA_NUM_CTX) and parses the
reply through the existing JSON path and fallback. A failed call answers
500 and stores nothing. Any other value keeps claude-opus-4-8 with the
same parameters.

Each stored statement records provider and model as attributes of its
existing Delphi_CollectiveStatement item (no new table or column), and
the POST response also returns created_at, model and provider, which the
report already reads. When the statement fires is unchanged.

client-report: a LocalModelNote shows "Generated by a local model
(<model>); quality differs from the hosted model." under a statement
(topic page, statement modal, statements report) and under a narrative
section (comments report, topic report, topic page) whose provider is
"ollama". Hosted output renders as before.

docs/configuration.md: LLM_PROVIDER=ollama now covers topic names,
narratives and statements; models, the four knobs and the quality note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
colinmegill added a commit that referenced this pull request Oct 3, 2026
…ocal narratives

example.offline.env sets OLLAMA_HOST=http://ollama:11434: the server
reads it for the narrative and collective statement and would otherwise
use localhost. docs/offline.md takes the lines the data export PR
(#2940) and the Ollama narrative PR (#2945) left for it: checklist item 6
adds the conversation data export zip, item 7 has the narrative and
collective statement from the local model, and the known gaps drop the
Anthropic-only line and describe the streamed export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… nothing, statement timeout under nginx's limit

- client-report: LocalModelNote takes as="span"; the topic page renders
  it that way inside the statement's metadata paragraph (a <p> cannot
  contain a <p>).
- delphi: error placeholders are written only after at least one section
  was stored. A local run in which every section fails writes no rows and
  fails the job, so the report (which shows the newest job that has rows)
  keeps showing the previous good run.
- server: the local statement request timeout is capped at 240 s, below
  nginx's 300 s proxy_read_timeout, so a slow local model returns a clean
  500 and stores nothing instead of a 504 followed by a stored statement.
  Documented in docs/configuration.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 0 100%
components/init.py 1 0 100%
components/config.py 165 70 58%
conversation/init.py 2 0 100%
conversation/conversation.py 1290 264 80%
conversation/group_vote_counts.py 38 2 95%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 178 55%
database/postgres.py 399 101 75%
empty_output.py 29 2 93%
engine_adapter.py 316 264 16%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 38 85%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/group_k_smoother.py 28 0 100%
pca_kmeans_rep/legacy_kmeans.py 197 7 96%
pca_kmeans_rep/pca.py 162 19 88%
pca_kmeans_rep/repness.py 226 5 98%
poller/init.py 4 0 100%
poller/admission.py 318 15 95%
poller/backfill.py 936 32 97%
poller/capacity.py 444 17 96%
poller/capacity_manifest.py 260 13 95%
poller/coordinator_bridge.py 316 316 0%
poller/large_class.py 149 2 99%
poller/math_writer.py 113 3 97%
poller/promotion.py 127 2 98%
poller/readiness.py 368 12 97%
poller/service.py 641 39 94%
poller/worker_pool.py 140 0 100%
queue/executor.py 223 111 50%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 891 640 28%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 130 85 35%
replay/init.py 2 0 100%
replay/attribution_capture.py 49 0 100%
replay/axis_continuity.py 347 24 93%
replay/certify.py 999 59 94%
replay/crosslang.py 128 2 98%
replay/decision_ties.py 55 8 85%
replay/diagnostics.py 50 7 86%
replay/driver.py 73 1 99%
replay/dsn_admission.py 36 1 97%
replay/equiv_query.py 10 0 100%
replay/event_ingress.py 95 1 99%
replay/fixture_bundle.py 815 62 92%
replay/fixture_config.py 266 11 96%
replay/fixture_extract.py 283 12 96%
replay/fixture_generate.py 103 1 99%
replay/fixture_samples.py 157 14 91%
replay/fixture_selection.py 119 0 100%
replay/fixture_survey.py 147 11 93%
replay/legacy_pca.py 29 17 41%
replay/polarity.py 319 25 92%
replay/poller_equiv.py 777 99 87%
replay/prodclone.py 193 2 99%
replay/real_data.py 146 10 93%
replay/schedule.py 205 10 95%
replay/served_math.py 159 4 97%
replay/shard_bench.py 189 65 66%
replay/stagecompare.py 890 161 82%
replay/stages.py 256 40 84%
replay/stepcompare.py 86 1 99%
replay/store.py 120 11 91%
replay/tie_capture.py 107 19 82%
replay/types.py 71 6 92%
run_math_pipeline.py 264 114 57%
types.py 22 0 100%
umap_narrative/501_calculate_comment_extremity.py 112 53 53%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 434 275 37%
umap_narrative/801_narrative_report_batch.py 856 682 20%
umap_narrative/803_check_batch_status.py 185 185 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 265 265 0%
umap_narrative/narrative_local.py 114 114 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 331 289 13%
umap_narrative/polismath_commentgraph/utils/storage.py 584 470 20%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 374 230 38%
umap_narrative/topic_naming.py 148 148 0%
utils/clj_hash.py 44 0 100%
utils/env_flags.py 13 0 100%
utils/general.py 63 39 38%
utils/output_profile.py 66 2 97%
utils/serialization.py 10 5 50%
utils/vote_convention.py 29 2 93%
Total 23135 8309 64%

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant