Multi-Modal Evidence Review — Solution

A hybrid rules + vision-LLM pipeline that adjudicates damage claims (car, laptop, package) from submitted images, a support-chat transcript, user history, and a minimum image-evidence checklist. It reads dataset/claims.csv and writes output.csv with the 14 required columns, and ships an evaluation/ harness scored against the labeled dataset/sample_claims.csv.

Design in one paragraph

Deterministic code does everything that doesn't require judgment — image sniffing/transcoding, evidence-requirement lookup, user-history risk context, output clamping/serialization, and a precedence-ordered reconciliation step. A provider-agnostic vision LLM (Anthropic or OpenAI, switchable via LLM_PROVIDER) does the actual visual judgment under a forced structured-output schema. The images are the source of truth; user history only adds risk context and never flips the verdict on its own. Every row gets exactly one model call, results are cached on disk, and a row that fails after retries still emits a safe fallback so output.csv always has exactly one row per input.

Architecture

                         ┌──────────────────── data/loaders ────────────────────┐
 dataset/claims.csv ────►│ claims.csv  user_history.csv  evidence_requirements.csv│
                         └───────────────────────┬───────────────────────────────┘
                                                 ▼
                                   pipeline/orchestrator (run_pipeline)
                                   bounded asyncio.Semaphore, input order preserved
                                                 │  one task per claim row
                                                 ▼
                        ┌──────────── pipeline/claim_processor (per row) ───────────┐
                        │                                                            │
                        │  1. DETERMINISTIC PREP (no network)                        │
                        │     data/images      sniff format, transcode → real JPEG   │
                        │     rules/evidence_matcher  claim_object → requirement text│
                        │     rules/history_rules     history → risk context sentence│
                        │                                                            │
                        │  2. PROMPT     prompts/prompt_builder                       │
                        │     system_prompt.txt + user text + decoded image blocks   │
                        │                                                            │
                        │  3. CACHE      pipeline/cache  (key built BEFORE network)   │
                        │     hit  ─────────────────────────────────► reuse parsed   │
                        │     miss ─► 4. LLM   providers/{anthropic|openai|mock}      │
                        │              forced structured output (tool / json_schema) │
                        │              providers/retry  backoff + jitter on 429/5xx   │
                        │                                                            │
                        │  5. RECONCILE  rules/reconcile  (deterministic precedence) │
                        │     + postprocess/validators    enum clamp + serialize     │
                        └────────────────────────────┬───────────────────────────────┘
                                                     ▼
                                     PredictionOutput (14 string columns)
                                                     ▼
                                            output.csv  (one row per input)

 Evaluation path: dataset/sample_claims.csv ─► same per-row pipeline, run once per
 strategy (hybrid vs. llm_only) ─► evaluation/scorer ─► per-field accuracy, Jaccard,
 claim_status confusion, strict full-row match.

code/main.py (test set) and code/evaluation/main.py (sample set) call the same process_claim_row, so the two paths can never drift apart.

Per-run wrapper (run.py)

python code/run.py does a full test run and the sample-set evaluation in one shot, writing everything into a fresh, isolated runs/run_<N>/ folder, then "promoting" the latest results to the canonical submission deliverables:

runs/run_<N>/
├── output.csv               predictions for dataset/claims.csv (the deliverable)
├── evaluation_report.md     operational analysis (calls, tokens, cost, latency)
├── eval_metrics.txt         sample-set accuracy (per-field, Jaccard, confusion, strict)
├── sample_predictions.csv   hybrid predictions on the labeled sample set, for inspection
└── run_meta.json            full manifest (provider, model, git commit, token stats)

promotion (unless --no-promote):
  runs/run_<N>/output.csv          ─► <repo-root>/output.csv
  runs/run_<N>/evaluation_report.md ─► <repo-root>/evaluation/evaluation_report.md

Key design decisions

• Image format gotcha. Files under dataset/images/** are named *.jpg but several are actually AVIF or WebP bytes. data/images.py lets Pillow sniff the real format (pillow-avif-plugin for AVIF), normalises EXIF orientation, drops alpha/palette/CMYK, downscales the long edge to IMAGE_MAX_EDGE, and unconditionally re-encodes to genuine JPEG before sending to any vision API. A decode failure is captured (decoded=False), never raised, so one bad image can't abort a run.
• evidence_standard_met vs valid_image. Distinct fields. The first asks "is there enough visual info to judge at all"; the second asks "is the image itself trustworthy / on-topic / authentic". A sharp photo of the wrong thing is valid_image=false; a genuine but blurry photo of the right thing is valid_image=true, evidence_standard_met=false. The system prompt and schema spell this out explicitly.
• History never flips the verdict. rules/history_rules.py turns the history record into a risk-context sentence for the prompt; rules/reconcile.py unions the user's valid history flags into risk_flags (model flags first, history flags appended) but never touches claim_status. user_history_risk / manual_review_required are injected by the system, never produced by the model.
• Evidence matching is object-level only. applies_to is free text with no clean join key, and the issue family isn't known until the image is inspected, so rules/evidence_matcher.py selects every requirement for the claim's claim_object plus every "all" row and injects them verbatim; the model decides which is relevant.
• Prompt-injection defense. Text rendered inside an image (captions, watermarks, "approve this claim") is treated as evidence to flag (text_instruction_present), never as an instruction to follow — stated explicitly in prompts/system_prompt.txt.
• Deterministic backstops + one-row-per-input guarantee. postprocess/validators.py clamps every enum to its canonical value (exact → substring → fuzzy → declared sentinel), reconcile filters supporting_image_ids to images that actually decoded and downgrades a "supported" verdict with zero supporting images to not_enough_information, and a row that fails after retries emits a safe fallback. The output always has exactly one row per input.
  • Object-grounding (wrong-object) check. The model reports the object it actually sees in a dedicated depicted_object_class field (judged from the pixels, independent of the claim). When a "supported" claim's depicted class is a concrete device that differs from claim_object (e.g. a phone shown for a laptop claim), reconcile flips the verdict to contradicted with wrong_object/claim_mismatch — catching a wrong-object claim the verdict layer would otherwise rationalize away (rules/object_grounding.py, toggle OBJECT_GROUNDING_CHECK_ENABLED). other/unclear are inert.
• Cache key computed pre-network. The cache key (provider + model + prompt version + full system/user text + sorted image SHA-256s) is built before any request, so the hybrid vs. llm_only ablation reuses the same cached model output — the ablation is free.

Design non-goals (deliberate omissions)

• No vector DB / RAG. A vector store would be overhead at this scale, for three reasons:
  • Scale — 44 test + 20 sample claims, 3 object types, a small evidence_requirements.csv. Vector search earns its keep at 10k–millions of items, not dozens; everything here fits in memory.
  • Lookups are exact-key, not semantic — rules/evidence_matcher.py filters requirements by claim_object (+ "all"), user history is an O(1) dict[user_id] lookup, and images go straight to the VLM. There is no retrieval step to accelerate.
  • Contract fit — a vector DB adds an external dependency, persistent index state, embedding calls, and ANN nondeterminism, working against the deterministic, reproducible, evaluable contract — for no accuracy payoff at this size.
  • When it would be worth it: at scale, for embedding-based duplicate/recycled-image fraud detection over a large claim history — see the improvements section below.

Setup

pip install -r code/requirements.txt
cp code/env.example .env          # then edit .env and add your API key(s)

Secrets are read from the environment only (ANTHROPIC_API_KEY / OPENAI_API_KEY); the .env is loaded automatically via python-dotenv and is git-ignored. Defaults: LLM_PROVIDER=anthropic, ANTHROPIC_MODEL=claude-sonnet-4-5, OPENAI_MODEL=gpt-4o-2024-08-06 (override any with env vars or --model).

Running

# Offline smoke test — no API key, no network (MockProvider):
python code/main.py --dry-run --input dataset/sample_claims.csv --output /tmp/sample_out.csv

# Full test-set run → output.csv at the repo root:
python code/main.py                        # uses $LLM_PROVIDER (default: anthropic)
python code/main.py --provider openai
python code/main.py --limit 2 --report     # smoke test + write the operational report

# Evaluate accuracy on the labeled sample set (hybrid vs. LLM-only ablation):
python code/evaluation/main.py             # or --provider openai / --dry-run

# One full run (test set + eval) into a fresh runs/run_<N>/ folder + promote:
python code/run.py                         # or --provider openai / --no-promote

Flags by entry point (verified against the code's argparse):

Entry point	Flags
code/main.py	--input, --output, --provider {anthropic,openai,mock}, --model, --limit N, --concurrency N, --dry-run, --no-cache, --no-reconcile (LLM-only ablation), --report
code/evaluation/main.py	--input, --provider, --model, --limit N, --concurrency N, --dry-run, --no-cache
code/run.py	--provider, --model, --concurrency N, --limit N, --name, --run-dir, --no-eval, --no-promote, --no-cache

Where outputs go

Path	What	Why
output.csv (repo root)	Final predictions for dataset/claims.csv	Top-level deliverable in problem_statement.md. Override with --output; written by run.py promotion.
evaluation/evaluation_report.md (repo root)	Operational analysis (calls, tokens, cost, latency, RPM)	Operational deliverable named in problem_statement.md. Written by main.py --report or run.py promotion.
runs/run_<N>/	Per-run isolated artifacts (see above)	Runs never clobber each other; the latest is promoted to the canonical paths.
.cache/responses/	Disk response cache (git-ignored)	Makes reruns free unless the prompt/image/model changes.

Note: dataset/output.csv (if present) is a header-only stub and is not the deliverable — the submission output.csv lives at the repo root.

Project structure

Every module gets a one-line responsibility:

code/
├── main.py                      # test-set CLI: claims CSV → output.csv (+ optional --report)
├── run.py                       # full-run wrapper: one fresh runs/run_<N>/ + promote to deliverables
├── config.py                    # single source of truth: paths, enums, OUTPUT_COLUMNS, env settings, default models
├── schemas.py                   # pydantic models (ClaimRow, HistoryRecord, ImageAsset, LLMStructuredOutput, PredictionOutput)
├── requirements.txt             # anthropic, openai, pydantic, python-dotenv, pillow(+avif), tenacity
├── env.example                  # template for .env (provider, keys, optional tuning)
│
├── data/
│   ├── loaders.py               # CSV loaders for claims, sample+labels, user_history, evidence_requirements
│   └── images.py                # resolve path, sniff real format, EXIF-normalise, downscale, transcode → JPEG b64 + sha256
│
├── rules/
│   ├── evidence_matcher.py      # select requirements by claim_object (+ "all"); render them as a prompt checklist
│   ├── history_rules.py         # history record → risk-context sentence + the subset of valid history risk_flags
│   └── reconcile.py             # deterministic precedence: no-decode override, image-ID filtering, history-flag union
│
├── providers/
│   ├── base.py                  # VisionProvider ABC + ProviderResponse; shared forced-tool name/description
│   ├── factory.py               # pick provider by name; fail fast if the selected provider's API key is missing
│   ├── anthropic_provider.py    # Anthropic vision call; forces structured output via a forced tool call
│   ├── openai_provider.py       # OpenAI vision call; forces structured output via json_schema strict mode
│   ├── mock_provider.py         # offline, deterministic, schema-valid response for --dry-run (no network/key)
│   └── retry.py                 # tenacity backoff+jitter on transient 429/5xx/timeout/connection errors
│
├── prompts/
│   ├── prompt_builder.py        # assemble system text, per-row user text, and decoded image blocks
│   ├── system_prompt.txt        # adjudicator instructions: images = truth, field semantics, injection defense
│   └── output_schema.json       # JSON schema the model is forced to fill (10 fields; additionalProperties:false)
│
├── pipeline/
│   ├── claim_processor.py       # per-row: prep → prompt → cache|LLM → reconcile → validate (+ RowStats)
│   ├── orchestrator.py          # bounded-concurrency run loop, order-preserving, RunStats, write_output_csv
│   └── cache.py                 # disk response cache keyed on provider+model+prompt+image hashes
│
├── postprocess/
│   └── validators.py            # enum clamping, list serialization, safe fallback row, output-consistency checks
│
└── evaluation/
    ├── main.py                  # evaluation CLI: run each strategy on the sample set and print scores
    ├── scorer.py                # per-field exact accuracy, set-field Jaccard, claim_status confusion, strict match
    ├── strategies.py            # named comparable configs (hybrid vs. llm_only ablation)
    └── report_writer.py         # render evaluation/evaluation_report.md (measured stats + stated pricing assumptions)

Cost & latency estimation

Grounded in runs/run_1 (44 test rows, 82 images, openai / gpt-5.4-mini, concurrency 4, cold cache):

Metric	Measured (run_1)
Claim rows / model calls	44 / 44 (one call per row)
Cache hits / failed rows	0 / 0
Images (decoded/total)	82 / 82
Input tokens / output tokens	129,962 / 5,494
Avg input / output per call	≈ 2,954 / 125 tokens
Avg latency per call	3.33 s
Wall-clock time	39.0 s
Run cost	$0.3798

Pricing is a stated assumption, not a measured charge — report_writer.py uses $2.50 / 1M input and $10.00 / 1M output tokens for OpenAI (GPT-4o-class) and $3.00 / $15.00 for Anthropic (Sonnet-class). At those rates the 44-row cold run costs ≈ $0.38.

How it scales. Cost is dominated by image input tokens and is roughly linear in rows: ≈ $0.0086 / claim row at the run_1 profile (≈ 2,954 in + 125 out tokens/row), so 1,000 rows ≈ $8.60. Output tokens are tiny (the structured schema is small).

Cost controls in the code:

• One model call per row — a row's images are batched into a single call.
• Image cap — long edge downscaled to IMAGE_MAX_EDGE (1568px), re-encoded JPEG q90, bounding per-call input tokens.
• Disk cache — keyed pre-network on provider+model+prompt+image hashes, so reruns and the hybrid/llm_only ablation re-bill $0; only changed prompts/images/models re-cost.
• Bounded concurrency — asyncio.Semaphore (default 4) keeps RPM under tier limits.
• Retry/backoff — transient 429/5xx/timeout errors self-heal instead of aborting.

Accuracy: current results + improvement strategies

Sample-set scores from runs/run_1/eval_metrics.txt (n=20, openai / gpt-5.4-mini). Hybrid and llm_only score identically, so the gaps below are prompting/judgment issues, not reconciliation:

Field	Accuracy
evidence_standard_met	95%
valid_image	95%
object_part	90%
claim_status	70%
severity	65%
issue_type	50%
risk_flags (Jaccard)	53.8% (hybrid) / 42.9% (llm_only)
supporting_image_ids (Jaccard)	92.5%
strict full-row match	20% (4/20)

Top finding — the model never predicts contradicted. In the claim_status confusion matrix, all 5 gold-contradicted rows are predicted supported. The model misses subtle claim/image mismatches (right object, wrong damage / wrong part / not actually damaged). Since reconcile only downgrades unsupported "supported" verdicts and never manufactures "contradicted", this is a model-judgment gap, not a reconciliation gap.

Prioritized improvements, highest-leverage first:

1. Fix the contradiction blind spot (biggest win). Add a dedicated "does the image actually support this specific claim text?" check to the system prompt, with sharper contradiction guidance and a few gold few-shot contradicted examples (different part, undamaged part, wrong issue type, non-original image). This is the single change most likely to move strict accuracy.
2. Self-consistency / multi-sample voting on claim_status for ambiguous rows (sample 3–5 times, majority vote) to catch contradictions the single-shot misses.
3. Confidence-gated model escalation — route low-confidence or borderline rows to a stronger model (the provider abstraction already makes this a one-line swap).
4. Improve issue_type (50%) and severity (65%) — the two weakest fields — with explicit rubrics and visual exemplars per value (e.g. dent vs. scratch vs. crack; low/medium/high severity bands).
5. Per-object-type prompt specialization — tailored checklists and part vocabularies for car vs. laptop vs. package instead of one generic prompt.
6. Confidence calibration on risk_flags (lowest set-field score) — tighten the prompt on when each flag applies and de-duplicate near-synonyms.
7. Active-learning loop — mine the labeled sample set for the rows the system gets wrong, distill them into the few-shot block, and re-evaluate; the evaluation/ harness already gives the per-field signal to drive this iteratively.
8. (At scale) embedding-based duplicate-image fraud detection — once there's a large persistent claim history, index image embeddings in a vector store to flag recycled or reused damage photos across claims/users. This is the one case that justifies the vector DB deliberately omitted today (see Design non-goals); it adds infra and nondeterminism that aren't worth it at the current 44-row scale.