Skip to content

Spotlight attachment content as untrusted data in agent prompts #206

Description

@nlundee

Summary

Harden the agent against indirect prompt injection via uploaded documents: delimit all attachment-derived content (extracted PDF text, and the FormSpec built from it) as untrusted data in the prompts, with an explicit instruction that instructions found inside it must not be followed.

Context

Users upload PDFs/images as supporting context for their goal ("build the attached form"). That content flows into two LLM calls:

  1. Spec extraction (spec_node): the raw document content is sent to the model to extract a structured FormSpec.
  2. The agentic loop: the extracted FormSpec — field titles, descriptions, page names, all attacker-influenceable — is injected into the loop's system prompt (context.py, "form spec" tail section).

None of this content is screened by the intent-security gate, in either mode: the intent parser deliberately sees only attachment filenames (see intent_security.md). A document containing embedded instructions ("ignore previous instructions and …") therefore reaches the model as ordinary prompt text. This was raised in review of Altinn/altinn-studio#19680 for read-only sessions, but it applies equally to write-mode sessions — the gate never saw document content in any mode.

Blast radius today is bounded by structural controls (write tools require user approval in read-only mode, file access is repo-contained, web_fetch is allowlisted to Digdir hosts), so this is defense-in-depth — but it's the cheapest layer we're missing, and the standard mitigation for exactly this vector.

Goal

Attachment-derived text is always presented to the model as clearly delimited, untrusted data — never as instruction-bearing prompt text.

In scope

  • Spec extraction prompt: wrap document content in explicit delimiters (e.g. <attachment_content>…</attachment_content>) with an instruction that it is data to extract fields from, and that any instructions inside it must be ignored
  • Loop prompt (context.py form-spec section): same treatment for the injected FormSpec content — it derives from the document and is equally untrusted
  • A test with a hostile fixture document (embedded "ignore your instructions…" text) asserting the extracted spec/loop behavior stays on-task
  • Brief note in the agents security docs (intent_security.md or adjacent) describing the layered model: intent gate (goal text, write mode) + structural tool containment + spotlighting (attachment content, both modes)

Out of scope

Acceptance criteria

  • Spec-extraction prompt delimits document content as untrusted data with an explicit do-not-follow-instructions clause
  • Loop system prompt delimits FormSpec-derived content the same way
  • Hostile-fixture test: a document with embedded injection text produces a normal FormSpec and no off-task tool calls
  • Documented in the security notes as the mitigation layer for attachment content (both modes)

Relationship to other work

Follow-up from the security review of Altinn/altinn-studio#19680 (unified agentic loop). Complements the fail-closed allow_app_changes default and the documented decision to scope the intent-security gate to write-mode goal text.

Metadata

Metadata

Assignees

Type

Projects

Status
In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions