Cite documents agents find via document search

[Paul Lizer (paullizer)]

Fixes #1239

The problem

Documents an agent retrieved through the document search action never became document citations.

	Standard document search	Agent document search (before)
Retrieval	route-level hybrid_search()	DocumentSearchPlugin.search_documents()
Stored as	hybrid_citations	agent_citations only
Sources disclosure	clickable document buttons	raw JSON tool modal
Enhanced citation / PDF viewer	yes	no
Eligible for cited_hybrid_citations	yes	no
Counted in Used documents	yes	no

The retrieved documents were already sitting inside the plugin result payload — nothing ever converted them into the document citation shape. functions_workflow_runner.py had the identical gap.

The fix

New functions_agent_document_citations.py derives real document citations from search_documents, retrieve_document_chunks, and summarize_document payloads, and merges them into hybrid_citations before build_cited_source_subsets() runs — so agent-discovered documents flow through the existing sources → cited references → used documents pipeline unchanged.

Wired into every path that writes an assistant message:

• document action path
• non-streaming chat
• streaming completion
• streaming cancellation (partial content)
• streaming interruption
• functions_workflow_runner._create_assistant_message()

Document search payloads now also carry a copy-ready citation marker formatted exactly as the tracker matches — (Source: Policy.pdf, Page: 3) [#doc-1_3] — plus citation_instructions, and the three kernel function descriptions tell the model to reuse it verbatim. When it does, the document is promoted into cited_hybrid_citations and the conversation's used_documents.

Sources are not cited references

Derived citations are sources and are deliberately uncapped — if an agent sources 500 chunks, all 500 are recorded. functions_citation_tracking.build_cited_source_subsets() remains solely responsible for narrowing to what the answer actually cited.

Because nothing is capped server-side, the per-message Sources disclosure now renders the first 25 entries and collapses the rest behind a Show N more sources control. No stored data is discarded. The Used documents drawer is unaffected — it lists cited documents, not source chunks.

Correctness details

• Locators: chunk keys are not always <document_id>_1. Video chunks are keyed by second and start at <document_id>_0. summarize_document_content() now reports a citation_chunk describing its real first source chunk, and summaries anchor to it instead of a synthesized locator. A valid page or sequence of 0 is preserved rather than relabelled as page 1 — this also corrects resolve_citation_location(), which is safe because every pre-existing caller already coerced 0 to 1 before calling.
• Cancelled and interrupted streams: streaming plugin invocations only reach the agent citation list on normal completion, so those two paths also read raw invocation records from the plugin logger, which is cleared per chat request. Merging is deduplicated, so supplying both sources never double-counts.
• Dedupe: by citation_id, falling back to document_id + chunk_id + page_number. Existing route-level records always win, so a chunk retrieved by both the toggle and an agent is listed once with its original metadata.
• Capability metadata: workspace.search_used / result_count now report document usage for agent-only document turns, which previously under-reported as unused.
• Access control: no new access surface. Everything derives from payloads produced by functions_search_service, which already resolves personal, group, and public scope against the current user.

Validation

functional_tests/test_agent_document_search_citations.py — 12/12 passing, covering derivation for all three plugin functions, dedupe without truncation on a 500-result set, provenance tagging, zero-indexed locator handling, raw-invocation support for cancelled streams, marker-to-cited-reference promotion, tabular sheets, JSON-string payloads, and wiring.

Test 11 is a branch-aware AST check: for every function that builds cited subsets, a merge must precede tracking and persistence on that branch, so a merge inside one finalization branch cannot vouch for another. It was confirmed to have teeth — removing the interrupted-stream merge makes it fail with the exact branch and line, and it independently caught a missing merge during development.

Regressions clean: test_chat_cited_source_tracking.py, test_agent_citations_fix.py, test_agent_citations_per_message_fix.py, test_chat_capability_usage_metadata.py, test_markdown_citation_lookup_fallback.py, test_stored_xss_chat_workspace_rendering_fix.py, test_mixed_source_chat_search_consistency.py, and the three route_tests policy suites. Remaining repo test failures are pre-existing and fail identically on the base branch (they require live Azure configuration).

Known limitation

PluginInvocationLogger is process-global and filters only by user and conversation, so a superseding request can clear invocations a prior request has not finished reading. This predates this change and affects existing agent tool citation capture identically. Making it exact requires tagging invocations with a request or run id across shared plugin infrastructure, which is better handled separately.

Docs and version

• docs/explanation/fixes/AGENT_DOCUMENT_SEARCH_CITATION_FIX.md
• Release notes entry under v0.250.219
• VERSION bumped to 0.250.219

Development was merged in and this fix renumbered from 0.250.218 to 0.250.219, since #1270 landed on 0.250.218 first.