Preserve whitespace after inline document citations

application/single_app/config.py

@@ -96,7 +96,7 @@
 EXECUTOR_TYPE = 'thread'
 EXECUTOR_MAX_WORKERS = 30
 SESSION_TYPE = 'filesystem'
-VERSION = "0.250.228"
+VERSION = "0.250.229"
 IS_DEVELOPMENT = is_development_env_enabled()
 
 SESSION_COOKIE_SAMESITE = os.getenv('SESSION_COOKIE_SAMESITE', 'Lax')

application/single_app/static/js/chat/chat-citations.js

@@ -45,6 +45,11 @@
   const citationRegex = /\(Source:\s*((?:(?!\(Source:).)+?),\s*(Page(?:s)?|Sheet(?:s)?|Location):\s*((?:(?!\(Source:).)+?)\)\s*((?:\[#.*?\]\s*)+)/gi;
 
   let result = message.replace(citationRegex, (whole, filename, locationLabel, locations, bracketSection) => {
+    // The bracket group's trailing \s* consumes whatever whitespace followed the last
+    // [#citation-id] marker, including the blank line that separates the citation from
+    // the next markdown block. Capture it so it can be restored on the way out.
+    const trailingWhitespaceMatch = /\s*$/.exec(bracketSection);
+    const trailingWhitespace = trailingWhitespaceMatch ? trailingWhitespaceMatch[0] : '';
     const trimmedFilename = filename.trim();
     const safeFilenameText = escapeHtml(trimmedFilename);
     let filenameHtml = safeFilenameText;
@@ -129,14 +134,26 @@
     });
 
     const linkedPagesText = linkedTokens.join(', ');
-    return `(Source: ${filenameHtml}, ${escapeHtml(locationLabel)}: ${linkedPagesText})`;
+    return `(Source: ${filenameHtml}, ${escapeHtml(locationLabel)}: ${linkedPagesText})${trailingWhitespace}`;
   });
 
   // Cleanup pass: strip any remaining [#guid...] bracket groups that the main regex didn't match.
   // These appear when the model uses non-standard citation formats (e.g. "passim" instead of "Page: N").
   // Pattern matches brackets containing one or more UUID-like citation IDs (with optional _suffix parts).
-  const guidBracketRegex = /\s*\[#?[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}[^\]]*\]/gi;
-  result = result.replace(guidBracketRegex, '');
+  const guidBracketPattern = '\\[#?[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}[^\\]]*\\]';
+  const guidBracketRunPattern = `(?:${guidBracketPattern}[ \\t]*)+`;
+
+  // A bracket run that occupies a whole line is removed with its line, so the blank lines
+  // around it are not merged into the preceding paragraph.
+  result = result.replace(new RegExp(`^[ \\t]*${guidBracketRunPattern}\\r?\\n`, 'gim'), '');
+
+  // A bracket run that opens a line is removed along with the spacing that follows it, so the
+  // paragraph it introduces starts cleanly.
+  result = result.replace(new RegExp(`^[ \\t]*${guidBracketRunPattern}`, 'gim'), '');
+
+  // Anything still left is inline, so only the spaces or tabs in front of it are consumed.
+  // Consuming newlines here would collapse the paragraph break that precedes the bracket.
+  result = result.replace(new RegExp(`(?:[ \\t]*${guidBracketPattern})+`, 'gi'), '');
 
   return result;
 }

docs/explanation/fixes/CHAT_CITATION_WHITESPACE_COLLAPSE_FIX.md

@@ -0,0 +1,116 @@
+# Chat Citation Whitespace Collapse Fix
+
+Fixed in version: **0.250.229**
+
+Related issue: [#1289](https://github.com/microsoft/simplechat/issues/1289)
+
+## Issue Description
+
+Inline document citations in assistant chat messages deleted the whitespace that followed them. The citation link itself rendered correctly, but the text after the citation was jammed onto the end of the closing parenthesis instead of starting a new paragraph, list item, or sentence.
+
+In the reported message the model produced normal, well-formed markdown, but the browser rendered:
+
+- `... used to support cited answers in chat. (Source: application_workflows.md, Page: 1)Admins can configure the extraction approach ...`
+- `... warrants it (Source: document-intelligence.md, Page: 1)For best results, upload clear, readable images.`
+- `... (Source: uploading_documents.md, Page: 1)Thank you, Paul.`
+
+Because the first citation was inside a numbered list, the paragraph that followed it was also absorbed into list item 5 rather than closing the list.
+
+## Root Cause Analysis
+
+`parseCitations()` in `application/single_app/static/js/chat/chat-citations.js` matched inline citations with:
+
+```js
+const citationRegex = /\(Source:\s*(...),\s*(Page(?:s)?|Sheet(?:s)?|Location):\s*(...)\)\s*((?:\[#.*?\]\s*)+)/gi;
+```
+
+The trailing `\s*` inside the repeated bracket group `((?:\[#.*?\]\s*)+)` is greedy and matches newlines, so it consumed the whitespace that followed the last `[#citation-id]` marker. The replacement callback returned only the rebuilt `(Source: ...)` string and never re-emitted that whitespace, so the blank line separating the citation from the next block was silently deleted.
+
+`parseCitations()` runs on raw markdown **before** `marked.parse()` in `renderAiMessageContent()` (`chat-messages.js`). Losing a `\n\n` therefore did not merely remove a space — it changed how markdown parsed the remainder of the block, which is why the following paragraph became a continuation of the preceding list item.
+
+Reproduced in isolation against the production regex:
+
+```text
+Input : "... in chat. (Source: application_workflows.md, Page: 1) [#181b54f7-..._1]\n\nAdmins can configure ..."
+Output: "... in chat. (Source: application_workflows.md, Page: 1)Admins can configure ..."
+```
+
+### Secondary instance of the same defect
+
+The cleanup pass that strips leftover `[#guid]` brackets (used when the model emits a non-standard citation format) had the same class of bug in the opposite direction:
+
+```js
+const guidBracketRegex = /\s*\[#?[0-9a-f]{8}-...[^\]]*\]/gi;
+```
+
+Its leading `\s*` also matched newlines, so a stray bracket that opened a paragraph took the preceding blank line with it (`text.\n\n[#guid] More` became `text. More`).
+
+## Technical Details
+
+### Files Modified
+
+- `application/single_app/static/js/chat/chat-citations.js`
+- `application/single_app/config.py`
+- `functional_tests/test_chat_citation_whitespace_preservation.py`
+- `ui_tests/test_chat_citation_paragraph_spacing.py`
+
+### Code Changes Summary
+
+- `parseCitations()` now captures the trailing whitespace from the matched bracket group and appends it to the rebuilt citation string. Whitespace is restored exactly as the model emitted it, so no spacing is invented and a citation followed immediately by punctuation renders byte-identically to before.
+- The leftover `[#guid]` cleanup pass was split into three ordered passes so line structure survives:
+  1. A bracket run that occupies a whole line is removed together with its line.
+  2. A bracket run that opens a line is removed along with the spacing that follows it, so the paragraph it introduces starts cleanly.
+  3. Any remaining inline bracket run consumes only the spaces or tabs in front of it, never newlines.
+- Both passes now also handle consecutive bracket runs such as `[#id-a] [#id-b]` as a unit.
+- Updated `config.py` to version `0.250.229` for this fix.
+
+The emitted citation HTML is unchanged. These are whitespace-only changes, so there is no change to escaping, sanitization, or the XSS surface.
+
+## Validation
+
+### Test Results
+
+`functional_tests/test_chat_citation_whitespace_preservation.py` executes the real `parseCitations()` in a Node sandbox and then renders the result with the vendored `marked` bundle, so it asserts the user-visible block structure rather than just the regex output.
+
+| Check | Result |
+| --- | --- |
+| Reported message keeps its paragraph breaks | Pass |
+| Inline citation spacing preserved (same-line text, trailing punctuation, back-to-back citations, citation id on the next line) | Pass |
+| Stray `[#guid]` cleanup keeps line structure | Pass |
+| Source guards for the whitespace restoration mechanism | Pass |
+
+All four checks fail against the pre-fix source and pass after it, so the test is a genuine regression guard.
+
+`ui_tests/test_chat_citation_paragraph_spacing.py` seeds the reported assistant message into the chat page and asserts the rendered DOM: the numbered list item ends at its citation, the following text renders as its own `<p>`, and no sentence collides with a citation's closing parenthesis.
+
+### Before / After
+
+Rendered HTML for the reported message, before:
+
+```html
+<ol start="5">
+<li><strong>Grounded chat:</strong> ... in chat. (Source: application_workflows.md, Page: <a ...>1</a>)Admins can configure the extraction approach ... The available modes are:</li>
+</ol>
+```
+
+After:
+
+```html
+<ol start="5">
+<li><strong>Grounded chat:</strong> ... in chat. (Source: application_workflows.md, Page: <a ...>1</a>)</li>
+</ol>
+<p>Admins can configure the extraction approach for images and PDFs under <strong>Admin Settings &gt; Search &amp; Extract</strong>. The available modes are:</p>
+```
+
+### User Experience Improvements
+
+- Paragraphs, bullets, and numbered lists after a citation render in the structure the model actually produced.
+- A citation followed by more text on the same line keeps its separating space.
+- Back-to-back citations no longer collide.
+- Copied and exported message markdown is built from the same parsed output, so it keeps its line breaks too.
+
+## Cross-References
+
+- Functional test: `functional_tests/test_chat_citation_whitespace_preservation.py`
+- UI test: `ui_tests/test_chat_citation_paragraph_spacing.py`
+- Related: [Citation Improvements](../features/v0.238.024/CITATION_IMPROVEMENTS.md)

docs/explanation/fixes/index.md

@@ -31,3 +31,4 @@ category: Version History
 - [Cosmos Container Throughput Deployer Fix](COSMOS_CONTAINER_THROUGHPUT_DEPLOYER_FIX.md)
 - [Workflow Task Document Picker Fix](WORKFLOW_TASK_DOCUMENT_PICKER_FIX.md)
 - [Workflow File Sync Prompt Context Fix](WORKFLOW_FILE_SYNC_PROMPT_CONTEXT_FIX.md)
+- [Chat Citation Whitespace Collapse Fix](CHAT_CITATION_WHITESPACE_COLLAPSE_FIX.md)

docs/explanation/release_notes.md

@@ -2,6 +2,17 @@
 
 For feature-focused and fix-focused drill-downs by version, see [Features by Version](/explanation/features/) and [Fixes by Version](/explanation/fixes/).
 
+### **(v0.250.229)**
+
+#### Bug Fixes
+
+*   **Citations No Longer Eat the Line Break After Them**
+    *   Text that came after an inline document citation was jammed onto the end of the closing parenthesis instead of starting a new paragraph — you would see `(Source: uploading_documents.md, Page: 1)Thank you, Paul.` with no break at all.
+    *   The citation parser matched the `[#citation-id]` marker along with the whitespace that followed it, then rebuilt the citation without putting that whitespace back. Because this runs on the raw markdown before it is rendered, a deleted blank line did not just remove a space — it changed how the rest of the block was read, so a paragraph after a cited list item got absorbed into the list item itself.
+    *   Spacing is now restored exactly as the model wrote it. Paragraphs, bullets, and numbered lists after a citation render in their intended structure, a citation followed by more text on the same line keeps its space, and back-to-back citations stop colliding. Copied and exported message text keeps its line breaks for the same reason.
+    *   The cleanup pass for leftover citation markers had the same flaw in reverse and could swallow the blank line *before* a stray marker. It now only removes horizontal spacing, or the marker's whole line when it sits on one.
+    *   (Ref: #1289, `chat-citations.js`, `parseCitations()`, chat message rendering)
+
 ### **(v0.250.228)**
 
 #### Bug Fixes