Skip to content

fix(mem_os): strip only anchored markdown fences in clean_json_response (#2420) - #2639

Merged
endxxxx merged 3 commits into
dev-v2.0.37from
codex/maintainer-edit-pr-2590-20261119
Nov 20, 2026
Merged

endxxxx merged 3 commits into
dev-v2.0.37from
codex/maintainer-edit-pr-2590-20261119

Conversation

@syzsunshine219

Copy link
Copy Markdown
Collaborator

The source branch for the change proposed in #2590 now lives in MemTensor/MemOS, so repository writers can contribute fixes through the normal upstream branch permissions.

The code and commit authors are unchanged. This PR retains target branch main and exact source head 41269f335d7c08a13cc1242eff662ff688203434 from #2590.

Previous discussion and review remain in #2590. Review and checks must be re-evaluated for this replacement PR; no previous approval or new test result is asserted by this migration.

Previous PR description and implementation details

Restores the change proposed in #2489 for review against main after its former base branch, dev-v2.0.36, was deleted. The source commits and authors are preserved.

Source head: 41269f335d7c08a13cc1242eff662ff688203434

This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2489.


Original change description

Summary

clean_json_response used a global response.replace("```json", "").replace("```", ""), which removed every triple-backtick sequence — including ones inside JSON string values. Any LLM response whose payload contained a code snippet came out silently corrupted before json.loads.

Root cause

src/memos/mem_os/utils/format_utils.py:1416 (pre-fix):

return response.replace("```json", "").replace("```", "").strip()

Reproducer from the issue:

clean_json_response("```json\n{\"text\": \"```py\\ncode```\"}\n```")
# before:  "{\"text\": \"py\\ncode\"}"      ← fences stripped from the value
# after:   "{\"text\": \"```py\\ncode```\"}"

Both api/handlers/chat_handler.py and api/handlers/suggestion_handler.py pipe responses through this function, so the bug surface was the entire chat-metadata and suggestion pipeline.

Fix

Replace the global string replace with two anchored regex substitutions that only strip the outer fences:

  • Opening: re.compile(r"^```[^\n`]*\n?") — covers the optional info string (json or otherwise) and trailing newline
  • Closing: re.compile(r"\n?```\s*$") — at the very end only

Interior ``` sequences inside a JSON string value are preserved verbatim. As a side effect the function now correctly handles responses that have only an opening or only a closing fence (the original implementation would leave those stranded mid-string).

Tests

Added four cases to tests/mem_os/test_format_utils.py:

  1. test_clean_json_response_preserves_internal_fenced_code — the issue reproducer (red against unpatched main)
  2. test_clean_json_response_preserves_unclosed_fence_in_value — a value containing a single unclosed fence
  3. test_clean_json_response_handles_only_opening_fence — best-effort handling when the closing fence is missing
  4. test_clean_json_response_handles_only_closing_fence — best-effort handling when the opening fence is missing

Existing tests (single backticks in JSON content, plain JSON, complex JSON, etc.) all continue to pass — test_clean_json_response_preserves_internal_backticks was a good seed for the new tests.

Verification

PYTHONPATH=src pytest tests/mem_os/test_format_utils.py -v
============================== 12 passed in 3.09s ==============================

Ruff: ruff format --check and ruff check both clean on touched files (the 2 pre-existing findings were auto-fixed by ruff check --fix during this PR — both were E501 line-too-long in clean_json_response itself, not introduced by this change).

Out of scope

  • No new dependencies
  • No public API / OpenAPI contract changes
  • No DB schema, migration, or graph model changes

Closes #2420

RerankerGuo and others added 2 commits November 4, 2026 21:58
…se (#2420)

clean_json_response previously used a global
`response.replace("```json", "").replace("```", "")`, which removed
every triple-backtick sequence in the response — including ones that lived
inside JSON string values. Any LLM response whose payload contained a code
snippet (a fenced Python block, an unclosed partial fence, etc.) came out
silently corrupted before json.loads, so suggestions and chat metadata
carrying markdown snippets were returned to callers with mangled content.

Both api/handlers/chat_handler.py and api/handlers/suggestion_handler.py
pipe responses through clean_json_response, so the bug surface was the
entire suggestion and chat pipeline.

The fix replaces the global string replace with two anchored regex
substitutions that only strip the outer fences:

- opening fence (^\`\`\`[^\\n`]*\\n?)  — covers the optional
  info string (`json` or otherwise) and trailing newline
- closing fence (\\n?\`\`\`\\s*$) — at the very end only

Interior \`\`\` sequences inside a JSON string value are preserved
verbatim. As a side effect the function now correctly handles responses
that have only an opening or only a closing fence (the original
implementation would leave those stranded mid-string).

- Add module-level _JSON_FENCE_OPEN_RE / _JSON_FENCE_CLOSE_RE
- Rewrite clean_json_response body to use anchored re.sub
- Four new tests in tests/mem_os/test_format_utils.py pin the contract:
  the issue reproducer (interior fenced code preserved), unclosed-fence-in-value,
  opening-only response, and closing-only response

Fixes #2420.
… pattern

Move the fence regex constants below the memos imports so ruff's E402
(module-import-not-at-top) passes again; the constants had been inserted
between the stdlib and memos import blocks, failing every build-matrix
job at the Ruff checks step.

Restrict the opening-fence info string from the greedy not-newline-not-
backtick class to a blank-padded short identifier. The greedy class
matched JSON payload characters up to the next backtick, so a
single-line fenced payload — triple-backtick + JSON + triple-backtick
with no newline after the fence, or with an info string followed by a
space — was swallowed whole and clean_json_response returned an empty
string, breaking json.loads downstream. The tightened class only accepts
an actual info string and keeps the newline optional so truncated fences
are still stripped.

Add regression tests for both single-line fenced forms.
@syzsunshine219 syzsunshine219 added the area:core MOS 编排层 / 框架底座 / 跨模块问题 label Nov 19, 2026
@Memtensor-AI Memtensor-AI added the status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 label Nov 19, 2026
@Memtensor-AI

Memtensor-AI commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2639
Task: d96e3012b03e74cc
Base: dev-v2.0.37
Head: codex/maintainer-edit-pr-2590-20261119
Head SHA: 93fe9a83b8fadf730e2c5b3e32e4804921deb23f

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

Generated by cloud-assistant via Open Code Review.

@endxxxx
endxxxx changed the base branch from main to dev-v2.0.37 November 20, 2026 02:57
@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Nov 20, 2026
@endxxxx
endxxxx merged commit 95d1df9 into dev-v2.0.37 Nov 20, 2026
17 checks passed
@endxxxx
endxxxx deleted the codex/maintainer-edit-pr-2590-20261119 branch November 20, 2026 03:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core MOS 编排层 / 框架底座 / 跨模块问题 status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发

Projects

None yet

Development

Successfully merging this pull request may close these issues.

clean_json_response corrupts JSON values containing code fences

6 participants