Fixes #4950: Preserve Unicode characters when applying diffs

[roomote-agent]

This PR fixes issue #4950 where Claude Sonnet 4 was converting Unicode apostrophes and quotes to ASCII equivalents when editing files using the apply_diff tool.

Root Cause

The diff strategies were normalizing Unicode characters to ASCII for comparison but not preserving the original Unicode characters in the replacement content.

Solution

Added a preserveUnicodeCharacters function to both diff strategies that maps Unicode characters from the original content and preserves them in replacements.

Changes

• Added preserveUnicodeCharacters function to multi-search-replace.ts and multi-file-search-replace.ts
• Integrated Unicode preservation into replacement logic
• Added comprehensive tests for Unicode character preservation
• Handles Unicode quotes (U+201C, U+201D) and apostrophes (U+2018, U+2019)

Testing

• Added unicode-preservation.test.ts with multiple test scenarios
• Verified Unicode characters are preserved during diff operations
• Confirmed backward compatibility with ASCII-only content

Closes #4950

---

Important

Fixes Unicode character conversion issue in apply_diff by preserving original Unicode characters during replacements.

• Behavior:
  • Fixes Unicode character conversion issue in apply_diff tool by preserving original Unicode characters during replacements.
  • Handles Unicode quotes (U+201C, U+201D) and apostrophes (U+2018, U+2019).
• Functions:
  • Adds preserveUnicodeCharacters function to multi-search-replace.ts and multi-file-search-replace.ts.
  • Integrates Unicode preservation into replacement logic in both files.
• Testing:
  • Adds unicode-preservation.test.ts with tests for Unicode character preservation.
  • Verifies preservation of Unicode characters and backward compatibility with ASCII content.

^{This description was created by for a1821c7. You can customize this summary. It will automatically update as commits are pushed.}

[ellipsis-dev]

The preserveUnicodeCharacters function here uses a simple mapping that overwrites the mapping for ASCII quotes (" and ') when both opening and closing Unicode variants are present. This means all occurrences of a quote in the replacement will use the last mapped Unicode character, potentially losing the distinction between opening and closing quotes. Consider using a sequential approach or separate mappings to preserve paired quotes correctly. Also, the function is duplicated in another file; consider extracting it into a shared utility module.

^{This comment was generated because it violated a code review rule: irule_tTqpIuNs8DV0QFGj.}

[daniel-lxs]

This PR seems to address the symptoms rather than the root cause. While the model can correctly write Unicode to files, the only issue I’ve seen so far is with quotes being replaced.

The proposed solution feels a bit overengineered, especially since it adds ongoing checks for Unicode quotes that don’t appear to be common.

That’s not to say the issue isn’t valid, but it might be better to investigate the underlying cause and fix it directly instead.