Skip to content

test: add regression test for multibyte UTF-8 truncation in compile history summary #81

Description

@yvonnedevlinrh

Context

PR #75 fixed a bug in buildCompiledArticle (tools/compile.go:585-589) where the history summary truncation used byte-slicing (summary[:80]) instead of rune-slicing. When a multibyte UTF-8 character (e.g. em-dash U+2014, 3 bytes) spanned the byte-80 boundary, the slice cut the character mid-encoding, producing invalid UTF-8 (e.g. 0xE2 0x80 0x2E).

The fix converts to []rune before slicing:

if runes := []rune(summary); len(runes) > 80 {
    summary = string(runes[:80]) + "..."
}

## What's needed
A regression test that:

  1. Constructs a Cluster with a learning whose Content contains a multibyte character positioned so it would span byte offset 80 under the old byte-slicing approach (e.g. an em-dash at rune position 79-80, or a string of 79 ASCII chars followed by an em-dash)
  2. Calls buildCompiledArticle and extracts the history table row
  3. Asserts the output is valid UTF-8 (utf8.ValidString)
  4. Asserts the summary ends with "..." and does not contain a corrupted byte sequence
  5. Ideally uses a table-driven approach covering multiple multibyte scenarios: em-dash (3-byte), CJK character (3-byte), emoji (4-byte), and a string that is exactly 80 runes (no truncation needed)

Convention pack reference

TC-006 MUST: Bug fixes MUST include a regression test that reproduces the original failure and verifies the fix. This was deferred from PR #75 to a follow-up issue.

Acceptance criteria

  • Test exists in tools/compile_test.go (or a new tools/compile_truncation_test.go)
  • Test fails if the fix is reverted to byte-slicing (summary[:80])
  • Test passes with the current rune-slicing implementation
  • go test -race -count=1 ./tools/... passes
  • Valid UTF-8 output is asserted for all multibyte boundary cases

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions