Text break Improvements - #955
Conversation
nat3Github
commented
Aug 14, 2026
nat3Github
commented
- word breaking: break on hyphen/slash in addition to space, and collapse trailing whitespace at wrap points so the next line doesn't start with leading spaces.
- chines-japanese-korean-aware keep-with punctuation: don't start a wrapped line with closing punctuation (、。」) or end one with opening punctuation (「（), matching how text is conventionally wrapped.
- test against a subset of Unicode's LineBreakTest.txt
Break on hyphen/slash in addition to space (matching browser convention), and collapse trailing whitespace runs at a break point so the next line doesn't start with leading whitespace.
When no space/hyphen/slash break point exists (the common case for CJK, which has no interword spacing), the width cutoff already breaks between any two codepoints. Add keep-with rules so that break doesn't start the next line with closing punctuation (、。」 etc.) or end the current line with opening punctuation (「（ etc.).
Curated subset of Unicode's public-domain LineBreakTest.txt (real test lines, hand-picked to cover the classes dvui's break rules support: whitespace, hyphen, and CJK opening/closing punctuation) referenced via @embedfile in the test file. Add legalBreakBefore(), a pure boundary-legality predicate built from the same isCjkClosingPunct / isCjkOpeningPunct helpers the real width-cutoff algorithm uses, so the test stays in sync with production logic by construction. Boundaries outside dvui's supported classes (e.g. plain letter-letter, where dvui deliberately breaks mid-word as a fallback, unlike strict UAX david-vanderson#14) are filtered out rather than asserted on.
CJK brackets are unambiguous glyphs (always open or always close), but Western quotation marks aren't: " is closing in German but opening in English, and „ (German opening low quote) wasn't handled at all. Rather than guessing a direction, treat quotation marks as a separate class with a symmetric no-break rule on both sides, matching UAX david-vanderson#14's LB19/LB19a. Removes the previously-wrong '/'" from the CJK opening/closing sets and adds isQuotationMark() instead. Oracle test extended with two more verbatim LineBreakTest.txt lines covering both directions of the rule.
|
note: whats still missing is correct breaking behavior for emojis glyphs. i have that in a seperate branch but it's not really relevant currently. |
|
edit: my verdict is that all font related issues in dvui (arabic/asian lang support, correct shaping, font discovery, bidi, text breaking, styling, var fonts, emoji fonts) cover such an ambious spectrum that it would be more efficient combining effort and outsourcing this into a seperate library. |
|
Sounds good. I am hoping that kb_text_shape (see #666) gives us shaping, bidi, breaking, and part of emoji support, but haven't gotten that far yet. |
|
i am working on a lib with similar scope to kb_text_shape but its a bit more ambitious, will try to integerate / co develop with dvui when its ready, i think it will solve a lot of issues |
Sounds awesome! One thing I've been trying to wrap my head around is how to achieve the current dvui kb_text_shape doesn't have that kind of api - you have to pass all the bytes up front (unless I'm totally missing something). As you are investigating this, any information on how to do this in a streaming fashion would be super useful! |
2 participants
