Fix OCR interpretation of stylistic lines

[MayVerse4]

Updated the OCR processing logic to correctly identify and ignore stylistic lines, such as horizontal lines made of underscores or hyphens, to prevent them from being misinterpreted as placeholders. This change ensures that the OCR output is consistent with the Ground Truth by not including underscores for stylistic lines.

[kushdab]

Thanks for tackling this — it's adjacent to something I dug into on #59, but I think the fix as written has a mechanism problem that will make it unreliable and, in some cases, silently drop legitimate output.

The core issue: delta here is a per-token streaming chunk, not a line.

delta = chunk["choices"][0]["delta"].get("content", "")

This is one SSE chunk from the SGLang/vLLM /v1/chat/completions stream — typically a single BPE token or subword piece, arriving one at a time as generation proceeds. A "stylistic line" like ____________________ is not emitted as one delta; it's spread across however many tokens the tokenizer's BPE merges happen to produce for a run of repeated characters — could be one 20-char token, could be ten 2-char tokens, depends entirely on how that specific run was tokenized (and that's model/tokenizer-specific, not something this check can rely on).

def is_stylistic_line(text):
    return text.strip() in ('_', '-', '—')

This only matches a delta whose entire stripped content is exactly one of those three single characters. Two consequences:

1. It will rarely catch the actual pattern it's meant for. A real decorative line of 20+ underscores almost never tokenizes into a stream of single-char _ tokens one at a time — it's far more likely to come through as a handful of multi-character chunks ("____", "________", etc.), none of which equal '_' after stripping. So for the common case, this filter is a no-op.

2. It will drop legitimate single-character content that has nothing to do with decorative lines. Any delta that happens to be exactly -, _, or — gets silently discarded from both chunks and the output file — a hyphen in a compound word split at a token boundary ("well" + "-" + "known"), an underscore in a snake_case identifier the model is transcribing, a markdown bullet -, an em-dash used as sentence punctuation. There's no way to tell from a single delta whether it's part of a 20-character decorative rule or a genuine hyphen — that context only exists once you look at the assembled line, not the token in isolation.

Suggested approach: this needs to operate on the assembled text, not per-token deltas — e.g. post-process each completed line in chunks/text (or buffer until a newline) and check whether that whole line matches a decorative-rule pattern, something like:

import re

_STYLISTIC_LINE_RE = re.compile(r'^[\s]*[_\-—―‐‑‒–]{4,}[\s]*$')

def is_stylistic_line(line: str) -> bool:
    """A full line consisting only of repeated dash/underscore/em-dash
    characters (a decorative horizontal rule), not a single character."""
    return bool(_STYLISTIC_LINE_RE.match(line))

applied line-by-line to the final "".join(chunks) (or buffered per-line during streaming, splitting on \n), not to each raw delta. That also sidesteps the tokenization-boundary problem entirely, since it only ever evaluates a complete line.

One more thing worth separating out: this PR's framing ("prevent misinterpretation as placeholders") reads as a different problem from what #59 was actually about. #59 wasn't the model transcribing a genuine decorative line in a document — it was verbatim regurgitation of eval-rubric text ("Rule 2 UNDERSCORE & LINE RULES", "Ground Truth") from what looks like training-data contamination from an LLM-as-judge pipeline. If this PR is aimed at the #59 pattern specifically, filtering standalone decorative lines from the output won't address that — the contaminated example produces a lot more than just a line of underscores (full rubric section headers, etc.), and the fix for that lives in the training data, not client-side output filtering. If this is aimed at a separate, genuine "document has a decorative rule and OCR renders it as underscores" case, that's a legitimate and different problem worth solving — just flagging so it's clear which one this patch is targeting before it merges.

[MayVerse4]

Thanks for the detailed review — you’re absolutely right. I was treating delta as if it carried line-level context, but it’s only a streaming token/subword chunk, so the previous check was both ineffective for repeated-rule output and unsafe for legitimate single-character punctuation/identifiers.

I’ve updated the PR to remove all per-delta filtering. The stream is now collected verbatim into chunks, then the fully assembled text is post-processed line-by-line with a regex for standalone decorative rules only:

STYLISTIC_LINE_RE = re.compile(r"^[\s]*[_\-—―‐‑‒–]{4,}[\s]*$")

and the output file is written from that post-processed assembled text. This preserves single -, _, and — deltas in normal content such as well-known, snake_case, bullets, or sentence punctuation, while removing only complete lines made of 4+ rule characters.

I also agree with your distinction from #59. This patch is only intended for the separate/client-side case where the OCR output contains a decorative horizontal rule from the source document. It is not meant to solve the rubric/training-contamination pattern from #59, which would require a different upstream/data fix.