fix: settle the search bundle's language order between bakes - #451
Merged
Conversation
Pagefind names each language's index in one JSON map in pagefind-entry.json
and writes that map in whatever order it iterated, which is not the same
order twice. Two bakes of one source then produce two spellings of the same
fact: identical index hashes and page counts, different language order.
A site with one language has no order to get wrong, so this site does not
show it yet -- every page is indexed as the wiki's content language today.
It appears the moment translations are indexed in their own, which is what
SifterSearch#69 does: a bake against that build holds three languages, and
the two-bake check fails on this file alone, in the root bundle and in both
skin copies of it.
< {"languages":{"ko":{...},"en":{...},"km":{...}},...}
> {"languages":{"en":{...},"ko":{...},"km":{...}},...}
Landing it before the pin moves is the point: the bump that brings the fix
in should just pass, rather than failing the reproducibility check for a
reason that has nothing to do with the bump.
Reproducibility is this build's promise (#411), not the indexer's, so the
build keeps it -- the same work StripBuildStamps does for the per-request ids
MediaWiki leaves in a page. Key order carries no meaning in JSON and the
client looks a language up by name, so sorting the map changes nothing any
reader of the bundle can tell.
The re-encode does not try to reproduce Pagefind's bytes, only the same bytes
every time, which is all the promise asks. The flags keep it close anyway, so
a diff against an older bake stays readable -- escaping the character list to
‿ would make the file differ for no reason, which is the very thing being
fixed. An entry file that cannot be parsed, or holds no map, is left exactly
as it was: a bundle this does not understand is not one to rewrite.
It runs in the orchestrator, after the index job runJobs() holds back to the
end and before the skin passes, so the one bundle they all copy is already
settled and nothing has to be done per pass.
The fixtures in the test are the real two files, off one runner, one source.
typos reads the "ba" in a Pagefind hash as a misspelling, so the ignore list
learns their shape, beside the entry already there for Translate's own hex.
Verified in CI on #447, which pins the nightly carrying SifterSearch#69: the
bake produced en, ko and km indexes and the two bakes came out identical.
---
_Generated by [Claude Code](https://claude.ai/code/session_01NiyBN88bnGhcXVUXkpD7Mc)_
Co-authored-by: Claude <noreply@anthropic.com>
lens0021
marked this pull request as ready for review
August 17, 2026 11:19
lens0021
pushed a commit
that referenced
this pull request
Aug 17, 2026
Picks up SifterSearch v0.7.2, whose fix indexes each page in the language it is written in rather than the wiki's content language (chaotic-ground/SifterSearch#69). Pagefind builds one index per language it finds and its client selects by the language of the page the reader is on, so until now this site shipped one English index holding all 59 pages: Korean and Khmer were tokenised and stemmed by English rules, and a reader searching from a translated page was answered out of the English index because there was no other to fall back to. A bake now produces three -- en 39, ko 19, km 1. The smoke test says so out loud and fails if English and Korean are not indexed apart. It is the assertion this site was missing: nothing here would have caught a search answering a Korean reader out of an English index, because the search still worked. The existing Pagefind checks assert a bundle exists per skin copy (#399); none of them looked at what was in it. Khmer shows in the printed counts without being required, one page being a thin basis to gate on. Reproducibility is already handled: three languages made Pagefind's language map come out in a different order each bake, which #451 settled ahead of this so the bump would not fail the two-bake check for a reason unrelated to it. Verified before this pin moved, against the nightly carrying the same fix. Closes #400. --- _Generated by [Claude Code](https://claude.ai/code/session_01NiyBN88bnGhcXVUXkpD7Mc)_ Co-authored-by: Claude <noreply@anthropic.com>
lens0021
added a commit
that referenced
this pull request
Aug 17, 2026
) Bumps SifterSearch to **v0.7.2** and adds the assertion that proves what it fixes. Rebased onto the current `main`; the reproducibility half that was on this branch went in separately as #451. Closes #400 — but only its language half. The other half is filed as #454; see the end of this description. ## What changes for a reader SifterSearch indexed every page as the wiki's content language, whatever it was written in (chaotic-ground/SifterSearch#69). Pagefind builds one index per language it finds and its client picks the index by the language of the page the reader is on, so this site shipped **one English index holding every page**: ```console $ ls dist/pagefind/fragment | sed 's/_.*//' | sort | uniq -c 59 en ``` Two consequences. Korean and Khmer were tokenised and stemmed by English rules, so relevance for a translated page was whatever English stemming happened to give it. And a reader searching from a translated page was answered out of the English index — the client looked for `ko`, found none, and fell back to the only one there was. A bake now produces three indexes: **en 39, ko 19, km 1.** Searching from a page now answers in that page's language. Content not translated into the reader's language is no longer surfaced to them, which is the intended trade and the reason per-language indexes exist. ## The assertion `smoke` prints what the bundle holds and fails if English and Korean are not indexed apart: ``` Search indexes built: Indexed pages per language: ``` This is the check the site was missing. Nothing here would have caught a search answering a Korean reader out of an English index, **because the search still worked** — Pagefind falls back to whatever index exists. The existing Pagefind checks assert a bundle exists per skin copy (#399); none of them looked at what was inside it. Khmer appears in the printed counts without being required to pass; one page is a thin basis to gate on. ## Reproducibility, already handled Three languages made Pagefind's language map come out in a different order each bake, breaking #411's byte-identical check on `pagefind-entry.json` alone. #451 settled that ahead of this bump, so this PR does not fail the two-bake check for a reason unrelated to what it changes. ## How this was verified The whole chain was proven against `nightly-2026-08-17` before the pin moved to the release: | claim | evidence | |---|---| | Pagefind indexes per language off `<html lang>` | ran its own binary over the HTML `BuildIndexJob` emits — 1 index when all pages claim `en`, 2 when they tell the truth | | `getPageLanguage()` returns the translation's language for Translate subpages | a real bake of this site: `en`/`ko`/`km` | | the fix does not break the two-bake check | smoke green end to end, once #451's normalisation was in | ## What #400 asked for that this does not do #400 raised two faults. This closes the first and leaves the second, now tracked as **#454**: a translatable page reaches the export twice in the same language — as the source page and as the `/en` translation Translate materialises — so the English index holds 39 entries for 20 pages, and an English reader is offered each one twice. No language split can separate those two; they are the same language by construction. It needs a decision about *what* to index, and an extension point in SifterSearch to express it, since `SifterSearchNamespaces` cannot tell a source page from its translation. Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #447, which verified it. This is the half that can land now; the nightly pin and the smoke assertion that goes with it stay there, since they need each other and both wait on a formal SifterSearch release.
The bug
Pagefind names each language's index in one JSON map in
pagefind-entry.jsonand writes that map in whatever order it iterated, which is not the same order twice. Two bakes of one source then produce two spellings of the same fact — identical index hashes and page counts, different language order:This site does not show it yet. Every page is indexed as the wiki's content language today, so there is one language and no order to get wrong. It appears the moment translations are indexed in their own, which is what chaotic-ground/SifterSearch#69 does — a bake against that build holds three (en 39, ko 19, km 1) and #411's two-bake check fails on this one file, in the root bundle and in both skin copies of it.
Why it lands before the pin moves
So the bump that brings the language fix in just passes. Otherwise that PR fails the reproducibility check for a reason that has nothing to do with what it changed, and whoever opens it — updatecli, most likely — gets to work out why.
Where the fix belongs
Reproducibility is this build's promise (#411), not the indexer's, so the build keeps it. That is the same work
StripBuildStampsalready does for the per-request ids MediaWiki leaves in a page: something upstream writes a value that varies per build, and the build settles it.Key order carries no meaning in JSON and the client looks a language up by name, so sorting the map changes nothing any reader of the bundle can tell.
Build::stabilizeSearchIndex()runs in the orchestrator, after the index jobrunJobs()holds back to the end and before the skin passes, so the one bundle they all copy is already settled — nothing to do per pass.What the re-encode is careful about
It does not try to reproduce Pagefind's bytes, only the same bytes every time, which is all the promise asks — so a change in Pagefind's own formatting cannot quietly break this. The flags keep it close anyway, so a diff against an older bake stays readable; escaping the
include_characterslist to‿would make the file differ for no reason, which is the very thing being fixed.An entry file that cannot be parsed, or that holds no language map, is returned as
nulland left exactly as Pagefind wrote it. A bundle this does not understand is not one to rewrite.Tests
Search::stableIndexEntry()is pure, soSearchTestcovers it directly. The fixtures are the real two files, off one runner, from one source — and the test asserts they differ before asserting they normalise to the same thing, so it cannot pass vacuously.typosreads thebain a Pagefind hash (ko_72ba3bbda7) as a misspelling ofby, so.typos.tomllearns their shape, beside the entry already there for Translate's own hex stamps.Verification
Proven in CI on #447, which pins the nightly carrying SifterSearch#69: the bake produced
en,koandkmindexes, and the two bakes came out identical — the step that failed before this fix.On this branch, with the pin still at
v0.7.1, the change is a no-op in effect: one language, nothing to reorder.Generated by Claude Code