Skip to content

fix: settle the search bundle's language order between bakes - #451

Merged
lens0021 merged 1 commit into
mainfrom
claude/stable-search-index
Aug 17, 2026
Merged

fix: settle the search bundle's language order between bakes#451
lens0021 merged 1 commit into
mainfrom
claude/stable-search-index

Conversation

@lens0021

Copy link
Copy Markdown
Collaborator

Split out of #447, which verified it. This is the half that can land now; the nightly pin and the smoke assertion that goes with it stay there, since they need each other and both wait on a formal SifterSearch release.

The bug

Pagefind names each language's index in one JSON map in pagefind-entry.json and writes that map in whatever order it iterated, which is not the same order twice. Two bakes of one source then produce two spellings of the same fact — identical index hashes and page counts, different language order:

< {"languages":{"ko":{"hash":"ko_72ba3bbda7",…},"en":{"hash":"en_e66da688eb",…},"km":{…}},…}
> {"languages":{"en":{"hash":"en_e66da688eb",…},"ko":{"hash":"ko_72ba3bbda7",…},"km":{…}},…}

This site does not show it yet. Every page is indexed as the wiki's content language today, so there is one language and no order to get wrong. It appears the moment translations are indexed in their own, which is what chaotic-ground/SifterSearch#69 does — a bake against that build holds three (en 39, ko 19, km 1) and #411's two-bake check fails on this one file, in the root bundle and in both skin copies of it.

Why it lands before the pin moves

So the bump that brings the language fix in just passes. Otherwise that PR fails the reproducibility check for a reason that has nothing to do with what it changed, and whoever opens it — updatecli, most likely — gets to work out why.

Where the fix belongs

Reproducibility is this build's promise (#411), not the indexer's, so the build keeps it. That is the same work StripBuildStamps already does for the per-request ids MediaWiki leaves in a page: something upstream writes a value that varies per build, and the build settles it.

Key order carries no meaning in JSON and the client looks a language up by name, so sorting the map changes nothing any reader of the bundle can tell.

Build::stabilizeSearchIndex() runs in the orchestrator, after the index job runJobs() holds back to the end and before the skin passes, so the one bundle they all copy is already settled — nothing to do per pass.

What the re-encode is careful about

It does not try to reproduce Pagefind's bytes, only the same bytes every time, which is all the promise asks — so a change in Pagefind's own formatting cannot quietly break this. The flags keep it close anyway, so a diff against an older bake stays readable; escaping the include_characters list to would make the file differ for no reason, which is the very thing being fixed.

An entry file that cannot be parsed, or that holds no language map, is returned as null and left exactly as Pagefind wrote it. A bundle this does not understand is not one to rewrite.

Tests

Search::stableIndexEntry() is pure, so SearchTest covers it directly. The fixtures are the real two files, off one runner, from one source — and the test asserts they differ before asserting they normalise to the same thing, so it cannot pass vacuously.

typos reads the ba in a Pagefind hash (ko_72ba3bbda7) as a misspelling of by, so .typos.toml learns their shape, beside the entry already there for Translate's own hex stamps.

Verification

Proven in CI on #447, which pins the nightly carrying SifterSearch#69: the bake produced en, ko and km indexes, and the two bakes came out identical — the step that failed before this fix.

On this branch, with the pin still at v0.7.1, the change is a no-op in effect: one language, nothing to reorder.


Generated by Claude Code

Pagefind names each language's index in one JSON map in pagefind-entry.json
and writes that map in whatever order it iterated, which is not the same
order twice. Two bakes of one source then produce two spellings of the same
fact: identical index hashes and page counts, different language order.

A site with one language has no order to get wrong, so this site does not
show it yet -- every page is indexed as the wiki's content language today.
It appears the moment translations are indexed in their own, which is what
SifterSearch#69 does: a bake against that build holds three languages, and
the two-bake check fails on this file alone, in the root bundle and in both
skin copies of it.

  < {"languages":{"ko":{...},"en":{...},"km":{...}},...}
  > {"languages":{"en":{...},"ko":{...},"km":{...}},...}

Landing it before the pin moves is the point: the bump that brings the fix
in should just pass, rather than failing the reproducibility check for a
reason that has nothing to do with the bump.

Reproducibility is this build's promise (#411), not the indexer's, so the
build keeps it -- the same work StripBuildStamps does for the per-request ids
MediaWiki leaves in a page. Key order carries no meaning in JSON and the
client looks a language up by name, so sorting the map changes nothing any
reader of the bundle can tell.

The re-encode does not try to reproduce Pagefind's bytes, only the same bytes
every time, which is all the promise asks. The flags keep it close anyway, so
a diff against an older bake stays readable -- escaping the character list to
‿ would make the file differ for no reason, which is the very thing being
fixed. An entry file that cannot be parsed, or holds no map, is left exactly
as it was: a bundle this does not understand is not one to rewrite.

It runs in the orchestrator, after the index job runJobs() holds back to the
end and before the skin passes, so the one bundle they all copy is already
settled and nothing has to be done per pass.

The fixtures in the test are the real two files, off one runner, one source.
typos reads the "ba" in a Pagefind hash as a misspelling, so the ignore list
learns their shape, beside the entry already there for Translate's own hex.

Verified in CI on #447, which pins the nightly carrying SifterSearch#69: the
bake produced en, ko and km indexes and the two bakes came out identical.

---
_Generated by [Claude Code](https://claude.ai/code/session_01NiyBN88bnGhcXVUXkpD7Mc)_

Co-authored-by: Claude <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to preview August 17, 2026 11:09 Destroyed
@lens0021
lens0021 marked this pull request as ready for review August 17, 2026 11:19
@lens0021
lens0021 merged commit e7fc96c into main Aug 17, 2026
21 checks passed
@lens0021
lens0021 deleted the claude/stable-search-index branch August 17, 2026 11:19
lens0021 pushed a commit that referenced this pull request Aug 17, 2026
Picks up SifterSearch v0.7.2, whose fix indexes each page in the language it
is written in rather than the wiki's content language
(chaotic-ground/SifterSearch#69). Pagefind builds one index per language it
finds and its client selects by the language of the page the reader is on, so
until now this site shipped one English index holding all 59 pages: Korean
and Khmer were tokenised and stemmed by English rules, and a reader searching
from a translated page was answered out of the English index because there
was no other to fall back to.

A bake now produces three -- en 39, ko 19, km 1.

The smoke test says so out loud and fails if English and Korean are not
indexed apart. It is the assertion this site was missing: nothing here would
have caught a search answering a Korean reader out of an English index,
because the search still worked. The existing Pagefind checks assert a bundle
exists per skin copy (#399); none of them looked at what was in it. Khmer
shows in the printed counts without being required, one page being a thin
basis to gate on.

Reproducibility is already handled: three languages made Pagefind's language
map come out in a different order each bake, which #451 settled ahead of this
so the bump would not fail the two-bake check for a reason unrelated to it.

Verified before this pin moved, against the nightly carrying the same fix.

Closes #400.

---
_Generated by [Claude Code](https://claude.ai/code/session_01NiyBN88bnGhcXVUXkpD7Mc)_

Co-authored-by: Claude <noreply@anthropic.com>
lens0021 added a commit that referenced this pull request Aug 17, 2026
)

Bumps SifterSearch to **v0.7.2** and adds the assertion that proves what
it fixes. Rebased onto the current `main`; the reproducibility half that
was on this branch went in separately as #451.

Closes #400 — but only its language half. The other half is filed as
#454; see the end of this description.

## What changes for a reader

SifterSearch indexed every page as the wiki's content language, whatever
it was written in (chaotic-ground/SifterSearch#69). Pagefind builds one
index per language it finds and its client picks the index by the
language of the page the reader is on, so this site shipped **one
English index holding every page**:

```console
$ ls dist/pagefind/fragment | sed 's/_.*//' | sort | uniq -c
     59 en
```

Two consequences. Korean and Khmer were tokenised and stemmed by English
rules, so relevance for a translated page was whatever English stemming
happened to give it. And a reader searching from a translated page was
answered out of the English index — the client looked for `ko`, found
none, and fell back to the only one there was.

A bake now produces three indexes: **en 39, ko 19, km 1.**

Searching from a page now answers in that page's language. Content not
translated into the reader's language is no longer surfaced to them,
which is the intended trade and the reason per-language indexes exist.

## The assertion

`smoke` prints what the bundle holds and fails if English and Korean are
not indexed apart:

```
Search indexes built:
Indexed pages per language:
```

This is the check the site was missing. Nothing here would have caught a
search answering a Korean reader out of an English index, **because the
search still worked** — Pagefind falls back to whatever index exists.
The existing Pagefind checks assert a bundle exists per skin copy
(#399); none of them looked at what was inside it.

Khmer appears in the printed counts without being required to pass; one
page is a thin basis to gate on.

## Reproducibility, already handled

Three languages made Pagefind's language map come out in a different
order each bake, breaking #411's byte-identical check on
`pagefind-entry.json` alone. #451 settled that ahead of this bump, so
this PR does not fail the two-bake check for a reason unrelated to what
it changes.

## How this was verified

The whole chain was proven against `nightly-2026-08-17` before the pin
moved to the release:

| claim | evidence |
|---|---|
| Pagefind indexes per language off `<html lang>` | ran its own binary
over the HTML `BuildIndexJob` emits — 1 index when all pages claim `en`,
2 when they tell the truth |
| `getPageLanguage()` returns the translation's language for Translate
subpages | a real bake of this site: `en`/`ko`/`km` |
| the fix does not break the two-bake check | smoke green end to end,
once #451's normalisation was in |

## What #400 asked for that this does not do

#400 raised two faults. This closes the first and leaves the second, now
tracked as **#454**: a translatable page reaches the export twice in the
same language — as the source page and as the `/en` translation
Translate materialises — so the English index holds 39 entries for 20
pages, and an English reader is offered each one twice.

No language split can separate those two; they are the same language by
construction. It needs a decision about *what* to index, and an
extension point in SifterSearch to express it, since
`SifterSearchNamespaces` cannot tell a source page from its translation.

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants