Skip to content

Add selectable dictionary sources and fix unreliable word lookups - #6

Open
hassanraza464 wants to merge 3 commits into
cartertemm:masterfrom
hassanraza464:dictionary-sources
Open

Add selectable dictionary sources and fix unreliable word lookups#6
hassanraza464 wants to merge 3 commits into
cartertemm:masterfrom
hassanraza464:dictionary-sources

Conversation

@hassanraza464

Copy link
Copy Markdown

Hi, thanks for the add-on. While using it I kept hitting HTTP Error 502: Bad Gateway on word lookups, and the digging turned up a second, quieter problem. This fixes both, then adds a way to avoid depending on a single dictionary.

Happy to split this into smaller PRs, drop the feature half, or adjust anything, just say.

The 502s

api.dictionaryapi.dev returns 200/404/502 nondeterministically per request. Their Cloudflare layer caches with max-age=14400, so a failure is served as a cache HIT for hours and looks deterministic; forcing a MISS with a cache buster showed the origin itself is unstable. Reproducible from unrelated networks, so it is not local to me.

The quieter problem

is, was and be return 404 from a fresh origin request, with the API's own {"title":"No Definitions Found",...} body, byte identical to what a nonsense word returns. The add-on cannot distinguish "no entry" from "not a word", so it was speaking "unable to find definition for word" for is. That behaviour survives the outage, since it is a property of the data rather than the downtime.

Changes

Reliability:

  • Retry 500/502/503/504 and connection errors up to three times with a short backoff. 404 still fails immediately so unknown words stay responsive. Exhausted 5xx speaks a message rather than the raw urllib error.
  • Verify a 404 against Wiktionary before reporting a word as unknown. Only if both sources agree is "unable to find definition for word" spoken; a word that exists but has no entry gives "definition unavailable". The check requires an en key, so pages existing only for other languages (e.g. hunde) are not mistaken for English words.

Word selection:

  • Strip punctuation from the ends of a word before looking it up, so a word at the end of a sentence or inside quotes or brackets resolves. Internal apostrophes and hyphens are preserved (let's, so-so), the unstripped form is tried as a fallback for 'tis and e.g., and text that is only punctuation no longer triggers a lookup. Type detection still runs on the original text, so the IP, URL and ISBN paths are untouched.

Sources:

  • A small provider abstraction returning normalised entries, so the formatting and audio code no longer cares where a definition came from. This is deliberately the shape dictionaryapi.dev already returns, so format_word_meaning and the audio dialog are unchanged.
  • Wiktionary as a second selectable source, with HTML stripping, filtering to English entries, and flattening of its nested senses. Its parent senses embed sub-senses as nested <ol>, which duplicates every child inside its parent, and a parent's parsedExamples aggregates all of its children's (13 for be item 1, 31 for item 8), so examples are allocated per part of speech rather than taking the first available.
  • A settings panel to pick the dictionary, plus an optional fallback to the other source. Defaults leave current behaviour untouched, so nobody's experience changes unless they open the panel.
  • Pronunciation fallback to Wikimedia Commons when a recording will not play, which also came up during this outage since their media host was 502ing every pronunciation URL. It uses Wikimedia's mp3 transcodes, so the existing MCI player handles them and no new audio backend is needed. A failed playback now reports "pronunciation unavailable" rather than only beeping.
  • Wikimedia responses are requested gzipped, taking a large entry from 69KB to about 9KB.

Also syncs manifest.ini with the version already set in buildVars.py, adds the two new modules to pythonSources so their strings are extracted, documents the settings in the readme, and fills in the new strings for es/fr/ru. I am not a native speaker of those, so please treat the translations as a starting point, or drop them if you would rather they went through the translators. No version bump, as that is your call.

Testing

Manually against NVDA on Windows 11, with both sources, the fallback on and off, and during the live outage. Verified the previously broken words (is, be, was) now return definitions from Wiktionary, that punctuation-adjacent selections resolve, that let's and so-so are unaffected, and that Commons audio substitutes when the dictionary's mp3 fails. Two pre-existing fuzzy entries, "Definition for {0}" and "Play {0}", are left as they were.

The Free Dictionary API intermittently returns 5xx errors, and returns its
ordinary "No Definitions Found" 404 for words it has no entry for. That 404 is
indistinguishable from a word not existing, so common words like "is" and "be"
were reported as not being words at all.

Reliability:

* Retry 500/502/503/504 and connection errors up to three times with a short
  backoff. 404 still fails immediately, so unknown words stay responsive.
* Speak a message for an exhausted 5xx rather than the raw urllib error.
* Verify a 404 against Wiktionary before reporting a word as unknown, so
  "unable to find definition" is only spoken when both sources agree. Requires
  an "en" entry, so pages that exist for other languages are not mistaken for
  English words.

Word selection:

* Strip punctuation from the ends of a word before looking it up, so words
  adjacent to punctuation resolve. Internal apostrophes and hyphens are kept,
  and the unstripped form is tried as a fallback for words like 'tis. Text that
  is only punctuation no longer triggers a lookup. Type detection still runs on
  the original text, leaving IP, URL and ISBN handling unchanged.

Sources:

* Add a provider abstraction returning normalised entries, so the formatting and
  audio code is independent of where a definition came from.
* Add Wiktionary as a selectable source, including HTML stripping, filtering to
  English entries and flattening its nested senses. Examples are allocated
  across each part of speech so a heading does not take the sentence belonging
  to the sense beneath it.
* Add a settings panel to choose the dictionary, plus an optional fallback to
  the other source when the chosen one lacks a definition or pronunciation.
  Defaults leave existing behaviour unchanged.
* Fall back to Wikimedia Commons pronunciations, using Wikimedia's mp3
  transcodes so the existing player handles them, when a recording cannot be
  played. Report a message instead of only beeping when none can be.
* Request Wikimedia responses gzipped, which reduces a large entry from 69KB to
  around 9KB.

Also sync manifest.ini with the version already set in buildVars.py.
Covers the dictionary selection, the fallback option, punctuation handling and
the "definition unavailable" message in the readme, and fills in the five new
strings for the Spanish, French and Russian catalogues. The settings panel title
reuses the existing "Text Information" message, so it needed no new translation.

Two entries, "Definition for {0}" and "Play {0}", were already marked fuzzy
before this change and are left as they are.
The dictionary API has been seen returning an empty body with a 200 status while
it is unwell. Parsing happened outside the try, so the lookup thread died with a
JSONDecodeError and the user was told nothing at all. Parsing now happens inside
it, and an unusable body is reported like any other failure. This also covers a
response that is valid json but an empty list, which raised IndexError.

Looking elsewhere for a pronunciation ran between the beep and the definition
being spoken, so a lookup paused for about half a second before saying anything.
It happens whenever the fallback option is enabled and the dictionary carries no
recording for a word, which is why it was intermittent. The definition is now
spoken first and the recording fetched afterwards.

That leaves a short window where a definition has been spoken but its audio has
not arrived, so opening the dialog within it shows no play buttons. Reaching that
needs a double press faster than the lookup, and the buttons appear on a second
attempt.
@hassanraza464

Copy link
Copy Markdown
Author

Pushed two fixes found while testing:

  • json.loads was outside the try in fetch_word_entry, so a 200 response with an empty body (which the API does while it is unwell) killed the lookup thread with a JSONDecodeError and the user was told nothing at all. Parsing is now inside the try, and an unusable body reports like any other failure. This also covers a valid but empty list, which raised IndexError on the [0].
  • Looking elsewhere for a pronunciation ran between the beep and ui.message, so a lookup paused for roughly half a second before speaking whenever the fallback was enabled and the dictionary had no recording for that word. The definition is spoken first now and the recording fetched afterwards.

The second leaves a short window where a definition has been spoken but its audio has not arrived, so opening the dialog inside it shows no play buttons. Reaching it needs a double press faster than the lookup, and a second attempt has them. Happy to close that properly if you would rather, though every option I looked at trades it for either a delayed dialog or one that grows buttons while being read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant