feat: enhance documentation with user guides, architecture, internationalisation, persistence model, and troubleshooting sections

Author: PierreRaybaut

README.md

@@ -0,0 +1,33 @@
+# DataLab-Web documentation
+
+Guides and reference material for DataLab-Web. Start with the [project README](../README.md) for the overview and quickstart.
+
+## User guide
+
+- [Welcome](userguide/welcome.md) — what DataLab-Web is and what you can do in it.
+- [Computation engine](userguide/computation-engine.md) — how Sigima runs in the browser.
+- [Differences from the desktop app](userguide/differences-from-desktop.md).
+
+## Concepts & architecture
+
+- [Architecture](architecture.md) — layer view, component view, worker protocols and end-to-end flows.
+- [Persistence model](persistence.md) — the HDF5 workspace as the single source of truth.
+- [Internationalisation](i18n.md) — translation framework and contributor workflow.
+
+## Subsystems
+
+- [Notebooks](notebooks.md) — multi-tab notebook panel, `.ipynb` import/export, macro conversion.
+- [Plugins](plugins.md) — Qt-compatible `PluginBase` API, hot-reload and the bundled vitrine plugin.
+- [Shim registry](shim-registry.md) — tracking temporary backport shims.
+
+## Contributing & release
+
+- [Testing strategy](testing-strategy.md) — the test pyramid, decision tree, and how to run each suite locally.
+- [Releasing & distribution](releasing.md) — versioning, the release pipeline, and the app + SDK tarballs.
+- [Troubleshooting](troubleshooting.md) — known browser quirks and workarounds.
+- [Roadmap](roadmap.md) — short- and long-term plans.
+- [CONTRIBUTING](../CONTRIBUTING.md) — project rules, including the Generative AI policy.
+
+## Integration examples
+
+- [Angular host app](examples/angular/README.md) — embedding DataLab-Web via the TypeScript SDK.

doc/README.md

@@ -14,7 +14,7 @@ in the browser.
 
 The whole stack runs **inside a single browser tab**: there is no backend.
 The HDF5 workspace file is the only durable artifact (see
-[Persistence model](../README.md#persistence-model)).
+[Persistence model](persistence.md)).
 
 ---
 

doc/architecture.md

@@ -0,0 +1,33 @@
+# Internationalisation
+
+DataLab-Web ships a working internationalisation framework. English is the **source language** (the `msgid`) and French is the first translated locale. The UI auto-detects the user's regional preference and renders fully in their language, including the labels that come from Python (Sigima / guidata) through the Pyodide bridge.
+
+## How it works for users
+
+The active locale is resolved **once per page load** in this order (first match wins):
+
+1. `?lang=<code>` URL query parameter (handy for sharing links and E2E tests);
+2. an explicit choice persisted in `localStorage["datalab-web:lang"]` (set via the language selector in the menu bar);
+3. the browser's regional preference (`navigator.languages`);
+4. English (`DEFAULT_LOCALE`) as the fallback.
+
+Switching language from the menu-bar selector persists the choice and triggers a **full page reload**, because the Pyodide instance pins its `LANG` at boot and Sigima/guidata cache their gettext labels at import time — so a fresh instance must boot with the new locale.
+
+## Architecture
+
+There are two translation surfaces, kept in sync:
+
+- **TypeScript/React strings** go through a lightweight `t()` helper ([`src/i18n/translate.ts`](../src/i18n/translate.ts)). The English string _is_ the key; for English the helper is the identity function (no `en.json` is shipped), and `{name}` placeholders are interpolated from a `vars` argument. Catalogs live in [`src/locales/<code>.json`](../src/locales/) and are merged in [`src/i18n/catalogs.ts`](../src/i18n/catalogs.ts). React components read the locale via `useTranslation()` from [`src/i18n/I18nProvider.tsx`](../src/i18n/I18nProvider.tsx); non-React code (the action registry, `runtime.ts`) reads it directly from [`src/i18n/locale.ts`](../src/i18n/locale.ts).
+- **Python-origin labels** (signal/image creation types, processing/operations/analysis menu entries, parameter-dialog field labels) are translated by Sigima/guidata's own gettext `.mo` catalogs. The active locale is mapped to a `LANG` value by `pyodideLang()` (English → POSIX `C`, i.e. untranslated `msgid`; any other locale → its bare code such as `fr`) and exported into the Pyodide environment **before** the first guidata/sigima import in [`src/runtime/runtime.ts`](../src/runtime/runtime.ts), [`src/runtime/macroWorker.ts`](../src/runtime/macroWorker.ts) and [`src/runtime/notebookWorker.ts`](../src/runtime/notebookWorker.ts).
+
+Applying `t()` to a Python-origin label is **safe**: a label that Sigima already owns in French has no key in `fr.json`, so `t()` returns it unchanged; only the English override strings defined in [`src/runtime/processor.py`](../src/runtime/processor.py) and the React-owned strings get looked up.
+
+## Contributing translations (developer workflow)
+
+1. **Wrap every new user-facing string** in `t("…")` (import from `src/i18n/translate`). Use `t("Delete {count} objects?", { count })` for interpolation. Do **not** translate brand names (e.g. _DataLab Web_) or AI-assistant system prompts.
+2. **Extract the keys.** Run `npm run i18n:extract` to merge newly discovered `t("…")` keys into `src/locales/fr.json` (existing translations are preserved; new keys get empty placeholders). Keys referenced only through a variable (never as a string literal) must be listed in [`src/locales/_dynamic-keys.json`](../src/locales/_dynamic-keys.json) so the extractor seeds them.
+3. **Fill in the translations** in `src/locales/fr.json`.
+4. **Verify** with `npm run i18n:check` (fails on missing or empty keys; this also runs the static scan). It is wise to run it in CI.
+5. **Add a new locale** by: adding its code to `SUPPORTED_LOCALES` and `LOCALE_LABELS` in [`src/i18n/locale.ts`](../src/i18n/locale.ts), registering its catalog in [`src/i18n/catalogs.ts`](../src/i18n/catalogs.ts), adding the code to `LOCALES` in [`scripts/i18n-extract.mjs`](../scripts/i18n-extract.mjs), and creating `src/locales/<code>.json`. Python-side labels are picked up automatically if the matching `.mo` catalogs are bundled in the Sigima/guidata wheels.
+
+Coverage is enforced by Vitest unit tests ([`tests/ts/i18n/i18n.test.ts`](../tests/ts/i18n/i18n.test.ts)) and an end-to-end Playwright spec ([`tests/e2e/i18n.spec.ts`](../tests/e2e/i18n.spec.ts)) that boots the app with `?lang=fr` and asserts both a translated UI menu and a French Sigima label coming through the Pyodide bridge.

doc/i18n.md

@@ -0,0 +1,22 @@
+# Persistence model
+
+DataLab-Web treats the **HDF5 workspace file as the single durable source of truth**. Everything else — IndexedDB caches, the recent notebooks/macros menus, even the in-memory Python object model — is ephemeral and reset on a hard reload of the Pyodide instance.
+
+| Asset class                               | Survives F5 reload?              | How to make durable                                                              |
+| ----------------------------------------- | -------------------------------- | -------------------------------------------------------------------------------- |
+| Signals & images                          | **No** — wiped with Pyodide      | **File → Save HDF5 workspace…**                                                  |
+| Groups, ROIs, metadata, plot annotations  | **No** — wiped with Pyodide      | **File → Save HDF5 workspace…**                                                  |
+| Macro **content**                         | Reachable via the _Recent…_ menu | **File → Save HDF5 workspace…** for the full workspace, or download individually |
+| Notebook **content**                      | Reachable via the _Recent…_ menu | **File → Save HDF5 workspace…**, or **Save notebook as…** for a `.ipynb`         |
+| Notebook **outputs / execution counters** | **No** — outputs aren't cached   | Save HDF5 workspace (outputs are persisted there too)                            |
+
+Macros and notebooks you edit are kept in a roll-over IndexedDB cache so you can re-open them later, but they are **not** silently restored into a fresh session: they only appear in each panel's **Recent…** menu. Pristine, auto-created sample documents are never cached.
+
+How this surfaces in the UI:
+
+- The window title shows `DataLab-Web — <filename or "Untitled">`, with a `•` marker as soon as the workspace contains unsaved changes, cleared on the next **Open / Save HDF5 workspace…**.
+- A `beforeunload` confirmation prompt fires only when the workspace is dirty.
+- A one-time informational banner appears at cold start if the IndexedDB cache holds edited macros or notebooks, pointing you at the **Recent…** menus where you can re-open them. **Dismiss** hides it for the session.
+- Fresh sessions are labelled **Untitled**. The first **File → Save HDF5 workspace…** proposes a timestamped name (`workspace-YYYYMMDD-HHMMSS.h5`); subsequent saves reuse the last filename associated with the session.
+
+The behaviour mirrors DataLab desktop: closing without saving loses unsaved work; opening an HDF5 workspace replaces the in-memory state.

doc/persistence.md

@@ -0,0 +1,38 @@
+# Releasing & distribution
+
+## Versioning
+
+The application version is declared **once**, in `package.json`, and is injected into the bundle at build time via Vite's `define` option (see `vite.config.ts`). The _Help → About_ dialog reads it from `import.meta.env.VITE_APP_VERSION`.
+
+To bump the version, use the standard npm command (it edits `package.json`, creates a commit, and tags it `vX.Y.Z`):
+
+```powershell
+npm version patch   # bug fix:  0.1.0 → 0.1.1
+npm version minor   # feature:  0.1.0 → 0.2.0
+npm version major   # breaking: 0.1.0 → 1.0.0
+```
+
+The next `npm run dev` or `npm run build` automatically picks up the new value — no other file needs to be edited.
+
+> **Keep `packages/sdk/package.json` in sync** — bump its `version` to the same value before tagging. The release CI fails if the two `package.json` files disagree.
+
+> **What `git push --tags` triggers** — the [`Release tarballs`](../.github/workflows/release.yml) workflow runs, in order: version coherence check (tag ↔ both `package.json` files) → `pytest tests/python` (3.11 + 3.12) and Playwright E2E (in parallel) → lint + Vitest + build + pack the two `.tgz` → publish a GitHub Release with the tarballs and auto-generated notes → deploy `dist/` to GitHub Pages. Any failing gate aborts the release **and** the deploy.
+
+## Distribution: app bundle + SDK tarballs
+
+DataLab-Web is shipped to integrators as **two `.tgz` artefacts** produced by the release pipeline:
+
+| Tarball                                  | Contents                                               | Consumer action                               |
+| ---------------------------------------- | ------------------------------------------------------ | --------------------------------------------- |
+| `datalab-web-<version>.tgz`              | Static bundle (everything under `dist/`) + `DEPLOY.md` | Unpack under any web server                   |
+| `datalab-platform-web-sdk-<version>.tgz` | Host-side TypeScript SDK (`@datalab-platform/web-sdk`) | `npm install ./datalab-platform-web-sdk-…tgz` |
+
+Generate them locally:
+
+```powershell
+npm run release:pack   # lint → test → build → SDK pack → app pack → summary
+```
+
+Or invoke each step independently (`npm run sdk:pack`, `npm run app:pack`). Output lands in `release/`.
+
+The two artefacts share the same release version. The wire-protocol they negotiate (`MAJOR.MINOR`, exposed as `client.protocolVersion`) is independent: a SDK and a bundle from different release versions inter-operate as long as the protocol `MAJOR` is unchanged. See [examples/angular/README.md](examples/angular/README.md) for the integrator-facing compatibility matrix.

doc/releasing.md

@@ -0,0 +1,13 @@
+# Roadmap
+
+## Short-term
+
+- Generic results-table view aligned with the desktop _Results_ panel.
+- Richer image data preview (numeric grid with virtualised scrolling).
+- Move the main `DataLabRuntime` off the UI thread (macros already run in a dedicated Web Worker; the main computation Pyodide instance still lives on the main thread).
+- Additional file formats through `sigima.io` (currently focused on text and HDF5).
+
+## Longer-term
+
+- Remote control bridge to a real DataLab desktop instance via the Web API.
+- Collaborative sessions through shared workspace files.

doc/roadmap.md

@@ -17,6 +17,50 @@ Playwright runs serially (`workers: 1`, ~3 min Pyodide boot per spec
 file in CI), so adding E2E tests has a real, compounding cost. Treat
 them as a scarce resource.
 
+A continuous layer ties the three together: GitHub Actions (`tests.yml`) runs all three on every push / PR. The Python layer runs `bootstrap.py` directly under CPython through fixtures that stub the Pyodide-only modules (`js`, `pyodide.ffi`); this gives fast feedback and high coverage without booting WebAssembly.
+
+## Running the suites locally
+
+```powershell
+# One-time: copy the environment template and create the project venv
+# (Python 3.11 or 3.12 — earlier versions trip a quirk in
+# ``isinstance(list[T], type)`` that breaks Sigima's processor
+# introspection).
+Copy-Item .env.template .env
+py -3.11 -m venv .venv
+.\.venv\Scripts\python -m pip install -r requirements-dev.txt
+
+# Python unit tests + coverage report (htmlcov-python/)
+.\.venv\Scripts\python -m pytest tests/python --cov=src/runtime --cov-report=html:htmlcov-python
+
+# TypeScript unit tests + coverage report (coverage-ts/)
+npm test
+npm run test:cov
+
+# End-to-end browser tests (boots Pyodide in Chromium ~1.5 min)
+npx playwright install chromium   # one-time
+npm run test:e2e
+
+# Performance benchmarks (opt-in — ~5 min). Includes the image-display
+# benchmark and the 50k-sample binary transfer probe.
+npx playwright test --project=perf
+PERF=1 npm run test:e2e
+
+# Run a throwaway `_repro_*` probe (the default suite ignores them).
+$env:PW_REPRO=1; npx playwright test --project=repro tests/e2e/_repro_x.spec.ts
+```
+
+Test layout:
+
+```text
+tests/
+├── python/          # pytest suite — exercises bootstrap.py headlessly
+├── ts/              # Vitest suite — pure TypeScript modules
+└── e2e/             # Playwright specs — real browser smoke tests
+```
+
+VS Code tasks are provided under `.vscode/tasks.json` (`🚀 Pytest`, `🟢 Vitest`, `🎭 Playwright`, …). The default test task (`Ctrl+Shift+P → Run Test Task`) launches the Python suite.
+
 ## Default suite vs perf project
 
 The default Playwright project (`chromium`) runs the regression suite and intentionally **excludes** performance benchmarks and `_repro_*` throwaway probes. The costly `perf` and `benchmark` projects are **opt-in**: each is registered only when explicitly requested, either via its `--project=<name>` flag or its `PW_<NAME>` env var (see `wantsProject` in [playwright.config.ts](../playwright.config.ts)). A bare `playwright test` — and therefore the default CI run, which calls `npm run test:e2e` (`--project=chromium`) — never pays for them.

doc/testing-strategy.md

@@ -0,0 +1,9 @@
+# Troubleshooting
+
+## Microsoft Edge: slow first load (several minutes instead of ~30 s)
+
+If the first load takes several minutes instead of ~30 s on Microsoft Edge, Edge's _Enhance your security on the web_ setting is disabling the WebAssembly JIT compiler on this site. Microsoft [documents this behaviour and explicitly recommends the workaround](https://learn.microsoft.com/en-us/deployedge/microsoft-edge-security-browse-safer) — quoting the page:
+
+> Developers should be aware that the WebAssembly (WASM) interpreter running in enhanced security mode might not yield the expected level of performance. We recommend adding your site as an exception to opt-out of enhanced security mode for site users.
+
+To add the exception: open `edge://settings/privacy` → _Security_ → _Manage enhanced security for sites_ → under _Never use enhanced security for these sites_, click _Add a site_ and enter `https://datalab-platform.com` (or your deployment URL). Reload the page; load time drops back to ~30 s. Chrome, Firefox and Safari are not affected.