v0.5.0

v0.5.0 (2026-06-29)

Major UI/UX overhaul — four new layout variants (V1–V4), six design channels,
SQL compare filter, accessibility and contrast fixes across all themes.

Added

• Four layout variants (V1–V4) — distinct UI shells for the classic sidebar
  experience:

  • V1 Grid Hero — dashboard-style grid with hero metrics card
  • V2 Dock — bottom-docked settings panel with slideout, SVG icon buttons
  • V3 Tabs — top-tabbed setup (Endpoints / Models / Settings) with full
    keyboard-accessible role="tab" ARIA pattern
  • V4 Overlay — full-screen overlay settings with label[role="button"]
    keyboard support

• Six design channels (D1–D6) — complete colour-system variants applied
  per-layout via html[data-design="vX"] + html[data-channel="dY"]:

  • D1 Default (phosphor accent), D2 Instrument (teal/copper), D3 Monitor
    (IBM Plex), D4 Startup (Space Grotesk), D5 Arctic Light (light theme,
    WCAG AA), D6 Terminal (monospace amber)

• Channel selector in sidebar — oscilloscope-style channel picker (D1–D6)
  replaces the old 3-way theme toggle; choice persists in localStorage.

• SQL Compare model filter — universal text filter anchored over the
  Model/Run column in the comparison heatmap bar; live visibility counter,
  MutationObserver-guarded against microtask loops; per-channel focus rings
  and accent borders.

• Alternative visual themes per V-layout — each V1–V4 layout has a second
  standalone CSS theme loaded disabled by default:

  • V1 Grid Hero ↔ Instrument (Fira Code / Fira Sans)
  • V2 Dock ↔ Monitor (IBM Plex Sans / JetBrains Mono)
  • V3 Tabs ↔ Startup (Space Grotesk / DM Sans)
  • V4 Overlay ↔ Terminal (Share Tech Mono / Fira Code)
    Theme switcher button in each V-layout nav persists via localStorage.

• SQL heatmap channel polish — full D5 Arctic Light override block
  (sticky headers, hover states, compare-head, result chips, pass/fail cells
  on white background); per-channel tinted hover + compare-head rows for D1–D4
  and D6; D5 compare-bar visibility fix.

Fixed

• V2 Dock slideout CSP violation — data-dock-action attributes replace
  inline onclick handlers; slideout toggle/open/close bound in app.js.

• V3 Tabs accessibility — label[role="tab"] now syncs aria-selected
  with radio :checked state for screen-reader tab announcements.

• V4 Overlay keyboard — label[role="button"] responds to Enter/Space;
  mobile .grid-viewport inset restored (0 0 60px 0).

• CSS deprecated word-break: break-word replaced with overflow-wrap
  across v1-instrument, v2-dock, v2-monitor, v3-startup, v3-tabs, sql-grid.

• .sr-only rules gain clip-path: inset(50%) alongside legacy
  clip: rect() for modern browser support (v1-instrument, v3-startup).

• WCAG AA contrast — --text-dim raised in v4-terminal.css and
  c1-clean-lab.css (#767676 / #8A8A8A) to meet 4.5:1 on their respective
  backgrounds.

• D5 compare bar — sql-compare-bar background made visible on the light
  Arctic Light surface.

Removed

• A1/A2/A3 and index-themes layout experiments — dead code, never linked
  from production, no nav switcher, no routes. Deleted 7 files
  (index-a1/a2/a3.html, index-themes.html, variants/a1/a2/a3.css), removed
  A2 tab-switching from app.js, cleaned up variants/index.html.

Tests

• Test suite: 217 passed (up from 213).

Verification

• python -m pytest → 217 passed
• create_app() boots and registers routes
• All 6 layout/channel combinations load without CSS/JS errors

v0.4.0

v3 Phosphor default design, oscilloscope-style channel selector, Think column
in SQL matrices, and Gemma thinking-mode tool-call fixes.

Added

• v3 Phosphor design — new default design with phosphor accent palette,
  replacing the previous default. Switchable via a 3-way theme toggle in the
  sidebar. (#54)
• Think column in SQL matrices — SQL run comparison heatmap and model
  grouping now include a Think column showing thinking-mode status per
  question, enabling quick scan of which questions used reasoning. (#56)
• Oscilloscope-style channel selector — the design toggle is replaced with
  an oscilloscope-style channel selector for switching between v1/v2/v3
  designs.

Fixed

• Model header row background — solid dark background across the entire
  model header row in all themes, replacing broken colspan-based layouts.
• Model header hover squares — hover state indicators work correctly in
  both v2 and v3 themes.
• SQL heatmap category border — category border no longer overlaps result
  cells in v3 design.
• Sidebar sticky header overlap — sticky header overlap on scroll fixed in
  v2/v3 themes.
• Compare runs — unanswered questions are no longer treated as errors in
  the comparison view.
• Gemma thinking-mode tool calls — parse <|call:> and bare
  run_sql_query formats from Gemma models in thinking mode, preventing
  TypeError on non-string SQL args. (#55)
• Think column comparison grouping — fixed comparison grouping for
  thinking_mode across SQL matrices. (#56)
• Text-fallback SQL success path — missing assistant message added to
  conversation on text-fallback SQL success. (#52)
• Duplicate SQL detection — TOOL RESULT appended to conversation when
  duplicate SQL is detected.

Tests

• Test suite: 217 passed (up from 213).

v0.3.0

Design toggle, SQL run comparison heatmap, Gemma-style tool-call parsing, and
text-fallback SQL retry improvements.

Added

• Design toggle — ✦ button in the sidebar header switches between the default
  dark-blue theme and a new instrument-palette theme (teal accent, copper secondary,
  Sora for headings, JetBrains Mono for data). Preference persisted in localStorage.
• Sparkline throughput bars — miniature animated bar charts in the speed results
  table showing per-run decode throughput, using teal/copper/red gradient.
• SQL run comparison heatmap — visual heatmap table comparing SQL accuracy across
  runs, with colour-coded cells for fast scanning of pass/fail per question. (#46)
• Model grouping in SQL run comparison heatmap — heatmap rows are grouped by model
  name for easier cross-run comparison. (#48)

Changed

• CSS architecture — all v2 design overrides live in a single
  html[data-design="v2"] block; the default v1 design is untouched.

Fixed

• SQL leading comments break text-only fallback — leading -- comment lines are
  now stripped before the SQL heuristic check. (#45)
• SQL execution errors lost in text-fallback branch — widened to
  (duckdb.Error, ValueError, TypeError); error is reported back as a tool result. (#47)
• stop_reason label clarity — retry and fallback labels now distinguish the two
  paths. (#47)
• Gemma-style <tool_call> blocks not parsed — parser now recognizes Gemma XML
  format alongside existing pipe-prefixed format. (#50)
• Last TOOL RESULT missing from conversation — appended before the retry-limit
  check so the conversation always includes the terminal error. (#50)
• Think-mode wasted retry budget — MAX_NO_TOOL_CALL_RETRIES reduced from 3 to 1
  when thinking_mode="on". (#50)

Tests

• 26 new tests; test suite 213 passed (up from 187).

v0.2.3

chore(v0.2.3): release — run comment, HTTP pooling, SQL watchdog, sco…

v0.2.2

Major refactor — code-review cleanup pass closes an SSRF gap in the SQL path, stops the backend from overriding server-side sampling settings, surfaces the tool-call budget and stop reason in the SQL detail card, adds a no-hardcode version resolver, and removes a pile of dead/misleading code uncovered by review.

⚠️ MAJOR — Heads up before you re-run anything

Sampling parameters (temperature / top_p / presence_penalty / frequency_penalty) are no longer sent in any LLM request.

Previously the backend pushed these values into every OpenAI-compatible payload. On OpenAI-compatible servers the client-sent value replaces the model's registered preset, so whatever you configured in LM Studio / llama.cpp / Ollama was being silently discarded by us. From v0.2.2 the server uses its own preset — that's the source of truth now.

Why the old behaviour was actively harmful for SQL benchmarks:
the SQL tool-calling path was pinned to temperature: 0.1 (hard-coded in call_llm_single and call_llm_tool_calling). That value runs the model in near-deterministic mode. Combined with the tool-calling retry loop (run_question_tool_calling — up to max_retries=3 SQL retries after a DuckDB error, plus the MAX_TOOL_CALLS=10 overall ceiling) the result was:

1. Model generates wrong SQL on attempt 1.
2. DuckDB returns a BinderException / ParserException.
3. We feed the error back and ask for a fix.
4. At temperature=0.1 the model produces the same wrong SQL again — sampling is too tight to escape the local optimum.
5. Repeat until MAX_TOOL_CALLS is hit. Question fails with stop_reason=tool_call_limit even though the model had room to try something different at any other temperature.

What you should do after upgrading:

• Re-run anything you compared between models — earlier "model X can't recover from BinderException" results were partly our fault, not the model's.
• Set your sampling preset on the LLM server side (LM Studio model settings, Ollama Modelfile, llama.cpp launch flags). The backend no longer pushes back.
• If you want strict determinism for reproducibility runs, set temperature=0 in the server preset — but understand the SQL retry loop will degrade exactly the way described above. For benchmarking, temperature=0.7 or whatever the model recommends is usually saner.

Added

• GET /api/version + sidebar version tag — backend resolves the running app's version on first call and caches the result; the UI fetches it at startup and renders it next to the "LLM Testbench" title with a tooltip showing where the string came from.
• .github/workflows/release.yml — on tag push (v*) it writes the release tag into ./VERSION, zips the working tree, and attaches the archive to the GitHub release.
• SQL detail card — Tool-call budget — the meta strip now shows Tool calls: 5 / 10, highlighting the chip in amber at ≥80% of MAX_TOOL_CALLS and red at 100%.
• History table — Questions and Thinking columns — two new columns so you can scan saved runs and tell at a glance which SQL questions were exercised and which thinking variant ran.
• SQL detail card — Stop reason — the backend stop_reason is surfaced as a coloured chip with a plain-English label. Hidden on the happy path (results_ok + success=true).
• CSS — --warn and --danger design tokens added to :root.

Security

• SSRF guard now covers the SQL path — call_llm_single and call_llm_tool_calling were issuing HTTP requests to user-supplied endpoints without the _validate_endpoint_url check that speed-mode runs already enforce.

Fixed

• ⚠️ Stopped overriding server-side sampling parameters — see the heads-up above.
• timeoutMs default was 12000 seconds (~3.3 hours) — fixed to 120 (= 120 s real timeout).
• SQL execution narrowed to except duckdb.Error could still crash the job — widened to (duckdb.Error, ValueError, TypeError).
• Speed benchmark lost all in-progress results on crash — run_sequential and run_parallel now flush after every appended row.
• fallback_completion_tokens reported chunk count, not token count — replaced with explicit None (UI shows n/a).
• Mixed latency_ms and latency_s rows were averaged without conversion — normalised seconds → ms before averaging.
• "Stop failed" misreported as "Start failed" — copy-paste error fixed.
• Speed table colspan="12" against an 11-column table — fixed to 11.
• Dead len(models) < 1 guard in BenchmarkRequest.from_dict — removed.

Changed

• CI actions bumped to Node 24 runtimes — clears the Node.js 20 actions are deprecated warning.
• Test suite: 150 tests passing (142 baseline + 8 new in tests/test_version.py).

v0.2.1

Fixed

• _finalize_tool_run crash on invalid LLM-generated SQL — duckdb.Error (e.g. BinderException for missing GROUP BY) was not caught, crashing the entire benchmark job instead of recording a failed result. (#32)

LLM Testbench v0.2.0

v0.2.0 (2026-06-10)

Major refactor — backend split into focused modules, frontend extracted to separate CSS/JS files, SSRF security hardening, performance optimizations, and numerous UI/UX fixes across speed and SQL views.

Security

• SSRF guard for provider endpoints — _validate_endpoint_url resolves hostnames via socket.getaddrinfo and rejects targets whose resolved IPs fall into multicast/reserved/unspecified ranges. Loopback and RFC 1918 ranges are permitted for local use. Wired into _benchmark_openai, _benchmark_ollama, and _probe_provider.

Added

• tests/test_speed_unit.py — 26 regression tests for aggregates, SSRF guard, OpenAI/Ollama benchmark paths, JSON I/O, phase atomicity, aggregate cache, and save-task draining.

Changed

• Server split into focused modules — python/server.py reduced from 3035 to 168 lines. Core logic extracted into:
  • python/models.py — dataclasses and registries (EndpointCandidate, BenchmarkRequest, RunState, JobState, etc.)
  • python/aggregates.py — per-model speed rollup
  • python/json_io.py — JSON load/read with path-tagged errors
  • python/ssrf.py — endpoint URL private-IP guard
  • python/speed_row.py — uniform row builder for success/failure/stopped
  • python/persistence.py — job save/load/export (538 lines)
  • python/job_runner.py — benchmark orchestration (825 lines)
  • python/benchmark_server.py — HTTP routes and delegate methods (1012 lines)
• Frontend split into static/style.css and static/app.js — index.html shrinks from 3971 to 321 lines. CSS and JS are loaded via <link> and <script> tags.
• Redundant OPTIONS catch-all route removed — aiohttp CORS middleware already handles preflight.
• Persisted aggregated_speed — on-disk job records now include pre-computed aggregates so history opens in aggregated mode immediately.
• set_phase is now atomic — five phase fields written as a single RunState snapshot under asyncio.Lock.
• Per-job save lock + tracked save tasks — serialises snapshot+write per job; parallel jobs don't race.
• _compute_speed_aggregates memoised per JobState — keyed on (len(results), last_result_timestamp). Cache turns O(n) poll walk into O(1).
• _build_speed_row helper — unified row-dict builder, eliminated ~80 lines of duplication.
• Silent except Exception blocks narrowed — all backend modules now catch specific exception types.
• Per-row DOM diff in speed results — Map<rowKey, {row, version}> cache eliminates DOM mutations on unchanged poll ticks.
• Frontend helpers — setStatusBoth, emptyState, getActiveJobId, formatters object, OUTCOME_META map centralise repeated UI patterns.
• Accessibility — aria-live, role="status", aria-busy, aria-label added to status and results elements.
• Poll timer leak fixed — stopPolling() clears pending setTimeout; registered on pagehide + beforeunload.
• Inline onclick migrated to data-action delegation — works under strict CSP.
• Removed dead code — BenchmarkServer._read_jsonl, unused frontend helpers, unused DOM ids, unused imports, empty __init__.py.

Fixed

• Speed decode_tps ~2× lower for reasoning models — TTFT anchor now triggers on first delta.reasoning_content or delta.content, matching LM Studio's eval_count / eval_duration.
• Ollama latency_ms was actually ttft_ms — now latency_ms == total_time_ms for both providers.
• Bare json.loads(...) produced errors with no file name — re-raised as ValueError("Invalid JSON in {name}: ...").
• Incremental saves could be lost on SIGTERM — drain_pending_saves() + BenchmarkServer.shutdown() await every pending save before exit.
• Speed results stay visible during live run — failed-only models now appear in aggregated projection.
• Aggregated/Individual toggle works mid-run — renderResults respects state.speedViewMode without silent fallback.
• Live updates not blocked by open history view — startNextQueuedJob resets activeHistoryJobId on new benchmark.
• Speed results no longer show "—" for missing usage block — falls back to chunk-counting when backend omits usage.
• HTTP 200 with SSE error body is now a hard failure — raises RuntimeError instead of producing phantom passes.
• Parallel mode no longer drops results on cancellation — CancelledError converted to stopped_result.
• Speed results columns aligned — table-layout: fixed, tabular-nums, explicit per-column widths across raw, aggregated, and detail tables.
• History Open race condition fixed — debounce flag, generation counter, deduplication guard.
• Speed view toggle preserves global state — syncSpeedToggleButton() is single source of truth.
• History Close clears speed table — both resultsBody and speedResultsContainer emptied.
• History Open updates status — shows opened job ID and status.
• Raw/aggregated table widths aligned — toggling no longer causes visible width jump.
• Aggregated TTFT in seconds — header and values consistently use seconds.
• Timeout inputs in seconds — labels, defaults, and JS payloads use seconds.
• Number inputs right-aligned — consistent numeric presentation.
• Sidebar visual overhaul — modernised spacing, typography, and interactive states.
• CSS specificity of aggregated/detail columns raised — scoped under #speedResultsContainer > table.speed-aggregated-table and #speedResultsContainer .run-detail-table to beat raw-view rules.
• Expand-toggle button padding fixed — padding: 0 prevents inheriting 6px 12px from generic button rule.

LLM Testbench v0.1.1

What's Changed

• Add repository triage automation setup by @bashrusakh in #1
• Fix/speed benchmark max tokens by @bashrusakh in #2
• fix: add gitattributes for line endings by @bashrusakh in #3
• Feat/speed UI aggregated by @bashrusakh in #4
• Feat/speed UI aggregated by @bashrusakh in #5
• fix(speed): decode t/s for reasoning models, aggregated view, column … by @bashrusakh in #6

New Contributors

• @bashrusakh made their first contribution in #1

Full Changelog: v0.1.0...v0.1.1

LLM Testbench v0.1.0

Removed

• Removed BFCL benchmark support from the backend, UI, fixtures, tests, README, and roadmap.
• Removed BFCL fixture data and the dedicated BFCL adapter module.

Fixed

• Model selection controls are now locked while a benchmark job is queued, running, or stopping.
• SQL result rendering no longer shows empty tool-call analysis blocks for empty results_ok confirmation calls.
• SQL model quantization badges now recognize Unsloth Dynamic GGUF labels such as UD_IQ1_S instead of falling back to an unknown format.

Changed

• Startable benchmark modules are now limited to Speed and SQL Accuracy.
• Fixture validation now covers SQL, Coding Micro, JSON Schema, and Prompt Replay only.
• The README and roadmap now describe the project as a fast, lightweight local benchmark workbench.
• The test badge and development instructions now report 116 passing tests.

Verification

• Test suite: 116 passed.