docs: add end-to-end tutorial with free APIs and local embeddings

examples/end-to-end-tutorial/.gitignore

@@ -0,0 +1,3 @@
+# Generated by running the notebook / `oaeval run config.yaml` — not committed,
+# same convention as the repo root's /reports/ (see ../../.gitignore).
+reports/

examples/end-to-end-tutorial/README.md

@@ -0,0 +1,159 @@
+# End-to-End Tutorial — Free APIs + Local Embeddings
+
+A complete, **executed** walkthrough of the OpenAgent Eval workflow using only free-tier
+services: [Google Gemini](https://aistudio.google.com/apikey) as the LLM and a local
+`sentence-transformers` model for embeddings. No paid API key, no external vector database, no
+GPU. Built for [issue #241](https://github.com/OpenAgentHQ/openagent-eval/issues/241).
+
+Start with [`tutorial.ipynb`](tutorial.ipynb) — this README covers setup, the honest notes from
+actually running it, and troubleshooting.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `tutorial.ipynb` | The 6-section notebook: setup → data → config → run → analyze → improve |
+| `config.yaml` | Evaluation configuration referenced throughout the notebook |
+| `data/sample_qa.json` | 16 hand-authored QA pairs (question, ground truth, context) |
+| `data/corpus.json` | 19 passages indexed by the local retriever (16 relevant + 3 distractors) |
+| `requirements.txt` | Dependencies for this tutorial |
+
+## Setup
+
+### 1. Get a free Gemini API key
+
+1. Go to <https://aistudio.google.com/apikey> and sign in with a Google account.
+2. Click **Create API key**.
+3. Export it in your shell before starting Jupyter:
+   ```bash
+   export GEMINI_API_KEY="your-key-here"
+   ```
+   (Never put the key in `config.yaml` or a notebook cell that gets committed — see
+   "How the API key is handled" below.)
+
+### 2. Install and run
+
+```bash
+cd examples/end-to-end-tutorial
+pip install -r requirements.txt
+jupyter notebook tutorial.ipynb
+```
+
+Or open it directly in Colab via the badge at the top of the notebook (you'll need to enter
+your key with the `getpass` prompt in Section 1, or store it in Colab's Secrets manager under
+the name `GEMINI_API_KEY`).
+
+Works the same way on Windows, macOS, and Linux — everything here is plain Python and YAML, no
+OS-specific paths or shell syntax beyond the one `export` line above (on Windows, use
+`set GEMINI_API_KEY=your-key-here` in `cmd`, or `$env:GEMINI_API_KEY="your-key-here"` in
+PowerShell).
+
+## How the API key is handled
+
+`config.yaml` never sets `llm.api_key`. When it's left unset, OpenAgent Eval's Gemini provider
+falls back to the `GEMINI_API_KEY` environment variable automatically
+(`openagent_eval/providers/llm/gemini.py`):
+
+```python
+resolved_api_key = api_key or os.environ.get("GEMINI_API_KEY")
+```
+
+So the key lives only in your shell/OS environment (or Colab Secrets) — it is never written to
+disk, logged, or committed.
+
+## What was actually run and verified
+
+Everything below was executed for real while building this tutorial, not written from memory:
+
+- **`data/sample_qa.json`'s 16 facts** were each independently verified by running the
+  corresponding `sqlite3` code in this environment before being written down. Two facts that
+  were initially planned — `Connection.backup()` and cross-thread reuse raising
+  `ProgrammingError` — were **dropped** because both hung indefinitely in this sandbox
+  (likely a threading restriction); they were replaced with facts confirmed to actually run
+  (`cursor.description`, `check_same_thread=False`).
+- **`sentence-transformers`** (`all-MiniLM-L6-v2`, CPU) was installed and loads real vectors —
+  confirmed standalone in Section 2 of the notebook (`(2, 384)` float32 output) and again as
+  part of the `memory` retriever during the live run in Section 4.
+- **The full pipeline** (`oaeval run config.yaml`) was executed against the **live Gemini API**:
+  6/6 items succeeded, 0 errors, with real generated answers and real metric scores (see
+  Section 4-5 of the executed notebook for the exact numbers).
+- **`jupyter nbconvert --to notebook --execute`** ran the whole notebook end to end with no
+  errors; its outputs are committed (see "Notebook outputs" below).
+
+## Two things that did NOT work as the issue's own config example assumed
+
+The issue's proposed `config.yaml` used `api_key: ${GROQ_API_KEY}` / `${VAR}`-style
+interpolation. That syntax is **not implemented anywhere in this codebase** — there is no
+`${VAR}` expansion in `openagent_eval/config/loader.py` or elsewhere. Setting a literal
+`${GEMINI_API_KEY}` string as `llm.api_key` would pass that literal (broken) string to the
+provider. The working pattern, used throughout this tutorial, is to **omit `api_key` entirely**
+and let each provider read its own environment variable directly (verified above).
+
+The issue's config also nested embeddings under `retriever.settings.embeddings.provider`. The
+actual schema (`openagent_eval/config/models.py`) is a sibling `retriever.embedder` block, not
+a `settings.embeddings` sub-key — see `config.yaml` in this folder for the verified shape.
+
+## Troubleshooting
+
+**`gemini-2.5-flash` returns a 429 "quota exceeded" error after a handful of calls.** While
+building this tutorial, `gemini-2.5-flash` hit a **free-tier daily quota of 20 requests per
+project** (`quotaId: GenerateRequestsPerDayPerProjectPerModel-FreeTier`) — a stricter limit than
+the commonly-quoted 15 requests/*minute* figure, and one that resets daily rather than
+per-minute. `gemini-2.5-flash-lite` (used in `config.yaml`) had independent quota headroom and
+is what this tutorial actually runs against. If you hit a 429 on whichever model you're using,
+either wait for the daily reset or switch to a different Gemini model / provider (Groq's free
+tier is a good alternative — see the "Other providers" note in Section 3 of the notebook).
+
+**`gemini-2.5-flash` intermittently returns `503 UNAVAILABLE: high demand`.** Observed under
+`parallel: true` (the default) with 4 concurrent requests. `config.yaml` sets `parallel: false`
+so requests go out one at a time — slower, but far more reliable on the free tier.
+
+**`context_precision` / `context_recall` / `mrr` all read `0.0`.** These metrics compare
+*retrieved* contexts against `ground_truth_contexts` (a **list** field) — not the singular
+`context` field also present on each dataset item. `data/sample_qa.json` sets both;
+if you add your own QA pairs, make sure to populate `ground_truth_contexts` too.
+
+**`sentence-transformers` import/first-load is slow (10-20s or more).** That's normal —
+it downloads and loads `all-MiniLM-L6-v2` (~90MB) on first use and caches it under
+`~/.cache/huggingface/hub/` afterwards. Subsequent runs are much faster.
+
+**`ModuleNotFoundError` for `openai`/`anthropic`/`groq`.** The retriever/LLM provider factory
+imports several provider SDKs unconditionally, so `pip install openagent-eval` alone is not
+enough — install with the `providers` extra: `pip install "openagent-eval[providers]"` (already
+in `requirements.txt`).
+
+## Notebook outputs
+
+The committed `tutorial.ipynb` has **executed outputs**, matching the precedent set by
+[`examples/openagent_eval_colab_tutorial.ipynb`](../openagent_eval_colab_tutorial.ipynb) (PR
+#226), which also ships with real, executed output rather than a blank notebook — outputs are
+what make "this actually runs" checkable without re-running it yourself. Absolute paths and the
+API key were confirmed absent before committing (see the PR description for the verification
+commands). One cell's animated progress-bar output (hundreds of near-duplicate spinner frames
+from the live 6-call Gemini run) was collapsed to its header and final summary line, to keep the
+diff readable — no content was fabricated, only repeated terminal-redraw frames were trimmed.
+
+Generated evaluation reports (`reports/*.json`, `reports/*.md`) are **not** committed — they are
+regenerated every time you run the notebook, the same convention the repo root already uses for
+its own `/reports/` directory (see the local `.gitignore` in this folder).
+
+## Note for maintainers: `config.yaml` needed a forced add
+
+The repo root `.gitignore` has a blanket, unanchored `config.yaml` rule (and `config.*.yaml`) —
+originally added to stop contributors' local eval configs from being committed by accident. It
+also silently matches `examples/end-to-end-tutorial/config.yaml`, which the issue explicitly
+asks for by that exact name. This file was added with `git add -f`, so it is intentionally
+tracked despite being `.gitignore`d; anyone running a blanket `git add -A`/`git add .` in this
+directory later won't accidentally re-add or drop it, since it's already tracked, but `git
+status` conventions that assume "ignored == not a real file" may find that surprising. Worth a
+follow-up: either carve an exception for `examples/**/config.yaml` in the root `.gitignore`, or
+rename this file if that's preferred.
+
+## Scope note on the issue's acceptance criteria
+
+Every item in [issue #241](https://github.com/OpenAgentHQ/openagent-eval/issues/241)'s
+requirements is met, with one deliberate scoping choice: the live-executed run in Section 4
+evaluates **6 of the 16** dataset items (`dataset.limit: 6` in `config.yaml`), not all 16, to
+stay comfortably inside the Gemini free tier's per-project daily quota discovered above. The
+full 16-item dataset is present, valid, and ready to run — delete or raise `dataset.limit` to
+evaluate all of it; expect that to take several minutes and to use closer to the daily quota.

examples/end-to-end-tutorial/config.yaml

@@ -0,0 +1,55 @@
+# config.yaml — free-tier, local-embeddings example for the end-to-end tutorial.
+#
+# LLM: Google Gemini (free tier: 15 requests/minute, 1M tokens/day). Get a key
+# at https://aistudio.google.com/apikey and export it before running:
+#   export GEMINI_API_KEY="your-key-here"
+#
+# `api_key` is intentionally NOT set below. When it is omitted, the Gemini
+# provider falls back to the GEMINI_API_KEY environment variable on its own
+# (see openagent_eval/providers/llm/gemini.py) — never put a real key in this
+# file or commit one.
+
+dataset:
+  path: data/sample_qa.json
+  format: json
+  limit: 6   # keep the live Gemini call count small and friendly to the free tier;
+             # drop this line (or raise it) to evaluate the full 16-item dataset
+
+llm:
+  provider: gemini
+  model: gemini-2.5-flash-lite   # free tier; see README for why not gemini-2.5-flash
+  temperature: 0.0
+
+retriever:
+  provider: memory       # process-local, no external vector DB needed
+  settings:
+    documents_path: data/corpus.json
+    k: 3
+  embedder:
+    provider: sentence_transformers   # local, free, CPU-only
+    model: all-MiniLM-L6-v2
+
+metrics:
+  retrieval:
+    - context_precision
+    - context_recall
+    - mrr
+  generation:
+    - faithfulness
+    - answer_relevancy
+    - exact_match
+    - f1_score
+  performance:
+    - latency
+  cost:
+    - token_count
+
+report:
+  output: markdown
+  output_dir: reports
+
+# Sequential, not parallel: the free Gemini tier caps out at 15 requests per
+# minute, and concurrent requests were observed to trip transient 503s under
+# load in this environment. One request in flight at a time keeps every call
+# well under the limit for a 16-item dataset like this one.
+parallel: false

examples/end-to-end-tutorial/data/corpus.json

@@ -0,0 +1,78 @@
+[
+  {
+    "id": "sqlite3-connect",
+    "content": "The sqlite3 module's connect() function opens a connection to an SQLite database file, creating it if it does not already exist. Calling sqlite3.connect(':memory:') instead opens a temporary, private database that lives only in RAM for the lifetime of that connection and disappears when it is closed."
+  },
+  {
+    "id": "sqlite3-cursor",
+    "content": "A Cursor object, created with connection.cursor(), is used to run SQL statements via cursor.execute(sql, parameters) and to read back results afterwards with cursor.fetchone() (one row), cursor.fetchmany(size) (a batch), or cursor.fetchall() (every remaining row)."
+  },
+  {
+    "id": "sqlite3-params",
+    "content": "sqlite3 supports parameterized queries using '?' placeholders, for example cursor.execute('SELECT * FROM users WHERE id = ?', (user_id,)). Values passed this way are sent separately from the SQL text, so untrusted input can never be interpreted as SQL syntax, which is what prevents SQL injection."
+  },
+  {
+    "id": "sqlite3-commit",
+    "content": "Changes written to an SQLite database are not persisted until connection.commit() is called, and connection.rollback() discards any changes made since the last commit. Using the connection itself as a context manager, 'with connection:', automatically commits the block on success or rolls it back if an exception propagates out of the block."
+  },
+  {
+    "id": "sqlite3-executemany",
+    "content": "cursor.executemany(sql, seq_of_parameters) runs the same parameterized SQL statement once for every tuple in seq_of_parameters. This is the recommended way to perform a bulk insert, since it avoids the overhead of calling cursor.execute() separately in a Python loop."
+  },
+  {
+    "id": "sqlite3-row-factory",
+    "content": "Setting connection.row_factory = sqlite3.Row changes query results so each row supports both dictionary-style access by column name (row['name']) and the usual tuple-style access by position (row[0]), without changing anything else about how queries are written."
+  },
+  {
+    "id": "sqlite3-integrity-error",
+    "content": "sqlite3.IntegrityError is raised when an SQL statement would violate a database constraint, for example inserting a value that already exists in a column declared UNIQUE or as a PRIMARY KEY. The insert or update that triggered the violation is rejected and no row is written."
+  },
+  {
+    "id": "sqlite3-isolation-level",
+    "content": "The Connection object's isolation_level attribute controls sqlite3's implicit transaction handling. Setting isolation_level=None switches the connection to autocommit mode, so statements take effect immediately and the connection never reports being inside an open transaction, unlike the default deferred-transaction behavior."
+  },
+  {
+    "id": "sqlite3-foreign-keys",
+    "content": "SQLite does not enforce foreign key constraints by default; a fresh sqlite3 connection reports 'PRAGMA foreign_keys' as 0 (off). Enforcement must be turned on explicitly per connection with the statement 'PRAGMA foreign_keys = ON', because it was left disabled by default for backward compatibility with older databases."
+  },
+  {
+    "id": "sqlite3-executescript",
+    "content": "cursor.executescript(sql_script) executes multiple semicolon-separated SQL statements in a single call, which makes it convenient for running a whole schema definition at once, such as several CREATE TABLE statements followed by seed INSERT statements."
+  },
+  {
+    "id": "sqlite3-lastrowid",
+    "content": "After a single-row INSERT run through cursor.execute(), cursor.lastrowid holds the integer row id that SQLite assigned to the row that was just inserted. This makes it easy to obtain the primary key of a newly created record without issuing a separate SELECT statement."
+  },
+  {
+    "id": "sqlite3-check-same-thread",
+    "content": "By default, a sqlite3 Connection object may only be used by the thread that created it; sqlite3.connect() accepts check_same_thread=False to disable that safety check when the caller has another way of guaranteeing the connection is not accessed concurrently from multiple threads."
+  },
+  {
+    "id": "sqlite3-cursor-description",
+    "content": "After a SELECT statement has been executed, cursor.description holds a tuple of 7-element tuples, one per selected column; the first element of each is the column's name. This is a convenient way to read back column names without hard-coding them, for example when printing a result table."
+  },
+  {
+    "id": "sqlite3-context-manager",
+    "content": "Opening a connection with 'with sqlite3.connect(...) as conn:' does not close the connection when the block exits — the with-block only commits the transaction on success or rolls it back on an exception. The connection stays open afterwards, so conn.close() must still be called explicitly to release it."
+  },
+  {
+    "id": "sqlite3-vacuum",
+    "content": "Running the SQL command 'VACUUM' rebuilds the entire database file, repacking it into the minimum amount of disk space and defragmenting it. This is typically used to reclaim space after a large number of rows have been deleted, since SQLite does not automatically shrink the file on every delete."
+  },
+  {
+    "id": "sqlite3-detect-types",
+    "content": "Passing detect_types=sqlite3.PARSE_DECLTYPES to sqlite3.connect() makes the module automatically convert column values into richer Python types, such as datetime.date, based on the declared column type in the CREATE TABLE statement. As of Python 3.12 the built-in default date/datetime adapters and converters that make this work are deprecated in favor of registering explicit adapters."
+  },
+  {
+    "id": "python-csv-distractor",
+    "content": "The csv module's csv.reader() and csv.writer() functions handle reading and writing comma-separated value files, including quoting and delimiter edge cases, without requiring the caller to hand-parse each line."
+  },
+  {
+    "id": "python-argparse-distractor",
+    "content": "The argparse module builds command-line interfaces by declaring arguments with parser.add_argument() and then calling parser.parse_args(), which also auto-generates a --help message from the declared arguments."
+  },
+  {
+    "id": "python-json-distractor",
+    "content": "The json module's json.dumps() and json.loads() functions convert between Python objects and JSON-formatted strings, and json.dump()/json.load() do the same directly against a file object."
+  }
+]