diff --git a/examples/end-to-end-tutorial/.gitignore b/examples/end-to-end-tutorial/.gitignore new file mode 100644 index 0000000..06591e2 --- /dev/null +++ b/examples/end-to-end-tutorial/.gitignore @@ -0,0 +1,3 @@ +# Generated by running the notebook / `oaeval run config.yaml` — not committed, +# same convention as the repo root's /reports/ (see ../../.gitignore). +reports/ diff --git a/examples/end-to-end-tutorial/README.md b/examples/end-to-end-tutorial/README.md new file mode 100644 index 0000000..84ec0e4 --- /dev/null +++ b/examples/end-to-end-tutorial/README.md @@ -0,0 +1,159 @@ +# End-to-End Tutorial — Free APIs + Local Embeddings + +A complete, **executed** walkthrough of the OpenAgent Eval workflow using only free-tier +services: [Google Gemini](https://aistudio.google.com/apikey) as the LLM and a local +`sentence-transformers` model for embeddings. No paid API key, no external vector database, no +GPU. Built for [issue #241](https://github.com/OpenAgentHQ/openagent-eval/issues/241). + +Start with [`tutorial.ipynb`](tutorial.ipynb) — this README covers setup, the honest notes from +actually running it, and troubleshooting. + +## Files + +| File | Purpose | +|---|---| +| `tutorial.ipynb` | The 6-section notebook: setup → data → config → run → analyze → improve | +| `config.yaml` | Evaluation configuration referenced throughout the notebook | +| `data/sample_qa.json` | 16 hand-authored QA pairs (question, ground truth, context) | +| `data/corpus.json` | 19 passages indexed by the local retriever (16 relevant + 3 distractors) | +| `requirements.txt` | Dependencies for this tutorial | + +## Setup + +### 1. Get a free Gemini API key + +1. Go to and sign in with a Google account. +2. Click **Create API key**. +3. Export it in your shell before starting Jupyter: + ```bash + export GEMINI_API_KEY="your-key-here" + ``` + (Never put the key in `config.yaml` or a notebook cell that gets committed — see + "How the API key is handled" below.) + +### 2. Install and run + +```bash +cd examples/end-to-end-tutorial +pip install -r requirements.txt +jupyter notebook tutorial.ipynb +``` + +Or open it directly in Colab via the badge at the top of the notebook (you'll need to enter +your key with the `getpass` prompt in Section 1, or store it in Colab's Secrets manager under +the name `GEMINI_API_KEY`). + +Works the same way on Windows, macOS, and Linux — everything here is plain Python and YAML, no +OS-specific paths or shell syntax beyond the one `export` line above (on Windows, use +`set GEMINI_API_KEY=your-key-here` in `cmd`, or `$env:GEMINI_API_KEY="your-key-here"` in +PowerShell). + +## How the API key is handled + +`config.yaml` never sets `llm.api_key`. When it's left unset, OpenAgent Eval's Gemini provider +falls back to the `GEMINI_API_KEY` environment variable automatically +(`openagent_eval/providers/llm/gemini.py`): + +```python +resolved_api_key = api_key or os.environ.get("GEMINI_API_KEY") +``` + +So the key lives only in your shell/OS environment (or Colab Secrets) — it is never written to +disk, logged, or committed. + +## What was actually run and verified + +Everything below was executed for real while building this tutorial, not written from memory: + +- **`data/sample_qa.json`'s 16 facts** were each independently verified by running the + corresponding `sqlite3` code in this environment before being written down. Two facts that + were initially planned — `Connection.backup()` and cross-thread reuse raising + `ProgrammingError` — were **dropped** because both hung indefinitely in this sandbox + (likely a threading restriction); they were replaced with facts confirmed to actually run + (`cursor.description`, `check_same_thread=False`). +- **`sentence-transformers`** (`all-MiniLM-L6-v2`, CPU) was installed and loads real vectors — + confirmed standalone in Section 2 of the notebook (`(2, 384)` float32 output) and again as + part of the `memory` retriever during the live run in Section 4. +- **The full pipeline** (`oaeval run config.yaml`) was executed against the **live Gemini API**: + 6/6 items succeeded, 0 errors, with real generated answers and real metric scores (see + Section 4-5 of the executed notebook for the exact numbers). +- **`jupyter nbconvert --to notebook --execute`** ran the whole notebook end to end with no + errors; its outputs are committed (see "Notebook outputs" below). + +## Two things that did NOT work as the issue's own config example assumed + +The issue's proposed `config.yaml` used `api_key: ${GROQ_API_KEY}` / `${VAR}`-style +interpolation. That syntax is **not implemented anywhere in this codebase** — there is no +`${VAR}` expansion in `openagent_eval/config/loader.py` or elsewhere. Setting a literal +`${GEMINI_API_KEY}` string as `llm.api_key` would pass that literal (broken) string to the +provider. The working pattern, used throughout this tutorial, is to **omit `api_key` entirely** +and let each provider read its own environment variable directly (verified above). + +The issue's config also nested embeddings under `retriever.settings.embeddings.provider`. The +actual schema (`openagent_eval/config/models.py`) is a sibling `retriever.embedder` block, not +a `settings.embeddings` sub-key — see `config.yaml` in this folder for the verified shape. + +## Troubleshooting + +**`gemini-2.5-flash` returns a 429 "quota exceeded" error after a handful of calls.** While +building this tutorial, `gemini-2.5-flash` hit a **free-tier daily quota of 20 requests per +project** (`quotaId: GenerateRequestsPerDayPerProjectPerModel-FreeTier`) — a stricter limit than +the commonly-quoted 15 requests/*minute* figure, and one that resets daily rather than +per-minute. `gemini-2.5-flash-lite` (used in `config.yaml`) had independent quota headroom and +is what this tutorial actually runs against. If you hit a 429 on whichever model you're using, +either wait for the daily reset or switch to a different Gemini model / provider (Groq's free +tier is a good alternative — see the "Other providers" note in Section 3 of the notebook). + +**`gemini-2.5-flash` intermittently returns `503 UNAVAILABLE: high demand`.** Observed under +`parallel: true` (the default) with 4 concurrent requests. `config.yaml` sets `parallel: false` +so requests go out one at a time — slower, but far more reliable on the free tier. + +**`context_precision` / `context_recall` / `mrr` all read `0.0`.** These metrics compare +*retrieved* contexts against `ground_truth_contexts` (a **list** field) — not the singular +`context` field also present on each dataset item. `data/sample_qa.json` sets both; +if you add your own QA pairs, make sure to populate `ground_truth_contexts` too. + +**`sentence-transformers` import/first-load is slow (10-20s or more).** That's normal — +it downloads and loads `all-MiniLM-L6-v2` (~90MB) on first use and caches it under +`~/.cache/huggingface/hub/` afterwards. Subsequent runs are much faster. + +**`ModuleNotFoundError` for `openai`/`anthropic`/`groq`.** The retriever/LLM provider factory +imports several provider SDKs unconditionally, so `pip install openagent-eval` alone is not +enough — install with the `providers` extra: `pip install "openagent-eval[providers]"` (already +in `requirements.txt`). + +## Notebook outputs + +The committed `tutorial.ipynb` has **executed outputs**, matching the precedent set by +[`examples/openagent_eval_colab_tutorial.ipynb`](../openagent_eval_colab_tutorial.ipynb) (PR +#226), which also ships with real, executed output rather than a blank notebook — outputs are +what make "this actually runs" checkable without re-running it yourself. Absolute paths and the +API key were confirmed absent before committing (see the PR description for the verification +commands). One cell's animated progress-bar output (hundreds of near-duplicate spinner frames +from the live 6-call Gemini run) was collapsed to its header and final summary line, to keep the +diff readable — no content was fabricated, only repeated terminal-redraw frames were trimmed. + +Generated evaluation reports (`reports/*.json`, `reports/*.md`) are **not** committed — they are +regenerated every time you run the notebook, the same convention the repo root already uses for +its own `/reports/` directory (see the local `.gitignore` in this folder). + +## Note for maintainers: `config.yaml` needed a forced add + +The repo root `.gitignore` has a blanket, unanchored `config.yaml` rule (and `config.*.yaml`) — +originally added to stop contributors' local eval configs from being committed by accident. It +also silently matches `examples/end-to-end-tutorial/config.yaml`, which the issue explicitly +asks for by that exact name. This file was added with `git add -f`, so it is intentionally +tracked despite being `.gitignore`d; anyone running a blanket `git add -A`/`git add .` in this +directory later won't accidentally re-add or drop it, since it's already tracked, but `git +status` conventions that assume "ignored == not a real file" may find that surprising. Worth a +follow-up: either carve an exception for `examples/**/config.yaml` in the root `.gitignore`, or +rename this file if that's preferred. + +## Scope note on the issue's acceptance criteria + +Every item in [issue #241](https://github.com/OpenAgentHQ/openagent-eval/issues/241)'s +requirements is met, with one deliberate scoping choice: the live-executed run in Section 4 +evaluates **6 of the 16** dataset items (`dataset.limit: 6` in `config.yaml`), not all 16, to +stay comfortably inside the Gemini free tier's per-project daily quota discovered above. The +full 16-item dataset is present, valid, and ready to run — delete or raise `dataset.limit` to +evaluate all of it; expect that to take several minutes and to use closer to the daily quota. diff --git a/examples/end-to-end-tutorial/config.yaml b/examples/end-to-end-tutorial/config.yaml new file mode 100644 index 0000000..52041ef --- /dev/null +++ b/examples/end-to-end-tutorial/config.yaml @@ -0,0 +1,55 @@ +# config.yaml — free-tier, local-embeddings example for the end-to-end tutorial. +# +# LLM: Google Gemini (free tier: 15 requests/minute, 1M tokens/day). Get a key +# at https://aistudio.google.com/apikey and export it before running: +# export GEMINI_API_KEY="your-key-here" +# +# `api_key` is intentionally NOT set below. When it is omitted, the Gemini +# provider falls back to the GEMINI_API_KEY environment variable on its own +# (see openagent_eval/providers/llm/gemini.py) — never put a real key in this +# file or commit one. + +dataset: + path: data/sample_qa.json + format: json + limit: 6 # keep the live Gemini call count small and friendly to the free tier; + # drop this line (or raise it) to evaluate the full 16-item dataset + +llm: + provider: gemini + model: gemini-2.5-flash-lite # free tier; see README for why not gemini-2.5-flash + temperature: 0.0 + +retriever: + provider: memory # process-local, no external vector DB needed + settings: + documents_path: data/corpus.json + k: 3 + embedder: + provider: sentence_transformers # local, free, CPU-only + model: all-MiniLM-L6-v2 + +metrics: + retrieval: + - context_precision + - context_recall + - mrr + generation: + - faithfulness + - answer_relevancy + - exact_match + - f1_score + performance: + - latency + cost: + - token_count + +report: + output: markdown + output_dir: reports + +# Sequential, not parallel: the free Gemini tier caps out at 15 requests per +# minute, and concurrent requests were observed to trip transient 503s under +# load in this environment. One request in flight at a time keeps every call +# well under the limit for a 16-item dataset like this one. +parallel: false diff --git a/examples/end-to-end-tutorial/data/corpus.json b/examples/end-to-end-tutorial/data/corpus.json new file mode 100644 index 0000000..38b2fed --- /dev/null +++ b/examples/end-to-end-tutorial/data/corpus.json @@ -0,0 +1,78 @@ +[ + { + "id": "sqlite3-connect", + "content": "The sqlite3 module's connect() function opens a connection to an SQLite database file, creating it if it does not already exist. Calling sqlite3.connect(':memory:') instead opens a temporary, private database that lives only in RAM for the lifetime of that connection and disappears when it is closed." + }, + { + "id": "sqlite3-cursor", + "content": "A Cursor object, created with connection.cursor(), is used to run SQL statements via cursor.execute(sql, parameters) and to read back results afterwards with cursor.fetchone() (one row), cursor.fetchmany(size) (a batch), or cursor.fetchall() (every remaining row)." + }, + { + "id": "sqlite3-params", + "content": "sqlite3 supports parameterized queries using '?' placeholders, for example cursor.execute('SELECT * FROM users WHERE id = ?', (user_id,)). Values passed this way are sent separately from the SQL text, so untrusted input can never be interpreted as SQL syntax, which is what prevents SQL injection." + }, + { + "id": "sqlite3-commit", + "content": "Changes written to an SQLite database are not persisted until connection.commit() is called, and connection.rollback() discards any changes made since the last commit. Using the connection itself as a context manager, 'with connection:', automatically commits the block on success or rolls it back if an exception propagates out of the block." + }, + { + "id": "sqlite3-executemany", + "content": "cursor.executemany(sql, seq_of_parameters) runs the same parameterized SQL statement once for every tuple in seq_of_parameters. This is the recommended way to perform a bulk insert, since it avoids the overhead of calling cursor.execute() separately in a Python loop." + }, + { + "id": "sqlite3-row-factory", + "content": "Setting connection.row_factory = sqlite3.Row changes query results so each row supports both dictionary-style access by column name (row['name']) and the usual tuple-style access by position (row[0]), without changing anything else about how queries are written." + }, + { + "id": "sqlite3-integrity-error", + "content": "sqlite3.IntegrityError is raised when an SQL statement would violate a database constraint, for example inserting a value that already exists in a column declared UNIQUE or as a PRIMARY KEY. The insert or update that triggered the violation is rejected and no row is written." + }, + { + "id": "sqlite3-isolation-level", + "content": "The Connection object's isolation_level attribute controls sqlite3's implicit transaction handling. Setting isolation_level=None switches the connection to autocommit mode, so statements take effect immediately and the connection never reports being inside an open transaction, unlike the default deferred-transaction behavior." + }, + { + "id": "sqlite3-foreign-keys", + "content": "SQLite does not enforce foreign key constraints by default; a fresh sqlite3 connection reports 'PRAGMA foreign_keys' as 0 (off). Enforcement must be turned on explicitly per connection with the statement 'PRAGMA foreign_keys = ON', because it was left disabled by default for backward compatibility with older databases." + }, + { + "id": "sqlite3-executescript", + "content": "cursor.executescript(sql_script) executes multiple semicolon-separated SQL statements in a single call, which makes it convenient for running a whole schema definition at once, such as several CREATE TABLE statements followed by seed INSERT statements." + }, + { + "id": "sqlite3-lastrowid", + "content": "After a single-row INSERT run through cursor.execute(), cursor.lastrowid holds the integer row id that SQLite assigned to the row that was just inserted. This makes it easy to obtain the primary key of a newly created record without issuing a separate SELECT statement." + }, + { + "id": "sqlite3-check-same-thread", + "content": "By default, a sqlite3 Connection object may only be used by the thread that created it; sqlite3.connect() accepts check_same_thread=False to disable that safety check when the caller has another way of guaranteeing the connection is not accessed concurrently from multiple threads." + }, + { + "id": "sqlite3-cursor-description", + "content": "After a SELECT statement has been executed, cursor.description holds a tuple of 7-element tuples, one per selected column; the first element of each is the column's name. This is a convenient way to read back column names without hard-coding them, for example when printing a result table." + }, + { + "id": "sqlite3-context-manager", + "content": "Opening a connection with 'with sqlite3.connect(...) as conn:' does not close the connection when the block exits — the with-block only commits the transaction on success or rolls it back on an exception. The connection stays open afterwards, so conn.close() must still be called explicitly to release it." + }, + { + "id": "sqlite3-vacuum", + "content": "Running the SQL command 'VACUUM' rebuilds the entire database file, repacking it into the minimum amount of disk space and defragmenting it. This is typically used to reclaim space after a large number of rows have been deleted, since SQLite does not automatically shrink the file on every delete." + }, + { + "id": "sqlite3-detect-types", + "content": "Passing detect_types=sqlite3.PARSE_DECLTYPES to sqlite3.connect() makes the module automatically convert column values into richer Python types, such as datetime.date, based on the declared column type in the CREATE TABLE statement. As of Python 3.12 the built-in default date/datetime adapters and converters that make this work are deprecated in favor of registering explicit adapters." + }, + { + "id": "python-csv-distractor", + "content": "The csv module's csv.reader() and csv.writer() functions handle reading and writing comma-separated value files, including quoting and delimiter edge cases, without requiring the caller to hand-parse each line." + }, + { + "id": "python-argparse-distractor", + "content": "The argparse module builds command-line interfaces by declaring arguments with parser.add_argument() and then calling parser.parse_args(), which also auto-generates a --help message from the declared arguments." + }, + { + "id": "python-json-distractor", + "content": "The json module's json.dumps() and json.loads() functions convert between Python objects and JSON-formatted strings, and json.dump()/json.load() do the same directly against a file object." + } +] diff --git a/examples/end-to-end-tutorial/data/sample_qa.json b/examples/end-to-end-tutorial/data/sample_qa.json new file mode 100644 index 0000000..df85c9c --- /dev/null +++ b/examples/end-to-end-tutorial/data/sample_qa.json @@ -0,0 +1,178 @@ +[ + { + "question": "How do I open a temporary, in-memory-only SQLite database in Python?", + "ground_truth": "Call sqlite3.connect(':memory:'), which opens a private database that lives only in RAM and disappears when the connection is closed.", + "context": "The sqlite3 module's connect() function opens a connection to an SQLite database file, creating it if it does not already exist. Calling sqlite3.connect(':memory:') instead opens a temporary, private database that lives only in RAM for the lifetime of that connection and disappears when it is closed.", + "metadata": { + "source_id": "sqlite3-connect" + }, + "ground_truth_contexts": [ + "The sqlite3 module's connect() function opens a connection to an SQLite database file, creating it if it does not already exist. Calling sqlite3.connect(':memory:') instead opens a temporary, private database that lives only in RAM for the lifetime of that connection and disappears when it is closed." + ] + }, + { + "question": "What Cursor methods do I use to read back rows after running a SELECT?", + "ground_truth": "cursor.fetchone() for a single row, cursor.fetchmany(size) for a batch, or cursor.fetchall() for every remaining row.", + "context": "A Cursor object, created with connection.cursor(), is used to run SQL statements via cursor.execute(sql, parameters) and to read back results afterwards with cursor.fetchone() (one row), cursor.fetchmany(size) (a batch), or cursor.fetchall() (every remaining row).", + "metadata": { + "source_id": "sqlite3-cursor" + }, + "ground_truth_contexts": [ + "A Cursor object, created with connection.cursor(), is used to run SQL statements via cursor.execute(sql, parameters) and to read back results afterwards with cursor.fetchone() (one row), cursor.fetchmany(size) (a batch), or cursor.fetchall() (every remaining row)." + ] + }, + { + "question": "Why do '?' placeholders in cursor.execute() protect against SQL injection?", + "ground_truth": "Because parameter values are sent to SQLite separately from the SQL text, so untrusted input can never be interpreted as SQL syntax.", + "context": "sqlite3 supports parameterized queries using '?' placeholders, for example cursor.execute('SELECT * FROM users WHERE id = ?', (user_id,)). Values passed this way are sent separately from the SQL text, so untrusted input can never be interpreted as SQL syntax, which is what prevents SQL injection.", + "metadata": { + "source_id": "sqlite3-params" + }, + "ground_truth_contexts": [ + "sqlite3 supports parameterized queries using '?' placeholders, for example cursor.execute('SELECT * FROM users WHERE id = ?', (user_id,)). Values passed this way are sent separately from the SQL text, so untrusted input can never be interpreted as SQL syntax, which is what prevents SQL injection." + ] + }, + { + "question": "What happens if an exception is raised inside a 'with connection:' block in sqlite3?", + "ground_truth": "The transaction is automatically rolled back; on success it is automatically committed instead.", + "context": "Changes written to an SQLite database are not persisted until connection.commit() is called, and connection.rollback() discards any changes made since the last commit. Using the connection itself as a context manager, 'with connection:', automatically commits the block on success or rolls it back if an exception propagates out of the block.", + "metadata": { + "source_id": "sqlite3-commit" + }, + "ground_truth_contexts": [ + "Changes written to an SQLite database are not persisted until connection.commit() is called, and connection.rollback() discards any changes made since the last commit. Using the connection itself as a context manager, 'with connection:', automatically commits the block on success or rolls it back if an exception propagates out of the block." + ] + }, + { + "question": "What is the recommended way to bulk-insert many rows with sqlite3?", + "ground_truth": "Use cursor.executemany(sql, seq_of_parameters) to run the same parameterized statement once per tuple, rather than looping over cursor.execute().", + "context": "cursor.executemany(sql, seq_of_parameters) runs the same parameterized SQL statement once for every tuple in seq_of_parameters. This is the recommended way to perform a bulk insert, since it avoids the overhead of calling cursor.execute() separately in a Python loop.", + "metadata": { + "source_id": "sqlite3-executemany" + }, + "ground_truth_contexts": [ + "cursor.executemany(sql, seq_of_parameters) runs the same parameterized SQL statement once for every tuple in seq_of_parameters. This is the recommended way to perform a bulk insert, since it avoids the overhead of calling cursor.execute() separately in a Python loop." + ] + }, + { + "question": "What does setting connection.row_factory = sqlite3.Row change about query results?", + "ground_truth": "Rows support dictionary-style access by column name (row['name']) in addition to the usual tuple-style access by position (row[0]).", + "context": "Setting connection.row_factory = sqlite3.Row changes query results so each row supports both dictionary-style access by column name (row['name']) and the usual tuple-style access by position (row[0]), without changing anything else about how queries are written.", + "metadata": { + "source_id": "sqlite3-row-factory" + }, + "ground_truth_contexts": [ + "Setting connection.row_factory = sqlite3.Row changes query results so each row supports both dictionary-style access by column name (row['name']) and the usual tuple-style access by position (row[0]), without changing anything else about how queries are written." + ] + }, + { + "question": "Which exception does sqlite3 raise when an INSERT violates a UNIQUE constraint?", + "ground_truth": "sqlite3.IntegrityError, and the offending row is not written.", + "context": "sqlite3.IntegrityError is raised when an SQL statement would violate a database constraint, for example inserting a value that already exists in a column declared UNIQUE or as a PRIMARY KEY. The insert or update that triggered the violation is rejected and no row is written.", + "metadata": { + "source_id": "sqlite3-integrity-error" + }, + "ground_truth_contexts": [ + "sqlite3.IntegrityError is raised when an SQL statement would violate a database constraint, for example inserting a value that already exists in a column declared UNIQUE or as a PRIMARY KEY. The insert or update that triggered the violation is rejected and no row is written." + ] + }, + { + "question": "How do I put a sqlite3 connection into autocommit mode?", + "ground_truth": "Set isolation_level=None (either at connect() time or on the Connection object), which makes statements take effect immediately with no open transaction.", + "context": "The Connection object's isolation_level attribute controls sqlite3's implicit transaction handling. Setting isolation_level=None switches the connection to autocommit mode, so statements take effect immediately and the connection never reports being inside an open transaction, unlike the default deferred-transaction behavior.", + "metadata": { + "source_id": "sqlite3-isolation-level" + }, + "ground_truth_contexts": [ + "The Connection object's isolation_level attribute controls sqlite3's implicit transaction handling. Setting isolation_level=None switches the connection to autocommit mode, so statements take effect immediately and the connection never reports being inside an open transaction, unlike the default deferred-transaction behavior." + ] + }, + { + "question": "Are foreign key constraints enforced by default in a new sqlite3 connection?", + "ground_truth": "No. 'PRAGMA foreign_keys' reports 0 (off) by default and must be turned on explicitly with 'PRAGMA foreign_keys = ON'.", + "context": "SQLite does not enforce foreign key constraints by default; a fresh sqlite3 connection reports 'PRAGMA foreign_keys' as 0 (off). Enforcement must be turned on explicitly per connection with the statement 'PRAGMA foreign_keys = ON', because it was left disabled by default for backward compatibility with older databases.", + "metadata": { + "source_id": "sqlite3-foreign-keys" + }, + "ground_truth_contexts": [ + "SQLite does not enforce foreign key constraints by default; a fresh sqlite3 connection reports 'PRAGMA foreign_keys' as 0 (off). Enforcement must be turned on explicitly per connection with the statement 'PRAGMA foreign_keys = ON', because it was left disabled by default for backward compatibility with older databases." + ] + }, + { + "question": "How can I run a whole schema of several CREATE TABLE statements in one call?", + "ground_truth": "Pass them all, semicolon-separated, to cursor.executescript(sql_script).", + "context": "cursor.executescript(sql_script) executes multiple semicolon-separated SQL statements in a single call, which makes it convenient for running a whole schema definition at once, such as several CREATE TABLE statements followed by seed INSERT statements.", + "metadata": { + "source_id": "sqlite3-executescript" + }, + "ground_truth_contexts": [ + "cursor.executescript(sql_script) executes multiple semicolon-separated SQL statements in a single call, which makes it convenient for running a whole schema definition at once, such as several CREATE TABLE statements followed by seed INSERT statements." + ] + }, + { + "question": "How do I get the primary key of a row I just inserted with cursor.execute()?", + "ground_truth": "Read cursor.lastrowid immediately after the INSERT; it holds the row id SQLite assigned to that row.", + "context": "After a single-row INSERT run through cursor.execute(), cursor.lastrowid holds the integer row id that SQLite assigned to the row that was just inserted. This makes it easy to obtain the primary key of a newly created record without issuing a separate SELECT statement.", + "metadata": { + "source_id": "sqlite3-lastrowid" + }, + "ground_truth_contexts": [ + "After a single-row INSERT run through cursor.execute(), cursor.lastrowid holds the integer row id that SQLite assigned to the row that was just inserted. This makes it easy to obtain the primary key of a newly created record without issuing a separate SELECT statement." + ] + }, + { + "question": "How do I let a sqlite3 connection be used from a thread other than the one that created it?", + "ground_truth": "Pass check_same_thread=False to sqlite3.connect(), which disables the default same-thread safety check.", + "context": "By default, a sqlite3 Connection object may only be used by the thread that created it; sqlite3.connect() accepts check_same_thread=False to disable that safety check when the caller has another way of guaranteeing the connection is not accessed concurrently from multiple threads.", + "metadata": { + "source_id": "sqlite3-check-same-thread" + }, + "ground_truth_contexts": [ + "By default, a sqlite3 Connection object may only be used by the thread that created it; sqlite3.connect() accepts check_same_thread=False to disable that safety check when the caller has another way of guaranteeing the connection is not accessed concurrently from multiple threads." + ] + }, + { + "question": "How can I read the column names of a SELECT result without hard-coding them?", + "ground_truth": "After executing the SELECT, read cursor.description; the first element of each entry is that column's name.", + "context": "After a SELECT statement has been executed, cursor.description holds a tuple of 7-element tuples, one per selected column; the first element of each is the column's name. This is a convenient way to read back column names without hard-coding them, for example when printing a result table.", + "metadata": { + "source_id": "sqlite3-cursor-description" + }, + "ground_truth_contexts": [ + "After a SELECT statement has been executed, cursor.description holds a tuple of 7-element tuples, one per selected column; the first element of each is the column's name. This is a convenient way to read back column names without hard-coding them, for example when printing a result table." + ] + }, + { + "question": "Does 'with sqlite3.connect(...) as conn:' close the connection when the block exits?", + "ground_truth": "No. The with-block only commits or rolls back the transaction; conn.close() must still be called explicitly to release the connection.", + "context": "Opening a connection with 'with sqlite3.connect(...) as conn:' does not close the connection when the block exits \u2014 the with-block only commits the transaction on success or rolls it back on an exception. The connection stays open afterwards, so conn.close() must still be called explicitly to release it.", + "metadata": { + "source_id": "sqlite3-context-manager" + }, + "ground_truth_contexts": [ + "Opening a connection with 'with sqlite3.connect(...) as conn:' does not close the connection when the block exits \u2014 the with-block only commits the transaction on success or rolls it back on an exception. The connection stays open afterwards, so conn.close() must still be called explicitly to release it." + ] + }, + { + "question": "What does running the SQL command VACUUM do to a SQLite database file?", + "ground_truth": "It rebuilds the whole file, repacking it into the minimum disk space and defragmenting it, which reclaims space left behind after deletes.", + "context": "Running the SQL command 'VACUUM' rebuilds the entire database file, repacking it into the minimum amount of disk space and defragmenting it. This is typically used to reclaim space after a large number of rows have been deleted, since SQLite does not automatically shrink the file on every delete.", + "metadata": { + "source_id": "sqlite3-vacuum" + }, + "ground_truth_contexts": [ + "Running the SQL command 'VACUUM' rebuilds the entire database file, repacking it into the minimum amount of disk space and defragmenting it. This is typically used to reclaim space after a large number of rows have been deleted, since SQLite does not automatically shrink the file on every delete." + ] + }, + { + "question": "What does detect_types=sqlite3.PARSE_DECLTYPES do, and is it still recommended?", + "ground_truth": "It makes sqlite3 automatically convert column values (e.g. to datetime.date) based on the declared column type, but as of Python 3.12 the built-in adapters/converters it relies on are deprecated in favor of registering explicit ones.", + "context": "Passing detect_types=sqlite3.PARSE_DECLTYPES to sqlite3.connect() makes the module automatically convert column values into richer Python types, such as datetime.date, based on the declared column type in the CREATE TABLE statement. As of Python 3.12 the built-in default date/datetime adapters and converters that make this work are deprecated in favor of registering explicit adapters.", + "metadata": { + "source_id": "sqlite3-detect-types" + }, + "ground_truth_contexts": [ + "Passing detect_types=sqlite3.PARSE_DECLTYPES to sqlite3.connect() makes the module automatically convert column values into richer Python types, such as datetime.date, based on the declared column type in the CREATE TABLE statement. As of Python 3.12 the built-in default date/datetime adapters and converters that make this work are deprecated in favor of registering explicit adapters." + ] + } +] diff --git a/examples/end-to-end-tutorial/requirements.txt b/examples/end-to-end-tutorial/requirements.txt new file mode 100644 index 0000000..3dcf68c --- /dev/null +++ b/examples/end-to-end-tutorial/requirements.txt @@ -0,0 +1,9 @@ +# Requirements for the end-to-end tutorial (examples/end-to-end-tutorial/). +# +# openagent-eval[providers] pulls in google-genai (Gemini), plus the openai/ +# anthropic/groq SDKs the provider factory imports unconditionally. +# sentence-transformers is the local, free, CPU-only embedding backend used +# by the tutorial's `retriever.embedder`. +openagent-eval[providers]>=0.4.8 +sentence-transformers>=2.2.0 +jupyter>=1.0.0 diff --git a/examples/end-to-end-tutorial/tutorial.ipynb b/examples/end-to-end-tutorial/tutorial.ipynb new file mode 100644 index 0000000..adc8e5f --- /dev/null +++ b/examples/end-to-end-tutorial/tutorial.ipynb @@ -0,0 +1,1391 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "96adb54f", + "metadata": {}, + "source": [ + "# \ud83e\uddea OpenAgent Eval \u2014 End-to-End Tutorial (Free APIs + Local Embeddings)\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/OpenAgentHQ/openagent-eval/blob/main/examples/end-to-end-tutorial/tutorial.ipynb)\n", + "[![PyPI Version](https://img.shields.io/pypi/v/openagent-eval)](https://pypi.org/project/openagent-eval/)\n", + "\n", + "This tutorial walks through the **full** OpenAgent Eval workflow, end to end, using only\n", + "**free-tier services**: [Google Gemini](https://aistudio.google.com/apikey) as the LLM and a\n", + "**local** `sentence-transformers` model for embeddings \u2014 no vector database, no paid API key,\n", + "no GPU required.\n", + "\n", + "It extends the zero-setup [Colab Quickstart](../openagent_eval_colab_tutorial.ipynb) (which uses\n", + "the built-in offline `mock` providers) toward a **real, free, working RAG pipeline** with\n", + "hand-authored data preparation, as requested in\n", + "[issue #241](https://github.com/OpenAgentHQ/openagent-eval/issues/241).\n", + "\n", + "| Provider | Free tier | Used here |\n", + "|---|---|---|\n", + "| **Google Gemini** | 15 requests/min | \u2705 primary LLM (`llm.provider: gemini`) |\n", + "| Groq | 30 req/min, 14,400 req/day | not used \u2014 see the \"Other providers\" note in Section 3 |\n", + "| OpenAI | $5 trial credit | not used (not truly free) |\n", + "| Ollama | unlimited, local | not used \u2014 this environment does not have Ollama installed |\n", + "| **sentence-transformers** (`all-MiniLM-L6-v2`) | free, local, CPU-only | \u2705 embeddings (`retriever.embedder`) |\n", + "\n", + "**What's in this folder:**\n", + "\n", + "| File | Purpose |\n", + "|---|---|\n", + "| `README.md` | Setup instructions and the honest notes from actually running this |\n", + "| `tutorial.ipynb` | This notebook |\n", + "| `config.yaml` | The evaluation configuration referenced below |\n", + "| `data/sample_qa.json` | 16 hand-authored QA pairs (the eval dataset) |\n", + "| `data/corpus.json` | 19 passages (16 relevant + 3 distractors) indexed by the local retriever |\n", + "| `requirements.txt` | Pinned-loose dependencies for this tutorial |\n", + "\n", + "> \u26a0\ufe0f **Read this before \"Run all\":** Section 4 makes real calls to the Gemini API and needs\n", + "> `GEMINI_API_KEY` set in your environment. Every cell that calls Gemini is guarded \u2014 if the key\n", + "> is missing, the cell explains what to do and does nothing, so *Run all* stays safe. It will\n", + "> just skip the live parts." + ] + }, + { + "cell_type": "markdown", + "id": "a402c91c", + "metadata": {}, + "source": [ + "\n", + "## 1. Setup & Installation\n", + "\n", + "### Get a free Gemini API key\n", + "\n", + "1. Go to and sign in with a Google account.\n", + "2. Click **Create API key**. The free tier is generous: 15 requests/minute is plenty for this\n", + " 16-question dataset.\n", + "3. Export it before starting Jupyter (never paste it directly into a notebook cell that gets\n", + " committed):\n", + " ```bash\n", + " export GEMINI_API_KEY=\"your-key-here\"\n", + " ```\n", + " In Colab, the cell below also offers a `getpass` prompt as a fallback.\n", + "\n", + "### Install the dependencies\n", + "\n", + "`sentence-transformers` pulls in PyTorch (CPU build), so this cell can take a few minutes the\n", + "first time." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "68c6f8c8", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:44:55.535652Z", + "iopub.status.busy": "2026-07-27T17:44:55.534435Z", + "iopub.status.idle": "2026-07-27T17:45:03.881477Z", + "shell.execute_reply": "2026-07-27T17:45:03.870127Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.\u001b[0m\u001b[33m\r\n", + "\u001b[0m" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install -q \"openagent-eval[providers]>=0.4.8\" \"sentence-transformers>=2.2.0\" pytest" + ] + }, + { + "cell_type": "markdown", + "id": "20d5653c", + "metadata": {}, + "source": [ + "Quiet the library's debug logging and confirm the install." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "ce019b91", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:45:03.892187Z", + "iopub.status.busy": "2026-07-27T17:45:03.891783Z", + "iopub.status.idle": "2026-07-27T17:45:03.909324Z", + "shell.execute_reply": "2026-07-27T17:45:03.907902Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Environment ready.\n" + ] + } + ], + "source": [ + "import os\n", + "\n", + "os.environ[\"LOGURU_LEVEL\"] = \"WARNING\"\n", + "print(\"Environment ready.\")" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "27421cef", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:45:03.929341Z", + "iopub.status.busy": "2026-07-27T17:45:03.929105Z", + "iopub.status.idle": "2026-07-27T17:45:07.241336Z", + "shell.execute_reply": "2026-07-27T17:45:07.240207Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "openagent-eval 0.4.8\r\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m - Environment Check\r\n", + "\r\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[3m Environment Status \u001b[0m\r\n", + "\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\r\n", + "\u2503\u001b[1m \u001b[0m\u001b[1mComponent \u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1mStatus\u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1mDetails \u001b[0m\u001b[1m \u001b[0m\u2503\r\n", + "\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mPython \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\u001b[2m \u001b[0m\u001b[2mv3.13.14 \u001b[0m\u001b[2m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mopenagent-eval\u001b[0m\u001b[36m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\u001b[2m \u001b[0m\u001b[2mv0.4.8 \u001b[0m\u001b[2m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mtyper \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\u001b[2m \u001b[0m\u001b[2mCLI framework \u001b[0m\u001b[2m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mrich \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\u001b[2m \u001b[0m\u001b[2mTerminal UI \u001b[0m\u001b[2m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mpydantic \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\u001b[2m \u001b[0m\u001b[2mData validation\u001b[0m\u001b[2m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36myaml \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\u001b[2m \u001b[0m\u001b[2mConfiguration \u001b[0m\u001b[2m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mloguru \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\u001b[2m \u001b[0m\u001b[2mLogging \u001b[0m\u001b[2m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mjinja2 \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\u001b[2m \u001b[0m\u001b[2mHTML templates \u001b[0m\u001b[2m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mhttpx \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\u001b[2m \u001b[0m\u001b[2mHTTP client \u001b[0m\u001b[2m \u001b[0m\u2502\r\n", + "\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n", + "\u001b[3m API Key Availability \u001b[0m\r\n", + "\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\r\n", + "\u2503\u001b[1m \u001b[0m\u001b[1mProvider \u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1mEnvironment Variable\u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1mStatus \u001b[0m\u001b[1m \u001b[0m\u2503\r\n", + "\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mOpenAI \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[33m \u001b[0m\u001b[33mOPENAI_API_KEY \u001b[0m\u001b[33m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;2mNot set\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mGemini \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[33m \u001b[0m\u001b[33mGEMINI_API_KEY \u001b[0m\u001b[33m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mAvailable\u001b[0m\u001b[1m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mAnthropic \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[33m \u001b[0m\u001b[33mANTHROPIC_API_KEY \u001b[0m\u001b[33m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;2mNot set\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mGroq \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[33m \u001b[0m\u001b[33mGROQ_API_KEY \u001b[0m\u001b[33m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;2mNot set\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36mOpenRouter\u001b[0m\u001b[36m \u001b[0m\u2502\u001b[33m \u001b[0m\u001b[33mOPENROUTER_API_KEY \u001b[0m\u001b[33m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;2mNot set\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\r\n", + "\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n", + "\r\n", + "\u001b[1mConfiguration:\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m Found config: config.yaml\r\n", + "\r\n", + "\u001b[1mSummary:\u001b[0m\r\n", + "\u001b[32mOK\u001b[0m Python version is compatible\r\n", + "\u001b[32mOK\u001b[0m Available providers: Gemini\r\n" + ] + } + ], + "source": [ + "!oaeval --version\n", + "!oaeval doctor" + ] + }, + { + "cell_type": "markdown", + "id": "742e24c3", + "metadata": {}, + "source": [ + "### Provide the Gemini API key for this session\n", + "\n", + "Skips gracefully if you don't have one yet \u2014 you can still run Sections 1-3 and 5 (data,\n", + "config, and reading the pre-generated report) without it; only the live-call cells in Section 4\n", + "are skipped." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "e48cdc3b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:45:07.250052Z", + "iopub.status.busy": "2026-07-27T17:45:07.248930Z", + "iopub.status.idle": "2026-07-27T17:45:07.263423Z", + "shell.execute_reply": "2026-07-27T17:45:07.259956Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "GEMINI_API_KEY set: True\n" + ] + } + ], + "source": [ + "import getpass\n", + "import os\n", + "\n", + "if not os.environ.get(\"GEMINI_API_KEY\"):\n", + " try:\n", + " from google.colab import userdata # type: ignore\n", + " key = userdata.get(\"GEMINI_API_KEY\")\n", + " if key:\n", + " os.environ[\"GEMINI_API_KEY\"] = key\n", + " except Exception:\n", + " pass\n", + "\n", + "if not os.environ.get(\"GEMINI_API_KEY\"):\n", + " entered = getpass.getpass(\"Enter your GEMINI_API_KEY (or press Enter to skip): \")\n", + " if entered:\n", + " os.environ[\"GEMINI_API_KEY\"] = entered\n", + "\n", + "print(\"GEMINI_API_KEY set:\", bool(os.environ.get(\"GEMINI_API_KEY\")))" + ] + }, + { + "cell_type": "markdown", + "id": "2416025d", + "metadata": {}, + "source": [ + "\n", + "## 2. Prepare Your Data\n", + "\n", + "The dataset for this tutorial is **hand-authored** (not scraped or copied) so its licence is\n", + "unambiguous. The domain is Python's standard-library **`sqlite3` module** \u2014 a topic small enough\n", + "to write accurately and self-consistently about.\n", + "\n", + "Two files, both under `data/`:\n", + "\n", + "- **`sample_qa.json`** \u2014 16 QA pairs. Each item has `question`, `ground_truth`, `context` (the\n", + " single passage that answers it), and `ground_truth_contexts` (the same passage, as a list \u2014\n", + " the retrieval metrics `context_precision`/`context_recall`/`mrr` compare *retrieved* contexts\n", + " against this list, not the singular `context` field).\n", + "- **`corpus.json`** \u2014 19 passages: the 16 that answer a question above, plus 3 unrelated\n", + " \"distractor\" passages (about `csv`, `argparse`, and `json`) so retrieval has to actually\n", + " discriminate rather than trivially returning everything.\n", + "\n", + "Every fact in `sample_qa.json` was independently verified by *running* the corresponding\n", + "`sqlite3` code in this environment before being written down \u2014 see the PR description for the\n", + "verification transcript. Two initially-planned facts (`Connection.backup()` and cross-thread\n", + "`ProgrammingError`) were **dropped** because they hung this sandbox's threading, and were\n", + "replaced with facts that were confirmed to actually run." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "8ed09023", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:45:07.269678Z", + "iopub.status.busy": "2026-07-27T17:45:07.268767Z", + "iopub.status.idle": "2026-07-27T17:45:07.398910Z", + "shell.execute_reply": "2026-07-27T17:45:07.396069Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded 16 QA pairs and 19 corpus passages.\n", + "\n", + "Sample QA pair:\n", + "{\n", + " \"question\": \"How do I open a temporary, in-memory-only SQLite database in Python?\",\n", + " \"ground_truth\": \"Call sqlite3.connect(':memory:'), which opens a private database that lives only in RAM and disappears when the connection is closed.\",\n", + " \"context\": \"The sqlite3 module's connect() function opens a connection to an SQLite database file, creating it if it does not already exist. Calling sqlite3.connect(':memory:') instead opens a temporary, private database that lives only in RAM for the lifetime of that connection and disappears when it is closed.\",\n", + " \"metadata\": {\n", + " \"source_id\": \"sqlite3-connect\"\n", + " },\n", + " \"ground_truth_contexts\": [\n", + " \"The sqlite3 module's connect() function opens a connection to an SQLite database file, creating it if it does not already exist. Calling sqlite3.connect(':memory:') instead opens a temporary, private database that lives only in RAM for the lifetime of that connection and disappears when it is closed.\"\n", + " ]\n", + "}\n" + ] + } + ], + "source": [ + "import json\n", + "\n", + "with open(\"data/sample_qa.json\") as f:\n", + " qa_pairs = json.load(f)\n", + "with open(\"data/corpus.json\") as f:\n", + " corpus = json.load(f)\n", + "\n", + "print(f\"Loaded {len(qa_pairs)} QA pairs and {len(corpus)} corpus passages.\")\n", + "print()\n", + "print(\"Sample QA pair:\")\n", + "print(json.dumps(qa_pairs[0], indent=2))" + ] + }, + { + "cell_type": "markdown", + "id": "f06a131e", + "metadata": {}, + "source": [ + "### Local embeddings, standalone\n", + "\n", + "Before wiring `sentence-transformers` into the retriever, let's prove it works on its own: load\n", + "the model and embed a couple of the QA questions." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "4a8d2b23", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:45:07.435238Z", + "iopub.status.busy": "2026-07-27T17:45:07.434944Z", + "iopub.status.idle": "2026-07-27T17:45:25.958022Z", + "shell.execute_reply": "2026-07-27T17:45:25.956871Z" + } + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "1a6cd7ead13e4a26b82654f3403c92a5", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Loading weights: 0%| | 0/103 [00:00\n", + "## 3. Configure Evaluation\n", + "\n", + "`config.yaml` wires everything together: the dataset, the Gemini LLM, the local\n", + "`memory` retriever backed by the `sentence_transformers` embedder, and the metrics to compute.\n", + "\n", + "A few things worth calling out:\n", + "\n", + "- **`llm.api_key` is intentionally absent.** When it's omitted, the Gemini provider falls back\n", + " to the `GEMINI_API_KEY` environment variable\n", + " (`openagent_eval/providers/llm/gemini.py`) \u2014 so the key is never written to disk or committed.\n", + "- **`llm.model: gemini-2.5-flash-lite`, not `gemini-2.5-flash`.** While verifying this tutorial,\n", + " `gemini-2.5-flash` hit its **free-tier daily quota of 20 requests per project** (not the 15\n", + " requests/*minute* figure most documentation quotes \u2014 that's a separate, less restrictive\n", + " limit). `gemini-2.5-flash-lite` had independent headroom. See the README's Troubleshooting\n", + " section if you hit this yourself.\n", + "- **`retriever.provider: memory`** is a dependency-light, process-local vector store (cosine\n", + " similarity over NumPy arrays) \u2014 no Chroma/FAISS/Qdrant server required. It embeds\n", + " `data/corpus.json` lazily on first query using whatever `retriever.embedder` you configure.\n", + "- **`dataset.limit: 6`** caps the live run to 6 of the 16 questions, to stay well inside the free\n", + " tier during Section 4. Delete that line (or raise the number) to evaluate the full dataset \u2014\n", + " see the README for guidance on pacing that against the daily quota.\n", + "- **Other providers.** Swapping `llm.provider: gemini` for `groq` (30 req/min free) needs only a\n", + " `GROQ_API_KEY` and a `model:` change \u2014 the rest of this config is provider-agnostic." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "b3654646", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:45:26.015934Z", + "iopub.status.busy": "2026-07-27T17:45:26.015230Z", + "iopub.status.idle": "2026-07-27T17:45:27.073794Z", + "shell.execute_reply": "2026-07-27T17:45:27.033961Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "# config.yaml \u2014 free-tier, local-embeddings example for the end-to-end tutorial.\r\n", + "#\r\n", + "# LLM: Google Gemini (free tier: 15 requests/minute, 1M tokens/day). Get a key\r\n", + "# at https://aistudio.google.com/apikey and export it before running:\r\n", + "# export GEMINI_API_KEY=\"your-key-here\"\r\n", + "#\r\n", + "# `api_key` is intentionally NOT set below. When it is omitted, the Gemini\r\n", + "# provider falls back to the GEMINI_API_KEY environment variable on its own\r\n", + "# (see openagent_eval/providers/llm/gemini.py) \u2014 never put a real key in this\r\n", + "# file or commit one.\r\n", + "\r\n", + "dataset:\r\n", + " path: data/sample_qa.json\r\n", + " format: json\r\n", + " limit: 6 # keep the live Gemini call count small and friendly to the free tier;\r\n", + " # drop this line (or raise it) to evaluate the full 16-item dataset\r\n", + "\r\n", + "llm:\r\n", + " provider: gemini\r\n", + " model: gemini-2.5-flash-lite # free tier; see README for why not gemini-2.5-flash\r\n", + " temperature: 0.0\r\n", + "\r\n", + "retriever:\r\n", + " provider: memory # process-local, no external vector DB needed\r\n", + " settings:\r\n", + " documents_path: data/corpus.json\r\n", + " k: 3\r\n", + " embedder:\r\n", + " provider: sentence_transformers # local, free, CPU-only\r\n", + " model: all-MiniLM-L6-v2\r\n", + "\r\n", + "metrics:\r\n", + " retrieval:\r\n", + " - context_precision\r\n", + " - context_recall\r\n", + " - mrr\r\n", + " generation:\r\n", + " - faithfulness\r\n", + " - answer_relevancy\r\n", + " - exact_match\r\n", + " - f1_score\r\n", + " performance:\r\n", + " - latency\r\n", + " cost:\r\n", + " - token_count\r\n", + "\r\n", + "report:\r\n", + " output: markdown\r\n", + " output_dir: reports\r\n", + "\r\n", + "# Sequential, not parallel: the free Gemini tier caps out at 15 requests per\r\n", + "# minute, and concurrent requests were observed to trip transient 503s under\r\n", + "# load in this environment. One request in flight at a time keeps every call\r\n", + "# well under the limit for a 16-item dataset like this one.\r\n", + "parallel: false\r\n" + ] + } + ], + "source": [ + "!cat config.yaml" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "47b81b87", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:45:27.091657Z", + "iopub.status.busy": "2026-07-27T17:45:27.089455Z", + "iopub.status.idle": "2026-07-27T17:45:28.182825Z", + "shell.execute_reply": "2026-07-27T17:45:28.180640Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m - Configuration Validator\r\n", + "\u001b[2mConfig: config.yaml\u001b[0m\r\n", + "\r\n", + "\u001b[1;36m1\u001b[0m\u001b[1m. Checking YAML syntax\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m YAML syntax valid\r\n", + "\r\n", + "\u001b[1;36m2\u001b[0m\u001b[1m. Validating configuration schema\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m Configuration schema valid\r\n", + "\r\n", + "\u001b[1;36m3\u001b[0m\u001b[1m. Checking API keys\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m All required API keys configured\r\n", + "\r\n", + "\u001b[1;36m4\u001b[0m\u001b[1m. Checking dataset\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m Dataset found: data/sample_qa.json\r\n", + " \u001b[2mSize: \u001b[0m\u001b[1;2;36m15.2\u001b[0m\u001b[2m KB\u001b[0m\r\n", + "\r\n", + "\u001b[1;36m5\u001b[0m\u001b[1m. Checking output directory\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[32mOK\u001b[0m Output directory exists: reports\r\n", + "\r\n", + "\u001b[1;36m6\u001b[0m\u001b[1m. Checking provider configuration\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[2mLLM: gemini \u001b[0m\u001b[1;2m(\u001b[0m\u001b[2mgemini-\u001b[0m\u001b[1;2;36m2.5\u001b[0m\u001b[2m-flash-lite\u001b[0m\u001b[1;2m)\u001b[0m\r\n", + " \u001b[2mRetriever: memory\u001b[0m\r\n", + "\r\n", + "\u001b[1;36m7\u001b[0m\u001b[1m. Checking metrics\u001b[0m\u001b[1;33m...\u001b[0m\r\n", + " \u001b[2mConfigured: \u001b[0m\u001b[1;2;36m9\u001b[0m\u001b[2m metrics\u001b[0m\r\n", + " \u001b[2mRetrieval: context_precision, context_recall, mrr\u001b[0m\r\n", + " \u001b[2mGeneration: faithfulness, answer_relevancy, exact_match, f1_score\u001b[0m\r\n", + " \u001b[2mPerformance: latency\u001b[0m\r\n", + " \u001b[2mCost: token_count\u001b[0m\r\n", + "\r\n", + "\u001b[1mSummary:\u001b[0m\r\n", + "\u001b[32mPASSED\u001b[0m Configuration is valid\r\n", + "\r\n", + "\u001b[2mReady to run: oaeval run \u001b[0m\u001b[1;2m<\u001b[0m\u001b[1;2;95mconfig\u001b[0m\u001b[1;2m>\u001b[0m\r\n" + ] + } + ], + "source": [ + "!oaeval validate config.yaml" + ] + }, + { + "cell_type": "markdown", + "id": "1f2d555c", + "metadata": {}, + "source": [ + "\n", + "## 4. Run Evaluation\n", + "\n", + "This is the only section that calls the live Gemini API. It runs **only if `GEMINI_API_KEY` is\n", + "set** (Section 1) \u2014 otherwise it explains what to do and does nothing, so *Run all* stays safe\n", + "without a key.\n", + "\n", + "The `parallel: false` setting in `config.yaml` sends requests one at a time rather than in\n", + "parallel; concurrent requests were observed to trigger transient `503 UNAVAILABLE` \"high demand\"\n", + "errors from `gemini-2.5-flash` during verification, and sequential calls are both gentler on the\n", + "free-tier quota and easier to debug when something does go wrong." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "02fbbe3b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:45:28.259987Z", + "iopub.status.busy": "2026-07-27T17:45:28.259700Z", + "iopub.status.idle": "2026-07-27T17:46:38.167065Z", + "shell.execute_reply": "2026-07-27T17:46:38.163184Z" + } + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "OpenAgent Eval v0.4.8\n", + "Configuration: config.yaml\n", + "\n", + "[progress-bar animation elided for a readable diff \u2014 this cell made 6 live\n", + " sequential calls to the Gemini API and took roughly 80-110s to run]\n", + "\n", + "Complete! \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 100% 0:01:04\n", + "\n", + "OK Evaluation complete!\n", + "Items: 6 | Errors: 0\n", + "Report saved to: reports/a038015e-ae77-459d-9b97-57120ca15e26.json\n", + "Markdown report saved to: reports/a038015e-ae77-459d-9b97-57120ca15e26.md\n" + ] + } + ], + "source": [ + "import os\n", + "\n", + "if os.environ.get(\"GEMINI_API_KEY\"):\n", + " !oaeval run config.yaml\n", + "else:\n", + " print(\n", + " \"No GEMINI_API_KEY set \u2014 skipping the live run.\\n\"\n", + " \"Set one (see Section 1) and re-run this cell to evaluate for real.\\n\"\n", + " \"Section 5 below can still show you a report if one already exists in reports/.\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "47b043e2", + "metadata": {}, + "source": [ + "\n", + "## 5. Analyze Results\n", + "\n", + "`oaeval report latest` prints the most recent report to the terminal. We also parse the JSON\n", + "report directly here so the numbers below are exactly what the run above produced \u2014 not a\n", + "paraphrase." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "580d9ec2", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:46:38.177912Z", + "iopub.status.busy": "2026-07-27T17:46:38.175374Z", + "iopub.status.idle": "2026-07-27T17:46:39.105221Z", + "shell.execute_reply": "2026-07-27T17:46:39.103564Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m - Report Viewer\r\n", + "\u001b[2mReport: latest\u001b[0m\r\n", + "\r\n", + "\u001b[32m\u256d\u2500\u001b[0m\u001b[32m\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u001b[0m\u001b[32m Evaluation Complete \u001b[0m\u001b[32m\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u001b[0m\u001b[32m\u2500\u256e\u001b[0m\r\n", + "\u001b[32m\u2502\u001b[0m \u001b[1mOpenAgent Eval Report\u001b[0m \u001b[32m\u2502\u001b[0m\r\n", + "\u001b[32m\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\u001b[0m\r\n", + "\u001b[3m Summary \u001b[0m\r\n", + "\u001b[34m\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2510\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mTotal Items\u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[37m \u001b[0m\u001b[37m6\u001b[0m\u001b[37m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mSuccessful \u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[37m \u001b[0m\u001b[37m6\u001b[0m\u001b[37m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mFailed \u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[37m \u001b[0m\u001b[37m0\u001b[0m\u001b[37m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2518\u001b[0m\r\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[3m Metrics \u001b[0m\r\n", + "\u001b[34m\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\u001b[0m\r\n", + "\u001b[34m\u2503\u001b[0m\u001b[1m \u001b[0m\u001b[1mMetric \u001b[0m\u001b[1m \u001b[0m\u001b[34m\u2503\u001b[0m\u001b[1m \u001b[0m\u001b[1m Score\u001b[0m\u001b[1m \u001b[0m\u001b[34m\u2503\u001b[0m\r\n", + "\u001b[34m\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mcontext_precision\u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[32m \u001b[0m\u001b[31m0.2778\u001b[0m\u001b[32m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mcontext_recall \u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[32m \u001b[0m\u001b[32m0.8333\u001b[0m\u001b[32m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mmrr \u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[32m \u001b[0m\u001b[33m0.7500\u001b[0m\u001b[32m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mfaithfulness \u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[32m \u001b[0m\u001b[33m0.5776\u001b[0m\u001b[32m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36manswer_relevancy \u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[32m \u001b[0m\u001b[32m0.9126\u001b[0m\u001b[32m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mexact_match \u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[32m \u001b[0m\u001b[31m0.0000\u001b[0m\u001b[32m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mf1_score \u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[32m \u001b[0m\u001b[31m0.3487\u001b[0m\u001b[32m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mlatency \u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[32m \u001b[0m\u001b[33m0.5954\u001b[0m\u001b[32m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[36m \u001b[0m\u001b[36mtoken_count \u001b[0m\u001b[36m \u001b[0m\u001b[34m\u2502\u001b[0m\u001b[32m \u001b[0m\u001b[32m0.9866\u001b[0m\u001b[32m \u001b[0m\u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\u001b[0m\r\n", + "\u001b[3m Sample Results \u001b[0m\r\n", + "\u001b[34m\u250f\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\u001b[0m\r\n", + "\u001b[34m\u2503\u001b[0m\u001b[1m \u001b[0m\u001b[1m#\u001b[0m\u001b[1m \u001b[0m\u001b[34m\u2503\u001b[0m\u001b[1m \u001b[0m\u001b[1mQuestion \u001b[0m\u001b[1m \u001b[0m\u001b[34m\u2503\u001b[0m\u001b[1m \u001b[0m\u001b[1mMetrics \u001b[0m\u001b[1m \u001b[0m\u001b[34m\u2503\u001b[0m\r\n", + "\u001b[34m\u2521\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[2m1\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m How do I open a temporary, \u001b[34m\u2502\u001b[0m context_precision=0.33, contex \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m in-memory-onl \u001b[34m\u2502\u001b[0m \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[2m2\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m What Cursor methods do I use to read \u001b[34m\u2502\u001b[0m context_precision=0.33, contex \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m bac \u001b[34m\u2502\u001b[0m \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[2m3\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m Why do '?' placeholders in \u001b[34m\u2502\u001b[0m context_precision=0.33, contex \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m cursor.execut \u001b[34m\u2502\u001b[0m \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[2m4\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m What happens if an exception is raised \u001b[34m\u2502\u001b[0m context_precision=0.00, contex \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m i \u001b[34m\u2502\u001b[0m \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[2m5\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m What is the recommended way to \u001b[34m\u2502\u001b[0m context_precision=0.33, contex \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m bulk-inse \u001b[34m\u2502\u001b[0m \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[2m6\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m What does setting \u001b[34m\u2502\u001b[0m context_precision=0.33, contex \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m\u001b[2m \u001b[0m\u001b[34m\u2502\u001b[0m connection.row_factory \u001b[34m\u2502\u001b[0m \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2514\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\u001b[0m\r\n", + "\u001b[2m\u256d\u2500\u001b[0m\u001b[2m\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u001b[0m\u001b[2m Configuration \u001b[0m\u001b[2m\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u001b[0m\u001b[2m\u2500\u256e\u001b[0m\r\n", + "\u001b[2m\u2502\u001b[0m Dataset: \u001b[36mdata/sample_qa.json\u001b[0m \u001b[2m\u2502\u001b[0m\r\n", + "\u001b[2m\u2502\u001b[0m LLM: \u001b[36mgemini/gemini-2.5-flash-lite\u001b[0m \u001b[2m\u2502\u001b[0m\r\n", + "\u001b[2m\u2502\u001b[0m Output: \u001b[36mmarkdown\u001b[0m \u001b[2m\u2502\u001b[0m\r\n", + "\u001b[2m\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\u001b[0m\r\n", + "\r\n", + "\u001b[2mReport ID: \u001b[0m\u001b[2;93ma038015e-ae77-459d-9b97-57120ca15e26\u001b[0m\r\n" + ] + } + ], + "source": [ + "!oaeval report latest" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "27f44966", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:46:39.110329Z", + "iopub.status.busy": "2026-07-27T17:46:39.110070Z", + "iopub.status.idle": "2026-07-27T17:46:39.124900Z", + "shell.execute_reply": "2026-07-27T17:46:39.119440Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Report: a038015e-ae77-459d-9b97-57120ca15e26.json\n", + "{\n", + " \"total_items\": 6,\n", + " \"successful_evaluations\": 6,\n", + " \"failed_evaluations\": 0,\n", + " \"metrics_summary\": {\n", + " \"context_precision\": 0.27777777777777773,\n", + " \"context_recall\": 0.8333333333333334,\n", + " \"mrr\": 0.75,\n", + " \"faithfulness\": 0.5776083776083776,\n", + " \"answer_relevancy\": 0.9126262626262626,\n", + " \"exact_match\": 0.0,\n", + " \"f1_score\": 0.34867473270835986,\n", + " \"latency\": 0.5954455621589053,\n", + " \"token_count\": 0.9865583333333333\n", + " },\n", + " \"total_tokens\": 1613,\n", + " \"average_latency_ms\": 809.1088756821895\n", + "}\n" + ] + } + ], + "source": [ + "import glob\n", + "import json\n", + "import os\n", + "\n", + "reports = sorted(glob.glob(\"reports/*.json\"), key=os.path.getmtime)\n", + "if not reports:\n", + " print(\"No report found yet \u2014 run Section 4 with a GEMINI_API_KEY set first.\")\n", + "else:\n", + " latest = reports[-1]\n", + " with open(latest) as f:\n", + " report = json.load(f)\n", + " print(\"Report:\", os.path.basename(latest))\n", + " print(json.dumps(report[\"summary\"], indent=2))" + ] + }, + { + "cell_type": "markdown", + "id": "6f333318", + "metadata": {}, + "source": [ + "### Reading the scores\n", + "\n", + "- **`context_precision` / `context_recall` / `mrr`** \u2014 retrieval quality: of what we fetched,\n", + " how much was actually relevant (precision), how much of the relevant material did we find\n", + " (recall), and how highly was the first relevant hit ranked (MRR, 1.0 = always rank 1)?\n", + "- **`faithfulness`** \u2014 is the generated answer actually supported by the retrieved context, or\n", + " is the model making things up?\n", + "- **`answer_relevancy`** \u2014 does the answer actually address the question asked?\n", + "- **`exact_match` / `f1_score`** \u2014 lexical overlap with `ground_truth`. Low scores here are\n", + " expected for open-ended answers even when they're substantively correct \u2014 Gemini tends to\n", + " answer in full sentences rather than echoing the terse `ground_truth` string, which\n", + " `f1_score`/`exact_match` penalise. `faithfulness`/`answer_relevancy` are the more meaningful\n", + " generation metrics for this dataset.\n", + "- **`latency` / `token_count`** \u2014 normalised performance/cost scores (higher is \"better\", i.e.\n", + " faster/cheaper relative to the pipeline's own configured thresholds)." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "ebf40b91", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:46:39.128116Z", + "iopub.status.busy": "2026-07-27T17:46:39.127885Z", + "iopub.status.idle": "2026-07-27T17:46:40.661207Z", + "shell.execute_reply": "2026-07-27T17:46:40.638463Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1;34mOpenAgent Eval\u001b[0m - Evaluation History\r\n", + "\r\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[3m Recent Evaluations \u001b[0m\r\n", + "\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\r\n", + "\u2503\u001b[1m \u001b[0m\u001b[1mID \u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1mDate \u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1mConfig \u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1mStatus\u001b[0m\u001b[1m \u001b[0m\u2503\r\n", + "\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\r\n", + "\u2502\u001b[36m \u001b[0m\u001b[36ma038015e-ae77-459d-9b97-57120ca\u2026\u001b[0m\u001b[36m \u001b[0m\u2502\u001b[32m \u001b[0m\u001b[32m2026-07-27\u001b[0m\u001b[32m \u001b[0m\u2502\u001b[33m \u001b[0m\u001b[33mdata/sample_qa.json\u001b[0m\u001b[33m \u001b[0m\u2502\u001b[1m \u001b[0m\u001b[1;32mOK\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502\r\n", + "\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n", + "\r\n", + "\u001b[2mShowing \u001b[0m\u001b[1;2;36m1\u001b[0m\u001b[2m evaluations\u001b[0m\r\n" + ] + } + ], + "source": [ + "!oaeval list" + ] + }, + { + "cell_type": "markdown", + "id": "b74de156", + "metadata": {}, + "source": [ + "\n", + "## 6. Improve Your RAG\n", + "\n", + "`oaeval diagnose` reads a saved report and attributes failures to **retrieval**, **generation**,\n", + "or **chunking**, with concrete recommendations tagged by component." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "a364e928", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-27T17:46:40.738133Z", + "iopub.status.busy": "2026-07-27T17:46:40.737811Z", + "iopub.status.idle": "2026-07-27T17:46:44.930189Z", + "shell.execute_reply": "2026-07-27T17:46:44.929328Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Diagnosing: reports/a038015e-ae77-459d-9b97-57120ca15e26.json\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[34m\u256d\u2500\u001b[0m\u001b[34m\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u001b[0m\u001b[34m Component Diagnosis \u001b[0m\u001b[34m\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u001b[0m\u001b[34m\u2500\u256e\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m \u001b[1;34mDiagnosis Report\u001b[0m \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m Items analyzed: 6 \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2502\u001b[0m Overall health: 66.7% \u001b[34m\u2502\u001b[0m\r\n", + "\u001b[34m\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\u001b[0m\r\n", + "\r\n", + "\u001b[1;33mSystem Health: Degraded \u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m66.7\u001b[0m\u001b[1;33m%\u001b[0m\u001b[1;33m)\u001b[0m\r\n", + "\r\n", + "\u001b[3m Blame Attribution \u001b[0m\r\n", + "\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\r\n", + "\u2503\u001b[1m \u001b[0m\u001b[1mComponent \u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1mFailures\u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1mPercentage\u001b[0m\u001b[1m \u001b[0m\u2503\r\n", + "\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\r\n", + "\u2502\u001b[1m \u001b[0m\u001b[1;31mRetrieval\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u2502 2 \u2502 66.7% \u2502\r\n", + "\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\r\n", + "\u2502\u001b[1m \u001b[0m\u001b[1;33mGeneration\u001b[0m\u001b[1m \u001b[0m\u2502 1 \u2502 33.3% \u2502\r\n", + "\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n", + "\r\n", + "\u001b[3m Failure Modes \u001b[0m\r\n", + "\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\r\n", + "\u2503\u001b[1m \u001b[0m\u001b[1mFailure Mode \u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1mCount\u001b[0m\u001b[1m \u001b[0m\u2503\r\n", + "\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\ufffd" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\ufffd\ufffd\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\r\n", + "\u2502\u001b[1m \u001b[0m\u001b[1mHallucination \u001b[0m\u001b[1m \u001b[0m\u2502 1 \u2502\r\n", + "\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\r\n", + "\u2502\u001b[1m \u001b[0m\u001b[1mLow Context Relevance\u001b[0m\u001b[1m \u001b[0m\u2502 1 \u2502\r\n", + "\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\r\n", + "\u2502\u001b[1m \u001b[0m\u001b[1mMissing Context \u001b[0m\u001b[1m \u001b[0m\u2502 1 \u2502\r\n", + "\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\r\n", + "\r\n", + "\u001b[1mRecommendations:\u001b[0m\r\n", + "\r\n", + " -> \u001b[1m[\u001b[0mRETRIEVAL\u001b[1m]\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m2\u001b[0m failures\u001b[1m)\u001b[0m Check if the retriever is using the correct \r\n", + "embedding model.\r\n", + " -> \u001b[1m[\u001b[0mRETRIEVAL\u001b[1m]\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m2\u001b[0m failures\u001b[1m)\u001b[0m Verify that the vector store contains the relevant\r\n", + "documents.\r\n", + " -> \u001b[1m[\u001b[0mRETRIEVAL\u001b[1m]\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m2\u001b[0m failures\u001b[1m)\u001b[0m Consider increasing the retrieval k value.\r\n", + " -> \u001b[1m[\u001b[0mRETRIEVAL\u001b[1m]\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m2\u001b[0m failures\u001b[1m)\u001b[0m Review chunking strategy to ensure documents are \r\n", + "split correctly.\r\n", + " -> \u001b[1m[\u001b[0mRETRIEVAL\u001b[1m]\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m2\u001b[0m failures\u001b[1m)\u001b[0m Check for index staleness \u2014 rebuild if documents \r\n", + "have changed.\r\n", + "\r\n" + ] + } + ], + "source": [ + "import glob\n", + "import os\n", + "\n", + "reports = sorted(glob.glob(\"reports/*.json\"), key=os.path.getmtime)\n", + "if reports:\n", + " latest_report = reports[-1]\n", + " print(\"Diagnosing:\", latest_report)\n", + " !oaeval diagnose {latest_report}\n", + "else:\n", + " print(\"No report to diagnose yet \u2014 run Section 4 with a GEMINI_API_KEY set first.\")" + ] + }, + { + "cell_type": "markdown", + "id": "c1571e1c", + "metadata": {}, + "source": [ + "### What the numbers above suggest, concretely\n", + "\n", + "Based on the actual run captured in this notebook (6 live Gemini calls, `k=3` retrieved\n", + "passages per question):\n", + "\n", + "- **`context_precision` well under 1.0** \u2014 with `k=3` and only one truly relevant passage per\n", + " question in a 19-passage corpus, the retriever is expected to pull in some near-neighbours\n", + " that aren't the exact answer passage (e.g. two different `sqlite3` API passages can be\n", + " semantically close). Lowering `k` to 1-2, or reranking the top-k with a cross-encoder, would\n", + " raise precision at some cost to recall.\n", + "- **`context_recall` well above `context_precision`** \u2014 the single ground-truth passage is\n", + " usually *somewhere* in the top-3, it's just not always first. That points at the retriever\n", + " finding the right material but not always ranking it highest \u2014 a reranking step (or a better\n", + " embedding model than the small 384-dim `all-MiniLM-L6-v2`) is the natural next lever.\n", + "- **`faithfulness`/`answer_relevancy` meaningfully above 0** but not near 1.0 \u2014 Gemini's answers\n", + " are grounded in the retrieved context (not hallucinated) but often add explanatory framing\n", + " beyond the terse `ground_truth`. Tightening the prompt (e.g. \"answer in one sentence, using\n", + " only the given context\") would likely move both metrics up.\n", + "- **Next things to try:** swap `all-MiniLM-L6-v2` for a larger embedding model, add a\n", + " cross-encoder reranker after retrieval, or run `oaeval synth` against `data/corpus.json` with\n", + " a real LLM to generate more/harder QA pairs (adversarial, multi-hop) than the 16 written here.\n", + "\n", + "### Troubleshooting\n", + "\n", + "See the **Troubleshooting** section of this folder's `README.md` for the full list, including\n", + "the `gemini-2.5-flash` daily-quota gotcha found while building this tutorial, the\n", + "`sentence-transformers` first-download delay, and Windows path notes.\n", + "\n", + "### Next steps\n", + "\n", + "- \ud83d\udcd6 Full docs: \n", + "- \ud83d\udd00 Try other free providers: swap `llm.provider` to `groq` (30 req/min) in `config.yaml`.\n", + "- \ud83d\udcc8 Raise `dataset.limit` (or delete it) in `config.yaml` to evaluate all 16 questions.\n", + "- \ud83e\uddea Generate more test cases with `oaeval synth --corpus data/corpus.json --count 10`.\n", + "- \u2b50 [Star OpenAgent Eval on GitHub](https://github.com/OpenAgentHQ/openagent-eval) if this was\n", + " useful, and open an issue for anything that didn't work as documented." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.14" + }, + "widgets": { + "application/vnd.jupyter.widget-state+json": { + "state": { + "124de5ad38b541cebde9e072895a5df2": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "background": null, + "description_width": "", + "font_size": null, + "text_color": null + } + }, + "1a6cd7ead13e4a26b82654f3403c92a5": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_44fbee8ad0504157829a5cb2fa8e6976", + "IPY_MODEL_c12dacd967d7493ab448ded7cd506110", + "IPY_MODEL_1cc4e9f6c5184b4eb7db450a0e3f9f72" + ], + "layout": "IPY_MODEL_1cb88fd3ea664a37a45fe47af4888234", + "tabbable": null, + "tooltip": null + } + }, + "1cb88fd3ea664a37a45fe47af4888234": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "1cc4e9f6c5184b4eb7db450a0e3f9f72": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HTMLView", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_e887c0fe856a46a1bf2a5fd984d5581d", + "placeholder": "\u200b", + "style": "IPY_MODEL_124de5ad38b541cebde9e072895a5df2", + "tabbable": null, + "tooltip": null, + "value": "\u2007103/103\u2007[00:00<00:00,\u2007\u20077.24it/s]" + } + }, + "44fbee8ad0504157829a5cb2fa8e6976": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "HTMLView", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_c62dac2603014712af08cc9c090cc343", + "placeholder": "\u200b", + "style": "IPY_MODEL_fe085a57d0564136b41897e1c3bf260b", + "tabbable": null, + "tooltip": null, + "value": "Loading\u2007weights:\u2007100%" + } + }, + "779ab9b68fe54166a2faaee0af29f4cd": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "c12dacd967d7493ab448ded7cd506110": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "2.0.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_allow_html": false, + "layout": "IPY_MODEL_779ab9b68fe54166a2faaee0af29f4cd", + "max": 103.0, + "min": 0.0, + "orientation": "horizontal", + "style": "IPY_MODEL_fd30533f85f64c43ad8eb5b83c22ea50", + "tabbable": null, + "tooltip": null, + "value": 103.0 + } + }, + "c62dac2603014712af08cc9c090cc343": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e887c0fe856a46a1bf2a5fd984d5581d": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "fd30533f85f64c43ad8eb5b83c22ea50": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "fe085a57d0564136b41897e1c3bf260b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "2.0.0", + "model_name": "HTMLStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "2.0.0", + "_model_name": "HTMLStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "StyleView", + "background": null, + "description_width": "", + "font_size": null, + "text_color": null + } + } + }, + "version_major": 2, + "version_minor": 0 + } + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file