docs: add end-to-end tutorial with free APIs and local embeddings - #249
Merged
himanshu231204 merged 1 commit intoJul 28, 2026
Merged
Conversation
`examples/end-to-end-tutorial/`: a 6-section, executed Jupyter notebook
(setup -> data -> config -> run -> analyze -> improve) plus config.yaml,
requirements.txt, and a 16-QA-pair dataset with a 19-passage corpus, all
hand-authored and independently verified in this environment.
Primary LLM is Google Gemini (free tier), embeddings are local
sentence-transformers/all-MiniLM-L6-v2 (CPU-only, no external service).
The pipeline (`oaeval run config.yaml`) was executed against the live
Gemini API: 6/6 items succeeded with real generated answers and real
metric scores, and the full notebook was run end-to-end with
`jupyter nbconvert --execute` with no errors.
Notable findings baked into the README/config rather than papered over:
gemini-2.5-flash hit a 20-requests/day free-tier quota (not the commonly
quoted 15 req/min) partway through verification, so the config uses
gemini-2.5-flash-lite instead; concurrent requests intermittently 503'd
under load so the config runs sequentially (parallel: false); the
issue's own config example used unsupported ${VAR} interpolation and a
retriever.settings.embeddings nesting that doesn't match the actual
schema (retriever.embedder) - both corrected here; and the repo root's
blanket `config.yaml` .gitignore rule silently excluded this file, so
it was added with `git add -f` (see the README note for maintainers).
Fixes OpenAgentHQ#241
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
🎉 Congratulations @Nitjsefnie! Your pull request has been successfully merged into main. 🚀 Thank you for contributing to OpenAgentHQ and helping improve the project. We truly appreciate your contribution and hope to see you back with more amazing PRs! Happy Open Sourcing! ❤️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
examples/end-to-end-tutorial/— a runnable notebook that takes a user from data preparation to evaluation results using only free-tier services: Google Gemini for the LLM and localsentence-transformersembeddings, with no paid API required.The notebook is committed with its executed outputs, matching the precedent set by the Colab tutorial in #226, so a reader can see real numbers without running anything.
Type of Change
Related Issues
Closes #241
How Has This Been Tested?
The notebook was executed end to end and the committed outputs are from that run:
Live Gemini calls, not mocks: the Section 4
oaeval run config.yamlcell completed 6/6 items with 0 errors.Local embeddings really run:
SentenceTransformer('all-MiniLM-L6-v2', device='cpu')produced a(2, 384)float32 array in Section 2, and served thememoryretriever during the live run.Metrics from that run:
context_precision=0.278, context_recall=0.833, mrr=0.75, faithfulness=0.578, answer_relevancy=0.913, f1_score=0.349, exact_match=0.0.The 16 QA pairs are hand-authored about Python's
sqlite3, and each fact was checked by actually running the code rather than written from memory.Unit tests pass (
uv run pytest) —969 passed, 5 skippedLinter passes (
uv run ruff check .) —ruff check examples/end-to-end-tutorial/is clean; the 221 errors onmainare pre-existing and untouchedType checker passes (
uv run mypy openagent_eval/) — not run to completion; it exceeds 90s here and this PR adds no.pysourceManual testing performed
Checklist
Additional Notes
Three things worth your attention, all verified rather than assumed:
The issue's config example does not match the implemented schema. #241 shows
${VAR}interpolation and nests embeddings underretriever.settings.embeddings. Neither exists in the code — interpolation is not implemented, and the real schema uses a siblingretriever.embedderblock. The committedconfig.yamlfollows the code, and the README documents the divergence so a reader who compares them is not confused..gitignoresilently swallows the example config — filed as #248. The root rule at.gitignore:240is a bareconfig.yaml, which is unanchored and matches at any depth, soexamples/end-to-end-tutorial/config.yamlcould not be staged normally. I usedgit add -fso this PR is complete, but the next person adding an example config will hit the same silent drop.gemini-2.5-flashfree tier enforces a 20-requests-per-day-per-project quota (quotaId: GenerateRequestsPerDayPerProjectPerModel-FreeTier), separate from and stricter than the widely-quoted 15/min. I exhausted it during verification, which is why the shipped config usesgemini-2.5-flash-lite. I have one data point, so I can't tell you whether that ceiling is project-specific or a general free-tier constant — flagging it as observed here, not as a general fact about Gemini.I did not touch
docs/examples/index.mdormkdocs.yml, which #226 updated. That felt like your call rather than mine; happy to add it if you want the tutorial linked from the docs nav.Generated by Claude Opus 5 (brief, review), Claude Sonnet 5 (implementation, testing)