Skip to content

docs: add end-to-end tutorial with free APIs and local embeddings - #249

Merged
himanshu231204 merged 1 commit into
OpenAgentHQ:mainfrom
Nitjsefnie-OSC:docs/241-end-to-end-tutorial
Jul 28, 2026
Merged

docs: add end-to-end tutorial with free APIs and local embeddings#249
himanshu231204 merged 1 commit into
OpenAgentHQ:mainfrom
Nitjsefnie-OSC:docs/241-end-to-end-tutorial

Conversation

@Nitjsefnie

Copy link
Copy Markdown
Contributor

Description

Adds examples/end-to-end-tutorial/ — a runnable notebook that takes a user from data preparation to evaluation results using only free-tier services: Google Gemini for the LLM and local sentence-transformers embeddings, with no paid API required.

The notebook is committed with its executed outputs, matching the precedent set by the Colab tutorial in #226, so a reader can see real numbers without running anything.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Test update
  • CI/CD update

Related Issues

Closes #241

How Has This Been Tested?

The notebook was executed end to end and the committed outputs are from that run:

jupyter nbconvert --to notebook --execute \
  --ExecutePreprocessor.timeout=300 --output tutorial.ipynb tutorial.ipynb
# completed with 0 errors; all 13 code cells produced output
  • Live Gemini calls, not mocks: the Section 4 oaeval run config.yaml cell completed 6/6 items with 0 errors.

  • Local embeddings really run: SentenceTransformer('all-MiniLM-L6-v2', device='cpu') produced a (2, 384) float32 array in Section 2, and served the memory retriever during the live run.

  • Metrics from that run: context_precision=0.278, context_recall=0.833, mrr=0.75, faithfulness=0.578, answer_relevancy=0.913, f1_score=0.349, exact_match=0.0.

  • The 16 QA pairs are hand-authored about Python's sqlite3, and each fact was checked by actually running the code rather than written from memory.

  • Unit tests pass (uv run pytest) — 969 passed, 5 skipped

  • Linter passes (uv run ruff check .) — ruff check examples/end-to-end-tutorial/ is clean; the 221 errors on main are pre-existing and untouched

  • Type checker passes (uv run mypy openagent_eval/) — not run to completion; it exceeds 90s here and this PR adds no .py source

  • Manual testing performed

Checklist

  • My code follows the project's coding standards
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works — not applicable; this is documentation. The notebook executing cleanly is the equivalent check.
  • New and existing unit tests pass locally with my changes

Additional Notes

Three things worth your attention, all verified rather than assumed:

The issue's config example does not match the implemented schema. #241 shows ${VAR} interpolation and nests embeddings under retriever.settings.embeddings. Neither exists in the code — interpolation is not implemented, and the real schema uses a sibling retriever.embedder block. The committed config.yaml follows the code, and the README documents the divergence so a reader who compares them is not confused.

.gitignore silently swallows the example config — filed as #248. The root rule at .gitignore:240 is a bare config.yaml, which is unanchored and matches at any depth, so examples/end-to-end-tutorial/config.yaml could not be staged normally. I used git add -f so this PR is complete, but the next person adding an example config will hit the same silent drop.

gemini-2.5-flash free tier enforces a 20-requests-per-day-per-project quota (quotaId: GenerateRequestsPerDayPerProjectPerModel-FreeTier), separate from and stricter than the widely-quoted 15/min. I exhausted it during verification, which is why the shipped config uses gemini-2.5-flash-lite. I have one data point, so I can't tell you whether that ceiling is project-specific or a general free-tier constant — flagging it as observed here, not as a general fact about Gemini.

I did not touch docs/examples/index.md or mkdocs.yml, which #226 updated. That felt like your call rather than mine; happy to add it if you want the tutorial linked from the docs nav.

Generated by Claude Opus 5 (brief, review), Claude Sonnet 5 (implementation, testing)

`examples/end-to-end-tutorial/`: a 6-section, executed Jupyter notebook
(setup -> data -> config -> run -> analyze -> improve) plus config.yaml,
requirements.txt, and a 16-QA-pair dataset with a 19-passage corpus, all
hand-authored and independently verified in this environment.

Primary LLM is Google Gemini (free tier), embeddings are local
sentence-transformers/all-MiniLM-L6-v2 (CPU-only, no external service).
The pipeline (`oaeval run config.yaml`) was executed against the live
Gemini API: 6/6 items succeeded with real generated answers and real
metric scores, and the full notebook was run end-to-end with
`jupyter nbconvert --execute` with no errors.

Notable findings baked into the README/config rather than papered over:
gemini-2.5-flash hit a 20-requests/day free-tier quota (not the commonly
quoted 15 req/min) partway through verification, so the config uses
gemini-2.5-flash-lite instead; concurrent requests intermittently 503'd
under load so the config runs sequentially (parallel: false); the
issue's own config example used unsupported ${VAR} interpolation and a
retriever.settings.embeddings nesting that doesn't match the actual
schema (retriever.embedder) - both corrected here; and the repo root's
blanket `config.yaml` .gitignore rule silently excluded this file, so
it was added with `git add -f` (see the README note for maintainers).

Fixes OpenAgentHQ#241

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@himanshu231204
himanshu231204 merged commit 8e72c5b into OpenAgentHQ:main Jul 28, 2026
8 checks passed
@github-actions

Copy link
Copy Markdown

🎉 Congratulations @Nitjsefnie!

Your pull request has been successfully merged into main. 🚀

Thank you for contributing to OpenAgentHQ and helping improve the project.

We truly appreciate your contribution and hope to see you back with more amazing PRs!

Happy Open Sourcing! ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create end-to-end tutorial with free APIs and local embeddings

2 participants