[DOCS] [FEATURE] Complete Colab Notebook Tutorial + Video Walkthrough

[himanshu231204]

📌 Summary

Create a complete, beginner-friendly Google Colab notebook that walks users through every feature of openagent-eval — from installation to advanced evaluation workflows — plus a companion YouTube video explaining how to use the tool. The video will be released on the official OpenAgentHQ YouTube channel with full credit to the contributor.

---

🎯 Goal

Make OpenAgent Eval accessible to anyone with a Google account — no local setup, no terminal knowledge required. The Colab notebook should be a self-contained, runnable tutorial that teaches users the tool end-to-end. The video should serve as a visual companion for those who prefer watching over reading.

---

📋 Problem Statement

• New users face friction setting up Python environments, installing dependencies, and configuring API keys locally
• There is no interactive, zero-setup way to try OpenAgent Eval before committing to a local install
• Visual learners and video-preference users have no walkthrough content
• The existing examples/ notebooks (rag_evaluation_tutorial.ipynb, corpus_and_related_modules.ipynb) are good but assume local setup knowledge
• A Colab notebook eliminates ALL setup friction — users click "Run All" and learn

---

🛠️ What to Build

Part 1: Google Colab Notebook (examples/openagent_eval_colab_tutorial.ipynb)

Create a single, comprehensive Colab notebook with the following sections:

---

Section 0: Welcome & Overview

• Title cell with badges (Colab open, GitHub stars, license)
• Brief explanation: What is OpenAgent Eval? Why use it?
• Table of contents with clickable links
• Estimated time: ~30 minutes
• Prerequisites: None (everything runs in Colab)

Section 1: Installation & Environment Setup

# Cell 1.1: Install openagent-eval
!pip install openagent-eval

# Cell 1.2: Verify installation
!oaeval doctor

# Cell 1.3: Check version
!oaeval --version

• Explain each command and its output
• Show expected output so users can verify

Section 2: Terminal Basics for Colab Users

This section is critical — many Colab users are Python-notebook-first and may not be familiar with terminal/CLI workflows.

• Explain what a terminal/command line is
• Show how Colab provides a terminal (! prefix for shell commands)
• Walk through basic terminal navigation:

  !pwd                    # Print working directory
  !ls -la                 # List files
  !mkdir my_project       # Create directory
  !cd my_project && pwd   # Change directory

• Explain the ! prefix: how Colab cells run shell commands
• Show the difference between Python cells and shell cells
• Demonstrate file operations:

  !cat config.yaml        # View file contents
  !head -20 config.yaml   # View first 20 lines
  !nano config.yaml       # (mention it's not available in Colab, show alternatives)

• Explain how to edit files in Colab (using Python file I/O or %%writefile magic)
• Show how to download/upload files between Colab and local machine

Section 3: Understanding the oaeval CLI

• Full CLI reference table (from README.md)
• Explain each command with examples:
  • oaeval init — scaffold a config
  • oaeval run — run evaluation
  • oaeval report — view results
  • oaeval audit — corpus health check
  • oaeval diagnose — blame attribution
  • oaeval synth — generate test data
  • oaeval compare — compare experiments
  • oaeval test — CI/CD gating
  • oaeval list — list past evaluations
  • oaeval doctor — system diagnostics
• Show the --help output for each command

Section 4: Initialize Your First Configuration

# Create config using oaeval init
!oaeval init --interactive

# Show the generated config.yaml
!cat config.yaml

• Explain every field in config.yaml
• Walk through the YAML structure:

  dataset:
    path: ./data/test_cases.json
    format: json
  
  providers:
    llm:
      type: openai
      model: gpt-4
      api_key: ${OPENAI_API_KEY}
    retriever:
      type: chroma
      collection: my_docs
  
  metrics:
    retrieval:
      - context_precision
      - context_recall
    generation:
      - faithfulness
      - answer_relevancy
  
  reports:
    - terminal
    - markdown

• Show how to set API keys securely in Colab:

  import os
  os.environ["OPENAI_API_KEY"] = "your-key-here"  # Or use Colab secrets

• Explain Colab Secrets Manager (lock icon in sidebar)

Section 5: Prepare Sample Data

• Create a sample knowledge base (3-5 simple documents)
• Create sample test cases in JSON format
• Show the expected data structure:

  [
    {
      "question": "What is OpenAgent Eval?",
      "context": "...",
      "answer": "...",
      "metadata": {}
    }
  ]

• Save files using %%writefile magic:

  %%writefile sample_data.json
  [
    {
      "question": "What is RAG?",
      "context": "Retrieval-Augmented Generation (RAG) is a technique...",
      "answer": "RAG combines retrieval and generation...",
      "metadata": {"source": "docs"}
    }
  ]

Section 6: Run Your First Evaluation

# Run evaluation
!oaeval run config.yaml

# View the report
!oaeval report latest

• Show expected output (with screenshots if possible)
• Explain what each metric score means
• Walk through the terminal report output

Section 7: Corpus Health Audit

# Create a sample corpus directory
!mkdir -p sample_corpus
%%writefile sample_corpus/doc1.md
# Document 1 content here...

# Run corpus audit
!oaeval audit --corpus ./sample_corpus/

• Explain what contradiction, staleness, duplicates, and coverage mean
• Show how to interpret the audit results
• Explain why corpus health matters before running RAG

Section 8: Failure Diagnosis (Blame Attribution)

# Run diagnosis on a report
!oaeval diagnose --report latest

• Explain the three blame categories:
  • Retrieval: Wrong documents retrieved
  • Generation: LLM hallucinated or misinterpreted
  • Chunking: Bad document splitting
• Show how to read the diagnosis output
• Explain what actions to take for each blame type

Section 9: Synthetic Test Data Generation

# Generate test cases from corpus
!oaeval synth --corpus ./sample_corpus/

• Explain why synthetic test data is useful
• Show the generated test cases
• Explain how to review and edit generated cases

Section 10: Comparing Experiments

# Run two different configs
# First, create config_v2.yaml with different settings
%%writefile config_v2.yaml
# Modified config here...

!oaeval run config.yaml
!oaeval run config_v2.yaml

# Compare results
!oaeval compare <exp1_id> <exp2_id>

• Show how to iterate on evaluation settings
• Explain how comparison helps identify improvements

Section 11: SDK Usage (Python API)

import asyncio
from openagent_eval.core import Engine
from openagent_eval.config import load_config

async def main():
    config = load_config("config.yaml")
    engine = Engine(config)
    # Run evaluation
    # report = await engine.run(dataset)
    # print(report.summary)

# asyncio.run(main())

• Explain when to use SDK vs CLI
• Show how to integrate into custom Python workflows
• Demonstrate programmatic access to metrics

Section 12: CI/CD Gating

# Set threshold-based pass/fail
!oaeval test config.yaml -t faithfulness:gte:0.8 -t context_recall:gte:0.7

• Explain exit codes (0 = pass, 1 = fail)
• Show how to integrate into GitHub Actions
• Provide a sample .github/workflows/eval.yml:

  name: Evaluation Gate
  on: [push, pull_request]
  jobs:
    eval:
      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@v4
        - uses: actions/setup-python@v5
          with:
            python-version: '3.11'
        - run: pip install openagent-eval
        - run: oaeval test config.yaml -t faithfulness:gte:0.8

Section 13: Advanced — Custom Metrics

# Show how to create a custom metric
%%writefile custom_metric.py
from openagent_eval.metrics.base import BaseMetric, MetricResult

class MyCustomMetric(BaseMetric):
    name = "my_custom_metric"
    
    def evaluate(self, answer: str, context: str, **kwargs) -> MetricResult:
        # Custom logic here
        score = len(answer.split()) / 100  # Example: word count normalized
        return MetricResult(
            score=min(score, 1.0),
            reason=f"Answer has {len(answer.split())} words",
            metadata={"word_count": len(answer.split())}
        )

• Explain the BaseMetric ABC
• Show how to register and use custom metrics

Section 14: Tips, Tricks & Troubleshooting

• Common errors and fixes
• How to use Colab GPU for local LLM testing (Ollama)
• How to persist files between Colab sessions (Google Drive mount)
• How to share the notebook with others
• Performance tips for large evaluations

Section 15: Next Steps & Resources

• Links to documentation, GitHub, discussions
• How to contribute
• How to report issues
• Links to related tools (RAGAS, DeepEval, etc.)
• Call to action: Star the repo, join the community

Section 16: Feedback & Credit

• "Was this notebook helpful? Star us on GitHub!"
• Contributor credit section
• Link to the companion YouTube video
• How to suggest improvements

---

Part 2: Companion YouTube Video

The contributor will create a 10-15 minute video tutorial that:

1. Introduces OpenAgent Eval (1-2 min)

   • What it is, why it exists
   • The problem it solves

2. Walks through the Colab notebook (7-10 min)

   • Screen recording of running each section
   • Explains what's happening at each step
   • Shows expected outputs
   • Highlights key takeaways

3. Shows a real-world example (2-3 min)

   • Running evaluation on a realistic RAG system
   • Interpreting results
   • Taking action on findings

4. Wraps up (1 min)

   • How to get started
   • Where to get help
   • Call to action (star, subscribe, contribute)

Video Requirements:

• Resolution: 1080p minimum
• Clear audio (narration required)
• Screen recording with visible cursor
• Chapters/timestamps in description
• Include notebook link in video description
• Credit: "Created by [Contributor Name] for OpenAgentHQ"

Video Deliverables:

• Raw video file (MP4)
• Thumbnail image (1280x720)
• Video description with timestamps and links
• SRT subtitle file (for accessibility)

---

📁 Files to Create/Modify

File	Action	Description
examples/openagent_eval_colab_tutorial.ipynb	CREATE	Main Colab notebook
examples/README.md	UPDATE	Add link to new notebook
README.md	UPDATE	Add "Try in Colab" badge
docs/tutorials/colab.md	CREATE (optional)	Written companion guide

---

✅ Acceptance Criteria

Notebook Criteria

• Notebook opens and runs in Google Colab without errors
• All cells execute successfully with Runtime > Run all
• No hardcoded API keys (uses Colab secrets or prompts user)
• Every section has explanatory markdown cells (not just code)
• Expected output is shown or described for each command
• Terminal/CLI section clearly explains ! prefix and shell basics
• All oaeval commands are demonstrated with real output
• Error handling is shown (what happens when things go wrong)
• Notebook is under 500 cells (keep it focused)
• Total runtime < 10 minutes on free Colab tier
• Works with free Colab (no paid features required)
• Includes Colab badge at the top:

Video Criteria

• Video is 10-15 minutes long
• Clear narration explaining each step
• Screen recording shows notebook execution
• 1080p resolution or higher
• Chapters/timestamps in video description
• Notebook link in video description
• Contributor credited in video and description
• SRT subtitles included for accessibility
• Thumbnail image provided (1280x720)

Documentation Criteria

• README.md updated with Colab badge
• examples/README.md links to new notebook
• All CLI commands shown with --help output
• API key setup instructions are clear and secure
• Troubleshooting section covers common issues

---

🏷️ Labels

• documentation
• enhancement
• good first issue
• help wanted
• hacktoberfest (if applicable)

---

🎬 Video Credit

The contributor who builds this notebook and creates the video will receive:

1. Full credit in the YouTube video description
2. Credit in the notebook's acknowledgment section
3. Credit in the repository's CHANGELOG.md
4. Co-author credit on the notebook file
5. Featured in the organization's social media post promoting the video
6. The video will be published on the official OpenAgentHQ YouTube channel

---

📚 Reference Materials

• PROJECT.md — Product specification
• README.md — Installation and quick start
• QUICKSTART.md — Developer quick reference
• ARCHITECTURE.md — System architecture
• examples/ — Existing example notebooks
• CONTRIBUTING.md — Contribution guidelines

---

💡 Notes for Contributors

1. Start with the notebook — the video comes after the notebook is polished
2. Test on fresh Colab runtime — everything should work from zero
3. Use free-tier compatible packages — no paid API keys required for basic demo (use mock providers or free alternatives)
4. Include a "No API Key?" section — show how to use Mock providers or local Ollama
5. Keep cells focused — one concept per cell, clear markdown explanations
6. Use Colab magic commands — %%writefile, !pip install, etc.
7. Add table of contents — use Colab's table of contents feature
8. Test Runtime > Run all — the entire notebook must execute without manual intervention

---

🚀 Getting Started

1. Fork the repository
2. Create branch: feature/colab-tutorial
3. Create the notebook in examples/
4. Test in Google Colab (share link for review)
5. Create the companion video
6. Submit PR with notebook + video link

---

Happy documenting! 🎉