Skip to content

Write a plugin development guide with step-by-step example #108

Description

@himanshu231204

Description

OpenAgent Eval supports plugins via Python entry points, but there's no dedicated developer guide showing how to create a custom metric, provider, or dataset loader from scratch. The existing \plugins/examples/custom_metric.py\ is a good starting point but needs documentation to go with it.

What to Add

Create a new page \docs/plugin-development.md\ (or add to the existing plugin docs) covering:

  1. Overview of the plugin system architecture
  2. Step-by-step: Creating a custom metric (using WordCountMetric as example)
  3. Step-by-step: Registering a custom LLM provider
  4. How to configure \pyproject.toml\ entry points
  5. How to test a plugin locally
  6. Best practices and common pitfalls

Files to Look At

  • \openagent_eval/plugins/\ — plugin system implementation
  • \plugins/examples/custom_metric.py\ — example plugin
  • \pyproject.toml\ — entry point configuration
  • \docs/\ for MkDocs integration

Good First Issue ✅

A documentation task that helps you understand the plugin architecture deeply.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationgood first issueGood for newcomers

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions