Description
OpenAgent Eval supports plugins via Python entry points, but there's no dedicated developer guide showing how to create a custom metric, provider, or dataset loader from scratch. The existing \plugins/examples/custom_metric.py\ is a good starting point but needs documentation to go with it.
What to Add
Create a new page \docs/plugin-development.md\ (or add to the existing plugin docs) covering:
- Overview of the plugin system architecture
- Step-by-step: Creating a custom metric (using WordCountMetric as example)
- Step-by-step: Registering a custom LLM provider
- How to configure \pyproject.toml\ entry points
- How to test a plugin locally
- Best practices and common pitfalls
Files to Look At
- \openagent_eval/plugins/\ — plugin system implementation
- \plugins/examples/custom_metric.py\ — example plugin
- \pyproject.toml\ — entry point configuration
- \docs/\ for MkDocs integration
Good First Issue ✅
A documentation task that helps you understand the plugin architecture deeply.
Description
OpenAgent Eval supports plugins via Python entry points, but there's no dedicated developer guide showing how to create a custom metric, provider, or dataset loader from scratch. The existing \plugins/examples/custom_metric.py\ is a good starting point but needs documentation to go with it.
What to Add
Create a new page \docs/plugin-development.md\ (or add to the existing plugin docs) covering:
Files to Look At
Good First Issue ✅
A documentation task that helps you understand the plugin architecture deeply.