fix(cicd): defer pytest import so oaeval works on a clean install

[Nitjsefnie]

Description

oaeval dies on any clean install because the CLI import chain reaches a module-level import pytest, and pytest is a dev dependency rather than a runtime one.

The chain, traced on current main:

pyproject.toml:95                        oaeval = "openagent_eval.cli.main:app"
openagent_eval/cli/main.py:23            from openagent_eval.cli.commands.test import test_command
openagent_eval/cli/commands/test.py:13   from openagent_eval.cicd.plugin import OAEvalPlugin
openagent_eval/cicd/__init__.py:14       from openagent_eval.cicd.plugin import OAEvalPlugin
openagent_eval/cicd/plugin.py:31         import pytest          # module scope

In a virtualenv carrying the runtime dependencies but not pytest, oaeval --help raises ModuleNotFoundError: No module named 'pytest' before printing anything. The import is now deferred into the plugin entrypoints that actually run under pytest, so the module imports cleanly without it.

This still reproduces at 0.4.8, not only at the 0.4.6 the issue was reported against — the post-fix run below reports openagent-eval 0.4.8.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Test update
• CI/CD update

Related Issues

Closes #227

How Has This Been Tested?

Reproduced first, then fixed. In a virtualenv with the runtime dependencies and no pytest:

$ /tmp/oae-227-venv/bin/oaeval --help
Traceback (most recent call last):
  File "/tmp/oae-227-venv/bin/oaeval", line 3, in <module>
    from openagent_eval.cli.main import app
  ...
  File ".../openagent_eval/cicd/plugin.py", line 31, in <module>
    import pytest
ModuleNotFoundError: No module named 'pytest'

After the change, same virtualenv:

$ /tmp/oae-227-venv/bin/oaeval --help
 Usage: oaeval [OPTIONS] COMMAND [ARGS]...
 Open-source CLI framework for evaluating RAG systems and AI Agents.
$ /tmp/oae-227-venv/bin/oaeval --version
openagent-eval 0.4.8

Added tests/unit/test_cicd/test_no_pytest_import.py — two subprocess-based tests that block pytest in sys.modules and assert that openagent_eval.cicd.OAEvalPlugin and openagent_eval.cli.main.app import cleanly and that oaeval --help succeeds. Subprocess rather than in-process, because the import has already happened by the time an in-process test runs.

• Unit tests pass (uv run pytest) — 975 passed, 4 skipped on tests/unit; 54 passed on tests/integration; full run with coverage 1029 passed, 79.61% against the 75% floor. The plugin behaves identically with pytest present.
• Linter passes (uv run ruff check .) — exits 1 on 229 pre-existing findings repo-wide. plugin.py carries the same two findings on this branch as on main (UP035, B904, shifted one line by the added import); the new test file has none. Zero new findings.
• Type checker passes (uv run mypy openagent_eval/) — see Additional Notes.
• Manual testing performed

uvx --from build python -m build and uvx --from twine twine check dist/* both pass.

Checklist

• My code follows the project's coding standards
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation — the notebook workaround noted in cli: oaeval crashes on a clean pip install — pytest imported at load but not a runtime dependency #227 (installing pytest in its install cell) can be dropped once this lands, but that lives outside this repo
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Additional Notes

Why lazy rather than declaring pytest a runtime dependency. #227 offers both. Making a test framework a hard runtime requirement of an evaluation library, to satisfy an optional CI-integration module, seemed the wrong trade — it lands in every user's install to serve a path most never touch. Happy to switch if you'd prefer the dependency.

Adjacent, not fixed: openagent_eval/cicd/__init__.py:14 still eagerly imports OAEvalPlugin, so importing anything from openagent_eval.cicd pulls the plugin module in. That is harmless now the pytest import is deferred, but it means any future optional CI-only dependency added to plugin.py reintroduces exactly this class of failure. Left alone as out of scope.

Bug discovered while running your CI commands: #250 — CI type-check step targets a non-existent src/ directory, so mypy has never run on the package. It is why the type-checker box above is unticked; this PR template's own checklist already names the correct path.

Generated by Claude Opus 5 (brief, review), Kimi K2.7 Code (implementation)

[github-actions]

🎉 Congratulations @Nitjsefnie!

Your pull request has been successfully merged into main. 🚀

Thank you for contributing to OpenAgentHQ and helping improve the project.

We truly appreciate your contribution and hope to see you back with more amazing PRs!

Happy Open Sourcing! ❤️