fix(core): resolve deepcopy compatibility issues in evaluation components

[ployts]

OpenJudge Version

0.2.0

Description

This PR fixes deepcopy compatibility issues across core evaluation components in OpenJudge. The changes address problems that occurred when attempting to deep copy evaluation objects, which is essential for scenarios involving:

• Parallel evaluation execution
• Runner and strategy cloning operations

Checklist

Please check the following items before code is ready to be reviewed.

• Code has been formatted with pre-commit run --all-files command
• All tests are passing
• Docstrings are in Google style
• Related documentation has been updated (e.g. links, examples, etc.)
• Code is ready for review

[gemini-code-assist]

Summary of Changes

Hello @ployts, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses critical deepcopy compatibility issues within OpenJudge's core evaluation components. By transitioning from shallow copies to proper deep copies for grader instances and implementing a custom deepcopy mechanism for chat models, the changes ensure robust object isolation. This is essential for scenarios like parallel evaluation execution and cloning operations, preventing unintended state sharing and improving the reliability of the evaluation framework.

Highlights

• Implemented deep copying for BaseGrader instances: Replaced self.copy() with copy.deepcopy(self) in base_grader.py and grading_runner.py to ensure complete isolation of grader objects during evaluation, preventing unintended state sharing.
• Removed custom copy() method from BaseGrader: The custom shallow copy method in base_grader.py was removed as it is no longer needed after transitioning to copy.deepcopy.
• Added custom __deepcopy__ to BaseChatModel: A __deepcopy__ method was implemented for BaseChatModel to correctly handle deep copying while preserving the shared client instance, optimizing resource usage.
• Updated .gitignore: Added local/ to the .gitignore file to prevent local development artifacts from being committed.
• Corrected error message: A minor typo in the error message for VotingEvaluationStrategy was fixed for improved clarity and consistency.

Changelog

• .gitignore
  • Added local/ to the ignore list.
• openjudge/evaluation_strategy/voting_evaluation_strategy.py
  • Corrected a typo in the error message for VotingEvaluationStrategy.
• openjudge/graders/base_grader.py
  • Replaced self.copy() with copy.deepcopy(self) in _aevaluate method.
  • Removed the deprecated copy() method.
• openjudge/models/base_chat_model.py
  • Imported the copy module.
  • Implemented a custom __deepcopy__ method to handle model cloning while reusing the client.
• openjudge/runner/grading_runner.py
  • Imported the copy module.
  • Replaced grader.copy() with copy.deepcopy(grader) for isolated grader instances.
• tests/evaluation_strategy/test_voting_evaluation_strategy.py
  • Updated the expected error message in the test to match the corrected message in the strategy.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively resolves deepcopy compatibility issues within the evaluation components by replacing shallow copies with deep copies. This is a critical fix for ensuring state isolation during parallel evaluations. The changes are well-implemented, particularly the custom __deepcopy__ method in BaseChatModel to handle the client object correctly. My review includes a few minor suggestions to improve code clarity and documentation consistency.