Skip to content

feat: add SciMIF benchmark - #1674

Open
shenye7436 wants to merge 1 commit into
open-compass:mainfrom
shenye7436:support-scimif
Open

shenye7436 wants to merge 1 commit into
open-compass:mainfrom
shenye7436:support-scimif

Conversation

@shenye7436

Copy link
Copy Markdown

Summary

This PR adds support for the SciMIF benchmark in VLMEvalKit.

SciMIF contains 2,527 samples across Chemistry, Geography, Life Science, Materials Science, and Physics. It supports text-only, single-image, and multi-image samples.

Resources

Changes

  • Added the SciMIF dataset adapter.
  • Added direct loading from Hugging Face.
  • Added text-only, single-image, and multi-image prompt construction.
  • Preserved edit_question and instruction_list.
  • Added instruction-following evaluation.
  • Added unit tests.

The current integration evaluates instruction following only. CompassVerifier-based answer-correctness evaluation is not included.

Validation

  • Loaded all 2,527 samples successfully.
  • Verified text-only, single-image, and multi-image inputs.
  • Verified multi-image ordering.
  • Verified model predictions and instruction-following evaluation.
  • Tested generation model: gpt-4o.
  • Tested judge model: gpt-4.1.
  • Unit tests and all pre-commit checks passed.

License

SciMIF is constructed from multiple open-source scientific datasets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant