Skip to content

[Benchmark] Update SciDocBench official release and evaluation - #1688

Open
Henryexcellentw wants to merge 1 commit into
open-compass:mainfrom
Henryexcellentw:feat/scidocbench-release
Open

Henryexcellentw wants to merge 1 commit into
open-compass:mainfrom
Henryexcellentw:feat/scidocbench-release

Conversation

@Henryexcellentw

Copy link
Copy Markdown

Summary

  • point SciDocBench to the official, version-pinned Hugging Face release
  • materialize the portable document-image assets automatically while preserving fully offline use when they already exist
  • score only the final answer after Qwen thinking and route 57 audited closed-contract tasks to deterministic Python scorers
  • use GPT-5.4-mini for open-ended judging with an improved rubric and a content-addressed cache
  • support both all-images-first and interleaved inputs in English and Chinese, with partition/category summaries

Dataset: https://huggingface.co/datasets/HenryExcellent/SciDocBench
Paper: https://arxiv.org/abs/2609.05141

Validation

  • python -m py_compile vlmeval/dataset/scidocbench.py
  • flake8 vlmeval/dataset/scidocbench.py
  • 15 local scorer and portable-image regression tests passed
  • verified the pinned TSV: 496 instances (124 per partition), 2,758 unique images, and no missing references

The PR intentionally changes only vlmeval/dataset/scidocbench.py; benchmark data and images remain in the official Hugging Face dataset.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant