[tests] Update lm_eval VL tests to qwen 3

[brian-dellabetta]

SUMMARY:
Upgrade the lm_eval vision languge tests from Qwen 2.5 to Qwen 3. After updating to include apply_chat_template, the scores closely align with what was achieved with Qwen 2.5

• switch to neuralmagic/calibration dataset, based on suggestion here, to avoid tracing issues related to VL dataset.
• switch to chartqa task, to increase number of samples and reduce variance in accuracy.

TEST PLAN:
The 3 lm_eval VL tests were run, and the accuracies were updated

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[dsikka]

Why not just use mmmu_val instead of the literature task? This gives us around 0.53 for the dense model?

[brian-dellabetta]

Why not just use mmmu_val instead of the literature task? This gives us around 0.53 for the dense model?

mmmu_val is 900 evals total instead of 30. that would add probably ~40 minutes to each lm-eval run, and we run two for each config, so total test time would increase over 3 hours with that change

[dsikka]

Why not use mmmu_val

Why not just use mmmu_val instead of the literature task? This gives us around 0.53 for the dense model?

mmmu_val is 900 evals total instead of 30. that would add probably ~40 minutes to each lm-eval run, and we run two for each config, so total test time would increase over 3 hours with that change

The 30 datapoints has proven to be very noisy historically. A happy medium might be better but we should also just validate the runtime for batch size of 100

[kylesayrs]

Should we not always be applying a chat template?

[brian-dellabetta]

None of the others have, they are using add_bos_token=True instead. This is just to be backwards-compatible. Research does not always do so, the transforms benchmarks aren't using it -- https://github.com/neuralmagic/research/blob/bdellabe/transforms-benchmarks/examples/llm_compress_eval_example.py#L171-L176