Skip to content

[Test]Add accuracy test for multiple models(2) #4251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+47 −2
Open

[Test]Add accuracy test for multiple models(2) #4251

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b2318b8
add acc test
MrZ20 Nov 18, 2025
8e15521
add acc test
MrZ20 Nov 18, 2025
bc51d40
modefy format
MrZ20 Nov 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/vllm_ascend_test_nightly_a2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,12 +114,15 @@ jobs:
- Qwen3-VL-8B-Instruct
- Qwen2.5-Omni-7B
- Meta-Llama-3.1-8B-Instruct
- Mistral-7B-Instruct-v0.1
- Phi-4-mini-instruct
- os: linux-aarch64-a2-2
model_list:
- Qwen3-30B-A3B
- Qwen3-VL-30B-A3B-Instruct
- DeepSeek-V2-Lite
- Qwen3-30B-A3B-W8A8
- ERNIE-4.5-21B-A3B-PT
- os: linux-aarch64-a2-4
model_list:
- Qwen3-Next-80B-A3B-Instruct
Expand Down
15 changes: 15 additions & 0 deletions tests/e2e/models/configs/ERNIE-4.5-21B-A3B-PT.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
model_name: "PaddlePaddle/ERNIE-4.5-21B-A3B-PT"
hardware: "Atlas A2 Series"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,flexible-extract"
value: 0.72
- name: "ceval-valid"
metrics:
- name: "acc,none"
value: 0.85
num_fewshot: 5
tensor_parallel_size: 2
batch_size: 16
gpu_memory_utilization: 0.6
11 changes: 11 additions & 0 deletions tests/e2e/models/configs/Mistral-7B-Instruct-v0.1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
model_name: "mistralai/Mistral-7B-Instruct-v0.1"
hardware: "Atlas A2 Series"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.35
- name: "exact_match,flexible-extract"
value: 0.38
Copy link
Collaborator

@MengqingCao MengqingCao Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an accuracy issue on this model?

trust_remote_code: True
num_fewshot: 5
13 changes: 13 additions & 0 deletions tests/e2e/models/configs/Phi-4-mini-instruct.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
model_name: "LLM-Research/Phi-4-mini-instruct"
hardware: "Atlas A2 Series"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.81
- name: "exact_match,flexible-extract"
value: 0.81
trust_remote_code: True
num_fewshot: 5
batch_size: 32
gpu_memory_utilization: 0.8
7 changes: 5 additions & 2 deletions tests/e2e/models/configs/accuracy.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
DeepSeek-V2-Lite.yaml
ERNIE-4.5-21B-A3B-PT.yaml
InternVL3_5-8B.yaml
Meta-Llama-3.1-8B-Instruct.yaml
Mistral-7B-Instruct-v0.1.yaml
Phi-4-mini-instruct.yaml
Qwen2.5-VL-7B-Instruct.yaml
Qwen3-30B-A3B.yaml
Qwen3-8B.yaml
Expand All @@ -8,5 +13,3 @@ Qwen2-Audio-7B-Instruct.yaml
Qwen3-VL-30B-A3B-Instruct.yaml
Qwen3-VL-8B-Instruct.yaml
Qwen2.5-Omni-7B.yaml
Meta-Llama-3.1-8B-Instruct.yaml
InternVL3_5-8B.yaml
Loading