Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix some bugs of mmlu #2299

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

eyuansu62
Copy link
Contributor

We find the following bugs of mmlu:

  1. The doc_to_text function in mmlu_flan_cot_fewshot is incorrect, causing the few-shot data not aligning with the pre-defined examples specified in the fewshot_config for each subset. For example, now we get:
Q: Statement 1 | Every element of a group generates a cyclic subgroup of the group. Statement 2 | The symmetric group S_10 has 10 elements.
(A) True, True (B) False, False (C) True, False (D) False, True
A: Let's think step by step. (C)

But the right one should be:

Q: Statement 1 | Every element of a group generates a cyclic subgroup of the group. Statement 2 | The symmetric group S_10 has 10 elements.
(A) True, True (B) False, False (C) True, False (D) False, True
A: Let's think step by step. A cyclic group is a group that is generated by a single element. Hence a subgroup generated by a single element of a group is cyclic and Statement 1 is True. The answer is (C).
  1. Since not all subsets have 5 cot-examples in "_cot_prompts.json", we set the default num_fewshot to 4.
  2. We modify the filter setting of mmlu_flan_n_shot to solve the related issue https://github.com/EleutherAI/lm-evaluation-harness/issues/2094#issuecomment-2260772114.

@baberabb
Copy link
Contributor

Thanks for the PR. Looks reasonable to me but I want @lintangsutawika to have a look as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants