-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added TurkishMMLU to LM Evaluation Harness #2283
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks real good. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi there! One last thing: could you also add an entry to lm_eval/tasks/README.md
describing the task in 1 sentence as is done for the other entries in that table (mentioning in this sentence that your dataset is not translated from MMLU and not machine-translated!), and note that the language is Turkish?
We want to make sure others can discover this task and your work more easily!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated Readme addresses the raised issues
In this pull request, I would like to add our work TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish to LM Evaluation Harness.
You can find the details of our work in our repository:
https://github.com/ArdaYueksel/TurkishMMLU
Also, our dataset is made available in HuggingFace: https://huggingface.co/datasets/AYueksel/TurkishMMLU
Key Features: