CORE centered accuracy calculation and random baseline

[karpathy]

Hello, I was looking at the eval_meta_data.csv and I noticed that the "Random baseline" column seems somewhat arbitrarily (and incorrectly?) set in some cases. For example, when there are 4 choices in a multiple choice, the baseline is 25% - great, that makes sense and it's what I assumed from the paper. But why is siqa at 33% when it's a binary task? Or commonsense_qa at 20% when it's a 4-choice task. Or boolq at 62% when it's a binary task (this makes e.g. Olmo1B, with accuracy of only ~64% contribute a centered accuracy of -0.25 towards the average CORE score, which looks wrong/buggy). Can you help me understand where these numbers come from?

[jeffreywpli]

Hey @karpathy, thanks for bringing this to our attention! Overall, we intentionally did not set all baselines to 1/N (where N is the number of MC choices). However, after looking into your question more carefully, we found that we indeed had set several baselines incorrectly-very sorry about these errors (as well as the delay in response).

Our intention

We originally wanted to adjust some baselines to account for imbalance (when non-trivial) in the test sets. The rationale is that for ICL, models could potentially tend towards outputting the majority label based on provided examples. For instance, 62% of answers for boolq are yes, so we choose that as the baseline instead of 50%.

In more detail, we meant to follow the procedure

• Default Baseline: By default, we set baselines to be 1/N where N is the number of choices
• Adjusted Baseline: When the majority label ratio r exceeds 1/N + 0.05 AND we use ≥N in-context examples, we increase the baseline to be r

Where we went wrong

While looking into your question, we found that for some evaluations, the llmfoundry version (that we use) is different from the original "canonical" version. For instance commonsense_qa was originally a 5-way task but llmfoundry simplified it to 4-way.

Unfortunately, we both weren't aware of these changes and had set some baselines based on the canonical version. This applies to the following tasks in the CORE subset.

• commonsense_qa 20% --> 40.3% (simplified from 5-way to 4-way and smushed two labels together)
• bigbench_language_identification 9.1% --> 25% (each example is only 4-way)
• agi_eval_lsat_ar 20% --> 25% (simplified from 5-way to 4-way)

Does fixing these errors change conclusions?

We also did some analysis on how much these corrections might affect model rankings. Thankfully, it seems orderings remain highly consistent (Spearman correlation > 0.99) so while individual CORE scores will shift, we don't expect major changes in conclusions.

Fixes going forwards

For now, we plan to merge the following changes in our codebase (PR #115) which includes

• a corrected version of our eval_meta_data.csv
• added documentation for these changes at the top of the README.
• updated numbers in exp_data/evals and the table in the README

We also plan to reach out to authors who've already published results based on dclm to help recompute averages for a more centralized/updated leaderboard.

[karpathy]

Got it, makes sense to skew it by the statistics of the labels. Not just if it were ICL, but also e.g. based on initialization and what comes out as majority label you could see some weird stuff too, e.g. maybe the metric starting out high and then plunging down as you train, etc. Thank you for the response and a nice writeup!