Skip to content

WIP watch diff with upstream main branch - #6

Open
Jeronymous wants to merge 163 commits into
upstream-mainfrom
merge_hf_main
Open

Jeronymous wants to merge 163 commits into
upstream-mainfrom
merge_hf_main

Conversation

@Jeronymous

Copy link
Copy Markdown
Member

No description provided.

Oligou and others added 30 commits October 14, 2025 11:51
…q len (131072) is larger than the maximum number of tokens that can be stored in KV cache (130944). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine"
…and new version of the dataset is different)
Jeronymous and others added 30 commits September 24, 2026 11:00
(cherry picked from commit d0320da)
(cherry picked from commit cd13a15)
(cherry picked from commit da845c8)
…MODE lines)

The env-var tokenizer_mode entries left both engine-args dicts with a duplicate 'tokenizer_mode' key (the config-driven self._tokenizer_mode already followed and won in the sync path, while the async path only carried the env var). Keep the auto-detected self._tokenizer_mode in both paths.
Ports OpenLLM-France PR #10 into the restructured layout: default-suite mgsm-rev2:<lang> (tasks/tasks/mgsm.py) and multilingual mgsm_rev2_<lang> (multilingual/tasks/mgsm.py), on lightonai/mgsm-rev2 (test split only, zero-shot), reusing the standard MGSM prompts and metrics. Also make mgsm_prompt tolerate rows without a CoT 'answer' field (fall back to answer_number).
…rom train)

Point mgsm-rev2:<lang> at the combined dataset (corrected rev2 test + the standard MGSM CoT exemplars as a train split) and set few_shots_split=train, so few-shot (e.g. |8) shows the chain-of-thought 'answer' as separate user/assistant turns. Limited to the currently-uploaded languages (en, es, fr, de).
LIGHTEVAL_ENFORCE_EAGER (default on) toggles vLLM enforce_eager, and LIGHTEVAL_KV_CACHE_DTYPE lets the KV cache dtype be set, in both the sync and async engine args. Mirrors the fork commit.
…ailable, else lightonai

en/es/fr/de use OpenLLM-France/mgsm-rev2-with-train with few_shots_split=train (proper CoT few-shot); the languages not yet in that combined dataset (ru/zh/ja/th/sw/bn/te) keep the original lightonai/mgsm-rev2 (test only, zero-shot -- few-shot not optimal for them).
…e new API

Move blend, carte, eiffel, include, aime25_multilingual, piqa_multilingual and
bbh-fr (plus the bbh few-shot JSON files) from community_tasks/ into
src/lighteval/tasks/multilingual/tasks/, where they are auto-discovered when
multilingual tasks are loaded.

- Rename bbh-fr.py to bbh_fr.py so the module name is import-safe.
- Drop the removed 'suite' argument from LightevalTaskConfig calls.
- Define LETTER_INDICES locally in carte.py and eiffel.py (the old
  lighteval.tasks.default_prompts module no longer exists).
- Remove stray module-level debug prints in eiffel.py that ran at import.
…mode (not trying to download punkt at every sample)
…ex: Ministral Reasoning) + fix detection of thinking models for models like Ministral Reasoning, which have nothing special in their chat template. Note: Ministral Reasoning LLM still have the [THINK] and [/THINK] tokens to be turned NOT special, so that everything works fine with lighteval (that uses skip_special_tokens=True to decode strings).
(Added under multilingual/tasks and adapted to the new API: dropped the suite= argument.)
…t files (so that results are generated even if the detail serialization fails)
(Placed under multilingual/tasks and adapted to the new API: dropped the suite= argument.)
(README part of the upstream commit that replaced SCoolKID with RedTeachingBench.)
Drop dead COMET/MetricX imports (the enum uses CorpusLevelCOMET/MetricX) and
fix import sorting/formatting left by the cross-branch patch application.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.