Repository navigation
WIP watch diff with upstream main branch - #6
Open
Jeronymous wants to merge 163 commits into
Open
Jeronymous wants to merge 163 commits into
Jeronymous wants to merge 163 commits into
Conversation
…ion of the dataset)
…q len (131072) is larger than the maximum number of tokens that can be stored in KV cache (130944). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine"
…and new version of the dataset is different)
…t it has eos_token_id)
…o avoid failures or NaN
(cherry picked from commit d0320da)
(cherry picked from commit cd13a15)
(cherry picked from commit 27ab205)
(cherry picked from commit da845c8)
…MODE lines) The env-var tokenizer_mode entries left both engine-args dicts with a duplicate 'tokenizer_mode' key (the config-driven self._tokenizer_mode already followed and won in the sync path, while the async path only carried the env var). Keep the auto-detected self._tokenizer_mode in both paths.
Ports OpenLLM-France PR #10 into the restructured layout: default-suite mgsm-rev2:<lang> (tasks/tasks/mgsm.py) and multilingual mgsm_rev2_<lang> (multilingual/tasks/mgsm.py), on lightonai/mgsm-rev2 (test split only, zero-shot), reusing the standard MGSM prompts and metrics. Also make mgsm_prompt tolerate rows without a CoT 'answer' field (fall back to answer_number).
…rom train) Point mgsm-rev2:<lang> at the combined dataset (corrected rev2 test + the standard MGSM CoT exemplars as a train split) and set few_shots_split=train, so few-shot (e.g. |8) shows the chain-of-thought 'answer' as separate user/assistant turns. Limited to the currently-uploaded languages (en, es, fr, de).
LIGHTEVAL_ENFORCE_EAGER (default on) toggles vLLM enforce_eager, and LIGHTEVAL_KV_CACHE_DTYPE lets the KV cache dtype be set, in both the sync and async engine args. Mirrors the fork commit.
…ailable, else lightonai en/es/fr/de use OpenLLM-France/mgsm-rev2-with-train with few_shots_split=train (proper CoT few-shot); the languages not yet in that combined dataset (ru/zh/ja/th/sw/bn/te) keep the original lightonai/mgsm-rev2 (test only, zero-shot -- few-shot not optimal for them).
…e new API Move blend, carte, eiffel, include, aime25_multilingual, piqa_multilingual and bbh-fr (plus the bbh few-shot JSON files) from community_tasks/ into src/lighteval/tasks/multilingual/tasks/, where they are auto-discovered when multilingual tasks are loaded. - Rename bbh-fr.py to bbh_fr.py so the module name is import-safe. - Drop the removed 'suite' argument from LightevalTaskConfig calls. - Define LETTER_INDICES locally in carte.py and eiffel.py (the old lighteval.tasks.default_prompts module no longer exists). - Remove stray module-level debug prints in eiffel.py that ran at import.
…ad of (list, int) tuples
…ough bootstrap stderr
…mode (not trying to download punkt at every sample)
…g in the generation prompt
…ex: Ministral Reasoning) + fix detection of thinking models for models like Ministral Reasoning, which have nothing special in their chat template. Note: Ministral Reasoning LLM still have the [THINK] and [/THINK] tokens to be turned NOT special, so that everything works fine with lighteval (that uses skip_special_tokens=True to decode strings).
(Added under multilingual/tasks and adapted to the new API: dropped the suite= argument.)
…t files (so that results are generated even if the detail serialization fails)
(Placed under multilingual/tasks and adapted to the new API: dropped the suite= argument.)
(README part of the upstream commit that replaced SCoolKID with RedTeachingBench.)
Drop dead COMET/MetricX imports (the enum uses CorpusLevelCOMET/MetricX) and fix import sorting/formatting left by the cross-branch patch application.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.