Skip to content

About

No description, website, or topics provided.

Resources

Code of conduct

Contributing

Security policy

Stars

17 stars

Watchers

4 watching

Forks

Luciole-Training

Training pipeline for the Luciole series of multilingual (French-English) large language models, part of the OpenLLM-France initiative.

This repository covers the full LLM lifecycle: data preparation, tokenizer training, pretraining, fine-tuning, and evaluation. It is designed to run on the Jean-Zay HPC cluster (NVIDIA H100 GPUs) via SLURM.

Pipeline Overview

┌──────────────┐    ┌───────────────┐    ┌──────────────┐    ┌──────────────┐
│ 1. Data      │───▶│ 2. Tokenizer  │───▶│ 3. Tokenize  │───▶│ 4. Datamix   │
│ Processing   │    │ Training      │    │ Datasets     │    │ Creation     │
│ data/        │    │ tokenizer_    │    │ data/        │    │ pretrain/    │
│ processing/  │    │ train/        │    │ tokenization/│    │ datamix/     │
└──────────────┘    └───────────────┘    └──────────────┘    └──────┬───────┘
                                                                    │
                    ┌───────────────┐    ┌──────────────┐    ┌──────▼───────┐
                    │ 8. Evaluation │◀───│ 7. Convert   │◀───│ 5. Pretrain  │
                    │ evaluation/   │    │ to HF format │    │ pretrain/    │
                    └───────┬───────┘    │ pretrain/    │    │ train/       │
                            │            │ conversion/  │    └──────────────┘
                            │            └──────────────┘
                    ┌───────▼───────┐
                    │ 6. Fine-tune  │
                    │ (optional)    │
                    │ finetune/     │
                    └───────────────┘

Directory Structure

Directory Description Documentation
data/processing/ 50+ dataset-specific preprocessing scripts (FineWeb, DCLM, Gallica, StarCoder, ...) data/README.md
data/tokenization/ Tokenize processed datasets and compute statistics data/README.md
data/synthetic_generation/ Generate synthetic training data using LLMs data/synthetic_generation/README.md
tokenizer_train/ Train and evaluate custom BPE tokenizers tokenizer_train/README.md
pretrain/train/ Distributed pretraining via NeMo + SLURM pretrain/README.md
pretrain/conversion/ Convert NeMo checkpoints to HuggingFace format pretrain/conversion/README.md
finetune/ SFT and RL fine-tuning with NeMo finetune/README.md
evaluation/ Benchmark evaluation with lighteval evaluation/README.md

Quickstart

Prerequisites

  • Access to Jean-Zay HPC cluster (or similar SLURM-based cluster with H100 GPUs)
  • Set the following environment variables in your .bashprofile:
export OpenLLM_OUTPUT=/path/to/shared/output     # Shared output directory
export SLURM_ACCOUNT_GPU="your_account@h100"      # SLURM GPU account
export SLURM_ACCOUNT_CPU="your_account@cpu"        # SLURM CPU account

Example pipeline (train + convert + evaluate)

# 1. Process data
cd data/ && source set_env.sh
python processing/fineweb2 --local   # test locally
python processing/fineweb2            # run on SLURM

# 2. Tokenize
python tokenization/run_tokenization.py datasets.yaml $OpenLLM_OUTPUT/data/tokens \
    --tokenizer_name OpenLLM-France/Luciole-1B-Base

# 3. Train
cd ../pretrain/train/
python slurm_launcher.py --output_dir $OpenLLM_OUTPUT/pretrain/my_exp \
    --arch nemotron1b --mode phase1 --datamix datamixes/luciole_phase1.json

# 4. Convert checkpoints to HuggingFace format
cd ../conversion/
sbatch convert.slurm $OpenLLM_OUTPUT/pretrain/my_exp

# 5. Evaluate
cd ../../evaluation/
python evaluate_experiment.py $OpenLLM_OUTPUT/pretrain/my_exp tasks/en.txt

Supported Model Architectures

Recipes in pretrain/train/recipes/:

Family Sizes
LLaMA 1B, 3B, 8B, 21B, 24B, 70B
Mistral 12B, Small 24B
Mixtral 8x7B
Nemotron 1B, 4B, 8B, 22B, 23B + variants
NemotronH 8B, 47B (hybrid Mamba/Transformer)

Environment Setup

Each pipeline stage uses its own environment. See the README in each subdirectory for setup instructions:

  • Data processing: data/README.md — conda datatrove-env (Python 3.10) + datatrove fork
  • Pretraining: pretrain/README.md — NeMo module on Jean-Zay + zarr
  • Fine-tuning: finetune/README.md — NeMo-RL with uv
  • Evaluation: evaluation/README.md — conda eval-env + lighteval fork

Key Environment Variables

Variable Description
OpenLLM_OUTPUT Shared output directory for all pipeline stages
SLURM_ACCOUNT_GPU SLURM account for GPU jobs (e.g. wuh@h100)
SLURM_ACCOUNT_CPU SLURM account for CPU jobs (e.g. qgz@cpu)
HF_HOME HuggingFace cache directory
HF_MODELS_CACHE Local mirror of HuggingFace models
HF_DATASETS_MIRROR Local mirror of HuggingFace datasets

Linting

pre-commit run --all-files
# Or directly:
ruff check . --fix
ruff format .

License

This project is licensed under the GNU General Public License v3 — see LICENSE for details.

See NOTICE for copyright and attribution, including the ownership and licenses of the Luciole models and datasets.

About

No description, website, or topics provided.

Resources

Code of conduct

Contributing

Security policy

Stars

17 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors

Languages