Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 27 additions & 59 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,11 @@

**Yeto** fine-tunes language and diffusion models across cheap, geographically
scattered GPU capacity — spot instances, mixed regions, mixed clouds, even
mixed hardware families — via the [SkyPilot](https://skypilot.co) SDK. Its
asynchronous synchronization is based on **Decoupled DiLoCo**
([Douillard et al., arXiv 2604.21428](https://arxiv.org/abs/2604.21428)):
a Rust syncer merges parameter fragments from independent learner islands
(quorum + adaptive grace, token-weighted RDA, Nesterov outer step), so slow
links and preempted islands never block training.
mixed hardware families.

```
┌──────────────────────────────┐
│ syncer (Rust, hot path) │
│ syncer (hot path)
│ fragment ingest · RDA merge │
│ Nesterov outer step · bcast │
└──────┬───────┬───────┬───────┘
Expand Down Expand Up @@ -60,9 +55,16 @@ yeto status | logs <run> | down <run> # runs detach; Ctrl-C never kills them

## Architecture

Fleets launch via the [SkyPilot](https://skypilot.co) SDK; asynchronous
synchronization is based on **Decoupled DiLoCo**
([Douillard et al., arXiv 2604.21428](https://arxiv.org/abs/2604.21428)):
the syncer merges parameter fragments from independent learner islands
(quorum + adaptive grace, token-weighted RDA, Nesterov outer step), so slow
links and preempted islands never block training.

One protocol, one syncer, four learner backends. Every backend speaks the
same fragment protocol to the Rust syncer and runs the same DiLoCo step
boundary — the pull/merge/α-blend/push loop lives in one shared module
same fragment protocol and runs the same DiLoCo step boundary — the
pull/merge/α-blend/push loop lives in one shared module
(`yeto/diloco_sync.py`), so protocol changes land once and apply everywhere.

| backend | selector | scope | validation |
Expand All @@ -78,53 +80,6 @@ terminal contract — budget cutoff, authoritative final cut, checkpoint
marking — is shared by all backends (`finalization.py`,
`budget_finalization.py`, `final_marker.py`).

## Production source safety

Remote model and dataset branches or tags are resolved to immutable Hugging
Face commits before any learner is provisioned. Config, tokenizer, model,
dataset, prefetch, export, and sampling paths reuse those commits. Executable
model code is disabled by default; enabling `--trust-remote-code` is an
explicit trust decision and still uses the pinned commit. Saved artifacts
include `yeto_provenance.json` (or embedded diffusion provenance) with
resolved source identities and a Yeto source-tree digest.

Legacy pickled losses are disabled by default because deserialization executes
code. Existing custom/callable transport requires
`--allow-unsafe-pickled-loss` and is SHA-256 attested before every learner
loads it. See [docs/PROVENANCE.md](docs/PROVENANCE.md) for revision flags,
local-path behavior, custom diffusion-loader requirements, safe tensor
formats, and migration guidance.

## Diffusion

Yeto's diffusion learner (`--model-kind diffusion`) keeps the LM-style
contract: aliases are only repo-id shortcuts and raw Hugging Face ids pass
through unchanged. Models that Diffusers can load use the generic path first —
Yeto loads the pipeline, discovers the trainable denoiser, attaches LoRA,
builds conditioning from data rows, and reuses the normal DiLoCo sync
machinery. Model-family behavior that the generic path cannot infer lives in
behavior adapters under `yeto/diffusion/adapters/` (in-tree: PixArt); use
`--diffusion-adapter module:factory` for external models that need one.

Recover the authoritative merged adapter from a syncer checkpoint with:

```bash
yeto-diffusion-export --checkpoint yeto-state.ckpt --model <id> --output-dir out/
```

Pass the same LoRA, fragment-pattern, and external-adapter flags used by the
training run.

- `--diffusion-seed <integer>`: reproducible initialization, data ordering,
timestep sampling, and noise (RNG streams only; does not force
deterministic CUDA kernels). Diffusion-only.
- `--resize-mode center-crop` with `--height`/`--width`: preserve aspect
ratio by scaling to fill the target then center-cropping (default is
direct resize).

The complete backend architecture, data contract, artifact flow, adapter
boundary, and current limitations are in [docs/DIFFUSION.md](docs/DIFFUSION.md).

## Supported models

Aliases are sugar over `yeto/models.py` (this table is generated by
Expand All @@ -133,17 +88,29 @@ is the frozen-base footprint an island's GPUs must jointly hold (bf16 base,
LoRA) — add ~8 GB per GPU for activations/overhead, ×8 for full tuning;
"(Hub)" means the size is resolved from safetensors metadata at plan time.

Tested — a completed Yeto fine-tuning run on real hardware:

| alias | Hugging Face id | min island VRAM (GB) |
|---|---|---|
| `gemma4` | `google/gemma-4-12B-it` | 66 |
| `deepseek4flash` | `deepseek-ai/DeepSeek-V4-Flash` | 568 |
| `qwen3-0.6b` | `Qwen/Qwen3-0.6B` | (Hub) |
| `qwen36-27b` | `Qwen/Qwen3.6-27B` | 54 |
| `kimi-k3` | `moonshotai/Kimi-K3` | (Hub) |
| `glm52` | `zai-org/GLM-5.2` | 1488 |
| `laguna-s-2.1` | `poolside/Laguna-S-2.1` | (Hub) |

<details>
<summary>All other supported aliases (untested)</summary>

| alias | Hugging Face id | min island VRAM (GB) |
|---|---|---|
| `gemma4` | `google/gemma-4-12B-it` | 66 |
| `qwen3-8b` | `Qwen/Qwen3-8B` | 17 |
| `qwen35-4b` | `Qwen/Qwen3.5-4B` | 8 |
| `qwen35-9b` | `Qwen/Qwen3.5-9B` | 18 |
| `qwen35-9b-base` | `Qwen/Qwen3.5-9B-Base` | 18 |
| `qwen35-35b-a3b` | `Qwen/Qwen3.5-35B-A3B-Base` | 70 |
| `qwen35-397b-a17b` | `Qwen/Qwen3.5-397B-A17B` | 794 |
| `qwen36-27b` | `Qwen/Qwen3.6-27B` | 54 |
| `qwen36-35b-a3b` | `Qwen/Qwen3.6-35B-A3B` | 70 |
| `llama32-1b` | `meta-llama/Llama-3.2-1B` | 3 |
| `llama32-3b` | `meta-llama/Llama-3.2-3B` | 7 |
Expand All @@ -164,7 +131,6 @@ LoRA) — add ~8 GB per GPU for activations/overhead, ×8 for full tuning;
| `nemotron3-ultra` | `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16` | 1100 |
| `glm45-air` | `zai-org/GLM-4.5-Air` | 212 |
| `glm46` | `zai-org/GLM-4.6` | 714 |
| `glm52` | `zai-org/GLM-5.2` | 1488 |
| `llama4-scout` | `meta-llama/Llama-4-Scout-17B-16E-Instruct` | 218 |
| `llama4-maverick` | `meta-llama/Llama-4-Maverick-17B-128E-Instruct` | 800 |
| `qwen3-coder-480b` | `Qwen/Qwen3-Coder-480B-A35B-Instruct` | 960 |
Expand All @@ -190,6 +156,8 @@ LoRA) — add ~8 GB per GPU for activations/overhead, ×8 for full tuning;
| `deepseek31-base-bf16` | `unsloth/DeepSeek-V3.1-Base-BF16` | 1343 |
| `kimi-k2-base-bf16` | `unsloth/Kimi-K2-Base-BF16` | 2060 |

</details>

## Docs

[docs/DESIGN.md](docs/DESIGN.md) — merge math, blending, adaptive grace,
Expand Down
18 changes: 14 additions & 4 deletions scripts/gen_model_table.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,16 @@
import sys

sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from yeto.models import MODEL_ALIASES, MODEL_WEIGHT_GB # noqa: E402
from yeto.models import MODEL_ALIASES, MODEL_WEIGHT_GB, TESTED_ALIASES # noqa: E402

SECTION_START = "## Supported models"
SECTION_END = "## Docs"


def table() -> str:
def table(aliases) -> str:
rows = ["| alias | Hugging Face id | min island VRAM (GB) |", "|---|---|---|"]
for alias, hf in MODEL_ALIASES.items():
for alias in aliases:
hf = MODEL_ALIASES[alias]
gb = MODEL_WEIGHT_GB.get(alias)
size = f"{gb:g}" if gb is not None else "(Hub)"
rows.append(f"| `{alias}` | `{hf}` | {size} |")
Expand All @@ -35,7 +36,16 @@ def section() -> str:
LoRA) — add ~8 GB per GPU for activations/overhead, ×8 for full tuning;
"(Hub)" means the size is resolved from safetensors metadata at plan time.

{table()}
Tested — a completed Yeto fine-tuning run on real hardware:

{table(a for a in MODEL_ALIASES if a in TESTED_ALIASES)}

<details>
<summary>All other supported aliases (untested)</summary>

{table(a for a in MODEL_ALIASES if a not in TESTED_ALIASES)}

</details>

"""

Expand Down
2 changes: 1 addition & 1 deletion tests/test_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ def test_readme_table_matches_alias_table():
import re

readme = (pathlib.Path(__file__).resolve().parents[1] / "README.md").read_text()
table_aliases = set(re.findall(r"^\| `([a-z0-9\-]+)` \| `", readme, flags=re.M))
table_aliases = set(re.findall(r"^\| `([a-z0-9.\-]+)` \| `", readme, flags=re.M))
assert table_aliases == set(MODEL_ALIASES), (
"README model table is out of sync with yeto/models.py — regenerate it"
)
Expand Down
14 changes: 14 additions & 0 deletions yeto/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
"gemma4": "google/gemma-4-12B-it",
"deepseek4flash": "deepseek-ai/DeepSeek-V4-Flash",
# Qwen dense + MoE
"qwen3-0.6b": "Qwen/Qwen3-0.6B",
"qwen3-8b": "Qwen/Qwen3-8B",
"qwen35-4b": "Qwen/Qwen3.5-4B",
"qwen35-9b": "Qwen/Qwen3.5-9B",
Expand All @@ -50,6 +51,7 @@
"kimi-k2-thinking": "moonshotai/Kimi-K2-Thinking",
"kimi-k25": "moonshotai/Kimi-K2.5",
"kimi-k26": "moonshotai/Kimi-K2.6",
"kimi-k3": "moonshotai/Kimi-K3", # size via Hub metadata
# DeepSeek
"deepseek31": "deepseek-ai/DeepSeek-V3.1",
"deepseek-r1": "deepseek-ai/DeepSeek-R1",
Expand All @@ -71,6 +73,7 @@
"minimax-m3": "MiniMaxAI/MiniMax-M3", # size via Hub metadata
"kimi-k27-code": "moonshotai/Kimi-K2.7-Code", # size via Hub metadata
"mistral-small3": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
"laguna-s-2.1": "poolside/Laguna-S-2.1", # 118B-A8B agentic coding; size via Hub metadata
# Ornith (DeepReinforce agentic-coding family, MIT)
"ornith-9b": "deepreinforce-ai/Ornith-1.0-9B",
"ornith-31b": "deepreinforce-ai/Ornith-1.0-31B",
Expand Down Expand Up @@ -105,6 +108,17 @@
# fp8 repo yourself and pass the path via --model.
}

# Aliases with a completed Yeto fine-tuning run on real hardware. The
# README surfaces these first; everything else is supported but untested.
TESTED_ALIASES = {
"qwen36-27b",
"deepseek4flash",
"qwen3-0.6b",
"glm52",
"kimi-k3",
"laguna-s-2.1",
}

DIFFUSION_MODEL_ALIASES = {
"flux": "black-forest-labs/FLUX.1-dev",
"flux-schnell": "black-forest-labs/FLUX.1-schnell",
Expand Down
Loading