Skip to content

Add Cosmos3 FP8 quantization cookbook and serving examples #298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wkutak wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Add Cosmos3 FP8 quantization cookbook and serving examples #298

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b6bdabb
Add Cosmos3 FP8 quantization cookbook and serving examples
wkutak Jul 30, 2026
a00d3c3
Removed --revision argument from vllm-omni commands
wkutak Jul 31, 2026
b75431f
vLLM image version change from 'cosmos3' to 'nightly-cb8104839c141609…
wkutak Aug 4, 2026
a25fbc6
Convert all quantize paths to quantization, remove dead comments from…
wkutak Aug 4, 2026
e635690
Fixed vllm inference bugs, removed too long notebook output cells
wkutak Aug 7, 2026
38da028
Merge branch 'main' into cosmos3-fp8-quantization-recipe
wkutak Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 56 additions & 10 deletions cookbooks/cosmos3/generator/action/run_fd_with_vllm_omni.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,53 @@
"curl http://localhost:8001/v1/models\n",
"```\n",
"\n",
"The request cells below use whichever Nano or Edge server is active at the configured endpoint. To keep outputs from separate model runs, set `COSMOS3_VLLM_OUTPUT_ROOT` to a different directory before running the configuration cell.\n"
"The request cells below use whichever Nano or Edge server is active at the configured endpoint. To keep outputs from separate model runs, set `COSMOS3_VLLM_OUTPUT_ROOT` to a different directory before running the configuration cell.\n",
"\n",
"### Quantized Checkpoints\n",
"\n",
"FP8 builds are published only for `Cosmos3-Nano` and `Cosmos3-Super`; `Cosmos3-Edge` has no FP8 build. The FP8 weights live on the `fp8` revision of the same repo. Download `Cosmos3-Nano` FP8 to a local directory:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "785bfdde",
"metadata": {},
"outputs": [],
"source": [
"from huggingface_hub import snapshot_download\n",
"\n",
"snapshot_download(\"nvidia/Cosmos3-Nano\", revision=\"fp8\", local_dir=\"cosmos3-nano-fp8\")"
]
},
{
"cell_type": "markdown",
"id": "4e98f90a",
"metadata": {},
"source": [
"then serve that directory:\n",
"```bash\n",
"docker rm -f cosmos3-vllm-omni-notebook 2>/dev/null || true\n",
"\n",
"docker run -d --name cosmos3-vllm-omni-notebook \\\n",
" --runtime nvidia --gpus '\"device=0\"' \\\n",
" -e CUDA_DEVICE_ORDER=PCI_BUS_ID \\\n",
" -v \"/mnt/sdb/.cache/huggingface:/root/.cache/huggingface\" \\\n",
" -v \"$PWD/cosmos3-nano-fp8:/models/cosmos3-nano-fp8\" \\\n",
" -v \"$PWD:/workspace\" \\\n",
" -p 8001:8000 --ipc=host \\\n",
" vllm/vllm-omni:cosmos3 \\\n",
" vllm serve /models/cosmos3-nano-fp8 \\\n",
" --served-model-name nvidia/Cosmos3-Nano \\\n",
" --omni \\\n",
" --model-class-name Cosmos3OmniDiffusersPipeline \\\n",
" --allowed-local-media-path / \\\n",
" --port 8000 \\\n",
" --init-timeout 1800\n",
"\n",
"# Wait until this returns model metadata before running the inference cells.\n",
"curl http://localhost:8001/v1/models\n",
"```"
]
},
{
Expand Down Expand Up @@ -1343,8 +1389,10 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "fdvl-hand-pose-spec-code",
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import sys\n",
Expand Down Expand Up @@ -1391,9 +1439,7 @@
"print(\"caption:\", hand_pose_record[\"prompt\"])\n",
"print(\"action shape:\", tuple(hand_pose_sample[\"action\"].shape))\n",
"print(hand_pose_fd_input_path.read_text())\n"
],
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
Expand All @@ -1405,8 +1451,10 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "fdvl-hand-pose-run-code",
"metadata": {},
"outputs": [],
"source": [
"hand_pose_result = submit_forward_dynamics(\n",
" hand_pose_record,\n",
Expand All @@ -1416,9 +1464,7 @@
"hand_pose_generated_video = hand_pose_result[\"video_path\"]\n",
"assert hand_pose_generated_video.exists(), f\"missing generated video: {hand_pose_generated_video}\"\n",
"print(\"generated:\", hand_pose_generated_video)\n"
],
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
Expand All @@ -1430,15 +1476,15 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "fdvl-hand-pose-preview-code",
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import Video, display\n",
"\n",
"display(Video(str(hand_pose_generated_video), embed=True))\n"
],
"execution_count": null,
"outputs": []
]
}
],
"metadata": {
Expand Down
51 changes: 49 additions & 2 deletions cookbooks/cosmos3/generator/action/run_id_with_vllm_omni.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
"id": "license-header",
"metadata": {},
"source": [
"<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: OpenMDW-1.1 -->"
"<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n",
"SPDX-License-Identifier: OpenMDW-1.1 -->"
]
},
{
Expand Down Expand Up @@ -83,7 +84,53 @@
"curl http://localhost:8001/v1/models\n",
"```\n",
"\n",
"The request cells below use whichever Nano or Edge server is active at the configured endpoint. To keep outputs from separate model runs, set `COSMOS3_VLLM_OUTPUT_ROOT` to a different directory before running the configuration cell.\n"
"The request cells below use whichever Nano or Edge server is active at the configured endpoint. To keep outputs from separate model runs, set `COSMOS3_VLLM_OUTPUT_ROOT` to a different directory before running the configuration cell.\n",
"\n",
"### Quantized Checkpoints\n",
"\n",
"FP8 builds are published only for `Cosmos3-Nano` and `Cosmos3-Super`; `Cosmos3-Edge` has no FP8 build. The FP8 weights live on the `fp8` revision of the same repo. Download `Cosmos3-Nano` FP8 to a local directory:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "053b62b4",
"metadata": {},
"outputs": [],
"source": [
"from huggingface_hub import snapshot_download\n",
"\n",
"snapshot_download(\"nvidia/Cosmos3-Nano\", revision=\"fp8\", local_dir=\"cosmos3-nano-fp8\")"
]
},
{
"cell_type": "markdown",
"id": "3f148649",
"metadata": {},
"source": [
"then serve that directory:\n",
"```bash\n",
"docker rm -f cosmos3-vllm-omni-notebook 2>/dev/null || true\n",
"\n",
"docker run -d --name cosmos3-vllm-omni-notebook \\\n",
" --runtime nvidia --gpus '\"device=0\"' \\\n",
" -e CUDA_DEVICE_ORDER=PCI_BUS_ID \\\n",
" -v \"/mnt/sdb/.cache/huggingface:/root/.cache/huggingface\" \\\n",
" -v \"$PWD/cosmos3-nano-fp8:/models/cosmos3-nano-fp8\" \\\n",
" -v \"$PWD:/workspace\" \\\n",
" -p 8001:8000 --ipc=host \\\n",
" vllm/vllm-omni:cosmos3 \\\n",
" vllm serve /models/cosmos3-nano-fp8 \\\n",
" --served-model-name nvidia/Cosmos3-Nano \\\n",
" --omni \\\n",
" --model-class-name Cosmos3OmniDiffusersPipeline \\\n",
" --allowed-local-media-path / \\\n",
" --port 8000 \\\n",
" --init-timeout 1800\n",
"\n",
"# Wait until this returns model metadata before running the inference cells.\n",
"curl http://localhost:8001/v1/models\n",
"```"
]
},
{
Expand Down
Loading