FrescoDiffusion

Official inference code for FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion.

Project page · arXiv paper · FrescoArchive dataset

Theory

FrescoDiffusion is a training-free two-pass image-to-video method. The first pass generates a video at the native resolution of the Wan backbone and retains its latent trajectory as a global spatiotemporal prior. The second pass denoises overlapping high-resolution tiles and combines their velocity predictions with the upscaled prior through the paper's closed-form weighted least-squares update. This preserves fine local detail while keeping motion and composition coherent across the full canvas.

The released path implements the global FrescoDiffusion schedule used in the paper:

[ \lambda(t)=\lambda_{\mathrm{base}}\cos\left(\frac{\pi t}{2}\right)\mathbf{1}[t\leq\tau]. ]

Behavior

The implementation uses Wan2.2-I2V-A14B, 81 frames, 16 fps, six denoising steps, guidance scale 1.0, 480×832 tiles, 30% overlap, linear edge-aware blending, and the paper defaults lambda_base=1.5 and cutoff=0.1.

The six-step release path uses a pinned LightX2V adapter for both Wan2.2 noise experts. The paper reports TurboDiffusion for its experimental timing configuration; this minimal release packages the LightX2V path from the source implementation.

Inference was developed for Linux with CUDA and evaluated on an NVIDIA H100 80 GB GPU. The pipeline uses BF16 model weights, FP8 transformer quantization, and INT8 text-encoder quantization. FP8 execution requires a CUDA GPU with compute capability 8.9 or newer.

Create a Python 3.12 environment and install the pinned dependencies:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Place each source image beside a text file with the same basename:

inputs/
├── 001.png
├── 001.txt
├── scene.jpg
└── scene.txt

Run inference:

python inference.py \
  --input-dir inputs \
  --output-dir outputs

Each pair produces outputs/<basename>.mp4. Existing videos are skipped unless --overwrite is set. The input is isotropically limited to four million pixels by default; --max-pixels changes that ceiling. --lambda-base, --cutoff, and --seed expose the method's reproducible controls.

Model weights are downloaded from Hugging Face on first use. The release uses a pinned BF16 repack of the Apache-2.0 Wan2.2 weights. Enough local storage for Wan2.2-I2V-A14B is required.

Code Structure and Implementation

• inference.py owns the CLI, paired-file discovery, model loading, quantization, two-pass orchestration, and video export.
• pipeline.py owns tiled denoising, edge-aware weighted fusion, the cosine prior schedule, and the closed-form FrescoDiffusion update.

The repository intentionally contains one inference path. It does not include experimental baselines, metric logging, segmentation variants, compatibility branches, or a test suite.

Citation

@inproceedings{frescodiffusion2026,
  title={{FrescoDiffusion}: 4K Image-to-Video with Prior-Regularized Tiled Diffusion},
  author={Hugo Caselles-Dupré and Mathis Koroglu and Guillaume Jeanneret and Arnaud Dapogny and Matthieu Cord},
  booktitle={The 2nd Workshop on Multimodal Large Language Models for Unified Comprehension and Generation (MUCG)},
  year={2026},
  url={https://arxiv.org/abs/2603.17555}
}

License

Apache 2.0. The Wan2.2 model, acceleration adapter, and FrescoArchive source images remain subject to their respective licenses and usage terms.