We formulate guidance in diffusion models as a variational control problem. Intuitively, at each diffusion sampling step, we infer controls (via an optimization routine) which guide the noisy state towards a direction which best satisfies a terminal cost subject to a regularization penalty.
This repo builds on top of the official RED-Diff implementation. Clone the repository and navigate to the project directory:
git clone git@github.com:czi-ai/oc-guidance.git
cd oc-guidanceWe provide both a Conda environment file (environment.yml) and a requirements.txt. You can use either of the following methods to set up the environment:
conda env create -f environment.yml
conda activate oc-guidance
# OR
conda create --name <env_name> python=3.11
conda activate <env_name>
pip install -r requirements.txtIn this work, we use:
- Unconditional ImageNet checkpoints from the ADM repository.
- Pretrained unconditional FFHQ checkpoint from the DPS repository.
📁 For convenience, we recommend placing the checkpoints in a folder named
ckpts/under the corresponding dataset directory.
For example:ckpts/ ├── imagenet/ │ └── model.pt └── ffhq/ └── model.pt
Make sure to update the ckpt_root argument in config and bash files accordingly when running inference.
- For FFHQ, we selected the first 1000 validation images for all tasks except Blind Image Deblurring (BID), where we used only 100 images.
- For ImageNet, we randomly selected 1000 images.
We provide text files listing the selected image from Imagenet under the misc/ directory:
sr3_top1k.txt: used for all tasks except BID.sr3_top100.txt: used only for the BID task.
Configuration management is handled using Hydra.
All configuration files are located in the _configs/ directory, with algorithm-specific configurations.
We provide the exact scripts used in our experiments—both for our method and for the baselines—under the scripts/ directory. These scripts are organized by task (e.g., super-resolution, deblurring, inpainting) to ensure easy reproducibility.
Below is a minimal example command for running non-linear deblurring with our method on ImageNet:
python main.py \
--config-name=imagenet256_uncond \
diffusion=ddpm \
classifier=none \
algo=ndtm \
algo.denoise=False \
algo.M=2 \
algo.sigma_y=0.005 \
algo.u_lr=0.01 \
algo.u_lr_scheduler=linear \
algo.eta=0.1 \
algo.w_terminal=50.0 \
algo.w_score=ddim \
algo.w_control=ddim \
algo.init_xT=guided \
algo.init_control=zero \
algo.combine_fn.name=additive \
algo.combine_fn.gamma_t=4.0 \
deg=non-linear_deblur \
loader=imagenet256_ddrm \
loader.batch_size=1 \
loader.num_workers=2 \
dist.num_processes_per_node=1 \
dist.port=8051 \
exp.t_start=0 \
exp.t_end=600 \
exp.num_steps=50 \
exp.seed=0 \
exp.stride=ddpm_uniform \
exp.root=/path/to/experiment/root \
exp.name=samples \
exp.ckpt_root=/path/to/pretrained/checkpoints \
exp.samples_root=/path/to/save/samples \
exp.overwrite=True \
exp.use_wandb=False \
exp.save_ori=True \
exp.save_deg=True \
exp.smoke_test=1To simplify usage, you can instead run the provided shell script directly:
sh scripts/method_scripts/non_linear_deblur/imagenet/ndtm.sh
@misc{pandey2025variationalcontrolguidancediffusion,
title={Variational Control for Guidance in Diffusion Models},
author={Kushagra Pandey and Farrin Marouf Sofian and Felix Draxler and Theofanis Karaletsos and Stephan Mandt},
year={2025},
eprint={2502.03686},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2502.03686},
}For questions, clarifications, or collaborations, feel free to reach out to the authors:
- 📫 Kushagra Pandey – pandeyk1@uci.edu – kpandey008.github.io
- 📫 Farrin Marouf Sofian – fmaroufs@uci.edu – farrinsofian.com
This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to opensource@chanzuckerberg.com.
If you believe you have found a security issue, please responsibly disclose by contacting us at security@chanzuckerberg.com.
