Derive DDIM prev_timestep from the inference schedule - #14357
Open
MohammadSadeghSalehi wants to merge 2 commits into
Open
Derive DDIM prev_timestep from the inference schedule#14357MohammadSadeghSalehi wants to merge 2 commits into
MohammadSadeghSalehi wants to merge 2 commits into
Conversation
DDIMScheduler.step used a uniform stride, num_train_timesteps // num_inference_steps, to form prev_timestep. That matches leading and trailing spacing but not linspace: for 1000 train steps and 10 inference steps the schedule steps by 111 while the formula steps by 100, so alpha_prod_t_prev is read from the wrong index. Look up the previous value as the next entry of self.timesteps, as DDPMScheduler.previous_timestep already does. Leading and trailing stay bit-identical: intermediate prev values match the old stride, and the final step still selects final_alpha_cumprod whether the formula yields -100 or the list yields -1. Default spacing remains leading, so only callers that set linspace change. Apply the same helper to DDIMParallel, CogVideoXDDIM and CogVideoXDPM via # Copied from. PNDM and RePaint keep the stride formula: PLMS/PRK half-steps and RePaint jump schedules are not consecutive schedule entries. Fixes huggingface#12633 Fixes huggingface#11347
doc-builder style rewrites these four copied docstrings to the project max length of 119. Keep the copies in sync with make fix-copies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DDIMScheduler.stepformed the previous diffusion index with a uniform stride:That expression assumes every inference step advances by the same integer.
set_timestepswithtimestep_spacing="linspace"does not produce that schedule. For 1000 train steps and 10 inference steps the materialised list strides by 111 while the expression strides by 100, so every step readsalpha_prod_t_prevfrom the wrong index.DDIMSchedulerdefaults totimestep_spacing="leading". Only callers that explicitly selectlinspacechange behaviour. Default sampling outputs do not move.Proof that leading and trailing are no-ops
Full deterministic loops (
eta=0) comparing schedule lookup against the historical stride, same residuals:leading0.000000e+00trailing0.000000e+00linspace2.584383e+02On
leadingandtrailing, intermediate prev values match the old stride exactly. At the final step the formula can yield a large negative (for example-100) while the list yields-1; both are< 0and both selectfinal_alpha_cumprod, so the sample is still bit-identical. Onlinspace, 9 of 10 steps changealpha_prod_t_prev. Example prev sequence for 1000/10 linspace: schedule[888, 777, …, 0, -1]versus hardcoded[899, 788, …, 11, -100].Change
Add
previous_timestep()that returns the next entry ofself.timesteps(last entry →-1). Use it fromstep(and the parallel batch / variance paths). Same helper is# Copied frominto the three DDIM-family schedulers below;make fix-copiesis clean.Scope: four changed, two deliberately not
Changed
DDIMSchedulerDDIMParallelSchedulerstep,_get_variance, and batch pathCogVideoXDDIMSchedulerstepCogVideoXDPMSchedulerstepNone of those
stepmethods was a# Copied fromof another; the formula was independently duplicated. The new helper is the shared copy point.Not changed
PNDMSchedulerstep_prkuses half-stride intermediates that are not entries ofself.timesteps.step_plmsrewritesprev_timestepandtimestepwhencounter == 1. Withskip_prk_steps, the list contains deliberate duplicates, so a first-match index would pick the wrong row.RePaintSchedulert, while the stride formula always means one reverse diffusion step. List lookup would break jump steps.Off-schedule fallback
When
stepis called with atimestepthat is not an entry ofself.timesteps,previous_timestepfalls back to the historical stride. That is not a defensive guess: replacing the fallback with aValueErrorfails five existing tests that callstep()outside the inference loop:test_etatest_inference_stepstest_steps_offsettest_time_indicestest_timestep_spacingPipeline loops always pass schedule entries, so the linspace fix is not diluted by the fallback. Happy to convert the fallback to a raise and update those five tests if maintainers prefer a stricter API.
Issues
Closes #12633 and #11347. They are one root cause (stride vs
linspaceschedule), not two independent bugs.Test plan
test_prev_timestep_matches_schedulefails at pristine base4b8e466bfand passes heremax_abs_diff=0)tests/schedulers/test_scheduler_ddim.pyandtest_scheduler_ddim_parallel.py: 78 passedmake styleexit 0 (ruff + doc-builder + extra checks)make fix-copiesexit 0Platform: macOS arm64, CPU, CPython 3.12.
Self-review
Scope
Branch
prev-timestep-schedule-mismatchagainst base4b8e466bf.Files:
src/diffusers/schedulers/scheduling_ddim.pysrc/diffusers/schedulers/scheduling_ddim_parallel.pysrc/diffusers/schedulers/scheduling_ddim_cogvideox.pysrc/diffusers/schedulers/scheduling_dpm_cogvideox.pytests/schedulers/test_scheduler_ddim.pyBlocking issues
None.
Non-blocking issues
PNDM and RePaint still use the stride formula
Off-schedule
step()fallbackDead code
None introduced.
Copied-from
DDIMScheduler.previous_timestepprevious_timestep# Copied fromDDIM;make fix-copiescleanstepbodiesprevious_timestepEvidence
test_prev_timestep_matches_schedulefails at base (AttributeError: previous_timestep) and passes here.max_abs_diff=0); linspace changes (max_abs_diff≈2.58e2).timestep_spacing="leading".make styleandmake fix-copiesexit 0 on this branch (pre-existing tree-wide reformat noise fromruff format/ doc-builder on unrelated files was not committed).Verdict
READY.