Reject SDE algorithm types on DPMSolverMultistepInverseScheduler - #14356
Open
MohammadSadeghSalehi wants to merge 2 commits into
Open
Reject SDE algorithm types on DPMSolverMultistepInverseScheduler#14356MohammadSadeghSalehi wants to merge 2 commits into
MohammadSadeghSalehi wants to merge 2 commits into
Conversation
Under inversion sigmas increase with step, so lambda decreases and h = lambda_t - lambda_s is negative by construction. Measured on the reported sde-dpmsolver++ config: h=-0.458995 and 1-exp(-2h)=-1.504252. Both SDE variance terms then take the square root of a negative number and the first step is all-NaN. The deterministic branches contain no square root and are unaffected. The same NaN appears without Karras sigmas. Raise ValueError at construction for sde-dpmsolver and sde-dpmsolver++ instead of failing after a model is loaded. Leave the SDE arms in the copied update methods: they are synchronised with the forward scheduler where those types remain valid. Leave the uncopied step SDE arms so variance_noise stays used. Fixes huggingface#10748
The SDE algorithm types cannot be constructed on the inverse scheduler, so the class docstring must not describe stabilising an SDE variant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Under inversion the noise schedule runs from clean to noisy, so sigmas increase with step index. Since
lambda = log(alpha) - log(sigma), an increasing sigma makes lambda decrease, andh = lambda_t - lambda_sis therefore negative by construction. On the reported configuration (sde-dpmsolver++, Karras, 25 steps) the first step measureslambda_s=3.534699,lambda_t=3.075704,h=-0.458995, and1 - exp(-2h) = -1.504252.Both stochastic update branches then take the square root of a negative number:
algorithm_typeh < 0sde-dpmsolver++sqrt(1 - exp(-2h))exp(-2h) > 1sde-dpmsolversqrt(exp(2h) - 1)exp(2h) < 1The deterministic branches (
dpmsolver,dpmsolver++) contain no square root and step without NaN. The same first-step NaN appears withuse_karras_sigmas=False, so the defect is not specific to Karras.No published method defines stochastic DPM-Solver inversion. Inventing a sign-corrected variance term (for example
sqrt(exp(-2h) - 1)) would define an unvalidated sampler and turn a loud all-NaN failure into a quiet wrong latent. Inversion is deterministic by purpose: recovering the noise that produces a given sample. This PR therefore rejects the unsupported types at construction with a concise error, and corrects the class docstring so it only advertises the deterministic algorithms.Why the SDE code paths stay
Four of the SDE arms sit inside methods marked
# Copied fromDPMSolverMultistepScheduler, where those algorithm types remain valid. Deleting them would mean removing the copy headers and unlinking those methods from the forward scheduler for no behavioural change;make fix-copiesexits clean with the arms left in place. The two uncopied SDE arms instepstill consumevariance_noise; removing them would leave a parameter that is accepted and ignored, which is the defect already reported in #14353.Backwards compatibility
A saved config with
algorithm_type="sde-dpmsolver++"(orsde-dpmsolver) will now raiseValueErrorat load or construction instead of producing all-NaN samples at the first step. That config already could not invert correctly, so the change replaces a silent wrong answer with an immediate error. Happy to convert the raise into a deprecation warning with a transition window if maintainers prefer that.Test plan
test_rejects_stochastic_algorithm_typesfails at pristine base4b8e466bf(ValueError not raised) and passes on this branchsde-dpmsolverandsde-dpmsolver++raises; message names the scheduler, the type, and that inversion is deterministicdpmsolveranddpmsolver++remain finite (no NaN)tests/schedulers/test_scheduler_dpm_multi_inverse.py: 39 passedmake styleandmake fix-copiesexit 0Platform: macOS arm64, CPU, CPython 3.12.
Fixes #10748
Self-review
Scope
Commit range on
scheduler-correctnessagainst base4b8e466bf.Files:
src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.pytests/schedulers/test_scheduler_dpm_multi_inverse.pyBlocking issues
None.
Non-blocking issues
sde-dpmsolverdeprecate("algorithm_types dpmsolver and sde-dpmsolver", ...)keeps the historical key so existing warning filters stay stable. The user-facing message only deprecatesdpmsolver.Dead code (advisory): left on purpose
# Copied fromconvert_model_outputdpm_solver_first_order_updatemultistep_dpm_solver_second_order_updatemultistep_dpm_solver_third_order_updatestepvariance_noiseDocumentation impact
algorithm_typeandeuler_at_finalupdated so the docstring only describes supported behaviour.docs/sourcepage for this scheduler was changed.Copied-from blast radius
__init__# Copied fromblock# Copied fromthis classVerdict
READY. Fix before submitting: nothing. Leave for review: whether the construction-time raise should be a deprecation window instead.