The optimization of the latents and parameters of diffusion models with respect to some differentiable metric defined on the output of the model is a challenging and complex problem. The sampling for diffusion models is done by solving either the probability flow ODE or diffusion SDE wherein a neural network approximates the score function allowing a numerical ODE/SDE solver to be used. However, naïve backpropagation techniques are memory intensive, requiring the storage of all intermediate states, and face additional complexity in handling the injected noise from the diffusion term of the diffusion SDE. We propose a novel family of bespoke ODE solvers to the continuous adjoint equations for diffusion models, which we call AdjointDEIS. We exploit the unique construction of diffusion SDEs to further simplify the formulation of the continuous adjoint equations using exponential integrators. Moreover, we provide convergence order guarantees for our bespoke solvers. Significantly, we show that continuous adjoint equations for diffusion SDEs actually simplify to a simple ODE. Lastly, we demonstrate the effectiveness of AdjointDEIS for guided generation with an adversarial attack in the form of the face morphing problem.
We can guide the generative process by using AdjointDEIS. For example we use AdjointDEIS to create high quality face morphs which fool Face Recongition (FR) systems.
AdjointDEIS applied to face morphing achieves SOTA results.
We propose to solve the continous adjoint equations to solve the following optimization problem below. We exploit the unique structure of diffusion models to make these numerical solvers efficient.
Problem Statement Given a diffusion ODE solve the problem above.
The continuous adjoint equations for diffusion SDEs actually simplify to an ODE!
We can find the gradients for a number of quantities of interest and can find them for both diffusion ODEs and SDEs.
@inproceedings{blasingame2024adjointdeis,
title = {Adjoint{DEIS}: Efficient Gradients for Diffusion Models},
author = {Blasingame, Zander W. and Liu, Chen},
booktitle = {The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year = {2024},
url = {https://openreview.net/forum?id=fAlcxvrOEX},
}