Zander W. Blasingame

The optimization of the latents and parameters of diffusion models with respect to some differentiable metric defined on the output of the model is a challenging and complex problem. The sampling for diffusion models is done by solving either the probability flow ODE or diffusion SDE wherein a neural network approximates the score function or related quantity, allowing a numerical ODE/SDE solver to be used. However, naïve backpropagation techniques are memory intensive, requiring the storage of all intermediate states, and face additional complexity in handling the injected noise from the diffusion term of the diffusion SDE. We propose a novel method based on the stochastic adjoint sensitivity method to calculate the gradientwith respect to the initial noise, conditional information, and model parameters by solving an additional SDE whose solution is the gradient of the diffusion SDE. We exploit the unique construction of diffusion SDEs to further simplify the formulation of the adjoint diffusion SDE and use a change-of-variables to simplify the solution to an exponentially weighted integral. Using this formulation we derive a custom solver for the adjoint SDE as well as the simpler adjoint ODE. The proposed adjoint diffusion solvers can efficiently compute the gradients for both the probability flow ODE and diffusion SDE for latents and parameters of the model. Lastly, we demonstrate the effectiveness of the adjoint diffusion solvers onthe face morphing problem.

Training-free guided generation is a widely used and powerful technique that allows the end user to exert further control over the generative process of flow/diffusion models. Generally speaking, two families of techniques have emerged for solving this problem for gradient-based guidance: namely, posterior guidance (i.e., guidance via projecting the current sample to the target distribution via the target prediction model) and end-to-end guidance (i.e., guidance by performing backpropagation throughout the entire ODE solve). In this work, we show that these two seemingly separate families can actually be unified by looking at posterior guidance as a greedy strategy of end-to-end guidance. We explore the theoretical connections between these two families and provide an in-depth theoretical of these two techniques relative to the continuous ideal gradients. Motivated by this analysis we then show a method for interpolating between these two families enabling a trade-off between compute and accuracy of the guidance gradients. We then validate this work on several inverse image problems and property-guided molecular generation.

Diffusion models have quickly become the state-of-the-art for generation tasks across many different data modalities. An important ability of diffusion models is the ability to encode samples from the data distribution back into the sampling prior distribution. This is useful for performing alterations to real data samples along with guided generation via the continuous adjoint equations. We propose an algebraically reversible solver for diffusion SDEs that can exactly invert real data samples into the prior distribution.

Neural differential equations have quickly become the backbone behind many powerful generative models such as flow models and diffusion models. These models have quickly become the state-of-the-art for modeling high-dimensional distributions, thereby enabling high fidelity sampling of such complicated real-world data distributions, e.g., images, audio, video, proteins, &c. Within the context of generative modeling, neural differential equations provide the mechanisms for describing a transport map from some nice tractable distribution to our desired data distribution. Then by solving this neural differential equation we can map samples from the nice distribution (e.g., Gaussian samples) to our desired data distribution. We explore strategies for guiding the model to enable greater creative control of these powerful deep generative models. Additionally, we explore reversible solvers for enabling the ability to encode real data samples into the noise space whilst preserving two important properties: exact inversion and low distortion to edits (i.e., stability). Topics include: continuous adjoint equations for flow/diffusion models (e.g., how to backpropagate through flow/diffusion models), greedy guidance for efficient guidance strategies (e.g., how strategies which avoid backpropagating through the entire flow model), and reversible solvers for neural SDEs (e.g., numerical solvers for SDEs which enables exact reconstruction). We anticipate that this thesis will be of interest to anyone working on guided generation with flow/diffusion models or working latent editing of real data samples with such models. We hope this thesis provides a useful reference of the current state-of-the-art.