Zander W. Blasingame

The optimization of the latents and parameters of diffusion models with respect to some differentiable metric defined on the output of the model is a challenging and complex problem. The sampling for diffusion models is done by solving either the probability flow ODE or diffusion SDE wherein a neural network approximates the score function or related quantity, allowing a numerical ODE/SDE solver to be used. However, naïve backpropagation techniques are memory intensive, requiring the storage of all intermediate states, and face additional complexity in handling the injected noise from the diffusion term of the diffusion SDE. We propose a novel method based on the stochastic adjoint sensitivity method to calculate the gradientwith respect to the initial noise, conditional information, and model parameters by solving an additional SDE whose solution is the gradient of the diffusion SDE. We exploit the unique construction of diffusion SDEs to further simplify the formulation of the adjoint diffusion SDE and use a change-of-variables to simplify the solution to an exponentially weighted integral. Using this formulation we derive a custom solver for the adjoint SDE as well as the simpler adjoint ODE. The proposed adjoint diffusion solvers can efficiently compute the gradients for both the probability flow ODE and diffusion SDE for latents and parameters of the model. Lastly, we demonstrate the effectiveness of the adjoint diffusion solvers onthe face morphing problem.

Face morphing attacks seek to deceive a Face Recognition (FR) system by presenting a morphed image consisting of the biometric qualities from two different identities with the aim of triggering a false acceptance with one of the two identities, thereby presenting a significant threat to biometric systems. The success of a morphing attack is dependent on the ability of the morphed image to represent the biometric characteristics of both identities that were used to create the image. We present a novel morphing attack that uses a Diffusion-based architecture to improve the visual fidelity of the image and the ability of the morphing attack to represent characteristics from both identities. We demonstrate the effectiveness of the proposed attack by evaluating its visual fidelity via Fréchet Inception Distance (FID). Also, extensive experiments are conducted to measure the vulnerability of FR systems to the proposed attack. The ability of a morphing attack detector to detect the proposed attack is measured and compared against two state-of-the-art GAN-based morphing attacks along with two Landmark-based attacks. Additionally, a novel metric to measure the relative strength between different morphing attacks is introduced and evaluated.

Training-free guided generation is a widely used and powerful technique that allows the end user to exert further control over the generative process of diffusion models. In this work, we explore the guided generation from the perspective of optimizing the solution trajectory of a neural differential equation in a greedy manner. We present such a strategy as a unifying view on training-free guidance by showing that the greedy strategy is a first-order discretization of end-to-end optimization techniques. We show that a greedy guidance strategy makes good decisions and compare it to a guidance strategy using the ideal gradients found via the continuous adjoint equations. We then show how other popular training-free guidance strategies can be viewed in a unified manner from this perspective.

Diffusion models have quickly become the state-of-the-art for generation tasks across many different data modalities. An important ability of diffusion models is the ability to encode samples from the data distribution back into the sampling prior distribution. This is useful for performing alterations to real data samples along with guided generation via the continuous adjoint equations. We propose an algebraically reversible solver for diffusion SDEs that can exactly invert real data samples into the prior distribution.

Neural differential equations have quickly become the backbone behind many powerful generative models such as flow models and diffusion models. These models have quickly become the state-of-the-art for modeling high-dimensional distributions, thereby enabling high fidelity sampling of such complicated real-world data distributions, e.g., images, audio, video, proteins, &c. Within the context of generative modeling, neural differential equations provide the mechanisms for describing a transport map from some nice tractable distribution to our desired data distribution. Then by solving this neural differential equation we can map samples from the nice distribution (e.g., Gaussian samples) to our desired data distribution. We explore strategies for guiding the model to enable greater creative control of these powerful deep generative models. Additionally, we explore reversible solvers for enabling the ability to encode real data samples into the noise space whilst preserving two important properties: exact inversion and low distortion to edits (i.e., stability). Topics include: continuous adjoint equations for flow/diffusion models (e.g., how to backpropagate through flow/diffusion models), greedy guidance for efficient guidance strategies (e.g., how strategies which avoid backpropagating through the entire flow model), and reversible solvers for neural SDEs (e.g., numerical solvers for SDEs which enables exact reconstruction). We anticipate that this thesis will be of interest to anyone working on guided generation with flow/diffusion models or working latent editing of real data samples with such models. We hope this thesis provides a useful reference of the current state-of-the-art.