DiM: Diffusion Morphs

Zander W. Blasingame and Chen Liu

Clarkson University

DiM IEEE TBIOM 2024(Journal) and IJCB 2024 (Oral)
Fast-DiM IEEE Security & Privacy

DiM Paper DiM (TBIOM) Fast-DiM Paper Fast-DiM (IEEE S&P) Code Cite

We propose Diffusion Morphs (DiM) a state-of-the-art method for creating morphed faces using diffusion models. Our proposed method has unrivaled visual fidelity and outperforms previous GAN-based face morphs. We illustrate the difference between our morphs and those produced by GANs and landmark-based methods.

Abstract

Face morphing attacks seek to deceive a Face Recognition (FR) system by presenting a morphed image consisting of the biometric qualities from two different identities with the aim of triggering a false acceptance with one of the two identities, thereby presenting a significant threat to biometric systems. The success of a morphing attack is dependent on the ability of the morphed image to represent the biometric characteristics of both identities that were used to create the image. We present a novel morphing attack that uses a Diffusion-based architecture to improve the visual fidelity of the image and the ability of the morphing attack to represent characteristics from both identities.We demonstrate the effectiveness of the proposed attack by evaluating its visual fidelity via the Fréchet Inception Distance (FID). Also, extensive experiments are conducted to measure the vulnerability of FR systems to the proposed attack. The ability of a morphing attack detector to detect the proposed attack is measured and compared against two state-of-the-art GAN-based morphing attacks along with two Landmark-based attacks. Additionally, a novel metric to measure the relative strength between different morphing attacks is introduced and evaluated.

The face morphing attack attempts to fool a Face Recongition (FR) system by producing a single image which registers a false accept with both original identities. Previous face morphing attacks used landmark-based methods which align and warp the two faces before performing a pixel-wise average between the two faces. Such techniques are prone to considerably artefacts, especially outside the center of the face. Conversely, Generative Adversarial Network (GAN) based morphs produce more realistic looking faces but their performance, but their effectiveness in fooling the FR system leaves much to be desired. We propose DiM, a novel face morphing algorithm which combats this problem by producing highly effective morphs with high visual fidelity.

Proposed Face Morphing Method (DiM)

Diffusion models can be used to generate images by solving the Probability Flow ODE which comes with the nice property that it is reversible. This means that given an initial image we can encode it into the noisy latent representation in a bijective manner. Using this property and an additional image encoder, i.e. the diffusion autoencoder, we can encode our bona fide images into a conditional and latent representation. We then interpolate between these two representations to produce our morphed representation. Then this morphed representation is used as the starting point for the diffusion model, ultimately produce the morphed image.

Overview of DiM Illustration of the morph creation process where green traces indicate identity a, red traces denote identity b, and blue traces denote the morphed image.

Detailed comparison between DiM and MIPGAN-II. Notice the prominent artefacts in MIPGAN-II morph and strage high-frequency textures overlaid across the skin. In comparison the DiM morph looks much more natural.

Vulnerability of the FaceNet, VGGFace2, and ArcFace FR systems to different morphing attacks. Note, the newer and more accurate ArcFace FR system is more vulnerable to these attacks than older FR systems. Overall, DiM is the most effective morphing attack.

Ability of a morph detector to detect unseen morphing attacks. The morph detection algorithm is particularly vulnerable to DiM if not trained against it.

Impact of pre-processing and interpolation on morphed images. Variant A has no pre-processing and uses spherical linear interpolation (slerp) for the noisy latents. Variant B has no pre-processing and uses linear interpolation. Variant C uses a pixel-wise average pre-morph and slerp. Variant D uses an OpenCV based pre-morph and slerp. We observe that the interpolation strategy plays a large impact on the appearance of the morphed images, see Variant A vs Variant B. Variant C has slightly fewer visual artefacts than Variant A.

Fast-DiM: Towards Fast Diffusion Morphs

By using higher-order numerical ODE solvers to solve the Probability Flow ODE we can reduce the number of Network Function Evaluations (NFE) while maintaining comparable morphing performance to the vanilla DiM variants. We call this method which uses higher-order ODE solvers, Fast-DiM.

Fast-DiM Results Fast-DiM achieves similar performance to the vanilla DiM variants with a significant reduction in NFE.

Poster

BibTeX


        @article{blasingame_dim,
           title={Leveraging Diffusion for Strong and High Quality Face Morphing Attacks},
           volume={6},
           ISSN={2637-6407},
           url={http://dx.doi.org/10.1109/TBIOM.2024.3349857},
           DOI={10.1109/tbiom.2024.3349857},
           number={1},
           journal={IEEE Transactions on Biometrics, Behavior, and Identity Science},
           publisher={Institute of Electrical and Electronics Engineers (IEEE)},
           author={Blasingame, Zander W. and Liu, Chen},
           year={2024},
           month=jan, pages={118–131}}
        
        @article{blasingame_fast_dim,
           title={Fast-DiM: Towards Fast Diffusion Morphs},
           volume={22},
           ISSN={1558-4046},
           url={http://dx.doi.org/10.1109/MSEC.2024.3410112},
           DOI={10.1109/msec.2024.3410112},
           number={4},
           journal={IEEE Security & Privacy},
           publisher={Institute of Electrical and Electronics Engineers (IEEE)},
           author={Blasingame, Zander W. and Liu, Chen},
           year={2024},
           month=jul, pages={103–114}}