MobileDiffusion

What is MobileDiffusion?

MobileDiffusion is a novel approach designed for rapid text-to-image generation on mobile devices. It is an efficient latent diffusion model tailored specifically for mobile deployment, featuring a text encoder, diffusion UNet, and image decoder components. MobileDiffusion leverages DiffusionGAN for one-step sampling during inference, allowing for the generation of high-quality images in just half a second on premium iOS and Android devices. With a compact size of 520 million parameters, MobileDiffusion's efficiency in terms of latency and size makes it a promising option for on-device image generation while adhering to responsible AI practices.

Who created MobileDiffusion?

MobileDiffusion was created by a team that includes Zhisheng Xiao, Yanwu Xu, Jiuqiang Tang, Haolin Jia, Lutz Justen, Daniel Fenner, Ronald Wotzlaw, Jianing Wei, Raman Sarokin, Juhyun Lee, Andrei Kulik, Chuo-Ling Chang, and Matthias Grundmann. The company focused on developing an efficient latent diffusion model specifically designed for mobile devices, aiming to enable rapid text-to-image generation on mobile devices with a small model size of 520M parameters.

How to use MobileDiffusion?

To use MobileDiffusion for sub-second text-to-image generation on mobile devices, follow these steps:

1. Model Components:

   • MobileDiffusion consists of a text encoder using CLIP-ViT/L14, a diffusion UNet, and an image decoder.

2. Diffusion UNet:

   • The diffusion UNet combines transformer blocks and convolution blocks, focusing on the efficient utilization of transformer blocks in the model.

3. One-Step Sampling:

   • MobileDiffusion incorporates DiffusionGAN for one-step sampling to generate high-quality images efficiently.

4. Training Procedure:

   • The training involves initializing the generator and discriminator with a pre-trained diffusion UNet. This approach streamlines training by leveraging the existing model's internal features.

5. Image Generation:

   • Images generated by MobileDiffusion exhibit high quality and diversity, showcasing its capability for various domains.

6. Performance Evaluation:

   • MobileDiffusion runs efficiently on iOS and Android devices, producing a 512x512 image within half a second, enabling rapid text-to-image generation on mobile platforms.

7. Results:

   • Example images generated by MobileDiffusion with DiffusionGAN one-step sampling demonstrate the model's effectiveness and potential for on-device image generation.

By following these steps, users can leverage MobileDiffusion to efficiently generate high-quality images from text prompts on mobile devices.