Hands-On Generative AI with Transformers and Diffusion Models

Chapter 8. Creative Applications of Text-To-Image Models

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the eighth chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

This chapter will present creative applications that leverage text-to-image models and increase their capabilities beyond just using text as control. We will start with the most basic ones and then move on to more advanced ones.

Image-to-Image

Even though generative text-to-image diffusion models like Stable Diffusion can produce images from text from a fully noised image, as we learned in Chapters 4 and 5, it is possible to start from an already existing image instead of a fully noised image. That is, add some noise to an initial image and have the model modify it partially by denoising it. This process is called image-to-image, as an image is transformed into another image based on how much it is noised and based on the text prompt.

With the diffusers library, we can load a image-to-image pipeline to load the class. For example, let’s explore how to use SDXL for this ...