Lesson 9: Deep Learning Foundations to Stable Diffusion
Вставка
- Опубліковано 8 чер 2024
- (All lesson resources are available at course.fast.ai.) This is the first lesson of part 2 of Practical Deep Learning for Coders. It starts with a tutorial on how to use pipelines in the Diffusers library to generate images. Diffusers is (in our opinion!) the best library available at the moment for image generation. It has many features and is very flexible. We explain how to use its many features, and discuss options for accessing the GPU resources needed to use the library.
We talk about some of the nifty tweaks available when using Stable Diffusion in Diffusers, and show how to use them: guidance scale (for varying the amount the prompt is used), negative prompts (for removing concepts from an image), image initialisation (for starting with an existing image), textual inversion (for adding your own concepts to generated images), Dreambooth (an alternative approach to textual inversion).
The second half of the lesson covers the key concepts involved in Stable Diffusion:
- CLIP embeddings
- The VAE (variational autoencoder)
- Predicting noise with the unet
- Removing noise with schedulers.
You can discuss this lesson, and access links to all notebooks and resources from it, at this forum topic: forums.fast.ai/t/lesson-9-par...
0:00 - Introduction
6:38 - This course vs DALL-E 2
10:38 - How to take full advantage of this course
12:14 - Cloud computing options
14:58 - Getting started (Github, notebooks to play with, resources)
20:48 - Diffusion notebook from Hugging Face
26:59 - How stable diffusion works
30:06 - Diffusion notebook (guidance scale, negative prompts, init image, textual inversion, Dreambooth)
45:00 - Stable diffusion explained
53:04 - Math notation correction
1:14:37 - Creating a neural network to predict noise in an image
1:27:46 - Working with images and compressing the data with autoencoders
1:40:12 - Explaining latents that will be input into the unet
1:43:54 - Adding text as one hot encoded input to the noise and drawing (aka guidance)
1:47:06 - How to represent numbers vs text embeddings in our model with CLIP encoders
1:53:13 - CLIP encoder loss function
2:00:55 - Caveat regarding "time steps"
2:07:04 Why don’t we do this all in one step?
Thanks to fmussari for the transcript, and to Raymond-Wu (on forums.fast.ai) for the timestamps.
Wow this is such a treasure to have freely available and I am so thankful that you put this out for the community. Many many thanks good sir, your work towards educating the masses about AI and Machine Learning is so very much appreciated. 🎉❤
This is beautifully explained Jeremy! From real basics to some of the most complicated state of the art models we have today. Bravo.
I am so glad you took your time to correct the math mistake! Great work! And thank you for your mission of teaching us new findings in AI and deep learning 🙏
I absolutely love the style in which this is explained. Thank you very much!
Glad you like it!
Thank you so much for this insightful video. The lecture breaks down complex ideas into segments that are very easy to comprehend.
I thought I was going to have to wait until next year, thank you for making this content accessible
This is a nicely thought-through course. Amazing Jeremy! :)
Wonderful, I was waiting for these series of videos. Bravo!
Undoubtedly an accessible and insightful guide
🙏🙏🙏
Amazing information.
I knew bits and pieces, now I know the entire picture.
Very informative video. Thank you for taking the time to produce.
Great! I can only see 2019 version of Part 2, look foward to see the new Part 2 course available!
Liberating the world with this quality of education
Outstanding, the best description so far. God Bless Jeremy. Excellent service to curious souls.
Excellent intiution. You're doing the huge service to humanity
Awesome material! Thank you very much for sharing
My pleasure!
Thanks for the early release
Thank you so much for your lectures.
You might want to put this series into a playlist. I see you have playlists for all your other courses.
Wow this helped me a lot. Thank you!
Thanks -- great lecture!
Amazing video and really easy to follow up with the topics. Its neat how different research is coming together to build something more efficient and promising. So future of AI is how models fit together?
YAY its hereeee. My excitement!!
Thank you for this lecture
Thanks, very clear.
Thanks a ton! have been waiting.
Hope you like it!
Ottimo video. Continua a pubblicare altre cose del genere.
THANKS 🙏🏻🙏🏻
I love you, its so clear as usual, i owed you embeddings, now i owe you diffusion too.
Please open a tipee
Thank you 🙏
Very an excellent course, I would like to ask a question that the noise N(0,0.1) is added equally to each pixel or to the whole image at 1:21:51? These two are equivalent？
Different per pixel! You’re drawing a vector of random noise samples then reshaping it into an image, so you get many values but all from a dist with low variance. The python random numbers let you sample in the shape of an image directly, so you don’t need to manually reshape. But that’s just for convenience
That math correction was very essential to me. Coming from a mechanical background, I knew something was off, but then thought I didn't know enough about DL to figure out what it is, and that I was on the wrong. With the math correction, it clicked, and was something I knew all along.Thanks.
Thank you for great explanation .. I just wanted to know ans to 'what is U-net?' could not understand where is it used in whole process ? I mean what I could not get is what is the difference between VAE (Autoencoder) and an Unet
During *training*, you pass an actual image into the VAE ENcoder (to reduce the amount of data you have to deal with), which then passes the latent it produces on to the UNet, which does the learning involving noising/denoising the latent. During *inference* ("generating"), the UNet (after a lot of other stuff happens :D) passes out a denoised latent to the VAE DEcoder, which then produces an actual image
Looks like the part 2 2022 webpage is still not public right? or I didn't find the way?
At 1:13:20 aren't we supposed to add derivatives to pixel values since we are maximizing P? Unless, since P is binary and it looks like a classification problem, we are going to get negative logits, then deducting seems ok (not touching the sign). Great course!
Can I ask a video of these model that is used for colonization?
Already out, wow. Then I'd better push on with part 1!
Awesome
at 1:56:50 I'm having hard time understanding the cost function. I think we need to maximise ( Green Summation - Red Summation ) , for that reason we can't call it a cost function because cost functions are usually minimised. Please correct me If I'm wrong
28:36 - I'm here in February '24, where they are good enough to do it in 1 go with SDXL-Turbo / ADD (Nov '23) :)
where can I find the latest recommended cloud computing resource?
I've been working on/with diffusion models (and before that VQGANs!) for years now, so I'm pretty familiar (from the end-user/theoretical POV, not so much the math/code side heh) with samplers/schedulers - this is the first time I've conceived of them as optimizers, and that seems like a *really* fertile area to research. Have you (or anyone else, for that matter) made any progress in this direction? It's (not too surprisingly) VERY hard to prompt today's search engines to find anything to do with "denoise|diffusion|schedule|sample|optimize" and NOT come up with dozens of either HuggingFace docs pages, or pages w.r.t. Stable DIffusion ROFL
since we want to increase the probability of our image being a digit, we should "add" and not "substract" the grad of probability wrt to img. Is this right? or am I missing something ?
gold
Released already??
Good thing is that with git we can go back to the state of the code as of (11/20).10.2022
👏👏👏
Jerremy, Adam has eps, SGD momentum
UHUUUL ITS OUT
There is a small (not that important) correction: when you talk about 16384 bytes of latents, it is 16384 numbers, which are 65536 bytes in fact.
Each number is 4 bytes because it's a float32 precision?
52:12 - The upside down triangle is "nabla", "del" is the squiggly d that goes before each partial derivative. Also, I'm jealous of people who started calculus in high school lol
Nevermind! Getting to the next section edited in lol
1:04:12 wait you still mixed them up 😅 At this rate with your following, you're going to speak it into existence though lol. Math notation ultimately is whatever everyone agrees upon, so I could see it being possible.
1:05:22 - While I'm being nit-picky - Right-side-up triangle is called "delta", and just means change, not necessarily small
I was wondering if I could give some suggestions, you spent 20 mins to explain the course materials and different people. Why not start from main fun play first, then later introduce more about the course materials. People will lose interest to listen to 20 mins course materials.
I thought I was going to have to wait until next year, thank you for making this content accessible