If anyone is stuck with the code. The "i" should be a "t" in this line in the loop: ``` latents = scheduler.step(noise_pred, i, latents)["prev_sample"] ```
I've been using Stable Diffusion to _deCGI_ images. Take a screenshot from a game, run it through SD with a low noise rate, give it a detailed description of everything in the picture and it produces pretty solid photo recreations of the images. Also, often, it gets possessed by Eldritch gods and spews out monsters.
@@MattRose30000 This is a long way off. It isn't just that it currently takes my 3090 Ti about 5 minutes to do one frame at 1024x1024 but also it can't be playing a game at the same time and also-also it would be very disorienting because each frame will be a _different_ photo that isn't consistent from frame to frame but probably the worst part is that _you need to write a text prompt that reflects what is in the scene for each frame somehow._
@@DampeS8N that’s great. Have you messed around with reusing seeds across different frames? I imagine if you get an output you like you’d want to reuse that seed
The very concept of embeddings is amazing to me. It's literally "organize concepts themselves into points in space, where similar things are closer together, in many many dimensions; now you can do arithmetic on *the meanings of words, phrases, and sentences.* " Want to add the meaning of "horse" and the meaning of "male"? Well, just add these vectors together and the resulting coordinates will point right at "stallion"! They amaze me so much that, when I watched Everything, Everywhere, All At Once for the first time, I completely geeked out when I realized their description of the organization of the multiverse is effectively a well-embedded latent space 😅
I've been playing with Stable Diffusion (specifically the "InvokeAI" fork because I don't have 10gb VRAM), and I've found out that spamming the end with keywords like "realistic, 4k, trending on artstation, 8k, photorealistic, hyperrealistic" has more effect on how good the output image is than I thought.
I really liked the stable diffusion that came with the webui that you could install on your own computer, to avoid quotas or subscription costs, and it provided easy to use UI as well. With inpaint feature inside the UI as well. Shoutouts to people who make those applications from the rough code for regular people to use.
Mike asked himself what the use case for mixing two prompts is. I used this only yesterday, to produce a photorealistic painting of an owlbear from DnD... So it has practical uses!
I love it how you simplify and explain this heap of complexity that is in generative models like this. You gave me the impulse to play around with it, inspite of being pretty complicated code due to the depth of the abstraction. It's a lot of fun to fantasize about something and have the model come up with a visual representation.
"there are questions of ethics, there are questions on how it's trained. Let's leave those for another time" well, if that doesn't just sum up the tech industry.
@@paultapping9510 you weren't trying to make any point, otherwise you would have clarified. You were just trying to sound smart. Also, liking your own comments is pathetic.
Not necessarily what you're after, but if you "interrupt" a run, you can see what it's current progress was. Depending on your steps and how early you catch it, I've seen some very interesting early "noisy" images that were themselves inspiration for other images!
The current version of the reference notebook is already deprecated due to Hugging Face's API changes :) You try to operate on "image", which is now a DecoderOutput class: image = (image/ 2 + 0.5).clamp(0, 1) It is fixed by unpacking its tensor attribute with its sample method: image = (image.sample / 2 + 0.5).clamp(0, 1)
Same goes for pil_to_latent(): AutoencoderKL.encode() returns a AutoencoderKLOutput class: return 0.18215 * latent.mode() The desired DiagonalGaussianDistribution class is now a property ("latent_dist") of this new class: return 0.18215 * latent.latent_dist.mode()
in img2img, I just extract the code of add_noise and used int instead of floatTesnsor. Change add_noise function to the following. also notice the for loop now loop 51 times. Not sure if this is correct, but at least it works. # View a noised version noise = torch.randn_like(encoded) # Random noise for i in tqdm(range(51)): scheduler.sigmas = scheduler.sigmas.to(device=encoded.device, dtype=encoded.dtype) scheduler.timesteps = scheduler.timesteps.to(encoded.device) sigma = scheduler.sigmas[i].flatten() while len(sigma.shape) < len(encoded.shape): sigma = sigma.unsqueeze(-1) noisy_samples = encoded + noise * sigma img = latents_to_pil(noisy_samples)[0]
Love Mikes explanations, somehow he manages explain so complicated stuff in so simple and understandable way. It will be interesting to know Mikes opinion om Midjourney as it's seems like the winner for now among the picture creation AIs.
Anyone else surprised that diffusion models are the clear winners for image generation? And GANs have almost completely fallen from favor? I haven’t seen them in any recent SOTA work..
Mmm isnt it still kinda a GAN? Stable diffusion uses a transformer block not just for the diffusion but for identifying what the actual image is from the diffusion output too. So isn't that technically a GAN? Generate images from the diffusion model, then try to categorize them through an adversarial transformer network?
@@timmyt1293 Actually there is no adversarial training in diffusion models in general (in particular for stable diffusion model). The condition processing is used only for guidance (free classifier guidance in this case) and from a theoretical perspective the diffusions models are closer to hierarchical variational autoencoders where the encoders are fixed diffusion steps and decoders are denoising steps with the trained noise estimation model.
@@erikp7378 I wonder if you could impliment stable diffusion inside a GAN. So have the generator define the parameters for the stable diffusion based on an input and then give that to the classifier mixed in with non ai generated images
@@JadeNeoma its depends on which parameters you have in mind but the main point is that the operations must remain differentiable in order to optimize the model. And in the case of hyper parameters inference it is not trivial in many cases (e.g. the number of steps)
I doubt DALL-E 2 is the “biggest” image generator. Stable Diffusion is probably bigger. In my circle, the biggest one is NovelAI, which is a Stable Diffusion variant specialized in anime-style images. Notably, its training data is probably the best image dataset out there in terms of detailed labels. It’s already been causing a lot of drama in the community. One notable case involved someone feeding a WIP drawing to img2img, posting it, claiming it as their own drawing. When the actual artist posts their finished image, this person then proceeds to accuse the artist of copying “their” art.
Thanks for the explanations of how AIs are being trained. I can see a slight hint of a neural network here. I think the advantage now is that companies like Bluewillow is utilizing discord to quickly gain testers free of charge even.
Great video! Can anyone recommend any other videos that explain the text encoding and the whole clipping process used to guide the image generation based on input prompt?
The notebook can still work with a few minor tweaks: The text prompt should be multiplied by the batch size; The scheduler step takes in "t" instead of "i", and now it prefers scaling via scheduler.scale_model_input(latent_model_input, t) rather than with explicit sigma. Also, torch.autocast did not work on my local machine for some reason. Anyway, thanks a lot for the code.
For anyone trying to get the notebook to work and is getting this error: "TypeError: unsupported operand type(s) for /: 'DecoderOutput' and 'int'" change "image = (image / 2 + 0.5).clamp(0, 1)" to "image = (image.sample / 2 + 0.5).clamp(0, 1)". As noted at the top of the notebook it seems the huggin API has changed.
I'll copy your transcript and feed it to open.ai's playground and ask him to re-interpret your addresss for images but for my own audio interpolation in music. Brilliant.
[notebook error] Hello, Thanks for the fantastic video. I noticed that as of today the notebook does not run since there are some errors. I do not why, probably some library changed a bit.The first error is at line 50 of the cell with the first inference loop. Instead of 'i' there should be 't'. The second error appears at line 59. Now to access the image's tensor you have to write 'image["sample"]' instead of just 'image'.
Thank you for the SCIENTIFIC video! It got outta control after the "novelaileak", which it is very important to leave some information as realistic as it can. I'm quite sad about the sub-culture but I still have hope on the artist / researcher to snap out from the chaos.
This is literally the first episode of Computerphile ever that I didn't understand anything of what was explained. And judging from the comments I'm the only one. Looks like I totally missed the boat on this topic.
@@zwe1l1nkehaende thanks. I already found it. But I still don't really get it 😊 Doing some "best fit" on noise until a photorealistic image comes out still sounds like magic to me.
I didn't realise that this is basically the next evolution of the "AI Upscaling" technology that has been used to in videogame mods: Take an image and then add detail until it looks like what I think it's supposed to. It's still mind-bending how it results in what it does, but AI Upscaling wasn't so scary, so I suppose this feels a bit less scary now.
Hi Mike. This is the by far the most technically clear explanation of SD that I have seen so thank you for this! Now as you would be aware by now, the art community is up in arms against this tech and I would love to hear your opinion based on the factual knowledge you have. The main issue that keeps coming up is that SD tech is art theft because it steals copyrighted artwork then companies profit using the images. Another point artists are making is that SD is just a mish mash collage of original art so nothing generated by Ai is brand new. Would you agree or disagree with these points and why strictly based on from your technical knowledge.
What surprises me is how primitive a lot of these techniques seem to be under the hood, and how much further it can obviously be taken. These techniques are still in their infancy. For instance, there seem to be a lot of potential image-generating procedures that might converge faster than random high frequency noise. What if there could be stages with simulated random brush strokes, or generating geometric shapes, or input to 3d modelling software. If the tools that humans use to create digital art could be algorithmically leveraged by an AI, if might be even more effective. Also, if you could spatially embed the tags in the source image in a way it could be coupled to the segmentation, maybe it could be used as a tool to 'compose' an image. A blob of one color is tagged as a dog, a blob of another is tagged as a bench, and the AI interprets it with those spatially defined weights to start.
@@purplewine7362 Or you've worked in the tech field long enough to know how dangerous this is, and how it will be used against people eventually. As happens with all tech.
On line 56, the image is coming from the sample property of the DecoderOutput, change to 55: with torch.no_grad(): 56: image = vae.decode(latents).sample
> There are questions about ethics. There are questions about how these were trained. Maybe we deal with them another time. I really hope there is a discussion of this at some point. As a discipline that skews very white/male and enjoys relatively posh working conditions, it's very easy to insulate ourselves from the very real problems of the world. And because computers are so powerful it's also simple to automate oppression of many kinds, helping it continue to run smoothly. I think we have a responsibility to talk about these issues and I would love to see this channel model that in a constructive way.
Yep, negative prompts are great for things like getting hands right. It turns out Stable Diffusion, at least the 1.4 model everyone's been using so far, has trouble identifying where a hand or finger is supposed to stop, so you often get hands with too many fingers or fingers coming out of fingers as it keeps trying to "complete" a partial finger. Including a negative prompt for "hands" or "too many fingers" tends to produce much better results.
Only a matter of time until someone adapts this to 3d models. I mean, there are millions of 3d models on the internet in form of assets for all kind of engines and frameworks, all with a description to them, too.
Great video, really informative. I was hoping to try out your Google Colab code, although it seems broken at the moment. Are there any updates regarding this announcement regarding the known bugs? "Note: There might be a handful of bugs at the moment. The developers of this stable diffusion implementation keep changing the api. Everyone should know not to make breaking api changes so regularly! I'll do a pass over the code and fix bugs as soon as I can. Am away this week :) thanks to Michael d for bringing this to my attention."
Photoreal rarely works for me because the AI weirdiness is so obvious to the eye. I have really enjoyed creating images with various art styles though, it is extremely good at that. made some really competent artworks that (for me) are indistinguishable from a talented artist.
@14:47 - idea: hand draw your animation sequence.Give the first to image and text to AI and get the result. Then hand the resulting image, your next hand drawn frame and the text to generate the 2nd frame. Continue the process so that each new frame is a combination of the last and what you want it to look like combined. In this way the "flicker" might be reduced. But I haven't seen what you're talking about. I may be off.
Ddim, euler, lms, heun and dpm all produce identical results. The ones with "a" at the end (euler a, dpm2 a) are ancestral samplers and produce different results
@@havz0r I ment how they work under the hood. They've already explained how the network generates images from noise, but not how the different samplers work
The future of animation and film making is insane. Combine this with deep fake tech and in a few years you'll be able to produce whatever film you want to see. I think alot of animation studio jobs will be at risk.....
Where is the love for Disco Diffusion? I'd argue to be the most power of the bunch in terms of what you can modify within it, especially as programmer and artist.
Stable Diffusion is a tool that will greatly accelerate productivity in the creative sector. This is like the invention of the washing machine in terms of how much time this will save artists in producing high-quality work. You can focus more on the result, iterating on layouts and themes at the start, selecting from portions of generated images, adding your own marks and refinements, to produce an image of equal quality to your best work in a fraction of the time. I know several artists who have been quite pleased with the result - one produces serial fiction with illustrations, and Stable Diffusion has allowed them to include many more illustrations in the same span of time, which also leaves them more time for writing.
This won’t improve creativity. It will only atrophy people’s actual human creativity by supplanting it with artificial creativity. It’s the same pattern that has undermined many other aspects of humanity in recent years.
I love that the technology has develop to the point where even with little to none artistic skill, you can still generate some cool artworks, the only limit now is just your imagination, and maybe the picture resolution too. But still I just feel bad that there will always be people who will find ways to exploit the technology and ruin the fun form other people
In my opinion, the skill just moved to a different art. You still need some skill to craft the right prompt for the AI to generate the result you want.
@@adiaphoros6842 But then it can not be compared to the traditional art at the same category. Also the people who use AI to make art even go out of their way to say they are the artists. Like no, they didn't contribute any artistic skill to make this art, all they have was an idea, and everyone else also has the millions dollars idea too, the important part of an artist is the execution, is the making of the art. Maybe they have a clever way of setting the prompt for the AI to make the art but that is no where near being an artist. AI art itself is absolutely another form of art alright, but the person who only enters the prompt to make it isn't artist
@@BaoNguyen6742 Prompt crafting is a subcategory of writing, which is a kind of art. There is still artistry involved, it's just the writing is not the end goal.
Cool, very clear... but if you run in the notebook in 2024, you need to use the specific diffuser version 0.2.4, !pip install transformers diffusers==0.2.4 lpips accelerate
The code might have a bug, "TypeError: unsupported operand type(s) for /: 'DecoderOutput' and 'int'" on the line "image = (image / 2 + 0.5).clamp(0, 1)"
If you can make images by removing noise from random noise. Can you make P solutions from NP solutions the same way by training on known P solutions having "noise" added to make them NP?
I don't understand at all how the result of this reconstruction process (remove noise) is stored. Sounds a bit like witchcraft to me. Remove some noise, here we go. I mean in which form is the noise reduction saved? In a database? Does it save pixels or what exactly?
so on the quality difference, dalle2 is 1024x but for some reason pretty heavily jpeg compressed, stable diffusion is 512x but (at least on replicate) much much less jpeg compressed, if at all (sometimes i’ve gotten stuff that looked compressed but it might’ve been from being trained on compressed images, not sure). so while it’s a lower resolution, i’ve found that it’s a higher quality image, but i’m sure there there are hosted versions that are much lower quality. also i’m not sure what differs between them for inpainting but i’ve found that for stable diffusion i can’t just add a mask, i have to inpaint stuff myself and get it somewhat close, otherwise i get variations on that part i was trying to get something else at
oh and dalle2 is way way pricier than stable diffusion on replicate so i don’t know why they’re compressing the images so much, surely they should be able to afford storage for the images at the cost they charge
@@deathstroyer oh yeah the autoencoder vs directly diffusing the image. would be cool to see someone fork stable diffusion and add on a non-autoencoded diffusion final step to make the output higher quality
Mind giving a quick review of Bluewillow and which software does it utiized? I think you guys break down the whole infrastructure which is actually very informative.
Somebody asked that in a Discord AMA a couple of days ago. They're not telling. But it's very likely Stable Diffusion, using a finetuned custom model, or several. So it should be the same infrastructure
If anyone is stuck with the code. The "i" should be a "t" in this line in the loop:
```
latents = scheduler.step(noise_pred, i, latents)["prev_sample"]
```
Did you get the code working?. for me it's showing "unsupported operand type(s) for /: 'DecoderOutput' and 'int'" in line 59
@@alenmathew8115 in the last few lines, change this line
image = (image / 2 + 0.5).clamp(0, 1) to this image = (image.sample / 2 + 0.5).clamp(0, 1)
man this helped me. thanks bro :)
Thanks man because of you I solved this error
Also in the Image Loop section, this needs to be moved inside the for loop :
```
# Prep Scheduler
scheduler.set_timesteps(num_inference_steps)
```
I've been using Stable Diffusion to _deCGI_ images. Take a screenshot from a game, run it through SD with a low noise rate, give it a detailed description of everything in the picture and it produces pretty solid photo recreations of the images. Also, often, it gets possessed by Eldritch gods and spews out monsters.
So win-win, right?
now do it in real time with DLSS and you've got something huge
@@MattRose30000 This is a long way off. It isn't just that it currently takes my 3090 Ti about 5 minutes to do one frame at 1024x1024 but also it can't be playing a game at the same time and also-also it would be very disorienting because each frame will be a _different_ photo that isn't consistent from frame to frame but probably the worst part is that _you need to write a text prompt that reflects what is in the scene for each frame somehow._
@@DampeS8N that’s great. Have you messed around with reusing seeds across different frames? I imagine if you get an output you like you’d want to reuse that seed
@@DampeS8N making text to video is the easy part, making video to text is the hard part.
The very concept of embeddings is amazing to me. It's literally "organize concepts themselves into points in space, where similar things are closer together, in many many dimensions; now you can do arithmetic on *the meanings of words, phrases, and sentences.* " Want to add the meaning of "horse" and the meaning of "male"? Well, just add these vectors together and the resulting coordinates will point right at "stallion"!
They amaze me so much that, when I watched Everything, Everywhere, All At Once for the first time, I completely geeked out when I realized their description of the organization of the multiverse is effectively a well-embedded latent space 😅
@@mrteco4236 It literally is and is done all the time.
@@mrteco4236 It's... common, in fact. There's a whole video on this channel about embeddings. And it's how CLIP fundamentally works...
This is super fascinating, especially as someone studying Data Science just learning about vector spaces and their many uses!
@@mrteco4236 lol
@@mrteco4236 that is literally what it does bro
I've been playing with Stable Diffusion (specifically the "InvokeAI" fork because I don't have 10gb VRAM), and I've found out that spamming the end with keywords like "realistic, 4k, trending on artstation, 8k, photorealistic, hyperrealistic" has more effect on how good the output image is than I thought.
You should try negative prompts.
to add, try emphasis "((x))" for specific objects.
Edit: you can also use x(y), y being the weight value for that tag.
I really liked the stable diffusion that came with the webui that you could install on your own computer, to avoid quotas or subscription costs, and it provided easy to use UI as well. With inpaint feature inside the UI as well. Shoutouts to people who make those applications from the rough code for regular people to use.
Mike asked himself what the use case for mixing two prompts is.
I used this only yesterday, to produce a photorealistic painting of an owlbear from DnD...
So it has practical uses!
Maybe google is planning to create new, even more impossible captchas. "Select all the cat-dogs in the picture"
Does it hoot or roar??
@@dembro27 It hoots and growls, in fact, here at Aguefort's Adventuring Academy!
Its how I make my fish people too for tabletop. Tons of applications for DnD
@@euchale You get half-decent tieflings if you ask for a quarter human, a half lizard and the last quarter goat.
I love it how you simplify and explain this heap of complexity that is in generative models like this. You gave me the impulse to play around with it, inspite of being pretty complicated code due to the depth of the abstraction. It's a lot of fun to fantasize about something and have the model come up with a visual representation.
SD is just outstanding. It can mimic the other projects and the 1.4/1.5 models will be public domain. You can't beat that.
Lol just add "dall-e 2" to your prompts XD
1.5 model just went public today i think
@@paryska991 Ye
You can beat that with human creativity that doesn't require billions of calculations per second to brute force a synthetic result.
@@dgo4490 doesn't it though?
I like how your channel has adapted to the advent of the machine learning boom we are experiencing
"there are questions of ethics, there are questions on how it's trained. Let's leave those for another time"
well, if that doesn't just sum up the tech industry.
what ethics ? its just a tool, and its highly dependent on human input.
@Luiz remember the AI chatbot that became incurably racist because it was trained on data scraped from 4chan amongst other places? That sort of thing.
that sums up every industry. you think people didn't copy art before ai? it's just a tool
@@purplewine7362 lol, not even close to the point I was making. Never mind.
@@paultapping9510 you weren't trying to make any point, otherwise you would have clarified. You were just trying to sound smart.
Also, liking your own comments is pathetic.
I'm sorry but , "unlock your face with your phone" just cracked me up..
This is inadvertently an excellent poetic description of someone using the selfie camera to apply makeup.
Unlock your phace with your fone
I think he was referring to using the Energizer Power Max P18K whilst in bed... :)
Hahahaha didn't even notice!
I am reminded of an odd commercial from a few years ago: "apply directly to the forehead".
I would like to see a version of the code where it shows the result of each step, so you can see the noise getting reduced with each iteration
me too!!
I think I'm going to do it. I'm downloading the source code and save a png for each step
Not necessarily what you're after, but if you "interrupt" a run, you can see what it's current progress was. Depending on your steps and how early you catch it, I've seen some very interesting early "noisy" images that were themselves inspiration for other images!
There is already a script for that
If you run automatic1111 there’s a setting for that, uses slightly more vram, but it’s great to watch it work
The current version of the reference notebook is already deprecated due to Hugging Face's API changes :)
You try to operate on "image", which is now a DecoderOutput class:
image = (image/ 2 + 0.5).clamp(0, 1)
It is fixed by unpacking its tensor attribute with its sample method:
image = (image.sample / 2 + 0.5).clamp(0, 1)
The rest of the notebook is hard to fix, I tried but in vain. I think I'll wait for Mike's update.
Same goes for pil_to_latent():
AutoencoderKL.encode() returns a AutoencoderKLOutput class:
return 0.18215 * latent.mode()
The desired DiagonalGaussianDistribution class is now a property ("latent_dist") of this new class:
return 0.18215 * latent.latent_dist.mode()
in img2img,
I just extract the code of add_noise and used int instead of floatTesnsor.
Change add_noise function to the following.
also notice the for loop now loop 51 times.
Not sure if this is correct, but at least it works.
# View a noised version
noise = torch.randn_like(encoded) # Random noise
for i in tqdm(range(51)):
scheduler.sigmas = scheduler.sigmas.to(device=encoded.device, dtype=encoded.dtype)
scheduler.timesteps = scheduler.timesteps.to(encoded.device)
sigma = scheduler.sigmas[i].flatten()
while len(sigma.shape) < len(encoded.shape):
sigma = sigma.unsqueeze(-1)
noisy_samples = encoded + noise * sigma
img = latents_to_pil(noisy_samples)[0]
@@victorwesterlund4826 What is the 0.18215 for? I keep seeing it in the code but I can't find an explanation for what is does or how it's derived
"Simple, you just chip away all the stone that doesn't look like David."
"I saw the angel in the marble and carved until I set him free" - Michalangelo
Love Mikes explanations, somehow he manages explain so complicated stuff in so simple and understandable way.
It will be interesting to know Mikes opinion om Midjourney as it's seems like the winner for now among the picture creation AIs.
Immediately recognized the book on Dr. Ponds desk - Prof. Paar was one of my teachers when I studied IT sec. Nice to see it outside of Germany too!
this is so interesting and has so many unexplored use cases
Anyone else surprised that diffusion models are the clear winners for image generation? And GANs have almost completely fallen from favor? I haven’t seen them in any recent SOTA work..
Mmm isnt it still kinda a GAN? Stable diffusion uses a transformer block not just for the diffusion but for identifying what the actual image is from the diffusion output too. So isn't that technically a GAN? Generate images from the diffusion model, then try to categorize them through an adversarial transformer network?
@@timmyt1293 Actually there is no adversarial training in diffusion models in general (in particular for stable diffusion model). The condition processing is used only for guidance (free classifier guidance in this case) and from a theoretical perspective the diffusions models are closer to hierarchical variational autoencoders where the encoders are fixed diffusion steps and decoders are denoising steps with the trained noise estimation model.
@@erikp7378 I wonder if you could impliment stable diffusion inside a GAN. So have the generator define the parameters for the stable diffusion based on an input and then give that to the classifier mixed in with non ai generated images
@@JadeNeoma I don't know how that would work.
@@JadeNeoma its depends on which parameters you have in mind but the main point is that the operations must remain differentiable in order to optimize the model. And in the case of hyper parameters inference it is not trivial in many cases (e.g. the number of steps)
I doubt DALL-E 2 is the “biggest” image generator. Stable Diffusion is probably bigger. In my circle, the biggest one is NovelAI, which is a Stable Diffusion variant specialized in anime-style images. Notably, its training data is probably the best image dataset out there in terms of detailed labels.
It’s already been causing a lot of drama in the community. One notable case involved someone feeding a WIP drawing to img2img, posting it, claiming it as their own drawing. When the actual artist posts their finished image, this person then proceeds to accuse the artist of copying “their” art.
Imagen by Google and NUWA-infinity by Microsoft are probably superior.
Would your "circle" happen to fit after rule 33 and before rule 35?
The danbooru property labeling format, to be exact. Training is rather easy as the images in the booru databases are human-labeled.
This video finally explained the code to me in a simple way! Now im less confused!!! Amazing extra documentation from you guys
Thanks for the explanations of how AIs are being trained. I can see a slight hint of a neural network here. I think the advantage now is that companies like Bluewillow is utilizing discord to quickly gain testers free of charge even.
I generated thousands of images with stable diffusion. It's really fun and inpiring.
Great video! Can anyone recommend any other videos that explain the text encoding and the whole clipping process used to guide the image generation based on input prompt?
The notebook can still work with a few minor tweaks: The text prompt should be multiplied by the batch size; The scheduler step takes in "t" instead of "i", and now it prefers scaling via scheduler.scale_model_input(latent_model_input, t) rather than with explicit sigma. Also, torch.autocast did not work on my local machine for some reason.
Anyway, thanks a lot for the code.
So amazing ❤ I love stable diffusion
Playing around the few last weeks
For anyone trying to get the notebook to work and is getting this error: "TypeError: unsupported operand type(s) for /: 'DecoderOutput' and 'int'" change "image = (image / 2 + 0.5).clamp(0, 1)" to "image = (image.sample / 2 + 0.5).clamp(0, 1)". As noted at the top of the notebook it seems the huggin API has changed.
wow thank you very much
can confirm that this indeed solves it👍
In my case it outputs a Hugging Face Tokens page warning? It says that I need a token? Is it free?
@@koh8614 yes it is free. you need to create an account on the hugging face website and generate a token from your profile.
Thank you
Mike is a legend, truly great videos with him
Stable Diffusion in code? More like “Super great explanation that’s solid gold!” 👍
I'll copy your transcript and feed it to open.ai's playground and ask him to re-interpret your addresss for images but for my own audio interpolation in music. Brilliant.
[notebook error] Hello, Thanks for the fantastic video. I noticed that as of today the notebook does not run since there are some errors. I do not why, probably some library changed a bit.The first error is at line 50 of the cell with the first inference loop. Instead of 'i' there should be 't'. The second error appears at line 59. Now to access the image's tensor you have to write 'image["sample"]' instead of just 'image'.
same thing for the other inference loops
Thanks! this should be pinned
We need an entire "Frogs on stilts" channel.
Thank you for the SCIENTIFIC video!
It got outta control after the "novelaileak", which it is very important to leave some information as realistic as it can.
I'm quite sad about the sub-culture but I still have hope on the artist / researcher to snap out from the chaos.
what sub-culture?
This is literally the first episode of Computerphile ever that I didn't understand anything of what was explained. And judging from the comments I'm the only one. Looks like I totally missed the boat on this topic.
what was confusing?
@@dibbidydoo4318 it wasn't actually confusing because there wasn't anything to confuse. I had literally never heard of these developments before.
@@nkronert this is the followup video on the topic, check out the first one, where the whole thing is explained.
@@zwe1l1nkehaende thanks. I already found it. But I still don't really get it 😊
Doing some "best fit" on noise until a photorealistic image comes out still sounds like magic to me.
Thank you for trying to fix the code after the API update broke it
I didn't realise that this is basically the next evolution of the "AI Upscaling" technology that has been used to in videogame mods: Take an image and then add detail until it looks like what I think it's supposed to. It's still mind-bending how it results in what it does, but AI Upscaling wasn't so scary, so I suppose this feels a bit less scary now.
Hi Mike. This is the by far the most technically clear explanation of SD that I have seen so thank you for this! Now as you would be aware by now, the art community is up in arms against this tech and I would love to hear your opinion based on the factual knowledge you have.
The main issue that keeps coming up is that SD tech is art theft because it steals copyrighted artwork then companies profit using the images. Another point artists are making is that SD is just a mish mash collage of original art so nothing generated by Ai is brand new.
Would you agree or disagree with these points and why strictly based on from your technical knowledge.
great video. today SORA was launched, nad youvideos help to understand whats going on the background. many thanks!
What surprises me is how primitive a lot of these techniques seem to be under the hood, and how much further it can obviously be taken. These techniques are still in their infancy.
For instance, there seem to be a lot of potential image-generating procedures that might converge faster than random high frequency noise. What if there could be stages with simulated random brush strokes, or generating geometric shapes, or input to 3d modelling software. If the tools that humans use to create digital art could be algorithmically leveraged by an AI, if might be even more effective.
Also, if you could spatially embed the tags in the source image in a way it could be coupled to the segmentation, maybe it could be used as a tool to 'compose' an image. A blob of one color is tagged as a dog, a blob of another is tagged as a bench, and the AI interprets it with those spatially defined weights to start.
I don't know if this is more amazing or more frightening. Brilliant stuff.
If you aren’t frightened, you aren’t paying attention.
@@andybaldman if you're frightened, you're a luddite
@@purplewine7362 Or you've worked in the tech field long enough to know how dangerous this is, and how it will be used against people eventually. As happens with all tech.
Mikes explanations Aretha best ❤
Franklin true
@@JavierSalcedoC *slow clap*
Seeing that GPT-2 vid reminded me: we haven't had Robert Miles on in a fair while. Is he just too busy?
I love his content.
On line 56, the image is coming from the sample property of the DecoderOutput, change to
55: with torch.no_grad():
56: image = vae.decode(latents).sample
Amazing so stable diffusion helps un clutter all that extra pixel during the process of facial recognition.
Thanks for this video.
So the Steps is actually the Noise Level.
> There are questions about ethics. There are questions about how these were trained. Maybe we deal with them another time.
I really hope there is a discussion of this at some point. As a discipline that skews very white/male and enjoys relatively posh working conditions, it's very easy to insulate ourselves from the very real problems of the world. And because computers are so powerful it's also simple to automate oppression of many kinds, helping it continue to run smoothly. I think we have a responsibility to talk about these issues and I would love to see this channel model that in a constructive way.
Well, xkcd did pick the number 4 by die roll. Seems a random enough seed to me.
I had to scroll far too much to see this mentioned, but yes I agree 4 seemed quite a good random seed there...
Good timing with the NovelAI leaks
Another cool thing you can do is _negative prompts,_ that you can put in place of the "unconditioned" embedding.
Yep, negative prompts are great for things like getting hands right. It turns out Stable Diffusion, at least the 1.4 model everyone's been using so far, has trouble identifying where a hand or finger is supposed to stop, so you often get hands with too many fingers or fingers coming out of fingers as it keeps trying to "complete" a partial finger. Including a negative prompt for "hands" or "too many fingers" tends to produce much better results.
@@Onihikage Yes, that is precisely what I use it for too. I expect we got that advice from the same place.
A hybrid frog/snake is properly called a *SNOG*, obviously.
Frake news
great video and very educational
I'd love to hear you guys talk about textual inversion
Awesome explanation, thank you!
Such a fun and interesting tool. Wish it wasn't used to do bad things, like stealing people's artworks
Wow this is actually pretty amazing. Fascinating stuff
Only a matter of time until someone adapts this to 3d models. I mean, there are millions of 3d models on the internet in form of assets for all kind of engines and frameworks, all with a description to them, too.
Great video, really informative. I was hoping to try out your Google Colab code, although it seems broken at the moment. Are there any updates regarding this announcement regarding the known bugs? "Note: There might be a handful of bugs at the moment. The developers of this stable diffusion implementation keep changing the api. Everyone should know not to make breaking api changes so regularly! I'll do a pass over the code and fix bugs as soon as I can. Am away this week :) thanks to Michael d for bringing this to my attention."
13:47 reminds me of the wave function collapse algorithm.
On line 50, i should be changed to t (as we need the FloatTensor) 50: latents = scheduler.step(noise_pred, t, latents)["prev_sample"]
Photoreal rarely works for me because the AI weirdiness is so obvious to the eye. I have really enjoyed creating images with various art styles though, it is extremely good at that. made some really competent artworks that (for me) are indistinguishable from a talented artist.
"the AI weirdiness is so obvious to the eye"
You mean those weird artifacts in the AI caused by Perlin's Noise?
12:34 beautiful cityscapes 🏙️
@14:47 - idea: hand draw your animation sequence.Give the first to image and text to AI and get the result. Then hand the resulting image, your next hand drawn frame and the text to generate the 2nd frame. Continue the process so that each new frame is a combination of the last and what you want it to look like combined. In this way the "flicker" might be reduced.
But I haven't seen what you're talking about. I may be off.
Great explanation.
This was so helpful in understanding this new tech. thank you
Now Deep Dream Generator has just added a text to image diffusion generator too, and it's actually pretty decent.
Could you do a video about the different samplers? (eg. DDIM, Euler, Euler a, etc.) That part of the process is still a mystery for me
Ddim, euler, lms, heun and dpm all produce identical results. The ones with "a" at the end (euler a, dpm2 a) are ancestral samplers and produce different results
@@havz0r I ment how they work under the hood. They've already explained how the network generates images from noise, but not how the different samplers work
this video just put me on a wonderful path, thank you!
Great video. I would love to see a video about the recent controversy with GitHub copilot and GPL licenses.
So it is basically a morphing, blending and upscaling algorhythm of compressed/encoded data?
The future of animation and film making is insane. Combine this with deep fake tech and in a few years you'll be able to produce whatever film you want to see. I think alot of animation studio jobs will be at risk.....
no, i don't think these jobs will be at risk for a few decades at least.
7:20 My man Mike knows that when you use a proper random function, the result would be 4. Guaranteed to be random!
I would do just about anything for more Mike content!
this whole topic blows my mind even more than when i first heard of deepfakes
For those who try the code and get an error with the image putout of the decoder, just add [0], like this
image = vae.decode(latents)[0]
Excellent explanations, as always! Thanks!
Mike said link to code in description!
Now sorted!
Where is the love for Disco Diffusion? I'd argue to be the most power of the bunch in terms of what you can modify within it, especially as programmer and artist.
Cartoons and anime are going to be so amazing in 5 to 10 years
Anime-style drawings are already a thing and is causing a lot of drama.
@@theemathas well, at least you can have unique wallpapers and profile pictures.
Next, we need semi natural language software that can generate 3D vector models for use in Blender or Unreal Engine etc.
Would love to see a test to see how it works when it's trained with a limited dataset.
"Picking the nice ones" is doing a lot of heavy lifting here. When you focus more on the failures the truth about this tech becomes more salient.
nothing is stopping people from testing out the technology itself, SD is open source so there's no truth being hidden.
Stable Diffusion is a tool that will greatly accelerate productivity in the creative sector. This is like the invention of the washing machine in terms of how much time this will save artists in producing high-quality work. You can focus more on the result, iterating on layouts and themes at the start, selecting from portions of generated images, adding your own marks and refinements, to produce an image of equal quality to your best work in a fraction of the time. I know several artists who have been quite pleased with the result - one produces serial fiction with illustrations, and Stable Diffusion has allowed them to include many more illustrations in the same span of time, which also leaves them more time for writing.
This won’t improve creativity. It will only atrophy people’s actual human creativity by supplanting it with artificial creativity. It’s the same pattern that has undermined many other aspects of humanity in recent years.
I love that the technology has develop to the point where even with little to none artistic skill, you can still generate some cool artworks, the only limit now is just your imagination, and maybe the picture resolution too. But still I just feel bad that there will always be people who will find ways to exploit the technology and ruin the fun form other people
In my opinion, the skill just moved to a different art. You still need some skill to craft the right prompt for the AI to generate the result you want.
@@adiaphoros6842 But then it can not be compared to the traditional art at the same category. Also the people who use AI to make art even go out of their way to say they are the artists. Like no, they didn't contribute any artistic skill to make this art, all they have was an idea, and everyone else also has the millions dollars idea too, the important part of an artist is the execution, is the making of the art. Maybe they have a clever way of setting the prompt for the AI to make the art but that is no where near being an artist. AI art itself is absolutely another form of art alright, but the person who only enters the prompt to make it isn't artist
@@BaoNguyen6742 Prompt crafting is a subcategory of writing, which is a kind of art. There is still artistry involved, it's just the writing is not the end goal.
If you mention another video please also link it in the description!
SD 1.5 is so much better than the previous version, i can make great pics in less than a minute locally (150 steps)
"What amount of frog DO you want in this image?"
I WANT ALL THE FROG.
Cool, very clear... but if you run in the notebook in 2024, you need to use the specific diffuser version 0.2.4, !pip install transformers diffusers==0.2.4 lpips accelerate
7:18 is clearly a reference to xkcd 221
The code might have a bug, "TypeError: unsupported operand type(s) for /: 'DecoderOutput' and 'int'" on the line "image = (image / 2 + 0.5).clamp(0, 1)"
Same case here :(
change a line before to image = vae.decode(latents).sample, the .sample fixes it but now trying to get it to display
@@pb-vj1qs It worked now, thanks! The image is displayed here...
I finally have something in common with Mike - we both drink Costa
Excellent tutorial. Thank you.
Thank you for this video, it's really interesting!
3:07 earned my like. I need to go see that now. 😂
"They are essentially the same, but quite different."
Ah yes, the ol' computer science maxim of "same, but different"
Great video. However, could you explain what this line "latent_model_input = latent_model_input / ((sigma**2 + 1) ** 0.5)" does?
Fascinating.
If you can make images by removing noise from random noise. Can you make P solutions from NP solutions the same way by training on known P solutions having "noise" added to make them NP?
I was literally generating stable diffusion memes right now.
I don't understand at all how the result of this reconstruction process (remove noise) is stored. Sounds a bit like witchcraft to me. Remove some noise, here we go. I mean in which form is the noise reduction saved? In a database? Does it save pixels or what exactly?
I bet this code is responsible for the most lost hours of productivity since microsoft solitaire. I love it.
so on the quality difference, dalle2 is 1024x but for some reason pretty heavily jpeg compressed, stable diffusion is 512x but (at least on replicate) much much less jpeg compressed, if at all (sometimes i’ve gotten stuff that looked compressed but it might’ve been from being trained on compressed images, not sure). so while it’s a lower resolution, i’ve found that it’s a higher quality image, but i’m sure there there are hosted versions that are much lower quality. also i’m not sure what differs between them for inpainting but i’ve found that for stable diffusion i can’t just add a mask, i have to inpaint stuff myself and get it somewhat close, otherwise i get variations on that part i was trying to get something else at
oh and dalle2 is way way pricier than stable diffusion on replicate so i don’t know why they’re compressing the images so much, surely they should be able to afford storage for the images at the cost they charge
I would assume thats the imperfections resulting from the upsampling from 64x64 youre seeing
@@deathstroyer oh yeah the autoencoder vs directly diffusing the image. would be cool to see someone fork stable diffusion and add on a non-autoencoded diffusion final step to make the output higher quality
and it’s not a 64x image, it’s latent space
unstructures thoughts, "u kind of align the two until they sort of make sense"
Mind giving a quick review of Bluewillow and which software does it utiized? I think you guys break down the whole infrastructure which is actually very informative.
Somebody asked that in a Discord AMA a couple of days ago. They're not telling. But it's very likely Stable Diffusion, using a finetuned custom model, or several. So it should be the same infrastructure