MULTIPLE CHARACTERS In ONE IMAGE WITH CONTROLNET & LATENT COUPLE! THIS IS SO FUN!
Вставка
- Опубліковано 25 сер 2024
- Recently a brand new extension for Stable Diffusion was released called Latent Couple, which allows you to determine specific zones of an image and attribute it a prompt, making it possible to generate multiple characters in different styles in 1 style generation, no Inpainting required at all! And when combining the extension to ControlNET and the Composable LoRA extension you can generate multiple different characters in different styles and in different positions in one single image generation! This is so cool and so powerful! So in this video, I will show you how to install and use the Couple Latent extension and how to use it in combination with ControlNet and LatentCoupleHelper to get the best results possible! So let's go!
Did you manage to generate multiple characters in 1 image? Let me know in the comments!
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
SOCIAL MEDIA LINKS!
✨ Support my work on Patreon: / aitrepreneur
⚔️ Join the Discord server: bit.ly/aitdiscord
🧠 My Second Channel THE MAKER LAIR: bit.ly/themake...
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Runpod: bit.ly/runpodAi
Latent Couple Extension: github.com/opp...
Composable Lora: github.com/opp...
Latent Couple Helper : github.com/Zun...
divisions=1:1,1:3,1:3,1:3 positions=0:0,0:0,0:1,0:2 weights=0.2,0.8,0.8,0.8 end at step=50
All ControlNet Videos: • ControlNet
My previous ControlNet video: • GET PERFECT HANDS With...
NEXT-GEN MULTI-CONTROLNET INPAINTING: • NEXT-GEN MULTI-CONTROL...
CHARACTER TURNAROUND In Stable Diffusion: • CHARACTER TURNAROUND I...
EASY POSING FOR CONTROLNET : • EASY POSING FOR CONTRO...
3D Posing With ControlNet: • 3D POSING For PERFECT ...
My first ControlNet video: • NEXT-GEN NEW IMG2IMG I...
Special thanks to Royal Emperor:
- Merlin Kauffman
- Totoro
Thank you so much for your support on Patreon! You are truly a glory to behold! Your generosity is immense, and it means the world to me. Thank you for helping me keep the lights on and the content flowing. Thank you very much!
#stablediffusion #controlnet #aiart #stablediffusiontutorial
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
WATCH MY MOST POPULAR VIDEOS:
RECOMMENDED WATCHING - My "Stable Diffusion" Playlist:
►► bit.ly/stabled...
RECOMMENDED WATCHING - My "Tutorial" Playlist:
►► bit.ly/TuTPlay...
Disclosure: Bear in mind that some of the links in this post are affiliate links and if you go through them to make a purchase I will earn a commission. Keep in mind that I link these companies and their products because of their quality and not because of the commission I receive from your purchases. The decision is yours, and whether or not you decide to buy something is completely up to you.
Its crazy how fast Stable DIffusion outscales all Paid services. Thats the power of the people
Literally every day I master one aspect of controlnet and another appears. This space ceases to amaze me with the daily advances
Master?? That's a bit of an exaggeration, no?
I get paid to do both prompt engineering and Training Models for bigger companies for concept art, so I have to learn each extension and maximize its usage. It can only be considered an exaggeration based on how fast the technology updates. For example, this video hasn’t been out for 24 hours, and there is already something called Multidiffusion Region Control which is an extra add on to this that allows you to sketch a mask in each section rather than using rectangular boxes/ratios. I spent the past 6 hours learning how it works, and what limitations are and I’m in a lot of communities getting updates about this stuff. Anyway, if that was your focus in my comment, you missed the point of my awe in how fast all of this stuff keeps updating
@@depthbyvfx9604 which communities if you don’t mind? I’d like to master as much as I can and maybe in the future get a job in this
Wow, this is really fantastic! You could essentially create comic book panels with ease this way, with a separate prompt for each panel. Thanks for yet another great video!
Maybe yeah, would be interesting to try!
That's an interesting idea in general. Like I wonder if anyone has tried the ControlNet with straight lines, but instead of using it for a room/building, lay out panels for a comic book prompt? Separate from Latent Couple, I wonder if that would work? Or I guess canny probably would if not.
The only problem with this is that you maybe get up to 300 dpi with this technique, which is absolutely horrible resolution for a manga panel. You can probably get around it with upscaling and inpainting, but then you hit a wall if your story contains original characters (because of consistency).
This could be solved with loras, but I feel like you'd have to train a lot of them.
I really can't wait for elite to come out as an extension, as it might be able to solve the consistency problem.
@@pladselsker8340 Hey, what is that elite extension?
You're probably still best served by generating each panel separately for now. Say you do six generations of each panel- you get to mix and match the best for each panel. If you try to generate them all at once, each generation will take significantly longer and the odds of all the panels being exactly what you want are honestly pretty low. There's definitely interesting potential, though- and definitely worth experimenting. It seems particularly promising with a style where one panel blends seamlessly into the next instead of having gutters.
It’s scary how fast this entire space is improving
I was still working on learning blender posing
It's amazing!
@@F5alconsHouse I think ill just skip this and download mdoels from other users xD that will safe a lot of time and i never used blender so yeah
The word you're looking for is "exciting."
This video is literally ALREADY out of date, now you can draw colored masks in whatever shapes you want instead of being stuck with rectangles.
This is just insane, each day there is something new, when is this even gonna stop 😳Thanks as always for the tutorials!
try taking personal photography and throwing it into a program like this with img2img, it's mindblowing what can be generated. dude I took some personal photography that took days to shoot along the erie canal. threw them into img2img and was blown away with what it came up with with the right prompts and checkpoints.
HELLO HUMANS! Thank you for watching & do NOT forget to LIKE and SUBSCRIBE For More Ai Updates. Thx
please can you post which version of stable diffusion, wed UI, lora.. your using as I just get errors on lora and Dream booth. ? or could you give us and install order with version's used so we can replicate your install.
What if I installed the latest latent couple with composable lora, and no latent couple section appeared in txt2img? Only the composable Lora did
@@anastasiaklyuch2746 same here, any solutions to this problem yet?
@@anastasiaklyuch2746 never mind I got it, Open cmd in sd root dir then paste this line below
git apply --ignore-whitespace extensions/stable-diffusion-webui-two-shot/cfg_denoised_callback-ea9bd9fc.patch
restart sd not just UI
@@zeeshanzaffar1435 It worked! Thank you, my heroic technomancer!
HOLY CRAP!!! Another HUGE advance. Now I can say a character has blue eyes without everything else being blue. Plus ALL the other things I can do. Fantastic!!!
Yup it's really cool!
it's actually nuts how far SD has come from the time this video was made, along with the extentions.
OK, so I just spent like two hours getting this to work only to produce horrible abominations, and here are two very important tips:
1) *If you are generating monsters fusing together:* The first subprompt (before the first AND) will be applied to the whole image (if you use divisions like in this video). This means that if you leave the settings like they are in this video and enter "a man AND a woman", you will smear the man over the whole image and then denoise the woman into the left side of the image, and do nothing for the right side (you entered only 2 subprompts). _You need as many subprompts as there are divisions. And you need to pay attention to the ratios of the divisions._
2) *If you are generating a person who is cut in half and generating the other person on their other half:* Increase the width of the image. None of the models I tested were able to properly generate two full persons like this with the default 512 width. I assume that's because the models were trained on 512 images, meaning they try to adjust the size of a person to 512. If you halve this 512 width, you are not letting the model complete a full human.
+1: You can leverage latent coupling with img2img and controlnets to better guide generation to what you are trying to achieve.
but the best part is making horrible abominations depending how you look at it 😅
yeah it doesnt work for me neither, it just mashes the loras into one...
I had problem 2) solved it by increasing the width of my image to 1024 while keeping the height at 512. Now it produces two different characters as prompted. Thank you!
I just keep generating one person who is a mix of both characteristics... I have Latent Couple enabled, but not sure what's going on. Any advice?
@@Gins. If you divide the picture to 1x100%, 1x50 left side, and 1x50 right side then you need to enter something like this:
forest background AND black man AND white man
You also need to set the aspect ratio to be wider. This will create a wide picture of a black man on the left and a white man on the right, with a forest in the background.
The Stable Diffusion space is evolving faster than I can keep up with it. Or so fast that I can't learn the previous new features and get good at them before something else comes out.
Thing is once something comes out, and I watch an older video of it to catch up, there's been another video that has updated the previous feature. I kind of wish it would slow down a little bit, I still got to go to work in a few hours. 😆
Yeah I feel you :)
Heck I'm still learning about merging checkpoints.
Oh man, I really understand you. I don't have time to learn in practice one function, 3 weeks later there's another, better one coming out. Or even UI elements can become slightly different or move somewhere else in some cases 😁
@@SkyGeekWave Yeah, at this rate, by the time I catch up to where it is now, Stable Diffusion may have already replace the president.
it's growing very fast. I remember not even understanding how to make multiple characters generate and thought it was impossible
This was such a good explanation of how this works. I've seen other tutorials on this before but none that actually explained it like yours did. I am going to have a lot of fun with this now.
I think the tools are advancing but getting quite complex in a way so hope some people can streamline controll net and coupls soon, but this is barely year 1! so there is alot of progress ahead
...wait for Multidiffusion. Veeeery promising!
I agree (this new tool makes sense to me, but only because I did table-based layouts for websites back in the day lol). But I bet it'll get there pretty soon. Leonardo and Mage (and probably other sites) seem like they're trying to wrap up functionality in a more user-friendly package. This is a perfect case where a good UI would help. Like you drag over a region of the image and it pops open a prompt and strength slider. You even could build in a full 3d posing tool with the hands/feet without having to go into another app and exporting/importing stuff. And of course would have a big library of predefined poses, etc.
unfortunately the installation doesn't work for me. Compsable Lora tab is there, latent couple is not. latent couple is also no longer selectable in the extentions, via URL installation I get an error message that it already exists. everything is up to date. the folder is in extention folder as it should. I restarted the web ui and also in the browser. any ideas?
I did a git pull origin master and restart the webui and it appeared.
Finally got this to work. Some checkpoints really don't like to render more than one person without face issues (that not even higres nor fix faces will fix), amplified by using loras.
Different checkpoint model and reduced lora strength, fixed it. Good video :)
I hope in future it will respect the reduced height more. Especially people will just be cut off or have their heads cut off, lol. Somehow SD still tries to put a full person in there, instead of shrinking it. to respect the area.
The possibilities are insane ! Thx for the tutorial.
Glad you like it!
There's a pull request on the repo that allows you to draw masks (with multiple colours) instead of messing around with calculating partitions.
I tried it. It basically solves composition problems almost entirely.
Can you share repo with me, thank you
Wow ... my mind continues to get blown with how fast powerful tools are being created. Makes me really curious about there we will be by the end of the year!
Tried around with this a few days ago and was surprised by how well it worked. Did not know about the Latent Couple Helper though. Makes things a lot easier.
Such power 🤯
Thanks !!! I can see those extension will be helpful for doing Comic panel , more precise t- shirt - merch and even book cover mockup direct inside SD and A1111. Awesome !
Hello Ai Overlord K, has this extension been replaced by the "Regional Prompter manipulations"? Thanks!
Would you ever do a video on the merge block weighted gui extension? The extension allows for more in-depth control in merging models than the stock 'Checkpoint Merge' ui thats in the base Automatic 1111 and it has pretty great potential.
I saw it, I need to try it out first
omg just yesterday I thought to myself that something like this would be amazing. And there it is already. How could I miss this video =D ?
Thank you for the video! Exactly what I have been looking for a long time! 😊
14:33 I have two remaining questions after watching the video. What about LoRAs trained on concepts or characters? Can I put one specific character in one side of the image and another one in the other? I ask because I've tried numerous times without any success, I'm getting the worst imaginable results. Secondly, I suppose this doesn't support textual inversion at all? I have some very clean, very well trained character embeddings but none of the cool new stuff seems to waste time on text inversion anymore. I wonder why? They are still pretty damn powerful, nothing has really changed in that regard. And they are like 1/250th the size. Is there maybe a way to convert an embedding to a LoRA so I can make use utmost use of what I already have? I'm begging you, if you have any information for me, please share. You would not believe how much I've dug for an answer.
I will sum up for ease:
- What about character LoRAs? Can we group together two, three, five specific characters in one prompt/image? I asked because my attempts failed.
- Are text embeddings out of fashion? Why is no new tech supporting them? I see them on the same level as LoRAs
- Can I port my good textual inversions to a LoRA somehow without going through the process of training all over again, lol?
I tried to use Controlnet and Latent couple and it kept fusing my two LORA characters together into a badly rendered hybrid one. Still trying to figure that out myself.
If anyone knows how to use Latent couple in combination with separate LORAs any help would be appreciated.
Thanks for this video.
This + Composable LoRA could make really great images.
Absolutely!
Edit: Turns out the regional prompt extension (which i installed but haven't used and don't know how to set up properly right) was enabled.
Looks like it works now (to varying success at least)
Edit 2: Nah, was coincidence it seems. Still get horrible results 90% of the time. the 10% are at least showing SOMETHING on both sides, but most often it does closeups of 1 or ignores it completely.
Doesn't work for me. like I write Beach AND man AND woman (simplified example), so i get a beach background and then a horrible mishmash of a manwoman monster, merged in the middle.
Same settings as in the video and taking into account what the comment with the 2 tips said
Now it switched over to basically ignoring the man and just making the beach and woman (on the correct side, just ignores the 1st subprompt)
What a time to be alive!!!! Thanks for this 😍
Don't forget to squeeze your paper!
@@Aitrepreneur
I can't seem to get this to work, enabling the extension and generating an image just results in one character with aspects of both prompts merged into one character
The problem is the size of images, if you want to have 2 subjects consider a width of 1024, if you want 3.. consider a width of 1536 and so on because each character takes 512 pixels
I'll definitely need to play around with this tomorrow
This is cool... that said, I feel like this at the back end of a painted rgb mask would be the next step.
I have installed with this tutorial, follow step by step. And even so, my Latent Couple dont work, its enabled and i have even tried with this model and settings, looks like its not doing anything with the images. Anyone else have this problem?
Do you depent on Nerdy Rodent in any way?
You are awesome! Thats exactly what I needed! Thank you so much!
Wow so incredible ❤❤
Really cool. I'm gonna need a minute to digest this information. It blows my mind how far this technology has come in such a short time.
true!
ikr i didnt even finish last tutorial
Getting ready for more insane content!
The model I'm using needs keywords at the start of the prompt and it doesn't seem to work with latent couple
Would be super cool if the LoRA part comes earlier or if there was a chaper division in the video time bar.
Yes, you made a video on this, and the explanation was super easy to follow, as expected, thanks so much!
Instant sub. I needed this.
I notice this won't work with lowvram mode but worked with med same prompt
I shouldn't have watched this before lunch. The pizza made me hungry.
We are almost there.
The ultimate version Of Stable Diffusion is almost Here.
It will be a Blender Addon that will combine the recently released Blender Skeleton for MULTI-CONTROLNET that our AI overlord talked about.
Combined with the next version of this which will allow us to assign a Prompt, Hypernetworks and Multi-Controlnets to each Skeleton and or "Control Meshs" and the Background.
And once Text To 3D, AI Animation and Images to 3D are also inevitably implemented as Blender Addons The fusion of the 2D and 3D Workflows will be Complete.
And with it The full democratization of animation.
It Will be Glorious and at the rate we are going It will be here Sooner than we realize.
Blender is a program the vast majority of people won't be using. Those same types of tools will be included with WebUi etc. -- In my opinion.
Beautifully explained!
Man, your videos are Incredibles. Thank you!!
hey great explanations !, but how did you manage to make the characters interact with each other ? so that they touch or look at each other ?
I installed the extension but it does not appear on the UI
This is so cool! thanks for sharing!
seems to be broken
This doesn't work at all. I can't understand how you managed to get the picture. I typed your settings and prompt 1:1 into my automatic1111 installation and all I get is one merged person. I don't know what I am missing but it must be something hidden so badly that this is completely useless.
Dang, can't get it to work. It always gives me one subject ... even with the same settings and prompt as in the video. any ideas?
are you sure that you enabled the extension? Any errors?
I'm getting the same results. Either one person or a mutated blend of the two. Once in awhile I get what I prompted for.
@@Aitrepreneur it's enabled, no errors, just keeps merging everything together... so weird
Same for me, I have a bunch of other extensions installed, maybe there's a weird interaction between them.
same for me, maybe for the low gpu? I have 1060 6 gb
Thanks! Important to note that you need to give the canvas enough space or it will merge the two concepts - took me ages to figure out, but luckily I noticed your 1024x512 dimensions and tried that, which immediately worked where 512x512 was failing!
Interestingly, once you enable the ControlNet 512x512 works again, but did you find any way to make it not break and morph the two concepts again on Hires Fix?
Man, you really saved me from getting nuts. I was trying so hard with damn 512X512, and was getting ugly mutants, thought I was doing something wrong, no matter what I did, then I saw your comment, and increased the resolution, and finally I get normal 2 characters, not a creepy mutant from the movies from 80's 😄
@@SkyGeekWave
I also found you can get it to work on 512 with a controlnet fairly reliably
@@thrillgates idk why, I was trying with controlnet openpose in 512x512, and was always getting mutants, or half of the face was male half female (I wanted it to draw a couple walking on the street though). So they were fused together, getting terrible radiation melt faces too. But when I moved to 800x800 + controlNet openpose, it mostly got what I was asking it to generate :) so thanks again.
@@SkyGeekWave
Ah yes I did find this too with a normal controlnet now I recall, setting a canny controlnet worked, but required some collaging of two singularly generated characters which kind of defeats the purpose lol
Hey, thanks a lot for your videos they are really top notch, I have request, can you please make a video on upscaling in stable diffusion, It would great since there is no video online which explains everything on upscaling.
dont work for me
I tried playing around with this and sometimes I get 2 characters, and sometimes I get 1. But never before did I get a flower garden background, nor a man AND a woman. I always get 2 men, or 2 women.
What am I doing wrong? I'm following your exact steps, even using the same model.
looking forward to an installation tutorial on comfy UI! I'm a node nerd
Do not work for me
5:05 _it knows_
I live by the Avenue Q philosophy of the internet.
can you explain regional prompter ?
just got into all this ai art generation stuff and this was very useful, have subcribr
sadly this does not work for me in the slightest, i just keep getting half a body on one side and one girl on the other, not sure why as i followed everything to a T
I have the same issues. There is a thread on reddit explaining how supposedly get it to work but even with copying the settings it gets me the same merged person
thank you~! many inspiration !
Just discovered Stable Diffusion a few days ago which I thought was insane but this is fucking crazy
Now I can make a harem book cover!
Really cool. Don't have to do a huge inpaint area on a completed image.
Holy diffusion
I tried with Chilloutmix model, but it doesn't create 2 characters
Wow! So powerful.
Please help! I'm using a SD 1.5 and i installed the latent couple extenssion from the list yet no UI shows up. I tried wiping it out and reinstall through URL install option but still deosn't work. Any idea how to fix this?
I have this as well. Very strange
Been trying to make a half human face with half venom face without any luck, I hope this will finally make it work!
Do you enable the composable lora, didnt work on my setup. Already followed your step all the way, but there's a frame skip you didnt explain anything before generating image.
Its up and running for auto now?? epic!
Thanks, been trying to figure this out!..
Now if someone can integrate this with the segmentation ControlNet, and manual segmentation definition/coupling (basically just a "prompt brush"), I think we'll start seeing what the future of working with this tech is going be like...
I think I saw on reddit that someone was working on it but code not released yet.
@@ShawnFumo Check again, it just got released.
Look for MultiDiffusion Region Control extension. It's what you're asking for I believe.
@@Gh0sty.14 wow, thanks, ill check it out, the pace this is moving is just wild!..
@@ArielTavori Yeah it's incredible but so hard to keep up. This extension just released a few hours ago.
Question, since you can use AND for the positive prompt, can you also use AND in the negative prompt to give discrete negative details to the individual zones?
I believe someone else mentioned it is a common negative prompt unfortunately
hmmm I installed it, but it just won't show in the text2image tab... any ideas?
I seriously can never get latent couple working
I can't seem to get this to work for me. I followed step by step and work on it for 3 days straight.
This is huge. Thanks for all your hard work. I never regret having the bell on.
Awesome! Thank you! ;)
Thanks for the tutorial man .... 🔥🔥
No problem 👍
5:05 "weird position but pretty cool" 🤣DEAD
Can someone help,i follow the tutorial step by step,but my latent couple doesn't show up😥😥
Latent Couple is not working for me.. I am enabling it but still only 1 character is generated. Please help
Do I really need to fit all the prompts for one character in one line then enter, put "AND", then do the other character prompts? or it doesn't matter as long as I put the "AND" word to separate them?
Nice and looks powerful, but why couldn't they implement photoshop style square region selection tool, like img2img has.
It isn't working here, it is still merging the characters =(, i will try together with control net later to see if it helps (it didn't, maybe it just don't work with AMD)
Is there any way to use this or any similar tool like Latent Couple Helper in Mac OS?? Thanks in advance for the good quality content
Can you add different seeds to go with the prompts? That would produce ultimate consistency
Out of curiosity, did you cherry pick the image results for the video? I'm following along but the results are terrible. Maybe 1 out of 10 generations will show what I prompted. The rest only have one person, or tries to blend the two people into a mutant lol
Does it work for stable diffusion automatic1111 ? I tried composable-lora
Public but it didn't work, lora still mixed together, not separated. Not sure where I went wrong.
NICE TUTORIAL. Latent cople gives me an error when pressing Visualize. To me works Regional Prompter extension.
Does it work with SDXL and Forge? Thanks!
Has Regional Prompter replaced this?
What if I don't have consistent results? It is normal? referring to the latent couple, it is difficult for me to get 2 characters out. sometimes it works but most of the time it doesn't 🤧
It might be easier using the pose controlnet at the same time
i am going to try this.... definitely xD
This is evolving faster that I can learn it :(
My Characters are blended together in the center of the image, split down the middle, half one type, half the other... How do i get two different figures in the image?
nothing works
Nice work! Do you think it would be possible to take s picture of a room and clear out all the existing furniture with a prompt that only an empty room remains? And then add custom furnitures in custom places with the latent couple.
Very useful, thank you very much ! By the way, is there something similar for comfyui ?