Exploring the New Photomaker Project: Another IPAdapter?
Вставка
- Опубліковано 19 лип 2024
- In this video, I help you discover the latest Photomaker project, a tool that transforms faces into various artistic styles, and learn how to use it in this comprehensive tutorial.
Hello everyone, welcome to this video on PhotoMaker. In today's video, we explore the PhotoMaker project, a potentially groundbreaking tool that allows you to generate diverse images using faces as references.
The tutorial covers the installation process, solving errors, testing the custom nodes, and showcasing the results.
I appreciate if you can like and share the video if it was helpful.
Subscribe for more content soon!
[SUPPORT THE CHANNEL]
Patreon: bit.ly/44js1Xx
Paypal: bit.ly/45lJsIg
[Resources]
Original TencentARC PhotoMaker: github.com/TencentARC/PhotoMaker
ComfyUI Custom Node: github.com/ZHO-ZHO-ZHO/ComfyU...
Input image on Pexels.com: www.pexels.com/photo/a-woman-...
HuggingFace Gradio demo: huggingface.co/spaces/Tencent...
[SOCIAL MEDIA]
UA-cam Channel: bit.ly/47OterT
Twitter X: bit.ly/3ReP9D3
[PREVIOUS VIDEOS]
Advanced Custom Nodes: • Advanced Custom Nodes ...
Understanding Prompting: • Understanding Promptin...
ComfyUI End of Year Updates: • End of Year ComfyUI Up...
Custom Nodes: • Create Your Own Custom...
SDXL Turbo Gradio App: • How to Use My SDXL Tur...
SDXL Turbo: • How to Use SDXL Turbo ...
Python API for ComfyUI: • Building a Python API ...
Introduction to Gradio: • Introduction to Python...
[TIMESTAMPS]
00:00:00 Introduction
00:01:05 ComfyUI Custom Node
00:03:00 Installation of PhotoMaker in ComfyUI
00:07:39 The Custom Nodes
00:07:50 Loading ComfyUI workflows
00:09:08 Error loading the workflow file
00:09:22 Manually creating the workflow
00:12:37 Finding good portrait images from pexels.com
00:14:06 First test run
00:14:21 Error kernel not found
00:16:56 The output images - final results
00:19:50 Conclusion
00:20:28 Another error invalid value with black image
00:21:34 Final thoughts
00:23:24 HuggingFace Gradio Application
00:24:09 So that is it
00:24:47 Thank you for watching
00:24:51 I will see you in the next video
[TAGS]
Photomaker, ComfyUI, Digital Art, AI Technology, GitHub, Open Source, Face Generation, Artistic AI, Hugging Face, Tech Tutorial, stable diffusion, IPAdapter
[HASHTAGS]
#Photomaker #DigitalArt #ComfyUI #AI #TechTutorial #OpenSource #GitHub #StableDiffusion #IPAdapter
You have explained this extremely well. As with you nI end up running into the out of memory error even with 12GB of VRAM in my case. This project does have some potential but they need to fix some basic issues ASAP so it can actually work for the majority of people.
Thank you for the feedback, @spiffingbooks2903! The out of memory error seems to be a major issue. The TencentARC Github Repo suggest having 15GB of VRAM but like I showed in the video I have only 4GB of VRAM but was able to use it although super slow. I do have 32GB of system RAM which is what the workflow uses. From further testing, I noticed that the workflow uses 3GB of VRAM and 27GB of system RAM. So 12GB of VRAM should be plenty.
Excellent video and great tips applicable to multiple situations. I’ve learned a lot today! Very grateful amigo! ❤️🇲🇽❤️
Thank you so much, @WhySoBroke! I'm happy to hear that you found the video helpful. Your support means a lot.
valuable video, many thanks for taking the time to create it!
Thank you so much, @hippotizer! Glad the video was useful.
Awesome tutorial brother. Simple and to the point. Thank you very much👍
Thank you so much, @dreamzdziner8484! Appreciate the feedback.
I'd like to thank you, I'm always learning something new from your channel. In this tutorial, working with a Portable embeded Python application e.g., compared to working in a more typical VENV has helped my understanding of some subtle differences.
Happy to have helped you, @bwheldale!
Great video. it worked for me on the first try.
Glad to hear it worked, @bordignonjunior!
You make great informative videos. thank you very much..
Thank you so much, @SezginAhmetKaratas! I truly appreciate your encouragement.
First Thank you for teaching all the steps you might not know how rare it is to see someone doing all the part, showing all the errors etc...
And IPAdapter is still the king specially the new FACE ID SDXL specially with Upscale, tried Photomaker none of the image it made was close to the refrence image.
Thank you so much for the kind words, @0A01amir! I'm happy you appreciate my approach in explaining each steps.
Yes, I think so too. IPAdapter has so many use cases and gives more control over the output image. PhotoMaker is too early to compete right now. I am just glad that the community is exploring new possibilities.
@@CodeCraftersCorner I came out a little bit ungrateful, yes i too like this community and the new possibilities they create.
No no, you were fine. Happy experimenting!
Thank you !
You're welcome! Thank you for watching!
ty man
You're welcome! Thanks for watching.
Awesome video. Liked. I have several questions, if you don't mind answering... :)
a) What is the difference between the checkpoint and the base model? Do we need both?
b) You used 1 reference image, is there an ideal number of reference images we should choose? How do we set-up the pipeline to handle more than one reference image (in comfy)?
c) What's the intuition behind replacing bfloat16 with float16?
d) Any methods or parameters that you are aware of that will guide the generation process to as close as possible to photorealism?
Thank you for watching, @mehradbayat9665! Yes, no worries!
a) No, you do not need to have both checkpoint and the base model from hub. Only one is enough. If you have an SDXL model already downloaded as safetensors file, then use the checkpoint else use the base model from hub node to automatically download one.
b) The authors say that 1 reference image is ok but more is better. Ideally, you want to have the same character at different angle to get the best result. Add multiple LoadImage nodes, each with the character at different angle, then use the BatchImage node to combine the images into one image pipeline. You can have multiple BatchImage nodes as well.
c) bfloat16 is usually not supported by old graphic cards which is why I suggested changing to float16. If the question was expecting a technical answer, I suggest reading the blog post by PyTorch here: bit.ly/3Hpt1Ak.
d ) As far as my few testing went, I found out that I am able to get closer to my prompt when not using any style; no style option. Simply describing everything in the prompt. You can try same by adding photorealism tokens into your prompt.
Update on b): The ref image node has a dropdown where you can choose between single or multiple images. Change to multiple and set a folder path in the text field above. You can place all your input images in that folder. This should be easier but does not allow you to modify the images beforehand like cropping.
Thank you!
No worries!
I really like your videos!!!! I noticed that PhotoMaker does not produce full body pictures..
Thank you so much! Yes, that is correct.
do u know how to fix the SyntaxError: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5). when trying to queue prompt,i keep getting this error message no matter which prompt i use,and if have correctly setup the nodes as in the video,im really stuck im on a mac m2 by the way.
Hello @osmanhassan7577, I am not familiar with running ComfyUI on MacOS but you can try the following:
- Start ComfyUI like usual.
- Load a basic workflow you know will work (even SD1.5)
- Generate a quick image to make sure the ComfyUI is working correctly.
- Once the image is done, without restarting ComfyUI, clear the canvas.
- Make the workflow from the video and try generating an image.
In my experience, the JSON syntax error happens when using an API. It is saying that the JSON file is not in the correct format.
Running fine on rtx3090 ti ... But what do you think about the fact that you can add a path for pictures and indeed use more than one picture? How many pictures is the most senseful amount? 4, 8 or even 25 like when you would train a LoRA?
The original repo says that one picture is ok but more is better. I suggest experimenting with 1, 2, 4 and compare the results. I would choose images with different angle. The repo also mentioned: "Although we do not perform face detection, the face in the uploaded image should occupy the majority of the image." I personally do not think that using 8, 25 or more will make much of a difference as this will not train the model instead add more information to the model. If the information received from the images are the same, it does not make the model better.
What are your arguments when starting?
I have these --lowvram --force-fp16
Generating all steps and then it always ends with this
Allocation on device 0 would exceed allowed memory. (out of memory)
When I once plugged in the wrong Clipvision model into the IP Adapter, they also wrote to me about memory. So the message about lack of memory does not mean memory at all?
Hi @ponponych2, I am using the run_nvidia_gpu.bat file in root ComfyUI folder. This is the command: .\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build. I have not changed it.
Great comprehensive instructions! However, after clicking Queue Prompt, I get "Unknown Error". No other info... Not very helpful..
I appreciate your feedback, @Taymar78. I'm sorry to hear that the video didn't address the 'Unknown Error' issue. Personally, I haven't encountered this error. Hopefully, someone with a similar experience can offer assistance. I suggest waiting a few days and check the Github repository again for a new version and try again. It might help.
When i launch run_nvidia_gpu.bat file, it did open correctly but in the "Add Node" there is no such things as Photomaker, did i miss something ? Where does the actual Photomaker models is, because in the video you did not mention if we were supposed to install it. These questions may be stupid but i dont know none about coding at all and in informatics in general. But youre videos seems to be really great, Thanks even thought I didn't make it
Hello, were you able to install it? I showed the installation process in the video. This version does not require downloading the model manually. It will auto download when you run the workflow the first time. Alternatively, you may want to check out a new video I made on PhotoMaker here: ua-cam.com/video/CYx6I0OetF4/v-deo.html. It uses a different implementation but will give the same results.
SyntaxError: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5)
i have this issue i follow all in the video can you suggest how to resolve this
Hello @venkatasudhakarayenampudi! Thank you for watching! The developer made some changes and the nodes changed since the video. I am editing a new version. The video will be online this week. I will share new workflows.
I have a photos folder of faces. If I use img in my pompt - a woman's face is ALWAYS produced - even if my prompt specifies a man's face! So I put some male faces into the photos folder, but still the same. I stopped using img, and the results changed, so that a male face was possible. So, I am unsure as to what the folder with photos can do? Maybe there should only be one photo in there ... I will try.
Hello @JackTorcello, the repo got a lots of updates as mentioned by @uk3dcom. I have not tried the latest. It is possible img is no longer required. I will try during the weekends.
hey with the ..\..\..\ step it just says it cant find the path specified, any tips?
Hello @swagabrownie! I will first try the following to troubleshoot:
1. If you are not using Windows operating system (Linux and MacOS), then replace the backslashes with a forward slash like this ../../../. Backslashes will only work in Windows.
2. The two dots means navigating back to a parent folder from the current directory. So you may have to customize the command if you have installed ComfyUI manually and have a different virtual environment.
3. If you are using Windows: try going into the python_embedded folder > find the python.exe > right click on it > click on Copy as Path > go back to custom nodes > photomaker folder > type CMD in the address bar and press enter > in the terminal / CMD window, press CTRL + V to paste the path > press space > then type: -m pip install -r requirements.txt > press enter.
See if that helps.
@@CodeCraftersCorner thanks for the response! i’ll go through each step & report back
hey i got it working! just a bit slower than expected. im running on a 2060 super with 8gb of VRAM
@swagabrownie Glad to hear you got it working!
@@CodeCraftersCorner think it should be slow with my gpu?
Im runninng ComfyUI under Debian, with 32GB RAM and RTC2060 6GB VRAM, and I have tryed running on CPU only, but keep getting "out of memory"
Hi @BjrnChristiansens, while I do not have a Linux build, I understand that despite having a good system you are experiencing "out of memory" issues. I am assuming you manually install ComfyUI. Maybe you can try reinstalling using this script from Github: bit.ly/428yhSj. Other things I can think of for you to try
- avoid video generation with Stable Video Diffusion - High requirements.
- Try running the default workflow and see if you can generate with SD1.5 model.
- Try adding arguments to python main.py commands. You can find the commands in the ComfyUI repositories. Adding --lowram argument for example.
- Check if you have the correct PyTorch install with the correct compute platform selected.
Other than these, I am not sure what could be the issue.
I hope someone with a similar build sees the comment and assist.
Try with --lowvram on the command line.
That's a good tip to try.
As for me ComfyUI auto detects like this:
"Trying to enable lowvram mode because your GPU seems to have 4GB or less. If you don't want this use: --normalvram
Set vram state to: LOW_VRAM"
BTW, I have 32 GB of system RAM and when running the workflow, it uses 27GB of it.
@@CodeCraftersCornerComfyUI works normally perfect. Running it with --lowvram, have also tryed --novram and --cpu. I can run everything else...
I see... The only thing that I can suggest for now is to try this version of PhotoMaker: bit.ly/47LY2ci. It is implemented in the ComfyUI reference way of doing things. Hope this version works for you.
I appreciate the detailed explanation. However, I couldn't succeed. My GPU is RTX 4070 Ti 12GB, and I followed the instructions carefully, but I'm getting an error message like
"torch.cuda.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory) Currently allocated: 9.18 GiB Requested: 1024.00 MiB Device limit: 11.99 GiB."
Do I need to upgrade to an RTX 4090, or is it worth investing in such a node?
Thank you for the feedback, @KKsinsa! No, I do not suggest upgrading right now. The node is too early and many people have reported getting out of memory error. There is a better implementation of the PhotoMaker project here: bit.ly/47LY2ci. I am testing this new implementation and will get back if I get it to work. For now, do not invest in the node. Instead, it is better to wait for a new version. Hopefully, the developers will fix the issues soon. In my case, I noticed that the workflow uses 3GB of VRAM and 27GB of system RAM. Since you have 12GB VRAM for the GPU, it should be enough.
I have the exact same card and I would get this error frequently. I fixed it by going into my Nvidia control panel > manage 3d settings, and making sure that CUDA system fallback policy is set to 'prefer system fallback.
Now if I get that error, I simply close the error box and queue prompt again right away, and it works just fine. I hope this works for you.
good luck
Great find, @Taymar78!
@@CodeCraftersCorner Thank you for the kind response. I was able to resolve the issue using the method suggested by Taymar78. By going to Nvidia Control Panel > Manage 3D Settings > CUDA System Fallback Policy, and setting it to 'Prefer System Fallback,' the problem was resolved. I appreciate it. I've realized that it's wise to communicate with each other to solve any issues. Communication is crucial.
@KKsinsa, I am glad it worked. Happy experimenting with it!