Відео

depending on the setup - I showed Q8_0 quantized with CLIP FP16 on CPU, so that would be ~20GB, running on a 4090/3090 However, there are many quantized setups and people in community are running under 16GB, i cannot confirm, but possible to squeeze down to 12 if you offload CLIP to CPU, although that required over 32GB system ram, so many optimization options these days it's hard to test them all. for the safest bet and with the best quality on Local, 20GB is where i stake the signpost on this one.

thanks !

You know i love to share the goods :)

I had not heard about this and would love to learn more ! I do believe it is what is needed - we are limiting the potential of the tech by censoring datasets, machine learning is highly capable and are failing to truly comprehend the power of these LLM's

@FiveBelowFiveUK he is creator of Dolphin LLM & others without any restrictions, on huggingface. Prometheus of our time.

Correct ! we are spearheading a community led research effort, thanks to the hard work of Kijai who brought us the wrapper - i'm pretty much finding best settings with the help of all involved, so our workflows have the best params for quality, and efficiency on local GPU. Find my open research thread here: github.com/kijai/ComfyUI-MochiWrapper/issues/26 we just received the VAE Encoder, so image 2 video workflows are now up, but no video until tomorrow, because i'm so hungry and it got too late tonight :)

i used 100 steps, 7 cfg and 3 seconds (73 frames), so due to those choices is added a lot to each segment of the same prompt ~20 minutes each. I queued it up and left it running for some hours and came back to around 40 videos, then used my randomizer (shuffle video studio) to create two patterns which were put together. It's just one prompt mostly on random seed, the prompts were built in the last video with my ACE-HoloFS system and Claude. The decoding of the latents with the new spatial tiling VAE was the easy part and takes 30 seconds each. So it was all done today, but it does take a long time, when you double the frames and add some seconds :) I'm happy with it though as we are seeing real progression for Local video generation ! Should be more to talk about tomorrow too - I recently updated my workflow packs to include a working Image to Video setup - it used the new VAE Encoder !

@@FiveBelowFiveUK look fwd to your post tomorrow. Thanks.

thanks so much !

none of these node affect the installation of or changes to Torch. I can only advise users of Comfyui portable which used embedded python. If you receive errors like "Torch was not compiled" something else is going wrong, if you need helop you are welcome to come and ask, but it seems like you need to look at how you launched ComfyUI, a workflow cannot break Torch when it is not listed int he requirements.txt Chances are you just didn't launch it as you had on other occasions.

thanks again for adding this transcription ! I know they are very useful for everyone !

YES ! in fact some users with 3090 or 16GB GPU were also able to run it, further optimisations can be made at expense of speed to fit inside even less vram

​@@FiveBelowFiveUK Woah! Nice man! That's what I have been waiting for... Just got into this AI world and Mochi'1 for me, is the best video generator out there for local play.

Yes and yes I'm doing some tricks in the workflow, such as offloading the CLIP model to CPU, which removed a good amount of load from the VRAM allocation, also we used the Q8_0 (there is also a Q4_0 which halves the load again) I'm using a 4090, but users in my group are running it on even 3090 and <16GB GPU. (with Q8) These videos were generated in 2second (49 frames), with 50 steps, so they only took 5 minutes each, i have not used any video enhance or upscalers to give a good idea of the output quality - and i did not remove any videos from the sequence that was generated so what you see here has no cherry picking - to give an idea of consistency.

the project should be using a virtual environment, this is why the .bat files load the environment before running the scripts, I hope to improve this but i'm not sure what is not working for you exactly. Could you swing by our discord and post in my help section? I'd be glad to help as other may be affected by this and i might be able to change something to address it :) links in description !

That's a great achievement, keep embracing AI and it will take us all to new heights ! stop by my discord if you get time, i'd love to see this tool you have built too :)

Thanks so much - it's an honor !

yes a valid next step i want to look into is putting this tool into a Gradio style interface, this would make it much easier to use too - i wanted to get the functionality nailed in python first and then build the GUI over the top. Thanks for your kind comments and i totally agree you have to wrangle the LLM else he will gas light you hahaha :)

@@FiveBelowFiveUK 😆- Yeah! There is a very simple window interface which is not Gradio (I forget the name, but it has a feather as icon) - It's a simple window accommodating buttons checkbox sliders, that sort of thing. Sometimes this is all you need. When I was arguing with the LLM, some of the misunderstandings were about how Gradio works (It's a bit spoiled, so to speak and tends to store data in 'windows USER TEMP' which I very much dislike for the lack of space on C:) So it may be useful to have simpler interface and ask the LLM to always store TEMP data within its own folders. Your dedication and sharing is amazing! - Thank you ever so much!! 🤩

(PERPLEXITY) Answer: The simple window interface you are referring to, characterized by a feather icon in the top left corner, is likely associated with the Tkinter library in Python. Tkinter is the standard GUI toolkit for Python, allowing developers to create windowed applications.

thanks for watching !!

I'm a huge fan of voice to voice, if you can do good impressions, this can get you across the line. I showcases Kits.ai in the past, it's a paid service tho. We are trying to show off more Local stuff so RVC is the best imho, trouble is finding it working on local can be tricky. If i find a good one you can expect a feature on this channel ! With all the AI project you might notice i tend to go around in big circles - it's impossible to keep up with it all :)

Thanks for the feedback - it'#s really helpful for everyone to see what results you have! I must admit that the nodes for the mochi loader are pulling in the models every time, but i think i can improve that by updating my Runpod Provisioning script ! Expect an update on that front soon ;) Regarding the Decoder OOM's with no Tiling - I think this was a 100GB VRAM model (!) so we are still squeezing it in even on aa 48gb. I wanted to offer the "full fat" option for those people that used Runpod as a primary platform. I only decode 2 second clips without the VAE tiling, Video VAE decoding takes an insane amount of VRAM due to all the frames.

@@FiveBelowFiveUK no result yet mate hahah i think my runpod had a headache. also i was not aware that there is an i2v for mochi? perhabs thats why its not working as intended? but keep up the great work :)

Thank you! Cheers!

hm, i have not seen this problem on my PC, it's a new platform for me also, all i can suggest is to try and see if they have a support forum or discord where you could ask? If it was me i would have done the same thing, delete it, reinstall it and hope it works.

i think you meant Billions of images, however these models we are using no longer contain copyright images, only SD1.5 and a small amount of SD2.1, even smaller portion of the SDXL dataset contained recent works by living artists IIRC. When i speak about "training on your own work" we mean by that "finetuning checkpoints" or "Finetuning Lora patch models" for various diffusion models. You are not wrong, but in practice a good fine tune will only produce estimations of your work when prompted for that purpose. FLUX1 is the leading base model and if you have enough data (your work) and you caption it accurately (tutorials on this are on this channel) it is totally a viable route artists are using right now! Personally, if i had the budget i would train a true clean base model on only Public domain works, this has been the goal for some time now - although we cannot guarantee this due to the time it would take to audit 5 billion images by human eyes (~20 years with no sleep) !

@FiveBelowFiveUK wow, you're right. I'm interested in that finetuning technique for sure. Do you know Eric Hartford, the famous publisher of uncensored LLMs in UK?

AFAIK there is no VAE encoder, so all you can do is "Vision to Image" which will approximate and image using a complex description/captions. However this is only an update away if the author decided to add this, i was even tempted to try writing one myself. It's still early days so i decided to cover a few other things before coming back to it - hope that helps!

the filename definitely does not contain all the numbers etc with .incomplete extension, you were right to try re-downloading the files. "dw-ll_ucoco_384_bs5.torchscript.pt" is the full filename. It's using the huggingface cache, so you might need to delete that file and have it refetch. I worked with a mobile internet for years so i feel your pain, i've seen this before. Often the huggingface cache is not in the comfyui path but somewhere else in windows (if you are even on windows). direct download might not work because of how the HF cache works. Good Luck !

@@FiveBelowFiveUK I just used another node and It worked. AdvancedLivePortrait. I doesn't like It because It needs to have two animations nodes after the image. I'll try to fix this problem another time. Don't you know if there's some nodes for facefusion 3? They have liveportrait behind It now. I need something to generate audio through Text and lypsync it with some image, do you have any tutorial for comfu ui?

I would not like to guess as i do not have an AMD GPU, all i can say is try it and see? If it doesn't, let us know, because it will help others :)

Yes, there are people in my discord that are using 3090 with this model, you would use the V6 Fast settings or V5 (Q4/Q8 + T5 FP8 Scaled). I will be making new versions to support lower VRAM this week, i had some other things to cover first :) I explain in articles all the different setups if you can't wait :)

@FiveBelowFiveUK Thanks broski

Yes I have, I releases a new "Thoda-SD35-Pack" on my github (DJZ-Workflows) and also on CivitAI, check more recent video releases to catch the links. I have put out fixed workflows for SD3.5 Medium and Large already ! Training will get a video as soon as it's more practical for everyone to access :)

@ awesome thank u! Would love to figure out how to train SD 3.5. I trained 1.5 with the Dreambooth Notebook before but it’s been a while and maybe there’s new tools out for training.

github.com/chflame163/ComfyUI_LayerStyle -- this should be what you need, i think its in the comfyui manager - it is unloading the VRAM to help fit the models into VRAM on your GPU, i place them there to help with the crazy load these video models use. Full details in the links in description! feel free to ask more questions if you have them :)

sounds like you need to update the pytorch to at least 2.5.0, use the .bat in the update folder (comfyui) update with dependencies.

@@FiveBelowFiveUK ok, thanks I'll update. Do you have any optimized workflow to run on an rtx 3060 12 GB vRAM?

🤝

until there is a VAE decoder all we can do is use ipadapter or Vision type nodes to get tokens from an image and then prompt them, this is only an estimation not true img2video - but i don't see why a future update would not add this in time, i think it's less than a week old. I'll be covering it for sure ! ~ and Thanks so much :)

@@FiveBelowFiveUK Great! The most relieving thing is that there's no technical reason that the model itself would not bend to that purpose. This model is already so good that it's only a question of time when the open source AI-based generation of video will be crazy epic! Most models seem to fail in motion, and video 2 video is a huge help with that. For example, I have been creating music videos with Runway mostly using Runway but with ComfyUI we will get more control instead of doing expensive lottery. In my opinion, what would be needed in order to have a genuinely usable video tool for music videos and movies instead of animated portraits and talking heads, these inputs would be needed: 1. Driving video for motion 2. Reference image for background 3. Reference image for character 4. Text prompt for controlling background action, camera movement and such Keep on keeping on!

Thanks for providing the Summary !! it's a big help for people

Watts the problem

yeah i keep on looking away from the mic haha it's a new mic already see, i'm still getting used to it being on the desk and not on a boom arm.

hahaha :)

This guy is brilliant

Big Thanks everyone :)

because it's so technical i have not even covered it yet, but the short story is you have to have the files in the comfyui folder (where the startup.bat files for comfyui are) and you do "pip install filename.whl" but this can break things, so again, i hope to return to this in a future update.

What do you mean? it's just me :) I used to use after effects and rotoscoping, but now it's all in a comfyui workflow, if you see the first video on the channel the secret is hiding in plain sight. Depth + Openpose, and a wireless mouse :)

pleasure is all mine - thanks for watching !

I agree to some degree but the cloud services like minimax also generate much faster than what any highend PC can. Putting 5 in a queue and generating 2 at a time, sometimes within a few minutes simply isnt happening locally. I choose to use both cloud and local for now.

It's not only about money. It is about as much privacy and safety of your data as possible (assuming it is even remotely possible with AI technology).

sure but using a 48gig vram card on runpod is cheaper than my electric bill, so if its a matter of money some people might wanna take that into account.

@@quercus3290 I understand a lot of the reasoning and calculations behind it, but I always struggle when I ask myself 'why'. Some people probably makes models, videos and images as a hobby, i make images for fun myself, it literally replaced gaming for me. Though I can't help but notice that there's also a lot of people that starts with generative AI thinking they're gonna make some cash, or fame, or both- Sinking money into it with expensive time constrained services.

​@@TheGalacticIndian What's privacy anyway?

agreed - that is why i have not even covered adding flash attention, to be honest it's the difference between 20 minutes and 10 minutes, i can wait :)

@@FiveBelowFiveUK is there any alternatives that you would recommend?

For maximum quality, the latent sideloading workflows in V6 are almost perfect - certainly peak as far as this model is concerned, you can also double the steps. Some people use 200 steps but i found 50-100 was enough. 49 frames was enough for my use case, but i know some want longer clips. V6 Fast is a good example of shorter videos. However - the reason that is required so much power to decode the latent files (runpod) was i did not use VAE tiling, this is insane because that was how this was even able to run on local in the first place - i wanted to show that the quality always suffered from ghosting no matter what tiling setting you used. Its the seams showing you see.