SegMoE - The Stable Diffusion Mixture of Experts for Image Generation!

Mixture of experts. Seems hot for AI text generation... but what if you had a mixture of experts for IMAGE generation? Oh. Segmind just did that. Welcome to SegMoE - the mixture of experts for SDXL, SDXL Turbo and Stable Diffusion 1.5.
Want to support the channel?
/ nerdyrodent
== Links ==
huggingface.co/segmind
github.com/segmind/segmoe
github.com/segmind/segmoe/blo...
== More Stable Diffusion Stuff! ==
* Faster Stable Diffusions with the LCM LoRA - • LCM LoRA = Speedy Stab...
* How do I create an animated SD avatar? - • Create your own animat...
* Installing Anaconda for MS Windows Beginners - • Anaconda - Python Inst...
* Add anything to your AI art in seconds - • 3 Amazing and Fun Upda...
* Video-to-Video AI using AnimateDiff - • How To Use AnimateDiff...
* One image Gets You a Consistent Character in ANY pose - • Reposer = Consistent S...

[@Mediiiicc]

Need one of those experts to specialize in "hands" lol

[@sherpya]

one expert per finger 😂

[@greengoblin9567]

Per cohk

[@MarcSpctr]

Finally a finetuned model for hands and legs can be used as expert, and maybe some model which can understand stuff like ON, ABOVE, UNDER, INSIDE, etc.

[@MrGTAmodsgerman]

What you wanna generate with "inside" in relation to body parts?

[@paulpardee]

early days, as you say... I don't think this really gives the concept a fair shake. You have models that are better at one thing than others, but all the models currently out today are generalists who just happen to be slightly better at text or prompt adherence, or counting... An expert model would be focused on just text or just counting and those don't exist as far as I know.
I'd love to see models built for this that have markup built in to tell Moe what they specialize in so it could direct that work to them... It'd be even better if you could have a standard library of models and Moe would dynamically load the best ones based on your prompt.

[@worthstream]

This will be a game changer as soon as it's somewhat optimized. Expecially if they do manage to release a finetuning framework. Using prompts to compute gating functions is an ok starting point, but a (relatively) quick fine tune of that can make the difference.

[@elihusolano5993]

Hope you have a speedy recovery. Thanks for the great content.

[@ritpop]

I don't comment a lot but your content is great. Hope you get better soon.

[@aimademerich]

This is phenomenal!!

[@kariannecrysler640]

So few comments! I’m not used to that lol. Hope you’re good my nerdy friend ✌️💕🤘🥰 🐭

[@NerdyRodent]

Will be soon!

[@kariannecrysler640]

@@NerdyRodent very happy to hear that 😁

[@c0nsumption]

Fn love that you’re always willing to get dirty when there no community support bud. Thanks for the hard work 🙏🏽

[@ImAlecPonce]

looks so cool!! I only have 16 gig vram though

[@stephantual]

Thanks that was fun :) 🤠

[@sandy66555]

hoping you're feeling back to full rodent normal soon

[@elihusolano5993]

can this new MoE be applied to Loras?

[@AC-zv3fx]

I wonder if it is possible to create MOE of Pony diffusion, AnimagineXL 3, realistic model and a model that is based on illustrations or traditional paintings.

[@blacksage81]

I feel like these researchers skipped a whole breakthrough by skipping Qlora, and the myriad of quantization flavors we could have played with and went straight to moe, when nearly all the models are just finetunes of the sd base. Its odd.

[@fast_harmonic_psychedelic]

i guess its a little better but CLIP training with partiprompts would be just as good

[@yahiiia9269]

Could you theoretically use multiple LCM Turbo models?

[@AC-zv3fx]

I thought those experts must be trained with the model, so it can know what model to choose

[@nickolaygr3371]

its like computer processors evolution

[@fast_harmonic_psychedelic]

the whole MOE paradigm seems to me to be theoretically dubious lol

[@Alice_Fumo]

Ok, but why?

[@LouisGedo]

👋

[@poipoi300]

Wonder if we could truly consider this MoE. Haven't read the code, but I suspect all this does is amplify bias, probably akin to LCM but instead it's distributed.

[@sadshed4585]

what cuda do you have? my torch is not saying cuda is available

[@NerdyRodent]

I use 12.3 locally

[@DoorknobHead]

___m_/ o o \_m___
0:46 Can someone take the Segmoe Ferret to the vet and get that ringworm removed from it's neck? Thanx, in advance.

[@oquletz]

i don't realy understand what is this. this is a tool to merge models? does it work for sd 1.5?

[@aimademerich]

Wow this whole time I thought your voice was AI, get well soon

[@fast_harmonic_psychedelic]

theyre all general models, none of the constituents are experts on any particular thing lol

[@mattkupka1702]

How was this much different than a checkpoint merge

[@kallamamran]

Isn't this just the same as merged models?

[@renovacio5847]

By by Chat GPT4 😂.. i was using it because the image generation.. but know..

[@JavierGarcia-td8ut]

in the SDXL I think you are using too low CFG setting... maybe?

[@AliasArketer]

I boggle at what has been done, I boggle at what may yet BE done. We're in territory that we can't show grandparents and convince them it isn't magic anymore than other silly daftards can be convinced it isn't copy-paste.

[@Guytron95]

groovy. Too bad they didn't include image-to-image but still groovy.

[@raymond_luxury_yacht]

24gb humblebrag

[@erics7004]

Me, with 4gb vram GPU 😢😢

[@user-oz9tf9zp7k]

why didn't this take off? Use a face, hands, background etc... expert and get way better images