FluxMusic Text To Music Generation - Local Test With NVIDIA 3090TI & Gradio

In this video, we take a deep dive into the "FluxMusic Text To Music Generation" model by running a local test on an NVIDIA 3090TI GPU using the Gradio interface. FluxMusic is an innovative research project exploring the extension of diffusion-based rectified flow Transformers for text-to-music generation. The model, developed with PyTorch, is designed to convert textual prompts into expressive musical compositions, pushing the boundaries of AI-generated music.
We'll walk you through the process of setting up and running the FluxMusic model locally. This includes the training and inference scripts needed to get the model up and running, leveraging the powerful capabilities of the NVIDIA 3090TI GPU. You'll also see a live demo using Gradio, where we generate unique music clips based on different text inputs.
Key Highlights:
Model Overview: An introduction to FluxMusic, its architecture, and how it builds on diffusion-based Transformers for text-to-music generation.
Local Testing: Step-by-step guidance on running FluxMusic locally with an NVIDIA 3090TI, using PyTorch's Distributed Data Parallel (DDP) for efficient training.
Gradio Interface: A demonstration of how to use the Gradio GUI for generating music, showcasing the model's flexibility and potential.
Pre-trained Models and Checkpoints: How to utilize various checkpoints (e.g., FluxMusic-Small, Base, Large, Giant) and explore the pre-trained weights and data.
Acknowledgments:
Special thanks to the FluxMusicGUI team, including the contributors from curtified and camenduru repositories, for their incredible work in advancing text-to-music generation technology. This project is built upon the foundational work of the Flux and AudioLDM2 repositories.
If you’re interested in learning more, exploring the model’s codebase, or experimenting with your own text prompts, check out the FluxMusic repo and join the text-to-music generation revolution!

[@RaverSnowLep]

That's great! I'm glad someone is working on this because I don't want to stay shackled to Suno forever.

[@OminousIndustries]

Yes, it is exciting to see progress in the open source side of this stuff. Suno is insane but I always like non-subscription alternatives.

[@RaverSnowLep]

@@OminousIndustries I just like being able to run everything locally and do things the way I want. I had been using a cloud AI for images up until recently and the kind of results I'm able to get on my own computer are so much better after learning everything to make it work. I want the same for the music.

[@OminousIndustries]

@@RaverSnowLep Same here. My experience going from Dall-e to running image gen locally was fantastic, with the added benefit of being able to generate hilarious stuff as well LOL

[@dafoex]

I love stuff like this. Maybe it's just in the realms of "inspiration machine" right now, but I like that it has the inhumanity of the computer - once competently programmed it will do exactly what you tell it - because sometimes that is exactly what you need.

[@OminousIndustries]

Thanks very much for the kind words!

[@TheInternalNet]

That's amazing. Creating custom lofi or intro music. Thank you for showcasing it. Always a blast to see your posts.

[@OminousIndustries]

Thanks very much for the kind words! It is pretty cool and I believe like the Open Sora to CogVideo jump we saw recently that this tech will be vastly improved in the following months. Consider something like SUNO which is mind blowing and it's only a matter of time before open source gets to say, 1/3rd of that or so.

[@TheInternalNet]

@@OminousIndustriesI absolutely agree. It's so amazing how quickly all of this is evolving and growing. Like a year from now is going to be leaps and bounds from where we are currently

[@sertenejoacustic]

That’s cool!

[@OminousIndustries]

Thanks very much!!

[@LucidFirAI]

Pro tip: start videos with like 10 seconds of the best output you can get. Heist film style "now you might be wondering how we got here"

[@OminousIndustries]

Sometimes I have done that, definitely a good strategy for showcase style videos like this.

[@drmarioschannel]

thanks OI!
i wonder if these models can be trained.

[@OminousIndustries]

Sure thing! Yes, I saw on the main github mentions of training it and the test.py script gives some info on how your dataset should be structured. They also specifically reference using multiple cards with ddp so with a bit of know how and a dataset of your own it is very possible to begin training this: github.com/feizc/FluxMusic?tab=readme-ov-file#1-training