Building My Ultimate Machine Learning Rig from Scratch! | 2024 ML Software Setup Guide
Вставка
- Опубліковано 15 тра 2024
- Join me on an exhilarating journey as I dive into the world of Machine Learning by building my very own ML rig from scratch! This video is your all-in-one guide to assembling a powerful machine learning computer, explaining every component's role in ML tasks, and setting up the essential software to get you up and running in the world of artificial intelligence.
🔧 What's Inside:
Component Selection: Dive deep into the heart of my machine as I explain why I chose each piece of hardware for my ML rig, covering the CPU, GPU, RAM, and more. Understand the critical role these components play in machine learning projects and how they can accelerate your computations.
Parts list w/ cost breakdown: docs.google.com/spreadsheets/...
Building Process: Watch as I put together all the pieces, sharing tips and tricks for assembling your machine learning computer. Whether you're a seasoned builder or a first-timer, there's something for everyone to learn.
Software Installation: The right tools can make or break your ML projects. I'll walk you through installing and configuring key machine learning software and libraries such as PyTorch, TensorFlow, CUDA, local Large Language Models (LLMs), Stable Diffusion, and more. Get insights into each tool's unique strengths and find out how to leverage them for your projects.
📚 Why This Matters:
Building a dedicated machine learning rig can be a game-changer for hobbyists, students, and professionals alike. By customizing your setup, you unlock new potentials in processing speed, efficiency, and the ability to handle complex models and datasets. This video aims to demystify the process and empower you to take your first step into a larger world of AI and ML.
💡 Perfect For:
AI/ML enthusiasts eager to build their specialized hardware.
Anyone interested in the hardware aspect of machine learning.
Viewers looking to install and use major ML software and libraries.
📌 Timestamps:
0:00 - Intro
0:25 - Parts
3:04 - The Build
4:48 - Ubuntu Server 22.04 LTS
7:25 - Tailscale
8:10 - NVIDIA Drivers
8:54 - Ollama
10:01 - Code Server
11:30 - Anaconda
12:07 - CodeLlama
13:47 - PyTorch
14:37 - Stable Diffusion
15:41 - Docker w/ GPUs
16:37 - Isaac Gym
18:23 - Tensorflow & CUDA Toolkit
21:10 - Conclusion
🔗Links & Resources:
Graphics Drivers Issue: askubuntu.com/questions/14663...
Docker Issue: stackoverflow.com/questions/7...
Isaac Gym:
catalog.ngc.nvidia.com/orgs/n... docs.omniverse.nvidia.com/isa...
#MachineLearning #TechBuild #AISoftwareSetup #DIYComputerAssembly #MLRig #ArtificialIntelligence
You put so much effort and knowledge into this video! I watched all the way and it was interesting how you demonstrated the whole build from scratch. Keep it up!
Thank you so much for watching! Excited to make more ML videos 🙏
@@sourishk07 Better use llama.cpp instead of ollama, faster and more options including model switching or running multiple models simultaneously.
Thanks for the recommendation! I'll definitely take a look.
I like Ollama because of how simple it is to get up and running and that's why I chose to showcase it in the video.
@@sourishk07 You don't seem the kind of people who likes "simple" things. Anyway if you want to run llama.cpp in a simple way also you can do it.
Hi @sourishk07,
I had considered the same config as yourself but then changed my mind due to:
1. the unstable 14900K performance due to MoBo feeding the i9 insanely high power. Please Do make sure you enforce intels thermal limitations on the Asus MoBo bios settings. 😊
2. Instead of the NR200P I opted for AP201 case so that a 360mmAIO can be used for the CPU.
3. I went for a used 3090 as much of my focus will be on using the A100 or AMD Mi300x on the cloud.
ROCm has made huge progress, noteworthy is the efforts that George Hotz is taking to make ROCm more understandable for the ML community.
Overall congratulations buddy, hope you succeed at your goals.
Hi! Thanks for watching the video and sharing your setup. You bring up completely valid points.
1. I personally haven't had any issues with the 14900K stability. I didn't turn on AI overclocking in the BIOS and just left settings at stock (except XMP for RAM). I'm probably more wary with any sort of overclocking after that news came out now though lol
2. The reason I opted for the smaller case was because I wanted to try building in a SFF for the first time. The good thing is that cooling hasn't really been impaired, although a larger radiator never hurt
3. I should've considered a used 3090 as well, but because I wanted to do some computer graphics work as well, I opted for the newer architecture.
And while the advancements in ROCm do seem promising, I'm not sure anything will ever take me away from NVIDIA's vast software suite for ML/AI, but maybe one day, we'll see!
This was a great video. I had to pause it like 10 times to make bookmarks to all of the resources you dove into. The I saved it to Reader, and summarized it with ShortForm. Great stuff. You went into just enough detail to cover the whole project and still keep the video moving along.
That was a balance I was trying really hard to navigate, so I'm glad the video was useful for you! Hope you have as much fun setting up the software as I did!
I found your channel today and binged all the content. Please please please keep this up
Wow I'm glad you found my channel this valuable! Don't worry, I have many more videos coming up! Stay tuned :)
This is such a good video. Thank you, can't wait to see your channel grow so big, you're awesome and oh we share the same process of doing things 😅
I really appreciate those kind words! Tell me more about how our processes overlap!
Great video! Thanks for sharing
Thanks for watching! Stay tuned for more content like this!
Cant wait to build my first ML machine
Good luck! I'm really excited for you!
Great video!!
Thank you Soham!
Nice video, well done ! this is promising content! Can't wait to see you try some Devin like stuff or test other way to use LLMs.
Thank you so much for watching! It'll be really cool to be able to run more advanced LLMs as they continue to grow in capabilities! Excited to share my future videos
This, is days of work!!! Compile freely in 1 single video. Thanks!! For that.pricesless information here
Relax it's 2hrs hard work max.
Don't worry, I had a lot of fun making this video! Thanks for watching and I hope you're able to set up your own ML server too!
This video is truly exceptional.
I'm really glad you think so! Thanks for watching
This is actually sick
Thank you so much!
Super cool video!
Glad you liked it!
Cool vid! Not sure about pulling hot air from inside the case, through the rad, to cool the CPU though. But really a great video for anyone interested in setting up their own AI server!
Hi! That’s a good point, but from my testing, the max difference in temperature is only about 5 degrees Celsius. Also, keeping the GPU cooler is more important.
And because the only place in the case for the rad is at the top, I don’t want to have it be intake, because heat rises and the fans would suck in dust.
Thanks for watching though and I really appreciate you sharing your thoughts! Let me know if there were any other concerns that you had. Always open to feedback 🙏
Absolutely love this! Building a machine learning rig from scratch is no small feat, and your detailed guide makes it accessible for anyone looking to dive into ML. From hardware selection to software setup.
Thank you so much!!! I appreciate the support 🙏
Great work, KUDOS
Thanks a lot! I appreciate the comment!
GOLD!
Thank you so much!
Great video! What's the name of the font you are using in your terminal?
Thank you for watching! The font is titled "CaskaydiaCove Nerd Font," which is just Cascadia Code with icons added, such as the Ubuntu and git logos.
am just gonna build a budgeter PC than yours for ML this weekend with 5900x + 4060ti 16GB ( not a good card but enough VRAM .. ) will go through your video and follow the steps to setup everything hopefully all go as smooth as you did ! Thanks dude!
Thanks for watching and good luck with your build! I think for my next server build I want to use GPUs with more VRAM, but 16 GB should serve you fine for a budget build
@@sourishk07 do you think those pro card such as A4000 or higher is really necessary for casual ML given their price tags?
@@raze0ver No, probably not. Since those cards are originally targeted at enterprise, they're overpriced. What I should've done is gone for a used 3090 because that's the best bang for your buck when it comes to VRAM or a 4090 if you can afford it.
Thanks for this! I see you went with 96 GB system RAM and a 4080 with 16 GB VRAM. Curious whether the 16 vs 24 GB VRAM (e.g. in 4090) could make a difference for AI/ML, and especially LLM, apps? I realize a 4090 would have set you back another extra $1000 though. And is more system RAM helpful, what I'm reading is that GPU VRAM is more important.
Thanks for the question! Yes, VRAM is king when it comes to ML/AI. Always prioritize VRAM. More system memory will never hurt, especially with massive datasets, but I didn't want to elect for the 4090 because of its price tag. However, on FB marketplace, I've seen RTX 3090's with 24 GB of VRAM for as low as $500, which was an option I should've considered while I was choosing my parts.
immagine two of them with nvlink and the cracked version of P2P. Too bad you didn't see it earlier, it would have been a great combo.😢 @@sourishk07
@@sourishk07 RTX 3090's is a MONSTER ! fp 16 models loaded with 32k context running at 60 tokens are the future :D
Great video :)
Hi, what is your experience with this rig? Is it not a problem for the temperature that the case is so tight?
The temperature has not been an issue with the same case size
Great video, I never got the Continue extension working in code-server, Is there a step that I may have missed ?
Thanks for watching! And regarding the Continue extension, what is the issue you're running into?
@@sourishk07 thank you for offering support 🙂I have followed exactly your steps , however I don't get Continue text zone to ask question , not even the drop list to choose the LLM or setup. I tried Continue Release and Pre-release but both did not work. I the fact that I have Ubuntu Server running as a VM under Proxmox with GPU passthrough could have an impact ?
I don't believe the virtualization should affect anything. When you go to install the Continue extension, what version are you seeing? Is it v0.8.25?
I've tried twice to post help with this, but the youtube does not like me adding a url. Anyway, I found I needed to run code-server under HTTPS in order for Continue to run. If you open code-server under HTTP it will issue an error (lower right) that certain webviews, clipboard and other features may not operate as expected. This affects Continue. You can find the fix by searching for: Full tutorial on setting up code-server using SSL - Linux. This uses Tailscale which Mr. Kundu has already recommended.
is this system good for inference? Llama 70b will run on this? I wonder whether RAM really compensates for the VRAM
Hello! That's a good question. Unfortunately, 70b models struggle to run. Llama 13b works pretty well. I think for my next server, I definitely want to prioritize more VRAM
I'm not sure if I like the idea of an AIO or water cooling in a server context. If it springs a leak I think you're a lot less likely to be regularly maintenancing or keeping an eye on a server that should be definition be out of sight.
I'd also argue that the choice in CPU is kind of weird; I would personally have preferred to step down on the CPU to something like a 13600K on for a good sale or a 5900X personally; they're plenty fast for ML tasks which are predominantly GPU bound but you could have thrown the extra money from the CPU (and the cooler!) into a stronger GPU. The exact price difference depends on the context, but I could see the difference being enough to do something a bit different.
I also think that an RTX 4080 Super is a really weird choice of GPU. It sounds kind of reasonable if you're just taking a quick glance at new GPUs, the price to performance ratio is wack. It's in this weird territory where it's priced at a premium price but doesn't have 24GB of VRAM; I would almost say if you're spending that kind of money you may as well have gone for a 4090 if you need Lovelace specific features like lower precision compute or something. Otherwise, I'd argue that a used 3090 would have made significantly more sense, and you could possibly have gotten two of them if you'd minmaxxed your build, and a system with 48GB of VRAM would absolutely leave you with a lot more options than a system with 16GB. You could have power limited them, too, if that was a concern.
If you were really willing to go nuts, in a headless server I've seen MI100s go for pretty decent prices, and if you're doing "real" ML work where you're writing the scripts yourself ROCm isn't that bad on supported hardware nowadays, and that'd give you 32GB of VRAM (HBM, no less) in a single GPU, which isn't bad at all.
Personally I went with an RTX 4000 SFF due to power efficiency concerns, though.
Thank you so much for all of that feedback! Honestly, I agree with all of it, not to mention a couple other people also have commented similar things.
But in my specific use case, my "server" is right next to my desk so maintenance should be pretty easy. Not to mention that I've really never really had any issues with AIOs for the 7 years I've been using them. Sure, a leak is possible but I guess I'm willing to take that risk.
I think I might need to potentially switch this computer to be my main video editing computer and convert my current computer be the server because it has two PCIE slots.
This was my first time building a computer from scratch solely for ML so I appreciate the recommendations!
With this as a secondary machine, I wonder what his main gaming machine is!
LOL you'll be surprised at this: my main gaming machine is an Intel 12700K and a 3080 12 GB. ML comes before gaming 🙏
As an embedded engineer I using 'continue' extension directly with my openai api, especially gpt4-turbo for auto-completion. Seems my knowledge not enough for this world..😟
Hello from Kyiv💙💛
Hello to you in Kyiv! I completely understand the feeling. With the field of ML/AI changing at such rapid paces, it's hard sometimes to keep up! I struggle with this often too
Could have started the video with the budget you were targeting. When recommending systems to other people, knowing how much the person can spend can heavily dictate the parts they can choose.
Here are some questions I've thought of while looking at the video. Why choose a 4080 over a 3090? Why choose a gaming motherboard or one that is a MITX formfactor? Why choose a "K" SKU for a production focused workload? There's missed commentary there.
I know that you have tagged some of your other videos but it could have been better to point out that you already have a NAS tutorial. Linking that video with the introduction of the 1TB SSD would have been helpful.
And finally why is the audio not synced up with the video? It's very jarring when that happens. Other than that it was cool to see the various programs that you can use. However I feel that the latter part feels tacked because it's hard to gauge how the hardware you chose has an affect on the software you chose to showcase.
Wow, thank you so much for your in-depth feedback! I sincerely appreciate you watching the video and sharing your thoughts. I apologize that the video didn't initially clarify some of the hardware choices and budget considerations. In retrospect, you're absolutely right, and I'll ensure to include such details in future content.
I chose the 4080 Super because it has the newest architecture, along with the fact that I was able to get it at a discount. The extra VRAM from the 3090 would've helped with larger models like LLMs and Stable diffusion, but for a lot of my personal projects such as training a simple RL agent or even some work with computer graphics, the extra performance of the 4080 Super will serve me better. Again, something I should've added to the video.
For the "K" SKU, I got the CPU on sale at Best Buy for about $120 off and the motherboard has an "AI overclocking" feature, which I thought would be kinda on brand with the video lol. I didn't really get a chance to touch upon it in the video or even benchmark any potential performance gains the feature might've gained me. Regarding the SFF build, I chose the form factor just because I have a pretty small apartment and I don't have much space. These are things I'm sure the viewers of this video might've been interested to hear about, and I appreciate you inquiring about them.
I also agree with your point about my NAS video! I'll keep that in mind the next time I mention a previous video of mine.
And regarding the audio, everything seems fine on my end? I've played the video multiple times on my desktop, phone, and iPad. Hopefully, it was just a one-off issue. Also, I suppose the software I installed isn't really too dependent on this specific hardware, but rather its the suite of tools I would install on any machine where I plan on doing ML projects.
Thank you once again for such constructive feedback. I'm curious, what topics or details would you like to see in future videos? Your input helps me create more tailored and informative content.
what is the total cost of hardware?
I forgot to mention this in the video, but the final cost was $2.8k pre-tax. Check out the link in the description for a Google Sheets for a complete price breakdown.
It's kind of not typical PC building targets something other than gaming.
Yes you’re definitely right! The motherboard was definitely meant for only gamers 😂
I didnt hear what the total cost was
Thanks for the comment. While focusing on the small details of the video, I completely forgot some of the important information haha. The cost pre-tax was $2.8k although components like the motherboard do not have to be as expensive as what I paid. I was interested in the AI overclocking feature, but never got around to properly benchmarking it. Anyways, I've updated the description to include a Google Sheets with a complete cost breakdown.
@@sourishk07 thanks! This was a very good video. Thanks
Greate video, truly loved it ❤,but you should hide your id 😭😭
Thanks for watching! And do you mean my email id?
@@sourishk07 sorry i meant to type ip, but I think it's a local ip so no worries ❤️
@@sourishk07 thanks for your high quality content.
I appreciate it! And yeah, all the IPs in the video were my Tailscale IPs which are only accessible to me, so unless my Tailscale account gets hacked, I have nothing to worry about.
If it doesn't have 4 H100s that i cannot afford i don't want it 😂
LOL maybe if I get a sponsorship, it’ll happen 😂😂😂 That’s always been a dream of mine
@@sourishk07 i subbed to see that one day 🙌
Haha I appreciate it! Looking forward to sharing that video with you eventually
who builds a ML system with 4080? 16GB is actually not enough! Either go 4090 or 3090
Yeah you’re right. Thanks for the comment! Next time I build a server, I’ll keep this in mind!
Has anyone built this yet based on his recommendation?
Hi! Regarding the hardware side of things, I didn’t really mean to recommend these specific set of parts. I just wanted to share my experience!
However, the software are tools I definitely use on a day to day basis and cannot recommend enough!
I built a similar rig, but with a Silverstone RM44 rack-mount case and a Noctua NH-D12L with an extra fan for cooling instead of the water unit. Fitting the GPU in the case required a 90 Degree Angled Extension Cable from MODDIY (type B). I used at ASUS ProArt Z790 motherboard. All of the software recommendations were great.
Why do you move your head so much
this is por**graphy
LMAO