AI Networking is CRAZY!! (but is it fast enough?)
Вставка
- Опубліковано 30 тра 2024
- Learn more from Juniper Networks on how the network will save the AI data center: juniper.net/AIwars
Dive into the heart of the AI networking battleground where the demand for speed and efficiency tests the limits of technology. This video unravels the complexities of training AI models, highlighting the network bottlenecks that slow down progress and how innovations like RDMA, RoCE, ECN, and PFC are pivotal in the quest for seamless AI development. We explore the clash between Ethernet and Infiniband, dissecting their strengths and weaknesses in handling the massive data and low-latency requirements of AI training.
🔥🔥Join the NetworkChuck Academy!: ntck.co/NCAcademy
**Sponsored by Juniper
SUPPORT NETWORKCHUCK
---------------------------------------------------
➡️NetworkChuck membership: ntck.co/Premium
☕☕ COFFEE and MERCH: ntck.co/coffee
Check out my new channel: ntck.co/ncclips
🆘🆘NEED HELP?? Join the Discord Server: / discord
STUDY WITH ME on Twitch: bit.ly/nc_twitch
READY TO LEARN??
---------------------------------------------------
-Learn Python: bit.ly/3rzZjzz
-Get your CCNA: bit.ly/nc-ccna
FOLLOW ME EVERYWHERE
---------------------------------------------------
Instagram: / networkchuck
Twitter: / networkchuck
Facebook: / networkchuck
Join the Discord server: bit.ly/nc-discord
AFFILIATES & REFERRALS
---------------------------------------------------
(GEAR I USE...STUFF I RECOMMEND)
My network gear: geni.us/L6wyIUj
Amazon Affiliate Store: www.amazon.com/shop/networkchuck
Buy a Raspberry Pi: geni.us/aBeqAL
Do you want to know how I draw on the screen?? Go to ntck.co/EpicPen and use code NetworkChuck to get 20% off!!
fast and reliable unifi in the cloud: hostifi.com/?via=chuck
AI network challenges
High-performance computing for AI
Ethernet vs. Infiniband comparison
Optimizing AI data centers
RDMA technology explained
RoCE for AI networking
AI model training efficiency
Reducing AI network latency
Tail latency in AI development
GPU server networking for AI
High-speed networking for AI
AI data center infrastructure
Infiniband benefits for AI
Ethernet advancements in AI
Network bottlenecks in AI training
AI workload network solutions
High bandwidth requirements for AI
AI network traffic optimization
Low-latency networking for AI
Network protocols for AI
AI network congestion solutions
ECN and PFC in AI networking
Future of AI networking technologies
Building scalable AI networks
#ainetworking #infiniband #rdma - Наука та технологія
Learn more from Juniper Networks on how the network will save the AI data center: juniper.net/AIwars
Dive into the heart of the AI networking battleground where the demand for speed and efficiency tests the limits of technology. This video unravels the complexities of training AI models, highlighting the network bottlenecks that slow down progress and how innovations like RDMA, RoCE, ECN, and PFC are pivotal in the quest for seamless AI development. We explore the clash between Ethernet and Infiniband, dissecting their strengths and weaknesses in handling the massive data and low-latency requirements of AI training.
🔥🔥Join the NetworkChuck Academy!: ntck.co/NCAcademy
**Sponsored by Juniper
i will be the networkjustin of albania
/
Juniper is being acquired my HPE
I would be curious if they have *load-balancing* also available or if that is considered 'not available' due to the fact of so many GPU's running in parallel and every line is already hyper-saturated. though *ROCERDMA* [sounds like another DMA for net], *Explicit Congestion Notification* and *Priority Flow Control* are interesting.
Hope you have a great day & Safe Travels!
6:40 Roblox💥💯💯
I wonder what the security implications of RDMA is since it is bypassing the kernal
That was my question to. How can I ensure security to earn money when today's skills are obsolete?
Usually RDMA is used in an isolated environment
RDMA is secure in the sense that it's typically implemented on a closed off network that is completely separate from other portions of the network that a supercomputer might be using (I work with supercomputers so that is what I think of when it comes to RDMA). Even with no encryption across the wire, there are keys in place which allow certain NICs and devices to talk to each other so it's a zero-trust configuration. You almost have to treat the high-speed network fabric like it's the bus moving data from CPU0 to CPU1. We have implemented RDMA across two of the supercomputers I work on and the performance is insane, but there still is a potential security risk that needs to be accepted even though the RDMA network is very closed off.
@@lubu42im studying computer science , i wish to work with supercomputers or atleast to work as internet eng. but im not sure that my university is teaching me enough about it , can i get some advice how to get job as network eng or to work with supercomputer , anything wich isnt web developing ...
one overflow exploit away from extracting the memory and the secrets from the target system and nothing to detect / alert it … NOTHING is completely isolated any more
Anyone old enough to remember running Doom via IPX/SPX (Novell Netware) on a LAN? Remember when TCP/IP was introduced into the stack and slowing the post-Doom games down?
Pepperidge Farm remembers.
7:39 Someone is behind you man! Just Run 😀
I saw that too!
I got scared lmfao
What or who was that? Pause at your own risk. That will give you nightmares. I hope you’re still alive!
Who is that behind you
we need an answer @NetworkChuck who was that weird looking person/creature behind you?
Having worked with some of the networking and hardware setups for AI servers... The networking and liquid cooling I got to see there were awesome. Just about anything you can get your hands on is outpaced 10 fold or more in an AI datacenter.
It's nothing new. Just faster and bigger. But yeah
which site code?
Yep until the tree hugging greenies go on about power usage for cards and data centre cooling
Awesome video Chuck, you always do an amazing job breaking down explanations. Been subscribed for a while, and I have always noticed very good quality videos. Thank you!
Chuck, your tech videos are consistently awesome and helpful. Keep up the great work! Thanks for sharing your knowledge!
This is the best video I've seen on your channel and I've been watching for years. Your knowledge shone through at an atmospheric level. Please do more like this every so often as the AI landscape broadens!!
sounds like this is going to introduce a severe set of attack vectors, but none the less its pretty awesome
the AI ‘revolution’ is following similar lack of security consideration as when everyone started to migrate from legacy to cloud…. the legacy systems were not designed for the level of internet exposure -- the Data ingested process for AI is not designed to maintain the security and permissions from the sources the data comes from
You run the AI training on a dedicated private network. If people are breaking into your private, closed off network, then you have a bigger set of problems to worry about.
@@cheema33 its a new network stack using rdma, it probably won't even be able to communicate with ipv4/6. its going to have a bunch of vulnerabilities and be underdeveloped. Will it get better yes but the main security vector is that rdma. With that said i still think its freaking cool.
We've just done some testing at work with the guys from Juniper, those switches are beasts
Love it!!!!!!! Rdma SInce 2019!!!!!!!!!!!!!
I like that you take your time to make absolutely sure the video is on point before you publish.
Keep Going Chuck !
Where?
This is a great video. Its been a while since i have felt excited about networking and technology. but this video has me super interested about AI and networking. this is the first time I have seen network specific design for AI.
I'm not sure if I should be scared or excited
This whole video must be the most sneaky advertisement I have ever seen in my life.
the jump cuts on the other hand...
It’s not
Not an advert or not sneaky?@@SovietCustard
Fun to see this video right now, as I am planning a big cluster at work. The main argument for me for Ethernet is cost.
12:26 Chuck, this is precisely my beef with Ethernet. It did not "think" about the future, until a contender arrived!
If Ethernet could do improvements to our current networking tech, why it did not? Precisely because it did not care...until it saw it's survival on the line.
We could and should have a better network than what we currently use now, not just for AI.
It is better to have both so they push each other to advance
True thats how Adobe has opperated for decades.
That doesn't really make sense though? There's literally nothing to gain from "holding back progress". What is "it" in this context? Companies implementing network protocols? The people developing them? They all gain from having faster and more reliable products than the copetition.
@@DJTimeLockThats true, however it seems that the standard itself had nothing to compete against, hence the lack of new ethernet protocols in the past decades
@@pikpockett necessity creates innovation. That's very true. But to me it seems like it wasn't a necessity until the AI boom happend. As much as his analogy about a ferrari on blocked highway it goes both ways. What's the point in creating a 12 lane highway if 3 lanes are sufficient.
Awesome video chuck
Hey Chuck, fellow network engineer here (mostly cloud infrastructure though). First, I loved the video. This helped a ton to clear up my understanding at a high level of InfiniBand. Just some creative criticism, this definitely went much deeper technically than your typical content, so maybe it would be a good idea to state that up front or maybe even break content like this into a separate channel. I would love it if you made more deeply technical content, even beyond this level. You have a great gift for explaining these sorts of things in terms that people can relate to while still keeping it entertaining! Keep up the GREAT work!
I didn't even realize it was that deep technically. I guess it just makes sense.
separate channel, yes more advance would be the best solution
2:50 Before we air cooled data centers, we liquid cooled them.
Wow this was cool 😀 I like both of them. A new network system will be needed for Quantum Computers soon as well. Quantum teleportation where the state can not change. Interesting time to be alive . Thank you for this video😊
Thank You for helping us to know more details of this new networking technology.....!!! and by the way I support both..!!
I remember working with Infiniband in 2006. Its cool to see that they're still going.
Wow really amazing topic
Thanks so much Networkchuck
What's important is working with what is adopted. As you mentioned ethernet is worldwide and the cheaper option to adopt which means that is what tech's are going to be hired to handle as that is what companies will prefer. This is where tech's can be paid a lot more to apply for positions that will soon open up in connection to supporting infini-band. As IT specialists, we work with companies and their current infrastructure, if the advancement of networking technology is inevitable, we'll all be forced to move along with it!
THANK YOU FOR POSTING CHUCK!!!!!!
Aye Chuck team infi but without the @ and we saw the data center vid! 🎉🎉🎉 thanks for the support (open sourced) infiniband, just like containers/‘Docker; appreciate IT Chuck!
I love your videos! Learning so much, and getting better and better with my homeserver!
Keep it up! and give me more informationen xD!
I'm a long time (now retired CCIE #1937, R/S Emeritus) network engineer and I'm definitely on Team Ethernet.
I have no problems with A"i" (really just the same machine learning we've had forced upon us for 10 years) being slowed to a crawl.
Same. Infact, I am in *full support* of dismantling companies like Stability, Wombo, and Open. I have no problem if RatGPT is shut down, along with all of the other scummy A.I. applications.
hey chuck been a sub for a while always really like what you're doing, you and many others are also inspiring people like me to also maybe go down the same road as what you guys are doing work for myself and share knowledge with people that can make use of your skills and passion instead of always fighting with cooks at the normal workplace.. thanks again.
you're talking about infiniband and rdma being new.. they're ooold :D For instance Isilon used Infiniband for their backend network for over 10 years, they even have migrated away from it to ethernet already
Since we are exchanging data directly btw the memories of the two servers without going through the network stacks or layers, it also means that a large bandwidth will be required since there are no data formatting techniques such as compression and encoding.
1:23 Never thought I would hear Chuck say he wants to become a battery😂
Yes, more data centers networking and AI please
That thumbnail gets the attention!!
Insightful video. Thanks!
really good content again Chuck. I might have an easier time picking between the two if I understood how firewalls and QoS work in the Infinaband space. Or do they have an alternate technology?
Thanks awesome video! More of that topic please
I love your amazing work and keep making some hardware exploit videos they are really helpful.
1:53-1:54 there's a cool moment...
you know what am i talking about :0
I get the direct communication with gpu but the last thing you said chuck
first : Why would they reinvent the wheel (Standard Internet Protocols )?
second : the rest of the internet is still using tcpip and IEEE , so they either have to force everyone else or create conversion wrapper ?
whats the point ?
I don't understand why ?
Hey Chuck, wouldn't RDMA require the data to be transmitted in an unencrypted form? Since the CPU is not processing it, I don't think any sort of encryption standards would be plausible for such networking, and hence it leaves the whole network more vulnerable than ever before.
#1 This is usually local traffic. In the example Chuck mentioned most scenarios only involved switching and not a lot of routing.
#2 You can very easily do line speed encryption/decryption in the NIC.
Love your chan. Your videos are fun to watch and you are a relatable host. Thanks for all the great content!
Infiniband is not new. Ive had infiniband in one of my production clusters for neigh on 20yrs!
Infiniband doesn't have to be expensive. I picked up 2xQlogic 18port QDR switches for $15 on ebay(still dont know how it was possible)
I was able to pick up cards/cables for 4 servers for in my home lab rack for under $200 .
I run subnet manager on a couple hosts since these switches cant run it.
Love you content men keep it up
I work for MinIO and I want to refactor how the world sees and uses its data 🤩 🎉 Thanks for the inspiration :3
Ethernet for sure. Can imagine changing the for on global scale🤯
at 11:30 you talk about prioritising some packet over other, doesn't that kill the net neutrality principal? or am i in the left field and there is no relation?
Fantastic cutting edge content Chuck...
Based on your description of how infiniband works it seems like there being an internet connected device which will likely be running a full operating system controlling direct memory access between two computers could have some potential security implications. For instance hypothetically if a malicious actor were to gain access to a switch handling the connection between the two servers it could be possible to inject malicious instructions directly into the receiving servers memory allowing an attacker to potentially gain some level of control over the server having its RAM written to.
The "Animaniacs" part is a classic...😅😂 oh and yeah the AI slow down, good job Juniper..
There are numerous initiatives that aim to reduce these issues. One of them is CXL interconnect, which takes RDMA concept like a step further and throws networking to top of the rack only. Since CXL allows P2P rootless communication, you can eliminate the "server" part from the server, and leave the resources only - RAM, accelerators, GPUs, storage, RAM (yes, as an actual device - there are already CXL connected RAM carrier boards) - and if you want - an actual CPU. This enables for example RAM sharing across whole CXL domain, and any network capacity added to the system benefits all of the devices equally, with basically fixed latency. You could even export and virtualize whole parts of CXL tree over normal network to group systems with identical dataset or mix and match for power envelope control.
Of course it's not panacea - it's restricted to rack scale only solution, so you need same infrastructure anyway outside of it - just less of it, it's power hungry (current CXL switches eat up kilowatts of power), it's insanely expensive, and requires rewriting and validating most of current workload managers, which comes with its own challenges, kind of doesn't fit in normal systems, the hardware is quite fragile physically, and it's not something you can build over time - you need to set it all up in one go, which kind of raises difficulty of adopting it and raises entry bar for development, not to mention requires expert knowledge form much wider set of topics from the developers than otherwise.
We will most likely never see much from this in "normal" datacenter environment for many years, as this tech is heavily skewed toward supercomputer systems, but the innovation is always welcome.
What antivirus/security software do you use to protect your personal computer or your other devices?
Love your channel man
1:53 was crazy
I feel you! It's easy to get carried away. Just set a budget and stick to it, right💸
What is the name / link to the song playing in the begining of the video uptil 2:00 Anyone ?
Super cool. TCP or holograpic udp.
The big jump that is comming is photonic matrix processing chips.
And I agree, we prbably do not need a need a new protocol.
~
Though,
I do not think there is a mathematical posibility of exctracting
certainy from our current network topology.
We want more videos like this! They are awesome and full or information!
I'm not a network engineer by any means. My initial question would be how far can ethernet go compared to infiniband. infiniband is starting with these newer features for connectivity and such, but ethernet had to catch up to the change. So how much more can ethernet grow vs infiniband, is there a limit to either in the sense of growth and improvement?
At this point , Ethernet is a known quantity. I see no need to reinvent the wheel .
Based on the models I train at home: For RTX 3090/4090 cards, you'll hit VRAM limits before bandwidth limits for the size models you're working on. 10Gbps is more than enough. You don't need anything exotic. Also it's fine to put your GPUs in multiple cheaper computers; you don't need a single super expensive motherboard/CPU.
AI training is usually done over A100 or H100 and you sure will hit the bandwith issue. A mixture of experts of 8x220B parameters models is not your everyday at home. Training requires communicating the gradient and optimizer which is about 3 times the model. This is several GBs at each iteration, while the GPUs have the model sharded and they just need to execute one layer per iteration which is way faster.
@@joseleperez8742 Not sure if I care how you think its "usually" done. I see lots of papers presenting cutting edge work trained on 3090/4090 cards, sometimes just one. For home labs it makes no sense to purchase an A100 or H100 for many reasons.
omg, why am I barely hearing about this?! OK so word to the wise... if you're worried about AI replacing your networking job, start learning Infiniband and all associated technologies. Thank you Network Chuck and Juniper for this video!!!!!!
The lowkey roast at 6:26 😂
7:39 - Person in background scared me. lol
I can't sleep anymore...
Interesting video but very high level. Regarding the high power consumption/heat dissipation point on GPUs a topic I’d like to see a well researched video about is the possible avenues to improve code efficiency where at a given load an LLM could have more productivity, as that would counteract the FLOPs/time needed and hence reduce power per output on the AI side
I like you beard. It looks cool man.
Hi, why are you using the old TCP/IP model ?
You're awesome man 😎
great content !
does it know to slow down if the internet WAN link is like 500 Mbps? Or still lots of QoS need to be implemented ?
Great thumbnail
It has some "I have no mouth and I must scream" vibes :)
Can QoS be used for AI traffic like PFC? I guess PFC runs over ECN and QoS uses the TCP/IP setup.
I’m team infiniband for these reasons, Ethernet is getting hard to get up to speed, I love Ethernet but eventually we are going to need speed and reliability that Ethernet cannot keep up with without major complexity added. The improvements mentioned in the video are great but isn’t that a lot more complex then just figuring out infiniband and future proofing it then having to upgrade every few years? Anyways, I had another reason but I forgot it sooooooooo. Just to be clear, I love Ethernet and want it to work well but it just isn’t fast enough and is getting bottlenecked.
I would say ai slowing down is a good thing. Better yet would be some sort of impenetrable barrier that makes larger ai models (and smarter than human models) impractical or impossible
You're forgetting one crucial thing, sustainable power consumption at scale.
This is all nice and everything, but until we can find better innovations in sustainable energy management systems to scale, it doesn't matter what type of software or hardware you have.
This dude makes me want to learn
TCP/IP it is for me, as you said, that is well known, and easy to find help. But that will need a separated network to secure it.
Hmmm interesting conversation! I love Ethernet but I am not sure if it will last, the one thing good thing about Ethernet is cost, it is fairly cheap, great video Chuck!
Just won on 4RA today, bro. Feels great! 🤑💪
We need speed of thought network infrastructure...
Speed of thought chuck.
Speed of thought.
BRO NOWAY I WAS JUST THINKING ABOUT YOU AND THEN BOOM A NOTIFICATION
RDMA is also used in some Clustering stuff for databases or Storage systems
Damn chick in black in back was creepin 07:38
I didn't understand a word you said, but I really like your videos.
I used to work for a company which built a 76TBit/s ethernet network
Is PFC kind of upgraded sliding window?
You gotta really break this problem down. It's specific parts of the networking that are standing in the way of having better bandwidth. There are also things that are more important at times to allow hardware and software to be able to effectively complete their operations efficiently. Latency is one factor that I have seen in software that can cause some software to behave different than it was intended to. Latency also drives up software costs because measures to contend with the varying levels of latency need to be put in place.
I think we should be going hard into fully optical communications circuits. I mean, that everything is done via photons until the very last second. Decreasing lacency and allowing the bandiwdth to be even less constrained to the fabric/medium it was transmitted through.
I really like the thumbnail
Thanks for that absolutely cursed AI generated preview.
help along skynet, sounds good
@networkchuck , as long as videos are not being re-uploaded to UA-cam, would you be ok with others watching your videos on twitch livestreams?
awesome video as always - I am team ethernet as infiband is for a small niche
How do you stay in shape with all the things you do!
I'm on team ethernet on this one. 🌐 I think keeping with the standard that we've already pretty much decided on is best as long as it can keep up with other future hardware. 💻
I'm with Ethernet.
Sticking with the traffic analogy: we're investing billions in trying to make self-driving cars that function within our existing infrastructure. It's taking a lot of effort, but it will ultimately require less effort, time, and money than to rip out all our roads and streets in favor of a self-driving-specific infrastructure.
Using what we have and building within those constraints will require less (if any) overhaul AND the improvements to the stacks will only make things better for everyone, since everyone is already using the base tech.
Your freaking amazing 💙
Love your vids
As someone who has extensive knowledge in installation/testing/troubleshooting network equipment/ethernet etc. I welcome the endless work muahahahaha
4ra never just sticks to betting their events are next level always exceeding the norm 🎉🎈