(Edit: Errata below.) I think this is my longest video yet! Going back to weekly schedule, I can't produce videos this long every week though :) Discord: discord.gg/AgafFBQdsc Patreon: www.patreon.com/DrWaku Errata - Google won the Oracle vs Google case, not Oracle - "OpenAPI" should be "OneAPI" (re: Intel) - Oracle may have implemented IBM's API, not the other way around - Intel did not have an open architecture, AMD was granted a license - CUDA compiler now uses LLVM as well!
You should sticky the errata. In any case, this is my first time watching your content, brilliant work and excellent presentation. Looking forward to browse through your back library.
Watched you from the beginning. You manage to maintain a very very high quality channel with pretty unique content without resorting to clickbait/hype or other algorithmic bullshit.
Google vs. Oracle: The surpreme court ruled in favour of Google. AFAIK they didn't have to pay tons of money. And it wasn't about "Hey, that code looks like ours!" it was a "Hey the reimplemented our API! How dare they!"
Yeah I got the outcome of this ruling wrong. And yeah it was about API reimplementation originally, but the sticking point ended up being a tiny function if I remember correctly. Maybe it didn't have an impact on the final final ruling.
This was clickbait. Nvidia is in no trouble. AMD is certainly trying to catch up but RocM is far behind and in 3-5 years there will be some market share lost to Cerebas or Groq chips but like you said Nvidia is benefitting from their existing relationships and proven quality. The title made it seem like Nvidia was in trouble but you basically spent 30 minutes discussing how far ahead they are.
@@pruff3 the title doesn’t claim it’s losing its lead only losing its monopoly and I think he made a good case that while they will still be the clearly dominant leader though not maintain their monopoly.
@@Greg-xi8yx yeah that's true I guess it's not really clickbait but the case for Nvidia losing its lead (the thumbnail) is misleading (pun intended) and also even the title is like saying "I'm dying" (yes, we all are, slowly, here's the evidence for why eventually we will be dead...in the future)
People are responsible for how they take different perspectives. Dr Waku has always had a high quality video in many aspects. In any regard, if you believe differently create your own video. Different perspectives and different takes is what makes knowledge possible. Calling things as clickbait is just a cheap criticism.
Each time Apple switched processor architectures on the McIntosh, they used Rosetta. The first time switching from Motorola 680X0 architecture 2 power PC, then they used Rosetta again when they switched from our PC to X 86. The switch from X 86 to ARM- 64 was just the latest iteration of Rosetta. I don’t think it was ever possible to have a stack of two or three Rosetta translators on Apple hardware. Doing so would allow you to run software for the original McIntosh bye emulating the Motorola 68,000 on the emulated power PC on the emulated X 86 running on ARM 64
The world would be a better place if most "intellectual property" were recognized as shared by all rather than imposing artificial scarcity to control and subjugate all beneath the will of greed and domination.
Thank you for this exhaustive course on CUDA and its potential alternatives. I am a big investor in NVIDIA and you got me worried. I don't know where you found all this information, but it is great work.
This was an interesting take on situation. It reminds me of 2008, when I was buying my first computer. I've gotten an AMD 4850 GPU. I was super happy. It was much better technologically than nvidia's 8800gtx, their first CUDA GPU. Still with software, even though older, 8800 series was better due to CUDA. I figured, I just need to give AMD a bit more time to catch up. With such great hardware, how can they fail? By 2011 it was obvious to me that AMD cannot possibly come close to nvidia software, no matter how good their GPU hardware is. Back then I got myself nvida gtx 460 and I haven't been back to AMD since. I have no reason to think that now, 18 years later, AMD has gotten any wiser. It would be nice to have a real competitor, but I'm not holding my breath. Strictly looking at their core 3d GPU technology they both support today: DLSS vs AMD's FSR. AMD simply cannot compete. If they can't make their GPUs perform their main function any better, what chance do they have in more complex problems.
This video was a lot more interesting than I thought it would be before I clicked on it I almost thought it was going to be AI voice slop too but i saw there was a face on the video lol
Good information and video. The next one is also engaging. However, the sound and editing greatly reduce the positive impact on the audience. I won’t be able to finish the video on my end due to the poor sound quality. I hope the adjustments can be made in future.
@@DrWaku I hope that helps too. I do feel and appreciate a high quality mic in your videos. However, a sensitive recording device combined with the 2023 iPhone Pro, which delivers high-quality sound (especially via AirPods), might potentially amplify every mistake multifold. First-world problems, poor us, eh. Would you consider trying a one-take video next time? It would be nice to experience your personality and mind in a more natural and unedited way. On a side note, if I may ask, what’s the story behind your hat and gloves?
I didn't do a one take video but I did do an interview so it's a slightly different format, if you want to check out the latest. The gloves are medical. You can check out the disability playlist for more info.
Medical conditions. My hands are quite sore if I don't wear them. Fibromyalgia and rheumatoid arthritis. I made a couple of videos on my disability etc if you want to check it out
@DrWaku How dare you show evidence of a disability on camera! Take the gloves off and suffer while filming lest you remind the audience that such medical mallodies exist. /s obviously
I have to say, although the video has great contents, constantly appearing subtitles at the bottom are incredeibly annoying. I would reduce their quantity by 5 times, if I were you.
Thanks. Yeah, usually the subtitles are a lot less frequent but this time my editor just put them everywhere. I'll ask him to decrease the frequency next time.
you need to start using a pop shield for your mic and you need to do something against the overdrive distortions. your audio isn't the best it could be and not always pleasant to listen to. (Writing this at 10:54 . The content was interesting so far.)
Didn't the google Java case end up legalizing API copyrights? Am I imagining that? The idea is Oracle could own the API/ABI itself as IP and you cannot build a cleanroom implementation.
Here is another moat killer perspective, with the advancement of LLM's and Copilot etc, its getting easier and easier to write code from scratch for building a new stack etc. + open source of AMD, that seems like things can really speed up for AMD with a combination like that. That being said u might wana check out George Hotz who was trying to create a tinyML using AMD and gave up on it and switched to Nvidia either. Other then the Software Moat with CUDA their software itself is less buggy then that of AMD as well, also the fact that programmers are getting lazier and lazier with copilot and LLMs will only make the code more buggier imho
AMD drivers and support for Linux have also been poor at times. That's one good thing about the AI boom, forces these companies to treat Linux as a first-class citizen.
Well it’s not coming to an end but upgrade cycles are disrupted. If gpt5 is any good then new upgrade cycle will probably be bought. Until then the upgrade cycle will continue through smaller companies upgrading data centers and databases going into the near future
Nvidia's AI monopoly is coming to an end due to increasing competition from other tech companies developing advanced AI hardware and software. This shift is fostering innovation and providing more diverse options for AI solutions.
Some way into the video, I began to resent that I was quite enjoying what appeared to be mere clickbait. However, you skilfully justified the provocative thesis in the final moments - well done. That said, let the challengers bark as they may. As long as the visionary commander remains compos mentis and the coffers flush with cash, the Nvidia behemoth will continue its march, unfazed by their noise.
Haha thanks for the backhanded compliment. I didn't mean to be clickbait but I think a lot of people have strong feelings about this topic. Thanks for sticking it out.
Short story, monopoly is bad for consumers. A good competitive market is 3-5 major players. And none of those major players should have less than 10% each or more than 40% of the market share. Anyone who buys from a monopoly contributes to the problem, no buts about it.
GOD damn I have to give you so much respect for you intelligence! Thanks for your videos! But I cant give you an abo I hate your glasses OCD kicks so hard a shame. YOU are great. Keep going,.
dude! this was an awesome video. I as a linux user for 2 decades always hated Nvidia, u should have been the student at standford asking Eric Schmidt some tough questions on this he seems extremely confident on Nvidia did u watch that infamous leaked vid laughing at intel as a competition etc.? but i think u got it right, AMD is going to approach this with "open source" because they are way too behind... AMD is going to end up being the poor man's GPU for LLM's but by the time AMD catches up where will Nvidia be? These things are evolving quite quickly. Zuckerberg seems all in with Meta llama & Nvidia alliance.
As long as AI demands more and more compute, nVidia having the hardware advantage will ensure them a monopoly: in AI only the best matters, be it the LLM or the hardware running inference. Choosing second beast means you're at a disadvantage in a field where months are equal to 'old IT' years, and that equals death to your company. This video is a fantasy.
nvidia nearly lost it when microsoft chose directx.. direct x and opengl are the wrappers.. im not aware of a PC game that only runs on one brand or another. today with AI, nVidia has a way to speed things up. ATI, like most other hardware manufactures, went compliant and didn't compete with the 1 competitor they had... Cuda has been accelerating machine learning for 8 years, ATI/AMD hasn't shown much effort in competing. If anything, shame on ATI/AMD and replace the company leaders who don't innovate. Also, ARM processors will be replacing the PC processors we use today. and it's happening now. Now ATI will lose more market share thanks to new machine language processors, as microsoft is including in new laptops.. Basically 10 years ATI/AMD sat on there hands, and its time to replace leadership.
@@DrWaku Y'all are doing a great job but there is a strong reason I have never used tiktok and it has to do with things being hyper-aligned to attention algorithms. The firehose is disquieting for mature (tired) minds, in such a way it might be impossible to describe to anyone under 30.
I was captivated for the whole video, you speak clearly and with a very nice pace and build a good foundation before touching the more confusing parts of the topic (also thank you for using the companies logos every time you mention one…)
A lot of small, but important, mistakes: 1. Triton was supporting AMD in September 2023 and maybe even before. Also PyTorch uses Triton as a backend. 2. CUDA compiler is partly open source as is based on LLVM. Nvidia is a major LLVM contributor 3. The story with IBM and Oracle was the other way around 4. x86 and x86_64 are not open architectures 5. Google won vs Oracle
Thanks for your reply. I took the date of AMD support from when they edited the readme to add it. I figured that's when it was official. Did not know that the CUDA compiler is based on LLVM. I got the IBM story from an IBM employee, maybe they were biased. The rest I didn't check on, good catches thanks.
NV might be keeping some competitors out, but in case of AMD the thing holding them out is THEMSELVES, they have just been too sloppy and disregarding year after year for their sw ecosys and that way they never built value but destroyed their own footprint.
Agreed. AMD has always tried to do things with a lower budget and without an eye to long term sustainability, unfortunately. We'll see if that continues in the GPU division under Lisa Su.
AMD needed to avoid bankruptcy and become successful/profitable with their other products before they could fully focus resources on GPGPU. From the Xilinx acquisition and on, they have been investing very heavily. We'll see how smartly they can deploy, integrate, and scale ROI on those investments.
Intels thing is definitely not called "OpenAPI" because thats a standard for defining and documenting HTTP REST APIs. Intels thing seems to be called "oneAPI"
Nvidia GPUs are also built around floating-point registers, which are used to compute decimal values. While floating-point logic has gotten better in recent years, it remains more complex than integer logic. Simply put, it takes more wires and transistors to build a floating-point register than one made to process integers. More wires equals higher cost. More wires equals more resistance. More resistance means higher energy consumption and heat generation. AI Developers are re-engineering transformers and neural nets to make use of more efficient integer operations and eliminate matrix multiplication. This would allow AI Models to run on much faster, cheaper, and more energy efficient RISC-Based ARM Chips.
It was Oracle who wrote a simple API to pull stuff out of DB2. So the Oracle-IBM story is the other way around. There are t-shirts that said, Oracle had IBM talking to itself. Because using the API, you could set up one DB2 talking to another DB2 through Oracle, yet those two different DB2 databases were incapable of talking to themselves.
I wonder if each company implemented the other's API at some point. I got this story from someone who worked at IBM and who was proud of the work that IBM had done, lol. That said, DB2 is probably not very flexible or easy to use so I'm not surprised they made a t-shirt.
An excellent overview. AMD's HIP is a CUDA compatible-like API (function names are changed from 'cuda_' to 'hip_' for legal reasons) and this makes porting CUDA relatively easy with a script (hipify). Blender for example targets CUDA for NVIDIA GPUs and HIP for AMD GPUs which means 2/3rds of the code can be shared between them cutting down dev time. Yes HIP would be a generation behind but nobody is deploying code written with the very latest API version in mind because a) development takes time and b) there's no market share for it. So that's dumping some dirt into the moat. PyTorch abstracting code away from CUDA is a dump truck of dirt into the moat. Triton is the next big step which brings me to something very important.. You mentioned AMD and intel as competitors but left out NVIDIA's _biggest_ competitors. All of NVIDIA biggest customers, Google, Microsoft, Meta, Amazon, Tesla, have their own AI accelerators in various stages of design or production. Google's TPUs have been in production for years. Microsoft is onto their second generation of their MAIA chip. Tesla is training on their Dojo (D1) chips. And Amazon has their second generation Trainium and Inferentia chips available to rent on AWS. These all operate with the PyTorch framework and most have Triton support. NVIDIA doesn't have to worry about their competitors, they have to worry about their customers.
there's one important thing that's being over looked. The CUDA compiler isn't just converting higher level language to executable byte code. It's also optimizing the execution to maximize parallel processing, which is historically very hard. You can have great hardware, but it won't be faster if your compiler can't effectively parallelize the work. CUDA stack does a lot of fancy stuff to optimize the code. OpenCL is essentially dead and no one should use it going forward. I really want scale to succeed and 7 years is a long time to spend on a project. The better fix is to move neural networks forward so we don't need 20K H100 to train 1 trillion parameters. We need to understand what all those weights are doing and make models that are 10,000 smaller.
every compiler does fancy stuff to optimize the code. You think gcc doesn't vectorize your data to benefit from SIMD as well ? :p it's the programmer who needs first to not only accurately parallelize the code for maximum throughput , but also be careful to keep gpu threads coalescent in memory accesses. this is the hardest part , finding THE parallel algorithm.
@@emperorpalpatine6080 great point. GCC does a lot of fancy stuff and there are a ton of compiler options. I've never written a GPU shader compiler, but I do enjoy writing compilers in my free time. From my own experience, most developers have a poor understanding of how to parallelize the execution and often it isn't obvious. Even if you figure out the best way today, if the data changes over time it won't stay optimal. It reminds me of the saying "premature optimization is the root of all evil." I don't necessary agree with that statement for all situations, but I have seen it first hand. Hand rolled optimization is difficult to maintain as data and hardware evolves. Nvidia has done a good job of making CUDA handle a lot of that, but you still have to make sure you load your data efficiently
There are 300+ cuda libraries supporting all forms of scientific, engineering, and commercial applications. They cannot be placed or duplicated quickly by a new language or compiler.
It's interesting to see these big companies like Intel and Nvidia experience some troubles when they might have seemed untouchable in the recent past...
Vis a vis CUDA’s complexity for coders, how much would it take for Nvidia to make a custom AI that’s an expert at CUDA code creation? They certainly have both the financial and compute resources to throw at the problem, and it could remove one of the biggest hassles for users.
Entirely possible (see: AlphaTensor), but there’s little need for it. They already have out of the box easy to use libraries for most things you need (Cutlass, etc), so if you need to do something, you can generally find a kernel for it somewhere on the internet, meaning you don’t really need to be a CUDA master. The complexity of writing kernels is why I use CPUs of all things for deep learning (it requires different designs for the architecture, but you can get surprisingly close to parity between single GPU and a single CPU), as it’s a lot easier to do a fully optimal AVX kernel for your CPU versus a fully optimal CUDA kernel. I’m looking into Tenstorrent hardware in the future as they’re roughly the same programming model as CPU AVX kernels.
It seems a bit click-baity, controversy seeking, and conspiracy-founding. Nvidia has invested many man-decades of work (and billions of dollars) into defining a software technology that matches their hardware technology, all of which are about representation of ideas and speed. Why isn't their hardware design a moat? Nvidia doesn't owe any other company free access to its technology. Amd and others can design a better system from scratch and whoop whomever they please. So far, no other companies can run and pass all the benchmarks anyone has ever written, and beat everyone. In AI, general compute, graphics, etc. Why would they give the keys to their office to the competition?
How much? It would take magic, because there is no such thing as "AI". Despite all the marketing, there is currently no program that can replace or even meaningfully assist an expert in writing any sort of involved code. Large language models have so far failed and increasingly seem to be a dead end for that purpose (among many other purposes). AI is a meaningless buzzword. Unless you're a developer talking to investors, stop using it.
Jensen (nvidia ceo) said no programmers in a few years, so theyre working on stuff like that. But i think hes wrong and doesnt understand programming, LLMs can help, but they're not even close yet to actually replacing programmers, we need more scientific breakthroughs for it to happen. Maybe it will, but it seems very optimistic and hees just hyping up nvidia for profit
@@martinkrauser4029 I think you have to look at the trajectory, not the current status. To me there's a clear trend, and I don't see a bright line anywhere that would be an uncrossable threshold. Perhaps we need to distinguish between "programming" and "coding". I view the former as including system architecture and design, goal-setting (choosing which problems to solve) and the generally "creative" parts of the process, while coding is simply converting the output of those parts into executable code. While we're nowhere near having machines able to do the former, we're making rapid progress on the latter. Currently, I'd say the assessment depends on your definition of "meaningfully assist." The LLMs can't write complete systems or even error-free components, but they can certainly shoulder a lot of the gruntwork of code-cranking. They're even now useful an amplifier on an expert's ability to rapidly generate code that implements their design. We're nowhere near what's being hyped, but look at the progress from GPT3 to GPT4 o1 in what, 2 years? From my naive viewpoint, the progress is accelerating, if you look at the capabilities of 3 to 3.5 to 4 to 4o and now 4o1. (Of course there are similar developments in other systems too, notably Anthropic's Claude series.) I agree that "AI" is a bit of a misnomer, although I've come to question what "I" actually is in the first place. (Maybe we're all just LLMs with state memory, on-the-fly reinforcement learning and agency? 😉) But I do think we're already seeing tools that can significantly leverage a human's time, and that are being able to do so more and more effectively.
AMD being compatible with Intel is NOT because of an open standard - it is because AMD had something to LEVERAGE Intel into giving them a LICENSE. What they leveraged was NOT being Intel - when IBM wanted to start making PC's, it was the dominant player and had a policy of ONLY dealing with anything that has at least 2 suppliers. So, Intel and AMD came to an agreement that allowed IBM to then use Intel processors. I am sorry, your - well - history-fu is quite shallow on that one.
I am getting real tired of hearing all the hype for ARM even though ISA really doesn't matter. If any of you who claim it as gospel actually do some research, you would stop spreading it. chipsandcheese has outstanding article on the topic "ARM or x86? ISA Doesn’t Matter" and it is in essency meta-analysis on the different actually research papres on the topic. What actually matters more is the actual whole system design in addition to the actual CPU power efficiency. Also Jim keller has said the same thing about ISAs.
@25:30 After you grabbed my full attention 3 seconds into this incredibly high-quality and well architected documentary, I said to myself, "AI is our only way out of this..." and here we are. Your sentiment about Triton is dead-on about OpenAI making chips. Especially when you hear OpenAI boardroom discussions even entertaining $2,000/month for ChatGPT Premium. That number seems insane to us but not when one chatbot can fully replace an entire clerk's office that would cost you $50,000+/month in salary alone. And I, like you, have an Intel/Nvidia Machine and an AMD machine sitting right here at my desk: one for CUDA and one for a 12-core CPU.
Thank you very much for your comment. Yeah I've been using AMD CPUs for a long time since ryzen 1800X, but I used an Intel CPU in my latest deep learning build with a 3090!
Love the vibe love the storytelling technique tell the what you will tell them, tell them, then remind them about what you just told them;) superb. Introduction and greetings, then to wrap it all up a charming personality and great visual props. Very well done.
wow.... this is amazing! Not just well researched and comprehensive knowledge, but all of the information is well expressed with the appropriated level of detail... it's rare to sit through a video where I don't want to fast-forward... i was actually engaged throughout all of it, no loose ends, and the information is framed in a way that makes sense... thank you so much!
Thank you very much! I do try really hard to keep the level of information appropriate. Hope you get a chance to check out some of my other videos too!
It's 1:39 AM right now in Brazil. I should be sleeping for long time. But the presentation and the substance of this video kept me up. Instant subscribe! Edit: time correction
I haven't heard much about them other than that one of my friends got a job offer there, which was very generous, but decided not to work there. Slight concerns about bonuses and long-term stability I guess not sure. I have a video coming out soon about Groq which is in a similar space I guess.
A nice presentation that most Nvidia "critics" and "shorts" seem to have little appreciation of. A part of the moat not mentioned, I think, is that several of the megacap tech companies who are working on their own hardware and software are also intensely competing with each other to get there generative apps out and making money. That competition means they're so far dedicated to the Nvidia ecosystem and that means their hardware and software stack. Another thing that extends the time span of the moat is that Nvidia is not standing still at all--they've speeded up the hardware development cycle from about 2 years, targeting 1 year cycles from now on. Their generative AI virtual assistant bank of applications seems way ahead also. So it seems like the competition is building a car for a drag race with Nvidia and has a nice gear shift but is lacking tires and a transmission and the motor is questionable. By the time they assemble the parts for the race Nvidia, has a car that is several generations ahead of them. So, eventually other organizations will have something equivalent to what Nvidia offers today. The question to ponder is what newer and more advanced problems are they going to have new stacks to attack and solve--not AI per se but maybe in other areas where their parallel processing data centers excel and the others are not yet working toward.
@@DrWakumy problem with this vid is these problems existed with Volta , nvida has a install base moat not really a cuda moat …. Its deeper than ”cuda” you can put it a program on nvida and it RUNS EVERYWHERE
Sort of open architecture? Intel? Thats not even close to what it was. Edit: Wait, aren't you the guy from Dredd? You know Mama's IT guy in Peachtrees?
Really impressed by your research abilities and how much you're able to simplify the details and tie them up together, can you share how many hours a video like this takes you to produce from scratch if you're comfortable with that?
Of course, it takes about 8-12 hours to research and write a script, 2-3 hours for thumbnail/description/timestamps, 1.5 hours to record. I think this video took about 10+3+2 = 15 hours. I pay for an editor, thank goodness, or that would surely add another 10+ hours.
AMD is always a 2nd place company. They do less inventing and more copying. Nvidia will certainly have competition in the long run. That’s why they need to be hyper focused on next-generation products. Someone called the DOJ in to slow nVidia down but most investors are worried that all of the capital expenditures in GPU’s is not bringing any level of profit. Which is now weighing on nVidia’s stock price. NVidia’s new chip will make the stock price jump up again.
That was what I thought in the Bulldozer era, but I still supported them (AMD) anyway. Arguably since the release of Ryzen chips though, AMD has been one of the most innovative companies out there. Still second place in terms of resources. But I don't know that they deserve that designation on the whole. GPU division has been another story.
@@DrWaku You might be right, but AMD's Ryzen chip is an X86, which is not their invention. Also, the x86 might already be a dinosaur looking at a bright light and the light is an ARM based killer asteroid. Also, from day one, AMD was not the GPU of choice to be used for AI. Nvidia is in autonomous cars, robots, AI, LLMs, computer graphics and crypto. I think AMD will be lucky if they can stay in second place in the GPU game. IMO, Nvidia's biggest challenge is from Apple, Microsoft, Meta, or Google, because they have the cash. Even little ole Amazon has 8 times the cash that AMD does. Good video, thanks.
I've watched many of the videos from this channel at this point and learned a lot. I really appreciate the high quality in-depth explanatory videos you've made through the past year. First I was hesitant from pushing on to your videos from the list because of the length of the videos. However after watching many of them I really appreciated the density of the information and the proper break-down interpretation of the complicated subjects you try to convey to the audience in a much more palpable form. Nevertheless I decided to write a comment for the first time after watching the videos on your health and career change in the past year. In my opinion this channel's videos' content is very high quality, probably worth much more subscribers than the current subscribers the channel has. The content quality is better than other engineering knowledge channels which have 100k to 300k subscribers. In my opinion two main elements are acting as a huge roadblock for the growth of your channel. (1) thumbnail design, art style of the thumbnail and (2) the average length of the videos. I wasn't going to comment anything before because I wasn't sure if you really wanted growth in your UA-cam channel but after watching the videos regarding your personal career change and health, I decided to write this after watching them, hoping this helps the channel growth. (1) the thumbnail is the biggest roadblock from the clicks getting through because people these days associate "generative AI" images and bots, or low quality videos. It seems like you edit texts on the images after you generate AI images for the thumbnails, but still the whole channel's thumbnails all look too much like a collection of AI generated images. The thumbnails looking like it containing a low quality video does a huge disservice to your excellent quality knowledge educational videos. In my opinion, the video themselves does not need any additional cosmetics changes. It is plenty good for an engineering knowledge sharing channel, but unless the general look of the thumbnails change, very few people will notice that this is a in-depth serious engineering channel's video. The engineering UA-cam channels' thumbnails have a certain style of look to them. I doesn't have to match the style but there is a reason their thumbnails evolved in a certain style over the years. It attracts and grab attention of a certain types of people, technology nerds such as myself. The current thumbnails looks too complicated and unorganized. It needs to be much more simple looking thumbnails in order to be perceived as a video from an in-depth engineering channel. The thumbnails don't have to show a human face, as many tech channels do. There are channels that make good thumbnails without putting a human face in the thumbnail. I suggest a few channels that makes a good reference thumbnails that fits this channel's content as reference for you. hope this helps. Some of these channels are not engineering channels but they make very good thumbnails for their videos. (@TheTeslaSpace, @Fireship, @3blue1brown, @ExplainingComputers, @TwoMinutePapers, @domainofscience, @GamerMeld ) If making these style of thumbnail is difficult under your health condition, at least more emphasis on the text on your thumbnails would be a improvement. (2) The average length of the videos of this channel seems its around 20~25minutes. Although the nerdy audience is more tolerant with lengthy videos when it comes to technical subjects, because most of the channels that are full of lengthy videos host low quality, low-density videos most of the times, people will see that your channel having a videos length of 20~25minutes by average as an indicator of low quality, low-density videos which nerd absolutely dislikes. The videos has to be under 19m59s in this subject matter, unless it's an long-form interview video of a person. Preferably under 16minutes, but 19m59s is the absolute maximum, in my experience, for not alienating the engineering enthusiast audience. I know that your channel's videos, even with 30minutes lengths, are very dense with it's information. If your audience reach a certain number, there is a snowball effect that mitigates the bad first impression associated with lengthy videos, but the combination of the Generative AI thumbnails and the lengthy videos, it gives a very wrong first impression of the UA-cam channel in my opinion. There are ways to contain video lengths to under 16minutes even if it's approaching a complicated subject. For example separating the video of multiple sub-subjects, and linking each other, not just chopping them the videos in a linear fashion. I really enjoyed your videos and hope that this channel's reach more wider audience in the future. Good luck and I'll be looking forward for future videos Dr Waku.
Thank you very much for your in-depth analysis. I do want to grow my channel, though I'm getting an impression of how difficult it might be to be a larger UA-camr with all the discussions about the thumbnail on this video. When your audience keeps growing, there are inevitably people who won't like the content, or assume one thing and find another. I guess it can be rough. But I want the channel to be able to support itself or maybe even support me, in future. So I'll give it a go. I used to keep all my videos under 20 minutes, and would even apologise when it approached 18+ minutes. My audience at that time however liked the density of content, no matter how long. But I agree with your analysis. I really hate watching channels with low density of information (hence my style). So keeping to 20 minutes max is a helpful signal to new viewers. (Existing viewers probably don't mind either way.) For the thumbnails, I just really enjoy AI art, so I kept creating them this way even though I've had some feedback that it was not ideal. I checked out all the channels you linked. I see that some are still using AI art, but there is also more use of negative space and larger font, etc. Gives me some ideas. Really appreciate your feedback.
Fantastic video. Being a pytorch + CUDA developer myself, I appreciate your insight. Well researched, well explained, and lots of information I didn't know.
It would be hard to argue that this was a clean room reimplementation, if your competitor has an open source compiler. This is because the LLM was probably trained on all of that source code. Unfortunately, using LLMs isn't the best way to avoid copyright or legal challenges.
because "AI" doesn't exist, and the Large Language Models you're actually referring to are useless for even writing simple web-apps right, let alone a compiler, let alone for CUDA.
Good presentation, but your reasonings are wrong. Intel was big and arrogant and looked down on TSMC, ignored the fragmented scientific communities including AI. AMD has a chance but is very far behind and it is too committed to the PC market to have resources left for AI servers. The Chinese companies don’t even have working prototypes yet. I think challengers will take more time as NVIDIA had 20 years head start in AI acceleration. It is more likely coming a startup, not from incumbents. NextStep supported 68k, PA-RISC, SPARC, and x86 before it was ported to PowerPC and became MacOS when Apple bought NEXT and brought Steve Jobs back. MacOS has come full circle by moving to x86 in 2005 and back to RISC (ARM) in 2020.
I think that a huge price-performance advantage will be needed to break nvidia's moat if it is to happen quickly. Most companies won't take the risk of compatibility issues that can end up in downtimes and extra development costs for a small saving as those can eat the small saving and then some. A very large performance or price to performance advantage in a way that requires breaking compatibility with cuda is what could do it. If getting away from cuda grants you 10x or more compute or allows to scale models 10x, it is very hard to say no as some competitor of you will say yes and you will be left in the dust. I believe that some kind of combination of analog computing (a single transistor can perform a multiplication) with in memory computing could offer the orders of magnitude improvement required to lure people away from Nvidia and I hope that AMD or Intel could be brave enough to try doing that.
Discovery number 1 in terms of clarity, depth of thought, relevance and thought-leadership. Damn this channel is underrated! Happy to have found you and thank you so much for your insights!
THANKS! Huge gratitude in explaining all this mess. I’m looking to build an AI generative machine. The only options for 24gb gpu (4 gpu or more) are old hardware. I avoided ROCm for a year, after reading mix results. Now, AMD might be feasible Thanks
9:16 false. Compatibility goes only as far back as the 8088/8086. 8080 is not compatible, and compatibility with 8085 is only "source-code-level" so re-assembly is required.
NVIDIA is coming up against manufacturing constraints because of narrow minded production ideology. They change everything but that, so long term production scale, cant match what they're currently pushing. They are only seeing a few feet in front of them and trying to plan within that limited production window.. Just on the physics of it all. thats what many are saying. so right now, unless they have a radical shift in how they construct their processors, then other competitors ( which anticipated Nvidia's current issues in advance and with the nimbleness and attrition of a startup )will over take Nvidia eventually. Or at least be competitive with it. That forward planning, and nimbleness will translate across everything even the code.
Suggest comparison of gcc and llvm? I think this shows certain parts of copyright need fixing. Perhaps the ability to force open sourcing of judge declared monopoly closed source!
Very informative video, but Nvidia's AI monopoly is coming to an end? You just made the argument of the opposite. Sure, it will end, as we are going to die someday 😂 Nvidia is not sitting on its hands, it is investing heavily on new libraries using CUDA. Bypassing CUDA by the LLVM route seems to me the only practical solution long term, but there will have implement so much code to catch up to Nvidia. Like you said, it is not happening anytime soon.
One big thing in the video hardware requires way too much power for AI specialized hardware will require a lot less power and deliver a lot more performance.
Apple funds the LLVM project. I bet they're working with AMD to challenge NVidia. Apple has a long adversarial relationship with NVidia. It's why Apple products stopped using their GPUs years ago.
(Edit: Errata below.) I think this is my longest video yet! Going back to weekly schedule, I can't produce videos this long every week though :)
Discord: discord.gg/AgafFBQdsc
Patreon: www.patreon.com/DrWaku
Errata
- Google won the Oracle vs Google case, not Oracle
- "OpenAPI" should be "OneAPI" (re: Intel)
- Oracle may have implemented IBM's API, not the other way around
- Intel did not have an open architecture, AMD was granted a license
- CUDA compiler now uses LLVM as well!
You're time work and wisdom is much appreciated, thank you
thanks for explaining in such a clear way, it's so important for us to understand this technology which is shaping our lives
I appreciate the work you put into your vids.
You should sticky the errata.
In any case, this is my first time watching your content, brilliant work and excellent presentation. Looking forward to browse through your back library.
@@IceAce1Added errata to this same stickied comment, thanks for the suggestion, cheers
Watched you from the beginning. You manage to maintain a very very high quality channel with pretty unique content without resorting to clickbait/hype or other algorithmic bullshit.
That's very kind of you to say. Thank you. I'm of the impression that high quality pays off over time.
NVIDIA LOSING ITS LEAD!!! how is that not clickbait? Theres nothing even close to NVIDIA, let alone being in danger of losing their lead
Was trying to say "losing its monopoly" but that was too long. Made a new thumbnail, hope that helps.
@@glamdragCalm down, the whole clickbait shaming crusade is getting as out of hand as clickbait itself.
@@glamdrag are you okay?
Google vs. Oracle:
The surpreme court ruled in favour of Google. AFAIK they didn't have to pay tons of money. And it wasn't about "Hey, that code looks like ours!" it was a "Hey the reimplemented our API! How dare they!"
Yeah I got the outcome of this ruling wrong. And yeah it was about API reimplementation originally, but the sticking point ended up being a tiny function if I remember correctly. Maybe it didn't have an impact on the final final ruling.
Great info and good presentation, deservers way more than your 16k. Subscribed, cheers from Sydney.
Thanks, I appreciate it! Cheers
Dude, you know your sh&t. Very impressed with your breadth of knowledge. I’ll subscribe!
Excellent essay, one of the most digestible tech breakdowns Ive ever heard, thank you Dr. Waku.
Thank you very much! I appreciate it :) see you around.
yeah - interesting video
Mojo and TinyML are both other widly differing approaches to langs built to support more hardware.
This was clickbait. Nvidia is in no trouble. AMD is certainly trying to catch up but RocM is far behind and in 3-5 years there will be some market share lost to Cerebas or Groq chips but like you said Nvidia is benefitting from their existing relationships and proven quality. The title made it seem like Nvidia was in trouble but you basically spent 30 minutes discussing how far ahead they are.
I don’t really think the title suggested they were in trouble only that their monopoly would end…
@@Greg-xi8yx obviously it will end some day but it's definitely not "losing its lead" any time soon
@@pruff3 the title doesn’t claim it’s losing its lead only losing its monopoly and I think he made a good case that while they will still be the clearly dominant leader though not maintain their monopoly.
@@Greg-xi8yx yeah that's true I guess it's not really clickbait but the case for Nvidia losing its lead (the thumbnail) is misleading (pun intended) and also even the title is like saying "I'm dying" (yes, we all are, slowly, here's the evidence for why eventually we will be dead...in the future)
People are responsible for how they take different perspectives. Dr Waku has always had a high quality video in many aspects. In any regard, if you believe differently create your own video. Different perspectives and different takes is what makes knowledge possible. Calling things as clickbait is just a cheap criticism.
Each time Apple switched processor architectures on the McIntosh, they used Rosetta. The first time switching from Motorola 680X0 architecture 2 power PC, then they used Rosetta again when they switched from our PC to X 86. The switch from X 86 to ARM- 64 was just the latest iteration of Rosetta. I don’t think it was ever possible to have a stack of two or three Rosetta translators on Apple hardware. Doing so would allow you to run software for the original McIntosh bye emulating the Motorola 68,000 on the emulated power PC on the emulated X 86 running on ARM 64
Hi Doc, good to see you, best wishes from UK
Cheers Alan!
The world would be a better place if most "intellectual property" were recognized as shared by all rather than imposing artificial scarcity to control and subjugate all beneath the will of greed and domination.
Thank you for this exhaustive course on CUDA and its potential alternatives. I am a big investor in NVIDIA and you got me worried. I don't know where you found all this information, but it is great work.
This was an interesting take on situation. It reminds me of 2008, when I was buying my first computer. I've gotten an AMD 4850 GPU. I was super happy. It was much better technologically than nvidia's 8800gtx, their first CUDA GPU. Still with software, even though older, 8800 series was better due to CUDA. I figured, I just need to give AMD a bit more time to catch up. With such great hardware, how can they fail?
By 2011 it was obvious to me that AMD cannot possibly come close to nvidia software, no matter how good their GPU hardware is. Back then I got myself nvida gtx 460 and I haven't been back to AMD since. I have no reason to think that now, 18 years later, AMD has gotten any wiser. It would be nice to have a real competitor, but I'm not holding my breath.
Strictly looking at their core 3d GPU technology they both support today: DLSS vs AMD's FSR. AMD simply cannot compete. If they can't make their GPUs perform their main function any better, what chance do they have in more complex problems.
Very accurately explanations, well done!
I tried. A few inaccuracies as other commenters have pointed out. Hopefully the broad strokes were helpful. Cheers
thank you for another video, keep on the good work! love it
cheers from Brazil
Thank you very much for watching! Have a great evening.
How many percent of the cuda api is used by pytorch?
This video was a lot more interesting than I thought it would be before I clicked on it
I almost thought it was going to be AI voice slop too but i saw there was a face on the video lol
I like to hear it when my face overdelivers :))
Good information and video. The next one is also engaging. However, the sound and editing greatly reduce the positive impact on the audience. I won’t be able to finish the video on my end due to the poor sound quality. I hope the adjustments can be made in future.
Thank you for the feedback. I will be using a new pop filter for my next video, I hope that helps. It's a high quality mic, in theory...
@@DrWaku I hope that helps too. I do feel and appreciate a high quality mic in your videos. However, a sensitive recording device combined with the 2023 iPhone Pro, which delivers high-quality sound (especially via AirPods), might potentially amplify every mistake multifold. First-world problems, poor us, eh.
Would you consider trying a one-take video next time? It would be nice to experience your personality and mind in a more natural and unedited way.
On a side note, if I may ask, what’s the story behind your hat and gloves?
I didn't do a one take video but I did do an interview so it's a slightly different format, if you want to check out the latest. The gloves are medical. You can check out the disability playlist for more info.
The hat is because I was tired of people talking about my hairline lol
I have to ask. Why the gloves??
Medical conditions. My hands are quite sore if I don't wear them. Fibromyalgia and rheumatoid arthritis. I made a couple of videos on my disability etc if you want to check it out
@DrWaku How dare you show evidence of a disability on camera! Take the gloves off and suffer while filming lest you remind the audience that such medical mallodies exist. /s obviously
I have to say, although the video has great contents, constantly appearing subtitles at the bottom are incredeibly annoying. I would reduce their quantity by 5 times, if I were you.
Thanks. Yeah, usually the subtitles are a lot less frequent but this time my editor just put them everywhere. I'll ask him to decrease the frequency next time.
you need to start using a pop shield for your mic and you need to do something against the overdrive distortions. your audio isn't the best it could be and not always pleasant to listen to. (Writing this at 10:54 . The content was interesting so far.)
Didn't the google Java case end up legalizing API copyrights? Am I imagining that? The idea is Oracle could own the API/ABI itself as IP and you cannot build a cleanroom implementation.
i forgot to state what linus said to nvidia the iconic line.
Great video, great topic. 💯
Much appreciated!
really good business strategy analysis with sound technical underpinnings. rare. well done!
Is Teslas D1 not a GPU in the classical sense or why is it not mentioned here ?
Here is another moat killer perspective, with the advancement of LLM's and Copilot etc, its getting easier and easier to write code from scratch for building a new stack etc. + open source of AMD, that seems like things can really speed up for AMD with a combination like that. That being said u might wana check out George Hotz who was trying to create a tinyML using AMD and gave up on it and switched to Nvidia either. Other then the Software Moat with CUDA their software itself is less buggy then that of AMD as well, also the fact that programmers are getting lazier and lazier with copilot and LLMs will only make the code more buggier imho
3dfx was the best graphics card manufacturer. I don't know why it disappeared.
Fucking NVIDIA driver on Linux still borked for stuff like FP16. Seems like they don't even get their basic shit together.
AMD drivers and support for Linux have also been poor at times. That's one good thing about the AI boom, forces these companies to treat Linux as a first-class citizen.
Well it’s not coming to an end but upgrade cycles are disrupted. If gpt5 is any good then new upgrade cycle will probably be bought. Until then the upgrade cycle will continue through smaller companies upgrading data centers and databases going into the near future
Nvidia's AI monopoly is coming to an end due to increasing competition from other tech companies developing advanced AI hardware and software. This shift is fostering innovation and providing more diverse options for AI solutions.
Excellent presentation, thank you 🎯
Thanks for watching! And commenting. Cheers
28:42 maybe they are considering MI300X hahaua
What ever happened to ATI and the Radeon Series. There GPU was better and it's gone
AMD bought them, that's how they started making GPUs I believe
AMD bought them and ATi Radeon became AMD Radeon
Some way into the video, I began to resent that I was quite enjoying what appeared to be mere clickbait. However, you skilfully justified the provocative thesis in the final moments - well done.
That said, let the challengers bark as they may. As long as the visionary commander remains compos mentis and the coffers flush with cash, the Nvidia behemoth will continue its march, unfazed by their noise.
Haha thanks for the backhanded compliment. I didn't mean to be clickbait but I think a lot of people have strong feelings about this topic. Thanks for sticking it out.
They would have found it much quicker, easier and cheaper to use Julia.
Short story, monopoly is bad for consumers. A good competitive market is 3-5 major players. And none of those major players should have less than 10% each or more than 40% of the market share.
Anyone who buys from a monopoly contributes to the problem, no buts about it.
I don't get why people especially gamers are poo pooing Moore Thread. Do they want to be in Nvidia clutch forever?
GOD damn I have to give you so much respect for you intelligence! Thanks for your videos!
But I cant give you an abo I hate your glasses OCD kicks so hard a shame.
YOU are great. Keep going,.
dude! this was an awesome video. I as a linux user for 2 decades always hated Nvidia, u should have been the student at standford asking Eric Schmidt some tough questions on this he seems extremely confident on Nvidia did u watch that infamous leaked vid laughing at intel as a competition etc.? but i think u got it right, AMD is going to approach this with "open source" because they are way too behind... AMD is going to end up being the poor man's GPU for LLM's but by the time AMD catches up where will Nvidia be? These things are evolving quite quickly. Zuckerberg seems all in with Meta llama & Nvidia alliance.
Good discussion, would be good to include Huawei's Mindspore. They will dethrone Nvidia in China and the world in the next few years.
As long as AI demands more and more compute, nVidia having the hardware advantage will ensure them a monopoly: in AI only the best matters, be it the LLM or the hardware running inference. Choosing second beast means you're at a disadvantage in a field where months are equal to 'old IT' years, and that equals death to your company. This video is a fantasy.
Thank you for this video, it's very informative. But... Is it just me or are your eyes super dilated? 😄
They are. It's just a natural thing for me, but it makes my sensitivity to light a lot worse unfortunately. Check out some of my disability videos
What the hell is even that?
nvidia nearly lost it when microsoft chose directx.. direct x and opengl are the wrappers.. im not aware of a PC game that only runs on one brand or another. today with AI, nVidia has a way to speed things up. ATI, like most other hardware manufactures, went compliant and didn't compete with the 1 competitor they had... Cuda has been accelerating machine learning for 8 years, ATI/AMD hasn't shown much effort in competing. If anything, shame on ATI/AMD and replace the company leaders who don't innovate. Also, ARM processors will be replacing the PC processors we use today. and it's happening now. Now ATI will lose more market share thanks to new machine language processors, as microsoft is including in new laptops.. Basically 10 years ATI/AMD sat on there hands, and its time to replace leadership.
neat
Because Nvidia logo use dark mode...
Hahahaha. Right. When AMD can run Torch with comparable ease of use let me know.
Crap click-bait. Nvidia just announced their record annual earnings and their future looks bright.
I'm not saying Nvidia is in trouble. Just that they shouldn't count on having a monopoly anymore.
thanks nerd!
Text at the bottom is super annoying
My editor put a lot more subtitles than usual into this one. I will definitely decrease the frequency in the future.
@@DrWaku Y'all are doing a great job but there is a strong reason I have never used tiktok and it has to do with things being hyper-aligned to attention algorithms. The firehose is disquieting for mature (tired) minds, in such a way it might be impossible to describe to anyone under 30.
I was captivated for the whole video, you speak clearly and with a very nice pace and build a good foundation before touching the more confusing parts of the topic (also thank you for using the companies logos every time you mention one…)
Thank you very much for your kind comment! You can thank my editor for the logos haha, he does a great job
Good informative roundup of the topic
14:07 I think you meant oneAPI instead of OpenAPI (stuff like this would deserved a pinned errata comment 🙂)
Thanks. I created an errata comment like you suggested, there are enough little errors in this video that it needs one
A lot of small, but important, mistakes:
1. Triton was supporting AMD in September 2023 and maybe even before. Also PyTorch uses Triton as a backend.
2. CUDA compiler is partly open source as is based on LLVM. Nvidia is a major LLVM contributor
3. The story with IBM and Oracle was the other way around
4. x86 and x86_64 are not open architectures
5. Google won vs Oracle
Thanks for your reply. I took the date of AMD support from when they edited the readme to add it. I figured that's when it was official.
Did not know that the CUDA compiler is based on LLVM. I got the IBM story from an IBM employee, maybe they were biased. The rest I didn't check on, good catches thanks.
Added some errata to the pinned comment
@@DrWaku What pinned comment ? I don't see any pinned.
What part of CUDA is open source?
@@DrWaku You added the errata, but forgot to pin the comment.
NV might be keeping some competitors out, but in case of AMD the thing holding them out is THEMSELVES, they have just been too sloppy and disregarding year after year for their sw ecosys and that way they never built value but destroyed their own footprint.
Agreed. AMD has always tried to do things with a lower budget and without an eye to long term sustainability, unfortunately. We'll see if that continues in the GPU division under Lisa Su.
AMD needed to avoid bankruptcy and become successful/profitable with their other products before they could fully focus resources on GPGPU. From the Xilinx acquisition and on, they have been investing very heavily. We'll see how smartly they can deploy, integrate, and scale ROI on those investments.
They focused on CPU. As Ryzen has become stable, hopefully they can focus on GPU, especially ROCm.
Intels thing is definitely not called "OpenAPI" because thats a standard for defining and documenting HTTP REST APIs.
Intels thing seems to be called "oneAPI"
Yeah, for some reason one reference called it open API by accident and that stuck in my head. This has been added to the errata list
Nvidia GPUs are also built around floating-point registers, which are used to compute decimal values. While floating-point logic has gotten better in recent years, it remains more complex than integer logic. Simply put, it takes more wires and transistors to build a floating-point register than one made to process integers. More wires equals higher cost. More wires equals more resistance. More resistance means higher energy consumption and heat generation. AI Developers are re-engineering transformers and neural nets to make use of more efficient integer operations and eliminate matrix multiplication. This would allow AI Models to run on much faster, cheaper, and more energy efficient RISC-Based ARM Chips.
It was Oracle who wrote a simple API to pull stuff out of DB2. So the Oracle-IBM story is the other way around. There are t-shirts that said, Oracle had IBM talking to itself. Because using the API, you could set up one DB2 talking to another DB2 through Oracle, yet those two different DB2 databases were incapable of talking to themselves.
I wonder if each company implemented the other's API at some point. I got this story from someone who worked at IBM and who was proud of the work that IBM had done, lol. That said, DB2 is probably not very flexible or easy to use so I'm not surprised they made a t-shirt.
An excellent overview.
AMD's HIP is a CUDA compatible-like API (function names are changed from 'cuda_' to 'hip_' for legal reasons) and this makes porting CUDA relatively easy with a script (hipify).
Blender for example targets CUDA for NVIDIA GPUs and HIP for AMD GPUs which means 2/3rds of the code can be shared between them cutting down dev time.
Yes HIP would be a generation behind but nobody is deploying code written with the very latest API version in mind because a) development takes time and b) there's no market share for it. So that's dumping some dirt into the moat.
PyTorch abstracting code away from CUDA is a dump truck of dirt into the moat.
Triton is the next big step which brings me to something very important..
You mentioned AMD and intel as competitors but left out NVIDIA's _biggest_ competitors.
All of NVIDIA biggest customers, Google, Microsoft, Meta, Amazon, Tesla, have their own AI accelerators in various stages of design or production.
Google's TPUs have been in production for years. Microsoft is onto their second generation of their MAIA chip. Tesla is training on their Dojo (D1) chips. And Amazon has their second generation Trainium and Inferentia chips available to rent on AWS.
These all operate with the PyTorch framework and most have Triton support.
NVIDIA doesn't have to worry about their competitors, they have to worry about their customers.
Pytorch recently started supporting MPS on Apple Silicon. I used it myself on my Mac, working like a charm. No nvidia needed.
there's one important thing that's being over looked. The CUDA compiler isn't just converting higher level language to executable byte code. It's also optimizing the execution to maximize parallel processing, which is historically very hard. You can have great hardware, but it won't be faster if your compiler can't effectively parallelize the work. CUDA stack does a lot of fancy stuff to optimize the code.
OpenCL is essentially dead and no one should use it going forward. I really want scale to succeed and 7 years is a long time to spend on a project. The better fix is to move neural networks forward so we don't need 20K H100 to train 1 trillion parameters. We need to understand what all those weights are doing and make models that are 10,000 smaller.
every compiler does fancy stuff to optimize the code. You think gcc doesn't vectorize your data to benefit from SIMD as well ? :p
it's the programmer who needs first to not only accurately parallelize the code for maximum throughput , but also be careful to keep gpu threads coalescent in memory accesses. this is the hardest part , finding THE parallel algorithm.
@@emperorpalpatine6080 great point. GCC does a lot of fancy stuff and there are a ton of compiler options. I've never written a GPU shader compiler, but I do enjoy writing compilers in my free time. From my own experience, most developers have a poor understanding of how to parallelize the execution and often it isn't obvious. Even if you figure out the best way today, if the data changes over time it won't stay optimal. It reminds me of the saying "premature optimization is the root of all evil."
I don't necessary agree with that statement for all situations, but I have seen it first hand. Hand rolled optimization is difficult to maintain as data and hardware evolves. Nvidia has done a good job of making CUDA handle a lot of that, but you still have to make sure you load your data efficiently
There are 300+ cuda libraries supporting all forms of scientific, engineering, and commercial applications. They cannot be placed or duplicated quickly by a new language or compiler.
Very informative content. Well done.
the moat is slowly being whittled away, and I'm here for it!
It's interesting to see these big companies like Intel and Nvidia experience some troubles when they might have seemed untouchable in the recent past...
Why didn't they made OpenCL compilers that automatically took care of the hardware-specific optimizations?
Vis a vis CUDA’s complexity for coders, how much would it take for Nvidia to make a custom AI that’s an expert at CUDA code creation? They certainly have both the financial and compute resources to throw at the problem, and it could remove one of the biggest hassles for users.
Entirely possible (see: AlphaTensor), but there’s little need for it. They already have out of the box easy to use libraries for most things you need (Cutlass, etc), so if you need to do something, you can generally find a kernel for it somewhere on the internet, meaning you don’t really need to be a CUDA master. The complexity of writing kernels is why I use CPUs of all things for deep learning (it requires different designs for the architecture, but you can get surprisingly close to parity between single GPU and a single CPU), as it’s a lot easier to do a fully optimal AVX kernel for your CPU versus a fully optimal CUDA kernel. I’m looking into Tenstorrent hardware in the future as they’re roughly the same programming model as CPU AVX kernels.
It seems a bit click-baity, controversy seeking, and conspiracy-founding.
Nvidia has invested many man-decades of work (and billions of dollars) into defining a software technology that matches their hardware technology, all of which are about representation of ideas and speed. Why isn't their hardware design a moat? Nvidia doesn't owe any other company free access to its technology. Amd and others can design a better system from scratch and whoop whomever they please. So far, no other companies can run and pass all the benchmarks anyone has ever written, and beat everyone. In AI, general compute, graphics, etc. Why would they give the keys to their office to the competition?
How much? It would take magic, because there is no such thing as "AI". Despite all the marketing, there is currently no program that can replace or even meaningfully assist an expert in writing any sort of involved code. Large language models have so far failed and increasingly seem to be a dead end for that purpose (among many other purposes). AI is a meaningless buzzword. Unless you're a developer talking to investors, stop using it.
Jensen (nvidia ceo) said no programmers in a few years, so theyre working on stuff like that. But i think hes wrong and doesnt understand programming, LLMs can help, but they're not even close yet to actually replacing programmers, we need more scientific breakthroughs for it to happen. Maybe it will, but it seems very optimistic and hees just hyping up nvidia for profit
@@martinkrauser4029 I think you have to look at the trajectory, not the current status. To me there's a clear trend, and I don't see a bright line anywhere that would be an uncrossable threshold.
Perhaps we need to distinguish between "programming" and "coding". I view the former as including system architecture and design, goal-setting (choosing which problems to solve) and the generally "creative" parts of the process, while coding is simply converting the output of those parts into executable code. While we're nowhere near having machines able to do the former, we're making rapid progress on the latter.
Currently, I'd say the assessment depends on your definition of "meaningfully assist." The LLMs can't write complete systems or even error-free components, but they can certainly shoulder a lot of the gruntwork of code-cranking. They're even now useful an amplifier on an expert's ability to rapidly generate code that implements their design. We're nowhere near what's being hyped, but look at the progress from GPT3 to GPT4 o1 in what, 2 years? From my naive viewpoint, the progress is accelerating, if you look at the capabilities of 3 to 3.5 to 4 to 4o and now 4o1. (Of course there are similar developments in other systems too, notably Anthropic's Claude series.)
I agree that "AI" is a bit of a misnomer, although I've come to question what "I" actually is in the first place. (Maybe we're all just LLMs with state memory, on-the-fly reinforcement learning and agency? 😉) But I do think we're already seeing tools that can significantly leverage a human's time, and that are being able to do so more and more effectively.
AMD being compatible with Intel is NOT because of an open standard - it is because AMD had something to LEVERAGE Intel into giving them a LICENSE. What they leveraged was NOT being Intel - when IBM wanted to start making PC's, it was the dominant player and had a policy of ONLY dealing with anything that has at least 2 suppliers. So, Intel and AMD came to an agreement that allowed IBM to then use Intel processors. I am sorry, your - well - history-fu is quite shallow on that one.
Have you heard of Mojo language from Chris lattner (creator of llvm)?
I am getting real tired of hearing all the hype for ARM even though ISA really doesn't matter. If any of you who claim it as gospel actually do some research, you would stop spreading it. chipsandcheese has outstanding article on the topic "ARM or x86? ISA Doesn’t Matter" and it is in essency meta-analysis on the different actually research papres on the topic. What actually matters more is the actual whole system design in addition to the actual CPU power efficiency. Also Jim keller has said the same thing about ISAs.
Great stuff, was very informative, and well presented too! Thanks
Thank you for your comment! Appreciate it
I subscribed because content was high quality and insightful
Thanks for the comment! And sub :)
@25:30 After you grabbed my full attention 3 seconds into this incredibly high-quality and well architected documentary, I said to myself, "AI is our only way out of this..." and here we are.
Your sentiment about Triton is dead-on about OpenAI making chips. Especially when you hear OpenAI boardroom discussions even entertaining $2,000/month for ChatGPT Premium. That number seems insane to us but not when one chatbot can fully replace an entire clerk's office that would cost you $50,000+/month in salary alone.
And I, like you, have an Intel/Nvidia Machine and an AMD machine sitting right here at my desk: one for CUDA and one for a 12-core CPU.
Thank you very much for your comment. Yeah I've been using AMD CPUs for a long time since ryzen 1800X, but I used an Intel CPU in my latest deep learning build with a 3090!
Love the vibe love the storytelling technique tell the what you will tell them, tell them, then remind them about what you just told them;) superb. Introduction and greetings, then to wrap it all up a charming personality and great visual props. Very well done.
Thank you very much :) yeah I come from the old school storytelling tradition I guess haha.
wow.... this is amazing! Not just well researched and comprehensive knowledge, but all of the information is well expressed with the appropriated level of detail... it's rare to sit through a video where I don't want to fast-forward... i was actually engaged throughout all of it, no loose ends, and the information is framed in a way that makes sense... thank you so much!
Thank you very much! I do try really hard to keep the level of information appropriate. Hope you get a chance to check out some of my other videos too!
It's 1:39 AM right now in Brazil. I should be sleeping for long time. But the presentation and the substance of this video kept me up. Instant subscribe!
Edit: time correction
What about Jim Keller's project TensTorrent?
I haven't heard much about them other than that one of my friends got a job offer there, which was very generous, but decided not to work there. Slight concerns about bonuses and long-term stability I guess not sure. I have a video coming out soon about Groq which is in a similar space I guess.
This is a very informative video and I watched it more than once. Even comment section has a lot of professional discussions.
Thank you very much! Yes I love how my viewers often leave very technical and informed viewpoints in the comments. Always much appreciated.
A nice presentation that most Nvidia "critics" and "shorts" seem to have little appreciation of.
A part of the moat not mentioned, I think, is that several of the megacap tech companies who are working on their own hardware and software are also intensely competing with each other to get there generative apps out and making money. That competition means they're so far dedicated to the Nvidia ecosystem and that means their hardware and software stack.
Another thing that extends the time span of the moat is that Nvidia is not standing still at all--they've speeded up the hardware development cycle from about 2 years, targeting 1 year cycles from now on. Their generative AI virtual assistant bank of applications seems way ahead also. So it seems like the competition is building a car for a drag race with Nvidia and has a nice gear shift but is lacking tires and a transmission and the motor is questionable. By the time they assemble the parts for the race Nvidia, has a car that is several generations ahead of them.
So, eventually other organizations will have something equivalent to what Nvidia offers today. The question to ponder is what newer and more advanced problems are they going to have new stacks to attack and solve--not AI per se but maybe in other areas where their parallel processing data centers excel and the others are not yet working toward.
Single company, single point of failure, think crowdstrike!
Exactly, right?
@@DrWakumy problem with this vid is these problems existed with Volta , nvida has a install base moat not really a cuda moat …. Its deeper than ”cuda” you can put it a program on nvida and it RUNS EVERYWHERE
Sort of open architecture? Intel? Thats not even close to what it was. Edit: Wait, aren't you the guy from Dredd? You know Mama's IT guy in Peachtrees?
Lol, I'm told I look like a lot of different people but that's a new one. Sorry about the intel thing, didn't look it up before filming.
@@DrWaku I hope I didnt offend, but for real, you could have played that guy!
Really impressed by your research abilities and how much you're able to simplify the details and tie them up together, can you share how many hours a video like this takes you to produce from scratch if you're comfortable with that?
Of course, it takes about 8-12 hours to research and write a script, 2-3 hours for thumbnail/description/timestamps, 1.5 hours to record. I think this video took about 10+3+2 = 15 hours. I pay for an editor, thank goodness, or that would surely add another 10+ hours.
AMD is always a 2nd place company. They do less inventing and more copying. Nvidia will certainly have competition in the long run. That’s why they need to be hyper focused on next-generation products. Someone called the DOJ in to slow nVidia down but most investors are worried that all of the capital expenditures in GPU’s is not bringing any level of profit. Which is now weighing on nVidia’s stock price. NVidia’s new chip will make the stock price jump up again.
That was what I thought in the Bulldozer era, but I still supported them (AMD) anyway. Arguably since the release of Ryzen chips though, AMD has been one of the most innovative companies out there. Still second place in terms of resources. But I don't know that they deserve that designation on the whole. GPU division has been another story.
@@DrWaku You might be right, but AMD's Ryzen chip is an X86, which is not their invention. Also, the x86 might already be a dinosaur looking at a bright light and the light is an ARM based killer asteroid.
Also, from day one, AMD was not the GPU of choice to be used for AI.
Nvidia is in autonomous cars, robots, AI, LLMs, computer graphics and crypto.
I think AMD will be lucky if they can stay in second place in the GPU game.
IMO, Nvidia's biggest challenge is from Apple, Microsoft, Meta, or Google, because they have the cash. Even little ole Amazon has 8 times the cash that AMD does.
Good video, thanks.
I've watched many of the videos from this channel at this point and learned a lot. I really appreciate the high quality in-depth explanatory videos you've made through the past year. First I was hesitant from pushing on to your videos from the list because of the length of the videos. However after watching many of them I really appreciated the density of the information and the proper break-down interpretation of the complicated subjects you try to convey to the audience in a much more palpable form. Nevertheless I decided to write a comment for the first time after watching the videos on your health and career change in the past year.
In my opinion this channel's videos' content is very high quality, probably worth much more subscribers than the current subscribers the channel has. The content quality is better than other engineering knowledge channels which have 100k to 300k subscribers. In my opinion two main elements are acting as a huge roadblock for the growth of your channel. (1) thumbnail design, art style of the thumbnail and (2) the average length of the videos. I wasn't going to comment anything before because I wasn't sure if you really wanted growth in your UA-cam channel but after watching the videos regarding your personal career change and health, I decided to write this after watching them, hoping this helps the channel growth.
(1) the thumbnail is the biggest roadblock from the clicks getting through because people these days associate "generative AI" images and bots, or low quality videos. It seems like you edit texts on the images after you generate AI images for the thumbnails, but still the whole channel's thumbnails all look too much like a collection of AI generated images. The thumbnails looking like it containing a low quality video does a huge disservice to your excellent quality knowledge educational videos. In my opinion, the video themselves does not need any additional cosmetics changes. It is plenty good for an engineering knowledge sharing channel, but unless the general look of the thumbnails change, very few people will notice that this is a in-depth serious engineering channel's video.
The engineering UA-cam channels' thumbnails have a certain style of look to them. I doesn't have to match the style but there is a reason their thumbnails evolved in a certain style over the years. It attracts and grab attention of a certain types of people, technology nerds such as myself. The current thumbnails looks too complicated and unorganized. It needs to be much more simple looking thumbnails in order to be perceived as a video from an in-depth engineering channel. The thumbnails don't have to show a human face, as many tech channels do. There are channels that make good thumbnails without putting a human face in the thumbnail. I suggest a few channels that makes a good reference thumbnails that fits this channel's content as reference for you. hope this helps. Some of these channels are not engineering channels but they make very good thumbnails for their videos.
(@TheTeslaSpace, @Fireship, @3blue1brown, @ExplainingComputers, @TwoMinutePapers, @domainofscience, @GamerMeld )
If making these style of thumbnail is difficult under your health condition, at least more emphasis on the text on your thumbnails would be a improvement.
(2) The average length of the videos of this channel seems its around 20~25minutes. Although the nerdy audience is more tolerant with lengthy videos when it comes to technical subjects, because most of the channels that are full of lengthy videos host low quality, low-density videos most of the times, people will see that your channel having a videos length of 20~25minutes by average as an indicator of low quality, low-density videos which nerd absolutely dislikes. The videos has to be under 19m59s in this subject matter, unless it's an long-form interview video of a person. Preferably under 16minutes, but 19m59s is the absolute maximum, in my experience, for not alienating the engineering enthusiast audience. I know that your channel's videos, even with 30minutes lengths, are very dense with it's information.
If your audience reach a certain number, there is a snowball effect that mitigates the bad first impression associated with lengthy videos, but the combination of the Generative AI thumbnails and the lengthy videos, it gives a very wrong first impression of the UA-cam channel in my opinion. There are ways to contain video lengths to under 16minutes even if it's approaching a complicated subject. For example separating the video of multiple sub-subjects, and linking each other, not just chopping them the videos in a linear fashion.
I really enjoyed your videos and hope that this channel's reach more wider audience in the future. Good luck and I'll be looking forward for future videos Dr Waku.
Thank you very much for your in-depth analysis. I do want to grow my channel, though I'm getting an impression of how difficult it might be to be a larger UA-camr with all the discussions about the thumbnail on this video. When your audience keeps growing, there are inevitably people who won't like the content, or assume one thing and find another. I guess it can be rough. But I want the channel to be able to support itself or maybe even support me, in future. So I'll give it a go.
I used to keep all my videos under 20 minutes, and would even apologise when it approached 18+ minutes. My audience at that time however liked the density of content, no matter how long. But I agree with your analysis. I really hate watching channels with low density of information (hence my style). So keeping to 20 minutes max is a helpful signal to new viewers. (Existing viewers probably don't mind either way.)
For the thumbnails, I just really enjoy AI art, so I kept creating them this way even though I've had some feedback that it was not ideal. I checked out all the channels you linked. I see that some are still using AI art, but there is also more use of negative space and larger font, etc. Gives me some ideas. Really appreciate your feedback.
Fantastic video. Being a pytorch + CUDA developer myself, I appreciate your insight. Well researched, well explained, and lots of information I didn't know.
Why not use AI to write a complete clean room compiler in open source?
It would be hard to argue that this was a clean room reimplementation, if your competitor has an open source compiler. This is because the LLM was probably trained on all of that source code. Unfortunately, using LLMs isn't the best way to avoid copyright or legal challenges.
because "AI" doesn't exist, and the Large Language Models you're actually referring to are useless for even writing simple web-apps right, let alone a compiler, let alone for CUDA.
Good presentation, but your reasonings are wrong.
Intel was big and arrogant and looked down on TSMC, ignored the fragmented scientific communities including AI. AMD has a chance but is very far behind and it is too committed to the PC market to have resources left for AI servers. The Chinese companies don’t even have working prototypes yet.
I think challengers will take more time as NVIDIA had 20 years head start in AI acceleration. It is more likely coming a startup, not from incumbents.
NextStep supported 68k, PA-RISC, SPARC, and x86 before it was ported to PowerPC and became MacOS when Apple bought NEXT and brought Steve Jobs back. MacOS has come full circle by moving to x86 in 2005 and back to RISC (ARM) in 2020.
I think that a huge price-performance advantage will be needed to break nvidia's moat if it is to happen quickly. Most companies won't take the risk of compatibility issues that can end up in downtimes and extra development costs for a small saving as those can eat the small saving and then some.
A very large performance or price to performance advantage in a way that requires breaking compatibility with cuda is what could do it. If getting away from cuda grants you 10x or more compute or allows to scale models 10x, it is very hard to say no as some competitor of you will say yes and you will be left in the dust.
I believe that some kind of combination of analog computing (a single transistor can perform a multiplication) with in memory computing could offer the orders of magnitude improvement required to lure people away from Nvidia and I hope that AMD or Intel could be brave enough to try doing that.
*Excellent* production values in this presentation. Very clear explanation, and well laid out.
I am do impressed with this video. Thank you.
Discovery number 1 in terms of clarity, depth of thought, relevance and thought-leadership. Damn this channel is underrated! Happy to have found you and thank you so much for your insights!
Their margin % may fall overtime but their volume will increase so gross profit $$$ may still grow over time..
THANKS! Huge gratitude in explaining all this mess.
I’m looking to build an AI generative machine. The only options for 24gb gpu (4 gpu or more) are old hardware. I avoided ROCm for a year, after reading mix results. Now, AMD might be feasible
Thanks
9:16 false.
Compatibility goes only as far back as the 8088/8086.
8080 is not compatible, and compatibility with 8085 is only "source-code-level" so re-assembly is required.
NVIDIA is coming up against manufacturing constraints because of narrow minded production ideology. They change everything but that, so long term production scale, cant match what they're currently pushing. They are only seeing a few feet in front of them and trying to plan within that limited production window.. Just on the physics of it all. thats what many are saying. so right now, unless they have a radical shift in how they construct their processors, then other competitors ( which anticipated Nvidia's current issues in advance and with the nimbleness and attrition of a startup )will over take Nvidia eventually. Or at least be competitive with it. That forward planning, and nimbleness will translate across everything even the code.
Suggest comparison of gcc and llvm? I think this shows certain parts of copyright need fixing. Perhaps the ability to force open sourcing of judge declared monopoly closed source!
Ironically AI will probably make coding a CUDA competitor easier
Musa is Arabic for Moses. So Musa will lead Moore Threads to the promised land of CUDA compatibility.
Very informative video, but Nvidia's AI monopoly is coming to an end? You just made the argument of the opposite. Sure, it will end, as we are going to die someday 😂 Nvidia is not sitting on its hands, it is investing heavily on new libraries using CUDA. Bypassing CUDA by the LLVM route seems to me the only practical solution long term, but there will have implement so much code to catch up to Nvidia. Like you said, it is not happening anytime soon.
One big thing in the video hardware requires way too much power for AI specialized hardware will require a lot less power and deliver a lot more performance.
Apple funds the LLVM project. I bet they're working with AMD to challenge NVidia. Apple has a long adversarial relationship with NVidia. It's why Apple products stopped using their GPUs years ago.