Energy, not compute, will be the #1 bottleneck to AI progress - Mark Zuckerberg
Вставка
- Опубліковано 20 кві 2024
- Full Episode: • Mark Zuckerberg - Llam...
Apple Podcasts: podcasts.apple.com/us/podcast...
Spotify: open.spotify.com/episode/6Lbs...
Transcript: www.dwarkeshpatel.com/p/mark-...
Me on Twitter: / dwarkesh_sp - Наука та технологія
Zuck is so much better at talking about nerdy shit than he is as a public-facing marketing guy lol
yeah noticed that too
Proof we should let people be real.
That's true, but the real question is WHY? The real answer isn't found in Zuckerberg's individual personality, but the wokeness and censoring nature of his employees and the surrounding Bay Area.
A sign he’s the real deal and not a show man.
Literally me lol
Zucc got his latest patch 1.8342
- improved facial animations
- improved vocal features
- improved body language
- custom hair
- integrated eye moisturizer (no more eye licking)
Still under development:
- show emotions on face, voice, body language
-
kids. kids is probably the biggest patch you can have in your life. after kids, people change. whether they want to or not. your priorities change, your emotions change, your maturity changes. the biggest cares and worries in your life also change.
it's not that zucc is more human or anything now. it's that he's a dad to 2 kids now. that demands a different persona
-fully working schlong
-fully working scchhhlong
😂 same thought when I saw video
Bro i was inhaling smoke when I read no more eye licking and I died
I never realized how much power was going into making these foundation models.
You should see how much energy is used for bitcoin - more than some countries.
So much for these big tech companies caring about climate, it’s all about AI dominance now.
@@erniea5843they only like to virtue signal when they can put the costs on average people
@@erniea5843 Trust me, if their AI can turn out a portrait of George Washington as an African American, then they care about climate change !
@@tuhaggis you should see how much energy is used for Christmas lights
Dwarkesh cut it out of this clip but he mentioned to Mark that Amazon just purchased a 960 MW (=close to 1 GW) data center from Talen energy next to a Talen nuclear power plant in Penssylvania... around the same time they invested 1 billion dollars in Anthropic, the company that made llm Claude. So I have a feeling we WILL actually see that major leap in capabilities next year, unlike what Mark believes, or seems to believe.
I completely agree with you!💯🎯
Just looked it up online and it's true
Are there pther compnies doigg NB the same?
I work for the company that built the switch yard for this data center. When construction started almost 3 years ago it was a for a bitcoin miner, half way through we heard the buyer had backed out. Funny how I find a random youtube video that tells me the outcome, as we have not heard anything.
@@G73ServerNot many companies have the ability to buy a GW plant that produces electricity 24/7.
Zuck got his human-look update
Stress and pressure will do that to a person.
i also noticed that, wtf is happened? 😅
@@Gramooneer I think it's mostly that he started growing his hair out. His hair looks much better at this length (makes his head look less like an alien). Also, his skin has more natural color to it. (via being in the sun or taking vitamins or something, idk)
It’s the hair. He got a normal haircut and is in really good shape. We are the same age and I’m very proud of him.
Yeah it’s the longer hair
energy always has been the determining factor of the advancement of society
Nuclear energy was always the answer
You can have desalination you can fight wildfire you can have infinite agriculture tech EVs etc
@KCJbomberFTW but noooo nuclear always goes kaboom
@@KCJbomberFTWwhat's the energy constraint for fighting wildfires?
@@sean_haz excess water in reservoirs to scoop with planes
Facts
glitch in the simulation at 0:40
Also at 0:30 the girl walks behind zucks head then reappears
2:37 another glitch - the man just flat out disappears
AI talking about why it's not possible to be developed 😂
im not sure if ur joking but its just a speed up probably
the zuck can bend the space time
Zuck looks more human than usual. That's sus. XD
I'm glad that you like his recent update
He was living a fantasy life, where in his mind he was this multimillionaire giga chad pioneering the future of humanity.
It seems he has gotten humbled by reality and realized that he isn’t what he thought he was. It’s good character development for him actually.
@@juanramonsilva1067💯
They got enough GPU compute now
@@juanramonsilva1067 Well put. VR and AR isn't really his thing either. You can tell how in-tune he is when it comes to neural networks and AI.
Mark Zuckerberg looks so normal and human here, that it feels weird af; we want our lizard robot boy back
You need to unlock the fusion energy tech tree, before you can build your super AI. Everyone knows that
Fission>>Super AI>>Fusion through Super AI is a much faster track. Just have to keep your population in check during the occasional crises meltdowns during the fission fase.
We don't need super AI. We already have fusion energy☀
@@dayhillbilly From my GPT friend: "To generate 1 gigawatt (GW) of power using solar panels, you would need approximately 4 square kilometers of solar panels, assuming an average solar panel efficiency of around 20%. This can vary based on factors like solar panel efficiency and sunlight availability. (Confidence: Medium)"
And then some batteries of cores.
Fission is literally good enough
It’s the cleanest cheapest safest energy ever made
What? Just build far more energy efficient transistors. The human brain uses bugger all power. Build biological computers if you have to.
When the most powerful companies in the US (Microsoft, Google, etc.) need to solve a bottleneck in order to make progress, regulatory and other constraints will be quickly resolved. There is too much at stake when we talk about AI progress. So, expect a 1 gigawatt data center next year.
Think so too
AI progress is also tied to national defense interest. So I think they will definitely clear regulat process out of the way
or just buy one like amazon done. "Amazon buys nuclear-powered data center from Talen"
"The data center, Cumulus Data Assets, sits on a 1,200-acre campus in Pennsylvania and is directly powered by the adjacent Susquehanna Steam Electric Station, which generates 2.5 gigawatts of power."
probably have it up and running within a year with the hardware installed.
@@percy9228 🎯
Amazing editing!
The cuts are really stunning
The efforts we put into replacing ourselves are remarkable
I work for a transmission system operator at the highest voltage level in the grid, and the number of connection requests for new data centers we're getting is just insane!
What company?
"1.21 Gigawatts!!! Great Scott!!"
- Mark EmmetBerg
Dude you grew like 20k subs in a span of a few days. awesome!
Who ever is doing his human training is killing it.
AI explaining that AI will reach a bottlneck to come people down
Remember when a computer filled up a huge room 🤔
i like ur thinking
Moores law is no longer applicable right?
remember when women were the computers, and lots of them were sitting in a room doing calculations
@@the_original_dudeyeah they still are. Bankers, accountants
@@Instant_Nerf no, they don't calculate manually anything
1.21 gigawatts?!?!
Is just 1 nuclear reactor
When this baby hits 88 exaflops an hour.... you're gonna see some serious sh
GREAT SCOTT!!!111!!
What the hell is a jigawat?
They just need to make sure their AI gets struck by lightning every time they need to ask it something. The trouble is, we don’t know when or where it’s going to strike.
@@GeneralKenobi69420 "88 exaflops an hour"😂
“How long can the exponential curve going for?” There’s going to be a Nyquist-Shannon equivalent limit to the max data and compute you can throw at a model before you can’t improve it anymore.
For example, if you trained an ai model on flipping a coin, it doesn’t matter if you use a 1-gigawatt cluster and 100 trillion coin flips as your dataset, the model is still going to be 50% chance. The question is how do you theoretically find that limit without throwing 10s of billions of dollars at training clusters.
What if you flipped the coin 5 times. You have an odd number, you are always going to have either more heads or more tails. If you flip it 6 times, there are a total of 64 possible permutations of the flip, 2/64 are all heads or all tails, 12/64 are one heads or one tails out of the 6, 30/64 are 2 heads or 2 tails out of the 6, and finally only 20 would give you a perfectly representative result of 50% heads and 50% tails. You'd only get this result 5/16, 31.25% of the time. This actually gets worse the more times you flip the coin, if you flip it ten times, you'd actually only get a perfect 50/50 heads and tails result 252/1024 times, 24.6%. there are always more non 50/50 results than 50/50 results. The best way would be to actually flip a coin some finite set of times within a larger finite set and then measure the distribution. Say sample 100 flips one billion times. And see if the distribution matches statistical expectations. But you run into kind of the same problem, just sort of at a higher dimension. Coin flips May be harder than you think, even trying to determine if a coin is really flipping 50% one way or the other
AWS has 1GW in the Columbia river basin in dozens of sites and buildings.
He specifically stated single site. Sure a bunch of service providers have a cluster of data center which equate to 1GW.
@@Neojhun riot blockchain will finish a 1GW site in Texas this year.
Just for some counter point the human brain runs on about 20 watts of power. Maybe needing Gigawatts and billions should be an indicator that this could be a wrong approach to reaching human level AI?
you got a better approach? lets stop everything because the human brain is so much efficient??
it's all progress. you build tech so it improves other tech. the first transistors where astronomically bigger compared to leading edge now. give it 25 years of advancements
@@percy9228 I just suspect this might be a bit of an AI tech bubble that bursts but gives us great chat bot technology but not real AI.
It's a bit like the crypto bubble where we have all this technology and money to push into something that looks good but may not deliver on the hype people are building for it.
Mind you if it does go full AI it will probably happen so fast our world would change completely within days. And I do not think we are ready for that.
qwerty-np2iw There are potential lower energy computing solutions that could just be around the corner e.g. graphine, memristor and photonic technologies could totally change the energy needs of modern computing if they can go mainstream.
@@percy9228 I mean we could get serious about education and up skill the whole world population. I bet even a small uplift in collective cognitive power would make a massive difference in the world
@@christophermcdermott2443 there has never been a better time in human history as there is now for access to elite level education. This is all thanks to the internet. I've met people who learnt languages themselves using the internet.
If you think about it, even the internet consumes ridiculously amount of energy, but no one on earth will argue to shut down the internet because of the tremendous good for humanity.
we are reliant on it so much.
AI will one day be considered in the same way. AI will shape how the world will work.
Internet took a while before it was fully accepted. nowadays you need it just to apply for things.
and I don't get your argument. you don't need to hand feed kids, just show them where to look to get answers. and now with AI in future they will have a personal tutor to guide them also. and parents who care already guide and parents who don't won't. no ones got time or energy to teach another kid when they busy earning food to pay for their family.
Energy is the #1 bottleneck of everything my guy
In other words, SMRs will be ideal for powering data centres/servers and super computers.
I can just imagine a bunch of private SMRs behind walled compounds that remain powered and cool while the rest of society sweats it out during blackouts
If they ever get built.
Build a gas fueled power plant in East Texas and you’re gucci Zuck. Plug it into your zillion gigawatt datacenter and let Jerry Jones take you to the promised land.
It's not just energy production, its infrastructure as well. Even if we have the technology, it would take at least five years to build the necessary infrastructure to deliver the energy and thats being optimistic.
So essentially they would have to place it next to a plant dedicated for it.
Bruce Power up in Ontario Canada is planning on adding more reactors to their site. They are in the planning phase for an added 4800 MW. Maybe bump that up to 6800 MW and then sign long term energy contracts with Microsoft, Facebook, Google and Amazon. Its right next to a massive lake so cooling water is not an issue and there is lots of land there.
check out "Amazon buys nuclear-powered data center from Talen", so now they just need to kit it out with AI hardware. so it's max a year away.
this probably drive other tech giants to buy something similar or get locked out for 5 years. and it wasn't that expensive at $560m
Yup its playing catch up.
Technology chases math, and energy chases technology, ultimately the technology gets manufactured implemented for consumption markets either publicly or not.
Will be interesting to see what kind of hardware advancements can be made to reduce the power requirements.
Yes but only if we keep making current microchip technology. If the transition to carbide silicon chips sooner then a massive 1GW center could ve powered by only a few MW with 5 times the computing power.
"The human generates more bio-electricity than 120-volt battery and over 25,000 BTUs of body heat. Combined with a form of fusion, the machines have found all the energy they would ever need."
hedge against losing energy suppliers in "the age of wars" (which is over-hyped, but kind of has to be) , do this by buying into renewable energy & refining existing tech, when that's built you've hopefully got it structured to sell it for minimal loss or better yet break-even on it..
above is the unlikely scenario tho, you will most likely, in my eyes get a sustainable growth vehicle for this project and in effect you have AI also relying on the sun like we do, which is kind of nice.
Zuck should look at what amzn did with Talen Energy.
Amzn got almost a whole nuke unit.
Energy over time is what all works run on, and how much energy that must be spent ultimately determine the cost..
Perhaps one day, we can all have dividends from Open sourced machine production commons, where the capacity is measured in energy.
The better way for energy not being a bottleneck will be to reduce compute elements to be not based on traditional electronics, but go atomic scale and use extremely low power. Current approach in AI computing uses brute force. Soon progress will depend on finding more intuitive algorithms that converge quicker.
It is literally what we have been working on for more than 15 years in laboratories around the world. Thanks for pointing out the obvious!
Two promising prospects I have seen to make computing more efficient are graphene chips the band gap problem they experience has been improved on recently. Also AI designed chips are designed 1000× faster then human designed ones, they aren't close to the computing power of human made designed chips but at the rate they are improving will soon close that gap
That’s what quantum computing is trying to do. But most power is consumed in transformers. They take the current and split it into a lower current and heat. And then needing cooling systems to dispense of the heat. Which is quite primitive to be honest. The electronic technology needs to mature. We are still at the infant stage of the electronic revolution
I would love to see a large data center connected to a huge solar plant system that uses full power during the day and is throttled down to a minimum at night for the training over weeks.
Current AI architectures are ridiculously inefficient compared to the human brain.
The efficiencies will only improve. Performance per watt is going up.
To be fair, we still don't know how the human brain works exactly. The neuron-activation model used in NN's is an extremely simplified way of understanding the brain, that doesn't even capture 1% of its true complexity. That being said, I'm surprised just how well our current (general) models surpass most humans in the tasks they do.
@@rapidreaders7741 Yes and that is the "problem". We don't even know how human brain works. These models are not self-aware. They just imitate self awareness, and that is 2 different things.
@@CamAlert2 Isn't that technological innovation? Thanks for pointing out the obvious.
I don't think this is the right way to put it. Many people like hinton believe nature took the way of training/ learning which is very efficient in terms of power consumption , it rejected other ways which does take a lot of power but could learn from a lot of data/ resource very fast without over fitting etc etc like SGD with back propagation. Today's ai uses learning algorithm which is very efficient in terms of time to train on large dataset and which fits very well, compromising power/computation.
A gigawatt is 3 million A100s or 1-2 million H100s.
About 10x more power consumption than the biggest data centers today.
what about the new Blackwell? by the time they build a nuclear power plant you'll have a new gpu
10x on an exponential curve is not that far off
After Zuck found MMA he’s been a completely different dude. COMPLETELY different dude.
It doesn’t have to be either or, I could see both energy and compute bottlenecks being the case, particularly if these exponentials continue for as long as the “optimists”, for lack of a better word, say that they will.
So he committed to build a datacenter that runs on 1.21 gigawatts to get back to the future...
The Zuck measuring things only in Gigglewatts
I think this is the model they keep at their headquarters, much more human-like.
Now he looks a lot less like a robot.
nyc typically consumes and anywhere from 5-10 gigawatts
They should build compute/data centers with power plants, no power lines going elsewhere.
Both need cooling and security.
It's a lot cheaper if you hook it up to the grid.
check out "Amazon buys nuclear-powered data center from Talen"
I have seen ideas of a modular mini nuclear plant that you could build into a building, supposedly they are safer and easier to maintain. I haven't even heard of a prototype being built, the government regulates this stuff heavily
So, we would need modular thorium power units, customized for datacenters, build right on-site..
..about time, that the US legislators wake up and make room in the laws for such new (hybrid) installations.
He's more human now. He updated his software 😂
what if you build the datacenter on Iceland? they kinda have unlimited geothermal energy which is cheap and green
And how do you transfer it? That’s infrastructure cited as a hurdle. Solution isn’t more power it’s engineering gpus and machines to work with less power consumption.
language model startups are building LM models that use a fraction of the computing power
r/whoosh
So symbolic to see people disapearing in the background as the talk about AI progresses.
lmfao
they also appear too
wait til the overlord disappears you too.
@@brozbro I'm planning to change my haircut, no chance...
Looking way too into it little bro
Pay people to generate electricity via peddling.
Holy shit you're a genius.
If you don’t have the brain, you must have the muscles.
Human efficiency ~ 25%, diesel engine, fossil fuel power plant ~50%. We can't compete.
All that Brazilian jiu-jitsu has made him more human!
I thought about this for a while: instead of building massive data centers where AI runs, why don't we distribute the compute across the globe in a decentralized fashion? This would also de-risk the power requirements, as it is much easier to power something that requires just a bit of energy.
They probably never thought of that. You are a visionary, they should pay you trillions for your unsolicited advice.
@@unmanned_mission ah the internet. Who doesn't love it ♥️
The volumes of data being moved in AI training necessitates everything being in a small space, even moving data between machines in the same rack becomes a bottleneck. Sometimes even the speed between individual chips can become a challenge (e.g. Nvidia developing NVLink for 900 GB/s)
Because its inefficient. Parallel compute needs to be in close proximity to provide low latency for AI clusters. The AI clusters also require high bandwidth networking like InfiniBand which is prohibitively expensive over long distances.
Thank you Mark. That was a very human statement to make.
Zack looks great with that haircut
The robot/alien has hair now
You can imagine a future in Zuck's head where the physical constraint of the AI energy throughput gets lifted such that there is an actual location with a giant super computer like the one in the 2005 film The Hitchhiker's Guide to the Galaxy. LMAO that would be epic
Fuck it. Capture the sun with a Dyson sphere
Zuck has never looked better, more human or more likeable. He's aging nicely. He's letting his hair grow. He's not the centre of attention. He's in observer mode. very enjoyable to see
Yep. I do interstate nat gas BD for the southwest US. We often get requests for data center or crypto interconnects so they can power gen sets or even small power plants and they don't know what to say when they learn that we and the competitors are sold out in perpetuity. They spend years planning development assuming that gas will just come out of the ground for them when they show up. There's no capacity left to connect the western US to supply because san juan basin is depleting and getting massive expansions or new builds to permian basin approved is a huge risk today that requires politicians to put their necks on the chopping block. But at the end of the day they aren't going to build the generation to address the massive projections for AI demand growth with unicorn farts. AI is particularly going to be screwed if LNG gets opened up because nobody will bother risking these huge multi billion interstate expansions for domestic use when they can connect to far better markets at the coast.
Human Intelligence is powered by tacos.
And then we will be turned into 🔋 like Morpheus said.😬
Computing and energy go hand in hand. The better the computational efficiency, the less energy will be needed. So, yes, both are a big problem, both current and future.
The hair def makes him look more human . Also the energy fields sound just like the matrix
This new updated version of Zuckerberg is much more human-like
Each time Zuckerberg says "Next Year" a person in this world vanishes. 2:34
Facebook et al are putting datacenters e.g. in the north of Sweden close to our hydro resources (and usually cheap cooling). It gives quite few direct employment opportunities, suck up a lot of power and raises energy prices in Sweden. It kindof outcompetes outdoor spas, incentivizes improved isolation in homes, some other energy hungry industries and so forth. We do look at new nuclear but that will take 10+ years to do, unfortunately, because it's so heavily regulation, just as he says.
Wow a nuclear power plant to train the model ? 😯😯
It's cz they want clean energy. 😅
I believe this was a few years ago. Before LLM’s was as prominent as they are now. So, the zuck may not be speaking about AI as we know it right now. 🤔
You wanna fix that? Cause there is a way, I mention it on the APEC propulsion conference open mic night from the end of April.
that is why you need to upgrade everything to B200 chip
Or transfer learning improves and you just build multiple smal and cheap clusters.
Yes we might be saved. My day just got brighter. Now im a but more optimistic about the future.
Gotta get that Dyson Sphere going.
need more solar, and eventually hopefully fusion
Solar is highly inefficient. Go with nuclear fission in the meantime. Fusion is still decades into the future.
i love this guy, so cool & poised
That’s why free sustainable solar is the only way forward
now it make total sense why Tesla bought solar city a long time ago
Character and empathy, not technology, will be the bottleneck to a better world.
How would the advent of scalable/cheap quantum computing affect the equation?
For that to happen, Universities need to pump out more scientists not socialists.
@@Matlockization you’re a bot right? LOL
@@JacobMoretti-NotABot When in doubt or triggered or offended, blame a bot. Right ?
@@Matlockization well your comment doesn’t have much to do with my comment
@@JacobMoretti-NotABot I replied to your bot question.
Confirmed: Zuck drinks Zippy Water™️
Seems odd to me that at the mid to high mega scale or even giga scale why they wouldn't be generating power on site with no grid tie. But having done work at high megawatt plants its not the kind of thing that you could build overnight. As far as I know Mitsubishi Power doesn't keep a lot of M501J Gas turbines sitting in stock in a warehouse. If you had a completely off the shelf design and could write a cheque you might be able to build one in 3 years.
Can't belive nobody trying to manufacture GPU production. Must be really tough
Intel
Tesla is manufacturing DoJo, which is a modular chip designed for machine learning. They are continually adding to it, and it will be one of the most powerful super computers soon. They are planning to use it to crack full self driving.
I just read the first few comments, so fantastic to see what kind of focus people have, a bunch of Dudes worried about how a Dude looks. There is a 3 prong issue 1. As Zuckerberg said, adequate power supplies are needed 2. GPU Processing Power and 3. Backhaul of all devices that can share the data sufficiently, traditional TCP-IP structure is insufficient as is copper over UTP. I'm here for my money, he's already got his.
Hence you see a glance if the future when nuclear fusion is on... no more limit on energy since it doesn't pollute...
Mark looks better with longer hair. This is a good look for him
We could use our body heat as an energy source...
Graphene is going to replace silicon soon. This will solve the problem.
Hmmmm “The Prisoners Dilemma”? 🤔
So they either need to make energy cheaper or the chips more efficient.
Solar + battery can be built anywhere
this is why Nvidia is so important, few people are talking about the energy savings from their more efficient chips
Zuck is apparently aware that a lightning bolt can generate the 1.21 Gigawatts required to produce whatever he requires.
Well said Mark
We should send datacenters out into space to train on solar energy.
Not if you have localized fusion production. No transmission, local generation, local consumption. Use AI to solve fusion and you’re done.
Algorithms always had and will have more potential to grow at exponential rates than physical infrastructure. Eventually energy won't matter.
I leave lights on in my house all the time, I have energy to spare but nobody is asking me for it. I guess I'll let Duckerberg buy it and be responsible for the carbon footprint.
Let’s not clutter the issue with facts.
Longest I've ever listened to Zuck speak tbh.