GPT-3 vs Human Brain
Вставка
- Опубліковано 31 лип 2020
- GPT-3 has 175 billion parameters/synapses. Human brain has 100 trillion synapses. How much will it cost to train a language model the size of the human brain?
REFERENCES:
[1] GPT-3 paper: Language Models are Few-Shot Learners
arxiv.org/abs/2005.14165
[2] OpenAI's GPT-3 Language Model: A Technical Overview
lambdalabs.com/blog/demystify...
[3] Measuring the Algorithmic Efficiency of Neural Networks
arxiv.org/abs/2005.04305 - Наука та технологія
GPT-3 has 175 billion parameters/synapses. Human brain has 100 trillion synapses. How much will it cost to train a language model the size of the human brain?
Not all of human brain synapses are used for language processing, though. It's gonna be super-human.
@@haulin I was thinking about the same. Not all parts of human brain is used to get there.
Today (2020) it costs a human life to train a human brain 🧠👀 👁 👅 👄 🩸💪 🦵
thanks Lex:)
It depends on whether the lottery ticket hypothesis is verified or not at brain scale. In this case the cognitive power of a much larger brain could be reached within a much smaller one.
I suspect new search mecanisms would have to be invented to discover these optimally sized architecture .
The level of brain plasticity observed on subjects that have lost part of their brains leans toward that hypothesis .
That's interesting because, if the trend continues, it will also cost $5M to train a human brain at college in 2032
College trains the human brain to be a good obedient worker slave for the big corps.. 9 to 5 9 to 5 9 to 5 9 to 5
And it extinguishes all sense of humor.
I am certain this comment was generated using GPT-3
I don't know about that trend as you would be multiplying with 0 if you did this in any civilized country.
This comment was made by Scandinavia gang
R M L college is what helped teach me Java, Python, JS, etc... but yeah, totally a scam 🙄
These short highly focused videos are a nice mental appetizer, and its easy to set aside 5 mins to watch them between consecutive unsuccessful model training runs
lol, watching a model train is soon going to be a trend (❁´◡`❁)
on point man
lmao
actually my model is training and i am watching this video. lol
@@AbhishekDubey-mp3ys I have some model trains. They're HO scale. I don't play with them much any more, though.
It would be awesome to see your breakdowns on GPT-3. Explain to us dummies how it works!
ua-cam.com/video/SY5PvZrJhLE/v-deo.html
Yannic's video does a good job explaining the paper but might be a bit long
How about writing a gpt-3 app that explains you how it works
it's basically the auto-predict-next-word feature on your phone after a few cups of coffee
GPT-800: I need your clothes, your boots and your motorcycle 😎😂
Lol
I think that model's name is T-800 :)
Don’t forget sunglasses
@@xyzabc6741 whoosh
More likely it will be GPT-5
You forget that the 100 trillion synapses doesn't only do language, it does vision, reasoning, biological function, fine motor control, and much more. The language part (if we can isolate from other parts) probably uses a fraction of those synapses
It might be hard to quantify how many neurons are associated with language, since language, vision, hearing and touch are very much interconnected in our brains. You can't learn a language if you can't see, hear and touch.
@@postvideo97 you actually can learn a language with any of these senses (e.g. touching is enough)
The brain is so interconnected it’s hard to put a figure on how many synapses are used for a single task although estimations are good.
@@thomasreed2427 That seems like a bit of an exaggeration to me. To replicate some of the behavior of a single brain neuron (eg xor), you would need 4 of our current neurons.
Let's take 10 times that and round upwards, 40≈100, to cover it more accurately. The structure of the brain could also give an additional 10, or even 100, times requirement with our type of neurons. Remember that just giving it an additional 10 times, is 10 times its current size, i.e. it could do the same job 10 different ways.
So personally I think you might need at most 100*100 = 10 000 times larger than 100 trillion. But idk ¯\_(ツ)_/¯
Wow u guys are all yt og's
Thank you for this post. Powerful topic. Excellent description of the potential for this platform
and hurdles involved.
Wow !
Surprised to find your comment here
Hello,
Thanks for your time and efforts! I love the idea of the short videos! I'm very grateful for all of your hard work!
Good job Lex, really like the format. Thank you for sharing the knowledge.
2:58 I would love an in-depth GPT-3 video, explaining how it works, the algorithms behind it, the results it has achieved, and its implications for the future.
If you're brave you can always try to read the papers much more informative , But I call them a total waste of time cause its allot. Look for "Autoregressive model that an easy start but before that read about entropy coding - Shannon. Then you basically have a little grasp whats going on just a little.
You can always go on youtube and search for it on your own. I recommend videos that are about 40 minutes long on the subject, else they are cutting too much of the details.
it is spyware
Love these quick videos! Keep it up!
absolutely love this!!!! need more videos and a jre appearance from you to explain Gpt 3 deeply!
These short vids are great, keep them coming man, nice job!
Thanks for feedback for more guidance from me on digital currencies...
W. H. A.. T.. S. A. P. P.........+1........4..........3..........5........2........2.........4..........5.........1..........5..........6@
These short videos are so good. Thanks for sharing them with us.
I think it is important that computer scientists use the term neuron and synapse very carefully. I am a molecular biologist and to equate neurons in neural networks to biological neurons, or even a synapse, is like calling an abacus a quantum computer. I don't say this to diminish machine learning at all, I use it as a biologist, and I've been showing my whole family AI Dungeon 2 utilizing GPT-3; it really is tremendous. But there is such a large difference between computer neurons (I'll just call them nodes) and biological neurons.
Each neuron itself could be represented as a neural network with probably 100s of trillions of nodes or maybe magnitudes more, and each of those nodes would itself consist of probably thousands or millions of nodes in their own neural network. This is to say that the computation involved in determining whether there is an action potential or not is truly massive. I wish I could put this into more precise words but the complexity of even a single neuron is far, far greater than the complexity of all human systems of all times compiled into even a single object.
I will try to exemplify this using a single example in my field of expertise, microRNA. The synapse consists of multiple protein complexes that work to transmit a chemical signal from outside the cell to inside the cell. In this case, the outside signal is created by another neuron. Every one of those proteins has dozens (and probably a lot more than that) of regulatory steps along the path of its production, localization, and function. These regulatory steps happen over time and themselves consist of other molecules produced/consumed by the neuron, each of which have their own regulation.
Now let's say we have neuron 1 and it is trying to form a synapse with neuron 2. At the position neuron 1 and 2 physically interact, communication has already happened and all the necessary players (small molecules, RNA, and protein) have been recruited to this location. The moment of truth arrives, neuron 1 has an an action potential. Neuron 2 starts to assemble a brand new synapse at that location but this does not end in the production of a new synapse. In neuron 2, perhaps hours or days previous it decoded a complex network of extracellular signals that culminated in the localization of a specific microRNA at the location of this potential synapse. At the same time neuron 2 receives the signal from neuron 1, that microRNA is matured and is made active over a period of minutes. Instead of this new synapse being formed on neuron 2, this specific microRNA causes the production of protein necessary for its completion to stop and the whole process is aborted.
At every step of every process in normally functioning neurons, these seemingly miraculous processes are occurring. They are occurring in our billions of neurons over our entire lives, existing in a body of trillions of cells that are all equally as complex, communicating with each other always and for our entire lives.
I say this not to demean or lessen the work of you, Lex, or any other computer scientist. But I say this to humble us, for us to be a little more careful when we say so casually, "it's just computation."
Wow - thanks for the comment this is very interesting!
Reading this, what's truly miraculous to me is that the organization of these 100 billion neurons with 100-1000 trillion synapses into something that can reason and see and hear and feel and smell and remember can be thought to be explained by natural selection occurring over a mere 4-5 billion years. Anyone who's tried to simulate evolution in computers with a tiny fraction of variables of the real world should have some idea of how tiny that time really is to produce our brains by evolution, especially while recognizing that complexity explodes with increasing variables. It's miraculous then that the idea that we are designed and created isn't the normative claim.
@@namaan123 I guess you should read more about evolution than what's given in the comments lol. Designer, my ass
Ahammed Jafar Saadique I know I don’t know enough about evolution to defend that position with scientific evidence; I defend it by other means. That said is your confidence to suggest that you can defend the contrary position with scientific evidence?
@@namaan123 The study on the evolution of human brain is pretty wide and diverse. I don't know what I'm supposed to prove here. Btw I can link you to some cool reads you can do in your free time to broaden your knowledge on human brain evolution. And I'm sorry if I came as arrogant in the last comment.
humanorigins.si.edu/human-characteristics/brains
www.yourgenome.org/stories/evolution-of-the-human-brain
These are great Lex!! Keep em coming !
the cost of training will be nothing compared to all the money they make on selling this as a service :o
funny that the money to pay all this AI services will be produced by other machines/AI's...at the end the human is practically out of the equation...nwo...cite:max tegmark,life 3.0
Let's hope it won't be about advertisements.
@@444haluk only humans have money though
@@444haluk I'd love to see how species will fare as the sun grows into a red giant. Before you say humans won't last long with the way we are polluting the earth. Humans will survive, the numbers will vary and many may perish. But the species will survive, we are the only ones with the best chance to turn into a spacefaring civilization.
baby bean That is so wrong it’s not even funny
Loving these short videos.
Would honestly love to see some lectures or video essays on these subjects from you
You're awesome, Lex. Keep up the incredible work.
Keep making "this kind of things" please! :) Bite-sized ideas and information!
This is what I love about 2020 and the Internet. Two decades ago a channel concentrated on the eclectic scientific subjects that Lex covers would have had little activity. But I was thrilled to see that this video, released only hours ago, has a ton of comments and likes on it already, just like a typical UA-cam "video star" channel! :D
On the darker side. The millions of dollars required to train a network like GPT-3 does torpedo somewhat the "democratization" of AI initiative. And yes, in X years the power required to train a GPT-3 system might fit in a smart phone. But when that happens there surely will be new hardware as powerful to that coming "genius" smart phone, as the computing cluster that GPT-3 was trained on is to the typical computing resources the average person can afford today. Perhaps it will be some astonishing combination of quantum computing and vast distributed parallel processing (or said more humorously by Marvin in The Hitchhiker's Guide to the Galaxy, a computing platform with the "brain the size of a planet") . Maybe that's just the way the Universe is and always will be??
I'd love to see what Google's/BERT's response is to GPT-3. After all, Google has the largest amount of compute resources in the world. Plus I'm sure their newest cloud TPU's can train GPT-3 much more quicker and efficiently than the many "general purpose" GPU's this exercise by OpenAI required.
Yep, but maybe we should improve ourselves too. Great technology become nothing, when it operated by idiots without power of will. We already drag behind of instruments, that could improve our live quality and this is already insane.
Thank you lex for explaining this. I am extremely grateful for your videos and explanations
thank you for all the great content bro
2:30 Looks like Ray Kurzweil’s prediction for the singularity is tracking pretty accurately.
Don't be so confidence
Remember that was what scientist said about the TOE back in 1980's and now in 2020 we are not even close
Now this what the whole world needed, Getting a bit of an idea from different articles stating what GPT-3 is but not really we got any update or clue.👋
This is the real thing that you have talked about Lex.👌👌👌👌👌👌
Good one....😺😺😺😺😺😺😺😺😺😺😺😺😺😺😺
Fantastic video, would love to see your thoughts on the potential of this technology and how you think it will impact the world
I love these little vids!
Awesome video, I hope you do keep making more like this. One question though, you account for the improving efficiency of neural networks leading to less expensive training, but is it not also true that compute will continue to get cheaper as well?
@stack Whale if it is, that's kinda what I'm asking about. I don't actually know if compute is factored into the increased efficiency he is talking about.
I see it as two factors of the same problem. I am happy to discuss how I may be wrong as I am always eager to learn.
@stack Whale I honestly wasn't trying to "click bait" anyone. If my question doesn't apply, I am happy to discuss that. I'm pretty sure we are actually both just on different pages and discussing this through the low bandwidth of the comments section clearly isn't going to rectify that issue. I frankly really don't appreciate this comment, especially as someone who is just trying to gain knowledged.
More of these videos! Especially from a philosophical standpoint
Liked this format, small, easy to digest 🙌
Thats the price of our last invention... After that...we might just be at best associate producers on everything.
Yep because we have more reason to decide
By ourselves
The Erudite
AI sees another field to takeover: “another one” ☝️
I like the short video, just started learning to code 3 months ago. Hopefully, I'll get the chance to work on some machine learning before the programs write themselves haha
Would love for a video on Lex's work station/setup, what laptop he uses, what OS, his daily tech bag etc etc.
I thought 175 billion parameters were a lot... They are actually! Wonderful!
Amazing short video. 👍
Absolutely brilliant Lex .
To be honest, I was expecting a figure in the ballpark of the hundreds of trillions of USD, more the entire World's GDP and stuff.
USD 2.6 billion doesn't sound impossible even in 2020. Maybe I'm poisoned by reading about billions too much, and startups like WeWork being worth dozens of billions - but some company/individual investing USD 2.6 bi / USD in 2020 to have a "maybe-too-close-to-human-like" language model, or at least something that is at least hundreds or thousands of times better than GPT-3 sound feasible to me.
Since I saw openAI play dota 5v5 it was clear to me that ML is capable of amazing things. Sure there are a lot of edge cases and weird stuff, but to see human behavior (like self sacrifice for the team good, intimidation tactics, baiting, object permanence, etc) emerge from a machine was just mind blowing. It would be really nice if you did a video about it or invite someone from the team on the podcast to talk about it.
Big fan! I love your videos!
Interesting reasoning. I wondered if we can continue doubling training efficiency at this rate. But yes... things might get interesting if there is a model containing 100 trillion parameters.
Can you please do a long-form interview with GPT-3? Its responses depend heavily on the questions asked and I think you could extract some fascinating answers from it.
Thanks for feedback for more guidance from me on digital currencies...
W. H. A.. T.. S. A. P. P.........+1........4..........3..........5........2........2.........4..........5.........1..........5..........6.s
“How do you snorgle a borgle?”
GPT3: “With a snorgle.”
I really like this idea, shorts are great.
Thanks Lex!
this is great thanks Lex
Love it! Great video....MOAR
When is GPT-6 come out lex cant wait to play it
Lex could you make a video explaining if this neural network powered technology could eventually lead to an AGI??
At this time there has came out faster and memory efficient training method: There is no need to train every synapse at each iteration, but only subset of them
Very sexy
More efficient learning was covered by the video
Not all human synapses are dedicated to image processing
Imagine a computer model that does
@Ronit ganguly on the way, we're more likely to encounter a partial AGI(which may or may not be misleading) though. Are there discussions about it(rather than full AGIs)?
@Ronit ganguly that's such an optimistic spirit :) I brought up about AGI on r/machinelearning and they confused it with those sensationalist fearmongering news that make AGI seem to be real-life Terminator/Matrix/Screamers, claimong AGI is jus a fantasy and such. They think MLPs being math functions mean they can't be intelligent or the like, even though that's a bit like saying "a bunch of proteins and electric signals can't have self-awareness".
Btw AFAIK most discussions regarding AGI hazard(like on Robert Miles' channel) seem to revolve around a hypothetical 'perfect'/'full' AGI, but what about the sub-AGIs we're likely to encounter first? Would they make different mistakes due to not being as intelligent?
Btw which papers have you worked on?
And what would be the cost of actually gathering the data on which to train the gpt3?
It's not an exaggeration when people say that the most valuable possession you have is your brain...
Not for long... :)
You are your brain
@@LordAlacorn Care to explain? I heard something about neurolink but I just can't comprehend the idea of uploading your consciousnesses.
@@victoraguirre92 neurolink is outdated. :)
Search for "artificial dopamine neuron" - we just invented it, no upload, direct brain expansion is possible.
Basically we invented possibility of a better brain for ourselves.
Alacorn Thanks
There was a paper "Adaptive Flight Control With Living NeuronalNetworks on Microelectrode Arrays" by Thomas B. DeMarseand Karl P. Dockendorf. Where they connect a chips to rat neuron cells and use it to train a "literal" neural network to create an adaptive flight control in a flight simulator. The weights of the network are adjusted via killing or stimulating growth of cells at specific location via low/high frequency electric pulse. There was also another channel "The thought emporium" where the content creator cultured a petri dish full of commercially bought human neuron cells (grown from induced stem cells, so no human was harmed) and hook them up with electrodes and attempt to do some basic neural network task like number recognition with it. So technically a neural network might be replicated within a human brain with a bit more technology. Although that machine version of neural network might not be the same as the natural version of neural network.
Can you go into deep detail about how GPT-3 works? Thanks.
Lex, it's not about how complex is the box, it's about how the box can interface with the world in a close loop of changing the world and being changed by the changes it makes upon the world.
86 billion neurons in the human brain. And about 10.000 connections between neurons (action potential), so it results 860 trillions (almost 1 quadrillion) of potentials connections what is called “connectome” and somehow make you - you.
Thank you!
Yes !!!! Always learn something new everyday !! :)
Grate work!
Thanks for feedback for more guidance from me on digital currencies...
W. H. A.. T.. S. A. P. P.........+1........4..........3..........5........2........2.........4..........5.........1..........5..........6.;
I'm pretty sure linear scaling does not apply here. I think N log(N) is probably more applicable, perhaps even N^2. AI training is an optimization problem, and optimization with linear scaling is like the holy grail. I would like to see some expert input here!
I realize this video is a year old, but just wanted to point out the incredible rate of progression on this stuff. Only a year after this video was made, there are now many models in the 1-2 trillion parameter range and Cerebras claims it can handle 175 trillion parameters (price unknown to the public as of yet).
There are also open source models that can be trained on commodity hardware that achieve benchmarks close to gpt-3.
Incredible (and somehwat frightening) stuff.
Loved the video
GPT-3 Sentence completion: Humanity is...[about to become obsolete] [unprepared for what's coming] [blissfully ignorant of the future they are about to experience]
all of the above
Would you have to train a neural net on a bunch of different things ( natural language, solving math problems, ....) to achieve general AI?
Great video, what about the fact that we don't use all 100 trillion for language but many other types of cerebral processing. Do we know how many synapses are used in the brain for language processing? Or is that not relevant as you can't process in isolation....
It took OpenAI about a year and a half to go from GPT2 to 3. Figure they will continue this path with 5-10X increase as the cost to train the network drops.
Good Morning Dave..
Is training complexity for neural networks really have linear dependency on size ? Bigger network requires bigger training data and more epochs. So, IMHO it will be at least quadratic dependence (or even more)
more GPT-3 please!
Have you thought about the implications of quantum computers on Machine Learning? A quantum computer would be exponentially faster at the optimization problems required as the matrix gets larger and they're scaling much faster than I thought they would.
Our Digital On Line Helper is closer than you think! And I am talking about the level of sophistication akin in the move " Her ."
GPT-3 is able to play chess via in-context learning, but not to a mastery level. do you think GPT-x, without task-specific training, could play chess at a mastery level?
Is there a coherent argument explaining how transistors compares to synapses for total computing power?
what about quantum computing in training models? how may that affect future ML algos and its associated costs? any idea if there are works going on with quantum computing and training models?
great job
waiting for the 20k challenge eagerly
Human brain has many synapses yes, but most are static or moving incredibly slow.
Perhaps a Trillion node neural network running at say 100Hz around the outside and quicker around sound and vision processing centers could fit on a single chip and run cool.
ok what would we do with it
Info like this can tell us how long it will take to populate a city with these models seem like it's going to take longer to mass produce than to actually achieve the science
Does the number of parameters in the language model need to be the same as the number of synapses to match the human brain ? What proportion of our synapses is required for language ?
I am not sure we should compare with the number of synapses of the whole brain but more with the part dedicated to language. How big is it?
Thanks for feedback for more guidance from me on digital currencies...
W. H. A.. T.. S. A. P. P.........+1........4..........3..........5........2........2.........4..........5.........1..........5..........6.s
Check out the gshard paper. Training a 600B params model with conditional computation with a few orders of magnitude less flops.
Are they programming a self referencing algorithm into the GPT-3 and if they did would it experience "something like it is to be a GPT-3" What would it feel like to be a GPT-3?
Yes, it passes the Turning test for language models, but it doesn't know what an apple is. That aspect is what I'm watching out for. Exciting times. GPT-3 would be amazing at grading grammar papers, though.
The singularity is rapidly approaching!
Technical details, please!
Question: What in we run a gpt3 on a NEST (neural simulation technology) using folding@home computational power?
we need more GPT3 videos
That is an interesting idea that you might unlock more capabilities at 100 trillion synapses. However even at 175 billion parameters, GPT-3 was hitting a performance ceiling that had more to do with the algorithm not actually having conscious thought. In other words, simply having access to more training data and more parameters will only get you so far before you need to have conscious thought which then steps into the metaphysical...
Singularly exciting!
with a tiny fraction of Jeff Bezos' fortune we would be able to train GPT-4 today.
Bezos could become god right now
@@VincentKun no he couldnt. Even with a 100 trillion parameter network you may not reach general intelligence. A scaled up language model isnt the same thing as an AGI.
@@xsuploader i was joking about it, of course you can't develope self consciousness with this GPT3
Vincenzo Gargano Do you need self consciousness to have an agi?
@@inutiledegliinutili2308 it's kinda yes and no.
For example in Asimov's books we have Strong AI, that is basically an AGI, with self-consciousness.
But in reality we don't know even what it is consciousness ... And when it appears.
So i can't answer for sure, i could also be wrong
Please make podcast on LINUS TORVALDS .
Excelent video
Maybe in the next years this IAs will help us to design the space ships for travel to other solar systems.
how well does GPT-3 do on a Turing test?
Someone did that and posted it here:
lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html
Mind blown. We're in an overhang where so much is feasible it's just a matter of trying it out.
don't know much about ai, but i would guess a lot of quantum mechanical mechanisms happen for any brain activity, which can not be easily simulated by matrix algebra, regardless of how complex the model is. put simply, as ppl know, quantum world is highly nonlinear. basically, i feel we massively underestimate the complexity of our brain.
To some degree it might be fairer to compare only a limited subsection of the brain to GPT-3. As GPT-3 doesn't have to perform sensory processing, homeostasis regulation, motor control. It doesn't make judgments regarding threats or rewards.
At least not yet.
are the numbers adjusted for inflation?
i like this, keep going