I really enjoyed this conversation with Ilya. Here's the outline: 0:00 - Introduction 2:23 - AlexNet paper and the ImageNet moment 8:33 - Cost functions 13:39 - Recurrent neural networks 16:19 - Key ideas that led to success of deep learning 19:57 - What's harder to solve: language or vision? 29:35 - We're massively underestimating deep learning 36:04 - Deep double descent 41:20 - Backpropagation 42:42 - Can neural networks be made to reason? 50:35 - Long-term memory 56:37 - Language models 1:00:35 - GPT-2 1:07:14 - Active learning 1:08:52 - Staged release of AI systems 1:13:41 - How to build AGI? 1:25:00 - Question to AGI 1:32:07 - Meaning of life
The time marks and the definitions and wiki links are really helpful lex thanks i find mind self revisiting podcast to listen to certain subjects this and your clips are very helpful
Im still on level 1: Pandas and Numpy and many things that are you mention are terra incognita but I have to thank you for videos. You truly add value to UA-cam
I dont mean to be offtopic but does anyone know a trick to get back into an Instagram account? I somehow forgot my login password. I appreciate any tricks you can offer me
@@appletree6741 About the backstabbing part : Don’t be assured that in the coming Netflix documentary Altman will have the role of the hero & Sutkever the role of the villain. There is still a lot we don’t know … Let’s hope we won’t remember this episode as the point of no return for humankind …
PLEASE, LEX!!! Bring Ilya back on for an update interview my friend. He's the man of the hour that people really want to know more about. (I do at least)
As a veteran AI graduate from the late 90s, who later chose to switch paths, watching this interview was one of the most enlightening experiences I have ever had on discussions related to the subject. Very rare to come across an interviewer who is extremely intelligent, exploring also a most intelligent interviewee's mind with such a thorough set of brilliant and cunning questions that the latter honestly answers without avoiding, and to a degree that provides an almost complete satisfaction for the viewer, leading to a formidable conclusion. THANK YOU SO MUCH!
I find it odd why someone would leave AI to begin with - I studied computer science for 3 years and left because the curriculum was boring and I was more interested in quantum encryption (which also wasn't taught, I just ran into it by accident.) AI was never mentioned and I didn't even know what it was until very many years later despite doing a research project with AI (I didn't know the software I was using was AI, I just found it handy.) I actually returned to the field precisely because this time all the cool shit is included in the curriculum.
Please bring Ilya back on the podcast! There has been so many advances and unforeseen things happening, and I wish to hear what this brilliant man has to say about the state of AI/AGI today and in near future.
I can't express how valuable these podcasts are. Listening to two very ,very intelligent humans chat and ask questions that make one another pause and ponder is golden.
It's like the Feynman quote goes, "Elementary means that very little is required to know ahead of time in order to understand it, except to have an infinite amount of intelligence."
Definetely! Especially since GPT-4 is out now! He already talked about the underestimation of deep learning at 29:35 but I'm curious about what he has to say now!
The quality of interaction in this interview was exceptional. Good questions that are challenging and pique the interest of the interviewee. And Ilya had exceptional answers as well.
such a shame Ilya Sutskever went on radiosilence after the whole debacle. He is like the most interesting person to listen too, no BS at all, just beautiful thoughts.
I love Lex because he captured this interview 3 years before generative AI became a household world and already knew who was behind it. and Of course Ilya and insightful genius.
What amazes me is how Ilya's face remains emotionless and serious any time Lex jokes about things like mortality, conscious AI, etc. He conveys that solving these things is not so much of a joke at this point, but a possibility in the observable future.
@@ymbhiojtukburtbuyt568 There are some cool episodes but it's definitely much easier to understand because they don't go into technical details. I think end boss was refering to difficulty rather than quality.
I like to click like on things that I feel compassion for. That doesn’t always make me laugh, or give me an insight that I sort of separate from the rest. It gives me experience of something that makes sense for me is there. We always go for more, we do not have some final goal we are thrived to reach, the only constant thing is change itself. We are about the change, not the goal.
That thought popped into my mind a couple of times too, he'd be superb, hope he does one day, he has those genius insights from someone who really worked in the stuff, understood it, and kept digging.
Hey Lex, with today's big news at OpenAI, it'd be awesome to have another interview with Ilya! I'm really curious to hear his thoughts on the recent leadership change and the new direction at OpenAI🤔
Really nice to see how rigorous Ilya is when speaking/thinking about any topic, it is very clear that he is always very precise with the definitions of the words he uses so he can actually arrive at some meaningful conclusion.
Great interview. Ilya is a 5 star guest. I love his focused response to the questions, he is really dialed in on the issues, and super quick with answers. What Ilya said: When the algorithm can determine the most likely truth. That will be a huge moment in moving forward IMHO.
Really great interview. Ilya is so precise in his language, and demands such precision in conversation, it's truly elevating to the listener. Great content for where I'm at in my data science path, very interesting to listen and compare to developments in neural networks in the 4 years since this interview. Quality interview! 👏👏👏
Its amazing to hear them about unification 2 years ago. How text and image generation models maybe related. Love it. It will be awesome to have him back and reflect.
What an enlightening conversation with Ilya! His perspective on decision-making, regret, and pride is truly thought-provoking. It's interesting to hear that despite his numerous academic accomplishments and breakthroughs in computer vision and language, he believes that happiness largely comes from our perspective on things. This is a powerful reminder that our outlook can greatly influence our sense of satisfaction and joy. I also appreciate his humility in acknowledging the uncertainty of these complex topics. The closing quote from Alan Turing about machine learning is a fascinating concept to ponder. Looking forward to the next podcast!
I have really enjoyed the diversity of guests on this podcast but I can't say I haven't missed deep learning :). Been waiting for this for ages; ever since I saw Ilya's lecture on this channel, in fact.
1:05:45 “there might be some unification towards a kind of multi-task transformers that could take on both language and vision tasks that would be an interesting unification” it is surreal to be living in the future where this exists… what a time to be alive! 😄
One of the most interesting things to pay attention to is when Ilya says things with absolute certainty and when he says he's not sure or doesn't know. Important high-level decision making for a researcher at his level.
Next step in podcasts: Have 2 guests come on . And we see them disagree and agree on various topics and that would be so amazing to watch. I'm sure the guests would enjoy it as well.
He dunked on 99.99% of humanity with one simple phrase. If that's Sutskever's view, I am not very optimistic about what a general AI would think about us.
Lex, where is Ilya? It's so important for him to be interviewed again, he's vanished with no explanation. Nobody seems to know what he's doing and it's been too long. Please find out, he's got the greatest mind in AI.
One of those podcasts I need to listen on 1x speed. Thanks for what you do Lex! Seems like you've nailed the production quality level on what looks like a mobile interview setup.
I love it how Lex is so romantic and is not afraid to appear vulnerable. Always asking about meaning of life, that’s very Russian somehow. He could try investigating all those priors in his own neural networks which led to reasoning in the way that he does. :) And I mean it in the nicest possible way.
Thank you Lex, it was an amazing podcast. It is rare to see that level of understanding of the subject in the host, so that questions aren't meaningless or general. I feel I have learned a lot from this podcast.
I should really listen from 1:30:00 onwards again. That was the question about the meaning of life and Ilya gave an incredibly well adjusted answer. I really really should listen to that again
Hey Lex, amazing seeing Ilya on the podcast once again. Can you please look to get a podcast with some of the students of greats like Marvin Minsky and John McCarthy etc. A lot of them went on to win great awards and achievements like Turing award in the field. Few of them are still there at MIT. You may be able to get in touch with them. Most of them are maybe in their 70's-80's. It would be great to hear their thoughts on the current state of Artificial Intelligence and some of their experiences from the early days.
Hello friend! I was a mentee of Marvin, and a colleague of Marvin's protege, Push Singh. Marvin thought very highly of the work on neural nets by Hinton and co. and thought it was genuinely 'fresh' progress in AI, a field that had stymied in the 2000s. Marvin very much believed that human intelligence was based in a 'Society of Mind' as he termed it, and his last work The Emotion Machine refined that further into the idea that Mind is really a collection of 'machines' and processes. It aligns quite well with a possible explanation of Double Descent, which is that, as model complexity grows, error increases because the model is full of half baked ideas, and then, as model complexity grows further, some of those ideas mature into successful hierarchical abstractions, or a 'Society of Mind'. I think as we continue to explore LLMs and their potential, what we learn about how they learn is going to teach us a helluva lot about how we think and what mind is. That's the nature of Mirrors. I for one am super excited to be living through this exciting time in history. Blessings!
You can almost hear the echo's of similar types of discussions had by Einstein, Heisenberg and Bohr in the late 20 & 30s when exploring quantum mechanics and the underlying models of the universe and god. A great discussion....especially now, looking back 3 years.
Sam is the perfect business guy, Ilya is the perfect scientist. They’re like ying and yang. Lex is the perfect interviewer. This is a legendary interview, I loved every second.
I really enjoyed this conversation with Ilya. Here's the outline:
0:00 - Introduction
2:23 - AlexNet paper and the ImageNet moment
8:33 - Cost functions
13:39 - Recurrent neural networks
16:19 - Key ideas that led to success of deep learning
19:57 - What's harder to solve: language or vision?
29:35 - We're massively underestimating deep learning
36:04 - Deep double descent
41:20 - Backpropagation
42:42 - Can neural networks be made to reason?
50:35 - Long-term memory
56:37 - Language models
1:00:35 - GPT-2
1:07:14 - Active learning
1:08:52 - Staged release of AI systems
1:13:41 - How to build AGI?
1:25:00 - Question to AGI
1:32:07 - Meaning of life
1:32:07 - Meaning of life
The time marks and the definitions and wiki links are really helpful lex thanks i find mind self revisiting podcast to listen to certain subjects this and your clips are very helpful
Im still on level 1: Pandas and Numpy and many things that are you mention are terra incognita but I have to thank you for videos. You truly add value to UA-cam
Thanks Lex
I dont mean to be offtopic but does anyone know a trick to get back into an Instagram account?
I somehow forgot my login password. I appreciate any tricks you can offer me
Can we have this person again on podcast and discuss further on the recent advancements in field of Ai ?
I will be very excited to watch 👍
Yes, me too.
@@joefish6546 here Ilya is a few days ago: ua-cam.com/video/Yf1o0TQzry8/v-deo.html&ab_channel=DwarkeshPatel
Your fans would love to have Ilya back Lex!
this podcast went a bit under the radar 2 years ago, but it's so fascinating and a lot of points make more sense today
Given what's happened in the last few days, it would be great to have him back on the podcast
I actually just came here to say this after I saw this on my recommend😭
He might have plenty of time if OpenAI ousts him after backstabbing Sam and then backstabbing the board and pretending he was never a part of it
@@appletree6741 if they oust him you think he will just sit there and play video games? elon will probably take him.
@@appletree6741 About the backstabbing part : Don’t be assured that in the coming Netflix documentary Altman will have the role of the hero & Sutkever the role of the villain. There is still a lot we don’t know … Let’s hope we won’t remember this episode as the point of no return for humankind …
I am a big believer in self-awareness he says. Following that philosophy, he owes us his personal testimony on what has been happening.!
Lex you gotta invite him again. Ilya is probably the most brilliant mind in computer science / AI and we need to see more of him documented.
Still feel that way after he destroyed OpenAI overnight?
@@FoeverDirt jesus christ calm down. nothing is destroyed
Elon Musk is the most brilliant mind in CS (after me of course) and he never even took CS.
@@magnetseclol.
bruh stop capping. he is incredible but not the best. there are many scientists who are as good as him.
PLEASE, LEX!!!
Bring Ilya back on for an update interview my friend. He's the man of the hour that people really want to know more about. (I do at least)
We want to know WHAT DID HE SEE?
@@lkrnpk Exactly!
please get ilya asap
As a veteran AI graduate from the late 90s, who later chose to switch paths, watching this interview was one of the most enlightening experiences I have ever had on discussions related to the subject. Very rare to come across an interviewer who is extremely intelligent, exploring also a most intelligent interviewee's mind with such a thorough set of brilliant and cunning questions that the latter honestly answers without avoiding, and to a degree that provides an almost complete satisfaction for the viewer, leading to a formidable conclusion. THANK YOU SO MUCH!
Well said.
👏👏👏
Have you watched the Wolfram interview? Killer stuff dude.
Absolutely 👏
I find it odd why someone would leave AI to begin with - I studied computer science for 3 years and left because the curriculum was boring and I was more interested in quantum encryption (which also wasn't taught, I just ran into it by accident.) AI was never mentioned and I didn't even know what it was until very many years later despite doing a research project with AI (I didn't know the software I was using was AI, I just found it handy.) I actually returned to the field precisely because this time all the cool shit is included in the curriculum.
Please bring Ilya back on the podcast! There has been so many advances and unforeseen things happening, and I wish to hear what this brilliant man has to say about the state of AI/AGI today and in near future.
The calls for Ilya to come back must be resonant now more than ever.
I can't express how valuable these podcasts are. Listening to two very ,very intelligent humans chat and ask questions that make one another pause and ponder is golden.
Love the way Ilya responds every answer in such a simple and first principe way .
"All concepts are very easy in retrospect" 9:27
Apparently even something as simple as the number zero took quite some while to discover, those things are much harder than people think.
It's like the Feynman quote goes, "Elementary means that very little is required to know ahead of time in order to understand it, except to have an infinite amount of intelligence."
Can we have Ilya again on podcast? It would be amazing to hear his perspective now
Lex’s podcast series needs to be archived and protected at all costs. It is an absolute compendium and distillation of humanity.
It's just talk radio on a web page, buddy
@@therainman7777 cool story bro
Lex will use it when he creates gis own AI..Lex AI
DAll-E 2 is too interesting! Please get Ilya back for another round of interview.
Definetely! Especially since GPT-4 is out now! He already talked about the underestimation of deep learning at 29:35 but I'm curious about what he has to say now!
"that's a good question"
Probably the most repeated thing said to Lex by most guests. Really shows the quality of the show! Thank you.
Definetily: Ilya II strikes back please Lex. I know maybe is not that easy to get... So, when you can... but you two talking now would be monstrous
Great interview - my only complaint is that it wasn't longer.
Can we have Ilya again? Nice to know about the vision for OpenAI
The quality of interaction in this interview was exceptional. Good questions that are challenging and pique the interest of the interviewee. And Ilya had exceptional answers as well.
such a shame Ilya Sutskever went on radiosilence after the whole debacle. He is like the most interesting person to listen too, no BS at all, just beautiful thoughts.
What debacle?
I love Lex because he captured this interview 3 years before generative AI became a household world and already knew who was behind it. and Of course Ilya and insightful genius.
What amazes me is how Ilya's face remains emotionless and serious any time Lex jokes about things like mortality, conscious AI, etc. He conveys that solving these things is not so much of a joke at this point, but a possibility in the observable future.
This aged well
Ah... Human made horrors beyond my comprehension
@@chrisant1715 Oh wait another 12 months, we've seen nothing yet😅
or... he's a psycopath lmao
Yes, compared with Elon Musk who has a great sense of humor , Ilya is expressionless….robotic computer guy.
Lex, please have Ilya back now!
Going from Rogan Experience episode to this, is like playing a game on level 1 and skipping all the way to the final boss
Daniel Clancy It’s entirely possible xD
@@ymbhiojtukburtbuyt568 There are some cool episodes but it's definitely much easier to understand because they don't go into technical details. I think end boss was refering to difficulty rather than quality.
This interview would be about half way thru the game. The final boss would definitely be eric Weinstein.
So you are comparing basically Gold with Shit...
@@Need_better_handle or stephen wolfram
I like to click like on things that I feel compassion for. That doesn’t always make me laugh, or give me an insight that I sort of separate from the rest. It gives me experience of something that makes sense for me is there. We always go for more, we do not have some final goal we are thrived to reach, the only constant thing is change itself. We are about the change, not the goal.
Ilya would make such a great professor. I like how he explains the deep learning concepts.
That thought popped into my mind a couple of times too, he'd be superb, hope he does one day, he has those genius insights from someone who really worked in the stuff, understood it, and kept digging.
You should bring this guy back, I’d really like to hear his perspective on he left OpenAI and his roadmap for developing a more secure platform
Ilya`s meaning of life answer was one of the best so far. Beautiful interview thank you, Lex Fridman.
With all that has transpired in the world of AI over the last few months (2023) you should have Illy back on the show! Было бы очень очень интересно!
Now that he's left openai, you should have him back and see what his thoughts are now and whether they've changed
Hey Lex, with today's big news at OpenAI, it'd be awesome to have another interview with Ilya! I'm really curious to hear his thoughts on the recent leadership change and the new direction at OpenAI🤔
UP
this ^
Hey lex with today’s big news I think would be a great idea to interview the openai board’s gaping asshole
Really nice to see how rigorous Ilya is when speaking/thinking about any topic, it is very clear that he is always very precise with the definitions of the words he uses so he can actually arrive at some meaningful conclusion.
One of the most underrated episodes this is gold!
It would be awesome to have a new podcast with Ilya Sutskever
Great interview. Ilya is a 5 star guest. I love his focused response to the questions, he is really dialed in on the issues, and super quick with answers. What Ilya said: When the algorithm can determine the most likely truth. That will be a huge moment in moving forward IMHO.
We need round 2 with this genius
Really great interview. Ilya is so precise in his language, and demands such precision in conversation, it's truly elevating to the listener. Great content for where I'm at in my data science path, very interesting to listen and compare to developments in neural networks in the 4 years since this interview. Quality interview! 👏👏👏
Damn this is probably the coolest guest you've had on the podcast so far 🤩 maybe one day you will interview Andrew Blumental
Its amazing to hear them about unification 2 years ago. How text and image generation models maybe related. Love it. It will be awesome to have him back and reflect.
Lex,can you please make another conversation with Ilya ? It is needed 😫😫
Jjst listening through this now i cant believe there arent newer decent interviews with him, we need him back. I wonder if he will do it
1:27:34
Lex: And a board can always fire a CEO.
Ilya: Yup 😅
I burst out laughter at this line!
perhaps ominous that they can't .... as more and more spectrum people controlling big tech
Lol
What an enlightening conversation with Ilya! His perspective on decision-making, regret, and pride is truly thought-provoking. It's interesting to hear that despite his numerous academic accomplishments and breakthroughs in computer vision and language, he believes that happiness largely comes from our perspective on things.
This is a powerful reminder that our outlook can greatly influence our sense of satisfaction and joy. I also appreciate his humility in acknowledging the uncertainty of these complex topics. The closing quote from Alan Turing about machine learning is a fascinating concept to ponder. Looking forward to the next podcast!
Lex, we need Ilya back on more than ever...
Ilya is a fantastic guest -- another episode would be much appreciated!
the question is not "what would you ask an AGI system"
the question is "what can an AGI ask us to make us recognize it as being AGI?"
vannallezzennogwat yes
An outstanding interview - thankyou to both interviewer and interviewee. The questions were as thoughtful as their answers. Super rich.
I have really enjoyed the diversity of guests on this podcast but I can't say I haven't missed deep learning :). Been waiting for this for ages; ever since I saw Ilya's lecture on this channel, in fact.
We demand to have Ilya again.
When a genius in his field( computer science) talks with a super genius in his field... this is beautiful!
ilya inspired me to lock down my brain to achieve accuracy in my thoughts and words.
Not many people can converse this way. I was transfixed and concentrating as hard as I could.
Lex, please invite Ilya again to the podcast!
1:05:45 “there might be some unification towards a kind of multi-task transformers that could take on both language and vision tasks that would be an interesting unification” it is surreal to be living in the future where this exists… what a time to be alive! 😄
It's so rewarding to see you two talking .
One of the most interesting things to pay attention to is when Ilya says things with absolute certainty and when he says he's not sure or doesn't know. Important high-level decision making for a researcher at his level.
Thank you for another brilliant guest, Lex! So much yes.
Want to see Ilya again on this platform
Next step in podcasts: Have 2 guests come on . And we see them disagree and agree on various topics and that would be so amazing to watch. I'm sure the guests would enjoy it as well.
I would love a Round 2 with Ilya. How can we create an unbiased curriculum for AI? Would it be the language of Math?
Lex: "humans continue to impress me"
Ilya: "Is that true?" (Cue apocalyptic robot movie intro music)
lol my favorite part!
timestamp by anychance?
@@WeMakeItRainz 28:06
He dunked on 99.99% of humanity with one simple phrase. If that's Sutskever's view, I am not very optimistic about what a general AI would think about us.
Another conversation with Ilya would be very interesting, given recent events
We want more podcasts with Ilya
It's really cool watching you when you're operating fully within your lane.
He is not. Both of them are playing around as this is supposed to be accessible .
1:27:36
Lex: "and the board can always fire the CEO"
OpenAI: "and we took that personal"
We need to have him again on the podcast, so much has happened since. @lexfridman please make it happen.
1:32:21 Ilya Sustkever at 100% CPU (all cores at 100% and the UI-thread hangs for a second)
(≧▽≦)
We want round 2
Lex, where is Ilya? It's so important for him to be interviewed again, he's vanished with no explanation. Nobody seems to know what he's doing and it's been too long. Please find out, he's got the greatest mind in AI.
Lex, it would be great if you could arrange a third round with Ilya Sutskever!
Let’s bring him again to the podcast.
40:30 Lex didn't realize Ilya had just explained double descent to him. I like how Ilya explains it again in simpler terms.
😅
Great interview, thanks Lex for the amazing questions and Ilya for answering them!!
I like how Ilya is not afraid to say "I don't know"
That's straight from Talmud: if you want to learn, first teach your mouth to say "I don't know".
One of those podcasts I need to listen on 1x speed.
Thanks for what you do Lex! Seems like you've nailed the production quality level on what looks like a mobile interview setup.
😊😊
😅😅😊🎉🎉🎉
🎉🎉🎉🎉🎉
Round 2 is urgently needed
I love it how Lex is so romantic and is not afraid to appear vulnerable. Always asking about meaning of life, that’s very Russian somehow. He could try investigating all those priors in his own neural networks which led to reasoning in the way that he does. :) And I mean it in the nicest possible way.
That’s what make him good interviewer and human
beautiful way to put it. Lex is pure.
This was one of the best podcast episodes in history! Instant classic.
Bring back Ilya!
Thank you Lex, it was an amazing podcast. It is rare to see that level of understanding of the subject in the host, so that questions aren't meaningless or general. I feel I have learned a lot from this podcast.
We need a round 2!
Time for a new interview?
I should really listen from 1:30:00 onwards again. That was the question about the meaning of life and Ilya gave an incredibly well adjusted answer. I really really should listen to that again
learned more during this interview than 10 books on the subject. fascinating! thank you!
Hey Lex, amazing seeing Ilya on the podcast once again.
Can you please look to get a podcast with some of the students of greats like Marvin Minsky and John McCarthy etc. A lot of them went on to win great awards and achievements like Turing award in the field. Few of them are still there at MIT. You may be able to get in touch with them. Most of them are maybe in their 70's-80's. It would be great to hear their thoughts on the current state of Artificial Intelligence and some of their experiences from the early days.
Oh yeah, great idea!
Hello friend! I was a mentee of Marvin, and a colleague of Marvin's protege, Push Singh. Marvin thought very highly of the work on neural nets by Hinton and co. and thought it was genuinely 'fresh' progress in AI, a field that had stymied in the 2000s. Marvin very much believed that human intelligence was based in a 'Society of Mind' as he termed it, and his last work The Emotion Machine refined that further into the idea that Mind is really a collection of 'machines' and processes. It aligns quite well with a possible explanation of Double Descent, which is that, as model complexity grows, error increases because the model is full of half baked ideas, and then, as model complexity grows further, some of those ideas mature into successful hierarchical abstractions, or a 'Society of Mind'. I think as we continue to explore LLMs and their potential, what we learn about how they learn is going to teach us a helluva lot about how we think and what mind is. That's the nature of Mirrors. I for one am super excited to be living through this exciting time in history. Blessings!
Really interesting to listen to this after GPT4 is out
You the man, Lex.
one of the most brilliant interviews I have ever watched! Thanks for uploading.
You can almost hear the echo's of similar types of discussions had by Einstein, Heisenberg and Bohr in the late 20 & 30s when exploring quantum mechanics and the underlying models of the universe and god. A great discussion....especially now, looking back 3 years.
This is wonderful!
Lex always has the movers and shakers first! Thanks Lex!
Looking at this from 2023 makes him sound like a prophet
I learned so much from this conversation, thanks Lex!
It's about time to have him back to the show Lex
the most underrated interview on internet
I think this is the best interview that I've ever seen.
Sam is the perfect business guy, Ilya is the perfect scientist. They’re like ying and yang. Lex is the perfect interviewer. This is a legendary interview, I loved every second.
Thank you for this! Currently writing a project on CNNs. What a wonderful timing
Time to Interview Ilya again!
We NEED Ilya back ❤❤❤❤❤