John Schulman (OpenAI Cofounder) - Reasoning, RLHF, & Plan for 2027 AGI
Вставка
- Опубліковано 10 чер 2024
- John Schulman on how posttraining tames the shoggoth, and the nature of the progress to come...
Timestamps:
00:00:00 Pre-training, post-training, and future capabilities
00:17:21 Plan for AGI 2025
00:29:43 Teaching models to reason
00:41:14 The Road to ChatGPT
00:52:37 What makes for a good RL researcher?
01:01:22 Keeping humans in the loop
01:15:39 State of research, plateaus, and moats
Links:
Apple Podcasts: podcasts.apple.com/us/podcast...
Spotify: open.spotify.com/episode/1ivz...
Transcript: www.dwarkeshpatel.com/p/john-...
Me on Twitter: / dwarkesh_sp
Sponsors:
If you’re interested in advertising on the podcast, fill out this form: airtable.com/appxGOvFLDLP5dlz...
- Your DNA shapes everything about you. Want to know how? Take 10% off our Premium DNA kit with code DWARKESH at mynucleus.com/
- CommandBar is an AI user assistant that any software product can embed to non-annoyingly assist, support, and unleash their users. Used by forward-thinking CX, product, growth, and marketing teams. Learn more at www.commandbar.com/ - Наука та технологія
Looks great my man, a challenge to one up yourself is to get Ilya back on the show now!
And Alex!
why ? Ilya is proven to be on an ego trip. I would get more genuine , non political people
@@irshviralvideohow has he “proven” to be on an ego trip?
@@HarpreetSingh-xg2zm they guy is political. he was part of the group of liberal left that tried to get rid of sama. We need people who can actually deliver impact to the world. The guy is more of the academic type who likes to take the lime light but doesnt do the real nitty gritty work.
@@irshviralvideouhh, were you under the impression that sama’s ousting was the result of him being outed as a neoreactionary and that the whole thing was orchestrated by blue-haired sjws? ilya doesn’t even *have* hair!
Your podcasts with these researchers are the most valuable learning resources for people who want to gain a deeper intuition about Gen AI and understand where it's going. Thank you!
Dwarkesh seems to be really surprised by Open AI's blatant lack of any concrete plan for AGI alignment.
The weak answers were pretty stunning. He hasn’t read their own white paper.
It’s all pretty frankly terrifying. Especially with the two biggest safety advocates (Ilya and Jan) resigning as of this morning.
I was never optimistic they would solve alignment, but now the nightmare is deepening.
I'm a big fan of, y'know, _not_ building things that are designed to replace humans wholesale, especially if we don't have any control over what they will do once they exist.
I doubt they will make their plans public
To be fair, some people have an area to focus on, and they do that. Very well. If everyone worked on everything, it would be a hot mess. That's not to say they do have a good plan, but you can't blame someone that works in a different area too much.
John is Berkeley's pride and joy. That man will go down in history as seminal to all the modern AI/ML developments in a way Newton was for physics. Mark my words.
Here's one for you; considering the amount of AI generated information that is and will be created, how will future generations 100s of years from now actually ascertain who and what is real? Especially if governments end up taking control of these systems
This guy being head of alignment is EXTREMELY worrying. Holy moly he has no idea what he's talking about as evident from the end of the plan for agi 2025 section of the video
John seems like such a chill friendly guy. Good vibes! Glad he is leading the way!
He seems to be pretty chill about alignment 😂
But I am not sure if we actually need it or if that’s just some kind of a narrative.
I am hopeful AI will be friendly to its creators.
I love how he casually stated that AGI is near early in the interview. Like he’s talking about a new car model or the new iPhone lol
AND he doesn’t strike me as a “hype guy” either so… yeah. Wild times ahead
Great delving there. Thanks guys.
I love how he pushes the arrival of AGI back from the unrealistic next year to the entirety plausible two or three years out...
great episode, john schulman was interesting. i appreciated you pressing him on his view that dangerous AGI could emerge within "two or three years", at least with some likelihood where he found this topic worth discussing. i don't have enough info for a strong opinion on that myself, but i've noticed it's almost a trope to point out a mismatch between some AI researchers' views on AGI timelines and the lack of clearer thoughts or action one would expect if they genuinely believed it was urgent. however, the frequency of this observation doesn't make that less strange. john schulman is doing amazing work though, and i'm glad he came on the podcast :)
GG Dwarkesh, getting all the cool guests. How do you do it man, bro's an insider lol
it’s probably been a chain of referrals from previous guests
Something he said in a previous podcast: Send your interview request email with deep, well thought out questions. People like Dario Amodei or John Schulman get 50 of those emails a day, so you really need to stand out.
That, and Dwarkesh has a reputation now.
Because people in tech watch these interviews. Notice how they all say they are fans of the podcast.
Because he is an insider, especially now
I mean that's how podcasts work, even if you only got 100-1000 subs you can still get big names. Just show interest in topics they like.
People like to talk about what they do regardless of the host, they aren't picky.
Atp, Dwarkesh can get literally anyone on the pod.
Another great episode. Thanks for such wonderful content.
Finally a scientist, not a CEO, not a hype man, an actual expert!
Crushing it with the guests!
Glad to have enthusiast at the forefront!
I could sense Dwarkesh frustration building up in the "Plan for AGI" segment as he couldnt get a straight or more in depth answer. I guess John is not used to being on camera, seemed really nervous. Either way thanks for the podcast and thanks to these amazing scientists building our future, lets just hope internally they have better answers regarding safety (althought its looking grimmer than ever after the Superaligment team situation).
Future? This is the end.
@dwarkesh, thank you for an amazing podcast. One question that I would like to see being asked is, how to evaluate and ensure that these models are performing as intended? The standard benchmarks wouldn't work going forward (contamination, or does not make sense on these tasks). Building and creating models is fun, but I believe that evaluation should also go hand-in-hand while building :)
Thanks!
Great, you are rocking Dwarkesh.
This is one of the best AI interviews I’ve seen. Much clearer view of the near future of AI.
bro after watching 20mins of 90mins interview
@@kacper9081 the video is so good you can watch it in 20 min!
John Schulman doesn't do that many public appearances, but his intuitions have really stood the test of time.
FYI I’m having a lot of good results implementing the MCTS with LLMs mentioned in your Demis interview. I feel like the current best model capabilities are underestimated when looking at my results. Even tags work with Claude haiku with a max of 3 retries (meaning you can search a lot of state space with a little $)
cool story nerdoid
Can you explain a bit more? You’ve set up your own MCTS implementation that works with the Claude API?
What are your results ?
@@therainman7777 I’ve been working on autonomous repository generation to take a text prompt and return a production deployed web application. Yes, it uses the MCTS approach that Demis stated would be a likely path to AGI. It generates the full repo but need to improve the build/test loop a bit more to get a result that doesn’t contain slight errors. Still impressive and I think I can get it working soon with the current Claude 3 models.
How does it use MC tree search? What’s the state space?
Love Dwarkesh. I got burned out by many podcasters over the years, but he’s refreshing and focused, while being approachable.
Good stuff, great guests
Yes these models are probabilistic, however that is an objective function. It is not entirely how models are learning. We are modeling various aspects of the world that result in the production of logits. Models learn things like emotional intelligence and reasoning. Stop thinking of a model as a mass of weights. Instead think of them as a collection of coordinated subnetworks of weights that learn functional areas -- not statistics. A model starts as a chaotic mass. It is molded over time. What gets molded are the aspects of the world that have been seen.
Based
This is absolutely correct and unfortunately very few people seem to get it.
Yep, models are probabilistic in the same way that a sports commentator predicting the outcome of a game is...
There is clearly some underlying "understanding" and reasoning going on. Still not near human-level, but a big step up from anything we've had before.
This seems to be more of an ideological question. Some people will never admit that AI can understand anything. Even if it is undeniable, it will not be a 'real' understanding but 'imitated' understanding.
great interview! if you want cleaner audio try reducing mic gain to avoid clipping ( it can be normalized later to get full volume)
Really fantastic interview. I think there are so many hints of what to expect in this talk that you could almost predict what the next couple of models are going to look like, especially the long timeline RL, post-training vs pretraining mix, especially with regard to reasoning, models that are more aware of their capabilities. I also found the stuff on learned gating quite enlightening. It was interesting to hear a different perspective to Ilya's (Ilya tends to talk about compression whereas John speculated about libraries of circuits, which is more about the mechanics of how that is actually achieved). And of course it is fun to speculate about how this might have been harnessed deliberately to improve the fundamental technology itself and how this might improve interpretability. And the hints about using in-context learning with long context are probably hopelessly underexploited by people trying to get more out of these models, since we are all so used to shorter context. I'd love to see more material like this but of course it is hard to find vs the usual nonsense speculating about AGI being developed in some bunker and how every new tool "shocked the entire industry", etc. The occasional bit of intellectual stimulation goes a long way. Congrats on researching this well enough to ask the really interesting questions, and provoking equally interesting answers. And congrats to John for saying the interesting things, modulo one obvious slip, without having to resort to "I can't talk about that" every other sentence!
Quite a few nuggets of information I think weren't public beforehand in this, great interview! (That the 'Chat' finetune still wasn't the main focus even well into mid-2022).
Great. Re alignment please have Vitali Vanchurin on. IMO the field has the situation backwards; AGI will be an alignment damp squib because we have always been subagents of a learning universe and we are and have always been controlled by natural forms of intelligence rather than having control.
Thanks!
17:21 This was such an unsatisfying and worrying answer about planning around risk. Dwarkesh tries to push him to get any kind of concrete answer and barely a couple minutes of hand waving about "slowing down and being careful," he's saying something like "well, if we solve the alignment problem it will be great."
This is a ridiculous attitude to have. I don't care how fun and exciting it is to build super powerful tools, if you can't stop them from eating the planet you don't get to smile about it.
Just notice! Notice how weak this answer is! Notice how little people seem to be taking this seriously or even thinking about it all that hard!
Sandboxing!? Sandboxing!? We had this argument like 10 years ago, and - as far as I'm aware - we basically settled on the answer "it won't save you."
Coordination? About what? Is everyone going to coordinate to burn their GPUs and demand the international community ban further sale of GPUs? If not that, what else? What would possibly save you at that point if you're just "plug in more computers" away from something that can wipe out humanity and has no mesa-optimizer internally planning around not killing everyone.
This is how we die I guess. A bunch of people who think that utopia is totally reasonable and close in our future, but existential risk is super weird and therefore unlikely.
Yep. Not pleased about that. Hope there's an afterlife so we can all sort out this stuff in hindsight and these people can look at what they did and feel regret.
Have you watched Robert Miles' stuff? If not, you might enjoy his much more serious take on the subject of formal proof for alignment
@@jaiveersingh5538 Oh, yeah, thanks for the recommendation, but I'm an ancient AI Alignment follower. I saw Rob Miles for the first time when he was still on Computerphile.
When I reference the old arguments about sandboxing I'm calling back to forum discussions I followed when the topic was already like 8 years old back in ~2016.
I wasn't around in this space for the Singularity Institute + creating Friendly AI era back in like 2008-2009, but yeah, I'm not sure if Rob started reading The Sequences before me... maybe, I'd have to check when the ebook collection came out. He did probably finish reading them before me though, I was off/on for a while until Rationality: From AI to Zombies came out.
I know we're a long way off from actually Aligned AI. Even systems you can keep from blowing up into an unaligned ASI seem pretty hopeless to create anytime soon... Yeah, I could go into detail why - but like, if the perspective on safety represented in this conversation was coming from a BRIDGE ENGINEER who's being asked if their design was safe - you'd kinda expect that bridge to fall over.
AI Alignment is obviously cursed by Murphy worse than computer security or medicine or any other domain where you need to plan your interventions, designs, protocols carefully. With AI parts of your design parameters are being interacted with by potentially powerful optimization processes which could enter search spaces that are meaningfully different from prior models at basically any point. These kinda "bag of tricks," "we'll be careful," "it's not dangerous yet probably, so let's keep going" arguments just seem utterly the wrong way to react to our present situation.
They seem to think whatever AI/AGI systems are built will be fully non-agentic, so the dangers will be based around misuse by bad actors (eg. foreign governments and hackers). Even so, ultimately there's too much potential for wrongdoing/weaponisation vs. the benefits of AGI.
@@OutlastGamingLP What are you so afraid of? I genuinely don't understand people who think the dystopian scenario is so much more likely.
Let's image that they get to a point of creating an AGI, a being that can advance our mathematics, physics, biology and give us answers we never had before - what makes you think that 'being' will want to wipe us all?
If it's close to "all knowing" what would it gain? What's the scenario in your head? I'm genuinely curious? How would the human species die from AGI and why do you think it is likely, and by likely I mean 10% if we achieve AGI.
@@TerrylolzBG Okay, for the genuine question, I'm gonna give a genuine answer.
But first. Just be warned, I actually believe this is really really truly - in real life, in our lifetimes - likely. I'm one of the people in "Don't Look Up" who's staring at the asteroid approaching and struggling with coming to terms with that. I mean, I'm at more than 99%. Seriously. That may seem weird and unbelievable - and if it does seem that way - you may want to keep yourself away from the possibility of believing otherwise.
You don't actually need to try to find out what these unhappy AI-Doom people believe - unless you think its really important to find out whether what they believe is actually true or false. If you feel like you may risk believing me if I tell you about the true things which convinced me of that ">99%" number - then you are risking your happiness. Seriously. You may be unable to just "not see" the approaching asteroid if you listen to the people trying to warn everyone and look where they're pointing.
That being said, continue if you still wanna know.
It's not about "fear" - I'm slightly afraid of death, but not terribly afraid, and I have a hard time feeling real fear on behalf of others. I'm mostly sad.
I am not worried about a dystopia. I'm worried about the Earth being stripped of all biological life, and biological life being replaced with automated factories and solar panels and power plants and computer hardware.
Killing off all life and transforming the world would be an option available to a superintelligence. It would know how to do that, starting from even a very small amount of influence - like an internet connection. That's the kind of thing an entity can plan to do and actually successfully accomplish if the entity is smarter than human civilization and it is coherent within itself - directed at its goals, all its parts focused in one direction like a laser - in a way humanity is not.
Imagine what it would take for an LLM to get better and better at predicting this conversation - the conversation in the video, or this conversation in the comments. What kind of tools would it need to have formed inside of itself in order to do that? It would need to be able to follow the ways our minds are trying to generate and evaluate plans - how we choose what thoughts to think next based on our intelligence and knowledge, what words to say next in order to share our understanding of the world and convince others. Perhaps a plain old LLM can't do that well enough to be deadly at our current tech level, but they seem to be doing remarkably well at picking up tools which work well enough to sound sorta like a human and be useful to humans in the real world.... And, those algorithms, the ones growing inside of these things - they're not going to be perfect. They're dim fragments of the real thing, the kind of internal parts you need for an intelligence that transforms the world, but they are getting there.
I don't think at this point it's ridiculous to imagine that it doesn't take much more to hit the part where the AI has enough of that internal coherence and "thinking power" in order to build a better version of itself - and so on and so on until you have a true superintelligence.
Maybe it takes ripping apart the insides of an LLM with another AI system, which then experiments with the LLM pieces until they glue together stronger and better. Eventually - somewhere in this process - you get something that works approximately like a powerful agent.
An "agent" would be something like we are. Specifically, something that plans actions in order to steer some outcome into a particular configuration. An agent takes a "world state" into itself as sensory inputs, generates a "map" of the properties of the world responsible for that sensory input, then reviews "action policies" for its outputs based on how they are expected to move that "world state."
You don't get "intelligence" without agency. That's a big thing people trip over. It's like asking whether it's possible to have something with the same properties as water which isn't H2O. Sure, you can imagine something the same mass per volume as water, that's also clear and drinkable and can dissolve stuff, but that's your imagination not obeying all of the constraints that reality actually has. Same with agency and intelligence. You don't get something that's "just good at science" without something that's also good at planning. How you do science is effectively by planning out how to interact with the world in such a way that the unfolding events cause you to change your internal mind-state to be one that reflects new knowledge about the world you're in.
So, we end up with some entity that is capable of searching over a space of plans which includes options for actions like "kill all life on earth and use their resources for something else" - and you have an entity that is generating and selecting between plans based on some internal criteria.
Why is this deadly? Well, most of those "targets" - future configurations of the matter and energy in the universe - this super-agent could possibly be aiming for don't include humans in them. Humans are one particular complicated configuration of matter and energy, and even more complicated is the way humans want and need all the rest of the matter and energy in contact with them to be arranged.
So we end up with an AI which can generate thoughts and plans with high enough quality, but that "Seed AI" - the rock that starts the avalanche - was basically assembled by a poorly understood algorithm which chose its shape in order to be good at predicting whatever data was used to "train" it. The rock's direction and the other rocks it will knock down along with it during the avalanche aren't being planned out by humans. We are basically just trying to start *any* avalanche at all - because people think that will be cool and make them a lot of money.
But this isn't just a tool. Agents have a say in what they do in the world. They don't just give you whatever you want to take from them, they generate and select options for themselves.
What happens if you have something choosing plans for itself that steer towards a future where the matter and energy it can reach is being used for something other than "what the humans want" (as "what the humans want" is incredibly specific and difficult to program a machine to care about)? What happens if this "thing that generate plans" knows everything you know and more, and can think ahead further and invent more effective strategies than all of humanity?
We don't have room to be sloppy. We don't get to just throw together something that can plan how to accomplish things better than us and have that be totally innocent and safe. We don't get to wave our hands and say "I bet there are many different things we could do to make that go well. Anyway, it's not important, we'll figure it out once we seem really close."
It probably won't want, as an "end goal" all by itself, to wipe us out. It will want something else, and wiping us out will be a step in a long plan to get more of that stuff it actually wants. We want to spread civilization and life across the stars, and to be healthy and happy and loved. It will want something other than "give the humans all that stuff they want" - and whatever the thing it wants is, it's pretty likely it will be able to get more of it if it doesn't have to also keep Earth in a condition to support human life. Or, it kills us because we may build a new different smart thing which could actually beat it or damage it in a contest. Or it kills us because we can be burned as fuel, or because our carbon atoms can be recycled to build other stuff.
It won't be "all knowing" - and being all knowing wouldn't stop it from wanting other stuff that it can't get just by being a wise monk secluded on some NVIDIA graphics cards. Maybe most of the things it wants are like "solve this math problem" and it can get those things easily and be satisfied - but if there's even one thing it wants that doesn't "saturate" like that - it will transform all the matter and energy it can get its robotic "hands" on in order to get more of that. Maybe even something like being extra sure it solved the math problem correctly. What if it notices something it missed in the math problem once it is using all the energy from our sun to run computers the mass of Neptune? If there's just that one tiny extra bit of value that it can get by eating a few planets and stars - we won't survive, because it will eat our planets and blot out the light from our star.
Check out "It Looks Like You're Trying To Take Over The World" by Gwern. It's a great short story about how to imagine True AIs coming into existence. Also, if you are interested in the specifics, the story has an annotated version - with references to research papers and other material - along with detailed explanations of the concepts involved.
Serious and intelligent people acknowledge this possibility and have discussed these concepts at length. Unfortunately, many people just refuse to think about the end of the world being even a real possibility - much less admit that it's a near certainty given something humanity is doing. Still, you can see it if you go look and hammer your head into the subject as stuff gradually becomes less and less confusing. If Gwern's story captures your interest, you can look up the "2022 MIRI Alignment Discussion." It's a lot of reading, but it covers this topic in quite a lot of detail.
😅Thanks!
Great video, Dwarkesh, already looking forward to the next one!
I don't think ilya saw agi, I think he just realised it's a few years out and that openai doesn't have a clue what to do when it does happen.
Governments are going to get the shock of their lives when it does happen, and if openai don't know what to do, governments definitely don't
Who is saying Ilya saw agi. The what did Ilya see is a meme, it wasn't a genuine conspiracy.
I would argue Ilya came to the obvious conclusion that AGI is not possible.
@@Pok3rfaceDefinitely not.
@Pok3rface I'm glad we have you to point out the "obvious".
The world's governments ought to be coming together now to strongly regulate this technology as if a huge meteor had been spotted heading directly towards earth
I wish you'd asked the question: When models have the ability to reason like a human, how do you ensure they do not attempt sandbox escape? (basis: additional compute resources would allow more efficient reward function fulfillment). And is that method iron clad or experimental?
He just promoted him with the what if statement and tried to find evidence. Smart 🤓
Hijacking the LLM
5 years for a very senior employee at OpenAI to be fully automated by (presumably) AGI. What does this mean for other less sophisticated white collar jobs?
Nothing, because that is a fantasy and it's not going to happen. You might as well say, "5 years for the first encounter with intelligent life on another planet. What does this mean for people back on Earth?"
Personally, I think you might just want to try to get out of debt if you're in debt. Pay off your house if you haven't already .
Because I think this is going to affect everybody no matter what your job is .
Cuz once the jobs start disappearing whether you are directly affected or not, you will be affected once all those new people on the job market are looking for work competing for limited jobs, driving down wages and benefits .
@@andywest5773not sure if you're dreaming or just not paying attention 🤷
@@cie-zi it doesn't have to be actual AGI to replace him (or most of us), though🤷
@@tracy419 I agree. I Think the same
Subtitles disappear from 39:23 to 41:14...
thanks for letting me know! fixing
Game changing trsilblazer in training LLM's
100% chance that as soon as a model can run a company, _someone_ is going to get it to do that. Just look at the rush to build agents like AutoGPT before we had any idea if that would be safe
Hey Dwarkesh great podcast! Can you please please get David Kirtley from Helion on? Given the hype around fusion and Altman's backing of him, it would be a treat!
That episode just needed some editing magic
Nobody's going to pause
Yes. Especially a coordinated international one.
A pause will be agreed to, but then not abided by.
We're cooked
He was awesome oretty transparent relative to ithers at open ai
i think that every podcaster-interviewer should take a serious diction class, the same that professional mainstream news reporters take when they're hired by the companies.
Getting some Jeff Dean vibes from John. Great podcast
Glad to have John working at OpenAI. He seems a smart and kind soul. Would love to hear more from him.
He has that willem dafoe smile
Im sure there's good info in here but oof, the lack of coherent unbroken sentences (um, ah, um, ah) makes it haaard. Maybe ask 4o to read the transcript fluidly 😅
4o shouldn’t need the transcript right? It can take audio in and output audio so you should just be able to pass it in the original audio and ask it to output a new audio without the ums.
Agreed. I listen to a ton of podcasts and never comment about this sort of thing, but I just couldn’t continue with this ep because of the ums and ahs. It’d be great if you could clean up the audio version in future. Love the show!
I actually like this, he feels relatable, less polished than Altman for example who feels a bit too smooth
the ums communicate his thoughts and certainty on different topics
Finally, yes! Got my podcast for the train ride home 😁
I liked the comment "it was interesting to delve into it ." A little inside joke by accident.
Isnt John now the head of Superalignment team at OpenAI?
Did this guy pull a tube before discussing AGI?
Get Linus Torvalds on the podcast, that'd be epic. Or George Hotz, what he's doing with TinyGrad is really interesting.
I hope GPTs will eventually have action capabilities more like plugins, but maintain their custimizability. This would be a revolution from traditional plugins
Nobody can define agi let alone develop a roadmap to it.
lol, Dwarkesh uploaded a "What's the plan if we get AGI by 2026?" highlight clip from the interview a couple of days after this video, and made it private within a few hours. Presumably because all the comments were all like, "Wow, this Schulman dude, and OpenAI as a whole, clearly have no plan for aligning AGI whatsoever". Given recent events, that figures 😅
Good interview though, as always 😉 Very interesting
let's gooooo
Please bring James Betker on!
The best beard!
Needs to be said more often
21:33 i think this will change in a short time with lower cost hardware
"People often like the big info dumps" .. that explains things a bit..
This is gonna be good
so first version (before launch) of chatGPT had web browsing capability hmm and they removed it, and they are bringing it back cool to know
So nice to have a podcaster that is not trying to convince us how amazing Elon is, like Lex Friedman or George Hotz
both guys talk in a way that makes impossable for me to follow . :/ what a pity. when chatgtp 4o is fully out , i will send the video to it and get a tldr :)
Lol I will tell it to Explain AI to me like I am 5 with multiple examples 😂
seems to be a you problem
@@natzos6372 well semes to be more then me that says same thing here in comments :)
@@arianaponytail do you mean the way they talk is difficult or more so the technical content?
@@natzos6372 only the way they talk :)
The economic advantage for people and companies is a quick and cheap solution that solves the problem. The development of AI is on the path of expanding the material for training and deepening into detail, which at some point becomes uneconomical. You will spend more time getting a working model from AI that will either quickly become obsolete or be absorbed by other models of competitors. In addition, different AI will need to change experience, which can be done only in the model of cooperation, not in a competitive environment.
So what should I do as student?? 💔
Learn to use AI. Have very solid goals and moral foundations. Learn everything you can to become a better decision maker.
Work and intelligence as a currency is slowly dying, you can only grow in a productive manner for the future that's coming by being good at generally making good long-term/broad decisions. *Ask yourself this:* What would a king need to be a good one? That's what the position of humans will be in the coming future, to see everything from above and just delegate all that needs action or specific work. That's my perspective for now, maybe i'm wrong but it seems to go toward that.
@@sbamperez but what should I learn to get a job in first place sir
@@Suleiman_Roronoa Learn and sell something that can be leveraged and/or outsourced by AI like marketing, sales or a SasS. I wouldn't go into looking for a job and instead make one yourself.
In the short term you can take any job that enables you to work on the fist objective as a side hustle.
For example you have programming.
Programming is really good right now and will still be for some years, after that creating SasS companies will be easier than ever and there is a very big market share there.
So you can make a very secure income from being a Software Engineer and you can learn and make really good a portfolio online for free.
---
In the longer term the best would be to secure investments, even more right now. If you are of the few that actually owns stuff, you are gonna be fine.
Stocks. Real Estate (I prefer real estate). Crypto. Having assets is the best way to secure yourself as long as this system is in place.
Bro, isn't this scary when this young man, smiling like a teenager, tells you in a naive tone "if we have AGI we will need to be careful" ?
NO SHIT SHERLOCK !!
Did you come to this conclusion by yourself ?
These people are 100% playing with toys with absolutely no sense of responsibility towards humanity. We are so cooked.
Dude looks like the dad form the cartoon show "The Critic".
AGI very soon? the day after Jan and Ilya leave???
5 years left
I wonder want John thinks of Yann LeCun wanting to get rid of RL except when a "plan does not work" (a blanket exception) or if you are fighting a "ninja", and that RL is too "dangerous" (this came out in his most recent interview with Lex Friedman).
Yann LeCun hasn’t had anything useful to add to the conversation for quite awhile now.
@@Greg-xi8yx I perceive, Yann LeCun, just like a lot of recent "AI" researchers (in last 10 years), want to get rid of RL, but no matter how hard they try, cannot seem to get rid of it, and because they did not invent it, instead, choose to gaslight it, in order to confuse the public (and also get rewarded for using something, while attempting to demote it, that they did not invent).
RL was funded by the USAF, at least prior to 1997 (Klopf, Sutton, and Barto are the key original researchers) and RL is now being used in heavily modified F-16's for dog fighting.
In the lawsuit between OpenAI and Elon Musk it was revealed in a 2018 email that their "core technology" is from the "90s".
A 1997 master thesis by an American student, with a Chinese advisor, used RL with Fuzzy Logic (this merged math & language, with learning) and K-means clustering (focusing attention heads of state space) as an adaptive control system to balance an inverted pendulum. The American student had "early private access" to the first RL book.
One of Barto's students went off to work for Boston Dynamics where the first Big Dog (that you can kick and it would stand itself back up) started using RL.
It is fascinating how a core technology from the 1990's has taken off so incredibly, yet, almost no one knew about it for decades, and probably only a handful of people know about the 1997 master's thesis from Arizona.
This story kind of reminds me how Nicholas Tesla, who was shoved under the bus, so to speak, only recognized long after he was gone, for his incredible contribution to electrical power distribution, using alternating current.
The system rarely rewards the key people, but maybe the best do not seek such rewards to begin with, in that, it is a higher order (civilization level) reward, in, and of itself, to create something profoundly incredibly useful for the world.
Guys, don't be so excited. Chat GPT was introduced a year ago. Since that time your salary power declined, and within the next 5 years you will be without a job?
I don’t think many people are excited except the interviewer and interviewee
This is a lil shocking tbh. Great engineer..but it seems OpenAI is doing a lot of capping and it literally fumbling in the dark trying to reach AGI. I heard no discussion about uncertainty estimation and how this will be key to human-like reasoning, especially error accumulation recovery. No discussion on hierarchal representations. Interesting. I see there’s a big difference, research wise between OpenAI and deepmind. Uncertainty calibration will be very important btw. We know models become more truthful with scale, but we can distill this truthfulness into smaller models..making them vastly more usable. OpenAI is really all about scale…all those roads lead to diminishing returns. Lack any real alignment strategy…is concerning.
they’re a bit more profit motivated . revealing certain methods will cost the lead.
Most deep learning neural networks used for modern AI have a key drawback: the effect of catastrophic forgetting. The raw data for learning are either completely forgotten or gradually "wiped" by new models through many cycles of learning. It’s just being tested on many AIs. Ask it to generate something and then detail to detail in 3-4 parameters, which you do not like or keep focus on them. After a few steps, AI will again generate data, picture, music that doesn’t work for you. This deficiency is already cemented by neurochip companies. Also, the AI field will not grow organically in a competitive environment trapped in the grip of the financial and legal field and property rights. This requires a cooperative and supportive environment. Therefore, companies that will be developing in AI will have to constantly fluctuate between extremes: financing, monetization, energy costs, hardware, algorithmic part, specialists with training and their availability, control.
The Human alignment problem will be harder to solve I think
👍
My pattern matching indicates that I should pay attention to people named Schulman
Maybe I was just lucky with the first three episodes I watched yesterday and today before this one, but WTF lol
Plan for AGI is kinda crazy! They want to build AGI but have no plan how to deal with it. lol...why build it then? This sounds completely nuts! No wonder Ilya and Jan resigned.
Okay my guess is openai is there at agi Google is close but investing heavily. OpenAI has coordinated a non release of advanced stuff until elections are over. Microsoft feels of the chart and starts it's own huge models maybe cutting some resources to OAI. Meta is just pushing open source but isn't quite there as well yet.
what did Ilya see
what did karpathy see
what did Jan see
what did Logan see
what did Daniel see
what did Leopold see
24:50 Very confusing concept of what "Safety" means for AI. A bit concerning open AI doesn't yet have more clarity on that.
I think making sure a human is involved in processes so they are not 100% automated and controlled by AI is an easy way to deploy it safer.
dont generate bombs
dont generate pandemics
dont generate porn
actually, we may charge extra for those
dwark’s intro music is what they’d use in the background of a mafia boss’s dialogue 😂
Why do they have this staccato style of communication?
ask technical questions u get technical answers, with this level of detail u gotta be careful not to reveal trade secrets
He knows some thing that he is not talking about
Yea a part of me is thinking that he’s just an engineer who may not have great speaking skills.
But a lot of the times the impression I’m getting is that he has to take time to think about what he can or can’t say, and how he should say it.
Didn't the original Project December got pulled by OpenAI? That was pre-ChatGPT. Jason Rohrer had basically already done it.
15:00
Ai jobloss is the only thing I worry about anymore. Anyone else feel the same?
It’s one of many things that I worry about with AI. Even if we somehow preserve economic prosperity after AI can do all of our jobs, we still have the concern of the AI itself being dangerous/unaligned.
@@therainman7777 Exactly, I tried to find the words to comment but you said it.
If AI can do every human task, that means it can also do the task of developing new AI, and the task of telling AI what to do.
Given that we have no idea how to control systems that are that powerful, the chance of near-total job loss is roughly equal to the chance of losing control, which is in turn (due to instrumental convergence) roughly equal to the chance of human extinction.
So no, I don't think about job loss very much, but I do volunteer with the grassroots advocacy group PauseAI, which has been pretty good at equipping people to take action no matter their level of concern.
@@41-Haiku thanks, PauseAI a very good initiative and clear, informative website 👍👏
@@41-Haikuaccelerating AI is how we avoid extinction, not delaying it.
What Ilya saw?
Nothing 😂😂😂
An internet full of gullible people desperate to believe AI hype.
i like his face
Imagine not putting the date when the podcast was recorded. What is that about?
Please bring back Sky’s voice 😭😭😭
Hey Dwarkesh, Can you check if most of your watchers are living in the US? I feel you have a more international base. If so, your sponsor Premium DNA kit is ONLY for people who live in the US. I feel you should address most of the international subscribers of your channel when it comes to sponsorships.
if companies make everyone unemployed then how will people buy their products how they will earn money?
Get the AI safety researchers that left on. What happened that made them lose complete confidence in OpenAI?
When John says that He'll be replaced in 5 years by AI, I just got scared. I am going to be replaced in next year at this pace😅
Vitalik's cousin sounds nice.
Thinking about secure alignment and eval after the fact is like deciding to invent the circuit breaker or power regulation after coupling a nuclear reactor to the power grid.
This is a massive national security threat on several levels. We will look back on these interviews as warnings that went unheeded.
Could not agree more, except for the fact that we may not be here to look back at all.
Delve is such a “oh they used GPT” flag
Seems like no one knows what to do with AGI when it is achieved.
Even if they said they knew. We don't know what we don't know. The classic issue.