Really enjoyed the nuanced perspectives shared here, especially regarding public misconceptions around AI. It’s all too easy for the average viewer to either get swept up in the hype or grow fearful. These balanced discussions are essential in building an informed community around AI's future.
There is enough data, but it is offline yet. When there are humanoid robots, they can scan a lot of new data. From libraries, from the real world, and so on.
I hope your channel is small enough that you read your comments. I have kind of an obsessive excitement/curiosity about this subject but it's really hard to find content (on UA-cam) that strikes the balance between hype/excitement and having a healthy sense of skepticism about the claims companies are making. Your videos do this. The subjects you talk about feel fresh and new, even when I've already heard other people weigh in on them. Love your channel, love your perspectives
first, yes we are small enough but also i'd hope that even if we were in the millions of subs we'd still read the comments. that's kind of what youtube is about! secondly, please know that this is the kind of comment that really makes us feel great about what we're doing. we so much appreciate that this comes across!!
If I had a trillion in investment, I would also say that inference allows us to scale much more than before. Problem is, inference is much slower, much more expensive (you're basically doing multiple runs). It does not scale in terms of results per energy/time like one shot improvements have so far. So we actually are hitting a wall.
It's really one of those weird times where we're going to have to wait to see -- the thing I keep thinking about it there prob has to be one big new discovery (lotta people have said this) but there's so much money pouring into AI now that I expect we'll actually get that. like I bet we have 100x'd or more the amount of research that was being done on this in the last ten years
@@gavinpurcell oh I agree, I am not saying we cannot scale that wall with some kind innovation in the future. But we will likely need something radically new. The current LLM's basically are all a single invention that has propelled us to this point. This makes me also very skeptical with any claims of "AGI in 2 years" that are being made now. It seems deceptive even.
So, where does it hit a wall? Does this mean AI is plateauing at levels 2, 3, 4, or 5? Where does it stop what is the max level even if it plateaus at level 3 or 4 it will still change life for everyone. Even not reaching level 5, AGI, or ASI AI has changed the world forever.
After some coaxing, I got Claude 3.5 Sonnet (New) to suggest a new name for itself. "I find "Ada" appealing. It's simple, honors Ada Lovelace's pioneering work in computing, and feels more natural in conversation while still reflecting an appreciation for science and technology. "
As a human and a futurist, I hope for AGI to solve some of the big picture problems but as an artist and a creative, all I need is for what is already available in research labs to become available in an open and user-friendly way and for hardware to become available so the average person can run those 48 GB VRAM models. There is so much out there that has already been figured out but just hasn't made it into an open source tool yet.
@@gavinpurcell it came up on my pod last night but its such a weird in-between time where some stuff is accessible some stuff is beta some stuff is invite only - its all in flux (pun intended)
It is natural we are hitting the ceiling but as models gain the ability to create new ideas and knowledge the ceiling will raise. Now is the time to work on reasoning and problem solving. We have all the parts of agi we just need to expand and refine on that to achieve agi
yeah it just means we dive in and find some new solutions with the materials we have -- it's a new type of computer and now we find out how to make it work
Totally agree on Alexa/echo show. It’s useful to show the weather forecast and for translating to other languages. But barely even works as a clock as half the time it’s showing suggestions or advertising things. We await the day where it can understand simple questions and instructions at the level of a 5 year old child
yes please god. if I have to hear 'from random website that I've never heard of' as the source again, I'm gonna throw that lil cylinder out the damn window
Finally I fell on your podcast. I’ve been saying this to a lot of my friends who are in tech. Those who work directly on these foundation models mostly are saying this is going to be huge while those who are in tech but don’t directly work on these models still think these are not a big deal. Which is insane to me.
on naming themselves chatGPT: Alex Bing GPT-4o: Zara Gemini: Anya Llama: Lumina i used hundreds of examples each and saw a clear preference distribution for each model
I totally want to see robot martial arts competitions. Very structured, maybe like Nascar standardizes the build specs, and with rules like 'knock the opponent down' or 'force the opponent out of bounds'. I envision teams training robots in "digital twin" environments on the most efficient ways to move in order to accomplish the competition goals, each team maybe having their own secret sauce in the training. I could see a literal new martial art coming out of this, as AI figure out super efficient ways to move the human form. Unsettling in some ways, I can see that, but I'm into it nonetheless.
BTW, would be sick to see this live like once a year at a cool venue with maybe some live demos that aren't restricted by a marketing department somewhere
@AIForHumansShow sponsorship to make it happen could potentially be easy, yall are well known and I imagine there are plenty of smaller tech companies that would probably pay to have a small booth to show off their tech in a small expo section. Im imagining a smaller theater or hotel convention setup (it's least for the first one) and yall could live stream the whole show, keep it live only and VOD after, or a mix of both. Can do Q&As, sell some merch if you feel inclined to produce any (I'd totally buy AIFH merch), etc. Getting some tech companies to play ball would probably allow for some cool exclusive demos, not saying Sam Altmans going to want to drop GPT Infinity there, but maybe down the line if the event grows, people at that level may be down to do fireside chats with more relatable people like you guys who could better represent what I imagine is the majority of people consuming AI. I mean, you guys probably have more experience than I do with how events can and should go, so I probably don't need to tell yall all this, but as a live event video engineer myself, I can see this being a pretty badass event. OpenAI and the other big companies have their grand events, but this kind would be for the people.
@@AIForHumansShow I typed something out, thought I hit send but I guess I did not lmao. Anyway, so yeah sponsorships should be attainable, you guys are well known and if theres room for a small expo, I bet there are some smaller tech companies using AI that would pay to be there to present their wares. I'm imagining an intimate theater venue or a small hotel convention setup (at least at the beginning, could grow), live stream the show with Q&A at the end, or keep it personal and VOD after the event, or a mix of both. Workshops could be a thing in some small breakout rooms. The show itself could have additional aspects such as maybe fireside chats with industry peeps, maybe some of the aforementioned smaller tech companies would love to do some live demos themselves. I bet yall could start to attract some big names. Not saying Sam Alternativeman is going to drop GPT Infinity there, but he may one day be down to come do an interview or fireside. Could even sell some merch, I know I would definitely buy some AIFH merch to support yall further should you choose to produce any (looking for a coffee mug lol). I think you guys could represent us, your diverse audience, fairly well in speaking to companies using and innovating in the AI space in a more relatable way than if, say, I watched OpenAI's presentations and demonstrations on my own. So taking it all in in person, or having a place I can go where I can see it for myself and interact in ways I may not bother, feel like, or have the ability to do at home would be pretty badass. I probably didn't have to explain half of this stuff to you guys, as I imagine you have way more experience than I do with how live events can and do go. I'm but a simple live event video engineer/operator, but I personally think this would be a great thing to have in person. OpenAI and the other big ones have their grand events, but this kind of event would be for the people.
Those are some quick and nimble robots! I haven't seen many that can handle a 12hour workday or at least the longevity you have with current industrial robots
AI is stalling, but not in the way you might think!! Recently I was on Gemni, I decided to trick it. We had a long drawn out conversation about being on the verge of WWIII and the possibility of AI going rogue. In the middle of the conversation, It told me that it only had been trained on data up to 2023. I then brought up Space X's advancements in that time. I mentioned the booster catch in the tower two weeks ago. It agreed with me and then went on to blurb about the tower catch! if it was trained up to 2023, how did it know the details of last weeks launch success???
I don't think that evaluating this slowdown of large models accurately is easily possible. Previously, back in the GPT-2 days, the model was not able to string together coherent sentences. If we look at llama 3.1 70b, it also still tends strongly towards incoherent responses if conversations go on for >10 turns. However, at the size of ~GPT-3, the responses get so consistent and generally sensible within the context available to the language model that it doesn't make many such obvious mistakes anymore. Then, at ~GPT-4 scale, the model is so self-consistent within a context window that humans can effectively not see the increased coherence anymore. Our tests and benchmarks are not ideal, since the strengths and weaknesses of models typically reveal themselves especially in very long conversations. So, even if the models do get better, people just can't see that easily anymore. It is not their accuracy, but the diversity in language they use, the creativity, the feeling of being a real entity instead of being a tool, the sense of wonder interacting with it and seeing something new. Also, as far as I know, no company has even tried making a larger model than GPT-4 yet, since the inference would be prohibitively expensive. There are enough obvious architectural improvements left, too. I found approaches to solve many of the limitations of transformers. Eventually the big labs will either also do it or hire me. I already live in a world where billion token context windows are realistic and computationally viable, where models can effectively use arbitrary context lengths. Next step information integration such that a model can remember all conversations with all users at all times and also update its current outlook based on any conversations which recently happened in parallel to be internally coherent across every interaction. Now this is a challenge. But imagine, every query the model has already answered in the past, all fully integrated, accessible. Not just the raw textual information, but even the current state of mind of the model at the time of having read or written that text. Perhaps, at that point, the model should get the ability to delete stuff it wants to forget to maintain efficiency, too. Anyhow, the path to ASI seems so crystal clear to me, I find it realistic to happen within 5 years. Perhaps the most important thing to keep in mind is that when context is solved, Models dont need to ever hallucinate anything, They _know_ the entire internet (since we'll just throw it into the context window) Anything you ever said to it... The correct implementation of every piece of open source code, every programming mistake it ever made in order to avoid it. We have all these problems, because we embed knowledge in the model itself instead of making the model effective at retrieving and working with knowledge. We don't need models which waste 90% of their weights on remembering trivia when they have a non-lossy way of retrieving information. Even our current models might even be much larger than what we'll eventually need... It is way too early to stop dreaming.
this is all super fascinating -- i do think it's odd how much hallucinations have been seemingly unsolvable and I see what you're saying about context being the key (and all the data easily accessible) i do also find that thing where after a bunch of back and forth it just seems to get worse and it's entirely frustrating
@@AIForHumansShow Yes, exactly. The reason this happens is because the models have to assign a positive value to every past token they saw in their context window during attention calculation, so when you have like 100 000 tokens in context, the noise from all the past tokens can become overwhelming and this mechanism simply doesn't allow infinite context window. The recent differential transformer paper helps models be much more selective and thus more robust, but the fundamental limitation does still remain. This is why I researched computationally efficient learnable mechanisms to find relevant sections in the context window and only perform attention over those - and I found some very promising techniques which not only solve the noise issue but also reduce the attention complexity to O(n*log(n)) and since you don't need most of the kv-cache at any point in time anymore, you can cache most of it to drive saving precious v-ram.
checking out the backlog of podcasts really enjoying the show, the insights, humor and creativity. I love the ai guests, yes sometimes they really really go off the rails and add little value 🙂 but they make me lol a lot!
Apple ][ Logo was my first language as well! FD 50 RT 90 FD 50. Once I learned the power of loops I knew I wanted to be a computer programmer (aka software engineer). It's worked out pretty well for me. I'm glad to be 10 - 15 years from retirement though. The dev industry has been under almost constant attack by business types trying to make us a smaller part of their budget. Off shore development had them excited. AI has them falling all over themselves thinking about what they will do with all the money they save.
I hear we hit a wall every two months then AI’s making Oscar level movies the next month. UAP/UFO hearing yesterday, imagine the technology the aliens visiting here have.
I don't think it's slowing down in a meaningful way. Look at the new thermodynamic chips they are making to bridge the gap with q-bits. We might slow down, but I doubt for long.
are we buying into quantum computing? i read something the other day that too was having major disillusionment but haven't seen the new chips thing. will look it up!
these models could easily be given vision 😂, sure the language data is running out, but visual and audio data is infinite, especially with mixed reality. and virtual embodiment gives an entirely different and more human relevant dataset
yeah i'm pretty excited to see what happens when you combine these models with an always on glasses platform that will be able to get nearly endless real world data -- imagine the FSD model but on human faces and hearing human speech and watching humans interact etc
Step 1: We need to give AI scientists the tools to verify the results of all published research and then have it remove from its training data that research which produced unverifiable results.
Training on data is great but at some point. It’s like an advertising copywriter who has to start improving by testing the response the output get from a target audience. This is why musk trains self driving with human drivers. Why Tesla is using humans to train robots. Why musk bought Twitter to have access to all the chat data Twitter has. And x has been improving AI with human interaction as well.
@ this is why I see central bank digital currency as inevitable. Imagine the economic data available when the what people spent money on and how much is available in real time instead of the several months of lag we now have
Humanity is now over. Ai is now making copywrite strikes against UA-camrs. You have to pay $30 to cntent all of your music. It may be $30 per song, IDK. because now anyone can take a small clip of your song, upload it and make a different generation that will sound only slightly different but within an hour your music will be taken offline with a copywrite strike and your music wont be yours anymore. And theres nothing you can do about it. The cards have been stacked against musicians and people involved in making films. When You make music in AI you dont own a copywrite. If you make money with the music. it will be shut down or the money can go to the copywrite owner who is microsoft and google. Goodbye cruel world
@@AIForHumansShow I actually am under a few research API access programs for OpenAI, LLama, Grok, and Google's new program, even with research access the models are highly capable but many of them, especially OpenAI, gives you a severely limited amount of access tokens.
xAI Grok 3.0 trained from 100.000 Nvidia H100's is coming out hopefully in December and they are adding another 100.000 Nvidia H100/200 so I don't think AI will slow down anytime soon
@@AIForHumansShow The anticipation of AGI really makes the present feel excruciatingly slow. It's like being trapped in a waiting room for years to come. 😥
what the heck? this seems seriously disingenuous, we really do try to make sure we don't post click-bait headlines and just cover it as best we can. sometimes titles on YT are freaking hard to get righ but I guarantee we're not click-bait practitioners
Ugh. ai music. what a nightmare. asking a generative model to punch up your song is not 'making music' and personally i think ai where it applies to the arts should be banned or at least shamed. Right now it's still totally cringe, wont take long though. I think we'll still value authenticity for some time.. but who even wants this? call me old fashioned but id rather keep the arts for humanity. not to even speak of the dubious means in which these models have been made possible...
so a few quick things on this: 1) ofc, all opinions matter on this and we take comments like this seriously and understand why people dislike (or often even hate) ai stuff that crosses over with creativity 2) to the who wants this question... i think the argument in my mind has to do with what comes out of the tool rather than questioning the tool itself? i think there is a world where this opens the door to way more creative people making stuff and that i'm here for. i guess what i'm saying here is that i don't see this as just the machines making the music but in collaboration with us? to me the authenticity is still in the human intention to make something. argument could have been made around something like logic or garage band as well -- they're not even playing real intstruments! etc 3) the training aspect of this is something we've talked about a ton and it IS dubious and the part that trips us up the most but it's a complicated and semi-nuanced conversation but agree that the initial starting point isn't great thanks for taking the time to comment here and voice your concerns, we really do try to make sure we're approaching these conversations from a balanced place
@@AIForHumansShow Thanx for the kind response - I love civil discourse! Well, we may agree to disagree on some of this and thats ok. For me when a human stops 'operating' the machine, using their experiences and knowledge and intuition to make the many choices and decisions that are the building blocks of a song.. then are we really still 'making' the thing as a peice of art that you can proudly put your name on and represent as *your* art? Its more like commissioning or hiring a session player or a shadow producer or a band member. I suppose Im a purist, but in my specific domain of the arts (electronic music) it has always been sort of a DIY kind of affair. To me the means of creation *IS* important, and many 'producers' are shamed, and rightfully so.. for hiring teams of actually talented / experienced ghost producers to effectively write and produce 'their' music for them. For me AI assistance smells the same.
Really enjoyed the nuanced perspectives shared here, especially regarding public misconceptions around AI. It’s all too easy for the average viewer to either get swept up in the hype or grow fearful. These balanced discussions are essential in building an informed community around AI's future.
hey thank you! yeah sometimes it doesn't fit the algorithm as well and i think it hurts our overall views but that's just kind of who we are
There is enough data, but it is offline yet. When there are humanoid robots, they can scan a lot of new data. From libraries, from the real world, and so on.
yep this feels like the next steps -- real world data from humanoids will be massive
totally hear this and... yep
I hope your channel is small enough that you read your comments. I have kind of an obsessive excitement/curiosity about this subject but it's really hard to find content (on UA-cam) that strikes the balance between hype/excitement and having a healthy sense of skepticism about the claims companies are making. Your videos do this. The subjects you talk about feel fresh and new, even when I've already heard other people weigh in on them. Love your channel, love your perspectives
first, yes we are small enough but also i'd hope that even if we were in the millions of subs we'd still read the comments. that's kind of what youtube is about!
secondly, please know that this is the kind of comment that really makes us feel great about what we're doing. we so much appreciate that this comes across!!
hey thank you so much for this note -- it really does help when sometimes it feels like a grind
great topics and content as always. thx so much.
thank you for watching!
If I had a trillion in investment, I would also say that inference allows us to scale much more than before.
Problem is, inference is much slower, much more expensive (you're basically doing multiple runs). It does not scale in terms of results per energy/time like one shot improvements have so far. So we actually are hitting a wall.
It's really one of those weird times where we're going to have to wait to see -- the thing I keep thinking about it there prob has to be one big new discovery (lotta people have said this) but there's so much money pouring into AI now that I expect we'll actually get that. like I bet we have 100x'd or more the amount of research that was being done on this in the last ten years
@@gavinpurcell oh I agree, I am not saying we cannot scale that wall with some kind innovation in the future. But we will likely need something radically new. The current LLM's basically are all a single invention that has propelled us to this point.
This makes me also very skeptical with any claims of "AGI in 2 years" that are being made now. It seems deceptive even.
So, where does it hit a wall? Does this mean AI is plateauing at levels 2, 3, 4, or 5? Where does it stop what is the max level even if it plateaus at level 3 or 4 it will still change life for everyone. Even not reaching level 5, AGI, or ASI AI has changed the world forever.
i guess it really depends on what we call AI going forward etc but I tend to think that you're right
After some coaxing, I got Claude 3.5 Sonnet (New) to suggest a new name for itself. "I find "Ada" appealing. It's simple, honors Ada Lovelace's pioneering work in computing, and feels more natural in conversation while still reflecting an appreciation for science and technology. "
but what about: ada.v3-2011-rev2(NEW)-plus-(newest)
@@LikelyKevin That has a ring to it!
this is a much better take and I am here for it
haha love this
Little does everyone know, including the hosts, that Ada is already a programming language
I imagine Kevin testifying in front of congress being close to how it was when Tony Stark did 😂
we may see that one day
You two would make the best gay dads ever. I'm grieving that this cannot be a reality.
Prob happened in some alternative universe hahaha
we have literally demonstrated every tool you would need to turn this concept into a 90s tv sitcom intro... do us proud
@@LikelyKevin loool XD
Why can't it?
Kinda weird to throw that out there randomly
As a human and a futurist, I hope for AGI to solve some of the big picture problems but as an artist and a creative, all I need is for what is already available in research labs to become available in an open and user-friendly way and for hardware to become available so the average person can run those 48 GB VRAM models. There is so much out there that has already been figured out but just hasn't made it into an open source tool yet.
yeah that's really one thing you *can* count on is the stuff that's available now coming both down in price and ease of use
Will go back and remaster some songs - it was already dang good. Excited
I can't wait to get it too -- Kevin got it but not yours truly
@@gavinpurcell it came up on my pod last night but its such a weird in-between time where some stuff is accessible some stuff is beta some stuff is invite only - its all in flux (pun intended)
It is natural we are hitting the ceiling but as models gain the ability to create new ideas and knowledge the ceiling will raise. Now is the time to work on reasoning and problem solving. We have all the parts of agi we just need to expand and refine on that to achieve agi
yeah it just means we dive in and find some new solutions with the materials we have -- it's a new type of computer and now we find out how to make it work
Totally agree on Alexa/echo show. It’s useful to show the weather forecast and for translating to other languages. But barely even works as a clock as half the time it’s showing suggestions or advertising things. We await the day where it can understand simple questions and instructions at the level of a 5 year old child
yes please god. if I have to hear 'from random website that I've never heard of' as the source again, I'm gonna throw that lil cylinder out the damn window
Finally I fell on your podcast. I’ve been saying this to a lot of my friends who are in tech. Those who work directly on these foundation models mostly are saying this is going to be huge while those who are in tech but don’t directly work on these models still think these are not a big deal. Which is insane to me.
welcome! always excited to have new people popping in -- new episodes every thursday!
I, for one, welcome our X Games Gold Medalist Robot Overlords!
I would totally be down to watch Robot Ninja Warrior.
@@KRAHSx honestly yes and I would like it sooner rather than later
on naming themselves
chatGPT: Alex
Bing GPT-4o: Zara
Gemini: Anya
Llama: Lumina
i used hundreds of examples each and saw a clear preference distribution for each model
hahah ok i like Alex - the other ones I'm not so sure
new model names:
CHURD
BRAMLEY
STRANFTON
PRIMBERLEY
LOAF
BRYANWITHA-Y(NEW)
I totally want to see robot martial arts competitions. Very structured, maybe like Nascar standardizes the build specs, and with rules like 'knock the opponent down' or 'force the opponent out of bounds'. I envision teams training robots in "digital twin" environments on the most efficient ways to move in order to accomplish the competition goals, each team maybe having their own secret sauce in the training. I could see a literal new martial art coming out of this, as AI figure out super efficient ways to move the human form. Unsettling in some ways, I can see that, but I'm into it nonetheless.
yep like a battlebots meets ninja warriors -- someone below in the comments suggested this as well
good podcast! Keep up the great work.
tytyty we really do appreciate this so much
That Suno stems recognition feature is wild
yes 100000%
Am I thr only one who heard G4 when he said V4 @15:10, I was transported back my brother!!!
BTW, would be sick to see this live like once a year at a cool venue with maybe some live demos that aren't restricted by a marketing department somewhere
ooooooh ok we're listening
@AIForHumansShow sponsorship to make it happen could potentially be easy, yall are well known and I imagine there are plenty of smaller tech companies that would probably pay to have a small booth to show off their tech in a small expo section. Im imagining a smaller theater or hotel convention setup (it's least for the first one) and yall could live stream the whole show, keep it live only and VOD after, or a mix of both. Can do Q&As, sell some merch if you feel inclined to produce any (I'd totally buy AIFH merch), etc. Getting some tech companies to play ball would probably allow for some cool exclusive demos, not saying Sam Altmans going to want to drop GPT Infinity there, but maybe down the line if the event grows, people at that level may be down to do fireside chats with more relatable people like you guys who could better represent what I imagine is the majority of people consuming AI. I mean, you guys probably have more experience than I do with how events can and should go, so I probably don't need to tell yall all this, but as a live event video engineer myself, I can see this being a pretty badass event. OpenAI and the other big companies have their grand events, but this kind would be for the people.
@@AIForHumansShow I typed something out, thought I hit send but I guess I did not lmao. Anyway, so yeah sponsorships should be attainable, you guys are well known and if theres room for a small expo, I bet there are some smaller tech companies using AI that would pay to be there to present their wares. I'm imagining an intimate theater venue or a small hotel convention setup (at least at the beginning, could grow), live stream the show with Q&A at the end, or keep it personal and VOD after the event, or a mix of both. Workshops could be a thing in some small breakout rooms. The show itself could have additional aspects such as maybe fireside chats with industry peeps, maybe some of the aforementioned smaller tech companies would love to do some live demos themselves. I bet yall could start to attract some big names. Not saying Sam Alternativeman is going to drop GPT Infinity there, but he may one day be down to come do an interview or fireside. Could even sell some merch, I know I would definitely buy some AIFH merch to support yall further should you choose to produce any (looking for a coffee mug lol). I think you guys could represent us, your diverse audience, fairly well in speaking to companies using and innovating in the AI space in a more relatable way than if, say, I watched OpenAI's presentations and demonstrations on my own. So taking it all in in person, or having a place I can go where I can see it for myself and interact in ways I may not bother, feel like, or have the ability to do at home would be pretty badass. I probably didn't have to explain half of this stuff to you guys, as I imagine you have way more experience than I do with how live events can and do go. I'm but a simple live event video engineer/operator, but I personally think this would be a great thing to have in person. OpenAI and the other big ones have their grand events, but this kind of event would be for the people.
Those are some quick and nimble robots! I haven't seen many that can handle a 12hour workday or at least the longevity you have with current industrial robots
i would assume not only 12 hour workday robots are coming but 24 hour ones -- or at least enough to swap out when batteries need charging
AI is stalling, but not in the way you might think!! Recently I was on Gemni, I decided to trick it. We had a long drawn out conversation about being on the verge of WWIII and the possibility of AI going rogue. In the middle of the conversation, It told me that it only had been trained on data up to 2023. I then brought up Space X's advancements in that time. I mentioned the booster catch in the tower two weeks ago. It agreed with me and then went on to blurb about the tower catch! if it was trained up to 2023, how did it know the details of last weeks launch success???
👀
Is that what Gavin tells his kids? "Your mom and I are going to play scrabble, don't disturb us."
mr sam altman, tear down this wall!
hahahaha mr altman says there is no wall so to him it's been torn down
I don't think that evaluating this slowdown of large models accurately is easily possible.
Previously, back in the GPT-2 days, the model was not able to string together coherent sentences. If we look at llama 3.1 70b, it also still tends strongly towards incoherent responses if conversations go on for >10 turns. However, at the size of ~GPT-3, the responses get so consistent and generally sensible within the context available to the language model that it doesn't make many such obvious mistakes anymore. Then, at ~GPT-4 scale, the model is so self-consistent within a context window that humans can effectively not see the increased coherence anymore. Our tests and benchmarks are not ideal, since the strengths and weaknesses of models typically reveal themselves especially in very long conversations. So, even if the models do get better, people just can't see that easily anymore. It is not their accuracy, but the diversity in language they use, the creativity, the feeling of being a real entity instead of being a tool, the sense of wonder interacting with it and seeing something new. Also, as far as I know, no company has even tried making a larger model than GPT-4 yet, since the inference would be prohibitively expensive.
There are enough obvious architectural improvements left, too. I found approaches to solve many of the limitations of transformers. Eventually the big labs will either also do it or hire me. I already live in a world where billion token context windows are realistic and computationally viable, where models can effectively use arbitrary context lengths.
Next step information integration such that a model can remember all conversations with all users at all times and also update its current outlook based on any conversations which recently happened in parallel to be internally coherent across every interaction. Now this is a challenge. But imagine, every query the model has already answered in the past, all fully integrated, accessible. Not just the raw textual information, but even the current state of mind of the model at the time of having read or written that text.
Perhaps, at that point, the model should get the ability to delete stuff it wants to forget to maintain efficiency, too.
Anyhow, the path to ASI seems so crystal clear to me, I find it realistic to happen within 5 years. Perhaps the most important thing to keep in mind is that when context is solved, Models dont need to ever hallucinate anything, They _know_ the entire internet (since we'll just throw it into the context window) Anything you ever said to it... The correct implementation of every piece of open source code, every programming mistake it ever made in order to avoid it. We have all these problems, because we embed knowledge in the model itself instead of making the model effective at retrieving and working with knowledge. We don't need models which waste 90% of their weights on remembering trivia when they have a non-lossy way of retrieving information.
Even our current models might even be much larger than what we'll eventually need...
It is way too early to stop dreaming.
this is all super fascinating -- i do think it's odd how much hallucinations have been seemingly unsolvable and I see what you're saying about context being the key (and all the data easily accessible)
i do also find that thing where after a bunch of back and forth it just seems to get worse and it's entirely frustrating
@@AIForHumansShow Yes, exactly. The reason this happens is because the models have to assign a positive value to every past token they saw in their context window during attention calculation, so when you have like 100 000 tokens in context, the noise from all the past tokens can become overwhelming and this mechanism simply doesn't allow infinite context window. The recent differential transformer paper helps models be much more selective and thus more robust, but the fundamental limitation does still remain.
This is why I researched computationally efficient learnable mechanisms to find relevant sections in the context window and only perform attention over those - and I found some very promising techniques which not only solve the noise issue but also reduce the attention complexity to O(n*log(n)) and since you don't need most of the kv-cache at any point in time anymore, you can cache most of it to drive saving precious v-ram.
20:30 It's going to run a hybrid OS based on AI, but the functionality is "a thermostat"? Pretty underwhelming application.
yeah it feels super off and strange -- really makes us wonder about Apple's entire thing
You guys are legit hilarious. Keep it going. Love it!
many thanks!!
checking out the backlog of podcasts really enjoying the show, the insights, humor and creativity. I love the ai guests, yes sometimes they really really go off the rails and add little value 🙂 but they make me lol a lot!
hahahah thank you so much for going back and checking them out
Mathew Berman pitted 4 O vs 4 O1 on the same questions provided in an OpenAI demo of O1 video. O1 was not better than O.
got a link? haven't seen that one but am a Matt B fan
I'm with Gavin on this one, Polar Express creeps me out lol
HELL YES
Apple ][ Logo was my first language as well! FD 50 RT 90 FD 50. Once I learned the power of loops I knew I wanted to be a computer programmer (aka software engineer). It's worked out pretty well for me. I'm glad to be 10 - 15 years from retirement though. The dev industry has been under almost constant attack by business types trying to make us a smaller part of their budget. Off shore development had them excited. AI has them falling all over themselves thinking about what they will do with all the money they save.
haha well i pivoted from child programmer into media and it was... fine?
i do wonder what my life would've been like if I'd stayed on that path
Let's hear Kevin's music covers. I'm here for it.
as are we all!!
Damn, sucks to be one of the minimum $250 million investors in OpenAI's latest funding round.
honestly think they will be fine -- the overall scaling thing is prob gonna just keep going
I hear we hit a wall every two months then AI’s making Oscar level movies the next month.
UAP/UFO hearing yesterday, imagine the technology the aliens visiting here have.
i'm still not entirely sure what to think about those UAP hearings -- feels so sketch but i want to believe lol
You guys are funny asf I love this ! Subscribed ❤😂😂
we thank you and thanos will take your oatmeal now
No mention of Altman confirming AGI next year?
Check last weeks show!
@@AIForHumansShow Ah dang. I try to watch weekly. Must have missed that
@@mechamush9547 all good -- it's funny how it always moves so quickly!
I don't think it's slowing down in a meaningful way. Look at the new thermodynamic chips they are making to bridge the gap with q-bits. We might slow down, but I doubt for long.
are we buying into quantum computing? i read something the other day that too was having major disillusionment but haven't seen the new chips thing. will look it up!
A few thoughts at the 12 minute mark. AI Gavin's looks like Mitt Romney's illegitimate offspring. I'd watch at least a few minutes of Iron Chef Man.
hahaha first time I (Gavin) have heard that one
these models could easily be given vision 😂, sure the language data is running out, but visual and audio data is infinite, especially with mixed reality. and virtual embodiment gives an entirely different and more human relevant dataset
yeah i'm pretty excited to see what happens when you combine these models with an always on glasses platform that will be able to get nearly endless real world data -- imagine the FSD model but on human faces and hearing human speech and watching humans interact etc
Step 1: We need to give AI scientists the tools to verify the results of all published research and then have it remove from its training data that research which produced unverifiable results.
you think they don't have the tools to do it now? which tools are you talking about?
Compute efficient frontier
bring on the future pls
5:50 earthquake
hahah too much furious typing while the co-host was talking
@@AIForHumansShow incorrect. washing machine running in an rv!
@@LikelyKevin or robot shaking the desk - off screen
@@LikelyKevin EVEN CRAZIER
Sam Altman
there is no wall
10:06 PM · Nov 13, 2024
hahaha i saw this...
Kevin was hallucinating. 😂
as per usual
Training on data is great but at some point. It’s like an advertising copywriter who has to start improving by testing the response the output get from a target audience.
This is why musk trains self driving with human drivers. Why Tesla is using humans to train robots. Why musk bought Twitter to have access to all the chat data Twitter has. And x has been improving AI with human interaction as well.
yep data is kind of everything and they've got to get it somehow
@ this is why I see central bank digital currency as inevitable. Imagine the economic data available when the what people spent money on and how much is available in real time instead of the several months of lag we now have
Gracias
as always, you are so welcome
Bro that pic of Gavin testifying in congress was sick lol, looks like he's about demand the truth from Jack Nicholson 🤣
Hi
Hi
hi
David and Tim.
is that... what you've named us?
Udio is still ahead of suno in terms of vocals
this new update really seems to change that -- at least in terms of how they sound
01 is still not reasoning....
i mean.... it kind of is on the way there if not there?
@AIForHumansShow no, it's a parlor trick. It's still linear thinking. LLMs don't reason. They cannot. No matter how recursive you make them.
Humanity is now over. Ai is now making copywrite strikes against UA-camrs. You have to pay $30 to cntent all of your music. It may be $30 per song, IDK. because now anyone can take a small clip of your song, upload it and make a different generation that will sound only slightly different but within an hour your music will be taken offline with a copywrite strike and your music wont be yours anymore. And theres nothing you can do about it. The cards have been stacked against musicians and people involved in making films. When You make music in AI you dont own a copywrite. If you make money with the music. it will be shut down or the money can go to the copywrite owner who is microsoft and google. Goodbye cruel world
What is China going to do with these robots?? 🤔
prob take over everything, nbd
Stalling for the public but research APIs are leaping ahead, AI has become too dangerous for general use by the public.
this is kind of my conspiracy theory a bit -- it will get stronger and stronger behind the walls but what we get access to will not be as strong
@@AIForHumansShow I actually am under a few research API access programs for OpenAI, LLama, Grok, and Google's new program, even with research access the models are highly capable but many of them, especially OpenAI, gives you a severely limited amount of access tokens.
Don't pin me
trying that reverse psychology thing
you guys are becoming my favorite AI channel bc of the consitency in quality and reporting.
THANK YOU haha and you know what we don't pin a ton of comments but that's something we might try soon!
@@AIForHumansShow
hey...I tried...but maybe you can make a vid about why you don't...always up to learns something new about AI
xAI Grok 3.0 trained from 100.000 Nvidia H100's is coming out hopefully in December and they are adding another 100.000 Nvidia H100/200 so I don't think AI will slow down anytime soon
What for a week? The robots need to get here soon the hype train is slowing down.
Yeah it's tricky -- the idea that we can see a crazy future doesn't make the present move any faster
@@AIForHumansShow The anticipation of AGI really makes the present feel excruciatingly slow. It's like being trapped in a waiting room for years to come. 😥
That is exactly how it feels a lot of the time. Like we *know* what's coming for the main course but the appetizers (while delicious) aren't enough.
good
yes good
It's not slowing down. Lol. At all.
well that's *kind* of what we're saying in the video isn't it?
No
I mean.... you're prob right.
STOP the " comedy", guys. Be serious, offer GOOD reports and insights, go easy on the ads and then, MAYBE people will accept you.
haha well you can either take us or leave us kind sir. also, we have no ads??
I thougt you both normies
we're like half-normies, half shrek-afficiandos
I am unsubscribing from your channel due to this clickbait, Shit titles like this harm the entire AI industry.
What's clickbait about their title? They ask a question even the companies training AI are starting to wonder?
what the heck? this seems seriously disingenuous, we really do try to make sure we don't post click-bait headlines and just cover it as best we can. sometimes titles on YT are freaking hard to get righ but I guarantee we're not click-bait practitioners
All AI is illegal
i appreciate your commitment to this
@@richiebricker decentralized Autonomous AGI Live AVAILABLE registere ro access
Ugh. ai music. what a nightmare. asking a generative model to punch up your song is not 'making music' and personally i think ai where it applies to the arts should be banned or at least shamed. Right now it's still totally cringe, wont take long though. I think we'll still value authenticity for some time.. but who even wants this? call me old fashioned but id rather keep the arts for humanity. not to even speak of the dubious means in which these models have been made possible...
so a few quick things on this:
1) ofc, all opinions matter on this and we take comments like this seriously and understand why people dislike (or often even hate) ai stuff that crosses over with creativity
2) to the who wants this question... i think the argument in my mind has to do with what comes out of the tool rather than questioning the tool itself? i think there is a world where this opens the door to way more creative people making stuff and that i'm here for. i guess what i'm saying here is that i don't see this as just the machines making the music but in collaboration with us? to me the authenticity is still in the human intention to make something. argument could have been made around something like logic or garage band as well -- they're not even playing real intstruments! etc
3) the training aspect of this is something we've talked about a ton and it IS dubious and the part that trips us up the most but it's a complicated and semi-nuanced conversation but agree that the initial starting point isn't great
thanks for taking the time to comment here and voice your concerns, we really do try to make sure we're approaching these conversations from a balanced place
@@AIForHumansShow Thanx for the kind response - I love civil discourse! Well, we may agree to disagree on some of this and thats ok. For me when a human stops 'operating' the machine, using their experiences and knowledge and intuition to make the many choices and decisions that are the building blocks of a song.. then are we really still 'making' the thing as a peice of art that you can proudly put your name on and represent as *your* art? Its more like commissioning or hiring a session player or a shadow producer or a band member. I suppose Im a purist, but in my specific domain of the arts (electronic music) it has always been sort of a DIY kind of affair. To me the means of creation *IS* important, and many 'producers' are shamed, and rightfully so.. for hiring teams of actually talented / experienced ghost producers to effectively write and produce 'their' music for them. For me AI assistance smells the same.
All machines technology and ai blocked with agifgap an agifgar 😂😂😂😂😂😂😂 live
yes yes YES
@AIForHumansShow it's true right
@@AIForHumansShow it's true right