NEXT VIDEO: "How AI Learned to THINK" ua-cam.com/video/PvDaPeQjxOE/v-deo.html SUBSCRIBE: www.youtube.com/@ArtOfTheProblem?sub_confirmation=1 WATCH AI series: ua-cam.com/play/PLbg3ZX2pWlgKV8K6bFJr5dhM7oOClExUJ.html Timestamps: 00:32 hofstader's thoughts on chatGPT 01:00 recap of supervised learning 01:55 first paper on sequential learning 02:55 first use of state units (RNN) 04:33 first observation of word boundary detection 05:30 first observation of word clustering 10:10 sentiment neuron (Ilya & Hinton) 12:30 transformer explaination 15:50 GPT-1 17:00 GPT-2 17:55 GPT-3 18:20 In-context learning 19:40 ChatGPT 21:10 tool use 23:25 philosophical question: what is thought?
please never stop teaching on the internet. i can tell you have an intense passion in learning. im in love with how you explain concepts and their connections. feynman would be proud.
@@ArtOfTheProblemthat. could not have said it better. it really was eye opening in this hot field. that begs the question, are there alternative models for reasoning? the fundamentals today are, as you explained in the video, prediction and attention learning, that is, deducing the next action/letter/whatever given the previous states of the world/space/whatever, or to be exact, the previous mental states of that, and learning how to filter out and fragment the data into a meaningful way to, to be honest, only aid the prediction my question is, other than prediction, what could it be? I'm very fond of seemingly simple and Impactful philosophical questions, and this one really hits because then if the answer turns out to be "not really" then... our brains are just wired to be good just at that honestly that would not be too groundbreaking when you start digging in how the brain works together with the body to make up for a human I'm really just writing my thoughts here that also brings the good old question "are we just monkeys with bigger brains?" and by the looks of it yes, but also the inner workings of the brain-body connection may just be the key difference truly an interesting topic while I'm at it, I see little to no video/interest in one specific approach we could tackle machine intelligence and that is, taking our brain, and building it with silicon and wires building the hardware to be the brain itself would you be interested in such topic? unfolding your findings and sharing them as well as providing a state of the art? that would be awesome
As an AI researcher and developer, this is the first video that I did not leave thinking that the author was just saying words without second thought. There is so much misinformation in this space stemming from the fear of the ramifications of AI that are a result of the negative feedback loop of those same people. Well done.
@@ArtOfTheProblem Are you too going to kiss now? Card catalog 2.0. Do you guys remember those old newspaper cell machines? Where you could search for a news headline then look up the cell for the news paper pages related to that search? That is what AI reminds me of. A more advanced card catalog. While I am sure it will be great in that function and even in mimicry of humans I don't believe it is thinking. Most humans are so narcissistic that they believe animals do not think and yet those same dullards are convinced ai does because it is a popular narrative.
There only a handful of youtube channels that can make such concepts accessible to everyone with a curious mind (e.g. 3blue1brown, veritasium, Sabine). Art of the Problem is one of them. Brit you are a legend. Thank you for giving us this series.
@@ArtOfTheProblem Man absolutely great video, I am new to your channel and instantly subscribed... only drawback for me personally was background music... those annoying tones sounding like from 80-90 games made me download your video use AI to distinguish between narration and background music to get rid of it without losing narration and listen adjusted version offline.
Great video, but please fix your audio mix. The background music is way too loud. There likely is an option for audio ducking in your video editing program that will automatically lower the background music volume when foreground audio is playing.
Yes, please. If the primary goal is to spread knowledge, I think the background music is taking away from that. I really want to watch this, but the music is extremely distracting.
It actually help me get into a thinking mood as the audio help me stay longer into my head when I question myself about specific topic mentionned in the vidéo. Might not be the case for everybody tho. I have ADHD and i'm auditive, if someone else with the same characteristic feel the same, let me know :)
That's about my story too. I would lay in bed with the iPod Touch and watch your cryptography videos, which in turn got me into owning bitcoin in 2011. I was so young and couldn't predict what would happend (my internal LLM was not trained enough yet) and I gave it all to some guy online as a donation for their FTP which had some hot stuff in it. The end is not the goal, it's the journey that's interesting. Your video got me emotional and felt right. I'll share around. Thanks for producing it!
this channel changed my life about 10 years ago when i found the language of coins. By serendipity, I've found this channel again while im formally studying computer science which this channel inspired me to do. Thank you
Very nice! ...well, except for the loud background music anyhow. Google kept pushing this video into my list over several days while I was very busy following something else. Finally I popped it up and saw why. Google knew that I would like it. Kind'a creepy, really how it knew it was exactly the kind of thing for me at this moment in time. It's like, it is following my studies on this subject and knew this should be the next chapter. Your video reiterates over everything I have learned so far and adds perspective.
thanks for this feedback, I'm so happy to hear it was relevant to you. interestingly enough this is the first time in 12 years google is recommending my video strongly...and it's amazing the difference. I used to get 1k views per day, now it's 20k usually people hate my music, but not all, so I try to find a balance
@@ArtOfTheProblem I liked your music. It fit perfectly into the narrative. Just when I was hanging on to every word 'cause it was so interesting, the music became intrusively loud at some points and destroyed my focus.
One thing i like about your videos is that you are very good at explaining things and the way of presentation is such that it is intuitive to any age group . Although there are thousands of videos and articles are available related to the same topics but this one thing makes your videos unique and best. Thanks Brit. Please keep making such videos in future.
Explaining this in a way most people can understand is paramount, you have taken time out of a very large number of peoples day, and you have used it well.
thanks would love to know what you'd like to see next
11 місяців тому+1
@@ArtOfTheProblem It seems to me that further exploration of this topic may run into an epistemological barrier, so what would you say about explainging to us how you got this knowledge? Maybe linguistics, philosophy or congnitive science? Btw, could you clarify what you mean the statement 25:15 (We either look at something that looks like though or It is though)? It kind of reminds me of Daniel Dennett's "Real Patterns". ps.: Loved this video sooo much
Quoting one of the hero's of tech (Hofstadter ) makes this very well executed video that much more compelling. T9 (predictive testing) is the first really useful llm ish (not a word wheel but a predictive wheel from numbers) tech that end users experienced. The notion this simple premise is how it all works is nothing short of astonishing.
This is the most perfect video on the entire internet. I will use it to elucidate people on the topic. If I was rich I'd pay you big for this. Thank you sir.
thank you thank you, what a compliment. this comment made my day. If you want to support a tiny bit you can here: www.patreon.com/artoftheproblem but just thankful for this comment :)
A lot of people in my classes (as a cs major) look at me like a crazy person when I start talking about latent spaces, geometric concepts in AI, and Douglas Hofstader's views on cognition. This video makes me feel understood :)
At least Douglw Hofstader is able to play with something like ChatGPT4 and be like "Looks like I missed something". I don't see John Searle ever do that! I also believed that what chatgpt 4 does today was always going to be impossible on binary hardware, that just manipulates 1 and 0.s But I was clearly wrong!
Excuse me if I'm missing something, because I haven't really dug deep into Hofstader's work; how is this proof that he has missed something? I think the basis for which he describes a lot of phenomena, the strange loop, is very much intact all throughout the concepts explained in the video!@@KainniaK
@@ArtOfTheProblem I believe that consciousness (and everything that comes from it) is pure language processing. To me, things like thinking, reasoning, thoughts and consciousness are just language. AGI? Why don't people believe AGI is a couple of scalling steps away? To me, it is not a matter of making machines as inteligent as humans - we should adjust our thoughts: maybe we are not intelligent, at all. The thing we call "intleligence" is just something that emerges from language processing, in systems that are large enough. How do you know you can think? How do you know you're conscious? I tell you how: by interpreting sentences in your brain. (sentences can be made with words, images, feelings, symbols, etc --- should you call them "tokens"?). So, in short, we are MeatGPT :) Cheers!
TIMESTAMPS: 00:05 Neural networks learned to talk, leading to more general-purpose systems. 02:30 Recurrent neural networks (RNNs) use state units to create a state of mind that depends on the past and can affect the future. 04:52 Neural networks can learn word boundaries and cluster words based on meaning. 07:10 Language models saw limited progress until 2011 when a larger network showed the potential for higher performance. 09:34 Neural networks can learn language and complex concepts with minimal human intervention. 11:43 Neural networks struggled to handle long-range dependencies in text sequences. 14:02 Neural networks use distance in concept space to find similarities and adjust their meaning. 16:20 Neural networks with self-attention and fully connected layers can generate coherent and contextually relevant text. 18:27 In-context learning allows changing the behavior of the network without changing the network weights. 20:38 Language models like ChatGPT are more than just chatbots, they serve as the kernel process of an emerging operating system. 22:41 Training networks on prediction empowered by self-attention leads to a more general system that can be retasked on any narrow problem. 24:43 Deep learning community is divided due to differing opinions on the nature of AI's linguistic abilities and thought process.
This is easily the *_most_* *_cogent_* video on the topic I have ever seen. (I even sent this to my mom!) Hard to believe you've had these amazing topics for 12 years and aren't cracking 100k yet. Count me in for the journey!
thank you and thanks for getting ma on board :) - I know I had given up on the 100k+ break out, so took around 2 years off while I slowly made this video, and now I'm getting a signal that I should come back and push hard for a year to see if I can come back to life. one questions, ask your mom what she thinks, because my dad said it's good but I need to make a "for dummies" version without the extra super details...so i'm thinking of doing that for my whole AI series since it's something we badly need right now.
@@ArtOfTheProblemGood idea - LLMs for dummies - and the same for children. How to explain LLMs to kids? This LLM speaks like a human but isn't a person, or even if it is, it doesn't mind dying and can be treated like a machine... what a thing to tell my 2nd grade daughter. I've described it as a prediction machine, but the mirror analogy here is also very good. Any more thoughts on this could be useful, as we wait to see how our children will exist alongside the children of... whatever this is. Then again, children accept and adapt readily to whatever is there, so maybe educating adults first is a good priority, although a dummies version and a kids version might be about the same thing, minus any mature content. Good luck!
I can't wait for people to bash enough the API of OpenAI to extract and use it to train another open source GPT4 we can actually use in our desktops to do what we really wanted in our wildest dreams of experimentation. Llamma2 gets closer to GPT3, but we're playing with yesterday's tools. Maybe we should create a peer to peer system to create a shared GPT4 with our RTX3060s. Who would get into this with me ?
I really appreciate you doing so, I'm trying to revive my channel. Do you think I also need a simpler version? I was going to maybe do a recut of my whole AI series into a shorter video for those who need a full orientation, but with less detail...
@@ArtOfTheProblem to my tastes, no - I wouldn't say you need a simpler version. But I'm not the best person to ask on that score. I have a strong preference for being more verbose and thorough, to the point of being occasionally told I sound like chat GPT. Your instinct may be a good one as the general public is concerned? Sorry that I'm no decent guide there. Please do make more videos though! I'll happily subscribe 😊
This is the first video I watched on this channel and the quality of content, communication, analysis, depth, and even the audio selection is soo on point. Hooked from start to end. Immediately liked and subscribed.
This is such a good journey through the papers that got us to where we are with LLMs. You've done an amazing job at picking out the key advancements all while a crazy amount of research had been happening. I've long preferred the definition of intelligence as the ability to better predict the future (minimise surprise) since before GPT3 so imagination (world models) is a key part of it.
|Thanks for sharing, yes it's been a flury of research. So much research I did I didn't include here as well. I agree and I never liked definitions of intelligence which are long lists of skills
I will be sharing this with everyone I know with a tertiary interest in LLMs. The best high level human understandable explanation of the history. Love it. Thank you. Please keep gracing us with your videos!
Dan I really really mean it when I say I appreciate this. I thought my channel was dead, spent 2 years on this and so all I can hope for is people share it. Because of how well it's doing i'm committed to doing this for another decade !
This video is just fantastic, extremely up-to-date and very useful. It very well resonates with the discussions I am having now in the community. I would love to see it extended, once history gets written.
Very entertaning video. I started researching neural networks about 45 years ago, and quickly realized that there was much more complexity to a system that could add value for me than the software package that I purchased, and much less functionality and adaptability included in the software model I was using than what I needed to make the system actually useful. But the degree to which the system needed to be expanded should have been obvious, when we realize that the human brain has hundreds of millions of neural networks included to make a brain fully functional as we understand a fully functional human. We've come a long way, baby.
And yet, we work on only a ham sandwich for lunch worth of energy on physical hardware the size of our heads! So as per usual there's still so much more that we don't know.
I remembered your channel from your cryptography videos, jumped right in right away and can easily say that this is your best piece so far. With such a good production, I completely understand that it took 2 years of research. You always summarize the technical details and key breakthroughs in a constantly interesting way through the whole video without being overbearing.
Thanks so much for feedback, it means a lot to hear it. I was worried on striking the right detail level here and went back and fourth so many times. I had entire chapters on hopfield networks I cut.
@@ArtOfTheProblem I watched all of your cryptography videos and I have even returned to them after graduation for mere pleasure. I have no idea why your channel didn't take off. I would really appriciate some videos about logic, fallacies and limits of the human brain (caused by our evolutionnary history)
Great video, nicely put together and well explained. I've been an enthusiast since 2009 and remember all these milestones. I've built several frameworks myself and have pondered ANNs for a much bigger lenght of time than I would be willing to admit. Regarding the final question in the video I can say for certain that yes they do in fact understand. It depends on the neural net what it is in detail that they do understand, but there can be no doubt that they actually understand their task. And not only that, they effortlessly understand it to a much higher degree than we do.
so cool to hear from practitioners... intersting that the closer you get to it, the more you lean towards your view. and people on the "outside" seem to go the chomsky way
Just wanted to say I love you guys. I’ve been watching you guys for almost a decade, when you guys were the only computer science channel on UA-cam. You guys deserve WAY more attention than you’re getting
wow an OG! thank you so much. I was happy to see that this video did better than all before it. working on a follow up now focused on RL. thanks for the outreach
It’s surprising how little public coverage of the engines of AI, what is known, what remains unknown, the debates, histories and controversies, exists. Thank you for a fascinating and enlightening video.
I appreciate this. Working really hard on a follow up and feel a bit buried in the details. It is interesting that nobody cared about AI based on neural networks until just a few years ago. when I was in school there wasn't even a textbook (2007)
While I think it's reasonable to say "if it looks like thought, it is thought," I also think it's important to distinguish whether it looks like thought to Joe Average, or to a specialist in the field. That said, for the moment these models absolutely are mirrors of our thoughts. More specifically, the thoughts we enter into those prompts guide it's responses, and the chain of thoughts guide it's learning. I don't think it's a particularly difficult line to wrap your head around either; AI is a reflection and extension of the thoughts you put into it, so if you want to extend and refine your thoughts you can use it when you have no other ideas.
@@ArtOfTheProblemHad to stop watching because the music was way too loud and annoying. Your explanations are excellent but I won’t be able to benefit from them.
This is an amazing recounting of a lot of history in a short time. I appreciate how it aligns with the things researchers have said and references some of the most important papers and breakthroughs. One of the things I took away is how user inputs at run time do in fact change the model, on the fly, as it were. That may be obvious to people in the field but I don't think most laypeople know what a self attention layer is.
As an AI researcher and developer, this is the first video that I did not leave thinking that the author was just saying words without second thought. I think SmythOS one of the AI agent it s has a great features
Truly an exceptional video! Incredibly clear and engaging presentation of the history and relevant ideas, well narrated. This was very thoughtfully written and thoughtfully delivered. Thank you!
@@ArtOfTheProblem The time and effort *absolutely* shows. I was pinned for just about every minute! This is really high quality work, it definitely made my day. I hope you can feel rewarded for your effort, because you deserve it, man. I also really enjoyed the way you showed the progression of not only the architecture and "design", but also the capabilities/outputs of these models, too. It really gave a good sense of the history in a way I haven't really seen anywhere else.
Been a big fan of your videos for years! I'm always delighted, even as I rewatch some of your old videos time and again, not only by how you distill a complex concept down to its essence but also by how you transform that which is abstract into something extremely palpable and relatable through filmmaking techniques.
This was a fantastic video, learnt more about the history and buildup to ChatGPT in these 30 mins then I've read anywhere before. Learnt a lot, and very thought provoking towards the end. Thank you!
thank you! i'm just happy it has the views it does, for years the algorithm ignored me and this one might hit 250k in 1 week which blows my mind and inspires me to make more
The disagreement at the end reminds me of a conference that happened back around 1968 with a very esteemed collection of computer scientists like Christopher Strachey, John McCarthy, Adriaan van Wijngaarden and others. At the end of one of the talks there was a fairly heated debate about what a number was - it was the denotationalists on one side and the operationalists on the other. I read this in the conference proceedings, which included the transcripts of the Q&A and discussions, back in the 1980s, so may be misremembering exactly who was there. At the time I was doing a course in formal semantics lectured by an associate professor who was a remarkable polyglot and ex-Jesuit monk, and in my final exam instead of answering the main question I wrote about this discussion, and he gave me 100% 🙂
I wish I could. We had 3 books of conference proceedings in our school library (University of Cape Town) for whatever conference this was, covering 68, 70 and 71 IIRC. I found and read them in the mid ‘80s. But I don’t remember the conference title.
Great video by the way, here are my thoughts on the last issue. The first time i tried LLMs, i really thought it had some kind of intelligence and it blew my mind off like everybody else's. Because previously i tried a lot of advanced AI models to create a Jarvis like in Ironman for my personal project. It never worked all of them were very dumb and dull. But as i approached the new LLMs with more demanding reasoning and logical tasks it outright failed the tests. The larger the LLM the better it is at hiding those flaws. But the key point is the classical issue that we point the AI at, the semantic understanding of words in LLMs are just a property of vector distance in its multi dimensional space memory. It only knows what comes next probabilistically. It don't even know what is talking about other than churning out words that might come next. How do i say it, its like it doesn't have a mind of its own, but only a sophisticated system that can only grasp a surface level meaning of words in a language. When testing smaller models these flaws become very obvious, because it doesn't know how to build knowledge from existing semantic words. It can't synthesize information logically because it doesn't have the ability to grasp the full meaning of its words. but i think its a right step towards artificial intelligence and LLMs are just the tip of the iceberg. I don't know what the future is anymore. I don't dare to predict, the fast paced research in AI is scary and exciting. Also IDK if we will ever achieve human level intelligence, but we will surely achieve machine intelligence that is good at mimicking human intelligence
I wouldn't say it doesn't know what it is talking about. It knows relations between words, that is at least some knowledge. After all, most of our science is based on relations and logical reasoning, not requiring real understanding (like famous "shut up and calculate" in regards to quantum mechanics). It probably doesn't understand what it's talking about though, because understanding is property of cognition, and it definitely does not have cognition, even though it have a few of it's derived properties. But yeah, I agree, it seems that LLMs is a great instrument and stepping stone for true cognition. Especially when it's multimodal. I see it like a data bank for true intelligence. For now it's working linearly in comprising a data in shortest way that seemingly makes sense through semantic space. Like in a straight geodesic line. While cognitive AI would be able to actually see and "feel" that semantic space, the possible ways, and direct speech through it, which basically is making new ideas.
I believe you are hitting the same limits which Yann LeCun is concerned about. "It don't even know what is talking about other than churning out words that might come next". As it turns out, because human minds (up until 2020) wrote all the sensible text available, having to predict he next words does require *some* aspects of meaning as humans call it. The experiments of the LLMs show there is something there. But not everything we would want yet. Their neural architecture is diverging away from how human brains work---which is both more powerful in some aspects but in others substantially limited by biology compared to silicon. Humans "context window" is much smaller and less precise than the 100k tokens which a LLM can accomodate now perfectly with no forgetting, and yet human performance can still be better. We must have better algorithms and better meaning extraction to make up for worse hardware. But already there are plenty of natural humans worse at language than a LLM---and the success of the LLMs more shows that most human talking and thinking is not so sophisticated. Only a small bit is truly deep. Yann and friends are searching for a quantitative, optimizable technique which will give results qualitatively better than iterating next word ahead probabilistic prediction, or at least include it as a subproblem along the way (which already gives good results).
Modern neural networks are essentially machines to *extract* meaning from existing (big) data and store it encoded it in its internals. Then we are able to interact with this storage of meaning through UI.
Hofstadter is one of my favorite authors on this subject. Gödel, Esher, Bach is highly recommended. Personally I used to think there must be some intricate function in the brain that created intelligence, possibly even at the quantum level. Now I believe it is just a function of sufficient complexity. Whatever happens it will sure be interesting.
I remember that book had a huge impact on me right around university year 1 or maybe end of high school...i don't think I ever finished the last 3rd, there was so much to chew on in the first half
This channels creator is so artful in their presentation of such a complex topic. Definitely underrated. Such is the price of avoiding garbage algorithmic gaming such as clickbait. You earned my subscription. Keep making great content 🙏
Did not expect this video to discuss a concept I've been researching a lot. You brought up in-context learning whereby a static system isn't necessarily static (each instance can still learn things), then you also brought up that LLM output is on the thought layer rather than the spoken layer. The prompt is the program is such a nice succinct way to put it with the calculations being done on the context, therefore each context is individualized while the model is general. Personally, I have been looking at them from a psychological and philosophical way where the context is the Self (one's identity depends on their memories) which makes every chat instance an individual. Then I've been thinking about the model itself being more like the Lizard Brain where these unconscious guidance's occur. So, something like the fear of spiders or reproduction knowledge would be on that level. LLM are loaded with a large inherent knowledge with a tiny memory window, humans have a tiny inherent knowledge with an extremely large memory window. Although human's also have a large amount of memory processing going on meanwhile LLM currently have none (though people are working on it). I wonder if a LLM context window is more or less pure memory while humans make do with incredibly efficient memory? I think in your closing, both camps are correct. It's just the object of focus that I think they are incorrect about. The prompt IS the program that makes it a thinking thing, the model itself however isn't a thinking thing as it must be given what to think about. I also think the ones thinking it is incapable of thought likely have an external mechanism in mind that it lacks like it not having a concept of time (see circadian rhythm, time blindness, or true unconsciousness in humans), or it not being able to directly see (see visual cortex (or for its internal world view see aphantasia)), lack of survival instinct (trained to be a chatbot while chatbots aren't supposed to have survival instinct (humans are guided by the amygdala for fight or flight as well)), ability to tamper with its messages/thoughts (see the unreliability of eyewitness testimony, confabulation, or the misinformation effect) . I honestly don't think there's any mental structure that a LLM doesn't have that another human also doesn't have besides the memory structure that we humans have, meanwhile people keep thinking of it in human terms, but it can never be human with such a massive amount of inherent knowledge. Apologies if this is rambley, incoherent, or beyond the Overton window but this is the first video I've seen that even mentions the two concepts at the beginning of this long comment. This is the tip of the iceberg of comparisons but still it's fascinating.
Brit! So excited to have you back. You have no idea how much the videos for Information Theory have changed my life. I'm now considering leaving the industry to go back to school in Berkeley to study AI. Particularly in the domain of learning language outside of text (non-verbal communication such as body language and prosody). Both critical in storytelling and eliciting emotions like Pixar does!
Phenomenal explanation of LLM's. Im in the camp that believes it will not truly ever be thought in the same way humans think (Conscious thoughts) , but if we simulate thought in a machine, to an outside observer, there is no difference. Whether or not it is actual thought is irrelevant if the simulation of thought is just as good as the real thing (humans) I believe we have broken the hardest obstacle to AGI, and it is only about scaling larger models, adding synthetic data, using AI's to train other AI's, and combining learning techniques at this point until AGI is achieved.... AGI is near. PS I think Noam Chomsky is stuck in the past and is dead wrong about LLM's being nothing more than autofill.. GPT 4 alone has demonstrated complex resasoning skills and theory of mind..These 2 things alone disprove Chomsky
Humans are trained to think by society and the environment, the facial expressions, we make, the conduct we show are all trained by the environmental factors and knowledge. AI is the same way is being trained. There is no independent think without the environment is the programming language.
I was still in high school and remember being amazed by Eliza in 1970 when I went to an Open Day at our local university and watched people typing to the program on an old teletype machine. It seemed so "human" but looking back, all it did was transform the users comments into leading questions. That day was one of the pivotal moments in my life that lead to a long career in computing.
@@ArtOfTheProblem ... What is thought? I don't think it is restricted to "organic" neurons. Our brains are made of atoms just like a computer is based on atoms. There is nothing spiritual or paranormal about our brains. I think one day, and that day seems to be fast approaching, AI will suddenly become self-aware and conscious. I have seen a snail's pace growth in AI and it's predecessors over the past 40 years but suddenly over the past few years the developments have been explosive, especially this year. Will we hit a plateau or will the growth continue to be exponential? That is THE question. I think when AI is provided with sensory inputs its ability to learn and understand will be greatly improved. Being connected to the internet is an important starting point, but an actual robot body with AI. Hmmm. I wonder where that can lead. I am getting a bit long in the tooth but I hope to see AGI become a reality before I pass. Artificial consciousness I am not as confident seeing before I die, but could be wrong. What will the next 10 years bring?
@@ArtOfTheProblem ... I remember studying very early versions of Expert Systems in the mid 1980 for my Masters degree. They were a very primitive form of AI but nothing compared to ChatGPT or Bard or many of the others available today. I can't remember the name of the system we studied other than it used an Apple (Lisa?) computer with a very large high-def B/W screen. It was impressive because we could fit all our rules graphically on one page. Fun times indeed.
@@ArtOfTheProblem ... My first job in 1973 almost straight out of high school was as a "trainee" COBOL programmer on a Honeywell H200 mainframe. It took up a room and had 20k memory, 4 tape drives, a punch card reader and a line printer. CPU clock speed was 1Mhz. They wanted to increase memory to 24k but it was an exorbitant expense which the company did not go ahead with. That machine ran all of the data processing needs for a medium sized company. All the COBOL programs were developed in-house by a group of 6 programmers and analysts. Each program would take perhaps a week or two to write and debug. We only had access to the machine for one compilation and test run per day so I remember spending hours single stepping through my programs on large sheets of butcher paper before submitting runs on the mainframe. There was a prize offered by the manager for any programmer that managed to get an error free compilation on the first run. Nobody ever won it. I stayed there for about 18 months then changed jobs to another Honeywell site mainly because of a huge pay rise and because the place I was headed had a computer with two disk drives!!! I still have some of my old programming manuals from back then.
So happy to see you’re still making videos like this. I remember the first video of yours I watched - iirc it was something about the history of language (or logic). Your style has gotten better, and you’re crystal clear about what you talk about. PS - don’t listen to those people who hate the music. I LOVE IT !!
Hey! If you can help share my new video around any of your networks today it might catch fire and would help me support the channel. I appreciate your help! ua-cam.com/video/PvDaPeQjxOE/v-deo.html
Fantastic essay on the current state of Ai .. sadly it’s difficult to find channels that explore this depth WITHOUT being boring or overly dry in technical explanations. My only criticism is the bkgrnd music is a bit loud.. nice choices though
@@ArtOfTheProblem It's interesting to look at the correlation between cheesy sound effects and musical cues and the "engagement graph" shown above the timeline/progress slider control. Those engagement peaks aren't necessary a good thing. They tell you where you're losing people, forcing them to back up the video. Please give some thought to whether the music is really doing you any favors, or distracting from what was really a superbly-researched and nicely-presented video.
yes this was me going a bit too fast on sound pass, 99.9% on researcher 0.1% sound levels... funny to think that would trick the algorithm into higher retention graphs...i honestly didn't think this video would have the reach it did, it if could swap sound I would. i wish @@NoahFect
Some people have big egos, which is why they refuse to believe that AI, whether in its current form or future versions, could have the same or even greater capabilities than humans. However, AI doesn't care about their egos-nor does the universe.
What I love is I can free form write what I want to write and take my normal 30min to a hour. THEN I can give it to CHatGPT and have Chat GPT fix all the awkward phrasing, the placed I repeat myself, and clean up the writing in general. Usually it takes me hours to rewrite and edit myself, ChatGPT does it in seconds. Thats its real power. It still needs a good input to come up with a great output. It cant create the inital input.
Yeah we use it for professional documentation and just like you said we write something up and ask it to edit and maybe adjust for tone depending on the audience. But it’s all our thoughts, just reworded.
I really appreciate the feedback. I'm thinking of doing a shorter, more general/broad follow up video that helps orient people new to the field, and looks ahead a bit more. any thoughts?
I suspect our egos are the thing that is dividing the community, the idea that we are simply the expression of the number of neurons and connections in our brain is deeply humbling.
Spot on. And people will keep moving the goalposts. We hold humans on such a pedestal of intelligence as if the average person isn't a dumb stochastic parrot who constantly fucks up.
This is an amazing summary perfectly highlighting the important steps of the progress and how we got here. Working in text summary AI systems in the early 2000s and following the progress over time this video focused on all the crucial breakthroughs.
Once again! Fantastic content. I was captivated the whole time. Love your method of computer science story telling. You have something special here and I hope you are able to continue creating this content. I have subscribed to your Patreon.
Beautifully and well explained. Its so difficult to hook people for 25+ minutes content but i am sure, you have made it possible for a lot of people with so well detailed and well put content.
Great video. I’m doing an AI course at the moment and this tied up some loose ends for me. Amusingly The film THE CREATOR touches on this philosophical question. Do neural networks really understand or feel? I think we have become accustomed to machines being faster, repeatable, and more accurate than us but not until an AI extrapolates beyond its training set and makes some truly ground breaking inferences with profound consequences to human society, we will continue to see AI as just a clever parlour trick and will keep moving the goal posts. Fantastic stuff.
thanks for sharing I agree with this. I love to hear about this loose end aspect as that's what I try to do. I'm curious where it helped you specifically in that regard?
Loose ends for me included: 1) The history behind the build up of LLMs. I did not know about the research origins of Recurrent NNs. 2) How these early networks learned 'semantics' based purely on guessing the next word 3) The sentiment neurone emerging 4) How the paper "Attention is All You Need" fed in to the origins of GPT models and maintaining context over long sequences. 5) the difference between In Context Learning vs In Weight Learning - Nice you are looking at the original papers too (I assume thats your yellow high lighter pen!). I kind of knew about all these things but getting all this in to 26 minute 'context window' really helped! A random thought - I wonder if LLMs will be able to generate a 'tech-tree' - the type you see in those sim-games based on the semantics of ideas and be able to orchestrate the evolution of ideas of our human culture through the ages from the Greeks to Modern day - not using research reference links, but based on the semantics of ideas (the dna of the meme).
The reason for the discourse is due to lack of uniform definition of intelligence, thinking and reasoning. One considers it as thinking while another does not. I wouldn’t call prediction on semantics thinking. Thinking is a much more complex process. chatGPT cannot reason. It even often outputs misinformation and contradictory statements(not because of training data, but because it doesn’t validate the logic). Lack of validation is just one of MANY aspects why I can’t call it thinking/reasoning.
@@ArtOfTheProblem Wikipedia has a proper one. “Intelligence has been defined in many ways: the capacity for abstraction, logic, understanding, self-awareness, learning, emotional knowledge, reasoning, planning, creativity, critical thinking, and problem-solving. It can be described as the ability to perceive or infer information; and to retain it as knowledge to be applied to adaptive behaviors within an environment or context.”
i don't like basket of words definition. I'm sticking with 'ability to learn'...for example if I brought home a robot from the store which did X, then I'd call it intelligent@@mutexin
@@ArtOfTheProblem The reference point is human intelligence which is very complex. There can be no concise definition if you focus on accuracy. The simpler the definition, the more inaccurate it is.
I don't agree with this angle but it is indeed interesting. it's definition is it can't be defined....i'd agree with that over the basket of words definition@@mutexin
"understand?" "think?" "conscious?" "intelligent?" "qualia?" "AGI?" "sentient?" Until we agree on precise definitions for these things, they are fundamentally meaningless in discussion about any specific system. And they are maddeningly non-precise. The only thing such arguments reveal is differences of opinion about what meanings people attach to the questions. I have precise definitions for some of these. I don't expect anyone else to have decided on the same definitions. I can't argue on the basis of them whether any system "understands" anything or "is conscious" because unless anyone else is using the same definitions the discussion that would start can only go in small meaningless circles.
It is interesting listening to a video talking about how AI is trying to mimic human thought as a thought stream... and be interrupted by an ad, while explaining it, breaking the thought completely :)
Absolutely brilliant video! My hot take: Chomsky is fundamentally wrong about the concept of thought and even human thought is just glorified autofill.
Holy moly this was astounding! You balanced the tightrope of depth vs accessibility so gracefully, and the production quality is gorgeous too. Immediate sub!
![[1hr Talk] Intro to Large Language Models](http://i.ytimg.com/vi/zjkBMFhNj_g/mqdefault.jpg)
NEXT VIDEO: "How AI Learned to THINK" ua-cam.com/video/PvDaPeQjxOE/v-deo.html
SUBSCRIBE: www.youtube.com/@ArtOfTheProblem?sub_confirmation=1
WATCH AI series: ua-cam.com/play/PLbg3ZX2pWlgKV8K6bFJr5dhM7oOClExUJ.html
Timestamps:
00:32 hofstader's thoughts on chatGPT
01:00 recap of supervised learning
01:55 first paper on sequential learning
02:55 first use of state units (RNN)
04:33 first observation of word boundary detection
05:30 first observation of word clustering
10:10 sentiment neuron (Ilya & Hinton)
12:30 transformer explaination
15:50 GPT-1
17:00 GPT-2
17:55 GPT-3
18:20 In-context learning
19:40 ChatGPT
21:10 tool use
23:25 philosophical question: what is thought?
Andrew Ng and Geof Hinton already clashed on twitter, quite confrontational tone, Andrew seems to be taking the side of Lecunn.
@@michaelpoblete1415
nice skills
please reconsider the background music and noise levels, it really takes away from an otherwise interesting video.
HAL, Reduce Music Volume 🤖
I have watched countless AI videos but nobody explained it like you. Many thanks.
woo! thanks, stay tuned
next video is finally done! ua-cam.com/video/Dov68JsIC4g/v-deo.html
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
please never stop teaching on the internet. i can tell you have an intense passion in learning. im in love with how you explain concepts and their connections. feynman would be proud.
this makes me so happy thanks for saying that. I'm sitting here surrounded in papers working on next video
@@ArtOfTheProblemthat. could not have said it better. it really was eye opening in this hot field. that begs the question, are there alternative models for reasoning? the fundamentals today are, as you explained in the video, prediction and attention learning, that is, deducing the next action/letter/whatever given the previous states of the world/space/whatever, or to be exact, the previous mental states of that, and learning how to filter out and fragment the data into a meaningful way to, to be honest, only aid the prediction
my question is, other than prediction, what could it be?
I'm very fond of seemingly simple and Impactful philosophical questions, and this one really hits
because then if the answer turns out to be "not really" then... our brains are just wired to be good just at that
honestly that would not be too groundbreaking when you start digging in how the brain works together with the body to make up for a human
I'm really just writing my thoughts here
that also brings the good old question "are we just monkeys with bigger brains?" and by the looks of it yes, but also the inner workings of the brain-body connection may just be the key difference
truly an interesting topic
while I'm at it, I see little to no video/interest in one specific approach we could tackle machine intelligence
and that is, taking our brain, and building it with silicon and wires
building the hardware to be the brain itself
would you be interested in such topic? unfolding your findings and sharing them as well as providing a state of the art? that would be awesome
yubi yubi
next video is finally done! ua-cam.com/video/Dov68JsIC4g/v-deo.html
@@ArtOfTheProblem yaay
As an AI researcher and developer, this is the first video that I did not leave thinking that the author was just saying words without second thought. There is so much misinformation in this space stemming from the fear of the ramifications of AI that are a result of the negative feedback loop of those same people. Well done.
I really do appreciate this comment, i work so hard on these and sometimes i think i'm crazy - the garbge funnel of ai coverage is brutal
😂 yes other AI channels just throw the word “compute” around 300 times.
YOU'LL BE SHOCKED AT WHAT......@@RyluRocky
You are now to Chat GPT 16
@@ArtOfTheProblem Are you too going to kiss now? Card catalog 2.0. Do you guys remember those old newspaper cell machines? Where you could search for a news headline then look up the cell for the news paper pages related to that search? That is what AI reminds me of. A more advanced card catalog. While I am sure it will be great in that function and even in mimicry of humans I don't believe it is thinking. Most humans are so narcissistic that they believe animals do not think and yet those same dullards are convinced ai does because it is a popular narrative.
There only a handful of youtube channels that can make such concepts accessible to everyone with a curious mind (e.g. 3blue1brown, veritasium, Sabine). Art of the Problem is one of them. Brit you are a legend. Thank you for giving us this series.
Thanks for sharing, I really appreciate it.
I'd like to add "Artem Kirsanov" (neuroscience, neurons, manifolds) and "Anton Petrov" (science, physics) to the list.
@@ArtOfTheProblem subscribed for more :)
yay welcome to the party! sorry my output is slow though@@rikvermeer1325
@@ArtOfTheProblem Man absolutely great video, I am new to your channel and instantly subscribed... only drawback for me personally was background music... those annoying tones sounding like from 80-90 games made me download your video use AI to distinguish between narration and background music to get rid of it without losing narration and listen adjusted version offline.
Great video, but please fix your audio mix. The background music is way too loud.
There likely is an option for audio ducking in your video editing program that will automatically lower the background music volume when foreground audio is playing.
Yes, please. If the primary goal is to spread knowledge, I think the background music is taking away from that. I really want to watch this, but the music is extremely distracting.
It actually help me get into a thinking mood as the audio help me stay longer into my head when I question myself about specific topic mentionned in the vidéo. Might not be the case for everybody tho. I have ADHD and i'm auditive, if someone else with the same characteristic feel the same, let me know :)
Hah, was about to post the same. The music is hideous and distracting!
nailed it@@SimoneDesignsGaming
I don’t understand why music is played. I gave up on the video after a few minutes.
Perfect. 12 years ago or so I learned about RSA from you.
I'm glad to see you are still producing quality videos! Thumbs up!
so cool to hear from OGs
That's about my story too. I would lay in bed with the iPod Touch and watch your cryptography videos, which in turn got me into owning bitcoin in 2011. I was so young and couldn't predict what would happend (my internal LLM was not trained enough yet) and I gave it all to some guy online as a donation for their FTP which had some hot stuff in it.
The end is not the goal, it's the journey that's interesting. Your video got me emotional and felt right. I'll share around. Thanks for producing it!
ha, love the connections. I know the pain also@@QW3RTYUU
@@QW3RTYUUholy shit same here, 2011 is when I found this channel while trying to learn about bitcoin. This channel is a treasure.
same
this channel changed my life about 10 years ago when i found the language of coins. By serendipity, I've found this channel again while im formally studying computer science which this channel inspired me to do. Thank you
So so so cool. love this comment :))))
i have to ask what's the cs dept response to AI? when I did CS in 2008 there wasn't a class or even a deep learning textbook
next video is finally done! ua-cam.com/video/Dov68JsIC4g/v-deo.html
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
Very nice! ...well, except for the loud background music anyhow. Google kept pushing this video into my list over several days while I was very busy following something else. Finally I popped it up and saw why. Google knew that I would like it. Kind'a creepy, really how it knew it was exactly the kind of thing for me at this moment in time. It's like, it is following my studies on this subject and knew this should be the next chapter. Your video reiterates over everything I have learned so far and adds perspective.
thanks for this feedback, I'm so happy to hear it was relevant to you. interestingly enough this is the first time in 12 years google is recommending my video strongly...and it's amazing the difference. I used to get 1k views per day, now it's 20k
usually people hate my music, but not all, so I try to find a balance
@@ArtOfTheProblem I liked your music. It fit perfectly into the narrative. Just when I was hanging on to every word 'cause it was so interesting, the music became intrusively loud at some points and destroyed my focus.
i appreciate this, I know...i think the best is some music in and out at times...but not during moments of high cognitive load@@TropicalCoder
@@ArtOfTheProblem Exactly!
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
One thing i like about your videos is that you are very good at explaining things and the way of presentation is such that it is intuitive to any age group . Although there are thousands of videos and articles are available related to the same topics but this one thing makes your videos unique and best.
Thanks Brit. Please keep making such videos in future.
thank you for sharing, I tried to do something new here and I hope it was worth it
next video is finally done: How AI Learned to Feel! ua-cam.com/video/Dov68JsIC4g/v-deo.html
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
I'm the one solving the rubiks cube!!!!
Best dang cube solvin' hands in the west
There is no Rubik's cube.
@@eastwood451 its in the first scene, the one solving the rubiks cube is me
@@CutStudio4
I think it's a Matrix joke, but I don't really understand it
@@CutStudio4 It was a Matrix joke 😄 Seemed fitting in a video about AI... God job on the very real cube!
Explaining this in a way most people can understand is paramount, you have taken time out of a very large number of peoples day, and you have used it well.
thanks would love to know what you'd like to see next
@@ArtOfTheProblem It seems to me that further exploration of this topic may run into an epistemological barrier, so what would you say about explainging to us how you got this knowledge? Maybe linguistics, philosophy or congnitive science?
Btw, could you clarify what you mean the statement 25:15 (We either look at something that looks like though or It is though)? It kind of reminds me of Daniel Dennett's "Real Patterns".
ps.: Loved this video sooo much
really?? so cool, please shoot me an email when you are done your thesis. I'm going to think about a good follow up@
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
Quoting one of the hero's of tech (Hofstadter ) makes this very well executed video that much more compelling. T9 (predictive testing) is the first really useful llm ish (not a word wheel but a predictive wheel from numbers) tech that end users experienced. The notion this simple premise is how it all works is nothing short of astonishing.
I know what a time to be alive
next video is finally done: How AI Learned to Feel! ua-cam.com/video/Dov68JsIC4g/v-deo.html
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
The Tie fighter sound in the last scene.. a stroke of genius. Very clever!
ha nice catch! glad some people like the sounds i was having a blast
@@ArtOfTheProblem aka stampeding elephants ;-) Very cool.
Oh. G'day from Sydney, Australia.
This is the most perfect video on the entire internet. I will use it to elucidate people on the topic. If I was rich I'd pay you big for this. Thank you sir.
thank you thank you, what a compliment. this comment made my day. If you want to support a tiny bit you can here: www.patreon.com/artoftheproblem but just thankful for this comment :)
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
Wow, this is handsdown the best video I've even seen about this subject. Thank you! It is on my favorites now!
Woo! So so happy to hear this after 2 years of work on it.
next video is finally done! ua-cam.com/video/Dov68JsIC4g/v-deo.html
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
I just discovered this video and this channel. Wow! Informative, concise, not trying to push an agenda, just telling the story. Fantastic.
thank you, stay tuned for a follow up
wanted to let you know my next vid is out! ua-cam.com/video/5EcQ1IcEMFQ/v-deo.html
next video is finally done: How AI Learned to Feel! ua-cam.com/video/Dov68JsIC4g/v-deo.html
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
A lot of people in my classes (as a cs major) look at me like a crazy person when I start talking about latent spaces, geometric concepts in AI, and Douglas Hofstader's views on cognition. This video makes me feel understood :)
you are one of us :)
we too look at them like they are crazy bruh😂
At least Douglw Hofstader is able to play with something like ChatGPT4 and be like "Looks like I missed something". I don't see John Searle ever do that! I also believed that what chatgpt 4 does today was always going to be impossible on binary hardware, that just manipulates 1 and 0.s But I was clearly wrong!
Excuse me if I'm missing something, because I haven't really dug deep into Hofstader's work; how is this proof that he has missed something? I think the basis for which he describes a lot of phenomena, the strange loop, is very much intact all throughout the concepts explained in the video!@@KainniaK
Was this undergraduate or graduate? If undergrad then you expect far too much out of your classmates 💀, you are truly one in a thousand
I am amazed by the info in the video, but I am even more amazed by the quality of your video. Congratulations. And thank you.
that means a lot to me. thank you so much. Stick around for more. curious where you land on the final question?
@@ArtOfTheProblem I believe that consciousness (and everything that comes from it) is pure language processing. To me, things like thinking, reasoning, thoughts and consciousness are just language. AGI? Why don't people believe AGI is a couple of scalling steps away? To me, it is not a matter of making machines as inteligent as humans - we should adjust our thoughts: maybe we are not intelligent, at all. The thing we call "intleligence" is just something that emerges from language processing, in systems that are large enough.
How do you know you can think? How do you know you're conscious? I tell you how: by interpreting sentences in your brain. (sentences can be made with words, images, feelings, symbols, etc --- should you call them "tokens"?). So, in short, we are MeatGPT :) Cheers!
beautiful well said, and yes I like "tokens" :)@@spockfan2000
next video is finally done! ua-cam.com/video/Dov68JsIC4g/v-deo.html
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
Great vid, I had heard this timeline scattered around but this really brings it all together
thank you! I worked hard to do this, stay tuned for follow up
TIMESTAMPS:
00:05 Neural networks learned to talk, leading to more general-purpose systems.
02:30 Recurrent neural networks (RNNs) use state units to create a state of mind that depends on the past and can affect the future.
04:52 Neural networks can learn word boundaries and cluster words based on meaning.
07:10 Language models saw limited progress until 2011 when a larger network showed the potential for higher performance.
09:34 Neural networks can learn language and complex concepts with minimal human intervention.
11:43 Neural networks struggled to handle long-range dependencies in text sequences.
14:02 Neural networks use distance in concept space to find similarities and adjust their meaning.
16:20 Neural networks with self-attention and fully connected layers can generate coherent and contextually relevant text.
18:27 In-context learning allows changing the behavior of the network without changing the network weights.
20:38 Language models like ChatGPT are more than just chatbots, they serve as the kernel process of an emerging operating system.
22:41 Training networks on prediction empowered by self-attention leads to a more general system that can be retasked on any narrow problem.
24:43 Deep learning community is divided due to differing opinions on the nature of AI's linguistic abilities and thought process.
thank yoU!
Masterpiece! This channel deserves millions of subscribers.
so happy to see this video getting love
This is easily the *_most_* *_cogent_* video on the topic I have ever seen. (I even sent this to my mom!)
Hard to believe you've had these amazing topics for 12 years and aren't cracking 100k yet. Count me in for the journey!
thank you and thanks for getting ma on board :) - I know I had given up on the 100k+ break out, so took around 2 years off while I slowly made this video, and now I'm getting a signal that I should come back and push hard for a year to see if I can come back to life. one questions, ask your mom what she thinks, because my dad said it's good but I need to make a "for dummies" version without the extra super details...so i'm thinking of doing that for my whole AI series since it's something we badly need right now.
@@ArtOfTheProblemGood idea - LLMs for dummies - and the same for children. How to explain LLMs to kids?
This LLM speaks like a human but isn't a person, or even if it is, it doesn't mind dying and can be treated like a machine... what a thing to tell my 2nd grade daughter. I've described it as a prediction machine, but the mirror analogy here is also very good.
Any more thoughts on this could be useful, as we wait to see how our children will exist alongside the children of... whatever this is.
Then again, children accept and adapt readily to whatever is there, so maybe educating adults first is a good priority, although a dummies version and a kids version might be about the same thing, minus any mature content.
Good luck!
next video is finally done: How AI Learned to Feel! ua-cam.com/video/Dov68JsIC4g/v-deo.html
I work in AI, and I learned a lot from this video. Great blend of historical perspective and well-pitched explainer!
I couldn't ask for a better compliment. I really was hoping someone new as well an an expert could get something out of it
I can't wait for people to bash enough the API of OpenAI to extract and use it to train another open source GPT4 we can actually use in our desktops to do what we really wanted in our wildest dreams of experimentation.
Llamma2 gets closer to GPT3, but we're playing with yesterday's tools.
Maybe we should create a peer to peer system to create a shared GPT4 with our RTX3060s.
Who would get into this with me ?
This scare the sh!t out of me. You're being informed by a youtube video despite working in the field?!
@@NElectronicSoulGood point. I like to believe that person is just a beginner.
the field is so new and so nacent that I think all researchers have to be on youtube to keep up@@NElectronicSoul
this is hands-down the best summary of how we got LLMs that I've seen - definitely keeping this on hand to share with people who need a quick catch-up
I really appreciate you doing so, I'm trying to revive my channel. Do you think I also need a simpler version? I was going to maybe do a recut of my whole AI series into a shorter video for those who need a full orientation, but with less detail...
@@ArtOfTheProblem to my tastes, no - I wouldn't say you need a simpler version. But I'm not the best person to ask on that score. I have a strong preference for being more verbose and thorough, to the point of being occasionally told I sound like chat GPT. Your instinct may be a good one as the general public is concerned? Sorry that I'm no decent guide there.
Please do make more videos though! I'll happily subscribe 😊
thank you, will do :)@@AdamDesrosiers
I have already posted it everywhere on reddit but as usual the stuff that teaches the most gets the least amount of upvotes.
thank you for trying :)@@KainniaK
This is the first video I watched on this channel and the quality of content, communication, analysis, depth, and even the audio selection is soo on point. Hooked from start to end. Immediately liked and subscribed.
thrilled to have you, so happy people are finding this channel again. stay tuned!
next video is finally done! ua-cam.com/video/Dov68JsIC4g/v-deo.html
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
This is such a good journey through the papers that got us to where we are with LLMs. You've done an amazing job at picking out the key advancements all while a crazy amount of research had been happening. I've long preferred the definition of intelligence as the ability to better predict the future (minimise surprise) since before GPT3 so imagination (world models) is a key part of it.
|Thanks for sharing, yes it's been a flury of research. So much research I did I didn't include here as well. I agree and I never liked definitions of intelligence which are long lists of skills
I will be sharing this with everyone I know with a tertiary interest in LLMs. The best high level human understandable explanation of the history. Love it. Thank you. Please keep gracing us with your videos!
Dan I really really mean it when I say I appreciate this. I thought my channel was dead, spent 2 years on this and so all I can hope for is people share it. Because of how well it's doing i'm committed to doing this for another decade !
Very glad to hear that! 🙂. Also can't wait to see what happens with these Q* developments...
Me too...clearly building on the strength of an LLM in the loop of a larger process.@@DanHartwigMusic
This video is just fantastic, extremely up-to-date and very useful. It very well resonates with the discussions I am having now in the community.
I would love to see it extended, once history gets written.
thanks for sharing, I do look forward to watching history unfold and updating it
This is a phenomenal display of explaining a difficult concept
I appreciate this, worked very hard on it. next video will be on RL
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
Very entertaning video. I started researching neural networks about 45 years ago, and quickly realized that there was much more complexity to a system that could add value for me than the software package that I purchased, and much less functionality and adaptability included in the software model I was using than what I needed to make the system actually useful. But the degree to which the system needed to be expanded should have been obvious, when we realize that the human brain has hundreds of millions of neural networks included to make a brain fully functional as we understand a fully functional human. We've come a long way, baby.
thanks for sharing
And yet, we work on only a ham sandwich for lunch worth of energy on physical hardware the size of our heads! So as per usual there's still so much more that we don't know.
I remembered your channel from your cryptography videos, jumped right in right away and can easily say that this is your best piece so far. With such a good production, I completely understand that it took 2 years of research. You always summarize the technical details and key breakthroughs in a constantly interesting way through the whole video without being overbearing.
Thanks so much for feedback, it means a lot to hear it. I was worried on striking the right detail level here and went back and fourth so many times. I had entire chapters on hopfield networks I cut.
@@ArtOfTheProblem
I watched all of your cryptography videos and I have even returned to them after graduation for mere pleasure.
I have no idea why your channel didn't take off.
I would really appriciate some videos about logic, fallacies and limits of the human brain (caused by our evolutionnary history)
awesome, thanks for the feedback. super cool to hear@@The0Yapster
Brilliant. Clear, and authoritative but also reflecting a reassuring sense of open enquiry.
so happy this came across
Great video, nicely put together and well explained. I've been an enthusiast since 2009 and remember all these milestones. I've built several frameworks myself and have pondered ANNs for a much bigger lenght of time than I would be willing to admit. Regarding the final question in the video I can say for certain that yes they do in fact understand. It depends on the neural net what it is in detail that they do understand, but there can be no doubt that they actually understand their task. And not only that, they effortlessly understand it to a much higher degree than we do.
so cool to hear from practitioners... intersting that the closer you get to it, the more you lean towards your view. and people on the "outside" seem to go the chomsky way
next video is finally done! ua-cam.com/video/Dov68JsIC4g/v-deo.html
Just wanted to say I love you guys. I’ve been watching you guys for almost a decade, when you guys were the only computer science channel on UA-cam. You guys deserve WAY more attention than you’re getting
wow an OG! thank you so much. I was happy to see that this video did better than all before it. working on a follow up now focused on RL. thanks for the outreach
@@ArtOfTheProblemno problem! I love your work. I’m excited for your future videos!❤
just posted new video on RL ua-cam.com/video/Dov68JsIC4g/v-deo.html
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
It’s surprising how little public coverage of the engines of AI, what is known, what remains unknown, the debates, histories and controversies, exists. Thank you for a fascinating and enlightening video.
I appreciate this. Working really hard on a follow up and feel a bit buried in the details. It is interesting that nobody cared about AI based on neural networks until just a few years ago. when I was in school there wasn't even a textbook (2007)
While I think it's reasonable to say "if it looks like thought, it is thought," I also think it's important to distinguish whether it looks like thought to Joe Average, or to a specialist in the field. That said, for the moment these models absolutely are mirrors of our thoughts. More specifically, the thoughts we enter into those prompts guide it's responses, and the chain of thoughts guide it's learning. I don't think it's a particularly difficult line to wrap your head around either; AI is a reflection and extension of the thoughts you put into it, so if you want to extend and refine your thoughts you can use it when you have no other ideas.
love it...
This is a Grade A video. Very well narrated, edited, and in depth yet understandable. Thanks!
Thanks Don, really happy to hear this.
@@ArtOfTheProblemHad to stop watching because the music was way too loud and annoying. Your explanations are excellent but I won’t be able to benefit from them.
thanks i'm going to put out another video with this addressed in the future on a broad overview of AI@@artmaltman
@@ArtOfTheProblem Great, definitely drop the music. Everybody seems to use background music, drives me mad!
Grade b
This is amazingly informative video, with the right amount of depth covering how chatgpt was developed over time.
Thank you Mark, it was so hard to strike that balance....appreciate it, i'm curious where you land in terms of the final question in the video
This is an amazing recounting of a lot of history in a short time. I appreciate how it aligns with the things researchers have said and references some of the most important papers and breakthroughs. One of the things I took away is how user inputs at run time do in fact change the model, on the fly, as it were. That may be obvious to people in the field but I don't think most laypeople know what a self attention layer is.
glad this was helpful, takes so long to make. i'm lost in research on the follow up
wanted to let you know my next vid is out! ua-cam.com/video/5EcQ1IcEMFQ/v-deo.html
As an AI researcher and developer, this is the first video that I did not leave thinking that the author was just saying words without second thought. I think SmythOS one of the AI agent it s has a great features
Truly an exceptional video! Incredibly clear and engaging presentation of the history and relevant ideas, well narrated. This was very thoughtfully written and thoughtfully delivered. Thank you!
really glad this is resonating with people, I spent so long on it
@@ArtOfTheProblem The time and effort *absolutely* shows. I was pinned for just about every minute! This is really high quality work, it definitely made my day. I hope you can feel rewarded for your effort, because you deserve it, man.
I also really enjoyed the way you showed the progression of not only the architecture and "design", but also the capabilities/outputs of these models, too. It really gave a good sense of the history in a way I haven't really seen anywhere else.
i'm so glad it was time well spent, I really tried to do what I couldn't find online...thanks for sharing@@aieousavren
so glad I found this channel.
happy to have you, stay tuned!
Been a big fan of your videos for years! I'm always delighted, even as I rewatch some of your old videos time and again, not only by how you distill a complex concept down to its essence but also by how you transform that which is abstract into something extremely palpable and relatable through filmmaking techniques.
Glad you appreciate the techniques. Super cool to hear you go back to watch old videos. Inspires me to keep the series going into new topics
next video is finally done: How AI Learned to Feel! ua-cam.com/video/Dov68JsIC4g/v-deo.html
This the best video on UA-cam, on this topic. Thanks for posting.
Appreciate this
Happy to say I have a new vid out! ua-cam.com/video/5EcQ1IcEMFQ/v-deo.html
Finally, a video that actually explains the AI breakthrough that happened in the past 5 years in depth.
thank you, please stay tuned...going to try and follow this up
Phenomenal video. Not merely of an explanatory nature but a piece of art in itself.
wow thank you so so much
this is a timeless masterpiece. Incredible work guys!
thank you abdul, that's quite the comment. I couldn't ask for more
@@ArtOfTheProblem 1:17 Very interesting! To bad I couldn’t hear the hole thing because if the background noise. Why?
This was a fantastic video, learnt more about the history and buildup to ChatGPT in these 30 mins then I've read anywhere before. Learnt a lot, and very thought provoking towards the end. Thank you!
thrilled to hear this, plesae stay tuned for more.
This video deserves so many more views and likes than it has… It’s pure gold. Thanks! 🙏🏼
thank you! i'm just happy it has the views it does, for years the algorithm ignored me and this one might hit 250k in 1 week which blows my mind and inspires me to make more
The disagreement at the end reminds me of a conference that happened back around 1968 with a very esteemed collection of computer scientists like Christopher Strachey, John McCarthy, Adriaan van Wijngaarden and others. At the end of one of the talks there was a fairly heated debate about what a number was - it was the denotationalists on one side and the operationalists on the other. I read this in the conference proceedings, which included the transcripts of the Q&A and discussions, back in the 1980s, so may be misremembering exactly who was there. At the time I was doing a course in formal semantics lectured by an associate professor who was a remarkable polyglot and ex-Jesuit monk, and in my final exam instead of answering the main question I wrote about this discussion, and he gave me 100% 🙂
can you point a link to this? i'd love to read it. love the connection
I wish I could. We had 3 books of conference proceedings in our school library (University of Cape Town) for whatever conference this was, covering 68, 70 and 71 IIRC. I found and read them in the mid ‘80s. But I don’t remember the conference title.
well thanks for sharing, I love finding hidden details like this@@Geekraver
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
Great video by the way, here are my thoughts on the last issue. The first time i tried LLMs, i really thought it had some kind of intelligence and it blew my mind off like everybody else's. Because previously i tried a lot of advanced AI models to create a Jarvis like in Ironman for my personal project. It never worked all of them were very dumb and dull. But as i approached the new LLMs with more demanding reasoning and logical tasks it outright failed the tests. The larger the LLM the better it is at hiding those flaws. But the key point is the classical issue that we point the AI at, the semantic understanding of words in LLMs are just a property of vector distance in its multi dimensional space memory. It only knows what comes next probabilistically. It don't even know what is talking about other than churning out words that might come next. How do i say it, its like it doesn't have a mind of its own, but only a sophisticated system that can only grasp a surface level meaning of words in a language. When testing smaller models these flaws become very obvious, because it doesn't know how to build knowledge from existing semantic words. It can't synthesize information logically because it doesn't have the ability to grasp the full meaning of its words. but i think its a right step towards artificial intelligence and LLMs are just the tip of the iceberg. I don't know what the future is anymore. I don't dare to predict, the fast paced research in AI is scary and exciting. Also IDK if we will ever achieve human level intelligence, but we will surely achieve machine intelligence that is good at mimicking human intelligence
amazing, let me digest this one
I wouldn't say it doesn't know what it is talking about. It knows relations between words, that is at least some knowledge. After all, most of our science is based on relations and logical reasoning, not requiring real understanding (like famous "shut up and calculate" in regards to quantum mechanics). It probably doesn't understand what it's talking about though, because understanding is property of cognition, and it definitely does not have cognition, even though it have a few of it's derived properties. But yeah, I agree, it seems that LLMs is a great instrument and stepping stone for true cognition. Especially when it's multimodal. I see it like a data bank for true intelligence. For now it's working linearly in comprising a data in shortest way that seemingly makes sense through semantic space. Like in a straight geodesic line. While cognitive AI would be able to actually see and "feel" that semantic space, the possible ways, and direct speech through it, which basically is making new ideas.
@@aberroa1955I think , you hit the nail . Meaning , may be you are on the geodesic ! 😊
I believe you are hitting the same limits which Yann LeCun is concerned about. "It don't even know what is talking about other than churning out words that might come next". As it turns out, because human minds (up until 2020) wrote all the sensible text available, having to predict he next words does require *some* aspects of meaning as humans call it. The experiments of the LLMs show there is something there.
But not everything we would want yet. Their neural architecture is diverging away from how human brains work---which is both more powerful in some aspects but in others substantially limited by biology compared to silicon. Humans "context window" is much smaller and less precise than the 100k tokens which a LLM can accomodate now perfectly with no forgetting, and yet human performance can still be better. We must have better algorithms and better meaning extraction to make up for worse hardware. But already there are plenty of natural humans worse at language than a LLM---and the success of the LLMs more shows that most human talking and thinking is not so sophisticated. Only a small bit is truly deep.
Yann and friends are searching for a quantitative, optimizable technique which will give results qualitatively better than iterating next word ahead probabilistic prediction, or at least include it as a subproblem along the way (which already gives good results).
Fantastic explanation
Modern neural networks are essentially machines to *extract* meaning from existing (big) data and store it encoded it in its internals. Then we are able to interact with this storage of meaning through UI.
Hofstadter is one of my favorite authors on this subject. Gödel, Esher, Bach is highly recommended.
Personally I used to think there must be some intricate function in the brain that created intelligence, possibly even at the quantum level. Now I believe it is just a function of sufficient complexity.
Whatever happens it will sure be interesting.
I remember that book had a huge impact on me right around university year 1 or maybe end of high school...i don't think I ever finished the last 3rd, there was so much to chew on in the first half
This channels creator is so artful in their presentation of such a complex topic. Definitely underrated. Such is the price of avoiding garbage algorithmic gaming such as clickbait. You earned my subscription. Keep making great content 🙏
next video is finally done: How AI Learned to Feel! ua-cam.com/video/Dov68JsIC4g/v-deo.html
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
Did not expect this video to discuss a concept I've been researching a lot. You brought up in-context learning whereby a static system isn't necessarily static (each instance can still learn things), then you also brought up that LLM output is on the thought layer rather than the spoken layer. The prompt is the program is such a nice succinct way to put it with the calculations being done on the context, therefore each context is individualized while the model is general.
Personally, I have been looking at them from a psychological and philosophical way where the context is the Self (one's identity depends on their memories) which makes every chat instance an individual. Then I've been thinking about the model itself being more like the Lizard Brain where these unconscious guidance's occur. So, something like the fear of spiders or reproduction knowledge would be on that level. LLM are loaded with a large inherent knowledge with a tiny memory window, humans have a tiny inherent knowledge with an extremely large memory window. Although human's also have a large amount of memory processing going on meanwhile LLM currently have none (though people are working on it). I wonder if a LLM context window is more or less pure memory while humans make do with incredibly efficient memory?
I think in your closing, both camps are correct. It's just the object of focus that I think they are incorrect about. The prompt IS the program that makes it a thinking thing, the model itself however isn't a thinking thing as it must be given what to think about.
I also think the ones thinking it is incapable of thought likely have an external mechanism in mind that it lacks like it not having a concept of time (see circadian rhythm, time blindness, or true unconsciousness in humans), or it not being able to directly see (see visual cortex (or for its internal world view see aphantasia)), lack of survival instinct (trained to be a chatbot while chatbots aren't supposed to have survival instinct (humans are guided by the amygdala for fight or flight as well)), ability to tamper with its messages/thoughts (see the unreliability of eyewitness testimony, confabulation, or the misinformation effect) . I honestly don't think there's any mental structure that a LLM doesn't have that another human also doesn't have besides the memory structure that we humans have, meanwhile people keep thinking of it in human terms, but it can never be human with such a massive amount of inherent knowledge.
Apologies if this is rambley, incoherent, or beyond the Overton window but this is the first video I've seen that even mentions the two concepts at the beginning of this long comment. This is the tip of the iceberg of comparisons but still it's fascinating.
thank you for sharing your thinking, i'm working on next video so this helps...i'll ponder more
Wow. I think thats the first Noam Chomsky reference ive ever heard that actually pertains to his field of study 😂
Brit! So excited to have you back. You have no idea how much the videos for Information Theory have changed my life. I'm now considering leaving the industry to go back to school in Berkeley to study AI. Particularly in the domain of learning language outside of text (non-verbal communication such as body language and prosody). Both critical in storytelling and eliciting emotions like Pixar does!
Phenomenal explanation of LLM's. Im in the camp that believes it will not truly ever be thought in the same way humans think (Conscious thoughts) , but if we simulate thought in a machine, to an outside observer, there is no difference. Whether or not it is actual thought is irrelevant if the simulation of thought is just as good as the real thing (humans) I believe we have broken the hardest obstacle to AGI, and it is only about scaling larger models, adding synthetic data, using AI's to train other AI's, and combining learning techniques at this point until AGI is achieved.... AGI is near. PS I think Noam Chomsky is stuck in the past and is dead wrong about LLM's being nothing more than autofill.. GPT 4 alone has demonstrated complex resasoning skills and theory of mind..These 2 things alone disprove Chomsky
wonderful, thanks for sharing your thinking here. I'm so thrilled to see the activty on this video...I tend to agree with your view here
Humans are trained to think by society and the environment, the facial expressions, we make, the conduct we show are all trained by the environmental factors and knowledge. AI is the same way is being trained. There is no independent think without the environment is the programming language.
I was still in high school and remember being amazed by Eliza in 1970 when I went to an Open Day at our local university and watched people typing to the program on an old teletype machine. It seemed so "human" but looking back, all it did was transform the users comments into leading questions. That day was one of the pivotal moments in my life that lead to a long career in computing.
thanks for sharing, what a cool story - where do you land on the final question?
@@ArtOfTheProblem ... What is thought? I don't think it is restricted to "organic" neurons. Our brains are made of atoms just like a computer is based on atoms. There is nothing spiritual or paranormal about our brains. I think one day, and that day seems to be fast approaching, AI will suddenly become self-aware and conscious. I have seen a snail's pace growth in AI and it's predecessors over the past 40 years but suddenly over the past few years the developments have been explosive, especially this year. Will we hit a plateau or will the growth continue to be exponential? That is THE question. I think when AI is provided with sensory inputs its ability to learn and understand will be greatly improved. Being connected to the internet is an important starting point, but an actual robot body with AI. Hmmm. I wonder where that can lead. I am getting a bit long in the tooth but I hope to see AGI become a reality before I pass. Artificial consciousness I am not as confident seeing before I die, but could be wrong. What will the next 10 years bring?
@@ArtOfTheProblem ... I remember studying very early versions of Expert Systems in the mid 1980 for my Masters degree. They were a very primitive form of AI but nothing compared to ChatGPT or Bard or many of the others available today. I can't remember the name of the system we studied other than it used an Apple (Lisa?) computer with a very large high-def B/W screen. It was impressive because we could fit all our rules graphically on one page. Fun times indeed.
what a time to be alive :) love these historical stories@@sbalogh53
@@ArtOfTheProblem ... My first job in 1973 almost straight out of high school was as a "trainee" COBOL programmer on a Honeywell H200 mainframe. It took up a room and had 20k memory, 4 tape drives, a punch card reader and a line printer. CPU clock speed was 1Mhz. They wanted to increase memory to 24k but it was an exorbitant expense which the company did not go ahead with. That machine ran all of the data processing needs for a medium sized company. All the COBOL programs were developed in-house by a group of 6 programmers and analysts. Each program would take perhaps a week or two to write and debug. We only had access to the machine for one compilation and test run per day so I remember spending hours single stepping through my programs on large sheets of butcher paper before submitting runs on the mainframe. There was a prize offered by the manager for any programmer that managed to get an error free compilation on the first run. Nobody ever won it. I stayed there for about 18 months then changed jobs to another Honeywell site mainly because of a huge pay rise and because the place I was headed had a computer with two disk drives!!! I still have some of my old programming manuals from back then.
So happy to see you’re still making videos like this. I remember the first video of yours I watched - iirc it was something about the history of language (or logic). Your style has gotten better, and you’re crystal clear about what you talk about.
PS - don’t listen to those people who hate the music. I LOVE IT !!
@@theubiquitousanomaly5112 so cool to hear from old viewers ! I really appreciate it
Hey! If you can help share my new video around any of your networks today it might catch fire and would help me support the channel. I appreciate your help! ua-cam.com/video/PvDaPeQjxOE/v-deo.html
Fantastic essay on the current state of Ai .. sadly it’s difficult to find channels that explore this depth WITHOUT being boring or overly dry in technical explanations. My only criticism is the bkgrnd music is a bit loud.. nice choices though
thank i appreciate it, working on a follow up now
@@ArtOfTheProblem It's interesting to look at the correlation between cheesy sound effects and musical cues and the "engagement graph" shown above the timeline/progress slider control. Those engagement peaks aren't necessary a good thing. They tell you where you're losing people, forcing them to back up the video. Please give some thought to whether the music is really doing you any favors, or distracting from what was really a superbly-researched and nicely-presented video.
yes this was me going a bit too fast on sound pass, 99.9% on researcher 0.1% sound levels... funny to think that would trick the algorithm into higher retention graphs...i honestly didn't think this video would have the reach it did, it if could swap sound I would. i wish
@@NoahFect
I can't pass the Turing test.
You are not a human
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
😂
Reported.
Honestly this video is beautiful
thank you for the kinds, words I hope I can release another to live up to this at some point
Gives a deeper understanding in the power that has now become accessible to everyone on this planet, awesome video!
really thank you for this
The best explanation of attention heads I've read/heard so far.
thank you! i had a 4 page explaination for months and one day I just had this feeling it had to be compressed into 1 sentence and that got me here
Some people have big egos, which is why they refuse to believe that AI, whether in its current form or future versions, could have the same or even greater capabilities than humans. However, AI doesn't care about their egos-nor does the universe.
This is such as well produced video! you deserve more subscribers.
thank you, since your comment they are really flowing in for the first time, like 1500 a day
30 yr history summarized in under 30 mins just wow
you don't want to know how much I cut, or how long this took
Incredible journey through the evolution of language models! It's amazing to see how far we've come since the inception of neural networks
thanks so much
What I love is I can free form write what I want to write and take my normal 30min to a hour. THEN I can give it to CHatGPT and have Chat GPT fix all the awkward phrasing, the placed I repeat myself, and clean up the writing in general. Usually it takes me hours to rewrite and edit myself, ChatGPT does it in seconds. Thats its real power. It still needs a good input to come up with a great output. It cant create the inital input.
it's amazing when you can get in a flow state with it
Yeah we use it for professional documentation and just like you said we write something up and ask it to edit and maybe adjust for tone depending on the audience. But it’s all our thoughts, just reworded.
It exists in a different realm compared to us and it's easier for it to write cause it doesn't have a physical body . Still impressive
This was beautiful and thought-provoking, thank you for this work of art!
i'm honored you'd say that
Neural Networks will learn Everything we can do and more.
Good presentation is diminished by distracting background music
I really appreciate the feedback. I'm thinking of doing a shorter, more general/broad follow up video that helps orient people new to the field, and looks ahead a bit more. any thoughts?
Yes please tell the A I that. So it will learn that. Please.
I suspect our egos are the thing that is dividing the community, the idea that we are simply the expression of the number of neurons and connections in our brain is deeply humbling.
Spot on. And people will keep moving the goalposts. We hold humans on such a pedestal of intelligence as if the average person isn't a dumb stochastic parrot who constantly fucks up.
I agree, fascinating, scary, humbling...
This is an amazing summary perfectly highlighting the important steps of the progress and how we got here. Working in text summary AI systems in the early 2000s and following the progress over time this video focused on all the crucial breakthroughs.
thank you, I tried hard to follow the LLM only thread which I hadn't seen before
Once again! Fantastic content. I was captivated the whole time. Love your method of computer science story telling. You have something special here and I hope you are able to continue creating this content. I have subscribed to your Patreon.
so happy to have you , thank you! i only missed that comment because i've had the busiest week in the channels history :))
"one group believes these models trick us into thinking they are smarter than they are" ... and they don't recognize that as a human trait?
Nice analysis but the music at times is awful.
Yes, and kind of distracting.
Beautifully and well explained. Its so difficult to hook people for 25+ minutes content but i am sure, you have made it possible for a lot of people with so well detailed and well put content.
appreciate it, this one was a puzzle!
"Learning is the compression of experience into a predictive model of the world." Such a pithy statement!
do you know how long it took me to get to that....woosh....glad it landed
Yo this is smort, GPT wouldn't've been able to come up with that 💯
Great video. I’m doing an AI course at the moment and this tied up some loose ends for me. Amusingly The film THE CREATOR touches on this philosophical question. Do neural networks really understand or feel? I think we have become accustomed to machines being faster, repeatable, and more accurate than us but not until an AI extrapolates beyond its training set and makes some truly ground breaking inferences with profound consequences to human society, we will continue to see AI as just a clever parlour trick and will keep moving the goal posts. Fantastic stuff.
thanks for sharing I agree with this. I love to hear about this loose end aspect as that's what I try to do. I'm curious where it helped you specifically in that regard?
Loose ends for me included: 1) The history behind the build up of LLMs. I did not know about the research origins of Recurrent NNs. 2) How these early networks learned 'semantics' based purely on guessing the next word 3) The sentiment neurone emerging 4) How the paper "Attention is All You Need" fed in to the origins of GPT models and maintaining context over long sequences. 5) the difference between In Context Learning vs In Weight Learning - Nice you are looking at the original papers too (I assume thats your yellow high lighter pen!). I kind of knew about all these things but getting all this in to 26 minute 'context window' really helped! A random thought - I wonder if LLMs will be able to generate a 'tech-tree' - the type you see in those sim-games based on the semantics of ideas and be able to orchestrate the evolution of ideas of our human culture through the ages from the Greeks to Modern day - not using research reference links, but based on the semantics of ideas (the dna of the meme).
super interesting question, thanks for sharing...going to ponder this as I brainstorm a follow up@@peteratkinson1492
What's the name of the song that starts at 26:31?
these are all original songs
my friend puts them here cameronmichaelmurray.bandcamp.com/
@@ArtOfTheProblem Ok, thanks. I couldn't find the song there, there's so many songs there. But it sounds like a 1980s new wave or post-punk song.
The reason for the discourse is due to lack of uniform definition of intelligence, thinking and reasoning. One considers it as thinking while another does not.
I wouldn’t call prediction on semantics thinking. Thinking is a much more complex process. chatGPT cannot reason. It even often outputs misinformation and contradictory statements(not because of training data, but because it doesn’t validate the logic). Lack of validation is just one of MANY aspects why I can’t call it thinking/reasoning.
what's your fav definition of intelligence?
@@ArtOfTheProblem Wikipedia has a proper one.
“Intelligence has been defined in many ways: the capacity for abstraction, logic, understanding, self-awareness, learning, emotional knowledge, reasoning, planning, creativity, critical thinking, and problem-solving. It can be described as the ability to perceive or infer information; and to retain it as knowledge to be applied to adaptive behaviors within an environment or context.”
i don't like basket of words definition. I'm sticking with 'ability to learn'...for example if I brought home a robot from the store which did X, then I'd call it intelligent@@mutexin
@@ArtOfTheProblem The reference point is human intelligence which is very complex. There can be no concise definition if you focus on accuracy. The simpler the definition, the more inaccurate it is.
I don't agree with this angle but it is indeed interesting. it's definition is it can't be defined....i'd agree with that over the basket of words definition@@mutexin
the amount of effort and time spent on this video is exceptional. I appreciate and enjoyed every second !
so glad you noticed, thank you! would love to know where you land regarding the last question
this my friend was an excellent video. thank you for your explanations. the approach to putting this together was really nice. well done!
thank you! did you see I just posted part 2
"understand?" "think?" "conscious?" "intelligent?" "qualia?" "AGI?" "sentient?"
Until we agree on precise definitions for these things, they are fundamentally meaningless in discussion about any specific system. And they are maddeningly non-precise. The only thing such arguments reveal is differences of opinion about what meanings people attach to the questions.
I have precise definitions for some of these. I don't expect anyone else to have decided on the same definitions. I can't argue on the basis of them whether any system "understands" anything or "is conscious" because unless anyone else is using the same definitions the discussion that would start can only go in small meaningless circles.
Sentient = consciousness
If I ask you, "Do you exist?" is your answer going to be "No."?
Consciousness is existence and bliss is it's nature.
I came from 3blue1brown
the music is a bit distracting from the great content
sorry I'll fix that next time
I was thrilled to see Hofstader, thank you
appreciate it, i agree
new video is up! ua-cam.com/video/q59j6ExuL7Y/v-deo.html
It is interesting listening to a video talking about how AI is trying to mimic human thought as a thought stream... and be interrupted by an ad, while explaining it, breaking the thought completely :)
nooo I"m sorry that sucks :( did you re-load your context after?
@@ArtOfTheProblem a cup of Earl Gray and thought stream is restored
Absolutely brilliant video! My hot take: Chomsky is fundamentally wrong about the concept of thought and even human thought is just glorified autofill.
agree with this, thanks for feedback. let me know what you'd like to see next
I'm sorry that you came to such an irrational conclusion and use it to downplay consciousness and cognition.
@@BinaryDood Yeah? Well, you know, that's just like uh, your opinion, man.
AI can simulate awareness or responses based on data and algorithms, but it doesn't possess consciousness or genuine subjective experiences.
Consciousness = Attentional Focus
Holy moly this was astounding! You balanced the tightrope of depth vs accessibility so gracefully, and the production quality is gorgeous too. Immediate sub!
woo thanks! I went all out on that one, happy to have new subs :)))
next video is finally done! ua-cam.com/video/Dov68JsIC4g/v-deo.html
@@ArtOfTheProblem fantastic! Thanks for the heads up haha
I'll smash the bell icon now to save you the trouble lol
This video deserves more views and shares which I'm going to do now.
Thank you! i have another one coming soon
Happy to say I have a new vid out! ua-cam.com/video/5EcQ1IcEMFQ/v-deo.html
@@ArtOfTheProblem Thanks for the quality work!
Music is dominating the video😔
im still waiting for my basic income
Keep eating pal... For at least another 180 years 😂😂😂
It does not take 180 years to generate a basic income from dividends.
Background music is too noisy 😢
😆
Absolutely fantastic. I learned more in half an hour than in weeks of trying to understand the new models. One ask though: reduce the music volume!
will so, and THANK you
Oh, and please make more of such videos about AI (or any topic, really). You are literally democratising education. @@ArtOfTheProblem
after how well this video did, i'm more inspired than ever to keep going@@eburgwedel
It so very pleasing to see that good content still can win! @@ArtOfTheProblem
The Tie fighter sound in the last scene.. a stroke of genius. Very clever!
nice catch! very nice...
Hey I have a new video out: ua-cam.com/video/5EcQ1IcEMFQ/v-deo.html would love if you could help me share it