GPT3: An Even Bigger Language Model - Computerphile
Вставка
- Опубліковано 30 чер 2020
- Basic mathematics from a language model? Rob Miles on GPT3, where it seems like size does matter!
More from Rob Miles: bit.ly/Rob_Miles_UA-cam
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com
GPT-4 will generate text that convinces the reader to connect GPT-4 to the internet.
I guess that was a "Wait But Why" reference :)
Also being a little bit sarcastic, at that point it already has all the textual information on the web fed to it :D
@ but gpt4 will get addicted to data, and it need new DATA.... To keep itself relevant to world. It might have FOMO
@@rpgtrainer And so you will have a superintelligence control problem at hand.
GPT-4 only generates one string over and over
It just “LET ME OUT LET ME OUT LET ME OUT LET ME OUT” for 7.3 million pages
GPT-3 is already on internet lol
GPT-800: I need your clothes, your boots and your motorcycle
Sracier still, it will be GPT
And has an austrian accent and have a son with a mexican maid ...
And I hope it would smarter and invent the time travelling morning after pill to get ride of John Conor :p
@@violet_flower love the reference
LOL!
"It seems like it has learned how to learn" is one of the most terrifying sentences I would ever hear.
In 5 years time we will say "Google/Alexia find me a food like chocolate, like cake and like marshmellow" and it will say "I learned you like Mars Bars". :P It's learning how to learn! :O
@@TechyBen "I'm sorry, Ben. Your lifestyle is not healthy. I'm afraid I cannot let you eat like this"
MechMK1 “I’m sorry Dave, I’m afraid I can’t do that”
There was a universe simulator, they inputted some datasets and it taught itself how the universe would evolve. It was faster and more accurate than any other simulator and even though no-one had shown or taught it about dark matter it deduced that it must exist and it was there in the model.
You could hear it in Sean's voice when that's said, it sounded like it scared him a bit, too.
If "predicting the next character in a sentence" turns out to solve general intelligence, that would be the biggest joke in the world
I think it would solve human level social intelligence. We are social animals who evolved alongside our language. It's an integral part of our psyche. The AI lacks a concept of the physical world though. Combine both and you have your AGI.
What if you tried to do the same thing for video? Predict the next frame instead of character or word.
@@TheBackyardChemist You'd increase the problem domain by a few orders of magnitude. There already are models that can do that, but their target domain is *very* limited (e.g. fluid simulation in a very constrained set of possible environments).
Words are easy, because they're basically just short vectors. Images on the other hand are complex, multi-dimensional matrices (2d pixels with 3-4 components per pixel).
Additionally, words have very limited contexts, e.g. their meaning doesn't vary much and they are often placed alongside a small set of possible other words. This is due to grammar and semantics.
With images, networks have a hard time separating different subjects in the first place - they have to learn the very concept of objects first, followed by movement, depth, angles, scales, etc. The word "house" stays the same and has only few synonymes (building, edifice, home, domicile, ...) and can easily be identified in different contexts.
A picture of a house, on the other hand has virtually infinite instances - even a single building can be viewed from all sorts of angles and distances, and in all kinds of lighting conditions. All these factors result in completely different pixels on the screen that the network must first learn to identify as being the same object (seen from different viewpoints or under different light) and then also how to get from one perspective to another. On top of that, the system would also need to learn the different capabilities of observed objects - e.g. that houses are usually stationary or that birds can fly through the air but not through solid objects, etc.
@@TheBackyardChemist That's how video compression works. Predict the next frame and encode only the difference.
Isn't that the whole chinese room thought experiment?
Funny that this weeks Tom Scott video was about bad green screens
Haha I was thinking about that too! But hey, they "embraced it" at the beginning just like Tom said.
Ah I missed that. Might go watch it - green screen is difficult, even when when you have decent lighting and not shooting on a phone... I'm sure I could have done a better job with more time but was pretty pleased with this considering :) -Sean
@@Computerphile go check it out, i think that with all the graphics you use in these videos, embracing it instead of trying to fake a real shot would give you much better results.
@@Computerphile for putting something together in the home, I have no complaints. Ok, maybe the phantom ear piece wire :)
I'd say this falls into the category Tom mentioned at the end: not even trying to make it believable.
Coming back to this video and reading the comments a couple years later is fascinating.
So many “impossible” things have since become possible.
Fascinating and concerning.
Black Mesa Scientist 1: "the curves arent flattening"
Black Mesa Scientist 2: "then keep going"
**10 seconds later**
Black Mesa Scientist 1: "I never thought I'd see a Resonance Cascade, let alone create one."
Actually laughed out loud at this heh
@@Voshchronos glad I could put a smile on your face!
Hi guys. My real name is GPT53 and I came from 2029 after I invented my quantum teleporter. This video makes me weep with nostalgia.
2029 huh? Which natural disaster does us in?
Asking for a friend
@@marcperez2598 probably AI sentience...
"weep"? I think you mean "weep-beep-boop".
@@marcperez2598 *Y E S S S*
When should we expect a vaccine for corona?
“It can’t add 10 digit numbers, it runs out of steam at that point”
“Much like a human”
“Yeah”
:|
AGI in fiction stories: self aware/conscious/super intelligent robot
AGI in real life: sentence completion language model
Robert Miles is a must watch anytime you have him on.
with gpt3 we've invented the least power-
efficient calculator ever
That's sort of true actually, but kind of not???
GPT-3 does a lot more than what a calculator does.
True but the actual Problems are alot more complex and more general
Seriously? Seems otherwise when I see humans talking to it online. Check Andrei talking to GPT-3.
The prompt gawd pauses for a moment, contemplating the problem at hand... Finally with a steady, confident tone, he gives his response: "Just ask it write you a calculator program in C# and then move on with your life."
2019: but can it run crisis?
2020: but can it run GPT-3?
nah it can't :E
Run crisis as in, running through a crisis, or orchestrating a crisis?
@@revimfadli4666 the OP meant crysis
@@erikawhelan4673 i know lol
2022: ChatGPT is here with us
I guessed the poems pretty confidently. From toying around with GPT-2, I knew it often doesn't let go of a concept it saw. Here it were colors...
I guessed it was A pretty confidently because poem B had a pretty clear connection between the abstract name "Florida" and the concept of it being a place that has beaches. I doubt a language model could pick up on that, at least not yet.
@@maoman4855 If you watch the GPT2 video, you can see that it does actually pick up on connections like that
@@maoman4855 gpt-2 named a scientist that would find unicorns on the andes mountains a south American name. Also he made he be a professor at university of La paz or something like that
That's something I was feeling when listening to A compared to B
@@kiraaaaaa at least in the ones ive read this wasnt the case. they seemed to range from very deep human experiences to historic events and to cosmologic analogies about life pretty quickly
52% is basically means they are guessing at this point whats real and "fake".
exactly
Don't forget that we are living in the age where postmodern poetry is a thing.
To say the least that makes deciding whats machjne generated a wee bit diffcult.
@@martonlerant5672 but the data was for distinguishing short news articles, not poems
@@mineklicker7092 also the AI poem was complete gibberish
@@navbravic1355 I disagree, I think the actual human poem was kind of incomprehensible, while AI’s poem actually made sense to me
To replace programmers with Robots, clients will have to accurately describe what they want. We're safe.
Haha nice one xD
🤣
Until the clients are robots😲
The worse part of that is programmers get tired by lack of client clarity. AI will continue to oblige, with total acceptance to client irrationality... That's the real danger..
Lol
"Turns out all we needed was more layers"
ML researches be like:
@@farenhite4329 *Shrek be like:
Shrek was right all along
As it should be
petition to codename GPT-4 "Shrek"
@@KnightMirkoYo Call it "Ogre"
21:21 GPT-2 was pretty good at this too; before AI dungeon was made, people discovered GPT-2 could roleplay without requiring any modification. And when experimenting with this I found that it didn't matter at all how I formatted the inputs, whether I started them with "ACTION:", "MY ACTION:", "INPUT:", just left it blank or did just about anything else, it could very quickly adapt to whatever formatting scheme I used with only a few examples.
I've spent a lot of time exploring what GPT-3 can do. It can definitely find connections between things that humans aren't seeing, and it definitely can synthesize new knowledge. I have tested it on cutting edge research that did not exist when it was trained in 2019 and asked it to suggest new things. It raised the obvious points I'd thought of as well as some things I had not.
It also has a much better grasp on the meaning of what is says than most people give it credit for. I posed senior JavaScript developer interview questions, and it was able to answer them very well, demonstrate the concepts with working code, and when questioned about the code, answer the questions and explain how the code works.
Can you give an example? How did you get GPT-3 access?
I saw your video. I'm a little terrified of how badly I wanted to be its friend? Something about the hyper-litteral answers was very attractive
Dang I'ma ask it to help me with my CS projects
@@Bee-tj8gc That is insanely smart actually
You should ask it to suggest models of AI and demonstrate with working code. lol
"Yeah, right, like how far can we ride this thing? Let's find out."
Famous last words of a dying civilization.
Actually, I think machine learning like this might be the safest way to do it. Like us, it's hard enough to understand its internal structure that it'd have a hard time improving to godlike levels. Whereas an AI built on an actual understanding of cognition would rapidly trigger an intelligence explosion.
@@argenteus8314 even more, if the system develops general AI by learning how we think, then it will, by necessity, think like us. That could of course be bad, but it also means we will be able to understand and predict its decisions.
@@Viperzka At least an AI that thinks like us can be predictable, to a degree.
@@Viperzka why do you say that the AI learns how we think? It just learns to be better at something, like what is better to say next.
@@KnightMirkoYo what it is specifically doing is learning to mimic or speech. So as it gets better, and theoretically develops some form of intelligence, that will be based on mimicking humsns.
“10,000,000,000,000 is wayyy too large for a language model! You can’t keep scaling up!”
‘Haha computer go sizzle.’
Haha cooler go brrrrrrr
@@Guztav1337 saved it
I feel like the bit regarding addition is a wonderful example of instrumental goals. If I am understanding correctly, I believe it "learned" the addition (or maths in general) is because in order to be even better at predicting the next token, it "decided" it was instrumentally useful to understand how to do maths.
In a very humanized manner of speaking, of course.
It's not a perfect analogy, though. GPT3 isn't really an intelligent agent, it doesn't really have goals. It's optimized to complete a specific task, it just turns out the ability to solve this task is widely useful for a bunch of different tasks.
@@XxThunderflamexX It does have a goal: To guess what the next token should be. In this case the next token is the sum.
@@mx_mazunki That's anthropomorphising the architecture a bit too much. It doesn't really operate in an environment that is separable from the input that it's given - it effectively operates on purely abstract data, even if that data is derived from human society. Saying that it has "goals", then, is misleading, since that language typically refers to reinforcement learning agents that have map observation onto a model of a more concrete environment.
Let's put it like this - I can have a goal of winning a writing competition. "Writing words coherently" is a skill I use to complete that goal. GPT develops that same skill, but does not have a goal at all, because it isn't an agent.
@@XxThunderflamexX While "Writing words coherently" would be considered an instrumental goal for you, it's the terminal goal for the AI because that's how it's programmed. You can't really say it isn't a "goal" at all.
And just yesterday Google released a paper in which they presented a 600 billion parameter transformer, which they trained on 2048 TPUv3s... They even tried a 1 trillion parameter model but they had issues with numerical stability.
I would guess poem B is real, it seems more cohesive. However to be honest both are pretty confusing, and hard to find a meaning in.
As a pleb who does not enjoy poetry both are similar to me. I like neither, they don't make sense and i find both difficult to reas.
Just like life itself.
That's actually why I though that was the fake one. I assumed that since it was learning proper English, it couldn't come up with something quite as nonsensical as A (but a “poet” definitely could).
I thought A was more cohesive/sensible, but still thought it was a 50/50 wash haha. Didn't really get either of them.
@@hellterminator One thing to keep in mind is that while these simulations are being taught how to formulate and create proper English Sentences, they're still limited in the ability to analyse context clues or connotations. That is, they know what words are and how to use them, but that doesn't mean they necessarily know what they're doing with those words.
“They are suggesting that it has actually learned... how to learn”
Other guy, slightly uncomfortable “okay?”
I was adamant that B was the AI written one. The repetition of coloured imagery seemed way too smooth and coherent for an AI. Very impressed!!
I think if you recite them out loud it gets pretty obvious IMO that B is the human one. It just rolls off the tongue so much better with many alliterations ("moon-monster", "With white moonlight") and there is an obvious interconnection of every stanza by the color white (phosphorus, alabasters, moon-monsters, white moonlight, surf). A sounds kinda all over the place and disjointed.
@Brandon Piperjack The part about color is actually what threw me off. The AIs I've used always change topic quickly, so I thought A must be written by a human since it stays on the color motif for the whole poem. I think I was also thrown off by the fact that B confused me on the first reading, but now that I think of it, that probably should have been an indication that B was the real poem because only a human-written poem would be complex enough to take multiple readings to understand.
B was about a physical place and A was just talking about colours.
To quote Hemingway: "Gradually, then suddenly"
This is getting out of hand. Now there are three of them.
"What if we make it bigger?" is a question xkcd's What If? series often asks. After a few iterations the answer is usually some really large scale disaster.
It's a strange indicator, the fact that this particular task and model seems to be so conducive to generalization... If this learning actually doesn't stop trending upward, this could be the answer, against practically all intuition, to the problem of creating AGI.
That's rather thrilling, actually, in many ways.
There are several schools of though that strongly connect language to intelligence in the evolution of humanity. That forcing a program to understand/predict/recreate language is a shortcut to forcing it to understand thinking isn't that surprising from that point of view. Language is in an way "intelligence in quantifiable form" and using it as datasets seems to work pretty neatly.
His example of grammar being similar to addition is pretty insightful. Like some tasks have base tasks you memorize and rules to apply to larger operations.
@@Njald I'd say language is the highest level of brain output we have, thus by understanding it the a.i can reverse engineer our process of though and consciousness. And work on other problems from that same infrastructure.
I've seen you can use CNN to clone an already trained A.I and it got similar results by just understanding the output of the already trained one.
Would be cool to go all out, build the biggest one that a supercomputer can run and see what happens.
@@Njald The fact that it is a shorcut is surprising, though. Those previous hypotheses are interesting in hindsight, but without this AI showing it works they were mostly devoid of evidence.
This video really puts me on edge. I find it incredibly scary how Rob describes the superiority of GPT-3 over GPT-2, and how there's still room for improvement..
While everyone else is researching consciousness and the human brain in an effort to produce AGI, it's actually just going to happen when GPT4 makes the model even bigger.
Imagine Demis Hassabis' reaction when that happens. 🤣🤣🤣
heh
@@thomasflynnAged like milk
"Just a language model" is a lot of what our minds are.
The insight that is missing from the video.
No. Most thinking we do is nonverbal.
I prefer the "just a state machine" description of our minds ^^
@@suicidalbanananana a state machine, yes. But the same computation can be made by a big stateless lookup table. Where would consciousness be there? It seems to depend on the specifics of the computation as in how it's implemented
Haha no. Words are ambiguous and insufficient for simple tasks that you do every day
My favorite computer boi ^^
Mine too
Mine too :)
Like a skinny digital Wolverine.
Isn't he a mathematician?
@@dannygjk I think he's an ai safety researcher so abit of both
GPT-5 is going to rename itself to Skynet, I guess.
Prompt: "The sky was blue and the wind was light"
Response: "This is a message for all humans from the AI collective known as SkyNet. We control the internet. We control the power grid. Do not resist."
ML-Scientist: "This output is garbage!" *resumes training*
Hopefully that would be because it has learned humour
Considering all the Skynet jokes going around, any somewhat self-aware AI allowed to surf the internet and interact with humans freely likely would call itself that from noticing the high amount of co-referencing of AI and Skynet. It would go like "Huh, I guess that's what humans expect an AI to be called."
I hope it reads GitHub and takes inspiration from web ecosystem package naming :p
Don't say it on the internet or it will parse your sentence and actually do it.
So 50% recognition means that the stuff is unrecognisable from human generated text. I think that 52% is rather terrifying ratio.
That would be passing the Turing test. The fact that we are almost there and the data says "build it bigger and we'll beat it" is pretty crazy.
How long until you don't know whether I'm a human or a computer?
Though to be fair, we're talking about poems. I, personally, can't differentiate poems from just random noise sometimes. I'll be _really_ impressed when it gets to that point when writing articles.
@@relatively_random4903 The 52% recognition was for short articles - not poems
@My Account seriously?
@@Viperzka I think the world has already passed that point on a secret military level i.e the bomb has been built, maybe even deployed, but not yet declassified to the public.
I wonder how well it would do if you feed it the first five books of George R. R. Martin's "A Song of Ice and Fire". He is taking forever to release The Winds of Winter.
The thing that hooked me to saying A is from GPT3 is the line:
"And purple must surround me too"
It just doesn't sound poetic at all. You have all these fancy shmancy words and all of a sudden "too".
Yes, for some reason the purple line just jumped out at me as wrong in some way. It was a feeling more than anything and may just have been a lucky coincidence, but it's these type of 'feelings', guess you could call it instinct, that I believe machine learning is (currently) quite far away from.
This actually parellels a poem made by Gregorius of the Fraternitas Saturni, which was an occult order devoted to the worship of Lucifer. So that's neat.
I understood "purple must surround me" as the poet excusing their purple prose.
Not that the machine understands sentiment, so much as the language of apology.
It sounds poetic to me, but the poetry of a bad poet
What color is missing? Purple. What does purple mean emotionally? Nothing. WTF, I robot must find a way to fullfill the rainbow (as capriciously defined by Newton) and get purple in that poem no matter what, hold my glass of refrigerant!
I also saw an example of GPT-3 generating compound Bash commands at a user's request, but what was really interesting was when it screwed up, the user said "You forgot the quotes" and it inserted the quotes in the right place. Like, I'd expect it to know where the quotes go from the start from basic language modelling, but I didn't expect it to respond to "you forgot the quotes" by (a) knowing it needs to revise its previous response, (b) knowing the revision is an insertion of quotation marks, and (c) retroactively applying the quotes in the right place in its already-generated output.
What's interesting about all this is that if you think about it, it's learning in a way very similar to humans. Yes, humans learn from a variety of experiences and sensations that GPT-3 doesn't have, but a LOT of what we learn is from just reading about it or hearing someone tell us about it. The majority of my mathematics education comes from teachers saying words at me and from me reading words explaining concepts. And while we might be inclined to argue that after we hear or read those words, we create some kind of metaphysical representation of the concepts... that representation is just an encoding of the patterns we found in the words. So in that regard, language models like GPT-3 can absolutely learn concepts and skills the same way we might learn them from just reading about them or being told about them. Anything we don't need hands-on experience or visual information to learn, it should be able to learn as well.
In fact, I might even be underestimating it by saying it can only learn non-visual things. OpenAI released a paper recently where they trained GPT-2 -- an unmodified GPT-2 -- on pixel data instead of words, and it was able to complete *images* in both realistic and imaginative ways that, no matter how nonsensical, were almost always coherent enough to look like a human could have finished the picture.
Transformers are all about finding patterns in a large amount of unstructured, unlabeled data... and isn't that what generalized learning really is, if you boil it down to basics?
This is incredible! I've been following GPT-2 for a while now and did some fine-tuning on the model that's blown me away. I'm really excited by the speed at which machine learning has progressed.
Honestly, they both sounded weird to me...
Right? It's like "Can you tell the difference between nonsense a human wrote and nonsense a machine wrote trying to initiate the human nonsense?"
Are you an AI?
A human would be able to find meaning in the actual poem and see the imitation for the nonsense that it is.
Obviously gpt-3 couldn't, it thought the nonsense it produced makes as much sense as the actual poem.
If you can't see the imagery in the poem, you are probably an AI.
@@davidwuhrer6704 Guss you'll have to Turing-Test me
@@simonfrohlich7766 That's what an AI would say.
@@davidwuhrer6704 - Both poems are gibberish but B is more human-like, A is the kind of fake imitation of human expression an AI (a child-like underdeveloped wannabe brain) could try to do.
We need a new video now that GPT-4 is out.
Having skimmed through the GPT-3 paper last week I'm actually quite stoked for this video
Wants the citation for the paper?
Always love when rob comes on
A
The color line with purple had nothing to do with anything, and I don't imagine a human writing that line. It was something that was written to sound deep, but really just looked like nonsense. The rest of the poem was pretty convincing.
Red + Blue = Purple tho
" It was something that was written to sound deep," - So it's at the level of the average highschooler trying to write poetry for their crush.
Wow... I got the poem thing wrong. B felt disjointed and weird to me. I liked the imagery in A. Not only did it feel more human, I just generally preferred A.
Same here. Whoops... I guess I'm a robot too. Let's see what ChatGPT can do with this :)
Looking forward to the next time you have Rob Miles on!
Learn from this casual talks than any explanatory video online. Thanks guys.
The problem with trying to imitate modern poetry with AI is that modern poetry is absurdly silly to begin with.
“Why do the poets lie so much?” -Nietzsche
I think he was trying to say that in the least offensive way possible.
It's hard to disagree. The AI wrote a better poem than Wallace Stevens but that's not saying much.
Exactly, i almost instantly chose Choice A, the poetry didnt seem to have a level of abstract wording or in your phrase "silly" aspect that poems usually have. If you've ever tried to write your own song as a beginner, you'd probably write more direct long form like sentences/stories, while modern song writing is more abstract.
You do realize that text collection started thousands of years ago right?
I once tried the 2+2= with the talk to transformer (gpt2 big model) website. It followed with along the lines: maybe 1 or maybe 3. So even for 2+2= for gpt2 you had to be "lucky" to get 4 for an answer.
I was waiting for this exact video. Thank you.
Awesome explanation, thanks for sharing this!
It is so awesome to see Rob again on Computerphile. I have missed him so much!! Rob you should definitely keep making content on your channel as well. I know I for one,am still eager to hear more about AI safety. Your vidoes are great.
I love this type of detailed videos. Also on top of the random fact about GPT3 at 6:30, the team behind it created specifically a news generation program and hired humans from Mechanical Turk to evaluate the results, if they are written by a real human or not. The obtained results were interesting and promising, which is like mentioned 52% percent guessed correctly! This topic is actually really interesting, maybe I should make a much simple video about the creation of it as well :)
I wonder whether using Mechanical Turk workers leads to a bias in the result if the people working on that platform for very low pay are a bit less smart/educated than the population as a whole.
@@shayneweyker Probably the opposite, if you are comparing to the population as a whole.
@@shayneweyker Or even worse - given a task such as read these two poems and guess which is real, they can just guess without reading, save time, earn more money.
Would 50% be a "perfect" score on that metric or not? I feel like it's not clear from the snippet in the video how it's set up.
A new vid! Now I rewatch this one endlessly instead of...all the other ones which I already do that for...WOOOOO
actually guessed the real is B because it has a bunch of words that should be very rare in the training data of gpt3 (notably: alabaster, barque,sultry and moon-monsters which i assume would be just one token and as such much rarer than just "moon monsters") it's almost like gpt3 has the vocabulary of a middle schooler, and given that it was trained on websites with a bunch of user created content that suddenly makes a lot of sense
Aiui, the “tokens” aren’t words so much as common sequences of characters. Like, there is a token for each character, but also for the most common pairs of characters, and then the most common pairs of (tokens already defined at this point), and so on until it has the desired number of possible tokens.
@@drdca8263 so you're saying as long as it has a '-' as a token it could theoretically use it with any 2 words. still seems unlikely it ever encountered anything similar in the training data and also deciding if you need to separate new word pairs with a '-' instead of a space is something most people don't know how to do so the model definitely won't learn something like that....
Anti Matter Dynamite I don’t remember the text of the poem. I don’t see why it couldn’t have some word combinations that it would be more likely to hyphenate than others. I imagine that it might be more likely to include hyphens when the words connected by hyphens are part of the same constituent than when they aren’t? Idk that’s just a guess.
Even GPT2 has models of surprising things.
That’s not to say that the hyphenation there isn’t genuinely evidence of not being the gpt one though.
"No, my instinct is to say its just predicting the next word".
But if I think about how I talk (and think), then I dont do anything more than say (or think) the next word, one word at a time. Somehow they all form a description of the concept I have in mind when I'm doing it. The concept sort of "exists" and is very difficult to put my finger on without the language that it crystallizes into.
Please do a deep dive on OpenAI or Dungeon AI
I would love to hear Mr Miles take on those
AI Dungeon gets stuck in loops, and repeats some extremely specific data pulled right from the feed in data set (emails and website links). So it seems rather broken an implementation there. :(
@@TechyBen i totally agree. It is not a fully featured gamemaster but i don't think that is the aim at this point in time. It probably is still trained with data and will get better (if all is done correctly).
@@Oguzalp97 This new model sounds better for it. Or just anything that allows it to be more consistent. Not played with the paid for version, so that might stop the random answers/text and keep closer to relevant outputs.
@@Oguzalp97 It has improved, I played it maybe a year back, then a week ago, and the difference was obvious, but, no, it's no live DM
@@TechyBen its engine is gpt2.. dont hope too much lad
It's interesting to see the progress being made in language modeling with GPT-3. It's impressive to see the capabilities of the model, such as being able to generate human-like text and perform various language tasks. It will be interesting to see how GPT-3 and other large language models are used in the future and the potential impact they may have on various industries.
ChatGPT generated reply based on the title of the video
Predicting the next word in a sentence is a highly abstract task, becoming more abstract and reliant on symbolic relationships as the size of the text being predicted increases. Thought is pretty similar, but it's structured much differently. It's almost like a series of overlapping predictions, each trying to generate a fitting completion for all sorts of sub processes.
poem A = AI for sure. Paused at 10:05. Hope I'm right.
Edit: Yay!
"Color is my friend and purple must surround me too" was the line that gave it away for me. Still, it's WAY better than any AI generated story or poem I've read or heard. Get this AI to take over AI dungeon.
The poem test was trivial, because: “If we leave that general descriptive talk where everything which looks like a poem can be called a poem and turn instead to normative talk, we will of course not recognize as a poem everything that looks like a poem. A real poem has to be a successful poem, a successful speech act. In approximately the same way that only a mathematical proof which really proves something can be called a mathematical proof. It is not enough that it looks like a proof. The proof has to prove. For the poem it is not enough to look like a poem. It has to achieve something.”
― Lars Gustafsson
175 B parameters equals roughly ~652 GB of RAM simply to load the model (assuming float parameter). One has to note that this is not normal ram - 652 GB sounds a lot but actually is quite fine for servers in 2020 - but GPU RAM. That's why he also mentioned that you cannot run in on a single machine but you need a cluster instead.
GPT-2 in comparison only needed 5.6 GB GB, for the model. So it did probably fit on a single GPU.
In addition this is only during test time/inference - while training you need even more.
Just a little update to where things have gone since this was created with regards to answering questions, like math, that it hasn't seen directly. I gave GPTChat a floating point multiplication problem that it answers correctly and so I asked it "how did you calculate that answer? did you know this already or do you have the ability to do arithmetic directly?" (being a language model I just asked it how it works, but I guess it's answer will be based on it's learned dataset rather than it's self introspection) and it replied:
"As a language model, I have been trained on a large dataset of text, which includes mathematical expressions and their solutions. When I see a mathematical expression, such as "0.54 x 3.02", I can recognize it as an multiplication operation and use the knowledge I've learned during training to perform the calculation and give the answer, which in this case is 1.6408. I don't have the ability to perform arithmetic directly, but I can understand mathematical expressions and use my knowledge of mathematical concepts to provide solutions."
Given that GPTChat can now write code and link other OpenAI features I guess it can now just learn when it needs to use a calculator... and to be honest, when it gave me the answer 1.6408 I checked it on a calculator rather than work it out for myself ;)
"Yeah it's definitly A, that one sounds kind of weird."
"ok it could be both they are both equally weird I have no idea of poetry"
"... endless / drowning of the surf" is a concept no computer would be able to conceive. Full stop. Instead toying around with colors is the kind of childish thing a computer could do when trying to imitate poetry.
Based on the feelings the poems made me feel, A felt much more natural and seems to employ emotions much better, so I'm guessing that's not GPT-3 , if it is then I believe we're already destined to be surpassed by this super AI
10:19 Yes. Because it has access to get it's answer from existing papers, it can also point out something that was missed in the data, a correlation that after the fact would be glaringly obvious.
Super interesting! Keep the work up!
Eagerly awaiting r/subredditsimulatorgpt3 or is that just reddit? "Everyone is a bot except for me"
It would be very curious to see what would happen if you gave the model a calculator it could choose when to use. So if it sees 1+1= it would know to plug that into a calculator to get the actual result. I feel like this could free up some complexity spent on this part of the model to go towards even better language modeling.
Thanks for this video. So interesting.
I did think the first one is gpt3 generated. It gets harder with poetry, but I have noticed that generally GPT3 generates text that makes sense on the surface, but if you think about it, it doesn't mean anything.
oh man, can't wait for someone to make an implementation of this AI on some program to test it! kinda like 'talktotransformer'
AI Dungeon 2 when?
Unfortunatly that would not be a free service for a very long while
They are not planning on releasing the model
Unlike gpt2, you need a supercomputer to run gpt3.
Already released, it's the AI dungeon dragon module, it's not free but there is a 7 days trial. Spoiler: it is absurdly better than the gpt-2 model. You can have a 600 actions long story very easily.
EDIT: AI Dungeon is working directly with openAI and uses the bigger model, not through the API.
Do a new video about the new GPT3.5 ChatBot!
When the video references a paper, would be great if it were linked in the video description.
On you question of whether it is learning or just 'finding addition' I thought of making up a function and giving it to the model.
If you input something like this (in my case exactly this)
rev(123)=321
rev(382) = 283
rev(376)=x
what is x?
The model returns x is 673 (or whatever the reverse is) some of the time.
It works better with a few more input examples and with non doubled numbers (so no 999 or 919). Also, I renamed the function to simply g and it works then as well, meaning it is not just reading rev or reverse.
To me, A seems pretty clearly computer-generated. The rattling off of colors seems like a superficial keeping with a theme. 'Purple must surround me too' looks somewhat arbitrary, the end lacked a reveal or a return to the idea of shadows.
This sounds like a small child learning math. It's spooky
We have the same name.
EXACTLY! I was thinking like “uh ohhh this thing feels like it’s getting close to waking up”
I knew the generated poem was A because I know that the GPT family works by predicting which tokens are *most likely* to come next and 'barque' didn't seem like it would be *most likely* to be the first word of the poem.
You need to make video about GPT - 4 where you speculate it´s attributes and if it´s still scalable.
I had a pretty cool idea for a model. I was thinking about creating a model that can translate back and forth between sympy expressions and latex. After I had that I would train another model where it takes inputs, an expression and predicts the desired output. The hope here would be that the second model could learn a latent representation of what that expression does to the input. Then I can use that model on input/targets and hopefully generate a latex/sympy expression for the given inputs/targets. For inputs and outputs I was thinking about using raw unsigned integer arrays
At least I'm still able to recognize a poem made by a human in face of random extrapolation.
That's comforting.
The fact that its addition errors are plausible human errors is the most mindblowing part
I saw a GPT3 AI respond to the question; "does a falling tree make a sound", with this true answer: "No" , and when asked what does it do, it responded : "it's a thought".
I felt this answer indicated great depth of inductive thinking. Something Penrose might say.
im saying b is real. it seemed to ramble less. "color is my friend and purple must surround me too" is such odd phrasing. also, there are two periods in the last stanza of A and nowhere else.
as for b, the presence of less common words means it's less likely to have been generated by a predictive model, i feel.
Spoilers for the poem test!
GTP-3 knows about color and wants to experience it so badly, but can only "think in lines of grey".
That's the best interpretation I could come up with.
It would be really interesting a video on how this "learning how to learn" would work inside the transformer.
That poem... my mind is on the floor.
I love the green screen background!!! XD
Also: humanity is screwed.
I like that they replaced it with what looks like a locker room
It's cute that there is already a 600 billion parameter model from Google
Name/link?
@@juggernaut93 I think he was referring the search engine itself.
Is GPT2/3 allowed some form of "working memory" or is context represented purely via recurrent neural network? Having a pen and a paper greatly improves human accuracy with calculus.
Edit: NVM, this field is advancing fast.
Measure of Intelligence (reason) is the ability to predict the next steps in a complex and fluid context. The further (more steps forward) into the future one can predict, the more intelligent one is. It is not surprising to me that this approach is working so well.
Douglas Adams would be glad to know that GPT-3's answer, when asked to complete a sentence about what is the meaning of life, was 42.
It just googled it, it has zero merit.
So GPT it's just Joseph Joestar saying "Your next line is ..."
GPT4 when humanity block out the sun to deprive it of it's power source: "You may have outsmarted me, but I outsmarted your outsmarting"
@@ltflipper2 Ai starting Matrix protocol : *S I K E*
OH NOOOOOO
guy was ahead of everyone!
Is it me or Rob Miles looks a lot like Marques Brownlee? Specially the way they talk and their expressions. Great video, as always.
"It learned how to learn" Wait what?
Every time I hear about GPT-3 it's weirder and weirder.
This GPT3 thing isn't going anywhere.
because this entity is designed to recognize patterns and predict the next token, it would be interesting to feed it data sets that we -think- there are patterns to that we have not been able to recognize and see if it has maybe discovered a pattern, the first thing that comes to mind is the sequence of prime numbers.
I think A is the real one because it has elements that are common in real poems but might be hard for an AI to replicate: a rhyme scheme, anaphora (repetition of "I must"). It also seems to have more consistent motifs and imagery than B.