Create NPCs with LLM-powered Convai!
Вставка
- Опубліковано 13 тра 2024
- @convai recently made headlines because of their conversational AI tools that use large language models (LLMs) like GPT to create intelligent non-player characters (NPCs) in games. In this sponsored video I chat with Convai's CEO Purnendu Mukherjee about the progress they're making in building these tools and even tried making my own NPC to see how well it works in practice.
This episode was sponsored by Convai and I received compensation for my time producing this video.
If you're interested in finding out more about Convai's work, some links are provided below:
Website: Convai.com
UA-cam Channel: / @convai
Discord: / discord
Twitter: / convaitech
LinkedIn: / convaitech
Convai Demo with NVIDIA ( • Live demo of Tour Guid... )
Download and play with the Convai demos:
drive.google.com/drive/folder...
[00:00] Intro
[02:29] Who Are Convai?
[05:41] Building NPCs for Conversation
[09:59] Testing Out the Tech
[19:24] Building for the Future
[26:56] Closing
[27:51] Credits
--
AI and Games is a UA-cam series on research and applications of Artificial Intelligence in video games.
Follow 'AI and Games' for educational videos and essays:
/ @aiandgames
My second channel 'AI and Games Plus' is dedicated to bonus content, deep-dive analysis, revisiting old topics and more:
/ @aiandgamesplus
Listen to my podcast 'Branching Factor', as I discuss game development and research with my co-hosts and guests:
shows.acast.com/branchingfactor
Watch Livestreams: / aiandgames
Join our Community: discord.aiandgames.com
Buy our Merch: store.aiandgames.com
--
You can support AI and Games via Patreon and UA-cam memberships, and in return, you get bonus rewards!
/ aiandgames
www.youtube.com/@AIandGames/join
--
Follow 'AI and Games' on social media.
Twitter: / aiandgames
Mastodon: mastodon.gamedev.place/@AIand...
TikTok: / aiandgames
Instagram: / aiandgames
Threads: www.threads.net/@aiandgames
--
If you're interested in working with us, check out the following:
Business Enquiries: contact@aiandgames.com
Arrange Meeting: calendly.com/aiandgames - Наука та технологія
Thanks again to Purnendu and the team at Convai for their support in putting this episode together.
If you're keen on trying out Convai's teach, head over to: www.convai.com
I think AI generation is strongest when not asked to improvise too much, so it seems especially great to fill the seams while keeping things like quests and such with predesigned content.
I wish for an elder scrolls type game where you can interact with every NPC via AI.
Imagine walking into an inn and having a chat with an NPC about the area, points of interest or what mission they are on - and if they want to join your team or not, there are so much opportunities for this and I bet bethesda has already thought about this, and my bet is that they will make an attempt to implement such a system.
There is a nexus mod for that in Skyrim
@@DassVeryGood I mean more a game that is build up with this from the ground up and not just being able to listen to generated dribble from npc.
That's the dream. I hope they make it less chatty too. Every AI I have seen so far talks too much and are averse to short sentences, which are more natural.
I'd love to be able to give simple commands to NPCs that are in your group like go over there, sneak there, attack that thing, go open that chest. I think this is something possible in the future with this tech
Lol, that would also mean that Bethesda make quality games xD
So as I understand it, they are essentially building question answering systems on top of some provided lore. I believe the QA approaches with language models, at least thus far, are based around comparing the query embeddings with the pre-calculated embeddings of the knowledge source, and providing the section of the knowledge source that matched best, along with the query, in the prompt sent to the language model. That sounds like it is what's happening here, perhaps with some scope to add some stylistic context in too (answer as a curt but witty character, etc.). I could see this being an interesting way of turning a structured lore bible, with information tagged to easily distinguish which people or groups of people know it, into dialogue for conversing with background characters, or perhaps for drafting scripts. Only, that needs the careful construction of such a bible, which may end up being more work than just fully scripting the dialogue from the outset.
Only, this is still be relying on inferencing a large language model. That's too costly to be done locally, and would thus mean an external API call for every dialogue response. For anything widespread, either convai would have to have an in-house LLM that is robust to scale, or any game developers using this will have to source their own.
Once we get past the human encyclopedia phase, I'm really looking forward to the LLM-generated NPCs who will tell me they don't want to talk to me, just like in real life.
The future is looking bright!
Exactly! We need npcs with an attitude problem and a generally negative outlook on life and even those with some contempt for human players.
I swear if this isn't getting into tje elder scrolls 6 i'm going to cry
I wonder if this could be made to work better (as in: less verbatim repetition of an input, less "lore dump" moments) by using a format like that used in "second brain" note-taking software like Obsidian, which has backlinks between notes (like how Wikipedia links non-hierarchically between pages).
Generally, notes in systems like this (or if using Zettlekasten) are much much shorter. They're generally "atomic" notes that contain the least viable information, more like little facts, then they are linked together.
That way, one "Backstory" note might list where Dave was "born", but the specific city would be backlinked rather than elaborated on in that note. Likewise, his "parents" would be backlinked so, if you ask more about the parents, you get more about the parents, but he doesn't just tell you his entire backstory all at once. Similarly, specific "memories" could be backlinked. That way, if you ask about his backstory, he doesn't read all the text, he just mentions that he grew up there and has fond memories of childhood. If you ask about fond memories of childhood, it could then use more from the notes about "memories".
Thanks for the info
If i didnt just accept a full time faculty position, I'd be excited to continue my undergraduate research on NPCs that attach emotional sentiment and recursive trust vectors to connected information to recall and adapt relationships. I couldn't use natural language back then so it was clunky, but now it's pretty trivial to run a local LLM and ChromaDB for natural language generation and recall. My masters was also on finding different kinds of associations between words and information based on variable attention windows, so complex memory would be a fun place to explore. Great video, fun to see the work on these kind of systems!
I fully agree that the voice in/out is a big limiter. I'm more interested in creative aid than full generation, and I could see it more helpful right now to generate obscene amounts of branching text dialogue and discrete options that cover a huge range that wouldn't be authorable, but could be revised.
Well hey, let's take a moment to celebrate that new faculty gig, congratulations!
what i want is to see remakes of stuff that is bound to sell, like wotlk with all new skins, matching newer stuff. there is also the unreal engine demos of if floating around youtube, which would be neat too.
wotlk, if?
It seems like a decent enough interface between the different llms, but it sounds like this technology won't really take off for a few more years, when the language models are smaller and more specific, and able to run locally.
I mean.. Microsoft phi is already pretty damn good and fast at only 2 gigs. Uses very little vram. A little direction for it and it could be make an excellent ai.
I wish you would have declared more clearly that this is sponsored instead of a tiny text in the thumbnail and a mention if you extend teh description. I like your videos but this is somewhat dishonest.
It is made clear this is a sponsored video twice in the video itself. There's a lower third in the video at 01:09 and again in the video title at 02:22.
Sorry if it didn't seem obvious, but between the video, the thumbnail, the description and the tag in UA-cam, there are 5 different parts of the whole package pointing out that it's sponsored.
@@AIandGames As someone who often listens to videos instead of watching them, a quick audio call out could be added. You clearly did nail the visual requirements though.
@@AIandGames I missed all of those until I saw the thumbnail bigger size when I linked it on discord, probably because I was mostly listening instead of watching.
But fair enough I guess.
@@AIandGamesyou gotta say it lol, anything short of that is dishonest
@@AIandGamesI missed all the visual clues because I was focusing on the audio, trying to sort out your funny and (sadly) uncommon accent. English is not m'y native language, and understanding everything you say including jokes and familiar expressions required all my attention. You could easily have replaced one of the visual clues with one oral mention of the sponsorship.
The question I have is why? If I am getting a quest from an NPC why would I care about that NPC? Why would I talk to a Boss? The novelty will wear off pretty quickly. After messing with it, I likely would use key words/sentence to skip talking to it. I don't see the point beyond entirely narrative driven games, and even then the appeal is the tightly written story.
There is a game called Façade released in 2005!!! that basically does what this does without all the fanciness, and certainly better. It's essentially a short text adventure, but fully voiced and animated, down to emotional response. It does a surprisingly good job at parsing an input and picking an output. How it ends is depends entirely on you. You can insult, close the door on them, break up the couple or reconcile or status quo, act weird, make wild claims like sleeping with his girlfriend.
So yeah, colour me unimpressed.
Indeed, longtime viewers will be *very* familiar with Facade. I explain how it works in episode #49 of AI and Games. It's one of my most popular episodes.
You are thinking to small. Imagine unlimited versions of quests. You would HAVE too listen to him if everytime the quest was different.
@@xanmal7042 We already procedurally generated quests/content/descriptions since forever. An LLM can't generate quests, what it can do is fluff up the quest's description, obscuring the critical information of when, what, where, who, how.
Imagine being forced to listen to a generic backstory to every pizza delivery to get the address, that would get tiresome real quick. Replace pizza delivery with infinity stones, adding generated text doesn't change the genericness.
'Ai' is intentionally mis-marketed because of capitalism. No one ask, what does it actually do. Once you get over the magical hype you will be like, that is dumb af, why did they think that was appropriate, oh right, they wanted to pump up the share price, look busy or fleece start up investors.
In the film industry, they found that VFX are more asymptotic than linear. The Uncanny Valley gets worse as you approach photorealism, and the last 10% seems to take significantly more time and effort than the first 90%. I feel like AI characters are going to have the same issues in the near term. I'm wondering if there are ways to reduce people's expectations so the AI can exceed them?
Just one idea, but I'd love to see a database system where the developers can generate curated, high-quality answers to use in real-time. If the player asks a new question, the AI could respond with "I don't know that one, I'll have to ask my colleagues" (or whatever), and add the question to a list for the developers to review. It shouldn't take too long for a human with an "approve, regenerate, blacklist" button to churn through a few hundred questions and collect a semi-authored set of reasonable responses.
I just hope that "See that guy? You can talk to him!" doesn't become the next "See that mountain? You can climb it!" The technology is interesting, but the gimmick potential seems pretty real, too.
My question for you Tommy, did you try to have 2 bots interact with each other? Just curious what a convo with two bots would turn out as...
No, none of the demos were built to support two bots talking to one another. Nor do I believe, at present, the API supports it? I think it's purely one sided in how conversation is initiated.
That said, I imagine if that did happen, it would just be never ending. 😅
@@AIandGames Hey, Convai dev here. Guess what...
We now have NPC2NPC for both UE and Unity
Integrating interactive AI systems into games is not as straightforward as it might seem, especially with character dynamics.
Take for instance an NPC who has something to hide from the player. Over time, the narrative is such that the player builds a bond with this NPC. Ideally, as this trust develops, the NPC would begin to share its secrets. But achieving this transition isn’t as simple as flipping a switch.
- Firstly, a considerable amount of writing still has to underpin these interactions. We can't just rely on AI to generate dialogue without a solid foundation. This means developers and writers still have a significant role, even with advanced AI support.
- Additionally, there's the unpredictability factor. AI might produce a seamless interaction for one player but might stumble with another, offering them a dialogue that doesn’t quite fit the moment.
- Then there's the matter of keeping the NPC consistent with its intended role. How do we ensure that an AI-powered NPC stays in character, especially when balancing complex situational constraints?
And of course, there's the distinction between primary and secondary characters. Using AI for a random NPC is one thing. But for primary characters, with their elaborate backstories and significant role in future plot events, the stakes are much higher. They can't just have a generic AI response; they need depth and consistency that aligns with the broader storyline.
So if players get used to dynamic, AI-driven secondary NPCs, and then encounter main characters that seem static in comparison, it might break the immersive experience.
In sum, while the idea of AI-enhanced gaming interactions is intriguing, there are plenty of hurdles to consider. It’s not just about technological capability, but about preserving the essence and integrity of a game's narrative.
Meanwhile, I would love to see it used in indie titles that may have no issues mentioned above, say "The long drive" NPC or alike.
And hey, if writing was easy for AI, I would've gotten this on the first try!
Hello, we understand these concerns at Convai. It is not trivial and we will work on enabling more complex characters, with a deeper sense of who they are and keep consistency. We are also working on how to maintaining long term memory and understand the players. There is research and quite a bit of work here. Together with narrative designers and story tellers we hope to bring to life the characters they envisioned and for those to be able to converse about anything the player wants while maintaining the identity given to it.
That AI voice is amazing...
I thought you made NPCs through Coitas. That’s what the nice man in the Mc’donalds drive through told me. He said that my children would be NPCs is this valid. Somebody please tell me.
Well, I wouldn’t say he’s wrong
Now you test Inworld AI.
Yessir
Oh hey, I'm first!
Wrong
Lol can you stop talking please
No.
Referring to your comment at ua-cam.com/video/nZE0gkFKMt8/v-deo.html 😂@@AIandGames
Yeah sorry, I replied, realised what you meant and then forgot to delete my message 🤣
damn!! this sucks!