Hi! I really enjoyed this video! And can I know how you programmed these gpts? Like... did you just write a system prompt (or create a customized GPT) and use the Openai's normal text-to-speech technology? Or did you do any special coding/handling? And if you don't mind, can you share your system prompt? They are so natural and stay in their characters so well! Especially the last one ---- it feels almost scripted. I am trying to do some study on playing games with AI, so I would really appreciate it if I could use your case as an example case! Thanks a lot!
Hey! Thanks for watching the video :) Apologies for taking so long to respond. Sure I can share the code and talk a bit about it. Here's a link to the zip in onedrive: drive.google.com/file/d/1z2m5RtZw1O97-A-ZRBzNy-R1ijhUFzZy/view?usp=drive_link It's a bit of a mess so it's not ready to go up on github lol, but that's the code base I used for the video. It's been several months since I've even tested it, so with api and library changes I'm not sure if it's still functional as is. But it should show how I made the prompts and handled the flow. I used gpt-4-1106-preview as the llm, I just used it's pre-existing knowledge of the game to run it, so no advanced rag setup or anything like that to load in documentation about the game. I used the openai/whisper-small model to do speech to text, and then eleven labs for text to speech to give the AI it's voice. There's a lot of optimization that can be done with this, it takes almost a second before the AI responds to user input, a lot of this has to do with the speech to text using whisper because I'm waiting for the user to stop speaking before transcribing (in my latest video making a discord bot I learned how to do live transcription with the model locally, but it's not included in this file), so there's a lot of delay there, but I'm sure there are other aspects that could help improve it's responsiveness. All of the model interactions in the video are real I'm glad to say :) But that's not to say the model was perfect every time. Marcus (the evil AI at the end) was particularly hard to get right. Had the idea for what I wanted the ai to do and had to modify the prompts quite a bit to get it right, but everything you see in the video was generated by the AI and not scripted by me. Even the final closing monologue lol Honestly the project could use a lot of improvements and refinements, and I'd like to revisit GPT playing games like this. I've learned a lot about better prompting, and how to fine tune models and reduce latency, and I'd like to experiment with ways to make the conversation feel more natural for the end user. Feels a bit stiff when actually playing with it. I'd also like to move away from paid services like openai and eleven labs and go full local, been doing a lot of work with open LLM models and fine tuning them, but haven't played with local text to speech yet. But anyway, think I've rambled on long enough lol. LMK if you have issues downloading the code, I cant' promise the code will run since apis and libraries around LLMs change so often, particularly with Eleven Labs I think they had a breaking update since this video. But I'll try and answer any questions I can :) Good luck! And if you build anything cool please feel free to share it :)
@@ColdstartCoder wow thanks for sharing your code! (And luckily I am a coder myself so I can totally understand it!) Yeah I have no doubt that the script is all AI-generated, and the generated content indeed is very impressive! I do wonder if the game manual pdf is loaded, will these AIs perform better. Speaking of the local model -> I agree. I feel that for quick prototyping it is nice to use Openai's model, etc because it is easy - just an API away, but fine-tuning a local model will give more control. Uh speaking of my project - I noticed the trend of AI games (AI story co-creating games like AI Dungeon, AI murder mystery games, etc) focusing on replacing humans by AI. I wonder if the experience is actually better. Or in which way is better, in which way is worse ----> that's why I am particularly interested in your video because I can compare this video with my experience playing this game with my friend. On top of that, I am trying to make an experimental game that explores the non-human aspects of AI as game mechanics or narratives. If you are interested, I can share more details (but it is probably not suitable for spamming your comment area hahaha)
Before watching I wil already say:
It was nice to know you
Hi! I really enjoyed this video! And can I know how you programmed these gpts? Like... did you just write a system prompt (or create a customized GPT) and use the Openai's normal text-to-speech technology? Or did you do any special coding/handling?
And if you don't mind, can you share your system prompt? They are so natural and stay in their characters so well! Especially the last one ---- it feels almost scripted.
I am trying to do some study on playing games with AI, so I would really appreciate it if I could use your case as an example case! Thanks a lot!
Hey! Thanks for watching the video :)
Apologies for taking so long to respond.
Sure I can share the code and talk a bit about it.
Here's a link to the zip in onedrive: drive.google.com/file/d/1z2m5RtZw1O97-A-ZRBzNy-R1ijhUFzZy/view?usp=drive_link
It's a bit of a mess so it's not ready to go up on github lol, but that's the code base I used for the video.
It's been several months since I've even tested it, so with api and library changes I'm not sure if it's still functional as is. But it should show how I made the prompts and handled the flow.
I used gpt-4-1106-preview as the llm, I just used it's pre-existing knowledge of the game to run it, so no advanced rag setup or anything like that to load in documentation about the game.
I used the openai/whisper-small model to do speech to text, and then eleven labs for text to speech to give the AI it's voice. There's a lot of optimization that can be done with this, it takes almost a second before the AI responds to user input, a lot of this has to do with the speech to text using whisper because I'm waiting for the user to stop speaking before transcribing (in my latest video making a discord bot I learned how to do live transcription with the model locally, but it's not included in this file), so there's a lot of delay there, but I'm sure there are other aspects that could help improve it's responsiveness.
All of the model interactions in the video are real I'm glad to say :) But that's not to say the model was perfect every time. Marcus (the evil AI at the end) was particularly hard to get right. Had the idea for what I wanted the ai to do and had to modify the prompts quite a bit to get it right, but everything you see in the video was generated by the AI and not scripted by me. Even the final closing monologue lol
Honestly the project could use a lot of improvements and refinements, and I'd like to revisit GPT playing games like this. I've learned a lot about better prompting, and how to fine tune models and reduce latency, and I'd like to experiment with ways to make the conversation feel more natural for the end user. Feels a bit stiff when actually playing with it. I'd also like to move away from paid services like openai and eleven labs and go full local, been doing a lot of work with open LLM models and fine tuning them, but haven't played with local text to speech yet.
But anyway, think I've rambled on long enough lol. LMK if you have issues downloading the code, I cant' promise the code will run since apis and libraries around LLMs change so often, particularly with Eleven Labs I think they had a breaking update since this video. But I'll try and answer any questions I can :)
Good luck! And if you build anything cool please feel free to share it :)
@@ColdstartCoder wow thanks for sharing your code! (And luckily I am a coder myself so I can totally understand it!)
Yeah I have no doubt that the script is all AI-generated, and the generated content indeed is very impressive!
I do wonder if the game manual pdf is loaded, will these AIs perform better.
Speaking of the local model -> I agree. I feel that for quick prototyping it is nice to use Openai's model, etc because it is easy - just an API away, but fine-tuning a local model will give more control.
Uh speaking of my project - I noticed the trend of AI games (AI story co-creating games like AI Dungeon, AI murder mystery games, etc) focusing on replacing humans by AI. I wonder if the experience is actually better. Or in which way is better, in which way is worse ----> that's why I am particularly interested in your video because I can compare this video with my experience playing this game with my friend. On top of that, I am trying to make an experimental game that explores the non-human aspects of AI as game mechanics or narratives. If you are interested, I can share more details (but it is probably not suitable for spamming your comment area hahaha)