Love this! I was thinking about building something similar but using Voiceflow to handle the conversation dialog for the NPCs. Thank you for the walkthrough - your series is awesome!
Thanks for the video! It's really impressive, I don't have the knowledge to comprehend and understand it all but I try. You do really cool and necessary things. I dream of having such NPCs in Virtual Reality and being able to fully interact with them. Thanks again. to you for your work and for showing it all. I was able to repeat the version of the text box and so far I think I'll stop there, but I have great interest and pleasure to follow your videos and recommend them to my friends.
I started with the file from the first video. I was able to scrub through this video to pick out the new components and set those up. Everything works except for the Whisper component. It functions erratically. The NPC doesn't hear the question and "Sent Message" doesn't display the question. It showtimes shows a couple of periods or a random question. The "Received Message" appears, but includes the NPC's name: "Ruth: How can I help..." One time the entire conversation appeared in the "Received Message" field for both Avatar and NPC. The standard "Input Field" and "Send Button" work as expected with the NPC speaking.
I was able to fix the problems by going through every script and comparing the various version in the different branches. In addition, I updated the packages and made sure the correct version of ChatGPT was being used. One future modification would be to have different animations while playing back the audio. Thank you Sarge for providing these excellent video tutorials.
I love your videos! Can I ask you one thing? I am following through with using chatGPT, and I am curious how can I change the height or width of the text boxes. The height of Sent Message/User Message seems to be fixed in 30, and I am struggling with changing the padding between the texts. Can you please help? thank you.
Got the error:Assets\Scripts\TextToSpeech.cs(17,51): error CS0103: The name 'Credentials' does not exist in the current context. Could you please help?
Great video, I'm currently working on this project, but I encounter some issue with the response where the response get cut-off at the end, and I don't know how to fix it. meaning when if it should reply "what kind of assist you need" it reply "what kind of" instead of the whole sentence. Thanks for the video, it's really cool!
Hi Sarge! Thank you for your amazing videos! I have an issue - I used your scripts in my project but the sentences generated from GPT are not completed, they are missing the last few words. I believe the issue is in the GPT and not AWS Polly since the text itself is getting cut, and the speech is according to the cut text. Did anyone else also have that problem and knows how to solve it?
Hi Neta, I never encountered this myself but remember others mentioning this before. This might also be due to max token limit. You might wanna change that. Also, to simplify smart NPC integration I developed Neocortex, you might wanna check it out! neocortex.link
It helped me a lot by watching the video. It was a project description that fits the direction I wanted to do. I just have a question. Can I set the voice of that NPC differently? I want to make the voices of NPCs sound different.
Hi, sure you can set different voices. I was using AWS Polly which provided around 10 different male and female voices. Simply you could change the preset, that I showed in AWS Polly video.
@@sgt3v As far as I know, when you set up Prompt, you set it up for NPC. However, is there a maximum text limit for the content required for these settings? If there is, how much total is allowed?
Awesome, thanks so much for the videos and the free code! I have a question: why did you tag the starting prompt as "user" and not "system"? OpenAI says a system prompt is more respected, so it seems it might work better to keep the character in line?
As far as I understand a user must interface a system or assistant, in this case, I as a user, tell the assistant what to do. platform.openai.com/docs/guides/chat
@@sgt3v Well as I understand it the user in this case is the player. In your link OpenAI used the system prompt as setting up the general role of what ChatGPT is supposed to be, and the user is the one interacting with that role. At least that's how I use it in my experiments.
@@krawlak The message structure is pretty much the same as ChatGPT user interface itself. As user, first you provide an instruction and expect the replies in a form of assistant or system. When you go to ChatGPT, with the user role you tell the assistant what it should do.
@@sgt3v Yes, but as I understand via API you can use the system prompt to keep the AI more tied to some general setup, which gives you more stability than possible in the web ui.
@@krawlak seems like according to the docs, it's supposed to start with system then goes as user/assistant. So initial prompt can be delegated to system role. I'll give this a try, thank you for pointing that out. platform.openai.com/docs/guides/chat/introduction
Hi! I am calling SendReply() in the Start() method, but GPT is doing all the tasks at the same time even using ... Any ideas? myPrompt: First greet the user and wait for his response, Then ask for his money Then ask for his weapons . Gpt never waits for user's response... Thanks!
Does the project file contain the final version shown in this video or do you need to assemble your own assets? If that's not the case, what's the name of the completed scene?
Hi Juan, tbh I haven't use the Spatial world creator myself. But if there is support for external packages it should work. Considering that you can almost hoist games there now, should be possible.
Woow very interesting Sarge mind blow is awesome thanks for help us with code too l have subscribed and l am going to watch all videos when l get data 😁 wifi data is expensive
Hi Sarge! Thank you so much, I'm working on a similar project and your videos are very useful! Are you considering using embeddings to reduce API costs? For example, if you wanted to save the current game state (including all conversations) and resume the game from where it left off, how could you minimize API calls (and tokens) as much as possible? Will you be making a video about it?
Hi Hina, embedding endpoint is available through the package and there are tests for it in the package, you can already embed certain initial prompts using that, and use the returning model id instead to use it. I do plan to put my NPC creation prompt engineering tutorials into more structure but I will need to find the time for it.
@@sgt3v Hello, sorry but I can't understand how to pass the model ID as a prompt, could you explain it to me better? I'm using the gpt-3.5-turbo model.
I suddenly have been getting this error when trying to get GPT response: JsonSerializationException: Could not find member 'warning' on object of type 'CreateChatCompletionResponse'. Path 'warning', line 2, position 12. I haven't changed the json and things were working before. PLS Help.
I got same error so I asked gpt and got this response "The error you're encountering appears to be related to JSON deserialization. You're trying to deserialize some JSON into an object of type CreateChatCompletionResponse, but the deserialization process is failing because it's expecting to find a member (or property) named warning in the object, which isn't present." So I changed jsonSerializerSettings in OpenAiApi.cs MissingMemberHandling = MissingMemberHandling.Error, to MissingMemberHandling = MissingMemberHandling.Ignore, error gone!
I recommend keeping the MissingMemberHandling.Error, so you can know if there is an issue with the incoming data. OpenAI added a Warning field which was not there before, the issue was due to that. The field is introduced to the response types in 0.1.15
@@sgt3v The irony here is that I'm using chatGPT to correct coding errors since I don't know how to code. My goal is to create a working prototype; it doesn't have to be clean for me.
Hi, I'm getting the error message "Error Message: You exceeded your current quota, please check your plan and billing details." I have the free plan for the Open AI API, but I've only used $0 out of $18. Can't I use the ChatGPT API in Unity with the free plan? Do I need to upgrade to a higher plan?
Im trying to add the package from github but i get an error when installing. "does not point to a valid packet". Have tried downloading and importing from folder also. Any suggestions? I've tried on 2 different computers, Mac and Windows.
When using the async method, you can pass a cancellation token to stop it when you want. However, there is no endpoint to tell OpenAI API to stop producing tokens.
This is great but I'm having trouble getting phrase embedding to work, at first I was trying to get it to add a code when it mentioned certain information in its response but I've even tried getting it to add a code if the player mentioned something - then mentioning it but I'm getting nothing added to the response. Was there something else you had to do to get that to work? I even tried copying your exact phrasing and have it return a code when my character is saying good bye
You might try to use a reduced temperature value which would be more strict in following the prompt. platform.openai.com/docs/api-reference/chat/create#chat/create-temperature
@@sgt3v I still haven't managed to get this working. Have you been able to add additional phrases? This is my prompt now, I've not included any of the game world or background information just to keep it really simple This is a murder mystery game. Act as the police constable assisting the detective. Reply to questions in the first person, incorporating your personality and knowledge. Do not mention that you are an AI and do not break character. If a question is outside your knowledge of the case, respond that you do not know if my reply indicates that I want to end the conversation, finish your sentence with the phrase END_CONVO if my reply indicates that I want to obtain a search warrant, finish your sentence with the phrase SEARCH_WARRANT if my reply indicates that I want to do a background check, finish your sentence with the phrase BACKGROUND_CHECK If I say goodbye, it does finish it off with END_CONVO If I ask for a search warrant, it asks for the reason and doesn't include the phrase. If I ask for a background check it tells me its an AI language model and doesn't have access to that information :(
You can try to record chunks of audio in js and send them to unity to be processed, this would require audio processing knowledge most probably bus should be possible. Until Unity supports Microphone in WebGL there is no native way to do this.
Thanks for this video sarge this has been a lifesaver, buy I'm trying to activate the startRecording() function with my oculus trigger button and its not really responding can you help me out please.
Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games
neocortex.link
This video just came in the perfect timing to help my work! Thank you!
Glad to hear it Hsiang!
Also for me!
Love this! I was thinking about building something similar but using Voiceflow to handle the conversation dialog for the NPCs. Thank you for the walkthrough - your series is awesome!
Your video series are great. continue with it. Thanks a lot
You're welcome!
Outstanding! Already waiting for the next tut :)
Excellent work Sarge !! you saved my "days"...not "day" !
Thanks for the video! It's really impressive, I don't have the knowledge to comprehend and understand it all but I try. You do really cool and necessary things. I dream of having such NPCs in Virtual Reality and being able to fully interact with them. Thanks again. to you for your work and for showing it all.
I was able to repeat the version of the text box and so far I think I'll stop there, but I have great interest and pleasure to follow your videos and recommend them to my friends.
Thank you Max, really appreciate it.
Awesome! Just i want to have in my world. And further on in games.
Dude, possibilities are endless... I think we already living in the future 😆
I started with the file from the first video. I was able to scrub through this video to pick out the new components and set those up. Everything works except for the Whisper component. It functions erratically. The NPC doesn't hear the question and "Sent Message" doesn't display the question. It showtimes shows a couple of periods or a random question. The "Received Message" appears, but includes the NPC's name: "Ruth: How can I help..." One time the entire conversation appeared in the "Received Message" field for both Avatar and NPC. The standard "Input Field" and "Send Button" work as expected with the NPC speaking.
I was able to fix the problems by going through every script and comparing the various version in the different branches. In addition, I updated the packages and made sure the correct version of ChatGPT was being used. One future modification would be to have different animations while playing back the audio. Thank you Sarge for providing these excellent video tutorials.
But you need to have added credit card on openai, so it does not run for free?
You can use it for free a time span with a certain quota, later you will need to pay.
I love your videos! Can I ask you one thing? I am following through with using chatGPT, and I am curious how can I change the height or width of the text boxes. The height of Sent Message/User Message seems to be fixed in 30, and I am struggling with changing the padding between the texts. Can you please help? thank you.
Got the error:Assets\Scripts\TextToSpeech.cs(17,51): error CS0103: The name 'Credentials' does not exist in the current context. Could you please help?
It's most likely because you copy and pasted the code from github to your file. You have to change "Credentials" to your actual keys from Amazon AWS
Hello friend, does this work with android/ios? I bought 2 assets from the asset store, unfortunately none of them are compatible.
I wonder if this is possible in AR too
Great video, I'm currently working on this project, but I encounter some issue with the response where the response get cut-off at the end, and I don't know how to fix it. meaning when if it should reply "what kind of assist you need" it reply "what kind of" instead of the whole sentence. Thanks for the video, it's really cool!
same problem , did you solve the problem by any chance 🥲
ADAM YA GELDİ ADAM
:0)
Hi Sarge! Thank you for your amazing videos!
I have an issue - I used your scripts in my project but the sentences generated from GPT are not completed, they are missing the last few words. I believe the issue is in the GPT and not AWS Polly since the text itself is getting cut, and the speech is according to the cut text.
Did anyone else also have that problem and knows how to solve it?
Hi Neta, I never encountered this myself but remember others mentioning this before. This might also be due to max token limit. You might wanna change that.
Also, to simplify smart NPC integration I developed Neocortex, you might wanna check it out! neocortex.link
Outstanding series! Did you try Ready Player Me with Media pipe facemesh libraries? Would be a game changer :)
Yes, I'll drop a react tutorial on this tomorrow 🤞
It helped me a lot by watching the video.
It was a project description that fits the direction I wanted to do.
I just have a question. Can I set the voice of that NPC differently?
I want to make the voices of NPCs sound different.
Hi, sure you can set different voices. I was using AWS Polly which provided around 10 different male and female voices. Simply you could change the preset, that I showed in AWS Polly video.
@@sgt3v As far as I know, when you set up Prompt, you set it up for NPC.
However, is there a maximum text limit for the content required for these settings? If there is, how much total is allowed?
Thank you!
You're welcome!
this is amazing 🤩🤩🤩🤩😍😍😍😍😍
Awesome, thanks so much for the videos and the free code! I have a question: why did you tag the starting prompt as "user" and not "system"? OpenAI says a system prompt is more respected, so it seems it might work better to keep the character in line?
As far as I understand a user must interface a system or assistant, in this case, I as a user, tell the assistant what to do.
platform.openai.com/docs/guides/chat
@@sgt3v Well as I understand it the user in this case is the player. In your link OpenAI used the system prompt as setting up the general role of what ChatGPT is supposed to be, and the user is the one interacting with that role. At least that's how I use it in my experiments.
@@krawlak The message structure is pretty much the same as ChatGPT user interface itself. As user, first you provide an instruction and expect the replies in a form of assistant or system.
When you go to ChatGPT, with the user role you tell the assistant what it should do.
@@sgt3v Yes, but as I understand via API you can use the system prompt to keep the AI more tied to some general setup, which gives you more stability than possible in the web ui.
@@krawlak seems like according to the docs, it's supposed to start with system then goes as user/assistant. So initial prompt can be delegated to system role. I'll give this a try, thank you for pointing that out.
platform.openai.com/docs/guides/chat/introduction
Hello Bravoo !!! ur Genius
Hi! I am calling SendReply() in the Start() method, but GPT is doing all the tasks at the same time even using
... Any ideas? myPrompt: First greet the user and wait for his response,
Then ask for his money
Then ask for his weapons
. Gpt never waits for user's response... Thanks!
SOLVED Model = "gpt-3.5-turbo-0301",
Messages = messages,
MaxTokens = 20,
Temperature = 0,
Stop = "
" //This is very important
You are the best!
Does the project file contain the final version shown in this video or do you need to assemble your own assets? If that's not the case, what's the name of the completed scene?
Hi, it has everything that you in the video. There are different branches in github for first and second video make sure to get the right one.
@@sgt3v Thank you fro the quick reply. I probably downloaded the wrong version. I'll try again.
Hey! How can I add Role.System content in order to work more like an assistant chatbot? Thanks!
You do it in the chat message objects in the role field as a string.
Thanks Sarge, great video! This means if we upload and publish this on Spatial it will work the same way?
Hi Juan, tbh I haven't use the Spatial world creator myself. But if there is support for external packages it should work. Considering that you can almost hoist games there now, should be possible.
@@sgt3v Thanks for the answer! Will try it in a few weeks and will let you know.
just perfect !
Woow very interesting Sarge mind blow is awesome thanks for help us with code too l have subscribed and l am going to watch all videos when l get data 😁 wifi data is expensive
Hi Sarge! Thank you so much, I'm working on a similar project and your videos are very useful! Are you considering using embeddings to reduce API costs? For example, if you wanted to save the current game state (including all conversations) and resume the game from where it left off, how could you minimize API calls (and tokens) as much as possible? Will you be making a video about it?
Hi Hina, embedding endpoint is available through the package and there are tests for it in the package, you can already embed certain initial prompts using that, and use the returning model id instead to use it. I do plan to put my NPC creation prompt engineering tutorials into more structure but I will need to find the time for it.
@@sgt3v Thank you a lot! I will try it 😄
@@sgt3v Hello, sorry but I can't understand how to pass the model ID as a prompt, could you explain it to me better? I'm using the gpt-3.5-turbo model.
I suddenly have been getting this error when trying to get GPT response:
JsonSerializationException: Could not find member 'warning' on object of type 'CreateChatCompletionResponse'. Path 'warning', line 2, position 12. I haven't changed the json and things were working before. PLS Help.
I got same error so I asked gpt and got this response
"The error you're encountering appears to be related to JSON deserialization. You're trying to deserialize some JSON into an object of type CreateChatCompletionResponse, but the deserialization process is failing because it's expecting to find a member (or property) named warning in the object, which isn't present."
So I changed jsonSerializerSettings in OpenAiApi.cs
MissingMemberHandling = MissingMemberHandling.Error,
to
MissingMemberHandling = MissingMemberHandling.Ignore,
error gone!
@@yjlee4386 OMG Thank you! That worked! I have no idea how I got into that state. Everything was working one day, then it was just broken the next!
I recommend keeping the MissingMemberHandling.Error, so you can know if there is an issue with the incoming data.
OpenAI added a Warning field which was not there before, the issue was due to that. The field is introduced to the response types in 0.1.15
@@sgt3v The irony here is that I'm using chatGPT to correct coding errors since I don't know how to code. My goal is to create a working prototype; it doesn't have to be clean for me.
Hi, I'm getting the error message "Error Message: You exceeded your current quota, please check your plan and billing details." I have the free plan for the Open AI API, but I've only used $0 out of $18. Can't I use the ChatGPT API in Unity with the free plan? Do I need to upgrade to a higher plan?
I can't really know maybe you are using a wrong api-key, or the free usage time is expired? Something to consult to OpenAI.
can solved this? same problem here
@@AIforKidss No :s
@@SrKeitaro i think work just paid mod
Im trying to add the package from github but i get an error when installing. "does not point to a valid packet". Have tried downloading and importing from folder also. Any suggestions? I've tried on 2 different computers, Mac and Windows.
maybe you forgot a space at the end or beginning?
@@sgt3v what do you mean?
@@clobman when you paste the package .git url, if there is a space at the end accidentally, it will not work.
@@sgt3v just triple checked and that didn't seem to be the issue. do I need any packages prior to installing this one?
@@clobman it should bring the dependencies with itself, can you check if you pasted the git url like this:
github.com/srcnalt/OpenAI-Unity.git
Is your ChatGPT project capable of canceling responce like "Stop Generate" button in ChatGPT?
When using the async method, you can pass a cancellation token to stop it when you want. However, there is no endpoint to tell OpenAI API to stop producing tokens.
This is great but I'm having trouble getting phrase embedding to work, at first I was trying to get it to add a code when it mentioned certain information in its response but I've even tried getting it to add a code if the player mentioned something - then mentioning it but I'm getting nothing added to the response. Was there something else you had to do to get that to work? I even tried copying your exact phrasing and have it return a code when my character is saying good bye
You might try to use a reduced temperature value which would be more strict in following the prompt.
platform.openai.com/docs/api-reference/chat/create#chat/create-temperature
@@sgt3v I still haven't managed to get this working. Have you been able to add additional phrases? This is my prompt now, I've not included any of the game world or background information just to keep it really simple
This is a murder mystery game. Act as the police constable assisting the detective. Reply to questions in the first person, incorporating your personality and knowledge. Do not mention that you are an AI and do not break character. If a question is outside your knowledge of the case, respond that you do not know
if my reply indicates that I want to end the conversation, finish your sentence with the phrase END_CONVO
if my reply indicates that I want to obtain a search warrant, finish your sentence with the phrase SEARCH_WARRANT
if my reply indicates that I want to do a background check, finish your sentence with the phrase BACKGROUND_CHECK
If I say goodbye, it does finish it off with END_CONVO
If I ask for a search warrant, it asks for the reason and doesn't include the phrase. If I ask for a background check it tells me its an AI language model and doesn't have access to that information :(
Does the oculus lipsync support IOS?
unfortunately not as far as I know.
Great Content,
But How can I use this project in WebGL, because Unity doesn't provide direct integration to Microphone.
You can try to record chunks of audio in js and send them to unity to be processed, this would require audio processing knowledge most probably bus should be possible. Until Unity supports Microphone in WebGL there is no native way to do this.
@@sgt3v Don't know how to do this
But I'll try
Thanks
Thanks for this video sarge this has been a lifesaver, buy I'm trying to activate the startRecording() function with my oculus trigger button and its not really responding can you help me out please.
The issue must be UI Event Handler related, can't really know why.
@@sgt3v what do you suggest I do if it’s a problem from the ui event handler
Is there a problem with Unity example files not working?