Sarge
Sarge
  • 30
  • 290 724
Making Smart NPCs with Neocortex in Unity
Hi folks! I’m excited to share that I’ve finally brought my side project, Neocortex, to a point where I can release it and share the progress with all of you. The project is now live, and you can create an account to start exploring it right away!
Neocortex is a node-based web platform and a set of SDKs designed to help you create and use action-based smart non-playable characters (NPCs) effortlessly. With Neocortex, you can integrate conversational AI - via text or audio, into your games or virtual assistant projects with ease.
I’d love to hear your thoughts and feedback as you try it out!
(And sorry for my messy hair, hair band broke and could not delay making the vid.)
📌Neocortex Web Platform: neocortex.link
📌Join Neocortex Discord: discord.gg/sG7JpcPaQe
📌Follow on X: x.com/neocortexlink
📌Follow on GitHub: github.com/neocortex-link
⏲️ Timestamps:
00:00 ​ Introduction
01:02 Creating an Account
01:32 Neocortex Dashboard
01:59 What a Project Looks Like
03:30​ Creating a New Project
04:58​ Adding Nodes
06:06 Adding Objects and Actions
08:03 Adding Intents to Actions
10:28 Importing the SDK in Unity
12: 28 Create an API Key
12:45 Copying the Project ID
13:06 Testing the Chat Sample
13:55 Testing the Audio Sample
16:05 Your Feedback is Valuable!
#ai #gamedev​ #indiedev​ #unity​ #madewithunity
Переглядів: 1 322

Відео

Making your own AI Assistant!
Переглядів 1,1 тис.8 місяців тому
Hello folks, long time no see! I wanted to let you know that I am alive and am working on some cool things on the side. Last week Marques Brownlee @mkbhd roasted Humane AI Pin and Rabbit R1 handheld device pretty roughly. Here I want to show you why was that so and how easy it is to make such tools actually. In this video I wanted to show you how easy it will be making AI assistants using Neoco...
[SIGGRAPH Asia 2023, Real Time Live] Geometry Aware Texturing - Ready Player Me
Переглядів 577Рік тому
I had the opportunity to present our paper on Geometry Aware Texture Generation, and showcase its demo live at SIGGRAPH Asia 2023 in Sydney, Australia. It was an amazing feeling and experience to be at the big stage at SIGGRAPH and was the biggest audience ever that I had in front of me. Try the demo at: siggraph.readyplayer.me Note: The model in demo requires cold-start, so your first try migh...
Unity Netcode x Ready Player Me - Multiplayer Quickstart
Переглядів 1,2 тис.Рік тому
Unity now provides their own networking solution Unity Netcode for GameObjects, which helps you create multiplayer games without needing third party service. With Ready Player Me Netcode sample you can kickstart your multiplayer games with personal avatars without having any trouble. 🐦Follow me on Twitter: sgt3v​ Sources: 📌GitHub Repo of Netcode Support: github.com/readyplayerme/rpm...
Photon x Ready Player Me - Multiplayer Quickstart
Переглядів 3,5 тис.Рік тому
Having Photon Engine PUN2 support was one of the most asked things around Ready Player Me. One of the main reasons for having personal avatars is of course using them in social multiplayer games and applications. However that was a tricky thing to work with due to how Photon handles things in editor and not really flexible with Runtime character creation. The Unity package showcased in this vid...
Google Mediapipe Face Tracking in React with Ready Player Me Avatars!
Переглядів 12 тис.Рік тому
Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link Google finally released a new version of Mediapipe which helps with ARKit blend shapes usage. So I wanted to test this on Ready Player Me avatars and share the tutorial with you. 🐦Follow me on Twitter: sgt3v​ 📌GitHub Repo for this project: github.com/srcnalt/rpm-face-tracking 📌Tes...
Making NPCs with ChatGPT in Unity! 2
Переглядів 17 тис.Рік тому
Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link In this tutorial, we will continue the smart NPCs (non-player characters) project using Ready Player Me avatars, Unity Starter Assets, and the OpenAI API. Since the first video, I made videos about many different features of OpenAI API and other things around making smart NPCs, and this video...
Face Animation with Generated Audio using Oculus LipSync!
Переглядів 18 тис.Рік тому
Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link Previously we generated audio using Amazon Polly but that alone is not enough to complete the Smart NPC loop. We can animate the face using the audio using Oculus LipSync via Viseme Blendshapes. In this video, I continue from the Amazon Polly implementation and show you how you can use Oculus...
Streaming Responses with ChatGPT in Unity!
Переглядів 4,7 тис.Рік тому
Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link This weekend I worked on fixing the Android and IOS issues on building with the OpenAI Unity package. And also brought support for streaming responses to ChatGPT and text completion endpoints. 📌Follow me on Twitter: sgt3v​ 📌First Video: ua-cam.com/video/DjX8mjBCMcg/v-deo.html 📌Pre...
Text to Speech with AWS Polly in Unity!
Переглядів 16 тис.Рік тому
Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link Text to Speech has been the most requested topic to cover in the comments of the previous videos and it is now here. There are multiple different services such as Microsoft Cognitive Services and IMB Watson to help out with this but we will use Amazon Web Services (AWS) Polly in this video. 📌...
Speech to Text with OpenAI Whisper in Unity!
Переглядів 29 тис.Рік тому
Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link One of the latest abilities of OpenAI API is Speech to Text functionality provided using the Whisper model. Using this model we can send audio data to OpenAI API in Unity and receive a text back. The endpoint also supports auto-translation of many languages so you can upload a foreign languag...
ChatGPT vs Text Completion - OpenAI API in Unity!
Переглядів 6 тис.Рік тому
Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link Previous I created a ChatGPT like system using OpenAI API's Text Completion endpoint. Now that OpenAI released an endpoint for ChatGPT, I implemented that feature in my Unity package for OpenAI API and compared the Chat Completion Endpoint to Text Completion Endpoint. With Chat Completion End...
Making ChatGPT in Unity! Android, IOS and WebGL Builds
Переглядів 8 тис.Рік тому
Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link AI-generated content on the internet is booming and with OpenAI API you can make ChatGPT-like apps or use DallE to generate images using your OpenAI account. This tutorial is a continuation of the first video, where I cover the updates and how to handle the API Key side of your projects, and ...
Prompt Engineering in ChatGPT for making NPCs!
Переглядів 6 тис.Рік тому
Prompt Engineering in ChatGPT for making NPCs!
Making NPCs with ChatGPT in Unity!
Переглядів 53 тис.Рік тому
Making NPCs with ChatGPT in Unity!
Making ChatGPT in Unity!
Переглядів 48 тис.Рік тому
Making ChatGPT in Unity!
What is Ready Player Me Avatar API - Unity Tutorial
Переглядів 3,9 тис.2 роки тому
What is Ready Player Me Avatar API - Unity Tutorial
Ready Player Me x Unity Live Capture / Face Capture App
Переглядів 31 тис.3 роки тому
Ready Player Me x Unity Live Capture / Face Capture App
Unity Third Person Starter Assets and Ready Player Me Avatars
Переглядів 17 тис.3 роки тому
Unity Third Person Starter Assets and Ready Player Me Avatars
Level Selection UI - Unity Indie Game Devlog #9
Переглядів 8993 роки тому
Level Selection UI - Unity Indie Game Devlog #9
Making the Mouse Swim - Unity Indie Game Devlog #8
Переглядів 7283 роки тому
Making the Mouse Swim - Unity Indie Game Devlog #8
Making Audio Manager and Settings Window - Unity Indie Game Devlog #7
Переглядів 5743 роки тому
Making Audio Manager and Settings Window - Unity Indie Game Devlog #7
Designs UI and Buttons For My Game - Unity Indie Game Devlog #6
Переглядів 5033 роки тому
Designs UI and Buttons For My Game - Unity Indie Game Devlog #6
Why I Almost Gave Up On My Game Project - Unity Indie Game Devlog #5
Переглядів 6283 роки тому
Why I Almost Gave Up On My Game Project - Unity Indie Game Devlog #5
Tween Animations and Notification System - Unity Indie Game Devlog #4
Переглядів 1,9 тис.3 роки тому
Tween Animations and Notification System - Unity Indie Game Devlog #4
Fog of War and Player Map - Unity Indie Game Devlog #3
Переглядів 2,9 тис.3 роки тому
Fog of War and Player Map - Unity Indie Game Devlog #3
Tools Programming in Unity - Unity Indie Game Devlog #2
Переглядів 1,4 тис.3 роки тому
Tools Programming in Unity - Unity Indie Game Devlog #2
My Game's Back Story & Motivation - Unity Indie Game Devlog #1
Переглядів 1,4 тис.3 роки тому
My Game's Back Story & Motivation - Unity Indie Game Devlog #1

КОМЕНТАРІ

  • @tofinoguy
    @tofinoguy 2 дні тому

    There's no sample code when I imported it.

    • @sgt3v
      @sgt3v 2 дні тому

      It is imported via Samples tab in package manager.

  • @early21century
    @early21century 4 дні тому

    is it fully free for personal use and does it allow unrestricted nsfw conversations with npcs.

    • @sgt3v
      @sgt3v 3 дні тому

      It has credit limits for free users. And conversations can be about anything given that you provided right context.

    • @early21century
      @early21century 3 дні тому

      @@sgt3v hi thanks for responding how long does credits last.

    • @sgt3v
      @sgt3v 3 дні тому

      For now it lasts around 100 back and forth conversation exchange. You can find more info on the web page. I am working on making it cheaper for me so I can provide more free credits to new users.

  • @early21century
    @early21century 4 дні тому

    so if i use it for personal use its completely free also is it unrestricted like allowing nsfw conversations.

  • @AkhilanSKmd24b001
    @AkhilanSKmd24b001 4 дні тому

    Can you give website or any other way to download whisper ai as unity package

    • @sgt3v
      @sgt3v 3 дні тому

      Hi, there is no package that helps with that as much as I know. But I am planning to include it into my neocortex.link Unity SDK.

    • @akhilans.k.2965
      @akhilans.k.2965 3 дні тому

      can you say how to integrate whisper Ai(offline ) in unity

  • @mustafatanugur8276
    @mustafatanugur8276 12 днів тому

    Hocam merhabalar, bir oyun tasarlıyorum ve oyunumda büyüleri konuşarak kullanmak istiyorum. OpenAI ile yapılabilir mi ?

    • @sgt3v
      @sgt3v 12 днів тому

      Selam mustafa, bu islemlerin daha kolay yapilabilmesi icin Neocortex adinda bir proje yayinladim gecen ay, ve kisa bir video hazirladim. Hesap acip onu deneyebilirsin. ua-cam.com/video/Mw4EWoGcuLc/v-deo.html

  • @اخبارمنالعالم-ل3ج

    i want to ask you, can i create my own model like i use gpt3 and train him with my own data and use its api so he can answer me realated of the files that ive trained him with ?

    • @sgt3v
      @sgt3v 21 день тому

      Hi! Yes you can, however it is a very expensive and time consuming process and you would need a lot of data to do it. You can "fine tune" existing models this is one option, or the simplest way is to doing good prompt engineering.

  • @legendtreemedia2968
    @legendtreemedia2968 25 днів тому

    This is so cool! I'm so excited to play with this and support you as you keep building! Let me/us know if there is anything specifically you want to test. Thank you for making this!

    • @sgt3v
      @sgt3v 24 дні тому

      Thank you so much, really appreciate it. Subscriptions to the platform, and projects of developers who are using the tool will be the biggest help I can get, which would provide the biggest feedback. I am planning to spend entire Xmas holiday on improving and upgrading things,

  • @rexthedentaldinosaur1963
    @rexthedentaldinosaur1963 26 днів тому

    Awesome stuff man! This is very useful!

    • @sgt3v
      @sgt3v 25 днів тому

      Thank you, glad you found it useful!

  • @yunusanacal61
    @yunusanacal61 28 днів тому

    It looks perfect! So I am wondering if I want to more npcs and they have different jobs, it means am I create a new project for each npcs. Because I am tried to add new agent node but I couldn't connect the result node

    • @sgt3v
      @sgt3v 28 днів тому

      Hi Yunus you are right, at the moment it will have to be a new project. Agent input of the result node accepts only one input. I am planning to improve the visual language of the nodes and also want to provide some templates for developers to quick start as well.

    • @yunusanacal61
      @yunusanacal61 28 днів тому

      @ Okey, thanks for help! Me and my friends trying to make a VR project so its looks so helpful. Can we contact with you if we encounter a problem?

    • @sgt3v
      @sgt3v 28 днів тому

      @@yunusanacal61 Absolutely! I also want to create a VR demo project to show some use cases, would be happy to help.

  • @vishalpratap3267
    @vishalpratap3267 28 днів тому

    Bro i have question to you , how it will sychronuse over the network @sgt3v

    • @sgt3v
      @sgt3v 28 днів тому

      You can use a networking system such as netcode or photon to transmit the float values.

    • @vishalpratap3267
      @vishalpratap3267 28 днів тому

      @sgt3v I am currently using fishnet

    • @vishalpratap3267
      @vishalpratap3267 28 днів тому

      ​@@sgt3v I am currently using fishnet

  • @douglasbarbosa1463
    @douglasbarbosa1463 29 днів тому

    But using this credit system for a commercial application is very expensive

    • @sgt3v
      @sgt3v 28 днів тому

      Hi Douglas, soon I will be changing providers which will help reduce the costs immensely, this will help lift things off. I will also bring on device TTS support to the SDK. At the moment I am paying for the entire project from my pocket. This project does not have any investors or supporters.

  • @charlesdongoyo9187
    @charlesdongoyo9187 Місяць тому

    I really enjoyed playing with it. Great work👏

    • @sgt3v
      @sgt3v Місяць тому

      Thank you Charles. It is still early work, will be much better soon!

  • @mendieleelin5510
    @mendieleelin5510 Місяць тому

    Im super excited to play around with this! Thank you so much for creating this ❤

    • @sgt3v
      @sgt3v Місяць тому

      Thank you Mendie! Make sure to share what you work on with me!

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @sgt3v
    @sgt3v Місяць тому

    Checkout my new project Neocortex for simplest way to integrate Smart NPCs in your games neocortex.link

  • @Netad97
    @Netad97 Місяць тому

    Hi Sarge! Thank you for your amazing videos! I have an issue - I used your scripts in my project but the sentences generated from GPT are not completed, they are missing the last few words. I believe the issue is in the GPT and not AWS Polly since the text itself is getting cut, and the speech is according to the cut text. Did anyone else also have that problem and knows how to solve it?

    • @sgt3v
      @sgt3v Місяць тому

      Hi Neta, I never encountered this myself but remember others mentioning this before. This might also be due to max token limit. You might wanna change that. Also, to simplify smart NPC integration I developed Neocortex, you might wanna check it out! neocortex.link

  • @makled
    @makled Місяць тому

    Does this only work on the Editor? Can you get live facial animations on builds with this?

  • @magatoki_ohma
    @magatoki_ohma Місяць тому

    It's wonderful! It is node-based and very easy to understand! Are the languages supported only in English?

    • @sgt3v
      @sgt3v Місяць тому

      At the moment only English, however I am planning to add a language node so that an agent can speak multiple languages selectively.

  • @williamowens6049
    @williamowens6049 Місяць тому

    Awesome! I'm very excited to test this in Unity. I'm also interested in the addition of text-to-speech.

    • @sgt3v
      @sgt3v Місяць тому

      Thank you!

  • @harrisonhough8872
    @harrisonhough8872 Місяць тому

    Super cool! Will there be support for other game engines?

    • @sgt3v
      @sgt3v Місяць тому

      Sure, Unreal engine SDK will be next up!

  • @graphicx1000
    @graphicx1000 Місяць тому

    Wow this is awesome l will test it out

    • @sgt3v
      @sgt3v Місяць тому

      Thank you, please let me know if anything breaks!

    • @graphicx1000
      @graphicx1000 Місяць тому

      Will you integrate text to speech also. Is they a way to make work offline 😁 remember not all of us have access to wifi also think of the end user to you would create way to store a local data base on device which can but updated

    • @sgt3v
      @sgt3v Місяць тому

      Surely will do! At the moment SDK handles only text-to-text and speech-to-speech (along with text), I can add this feature easy.

    • @sgt3v
      @sgt3v Місяць тому

      @@graphicx1000 I have a few offline features in mind, TTS being one of them - where developer can decide on the project size tradeoff.

  • @kritimukherjee2640
    @kritimukherjee2640 Місяць тому

    It's showing page not found of rpm unity sdk

    • @sgt3v
      @sgt3v Місяць тому

      Updated.

  • @Tmoproject
    @Tmoproject 2 місяці тому

    Never knew this. Thanks a lot!

  • @DiamondNinjaPig
    @DiamondNinjaPig 2 місяці тому

    Thank you so so much for this tutorial! Currently making a Unity VR project with AI for university, this series of videos is extraordinarily helpful!

  • @dimasrahmat3949
    @dimasrahmat3949 3 місяці тому

    does it work on android device?

  • @lettuce8463
    @lettuce8463 3 місяці тому

    I wonder if this is possible in AR too

  • @lettuce8463
    @lettuce8463 3 місяці тому

    Please make a full tutorial 🥹

  • @lettuce8463
    @lettuce8463 3 місяці тому

    Is there a way to train the ai to answer question beased on a script

  • @Wonder_Verse_Tech
    @Wonder_Verse_Tech 4 місяці тому

    Error Message: You exceeded your current quota, please check your plan and billing details. even for a new acount, I haven't used it , In very first use how can it say that quota exhausted ?

  • @gamitcrs4926
    @gamitcrs4926 4 місяці тому

    Hi, any advice on how to connect a custom assistant to Unity? What endpoint should it be called?

  • @sedanurylmaz7568
    @sedanurylmaz7568 4 місяці тому

    How can i install openAI package?

  • @sedanurylmaz7568
    @sedanurylmaz7568 4 місяці тому

    What is the openAI Unity package? Why we are using it

  • @vincentschuijt9778
    @vincentschuijt9778 5 місяців тому

    Your solutions are amazing :D However, looks like OpenAI changed it's formatting for the messaging. Not sure if you are planning an update :) in the meantime working on a way around it.

  • @saadaamir4933
    @saadaamir4933 5 місяців тому

    Thanks Man you are Awesome!

  • @ErenBR562
    @ErenBR562 5 місяців тому

    Hello friend, does this work with android/ios? I bought 2 assets from the asset store, unfortunately none of them are compatible.

  • @kirankumar3114
    @kirankumar3114 5 місяців тому

    I have created the .openai folder in my desktop. But the unity is not referring to the folder itself. So, where should I define the path in the C# script?

  • @TasseltipTechnologies
    @TasseltipTechnologies 6 місяців тому

    Good stuff. I wanted to change the basic movement to the joysticks that come along with the third person controller. I did convert the RPM_Test photon avatar to a standard asset. I am still not able to get it to work. Any suggestions ?

    • @TasseltipTechnologies
      @TasseltipTechnologies 6 місяців тому

      I meant third person controller that comes with starter assets.