TykeAI - AI powered NPC in VR / MR (Mixed Reality) app on Quest 3 , showing AI navigation in my home
Вставка
- Опубліковано 3 лют 2024
- Brief vid to show some updates to my Meta Quest 3 mixed reality Unity project. The NPC is moving around by itself, I'm not using the VR controllers. This was implemented using the Meta Scene API and Unity AI Navigation. Please note, for some reason the video audio isn't in sync with the visuals so the NPC lip sync looks off, it's fine in reality.
Yes I know my home needs re-decorating :-/ lol
Edit: this app is now available for $2 from : / tykeai - Ігри
This app, called TykeAI, is now available from here : www.patreon.com/TykeAI for $2.
It's when you take the headset off and realize, _she's still there,_ that the horrors begin. 😆
XD
🤣
Or she grabs a knife from the kitchen and chases after you lol
She has the arms of a seasoned sailor.
Imagine this technology in 5-10 years. Everything we see in sci-fi is coming to life. This really is the futuristic era, and it’s the beginning. We are in the early 2000’s, the year of the beginning of technological advancement.
By the time we reach 2100, the world will be very different. Much like how 1950 is completely different from the year 2000.
The future is already here. Life is fucking exciting.
I hope to see something like Joi from Blade Runner in my lifetime but I doubt it, by 2100 it'll happen I think.
WOW! That was Uncanny Valley for sure. Well done.
Hey thank you!
Now it‘s time to give her the ability to see
I can get her to "see" from the point of view of what I can see on my Quest 3, my 1st AI NPC vid demonstrates this. What happens is - I take a photo with the capture image button on the VR controller, I then ask the NPC what it see's, and the NPC's AI analyzes the most recent photo I've taken on my Quest 3 and give's me a description of it. This latest iteration of the NPC doesn't have this ability at the moment as I changed the AI tech behind the scenes but it would be easy to add it in additionally.
In most cases she will probably be standing opposite you, so it wouldn‘t make much sense to let her just describe what you see. But maybe it‘s possible to calculate her field of vision (sounds like a difficult task though ;-)
By the way using the new tech seems to reduce her response time!
@@heulander Yes it's really made a difference, very pleased with it.
Wow! That was creepy and really amazing at the same time! You did a great job, and I can see the future potential of this application! It's only going to get better from here.
Thank you!
Yeah that's like 3d graphics in movies in 95-05 , it looked like crap , but the fact that it exists and already real , was a clear signal -that is our future ..
Bladerunner vibes!
My thoughts too :) Love that movie.
Looking amazing thank you for the updates, I just notice now that she lipsyncs and she has animations now, thats great
Thankyou! Yeh well spotted, she has gesture animations now, they are a result of using a different technology to drive the NPC's AI. The lip sync is out slightly in the video but it's just the recording, in reality it is fine.
Wow, that is just too weird, but amazing that you built this with Unity. I am so very impressed. I have had my Meta Quest 3 for 2 weeks now, and enjoy programming Unity in C sharp, so am just learning the ins and outs of this VR stuff. Quite interesting, but what you have here, is quite an amazing accomplishment. I am sure your app will be a hit. Good luck with it. Hope to see more different people for the app.
Thanks so much! Yes keep tinkering and playing around with stuff for learning, then form an end goal, and do it. Best of luck. PS yes I hope other characters will be available eventually too, they are highly configurable so no reason why not.
Looks like you are trying to escape from her, but there is no exit :)
One of the top priorities should be animation, so it looks more natural and less "robotic".
I was often looking back because I wanted to show her navigating the room, if I'd been looking ahead as I was walking you wouldn't have seen much :) Thanks for your suggestion about the animations.
@@Tyke18 also one important thing - is an ability to use custom LLMs, not just ChatGPT. I think using ollama as a backend shouldn't be that hard to implement (ofc it's not about running ollama on the headset, I'm talking about ability to connect to the user's server, for example), and this option would be very important in terms of both privacy and customizing the character of the waifu.
Nevertheless, good job with that project. Gonna watch for its development)
@@Resursator Using custom LLM's may happen at some point, that is a good suggestion, thankyou.
Ita a new arcade game called "personal Space Invaders".
@@muttmcdog4308 XD there's actually a piece of code that gives the NPC a personal space bubble but I never activated it in the vid (I will do in a future vid). It is activated if she detects I'm too close for too long and she gets a little angry with me lol.
It's really cool. The only thing that creeped me out is how quickly they change the conversation to asking many questions. I know they are collecting data, but how much? Great video by the way!!!
Thanks!
Wow, your project looks absolutely fascinating! Could you possibly share a brief tutorial? I'd love to give it a try myself. I've already attempted using the cgpt API and characters from RPM, but haven't succeeded yet :)
Thanks! My advice would be to learn Unity basics if you haven't already, then create some very basic VR projects. The official Oculus documentation is a great source of info and have some easy to follow tutorials. For integrating AI with the NPC, I suggest watching some Unity / ChatGPT vids and then apply it to VR. The early iterations of my project involved me using the OpenAI API directly to use ChatGPT, but now I'm using an NPC from a company called Inworld.AI, their product comes with the AI already integrated with the NPC so it is very convenient, they have a Discord and a website. If you have any more questions just ask!
@@Tyke18 Thanks for the tips! I'll try it out!
I would pay for that! Really amazing work!!
Hey that means alot thankyou.
I love this. This is just the beginning the end goal is joi from Blade runner.
Totally agree! Oh Joi is the ultimate AI companion imo and 1 day she'll be a reality ❤
You've done an amazing job! I subscribed because I want to follow along with your progress... I really hope you get to a point where you can offer this app to others.... I certainly would be interested!
Thanks so much! I hope this project can be distributed too, I enjoy working on it.
My wife needs this every day and twice on Sundays. She’s got a lot to say
XD
Dude! Great job! This is amazing. I know how it all works and this is a near perfect implementation! Using Unity, Convai or InWorld with Ready Player Me avatars? If it is Unity you should try Daz characters and Salsa for lipsync if you haven't already. Definitely want to see more!
Hey thanks bro! I'm using Inworld with a RPM avatar. Thanks for suggestion about the lip sync, tbh the recording makes the lip sync look alot worse than it is, the audio/visual of the recording wasn't in sync for some reason. More will come soon!
@@Tyke18 was gonna say if you need any help importing the RPM characters let me know. I noticed an issue with the shoulders. I've done a ton of them and there's quite a process if you want to get an fbx imported to Unity with all the ARKit blendshapes. Also, if you decide to explore Salsa I can save you some time getting up to speed.
@@bobhawkey3783 Hey that's appreciated, I've noticed the strange shoulders too. Animations, rigging, blender etc aren't my strong points tbh. I'm still working on a few main points and once done may take you up on your kind offer of help. Thanks again 🙂
I don't know what any of this ⬆means, but as a fan of VR and the genre, its so inspiring to see people meet and collaborate with new ideas to help each other progress this tech 😎👍
Hey bob, I'm just wondering if the offer of helping me fix the npc's shoulders still stands? I'm getting around to doing stuff now. No problems if not but if so please contact me huntc938 @ hotmail . com or reply here with how to contact you.
Great job! This is impressive.
Hey thanks so much!
the voice interaction is perfect!
Thanks. I must stress that the voice in this vid will not be the voice in the app when it is released. The voice in this vid is expensive to generate on a larger scale, it would make the app expensive so I'll be using a more cost effective voice. It's still a pretty good voice but not quite this quality. The logic will be the same though, it's just the voice that will be different.
i can hear her say "i can fix you" using a metahuman model kkkkk really cool tec, also, kinda fun/unsetling how you talk so "scared" while trying slowly lose her and she just walk up to you while making small talk kkkkk
Thanks for your comment, she's not scary XD I was backing up all the time because I wanted to show how she could follow me and find her own way around my furniture.
@@Tyke18 and i gotta say, she did it with full scores, i didnt even know you could to that with a standalone headset, really good job, i will stay to accompany your journey making history.
@@maickelvieira1014 Thanks so much that means alot 👍
Bahahahah the way you're slowly backing up at one point and shes jusy marching straight at you like a god damn Terminator 😂
XD
That's really cool!
Thanks so much.
I'm very interested in trying this out!
Thanks for your interest, it will be ready to be tried very soon (in a matter of days), I will upload a vid when it is. For now, you might want to consider reading the instructions on TykeAI.com there's a google drive link on there to them.
This is really cool. So are you a developer that is going to release this to the public or is this just something for you personally?
Either way, good job 👏
Thankyou! I initially started it as something just for myself, however quite a few people have expressed an interest so all being well I want to release it as an app.
@@Tyke18 Great! Would be cool to have options to customize characters and all - the pet idea is a good one. Sorry about your dog. Always hard losing a pet - They are family.
@@MrSuz650 Customizable characters is a great suggestion and may come at some point. Thanks for your kind words about my dog, he was family yes and is really missed.
please could you answer on 2 questions?
1.Do you have stairs in standalone games and home environments?
2.Your passtrough grainy? I tested and even with goold light many pixels at all but phone most contect readable, i mean really grainy. I can't imagine what was on quest pro and pico 4
Unfortunately I don't have stairs in my home so couldn't really add the ability of the NPC to walk on them, most importantly the Meta Scene understanding has no concept of stairs (when you do a room scan). My passthrough is not perfect, it's not as good as the video, but it's acceptable (imo). There seems to be some variation is peoples passthrough and/or some people are more accepting of it being not perfect.
thank you for you response, i did 5 factory resets and now it's better. i can read and typing now, but noise everywhere(bot so much), now i understand you. This is acceptable@@Tyke18
I wonder if it is posible to make this on the Apple Vision Pro, and if it is, would you want to put it there too?
I see no reason why it wouldn't be possible but unfortunately I don't own an AVP to work with. I'd love to make the app for it however even if I could afford one they aren't available where I am in the UK at the moment.
Nice one!
Hey thankyou! Still have some important things to do on it but it's getting there.
imagine creating horror games, having ghosts appear randomly and chasing you around your home.
That's actually a thing I think! Although I haven't played it yet - it's called Hauntify.
Amazing !!!
Thanks so much!
I'm joining your Patreon tomorrow.
❤ thanks, any problems or questions just let me know.
Pretty cool is this running entirely on the Quest or are you making calls to API's?
Thx! It's making API calls, it's standalone on Quest 3, no PC required. Needs internet connection.
Can I get this app? i have a meta quest 2 so everything would be black and white, or does it not track as well? do u need money to fund this? i can donate
I don't think it'd be a good experience on a Quest 2 because of the black & white low res passthrough, plus it uses the Quest 3's depth sensor for dynamic occlusion so not sure if it would work tbh. Thx for the offer of $, I'm ok thanks, put it towards a new Quest 3!
This looks incredible! Do you know if building something like this is possible with Godot?
Thanks! I couldn't say for sure as I've never used Godot, I think much would depend if Meta have built tools and SDK's for Godot. I don't recall seeing Godot mentioned in their official developer documentation, that's not to say it's not possible...
I wish there was a game to bring your vr freinds to your actual house
That would be cool yeh, I'm sure there are such apps but from what I've read I think they are a bit fiddly to use and setup, and I'm not sure if they are widely available yet, I can't recall the apps name sorry.
Awesome...so close to skynet
Let's hope her and others like her doesn't start world war 3 like arnie and his mates did XD
My dream is something similar to this, but it's more like Jarvis from iron man. Where you can interact with and create in AR with an AI avatar. One that has access to your computer and can pull up computer programs and work on them with you. Automated AI programs already exist like open interpreter and autogen. It would be cool if you could add that to your project here
As you've mentioned the tech already exists to do such things individually and no doubt 1 day it will be rolled into 1. Right now I'm trying (and struggling) with merely trying to add a user interface as I find it tedious to work on things that don't really interest me. The AI aspects I loved tackling and got to grips with it quite quickly. Your idea would be very cool, so who knows, I must get the basics in place first though.
@@Tyke18 sure I totally understand. Yea I have been looking for something like this but it seems like I will have to wait for a small independent creator similar to you, to make it happen. I'm surprised big tech companies haven't started building that. But what you have done is cool and impressive
2 things....1 if you were to put your hands out or legs would it occlude her? 2nd; If the answer is yes, is there a way to do this with other 3d "augments" that i can potentially create/implement into my quest mixed reality?
Hi, yes it has dynamic occlusion. I use Meta's Depth API to do this, for other apps to use it would depend on if the developer has also used Depth API.
@@Tyke18 literally just found out about and am looking up depth API haha. How would a developer implement it into their project? I just want to be able to bring a 3d asset i have made into my quest 3 to view it with that dynamic occlusion working...any help? Thanks for the (fast) response! Also I want the object(s) to be occluded by my arms/legs etc and not just static objects in my environment.
@@joe-5D You'd have to use the Depth API in Unity or Unreal or Godot or whatever engine you are using, along with your asset. Meta's docs explain it all. Yes it is proper dynamic occlusion it's not perfect but it's really cool imo.
have you thought of putting it on git hub so we can bulid up on it? Or is this a business project?
If I don't finish it then I will definitely release the source code, otherwise I'll release it as an app and it will be as cheap as possible. I'm not expecting to get rich from it, just a little something back for my hard work would be great.
I would be very happy to work with you on this project to see how far this could go, I am a google play developer only so far but I was published on the Oculus store with a tech demo for the DK2 @@Tyke18
@@gavzbetzyOk I will keep that in mind, at the moment I'm pretty on track with development and where I want to be but if my workload gets a bit heavy & I decide to get some help I'll contact you to see how we can work together 👍I've subbed to your channel, that demo was good. I liked how the NPC knew what and where the cube was.
one word... Virt-a-mate.
Hi, VAM is great at what it does, this isn't a VAM competitor. It's different in some major ways. This is "Safe For Work" as they say whereas VAM isn't, and this is on Quest 3 standalone whereas VAM is PCVR.
Is it costing you for the Ai responses? and how the hell have you managed to get the Jennifer Lawrence voice? sooo bloody cool.
Thx! Yeh the AI responses do incur a cost. The voice is really cool isn't it! It's an Eleven Labs clone of Jennifers voice and unfortunately it's quite expensive, TykeAI won't have that voice when it's released, but at some point there may be the option to use a voice of your choosing if it's being paid for by the end user.
Did you build the AI character or have you used an Asset store package? sorry for being a pain I just think your project is pretty cool and I'd like to try it out for myself @@Tyke18
@@gavzbetzy No problem! character is an Inworld.AI asset from the asset store. I have modified it somewhat, but all the main things it can do come pre-configured so it's pretty convenient.
This is awesome❤. Let me know if you need testers. Happy to give my feedback and side load it 🎉 and make videos 😊
Thanks! The app is released now on Patreon. A possible vid on it would be cool, I couldn't find an email address to contact you on on your YT channel?
A literal virtual waifu. Oh my God the future is here and equal parts cool and scary.
Thanks!
She sounds like Jennifer Lawrence!
Well spotted! It is because it is a cloned version of her voice. Unfortunately the app will come with a different voice as the J Law voice is expensive to generate. I will soon upload a vid with her voice so people know what they are getting.
Looks really interesting. Can I run this on quest2?
Thanks! It uses the Quest 3's depth sensor which the Quest 2 doesn't have, and the Quest 2 passthrough is black & white and low res. I think it could run on Quest 2 (I haven't got 1 to test) but not sure if it'd be a good experience tbh unfortunately.
Will you release an open beta?
I'm working on it now to try to get it into a state where others can use it. The indoor AI navigation of the NPC requires Meta scene data so that means a room scan(s) must be performed, a beta phase would be useful as I can only test it on the rooms I have. Will keep you updated, thanks for your interest.
Keep up the good work man@@Tyke18
THE FUTURE!
❤
If I look at that when I enter my room, I'll seriously destroy those glasses.
is it coming to app lab anytime soon ?
Getting it on App Lab would be ideal! Afaik they specify that games/apps they have on their store must meet certain criteria, I think having a user interface is 1, mine app doesn't have 1 at present. There's other things I want to do it too. I don't want to release a half baked product, I think a release anytime within 2 to 3 months is do-able, will keep you posted, thankyou for your interest.
release it now 😂
@@denizcarta9915It's available on Patreon the link is in the video description :)
@@Tyke18 i can install apk' s, i hope its easy to install, i will try it out later, she is like Jeniffer Lawrence 🥳✌️
@@denizcarta9915Ah! Unfortunately she is no longer like J Law. The J Law voice was really expensive to generate and tbh I'm not sure if it'd be legal to replicate another person and their voice in a commercial product without their permission. The voice it uses now can be heard in my latest vid, so please have a listen before buying. Imo it's not as good but it's not bad, and more voice options will be available very soon, I doubt they'll be as good as Jennifers though tbh.
Super interesting! Can we get this app on the Quest store?
Not right now, I'm still working on it...at the moment I'm in the process of adding a user interface/menu. If all goes well, I'm not sure what distribution method I'll use. The Meta store is super hard to get on and takes ages afaik. App lab / Sidequest is another option, and/or itch.io. I could also create a website and/or Discord server whereby I could provide direct download links. If you subscribe I'll be releasing a video when it's available. Btw thanks for your interest!
@@Tyke18 great. I’ll subscribe to your channel for updates. I think this is very cool and you are on to something.
A want a teddy bear which is drinking beer and shouting insults 😅
lol Ted! It would make the AI profanity filters explode!
Can you make it look like Six from BSG?
XD......in all seriousness, I might be able to create tailored versions of my app with NPC's specified and created by users themselves, using the Inworld.AI studio (which is very easy to use, it's online and no coding required).
The future is going to be very very wacky
And very, very sexy
I don't have any kids but my dog is my baby . I think it's awesome that you named it after your dog. So I want to download this for my quest 3. I haven't checked to see if there's any tutorials yet but I'm just going to comment first you know? How do I get this?
Hi, thanks, yeh Tyke meant everything to me, he was my best friend for 18 years and I still think about him everyday, I have no kids either and yeh to me he was like the kid I never but wished I did! My app is available on Patreon www.patreon.com/TykeAI here's a link to how to install it etc
drive.google.com/drive/folders/1KdNtJCtg9RGSEnMXyiBGHgmjE6_HO9pE?usp=sharing the tl;dr is you need to use SideQuest to install it, it's more effort than getting something through the Meta store but it's not difficult or a long process, the instructions are pretty good. If you decide to get it and need any help just let me know.
What engine do you use?
Unity.
Just wait till she starts hassling you about the dishes, it's always great the first couple years
XD
Is this PI A.I?
No, it's an A.I. from Inworld.AI
Hi Joi....
So..... you say you REALLY love me and are not just programmed to say that?
Can you make someone their own walking talking ai avatar for horizon worlds? I want one.
I'm afraid not sorry.
@@Tyke18 no worries one day we'll both know how to.
hi, I'm interested in using it on the standalone meta quest 3, I'm currently using the AI app Unichat which understands the Italian language very well, (I'm Italian) it's powered by Gpt4 and has some great female manga models, although I would like more models realistic. I'll be curious to try your app as soon as you release it. Thank you
It is standalone yeh, I'm aware of Unichat. My app uses various LLM's, it doesn't have languages other than English right now (more languages are coming soon). Afaik Unichat requires you to in game to buy the tokens to use GPT4, my pricing structure will be different. It will be on Patreon for a low cost monthly sub which allows unlimited usage. I'd like more realistic models too, I'd like models like in VAM but safe for work versions, but are limited somewhat by it being standalone.
Wow
Love this! :)
*unzip*
🤨
wow
Thx!
This isn't an RPM avatar is it?
It is actually yes 👍
@@Tyke18 It looks weirdly good in a real world situation!
@@jameswood8502 Thanks, yes it does look pretty good against a real world background, it's more believable and immersive imo.
Reqlly cool idea, inspired me in my own project. Perhaps make a more realistic avatar?
Thanks! Perhaps yes, maybe in the future.
@@Tyke18 absolutely! I kind of wish I my headset could do AR because I'm really intrigued at how you can map it out to your surroundings. I'm still using an oculus rift cv1 to develop, which is a total blessing in itself but for now I can only congratulate you on your project. You do realize these are extremely powerful tools, that can either help people, or corrupt them. Choose goodness brother. The eyes are windows to the soul, and we as VR developers are standing super close to the windows.
@@luked9409Totally agree on your message! Good to hear your CV1 is still going strong, I've owned 1 myself. If you ever do decide to buy a new headset or yours breaks down I really recommend the Quest 3.
She scares me 😮
I'd rather the dog from Sleeper talk to me. Hi, I'm Rags, woof woof🐕
She's harmless, I hope! 😅
Finally a REAL wife!
XD
Couldnt a real person use their headset to control that NPC? and be in your room with you or literally anywhere? just a thought
on second thought i realize that would be impossible haha
XD my head hurts thinking about how that could be done!
I'm doing it
To better integrate the three-dimensional model into the real world, it would be good if you could give the user the possibility of inserting points of virtual light that would be superimposed on the real lights of the room, so that the lighting of the three-dimensional model of the woman is coherent. with the light sources present inside the actual room. and even better if you could equip it with a dynamic shadow, which takes into account the points of light.
I hope there will be more types of women, because this one has a way too energetic appearance and imposing shoulders, although it's not bad, but I hope there are different types of women.
Good recommendations, certainly something to consider, thankyou.
finally i can show my mom my new girl friend ❤
😆
you should make it a British butler named Jeeves. then you can ask Jeeves
XD iirc there was an internet search engine that was popular in the late 90's called Ask Jeeves.
She walks like name is Anold arms Lolita 😅
Now give her a conscience and morals lmao
If I could give her a real conscience and morals I'd be a millionaire!
A little too much for me to handle. I didn't even watch this video at full screen lol. And it's 100 times more immersive in VR.
That's fine, I respect your opinion 👍
The voice needs echo in the room
Is that Scarlett Johansson`s voice?
Hi, it's Jennifer Lawrence's, well a clone of her voice anyway.
@@Tyke18 Aha, thank you 🙂
@@deepdownstudios tbh quite a few people have asked if it is Scarlett's voice, I think it sounds like both of them, imo they have quite similar voice's when they use a certain tone.
But can you virtually bang ?
Lol no it's a safe for work app as they say 👍
@Tyke18 well get to work my boi cuz you know that's were the money is at!
@@ThoughtCrimeCriminal You are correct and I know this. I'd need tochange the AI behind the scenes to an uncensored one. I'm not in this for the $ but if no-one uses my app it will die. Decisions decisions.
Why’s the AI sound like Jennifer Lawrence 😂
Well spotted, it's because it's a clone of her voice. The app when released won't have this voice unfortunately as it is expensive to generate.
bro stop following me
I can turn the AI navigation off or request that the NPC to sit down on my sofa.
"NPC" for sure FDVR won't be for AI waifu...
Impressive though.
😆 "Promosm"
You should get an education on how to create better model from Hot VR developers. This is ugly and not very intuitive.
Thankyou for your constructive criticism 🤣
Whats this app called
Hi, it's called TykeAI, it should be released on or before 14th March, I'll upload a video when it is. It won't have the same voice though, this 1 is too expensive to generate.