The Face & Eye Tracking Rabbit Hole [VRChat]
Вставка
- Опубліковано 12 чер 2024
- I dove into the world of Face and Eye Tracking in VRChat! What seems like only a niche technology, hides a fascinating peek at the future of online communication. I interviewed leaders in the community, who spend their time shaping open-source hardware and software to bring this extra bit of realism to our interactions.
Checkout SideQuest’s physics based social VR app BANTER for a chance to win a FREE Quest 3: sidequestvr.com/app/10831/ban...
Join my Discord!:
/ discord
VRCFT: / discord
EchoTheNeko: / echotheneko
EyeTrackVR: / discord
PHIA's Socials
Twitter: / phia_bunny
Twitch: / phiabunny
Instagram: / phiabunny
TikTok: / phiabunny
Patreon: / phia
Voicemod Code "PHIABUNNY" 5% off:
www.voicemod.net/voicemod-par...
=TIMESTAMPS=
0:00 - Intro
1:35 - The Face Track Conversation
4:45 - Sponsor: SideQuest/Banter
5:31 - VRCFT
7:58 - Avatars
9:45 - EyeTrackVR
12:08 - PRIVACY!
Join the TVRS VRChat Group:
vrc.group/TVRS.0429
TVRS Official Twitter: / tvrs_official
TVRS SHORTS!
/ @tvrsshorts
PHIA Second Channel:
/ @phia_bunny
TVRS Studio Channel:
/ @tvrsstudio
Editing help by Moonwatcher & Thrill
/ moonwatcher68
Business Contact: phia@thevirtualreality.show - Розваги
Make sure to checkout SideQuest's Discord for Banter to get info on that Quest 3 giveaway :)
discord.gg/bantaverse
I’ve been a Banter user (ARK1701 in VRChat and Banter) for about four months. Banter is event driven and a very friendly community.
discrod lol
Hi Phia I contacted you a while back via Email about a potential research into VR etc. I have not heard back, maybe went to your spam folder. Let me know - you may find it very interesting.
Yeah I don’t want those headsets that fail and overheat.
Can i have some trackers so i can make a video on them
f a c e t r a c k i n g 👁️👄👁️
Cidil
trilseker
facetracking, but your eyes are covered by a magic brick that sends to you to cyberland
Psst, check out project babble to do diy lip tracking. Goes great with eyetrackVR
BABBLE BABBLE BABBLE
@@summerbreeze2167Where my headset mr programmer socks?
"If they don't do it, we will. That's the beauty of the open source community." Truer words have never been spoken.
Thank you so much for featuring EyeTrackVR! The future of facial tracking deserves to be open, and I'm incredibly honoured to be a part of such a talented community working towards it!
Ello!
@@RidgeXR_2hey ridge, lol
@@prohurtz *rigde
Thrill + PHIA is the ultimate VR power couple. They both cover very different but very significant sides of the industry and work off of each other really well.
Wishing you well with surgery, take as much time as need to recover ❤
We really need more headsets to be released with face and eye tracking, or at least eye tracking. It's annoying that headsets being released still don't all have eye tracking.
Sure, we can do it open source by ourselves, at least for eye tracking, but we shouldn't *need* to do it ourselves.
Looking at you, Meta...
EEE THANK YOU SO MUCH FOR INVITING ME TO BE A PART OF THIS!! Im really passionate about face tracking and I’m so happy a whole new audience is being exposed to it!
imagine you were still mute
yooo Echo :D
Are you currently taking any new commissions?
Being able to look someone in the eye while having a conversation is why I always thought VR is the first true social media application. Everything else feels para or anti social in comparison. Great vid phia !
I love the face track conversation having the virtual and actual perspectives, you've truly outdone yourself with this video!
Honestly, it is just mind boggling how much VR is being advanced these years, makes you wonder how far we will advance in just 1 or 2 years from now.
Disabled VRChatter here! I'm blind in an eye and one day I'll get my hands on face tracking and folks will be able to TELL because of how it just-- kinda follows my good eye but doesn't move on its own. It's hilarious.
ALSO Echo's face tracking is FANTASTIC. She's worked so hard on all the tracking and has spent YEARS in Blender [someone please bring my daughter back from the Blender wars]. I tell anyone who wants face tracking to hit her up because of how EXPRESSIVE it is.
So does that mean you have similar eye movement to actor Tiny Lister aka Zeus in No Holds Barred/WWF aka Deebo in the Friday movies? He was also blind in one eye, so he had what you described in his eye movements as well.
Wait could I hit her up? If you don't mind giving out that info. I'd love to learn from her!
Thank you for making this video. the VRCFT community has inspired me so much and is so special to me. Huge thanks to Ben, Jerry, Azmidi and all the module and avatar developers who made it happen. Happy to see my friends Hash and Kombi featured here, they are awesome!
Maybe you guys should do a VR chat podcast
Seeing someone with eye tracking look at you in VRChat hits different, especially if they are flirting with you.
As long as my QPro doesn't make my eyebrows start doing jumping jacks for an hour straight just because I raised one slightly
HOW DID I JUST FIND OUT U AND THRILL ARE TOGETHER?!?!
That's not Thrill?
@@NixonAxi it is..
Yeah is that really him?? Like i recognize the voice but i find it so wild???
@@NixonAxi Yeah it is.
I love face tracking, hope to get it eventually. it's so neat to see how completely transforming interactions can be when seeing peoples expressions.
hard to go without it once you got Facetracking! Just the subtle amount of movements you do in a conversation is crazy.
This is wild. That EyeTrackVR tech really opens doors to majority of us without native tracking headsets. I might explore this.
I'd say you guy's relationship are goals for any VR community member! I like these videos that feature the both of you!
EDIT - You just made countless thousands of people want face and eye tracking!
Wishing you well in your health and please take the time you need for yourself!
Love your videos.
I love seeing you both in a video together! My wife and I met in second life almost 20 years ago and we love being together all the time in real and VR! ❤
Haven't been on your channel in three or four years...I'm so glad I found it again! Amazing how much improvement in technology has come in the past few years!
Epic video. I've been loving Face Tracking on the Quest Pro. For my purposes, I only wish the microphone was far better for Content Creation and Streaming on that headset, to be honest.
It's also been fun learning about how it all works via OSC + VRCFT Software and more.
Using it with my own Karin and Rusk avatars that I've spent 100s of hours on in Unity, to perfect to my likings, has been really neat too.
Also I hope your surgery goes well and that you make a swift recovery. And once you're back to being able to make videos, I wish for you to be able to easily keep up the good work on the entertaining and informative content you produce. Take care. :)
Most informative. I'm Trying to decide between quest pro and quest 3. I want to game but I also want to be very expressive. I'm still trying to find something somewhere in between or choose one or the other.
The face tracking is SO impressive, and so was the way this video was edited. I was hooked 🙌
really glad to have been able to provide some footage. wasn't expecting my memes avis to be in there (I skibidied)
i ordered a quest pro a few days ago and have been so excited looking for new videos on it and HERE YOU GO 8 HRS AGO!
wooooo!
13:17 good to see vyde from New Eden making an appearance
I have struggled with applying facetracking on my VTuber Avatar myself, but in the end i am really happy, that i have accomplished it :3
What a great video! We always learn so much from you PHIA! Thank you.
1:26 "...pulled my partner Thrill[seeker] into a..." so it's official? 😉
Exactly what I was thinking! Always had my suspicions but didn't want to jump to any conclusions or anything
They’ve been out in the open about their relationship for a while now
yes, they even mention the fact they live together and see each other every day
XR and Social VR is the next phase in the evolution of telecommunications. Face tracking is another important part of that evolution.
We’re the early adopters, but in another ten years, I believe Social VR (VRChat, Banter, RecRoom, the Zuckerverse, etc) will be main stream.
I do think it’s super weird as much as an evolutionary step forward as the Apple Personas are, they don’t have their own Social VR platform, just FaceTime. Why take a step forward with one “foot”, only to make a diagonal step backwards with the other “foot”? 🤷🏻♂️
Fact that this channel didn't go viral yet, is for me simply mind-blowing
My favourite thing to do when I music mirror dwell, is to gesture sync to the song, basically acting along and syncing my face gestures to the song, acting asif my avatar is singing it in a play or something. It's amazing and fun to see others that KNOW their gestures and actorly naturally use them when talking or acting; I love doing it so people can really see how I'm reacting, and feel better immersed when talking to me ^-^
My main avatars are not picked by how they look and toggles, but the most thing I look for a diverse gestures with a wide range of emotions. ones with diffrent gestures on each hand, or ones that have perfect mixed gestures.
I always do different gestures per hand on my avatars, but am soon going to be adding a 3rd set for double gestures! :D I cant wait to use it ^-^ (and of course adding a toggle so people can turn off 2 of the gesture sets for when they just wanna chill or play games hehe.
i cant afford to use any vr stuff since im poor. but i like watching your vids and keeping up with the tech evolving and what people are doing with it. i hope by the time i can afford to get into vr, things like eye tracking will already be standard and accessible!
I really want a new headset to come out that has the same speakers + microphone quality as the index, But with the option to go wireless, better resolution, FOV and face + eye tracking. And of course it still needs the option for base station tracking as well, so I can use base-station tracked full body trackers.
Hope a speedy recovery
This is a pretty interesting aspect, I recently came across your channel recently and now I'm going to subscribe because I enjoy the editing and the ways your manage to get your content across. While I'm not too involved with this as all my characters are... knights/mass effect aliens/sci fi soldiers. This is good to stay in the know.
as a kid who got into vtubing first, then VRC, imagine the goldmine this was lol
I was surprised to learn that hey, the rigging knowledge I might have in 2D rigs might be useful in 3D rigs!
and the way that people make a living off it too!! I didn’t know it was like that in the 3D community- I thought it was limited to 2D models haha. I’m excited to see where all of this goes in a few years and maybe even get into it myself :0
Wow!! All the guest stars made this possibly the best video yet. I love the virtual reality show! Get feeling better, surgery is the worst 😭
Eye tracking is a fascinating technology. When i get a new headset I will certainly try to get eye tracking with it. Also i hope your surgery went well
I'm pretty excited on where VR is heading with tech like this, though I share the very based concerns about corporate abuse (IE Excessive advertising and data collection).
That said, I'm glad i'm recognizing some of the folks featured, makes me feel connected ^_^
finaly a new video. :)))
I love the long form content! This was so fascinating and I am hellbent on trying face and eye tracking out now!
Hope you have a good recovery from your surgery! ❤
To fellow commenters, where do I get a really solid option for a custom avatar?! I have been to so many avatar worlds for about two years of searching and I am struggling to find my fit. I think I need to have one made just for me.
Jerry's face tracking is amazing! Please check them out on gum road. Love my quest pro face and eye tracking. Only had it a week, and I do not think I could go back.
It's strange how many times I caught myself responding empathetically to you due to the micromovements you're eye-tracking was picking up.
Another big opensource project in this field is "Project Babble"
It's being worked on by a small but passionate development team, working on making it so that if you can get a camera onto your headset, you can use an opensource AI model made by the community to get face tracking up and running. Right now, it's somewhat picky, but they're working on getting more and more data, and using different techniques to train models. They've also been getting more and more people in to help render out new faces, and they recently announced that they'll even need to get help from the community just to train the next model, not even just making the images
Glad facetracking is getting more awareness, PHIA your tracking looks so good. Can I ask your set up?
Take as long as you need to recover. Love your vids
Sidenote but I absolutely love the thumbnail! Really great composition and colours.
so this is cool 1 minute ago it got posted thats cool that i got here this quick :)
FacetrackVR is so cool! I made slimes this winter! Now I can build eye tracking too!
it's great to see the community grow like this, face/eye tracking allows so much
Thank you so much for the video. Im super interested in what VR headset tracking is recommended by you and others or if DIY is still the way to go. Hope your surgery went well, best wishes
I think you've had that surgery by now... Wishing you the best on your recovery! Hoping everything goes as smooth as possible. Health stuff getting in the way is definitely frustrating.
Thanks for the video. Very informative and fun, as always!
As someone with the jankiest headset ever, the pimax crystal, setting up eye tracking and face tracking took me so deep down every reddit thread and discord server. After five third party programs and a lot of lost brain cells, I finally figured it out tho and its so cool :D
I'm planning on getting a meta pro soon and i cant wait! i have been playing vrc for 5+ years and i got fbt last year, i think I'm finally ready for the next step
I got mine along with a lot of other people for free at RDC, and while I never would've purchased it on my own its a fantastic device. My only issue with it is that because I'm on university wifi, it's on a separate connection so it HAS to be wired in order to access pcvr
I love the expressiveness Face and eye tracking give me with the Quest Pro. It's basically the only reason to visit Horizon. I had it set up in VRChat a few times but I need to relearn it and find or buy a good avatar.
Hope you have a successful & speedy recovery on your surgery 💖
I wish you best of success with your surgery!
As someone who has never had enough time to use vrchat often enough to memorize which hand gestures made which faces, face and eye tracking is a godsend. I can finally express myself in vrchat with literally no effort, as opposed to having to memorize hand gestures. Also, having started with vive wands never helped, since the circle trackpad was split into 5 sections for hand gestures. Because of the split like that, it's always been very un-intuitive and hard to memorize. It wasn't until 2 years ago when i upgraded to index controllers that i started to be able to more easily memorize hand gestures finally.
This is a great video. I had a Quest Pro for a bit for the eye tracking, but other factors had me go back to my Index, and Quest 3, but I really do miss the eye and face tracking. I have been interested in the DIY eye tracking, but I keep putting it off
Ever since I started using face tracking in vrchat with my quest pro, I can't live without it. I have a quest 3 that sits on my shelf because face tracking is just that important to me. People are always blown away. And people comment that they can really tell that I'm listening to them when they talk to me. It's very important.
It also keeps people from talking over me with the voice latency because they see my face expression forecast that I'm about to talk. It's great
Hope your surgery went well, keep being awesome~
i have been playing vrchat with face and eye tracking with my vive headset since 2021 and that has made it really difficult to adapt and move to another headset if I'm playing VRChat. it makes a hell of a difference to my experience in the game.
YOUR EYES. So expressive!! 🤩
6:47 That second of rage 🤣
hope the surgery went well!!
Face tracking is daunting, but once the technology is more available, I think it'll become commonplace. In his review of the AVP, Zuck said he's planning on bringing eye tracking back for future Meta headsets, and I expect that to influence the industry. I can't wait to get that extra layer of immersion myself!
I love my face and eye tracking no way could I ever go back to a headset without it everyone finds it so cool when they see it.
amazing video, absolutely loving it, I just wish that foveated rendering would have been talked about more. I think it is an amazing way to enhance vr performance with literally no downgrade in visual fidelity
I'm really excited to see where this goes; while I really want facial expression tracking I wouldn't really be able to afford it currently - and hoping that headsets eventually all upgrade to track expression. I can only imagine that soon there will be programs run to block out tracking/streaming data to the companies themselves, but I am hopeful.
future president of vrchat, you do us proud
I'm still using a rift. Best headset ever made period! They were built likes tanks, never broke as often as modern headsets and are hella cheap now. I just got my 5th one back in January for $60 and it had only been used twice in it's life.
as someone that motions a bit with their hands while talking the hand gestures in VRchat make it really hard to convey the proper emotions, being on the fence about jumping into the VR enthusiast rabbit hole I hope a gaming focused standalone headset comes along with native face/eye tracking.
I've always wanted more tracking available in VRChat. Initially, I thought VRC eye/facetracking seemed off. When anime characters are making these intricate and almost non-exaggerated expressions, it seemed to fall into an uncanny valley of sorts. It seems like it's a matter of how the original avatar was styled as, since IMO it looks a lot cleaner on you than Thrill.
I think that's why a lot of the furry avatars seem better for it.
I am fairly sure the Quest 4 will have facetracking because of quest pro's appeal for facetracking and the apple vision pro's face tracking, so I'm waiting for the Quest 4 to see if they have face tracking with a nice headset at an affordable price!
HTC has been really good at Discontinuing their useful products, which is why I left them for the Valve Index a long time ago and never looked back. At least I can mod a Valve Index somewhat to support face tracking.
I have gotten so good at puppeting my models with hand gestures and locking them on and off, people have thought I had face tracking multiple times now.
3:18 exactly just got facetracking 4 days ago it changed everything never going back its just too good!
Thanks for this great and nuanced video! I find the last part about larger corporations exploiting this information to be so interesting and important.
ALSO hope the surgery went well and wishing you a speedy recovery Phia!!!
I think it would be interesting to use the eye tracking to see if your eyes are open, so it doesn't have to spend a bunch of computing power rendering the world while you're sleeping, but you can still see it when you wake up and open your eyes.
I hope you get well soon... 2 months belated
For a while I had EyeTrackVR working on my valve index. To find out it was annoying. I use the portable mirror a lot for phantom touch/sense. And every time I peek at the mirror, the person(s) near me got confused where I am lookin at, even looking over their own shoulder to see if maybe someone else sneaked up next to them. But I was just checking the mirror for touch feelings, spacemove positioning (clipping in chairs) and if eyetrackvr wasn't glitching out and needed recalibration. In the end, I only found it usefull for blinking detection, and not for eyemovements itself. I just prefer to have phantom touch over eye movements, without other people getting confused.
I hope your surgery went well!
We're getting there! Can't wait for a future where face and eye tracking is available in most mainstream headsets
I personally bought a quest pro purely for the added built in Face and Eye (and now tongue yippee) tracking. It just adds an extra level of expression that feels way more natural to me than using controllers to determine how my avatar will look
Star citizen has had this tech for several years, it really is a game changer, it has head, face and eye tracking, and it will also get vr according to the guy dealing with the vulkan integration for it in his words "even if I have to code it at home in my own time, it will have vr" lol when you might ask? who can say they are not known for rushing things lol. but the eye, head and face tracking and proximity voip are all in already.
If you want to see it in all its glory check out any of the multitudes of Star citizen machinima that exists because of it. :)
I have FBT and face tracking and alot of my friends do also. It absolutely gives a since of presence. I exclusively look for face tracked avatars now. Being able to smile and make eye contact is huge. I have a quest pro and with a few mods is an amazing and comfortable head set. Pitty they droped it.
Wishing you a speedy recovery
.WingZero.
and once you do express yourself out of gestures, you tend to forget to use it on non facetrack avatar ^^
I'm deep deep into this rabbit-hole and I like it.
the skibidi toilet at the beginning caught me off guard 😭 i can't escape it
I’m glad you mentioned the need for open-source technologies. It’s very important to not let companies rule this realm too.
I bought a Quest pro (used for cheap) last christmas, and one thing that most people don't talk about is the availability of face + eye tracked avatars. I found 2 worlds with public face + eye tracked avatars, and about 80% of the avatars were furries or just memes.
I ended up buying a rigged one instead.
nyaw, hope u get well soon phia
GOOD LUCK PHIA!
I recall seeing a listing for plug and play HTC Vive eye trackers... was 1400 euros back in 2020. 1400 for two plastic clips that hold a single camera pointed at the eye, and patched into the HTC Vive Camera... the price for convince will remain high so long as most people want the simple solution that doesn't require soldering or setup, and if it isn't natively supported in the application, they likely will not blame the +$1k addon. Didn't know they had a solder solution, and will be looking forward to that, as last I checked there were no good solutions :D
I just got a Quest 2 and was trying VRChat out and trying to learn the avatar facial expressions was like trying to play guitar or something.
I am pretty surprised that this didn't cover babble, and a little sad. Hopefully it will in the future, we have a pretty great community working on it
I don't think a large company will invest any more money in face/eye tracking.
meta only made one attempt with it.
Last year I asked myself how I should improve my Vtubing. The meta quest pro was on the short list for a long time. But then I decided on an iPhone 14.
Why is easily explained because I can continue to use my vriod model without any additional costs.
The funny thing is, I'm still faced with the same question: where can it be better and more current? Is the question a quest 3 or a quest pro or wait for appel or a meta pro 2
So with many thanks for this video, it doesn't make the decision any easier, but it shows me alternatives, thanks for that❤