If your goal is telepresence, consider the standards set by VR and video games for input lag, response and motion sickness. I'll be impressed if you can navigate all the bottlenecks w/ hobby servos. High speed FPV drones require very fast input/response. Also, there's some great research on inducing directional effects (yaw/tilt/etc) in the brain via inner ear through soundwaves from headphones - could fit a telepresence build
This this this this! Also use a pair of 360 camera and project it on your view in unity! You don't need to rotate on yaw at all! One ball camera is all you need!
you can compensate for latency by sending the robot's pose along with the video and use it to position the video-showing plane in unity. that way if you - for example - look left, the image in your headset will immediately move right, bypassing the latency of sending commands to the robot and the video stream. it's called asynchronous reprojection and it's not perfect, but the imperfections like distortion and black sides only show when the latency is terrible AND you're turning fast. also the non-linear linkage problem can be solved using calibration by computer vision: track the angle to a point in space (eg a checkboard corner) for all actuator angles and then use the inverse. the benefit is that it compensates for 'as-built' inaccuracies as well as the non-linear motion itself.
I came here to mention the first thing, but it now occurs to me that that would defeat his plan of making it stereoscopic. Once the second camera is in, the geometric distortion of two views, each at a fixed point in space, suddenly rotating around in a reprojection screen would be nauseating. That said, I recall that some VR games already have a correction algorithm or technique for this, so it still bares considering. I do remember once having to stream from the camera to mpeg via flask, so that IS possible, using ffmpeg, IIRC. My application was for machine learning and object identification, but it should cross over just fine.
@@Corbald afaik stereo asynchronous reprojection works just as well, all the quests use it all the time even when streaming over wifi (ie when there aren't depth buffers available to combat distortion)
@@Corbald dual 64mm spaced 360 cameras fixes this entirely, you don't need to move the "head" of the robot at all, it's just a "camera ball on the skybox". I've done this in unity already for another project and it works great. You use equirectangular over-under video and bob's yer uncle. It even automatically gives you the "environment scale" of the robot itself, like you were shrunk with a shrinking ray. It's very cool. You just move your head and it changes the view for you in vr, nothing on the robot itself moves for vision. Solves that problem completely. Then you just use tank locomotion with the controller.
I really want to get one of those bi-directional ultrasonic BMIs. Sometimes my hands aren't as accurate as I would like them to be. I got the feeling that adding brain correction would make interfacing like butter.
mechatronics engineer here, currently designing a robotic tank but want to implement some VR capability as I recently got an oculus quest 2 as a gift. this video has been very helpful, i originally was going to use a esp32 cam but there is too much latency. this video has given me a nice structure. awesome video, you are a legend
So glad to see you back to actively creating! Your designs have been such a great backbone for so many, and it is great to see you breaking new ground! I do wish that you would revisit some of your older stuff though and update and improve those. For example your mouth mech was such a great starting point, but it feels unfinished. Anyway, keep up the great work!
I've been wanting to try and tackle this myself for a long time. But I also want to control robot arms with the Quest controllers. I have a D&D group that I Zoom into every week and I'd love to have a mini-me there that actually looked around and rolled dice there, instead of just being a static camera.
HA! That would be great. Eventually we'd just get to the point in that Bruce Willis movie "Surrogates". But silly question, do not the US Walmarts (I'm assuming You're in the US) have remote shopping? They do here (I'm in Canada) For a couple bucks you shop online and then they bring it out to the car. No going in at all.
@@Runescope Sure they do, but... robot! Seriously, I'm not sold on the whole remote shopping idea. Besides, teleoperated avatars with physical capabilities represents a whole market for disabled people. Something like I described would serve a lot of people in a lot of ways, not just shopping.
@@Corbald When it first started up, I used the "Delivery to door" service all the time and it was great. But I found it was more convenient to schedule a pickup time and just let them grab it all. I've been using it for almost four years now and no complaints. But yes, robots would be supremely cool and offer so much for people with disabilities, and if I could wander through a store remotely shopping, I would do it in an instant.
I just got in universal joints for my new head design two weeks ago. 😀 It's good to see we are on similar paths. I will have to try printing some as well. Your project is looking great! Can't wait to see more.
This is great stuff! I've been thinking about this topic for years but could never muster the effort to do anything, lol. I was thinking you could scale it up to the size of a person, which would be cool.
I love those funny moves the robot makes @6:56. Combined with the face like plate this gives the robot a fun personality. Now I need to make one too. Very useful to start working from home. That still scares me so it would be good to have a robot like this at home so I can control it from the office 🤣
Wow! Well done... Do you have a team of people working on this project with you? It is a huge project! Side note: It seems like the robot is quite wobbly. The camera's stability is a crucial factor. The user might experience motion sickness due to it. If you find it a real issue, maybe you can add some mass to it or smooth the motors outputs. You may also adjust the tolerances in the joints in case there are backlashes. Will adding friction in the joints help reduce the wobble? Since you 3d print those parts, you can orient them on the building plate so that the layer lines will create such friction.
Just got a chance to watch the video, this will definitely be a fun project. I'll try to download the files this weekend. Doing the telapresence with a robot would be a lot of fun, especially if you could do it with a human sized robot. Would be a hoot to "walk" around conventions aa the robot.
I was just thinking about what it would take to DIY one of these, so I can check up on the cat when away. I would instead use a 360 cam and project that into VR so you wouldn't need any moving parts to look around or worry about nauseating latency when moving your head - all the head movement would be handled by the VR app so although the stream would be delayed, head movements are immediate enough to not cause motion sickness.
Thanks for the tip about picamera2, I also worked on a Raspberry Pi Zero project with a camera module and was having the hardest time getting the stupid thing to work!
would love to see a gorilla tag style bot where you can climb around even your own home courses and interact with stuff, i just think it would be hilarious to be on my pc and have my friend pop in on a little gorilla bot and ask whats going on or somethinf
My setup for transmitting FPV from pi zero to a screen is kinda horrifyingly janky. But it works. I have the feed show on screen and then transmit the whole screen through VNC. So. Far too much lag for telepresence but eh, i like it. The whole project is a mess, so a bit of mess on the usage side is just on brand. Also it works and was easy to set up.
Love this idea. Looking at the math issue from my games background, I'd suggest avoiding angular (euler) representations as much as possible. If you can, stick to quaternions and vectors - only converting to angles at the last moment if absolutely necessary.
IMO you should use a 360 degree camera and crop the image based on the rotational values of the headset and then use the servos/motors for the translation part of 6DoF
just had an interesting discovery, look at stereo pi v2 board built on the raspberry pi, designed for stereoscopic vision, perhaps you have already seen this but I thought worth sharing it. ill be definitely buying it
Hey, I am working on the same type of project...also streaming from Unity to Flask! I never got the real time video stream right...tried web rtc in the past but I guess there should be better options! Super curious on seeing your sollution! 😊
I haven’t even started the video but have you taken a look at the headsets Remote Controlled airplanes people use??! They turn their head and the small camera mounted inside the RC plane does aswell, I don’t know how you would be able to combine all of these technologies together but there’s that.(Nevermind you did looked at them HA)
As soon as I seen that in the back It reminded of gundam stuff!! pls take some inspiration there!! Also, have you thought on getting one of those flat cameras on the eyes with also the same treatment of resin and UV, if even that could work then you could do that next step of bifocal! I'm no expert or even know how to do stuff, this just crossed my mind while watching your videos, and I know I'm asking you to sacrifice one of those flat cameras and I'm sorry for that, but if it works you only have to do it once more!
Wouldn't the latency be nauseating? Do you need to tie the camera to drive the head movement or could you use a wide angle lens and do the movement in unity(like how 180degree videos work), you could just have a fake head to indicate direction.
Great video, I am actually creating a similar thing currently. I was focusing more on the software as i find it the hardest part. Also went for the jpg updates every frame at first, since chatgpt recommended it. Unity is very bad at this, I actually created a webxr experience and made it asynchronously download the frames which was almost perfect. It still isn't perfect, I am currently working on sending the video stream theough webrtc, which should be the fastest way. Also my goal is that the quest can just connect to the robot through the web browser instead of downloading an apk, which is much more user friendly. My solution also has two cameras streaming to produce a truly stereoscopic vr view. I won a worldwide engineering competition with the demo I am talking about. We should connect and help eachother.
Hi Will, I've solved low latency video to Unity from the Raspberry Pi using multiple cameras. I built a robot to rescue a spanner from under my workshop, long story and video coming soon, but I wrote a Unity based application for the Steam Deck that controlled it all with no noticeable latency. I've also a cargo robot, Johnny Five inspired robot, and a bunch of other things that may be relevant. Give me a shout if I can help with any of it as I'd be happy to share my mistakes to give you a headstart! Pun entirely intended...
Just use analog fpv. Rids you completely of staying within brand ecosystems like walksnail, dji, etc. there are even okayish AIO analog cam modules for tinywhoops
I think you are making a vomit inducing machine, because the mechanical systems you are using have backslash and will inherently have a little latency and mismatch being physical and all, I'd much rather work on a solid state device, that can be bolted on the surface you are working on (or robot) that uses a wide angle lens, like an insta 360, but you either stream the whole picture or somehow (might be impossible) request a certain angle frame from the device, that way you get rid of all the previous problems, but induce other like lower definition and maybe bandwidth problems, who knows, I wish I could work on this. I too am a fan of Alastair Reynolds and his remote presence concept.
@@OpreanMircea I can't say I have, but I'd imagine that they're just tilted so that the overlap cuts out the other camera. Seems like a harder issue to solve for 360 without creating a big old phantom blob if you look the wrong way
"What is my purpose?"
"Butter, your purpose is to pass butter"
Oh, my, god
If your goal is telepresence, consider the standards set by VR and video games for input lag, response and motion sickness. I'll be impressed if you can navigate all the bottlenecks w/ hobby servos. High speed FPV drones require very fast input/response. Also, there's some great research on inducing directional effects (yaw/tilt/etc) in the brain via inner ear through soundwaves from headphones - could fit a telepresence build
This this this this!
Also use a pair of 360 camera and project it on your view in unity! You don't need to rotate on yaw at all! One ball camera is all you need!
you can compensate for latency by sending the robot's pose along with the video and use it to position the video-showing plane in unity. that way if you - for example - look left, the image in your headset will immediately move right, bypassing the latency of sending commands to the robot and the video stream.
it's called asynchronous reprojection and it's not perfect, but the imperfections like distortion and black sides only show when the latency is terrible AND you're turning fast.
also the non-linear linkage problem can be solved using calibration by computer vision: track the angle to a point in space (eg a checkboard corner) for all actuator angles and then use the inverse. the benefit is that it compensates for 'as-built' inaccuracies as well as the non-linear motion itself.
I came here to mention the first thing, but it now occurs to me that that would defeat his plan of making it stereoscopic. Once the second camera is in, the geometric distortion of two views, each at a fixed point in space, suddenly rotating around in a reprojection screen would be nauseating. That said, I recall that some VR games already have a correction algorithm or technique for this, so it still bares considering.
I do remember once having to stream from the camera to mpeg via flask, so that IS possible, using ffmpeg, IIRC. My application was for machine learning and object identification, but it should cross over just fine.
@@Corbald afaik stereo asynchronous reprojection works just as well, all the quests use it all the time even when streaming over wifi (ie when there aren't depth buffers available to combat distortion)
@@Corbald dual 64mm spaced 360 cameras fixes this entirely, you don't need to move the "head" of the robot at all, it's just a "camera ball on the skybox".
I've done this in unity already for another project and it works great. You use equirectangular over-under video and bob's yer uncle.
It even automatically gives you the "environment scale" of the robot itself, like you were shrunk with a shrinking ray. It's very cool.
You just move your head and it changes the view for you in vr, nothing on the robot itself moves for vision. Solves that problem completely.
Then you just use tank locomotion with the controller.
I really want to get one of those bi-directional ultrasonic BMIs.
Sometimes my hands aren't as accurate as I would like them to be.
I got the feeling that adding brain correction would make interfacing like butter.
mechatronics engineer here, currently designing a robotic tank but want to implement some VR capability as I recently got an oculus quest 2 as a gift. this video has been very helpful, i originally was going to use a esp32 cam but there is too much latency. this video has given me a nice structure. awesome video, you are a legend
So glad to see you back to actively creating! Your designs have been such a great backbone for so many, and it is great to see you breaking new ground! I do wish that you would revisit some of your older stuff though and update and improve those. For example your mouth mech was such a great starting point, but it feels unfinished. Anyway, keep up the great work!
Will, you're brilliant. Keep doing what you're doing.
I've been wanting to try and tackle this myself for a long time. But I also want to control robot arms with the Quest controllers. I have a D&D group that I Zoom into every week and I'd love to have a mini-me there that actually looked around and rolled dice there, instead of just being a static camera.
I've had similar thoughts, but with grocery shopping. I hate it normally, but it'd be fun to drive a robot shopping cart with arms around the Walmart.
HA! That would be great. Eventually we'd just get to the point in that Bruce Willis movie "Surrogates". But silly question, do not the US Walmarts (I'm assuming You're in the US) have remote shopping? They do here (I'm in Canada) For a couple bucks you shop online and then they bring it out to the car. No going in at all.
@@Runescope Sure they do, but... robot! Seriously, I'm not sold on the whole remote shopping idea. Besides, teleoperated avatars with physical capabilities represents a whole market for disabled people. Something like I described would serve a lot of people in a lot of ways, not just shopping.
@@Corbald When it first started up, I used the "Delivery to door" service all the time and it was great. But I found it was more convenient to schedule a pickup time and just let them grab it all. I've been using it for almost four years now and no complaints. But yes, robots would be supremely cool and offer so much for people with disabilities, and if I could wander through a store remotely shopping, I would do it in an instant.
your videos are amazing!
as a newbie in robotics, you're an inspiration to show what can be achieved!
I just got in universal joints for my new head design two weeks ago. 😀 It's good to see we are on similar paths. I will have to try printing some as well. Your project is looking great! Can't wait to see more.
This is great stuff! I've been thinking about this topic for years but could never muster the effort to do anything, lol. I was thinking you could scale it up to the size of a person, which would be cool.
You're my hero! Amazing work and motivation!
nice!, Not the surgery on a grape meme XD.
No one ever tells you the next day the grape died
I love those funny moves the robot makes @6:56. Combined with the face like plate this gives the robot a fun personality.
Now I need to make one too. Very useful to start working from home. That still scares me so it would be good to have a robot like this at home so I can control it from the office 🤣
Wow! Well done... Do you have a team of people working on this project with you? It is a huge project! Side note: It seems like the robot is quite wobbly. The camera's stability is a crucial factor. The user might experience motion sickness due to it. If you find it a real issue, maybe you can add some mass to it or smooth the motors outputs. You may also adjust the tolerances in the joints in case there are backlashes. Will adding friction in the joints help reduce the wobble? Since you 3d print those parts, you can orient them on the building plate so that the layer lines will create such friction.
Love your work Will
Just got a chance to watch the video, this will definitely be a fun project. I'll try to download the files this weekend. Doing the telapresence with a robot would be a lot of fun, especially if you could do it with a human sized robot. Would be a hoot to "walk" around conventions aa the robot.
Loving the classy CEX sticker on the headset. Keepin it real
Great project! I definitely subscribed.
I have been wanting this since I got vr, a lil creature to control
I was just thinking about what it would take to DIY one of these, so I can check up on the cat when away. I would instead use a 360 cam and project that into VR so you wouldn't need any moving parts to look around or worry about nauseating latency when moving your head - all the head movement would be handled by the VR app so although the stream would be delayed, head movements are immediate enough to not cause motion sickness.
I was thinking the same. The 360 camera would automatically kill the vr nausea issue caused by the lag.
Thanks for the tip about picamera2, I also worked on a Raspberry Pi Zero project with a camera module and was having the hardest time getting the stupid thing to work!
would love to see a gorilla tag style bot where you can climb around even your own home courses and interact with stuff, i just think it would be hilarious to be on my pc and have my friend pop in on a little gorilla bot and ask whats going on or somethinf
My setup for transmitting FPV from pi zero to a screen is kinda horrifyingly janky. But it works. I have the feed show on screen and then transmit the whole screen through VNC. So. Far too much lag for telepresence but eh, i like it. The whole project is a mess, so a bit of mess on the usage side is just on brand. Also it works and was easy to set up.
7:26 cheeky bastard😂
love the colours you're printing with what colours did you use ?
Love this idea.
Looking at the math issue from my games background, I'd suggest avoiding angular (euler) representations as much as possible.
If you can, stick to quaternions and vectors - only converting to angles at the last moment if absolutely necessary.
IMO you should use a 360 degree camera and crop the image based on the rotational values of the headset and then use the servos/motors for the translation part of 6DoF
just had an interesting discovery, look at stereo pi v2 board built on the raspberry pi, designed for stereoscopic vision, perhaps you have already seen this but I thought worth sharing it. ill be definitely buying it
instead of servos, those where hydrolic type mechanism, it could allow much more freedom...
Great project!
check out the radxa zero 3 and openipc, that could be a nice hw/sw base to build on
Your designs are sleek
Hey, I am working on the same type of project...also streaming from Unity to Flask!
I never got the real time video stream right...tried web rtc in the past but I guess there should be better options!
Super curious on seeing your sollution! 😊
Print in place universal joints have a lot of slop, why not try a flexture? Also you'll need to use stereoscopic cameras to get the best of this.
I haven’t even started the video but have you taken a look at the headsets Remote Controlled airplanes people use??! They turn their head and the small camera mounted inside the RC plane does aswell, I don’t know how you would be able to combine all of these technologies together but there’s that.(Nevermind you did looked at them HA)
As soon as I seen that in the back It reminded of gundam stuff!! pls take some inspiration there!!
Also, have you thought on getting one of those flat cameras on the eyes with also the same treatment of resin and UV, if even that could work then you could do that next step of bifocal! I'm no expert or even know how to do stuff, this just crossed my mind while watching your videos, and I know I'm asking you to sacrifice one of those flat cameras and I'm sorry for that, but if it works you only have to do it once more!
extrapolate the dimensions of that base to something that can hold a chair for a car sim, pretty pleaseeeee
Which filament did you use for that? I’ve been looking for that shade of green
Do you have the STL file for the universal joint you used?
Wouldn't the latency be nauseating? Do you need to tie the camera to drive the head movement or could you use a wide angle lens and do the movement in unity(like how 180degree videos work), you could just have a fake head to indicate direction.
Could you do the same thing with 360 camera and cut out all the moving parts? I've always wanted to do that on a drone.
whats that white headset behind you?
Unfortunately, all I see in that thumbnail is "Kermit Goatse"
Great video, I am actually creating a similar thing currently. I was focusing more on the software as i find it the hardest part. Also went for the jpg updates every frame at first, since chatgpt recommended it. Unity is very bad at this, I actually created a webxr experience and made it asynchronously download the frames which was almost perfect. It still isn't perfect, I am currently working on sending the video stream theough webrtc, which should be the fastest way. Also my goal is that the quest can just connect to the robot through the web browser instead of downloading an apk, which is much more user friendly. My solution also has two cameras streaming to produce a truly stereoscopic vr view.
I won a worldwide engineering competition with the demo I am talking about. We should connect and help eachother.
Hi Will, I've solved low latency video to Unity from the Raspberry Pi using multiple cameras. I built a robot to rescue a spanner from under my workshop, long story and video coming soon, but I wrote a Unity based application for the Steam Deck that controlled it all with no noticeable latency.
I've also a cargo robot, Johnny Five inspired robot, and a bunch of other things that may be relevant. Give me a shout if I can help with any of it as I'd be happy to share my mistakes to give you a headstart! Pun entirely intended...
okay this is cool
Hack the 3 axis gimbal cam
this reminds me of the electric state
OMG another 4 yr project. I can't do it, by the time you finish things things have changed, no longer Moores law even faster now.
Just use analog fpv. Rids you completely of staying within brand ecosystems like walksnail, dji, etc. there are even okayish AIO analog cam modules for tinywhoops
I can tell you how to do inverse kinematics by hand/code
3:18 pro tip; use claude 3.5
Your videos are lovely, but the background music is annoying, I now watch it with mute and CC
you might be enslaved by skynet to make them terminators 💀
I think you are making a vomit inducing machine, because the mechanical systems you are using have backslash and will inherently have a little latency and mismatch being physical and all, I'd much rather work on a solid state device, that can be bolted on the surface you are working on (or robot) that uses a wide angle lens, like an insta 360, but you either stream the whole picture or somehow (might be impossible) request a certain angle frame from the device, that way you get rid of all the previous problems, but induce other like lower definition and maybe bandwidth problems, who knows, I wish I could work on this. I too am a fan of Alastair Reynolds and his remote presence concept.
Stereo 360 would be a hell of an engineering problem to solve
Edit: nvm I thought he was using a stereo camera, apparently not
@@alfiegordon9013 it's already solved for 180°, have you seen the rigs people use to shoot 3d porn?
@@OpreanMircea I can't say I have, but I'd imagine that they're just tilted so that the overlap cuts out the other camera. Seems like a harder issue to solve for 360 without creating a big old phantom blob if you look the wrong way
Nice idea but so evercomplicated for what it is.
First 👍
People talking into a mic like that are not people I'd subscribe to. The project was cool however.
Will Cogley do you have discord?
I do! Link is here: discord.gg/qXKNRgHKNB