Firstly, this is a great tutorial! But I found that I can communicate with ROS in my Unity, but deploying all of this to my Hololens2 seems to be ineffective. I think this is a problem with ROS-TCP Connector. Do you have any recommended methods to solve this problem?
Thank you so much Bryce for this amazing tutorial! I seem to have almost everything working... I'm on Win11. I've successfully built the HoloLensBaseProject and uploaded it to my Hololens device, and I've properly established a connection between my ROS on WSL2 and my Hololens device. I even get the number counter incrementing! There is just one issue, on Hololens, the first button is forever stuck with the subtitle "Set IP Address". When clicked it opens up the custom keypad. This keypad is pretty much unresponsive. The textbox is stuck saying "Enter IP Address..." and all numberpad keys are unresponsive. Hitting clear, backspace, or OK does not interfere with my ROS connection whatsoever. My connection with ROS is working, but that's because I set up the correct IP Address while building the project in Unity under Robotics > ROS Settings. What is weird is that when I press play in the Unity editor, I can also connect to my WSL ROS, but there, the buttons work ! The first button has my chosen ROS IP as a subtitle, and the keypad keys are responsive. I can change my IP with the keypad to disconnect and to reconnect. Would you have any idea why the menu and keypad wouldn't work on Hololens vs on Unity? Thank you very much in advance, -Liam
Hmm this is a tough one. It's weird that it works in play mode but not during deployment. Have you checked to make sure you are building the correct scene?
@@游勇-v7d I unfortunately haven't figured out how to fix this issue. I will write again if I find a solution someday. @bryceikeda7782 I believe I am building the correct scene (the only scene I have in my project).
I would make sure that your firewall is not blocking connections to or from Unity. It should be in your firewall inbound rule settings under the Unity version you are using. You should also check to make sure Ubuntu is under the same network as the Unity application. If you are using WSL or another VM, its network should be bridged. If you are doing this to two separate computers, they should both be on the same wifi network.
Hello Sir, I hope you are doing well, I'm working on my thesis Franka Panda_robot with ROS and Unity based augmented reality. I want to know where can I get this model for example a useful github that contains the needed files. I saw many videos on youtube but without a reference. I really appreciate your help
Hi, Thank you so much for your video it is helpful to make my project topic based on augmented reality. But im facing an issue that is in my number pad input (script), There are no any tabs for ROS and connect IP. Only two tabs are available it is script and set IP button text. How to get those ROS and connect IP tabs? (For your ref: you explained this in your video on 9:01).
It's been a while since I've worked with this repo. However, I think I updated the directions so the video is slightly out of date. I would follow the REAME directions in the repository itself. If that still doesn't work let me know!
Hey Bryce, Thanks so much for your videos since they have been a huge help with my senior design project thus far. I'm at a sticking point in my project and was wondering if you could provide some guidance. I am working on connecting a Unity Project for a HoloLens 2 headset to a RPi 4 running ROS2 to control it remotely. I set up the TCP Connection successfully thanks to you but now I am unsure how to send ROS msgs from Unity to ROS2. More specifically, under the "Robotics" tab, I am unsure which path to put for the "ROS message path". Do I need to have ROS2 installed on my local machine that I am running Unity on? If so, I might be in trouble since I am running on Windows 11 and not 10. Any help would be greatly appreciated. Thanks :)
Hi, glad you were able to get this working with a RPi 4 running ROS2. I haven't worked with ROS2 yet but it's great to hear people are having success with it. In terms of the ROS message path, I think all you need to do is download the relevant message files. For example, it looks like github.com/ros2/common_interfaces holds most of the common files for ROS 2. Then you can set the ROS message path to wherever you downloaded those files to so that you can generate the messages for Unity.
Also, you can follow this video I made ua-cam.com/video/LxK6jxpwNa8/v-deo.html which actually shows you how you can use get Ubuntu running on WSL 2 in Windows 11. That way, if you wanted to, you can use ROS with windows! Let me know if you have any other questions.
@@bryceikeda7782 That worked!!!!! Thanks so much!!! One more question off of that would be how I would access those messages in a separate script? I'm following Unity Robotics Hub's tutorial for creating a publisher script. When I create my own script, how would I access those message types from that repo when using a function like "ros.RegisterPublisher(topicName);" instead of found in the example script Thanks again for all your help!
@@mad4234 So you would need to include the namespace those messages are in. If you click the RosMessages folder, and find the script with the message type you want to use, check out the namespace. For example, it could say namespace RosMessageTypes.Control. In the script you want to use it in, put at the top "using RosMessageTypes.Control" and then in that script you can use whichever message for the publisher.
hi Bryce, thanks for your video. I can connect to the ros on Ubuntu via Unity on Windows (Green on IP position). However, the hololens2 can not connect to the ros (Nothing happened). I can not find the problem.
@@bryceikeda7782 I can't ping Hololens from Ubuntu (VM), but I can access to the 1XX.XXX.XXX.XX to control the Hololens on the web browser... btw I can ping my windows from Ubuntu.🤔
@@bryceikeda7782 Hey Bryce, I also have the same issue as Jiazhen. I can't ping the Hololens IP from Ubuntu and from what I have read online it seems Microsoft blocks ping requests to the HoloLens? Are you able to ping your HL?
@@zyx-kb3cl Ah okay, that rings a bell, yeah you probably can't ping the HoloLens. Networking issues are tough to diagnose. I would recommend you use holographic remoting, that way you can run everything on Unity and it just streams to the HoloLens. Since the network is already working in this Ubuntu-> Unity on windows setup.
I have not been able to connect ROS with Unity. Could you help in pointing out on how to correctly configure the network configuration on both Unity and ROS on VM?
Thanks so much for this tutorial!!! But I configured it under Ubuntu 20.04, and after reporting the error, I couldn't solve the problem by sudo apt install python-future. Is there any other way to solve this problem?
Hello Bryce! Would it be possible to send video from ROS to unity with this method? Would it also be possible to use the Unity application(with the video) on oculus quest glasses in real time? Thank you so much!
Hi Bryce, I’m really new to Unity and ROS. I’m doing a robotic project involving HoloLens2 and I need the device to communicate with a robot via ROS2. So I was just wondering if this could also work with ROS2? Thanks
Hi Massittha, it will work with ROS2. If you go up to the Robotics tab in Unity, then click ROS Settings and switch the protocol to ROS2, it should work from there. However, I have not used it with ROS2 yet so I don't know how it differs from ROS1.
Thank you for this great tutorial Bryce!!! would it be possible if you show us how to move a gameobject according to real-time odometry data received from an Optitrack using the mocap_optitrack package and use the rosbridge server maybe to publish the data to the topic that the game object is subscribed to? the final result would be a game object that we could see on Hololens2 that moves according to the markers real-time movement captured by the Optitrack.
Hi N.J Production. I don't think I will be able to do that because I don't have access to an Optitrack system right now. However, if you have the odometry data from the optitrack package you should be able to subscribe to that by using this script as a template github.com/Unity-Technologies/Unity-Robotics-Hub/blob/main/tutorials/ros_unity_integration/subscriber.md in Unity. It looks like mocap_optitrack publishes either tf transforms, poses or 2D Poses so Unity Robotics Hub should already have those message types by default so you don't need to import them. In your Subscriber script you can add a public GameObject that will correspond to the game object in your scene. Then set the GameObject transform position and rotation to the position and orientation from the geometry_msgs/Pose you are receiving via your callback (the ColorChange method in the subscriber example). Let me know if you have more questions while you try to do this.
Nothing should be showing up on the HoloLens 2. It just builds and deploys. The rest is up to you to create things in the scene to show up on the HoloLens.
Hello Bryce , I'm beginner and I want to connect my hololens 1 to ROS Melodic on my computer but when I build the project on unity, I have the following compilation error message: error CS0234: The type or namespace name 'UrdfImporter' does not exsit in the namespace 'Unity.Robotics' Do you know about this mistake and how solve it ? Thank's a lot
When I import package on unity the git import URL does'nt work so I've download the GIT on my computer and I use add package from disk to find it,I think the problem comes from this step. Have you installed a module or performed a manipulation to be able to import a GIT from a URL?
Hmm I feel like that should be fine. Did you try using this url: github.com/Unity-Technologies/URDF-Importer.git?path=/com.unity.robotics.urdf-importer#v0.5.2
Yes, and I have many erros who appears : No git executable was found (but I have already install it on my computer) Cannot perform upm operation Error adding package
Gotcha, try this link. This is the one I use rather than the newer one. github.com/Unity-Technologies/URDF-Importer.git?path=/com.unity.robotics.urdf-importer#v0.5.0 also do you have an asmdef in your folder holding the scripts that are giving you the import error?
Hello Sir, I hope you are doing well, I'm working on my thesis Franka Panda_robot with ROS and Unity based augmented reality. I want to know where can I get this model for example a useful github that contains the needed files. I saw many videos on youtube but without a reference. I really appreciate your help
I would probably use the urdf from this repository: github.com/frankaemika/franka_ros/blob/develop/franka_description/robots/panda/panda.urdf.xacro and I think you will need to convert the xacro to urdf using one of the commands found on this thread answers.ros.org/question/10401/how-to-convert-xacro-file-to-urdf-file/
Firstly, this is a great tutorial! But I found that I can communicate with ROS in my Unity, but deploying all of this to my Hololens2 seems to be ineffective. I think this is a problem with ROS-TCP Connector. Do you have any recommended methods to solve this problem?
Thank you so much Bryce for this amazing tutorial!
I seem to have almost everything working... I'm on Win11. I've successfully built the HoloLensBaseProject and uploaded it to my Hololens device, and I've properly established a connection between my ROS on WSL2 and my Hololens device. I even get the number counter incrementing!
There is just one issue, on Hololens, the first button is forever stuck with the subtitle "Set IP Address".
When clicked it opens up the custom keypad. This keypad is pretty much unresponsive. The textbox is stuck saying "Enter IP Address..." and all numberpad keys are unresponsive. Hitting clear, backspace, or OK does not interfere with my ROS connection whatsoever.
My connection with ROS is working, but that's because I set up the correct IP Address while building the project in Unity under Robotics > ROS Settings.
What is weird is that when I press play in the Unity editor, I can also connect to my WSL ROS, but there, the buttons work ! The first button has my chosen ROS IP as a subtitle, and the keypad keys are responsive. I can change my IP with the keypad to disconnect and to reconnect.
Would you have any idea why the menu and keypad wouldn't work on Hololens vs on Unity?
Thank you very much in advance,
-Liam
Hmm this is a tough one. It's weird that it works in play mode but not during deployment. Have you checked to make sure you are building the correct scene?
Hi,I have encountered the same problem as you. Have you resolved it yet,Thank you very much in advance
@@游勇-v7d I unfortunately haven't figured out how to fix this issue. I will write again if I find a solution someday. @bryceikeda7782 I believe I am building the correct scene (the only scene I have in my project).
Great video though how what did you do in ubuntu? i can't connect
I would make sure that your firewall is not blocking connections to or from Unity. It should be in your firewall inbound rule settings under the Unity version you are using. You should also check to make sure Ubuntu is under the same network as the Unity application. If you are using WSL or another VM, its network should be bridged. If you are doing this to two separate computers, they should both be on the same wifi network.
Hello Sir, I hope you are doing well, I'm working on my thesis Franka Panda_robot with ROS and Unity based augmented reality.
I want to know where can I get this model for example a useful github that contains the needed files. I saw many videos on youtube but without a reference.
I really appreciate your help
Hi, Thank you so much for your video it is helpful to make my project topic based on augmented reality. But im facing an issue that is in my number pad input (script), There are no any tabs for ROS and connect IP. Only two tabs are available it is script and set IP button text. How to get those ROS and connect IP tabs? (For your ref: you explained this in your video on 9:01).
It's been a while since I've worked with this repo. However, I think I updated the directions so the video is slightly out of date. I would follow the REAME directions in the repository itself. If that still doesn't work let me know!
Hey Bryce,
Thanks so much for your videos since they have been a huge help with my senior design project thus far. I'm at a sticking point in my project and was wondering if you could provide some guidance.
I am working on connecting a Unity Project for a HoloLens 2 headset to a RPi 4 running ROS2 to control it remotely. I set up the TCP Connection successfully thanks to you but now I am unsure how to send ROS msgs from Unity to ROS2. More specifically, under the "Robotics" tab, I am unsure which path to put for the "ROS message path". Do I need to have ROS2 installed on my local machine that I am running Unity on? If so, I might be in trouble since I am running on Windows 11 and not 10. Any help would be greatly appreciated. Thanks :)
Hi, glad you were able to get this working with a RPi 4 running ROS2. I haven't worked with ROS2 yet but it's great to hear people are having success with it. In terms of the ROS message path, I think all you need to do is download the relevant message files. For example, it looks like github.com/ros2/common_interfaces holds most of the common files for ROS 2. Then you can set the ROS message path to wherever you downloaded those files to so that you can generate the messages for Unity.
Also, you can follow this video I made ua-cam.com/video/LxK6jxpwNa8/v-deo.html which actually shows you how you can use get Ubuntu running on WSL 2 in Windows 11. That way, if you wanted to, you can use ROS with windows! Let me know if you have any other questions.
@@bryceikeda7782 That worked!!!!! Thanks so much!!!
One more question off of that would be how I would access those messages in a separate script? I'm following Unity Robotics Hub's tutorial for creating a publisher script. When I create my own script, how would I access those message types from that repo when using a function like "ros.RegisterPublisher(topicName);" instead of found in the example script
Thanks again for all your help!
@@mad4234 So you would need to include the namespace those messages are in. If you click the RosMessages folder, and find the script with the message type you want to use, check out the namespace. For example, it could say namespace RosMessageTypes.Control. In the script you want to use it in, put at the top "using RosMessageTypes.Control" and then in that script you can use whichever message for the publisher.
@@bryceikeda7782 Awesome, thank you so so so much!!!!
hi Bryce, thanks for your video. I can connect to the ros on Ubuntu via Unity on Windows (Green on IP position). However, the hololens2 can not connect to the ros (Nothing happened). I can not find the problem.
Hi Jiazhen, are you able to successfully ping the HoloLens IP from Ubuntu?
@@bryceikeda7782 I can't ping Hololens from Ubuntu (VM), but I can access to the 1XX.XXX.XXX.XX to control the Hololens on the web browser... btw I can ping my windows from Ubuntu.🤔
@@bryceikeda7782 Hey Bryce, I also have the same issue as Jiazhen. I can't ping the Hololens IP from Ubuntu and from what I have read online it seems Microsoft blocks ping requests to the HoloLens? Are you able to ping your HL?
@@zyx-kb3cl Ah okay, that rings a bell, yeah you probably can't ping the HoloLens. Networking issues are tough to diagnose. I would recommend you use holographic remoting, that way you can run everything on Unity and it just streams to the HoloLens. Since the network is already working in this Ubuntu-> Unity on windows setup.
I have not been able to connect ROS with Unity. Could you help in pointing out on how to correctly configure the network configuration on both Unity and ROS on VM?
Sure, is your VM using a bridged network? Did you allow the Unity connection through your firewall?
Thanks so much for this tutorial!!! But I configured it under Ubuntu 20.04, and after reporting the error, I couldn't solve the problem by sudo apt install python-future. Is there any other way to solve this problem?
Which error are you referring to? Sorry, it's been a while since I recorded this tutorial.
Hello Bryce! Would it be possible to send video from ROS to unity with this method? Would it also be possible to use the Unity application(with the video) on oculus quest glasses in real time? Thank you so much!
Hi sorry for the late reply, I haven't tried this on the Oculus quest so I'm not too sure...
@@bryceikeda7782 Without taking into account oculus quest, could it be possible? Just send video into Unity from ROS.
@@EmilioMrDJ I know you can receive CompressedImage msgs. Not sure about any other message types for video as I haven't tested it any other way.
Hi Bryce, I’m really new to Unity and ROS. I’m doing a robotic project involving HoloLens2 and I need the device to communicate with a robot via ROS2. So I was just wondering if this could also work with ROS2? Thanks
Hi Massittha, it will work with ROS2. If you go up to the Robotics tab in Unity, then click ROS Settings and switch the protocol to ROS2, it should work from there. However, I have not used it with ROS2 yet so I don't know how it differs from ROS1.
@@bryceikeda7782 Thank you Bryce! I’ve tried deploying the Unity Publisher from Unity Robotics Hub on HoloLens2 and it works.
Thank you for this great tutorial Bryce!!! would it be possible if you show us how to move a gameobject according to real-time odometry data received from an Optitrack using the mocap_optitrack package and use the rosbridge server maybe to publish the data to the topic that the game object is subscribed to? the final result would be a game object that we could see on Hololens2 that moves according to the markers real-time movement captured by the Optitrack.
Hi N.J Production. I don't think I will be able to do that because I don't have access to an Optitrack system right now. However, if you have the odometry data from the optitrack package you should be able to subscribe to that by using this script as a template github.com/Unity-Technologies/Unity-Robotics-Hub/blob/main/tutorials/ros_unity_integration/subscriber.md in Unity. It looks like mocap_optitrack publishes either tf transforms, poses or 2D Poses so Unity Robotics Hub should already have those message types by default so you don't need to import them. In your Subscriber script you can add a public GameObject that will correspond to the game object in your scene. Then set the GameObject transform position and rotation to the position and orientation from the geometry_msgs/Pose you are receiving via your callback (the ColorChange method in the subscriber example). Let me know if you have more questions while you try to do this.
how does it look like on hololens 2?
Nothing should be showing up on the HoloLens 2. It just builds and deploys. The rest is up to you to create things in the scene to show up on the HoloLens.
Hello Bryce , I'm beginner and I want to connect my hololens 1 to ROS Melodic on my computer but when I build the project on unity, I have the following compilation error message:
error CS0234: The type or namespace name 'UrdfImporter' does not exsit in the namespace 'Unity.Robotics'
Do you know about this mistake and how solve it ?
Thank's a lot
I would make sure you have the URDF package by checking in your package manager. It should say URDF Importer and have a green check mark
When I import package on unity the git import URL does'nt work so I've download the GIT on my computer and I use add package from disk to find it,I think the problem comes from this step.
Have you installed a module or performed a manipulation to be able to import a GIT from a URL?
Hmm I feel like that should be fine. Did you try using this url: github.com/Unity-Technologies/URDF-Importer.git?path=/com.unity.robotics.urdf-importer#v0.5.2
Yes, and I have many erros who appears :
No git executable was found
(but I have already install it on my computer)
Cannot perform upm operation
Error adding package
Gotcha, try this link. This is the one I use rather than the newer one. github.com/Unity-Technologies/URDF-Importer.git?path=/com.unity.robotics.urdf-importer#v0.5.0 also do you have an asmdef in your folder holding the scripts that are giving you the import error?
Hi,I loved this video of yours.I am trying to controll UR5e robot using Unity game engine.It would be a very big help if you can tell me what to do.
In what way are you trying to control the UR5? I have not used the robot before.
Hello Sir, I hope you are doing well, I'm working on my thesis Franka Panda_robot with ROS and Unity based augmented reality.
I want to know where can I get this model for example a useful github that contains the needed files. I saw many videos on youtube but without a reference.
I really appreciate your help
I would probably use the urdf from this repository: github.com/frankaemika/franka_ros/blob/develop/franka_description/robots/panda/panda.urdf.xacro and I think you will need to convert the xacro to urdf using one of the commands found on this thread answers.ros.org/question/10401/how-to-convert-xacro-file-to-urdf-file/