Connecting ROS to a Unity project on the Hololens 2 using Ros Sharp and Ros Bridge
Вставка
- Опубліковано 20 вер 2024
- If you want to use Robotics Hub, here is the link to my tutorial on how to implement the ROS TCP Connector on the Hololens 2: • (Old) Unity Robotics H...
This is a tutorial on how to make a Unity project with Ros Sharp and the Mixed Reality Toolkit. I then run the application on the Hololens 2 emulator and demonstrate how to run it on the Hololens 2 hardware. I also go through the steps on how to connect to Ros Sharp using Ros Bridge on a virtual machine. I demonstrate a working connection by publishing to an odometry topic to get a cube game object in my Unity scene to move.
Desktop Software:
Unity 2019.4.15f1: unity3d.com/ge...
Visual Studio Community 2019 v16.8.3: visualstudio.m...
Windows 10 SDK v10.0.19041.0: developer.micr...
Hololens 2 Emulator v10.0.19041.1131: docs.microsoft...
Mixed Reality Toolkit v2.5.1: github.com/Mic...
Ros Sharp: github.com/Eri...
Ros Bridge: github.com/Rob...
Virtual Machine Software:
VMware Workstation Pro 16: www.vmware.com...
Ubuntu Bionic Weaver v18.04.5 LTS: releases.ubunt...
ROS Melodic: wiki.ros.org/me...
Hi Bryce,
Thank you for such an amazing tutorial!!
Can the method demonstrated in video be used for tele-operation of TurtleBot3 using Hololens2?
For sure, just make sure you run the rosbridge connection script on the raspberry pi running ROS that is connected to your Turtlebot3. As long as it is receiving the cmd_vel or whatever topic message type you need to command a turtlebot, it should work.
Hi Bryce,
Thankyou for the tutorial, I am trying to connect my already existing Unity project built on docker container with hololens2. Do you have any idea or hints on how to establish the connection between Unity ROS project and holoens2 this way?
Yeah! You can follow my other video that shows me connecting a docker container to the HoloLens2: ua-cam.com/video/3KMhdGV6Ql8/v-deo.html Hopefully it helps but I would strongly recommend using Unity Robotics Hub rather than Ros Sharp/Bridge: github.com/Unity-Technologies/Unity-Robotics-Hub
Thank you for this amazing tutorial.
I wanted to subscribe to the camera image msgs on ROS using a Hololens 2. Would you give me a hint, please? Thanks again Bryce.
Yeah so if you go into the RosSharp repository I linked, in the rosbridgeclient/ROS communication script folder there is a imagesubscriber script you can use.
Hi Bryce, first of all thanks for the tutorial, it helped me a lot. I followed the tutorial and it worked like charm on the Unity editor, but as soon as I deploy it to the hololens2 it doesn't connect to ROS, I assume it is some Firewall or some Unity Setting im missing (although I checked InternetClient, InternetClientServer and PrivateNetworkClientServer settings on the player settings). I'm clueless with this problem, maybe you have a solution how I can establish the connection in the Real Hololens2 device? I would be thankful.
Although I’m pretty sure it should all work if it works via the Unity editor, I would check to make sure your firewall settings on windows allows public and private connections. Going to the advanced settings, make sure wherever ROS is running allows all connections.
Hi Bryce,
Great tutorial so far. I'm stuck at the 'catkin_make' section and thought I'd ask your help. It mentions that there are some non_catkin packages and will therefore not install. I then used the catkin_make_isolated instead but that leaves out most of the packages. How would I go about installing the rest?
It looks like the repository has been updated since this video, so this will be a problem for all future viewers. Hopefully this answer will be here to help them.
Cheers
I just tested it out on my machine and it still works for me. After creating the 'rosbridge_connector' directory, did you make sure the rosbridge repository was cloned into 'src'?
@@bryceikeda7782 I'm having this same problem although there's no issue regarding the folders and everything. :(
@@jiwon4058 Hi Jiwon, sorry for the late response. Honestly, I would recommend using Unity Robotics Hub github.com/Unity-Technologies/Unity-Robotics-Hub. It is basically built off of ROS # and has way more support for roboticists to ask questions if something isn't working.
Maybe a little bit too late, but I had the same error. I'm using ROS, a Hololens 2 and a Kinect, where the Kinect requires ROS 1 to work. But the default branch of the Ros Bridge Repo is for ROS 2. After switching the branch my catkin_make worked.
Hello Bryce,
I am working on a project that involves a HoloLens 2 application subscribing to some compressed image data. I have managed to get Unity to display the compressed image on a plane in the scene. But I am running into some issues with deploying the application. I think it has something to do with the 999+ errors in the Unity editor. The errors look like this: GUID [b98b3e1f52d80b34a8c2a7a75d3e70ec] for asset 'Packages/com.microsoft.mixedreality.toolkit.standardassets/Textures/OculusRemoteController_white.png' conflicts with:
'Assets/MRTK/StandardAssets/Textures/OculusRemoteController_white.png' (current owner). Do you have any insight on how to resolve these conflicts?
I haven’t personally run into this issue but it seems like you have the same asset in two different folders that are conflicting. I would delete one of them via the file path it gives you.
@@bryceikeda7782 I managed to get video playing in the Hololens 2 that I had streamed from a ros env. I had to use this tutorial to setup most of the project: docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/tutorials/mr-learning-base-02?tabs=openxr and changed a few things here and there in conjunction with your video. Thank you for the help.
Hi Bryce, great video! Im working on a similar project but am looking to control a robot simulation on ROS using HMD position tracking. You video illustrates well how we can use ROS to manipulate 3D objects in unity but im wondering if you have any ideas of reversing this, letting the HMD tracking data dictate the orientation of a robot (arm) in ROS? I would assume instead of using the Microsoft tool kit to be our controller in Unity we could simply add a VR camera controller and link it to the ROS Connector game object. Any ideas on getting that movement data to manipulate a ROS simulation?
Hmm, if you are trying to use the Hololens position to control the robot arm you can make a call to the main camera transform to get its current position and orientation. Then you could use the PoseStampedPublisher.cs example in Ros sharp to send that information from Unity to Ros via the websocket.
your Resolution is too high. Can't see anything with the small size. Please look into the zoom feature that zooms the highlighted area or ( mouse tracking zoom ) or reduce your resolution.
Will do, thanks for the feedback!
Hey Bryce, I have an issue whenever I launch the game. I get a notification that the ros connector failed to connect to my websocket. The websocket is properly launched, the ip is correct with hostname -I, and my network settings are setup the same as yours (I think). I don't know why it keeps failing to connect. Any tips?
Sorry for the late reply. If it is Unity that can't connect, I would check to make sure your firewall allows inbound private and public connections. Also, if you are using a virtual machine, make sure it is bridged to the same network as your computer. You can check this by pinging your Unity IP address or the IP address of your main computer.
Hallo Bryce, i need your insight. I am using a Ubuntu WSL for windows 10 instead of virtual machine. Do you think it is possible to establish a connection with HoleLens2
Hi Mohan, I don't have any experience with WSL so I am not sure. However, I would assume it is possible since you should be able to run all of the packages.
Hi Mohan,
yes, it is possible. I have been working with WSL (1&2) and RosSharp for a few months now. Use "localhost" as an IP address in that case. You might run into some issues if you want to connect to the WSL instance via the local network (e.g. from another device and not only form your host-machine), which can be solved with port forwarding etc.
Hi Bryce
I am Japanese student. I'm sorry, I'm not good at English.
I've watched your video and practiced it, but I'm having trouble with an error. The Unity version, ROS #, and execution procedure are the same as in the video, but I get the following error:
Assembly'Library / ScriptAssemblies / Assembly-CSharp.dll' will not be loaded due to errors:
Reference has errors'RosBridgeClient'.
Assembly'Library / ScriptAssemblies / Assembly-CSharp-Editor.dll' will not be loaded due to errors:
Reference has errors'Assembly-CSharp'.
Assembly'Assets / RosSharp / Plugins / External / Newtonsoft.Json.Bson.dll' will not be loaded due to errors:
Unable to resolve reference'Newtonsoft.Json'. Is the assembly missing or incompatible with the current platform?
Reference validation can be disabled in the Plugin Inspector.
Assembly'Assets / RosSharp / Plugins / RosBridgeClient.dll' will not be loaded due to errors:
Unable to resolve reference'Newtonsoft.Json'. Is the assembly missing or incompatible with the current platform?
Reference validation can be disabled in the Plugin Inspector.
If you have a solution or opinion, can you tell me?
I using ...
Unity2019.4.15f.1
MRTK V2.5.1
EricVoll/ros-sharp
I'm sorry, maybe these errors were okay if they were displayed.
I will try it again.
Hi Kaze,
It sounds like you are missing the extra Plugins folder from ros-sharp. In the EricVoll/ros-sharp repo go to /ProjectSetup/RosSharpUnity/Assets/ and add the Plugins folder to the assets in your Unity project. Let me know if that does the trick.
@@bryceikeda7782
I'm sorry for the late reply.
I asked the question with ros-sharp-master / Unity3D / Assets RosSharp and ros-sharp-master / ProjectSetup / RosSharpUnity / Assets Plugins added.
So it looks like this after reflecting your opinion.
@@nezumi7195 Hmm, then I'm not really sure. That was the only way I could recreate your error. Perhaps try re-importing the ros sharp plugins/libraries? Sorry I couldn't provide more help.
@@bryceikeda7782 When I tried again, I was able to successfully connect ROS and Unity. I'm using ROS kinetic and it's still working.
Thanks to your video I was able to solve the problem. Now I can proceed with my research.
Thank you very much.