iPhone Sensors & Depth Camera in TouchDesigner Tutorial - Part 1!
Вставка
- Опубліковано 24 січ 2021
- Get access to 200+ hours of TouchDesigner video training, a private Facebook group where Elburz and Matthew Ragan answer all your questions, and twice-monthly group coaching/mastermind calls here: bit.ly/2zJ0lmt
If you’re a TouchDesigner Beginner, check out our TouchDesigner Tutorial Series! We teach you all the basics of how to use TouchDesigner, including mouse controls, hotkeys, parameters, the operator families, and more: interactiveimmersive.io/touch...
We have full-on super computers in our pockets. The latest iPhones have everything from built in AR, depth cameras, facial tracking, marker tracking, and a smorgasbord of sensors like gyroscopes, accelerometers, and more. You'd be surprised how easy it is to integrate all of these into TouchDesigner using an app called Zig Sim and Zig Sim Pro over OSC and NDI.
BOOM THERE IT IS!!! @5:31 So awesome - then it's data in TD and we can do anything we want w/it. This is so awesome.
Yay, i'm especially stoked about those depth cams.. lots of potential. thanks for the demo / tut!
No problem! Part 2 next week will show off the ARkit and some practical setups in TouchDesigner :)
@@TheInteractiveImmersiveHQ that NDI stream alone is pretty wild
@@LukeHildreth I'm very excited to use the depth map! It's super cool.
super helpful thank you so much
Our pleasure ;)
You got an instant subscribe
Thanks! Glad you enjoyed it and are getting value from our work :)
burh I just got the hang of arduino and touchdesigner and i come across THISSS!!! lol like why didn't this come first! lol thanks for the tutorial! :)
I feel like this cut before it was supposed to? We never see what AR Kit does.
Yup that was an edit issue on our part and UA-cam won't let us replace the video. Stay tuned next week for part 2 where I'll go even deeper on the ARkit side of things and even make a sample project in TouchDesigner for face tracking!
Very cool! It's also available on android. The pro version is ios only though.
Yup! The free version is available both on iOS store and Google Play store.
Thanks for the tutorial! I'm set up on a mac with an iphone, commercial license. I get OSC coming through just fine. Wondering if you or anyone reading has had trouble getting your phone to show up as source in an NDI in.
I've seen a few people have issues initializing the stream sometimes. Few things to try: 1) make sure you're on the same network on both devices, 2) try to fully close the app on your iPhone and then reopen it (sometimes the stream gets a bit stuck it seems), 3) try to disable firewall on your mac temporarily to see if that helps. If those things don't work, let me know and I can give you some more things to try.
@@TheInteractiveImmersiveHQ Thanks man, I had tried 1 and 2 already, and was about to kill the firewall and suddenly it worked. Quitting out of both apps twice(?) was maybe the "trick". Cheers!
@@timlillis2123 Great that you got it working. A bit annoying that it's not a clear and easy to solve issue. My guess is that the NDI stream isn't getting opened and closed properly, especially since in the newer builds of TouchDesigner it's using a new SDK version of NDI, so sometimes you need to close and re-open the apps to get them working. Hopefully it holds up a bit better!
@@TheInteractiveImmersiveHQ Hi I have the same issue, and followed all the tips but still not working. I can't find the source options in the NDI in top.
@@lishu9120 which version of TouchDesigner are you using? That sounds strange. Do you see the NDI stream from the phone if you use the NDI Studio Monitor on your computer?
Not sure how dumb this question is but can I get zig sim to communicate with a pc that's only connected with an ethernet cable?
Nice! SO awesome thanks! Is it somehow possible to control i.e. Ableton and make some music with this data?
For sure! You can send this data over to Ableton as MIDI information via the MIDI Out CHOP. If you're running Windows you'll also need a program like Virtual MIDI (www.tobias-erichsen.de/software/virtualmidi.html) to route the MIDI data to Ableton. The Math CHOP is also a useful tool for modifying your CHOP data to be in the right range for MIDI note values. Hope that helps!
@TheInteractiveImmersiveHQ Thanks for that tip! I am working on a Mac, so I do not need a virtual MIDI program? I also was able to connect TD Ableton to Ableton but I am still lost on how to get the Data out of TD and connect it with Ableton to control something like audio effects...
This !
Hopefully this will have auto captions soon.
It should get automatically added once UA-cam has them ready!
Thanks for this showcase! I bought the Pro version and am trying to connect an NDI stream to TD (Windows, Commercial license, latest build) but it is not showing up as an NDI source in the TOP. It successfully sends data into the OSC CHOP.
Did you come across any hiccups setting it up that might help solve?
Our pleasure! Hmm I didn't but it sounds like it could be a network issue or firewall issue. The first thing I'd try is to install NDI Tools from Newtek's website and see if inside of the Studio Monitor you can access the NDI stream from your phone. If you can, then it's probably a firewall issue and Windows is stopping TouchDesigner from talking on the network fully. If you don't see it in Studio Monitor as well, then it sounds like it is more of a network issue and for some reason NDI isn't able to send data around the network. Do you have 2 computers you could try sending NDI video streams between as test?
I had the same strange problem. Tested this out with Ndi HX Camera App. There the Ndi Connection works well. Just the ZigSimPro App somehow does not want to send out an Ndi Stream. Funny solution: Uninstalled the app and installed it again on my Iphone :)
@@dr.potatoo3547 If you see that issue, try force closing the app and re-opening it. I had that happen once or twice on my end as well.
@@dr.potatoo3547 Thank you. 감사합니다.
@@TheInteractiveImmersiveHQ Thank you. 감사합니다.
Hi is it only working when I m in the ZIG SIM App , or could I Have for example a video playing on the iPhone screen. ?
That's a good question! I haven't tested those capabilities out but it should be pretty easy to try. My guess though is that iOS's aggressive garbage collection would close the app pretty quickly once it's in the background. I've heard other apps have to do pretty hacky things to try to stay active in the background longer.
Thank you for this!! When I try feeding in NDI from Zig Sim Pro, within TD my NDI In TOP does not show anything under the "source" dropdown (it's empty). How do you recommend I troubleshoot the NDI into TD since I can't find the Zig Sim Pro source in the TOP?
Great question. Usually for troubleshooting I always try to simplify my setup. For example, a good way to simplify this setup is remove TouchDesigner from the equation and see if you can use the NDI Studio Monitor app that comes with NDI Tools to see if you can see the stream. If you can't see it in there, then the issue is likely on the network. I'd check to make sure your phone and computer are connected to the same network. You can also try temporarily disabling your computer's firewall to make sure that isn't blocking the NDI streams. Let me know once you try those if you have any success.
@@TheInteractiveImmersiveHQ Thanks for this, I've disabled my firewall which is bringing in the OSC stream great, but NDI still seems to not work...I also am not able to read in my iPhone's NDI input directly to NDI Studio Monitor. Do you know if there are specific packages to download to get my iPhone Zig Sim Pro streaming NDI into NDI Studio Monitor? Haven't been able to find much documentation. Thanks again for your help!
@@mikepastore9742 if all the network conditions are right, Zig Sim should be firing out the NDI stream and Studio Monitor should see it if you open the right click menu and look at sources. A few things to try, 1) force close and re-open Zig Sim on your iOS device. 2) Can you confirm they're on the same WiFi network? It sounds like maybe they're not on the same network, because otherwise the rest of the NDI setup should be pretty straight forward. 3) Can you try downloading another app from the app store that has NDI output from it that we can test?
@@TheInteractiveImmersiveHQ thanks! I had the same problem. I closed and open again both, TouchDesigner and ZIG SIM Pro App. After the connection works. Voila :-). Camera detected.
I had the same issue and got it working by turning active off and on again in the NDI In params
Can I use my iPhone as a Kinect sensor, do you have any tutorials linking both iPhone and fluid simulation
You can use the ARKit features in ZIG SIM PRO for facial and body tracking, including position and rotation of joints etc. At the moment we don't have a tutorial like that, but be sure to see the next video in the series for an example of working with ARKit!
*iPhone ARkit & Depth Sensor in TouchDesigner Tutorial - Part 2!* - ua-cam.com/video/dfKfVJfy7SI/v-deo.html
Hi Thanks for such a nice tutorial, I tried this first time using the free version zig sim, and it worked very well, but when I'm trying this time with the pro version, it keeps telling me unknown socket errors, and the free version doesn't work either...Do you know what is the problem?
Hmm that's interesting it isn't working anymore. Do you know if you've changed wifi networks or anything like that? Also on your phone, I always recommend fully force closing ZigSim between use, because sometimes if it stays in the app memory, I find it can not work the next time you open it.
@@TheInteractiveImmersiveHQ Thanks a lot, I will try restarting my iphone
@@nickchenhezhang9292 Let me know if that helps!
@@TheInteractiveImmersiveHQ Same issue, unknown socket error seems to be a recurring problem as of late.
@@zachbehrman8891 I haven't tried it in a moment, if you're seeing a socket error, sometimes this can be because either TouchDesigner or the iOS app hasn't properly closed the socket since the last time you tried it. I think if you can try fully restarting your computer and iOS device, see if that makes the socket error go away.
great thanks for sharing. Did you use an iPhone with Lidar? is this a must?
I used an iPhone 11 Pro for this video, so I'm not sure exactly what it has and doesn't have. It has a depth camera, but I think LiDAR is only available on the newer devices?
@@TheInteractiveImmersiveHQ Yes Lidar sensor is only available in iPhones 12 -13. I was wondering what is the difference of using Lidar in your process?
@@tiporight Good question! LiDAR is a bit different than depth cameras because depth cameras will just provide you a black/white texture that shows you how far or close things are from the camera, whereas LiDAR gives you a texture that is a point cloud, where each pixel contains XYZ positions in 3D space. They can have similar use cases a lot of the time, but they're fundamentally a bit different. I haven't tried iPhone's LiDAR so I can't speak specifically to how it works yet :)
@@TheInteractiveImmersiveHQ Thanks I appreciate your explanation. Did you try using ZigCam with iPhone 11pro?
@@tiporight I haven't, but I just downloaded it and can give it a try this week!
Hey dude, on mac.. got the gyro into TD via osc.. then got the pro version. with the NDI, IPHONE ZIG SIM PRO shows up in source name on TD just tried to use the NDI its just red... it also eventually comes up with a caution in the NDI in Top 'unable to find the selected source name, fail-over source may be used instead.. any ideas?
hmm just tried the NDI video monitor and it comes through no probs...
ha... never mind... it was little snitch :)
@@CultureDrift Great that you got it up and running. Especially with network protocols like NDI, always good to remember about firewalls and apps like LittleSnitch!
do you know if, the liar scanner on the iPhone can be used into touch designer?
You can use the LiDAR sensor to scan objects and import them as .OBJ models into TouchDesigner. Check out Crystal's article for a rundown of the process: 3D Scanning with Apple’s LiDAR - interactiveimmersive.io/blog/3d/3d-scanning-with-apples-lidar/
Does this work on Mac?
It should work just fine on macOS!
My operator is not outputting anything. I am on the same network with both devices
Can you try force closing the app on your phone and then opening it again? Also a good thing to try is temporarily disable the firewall on your computer and see if that starts working. On your computer you can try using NDI Studio Monitor included in the free NDI Tools that can help confirm if the issue is TouchDesigner or the whole system.
@@TheInteractiveImmersiveHQ thank you for your answer! In my case it was the firewall (as always:))
My ndi in top looks different with no borders and when i select it has a weird rectangular shape, and no purple borders, result i can't detect the source name, anyone can help?
Which device are you using and what operating system are you on? Do you also have NDI Tools installed? If so you can use the Studio Monitor tool to see if it's able to see the NDI stream as well. If you're not seeing the purple border, can you confirm you're using the NDI In TOP?
@@TheInteractiveImmersiveHQ ok, i'm running Big sur on mac, but no ndi tools installed (didn't know about it, oops) solved the purple border opening a new project. thank you hope i can fix it now :)
@@TheInteractiveImmersiveHQ Do you know where do i can find the ndi tools?
Still doesn’t work and with the studio monitor I’m not able get data, it’s because I’m using an iPhone 8?
@@jogi9206 Ok, so great first step is knowing it doesn't work in Studio Monitor either. That means the problem likely isn't in TouchDesigner. Can you confirm both the device and your computer are on the same network? It might also be a good test on your computer to fully disable the firewall in the operating system to make sure that isn't blocking NDI's network information. When you're in Studio Monitor, do you see any devices when you right click on the background? Im curious if it sees your phone but can't read the stream or if it doesn't even see your phone.
Is there any replace app for android?
Tough to say. I don't personally know any, but you could try doing a web search for "depth camera NDI" on the google play store. Also another person in the comments mentioned that one of the ZigSim versions might be available for Android, so that could be a good starting place.
@@TheInteractiveImmersiveHQ Thank you.
@@hadimcmxcv No problem, hopefully that works!