iPhone ARkit & Depth Sensor in TouchDesigner Tutorial - Part 2!
Вставка
- Опубліковано 31 січ 2021
- Get access to 200+ hours of TouchDesigner video training, a private Facebook group where Elburz and Matthew Ragan answer all your questions, and twice-monthly group coaching/mastermind calls here: bit.ly/2zJ0lmt
Picking up where we left off in part 1, we dive deeper into Zig Sim Pro and look at the capabilities it has when it comes to it's depth sensor and ARkit features. Not only do we bring the channels into TouchDesigner, Elburz shows you a few tricks on how you can use the depth data to displace a grid geometry, and how you can simply take channels from the ARkit face tracking and use them to control elements inside of your network.
ahh the amount of possibilities with this!!!!! awesome video :D
Thanks! If you make something, feel free to tag us on Instagram!
Awesome Video that's really very cool!
Thanks for watching!
Elburz‘ wink! Lol!!! Thank you so much for sharing it!
Haha, I thought the tongue out was pretty funny too!
Really nice tutorial. I’m just confusing about after I putting the data of normal map into the phong, that enable height map still cannot be clicked. But when I use undo, that the enable heigh map button is available. However the image in geometry comp will not show up anymore after I turn on the heigh map. Is there any solutions for that? Thanks a lot!
Thank you
Of course! Thanks for watching :)
awesome video. looking forward to digging into more TD this year. what to check when the source name doesn't show up in the NDI flydown?
Hey Bradon! Usually I'll force close and restart the app on my iOS device. Sometimes it seems like opening and closing it can trip up the NDI stream. If that doesn't work, I'd suggest trying with NDI Tools to see if their Studio Monitor can see the stream (if TouchDesigner cant), then from there trying to disable the computer's firewall temporarily. Let me know if those work!
@@TheInteractiveImmersiveHQ looks like a case of getting tripped up on the app, just closed and restarted. works great now! this was a cool tip. very fun to play with an iphone feed in TD.
@@bradonwebb2539 Great! Ya it's a little bit annoying but I'm glad that trick makes it work again. There's definitely a lot of potential !
Hey! Great stuff! I just wanted to know if it is possible to get an output of the depth sensor camera that I would be able to map to something else. I want to know if I'm able to use this like a kinect so I can use simple movements such as moving my hand closer or further from the camera to control to control MIDI information. Thanks in advanced!
Yup the depth camera works very similarly to kinect so you can crop areas of interest where the hands are and get the average value to use that to determine how close or far the hands are. Is that what you're looking for?
Hello Elburz!
So I am wanting to achieve something similar to a face-filter effect. I'm wondering if there is a relatively straightforward way to somehow take the 3D data from ARKit using Zig and re-create the face 3D mesh in TouchDesigner that then can have various textures applied to it?
Hmm that's a tricky one, I think it depends on how much processing power your computer has. Unfortunately we don't have access to that really nice mesh that's seen in the previewer, because those are native processes happening in iOS. Two ways you could approach it would be one to take as many of the CHOP channel data points as possible and then shuffle them all into TX, TY, and TZ channels. Then you'd feed that to a Limit SOP to create the points in SOP space. You could take that and then use something like an Add SOP to close the geometry or maybe a Skin SOP to skin it. Alternatively you could try to take the depth map, use thresholding to trim only a specific distance range for where the face will be, then you could either use a Trace SOP to trace a geometry (but it will be flat). Alternatively you might be able to cheat by having a flat grid geometry and then trying to use the depth map as a displacement map to warp your grid to the face shape. There's lot of options, most of them need some experimentation. Hopefully that helps!
Hi Elburz! The OSC in works in the previous tutorial set up. But the NDI in doesn't find my iphone with the zig Sim pro¿? Any hint what could be wrong? Same settings as the Zig sim regular.
One thing to try is to force close Zig Sim and try to open it again. Sometimes it seems like if you open it and close it too many times the NDI stream gets frozen. Also make sure you're on the same WiFi network as your computer. If those don't work, try temporarily turning off the firewall on your computer and see if that helps.
Amazing
Thank you!
@@TheInteractiveImmersiveHQ no, thank you!! I’ll be using this technique to trigger samples in response to face movements for my uni dissertation
@@VorosMedve That's awesome! We'd love to see how it comes out if you post it on Instagram or somewhere else :)
Thanks for the video! The only issue is when I add the height map, my geometry becomes all black..? Other than that, I'm excited to keep experimenting with the Zig Sim Pro.
What system are you on and what GPU does it have? Are you on macOS or Windows?
I found that adding an attribute create after the grid and turning compute normals on will solve this.
@@jaimeforson7712 Yes! This is a fix for macOS. Although I turned on Compute Tangents to make it work @The Interactive & Immersive HQ
How can I emit particles from this deforming height map grid? If I plug in the particle sop it will just emit from a single point.
Try using the Normal Map or Height Map textures as the particle emitter source for the particlesGPU (available in the Palette). particlesGPU allows you to use a TOP texture to emit particles, instead of having to convert it to SOP geometry.
Hello Elburz!
I have the ZIG SIM PRO app, and it works with the OSC signal in TouchDesigner. However, when I try using NDIin to incorporate the camera and LiDAR sensor, it initially works fine, but after approximately 10 seconds, the signal cuts out and freezes. The error message displayed is "Warning: Unable to find the selected source name, fail-over source may be used instead. (/project1/ndiin1)".
I have tried various troubleshooting steps, but the issue persists. I am using an iPhone 12 Pro and my setup includes an AMD Ryzen 9, 32 GB RAM, and NVIDIA GeForce RTX 3080 running on Windows 11.
Any assistance you can provide would be greatly appreciated.
Great question - the first thing I would check is that you're running the latest version of TouchDesigner and that you've downloaded any available updates for the Nvidia GPU driver. Looks like some folks were having a similar issue with an older version of TouchDesigner last year, but the built-in NDI SDK has been updated since then
@@TheInteractiveImmersiveHQ Thanks! I've already updated TouchDesigner to the latest version, and my GPU drivers are up to date as well. However, the issue still persists. The only possible culprit I can think of might be my second hand iPhone 12 Pro, as when I lower the fps to 10, it seems like the problem occurs a bit later... It's frustrating because the output is stunning, but I can only use it for a few seconds... Thanks!
Thanks for video! Everything works except ndi... I do not see my iphone in ndi in top... why can it be, pls help =)
Hi Alexey! Can you try force closing the app on your phone and try reopening it and enabling NDI once? I had sometimes when I turned it on and off a bunch it wouldn't work properly and I had to force close the app.
Thx for nice tutorial
But i have some problem from 6:34 my geo node turns out black what should i do?
Are you running on Windows or macOS? Also what kind of GPU does your system have? Some of the rendering parameters need to change based on those and there's certain features that macOS doesn't have yet in TouchDesigner.
@@TheInteractiveImmersiveHQ i use win but i had solution that i use sop attribute thx!!
Hi there, for some reason I am unable to get any video feeds into TouchDesigner. I am unable to select anything from the source name list, even though the OSC operator is clearly picking up a signal. Everything else works fine, it is just the NDI that doesn't work. I am using the iPhone 13 Pro Max with the ZIG SIM Pro app. I have tried connecting to different networks (both 2,4Ghz and 5Ghz, wired and wireless. With wired I get about 400mbs with a 10ms ping), tried different ports, connected my computer to a hotspot on my phone, added all TouchDesigner related apps I could find as an exception to my firewall, even turned off my firewall completely - but nothing. I am completely out of ideas why this does not want to work for me.
A good thing to try is see if the Zig Sim app can send NDI to NDI Studio Monitor on your computer. That would be a good way to see if this is an issue with TouchDesigner or the larger system. One thing to test as well is to force close the Zig Sim app on the phone and re-open it fresh.
@@TheInteractiveImmersiveHQ Had the same problem. Closing the app and reopening helped me. Thanks.
@@markus_knoedel Yup, it seems like the way iOS handles background apps can trip up / close the NDI stream and jam it up a bit. So force closing and reopening the app is a great approach.
Does anyone have any luck trying the rear depth camera? I have the ipad pro with a lidar sensor which would be really cool to use like this, but the app crashes any time I try to send rear depth data... :(
That's unfortunate, I don't have a new iPad with that feature to test. It seems like best thing to do would be send the company a quick message at their website and see if they've seen the same issue: www.1-10.com/
Alternative would be if there's a way to "Contact developer" directly from App Store to report the bug.
Why my NDI Image Type couldn't choose both?
That is a limitation of the ZigSim application unfortunately it's not in our control.
I can't see anything from geometry.
If i turn on height map of phong,
This might be due to the geometry not having tangent attributes - try adding an Attribute Create SOP directly after the grid1 SOP (and before the Geometry COMP). Within the Attribute Create SOP parameter window, turn the “Compute Tangents” parameter to On. Does that solve the problem for you?
Hi，I drag the normal map into phong it turnning black.....
This might be due to the geometry not having tangent attributes - try adding an Attribute Create SOP directly after the grid1 SOP (and before the Geometry COMP). Within the Attribute Create SOP parameter window, turn the “Compute Tangents” parameter to On. Does that solve the problem for you?
@@TheInteractiveImmersiveHQ Big tks for response!!!!problem solved!!!!
😉
Silly in the name of education! Worth it haha
@@TheInteractiveImmersiveHQ Always!
Suddenly feeling stupid about the grid columns/rows 😱
Don't! It's a VERY common thing that everyone does. You just are so used to saying width first and height after that you assume it's the same pattern. It's a totally understandable and natural mistake unless you REALLY look at the computer monitor for too long! haha