BlendArMocap - AR Pose, Hand, Face Detection in Blender Beta Release
Вставка
- Опубліковано 11 лип 2024
- Requires Blender 2.9+ or 3.0 with elevated privileges.
Get BlendArMocap:
github.com/cgtinker/BlendArMocap
Report Bugs:
github.com/cgtinker/BlendArMo...
Want to support?
/ cgtinker
Timecodes
0:00 - Introduction
1:02 - Start Blender with enhanced privileges
1:56 - Add-On installation
2:53 - Hand, Face and Pose Detection
4:17 - Rigify Rig
5:05 - Transfer Animation Data
5:50 - Clean up results - Фільми й анімація
Thanks so much man, you're doing so much for the community, i hope you get the exposure/recognition you deserve
Man, this is impressive! Thank you so much for sharing this progress!
The work you do is AMAZING, thank you so much for taking the time to make something like this for the community :)
Thank you soooooooooo much for the work you are doing! You are giving us little guys such a great boost in our work, and I will DEFINITELY have you in my credits after I finish creating my show.
Aaaare you freakin' kidding me...you and your work are absolutely inspiring and i really hope you'll get everything you need to continue this journey! This has a lot of potential.
Amazing!! I just discovered this great addon and is just what I was looking for. Also I have use OBS with the Virtualcam feature to use a pre-recorded video as input, then changing the Webcam device slot and it's brilliant, works great. You really deserve a honor place in the Blenderhalla! Big thanks for your work!!
Absolutely astonishing work, this is amazing. Thank you for your contribution to the community!!
Just saw this, amazing!!! I love your work, especially the BlendArTrack one.. Going to put this to good use since ive got a small project going....
Good lord, this is a game changer. You are amazing! And very generous to release such an addon!! Thank you!!!
That´s really an incredible work you are doing.
This is amazing. Thanks so much for creating this. The potential is enormous, to say the least.
dude, I just downloaded this, I will donate in the future, thank you for your work, sincerely
I'm a Senior year college student and I just found out about you today. You are a huge inspiration and a huge help! Thank you for helping the community. I was struggling since most apps I wanted for motion capture are restricted to iPhone and I have a Samsung Galaxy Note 20 Ultra. I hope I can use your app in some way to help me with my project, but if not, no worries. Your app is amazing and it works amazing on my phone. Keep it up!
I can't wait to give this a try, this may be exactly what I need! All i really need is head movement mouth open close. I'm so stoked
Thank you so much for your contributions to the community!!
This is amazing. Can't wait to put this into use!!
Awesome, I tried, great add-on, great tutorial 👍🏼👍🏼
Thanks
Thank you for creating this masterpiece!
Wow thanks so much man I've been trying to do this for so long
Dude you are awsom. Espacially just the plane tracking is so usefull. There is nothing like this in the whole community.
It's great you showed the install. Thanks.
thanks, this is amazing technology and I'm glad your putting this up for us
this is promising man..!! would love to try asap.
Thankyou dude this is focking awesome. Thankyou for your hard work
You do amazing work!
This is amazing! Thank you so much man!
You sir are a god. Thank you for YOUR hard work and give us blender the opportunity to do all of this. I will this for my video game big time as we are all seemingly one man teams ahaha. THANKS!!!!
Thank you sir for your hard work! This addon is incredible!
Thank you!
Will soon update and hopefully release soon (still beta state) planning to improve the transfer quality a lot
Bravo! Well done, sir!
😎 I just purchased your Blendartrack app! Thank you my friend for all your excellent work. Best Wishes
thanks, hope you enjoy using it :)
Amazing work!
You deserve a place on top of MoCapMountain. Absolut amazing man! Thank you for sharing.
Heck yeah man! Thanks for this!
Joining your patreon now
Lovely, thank you :)
You are my first patreon :)))
Great work. I will try this
Мужик, твой аддон просто находка! Давно ждал, когда кто-то сделает аналог Live Link Face для Android! Спасибо Вам за Ваши труды!
someday i will try your stuff n give you credit or else.. u are one of the most smartest bro,, thx u
Amazing Work !!
Amazing!!! Well done.
You are a blessing to the blender community
saw the World Blender Meetup presentation and I'm looking forward to trying it
nice to hear :)
Hey bro like your work and effort you are doing well
Would love to help support this as well and will be checking it out, nice work!
I think this add-on will save lives.
Excelent, You Deserve a lot more
Thanks for this great addon I was looking for exactly something like that. thank you you really help me a lot thank you very much
oh man really thanx for this gift
Amazing job!
This is DOPE brother! Thank god we have such a smart guy in Blender community. This add-on came in the right time for me. I'm a beginner user trying to make some cool stuff in music videos, and this tool is great!
But I'm noob about the drivers and it would be great if you include these offset settings in the add-on too!
When I have money, sure I will give you some support!
Thanks a lot - I plan to make this process easier in the future so just moving around empties will be required. But I wanted to get the toolset out there as I already worked on it for a while
@@cgtinker I really appreciate brother, thanks a lot! And I have a question: can I use a video file to make the capture? I'm asking because If I need to use a video file of a dancer or choreographer of another place it could be useful, since I can't dance 😂
@@rickmarques10 not yet - I plan to make that possible soon!
@@cgtinker looking forward to it!
@@rickmarques10 Isnt it possible to stream video to webcam output?
You sir.... are a Miracle.
Very cool! Thank you!
man,, u r simply amazing,, thx u
Thank you, follow you from Cuba.
Thanks for following!
Never have been there, but I guess it's beautiful over there isn't it? :o
Would love to live close to the sea...
amazing addon!!!!!
Thanks for your effort
love this you awesome man
huge thanks this is amazing :D
just thank You man you are amazing .... Thank you thank you thank thank you
Thaaaanksssssss amaaaazing work ... Cant stress this enough 😭.. we felt leftout from all those rokoko suite wearers ....
Haha yeah! Thanks 🙏
It still has a long way to go but it's already a lot of fun. Hope you can make good use of it already :)
so impressive 😮
Thanks nice video
This man is awesome
Amazing !
Wow no more deepmotion...thank you so much :)
Brilliant !
amazing!
Genius!!
This is amazing,!
Thank you, Sir
Mind blowing...
Muchas gracias
woohoo, awesome!
This is fucking awesome man!
That's cool!
Amazing
It is working, waoo
NICE 🎉
Thank you your work is very precious. I wanna ask you if there is a way to use this rig with the addon auto rig pro cause when I try it doesn't work
awesome
A quick note about the Linux case, since /usr/bin is surely already part of the PATH variable you can use the command: "sudo blender" from anywhere.
Yep guess usually it is - I actually don't have it added, so I included the path. :P
..I am not to much into linux - thx for the info, now I know what's wrong as it worked in the past
Will you consider to implement a local tensorflow or torch backend and allow people to train the model themselves?
Kudos for being the first person to do this though this is exactly what needed to be done to where we could have face tracking hand tracking and pose tracking right in blender so when you're animating all you do is start the add on and you don't even have to animate that part but it's just gotta be a FACECAP quality otherwise it's kind of a waste of time because we can't use this...... but the hand tracking looks perfect on the video when you're tracking it...the face tracking not so much and I couldn't get the body tracking to work
facecap is blend shape based as far as I know - that's a different concept and currently only possible using an iOS device.
It's a lot work, tweaking etc to make a one click solution possible. Soon quality will improve drastically.
@@cgtinker Yeah I really appreciate your effort.....you're taking the absolute best approach having it in Blender like you do if you master this and can make it... I meant as perfect as FaceCap is...at that level...then you could charge like a 100 bucks a pop for it on Gum road....just like they do for the FaceIt add on... And just clean up... just make a whole bunch of money for yourself cause everybody'll buy it cause you are taking the absolute best approach... I'm looking forward to your Future successes and if You do master this all the way as good as FaceCap(that level of quality)...everybody. .... will be happy to pay the 99 bucks for it 😎'good luck my friend
@@toapyandfriends interesting point. I didn't focus too much on the face yet. My current update plan is:
- no more arm stretching when pose tracking
- fix hand rotations (restrict to up/down)
- add y-finger angles
- add relative finger offsets
- improve mouth detection results (no more offsetting)
- add eyebrow tracking
- and lots of bug fixes
I am strictly working with bones and cloud data - so the results are based on points in space. With blend shapes face animation is way easier as you can preassign values and the animations are based on a single float - not based on vector math.
I think the face results will be decent but not on the level of a blend shape based application. I plan to develop one as soon it's possible either on desktop or android. However, this isn't too much about face tracking, it's only a part.
I prolly should charge a price - dunno. getting like 0.25$ per download won't help me to keep it up for ever, that's for sure :D. however the beta helps me to get most of the stuff right and prolly I find other ways to earn enough money to keep it up.
ur Crazy man 😍
👏👏
this is amazing thanks very much, i have one question tho how long can this becorded? mine is only 30 seconds and that's it Thanks
VERY IMPORTANT, PLEASE READ IF YOU ARE HAVING TROUBLE INSTALLING DEPENDENCIES! Only for Windows 10 though.
This worked for me, and it might work for you too. If you have any errors with installing the dependencies, follow these steps:
1. Make sure Blender has admin privileges, of course.
2. This is the most important step. Go to "System Environment Variables" and in the "User variables" section, find the path that leads to a python install. If this leads to a python install not within your blender directory, select it and click "Edit."
3. Now, you should see one or two paths that lead directly to a python directory. Leave that tab open. Then navigate to your blender python directory (...\Blender\2.91\python), copy and paste that into the variable leading to just Python. Then for the path leading to "Scripts" just go to "...\Blender\2.91\python\Scripts" and paste that into the variable. Now install dependencies. I might have explained this strangely, so please reply to this comment with questions if you have any.
Thanks, good point. I prolly add this in future setup tutorials 👍
i couldn't under it, please help me, if possible share sm tut
@@legendearn8570 what?
awesome but In order to cover the entire body, do I need to repeat this on the hand, face and body? How do I combine them all into one person and one time line?
Great work!
This addon takes videos from the phone camera or do i need an app to make the videos for the addon?
Blendarmocap works with a webcam and runs locally
Wow! I'm kind of amazed that it took this long for someone to make an open source app like this, so thanks for being that guy! The blender, and open source community thanks you 😀
Hi, thank you so much for this! I'm not a Blender user, but I think I'll learn it now. Let me ask you, is there a way to retarget that Blender Meta Rig to a Unreal Engine Metahuman? Or do you know if it's possible to bring a Metahuman rig from Unreal (or the standard skeleton rig) and use it with your BlendArMocap? Thanks!
hey there!
cool, glad you join the community!
Retargeting may work with some effort - never worked with Metahuman characters but I'll make some tutorials (also covering retargeting) as soon I'm done with updating blendarmocap :)
@@cgtinker Awesome! Thank you!! :)
6:19 , the desperation in your eyes in pronounce those words :D eheh
This is incredible. I came across this by looking for videos on motion capture in Blender. I do motion capture, camera tracking and matchmoving professionally, but am surprised by the lack of material on it. Do you perhaps know anything about mocap in Blender and using witness cameras?
Maybe. What you plan to track? Camera movements, Face, Bodies? Blenders internal tools mainly focus on camera motion tracking and can deliver great results.
Human motions can be tracked with lots of effort, but third party tools are easier..
This is really awesome ...my compliments you're a genius...coiple questions
1. Can you do a tutorial on how to skin it to a character
2. Or would you explain how to save it a a bvh, or fbx file for use in any human character
Thanks! ATM I am working on an update, as soon I am done I plan to do some tutorials :)
But I don't have webcam. How to import MP4 video into the add-on manual. Not from webcam live? Please reply...
Very cool; it's great that you're working on this!
Is there an easy way to bake these animations into actions? And perhaps a workflow of "get the animation in, bake it, then fix the rotations, etc., using the animation tools" might be more straightforward than the "tweak the driver constants"?
I kinda like drivers but you are right, I am planning to create a setup in the future which just requires to move around empties which will be much more convenient. But I wanted to get the beta out as it's already in process for a while.
Basically only constraints are used so baking shouldn't be much of a struggle. I didn't actually test it but blendartrack has a bake to action feature which probably works just fine
@@cgtinker Thanks for the quick reply! The main thing is that for these animations to be usable in projects you do need to, in the end, convert them to actions, so you can layer them on the NLA, edit them, concatenate them with others, etc. You can't, then, have your rig be constrained with the BlendAr constraints.
I think something like "ok, now that I see the animation is correct, bake it and remove the BlendAr constraints" will need to happen (at least for me) to be able to use this on real stuff - games or animations.
@@cgtinker And, yes, thanks for putting out the beta early! :-) Cheers.
@@t----m thanks for the input, I'll cover that in the future! the constraints only copy values from the drivers so it can be baked - no worries. The huge benefit of this setup is, that it is possible to use basically any rig because the driver data is available on the empties (at least if it gets properly linked and the rig includes ik chains).
However, the stretch constraints from the rigify rig cannot be baked but there are options to disable them entirely - so it is all doable :)
Thank you for doing so much for the community. Unfortunately I get an error when attempting to run the "Install dependencies" with enhanced dependencies. You have an idea what might b e going wrong? I have now tried with Blender 2.9 and 3.0,
Usually it's an environment error on windows - I am working on a fix, will be up soon
heyyy so i got a question i wanna convert this animation rig that've made with the programm ( the movements ) to another type of rig, "The Sims 4" one, is it possible?
it's possible to retarget rigid animations to any rig, but it's quiet a bit of effort. check out this video by cgvirus:
ua-cam.com/video/qnJFOSisDLQ/v-deo.html
Hi, is there a way to use Videofootage instead of the webcam?
Thanks for this service. But why I connect to the pose and hands? Is there any solution?
Bruder man hört das du deutsch bist 👌🏽
mist, muss mehr üben :D
This is amazing! Will there be a way to record the pose and the face at the same time with a head mounted camera in the future?
recording pose and face at the same time is possible while using the holistic option (will release a new version + vid tomorrow). I don't really get the "head mounted" camera point..
@@cgtinker Wow! Cant wait! For the camera, big budget mocap studio suits have a camera attached to a helmet. The camera can then record your face while you perform. No matter the pose, the face always stays in frame. I was planning on making my own low budget mocap face/body capture setup. And your addon might do the trick!
@@2121nath nice! yea it should do the trick :)