Blender & OpenCV - Windows Guide - Facial Motion Capture
Вставка
- Опубліковано 27 гру 2019
- This is a step by step guide to install OpenCV for use in Blender and configure the Vincent model to capture facial landmarks.
Github Repository:
github.com/jkirsons/FacialMot...
Download trained model lbfmodel.yaml (note - this is not for commercial use)
github.com/kurnianggoro/GSOC2...
Prerequisite Steps:
Open Command Prompt as Administrator
cd "C:\Program Files\Blender Foundation\Blender 2.81\2.81\python\bin"
python -m pip install --upgrade pip
python -m pip install opencv-contrib-python numpy
Then follow the Vincent model configuration steps in the video.
Citations:
C. Sagonas, E. Antonakos, G, Tzimiropoulos, S. Zafeiriou, M. Pantic. 300 faces In-the-wild challenge: Database and results. Image and Vision Computing (IMAVIS), Special Issue on Facial Landmark Localisation "In-The-Wild". 2016.
C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, M. Pantic. 300 Faces in-the-Wild Challenge: The first facial landmark localization Challenge. Proceedings of IEEE Int’l Conf. on Computer Vision (ICCV-W), 300 Faces in-the-Wild Challenge (300-W). Sydney, Australia, December 2013.
C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, M. Pantic. A semi-automatic methodology for facial landmark annotation. Proceedings of IEEE Int’l Conf. Computer Vision and Pattern Recognition (CVPR-W), 5th Workshop on Analysis and Modeling of Faces and Gestures (AMFG 2013). Oregon, USA, June 2013. - Наука та технологія
Thank you, mate! I'm not sure everyone realizes the full impact of what you've done here and offered up for public consumption. Seriously, it started to feel like a conspiracy was at foot, trying to piece all of this together myself, only to run into some missing crucial piece of information every time!
Excellent tutorial! I was able to get it working, pretty quickly. Now the goal is trying to get it working with my wireless camera. Thank you very much!
THIS IS GENIUNELY SO GOOD TO BE TRUE AND AUTHENTIC !!
THANK YOU
Alright! A "for dummies" version of this ;-). Amazing. Great work!
Thank you for this Video. It took me a few tries but I was able to get it working.
I got python script error will you please tell how i fix it?
Thank you, very usefully, it works perfectly.
Thank you sir. You earned a sub!
Thank you, it helped me a lot
thanks a lot man!! You are the best!
this is amazing , thank you !
excellent guide!
Hi! First, amazing tutorial! I was practicing and it's really cool. I have a question: I am a little outdated about performance techonologies. I have a NVidia Gtx 760 ti, which I know that is really old. What kind of NVidia Graphic Card should be enough for play with Facial Mocap faster? Thanks!
Thanks for your work ! What type of license do you have? Do you allow commercial use or ....?
Also, if I want to make it work in other rigs, not just Vincent, Do I have toI have to rewrite your code or are there easier ways?
Is it possible to use a gameplay (analog sticks) as a Input for recording animation ?
Maybe one stick fore the head turn and one fore blendshapes ore something like that.
Keep up the good work !!!
Thank you for this tut. It nicely works on Ubuntu 18.04 / windows 10.
Performance are very low (w10/Ubuntu) due to my very old pc, 2/4 FPS and low poly model:
PhenomII - 4gram - gtx650ti, video capturing from DroidCam on my smartphone.
I want to try install cuda-opencv for a new test.
Wow this was much easier than that other tutorial
Where I got nothing to work
Thanks! Simply , quickly, effectively! Will it work for the hole body?
thank you very much for ur great tutorial. if I download these dependencies will it affect my systems python as well or is isolated from my systems dependencies tht i downloaded previously?
Normaly, i do not commont on youtube videos. However, this approach is amazing. Definetely, this should be an integrated add-on within Blender.
Is it possible for you to develop an add-on?
thank you...its works well
Thanks a lot .
Just a question is there any doing full body mo-cap with this method?
Thanks for the tutorial!
Is it possible to capture not just the face, but also the hands movement? I would like to create something for sign language.
Hey! Great video! How can I configure my webcam? When I press the Capture button it gives me an error on line 203, something about Team Management and location unknown, everything else works fine. Thanks!
THK SIR ！ but i got 2 question here 1.why the eye ball do not Motion Capture?how to fix this ? 2. Does this script only work for the one character?, or will it also work with exported Adobe Fuse/Mixamo characters that come equipped with full face shape keys for blender?
got it working, ill become a contributor
It took a few days to set up this for me, but interesting is that errors of the scripts could be found by entering script in console line by line from the script file. Usually there are errors with imports and paths. I understood also that I needed to install exact version of Python on my Windows as Blender has and also I needed add Pythons cv2 folder to Blenders cv2 folder. And then animation goes.
Only thing i am interested also - how does the animation in Blender works with hands and legs.
Thank you for this great tutorial, I managed to get script running, Now I want to know how to export the rig with animation as fbx or gltf file? Somehow blender always freezes when I try it
Hello, I see you are using the landmark model lbfmodel.yaml. I would like to ask if I can build a landmark model from Tensorflow, and replace the lbfmodel.yaml. Is it possible? Or do you know any sources explained how lbfmodel.yaml was built? Thank you!
Followed all the steps and got up and running very quickly. However, the output seems really shaky and jittery. Do you think camera fps plays a part? I have some PS3 eye cameras that do 60 fps. Maybe try those?
Hey, thanks for this easy to follow tutorial, it is working perfectly fine for Vincent model. But I have an issue when I use this code in another model. I renamed my model's rig as Vincent's rig name and did necessary changes in bone's names also and now the script is working very well on another model. But the problem is, the movement is in opposite direction, for example while moving eyebrows up the model's eyebrow will go to downward direction, same thing is happening for the jaw and lips. I don't know how to fix this issue, it'd be great if you can help me!
It is necessary to have a face rig? or we can use blend shapes
Hi, very good video. But I have a problem. When I run it I get the following "name: 'bones' is not defined". But I have this on line 158, which is where it tells me the error is: "bones = bpy.data.objects["RIG-Vincent"].pose.bones"- Any ideas?
stuck 3:55 mark update the folder gl user name? where i have it saved? where do i find info? please & thanks on the help dumb down as can be
How do i record these facial animation and use it for other model in unity??
I've noticed that the lbfmodel.yaml is not for commercial use. How do I generate something that is for commercial use?
Is it possible to use a pre-recorded video instead of using a webcam? What should I do if possible?
It says
"Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'cv2'"
How come?
this project can be exported to UPBGE or Verge3D? or to which executable can it be exported?
I have no Idea how to do this on my model, although I want to test it so badly, I have hesitation to do it scared that it might broke blender (because I have many versions of blender extracted from zip file and I only know basic of phyton). I wish I have the wisdom and power to help you I hope my sub somehow help you. Eevie is really good complement for this. Studying blender the only best decision on my life!
When i click on capture blender starts hanging and a capture window open in that window some glitch type frames are there ..( i am using my phone camera as a webcam ) ... can you please help me to find issue
Very usefull but Do we need reinstall blender for the code?
Any idea what I could do against the massive jitter?
@Gadget Workbench How to apply this for our own model?
Hi
How I can use blender to capture hand , fingers and face live motion for sign language animation purposes
Does this script only work for the one character?, or will it also work with exported Adobe Fuse/Mixamo characters that come equipped with full face shape keys for blender?
Dude really wanted to know that
yeah you can retarget them onto mixamo etc chars eg using autorig pro -> remap
@@mareck6946 any tutorial video?
congratulations, but, I want to tell you, the Capture button that enables the webcam does not appear
is this worked on 2.79 blender version?
Hello good day. when I open the cmd and run "cd" C: \ Program Files \ Blender Foundation \ Blender 2.81 \ 2.81 \ python \ bin "I get an error that it cannot find the path
Does this only work with windows, or does it work with MAC too?
Thank you for this great video! I have one question. OpenCV seems to have some sort of twitchyness to it like its strugling to find and stay on top of main features of the face. Do you think this could be improved by using face markers for motion capture?
i know im late but do you mean physical markers on face?
@@roshanramsa794 Hi, no problem :) yeah like colored spots on the face for tracking, do you think it would help?
@@Consisto77 i don't think it would in the current code but i may be wrong. But if you are willing to put some colour on the face you might as well try out the official blender motion tracking
@@roshanramsa794 It been a while so I don't remember for sure, but I ended up manually tracking it in blender. But thanks anyways.
I get an error message: ModuleNotFoundError: No module named 'cv2'
can i use this with difrent kind of rig?
I am not ne able to download lbfmodel.yaml not downloading
if possible can you maybe make a video on how to do this on mac?
in the most recent versions of blender like 4.1 the .yaml are not supported, just tried doing it as of 2024 and it doesnt work, i really wanted to find a way to use OPENCV and RIGIFY
How can I apply this code with my fbx character that is rigged by auto rig pro? Is it possible? Because Vinchent has rigging python script. But it is only applicable for vinchent. What to do for my rigged character?
the openCVanim script does not work for me in blender and idk why. any help?
Will this work on any character?
please make tutorial to use this for makehuman character
please help its very urgent. when i try to run the OpenCVoperator the blender closes . how can I sort out this issue
HELP
Traceback (most recent call last):
File "C:\Users\1\OneDrive\Desktop\vincent.blend\Text.001", line 203, in modal
cv2.error: OpenCV(4.6.0) D:\a\opencv-python\opencv-python\opencv\modules\highgui\src\window.cpp:967: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'cv::imshow'
location: :-1
please help me solve the error path for yaml database is ok.
lbfmodel.yaml this is a binary code i cant download it like a file can you fix?
Thanks for the great guide, eventhough I fail to follow it haha. Any ideas what causes this error when clicking OpenCVAnimation button: WorkSpaceTool.setup(): error with keyword argument "options" - : 'REGISTER' not found in ('KEYMAP_FALLBACK')?
- Ended up skipping that script and just using bpy.ops.wm.opencv_operator() to run the other one, which works fine.
- Also had to do the int type definition for the cv2.circle x and y in order to make it work.
- In order to get this to work with a custom rig, I had to make sure that my bones (in Pose mode) were using Euler angles (in Transform tab) instead of Quaternions. This almost drove me nuts, spent almost 2 hours wondering why nothing was working before I realized this.
Couple things I can't get my head around:
1. The tracking points are really shaky and therefore causes issues for my face rig too. My webcam is 1080p60fps - is it too high resolution or fps? I tried to limit OpenCV fps but without a success.
2. I'm having really bad performance issues with my rig. Webcam preview runs prefectly fine, but my rig lags slowly behind it. (Simplify is enabled and tried to remove everything extra from my scene, having now only the head mesh of my character & rigs enabled).
3. Could you give any hints how should I calculate the model_points coordinates? That's atleast one thing that is off for me as I have completely different character in use.
I ran into the same issue but can't figure out how to solve this using the below steps, can you please help me here. Thanks
Looking to get some help on the landmark file path if you wouldn't mind getting back to me
blender 2.93 no capture video button shows
cv2.error: OpenCV(4.5.3) :-1: error: (-5:Bad argument) in function 'circle'
> Overload resolution failed:
> - Can't parse 'center'. Sequence item with index 0 has a wrong type
> - Can't parse 'center'. Sequence item with index 0 has a wrong type
Were you able to get this resolved
Python script fail in my blender
What i do
Why it doesn't work on an other 3d character
Can I import a video instead of webcam
When I click the OpenCV Animation tool in layout, python errors out with "ValueError: WorkSpaceTool.setup(): error with keyword argument "options" - : 'REGISTER' not found in ('KEYMAP_FALLBACK')". Using Blender 3.0.0 in Windows.
Were you able to get this resolved?
@@RohitSingh-zo8ci I didnt no. I gave up.
scripting problem：Tool 'ui_plus.opencv' already exists! anyone can help me out with this？
Python script failed, check the message in the system console
bpy.ops.text.run_script()
Python script failed, check the message in the system console
Python script failed, check the message in the system console
???
how FIX
Can you help with RIGIFY-RIG ? I can pay ofcourse. Im amateur user of blender. I have 100 models from CC3 / Iclone user. Thanks and replay pls. Have a good time
any one pls tell me what is this?
it worked the first time but my webcam wasnt updating so i restarted it and now im getting all the errors
So the capture has a lot of shake, resulting in shivering like animation!!
how to fix i dont see any data on C:\ drive..so "persistence.cpp" is problem?
cv2.error: OpenCV(4.2.0) C:\projects\opencv-python\opencv\modules\core\src\persistence.cpp:719: error: (-49:Unknown error code -49) Input file is invalid in function 'cv::FileStorage::Impl::open'
hi.
Have you fixed it?
I have a similar error
mine worked but doest have a camera button
Thank you but I have a problem with the lbfmodel.yaml file. If I use the raw version of your version, I get the message "python script failed, check the message console". After I run the OpenCVAnimOperator.py.
It is a syntaxerror "unicode 'unicodeescape' error codec can't decode bytes in position 2 - 3: truncated"
I added: r, open and double slashes but nothing worked. So I can't get capture under opencv animation.
How can I fix this?
However in object mode I see "openCV Animation" but not the button "capture"
I followed every step
Same problem I don't have the button "capture" and when I click run script of the OpenCVAnimOperator.py file blender crashes!
did you solve the problem ??
@@ranijbeli did you solve your issue coz I'm getting it too
it is not working what should i do ?
please give code for adding video instead of web cam, i added cv2.VideoCapture("video.mp4") but its read only 1 frame .
you should use while loops
Anyone get this to work on blender v 2.83.2?
I downloaded 2.81 and got it working by following the directions. It's not very accurate though.
To change the Blendrig Character then you just need to change this line to be the root node of the armature: @t
Python script failed
Could anyone help? Trying this on a mac, downloaded and installed the pip and the scripts. I see the capture button, but Blender (2.9) keeps crashing, also when I run it from Terminal. Should I install python 3 instead of the basic 2.7, cause I thought it would run from within Blender.
It doesn't run it from your C python. Blender has its own python version and it's 3.9. else, I don't have mac, so best I can tell, remove and start from scratch.
@@Mrationality yeah, I worked a bit on it and run it from terminal during my tests
Blender still crashes, but I get an error(something like this, I am not on pc and it’s been 3 days lol): -invalid driver
And it’s says that the data for real-time animation is not found for the lips (R and L)
Can I fix this? I downloaded the new scripts that someone commented a link to...
Can you make video with full body motion capture?
ps Hello from Russia with Love
I have python 3.10 on appdata folder. And after I installed Blender. OpenCV is installed with and on system's PIP (and folder). Now, I can import OpenCV (cv2) with Blender's python.exe on CMD, from Windows CMD Commander Prompt I can see Blender's Bundled Python version is 3.9.x .
THe Great Problem is: I can't import cv2 in Blender's Python Console which embedded in Blender GUI. How can i solve this problem with existent system Python?
WARNING: I'm not Expert. If your Personal Computers get damaged, it is your own risk. I'm not fucking responsible.
I'm just mad about spend a lot of hours to solve this. And Gadget Workbench it is not your fault, thanks for the tutorial.
Solution: I removed python 3.10.x. I delete folder adresses from PATH's for pip and python. I add path Blender bundled python's folder adress to PATH. Remove all residuals from Appdata/Roaming all about Python.
Install all packages opencv, imutils etc.
Where can I change PATH??? Windows+R, run sysdm.cpl. Advanced ==> Enviromental Variables ==> I changed both sytsem variables (Path) and User variables (PATH). I also changed PYTHONPATH to blender's python.exe folder.
where is pip.exe? ==> ~python\scripts
WARNING: I'm not Expert. If your Personal Computers get damaged, it is your own risk. I'm not fucking responsible.
NOW A NEW PROBLEM ARISES FROM MUYGUN'S ASHES!
COULD MUYGUN CHANGE SPYDER'S KERNEL TO BLENDER'S BUNDLED PYTHON?
Solved: Just I need install spyder-kernel
python -m pip install spyder-kernels==2.1.*
Can you motion capture yr body and make an animated movie more realistic?
do a version for 2.9?
( python script failed, check the message in the system console ),I get this message, what I do ?????
Do you get the answers
I had the same problem, but I had the file name of the YAML messed up. Then it worked fine
Blender versión ?
Guys you can also use your phone as camera if you dont have good webcamera. use droidcam for this and change cv2.VideoCapture(0) in OpenCVAnimOperator.py to VideoCapture(1)
i didn't have to change anything and droidcam worked for me... hmmm
No pip error ..help
How this can work with other characters
use the blenrig addon! its the same used for the model showed in this video
@Sterry Ks do you get the answer?
@Sterry Ks okzz
@Sterry Ks here is link for blenrig 5 for blender
cloud.blender.org/p/blenrig/5cb14ebd808c0e07cdde1d03
i can't run the script anyone can help me?
C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\bin>python -m pip install --upgrade pip
sir ibfmodel not directly download ???
You can open the repository, click on the file, right click on "View raw" then "Save link as..."
How can we keyframe this facial animation in blebder ????
would be interesting to know a smooth workflow for that :)
i tried but the second script fails to run, keeps telling me "Python script failed, check the message in the system console
"
i don't know why, thouug i can click on the camera opencv open, but it does not capture anything, its just a plain camera, no points no movement
Same issue!! were you able to solve it?