Real Time AI Face Landmark Detection in 20 Minutes with Tensorflow.JS and React
Вставка
- Опубліковано 19 вер 2020
- Ever wondered how snapchat filters work?
Three words...facial landmark recognition!
Facial landmark recognition allows you to detect a number of different points on your face that together make up your eyes, mouth, ears, nose and so on. From there you're able to apply overlays aka filters to get your snap! But there's so much more that it can be used for like emotion analysis and face tracking.
Sounds a little difficult though, right? Well, in this video you'll learn how to build your own face landmark detector app in 20 minutes using Javascript and Tensorflow.JS.
In this video you'll learn how to:
1. Install the Tensorflow.JS facemesh model
2. Build a Javascript React.JS face landmark detection app
3. Detecting landmarks on your face in real time using your webcam
Resources:
Tensorflow.Js models: www.tensorflow.org/js/models
HTML Canvas: developer.mozilla.org/en-US/d...
Create React App: reactjs.org/docs/create-a-new...
GitHub: github.com/nicknochnack/Facia...
Oh, and don't forget to connect with me!
LinkedIn: / nicholasrenotte
Facebook: / nickrenotte
GitHub: github.com/nicknochnack
Happy coding!
Nick
P.s. Let me know how you go and drop a comment if you need a hand!
Music by Epidemic Sounds - Наука та технологія
One of the best Facemesh tutorials I've seen so far online. Keep up the good work.
YESSS! Thanks so much @Thang!
why dont you have more subs/views? youre amazing at explaining your code and actually seem passionate unlike most coding tutorial creators. keep going!
Thanks soo much, I've only been going at it since May so we're only just getting started! But plenty more to come!!!
I haven't even started watching and can already tell you deserve more views.
🙏 thanks so much @Rav Ada! Getting there!
Oh yes!
WHOA YOU WILL RECEIVED ONE MILLIONS OF VIEWS THAT AMAZING CONTENT RIGHT HERE
Hell yeah, that's the goal, thanks so much 😊!!
Best 30 minutes spent in my life
🙏🙏🙏 thanks so much @Alfredo!
Absolutely great tutorial...And believe me, you will get dozens of subscriber soon..
All the Best!
Heya @Sufyan Khan, thanks a million! More tutorials to come 👨💻!
You are AWESOME!!! Your tutorials are easy to follow and I love how you break down the concepts as well.
As a point of feedback, I would love if you could mention some possible applications for noobs.
Thanks again,
Subscriber!
Solid suggestion, thanks @Tony!
As a backend dev, it wasn't difficult to understand the front-end part.
I like your teaching style. 👍
Thanks a bunch! So glad you enjoyed it.
Great content man, thank you !!!
You deserve more subs dude. I'm on your team now
Hell yeah!! Welcome to the team @Nico Leiva!
This is an amazing video, what could make it even better, is possible applications for the landmarks, for example, detecting certain dots, and based on it's movements, detect micro-expressions, that could be interesting
Thanks a lot man! looking for more quality content like these
Ayyy, cheers! Thanks so much!
This was super helpful thank you very much
Bro thanks very much for all these tutorials.
best tutorial ever
Thanks so much @mustafa! Definitely, in the works!
Best Face Landmark Detection + Coding Tutorial that I've seen!
Thanks so much @Timothy! So glad you enjoyed it. Same Timothy from LinkedIn?
@@NicholasRenotte Yes! :) It's me. I just sent you another epic video idea.
@@TagenigmaLLC yesss, alrighty will check it out!
Thank you friend for sharing the teachings learned here will be of great help for my projects
Dope tutorial, keep it going :)
Thanks so much @Diego Caumont! A bunch more coming. 😄
Thanks so much for your effort!
Plenty more to come @Panda! 🙏
Thanks for sharing!
Thank you to make me gradueated
great stuff man!
Thanks so much @scott!
You're the bestt!!!!
I try your GitHub repo and build in 1 command " npm i " and it's working so amazing :)
Awesome!! Glad you got it running so easily, YESSS!
wow sir, i love the Beginning ||||
Amazing tutorials!
Thanks so much @Jerry LEE 🙌 !
The key points for me was to switch to video.videoWidth instead of video.offsetWidth & second one is resetting canvas width and height for each loop. Thanks for the simple learning
Anytime! Awesom suggestions.
Love from India.... Hoping one day I can also figure it out by myself...... Ur videos are awesome..... Explained well..
Awesome! Let me know if you need anything along the way :)!
nice, thanks for your videos, i will learn
i from indonesia
Really awesome video.
Thanks so much @Vijay!
Good tutorial! ❤️👍🏻
Thanks so much @Tennyson 🙏🙏!
Great !
love your videos
Thanks so much @Mantas!
Is there any way to export this data into 3D software like blender?
Not too sure about pushing out to blender but you could definitely export the x,y and z coords for use elsewhere? I've seen it used alongside three.js for 3d viz @KriGeta.
@@NicholasRenotte sounds intresting
Love it!
🙏 thanks so much @Laura Estupiñan!
I would love to see a continuation of this project. Maybe create a filters of your own? Rabbit ears, whiskers, etc...
Yah, working on it as we speak. Trying to get the hang of 3d rendering for it!
@@NicholasRenotte If you have a patreon, buy me coffee.. or any sort of donation links I will personally donate 100 dollars if you cover how to make a snapchat like camera experience in terms of putting filters on ones face.
Heya @@haibertbarfian6343, I have a Patreon but no need to donate for this particular use case. Got it on the list already, what types of filters are you thinking of? Background removal, adding sunglasses etc?
@@NicholasRenotte You are a SAINT MY FRIEND!!!! I was thinking of a snapchat like experience where you can put sunglasses, and all sorts of fun filters on a face using react native. Many people have joined the RN community and we need to show the prowess that RN has obtained over the years.. if snapchat filters are possible in RN.. almost anything is! I know that currently there are SDKs that allow this functionality on WEB.. but no one has done it on RN for mobile apps... this is going to be bananas.
@@haibertbarfian6343 hell yesss! Definitely, I'm going to get to work on it soon!
Found your channel today! Great content!
🙏 Thanks so much @Yona, anything you'd like to see?
@@NicholasRenotte Yeah! I would love to be able to work with the detected landmark points and trigger certain areas (mouth, eyes). Something like gaze recognition or emotion detection.
@@yonakislinger6301 awesome stuff, was just planning iris detection today!
@@NicholasRenotte Amazing man! Really cool what and especially how you create!
@@yonakislinger6301 thanks so much, honestly appreciate it a ton 🙏!
Looking forward to converting this to Angular!
Yess, let me know when you do, would love to take a look at the code @Libby!
Thank you so mush Nicholas, this is a great content, so much valuable. I have just one confusion that why the Triangulation matrix is used and how did it formed ?. Again thank you so much.
Amazing thanks for such great videos, Can you please tell me how to just extract eyes coordinates
Awesome! Great tutorial. Are you planning for more tutorials using Posenet and React JS? (Using Teachable Machine)
Thanks so much @Haesun Shim, I sure am! I've done a Posenet basic video but have a few others planned with Tensorflow.JS as well: ua-cam.com/video/PyxsziqatFE/v-deo.html
Canva has a delay in relation to the video component. Hide the video and add the camera image to the canva, along with the points on the face. Congratulations on the video!
Awesome suggestion @Diego!
Your tutorial is very good and I understand it completely. Thank you so much for sharing.
How to use landmark coordinate point to generate animation avatar?
In other words, how to use landmark coordinates to compare with virtual components such as eyebrows, and then select a matching eyebrow for image assembly;
Working on this atm :) will share a tutorial once I'm done!
@@NicholasRenotte Is the tutorial out yet?
Hello! I loved your video, it was very helpful and easy to understand. Sorry for the inconvenience but would there be some way to recognize the face of a specific person and allow only this person to log in? Like in the phone?? Thank you very much for everything
hey Nicholas! this is so cool! do you think I can use this on a tiktok clone I already built using dart?
Excelent tutorial as usual. Do you have a tip about how to implement liveness in face detection?
are you able to implement
Greate stuff :-)
Thanks so much!
This is amazing! There is no video out there showing an actual implementation of face filters using react native. Do you think you would be able to pull this off? I believe that is a highly coveted topic and there are absolutely 0 videos on it out there.
😉 stay tuned!
@@NicholasRenotte github.com/haibert/graphics-project I want to share my attempt at this with you incase it helps or gives you any ideas.. I have used the expo-face-detector to get a 2D X Y coordinate of the nose and map that to the rotation of this 3D box I have generated on top of the camera using the libraries mentioned below. It is by no means near what it needs to be.. because I only have 2D data using expo face detector... but with the face landmarks you should be able to do much more interesting things. I'M SO STOKED TO SEE YOUR VIDEO!
"expo-gl": "~10.2.0",
"expo-graphics": "^2.0.0",
"expo-three": "5.4.0", (the package versions REALLY matter)
github.com/haibert/graphics-project
🥴
Great video! You’re really great at explaining explaining everything but is it possible you could help me out? Because I’m getting stuck at the Load Facemesh part. After coding everything It’s throwing an error saying “load” is not a function…
Is there a way to choose either to display or not the predictions based on a "confidence" level of the predicition? I mean, when I get off camera the model keeps trying to predict if there are any faces and draws over the objects in front of the camera.
Very good trainer. I wanna make an easy reconigtion app, first I would load an image with the name on (text) from phone camera to the database. The name input as watermask or another way the image's file name. . then at another location, this app could detect this item (from camera) and output the name on the live image. React Native + Mongobase.
Great work Nicholas. But, there is a typo on line 2673 of utilities.js: 'TRIANGULATION[i * 3 + 2],u' doesn't need the 'u', so guessing a typo. Really enjoyed though - thank you.
Thanks so much @David, ah i'll update the repo, I think I might have thrown it in just before committing to GH.
Awesome work honestly you deserve much more recognition :D One question though when I cloned it and tried to run it it keeps crashing before the detection with an Unhandled Rejection (Error): method must be bilinear or nearest, but was undefined error :) thanks for your help
Ah, baby steps, the channel is growing thanks to all the love from you guys 😍!! Can you share your code? Where exactly is the error happening in the trace?
@@NicholasRenotte I just cloned the code from your repo and tried to start the react app :) (I installed all the prerequisites). Thank you fro your help
@@alaeddineyacoub6225 you cloned it directly and it's still crashing? Just checking is your webcam activating when you start the app?
nice
Wow
Can you try with external usb connected camera plz?
I was wondering if you can make a separate video on who to map the detected face landmarks on a 2D/ 3D avatar
Working on it @Sarina, definitely planned!
Thanks for sharing bro, btw how to get blink eye, node, turn head etc (gesture face)
thanks before
like video filter,is it possible to change image instead of drawing line ? in here only to show us how to detect face! to change line to image ! what is best way ? if y know , plz tell me know ㅠㅠ
Great Tutorial! Can you please come up with react-native example of same tensor flow model using expo-gl?
Ya, in the works @Yash, will let you know once it's out!
@@NicholasRenotte Thanks! Appreaciated
@@yashthakkar3453 anytime!!
Hi, first of all, great job explaining. But i have a question, where do you get the triangulation data? Keep it up!
The landmark map? I got it from the official TF Model repository. Need a link?
@@NicholasRenotte Thanks for this video. I'm new the space but I also don't understand the triangulation data. It's hardcoded data. Why is there hard coded triangle data in there?
nice video
Thanks so much @Ekonaldi Hutabarat, glad you enjoyed it!
Loved the video it was awesome - do you know how you might remove the flickering of the mask? I'm trying to expand on this project to make a snapchat-like filter application and the flickering gets in the way.
Heya @Jacob, I haven't had a chance to fix it yet, I believe it's because the drawing is staged in two parts the dots and the lines. I'm working on a fix in the coming weeks though!
@@NicholasRenotte Thank you for the help!
@@jacobpiirsalu2862 anytime, will let you know once it's updated!!
Amazing Content man, spent a day trying to figure this out till I came across your video! Lastly, i was trying to use this model to use pre recorded videos. Do u have any resource/tips I could look into to change the input from webcam to pre recorded videos?
Heya @Tejas, does it have to be with .Js or are you comfy with Python as well? TBH it'll probs be easier to do with Python if you're wanting to work with pre-recorded vids. LMK I can chase it up for you!
Hey @@NicholasRenotte! Thank you for your response! I’m comfortable with Python but not sure how I would use it to implement this for pre recorded videos! And Yes! Can you please look into this/guide me?? Thanks a lot!!
@@tejaschoudhary3882 will add it to the list of videos. You should be able to do it with Media Pipe: google.github.io/mediapipe/solutions/face_mesh.html
@@NicholasRenotte Just had a look at it! The facemesh is implementable on Python but Iris tracking (separately on Media Pipe but included in the latest version of tensor flow) is unavailable on Python yet.
@@tejaschoudhary3882 ah got it!
Hi this project was very interesting I only have one doubt as to how are u getting the TRIANGULATION metric data since I can't wrap my head around it if you can help or explain
this is awesome. i came here coz i have a digital image processing project and i was wondering how to do the same thing in python (_Coz i'm weak in python). now i'm wondering why tf use python when u can do the same in js.(coz my strong point is javaScript. xD).
Hahahah, the beauty of those libraries is that you can do it in either right! Thanks for checking it out @Vijay!
I asked a question about this in your other video - but now that I looked at your latest push to your repo. Does the use of useEffect and RequestAnimationFrame prevent some of "lag" the previous implementation had? I used that with blaze pose in my code and it seems to have fixed any sense of lag which is great. I would love to know why this works if this is what fixed that issue. Thanks again for the content.
I was wondering the same thing
this is an amazing tutorial i just had to open my laptop and try it for myself! thank you so much. i have a question, i followed your tutorial, however the package @tensorflow-models/facemesh is no longer available. do you have any suggestions on how to resolve this issue? once again, thank you!
Heya @Putri, looks like the model name has changed from facemesh to face-landmarks-detection (github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection). Was looking into it over the last couple of days. There's three quick changes required to get it to work (plus I've updated the code here: github.com/nicknochnack/FacialLandmarkDetection):
1. Install the new package
// OLD Installation
npm install @tensorflow-models/facemesh
// NEW Installation
npm install @tensorflow-models/face-landmarks-detection
2. Change the import
// OLD Import
import * as facemesh from "@tensorflow-models/facemesh";
// NEW Import
import * as facemesh from "@tensorflow-models/face-landmarks-detection";
3. Change the load
// OLD Neural Network Load
const net = await facemesh.load({
inputResolution: { width: 640, height: 480 },
scale: 0.8,
});
// NEW Neural Network Load
const net = await facemesh.load(facemesh.SupportedPackages.mediapipeFacemesh);
4. Last but not least update the detection function
// OLD Detection
const face = await net.estimateFaces(video);
// NEW Detection
const face = await net.estimateFaces({input:video});
@@NicholasRenotte Thanks a lot for this update!
Hi Nicolas, you are great, I have a big ask for your, can you help me to create a good app by React and Face Detection functionality in your video for my graduated work in university
Hi Nick! I just completed your Media Pipe face mesh video and now i am here... one question is coming into my mind is ... can i build the same react app and integrate it with the MediaPipe one ?? please let me know if this is possible then i will try that integrating media pipe with this react app (I don't have much depth knowledge in react.js ). Also I feel media pipe is better than tf face mesh. Please let me know what are your answers/thoughts on this :)
Heya @Abhishek, yes you can definitely use medipipe on React in place of TF.js. I believe they're actually based on the same model so I think it's all about using what's easiest. I haven't tested out the js version yet but it's definitely on the list!
Hello! Thanks for the tutorial! What is your recommendation for the next steps to create a filter; such as distortion/displacement of the face or particles that attach to the mesh? I'm familiar with doing these things with a Kinect camera in Touchdesigner, but working in the browser is new territory for me. Thanks!
You could apply an viz overlay from the JS canvas inside of utilities.js! Where and how you adjust the canvas would be dependent on what points you want to attach the filter onto or from. Want me to make a video on this in the future?
@@NicholasRenotte yes!
@@nixcampbell5399 done, added to the list!
@@NicholasRenotte kindly,,,
@@NicholasRenotte Can you please share the link
Hi, Thank you so much for this tutorial! Can you make a video explaining how to do this in Java and Android studio? thank you so much!
Ooooh, not really my expertise @Daniel but I'll throw it on the list!
@@NicholasRenotte Thanks any way!
@@danielrotnemer2564 anytime!!
awesome tutorial👍🏼 btw, do we have to use react webcam? how do we make it for phone user? I'm a junior student in IT department. I just figured out how to make phone app with unity (using C#).. and wonder how can I combine it with ai face detection
Heya @Rhena thanks so much! Nope you don't! I just found it easy to work with. Oooooh Unity is something I'm super interested in. Have you checked out Unity ML?
Is there any way to optimize the model ? My laptop has 8gb ram and 2Gb gfx card but is barely able to run this smoothly, always hangs when I try to reload the app. Can I run it on an AWS server or something
Heya, it's all happening on the client-side so running it on a server won't boost performance. I'm working on tuning at the moment, should be out as part of the custom gesture vid 😊.
Is it possible to use this for facial expression recognition, with data point used for data science /machine learning?
Can you make something about body detection? I wanna make a virtual try on
Great video Nicholas!
I want to link this model to a react native based app. ( Hoping to determine whether a driver is drowsy or not) Can you refer me a direction to look for on how to do it? I'm kinda new for these environments. Thanks.
Heya @Chris, awesome use case. I'm actually working on a RN solution as we speak, will let you know as soon as it's out!
@@NicholasRenotte Okay. looking forward to it. Cheers!
I encounter error
Hey, great video! couldn't stop myself from implementing this! React-webcam is not working on localhost, do you have a way out? 🤔
Hmmm, any errors?
Hi, can you help me with text detection in an image using tensorflow js or opencv js
You're awesome man!👍👍👍
Instead of using create-react-app, can I use react native instead for this code to work? And how do I add my face mask image on the detected area of my face?
Sure can, but I think the webcam module is slightly different if you're doing it on RN. Working on the overlay code now, will share once I've got it up and running @Benjamin!
@@NicholasRenotte Yes, I think instead of webcam, you'll use RNCamera. i will be waiting for the overlay video. Thanks for your kind response.
@@benjaminikwuagwu6469 brilliant! Will let you know as soon as it's out!
@@NicholasRenotte I really appreciate :)
@@benjaminikwuagwu6469 anytime!
Which one is better py or javascript I am from the earth observation domain and I want to work on spatial using images for advanced applications in the Earth observation domain
i am curious if you could make a facial recognition using this face mesh?? hmm just like loading or storing dataset as 3d???
I'm trying to make an app where the user can try different shades of lipstick virtually.. I've found the annotations where it separates the lips out into 4 arrays, 3 of 11 and one of 10. Where I'm getting confused is how did you find the triangulation to draw the whole face mask? How would I map out drawing a texture for just the lips? Any advice or links to other tutorials would be greatly appreciated. It just doesn't seem to be well documented or I'm missing a keyword in my searches.
Yeah, I'm with you here, there isn't much out there on it. This isn't quite what you're looking for but it's a start: livecodestream.dev/post/detecting-face-features-and-applying-filters-with-javascript/ I've got some stuff planned for filters and textures planned!
thank you for your tutorial its help me for my task, but i get some error on const face = await estimateFaces(video); and this error like this estimateFaces is not defined. Can you help me to fix it?
Gg awesome stuff
Thanks soo much @ThunderStack!
I can detect which face it is based on these points? Like save them in a database an use for face detection
Awesome tutorial! btw, can you use the landmarks to get the actual measurement of your face? I'm trying to get the actual measurements of the eyes and nose something like that. its for my school project. TIA
Ya, I believe you could, will add it to the upcoming tutorials @Randy!
@@NicholasRenotte alright I will watch out for that, thanks for the response! thanks in advance for the tutorial
@@randycook6802 anytime, let me know if there's anything else you'd like to see. I'm always updating the video list!
@@NicholasRenotte i think you need a special webcam for that with 2 cameras
Hey ,using this can i able to detect which is live face which is image or photo by using only react or java script
how did you get those triangulation metrics? did you calculate them or are those standard? I am completely new to this, I was able to follow all along till this point...
No problems, great question!! They're standard triangulation metrics for that specific facemesh model.
how can we use this to make a blink detector/ face verification/ yawn detector
Can u help me for make tutorial liveness detection with face gesture like blink, nod, turn head etc?
Do u have same playlist with Angular?
Amazing tutorial. Appreciate it. How can I use it for user login? I want to grant access to a certain page after positive face id. Any resources I can look up? I didn't find anything on that on UA-cam, nor was I able to google it. Appreciate it.
Heya @Feruz Astanov, the technique that would allow you to do that is called Face Recognition. You can read more about it here: paperswithcode.com/task/face-recognition. Would you like me to do a video on it?
@@NicholasRenotte oh definitely ! Lol. I have a project to finish in 2 weeks, hope you can drop a video before then. Thank you so so much !
I love it the way you explained........ i have a query - ow do I overlay a 3D object using the TensorFlow "face-landmarks-detection" ML model?
trying to overlay 3d object on face like glasses but unable to render glasses on facemesh. We also looked into mediapipe but did not find appropriate solution for java script (web).
Any response will be highly appreciated.
Heya @Sachin, I'll be working on this over the next few weeks! Stay tuned!
@@NicholasRenotte thanks...I will be eagerly waiting for this.
@@NicholasRenotte any updates on it?
@@romanpulich16 yah, about 50% of the way in the code. Had to knock off some other stuff first.
@@NicholasRenotte can't wait for it to come
How To Convert Your Photo To Avatar Though Tensorflow Face Landmark Detector?
the new version of tensorflowjs is different and i am fully confused now! it doesn't know the model and i don't know how and from where it's requesting to download about 4MB models for each request!! so setinterval can not be the best practice for me! is there anyone here that can help me?!
Really good! Delayed but good, also I get this error "React Hook useEffect has a missing dependency: 'runFacemesh'. Either include it or remove the dependency array"
This is a warning, does the code still run?
@@NicholasRenotte Yes, it does
Error:
export 'load' (imported as 'facemesh') was not found in '@tensorflow-models/face-landmarks-detection' (possible exports: SupportedModels, createDetector, util)
What should I do now ?
Hey! Thank you for a great tutorial. Is there a way to save the facemash network alone as png or something we can use later on?
Yep, probably easier to do it with Python and cv2.imwrite ua-cam.com/video/pG4sUNDOZFg/v-deo.html
@@NicholasRenotte Awesome. Thank you for reply. 😊
@@fassoyangce1075 anytime!