Building a Real Time Sign Language Detection App with React.JS and Tensorflow.JS | Deep Learning
Вставка
- Опубліковано 21 лис 2020
- Tired of just using your Python Tensorflow Object Detection model in OpenCV?
Want to showcase your work in a standalone app?
Maybe, you want to do exactly this and help solve the sign language barrier!
Well you can!
In this video you’ll learn how to do exactly that in 45 minutes. You’ll convert an existing Python Tensorflow model and leverage it in a standalone app with real time detections. You’ll be able to see the results in real time! But what’s even better is that you can take this and apply to to your very own use cases, whatever they might be!
This video covers:
1. Converting a Tensorflow Object Detection API model to Tensorflow.JS Graph Model format
2. Hosting a trained Tensorflow deep learning model for applications
3. Downloading the React and Tensorflow.JS Computer Vision Template
4. Making real time detections using a deployed Tensorflow.JS model
5. Visualising detections within the HTML canvas
Completed Code:
github.com/nicknochnack/RealT...
Other Links
Template Code: github.com/nicknochnack/React...
Setting Up Cloud Object Storage on IBM Cloud: cloud.ibm.com/objectstorage/c...
Installing the Cloud Object Storage CLI: github.com/IBM-Cloud/ibm-clou...
Installing the Cloud Object Storage Plugin: github.com/IBM/ibmcloud-objec...
Tensorflow.JS : www.tensorflow.org/js
React: reactjs.org/
Oh, and don't forget to connect with me!
LinkedIn: / nicholasr. .
Facebook: / nickrenotte
GitHub: github.com/nicknochnack
Happy coding!
Nick
P.s. Let me know how you go and drop a comment if you need a hand! - Наука та технологія
Good watch! I learned a lot. I've been waiting for your Real Time Face Detection in Mobile Application with React.JS and Tensorflow. JS :D
OMG thank you so much for teaching stuff cause of your projects I don't find machine learning taught in my uni boring
The best tutorials in UA-cam, Thank you very much !!!!!
Thanks so much 🙏🙏🙏!!
So thankful that someone on youtube named Nicholas is making amazing tutorials for free about ML
Heyyyy, thanks so much @Mono Me!
Awesome and great tutorial Nick!!! Loved it and thanks for educating us. Also your recording style is great without any distractions. I just love it. By the way, could you please share how you record these sessions and what tools do you use and steps you follow? Would greatly appreciate if you can share the details.
Heya Bhushan, I use OBS for making them!
It is fantastic and informative! Thanks for the tutorials!
Thanks so much @Jerry, anytime!!
Thank you soooo much Sir 🔥🔥🔥 I just cant express how happy I am
God bless you with more success💖💖
Plss bring more tutorials for us like these
Definitely, plenty more coming @Jaskeerat!
Great work!!
love it. Thanks, brother.
Thanks so much @Eranda 🙏!
Thank you very much. This is awesome.
Thanks so much @Saman! Plenty more to come!
The best TFJS tutorial ever!
Oh! Thanks so much @Max!
Could you please help me to run the project? It's really urgent. I'm struggling for the last couple of days.
@@hasanfuad6631 sure, what's up?
bro well done!
THE BEST!!!! THANK YOU
Thanks so much 🙏!
thanks you man, and big Up
Great video!
🙏🙏 thanks @Nico!
Amazing job!! I was thinking to do the same with the sign language activity recognition video you have, is it possible to export and deploy that model the same way you did in here? I was researching about it and didn't find much.
I'd imagine it takes ages to make these long tutorials.Thanks you so much for this :D
Oh, you're absolutely spot on @Diego! I'm trying to make them as complete as possible though!
Yooo 🔥
Still waiting for the mustache detection video by the way 🤔 haha jokes! 😂
Super cool video man!
Can you do a video on how to deploy this app? Seems tricky ...
😂😂😂 I thought about sneaking in a new class of detections into the model! Yep, got deployment planned @Vikash!
Check it out @Vikash, deploying the app: ua-cam.com/video/2dwSXXMFhPY/v-deo.html
Thanks a million for all the mind blowing videos!! Much appreciated :) My 70% of Tensorflow knowledge were built through your tutorials... Looking forward to learn more
Thanks so much @Anisha! Check out this tutorial, it should help you extract the detection history: ua-cam.com/video/UPS54i7Km30/v-deo.html
@@NicholasRenotte Very interesting tutorial. I'll try this out. Alternatively I tweaked visualization_utils.py to get each class count. Cheers👍
@@anishaudayakumar1778 awesome work! Next project, Raspberry Pi Sentiment Analysis perhaps? ua-cam.com/video/yqkISICHH-U/v-deo.html
@@NicholasRenotte Woahhh! Super interesting and I can actually try out some retail use cases with this sentiment analysis😀
@@anishaudayakumar1778 yesss, awesome!
Hey Nick! Need some guidance. I trained a deep learning model of Inceptionv3 on my custom Static ASL dataset (my dataset, for which I used the handtrackingmodule library, has hand keypoints with bounding box in it). I have the trained model in .h5 format. Can I use this code with my model?
Thankyou so much Nicholas sir for this amazing video. I have decided to do this for my final year project.
The first part i.e. detecting sign language in real time using OpenCV is done but the project is still open-ended so I have made a website instead of a standalone app. So, Could you please guide me in integrating the model with website?
Hello bsed on what did you comput the 16.7 interaval? My monitor only has 60 hz rfrsh rate, so I was wondering, what is the optimal interaval that I can use
After several days I decided to continue teaching of my model. I was waiting for 7k steps. After training I generated new json file and it had another indexes for boxes, classes and scores. Previous time it was 0,1,6. Now it's 4,0,1. No idea how does it work but it works.
I have no idea why it changes each time @Max, I noticed it as well when I was prototyping the course!
@@NicholasRenotte Is there some way to recognize which one are the new ones? Having trouble with my custom model.
Anthony you are so good at explaining, could you do a video about R-CNN hand gesture detection with Keras, TensorFlow, and Deep Learning. ????????
Oh thanks so much! Definitely, I'll add it to the list1
Hello, i didn't understand where I can download "Custom object detection App" folder, help me please! And is there difference when u do this project on mac os?
How would you implement the cloud object storage part if you're using AWS instead of IBM Cloud Storage? Would love a step-by-step explanation!
Just a question.
How about doing all the processing stuff in backend django and the presenting the output in front end react?
I use django react combination.
Hey, Nicholas is there any way to transfer this code to Angular? That would be great if there was a simple tutorial on that too! Thanks!
Hi Nicholas! Amazing video! I was wondering, is there way for us to detect other signs besides the ones you showed us? Like the alphabet and such? Thanks!
Definitely @Andy, you could add in all the labels you want, you just need to collect and label the required images and train it!
Hey bro please respond to me I want to make a sign language translator project and I want to ask what should I learn to accomplish my goal I know Python3 and a very basic knowledge of Machine learning what should I learn now to make that project ?
But this will run on the PC where I am building this project, how can I open the camera for the client side ?
Good tutorial but when you upload your hand img ?
I want to ask about the model that I have created. here I have created and saved the model in .h5 or hdf5 file format. when I convert using tfjs.converters.save_hard_model(), can it be applied to your tutorial?
Hi, can you help me with text detection in an image using tensorflow js or opencv js
Hey nick,can i ask? Is this the continuation of your previous video which is sign language detection using tensorflow to make it as a stand alone app?
Yep, this is just a follow on from the initial walkthrough.
It was very helpful my friend, but I would greatly appreciate it if you could explain how to translate an entire sentence in addition to a word using this method.
Heya, you can store each translation inside of a string by passing it through to a React state component!
@@NicholasRenotte thank you 👌👌
@@ndwdesigners1848 anytime!
Why am I getting error like TensorList shape mismatch: Shapes -1 and 3 must match
I know this may sound like a dumb question but please what database does it use if it uses any. can we save the dataset in a database instead of just a repository
hi, I got an error :
Traceback (most recent call last):
File "Tensorflow/scripts/generate_tfrecord.py", line 18, in
import tensorflow.compat.v1 as tf
ModuleNotFoundError: No module named 'tensorflow'
can you tell me how to fix it please !
Heya @Fatima, looks like a missing module. Try running through the install video and it should be resolved: ua-cam.com/video/dZh_ps8gKgs/v-deo.html
@@NicholasRenotte okey thank u
@@fatimashow7174 anytime! Let me know how you go!
IBM cloud seems difficult to sign up this day. It requires Credit Card which I don't have one. Can you tell me how to use other stuff?
Hi, I just want to ask what other software or language do you use other than tensorflow.js and react.js or is it only tensorflow.js and react.js. Thanks
Mainly Python and Javascript but I code in some product specific languages as well for some IBM products.
Awesome tutorial!! But can you help me with a doubt that if I convert a .h5 keras model for my emotion detection project directly to tfjs and not freeze the graph and all will it work?
Ideally you want to freeze before you do the conversion!
@@NicholasRenotte ok so should I train it again in the way you did?
@@karansethi1830 yep, if you want to use the TFOD API it should hopefully make it a bit easier.
Very nice presentation. What are the challenges you faced? Are there any enhancements you have in mind? How do I contact you
Hey Nicholas, awesome stuff as always man! Quick question, if I do not want to host my model on IBM's cloud, can I power it locally? And if so, how?
Yup, I believe you can host it in local storage!
@@NicholasRenotte Heya, I'm getting the following error:
Unhandled Rejection (Error): TensorList shape mismatch: Shapes -1 and 3 must match
The code that produces this error is the same one as in your GitHub repo. Why is this showing up?
EDIT: The code does not crash with the same error on your model (bucket), however, there are no detections either. I followed your Sign Language Tensorflow tutorial.
@@ItzYeho Same here man
@@ItzYeho same. did you find any solution?
@@ItzYeho DId u find the solution?
Can you do a hand gesture recognition in Unity and this can run on Android with >30fps ? Thank you
Great tutorial sir.
Ask sir... How to put voice into tensorflow? I mean there is voice output when the object is detected.
Heya @Dwiki, like adding text to speech for this example?
Thank you so much for this tutorial sir, it's awesome 🔥🔥 But I would like to ask. I've created my own model, and after running 'npm start' I keep getting a blank page on my react web. Any suggestion for the problem, sir?
Getting errors?
I tried to run the react app as it is to test it out but It's not detecting the signs. Am I doing anything wrong? I updated all the packages as well.
For the convert to TFJS model step, when I run the tensorflowjs_converter command, windows does not recognize the command even though tensorflowjs is successfully installed. What to do?
Heya @Saran, can you try running through the full Object Detection API install? ua-cam.com/video/dZh_ps8gKgs/v-deo.html
Hey Nicholas, Can we make the same app using Flutter?
hello nick i got an error on browser which is mismatch shape -3 and 1. What should I do?
is it needed to follow whole process? i just clone your git and start but i always get "no"
Hi, is it possible to host our model to Azure Cloud? If it is possible then can please tell me how could host my model to the cloud? I already try it but it is giving me error ( Unhandled Rejection (TypeError): Failed to fetch). By the here I have used Azure Blob. please help me with this
Hi, did found any solution?
I need this video for react native
To build this projects what all things to be install in our machine?
Can we like make a separate section where we can show the text of what the sign language is being detected? like a section box showing "Thank You" in the website.
Yup, you just need to parse the detection_classes and map it to the label map object @Abhinav
Cool
Thanks so much @Novia!
hey guys want to ask which one is the link to the first tensorflow folder?
Here for the Mo again
😂 two more weeks!
Hi,
Someone is having problem with the detection model itself? Section 8, "Detect in Real-Time", I run the code but nothing is being detected...
very nice can video media pipe with tenserflow model Sign Language Detection react
what are the system specs to make this project work
I have exported the graph. How can I convert to Tflite model instead of TFJS??
Heya @Hans, check this out: gilberttanner.com/blog/convert-your-tensorflow-object-detection-model-to-tensorflow-lite
my coverted model.json and your are in different form. So i cannot use my model to apply with your react code? What should i do?
I'm on it, will have it updated for the next tutorial @Srey.
Hello Nicholas , I got an error "ValueError: 'images' must have either 3 or 4 dimensions." when running the part of Detect in Real-Time. Could you help me to fix that? Many thanks!!
Heya @Tianshi, try stopping your code then running cap.release(). This should free up your webcam to make new detections.
Hey, Nicholas! Thank you so much for yet another amazing tutorial. I have a few doubts of my own. If you could help me, that would be appreciated:
1. I started off by cloning the RealTimeObjectDetectionwithTFJS repo in my machine.
2. I checked the code and as you said, opened up a new terminal and ran the command "npm install" in the directory which contained package.json
3. It gave me around 3600 warnings out of which 19 were profiled high.
4. Then, I ran "npm start" which started the react page on localhost:3000. The webcam launched.
But, this is where I encountered a problem. The detections don't work or they are very inaccurate. My face gets a box drawn around it instead of my hand or I love you is detected as no. What could be the problem and how do I resolve it? Do I have to train the model more? If so how do I achieve that? Your help would be very valuable Nicholas, thank you.
Heya @Tejeshwar, ideally you would want to train the model a whole lot more. I've only trained it on a couple of images so it would likely be overfit on my particular background etc.
Sorry I'm being inquisitive here, but I'm just starting on this whole thing and could you tell me how I can train the model with more images?
@@TheDIYTechXpert yup, just freeze your checkpoint, then change the fine_tune_checkpoint path in the config to point to that file e.g. ckpt-0. Add your new images and recreate the tfrecord and kick off training again!
Hi Nick it's awesome video thanks! I have error like this:
Unhandled Rejection (RangeError): byte length of Float32Array should be a multiple of 4
Is the problem caused by my custom model? (ssd_resnet_50_fpn_coco)
When I give your model links to tf.loadGraphModel function it's work. But mine doesn't work. I set the public access from IBM cloud but still not working.
Heya @Niels, It might be because the model outputs have changed when converting. You might just need to work through to determine which value in the obj array maps to the different components, boxes, classes and scores. I show how to do it here: ua-cam.com/video/yqkISICHH-U/v-deo.html
@Nicholas Renotte I solved that thank you man you're my hero! too much question sorry but, it's work so slow. I have Gtx1050 and converted json file around 192 mb. I think it's performance that much. Is that possible to run this kind of applications on Google Colab?
@@nielsbohr3116 YESS, awesome work. You can run static image detections on Colab but getting a feed is a little trickier.
Hey I got the same error, how did you fix it?
@@finnsmither-may2691 Hey, I changed tensorflowjs version in package.json to "@tensorflow/tfjs": "^3.4.0" and don't forget to type 'npm install' command again.
If still not working, I tried this: in Nicholas's code console.log(obj) print an array includes 8 tensor object. I think different models have different output orders. Just look which one is represented by boxes, classes, scores, then change indexes for your model. For example I use:
const boxes = await obj[1].array()
const classes = await obj[6].array()
const scores = await obj[7].array()
Note: Tensor objects are very similar. You can seperate them from their size. But still there are 3 tensors with size 100 hard to detect. I tried combinations until find them.
When I run the App, it doesn't detect the sign language that I do. Any solutions to solve this problem? Thanks a lot
Hi can you please tell me that if i can create mobile app using these steps?
module 'tensorflow' has no attribute 'contrib'. How to solve this error?
can i use heroku instead of ibm
Is it necessary to host model on cloud storage bucket? I tried to import directly but didn't succeeded. I also tried to host a local server with node http-server but again that's a no-no. Btw thanks, got this far because of you.
Heya @Utsav, the easiest way is to host it in a bucket. I tried the other methods described here www.tensorflow.org/js/guide/save_load and they ended up being a bit of a pain (especially if you want to deploy later). The next best easiest option, would be to save it to localStorage within the browser and load from there.
@@NicholasRenotte Hello Nicholas firstly thank you very much for your videos, do you have a video that shows how to load from localStorage using React.js?
@@viniholtzreviews haven't got one done yet unfortunately. I saw it in the doco a while back but didn't have a chance to test it out.
Hey,
I am unable to get proper predictions...the code works fine om jupyter...but on react it hardly gives any predictions...i have also changed the values for the indexes of the boxes, classes and scores
Could be a bunch of things, what was your accuracy like during testing? Take a look at improving lighting as well?
@@NicholasRenotte the accuracy was around 80%....yes I'll improve the lighting...thank you!!
Could you record a video about how to recognize sign language by using video processing?? I really need it for my FYP, if possible could I have your contact?
Definitely, already done! Check this out: ua-cam.com/video/pDXdlXlaCco/v-deo.html
@Nicholas Renotte Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
I am getting this error how to fix it ? :) i tried to fix it but still getting the same error :)
Heya @rafia! Have you installed Python yet? We used Anaconda for this video.
@@NicholasRenotte I fixed it and finally i have done it ✌thank u so much😌
@@rafiaqadirmemon6078 AWESOME! Great work!
Why when i try your react app, the webcam just detect sign "no"
All Working Good but the webcam is not working when npm start it just shows a Blank page...I am using laptop and chrome having the default laptop webcam...Help
the same trouble as me. I hope we have solution😓
Heya @Suvinary, can you try an alternate default webcam inside of your browser?
Hi can we implement it and make mobile app?
hello bro can u make a video for multiple hand recongnation video
How would the UML diagram and class diagram would look like for this?
Don't have it available unfortunately @Sivesh!
Can it work on mobile?
This particular app doesn't but It can, I believe the react-webcam library needs to be swapped out for something that runs on mobile!
@@NicholasRenotte Thank you for the reply
Anytime!!
Thank you amazing tutorial! I have a problem thought.
I did all the steps and everything worked great exept for one thing.
When I run react.js(npm start and browser pops up) camera doesn't work. I only have a blue background. Code is pretty much copy-paste, no mistakes. Any ideas?
Hmmm, can you try changing the default webcam in your browser @RealEscapePlan?
@@NicholasRenotte I am also having the same issue that camera doesn't work and I am doing project on laptop Please Help...All going right but only the webcam doesn't work.
@@suvinaysawant9502 can you try changing your default webcam in the browser?
@@NicholasRenotteNo, I have just integrated webcam of laptop
@@suvinaysawant9502 no that's fine, just double check that it's selected in Chrome or Firefox for example.
hey can i use your project for my college project
Hello ..anyone getting an error saying text can't be used to index type ?
Hi, this is an amazing project the thing that is disturbing me is that how can we add new gestures to it and also it's quite time-consuming to label each image separately in real-time. Lets say if anyone who don't know about the programming and imglabeling and we are giving this software for use then how can we come up this situation as he/she won't be able to do imglabeling? Please help me out on this as I'm presenting such a project in my FYP.
Heya @Tamur, this is at this stage an app that needs to be trained locally then deployed. You probably could work something out that allows training from within the browser, might be easier to do it with image classification however.
Great thanks for guidance Nick. Cheers
@@tamurtaj2722 you're most welcome! Happy to help
Is there a way to recognise the whole alphabet? There are some dynamic gestures that requires video processing. I am not an expert, just learning. Thank u so much for your vids!!
Check out the action detection video on the channel, I show how to use action recognition there!
can we add in resume
Plss also tell how to deploy this
Coz I deployed the previous apps with Netlify and also posted code on Github
Plss tell a new approach or tell if Netlify is fine or not??
Netlify is all good, I normally deploy using IBM Cloud but if you've done it successfully no stress!!
Heya @Jaskeerat, check it out, deploying the app: ua-cam.com/video/2dwSXXMFhPY/v-deo.html
Hi, how can I make it work as a stand-alone app on mobile?
Heya @LEE, you could do it with React-Native but would need to use a different webcam library!
Nice tutorial.. Can you show how to deploy this app 😊.. Thanks
Thanks so much! I was just thinking about that for the next vid, will take a look this week @Ayushman!
Looking forward to the next video.
Thanks again for these tutorials
@@ayushmanburagohain8930 anytime, stay tuned!
Heya @Ayushman, you asked, it's here! Deploying the app: ua-cam.com/video/2dwSXXMFhPY/v-deo.html
@@NicholasRenotte stoked about it man.. Thanks
Hey Nick I'm facing some issue with the boxes ... I have different model but it is showing that the function it not defined when I use my model I can share the ss the code is exactly the same have tried to find the problem but no result. help please
Heya @Sourav, it might be because of the shape of the outputs from the model. Can you try your model in the baseline code: github.com/nicknochnack/RealTimeSignLanguageDetectionwithTFJS
@@NicholasRenotte I have actually tried that too but sadly it's no use 😅 Your model just doesn't seem to recognize my gestures.
@@NicholasRenotte Okay so I did some work and I changed it to this
const boxes = await obj[0].array()
const classes = await obj[4].array()
const scores = await obj[1].array()
your was - obj[1], obj[2], obj[4]
but for mine it is as above and hence my model works just fine.. thank you so much.. now I just gotta convert these to speech ... and I was thinking if we trained a model with black and white images then it would not have trouble recognizing different skin tones ... obviously we need to get the video in bw for the detection too I tried it but welp no results
@@souravde2029 YESSSS! Awesome work. Speech conversion would be sick. Correct, you would ideally want to grayscale before making detections then. Could probably do it with OpenCV for javascript.
Hi sir. Followed the same steps and finally the application started without any errors. But it is not able to predict any signs and objects. Can you please help me with this?
Heya @Sai Sri Mukka, I realised a little later on that the conversion doesn't always output the results in the same shape. You might need to play around with the indexes for the boxes, classes and scores variables to get the right values. I show how to do it here: ua-cam.com/video/yqkISICHH-U/v-deo.html
hallow i realy love your tutorials especially that of facemesh. Plz can you do real time(on camera) face recogition by reactjs with tensorflowjs or reactnative with tensoflowjs am react native fan especially with react native if possible using Resnet50 or Resnet34 models
As my idea was faceblaze model(for face detection)->landmark detection model(386points) ->then describe the data by (Resnet34/50 model-which is not on tensorflow js but in python) for disriptive -> then we apply ( Euclidean distance ) for recogntion. plz correct me if am wrong
Heya @Ninkambazi, sounds interesting. What's the end goal, to recognise faces? I tend to stay away from explicit facial recognition tech due to its potential for misuse. My friend's asked for something similar this weekend, perhaps I could do it on non-human faces? It's on the list to consider, just need to think it through!
@@NicholasRenotte woooow i cant wait for it seriously when will this be? am curious
@@ninkambazielitwaza7859 likely early next year 😊, got a bunch of vids planned for the rest of this year!
How will you deal with words that require multiple actions like "menu"????
Some words require 4-5 actions in one go.
That's part two that I've been working on, you can use a different network architecture that uses video as an input. It's called Tiny Video Networks.
Hello can this tutorial exported with react native ? On mobile smartphone?
Could use the same tfjs model on React Native. Just need to shift from pages to views and update the code to handle native cameras.
@@NicholasRenotte can you make the tutorial for react native and use the react native CLI not expo?
@@MrSANS ya, gotta brush up on my react native
Even though it creating the tensor object in console but not predicting on screen
Hmmm, what happens when you run the baseline code and swap it out with your model: github.com/nicknochnack/RealTimeSignLanguageDetectionwithTFJS
can we apply sign detection code on other applications...for example can we apply it on video calling..so tht whtever person performs sign on video call we can display the result....can we do that.....if yes thn plss make a tutorial on it sir....
Hmmm, might need to dig into this one a little more!
I tried the same thing but for me classes is not an array, classes.length is returning undefined. my code is something like obj[2].array().then(classes => {... where obj is this.net.executeAsync(expanded).then(obj => {
weird now boxes is not an array, but just 100
Yeah I noticed this was happening when I redid the project for a client. Take a look at the 5hr tutorial, I show how to resolve it in Project 2.
sir i have follow you same but sir i did not get the prediction from webcam uploading through reactjs
Hmm, any errors?
Shapes (7, 7, 3, 64) and (3, 3, 3, 32) are incompatible. i am getting this error what should i do
Heya @bruce, when is this happening? When you deploy the model or test it out?