Face detection and tracking on FPGA
Вставка
- Опубліковано 5 лют 2025
- Face detection and tracking has been an important and active research field because it
offers many applications, especially in video surveillance, biometrics, or video coding. The goal of this project was to implement a real-time system on an FPGA board to detect and track a human's face. The face detection algorithm involved color-based skin segmentation and image filtering. The face location was determined by calculating the centroid of the detected region. A software version of the algorithm was independently implemented and tested on still pictures in MATLAB. Although the transition from MATLAB to Verilog was not as smooth as expected, experimental results proved the accuracy and effectiveness of the real-time system, even under varying conditions of lights, facial poses and skin colors. All calculation of the hardware implementation was done in real time with minimal computational effort, thus suitable for power-limited applications.
people.ece.corn...
Thank you for your amazing work! I appreciate that you shared this video and especially the documentation. Thanks again and good luck for the future.
That is lit. Nice job to that student!
Cool stuff. I've done a lot of multiple face detection, human tracking from single POV. Finding the right combination of pixel manipulations can be difficult especially without fixed lighting in the environment. It's nice to see this in hardware.
You could use the skin detection method to detect probably objects and then calculate the size of the object detected.
Since humans generally have the same head size and shape and that we know the facial proportions of the human face, you should be able to calculate the locations to where you should look for the eyes and mouth for any given head at any given distance.
With the filter used here you could see a pattern that was very similar a round/oval shape for both heads wit some shadows.
I'm awestruck. The immensity of it!
Amazing Work.
Thanks.
WATCHING IT AFTER 10 YEARS ALMOST FROM INDIA ECE STUDENT
......
thank you so much for sharing documents of your work.you deserve the best.good job
Very nice project!
I was wondering why not use some basic characteristics of a human face like the distance between the eyes to determine whether there is a face in the image or not?
I would assume that finding two yes would be a simple algorithm to implement.
Thanks for sharing, very interesting! I will use a newer SoC board to reproduce the function mentioned in the report, also adding some machine learning techniques to based on the original design.
Thank you for the Verilog code. I am used to learn VHDL, but will force myself to learn Verilog HDL too. Great video. Just this board is too expensive for my taste.
One way or another, if you wan to do this project, you will need peripherials attached to the fpga. That is not cheap
@@ece4760 You are right. But my problem is which board to chose?! So far, I am working on cheap FPGA (Cyclone II), just board, 50 MHz oscillator and flash memory). For ADC, DAC and other thing, I have plan to use STM32 MCU, at least for audio part. For VGA, it may be problem, but maybe not - soldering some "R/2R resistor ladder" and everything for first video work. For webcam input, that might be the problem, but I will solve that somehow. Thanks anyway.
@@MilanKarakas I wrote this a few years ago
hackaday.io/project/5033-de0-nano-fpga-to-vga-output
Webcam input is quite difficult. The USB protocol is very complex. It is better to use a serial output camera.
@@ece4760 Well, yes. I noticed that when study USB on STM32. Just not familiar with whole concept. Serial output camera? Is that like IP camera or what?
Bruce, that is very impressive for a project. What university was this?
cornell university, masters project
Interesting project. Great job!
Thanks for the sharing video and open source file. I will give you credit for my project.
Yes, very nice and thank you for the effort for a small but very important breakthrough...... Thanks.... I understand your effort. And the end applications are inconmeasurable....
Sir in your report there are two code present one for matlab and one for verilog. Please tell what is the working of verilog code which is given in your report.
I am confused for verilog code. Please tell me what function is performed by verilog code.
Can i ask where do your students get sdram controller? I have de0 nano with 32 MB SDRAM and i was wondering how to achieve frame buffer for 640x480 camera resolution - i found two ways to be most appropriate for me - one using sdram controller from hamsterworks (quite popular vdhl sdram controller for de0 nano and sdram chip on it) and other method was to use Qsys (i prefer to have VHDL/verilog controller to instantiate it in my project without Qsys) - ive opened DE2_TV file from link in the description and found that You are using some kind of template from altera (still there are no files for sdram controller if im right, only sdram framebuffer instantiation Sdram_Control_4Port u6)
You are right. Thye students got it from the TV example. But now we use the University program SDRAM controller for Qsys on DE1-SoC
Hi!
Would be nice if you can provide in your report the amount of resources used (LE's, DSP's, memory, etc...).
A power estimation report would also be nice, for each of the synthesized cores.
Apart from that great work! Have you considered using other methods for the detection? And from what I can see, you're just detecting, not really tracking. Did you try some tracking methods using filters like particle filters, EKF's, UKFs.. etc?
I checked your report, and I couldn't find the answer to the above questions.
Again, great work and good luck with full completion of your master thesis.
***** Also, what camera sensor did you use? Was it the OV7670 by any chance?
***** NTSC analog TV camera
First i want to thank you for sharing everything ..
would it be possible to use the transition from MATLAB to VHDL instead of Vrilog ?
+Mohamed Na VHDL should work. But the conversion to Verilog was done manually, so you would have to convert it.
I did not understand the centroid concept. Can someone please clarify?
Average position of all the marked pixels. Average x and y positions separately.
All calculation of the hardware implementation was done in real time with minimal computational effort, thus suitable for power-limited applications.
Who will power the fpga
excellent work. Congratulations.
I want to implement this project but I am new to camera interfacing to fpga and i am not able to find much which help me understand from the basic ...it would a great help if you could share any helpful link (related to camera interfacing with fpga)
I would like to know more about Fpga and tracking, where can I find out more information about it?
Hi doctor Bruce, I was looking at the code why avg_out is being multiplied by 16 for value of avg2 register
did you read the report?
which camera have you used?
is it analog or digital?
How many frames per second can you expect from the fpga? I mean if the frame speed wasn't limited by the camera, screen etc...
it is probably limited by memory access to around 50million pixels/sec. so around 150 fps.
Very neat project! Makes me ashamed I never learned about FPGAs.
How well does the skin segmentation work on different skin colours? Especially some of the darker skin colours found in Sub Saharan Africa.
+MrScout It seems to work fairly well because intensity is factored out and everyone's skin color is based on one chemical with almost constant spectral characterisitcs.
+Bruce Land Thank you for the response. I hadn't heard of this technique before (I am by no means a CV expert), but it might work perfectly for my application - also with constrained resources. My current approach fails miserably on very dark skinned people.
there are lots of research papers on the skin color constancy
Hello!
The code for the VGA should be written in Verilog direclty or is there any code in Matlab that implements the VGA?
Code was tested in matlab, then converted manually into verilog. Matlab is not fast enough to do VGA.
Thank you for your response!
Sir how I convert matlab file into verilog file
Which camera module is used? can you provide the link or manufacture details. i just want to know whats the interface between camera and FPGA?
Did you read the report? The camera is a generic, NTSC camera connected to the NTSC codec on the DE2 board.
excuse me..
i want to ask about the type of your fpga board.
what type of your fpga board that you use?
full details at people.ece.cornell.edu/land/courses/eceprojectsland/STUDENTPROJ/2012to2013/tnn7/index.html
Bruce Land do you have vhdl code? i want to implement edge detection in fpga partan 3a, but i don't know how to insert edge detection code in there. can you help me?
*spartan
I do not have vhdl code, nor do we use Xilinx parts. Sorry
However, this approach to my home turns out not really good. The wall color reflecting the bulb light just make the false detection everywhere. I guess shape detection would be more reasonable.
hi I have connected a cctv camera and the output RCA cable is given to video in. but the display is not coming in monitor. may i know what might be the problem.
+Jeyakumar R How could I possibly know what the problem is with so little info. Does the camera produce actual NTSC video. Did you check it? Does the reference video code from Terasic work with your board?
+Bruce Land I am not exactly sure about the output format of my Camera. but when i connected my RCA jack to video in port of TV it works fine. but when the same is connected to video in port of FPGA DE2 board it is not showing anything. May i know what is the problem. ? I tried working with one more code by connecting terasic camera by connecting to GPIO pins. It is working fine.
Jeyakumar R Was the TV NTSC? When you hook a scope to the camera output is it running at 60 fields/sec? GPIO camera tells you nothing useful.
+Bruce Land yes i understood my problem. CCTV camera does not provide NTSC format. It provides only PAL output. Now how to convert PAL to NTSC format . any idea in DE2 board ?
+Jeyakumar R It is easier to change the code to work with PAL
sir ,
how to convert this project matlab code to hdl code
what are the steps or software required
+VAIBHAV MATHUR Learn HDL and convert it by understanding the algorithm
it detects the human skin not only face,so any idea how make just detect the face.
Good work!
Can I use a Webcam???.
Is Webcam video format is same as ntsc,pla,etc
YOu could easily look this up. How could it be? Every detail of the connection is different.
Interesting approach. Isn't an AI usually used to detect faces? This is adequate for this purpose though. I wonder if a small computer with OpenCV would be more energy efficient.
Don't know. Often, hardware which is purpose built is more efficient, but would be interesting to test.
I've heard that FPGAs usually use up a lot of energy though. ASICS are obviously the best solution but that's a bit hard :P. I'm impressed though. Good work =)
What exactly do I have to study to do this? Is it just EE or Computer Engineering?
Mikael Murstam
digital design and computer engineering
hi there,This camera is your designer which used on your project,what the verilog is your language,yeah i am interested in VHDL,ok can we just ... interacting ok ?..
How about you use complete sentences so I can figure out what you are saying?
Uh.. i mean you did nice . But the hardwear you used should be already made . What about try your best to design one. And so .. next programming..
My English language gets worse .don't mind
Hello doctor bruce.Can i implement this using a Zedboard?
probably. try it.
+Bruce Land I tried doing it but what OS is your FPGA using,Linux,linaro or Xilinx??
This FPGA was hardware only, no processor, no OS.
+Bruce Land Thank you
also why avg2 needs to be greater than 959.
Genious. What is that running on?
Did you read the linked report?
You could start with the link given in the video comment.
apologize to its algorithm converts the image into a matrix ?
+Primavera Argüelles Hmm not really. The image is only a matrix in MATLAB. But in Verilog, the image should be stored ( or buffered if it is a live video camera stream) in the fpga memory ( mostly block Ram) as an array of pixels. So after, porting the algorithm from MATLAB to Verilog using the MATLAB HDL coder, the algorithm should now be pointing to the block ram as its image input.
+alex nice This was not ported using HDL coder. It was ported by writing the Verilog by hand.
Bruce Land
Oh okay, please do u know if they used block ram as image storage ?
you could read the report
+alex nice She uses SRAM (maybe SDRAM) as buffer memory. Won't be able to store in BRAM because she uses 640 x 480 resolution unless she has an expensive FPGA.
can you provide the test bench for the same ? please that would be a great help
Which language and tool U r used ?
Did you read the linked report?
Sir how can I convert matlab code into verilog code
Detail any software or step which you used
the student converted the code by hand by understanding and optimizing each step. No software was used to convert.
Sir I have two code one for matlab and another for verilog but matlab give exact output but verilog code is not properly run.
I want to know what operation exactly perform in fpga board
The verilog code on the FPGA did all operations. Did you read the report?
Bruce Land
Yes sir I read all your report. In this you mentioned conversation of matlab code is not a easy task and also write an implementation of morphological filter in verilog is very difficult.
But in your report a verilog code also give, I want to ask is this code is sufficient for face recognition,if no so what step should I need?
So please help me on this....
This project does face DETECTION, it does not do face RECOGNITION. Recognition is much harder. There is a huge ligature on face recognition. Start reading
Ugh... and I am here trying to get the HPS to blink some LEDs for me.
The HPS is difficult! But since 2016 we have been using it. See people.ece.cornell.edu/land/courses/ece5760/index.html#links
what about face emotion detection on fpga is it possible.
Code that is not written can do anything. Just ask marketing
But google "emotion recognition fpga"
do you have a code of emotion recognition fpga.?
I do not
can i get full report in word document.
+Prashanth Reddy Nope. PDF.
pretty awesome concept.
complete tutorial please or at least how to access camera using fpga
Did you read the linked report?
Oh my bad, thanks anyway
Great work sir , Is it is possible to extend this technique to recognise face in real time on FPGA.
recognition is much harder than detection. I do not thik this technique will work for recognition.
Thanks for your reply. Sir. I thought, if i detect face with this technique, then only the detected portion of video match with data base and apply some technique of face recognition so we able to face recognise. but as your suggestion I leave it.
Sir I am using zedboard. Your verilog code is compatible with zedboard??
What steps should be taken to make it compatible with zedboard.
It is not. Completely recode all device-specific details, like block memory and multipliers.
Amazing work can u share the verilog code for this project
Did you follow the link in the video description?
Which FPGA is that?
did you read the report?
Sorry no. I was trying to get some project ideas.
And thank you, I got the pdf :) I will read.
sir materials of this project : just camera and fpga board and monitor
and special camera or any camera !!!
Did you read te linked report? NTSC camera
thank you to reponse this project my Graduation .....Yes I read I want to check only....so we need fpga and camera NTSC and monitro !!!!
@@ahceneouachemi1389 did you read the report? YOu need a DE2 board, a VGA monitor, an NTSC camera, and of course quartus software
Does he work by xilinx ise sir !!!!
It would mean completely rewriting the verilog, assuming that the xilinx device includes a video codec
yeah that's awesome
i want to thank you for this vedio, can you help me to write program code " read and write image" by C language in ISE ->EDK->XPS because i am beginer in FPGA and can we talk special
I do not use xilinx tools, sorry
awsome
do this user have an email address?
If you have a quesion, ask it here.
@@ece4760 . I am an undergrad student, and my final year project is titled as “Video Compression using FPGA”. To implement FPGA design, I need some guidance from your side.
I will also like to know that, if I could enroll myself in certain workshop or FPGA learning programs so as to achieve my objective.
I will be using Xilinx Spartan 6 FPGA family for this project. Please comment on that as well. Will I be able to implement the video compression algorithm on this sort of board provided by digilent atlys.
Kindly visit this link in order to get an insight to the board that I will be using. store.digilentinc.com/atlys-spartan-6-fpga-trainer-board-limited-time-see-nexys-video/
Your reply will be a favor
@@The2wistArk The question you are asking depends on your background. Have you studied design of digital sequential circuits? What do you know about orthogonal transform methods? What do you know about compression? Have you read the Xilinx tutorials for the FPGA. What software are you using? Why did you choose this project?
Also, of course, google "spartan 6 image compression". or "spartan 6 tutorial"
@@ece4760 I always wanted to tackle some challenges .. i knw alot about about digital logic design i hv done the course in my undergrad.. plus i dnt know about vhdl yet nd learning and designing the algo is the real challenge!
Genius
wow
that's so fucking great!!
cool
hi I wanna meet u can we
+Ieeeproject Guru Meet? Are you in upstate NY?
can you help for fpga
+Veysel Cambay Depends on what you are asking.
I have DE2 115FPGAs How do I run your program...I try not...
Veysel Cambay did you read the report?
yes ..but I CAN NOT RUN....CAN YOU HELP ME...
Maybe, but the only info I have on the project is the report
what you do now guys? 10 years last at this moment)
we now use a DE1-SOC. See course at people.ece.cornell.edu/land/courses/ece5760/index.html
Also check out Van Hunter Adams youtube channel. www.youtube.com/@hunteradams9430
I want to implement this project but I am new to camera interfacing to fpga and i am not able to find much which help me understand from the basic ...it would a great help if you could share any helpful link (related to camera interfacing with fpga)
+Kushagra Gupta Did you read the report?
I too am interest in this. Where is the link to the report?
did you read the video description?
+Kushagra Gupta If you read the description's pdf you can see that she sent the captured frames of the camera through composite(I think ALTERA DE2 has video input), and then did all the algorithms and stuff over that, at least that's what I understand.
That is correct. The DE2 has composite video (NTSC, PAL, SECAM) input. The hardware codec output is sent to the FPGA where the image is passed thru to DRAM, then analysed on the FPGA