Object tracking with YOLOv8 using Jetson Nano
Вставка
- Опубліковано 31 тра 2024
- In this tutorial I explain the theory of tracking algorithms and how to use them with YOLOv8 and Jetson Nano.
Jetson Nano jetpack version: R32.7.3
The project is here:
drive.google.com/drive/folder... - Наука та технологія
Awesome video as a learning resource. I was trying a similar thing, just wanted to know if you ran into any low memory situation. If so, how did you overcome it?
Hi Najib Murshed!
Thanks for watching my video!
Yes, I had low memory situations. I recommend you not open any windows or run programs that are not related to execution of your code. For really hardcore usage, you may delete GUI-related programs of jetson nano and access via SSH.
great video Thank you
Hi Faruk Cinemre!
Thanks for watching my video!
It is my pleasure if this video has helped you.
Wow I was trying to figure out how to do this yesterday and you published this yesterday. Nice.
I haven't looked through the entire video yet, but would it be possible to move a second sensor like a laser to wherever the object is, instead of a camera?
Also, do you know a good place to learn robot joints and mechanics? I know Solidworks and I have and know how to use a 3D printer, but I don't know how to make my own joints and the like.
Hi Racecar Johnny!
Thanks for watching my video.
It depends on your use case, but I think it is possible to move a laser towards an object.
I think you should just learn basic physics and mechanics to understand how joints work. If you are planning to make a robot with joints, I recommend you to buy parts that exist on the market since you cannot make a joint (like a spherical joint) with a 3d printer, you will need lathe and milling machine for that.
hello. You posted a great video, thanks. Could you tell the fps of this for detection and tracking, respectively? I read in the comments that it was aroun 6-8 fps, but you didn't mention what task, detection or tracking. Also, have you ever tested with other better edge devices such as jetson orin nano/nx/xavier? If yes, I would love to hear about them. Keep posting such videos
Hi Sokhib Tukhtaev!
Thanks for watching my video!
It gives 6~8 fps for detection. For tracking it is slightly slower since in this tutorial all calculations are done in a single thread. I made several tutorials with Xavier and Orin Nano, but haven’t done quantitative evaluation for fps.
appreciat it, @@robotmania8896. Hope to see more such educational content
Hey together and thanks for the nice demo. I was wondering whether the standard jetson nano is capable to run a bigger YOLOv8 model than YOLOv8n (s/m/l) in Real-time. I am currently working on a raspberry pi and I am frustrated with the computing speed. A quick feedback would be really nice. BR
Hi Johannes Bernhard!
Thanks for watching my video.
Regarding operations such as inference that involves GPU, jetson nano is much faster than Raspberry pi. I think with Jetson nano, YOLOv8s is the most that you can run. With bigger models, inference time will be very long (more than 1s) or it may not even run because of memory shortage of Jetson nano.
Hello, would like to ask but do you have a simple code that shows just the object detection? Thanks a lot. Been working around with your code but I can't seem to fix my pyrealsense2 issue.
I haven’t published any simple code that just shows object detection. But in case of yolov8, the code should be very simple. I think it will be like this.
from ultralytics import YOLO
model = YOLO(“model_directory”)
model.predict(“source”, save=True)
@@robotmania8896thanks for your videos! Just a comment on that, I wasn't able to run yolov8 on Jetson nano cause python3.8 is now required. I did run yolov5s though, with my custom model using python3.6.9, but now python3.8 is also a requirement for yolov5 package. Have you tested yolov8 on Jetson nano recently? Is there a way to overcome this? Even using a venv with python 3.8 did not work for me, as torch couldn't work. Thank you again
@@elizabethb5615 Hi elizabethB!
Thanks for watching my video!
No, I haven’t tested yolov8 with Jetson Nano recently. What error exactly did you get? At the ultralytics GitHub page it is written that PyTorch>=1.8 is required. Since in my tutorial we install PyTorch 1.8, there should not be problems from PyTorch version side.
@@robotmania8896 It is python 3.8 what causes the issue with jetpack. Not pytorch. My Jetpack works with python3.6.9 but yolov5 (as yolov8) requires python3.8 now. I had no problems with pytorch and python 3.6.9 previously to run Val.py and detect.py. I think it is pytorch 1.8. So, what I am trying now is to run yolov5 with an older repo that run with python 3.6.9. I will also try your video steps inside a virtual environment and give u updates.
Hello, please I have a question? When I use jetson nano to connect to a separate screen, can I set it on the separate screen? Or do I have to set up jetson nano on my laptop before using jetson nano on the separate screen?
Hi Nhật Phạm!
Thanks for watching my video!
You can set jetson nano up through SSH if you know IP address of jetson nano. But generally, I recommend you to connect monitor to the jetson nano and do all the required settings.
@@robotmania8896 thank you so much
Hey there! Awesome video. I have a question tho: I got this custom trained yolov8n model and i want to migrate the .pt file to my jetson nano and i can seem to get it to work. Could you please help me with a step by step guide? Thanks
Hi Marius!
Thanks for watching my video!
If I understand correctly, you would like to use your custom model with tracking script. Am I right?
Cool video, I have two questions: When robot will move is it still able to object tracking? What fps Jetson Nano gives you for object detect and track? Thanks!
Hi antoniospr!
Thanks for watching my video!
Yes, if the robot is not moving too fast, it should be able to track objects. FPS depends on YOLO model size. In this tutorial, fps is about 6~8.
Hey Man, for this project what library did you use to control servos? or communicate via i2c?
Hi Naz Louis!
Thanks for watching my video!
I used Dynamixel SDK. Communication is done using TTL.
@@robotmania8896 nice appreciate it!
sir, is it possible to convert to RT tensor model to speed up recognition? I mean when switching to tensor model RT will still control and track the object like this video. thanks a lot.
Hi minhhnguyen88!
Thanks for watching my video!
I have never tried to convert yolov5-obb to RT tensor model myself, but this repository may help you.
github.com/fish-kong/Yolov5-obb-Tensorrt-Infer
@@robotmania8896 i have watched a few other video tutorials but then i won't know which file to add my motor control to. That's why I want to ask you - who is more experienced than me. I'm a newbie so I don't know yet. very grateful if you can make such video.
@@minhhnguyen88 I am not very familiar with TensorRT too, so I cannot give you any valuable advice. Thank you for a suggestion though, I will consider it.
@@robotmania8896 Thank you very much
Great video my friend, have you worked with triton inference server?
Hi Guillermo Velazquez!
Thanks for watching my video!
I have never worked with triton inference, but I want to look into it.
I was wondering is it possible for this to work faster? Saw video from 3 years ago and car was following a ball in real time.
Hi Music Videos!
Thanks for watching my video!
Yes, there are several ways to make it faster. For example, we can divide inference thread and servo control thread instead of doing all stuff in one thread. Also, we just can use faster devices such as Agx Xavier. BTW, in the ball following task, they might not be using inference which makes control frequency much higher.
What is inference time per frame?
And which version of yolo did you use? - nano?
Hi Alex Polymath!
Thanks for watching my video!
Inference time per frame is about 100~120ms. Here I am using yolo version 8.
Wow, it's a nice tutorial. Would you mind making a tutorial on object tracking using a camera Gazebo like your tutorial "Object detection in Gazebo using Yolov5 and ROS2"?
Hi jafar3276!
Thank you for the suggestion, I will consider it!
I have trained a custom model on multiple classes using YOLOv8 in google colab. Can you please do a video on how to deploy that on Jetson Nano? Please I have project submission in 4 days and I am stuck into this.
Hi Adnan Ahmad!
Thanks for watching my video!
If you have a trained model (pt file), put it into the “yolov8_tracking” folder and then specify it while executing python script. Please watch from 19:00.
Weird but i dont see any pyrealsense2 folder in my python3.6. Just the dist-packages. I couldn't import pyrealsense2, too. Need help
Hi melonstreams9369!
Thanks for watching my video!
Has the installation process of librealsense been finished successfully?
@@robotmania8896 yeah. I created the object successfully
Hello sir, I am unable to execute it. It is giving me NO module named 'yolov8.ultralytics' i have cloned the repo what u gave but still unable to use the yolov8 model. I have a python3.8 environment and i have installed everything that u have given in the video but still unable to run. Hoping for you reply sir. Thank you sir.
Hi TEAM ACE!
Thanks for watching my video!
Are you using the zip file I have provided? There should be a “yolov8” folder and an “ultralitics” folder inside it.
hi, great video. Could you do a video for using YOLO OBB on ROS? Knowing orientation of bounding boxes could be really useful in robotics
Hi Luca Reitano!
I have already made several videos related to usage of YOLO with ROS.
ua-cam.com/video/XqibXP4lwgA/v-deo.html
ua-cam.com/video/594Gmkdo-_s/v-deo.html
May be these videos can help you?
@@robotmania8896 yes, i have seen them. However YOLO OBB (the most famous object detection with rotated bounding box) is unofficial and it doesn't work by simply weights substitution. For this reason i have asked to you.
@@lucareitano2737 I understand. I am also interested in rotated boundary boxes, so I will definitely look into it.
@@robotmania8896 I have tested OBB repo for training and inference and works perfectly. I want to use it for detect object orientation in pick and place operation, we can cooperate together. Thank you
@@lucareitano2737 Thank you for the suggestion. Unfortunately, my plans depend on busyness of my main job, so it is difficult for me to plan to cooperate with someone. But I am also interested in using yolo-OBB and MoveIt to solve the picking problem. So maybe I will make a video on this topic in the future.
nice video sir, i want to know how to implement camera tracking from using dynamixel servo to using ordinary motor servo. i still cannot figure it out and i am using TD8120MG.
Hi Aydin Rizqu!
Thanks for watching my video!
If you are using an ordinary servo, please use this library to operate it.
github.com/JetsonHacksNano/ServoKit
Hi i tried installing numpy. However there is a runtime error saying running cythonize failed. Why is it happening. I am using python 3 version 3.6.9.Is that the issue
Hi vishnu sudheer!
Thanks for watching my video!
Have you installed cyhon? This error may happen because you haven’t installed cython.
@@robotmania8896 I installed cython and the installation was successful. Can you help
Are you using jetson nano devloper toolkit and may i know your jetpack version
Hi Prabu s!
Thanks for watching my video!
Yes, I am using jetson nano developer toolkit. In this tutorial I used Jetpack version 4.6.
Bro i followed all procedure but when I run the last command in terminal, it doesnt do anything and return "Killed" after sometime.
I am doing object detection using custom YOLov8 model.
Hi Adnan Ahmad!
Thanks for watching my video!
Probably that happens due to memory shortage. Please close all other applications and execute the code.
@@robotmania8896 bro there is no other application opened.
@@adnanahmad4168 I used 4GB jetson nano for this tutorial. If you are using 2GB version, it may stop due to memory shortage.
@@robotmania8896 i also used 4GB Jetson with 64GB sd card
@@adnanahmad4168 In that case, I don’t know what could be an issue. If you are using a monitor, you can unplug it and try to run the code using VNC viewer, for example.
why i dont have the folder with pyrealsense2?
Hi Troyano8718!
Thanks for watching my video!
To install librealsense, please watch from 11:20.
have u fixed this? i also don't have the folder
Hi. I was trying to make a system for detection of drones using yolov8.can you tell me how to do it on jetson nano
Hi vishnu Sudheer!
Thanks for watching my video!
Basically, you should follow this tutorial. You just have to prepare your own dataset to recognize drones.
But I think detecting drones using camera will be quite difficult, since fling drone is just a black dot in the sky.
@@robotmania8896 I will try making it. Hope I will get some help if I get stuck. Because I am an actual noob in this and i am trying to learn
Can you also tell me what change i should make in the process if I am using a logitech 4k hdr webcam
@@vishnusudheer6049 If you are using webcam, please use “track_webcam.py” script. It should work with usual web camera.
Sir actuallly ultralytics(YOLO v8) supports only python version>=3.8 but here the default python3 version is python 3.6.9 So how did you manage to run the code because when I tried to install ultralytics library it showed error message.
Sir It would be very helpful if you try to help me with this issue :)
Hi Tejesh Kumar G N!
Thanks for watching my video!
If you are using python 3.6.9, you can not install ultralytics using pip. To do inference, please follow instructions in the video.
@@robotmania8896
what do you mean by inference sir?
I watched the full video. Really can't get it.
Kindly help me with this sir. :)
@@TejeshKumarGN-xg4jd By inference I mean recognition of objects. I think you should be able to run object detection even with python 3.6.9.
Hello, sir. I'm a college student majoring in robotics in Korea. The video was well described, so it helped me a lot in my studies. I checked that the results ran well and motor control was done according to the video. I studied using the file track_realsense. I'm trying to do other yolo files, but I'm having a hard time. Can you tell me a solution?
I changed the code to the yolo file name to use part 'Yolo_weight=weight/yolov5m.t' and the object information was the same. I'm having a problem here
Hi 김지환!
Thanks for watching my video!
What do you mean by “object information was the same”? You mean you used your own pt file but it hasn’t recognized objects?
@@robotmania8896 There was a little mistranslation using the translator. I'm sorry for this. I changed the pt file I made and ran it. The file I made did not run yolo, but it recognized the object like the video.
@@user-ie8dh8md2m So, you would like to say that you have made a custom model, but yolo still recognizes a class that was defined in the old model (which I have provided).
Hello friend, I really like your videos have helped me to develop my project I admire your videos, even that I would like to know how I can change the motors to stepper motors in the code, I would like to mount it to a drone. 😄
Hi Jairo_roke!
Thanks for watching my video!
If you have a library in which you can specify moving angle for stepper motor, you just have to replace lines 350 and 351 in the “track_using_servos.py” script with your function.
how to send usart with usb TTL to stm32?
Hi Zayn!
Thanks for watching my video!
Unfortunately, I am not very familiar with with STM32. But I found a tutorial that might help you.
embedded-lab.com/blog/stm32-serial-communication/
Suggestion: put all those libraries into a single file in the repo
Hi Al!
Thank you for your suggestion, I will consider it.
How to run with csi camera
Hi Kiên Nguyễn Bảo!
Thanks for watching my video!
I have never used csi camera, but I hope this article will help you.
www.hackster.io/salahuddin/nvidia-jetson-nano-with-csi-camera-9c6c59
why pyrealsense not include this file?
What do you mean by “pyrealsense not include this file”?
@@robotmania8896 i change bash.rc pyrealsense2 not include in folder python3.6
Traceback (most recent call last):
File"", line 1, in
ModuleNotFoundError : no modul named 'pyrealsense2'
@@aryalfanani8196 This error means that python can not find “pyrealsense2” library. It is difficult to say what is wrong only from this error message. Did you fallow all steps described in the video?
@@robotmania8896 yes i follow this video
I expected Jetson Nano record to be smoother
Hi diamond!
Thanks for watching my video!
This is the best that jetson nano can do, I guess.
Hello sir, could i send you mail, i need help about this project.
Hi Umut Baran Akkaya!
Thanks for watching my video!
Yes, send an e-mail to this address:
robotmania8867@yahoo.com
can't go through at 11:04
Hi M4rsquake!
Thanks for watching my video!
It will take time, so be patient.
@robot mania bro, I executed the yolovn.pt model and it rn down successfully this time. But when i CHANGED THE MODEL TO CUSTOM MODEL I GOT THE FOLLOWING ERRORS,
YOLOv8x summary (fused): 268 layers, 68132235 parameters, 0 gradients, 257.4 GFLOPs
Traceback (most recent call last):
File "track.py", line 365, in
main(opt)
File "track.py", line 360, in main
run(**vars(opt))
File "/home/jetson/.local/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "track.py", line 131, in run
model.warmup(imgsz=(1 if pt or model.triton else bs, 3, *imgsz)) # warmup
File "/home/jetson/yolov8_tracking/yolov8/ultralytics/nn/autobackend.py", line 354, in warmup
self.forward(im) # warmup
File "/home/jetson/yolov8_tracking/yolov8/ultralytics/nn/autobackend.py", line 255, in forward
y = self.model(im, augment=augment, visualize=visualize) if augment or visualize else self.model(im)
File "/home/jetson/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jetson/yolov8_tracking/yolov8/ultralytics/nn/tasks.py", line 194, in forward
return self._forward_once(x, profile, visualize) # single-scale inference, train
File "/home/jetson/yolov8_tracking/yolov8/ultralytics/nn/tasks.py", line 57, in _forward_once
x = m(x) # run
File "/home/jetson/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jetson/yolov8_tracking/yolov8/ultralytics/nn/modules.py", line 407, in forward
self.anchors, self.strides = (x.transpose(0, 1) for x in make_anchors(x, self.stride, 0.5))
File "/home/jetson/yolov8_tracking/yolov8/ultralytics/yolo/utils/tal.py", line 192, in make_anchors
stride_tensor.append(torch.full((h * w, 1), stride, dtype=dtype, device=device))
RuntimeError: CUDA error: the launch timed out and was terminated
Sentry is attempting to send 2 pending events
Waiting up to 2 seconds
Press Ctrl-C to quit
PLEASE HELP ME WITH IT
Hi Adnan Ahmad!
I have never encountered such an error. But I found this post.
forums.developer.nvidia.com/t/jetson-nano-runtimeerror-cuda-error-the-launch-timed-out-and-was-terminated/198530
I hope it will help you.