Impeccable piloting and astonishing craftsmanship! I knew you were my favorite Tim for a reason :D keep up the great work. I can't wait to see where you take your plane from here; mostly cause it's usually the other way around! xD
If you haven't started working on the waypoint system consider this: When a (return to) waypoint is designated, start recording the velocity of the craft and its time traveled, to return to the waypoint(s) you can just reverse this velocity (for binary waypoint travel) or have it loop through each velocity transposed by some algorithmn (depending on the last visited waypoint) for loop routines. Just a thought.
I think before you move on to altitude you need to determine the velocity of the plane (perhaps you can highlight key features in the video frame like a tree, building, etc and track how it's position changes over time). Then you can integrate the velocity (calculus is useful here) in terms of time to get altitidue (you only need the vertical component of the speed). Most computer mouses use this technique and a small image sensor to figure out this their position so I recommend research how they work . I also know that the Intel RealSense tracking camera uses 2d images to get velcoity (in the same way a mouse does) and uses velcotiy to get the real time 3d position. I wish you luck!
Good ideas! I don’t think visually tracking elements moving on the screen could determine a velocity though. There are so many confounding factors. To start with, altitude: you wouldn’t see much movement at high altitude and you will see a lot at low altitude, and we don’t even know the altitude at this point. Also, turning would cause things to move on screen, and we don’t know the rate of change of yaw either. There may be some complicated VSLAM algorithms that could do what you are describing, but I don’t think they can run fast enough, and I still doubt if they would work at all when flying high and little is changing on screen. Another consideration is that all of these methods could at best tell you ground speed, but what you need to know with airplanes is the airspeed. Recently, I have been thinking about using control surface authority as a proxy for airspeed. I might make a video on that in the future when I start working on that, but I have other tasks I need to complete first
Woah, nice work! It will be interesting to see how you approach the altitude issue. It doesn't seem like there's an easy approach. I wonder if a second camera that always faces down (two dimensional gimbal) could infer altitude by the speed of object movement through the field of view. You could also model the upper air windspeed and direction by subtracting what you expect from your plane given its performance, (turning radius and level flight considering engine output). A long climb out into wind would allow it to start building such a model that can be updated throughout the flight. I think that's how I'd approach it.
Exactly, it's not an easy issue to solve. Using the inferred ground speed of the plane could be problematic; if there's wind, this wouldn't be accurate. I'm wondering if there are any machine learning models that could be trained to infer segments of ground that are "far away". Even that has problems though. It might be slow, and what happens when the ground is not visible?
@@LimabeanStudios you’re right, it would work. However, my original plan was to only use computer vision, just as an exercise/challenge. I might end up abandoning that idea and incorporating some standard sensors.
WayPoint mission only with computer vision reminds me of back in the day flying my quadcopter in a familiar area was easy to find home because my vision provided the sight of familiar landmarks but if you are flying somewhere for the first time you need to follow a road or some type of a bearing in order to find home again, right? Seems storage of visual landmarks will need to be recalled same with flying to a WayPoint from scratch without a compass? Good way to get loss, it seems? I am really impressed with what you've done so far!!!
Flying a waypoint mission with computer vision would be challenging for sure. Not sure if I will be able to pull it off. I typically fly at the same field, so I could train it to recognize the corners of the field. I might also abandon that idea altogether. I just bought a GPS module, so I might use that instead.
because it's useless unless you wish to do aerobatics (assymertic flight) or if you must have a perfect lateral control during crosswind takeoffs and landings 😅 => I think both cases are irrelevant in your case (basically mono speed aircraft. takeoffs and landings from/to whatever ) Never forget that the best way to avoid a system failure, is by deleting it lol. Go on simplicity. Use programmable/mechanical aileron differential to avoid lateral dissymetry during flight ... but maybe you have no idea of how to adjust such parameters ... ?!? If your airplane has issues with lateral stability, increase the vertical surface. An approximate formula here Va*l/b*Sw=0.035 where : Va=vertical surface (rudder and fin) l=distance from the forward 1/4 of the wing's median chord to the forward 1/4 of the vertical surface's median chord b= wingspan Sw= wing surface all figures must be in metric decimeters. I really love your project. Ask for more info if you wish :) GL
Great project, but I dont think it would be able to fly high high because at that point, there would be no object to determine the horizon. Do you have any ideas as to how you will fix this?
The plane would still be able to see the horizon when it is flying high. However, I believe that as it flies higher, it might need to aim for some point above the horizon in order to maintain level flight; pointing directly at the horizon would cause it to go down. If this is the case then plane needs an altitude sensor to help it deal with this effect. I may never be able to test this since I am using an RC plane that cannot legally fly high.
i badly want this kind of stuffs for my project in thesis. especially UAV projects for reconnaisance using raspberry pi. is it possible for me to build it from scratch? even the plane itself?
No, but I would like to experiment with that. I looked into Coral TPUs a while back, and they were either unavailable or super expensive due to global supply chain issues. They aren’t necessary for what I am doing right now, but if I want to run an ML model, they would be critical.
May I ask how is the realtime video feed transmitted to you? What is the controller range of such a RC plane and how would you know it is about to go out of range? What happens if it fly out of range? Lost forever? Thanks!
There are two cameras on the plane. A raspberry pi camera 2 is used for the computer vision operations. It also records footage onboard, and that is what you see in my videos. It doesn’t transmit footage in real-time. I also have an FPV camera with an antenna that transmits footage back to my goggles. I have never really tested the range, because I fly within a park, but sometimes at the corners of the park, the image starts to get a fuzzy, so I turn back. The airplane transmitter has RSSI, so I get warnings if signal is low, but that rarely happens in practice.
@@timmarkhuff I wouldn't put mountains in the same category as trees and buildings. From the point of view of a bright/dark border finding algorithm, there are places where the (apparent) horizon isn't horizontal. If your plane has sufficient aerodynamic stability, you could calibrate the inclination and height of the apparent horizon while it is gliding without any rudder input. Then, from optical flow analysis of a donw facing camera signal, estimate yaw rate and start creating a "horizon map".
@@andreassittig2772 when you say sufficient aerodynamic stability, are you thinking of planes with dihedrals that naturally tend towards flying level, like gliders? That could be interesting. I have a glider like that, and in theory, my program could control it by only managing the pitch. The roll would be managed naturally by the aerodynamics of the plane. The idea of the downward facing camera is interesting, but I’m trying to keep it to one camera at the moment. I’ve purchased a GPS module, and might just use that to manage the yaw/waypoint finding of the plane, even though that is straying a bit from the original goal of this project.
@@timmarkhuff I'm thinking of all en.wikipedia.org/wiki/Aircraft_dynamic_modes having positive damping. Such behavior is not limited to gliders. In the human-carrying world, I would expect it from the likes of en.wikipedia.org/wiki/Cessna_172 (at idle, not necessarily at full power), not from a en.wikipedia.org/wiki/Swift_S-1 and certainly not from a en.wikipedia.org/wiki/Extra_EA-300. GPS, really? C'mon, there are optical alternatives: en.wikipedia.org/wiki/Celestial_navigation ;-)
Hi Sahil. Altimeters don’t keep a plane flying level; that’s the job of a gyroscope. My system can keep the plane flying level by visually identifying the horizon. Since it doesn’t have an altimeter, it will tend to lose or gain altitude over time. I’ve tried to think of ways to visually gauge the altitude of the plane, but haven’t been able to solve that problem yet
Seems it would have difficulty flying in mountainous terrain where the mountains extend above the horizon line. This reminds me of the AP2000i IR autopilot systems we’d use on RC helicopters 15 yrs ago. Those had an array of 4 IR sensors to detect sky from ground… purely analog. They worked for the most part but not so well on cool cloudy days.
It would struggle with mountains for sure. I’m interested in making a system that uses a machine learning model to find the horizon line. I’m thinking it could look at more features than just relative brightness, and therefore achieve better accuracy.
Wow, applied computer vision to analog FPV, no less. I wonder if we could look at your project the other way, as a cost effective sensor for an existing FC implementations like APM or iNav. Say over CAN, i2c, etc, to supplement accelerometers and gyros. My Esky Eagles (same plane) was a test platform for 1.2W fpv and 21700 cells, but iNav was good enough for me, lol. I can only imagine where you'll take this! And besides where the h%%l did you find a Pi4?! haha
Integrating the horizon detector into an existing flight controller would be a cool direction to take the project, however at the moment I am having fun building my own flight controller. I got my pi 4 a few years ago, before they became quite so expensive and scarce. I feel extra pressure to not crash when the pi 4 is onboard.
@@timmarkhuff Tim I salute your gumption, Sir. I hope your horizons stay flat as Florida, and that your contrast values are always perfect! I can't judge, I'm presently trying to use Ardurover to drift a WPL D12, much, much less successfully than your project lol
It needs the horizon to be in frame in order to work. If it fails to detect the horizon, it will maintain its current control outputs for a short period of time (one second) and then revert to a 0. This behavior helps when it fails to detect the horizon for a short period of time.
hay, awesome work, i am wanting to have OpenCV on Nano size drones with something like a pi zero or banana pi zero for obstacle avoidance proximity, could you help me out
I designate one of the channels on my radio to be for the autopilot switch. It is operated by a two-way switch. On the plane, there is a Raspberry Pi that uses its GPIO pins to listen to several channels coming from the receiver. One of those channels is the autopilot switch. When the program sees that the autopilot switch is pressed, autopilot takes over.
![Дон ДОН, Алаудинов и СБЕЖАВШИЕ из под Курска ахматовцы 😁 [Пародия]](http://i.ytimg.com/vi/dEfCf5IK26Q/mqdefault.jpg)
Awesome work! Truly game changing for the open source radio control community
And there will be more updates in the coming months!
Impeccable piloting and astonishing craftsmanship! I knew you were my favorite Tim for a reason :D keep up the great work. I can't wait to see where you take your plane from here; mostly cause it's usually the other way around! xD
1:38 very cool set up with the FPV goggles!😮
NIce to have flap turns without disengaging autopilot. Very cool project.
Good job with that system! You've actually explained this complex system so good and easy to understand
Great work, Tim!
If you haven't started working on the waypoint system consider this:
When a (return to) waypoint is designated, start recording the velocity of the craft and its time traveled, to return to the waypoint(s) you can just reverse this velocity (for binary waypoint travel) or have it loop through each velocity transposed by some algorithmn (depending on the last visited waypoint) for loop routines.
Just a thought.
Very cool project! I would love to see a future version that uses a down facing cam and a offline map to calculate position :D
Will be my new favourite channel.. Keep it up
Thanks for watching!
Awesome stuff
Thank you! I just released the next video in this series. Check it out if you are interested: ua-cam.com/video/BQiIkhdTP4o/v-deo.html
I think before you move on to altitude you need to determine the velocity of the plane (perhaps you can highlight key features in the video frame like a tree, building, etc and track how it's position changes over time). Then you can integrate the velocity (calculus is useful here) in terms of time to get altitidue (you only need the vertical component of the speed). Most computer mouses use this technique and a small image sensor to figure out this their position so I recommend research how they work . I also know that the Intel RealSense tracking camera uses 2d images to get velcoity (in the same way a mouse does) and uses velcotiy to get the real time 3d position. I wish you luck!
Good ideas! I don’t think visually tracking elements moving on the screen could determine a velocity though. There are so many confounding factors. To start with, altitude: you wouldn’t see much movement at high altitude and you will see a lot at low altitude, and we don’t even know the altitude at this point. Also, turning would cause things to move on screen, and we don’t know the rate of change of yaw either.
There may be some complicated VSLAM algorithms that could do what you are describing, but I don’t think they can run fast enough, and I still doubt if they would work at all when flying high and little is changing on screen.
Another consideration is that all of these methods could at best tell you ground speed, but what you need to know with airplanes is the airspeed.
Recently, I have been thinking about using control surface authority as a proxy for airspeed. I might make a video on that in the future when I start working on that, but I have other tasks I need to complete first
2:24 ''to trim the angle of attack''. Well technically it's to trim the attitude of the aircraft.
Good catch!
amazing
Woah, nice work! It will be interesting to see how you approach the altitude issue. It doesn't seem like there's an easy approach. I wonder if a second camera that always faces down (two dimensional gimbal) could infer altitude by the speed of object movement through the field of view. You could also model the upper air windspeed and direction by subtracting what you expect from your plane given its performance, (turning radius and level flight considering engine output). A long climb out into wind would allow it to start building such a model that can be updated throughout the flight. I think that's how I'd approach it.
Exactly, it's not an easy issue to solve. Using the inferred ground speed of the plane could be problematic; if there's wind, this wouldn't be accurate. I'm wondering if there are any machine learning models that could be trained to infer segments of ground that are "far away". Even that has problems though. It might be slow, and what happens when the ground is not visible?
@@timmarkhuff why wouldn't a generic barometric pressure sensor work for you?
@@LimabeanStudios you’re right, it would work. However, my original plan was to only use computer vision, just as an exercise/challenge. I might end up abandoning that idea and incorporating some standard sensors.
@@timmarkhuff You know, I spent like 5 minutes trying to I think of why it wouldn't work totally forgetting you were going for pure computer vision.
great
this is awesome, good luck
Awesome stuff man!
Appreciate it!
WayPoint mission only with computer vision reminds me of back in the day flying my quadcopter in a familiar area was easy to find home because my vision provided the sight of familiar landmarks but if you are flying somewhere for the first time you need to follow a road or some type of a bearing in order to find home again, right? Seems storage of visual landmarks will need to be recalled same with flying to a WayPoint from scratch without a compass? Good way to get loss, it seems?
I am really impressed with what you've done so far!!!
Flying a waypoint mission with computer vision would be challenging for sure. Not sure if I will be able to pull it off. I typically fly at the same field, so I could train it to recognize the corners of the field. I might also abandon that idea altogether. I just bought a GPS module, so I might use that instead.
Any plans to eventually intergrate this into something like Ardupilot? Super cool work, especially without an IMU.
I am interested in doing that, but at the moment I am having fun making my own flight controller.
Cool project😊 don’t bother with rudder control
Why no rudder control?
because it's useless unless you wish to do aerobatics (assymertic flight) or if you must have a perfect lateral control during crosswind takeoffs and landings 😅 => I think both cases are irrelevant in your case (basically mono speed aircraft. takeoffs and landings from/to whatever )
Never forget that the best way to avoid a system failure, is by deleting it lol.
Go on simplicity.
Use programmable/mechanical aileron differential to avoid lateral dissymetry during flight ... but maybe you have no idea of how to adjust such parameters ... ?!?
If your airplane has issues with lateral stability, increase the vertical surface. An approximate formula here
Va*l/b*Sw=0.035
where :
Va=vertical surface (rudder and fin)
l=distance from the forward 1/4 of the wing's median chord to the forward 1/4 of the vertical surface's median chord
b= wingspan
Sw= wing surface
all figures must be in metric decimeters.
I really love your project. Ask for more info if you wish :) GL
Amazin work, i need one. lol
I’ll let you know when we go into production :)
Great project, but I dont think it would be able to fly high high because at that point, there would be no object to determine the horizon. Do you have any ideas as to how you will fix this?
The plane would still be able to see the horizon when it is flying high. However, I believe that as it flies higher, it might need to aim for some point above the horizon in order to maintain level flight; pointing directly at the horizon would cause it to go down. If this is the case then plane needs an altitude sensor to help it deal with this effect. I may never be able to test this since I am using an RC plane that cannot legally fly high.
@Tim Huff That would make sense. This is a rather interesting and new project I've never seen before
i badly want this kind of stuffs for my project in thesis. especially UAV projects for reconnaisance using raspberry pi. is it possible for me to build it from scratch? even the plane itself?
Are you using some kind of edge computing device like google coral usb accelerator or intel ncs2?
No, but I would like to experiment with that. I looked into Coral TPUs a while back, and they were either unavailable or super expensive due to global supply chain issues. They aren’t necessary for what I am doing right now, but if I want to run an ML model, they would be critical.
May I ask how is the realtime video feed transmitted to you? What is the controller range of such a RC plane and how would you know it is about to go out of range? What happens if it fly out of range? Lost forever? Thanks!
There are two cameras on the plane. A raspberry pi camera 2 is used for the computer vision operations. It also records footage onboard, and that is what you see in my videos. It doesn’t transmit footage in real-time.
I also have an FPV camera with an antenna that transmits footage back to my goggles. I have never really tested the range, because I fly within a park, but sometimes at the corners of the park, the image starts to get a fuzzy, so I turn back. The airplane transmitter has RSSI, so I get warnings if signal is low, but that rarely happens in practice.
@@timmarkhuff Great stuff. Thanks!
Awesome! If you were flying close to a mountain ("bending" the horizon), your algorithm would tend to steer away from it, is that correct?
That’s correct. I’m interested in finding more robust techniques that can deal with things like mountains, trees, buildings and other obstructions.
@@timmarkhuff I wouldn't put mountains in the same category as trees and buildings. From the point of view of a bright/dark border finding algorithm, there are places where the (apparent) horizon isn't horizontal. If your plane has sufficient aerodynamic stability, you could calibrate the inclination and height of the apparent horizon while it is gliding without any rudder input. Then, from optical flow analysis of a donw facing camera signal, estimate yaw rate and start creating a "horizon map".
@@andreassittig2772 when you say sufficient aerodynamic stability, are you thinking of planes with dihedrals that naturally tend towards flying level, like gliders? That could be interesting. I have a glider like that, and in theory, my program could control it by only managing the pitch. The roll would be managed naturally by the aerodynamics of the plane. The idea of the downward facing camera is interesting, but I’m trying to keep it to one camera at the moment. I’ve purchased a GPS module, and might just use that to manage the yaw/waypoint finding of the plane, even though that is straying a bit from the original goal of this project.
@@timmarkhuff I'm thinking of all en.wikipedia.org/wiki/Aircraft_dynamic_modes having positive damping. Such behavior is not limited to gliders. In the human-carrying world, I would expect it from the likes of en.wikipedia.org/wiki/Cessna_172 (at idle, not necessarily at full power), not from a en.wikipedia.org/wiki/Swift_S-1 and certainly not from a en.wikipedia.org/wiki/Extra_EA-300.
GPS, really? C'mon, there are optical alternatives: en.wikipedia.org/wiki/Celestial_navigation ;-)
@@andreassittig2772 celestial navigation? Nice. Might be hard to test that out in my local park though :)
Hey tim, how did this drone to maintain level flight without use of altimeter during autopilot?
Hi Sahil. Altimeters don’t keep a plane flying level; that’s the job of a gyroscope. My system can keep the plane flying level by visually identifying the horizon. Since it doesn’t have an altimeter, it will tend to lose or gain altitude over time. I’ve tried to think of ways to visually gauge the altitude of the plane, but haven’t been able to solve that problem yet
Seems it would have difficulty flying in mountainous terrain where the mountains extend above the horizon line. This reminds me of the AP2000i IR autopilot systems we’d use on RC helicopters 15 yrs ago. Those had an array of 4 IR sensors to detect sky from ground… purely analog. They worked for the most part but not so well on cool cloudy days.
It would struggle with mountains for sure. I’m interested in making a system that uses a machine learning model to find the horizon line. I’m thinking it could look at more features than just relative brightness, and therefore achieve better accuracy.
I am very new to these thing forgive me for the stupid question but i am curious is it possible that it can take off and land on its own.
Currently no, I manually take off and land the plane
Wow, applied computer vision to analog FPV, no less. I wonder if we could look at your project the other way, as a cost effective sensor for an existing FC implementations like APM or iNav. Say over CAN, i2c, etc, to supplement accelerometers and gyros. My Esky Eagles (same plane) was a test platform for 1.2W fpv and 21700 cells, but iNav was good enough for me, lol. I can only imagine where you'll take this!
And besides where the h%%l did you find a Pi4?! haha
Integrating the horizon detector into an existing flight controller would be a cool direction to take the project, however at the moment I am having fun building my own flight controller. I got my pi 4 a few years ago, before they became quite so expensive and scarce. I feel extra pressure to not crash when the pi 4 is onboard.
@@timmarkhuff Tim I salute your gumption, Sir. I hope your horizons stay flat as Florida, and that your contrast values are always perfect!
I can't judge, I'm presently trying to use Ardurover to drift a WPL D12, much, much less successfully than your project lol
Real neat! But what if if looses sight of the horizon?
It needs the horizon to be in frame in order to work. If it fails to detect the horizon, it will maintain its current control outputs for a short period of time (one second) and then revert to a 0. This behavior helps when it fails to detect the horizon for a short period of time.
hay, awesome work, i am wanting to have OpenCV on Nano size drones with something like a pi zero or banana pi zero for obstacle avoidance proximity, could you help me out
I haven't done much with obstacle avoidance in drones, so I may not be much help, however I hope to tackle this challenge in the future.
how to switch on auto pilot mode from manual control,is there any option that already built in transmitter like in this video you turned on the switch
I designate one of the channels on my radio to be for the autopilot switch. It is operated by a two-way switch. On the plane, there is a Raspberry Pi that uses its GPIO pins to listen to several channels coming from the receiver. One of those channels is the autopilot switch. When the program sees that the autopilot switch is pressed, autopilot takes over.
@@timmarkhuff how can i contact you sir, i have some doubts regarding this..are you in Instagram..?can I get your I'd
Unfortunately Raspberry Pi is at scalping prices
do you continue work on this project?
Yes, I plan to publish some updates in the coming months. Stay tuned!
It's pronounced "et. Cetera". Not "eksetera"
It’s commonly pronounced both ways, but maybe we should stay true to the Classical Latin and pronounce the c as a k? 😊