How the DJI FPV Video system works! (Air Unit, Vista and FPV drone)
Вставка
- Опубліковано 25 лип 2024
- Find out how analogue FPV works. This is the first in a series of videos explaining different FPV video systems. Subscribe for HDZero/Sharkbyte up next!
Check out Patreon for Discord access and sneak peaks of secret projects: / chris_rosser
Check out my frames if you like a scientific approach to high performance frame design!
www.aos-rc.com
0:00 Intro
0:34 Summary of the process
1:37 DJI Camera
2:23 MIPI Interface
4:09 Video encoding and compression summary
5:28 Compression Methods
5:58 Prediction for video compression
7:07 Intra frame prediction
9:48 Inter prediction or motion compensation
11:15 Residual Transformation
12:38 Quantisation
13:51 Luma and Chroma Data
14:42 Digital Video Transmission
16:29 OFDM link
20:36 Digital Interference and retransmission
21:54 Demodulation, decoding and display
22:32 Advantages and disadvantages of DJI's approach
24:25 Outro
I am commenting just to promote this guy he is like underdog in fpv world
Lot of effort goes behind making such videos . Glad to see you have analysed this much . Really looking forward to more such videos and keep doing it, inspires me a lot 🤗
Thanks! Sharkbyte up next 😁
Finally, someone that teaches about the digital system!! (as an engineering student who also flies fpv this is GOLD!!!)
To add some things I've noticed, to minimize bandwidth spikes (decrease the size of retransmissions e.t.c.) and increase decode stability DJI utilizes what's called intra refresh in h.264. Aka they do not use keyframes, instead it uses what is essentially a rolling block that acts as a keyframe to which other frames can reference. You can see telltale artifacts when Vista switches from 60fps to 120fps mode on arming for example.
It's actually quite an efficient method with least amount of visual artifacts, back in the day 8-9 years ago when twitch allowed you to have fun with encoder settings but something like 1.5mbps max bandwidth, I used to stream with very custom settings that included intra refresh. You can see some examples of the artifacts on my channel in wow videos on 8-9 year vintage. Though DJI uses a lot more granular version of it. Afaik they axed it all a long time ago in the name of compatibility with cheap embeded decoders, tv boxes and such.
Wow that's really great information to have thank you! That explains how they avoid sudeen distortion due to a missed keyframe.
@@ChrisRosser It also really helps for getting latency down, it does not really help with initial connection, since you still need to wait for the refresh to cover the whole screen, but after that you avoid having big data spikes, for keyframes, since essentially you smear that data over time. Also it's not that big of a deal if you cant recover some of them - you'll just have some kind of an artifact in that block until the next refresh covers it up. Also should help recover a lot faster from dropped frames, since you can also just have some artifacts in places that were covered by refresh that was transmitted with those dropped frames.
Also that would allow them to use inter frame stuff, since they do not have to basically cache previous frames to reference, but just blocks between the refresh, thus also reducing latency and increasing efficiency for when you cant have frames buffered.
What's more interesting to me is focus mode, since afaik nothing of that sort is supported out of the box in h.26x series codecs, so I imagine that's their contribution, and since I think they kinda wanted a foviated rendering effect, but they ended up with very sharp transition point, I imagine it's a fairly simple hack ;)
The main downside is probably that generally it needs a bit more total data transmitted over time, compared to regular keyframes and a bit more complicated decode/encode (though I imagine it's not a issue nowdays, since even in 2010 I had no issue playing those streams on android phones of that time). And well.. seeking that type of video wont be instant if you want to avoid artifacts (you need to decode a bunch of previous frames to get a clear picture, or just deal with artifacts), but that's obvisouly irrelevant for realtime stuff.
Woow 😮. Such complex science and engineering. No wonder they sell it at that price.
Wonderful job making such an informative video. Thank you for your efforts buddy 😊
Thank you for watching and posting a kind comment! I really appreciate it.
This is so cool. Thank you for taking the time to make videos like this!
i love this, im trying to figure out the details of how they do it and you did a great overview
a lot of info packed in here, great job!
Appreciate it! I really went down the rabbit hole on this one.
This is a very useful presentation. I really appreciate your hard work!
Thank you for your deep and informative explanations. The amount of research you put in your videos warms my heart.
You are so welcome! I hope you enjoy the Sharkbyte video as well.
Now, they who always say digital the best or analog still thebest, can see cleary how the system works. Advantages and disadvantages always a good point to see, what a great video
Absolutely. There are tradeoffs with every system.
Love the deep dives
Very well done, Chris! :-)
Great explanation. Can't wait for the HDZero one (and I'm going to be pointing people to it in my next video)
Thanks, I just go tthe content approved by Carl from Divimath so we are good to go!
Awesome, cant wait to watch this later
Hope you enjoy it!
This is amazing! thanks for all your hard work. :D
My pleasure!
Thank u for this great video full of infrmation that i did not know
So cool, can't wait for the shark bite analysis
I really appreciate this content. Thank you sir!
My pleasure!
Great video
Glad you enjoyed it!
thank you for this video
Another wonderful video that explains things so well. Thank you again.
But it's hard to believe that DJI don't use error correction. IBM used it for tapes in the '70s, audio CDs in the '80s, DVDs and Blu-ray later, etc. It's so common and so well worked out.
With a small overhead EC could eliminate most of the retransmission activity.
I guess, as always, it's a trade-off between the overhead and the amount of retransmissions needed.
They do use some error correction (checksum based) but if a packet is too badly damaged a retransmission is needed.
Thank you once again. Great presentation as always. I have only ever flown with the DJI air unit as I am fairly new to FPV. After seeing your videos on how Analog works and now DJI I am curious about Shark bite. I am very curious about analog, I think I may want to try it at some point. The DJI system has a nice picture but when it breaks up and the latency increases it can be hard to fly. Once more thank you Chris Rosser!
analog is a mistake for you, use low latency mode, you can fly on dji more more far and penetrate than analog
look a video 25mw analog vs 25mw dji
I've switched from Analogue to DJI and am keen to try shark byte for the first time. I wouldn't go from DJI back to analogue because the image quality is much worse particularly when there is interference.
good stuff! able to stay connected when playback speed is at 2X. I guess another advantage for the DJI system is that no technician license required (U.S.)?
Wish you could do this kind of explanation for the other systems on a quad :P
To be honest, 75% of this video is about generic video/image processing & digital data transmission topics. Nicely presented but not really related to DJI. Looking at the image when the link builds up, it seems very probably that DJI is using a 3x3 scheme of independent video streams. They probably use something SRT. And progressive encoding (there is a scheme for h.264 etc) is somethings that is skipped here completely. Both SRT and progressive encoding are possible explanations why the latency stays quite constant and the image quality degrades so quickly when the link quality drops.
Wrt 120fps: This helps to get the frame faster. It also makes the video much smoother. You can strongly see that when flying fast and using 60 and 120fps alternatively. This is why in high-end VR there is the recommendation for >90fps. So the 120fps deliver an additional advantage.
Great comment! Thanks for all the extra detail. SRT may be a technology they use instead of simple UDP. Progressive encoding seems less likely to me as it seems more expedient to encode each frame with compression matched to the current link speed.
This is not just off the shelf wifi. You can find wifi frontend chips on the boards. That makes sense if you want to build a transceiver on that frequency range. The encoding itself I guess is a progressive encoding h.264 similar to what Ambarella offers as a real time profile. Using SRT one might prioritize the dropped frames that you really need.
One can also see after a video break up that the video is made out of a 3x3 matrix of probably separate streams. The use this in focus mode to reduce the bandwidth on the outer parts of the FoV.
@@Stefan_73 You two sound like you know this stuff well. DJI seems to be focusing less and less on fpv, and I've been wondering why there's not a good competing product out there yet. Is a viable competing digital FPV system really that hard to develop?
Could you do a similar video on how RF data communications for controls operate?
Great suggestion. I'll add it to the list.
thx for the information.
will be cool to see how does sharkbyte/DJI FPV compare after each stage.
example can we compare the compression stage , and latency it introduces ?
That would be extremely interesting. I wonder how it could be done? The DJI system is proprietary so I'm not sure how you could interrogate it for latency information.
@@ChrisRosser I imagine the only way to do it is to get some soft of debug output from them, but I doubt we'll get any of that ever.
Hi there, great video! I'm just wondering if the antennas are still the same? Would any 5.8ghz antennas work? Thanks!
The MAX285x series comes to mind.
Watching your video, that looks pretty darn close if I read its datasheet!
Great video by the way, if you want to test macro block vectoring visually I can highly recommend rotating a gradient at a low bitrate! You can really see the blocks "walk" across the screen guarded by a grid.
I've used patterns like that to tweak bandwidth on crappy ADSL lines for livestream events, it has been a while ago but I learned quite a bit about encoding :)
The modulation scheme kind of reminds me of the lattice of carriers approach that the fast phone modems used. ( a really long time ago )
I'm sure it has it origins in similar approaches!
Has anyone got an aspirin my head hurts. 😀. Thanks again Chris another great video.
Oh no! Don't worry its just your brain gorging itself on information! 😁 I got a cracking headache trying to put the slides together 😋
@@ChrisRosser It’s probably all that old information leaking out to make way for this great new stuff. 😀😀😀
Hi Chris, great video thanks! How does the digital video transmission handles reflected signals ? In analog we get a mess of sinusoids of the current and previous frames but how is this handled in digital transmission ? Thanks again for your video!
Using a circularly polarised signal helps. When a lhcp signal is reflected it becomes rhcp and if your receiving antenna is lhcp it rejects the rhcp signal. The same wish is used with analogue.
@@ChrisRosser Does this mean reflected signals are as bad for digital as they are for analog signals ? As far as I know analog is affected much worse by reflections than digitial transmission. Or is it just the complexity of the analog signal that makes it so bad. Thanks again!
@@hristiantodorov3923 With analogue you have no way of knowing if your signal is good or bad. With digital you can use checksums to discard corrupt data and request a retransmission.
OFDM is the thing that handles reflections and delay spread(ISI) in digital transmission as opposed to analog transmission.
In OFDM each symbol is transmitted on a different subcarrier and for longer time which makes the delay spread less destructive.
Great video and great explanation. Some additional information about the fact that the goggles are also sending packets to the VTX would be valuable. Why is it disturbing analog video more than you would think? (outside of the presumed frequency range of a certain channel). What frequency are the goggles using for TX? Also In what way is the DJI signal encoded, can you recieve it with any other equipment? Some facts about the spectator feature would be interesting in this regard as well. Does the VTX send the data to two goggles or is the first one relaying it to the second goggle? (25mbit to recieve data, 25mbit to relay it to another goggle). Does the first goggle need to establish two bidirectional links and does it mean four different frequencies?
Hey, do you think it is possible to take apart the video transmitter from the dji drone and the camera and make it work in another non dji drone? Or it might be some signal necessary feom the MB for it to work?
I think the system is pretty proprietary. It might be possible but I think just using the air unit or vista is easier.
Just bought the v2 goggles, and vista polar combo. Hooked it up yesterday. Still have not turned it on or activated the dji. I hope it all works, as I just took my analog quad out and it smoked?!?! No beeps or nything when I plug a battery in. But I get video. Might of smoked mu aio board? (nazgul5 v2 led ) I noticed some grass wrapped around the motors. The motors did not feel overly hot though.. Time to finish the dji build I guess.
Yikes, sorry to hear that. Make sure you have a 1000uF cap and a fettec spike absorber on every build. Haven't blown up an ESC since I started using the spike absorber.
@@ChrisRosser It is a bnf nazgul v2 led version (f7 aio) No smoke/smell at all. It has a cap. I LOVE this quad!!
@@ChrisRosser I just plugged it into betaflight. All of the transmitter commands are working fine. Must be the esc board. Good luck finding one of those right now. Can I do a check somehow? something in cli maybe.( complete newb lol )
Hello Chris. Learned a lot from this video. I have a question for you. Do you know the usage case of 2.4 ghz link between dji fpv goggles v2 and dji fpv drone ? Is it the radio link or video link?
I'm thinking to move digital with limited budget and cannot decide paying extra for v2 worth it ? I sometimes see v1 goes a lot cheaper than v2 s . I ll highly appreciate your input at this.
As I know your analog goggles were Aomway commander v1 . I have the same goggles. Never got to try the dji's . What do you suggest me ?
I think the DJI FPV drone uses 2.4 & 5.8GHz and switches between them automatically based on RSSI. If you are not going to buy the DJI FPV drone there is no benefit to the V2 goggles. For Air Unit and Vista V1 goggles are the same as V2.
@@ChrisRosser Thanks for the reply Chris. I'm aware of current use cases but trying to futureproof it. Lets see if I'm able to get it or not 🙏🙏
Is there any relay station or relay device can increase DJI FPV Video signal transmission distance？
I had my thoughts that digital used many more channels I didn't know it was such a high number of them. Talking of the carrier waves.
Yes I was surprised that wifi uses 64 carriers on a 20MHz channel. Sharkbyte uses 1024!
then why is my mipi cable from my hdzero cable which has 12 cables on it you say 4 so 4 then another 2 for power and ground so whats the other 6 cables for if only 4 are needed?
Great video as usual. But I always have this question, if the Vtx sends a packet and arrives corrupted to the goggles, the googles ask for a new packet. If the quad is far away, the packets are transmitted at, let's say 700mWatts, but what transmission power are the goggles using to send the requests at the same distance?
I believe the goggles transmit at the same power as the VTx is set to
@@ChrisRosser but then it should be extremely hot as the Vtx. I guess I should measure the current drawn by the goggles to be sure what's going on in that matter
@@DronoTron the goggles have quite a loud fan that runs continuously. That should keep them cool. Also I think decoding video is less computationally intensive than encoding.
@@DronoTron the goggles are quite a large form factor as well compared to the AU/Vista - maybe that + the loud fan helps dissipate the heat faster?
Makes sense. Hey how about explaining openHD ? It looks like a promising alternative
Colin Briggs in UK
Hi Chris do you intend to build a AOS 5.5 with an extended nose so there are no props in view. My friends and I like to fly with the DJI air unit and camera and not use a GoPro
Hi Colin, if you're not using a go pro you can run 5" props and be confident of no props in view.
I have one question. Is possible to use Kakute F7 HDV (FC) with connected Caddx Air Unit and external radio receiver and then set in betaflight this external receiver as signal source and stream video by Air unit to dji googles at the same time or do i have to use dji controller with air unit to stream video?
In short: Is it necessary to use a DJI controller to transmit video from the air unit to the google?
Not at all. I use my Caddx Vista with ExpressLRS control link on a Taranis radio. I don't use DJI controller.
@@ChrisRosser thanks for your response ;)
I couldn't find relevant information anywhere so decided to give it a try here. I playing little bit with my Vista unit because have problem with DJI FPV Cam. I do connecting and reconnecting camera while Vista were powered. After that I notice that without camera my bitrate is about 10Mbps. Could no image to process cause lower bitrate?
I think so. Compressing a black image will require much less data than an image with detail.
@@ChrisRosser Thanks!
You think it's possible to make a repeater for the dji fpv system? Figured if anyone knows it'd be you. And don't the FPV races use repeaters, or do they all race on analog?
Possible? Of course. But the technology would need to be similar to WiFi repeaters.
Awesome video! But is there a reason why analog systems are still preferable for long range instead of digital?
I'm not sure. DJI Digital can go 13.5km maximum due to a round trip limitation of 90 microseconds: ua-cam.com/video/wj5A_bEZfZ4/v-deo.html
If you need to go further than that then analogue is the only option. Sharkbyte might not have this limitation but you would need a lot more output power than any current Sharkbyte VTX.
Which has better video quality, DJI FPV or an Air Unit with the best camera?
So focus mode on is definitely useful, right?
It just trades off image quality in the middle of the field for image quality at the edge. I tend to run with it off. As often the edge of the screen is where you are looking to spot a turn etc.
I'm wondering at what point the image gets debayered. In cam's image processor, or later on, in vtx?
That will almost certainly happen in the image processing electronics within the camera before the frame is sent over MIPI.
@@ChrisRosser Most probably, yea. But also, 4 lane MIPI-CSI-2 with combo PHY layer should support transfer of raw image data. And you can apply initial processing to raw data in cam using red, blue and (green1+green2)/2 data.
Demosaicing the image in dji vtx would open possibilities to use more advanced algorithm and could help optimize encoding/compression.
I'm really curious how DJI decided to transfer/process/etc. images and what debayer algorithm they are using. Image I'm getting in googles does not look like cheap bilinear debayer to me.
Hey Chris, what would you say is the main reason why DJI is much better than Sharkbyte at penetrating buildings? Are they doing more compression or because they are retransmitting?
Also curious why this advantage does not carry over to penetrating foliage.
A good question. I think that reason DJI does so well in buildings is because it handles multipath interference really really well. OFDM in general handles multipath well because it acts as many separate narrow band signals with low symbol rate rather than one wideband signal. In a building I think DJI is bouncing around the obstacle rather than penetrating. Out in foliage there is no surface to bounce the signal around and so the penetration is similar to analogue.
Digital FPV is pretty much behind compared to the smartphone industry I suppose, I wonder how does it compare. This kind of video I hope could inspire more companies to invest in digital FPV system further. I'm wondering how far the current system (either DJI or SharkByte) could go, as it will need a better encoding / decoding process to achieve better quality, frame rate, latency and range, thus it will result in more hardware cost.
Not sure I agree. DJI have had to develop a custom ASIC to do their digital video encoding at a cost of ~$5m USD. This is pretty cutting edge technology. Smartphones cannot get close to the latency of DJI FPV. The best they can manage is ~100ms. That's why you don't see any clones of the DJI FPV system. Without custom silicon it's a non starter.
I've never seen the data on any other system outside FPV, I just expected that they have a similar system, such as in gaming or video broadcast. But, I can see that it's only within FPV / drone racing that ultra low latency video transmission is important, or at least important enough for DJI to produce it.
Hey Chris. Perhaps you’ve been asked this before but I couldn’t help wonder why the new runcam spilt camera recording system didn’t just split the wires from the camera and have one set go to a recording device and the other to a transmission device. It just seems like doing this should work rather than having the setup they have with one wire from the cam to the recording device and another from the recorder to the transmitting unit. Any ideas?
With a MIPI link the connection is two way so you can't split the feed easily (like you can with analog for example). If you did split the feed each device would only have half the bandwidth. In the RunCam split I believe the link is digital to the recording device where it is both saved to SDcard and converted to analog for the VTX.
@@ChrisRosser thank you so much!!’
Do you think there would ever be a hack or custom firmware on the DJI system?
Never say never, we have USB C video out now!
Somewhere below I read you saying that the goggles can also transmit at the same power as the vtx. Is that even healty to have that kind of transmitter onto your head?
Yes it's totally safe. The radiation emitted by our equipment is non-ionising. It's actually much less dangerous than the visible light from the sun! The only radiation you ever need to worry about is ionising radiation (UV, X ray and Gamma ray) everything else is completely safe (all it will do is gently warm things up) and I wouldn't listen to anyone who says different. 👍
@@ChrisRosser Thanks a lot! This makes me worry a lot less about all that radio tech in my hands and on my head.
Sometimes I feel like I'm too dumb for this hobby. This is one of those times 😆
You'd better explain why in the evening when it's getting dark before arming you have clear picture on nebula pro, but after arming the picture is getting dark and you're not able to fly. On polar this doesn't happen. Is it software trick to force people to buy "night" camera?
May be the frame rate switching from 60 to 120fps. Try high quality mode?
I wonder why DJI don't break each frame into bands and then you have less latency to decode a band, so even if you lag on one band, another band might transmit, you'd get flicker in bands, but perhaps less latency waiting all the data to send and repeat transmission
The engineers engineer
The nerd's nerd!
I'm pretty sure they don't use the "retransmit on first failure" method you've described here - this is something you'd do over a wire, where errors are rare. Digital RF systems, where some percentage of errors is expected in every packet, use forward error correction codes instead of checksums - these allow to correct up to x bit errors or decide that there are too many Generally, transmitter can dedicate any share of the channel's bandwidth to FEC, thus adapting to adverse conditions. This is literally RF 101, and it can't be that DJI didn't implement FEC, because obviously they don't produce the OFDM transceiver chips, they purchase or order them from people who know their shit. If the Rf link worked as you described, it... wouldn't work. Like, at all. And even at 1 meter between TX and RX, it would drop some frames constantly. I don't have these DJI goggles, but the huge scandal would certainly break into my bubble if they sold a ~1000$ FPV system unusable by design.
Even more absurd in your description is the idea that DJI retransmit whole frames... Come on, that's not how any of this works. Why would you retransmit a frame of live video? Receiver would certainly prefer a fresh frame to an old one (even if the old one was an I-frame and current one is a P-frame, really), and you have these new frames piling up in some buffer until you can retransmit... No, they don't do that. If this was the case, it would be noticeably degrade the system's performance, and recordings of "pause then fast-forward" artefacts from these goggles would be all over UA-cam.
Great video Chris! I'm glad to see that you are creating these easy to understand videos on how FPV hardware works. I'm a chip design engineer, and one thing I would add is that people don't understand how incredibly difficult it is to reduce the video encoding latency for use with FPV. Have you seen this interview with Davide Cavion? (ua-cam.com/video/KxqS-Uy76QI/v-deo.html) He explains how the cost of designing and producing an ASIC chip capable of doing what DJI is doing is beyond the budget of most companies that are supporting FPV hobbyists.
when is the mid-term exam?
No exams at the school of rip!
So why has this technology become available only last couple of years and previously no one believed it is possible to make HD FPV???
Millions of dollars spent on custom silicon to do the realtime low latency encoding by DJI and HDzero respectively.
first :)
Second, sorry!
So basically it's stupidly expensive WiFi in broadcast mode with separate encoding for the center of the image and the rest of it
second
Third, sorry!
Not sure the prediction algorithms are actual for realtime image transmission, and I can't believe in retransmission (two way communication) that is too complicated way to build final image, particularly for real time applications. In my opinion DJI simply compress image to a single frame (probably something like a progressive JPEG). Depending on signal quality the quantization coefficients for image blocks could be different because we need a good quality at center and less quality on peripheral.
DJI use proprietary silicon for their video compression and encoding so its impossible to be certain. There will be some prediction going on (even if only intra frame) in order to get the data rate down.
Nah, DJI codec is clearly based on h.264/h.265 that has been somewhat customised, or something very similair. If you messed with most of the codecs it's very clear by looking at the artifacts. For the customisations - focus mode is one for sure. But it's very much a derivative of those optimised for minimal latency and perf.
@@ooHotcooleRoo So would you expect them to be using intra and inter frame prediction along with the periodic intra refresh technique you spoke of?
@@ChrisRosser I imagine so, since I cant really see h.26x type codecs having any decent efficiency without inter frame prediction what so ever, since without it we are only left with i-frames (which are well.. keyframes themselves). Also AFAIK intra refresh is basically a P-frame only type deal. If you are interested in it a bit more there are a bunch of papers describing multiple variations of it (Periodic Intra Refresh) with various adaptive properties, different placements, hybrid approaches wtih having also periodic I-frames too e.t.c. Most of them relate to real time streaming and error resilience.
What they are not using for sure are B-frames (bidirectionally predicted frames), since that would entail at least an additional frame of latency.
But at this point I'm talking things I expect, not things I know, or I'm sure about.
Otherwise considering that the codec is probably h.264 derived and the quality vs bitrate is quite low, they are probably not using that much feature wise, but in i-frame only mode (intra frame predicted) it would be far more bandwidth intensive for sure.
Also it would be fascinating to read on encoder bandwidth control, how much headroom in there in the link and how focus mode works and decided on. Since there were papers proposing foveated algorithms for h.264 for a long time (one of the later ones with bitrate comparisons too - oliver-wiedemann.net/static/publications/msc_thesis_oliver_wiedemann.pdf) and focus mode is clearly a some sort of low effort/fast approximation of the same idea.
Forth
Fifth, sorry!
@@ChrisRosser close lol
25 mins to explain how 26 ms happens Awesome
First!
Yes!
Yay~
I always fly in a small park behind my house. Because it's in a middle of a housing area it must have a lot of WiFi interference. I started flying DJI channel 1 and boy the variable latency is unbearable even at very close distance. It become buttery smooth once I switch to channel 3 50Mbps.
@@LikWeiLee Yeah, 5GHz Wifi can really mess with DJI Video. Glad you were able to find a clear channel to rip on!
@@ChrisRosser Can't wait for your upcoming sharkbyte video. I wonder why it can lock at a fix latency, I'm guessing it is because it doesn't do re-transmit?
So you want to tell us there are idiots out there who say it's magic?
Nah, just a silly thumbnail!