DFT time complexity is O(n log n), but you can find period of string using Knuth-Morris-Pratt algorithm (which is for strings). As a substep of algorithm, it precomputes longest suffix which is also prefix. It's easy to show, that if string is periodic, then period is difference of the length of the string and the length of largest suffix which is also prefix.
How does this work when you have a much noisier dataset that has periodicity but not perfectly, like 123451234612345? I’m looking into tidal predictions and errors between measured water level and predicted water level. There seems to be some sort of periodicity but I haven’t dug into it. How should I?
It can be proven that one is only periodic if the other too and the other way around as well? Very interesting. So it is useful when the sequence is so long that it doesn't fit into the memory? There can be an issue with numeric stability. It needs very high precision.
Great observations! The DFT of a periodic sequence is not necessarily periodic, as the pivots may be all different, but it's divided into these blocks with zeroes. Yes, when sequences are really really long, this is super useful. For example, in Shor's algorithm, which factors very large numbers, really large sequences appear, and the point is to determine their period. It definitely requires high precision. The FFT (latest video!) is a good solution for this.
I think the result of the last row on "A word of warning" is not correct? The numbers should be the same as first 3 rows. Thanks for the great illustration nonetheless!
DFT time complexity is O(n log n), but you can find period of string using Knuth-Morris-Pratt algorithm (which is for strings). As a substep of algorithm, it precomputes longest suffix which is also prefix. It's easy to show, that if string is periodic, then period is difference of the length of the string and the length of largest suffix which is also prefix.
This may be the best exposition of the DFT. Ever.
Was eagerly waiting for this part❤️. I would love to make a elaborated video on this.
Thank you, glad to hear! I hope you enjoy it, and if you make a longer video please let me know so I can take a look at it and share it! :)
How does this work when you have a much noisier dataset that has periodicity but not perfectly, like 123451234612345? I’m looking into tidal predictions and errors between measured water level and predicted water level. There seems to be some sort of periodicity but I haven’t dug into it. How should I?
Why is there no HD version of the video. Thanks anyway for the good content.
It can be proven that one is only periodic if the other too and the other way around as well? Very interesting. So it is useful when the sequence is so long that it doesn't fit into the memory? There can be an issue with numeric stability. It needs very high precision.
Great observations! The DFT of a periodic sequence is not necessarily periodic, as the pivots may be all different, but it's divided into these blocks with zeroes.
Yes, when sequences are really really long, this is super useful. For example, in Shor's algorithm, which factors very large numbers, really large sequences appear, and the point is to determine their period. It definitely requires high precision. The FFT (latest video!) is a good solution for this.
I think the result of the last row on "A word of warning" is not correct? The numbers should be the same as first 3 rows.
Thanks for the great illustration nonetheless!
Oh! Yes you’re completely right, thank you! It should be two blocks, one starting with 18 and one with -6. Thanks so much, I hadn’t noticed.