If you want your model to have real validity, you need to find some way to use cross validation, else the model will be over-fit rather than generalized. Without some measurement of expected error ranges and confidence intervals, you're just trusting that a computer is able to replicate the past and therefore can predict the future. For time series, you could break it into chunks where the model is fit to the first 70% of the data and the remaining 30% is used for validation to simulate what would have happened had you built the model and used it to predict things from that 70% point forward. Another option is to randomly drop out 30% in segments of contiguous data and use those data as your measurement metric.
Fourier just extends the past, and nothing fits. Actually no algorithm i have used fitted, and people go wild if a single tick actually matches. I give up on predicting futures 😆
How did you provide that error feedback in the forecast? In order to calcualte the error, you will need the timeseries in the future daterange. The error feedback will help you get better decomposition, I agree. But you have the future timeseries itself, then what are you forecasting?
Perfect presentation Sr. ! I"m very interested to learn more about. Can you indicate a literature or a code to do what you told on 4:21, the outages ? How can be done to compensate the mentioned problem on 4:21 ? I'm trying to figure out how to code it With all respect. Robson
So in that first example, is there any forecasting actually being done at all? Because it seems like he already has the real data and is just doing an FFT on it and filtering out the high frequency components. Where is the past data and where is the future data? It seems to be all past data. Also, when you are able to take the error between the real data and predicted data, then it's no longer forecasting because you have the real data right? So how can you use the described error technique if, in a real forecasting example, you wouldn't have the real data yet so you wouldn't know the error until later? I think this first algorithm isn't explained in a way where we can use it easily. There's a lot of missing explanation around it.
5:14 is bottom line the error? between the red and black curves? it seems the error varies along time, but why the bottom line looks almost horizontal?
This is extremely good example, however i would like to see one which there more irregularities, it may have about 1-2 year long periodicity and we may not have sufficient data etc. Actually I came across such problem recently. Time series decomposition had very large values in the error part.
@@zhangshaojie9790 After a bit of reasoning I reached the same conclusion. The method works very well in particular for reducing the maximum forecasting error in a time series. I found that Facebook Prophet does also a good job in reducing the maximum error.
There appears to be a contradiction in the lecture. One advantage of FFT, he says, is that recursion can be used to fit errors into the model. Then, he says it’s difficult to incorporate “new curves” with FFT. So, why isn’t recursive error fitting using FFT a “new curve” - and what exactly is the definition of it?
Assalamualaikum brothers and sis, I am seeking Software Engineers, whom can code in Tensor flow and RNN Time series. I am paying.. I am based in USA.. Please check out lighttheory.page/ or email us at info@LightTheory.tech SALAM
Does anybody understand the part between 13:20- 13:58 . Don't really understand how encoder decoder things works. How exactly does the historical data in the encoder can be used in the decoder ?
This lecture series on recurrent neural nets by Andrew Ng should help. It explains encoder and decoder, and many other aspects of recurrent neural net architectures. The lectures assume that you already understand the basics of neural nets (a standard feed-forward neural network). ua-cam.com/play/PLkDaE6sCZn6F6wUI9tvS_Gw1vaFAx6rd6.html
I fell like this is way too simplistic for modeling financial time series data. Under extreme financial stress and boom regimes, memory is both very short-lived and extremely long-live i.e. a function of subsequence proximity to said regimes (stress or boom).
not in the additive way you saw in FFT. But if you implement attention in the encoder, you could interpret attentions weights similar to Auto Regression weights
Yea, you are right. This is because some trend market are often manipulated by market makers and such trends usually falls outside the 2 standard deviation move of price. I am glad that he talked about it. I see that his idea might work in a range bound market but not in a trend market.
These methods are interesting however they over complicate the forecasting process. A simple SARIMA model would do the trick, maybe even a Holt's Winter seasonal model. If u want to utilize Fourier terms a dynamic harmonic regression or sinusoidal regression might have been better.
I am sorry my friend , using a fft for forecasting is methodological nonsense De implicit assumption of a fft is that the timeseries is periodic Why would it be?
Maybe he is working with data that they can casually infer is periodic or seasonal trend data unlike perhaps the FOREX or stocks? - I think he states when to use it @6:02
Yes, it predicts an infinite repetition of the input by definition. That's why you have to remove mean and tend before application. Better to use this technique to remove noise, and then forecast by other means.
It always will be. There is nothing more ideal than an algorithm that can teach itself through trial and error, what its supposed to do. The more precise you apply the algorithm, the better the results. I.e. using seq2seq for language comprehension.
Good evening my Professor, Please sir, if we have the Yt series. To study the stationarity of this series, we can do the following decomposition (or filtering): Yt=F(t)+Ut, such that F(t) is a continuous function according to the trend (linen, nonlinear). And if we find the series Ut it is stationary, it implies that Yt is stationary, and the opposite is right? B.w
Well that’s funny. Almost everything advanced(or seems to be advanced) belongs to ‘deep learning’. In my opinion, this is just the state space model or hidden Markov models, isn’t it?
when i was young, we were supposed to learn, arima arfima, arch, armax, state space filter, and all these tools useful for time series.......... nowadays, no need for any skill, just do deeplearning, use tensorflow and/or lstm, and all the problems will be fixed ( whatever the problem, supplychain, wheather, finacial forecasting, ...)........ and that's the same for multidimensional analysis, econometrics, and so on........ sad.......... really sad i just made a comparison between a state space model, and a lstm... no need to say who was the winner, who gave a result nearly immediately, who did not need coding and debugging too much, who....
Are you saying that the various types of neural nets aren’t getting the job done as well as the old algorithm, or are you saying that they are inefficient in terms of man-hours (debugging and tuning) and computation? If the case is the former, then we should revive the old techniques, if the latter, then it’s completely sensible to keep the trend moving. Apart from nostalgia, I can’t say the abacus is the better application than the calculator on my phone or that paper-books are better for my learning than e-books on my G-Drive. Technology will keep on progressing, and we humans must adapt. If ARMA and GARCH must now live in museums, so be it! We who have had opportunities to work with them can always go visit on our own time.
There are still some definite advantages to the "old" style of model building. With deep learning, all you get is the result. Using techniques like ARIMA, GARCH, and partial differencing, you can get a lot more insight into the behavior of the data. You know what kind of AR and MA models you have, what that implies about the source, etc. With deep learning though, all you get is the point forecast and maybe an interval. It might be better for forecasting with a stable information set, but to be able to predict the impact of outside factors using explicit weighted coefficients and models is a lot more helpful for less informationally stable data sets.
i'm saying both of them......... for the computational time, i have no doubt that you gave it a try....... for the accuracy, i try things when i have got time.... i have no idea of the experience you have in the field of forecasting things in the ' non academical' field........ when i taught at the university, the first thing i said was what i was about to show was on data ' calibrated' for students to understand... when you show data in the real life ( retailers, to name things) , things are differents ........ erp and aps integrate ' forecasting tools' as welll...... they always give a result..... mmmmm is it useful? well... it depends....... most of the time when you want a solution, you have to build it on your own, step by step....... just for the fun i tried an lstm model on the tuberculosis data in china, which is a dataset you will find on the web..... make the comparison with a model from the sixties (!!!!!), and we will probably discuss some more about nostalgia! :-).......... eventhough when the lstm is properly calibrated, the old man runs faster and longer! ( ok, not much...)......... last but not least, i hope you have the mathematical background.... so you know that AI often reinvents the wheel, and gives anew name to existing things ( ok ok, i was not talking about cnn) never forget that the use of genetic algorithms, tabu search, or simulated annealing was supposed to dramatically improve things...... well..... cheers mate
definitely.......... most of the people forget that ann are blacboxes.......... when they work, it's perfect....... in my professionnel life, i never met anything perfect ( well, in more than 20 years, i had the time!)........ with an ann, trying to see were the problems are is impossible, the only thing you can do is change your model ( ann, cnn, kohonen, counterpropagated, and so on) or change the parameters ( learning rate, noise, number of layers, hidden units...) with old models, you can see step by step what's happening..... last but not least, in the real life, most of the time, when you want to fix a problem, you try several kind of tools.... a tool suitable for a probelm will not work for another one, which seems to be similar.......... you can't do that with neural networks... as an example, let's point out what you said.... your time series is not stable in it's parameters ( i'm not talking of integrated modelling...) tests will tell you there is something ' to monitor'........ nothing like this with an ann.... as new data are forecasted you can just see that the more you forecast, the bigger the error..... and you ' retrain' it ' to learn' ( well well well !!!) cheers
1. To perform Fourier analysis on a dataset it has to be L1-integrable. In the presented example, the time series is not L1-integrable. This method is good for 1st-year students, not for serious people. In such an example you should use proper models. 2. Did he just hugely overcomplicate the idea of autoregressive modeling?
@@mitalpattni1977 There is no trend but that is not what I mean. I'm trying to say that he has no idea that he is not allowed to use Fourier analysis for this type of data, because the data is not fulfilling the basic formal conditions required for Fourier analysis, such as L1-integrability (I hope this is the word, English is not my native language :) ). It looks like he has no education in calculus, signal theory or forecasting, and he is excited about the first data analysis tutorial that he found on youtube, but he doesn't even know that methods cannot be used freely without concern, you have to follow the rules.
@@jacekwodecki3530 hey jacek I believe you vocab is correct I just googled it, but I couldn't find a good explanation except for some definitions of L1 - integrable. It would be great if you could share some material on it.
What's different from Generalized Additive Models where smoothers are used? What he presents is a special case. Not very interesting, neither general. Plus, is a very different thing to find good techniques for data fitting from trying to understand what drives the stochastic process. Economists are far better to tell you about the second, without doing worse than people who do data fitting for forecasting purposes (call them statisticians, financial engineers, etc).
@@stefanosgiampanis4898 the well known book by Hastie. Otherwise, in a GAM model use f(X) = a+bx+ b^2x^2 ... As a very simple example of a nonlinear model
@@safishajjouz166 That makes sense, thank you. The non-linear f(X) is, however, specified (not entirely learned as in the case of a NN). Not criticizing GAMs or arguing in favor of NNs.
![Lp. Сердце Вселенной #60 РОЖДЕНИЕ ЛОЛОЛОШКИ [Финал] • Майнкрафт](http://i.ytimg.com/vi/YoR0pAV9FVQ/mqdefault.jpg)
1:14 Decomposition
3:49 FFT
14:19 Seq2seq
lol
@@iLLt0m it is really useful :D
I learned so much in 14min! Thank you for sharing your knowledge and experience!
Damn! this guy is good.
If you want your model to have real validity, you need to find some way to use cross validation, else the model will be over-fit rather than generalized. Without some measurement of expected error ranges and confidence intervals, you're just trusting that a computer is able to replicate the past and therefore can predict the future.
For time series, you could break it into chunks where the model is fit to the first 70% of the data and the remaining 30% is used for validation to simulate what would have happened had you built the model and used it to predict things from that 70% point forward. Another option is to randomly drop out 30% in segments of contiguous data and use those data as your measurement metric.
@@LawrenceReitan I have not used TensorFlow, so I do not know. When I do AI/ML stuff, I tend to write everything myself.
@@k98killer 💀
If you can't explain it in simple words you didn't understand it. This guy nails it perfectly that even my kid would get it
Stupid quote.
I came in here 2 years after I like my own comment without noticing it was mine 😅
Wonderful presentation, very clear, very precise. Thank you!
Using FFT for forecasting the future: that's just like repeating the past with extra steps...
On behalf of all of us who have tried to use FFTs to forecast the stock market and had a rude awakening: this is spot on.
@@rushthedj306 there is no periodicity of companies prices, so I dont see the point...
@@piotr780 yes that's the point
@Aden Jett why would you post that here in a discussion thread about FFTs and misguided attempts at market prediction?
Fourier just extends the past, and nothing fits.
Actually no algorithm i have used fitted, and people go wild if a single tick actually matches.
I give up on predicting futures 😆
I cant explain it like this. This guy trully explains it. Thanks for awesome video
How did you provide that error feedback in the forecast? In order to calcualte the error, you will need the timeseries in the future daterange. The error feedback will help you get better decomposition, I agree. But you have the future timeseries itself, then what are you forecasting?
I like this very much. Short and packed with actionable information. Thank you!
Outstanding presentation. Thank you.
Perfect presentation Sr. ! I"m very interested to learn more about. Can you indicate a literature or a code to do what you told on 4:21, the outages ? How can be done to compensate the mentioned problem on 4:21 ? I'm trying to figure out how to code it
With all respect.
Robson
I'm glad I actually watched. This is AMAZING
So in that first example, is there any forecasting actually being done at all? Because it seems like he already has the real data and is just doing an FFT on it and filtering out the high frequency components. Where is the past data and where is the future data? It seems to be all past data. Also, when you are able to take the error between the real data and predicted data, then it's no longer forecasting because you have the real data right? So how can you use the described error technique if, in a real forecasting example, you wouldn't have the real data yet so you wouldn't know the error until later? I think this first algorithm isn't explained in a way where we can use it easily. There's a lot of missing explanation around it.
100% agree
5:14
is bottom line the error? between the red and black curves?
it seems the error varies along time, but why the bottom line looks almost horizontal?
maybe this is errors after removing high freq noise?
+1 to the question
Hm, seems like he applied a threshhold to his errors, i.e., error0 = fitted - original; strange_error = {0, when abs(error0) =thr}
Would be great to have some code examples for FFT or seq2seq. Much appreciated, if someone can provide them!
Not code, but Reducible has an excellent video breaking down FFT in detail.
Thanks for the talk. Mind opening.
Amazing!
awesome vid! thank you for posting
The question remains: Why can't it beat the stock market?
is there a playlist for all related videos
Excellent presentation.
Couldn't a wavelet transform be used instead of a FFT to catch high frequency signals?
This is extremely good example, however i would like to see one which there more irregularities, it may have about 1-2 year long periodicity and we may not have sufficient data etc.
Actually I came across such problem recently. Time series decomposition had very large values in the error part.
Very good tutorial. Thank you for sharing!
Great video!!
This is very well explained!
Please help in prediction time circle in STOCK MARKET.
When you say "predict for each component of the time series decomposition and then you recombine the results" how exactly do you recombine them?
Same way they decompose them
@@zhangshaojie9790 After a bit of reasoning I reached the same conclusion. The method works very well in particular for reducing the maximum forecasting error in a time series. I found that Facebook Prophet does also a good job in reducing the maximum error.
There appears to be a contradiction in the lecture. One advantage of FFT, he says, is that recursion can be used to fit errors into the model. Then, he says it’s difficult to incorporate “new curves” with FFT. So, why isn’t recursive error fitting using FFT a “new curve” - and what exactly is the definition of it?
Can someone provide the name of the paper from which the very last bit was taken? (The prediction with the encoder-decoder NN)
This is true Gold!! thx :)))
Assalamualaikum brothers and sis, I am seeking Software Engineers, whom can code in Tensor flow and RNN Time series. I am paying.. I am based in USA.. Please check out lighttheory.page/ or email us at info@LightTheory.tech SALAM
Does anybody understand the part between 13:20- 13:58 . Don't really understand how encoder decoder things works. How exactly does the historical data in the encoder can be used in the decoder ?
This lecture series on recurrent neural nets by Andrew Ng should help. It explains encoder and decoder, and many other aspects of recurrent neural net architectures. The lectures assume that you already understand the basics of neural nets (a standard feed-forward neural network).
ua-cam.com/play/PLkDaE6sCZn6F6wUI9tvS_Gw1vaFAx6rd6.html
You deserve the like bro
You too man
I fell like this is way too simplistic for modeling financial time series data. Under extreme financial stress and boom regimes, memory is both very short-lived and extremely long-live i.e. a function of subsequence proximity to said regimes (stress or boom).
Of you master this you'll have the edge
thank you, very well explained
Got it, Thank you very much!
:)
Thank you for the good talk.
Great lecture!
Can we decompose the time series using Seq2Seq?
i think it cannot
not in the additive way you saw in FFT. But if you implement attention in the encoder, you could interpret attentions weights similar to Auto Regression weights
How about transformer?
It might work somewhat in a sideways market but otherwise time series data is non periodic.
Yea, you are right. This is because some trend market are often manipulated by market makers and such trends usually falls outside the 2 standard deviation move of price. I am glad that he talked about it. I see that his idea might work in a range bound market but not in a trend market.
Thanks
I recommend this.
These methods are interesting however they over complicate the forecasting process. A simple SARIMA model would do the trick, maybe even a Holt's Winter seasonal model. If u want to utilize Fourier terms a dynamic harmonic regression or sinusoidal regression might have been better.
that decomposition called multiplicative classical time series analysis, it's pretty simple, not too complicated.
I am sorry my friend , using a fft for forecasting is methodological nonsense
De implicit assumption of a fft is that
the timeseries is periodic
Why would it be?
Maybe he is working with data that they can casually infer is periodic or seasonal trend data unlike perhaps the FOREX or stocks? - I think he states when to use it @6:02
Why wouldn’t it be?
Yes, it predicts an infinite repetition of the input by definition. That's why you have to remove mean and tend before application. Better to use this technique to remove noise, and then forecast by other means.
Exactly
@@ColocasiaCorm Because it would violate stationary.
excellent!
love it!
so the solution is deep learning again
It always will be. There is nothing more ideal than an algorithm that can teach itself through trial and error, what its supposed to do. The more precise you apply the algorithm, the better the results. I.e. using seq2seq for language comprehension.
ikr lol
Good evening my Professor,
Please sir, if we have the Yt series. To study the stationarity of
this series, we can do the following decomposition (or filtering):
Yt=F(t)+Ut, such that F(t) is a continuous function according to the
trend (linen, nonlinear). And if we find the series Ut it is stationary,
it implies that Yt is stationary, and the opposite is right?
B.w
very nice!
THESE differences, not THIS differences.
Well that’s funny. Almost everything advanced(or seems to be advanced) belongs to ‘deep learning’. In my opinion, this is just the state space model or hidden Markov models, isn’t it?
No markov decision processes are pretty simplistic and don't utilize all the sequence data to calculate the probabilities
Linear regression
I'm not understanding anything I'm new to this field
when i was young, we were supposed to learn, arima arfima, arch, armax, state space filter, and all these tools useful for time series.......... nowadays, no need for any skill, just do deeplearning, use tensorflow and/or lstm, and all the problems will be fixed ( whatever the problem, supplychain, wheather, finacial forecasting, ...)........ and that's the same for multidimensional analysis, econometrics, and so on........
sad.......... really sad
i just made a comparison between a state space model, and a lstm... no need to say who was the winner, who gave a result nearly immediately, who did not need coding and debugging too much, who....
Are you saying that the various types of neural nets aren’t getting the job done as well as the old algorithm, or are you saying that they are inefficient in terms of man-hours (debugging and tuning) and computation? If the case is the former, then we should revive the old techniques, if the latter, then it’s completely sensible to keep the trend moving. Apart from nostalgia, I can’t say the abacus is the better application than the calculator on my phone or that paper-books are better for my learning than e-books on my G-Drive. Technology will keep on progressing, and we humans must adapt. If ARMA and GARCH must now live in museums, so be it! We who have had opportunities to work with them can always go visit on our own time.
There are still some definite advantages to the "old" style of model building. With deep learning, all you get is the result. Using techniques like ARIMA, GARCH, and partial differencing, you can get a lot more insight into the behavior of the data. You know what kind of AR and MA models you have, what that implies about the source, etc. With deep learning though, all you get is the point forecast and maybe an interval. It might be better for forecasting with a stable information set, but to be able to predict the impact of outside factors using explicit weighted coefficients and models is a lot more helpful for less informationally stable data sets.
i'm saying both of them.........
for the computational time, i have no doubt that you gave it a try.......
for the accuracy, i try things when i have got time.... i have no idea of the experience you have in the field of forecasting things in the ' non academical' field........ when i taught at the university, the first thing i said was what i was about to show was on data ' calibrated' for students to understand... when you show data in the real life ( retailers, to name things) , things are differents ........ erp and aps integrate ' forecasting tools' as welll...... they always give a result..... mmmmm is it useful? well... it depends.......
most of the time when you want a solution, you have to build it on your own, step by step.......
just for the fun i tried an lstm model on the tuberculosis data in china, which is a dataset you will find on the web..... make the comparison with a model from the sixties (!!!!!), and we will probably discuss some more about nostalgia! :-).......... eventhough when the lstm is properly calibrated, the old man runs faster and longer! ( ok, not much...).........
last but not least, i hope you have the mathematical background.... so you know that AI often reinvents the wheel, and gives anew name to existing things ( ok ok, i was not talking about cnn)
never forget that the use of genetic algorithms, tabu search, or simulated annealing was supposed to dramatically improve things...... well.....
cheers mate
definitely.......... most of the people forget that ann are blacboxes.......... when they work, it's perfect....... in my professionnel life, i never met anything perfect ( well, in more than 20 years, i had the time!)........ with an ann, trying to see were the problems are is impossible, the only thing you can do is change your model ( ann, cnn, kohonen, counterpropagated, and so on) or change the parameters ( learning rate, noise, number of layers, hidden units...)
with old models, you can see step by step what's happening.....
last but not least, in the real life, most of the time, when you want to fix a problem, you try several kind of tools.... a tool suitable for a probelm will not work for another one, which seems to be similar.......... you can't do that with neural networks...
as an example, let's point out what you said.... your time series is not stable in it's parameters ( i'm not talking of integrated modelling...)
tests will tell you there is something ' to monitor'........ nothing like this with an ann.... as new data are forecasted you can just see that the more you forecast, the bigger the error..... and you ' retrain' it ' to learn' ( well well well !!!)
cheers
Jacob Swe Great insight. What learning materials (books, courses) will you recommend to develop the understanding of time series methods?
1. To perform Fourier analysis on a dataset it has to be L1-integrable. In the presented example, the time series is not L1-integrable. This method is good for 1st-year students, not for serious people. In such an example you should use proper models.
2. Did he just hugely overcomplicate the idea of autoregressive modeling?
I am trying to understand what L1-integrable means, and in this context are you trying to say that there is no trend?
@@mitalpattni1977 There is no trend but that is not what I mean. I'm trying to say that he has no idea that he is not allowed to use Fourier analysis for this type of data, because the data is not fulfilling the basic formal conditions required for Fourier analysis, such as L1-integrability (I hope this is the word, English is not my native language :) ). It looks like he has no education in calculus, signal theory or forecasting, and he is excited about the first data analysis tutorial that he found on youtube, but he doesn't even know that methods cannot be used freely without concern, you have to follow the rules.
@@jacekwodecki3530 hey jacek I believe you vocab is correct I just googled it, but I couldn't find a good explanation except for some definitions of L1 - integrable. It would be great if you could share some material on it.
@@mitalpattni1977 You say that you have found definitions of L1-integrability, so what is that you need beyond that? Happy to help.
@@jacekwodecki3530 what I meant to say was some legitimate article on it instead of stack exchange answers.
That's why it doesn't work in trading.
Wow!
好不容易找了几个有效的例子 实际中没啥用啊
learn statistics and stochastic processes, at least
Nice :)
Sucks being new to this stuff lol
What's different from Generalized Additive Models where smoothers are used? What he presents is a special case. Not very interesting, neither general. Plus, is a very different thing to find good techniques for data fitting from trying to understand what drives the stochastic process. Economists are far better to tell you about the second, without doing worse than people who do data fitting for forecasting purposes (call them statisticians, financial engineers, etc).
Aren't GAM's linear models? seq2seq models are non-linear.
@@stefanosgiampanis4898 no, they can be non-linear. If they were only linear wouldn't bother to comment at the video.
@@safishajjouz166 Can you point me to a reference about how a GAM can be non-linear? en.wikipedia.org/wiki/Generalized_additive_model
@@stefanosgiampanis4898 the well known book by Hastie. Otherwise, in a GAM model use f(X) = a+bx+ b^2x^2 ... As a very simple example of a nonlinear model
@@safishajjouz166 That makes sense, thank you. The non-linear f(X) is, however, specified (not entirely learned as in the case of a NN). Not criticizing GAMs or arguing in favor of NNs.
He forgot the most effective and fastest algorithm for forecasting tabular data
which is?
This is outdated even at the time of release... seq2seq is no longer the best sequence model. Not easy to train, not accurate enough result.
Attention is all you need
Overfitting isn’t forecasting...it’s not necessary and useless
Time wasted.
RNNs are dead
бест опинион