How autocorrelation works
Вставка
- Опубліковано 9 чер 2024
- Part of the End-to-End Machine Learning School Course 212, Time-series Analysis at e2eml.school/212
To use autocorrelation in a weather prediction model, check out the Time-Series Analysis project course:
end-to-end-machine-learning.t...
Slides are available: docs.google.com/presentation/...
Follow me for announcements and update: / _brohrer_ - Наука та технологія
This is simply excellent. I haven't seen a better explanation of this concept.
Well explained! Very natural transition from topic to topic until I accidentally learned autocorrelation.
This is the best video ever on autocorrelation! Thank You!
Thanks Ani!
Woah, thank you, Sir. This short tutorial gets me to understand this entire auto-correlation thing than 10 page pdf explaining with all kinds of formulas .
Thank you very much. This give the insight for ACF, not just mindlessly calculating them
Autocorrelation 7:02
Partial Autocorrelation 9:45
Came to revise Autocorrelation. Ended up revising regression too. Great video.
One of the best videos about acf
The kind of explanation I was looking for! Thanks!!!
Great explanation with such a good example.
Easy to understand too.
Thanks!
Nice video, the best one I've seen yet about the concept of partial autocorrelation!
Excellent explanation with an easy-to-follow example!
Brandon's videos deserve a lot more likes and views. Very underrated.
I watched this 2 years ago. Didn't understand the concept. I watch again in quarantine days.
Cool description, very detailed, with lots of graphs
Best video on this topic. Thank you!
Another great and super clear explanation! Thanks.
Thank you for existing.
And this is where the info on autocorrelation starts: 6:57
You're an hero, bro.
Hello Brandon,
thank you for the explanation. Especially the code helped to understand whats happening.
I'm glad to hear it Bartosz. Thanks.
Very good video. In Numerical Methods it always helps to visualise what you are doing
This was explained excellently!
best explanation found till now
Great video! I liked how you presented the python code that would perform what you're explaining.
Very nice demonstration!
Really helpfull
And straight forward
U r genius man
this is best available in you tube thanks it was very helpful
Amazing video! Very useful. Thanks!
Great explanation, Thank you.
Very nice and clear video!
I really liked the autocorrelation part!
Nice and clear explanation. I will add at 4:30. That the points can fit the line very well but there is no correlation. It is the case when the fitted line is parallel to the x-axis or y-axis. It is the case of near zero variance variables.
Very clearly explained!
Great video
Many thanks!
You covered regression better than videos on regression 👏
Thanks a lot, view the code helpful to understand quickly
this was super helpful
Amazing explanation, subscribed!
Thanks Carlos :)
really helpful video, thank you for this
fantastic explanation
Excellent explanation, thank you!
Thanks yung!
Stellar explanation!
Auto correlation sounds useful for finding data cycles. Like every year temperature correlation, around ~365 days shift~, there is probably a max again. You can find cycles in shifts where correlations peak.
Also thanks for showing how it can be refined the same way as tailor series. Didn't cross-concept that correlation can give differentials weight for interpolation.
nice and concise video!!
This is amazing, thank you
Brendon, I would so love to learn ML from you. teaching code along with concept would definitely decrease the MSE between ignorance and knowledge. Thanks
Thanks Siddarth! In case you're interested, there's a whole collection of concept-with-code courses at e2eml.school/catalog
Good job on this video!
I wish it would have showed up when I searched for information on ARIMA, all the ones that did were pretty poor.
very good video!! Thank you
Thanks for the video!
Where you use the word "shift", when describing autocorrelations, wouldn't it be better to use the word "lag", as I've seen this appear in literature more often and am now wondering whether they're the same thing
they are used interchangeably, as well as the word "delay"
Instead of using list slicing it is a bit easier to use the "shift" method from the pandas library. The video in general is great :)
thanks for the great video, @4:57 should the greater than and smaller than signs be flipped in the moderate correlation plot ?
Thanks mohammed, you're exactly right :)
temp_change is not defined in your code is it? Something is missing there.
great video man
Very nice and explanatory video! Thanks! One thing that you could also do is mention real-world applications. A toy model is always very helpful for me to *understand* a topic but real-world application helps me to also *motivate* it.
Besides that keep it up your videos are super nice! :)
I am also currently learning so I'm not sure if my idea is entirely correct, you could use this for any data set, to see how it changes. You could use the auto-correlation on gas prices to see how it changes with time, on flight tickets to see how it changes on a day to day basis or even on the price of a big mac to see how it has changed over the years.
If in the time series , the residuals on temperatures day i after fiting temp(i-1) are correlated with actual temperatures of day i , then can it be said the there is partial auto correlation with lag beyond (i-1). If that method holds good ,then it could be much easier method of detecting PACF.
But I have liked your video also.
Very very nice explain .. many thanks
🙏
I was looking for good video to understand the concepts and this video has done that. I have the following questions:
- Can we do correlation to compare stock market bottoms or tops? If we compare the market price by its 1 day 1 week prior price then will this help in prediction of top or bottom?
- How can we compare two asset prices? I am thinking of below methods
- Do a ratio of two asset prices and run the n-1 vs n comparison of the ratio
- Do a ratio of two asset prices and compare the ratio vs one of the asset prices
- Compare the % return of each assets against each other on X and Y axis
Let me know what do you think
do you know how to do a white noise time series to test for seasonality?
Amazing video, but the autocorrelation loop contains a bug. for shift in range(10) will start from zero, and when you temp[ : -0] you will get an empty array, and your code won't work, because you will do a corrcoef of an empty matrix with another that contains data, thus, the correct loop will be something like: for shift in range(1,10)...thanks a lot for the vid
This is great. Thanks
Great Video
Thank you!
In partial autocorrelation, the graph for shift 2 is incorrect, it has to show the points tilted along the 1st fit.
How to compare two voice signal similarity in python?
wonderful!
great video
Wow thank you
Thanks a lot!
THANK YOU
I think you might be conflating 2 different concepts: 1. de-trending the timeseries before forecasting with AR, 2. partial auto-correlation.
For example the first de-trending step may not be just fitting a straight line, it could be non-linear or have seasonality etc. This is unrelated to PACF which teases out what marginally new information are gained with each auto-correlation step
Thank you! :D
It is not clear in the video how much residual of the previous periods to be added to the time series equation of the present period.
I think you have small error on screens, please 11:02 , on left graph should be "temperature on day i-1"
nice and clear explanation for this useful concept! for me its just not clear how to determine whether or not a value is still whithin the 95% confidence interval? R is giving some limits but there it remains unclear as well how to calculate those boundaries..
Thanks darmastadvideo! This is important, but unfortunately didn't make it into this video. I'll have to save it for a future one.
Intuitively, how does one gather the data for the temperature scatter plot for autocorrelation? This is my only stumbling block.
To see a detailed example in practice (including step-by-step data collection and organization), check out e2eML course 212: e2eml.school/212
Sounds like George Clooney explains it
Is the autocorrelation with zero lag always 1? Because I've read that Gauss-Markov processes have the following equation for autocorrelation:
autocorrelation(lag) = variance*exp(-constant*lag)
In the case of this formula, autocorrelation(0)=variance (!!!) why is it different in this case?
no question about this. with no lag, you're basically computing the correlation of a variable with itself. Any algorithm that gives you something different than 1 in this case simply fallacious.
Excellent
can i have the data sample sir on the basis of which you are calculating thanks
Thank you :)
Thanks !
im confused! So when you talk about correlating the shifted (lagged) dataset. Lets say in an AR(1) process you shift the Y values by 1 day, does that mean the Y-1 dataset will have 1 less data point than the original Y dataset? And by extension if you specify an AR(10) process are you then knocking off 10 data points and hence correlating two datasets where one dataset has 10 less data points than the other?
Sorry if i explained that like a moron, maybe this will help:
let's say
y = [1,2,3,4,5,6,7,8,9,10]
y-1 = [1,2,3,4,5,6,7,8,9]
y-2 = [1,2,3,4,5,6,7,8]
y-3 = [1,2,3,4,5,6,7]
y-n = [.........]
so for example if i'm interested in specifying an AR(3) process, to determine the autocorrelation would i just find the correlation coefficient between y & y-3:
y = [1,2,3,4,5,6,7,8,9,10]
y-3 = [1,2,3,4,5,6,7]
i am confused as to whether the fact you are correlating a [1,10) with a [1,7] is correct? or an issue? or what!
sorry this was really long!
yes .
Think of it like this , when you are using a lag of 1 , you are comparing every day value to its previous day value therefore automatically for day 1 you won't have anything to compare it to. But it doesn't matter , the data points left are enough to give an accurate correlation value.
Dhirendra Singh amazing, thanks so much for the response. I have since experimented with some data and it makes much more sense now! Cheers mate
Appreciated
Close your eyes and you can hear Matthew McConaughey explaining autocorrelation. The only thing missing in here "Alright, alright, alright"
he literally sounds nothing like him
Nice
Cool! You get a coordial thanks from me.
You are most welcome
Hi! Great video but there is an error at 5:24 - The correlation is not negative if just the slope of the line is negative. Correlation is negative if the data points and the line have converse slopes.
%matplotlib inline # JuPyter Notebooks
import matplotlib.pyplot as plt
import numpy as np
n = 10
x = -np.linspace(0, 1, n)
y = x + (0.3 * np.random.rand(n)) - 0.3/2
neg = np.corrcoef(x, y)[0, 1]
plt.plot(y, marker='o', markersize=4, ls='None', c='k')
plt.plot(x, label='negative ({:.2f})'.format(neg), c='r')
plt.legend(); plt.show()
No error there, video is correct. you are plotting x and y independently on the same plot, not y against x. correct code should be:
plt.plot(x,y)
cool !!!
Great video! Except:
Traceback (most recent call last):
NameError: name 'temp_change' is not defined
;)
At 4:20 in the video a solution is provided.
Give this video a like
good but i think the way you represent the days shift in the code using [1:] and [:-1] is wrong (i do not get the meaning) or please explain what you meant. tks
It isn't wrong, its just python notation to slice a array. Just google it :)
What is the payoff of all this autocorrelation stuff? Once you have fit the data to the line and calculated the correlation, it seems like that is the all the useful information. I can't see where the autocorrelation ads information that would be useful.
Autocorrelation helps understand seasonality/periodicity in time series data, which can determine when will a value repeat itself in the future. A high autocorrelation means the selected shift/lag is likely to be the period; and instead of guessing which lag value will yield a high autocorrelation, we can checkout something called the correlogram. This can be useful, as sometimes the time series has both a trend and seasonality, and the correlogram decomposes it (sort of).
Damn. So mf nice!!! 10 stars
Aw, thanks Marlon. I'm really happy it clicked.
👏
The graph at 10:46 and the graph at 10:52 are the same but the x-axis is labeled differently. Which is accurate?
And does the correlation coefficient, r, from the regression performed on the graph at 11:02 give you the point on the PACF (2, r)? And the graph at the right I'm confused by. If you are regressing the series with lag 2 against itself it would be a perfect fit. Why are there residuals at all?
@@zagwask78 doesn’t answer questions, sad :(
🤓
6:48 you'll pull off??
"degrees" is not a unit
Bad
Poor Explanation