Woah, thank you, Sir. This short tutorial gets me to understand this entire auto-correlation thing than 10 page pdf explaining with all kinds of formulas .
Nice and clear explanation. I will add at 4:30. That the points can fit the line very well but there is no correlation. It is the case when the fitted line is parallel to the x-axis or y-axis. It is the case of near zero variance variables.
Where you use the word "shift", when describing autocorrelations, wouldn't it be better to use the word "lag", as I've seen this appear in literature more often and am now wondering whether they're the same thing
Brendon, I would so love to learn ML from you. teaching code along with concept would definitely decrease the MSE between ignorance and knowledge. Thanks
Auto correlation sounds useful for finding data cycles. Like every year temperature correlation, around ~365 days shift~, there is probably a max again. You can find cycles in shifts where correlations peak. Also thanks for showing how it can be refined the same way as tailor series. Didn't cross-concept that correlation can give differentials weight for interpolation.
Amazing video, but the autocorrelation loop contains a bug. for shift in range(10) will start from zero, and when you temp[ : -0] you will get an empty array, and your code won't work, because you will do a corrcoef of an empty matrix with another that contains data, thus, the correct loop will be something like: for shift in range(1,10)...thanks a lot for the vid
Hi! Great video but there is an error at 5:24 - The correlation is not negative if just the slope of the line is negative. Correlation is negative if the data points and the line have converse slopes. %matplotlib inline # JuPyter Notebooks import matplotlib.pyplot as plt import numpy as np n = 10 x = -np.linspace(0, 1, n) y = x + (0.3 * np.random.rand(n)) - 0.3/2 neg = np.corrcoef(x, y)[0, 1] plt.plot(y, marker='o', markersize=4, ls='None', c='k') plt.plot(x, label='negative ({:.2f})'.format(neg), c='r') plt.legend(); plt.show()
I think you might be conflating 2 different concepts: 1. de-trending the timeseries before forecasting with AR, 2. partial auto-correlation. For example the first de-trending step may not be just fitting a straight line, it could be non-linear or have seasonality etc. This is unrelated to PACF which teases out what marginally new information are gained with each auto-correlation step
I was looking for good video to understand the concepts and this video has done that. I have the following questions: - Can we do correlation to compare stock market bottoms or tops? If we compare the market price by its 1 day 1 week prior price then will this help in prediction of top or bottom? - How can we compare two asset prices? I am thinking of below methods - Do a ratio of two asset prices and run the n-1 vs n comparison of the ratio - Do a ratio of two asset prices and compare the ratio vs one of the asset prices - Compare the % return of each assets against each other on X and Y axis Let me know what do you think
Very nice and explanatory video! Thanks! One thing that you could also do is mention real-world applications. A toy model is always very helpful for me to *understand* a topic but real-world application helps me to also *motivate* it. Besides that keep it up your videos are super nice! :)
I am also currently learning so I'm not sure if my idea is entirely correct, you could use this for any data set, to see how it changes. You could use the auto-correlation on gas prices to see how it changes with time, on flight tickets to see how it changes on a day to day basis or even on the price of a big mac to see how it has changed over the years.
What is the payoff of all this autocorrelation stuff? Once you have fit the data to the line and calculated the correlation, it seems like that is the all the useful information. I can't see where the autocorrelation ads information that would be useful.
Autocorrelation helps understand seasonality/periodicity in time series data, which can determine when will a value repeat itself in the future. A high autocorrelation means the selected shift/lag is likely to be the period; and instead of guessing which lag value will yield a high autocorrelation, we can checkout something called the correlogram. This can be useful, as sometimes the time series has both a trend and seasonality, and the correlogram decomposes it (sort of).
Is the autocorrelation with zero lag always 1? Because I've read that Gauss-Markov processes have the following equation for autocorrelation: autocorrelation(lag) = variance*exp(-constant*lag) In the case of this formula, autocorrelation(0)=variance (!!!) why is it different in this case?
no question about this. with no lag, you're basically computing the correlation of a variable with itself. Any algorithm that gives you something different than 1 in this case simply fallacious.
im confused! So when you talk about correlating the shifted (lagged) dataset. Lets say in an AR(1) process you shift the Y values by 1 day, does that mean the Y-1 dataset will have 1 less data point than the original Y dataset? And by extension if you specify an AR(10) process are you then knocking off 10 data points and hence correlating two datasets where one dataset has 10 less data points than the other? Sorry if i explained that like a moron, maybe this will help: let's say y = [1,2,3,4,5,6,7,8,9,10] y-1 = [1,2,3,4,5,6,7,8,9] y-2 = [1,2,3,4,5,6,7,8] y-3 = [1,2,3,4,5,6,7] y-n = [.........] so for example if i'm interested in specifying an AR(3) process, to determine the autocorrelation would i just find the correlation coefficient between y & y-3: y = [1,2,3,4,5,6,7,8,9,10] y-3 = [1,2,3,4,5,6,7] i am confused as to whether the fact you are correlating a [1,10) with a [1,7] is correct? or an issue? or what! sorry this was really long!
yes . Think of it like this , when you are using a lag of 1 , you are comparing every day value to its previous day value therefore automatically for day 1 you won't have anything to compare it to. But it doesn't matter , the data points left are enough to give an accurate correlation value.
nice and clear explanation for this useful concept! for me its just not clear how to determine whether or not a value is still whithin the 95% confidence interval? R is giving some limits but there it remains unclear as well how to calculate those boundaries..
And does the correlation coefficient, r, from the regression performed on the graph at 11:02 give you the point on the PACF (2, r)? And the graph at the right I'm confused by. If you are regressing the series with lag 2 against itself it would be a perfect fit. Why are there residuals at all?
If in the time series , the residuals on temperatures day i after fiting temp(i-1) are correlated with actual temperatures of day i , then can it be said the there is partial auto correlation with lag beyond (i-1). If that method holds good ,then it could be much easier method of detecting PACF. But I have liked your video also.
good but i think the way you represent the days shift in the code using [1:] and [:-1] is wrong (i do not get the meaning) or please explain what you meant. tks
This is simply excellent. I haven't seen a better explanation of this concept.
This is the best video ever on autocorrelation! Thank You!
Thanks Ani!
Autocorrelation 7:02
Partial Autocorrelation 9:45
Woah, thank you, Sir. This short tutorial gets me to understand this entire auto-correlation thing than 10 page pdf explaining with all kinds of formulas .
One of the best videos about acf
Well explained! Very natural transition from topic to topic until I accidentally learned autocorrelation.
Came to revise Autocorrelation. Ended up revising regression too. Great video.
Nice and clear explanation. I will add at 4:30. That the points can fit the line very well but there is no correlation. It is the case when the fitted line is parallel to the x-axis or y-axis. It is the case of near zero variance variables.
Nice video, the best one I've seen yet about the concept of partial autocorrelation!
Brandon's videos deserve a lot more likes and views. Very underrated.
And this is where the info on autocorrelation starts: 6:57
You're an hero, bro.
Thank you very much. This give the insight for ACF, not just mindlessly calculating them
The kind of explanation I was looking for! Thanks!!!
Where you use the word "shift", when describing autocorrelations, wouldn't it be better to use the word "lag", as I've seen this appear in literature more often and am now wondering whether they're the same thing
they are used interchangeably, as well as the word "delay"
Excellent explanation with an easy-to-follow example!
Thank you for existing.
I watched this 2 years ago. Didn't understand the concept. I watch again in quarantine days.
Best video on this topic. Thank you!
best explanation found till now
Hello Brandon,
thank you for the explanation. Especially the code helped to understand whats happening.
I'm glad to hear it Bartosz. Thanks.
I really liked the autocorrelation part!
You covered regression better than videos on regression 👏
Great explanation with such a good example.
Easy to understand too.
Thanks!
Great video! I liked how you presented the python code that would perform what you're explaining.
thanks for the great video, @4:57 should the greater than and smaller than signs be flipped in the moderate correlation plot ?
Thanks mohammed, you're exactly right :)
Instead of using list slicing it is a bit easier to use the "shift" method from the pandas library. The video in general is great :)
Very nice demonstration!
this is best available in you tube thanks it was very helpful
Very good video. In Numerical Methods it always helps to visualise what you are doing
Cool description, very detailed, with lots of graphs
Brendon, I would so love to learn ML from you. teaching code along with concept would definitely decrease the MSE between ignorance and knowledge. Thanks
Thanks Siddarth! In case you're interested, there's a whole collection of concept-with-code courses at e2eml.school/catalog
Great explanation, Thank you.
This was explained excellently!
Amazing explanation, subscribed!
Thanks Carlos :)
Very nice and clear video!
Auto correlation sounds useful for finding data cycles. Like every year temperature correlation, around ~365 days shift~, there is probably a max again. You can find cycles in shifts where correlations peak.
Also thanks for showing how it can be refined the same way as tailor series. Didn't cross-concept that correlation can give differentials weight for interpolation.
Amazing video, but the autocorrelation loop contains a bug. for shift in range(10) will start from zero, and when you temp[ : -0] you will get an empty array, and your code won't work, because you will do a corrcoef of an empty matrix with another that contains data, thus, the correct loop will be something like: for shift in range(1,10)...thanks a lot for the vid
Very clearly explained!
Great video
Many thanks!
Another great and super clear explanation! Thanks.
fantastic explanation
Really helpfull
And straight forward
U r genius man
Excellent explanation, thank you!
Thanks yung!
Amazing video! Very useful. Thanks!
Hi! Great video but there is an error at 5:24 - The correlation is not negative if just the slope of the line is negative. Correlation is negative if the data points and the line have converse slopes.
%matplotlib inline # JuPyter Notebooks
import matplotlib.pyplot as plt
import numpy as np
n = 10
x = -np.linspace(0, 1, n)
y = x + (0.3 * np.random.rand(n)) - 0.3/2
neg = np.corrcoef(x, y)[0, 1]
plt.plot(y, marker='o', markersize=4, ls='None', c='k')
plt.plot(x, label='negative ({:.2f})'.format(neg), c='r')
plt.legend(); plt.show()
No error there, video is correct. you are plotting x and y independently on the same plot, not y against x. correct code should be:
plt.plot(x,y)
temp_change is not defined in your code is it? Something is missing there.
really helpful video, thank you for this
Good job on this video!
I wish it would have showed up when I searched for information on ARIMA, all the ones that did were pretty poor.
Stellar explanation!
Sounds like George Clooney explains it
Intuitively, how does one gather the data for the temperature scatter plot for autocorrelation? This is my only stumbling block.
To see a detailed example in practice (including step-by-step data collection and organization), check out e2eML course 212: e2eml.school/212
Thanks a lot, view the code helpful to understand quickly
This is amazing, thank you
Close your eyes and you can hear Matthew McConaughey explaining autocorrelation. The only thing missing in here "Alright, alright, alright"
he literally sounds nothing like him
No
nice and concise video!!
I think you might be conflating 2 different concepts: 1. de-trending the timeseries before forecasting with AR, 2. partial auto-correlation.
For example the first de-trending step may not be just fitting a straight line, it could be non-linear or have seasonality etc. This is unrelated to PACF which teases out what marginally new information are gained with each auto-correlation step
Thanks for the video!
very good video!! Thank you
Very very nice explain .. many thanks
🙏
I was looking for good video to understand the concepts and this video has done that. I have the following questions:
- Can we do correlation to compare stock market bottoms or tops? If we compare the market price by its 1 day 1 week prior price then will this help in prediction of top or bottom?
- How can we compare two asset prices? I am thinking of below methods
- Do a ratio of two asset prices and run the n-1 vs n comparison of the ratio
- Do a ratio of two asset prices and compare the ratio vs one of the asset prices
- Compare the % return of each assets against each other on X and Y axis
Let me know what do you think
do you know how to do a white noise time series to test for seasonality?
In partial autocorrelation, the graph for shift 2 is incorrect, it has to show the points tilted along the 1st fit.
Very nice and explanatory video! Thanks! One thing that you could also do is mention real-world applications. A toy model is always very helpful for me to *understand* a topic but real-world application helps me to also *motivate* it.
Besides that keep it up your videos are super nice! :)
I am also currently learning so I'm not sure if my idea is entirely correct, you could use this for any data set, to see how it changes. You could use the auto-correlation on gas prices to see how it changes with time, on flight tickets to see how it changes on a day to day basis or even on the price of a big mac to see how it has changed over the years.
this was super helpful
What is the payoff of all this autocorrelation stuff? Once you have fit the data to the line and calculated the correlation, it seems like that is the all the useful information. I can't see where the autocorrelation ads information that would be useful.
Autocorrelation helps understand seasonality/periodicity in time series data, which can determine when will a value repeat itself in the future. A high autocorrelation means the selected shift/lag is likely to be the period; and instead of guessing which lag value will yield a high autocorrelation, we can checkout something called the correlogram. This can be useful, as sometimes the time series has both a trend and seasonality, and the correlogram decomposes it (sort of).
Is the autocorrelation with zero lag always 1? Because I've read that Gauss-Markov processes have the following equation for autocorrelation:
autocorrelation(lag) = variance*exp(-constant*lag)
In the case of this formula, autocorrelation(0)=variance (!!!) why is it different in this case?
no question about this. with no lag, you're basically computing the correlation of a variable with itself. Any algorithm that gives you something different than 1 in this case simply fallacious.
great video man
7:00 autocorr
It is not clear in the video how much residual of the previous periods to be added to the time series equation of the present period.
im confused! So when you talk about correlating the shifted (lagged) dataset. Lets say in an AR(1) process you shift the Y values by 1 day, does that mean the Y-1 dataset will have 1 less data point than the original Y dataset? And by extension if you specify an AR(10) process are you then knocking off 10 data points and hence correlating two datasets where one dataset has 10 less data points than the other?
Sorry if i explained that like a moron, maybe this will help:
let's say
y = [1,2,3,4,5,6,7,8,9,10]
y-1 = [1,2,3,4,5,6,7,8,9]
y-2 = [1,2,3,4,5,6,7,8]
y-3 = [1,2,3,4,5,6,7]
y-n = [.........]
so for example if i'm interested in specifying an AR(3) process, to determine the autocorrelation would i just find the correlation coefficient between y & y-3:
y = [1,2,3,4,5,6,7,8,9,10]
y-3 = [1,2,3,4,5,6,7]
i am confused as to whether the fact you are correlating a [1,10) with a [1,7] is correct? or an issue? or what!
sorry this was really long!
yes .
Think of it like this , when you are using a lag of 1 , you are comparing every day value to its previous day value therefore automatically for day 1 you won't have anything to compare it to. But it doesn't matter , the data points left are enough to give an accurate correlation value.
Dhirendra Singh amazing, thanks so much for the response. I have since experimented with some data and it makes much more sense now! Cheers mate
can i have the data sample sir on the basis of which you are calculating thanks
I like it! Thank you!
great video
How to compare two voice signal similarity in python?
Great Video
nice and clear explanation for this useful concept! for me its just not clear how to determine whether or not a value is still whithin the 95% confidence interval? R is giving some limits but there it remains unclear as well how to calculate those boundaries..
Thanks darmastadvideo! This is important, but unfortunately didn't make it into this video. I'll have to save it for a future one.
Thanks a lot!
wonderful!
The graph at 10:46 and the graph at 10:52 are the same but the x-axis is labeled differently. Which is accurate?
And does the correlation coefficient, r, from the regression performed on the graph at 11:02 give you the point on the PACF (2, r)? And the graph at the right I'm confused by. If you are regressing the series with lag 2 against itself it would be a perfect fit. Why are there residuals at all?
@@zagwask78 doesn’t answer questions, sad :(
Thank you :)
This is great. Thanks
If in the time series , the residuals on temperatures day i after fiting temp(i-1) are correlated with actual temperatures of day i , then can it be said the there is partial auto correlation with lag beyond (i-1). If that method holds good ,then it could be much easier method of detecting PACF.
But I have liked your video also.
Thank you! :D
THANK YOU
Excellent
Damn. So mf nice!!! 10 stars
Aw, thanks Marlon. I'm really happy it clicked.
Appreciated
good but i think the way you represent the days shift in the code using [1:] and [:-1] is wrong (i do not get the meaning) or please explain what you meant. tks
It isn't wrong, its just python notation to slice a array. Just google it :)
Give this video a like
Thanks !
Nice
cool !!!
👏
6:48 you'll pull off??
🤓
"degrees" is not a unit
a degree is a dimensionless unit of measurement
Bad
Poor Explanation
Thank you!
Wow thank you
Thanks!