Hurst exponent explained: Long-term memory in time series (Excel)
Вставка
- Опубліковано 11 січ 2021
- Do stocks follow random walks? How to test for market efficiency or time series dependency in the long term? Today we are addressing these questions and investigating a very insightful and elegant method for determining long-term memory in time series - the Hurst exponent.
Don't forget to subscribe to NEDL and give this video a thumbs up for more videos in Finance!
Please consider supporting NEDL on Patreon: / nedleducation
You can find the spreadsheets for this video and some additional materials here: drive.google.com/drive/folders/1sP40IW0p0w5IETCgo464uhDFfdyR6rh7
Please consider supporting NEDL on Patreon: www.patreon.com/NEDLeducation
Honestly an amazing video that explains and demonstrates how to calculate the hurst exponent. Very underrated!
Great video expaining with just the right amount of detail
I'm so glad I've found this channel. Amazing content 👍
So great! Awesome teacher, thank you
This is amazing dude!
Great contents. Many thanks!
Great Job Dear!
Thank You A lot.
Love and Respect from Nepal
Hi, thank you for the kind words! I’m glad you enjoyed the videos :)
thank you so much for this video
very good!
Excellent.
you are so good...thank you :)
Great channel i hope to come back here more time
Hi from Brazil
opa, br aq tb
so por curiosidade, pq ta procurando sobre o coeficiente d hurst>
excellent!
Thank you very much for all those great explanations. Can you help me to understand detrended fluctuation analysis (steps) . You re a great teacher.
Your work is really appreciated. I am interested in Multifractal Detrended fluctuation analysis. Can you please make a video on it.
Great!!!
Great video. SInce low Hurst exponent implying mean reverting, I guess half life concept possibly can indicate the same too. Together they possibly can be used in pair trading. If possibly may discuss the half life too.
Thanks so much for the comment and glad you enjoyed the video. Hurst exponent can be definitely related to half-life, and together they can augment one's perspective on the persistence or antipersistence of a time series.
Great video. Files to download are very helpful. QUESTION: Your first subset has 2 chunks. Reading Wikipedia, it appears they suggest that the first subset should be the whole initial series: "A time series of full length N is divided into a number of nonoverlapping shorter time series of length n, where n takes values N, N/2, N/4, ..."
If I apply this algo including the whole series (9 subsets) I get H=0.6266. Comments? I don't have the original paper by Mandelbrot & Wallis.
aydınlandım
excellent
Thank you for such a nice video. Have you gone through on centered detrended moving average analysis (CDMA) and Detrended Fluctuation Analysis for long run dependence in time series. I would appreciate if you make some videos in the near future. Many Thanx
Hi Muhammad, and glad you liked the video! Will definitely consider addressing this at some point in the future!
Great work, thanks for sharing! I wonder if you have any study applying mandelbrot's fractals to financial systems?
Hi Kemal, and glad you liked the video! The Hurst exponent can be considered a special case of broader multifractal model developed by Mandelbrot, among others. A general case is quite tricky computationally, but I am looking forward to implement it at some point.
After seeing this video, I understand the hurst exponential material, but I have problems with the sample data. how do you find data to be like this video
Thank you for the detailed explanation, may I seek your help on work around for the MAXIFS function for the "Subsample range segment"?
Hi Jansen, and glad you liked the video! In principle, you can workaround the MAXIFS or MINIFS function by building a simple MAX or MIN function on top of an IF function, just as I have done later in the video to calculate standard deviations. Hope it helps!
@@NEDLeducation could you post the respective command? thx...
Hello and thank you. May i ask if it is possible to take the same st.dev.S for all the columns and rows?? It sounds very weird to me. I work with Open Office (Neat Office)
Very interesting from math and technical point of view.
But your input data are stock prices that have a drift component included and thus they will always show a tendency over time to rise unless all net profits have been distributed as dividends. (very unlikely).
If you want to test randomness, you must eliminate the drift in the input data to reach a conclusion on whether the remaining components in the prices are random or not. If not, you will reach your conclusion they are not random, which may easily be a wrong conclusion, due to the composition of the input data.
Hi Hen, and glad you liked the video! As for your comment, the drift in the calculations is eliminated, as the rescaled range and the Hurst exponent are calculated based on demeaned returns. which takes care of the drift. Hope it helps!
@@NEDLeducation Demeeaned, does that mean detrended?
Anyway if drift is eliminated, can you translate that back into the prices you have used and get the series without drift?
Hello, Thanks for a great video. What if you only want to use a sample of 100 instead of a large sample such as 1024, then do you start with 16 subsamples with (64 observations) and continue until 256 sumsamples of 4 to have a total of 5 number of Observation, or do you simply do just 16 subsamples if 64 and have only 1 observation? Thanks 🙂
Hi Chintan, and glad you liked the video! Generally, Hurst exponent is calculated for samples of size equal to a power of 2, and you need enough data points to be able to estimate the Hurst exponent precisely enough. The smallest sample that still performs decently is (from my experience) 256.
Hi, I'm Facu from Argentina. This is the video that helped me the most. Thank you very much for that. I only have one question:
If I wanted to work with timeframes of 1 hour, instead of 1 day as you show in the video, should I take more than 1024 data? or less ?
Hi Facu, and glad you liked the video! As for your question, if you use hourly data, you can afford to go for longer intervals back in time (for example, 2048 hours, or even 4096 hours, which would be around a year or two years, respectively in terms of trading hours). However, keeping it at 1024 is reasonable as well so feel free to keep this specification for hourly data, especially if your investing horizons are not very long.
Thankiu for teaching and sharing!!!! Love your videos, I have one question,
I’ve try with your same data in R project, (with the package of “pracma”) the results are a little bit different.. you know why?
Hi Rodrigo, and glad you are enjoying the channel! Little discrepancies in caclulation results might be a result of methodology, for example, how many subsamples do you select (here I stop at 256), or which method do you use to calculate the Hurst exponent. I fit the slope using OLS, sometimes it is also done using maximum likelihood. The difference should not be substantial though. Hope it helps!
@@NEDLeducation I'm greatful!! thanks again, and please, keep making videos, your work really helps a lot to all
Hi Sava, noob question here, can the Hurst exponent be applied to time series of hourly observations rather than just time series of daily observations? I ask because I would like to apply this knowledge to short term trading (like futures), but I don't know if it would be the right approach. Also, thank you so much for your content, I just subscribed :)
Hi Noah, and many thanks for your subscription! Yes, you can apply Hurst at any frequency.
Amazing! Have you tried the multifractal model of asset returns from Mandelbrot? That would be a cool video
Hi, and glad you liked the video! Mandelbrot's multifractal model, if I recall correctly, can be considered as a generalisation of Hurst exponent (processes that can be described by Hurst exponent Mandelbrot himself calls "self-affine" and considers them a special case of broader self-similar processes). I might make a video on that in the distant future!
@@NEDLeducation that would be awesome thanks mate
@@NEDLeducation I would love to see a presentation on that as well. I'm trying to work through Mandelbrot (Mis-Behavior of Markets) and applying it in Python, but it's been challenging for me. thank you
If the time series has observation that are not divisible by power of 2 and the time series has odd number of observations in that case how to divide the time series and compute the Hurst exponent? Also how much optimum number of such divisions need to be done? how mum minimum number of dataset is required for computation of Hurst exponent?
Hi Debditya, and thanks for the question! Hurst exponent requires estimation to be undertaken on a power of 2 subsample, so you generally select the most recent 1024 (256, 512, 2048, etc.) trading days/candles to get the value. The selection of the optimal period is where finance becomes more of an art than a science, but you can always see whether the results are consistent in different specifications. I do address it in greater detail in my Python tutorial on Hurst exponents: ua-cam.com/video/v0sivj2wGcA/v-deo.html
great
I mean how is it possible for the formula to recognize the stdev of specific group since we use the '$' for the rows of returns ($C$17:$C$1040) ? It seems to me that it takes the whole stdev of all returns every time
Hi, and thanks for the great question! This formula works as we implement an IF function on top of the whole array of returns that only takes those that belong to the subsample we need.
@@NEDLeducation sorry but it doesnt work for me. Maybe its different for open office
@@NEDLeducation would it be trouble for you to send the file somewhere ?? Maybe, just in case you have the time
Hi, thanks a lot for this video. I wanted to know what my results mean:
Hurst exponent: 0.5662
std error: 0.01979
t-stat: 3.3475
p-value: 1.55%
Thank you for you're videos!!
Hi JaVale, and glad to see you are successfully applying the tests to your data! As for your question, the Hurst exponent in your case is higher than 0.5, and significantly so (p-value is lower than 5%). That would imply the data is persistent in the long-term (there are long-term trends rather than reversals). As such, the returns are not independent (the market is inefficient). For trading strategies, that would mean that long-term momentum strategies are likely to be profitable. Hope it helps!
By the way, the video on approximate entropy is in the pipeline and to be released in the next couple of weeks, so stay tuned :)
Cheers buddy, thanks a lot!!
Hello. I am a research scholar from India. Thank you soo much for this video. This helped me in my research and analysis. I wanted to know if it is necessary for sample size to be in power of 2 only? Because wen we check long-term memory in time series over a period, the sample size wouldn't necessarily be in even numbers.
Hi Shiney, and happy you are enjoying the channel! As for your question, it is preferable that the sample size is a power of 2 as it allows for the same length of all subsamples. This, in turn, leads to our estimate of Hurst to be the most precise.
Great video!!! I question that I have on Hurst is how to "keep track" of it on a daily basis if the sample size must follow a power of 2, I´ve seen some other implementations but they were as rigorous as yours. Best Regards!
Hi Guilherme, and glad you liked the video! The easiest approach that comes to mind is to continuously apply the Hurst exponent to the most recent 1024 (or any other power of two) observations and see how results change with time. Overall, there is nothing particularly special about powers of 2, it could well have been powers of 3, or even any sumbsample sizes that divide the total sample size, and here it is just crucial that your sample size is not a prime :) Powers of 2 are conventionally chosen as they give a reasonably high level of statistical power by generating the highest number of evenly spread points on a log-log plot which leads to a precise slope estimation (and that is what we are ultimately looking for in Hurst). Hope it helps!
@@NEDLeducation sure helped! Thanks for the video! Gret content as always, proud to be a patreon!
Hello, I am a Korean university student. Your video really helped me a lot and I really appreciate it. And I'm currently looking for a way to calculate the Hurst exponent using DFA (Detrended Fluctuation Analysis) proposed by Peng et al. (1994). Can I have some simple advice on this?
Hi, and glad you have enjoyed the video! Thanks for the excellent question. DFA can be viewed as a generalisation of the Hurst exponent, so if you want to use Hurst as presented in this video (or, in a more efficient way using Python, here: ua-cam.com/video/v0sivj2wGcA/v-deo.html), you can refer to it as a particular case of DFA. DFA and, for example, multi-fractal models of asset returns that are associated with Mandelbrot, relax the assumption that the scaling process is linear (that Hurst exponent assumes) and thus are much harder to explicitly test. Hope this makes sense :)
Thanks for sharing! And I want to learn how to compute the fractal feature (fractal dimension, Hurst exponent etc). Could you give me an advise how to start it ?
Hi, and glad you liked the video! As for your question, the fractal dimension can be calculated as two minus the Hurst exponent. For a more efficient way of calculating the Hurst exponent, you can have a look at this video that codes the procedure in Python: ua-cam.com/video/v0sivj2wGcA/v-deo.html
@@NEDLeducation Think you! Would you please recommend some books to me?
does my subsamlpe have to a number that is a power of 2. or can i have a number like 2982
Hi Ibrahim, and thanks for the question! It is preferable to have a power of 2 as a sample size. For your application, you can look at the most recent 2048 observations (2^11). Hope it helps!
Hi, is it possible to calculate the hurst exponent on a rolling basis? This seems to calculate the hurst exponent at a fixed point in time. I would like to see how the hurst exponent changes over different points in time. Is this possible?
Hi Jack, and thanks for the question! Yes, you can estimate it based on a rolling window of 1024 trading days, for example, and see how it evolves through time. Alternatively, the data can be divided into samples of 256 (a power of two closest to a full trading year), and then Hurst can be estimated on an annual basis. Hope it helps!
NEDL thanks for getting back to me so quickly. If I wanted to do that, would I have to change the structure of the spreadsheet used in the video?
Forgive me for the ignorance, but it seems like the sheet produces only one hurst exponent value, as the averages and index numbers are fixed. Ideally I’d like to track H over a rolling basis and then present that on a line chart.
I hope my question makes sense and thanks!
Hi again, and thanks for the follow-up! Yes, this spreadsheet is designed to estimate one Hurst exponent value. For a dynamic way of generating one for rolling windows, Excel is perhaps not the best software (as you need to calculate a new subsample breakdown and estimate a new regression for a box plot for each observation) and a more efficient solution would be to code it in Python. If you want to discuss it further, you can email me onto savvashanaev@yandex.ru. Hope it helps!
@@jackhaywood4331 Python . The secret is Python :-)
Thanks a lot. I am still interested in how to draw time-varying Hurst exponent. Could you please share some experience?
Hi Adam, and glad you liked the video! A time-varying Hurst exponent is quite a computationally intensive feat, so I might well do a Python tutorial on that in the future. The logic would be to either estimate it in subsamples of a power of 2 observations each, or move in a rolling window of a power of 2 observations. While Excel can do the former, it would be painful to accommodate the latter there.
@@NEDLeducation Actually, I am working on my dissertation, and I am considering whether I need to choose the time-varying Hurst Exponent to compare the changes of market efficiency with the influence before and after a special policy. I know that the time-varying Hurst Exponent needs to be smoothed, but this is only my shallow understanding. Indeed, it is quite difficult. Anyway, thanks for your reply and material!
@@NEDLeducation that will be awesome
Hello, great video, can you please help me interpret my findings:
Hurst exponent: 0.6735
Standard error: 0.0133
Expected Hurst: 0.50
t-stat: 13.029
number of obs: 8
degrees of freedom: 6
p-value: 0.00126%
Hi, and glad to see you have applied the technique to your own dataset! Your Hurst exponent in 0.67, and it is significantly higher than the null hypothesis of 0.5 (as p-value is very low), meaning that there is long-term persistence in your data. Hope it helps!
Спасибо за ваши видео, было бы круто, если на русский перевели бы так же. С вами учиться легче!)
Привет, и спасибо за комментарий! На самых первых видео есть русские субтитры, есть планы в будущем сделать их и для самых популярных более новых видео. Всегда стараюсь говорить на более-менее понятном английском, ну и разумеется альфа и бета на любом языке альфа и бета :)