Few questions: a) So alot of this just finds a strategy where over time there was a smooth profit curve by splitting it up randomly in a way and checking each segment separately? b) Between each run, does it only pass the settings onto the next run if it meets the pass filter criteria from the previous run? And it starts with a certain amount of tests right and reduces it down to one? What I like to do is to be able to see how many different variations of settings for one strategy (etc period, standard deviation for the stddev indicator etc) stayed profitable in the out of sample section versus in comparison to how many became unprofitable of the same strategy but with different settings for each test. Lets say I have 5 tests that uses an exponential moving average for example, only thing that changes is the EMA period between each test. I pass all 5 into the in-sample period, 4 out of 5 pass my criteria in the in-sample. Then with those 4 that passed, they would be passed to the out of sample. After the out of sample was completed, only 2 out of 4 passed (Using same criteria as in-sample). Which means I would get a score of 50% since only 2 remained profitable out of the 4 that I passed into the out of sample. The 50% score is what i'd like to see. It tells me if the strategy is only really profitable by specific periods/settings in the strategy and not so much the strategy logic itself. The more higher that percentage, the more likely the algo is profitable by the logic and not the settings (period, input parameters).
a) it finds the best parameters for the previous period to achieve the best (fitness function, eg. net profit), then apply those parameters to OOS b)the filters apply at the end of all optimization runs. between each run above point apply. WFO works as above, however you can do your own optimization and customize it as you like with testing IS separately and pass the one you need to OOS testing and you can automate all that with Projects. Of Course you can also use WFM and SPP which are multiple WFO combined into one plus you can do filters on them separately or collectively.
HI Ali, in your Builder data section you have chosen in sample and out of sample, you don't use In Sample Validation, can you explain a little bit what is that used for? For this method you used, IS and OOS are separate period, Robert Pardo suggested to use walk forward period contains both previous OOS and IS as current IS, what is your thoughts on that?
I don't use ISV, just to point out that not all features in SQX are meant to be used at the same time, so just like I don't use IS/OOS 70/30. The dev team implement most requested features regardless of their over all functionality.
this is great and easy to understand. how can the wf matrix and wf optimization help in achieving better profits? or is it more about confirmation the strategy is robust and therefore you can trust it to trade your own capital? sometimes results differ from sqx to the platfom traded on so im unsure if it is helpful in this way. be great to hear your thoughts on this
WFM is a robustness test, so it is used to vet your strategy if it is robust or not. Of course you can use it to pick the peak of your parameters to achieve the best Net Profit but that is not recommended at all. Will do an update video on it soon.
Hey Ali, in another video you did I think you mentioned you don't add commission & slippage, but i noticed you've included it in the optimization here. Do you find it better to add it in during the optimization?
a lot get confused with this. In general your average trade should be +3x your commission and slippage. so if you don't included then you need to keep it in your mind when looking at average trade. If you are not used to it then you can add it in immediately. One caveat, when developing a strategy, you might overlook some of them if the commission and slippage already included because the strategy will look like crap, so keep this in mind and don't throw a strategy before trying some filters.
@StatOasis - great video, thank you! couple of things that are bothering me, though. 1) If i have built/tested/optimized profitable strategy with period ie. 2000.01.01-2020.12.31, does that mean that most likely strategy is overfitted, as it has optimal settings for ie net profit/DD? In other words - does Walk Forward have a meaning? Is it always better to build strategies with periods 2000-2018 making sure there is a room for Walk Forward (2019-2020)? 2) I tested strategy with WF Matrix (3x3) and strangely it scores 83% rob. for: a) 30% OOS/ 15WF, b) 20% OOS/10WF, c) 15% OOS/5WF but all other are red. The way i see it, if a) passes the robustness test, how come it does for pass for 20% OOS and 10 WF?
This is a long subject, and I will try to be concise. As I mentioned in many other posts, robustness is a guideline and not a perfect science. Also keep in mind that no method are a guarantee that your developed strategy will work forever. Long Term strategies like Value, Trend Following, Momentum have been proven to work over a 100 years, but they do have decades of negative performance. Knowing when to switch ON/OFF a strategy will serve you better in the long run. Robustness tests in SQX are tools to filter out garbage strategies that based on history have an edge, but not a guarantee will keep its edge in the future. As with all other tools, it depends on how you use it. You need a sound repeatable framework to test your strategies, that only comes from experience or learning from an experienced person. You are on the right path though, and here are some guidelines: -it is not about how far back you go in data, its about how many market regimes in the data set. -if you think of all data points on a plain then preferably you don't want peaks and valleys, but rather smooth hills and ponds. so if 15%OOS makes $1000 and 30%OOS makes $1500, then 20%OOS should be within 1SD of all other samples. -if you use anchor WF, then the more data you include, the less the current added data will have effect on the outcome. -no one WF run should contribute more than 50% of the winnings of all data set. If you want to learn a sound framework, then please join my Algo Trading Masterclass.
@@StatOasis Hi, do you have a part in one of your videos explaining this part of the comment in a bit more context context ", its about how many market regimes in the data set."
You can search the channel for "Market Regime" and see the videos that come up. I did and this came up: ua-cam.com/video/KjZLiJ7AI-0/v-deo.html basically your strategy will have different performance in each market regime and you can trade base off that. There is a whole module in my course about market regime.
I am still trying to get my head around what the purpose of a walk forward is. Why would it be any better than just a standard forward 1/2 optimisation for the same total period?
walk forward will apply new settings to new data several time over the time series, so if a strategy is profitable over most of those segments and if the variables don't change that much over several segments, then the strategy is not over fitted. If you want to master WFA, then I suggest Robert Pardo (inventor of WF) book: The Evaluation and Optimization of Trading Strategies
@@StatOasis This software that that you use to do it, can it optimise an EA from its default settings with minimal user knowledge or intervention? Is there a free software I can use to get this done just by specifying the walk forward dates?
You will have to build the EA inside StrategyQuant X then it is just picking default settings to do the analysis, or you can pick your own values. SQX has 1 month free trial, so you can test it for free.
Hi, could you tell me that how to add WF Score of netprofit? when i create a custom view, i try to search column WF Score..., i can't find these "WF Score..." word.
Another excellent video, cheers Ali, very helpful.
Glad you enjoyed it
Very good explanations. Thank you.
Very well explained! Got everything now.
Glad you found it useful
Very good videos like always! You are great!
Thank you very much!
@@StatOasis thanks to you!
Few questions:
a) So alot of this just finds a strategy where over time there was a smooth profit curve by splitting it up randomly in a way and checking each segment separately?
b) Between each run, does it only pass the settings onto the next run if it meets the pass filter criteria from the previous run? And it starts with a certain amount of tests right and reduces it down to one?
What I like to do is to be able to see how many different variations of settings for one strategy (etc period, standard deviation for the stddev indicator etc) stayed profitable in the out of sample section versus in comparison to how many became unprofitable of the same strategy but with different settings for each test. Lets say I have 5 tests that uses an exponential moving average for example, only thing that changes is the EMA period between each test. I pass all 5 into the in-sample period, 4 out of 5 pass my criteria in the in-sample. Then with those 4 that passed, they would be passed to the out of sample. After the out of sample was completed, only 2 out of 4 passed (Using same criteria as in-sample). Which means I would get a score of 50% since only 2 remained profitable out of the 4 that I passed into the out of sample. The 50% score is what i'd like to see. It tells me if the strategy is only really profitable by specific periods/settings in the strategy and not so much the strategy logic itself. The more higher that percentage, the more likely the algo is profitable by the logic and not the settings (period, input parameters).
a) it finds the best parameters for the previous period to achieve the best (fitness function, eg. net profit), then apply those parameters to OOS
b)the filters apply at the end of all optimization runs. between each run above point apply.
WFO works as above, however you can do your own optimization and customize it as you like with testing IS separately and pass the one you need to OOS testing and you can automate all that with Projects.
Of Course you can also use WFM and SPP which are multiple WFO combined into one plus you can do filters on them separately or collectively.
It would be good if you made that video about the WFM rankings and those topics that you talk about in this video
sure will add it to post ideas
HI Ali,
in your Builder data section you have chosen in sample and out of sample, you don't use In Sample Validation, can you explain a little bit what is that used for?
For this method you used, IS and OOS are separate period, Robert Pardo suggested to use walk forward period contains both previous OOS and IS as current IS, what is your thoughts on that?
I don't use ISV, just to point out that not all features in SQX are meant to be used at the same time, so just like I don't use IS/OOS 70/30. The dev team implement most requested features regardless of their over all functionality.
this is great and easy to understand. how can the wf matrix and wf optimization help in achieving better profits? or is it more about confirmation the strategy is robust and therefore you can trust it to trade your own capital? sometimes results differ from sqx to the platfom traded on so im unsure if it is helpful in this way. be great to hear your thoughts on this
WFM is a robustness test, so it is used to vet your strategy if it is robust or not. Of course you can use it to pick the peak of your parameters to achieve the best Net Profit but that is not recommended at all. Will do an update video on it soon.
Hey Ali, in another video you did I think you mentioned you don't add commission & slippage, but i noticed you've included it in the optimization here. Do you find it better to add it in during the optimization?
a lot get confused with this. In general your average trade should be +3x your commission and slippage. so if you don't included then you need to keep it in your mind when looking at average trade. If you are not used to it then you can add it in immediately.
One caveat, when developing a strategy, you might overlook some of them if the commission and slippage already included because the strategy will look like crap, so keep this in mind and don't throw a strategy before trying some filters.
@@StatOasis Thank you :)
@StatOasis - great video, thank you! couple of things that are bothering me, though.
1) If i have built/tested/optimized profitable strategy with period ie. 2000.01.01-2020.12.31, does that mean that most likely strategy is overfitted, as it has optimal settings for ie net profit/DD?
In other words - does Walk Forward have a meaning? Is it always better to build strategies with periods 2000-2018 making sure there is a room for Walk Forward (2019-2020)?
2) I tested strategy with WF Matrix (3x3) and strangely it scores 83% rob. for: a) 30% OOS/ 15WF, b) 20% OOS/10WF, c) 15% OOS/5WF but all other are red. The way i see it, if a) passes the robustness test, how come it does for pass for 20% OOS and 10 WF?
This is a long subject, and I will try to be concise. As I mentioned in many other posts, robustness is a guideline and not a perfect science. Also keep in mind that no method are a guarantee that your developed strategy will work forever.
Long Term strategies like Value, Trend Following, Momentum have been proven to work over a 100 years, but they do have decades of negative performance. Knowing when to switch ON/OFF a strategy will serve you better in the long run.
Robustness tests in SQX are tools to filter out garbage strategies that based on history have an edge, but not a guarantee will keep its edge in the future. As with all other tools, it depends on how you use it.
You need a sound repeatable framework to test your strategies, that only comes from experience or learning from an experienced person.
You are on the right path though, and here are some guidelines:
-it is not about how far back you go in data, its about how many market regimes in the data set.
-if you think of all data points on a plain then preferably you don't want peaks and valleys, but rather smooth hills and ponds. so if 15%OOS makes $1000 and 30%OOS makes $1500, then 20%OOS should be within 1SD of all other samples.
-if you use anchor WF, then the more data you include, the less the current added data will have effect on the outcome.
-no one WF run should contribute more than 50% of the winnings of all data set.
If you want to learn a sound framework, then please join my Algo Trading Masterclass.
@@StatOasis Hi, do you have a part in one of your videos explaining this part of the comment in a bit more context context ", its about how many market regimes in the data set."
You can search the channel for "Market Regime" and see the videos that come up. I did and this came up:
ua-cam.com/video/KjZLiJ7AI-0/v-deo.html
basically your strategy will have different performance in each market regime and you can trade base off that.
There is a whole module in my course about market regime.
I am still trying to get my head around what the purpose of a walk forward is. Why would it be any better than just a standard forward 1/2 optimisation for the same total period?
walk forward will apply new settings to new data several time over the time series, so if a strategy is profitable over most of those segments and if the variables don't change that much over several segments, then the strategy is not over fitted.
If you want to master WFA, then I suggest Robert Pardo (inventor of WF) book: The Evaluation and Optimization of Trading Strategies
@@StatOasis This software that that you use to do it, can it optimise an EA from its default settings with minimal user knowledge or intervention? Is there a free software I can use to get this done just by specifying the walk forward dates?
You will have to build the EA inside StrategyQuant X then it is just picking default settings to do the analysis, or you can pick your own values.
SQX has 1 month free trial, so you can test it for free.
Hi, could you tell me that how to add WF Score of netprofit? when i create a custom view, i try to search column WF Score..., i can't find these "WF Score..." word.
you pick net profit as the variable, but in data menu, instead of portfolio pick WF