Hi there! If you want to stay up to date with the latest machine learning and deep learning tutorials subscribe below. Thank you for your support! ua-cam.com/users/decisionforest
Just so I understand: This method 1) doesn't de-trend the data or make it stationary, 2) uses just the current price as the only feature (no rolling window aggregates), correct?
Do you have relevant experience in machine learning survival analysis? For example xgbse , scikit survival and pycox package in python. I don't know how the data format is handled in this case, also time dependent ROC, calibration and DCA.
Nice! I have a time series problem where the data is at daily level for 3 years and we forecast for daily level 2 years out. Currently I use Prophet which works well. Prophet takes care of spike events (holidays) and weekly / yearly seasonality. Can I use XGBoost to do this? Your example here is forecasting one step out which will be like forecasting one day out for my case.
Happy it helped. I haven't created any video with technical indicators as features as it's pretty easy to extend my current videos with them. Let me know what results you get if you try it out. My results so far have been pretty poor with standard indicators although I wasn't surprised :)
Thanks for sharing.... Keep getting an error at the last line after print(rmse) ... ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
I read in a research paper to predict time series we may use multiple trajectories. Any insight on what these trajectories are in time series and how to calculate these?
looks good , the question is how can you predict into unknown future ? how do you give it number of days to keep looping on training updating prediction and retraining
That's exactly what we're doing here with walk forward validation, we keep retraining and predicting. That's why walk forward validation is so important for time series.
Just read a raft of papers telling me that tree based methods are not easily adapted to time series… if you give them the simplest model ie extrapolation, can’t do. Dude you do this for a living.
of course there are, and there are others claiming the opposite. We start with a theory, test it in practice and based on the results we decide what is good and what isn't. Papers are incredibly useful as they help you avoid wasting time but in the end what works in the real world matters.
@DecisionForest Great Video! Is it common for the model to take over 15 minutes? The validation walkthrough part to be exact. Mine has been running for a good amount of time.
@@DecisionForest Hi Thanks for replying. So once we do a walk forward validation on the train test split, I finalize my model. I use that model on the whole dataset. I am not able to understand what will the model return at that step?
Great video.. How do you cater for the case where there's a new major event like the pandemic, because the old trained model wouldn't work anymore? Thanks!
Thankyou for sharing. I got error in pred = xgb_predict(history, test_X[0]) and X, y = train[:, :-1], train[:, -1]. The error message was "IndexError: too many indices for array: array is 0-dimensional, but 2 were indexed". Could you please tell me how to resolve this error?
Good video, thanks for sharing this! Could you maybe indicate why can't you after fitting the model just pass the full X_test set and use the predict method available from XGBoost?
Glad it was helpful. Well that’s the trick and why walk forward validation is useful. why would you ignore new data when you have it, of course you could but you want to get the best possible results.
Thanks, that was helpful.. can you provide more videos on XAI (Shap, Lime, etc..) and how to inerpret models such as XGBoost, RandomForest, Deep learning NN models and so on
Glad it was helpful. I wanted to do a video on SHAP, I actually created the notebook just that it takes forever to run, it's extremely computationally expensive especially with high dimensional datasets. I'd encourage you to use LIME as it's more practical for real world scenarios even if SHAP may be more interesting on paper.
This tutorial video is crystal clear to me up until 13:59 minute, then i lost completly... What is the meant of 1.7967091070446082 ??? Is it the prediction of the next hour close price??? Or have i miss something??? i mean, how to print out that predictions for the net 1 hour close price???
Thanks, glad it was helpful. That is the RMSE, the root mean squared error, the metric that we want to optimize in this case. You need to optimise for small values of this metric. Hope this helped.
Hi there! If you want to stay up to date with the latest machine learning and deep learning tutorials subscribe below. Thank you for your support!
ua-cam.com/users/decisionforest
10:15 Explanation of Walk Forward Method
why do you put val into np.array? Why do not you use encoders and scaler before testing?
Just so I understand: This method 1) doesn't de-trend the data or make it stationary, 2) uses just the current price as the only feature (no rolling window aggregates), correct?
Great Information. Thank you for your time!!
Once you find the rmse for your train/test model. How can you predict for next 2 years?
Why did you make the prediction one sample at a time?
Excellent explaination of XGBoost and it's slimplifed things.
Do you have relevant experience in machine learning survival analysis? For example xgbse , scikit survival and pycox package in python. I don't know how the data format is handled in this case, also time dependent ROC, calibration and DCA.
hey,i have a question. can we measure the r2Score of this? I got a negative r2score. Can I know what is the reason?
you save my day!! thank you from VietNam
Great demonstration!
I didn't understand the val object,why did u reshape it.Plz give a clarity.
Thanks
Great video! Another tool in the ML toolbox. Is this different to the Timeseries Cross validation?
You're a great teacher!! Thanks for the lesson, ir really helped me!! 🙂
Thank you so much Charlotte, glad it was useful.
Can you show the end prediction results please? Would be very useful as a reference point :)
Nice! I have a time series problem where the data is at daily level for 3 years and we forecast for daily level 2 years out. Currently I use Prophet which works well. Prophet takes care of spike events (holidays) and weekly / yearly seasonality. Can I use XGBoost to do this? Your example here is forecasting one step out which will be like forecasting one day out for my case.
2:52 what is the purpose of debuggers ?
I am a student and this helped me greatly! Is there a video for incorporating technical indicators or other features to have a more accurate model?
Happy it helped. I haven't created any video with technical indicators as features as it's pretty easy to extend my current videos with them. Let me know what results you get if you try it out. My results so far have been pretty poor with standard indicators although I wasn't surprised :)
Thanks for sharing.... Keep getting an error at the last line after print(rmse)
... ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
Good video. Is there a way to predict the unknown future? or how can we adapt the function to this approach?
Thank you.
I read in a research paper to predict time series we may use multiple trajectories. Any insight on what these trajectories are in time series and how to calculate these?
Nice explanation
I have 450 points for train data, need to predict for next 10 samples, how can we do train test split
i tried to sign up to get the code but failed why ?
looks good , the question is how can you predict into unknown future ? how do you give it number of days to keep looping on training updating prediction and retraining
That's exactly what we're doing here with walk forward validation, we keep retraining and predicting. That's why walk forward validation is so important for time series.
Just read a raft of papers telling me that tree based methods are not easily adapted to time series… if you give them the simplest model ie extrapolation, can’t do. Dude you do this for a living.
of course there are, and there are others claiming the opposite. We start with a theory, test it in practice and based on the results we decide what is good and what isn't. Papers are incredibly useful as they help you avoid wasting time but in the end what works in the real world matters.
This looks great thanks :) just subbed today, loving your channel
Thank you for the support! Glad you find the content helpful.
@DecisionForest Great Video! Is it common for the model to take over 15 minutes? The validation walkthrough part to be exact. Mine has been running for a good amount of time.
Thanks Malcolm! Yes, the walk forward validation and any type of validation can take quite a long time depending on computing power.
Hi!
Great video. But this was only the validation of the model. Suppose we want to forecast it further. How do we do that?
Cheers Varun! Well after you are satisfied with the results, you train on the full dataset.
@@DecisionForest Hi Thanks for replying. So once we do a walk forward validation on the train test split, I finalize my model. I use that model on the whole dataset. I am not able to understand what will the model return at that step?
Great video.. How do you cater for the case where there's a new major event like the pandemic, because the old trained model wouldn't work anymore? Thanks!
Thank you for this very helpful video. Much appreciated!
Happy I could help!
Thankyou for sharing. I got error in
pred = xgb_predict(history, test_X[0]) and
X, y = train[:, :-1], train[:, -1].
The error message was "IndexError: too many indices for array: array is 0-dimensional, but 2 were indexed".
Could you please tell me how to resolve this error?
Good video, thanks for sharing this!
Could you maybe indicate why can't you after fitting the model just pass the full X_test set and use the predict method available from XGBoost?
Glad it was helpful. Well that’s the trick and why walk forward validation is useful. why would you ignore new data when you have it, of course you could but you want to get the best possible results.
can you give the link to the dataset
Thanks, that was helpful.. can you provide more videos on XAI (Shap, Lime, etc..) and how to inerpret models such as XGBoost, RandomForest, Deep learning NN models and so on
Glad it was helpful. I wanted to do a video on SHAP, I actually created the notebook just that it takes forever to run, it's extremely computationally expensive especially with high dimensional datasets. I'd encourage you to use LIME as it's more practical for real world scenarios even if SHAP may be more interesting on paper.
You’re great Sir!
You’re kind, thank you! Glad it was helpful.
This tutorial video is crystal clear to me up until 13:59 minute, then i lost completly...
What is the meant of 1.7967091070446082 ???
Is it the prediction of the next hour close price??? Or have i miss something???
i mean, how to print out that predictions for the net 1 hour close price???
Thanks, glad it was helpful. That is the RMSE, the root mean squared error, the metric that we want to optimize in this case. You need to optimise for small values of this metric. Hope this helped.
Great content
Thank you Sunil!
Amazing content thanks sir
Most welcome, happy for your support
Merci (:
How can we get the dataset, please?
Just updated the download file to contain the data.
You forgot the plot.
Your model is using one hour's value to predict the next? This is ridiculously useless model haha. But the explanations are great!
model with one variable xD