LSTM Time Series Forecasting Tutorial in Python
Вставка
- Опубліковано 9 лют 2025
- Thank you for watching the video! Here is the Colab Notebook: colab.research...
I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/g...
Learn Python, SQL, & Data Science for free at mlnow.ai/ :)
Subscribe if you enjoyed the video!
Best Courses for Analytics:
---------------------------------------------------------------------------------------------------------
IBM Data Science (Python): bit.ly/3Rn00ZA
Google Analytics (R): bit.ly/3cPikLQ
SQL Basics: bit.ly/3Bd9nFu
Best Courses for Programming:
---------------------------------------------------------------------------------------------------------
Data Science in R: bit.ly/3RhvfFp
Python for Everybody: bit.ly/3ARQ1Ei
Data Structures & Algorithms: bit.ly/3CYR6wR
Best Courses for Machine Learning:
---------------------------------------------------------------------------------------------------------
Math Prerequisites: bit.ly/3ASUtTi
Machine Learning: bit.ly/3d1QATT
Deep Learning: bit.ly/3KPfint
ML Ops: bit.ly/3AWRrxE
Best Courses for Statistics:
---------------------------------------------------------------------------------------------------------
Introduction to Statistics: bit.ly/3QkEgvM
Statistics with Python: bit.ly/3BfwejF
Statistics with R: bit.ly/3QkicBJ
Best Courses for Big Data:
---------------------------------------------------------------------------------------------------------
Google Cloud Data Engineering: bit.ly/3RjHJw6
AWS Data Science: bit.ly/3TKnoBS
Big Data Specialization: bit.ly/3ANqSut
More Courses:
---------------------------------------------------------------------------------------------------------
Tableau: bit.ly/3q966AN
Excel: bit.ly/3RBxind
Computer Vision: bit.ly/3esxVS5
Natural Language Processing: bit.ly/3edXAgW
IBM Dev Ops: bit.ly/3RlVKt2
IBM Full Stack Cloud: bit.ly/3x0pOm6
Object Oriented Programming (Java): bit.ly/3Bfjn0K
TensorFlow Advanced Techniques: bit.ly/3BePQV2
TensorFlow Data and Deployment: bit.ly/3BbC5Xb
Generative Adversarial Networks / GANs (PyTorch): bit.ly/3RHQiRj
I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/greghogg/30min
hey link doesn't work. i copied the code exactly and not working please help
you are my god
Buddy this is the best explanation I have seen on this topic. You are an excellent communicator, if you're not a professor or instructor you should be.
Glad to hear it!
Wow the amount of flexibility on the communication is huge, great skill and of course the LSTM skills and python are perfect, can’t say enough thanks Greg.
Glad to hear it!!
great video! Thanks for sharing. One comment, hard coding `i+5` in the `df_to_X_f` function will lead to some very unexpected results if the `window_size` is not set to 5. Better would be to use the `window_size` variable here to ensure the slice is always the same as the window. Cheers :)
Oops, good catch yes I definitely meant window size, thanks!
@@GregHogg I'm a complete newbie trying to complete his capstone project in a bootcamp. Knowing that you could make that kind of mistake that even I saw really is going to help me with my imposter syndrome moving forward. Also, 11/10 video
@@GregHogg You have used train set instead of test set during prediction
There was not even a single reason why I should leave without subscribing , leaving this comment and liking the video , what an explanation , what a model 🤗
Very nice of you to say! Thank you greatly for the support :)
Beautiful tutorial.
I would kindly like to get your attention towards some bad practice when using the Numpy skills.
You can use np.fromfunction( lambda x, y: x+y, shape=(n_windows, window_size) ) to create a matrix of indices and then slicing the df_as_np with the indices matrix to create X. Then you can use np.reshape to convert it to (n_windows, window_size, 1) shape. That avoids any explicit iterations (for loops) and allows numpy's backend to perform parallelism if possible.
Overall very good and clear tutorial!
Greg, this has been a great help. Even after 1 year, this remains relevant and super useful. Love it ✅
Glad to hear it! :)
You know @@GregHogg I saw this and right now I'm trying to apply this in different scenarios. I'm failing to get it right and I am hoping to reach you. Do you have an instagram? LinkedIn?
Thanks!
You're super super nice and I really appreciate it 😊
@@GregHogg Not at all sir! Great content!
@@ricgondo Thanks so much.
Good explanation overall for the concept. However, some quick suggestions to make the videos a bit more comprehensive is to explain the key details like what is a tensor and why did you design a 3-D matrix for the model, or the part where you flatten the predictions; instead of just saying that the output will have extra dimensions run the code and exhibit it and then apply flatten.
Given this, really like your work to help people like me understand Machine Learning from the very basics. 😊
I really appreciate feedback like this. Very specific on important ideas like the tensor. I will keep this in mind for sure, and I appreciate your kind words!
great video. It was explained in a very simple terms and precise. I watched few others videos from university professors but this one stands out. Great job Greg.
Your video is amazing, man. Thank you so much. I was having problems understanding time series forecasting but you just made everything so clear and easy to understand. Again, thank you😁😁
Probably the best video on youtube for LSTM. Question which I cant find anywhere, if I want to apply this and show what my next forecast would be, would I just use model.predict() and pass in the last 5 intervals to get my next hour prediction? I cannot find a good example anywhere of someone actually using the model to forecast.
That's very nice of you to say, Eric! Yeah, that's what you would do.
Thank you Greg, you cleared my fear of deep learning.
Thanks Greg, for the very informative content on LSTM. It's been great help!
You're very welcome Adithya!
really awesome video, learned a lot about LSTMs. thank you Greg!
Great to hear!!
@@GregHogg do you have anything to predict future stock price? :)
@@moondevonyt Yes check my recent Videos :)
@@GregHogg youre the man, thanks!
This video helped me greatly.
Amazing explanation.
Thank you.
Super glad to hear it :)
Thank you Greg, perfect explanation and presentation!
excellent video and covered all my doubts, thanks Greg...
Thanks for the video! Exactly what I needed
Really glad to hear that!
This is amazing, thank you for this great video.
Wow...the greatest tutorial ever...
Great VDO mate!!! Keep it up!!
Thanks so much!
This video was really helpful with implementing an LSTM in tensorflow! A lot of sources talk about it as either theoretical or building a toy one from scratch. Nice to actually see a tensorflow workflow used. When you are dealing with n-featured examples in the time series, how would you set up the model layers? For example: lets say you used barometric pressure in addition to temperature so your training matrix now looks like : [[ [x11, x12], [x21, x22], ... [xn1, xn2] ]] where x1 is the first entry in the series, x11 is the first feature of the first entry, and n is the window size. How would you set up the model layers? you would be dealing with a 3D matrix, where the third dimension is another feature matrix of window size n. Would the initial input layer just be a tuple of (5, 2)?
edit:
I just realized you have a whole other video on this so I will watch that lol
Great tutorial, thanks!!🎉
Very well explained! Thanks for sharing
ValueError: The filepath provided must end in .keras (Keras model format). Received: filepath=model1/
Solved this by providing model name with extension .keras
For absolute beginners like myself, this is how that line looks like. cp = ModelCheckpoint('model1.keras', save_best_only=True)
@@kyawhan3690 This was an annoying issue I was trying to fix for like 20 minutes. Thank you!
wow ,you're a GOD!!
Thank you haha
Great video! Thanks for the great content!
Hello Greg, first of all, thank you very much for the very lovely and exciting video. I do realize that in the test part, you are not updating the window with the new predictions. So mainly, you are doing predictions for every five timesteps, but then in the window, you are using the actual test values to predict the next value. So my question is did you try in this way as well, and if so, how was your results? Thanks again!
I have the same question!!
Great video, really helped a lot!
Glad to hear it :)
Thank you so much. It definitely was much needed.😇😇
You're very welcome and happy it helped!
Great Video Gregg!
Thanks Akash!
Thank you so much, it's very helpful
You're very welcome and I'm super glad to hear that :)
Bro you sound like Robert Greene and you are just as awesome as he is or even more awesome !
awesome tutorial bro!!!
Thank you!!
great video. Thank you very much
Very welcome!
awesome, great explanation!
Thanks Ken, I really appreciate that!
great video, thank you
You're very welcome!
Thank you. Great video
Great Video!
Thanks for this video
No problem!!
@@GregHogg its hard for me to understand this stuff i have a lower iq than most people :(
@@imveryhungry112 Keep fighting!
Love those voice cracks ❤
Pfffft idk what you're talking about :/
Amazing explanation. Next show how to apply attention mechanism/transformers to time series.
fantastic, thank you very much for sharing
You're very welcome!
Thanks a bunch! Do you have any advice on how to create confidence intervals for predictions?
I'd like to know that too =)
Great video! I have a question, do you have to do normalization in time series forecasting, I see it was not presented in the video.Thanks！
Idk why but I got an error "No file or directory found at model1/" when trying to save the best model
Great stuff Greg! Thanks to this vid I finally got the input dimensions of the LSTM right. One quick question: why is the extra `Dense(8, "relu")` layer necessary, and not just the final Dense(1)?
Thank you, and great to hear! I forget the final model but most likely this is for extra complexity
@@GregHogg Hey so, is it okay to put fully connected layer after LSTM Layer? Thank you
really helpful man, thank u
Amazing Professor appricate tooo.
Thanks a lot, Greg
Hi, great video! Is it possible to run the data in a sliding window way and update the model as it runs? Thank you
If I wanted to forecast 18-24 months into the future, do you have a video on this for a supply chain context? Thanks so much Greg, this video was really well put together for understanding the LSTM Time Series. Also, do you have a video showing how you started off with a baseline model (i.e., Linear Regression or Seasonal Moving Average), then iterated to better models with regression metrics shown?
Thank you! I probably do, although I don't remember sorry
Thank you so much
You're very welcome 🙂
Thank you very much
Nice Video ! Just think about mention François Chollet and Keras for their tutorial
hello
i'm a beginner of machine learning
is it important that the index on the dataframe must use a timestamp before being put into the train model?
is there any effect if the index still uses auto increment?
sorry for bad english
It doesn't do anything once it reaches the NumPy stage. It just allows the use of pandas functions to manipulate the dataframe in pandas
@@GregHogg Thank you so much for your attention
I have 1 more question
how to determine a good number of hidden layers and good activation function ?
I have tried the lstm model with different hidden layer configurations and activation functions several times with my data, but the training results are always not good
Thank you so much for the great lecture. Just a quick question, when we use the test data finally, we are not using the rolling prediction(basically the output predicted by the lstm is appended as the last input and then this sequence is the new input and so on ...). Basically in the absence of the true test data itself. Thank you.
Hello congrats amazing video!!
What should we do differently to consider more variables as X?
Merci 😊
Thank you captain.
Many thanks for this video. Can we use this method for forecasting univariate time series that involve outliers? If so, how to treat the outliers first?
Hello, and thanks for the fantastic video. If I may ask, what would you recommend if I look for a specific kind of time series forecasting in which I don't want to give the model the real values x_test? I am pretending to use the model's prediction to create an x_test_model and then use the model on this set to achieve a y_test_model that I can compare with the real y_test.
Hi! I have a question with respect to the 20:07 part where you define the 'linear' function, in case I'm using a variable that doesn't take negatives like the price of an action which one would you recommend? Thank you :)
Thanks for the video, I'm wondering how to use the first model to predict future values.
Don't you need to scale the data using MinMaxScale or StandardScale between 0 and 1 ? And then once you've done gone thru' the LSTM training, you scale the data back to original ?
Thank you.
What shall be modified in the code in case the X is not a single value (like temperature), but many features vector and y is a single value?
I will cover this in the future. But essentially, we'd probably want the single items in each list (the third dimension) to be more values. Then you'd have to change the input shape of the model accordingly
Thank you.
You're welcome!!
Great! What should I do to improve the accuracy on my trained LSTM model?
Dropout, regularization, adjust the model in any way you see fit!
Great video Greg. Thank you for sharing.
I have a question. Is it better to perform data normalization/standardization before we start the process? Thank you
Thank you. As long as you've preprocessed inputs in some form, you probably can't go too wrong
This is a fantastic video Greg. I was wondering can you do a video on GaussianProcessRegressor? thanks
Thanks very much! And to be honest I've never heard of that haha
Do you consider the prediction of first predicted hour in second input to do further prediction?
Nice tutorial! I'm struggling to conceptually understand what happens when you call predict. I understand the training portion, but how do I get values for say 30 days out? It seems I need to feed it 30 values that I already know the answer for? What is it actually returning when you call predict?!
For example, if I have a dataset of 100 values with a look back/ window of 5. I want to predict values 100 to 105. I train the model on 95 values and call model.predict on the remaining 5, do I get the predictions for 95-100 or 100-105?
thank you for the very good video ...but isnt working with np Arrays exhust the computer memory during fitting ? can we find another way to prepare the data before training... i am working on a multivariate multistep LSTM model ... i have 90 features to predict ine target value ... with many zeros in the target and also in the input features... it is very hallanging specially for multi step forcasting... any advice!?
awesome tutorial bro!!!,any thanks for this video, to confirm, could you say that the 5 data above predict 1 value?
Great video. Do you know where I can find more information on how to determine the number of layers when performing the model.add() commands. For example, I believe you used 16 for your LSTM.
Great video! Question: I don’t understand the difference between validation data and test data. You say that test data has not seen the model before, but so did the validation set? I thought we only divide into train and test data
great, please how should we prepare our data if we want to predict the temperature in different cities, (redundant dates)??
tnx.nice job
Thank you Greg very much for the course! I have one question: if I want to forecast the temperature at 06:00 based on not one X(parameter) but 10 Xs (parameters) from 00:00-05:00(sliding window of 5) how should I tweak my df_to_X_y function?
Great explanation, it really helped me , but i have one question, how can we forecast on the future dates? Knowing that we dont have the X_future?
Thank you! We can recursively feed our output back in as input to the model
@@GregHoggHi, can you give me a hint on how to do this? Please help...
Hi Greg, that was a lovely and informative presentation. but you are using the training data again in prediction. so why did we split the data into testing, training, and validation?
Amazing! How can i use this for raster data? Thank you!
When I try to predict a temperature, the imput should be the prior 5 temp records, right? But the model accept only one value in order to give a prediction, how is that possible?
Hi Greg,
Does your model predict different numbers each time you run it? Could you make it give range instead of one number? Kind of Monte Carlo simulation.
Greg, Thx for great video. I'm a complete noobie, just starting to learn Python. But I'm confused. I've seen a number of videos with output plots for RNNs where, like this one, the prediction line seems to mirror the actual line, often AFTER (to the right of) the actual. Of course, I could take a stubby pencil and look at the thermometer outside my window and write the temp down and say it was my prediction five minutes ago. Voila! One line of code. But the prediction line should be BEFORE the actual line, no? What am I missing? Am I just not seeing it due to the scale on your graph? Rob
i have a data set of about 4500 rows with daily prices and its dates, I am always getting a loss of around 1.2, and after that it keeps oscillating around this number . How do i resolve this issue ?
Can I use the R2 value to measure the accuracy of the LSTM model in time series prediction?
hi Greg, what if my target variable is not normally distributed but positively skewed? what should I do in such case? and also, is it recommended to datetime function as cos and sin of hour, weeks n all to provide it in the x variable?
instead of creating a window function, you can just use the pandas "shift" function to create columns that are shifted by 1 each, using a for loop
Why do you use rmse over r2 score? I would like to know thanks!
Excellent video Greg, I had two questions:
1. Why do you indicate a window size of 5, could I have chosen another value, maybe 10 or 20?
2. If after running the LSTM model I get a (loss: Nan) and the (MAE: Nan), what could I be doing wrong?
Thanks for your help. Regards
Thank you! 1. Pick any window you want. 2. Not sure unfortunately!
Hello Jesus. Your dataset might have some Nan or 0 values. Please check it.
@@alperenkoza9624 thanks Aperen Exactly I had those values in my dataset
Hello! How can I make further prediction using this code? I mean, we predicted the first 1-60000 values, but how to predict those that were not in the original dataset? Thank you for a tutorial!
You can use the model to predict the next value, and then use that prediction as input to the model to predict the following value and so on :)
@@GregHogg Thanks for the answer! I did this, and the prediction quickly degenerated into a straight line (after like 15 iterations)... Is this normal?
Hi! Excelent video! How you can apply the model with new data, i mean when you have new variables without the temperature values. You would like to predict the new (future) temperature values? I hope you can help me with this. Thanks!
Amazing! Congrats for great explanation, my friend. I have a question. If I wanted to forecast future temperature regarding date in the future, how could I do that? Regards, Lucas from Brazil.
Which data you as been taken in this video can you please show us or update a dataset your
fantastic video, can you also explain us how to work with multistep forecast?
This seems to be highly requested, yes I will make a video on this :)
This also seems only helpful if you want to predict a singular value in the future. What if you wanted a very large number of predictions?
Hi there! Found this really helpful, but I've managed to confuse myself about the data windowing and the test set. Since every target in y_test (except the last one) is somewhere in X_test, how are we not assuming our solution? I had a look at your stock price notebook and there's the bit about recursive predictions, but you don't have that here. Any clarification?
Would this approach work if you wanted to predict the next say 24 or 96 values, instead of just the next 1?