LSTM Time Series Forecasting Tutorial in Python
Вставка
- Опубліковано 3 жов 2021
- Thank you for watching the video! Here is the Colab Notebook: colab.research.google.com/dri...
I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/greghogg/30min
Learn Python, SQL, & Data Science for free at mlnow.ai/ :)
Subscribe if you enjoyed the video!
Best Courses for Analytics:
---------------------------------------------------------------------------------------------------------
+ IBM Data Science (Python): bit.ly/3Rn00ZA
+ Google Analytics (R): bit.ly/3cPikLQ
+ SQL Basics: bit.ly/3Bd9nFu
Best Courses for Programming:
---------------------------------------------------------------------------------------------------------
+ Data Science in R: bit.ly/3RhvfFp
+ Python for Everybody: bit.ly/3ARQ1Ei
+ Data Structures & Algorithms: bit.ly/3CYR6wR
Best Courses for Machine Learning:
---------------------------------------------------------------------------------------------------------
+ Math Prerequisites: bit.ly/3ASUtTi
+ Machine Learning: bit.ly/3d1QATT
+ Deep Learning: bit.ly/3KPfint
+ ML Ops: bit.ly/3AWRrxE
Best Courses for Statistics:
---------------------------------------------------------------------------------------------------------
+ Introduction to Statistics: bit.ly/3QkEgvM
+ Statistics with Python: bit.ly/3BfwejF
+ Statistics with R: bit.ly/3QkicBJ
Best Courses for Big Data:
---------------------------------------------------------------------------------------------------------
+ Google Cloud Data Engineering: bit.ly/3RjHJw6
+ AWS Data Science: bit.ly/3TKnoBS
+ Big Data Specialization: bit.ly/3ANqSut
More Courses:
---------------------------------------------------------------------------------------------------------
+ Tableau: bit.ly/3q966AN
+ Excel: bit.ly/3RBxind
+ Computer Vision: bit.ly/3esxVS5
+ Natural Language Processing: bit.ly/3edXAgW
+ IBM Dev Ops: bit.ly/3RlVKt2
+ IBM Full Stack Cloud: bit.ly/3x0pOm6
+ Object Oriented Programming (Java): bit.ly/3Bfjn0K
+ TensorFlow Advanced Techniques: bit.ly/3BePQV2
+ TensorFlow Data and Deployment: bit.ly/3BbC5Xb
+ Generative Adversarial Networks / GANs (PyTorch): bit.ly/3RHQiRj
I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/greghogg/30min
Wow the amount of flexibility on the communication is huge, great skill and of course the LSTM skills and python are perfect, can’t say enough thanks Greg.
Glad to hear it!!
Your video is amazing, man. Thank you so much. I was having problems understanding time series forecasting but you just made everything so clear and easy to understand. Again, thank you😁😁
great video! Thanks for sharing. One comment, hard coding `i+5` in the `df_to_X_f` function will lead to some very unexpected results if the `window_size` is not set to 5. Better would be to use the `window_size` variable here to ensure the slice is always the same as the window. Cheers :)
Oops, good catch yes I definitely meant window size, thanks!
@@GregHogg I'm a complete newbie trying to complete his capstone project in a bootcamp. Knowing that you could make that kind of mistake that even I saw really is going to help me with my imposter syndrome moving forward. Also, 11/10 video
Thank you Greg, you cleared my fear of deep learning.
Buddy this is the best explanation I have seen on this topic. You are an excellent communicator, if you're not a professor or instructor you should be.
Glad to hear it!
Thank you Greg, perfect explanation and presentation!
Greg, this has been a great help. Even after 1 year, this remains relevant and super useful. Love it ✅
Glad to hear it! :)
You know @@GregHogg I saw this and right now I'm trying to apply this in different scenarios. I'm failing to get it right and I am hoping to reach you. Do you have an instagram? LinkedIn?
you sir are a thousands times better than my professor appreciate it
excellent video and covered all my doubts, thanks Greg...
Beautiful tutorial.
I would kindly like to get your attention towards some bad practice when using the Numpy skills.
You can use np.fromfunction( lambda x, y: x+y, shape=(n_windows, window_size) ) to create a matrix of indices and then slicing the df_as_np with the indices matrix to create X. Then you can use np.reshape to convert it to (n_windows, window_size, 1) shape. That avoids any explicit iterations (for loops) and allows numpy's backend to perform parallelism if possible.
Overall very good and clear tutorial!
Very well explained! Thanks for sharing
Good explanation overall for the concept. However, some quick suggestions to make the videos a bit more comprehensive is to explain the key details like what is a tensor and why did you design a 3-D matrix for the model, or the part where you flatten the predictions; instead of just saying that the output will have extra dimensions run the code and exhibit it and then apply flatten.
Given this, really like your work to help people like me understand Machine Learning from the very basics. 😊
I really appreciate feedback like this. Very specific on important ideas like the tensor. I will keep this in mind for sure, and I appreciate your kind words!
There was not even a single reason why I should leave without subscribing , leaving this comment and liking the video , what an explanation , what a model 🤗
Very nice of you to say! Thank you greatly for the support :)
Thanks Greg, for the very informative content on LSTM. It's been great help!
You're very welcome Adithya!
This video was really helpful with implementing an LSTM in tensorflow! A lot of sources talk about it as either theoretical or building a toy one from scratch. Nice to actually see a tensorflow workflow used. When you are dealing with n-featured examples in the time series, how would you set up the model layers? For example: lets say you used barometric pressure in addition to temperature so your training matrix now looks like : [[ [x11, x12], [x21, x22], ... [xn1, xn2] ]] where x1 is the first entry in the series, x11 is the first feature of the first entry, and n is the window size. How would you set up the model layers? you would be dealing with a 3D matrix, where the third dimension is another feature matrix of window size n. Would the initial input layer just be a tuple of (5, 2)?
edit:
I just realized you have a whole other video on this so I will watch that lol
Hi Howell, thank you for sharing your valuable information through this channel. I am one of the new followers of time series. If possible, could you create a series on how to implement Transformers on time series data, covering both univariate and multivariate approaches? Focusing on operations like forecasting, classification, or anomaly detection-just one of these would be greatly appreciated. There are no videos available on UA-cam that have implemented this before. It would be extremely helpful for students and new researchers in the field of time series.
Thanks for the video! Exactly what I needed
Really glad to hear that!
This video helped me greatly.
Amazing explanation.
Thank you.
Super glad to hear it :)
Probably the best video on youtube for LSTM. Question which I cant find anywhere, if I want to apply this and show what my next forecast would be, would I just use model.predict() and pass in the last 5 intervals to get my next hour prediction? I cannot find a good example anywhere of someone actually using the model to forecast.
That's very nice of you to say, Eric! Yeah, that's what you would do.
Hello congrats amazing video!!
What should we do differently to consider more variables as X?
really awesome video, learned a lot about LSTMs. thank you Greg!
Great to hear!!
@@GregHogg do you have anything to predict future stock price? :)
@@moondevonyt Yes check my recent Videos :)
@@GregHogg youre the man, thanks!
Great VDO mate!!! Keep it up!!
Thanks so much!
Great video! Thanks for the great content!
Thank you so much. It definitely was much needed.😇😇
You're very welcome and happy it helped!
Great video, really helped a lot!
Glad to hear it :)
Many thanks for this video. Can we use this method for forecasting univariate time series that involve outliers? If so, how to treat the outliers first?
really helpful man, thank u
Hello Greg, first of all, thank you very much for the very lovely and exciting video. I do realize that in the test part, you are not updating the window with the new predictions. So mainly, you are doing predictions for every five timesteps, but then in the window, you are using the actual test values to predict the next value. So my question is did you try in this way as well, and if so, how was your results? Thanks again!
I have the same question!!
Greg, Thx for great video. I'm a complete noobie, just starting to learn Python. But I'm confused. I've seen a number of videos with output plots for RNNs where, like this one, the prediction line seems to mirror the actual line, often AFTER (to the right of) the actual. Of course, I could take a stubby pencil and look at the thermometer outside my window and write the temp down and say it was my prediction five minutes ago. Voila! One line of code. But the prediction line should be BEFORE the actual line, no? What am I missing? Am I just not seeing it due to the scale on your graph? Rob
Great Video Gregg!
Thanks Akash!
fantastic, thank you very much for sharing
You're very welcome!
awesome, great explanation!
Thanks Ken, I really appreciate that!
Hello, amazing video, may I ask something?
So lets say we were to predict a single value, we don't need to use the time somehow or not? Does the model kinda learns to which time the data sequence is relationed? like: "this values n sequence... must be winter so the prediction is: x!" right???
Great video. Do you know where I can find more information on how to determine the number of layers when performing the model.add() commands. For example, I believe you used 16 for your LSTM.
Hi! I have a question with respect to the 20:07 part where you define the 'linear' function, in case I'm using a variable that doesn't take negatives like the price of an action which one would you recommend? Thank you :)
awesome tutorial bro!!!,any thanks for this video, to confirm, could you say that the 5 data above predict 1 value?
Thank you so much, it's very helpful
You're very welcome and I'm super glad to hear that :)
Thanks a lot, Greg
Amazing! Congrats for great explanation, my friend. I have a question. If I wanted to forecast future temperature regarding date in the future, how could I do that? Regards, Lucas from Brazil.
Thank you Greg very much for the course! I have one question: if I want to forecast the temperature at 06:00 based on not one X(parameter) but 10 Xs (parameters) from 00:00-05:00(sliding window of 5) how should I tweak my df_to_X_y function?
Hello, and thanks for the fantastic video. If I may ask, what would you recommend if I look for a specific kind of time series forecasting in which I don't want to give the model the real values x_test? I am pretending to use the model's prediction to create an x_test_model and then use the model on this set to achieve a y_test_model that I can compare with the real y_test.
Thank you captain.
great video. Thank you very much
Very welcome!
Great video! I have a question, do you have to do normalization in time series forecasting, I see it was not presented in the video.Thanks！
Nice Video ! Just think about mention François Chollet and Keras for their tutorial
Nice tutorial! I'm struggling to conceptually understand what happens when you call predict. I understand the training portion, but how do I get values for say 30 days out? It seems I need to feed it 30 values that I already know the answer for? What is it actually returning when you call predict?!
For example, if I have a dataset of 100 values with a look back/ window of 5. I want to predict values 100 to 105. I train the model on 95 values and call model.predict on the remaining 5, do I get the predictions for 95-100 or 100-105?
great video, thank you
You're very welcome!
Hi Greg,
Does your model predict different numbers each time you run it? Could you make it give range instead of one number? Kind of Monte Carlo simulation.
I have a question. How do I line up the Dates with the corresponding actual values and predicted values? It seems like the lengths don't matchup
Great video. I do time series forecasting using ARIMA and Prophet models. I wanted to have a go at a different approach such as LSTM, and this video is very helpful. I do have a couple of questions for you, and all of the readers here. What would I need to do if I wanted to forecast temperature for the next 60 days (or x number of days) with time stamp as our index? Thank you in advance!
Thanks Jake!! I'll defer you to the TensorFlow tutorials for this one, it's where I based my lesson off. www.tensorflow.org/tutorials/structured_data/time_series
tnx.nice job
Great video Greg. Thank you for sharing.
I have a question. Is it better to perform data normalization/standardization before we start the process? Thank you
Thank you. As long as you've preprocessed inputs in some form, you probably can't go too wrong
Bro you sound like Robert Greene and you are just as awesome as he is or even more awesome !
Great stuff Greg! Thanks to this vid I finally got the input dimensions of the LSTM right. One quick question: why is the extra `Dense(8, "relu")` layer necessary, and not just the final Dense(1)?
Thank you, and great to hear! I forget the final model but most likely this is for extra complexity
@@GregHogg Hey so, is it okay to put fully connected layer after LSTM Layer? Thank you
Merci 😊
Hi Greg, that was a lovely and informative presentation. but you are using the training data again in prediction. so why did we split the data into testing, training, and validation?
awesome tutorial bro!!!
Thank you!!
Can I use the R2 value to measure the accuracy of the LSTM model in time series prediction?
Thank you.
What shall be modified in the code in case the X is not a single value (like temperature), but many features vector and y is a single value?
I will cover this in the future. But essentially, we'd probably want the single items in each list (the third dimension) to be more values. Then you'd have to change the input shape of the model accordingly
Hi, great video! Is it possible to run the data in a sliding window way and update the model as it runs? Thank you
thank you for the very good video ...but isnt working with np Arrays exhust the computer memory during fitting ? can we find another way to prepare the data before training... i am working on a multivariate multistep LSTM model ... i have 90 features to predict ine target value ... with many zeros in the target and also in the input features... it is very hallanging specially for multi step forcasting... any advice!?
Hi there! Found this really helpful, but I've managed to confuse myself about the data windowing and the test set. Since every target in y_test (except the last one) is somewhere in X_test, how are we not assuming our solution? I had a look at your stock price notebook and there's the bit about recursive predictions, but you don't have that here. Any clarification?
hi Greg, what if my target variable is not normally distributed but positively skewed? what should I do in such case? and also, is it recommended to datetime function as cos and sin of hour, weeks n all to provide it in the x variable?
Hi Gregg , when running the predictions :train_results = pd.DataFrame(data={'Train Predictions':train_predictions, 'Actuals':y_train1}) i have noticed that you've used data but no prior reference could you state why?
Great video! Question: I don’t understand the difference between validation data and test data. You say that test data has not seen the model before, but so did the validation set? I thought we only divide into train and test data
This is a fantastic video Greg. I was wondering can you do a video on GaussianProcessRegressor? thanks
Thanks very much! And to be honest I've never heard of that haha
This also seems only helpful if you want to predict a singular value in the future. What if you wanted a very large number of predictions?
Do you consider the prediction of first predicted hour in second input to do further prediction?
wow ,you're a GOD!!
Thank you haha
great, please how should we prepare our data if we want to predict the temperature in different cities, (redundant dates)??
thanks for video
Would this approach work if you wanted to predict the next say 24 or 96 values, instead of just the next 1?
Great! What should I do to improve the accuracy on my trained LSTM model?
Dropout, regularization, adjust the model in any way you see fit!
thx
Love those voice cracks ❤
Pfffft idk what you're talking about :/
Thank you so much
You're very welcome 🙂
Can you giv e an example of multivariate forecasting?Also if i have temperature datas in two different files and i want to train the LSTM on both the datasets, how will i include that ? Example: first dataset is say temperature on antartica (-10 to 5 degree celsius) and other file has temperature of south africa(30-50 degree celsius). I cant just simply append both data frames. So do i need to train it twice on both datasets?
Idk why but I got an error "No file or directory found at model1/" when trying to save the best model
Thank you.
You're welcome!!
I have tried creating a similar forecasting model but have to normalise the data, whereas you did not in your code. Is there another reason I'm not understanding that allowed your model to work with the real values?
fantastic video, can you also explain us how to work with multistep forecast?
This seems to be highly requested, yes I will make a video on this :)
instead of creating a window function, you can just use the pandas "shift" function to create columns that are shifted by 1 each, using a for loop
Thanks for this video
No problem!!
@@GregHogg its hard for me to understand this stuff i have a lower iq than most people :(
@@imveryhungry112 Keep fighting!
Thanks a bunch! Do you have any advice on how to create confidence intervals for predictions?
I'd like to know that too =)
how would the df_to_X_y function change if instead of only having a date and temperature column you also have a location column because now X would have the format (n_samples, window_size, 2)
If I wanted to forecast 18-24 months into the future, do you have a video on this for a supply chain context? Thanks so much Greg, this video was really well put together for understanding the LSTM Time Series. Also, do you have a video showing how you started off with a baseline model (i.e., Linear Regression or Seasonal Moving Average), then iterated to better models with regression metrics shown?
Thank you! I probably do, although I don't remember sorry
Great video with good explanation!
But if I want to forecast the values for the next month how should I do it?
I would my stock prediction video. And thank you :)
Why do you use rmse over r2 score? I would like to know thanks!
How would we do if we did not know the values of x that correspond to y that we want to forecast?
very well explained video, but in terms of the results you show, it looks like your predictions are lagging behind the actual values. Meaning the model doesn't generalize well enough and relay in its prediction on the last known value. It's seen there the trend of the blue line(prediction) always one step behind the trend of the orange line(actual values).
I realized it was the previous statement to create a problem:
model1.add(InputLayer((5, 1)))
I cancelled one parenthesis believing it was redundant and I wrote model1.add(InputLayer(5, 1))
but this causes a crash to the following line.
At the moment it is not exactly clear to me the reason. I have to study about.
In any case I can go on now.
Sorry for the inconvenience.
Thank you so much, I followed the tutorial and got my first LSTM network working. I am however having trouble with generating out of sample predictions, I feel like I have tried everything and just can’t crack it. I am using the last batch of the test set to predict the first out of sample data point, but it ends in an error. I feel like it’s a dimension thing. Any help, anyone?
How is it possible that my model performs well when training and predicting with a dataset with many examples, but when I use only one example (to make the prediction) it performs really bad? Is it a problem of how the data is structured or do I have to make adjustments to the model?
This video at 13.26 shows a time series function, but it is single-step forecasting only... can you show how to modify it for multistep. I have tried to add multiple labels in the loop but it is not working with LSTM. Thanks, Greg for this video also.
SMOOTH!!!!!
Hi! i used the same code but shape of X,y is not right. they give(1,20,1),(1,20) they are not giving correct n_samples, total number of samples are 365557, and after minus 20 as window size it coulde be 365537.where is the problem
Why doesn't it show the same performance when I put P bar instead of temperature? The temperature values are small and have negative values, so I don't understand, did you make an arrangement?
hey I need a help, I got an error saying ( UnimplementedError: Graph execution error:), what should I do?
does the lstm train only on 1 input ? i see this ((70086, 5, 1), for X.shape