There was not even a single reason why I should leave without subscribing , leaving this comment and liking the video , what an explanation , what a model 🤗
Buddy this is the best explanation I have seen on this topic. You are an excellent communicator, if you're not a professor or instructor you should be.
great video! Thanks for sharing. One comment, hard coding `i+5` in the `df_to_X_f` function will lead to some very unexpected results if the `window_size` is not set to 5. Better would be to use the `window_size` variable here to ensure the slice is always the same as the window. Cheers :)
@@GregHogg I'm a complete newbie trying to complete his capstone project in a bootcamp. Knowing that you could make that kind of mistake that even I saw really is going to help me with my imposter syndrome moving forward. Also, 11/10 video
Beautiful tutorial. I would kindly like to get your attention towards some bad practice when using the Numpy skills. You can use np.fromfunction( lambda x, y: x+y, shape=(n_windows, window_size) ) to create a matrix of indices and then slicing the df_as_np with the indices matrix to create X. Then you can use np.reshape to convert it to (n_windows, window_size, 1) shape. That avoids any explicit iterations (for loops) and allows numpy's backend to perform parallelism if possible. Overall very good and clear tutorial!
Wow the amount of flexibility on the communication is huge, great skill and of course the LSTM skills and python are perfect, can’t say enough thanks Greg.
You know @@GregHogg I saw this and right now I'm trying to apply this in different scenarios. I'm failing to get it right and I am hoping to reach you. Do you have an instagram? LinkedIn?
Good explanation overall for the concept. However, some quick suggestions to make the videos a bit more comprehensive is to explain the key details like what is a tensor and why did you design a 3-D matrix for the model, or the part where you flatten the predictions; instead of just saying that the output will have extra dimensions run the code and exhibit it and then apply flatten. Given this, really like your work to help people like me understand Machine Learning from the very basics. 😊
I really appreciate feedback like this. Very specific on important ideas like the tensor. I will keep this in mind for sure, and I appreciate your kind words!
Thank you so much for the great lecture. Just a quick question, when we use the test data finally, we are not using the rolling prediction(basically the output predicted by the lstm is appended as the last input and then this sequence is the new input and so on ...). Basically in the absence of the true test data itself. Thank you.
Your video is amazing, man. Thank you so much. I was having problems understanding time series forecasting but you just made everything so clear and easy to understand. Again, thank you😁😁
Probably the best video on youtube for LSTM. Question which I cant find anywhere, if I want to apply this and show what my next forecast would be, would I just use model.predict() and pass in the last 5 intervals to get my next hour prediction? I cannot find a good example anywhere of someone actually using the model to forecast.
ValueError: The filepath provided must end in .keras (Keras model format). Received: filepath=model1/ Solved this by providing model name with extension .keras
This video was really helpful with implementing an LSTM in tensorflow! A lot of sources talk about it as either theoretical or building a toy one from scratch. Nice to actually see a tensorflow workflow used. When you are dealing with n-featured examples in the time series, how would you set up the model layers? For example: lets say you used barometric pressure in addition to temperature so your training matrix now looks like : [[ [x11, x12], [x21, x22], ... [xn1, xn2] ]] where x1 is the first entry in the series, x11 is the first feature of the first entry, and n is the window size. How would you set up the model layers? you would be dealing with a 3D matrix, where the third dimension is another feature matrix of window size n. Would the initial input layer just be a tuple of (5, 2)? edit: I just realized you have a whole other video on this so I will watch that lol
Thank you Greg very much for the course! I have one question: if I want to forecast the temperature at 06:00 based on not one X(parameter) but 10 Xs (parameters) from 00:00-05:00(sliding window of 5) how should I tweak my df_to_X_y function?
Hello, and thanks for the fantastic video. If I may ask, what would you recommend if I look for a specific kind of time series forecasting in which I don't want to give the model the real values x_test? I am pretending to use the model's prediction to create an x_test_model and then use the model on this set to achieve a y_test_model that I can compare with the real y_test.
Nice tutorial! I'm struggling to conceptually understand what happens when you call predict. I understand the training portion, but how do I get values for say 30 days out? It seems I need to feed it 30 values that I already know the answer for? What is it actually returning when you call predict?! For example, if I have a dataset of 100 values with a look back/ window of 5. I want to predict values 100 to 105. I train the model on 95 values and call model.predict on the remaining 5, do I get the predictions for 95-100 or 100-105?
Can you giv e an example of multivariate forecasting?Also if i have temperature datas in two different files and i want to train the LSTM on both the datasets, how will i include that ? Example: first dataset is say temperature on antartica (-10 to 5 degree celsius) and other file has temperature of south africa(30-50 degree celsius). I cant just simply append both data frames. So do i need to train it twice on both datasets?
hi Greg, what if my target variable is not normally distributed but positively skewed? what should I do in such case? and also, is it recommended to datetime function as cos and sin of hour, weeks n all to provide it in the x variable?
When I try to predict a temperature, the imput should be the prior 5 temp records, right? But the model accept only one value in order to give a prediction, how is that possible?
Don't you need to scale the data using MinMaxScale or StandardScale between 0 and 1 ? And then once you've done gone thru' the LSTM training, you scale the data back to original ?
Hello Greg, first of all, thank you very much for the very lovely and exciting video. I do realize that in the test part, you are not updating the window with the new predictions. So mainly, you are doing predictions for every five timesteps, but then in the window, you are using the actual test values to predict the next value. So my question is did you try in this way as well, and if so, how was your results? Thanks again!
how would the df_to_X_y function change if instead of only having a date and temperature column you also have a location column because now X would have the format (n_samples, window_size, 2)
i have a data set of about 4500 rows with daily prices and its dates, I am always getting a loss of around 1.2, and after that it keeps oscillating around this number . How do i resolve this issue ?
thank you for the very good video ...but isnt working with np Arrays exhust the computer memory during fitting ? can we find another way to prepare the data before training... i am working on a multivariate multistep LSTM model ... i have 90 features to predict ine target value ... with many zeros in the target and also in the input features... it is very hallanging specially for multi step forcasting... any advice!?
Hello, amazing video, may I ask something? So lets say we were to predict a single value, we don't need to use the time somehow or not? Does the model kinda learns to which time the data sequence is relationed? like: "this values n sequence... must be winter so the prediction is: x!" right???
Hi! I have a question with respect to the 20:07 part where you define the 'linear' function, in case I'm using a variable that doesn't take negatives like the price of an action which one would you recommend? Thank you :)
Great video! Question: I don’t understand the difference between validation data and test data. You say that test data has not seen the model before, but so did the validation set? I thought we only divide into train and test data
Hi there! Found this really helpful, but I've managed to confuse myself about the data windowing and the test set. Since every target in y_test (except the last one) is somewhere in X_test, how are we not assuming our solution? I had a look at your stock price notebook and there's the bit about recursive predictions, but you don't have that here. Any clarification?
Hi Greg, Does your model predict different numbers each time you run it? Could you make it give range instead of one number? Kind of Monte Carlo simulation.
If I wanted to forecast 18-24 months into the future, do you have a video on this for a supply chain context? Thanks so much Greg, this video was really well put together for understanding the LSTM Time Series. Also, do you have a video showing how you started off with a baseline model (i.e., Linear Regression or Seasonal Moving Average), then iterated to better models with regression metrics shown?
hello i'm a beginner of machine learning is it important that the index on the dataframe must use a timestamp before being put into the train model? is there any effect if the index still uses auto increment? sorry for bad english
@@GregHogg Thank you so much for your attention I have 1 more question how to determine a good number of hidden layers and good activation function ? I have tried the lstm model with different hidden layer configurations and activation functions several times with my data, but the training results are always not good
Thank you. What shall be modified in the code in case the X is not a single value (like temperature), but many features vector and y is a single value?
I will cover this in the future. But essentially, we'd probably want the single items in each list (the third dimension) to be more values. Then you'd have to change the input shape of the model accordingly
How is it possible that my model performs well when training and predicting with a dataset with many examples, but when I use only one example (to make the prediction) it performs really bad? Is it a problem of how the data is structured or do I have to make adjustments to the model?
This video at 13.26 shows a time series function, but it is single-step forecasting only... can you show how to modify it for multistep. I have tried to add multiple labels in the loop but it is not working with LSTM. Thanks, Greg for this video also.
Hi! i used the same code but shape of X,y is not right. they give(1,20,1),(1,20) they are not giving correct n_samples, total number of samples are 365557, and after minus 20 as window size it coulde be 365537.where is the problem
Great stuff Greg! Thanks to this vid I finally got the input dimensions of the LSTM right. One quick question: why is the extra `Dense(8, "relu")` layer necessary, and not just the final Dense(1)?
Hi! Excelent video! How you can apply the model with new data, i mean when you have new variables without the temperature values. You would like to predict the new (future) temperature values? I hope you can help me with this. Thanks!
Why doesn't it show the same performance when I put P bar instead of temperature? The temperature values are small and have negative values, so I don't understand, did you make an arrangement?
in df_to_X_y() shouldn't row= [[a] for a in df_as_np[i:i+window_size]] and not i+5? I'm asking because i am new to python and i don't fully understand it yet but if feel to generalize this funtion it should be i+window_size and not i+5
Oh, sorry, ignore this. Found the answer in @mandem2735 comment from a year ago. Many thanks for the video. works great for temperature dataset. Struggling with stock prices but getting there...
Hi Gregg , when running the predictions :train_results = pd.DataFrame(data={'Train Predictions':train_predictions, 'Actuals':y_train1}) i have noticed that you've used data but no prior reference could you state why?
Many thanks for this video. Can we use this method for forecasting univariate time series that involve outliers? If so, how to treat the outliers first?
Great video. Do you know where I can find more information on how to determine the number of layers when performing the model.add() commands. For example, I believe you used 16 for your LSTM.
I have tried creating a similar forecasting model but have to normalise the data, whereas you did not in your code. Is there another reason I'm not understanding that allowed your model to work with the real values?
Hi Greg, that was a lovely and informative presentation. but you are using the training data again in prediction. so why did we split the data into testing, training, and validation?
Hello! How can I make further prediction using this code? I mean, we predicted the first 1-60000 values, but how to predict those that were not in the original dataset? Thank you for a tutorial!
@@GregHogg Thanks for the answer! I did this, and the prediction quickly degenerated into a straight line (after like 15 iterations)... Is this normal?
So... the predictions it makes are within what timeframe? By hours? By days? I put in data by month and for example the actual value is 200, I got the prediction is 1.2 or something like this. I am going to re-do it with daily data instead, but is this 1.2 supposed to be an hour, a day?
Okay, I changed learning rate to .1 and got way better values but they are all the same as if I am doing a simple exponential smoothing 😂😂 The data I'm working with must be horrible.
Amazing! Congrats for great explanation, my friend. I have a question. If I wanted to forecast future temperature regarding date in the future, how could I do that? Regards, Lucas from Brazil.
Great video Greg. Thank you for sharing. I have a question. Is it better to perform data normalization/standardization before we start the process? Thank you
Thank you so much, I followed the tutorial and got my first LSTM network working. I am however having trouble with generating out of sample predictions, I feel like I have tried everything and just can’t crack it. I am using the last batch of the test set to predict the first out of sample data point, but it ends in an error. I feel like it’s a dimension thing. Any help, anyone?
Hello, thank you so much for this video, after doing train_predictions = model1.predict(X_train1).flatten() train_results = pd.DataFrame(data={'Train Predictions':train_predictions, 'Actuals':y_train1}) train_results i got this error: ValueError: All arrays must be of the same length What should I do ?
What about packing (X,y) and doing shuffling and do train, val split. Or is it okay/valid to shuffle like this for this type of time series forecasting?
very well explained video, but in terms of the results you show, it looks like your predictions are lagging behind the actual values. Meaning the model doesn't generalize well enough and relay in its prediction on the last known value. It's seen there the trend of the blue line(prediction) always one step behind the trend of the orange line(actual values).
Just found the reason why…but still don’t understand why… Basically I was using a one column df…and that’s gave me this extra one. When I used a multiple column df from which i extracted the interesting column I obtained the good shape for X and y
Hi!! Thank you for the content!!! When I run the df to x y fonction on my data I have this shape (431,5,1,1) , (431,1) My data is one column of data and index as date time. No idea we here this extra 1 is coming from…
I realized it was the previous statement to create a problem: model1.add(InputLayer((5, 1))) I cancelled one parenthesis believing it was redundant and I wrote model1.add(InputLayer(5, 1)) but this causes a crash to the following line. At the moment it is not exactly clear to me the reason. I have to study about. In any case I can go on now. Sorry for the inconvenience.
I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/greghogg/30min
hey link doesn't work. i copied the code exactly and not working please help
you are my god
There was not even a single reason why I should leave without subscribing , leaving this comment and liking the video , what an explanation , what a model 🤗
Very nice of you to say! Thank you greatly for the support :)
Buddy this is the best explanation I have seen on this topic. You are an excellent communicator, if you're not a professor or instructor you should be.
Glad to hear it!
great video! Thanks for sharing. One comment, hard coding `i+5` in the `df_to_X_f` function will lead to some very unexpected results if the `window_size` is not set to 5. Better would be to use the `window_size` variable here to ensure the slice is always the same as the window. Cheers :)
Oops, good catch yes I definitely meant window size, thanks!
@@GregHogg I'm a complete newbie trying to complete his capstone project in a bootcamp. Knowing that you could make that kind of mistake that even I saw really is going to help me with my imposter syndrome moving forward. Also, 11/10 video
@@GregHogg You have used train set instead of test set during prediction
Beautiful tutorial.
I would kindly like to get your attention towards some bad practice when using the Numpy skills.
You can use np.fromfunction( lambda x, y: x+y, shape=(n_windows, window_size) ) to create a matrix of indices and then slicing the df_as_np with the indices matrix to create X. Then you can use np.reshape to convert it to (n_windows, window_size, 1) shape. That avoids any explicit iterations (for loops) and allows numpy's backend to perform parallelism if possible.
Overall very good and clear tutorial!
Wow the amount of flexibility on the communication is huge, great skill and of course the LSTM skills and python are perfect, can’t say enough thanks Greg.
Glad to hear it!!
you sir are a thousands times better than my professor appreciate it
Greg, this has been a great help. Even after 1 year, this remains relevant and super useful. Love it ✅
Glad to hear it! :)
You know @@GregHogg I saw this and right now I'm trying to apply this in different scenarios. I'm failing to get it right and I am hoping to reach you. Do you have an instagram? LinkedIn?
Good explanation overall for the concept. However, some quick suggestions to make the videos a bit more comprehensive is to explain the key details like what is a tensor and why did you design a 3-D matrix for the model, or the part where you flatten the predictions; instead of just saying that the output will have extra dimensions run the code and exhibit it and then apply flatten.
Given this, really like your work to help people like me understand Machine Learning from the very basics. 😊
I really appreciate feedback like this. Very specific on important ideas like the tensor. I will keep this in mind for sure, and I appreciate your kind words!
Thank you so much for the great lecture. Just a quick question, when we use the test data finally, we are not using the rolling prediction(basically the output predicted by the lstm is appended as the last input and then this sequence is the new input and so on ...). Basically in the absence of the true test data itself. Thank you.
Your video is amazing, man. Thank you so much. I was having problems understanding time series forecasting but you just made everything so clear and easy to understand. Again, thank you😁😁
Probably the best video on youtube for LSTM. Question which I cant find anywhere, if I want to apply this and show what my next forecast would be, would I just use model.predict() and pass in the last 5 intervals to get my next hour prediction? I cannot find a good example anywhere of someone actually using the model to forecast.
That's very nice of you to say, Eric! Yeah, that's what you would do.
Thanks!
You're super super nice and I really appreciate it 😊
@@GregHogg Not at all sir! Great content!
@@ricgondo Thanks so much.
Thank you Greg, you cleared my fear of deep learning.
Thanks Greg, for the very informative content on LSTM. It's been great help!
You're very welcome Adithya!
ValueError: The filepath provided must end in .keras (Keras model format). Received: filepath=model1/
Solved this by providing model name with extension .keras
For absolute beginners like myself, this is how that line looks like. cp = ModelCheckpoint('model1.keras', save_best_only=True)
This video was really helpful with implementing an LSTM in tensorflow! A lot of sources talk about it as either theoretical or building a toy one from scratch. Nice to actually see a tensorflow workflow used. When you are dealing with n-featured examples in the time series, how would you set up the model layers? For example: lets say you used barometric pressure in addition to temperature so your training matrix now looks like : [[ [x11, x12], [x21, x22], ... [xn1, xn2] ]] where x1 is the first entry in the series, x11 is the first feature of the first entry, and n is the window size. How would you set up the model layers? you would be dealing with a 3D matrix, where the third dimension is another feature matrix of window size n. Would the initial input layer just be a tuple of (5, 2)?
edit:
I just realized you have a whole other video on this so I will watch that lol
Thank you Greg, perfect explanation and presentation!
Thank you Greg very much for the course! I have one question: if I want to forecast the temperature at 06:00 based on not one X(parameter) but 10 Xs (parameters) from 00:00-05:00(sliding window of 5) how should I tweak my df_to_X_y function?
Idk why but I got an error "No file or directory found at model1/" when trying to save the best model
This video helped me greatly.
Amazing explanation.
Thank you.
Super glad to hear it :)
excellent video and covered all my doubts, thanks Greg...
Hello, and thanks for the fantastic video. If I may ask, what would you recommend if I look for a specific kind of time series forecasting in which I don't want to give the model the real values x_test? I am pretending to use the model's prediction to create an x_test_model and then use the model on this set to achieve a y_test_model that I can compare with the real y_test.
Nice tutorial! I'm struggling to conceptually understand what happens when you call predict. I understand the training portion, but how do I get values for say 30 days out? It seems I need to feed it 30 values that I already know the answer for? What is it actually returning when you call predict?!
For example, if I have a dataset of 100 values with a look back/ window of 5. I want to predict values 100 to 105. I train the model on 95 values and call model.predict on the remaining 5, do I get the predictions for 95-100 or 100-105?
Can you giv e an example of multivariate forecasting?Also if i have temperature datas in two different files and i want to train the LSTM on both the datasets, how will i include that ? Example: first dataset is say temperature on antartica (-10 to 5 degree celsius) and other file has temperature of south africa(30-50 degree celsius). I cant just simply append both data frames. So do i need to train it twice on both datasets?
hi Greg, what if my target variable is not normally distributed but positively skewed? what should I do in such case? and also, is it recommended to datetime function as cos and sin of hour, weeks n all to provide it in the x variable?
really awesome video, learned a lot about LSTMs. thank you Greg!
Great to hear!!
@@GregHogg do you have anything to predict future stock price? :)
@@moondevonyt Yes check my recent Videos :)
@@GregHogg youre the man, thanks!
When I try to predict a temperature, the imput should be the prior 5 temp records, right? But the model accept only one value in order to give a prediction, how is that possible?
Great video! I have a question, do you have to do normalization in time series forecasting, I see it was not presented in the video.Thanks！
Don't you need to scale the data using MinMaxScale or StandardScale between 0 and 1 ? And then once you've done gone thru' the LSTM training, you scale the data back to original ?
Thanks a bunch! Do you have any advice on how to create confidence intervals for predictions?
I'd like to know that too =)
Hello Greg, first of all, thank you very much for the very lovely and exciting video. I do realize that in the test part, you are not updating the window with the new predictions. So mainly, you are doing predictions for every five timesteps, but then in the window, you are using the actual test values to predict the next value. So my question is did you try in this way as well, and if so, how was your results? Thanks again!
I have the same question!!
how would the df_to_X_y function change if instead of only having a date and temperature column you also have a location column because now X would have the format (n_samples, window_size, 2)
Amazing explanation. Next show how to apply attention mechanism/transformers to time series.
i have a data set of about 4500 rows with daily prices and its dates, I am always getting a loss of around 1.2, and after that it keeps oscillating around this number . How do i resolve this issue ?
Great VDO mate!!! Keep it up!!
Thanks so much!
instead of creating a window function, you can just use the pandas "shift" function to create columns that are shifted by 1 each, using a for loop
Thanks for the video! Exactly what I needed
Really glad to hear that!
thank you for the very good video ...but isnt working with np Arrays exhust the computer memory during fitting ? can we find another way to prepare the data before training... i am working on a multivariate multistep LSTM model ... i have 90 features to predict ine target value ... with many zeros in the target and also in the input features... it is very hallanging specially for multi step forcasting... any advice!?
Wow...the greatest tutorial ever...
Very well explained! Thanks for sharing
Hello, amazing video, may I ask something?
So lets say we were to predict a single value, we don't need to use the time somehow or not? Does the model kinda learns to which time the data sequence is relationed? like: "this values n sequence... must be winter so the prediction is: x!" right???
Hi, great video! Is it possible to run the data in a sliding window way and update the model as it runs? Thank you
Hi! I have a question with respect to the 20:07 part where you define the 'linear' function, in case I'm using a variable that doesn't take negatives like the price of an action which one would you recommend? Thank you :)
Thanks for the video, I'm wondering how to use the first model to predict future values.
Would this approach work if you wanted to predict the next say 24 or 96 values, instead of just the next 1?
Can I use the R2 value to measure the accuracy of the LSTM model in time series prediction?
Great video! Question: I don’t understand the difference between validation data and test data. You say that test data has not seen the model before, but so did the validation set? I thought we only divide into train and test data
Hi there! Found this really helpful, but I've managed to confuse myself about the data windowing and the test set. Since every target in y_test (except the last one) is somewhere in X_test, how are we not assuming our solution? I had a look at your stock price notebook and there's the bit about recursive predictions, but you don't have that here. Any clarification?
Hi Greg,
Does your model predict different numbers each time you run it? Could you make it give range instead of one number? Kind of Monte Carlo simulation.
great, please how should we prepare our data if we want to predict the temperature in different cities, (redundant dates)??
Nice Video ! Just think about mention François Chollet and Keras for their tutorial
If I wanted to forecast 18-24 months into the future, do you have a video on this for a supply chain context? Thanks so much Greg, this video was really well put together for understanding the LSTM Time Series. Also, do you have a video showing how you started off with a baseline model (i.e., Linear Regression or Seasonal Moving Average), then iterated to better models with regression metrics shown?
Thank you! I probably do, although I don't remember sorry
hello
i'm a beginner of machine learning
is it important that the index on the dataframe must use a timestamp before being put into the train model?
is there any effect if the index still uses auto increment?
sorry for bad english
It doesn't do anything once it reaches the NumPy stage. It just allows the use of pandas functions to manipulate the dataframe in pandas
@@GregHogg Thank you so much for your attention
I have 1 more question
how to determine a good number of hidden layers and good activation function ?
I have tried the lstm model with different hidden layer configurations and activation functions several times with my data, but the training results are always not good
Thank you.
What shall be modified in the code in case the X is not a single value (like temperature), but many features vector and y is a single value?
I will cover this in the future. But essentially, we'd probably want the single items in each list (the third dimension) to be more values. Then you'd have to change the input shape of the model accordingly
How is it possible that my model performs well when training and predicting with a dataset with many examples, but when I use only one example (to make the prediction) it performs really bad? Is it a problem of how the data is structured or do I have to make adjustments to the model?
Bro you sound like Robert Greene and you are just as awesome as he is or even more awesome !
Why do you use rmse over r2 score? I would like to know thanks!
This video at 13.26 shows a time series function, but it is single-step forecasting only... can you show how to modify it for multistep. I have tried to add multiple labels in the loop but it is not working with LSTM. Thanks, Greg for this video also.
Hi! i used the same code but shape of X,y is not right. they give(1,20,1),(1,20) they are not giving correct n_samples, total number of samples are 365557, and after minus 20 as window size it coulde be 365537.where is the problem
does the lstm train only on 1 input ? i see this ((70086, 5, 1), for X.shape
The df_to_X_y function doesn't actually work for the df, just for the temp array. Confusing naming
Great stuff Greg! Thanks to this vid I finally got the input dimensions of the LSTM right. One quick question: why is the extra `Dense(8, "relu")` layer necessary, and not just the final Dense(1)?
Thank you, and great to hear! I forget the final model but most likely this is for extra complexity
@@GregHogg Hey so, is it okay to put fully connected layer after LSTM Layer? Thank you
Hi! Excelent video! How you can apply the model with new data, i mean when you have new variables without the temperature values. You would like to predict the new (future) temperature values? I hope you can help me with this. Thanks!
This also seems only helpful if you want to predict a singular value in the future. What if you wanted a very large number of predictions?
Why doesn't it show the same performance when I put P bar instead of temperature? The temperature values are small and have negative values, so I don't understand, did you make an arrangement?
in df_to_X_y() shouldn't row= [[a] for a in df_as_np[i:i+window_size]] and not i+5? I'm asking because i am new to python and i don't fully understand it yet but if feel to generalize this funtion it should be i+window_size and not i+5
Oh, sorry, ignore this. Found the answer in @mandem2735 comment from a year ago. Many thanks for the video. works great for temperature dataset. Struggling with stock prices but getting there...
Hi Gregg , when running the predictions :train_results = pd.DataFrame(data={'Train Predictions':train_predictions, 'Actuals':y_train1}) i have noticed that you've used data but no prior reference could you state why?
Great tutorial, thanks!!🎉
Do you consider the prediction of first predicted hour in second input to do further prediction?
Many thanks for this video. Can we use this method for forecasting univariate time series that involve outliers? If so, how to treat the outliers first?
Great Video Gregg!
Thanks Akash!
Great video. Do you know where I can find more information on how to determine the number of layers when performing the model.add() commands. For example, I believe you used 16 for your LSTM.
Great! What should I do to improve the accuracy on my trained LSTM model?
Dropout, regularization, adjust the model in any way you see fit!
How would we do if we did not know the values of x that correspond to y that we want to forecast?
awesome tutorial bro!!!,any thanks for this video, to confirm, could you say that the 5 data above predict 1 value?
I have tried creating a similar forecasting model but have to normalise the data, whereas you did not in your code. Is there another reason I'm not understanding that allowed your model to work with the real values?
Great explanation, it really helped me , but i have one question, how can we forecast on the future dates? Knowing that we dont have the X_future?
Thank you! We can recursively feed our output back in as input to the model
@@GregHoggHi, can you give me a hint on how to do this? Please help...
Hello congrats amazing video!!
What should we do differently to consider more variables as X?
Hi Greg, that was a lovely and informative presentation. but you are using the training data again in prediction. so why did we split the data into testing, training, and validation?
Which data you as been taken in this video can you please show us or update a dataset your
Hey, can you also forecast time series data like that but using random forest??
Thank you so much. It definitely was much needed.😇😇
You're very welcome and happy it helped!
Hello! How can I make further prediction using this code? I mean, we predicted the first 1-60000 values, but how to predict those that were not in the original dataset? Thank you for a tutorial!
You can use the model to predict the next value, and then use that prediction as input to the model to predict the following value and so on :)
@@GregHogg Thanks for the answer! I did this, and the prediction quickly degenerated into a straight line (after like 15 iterations)... Is this normal?
So... the predictions it makes are within what timeframe? By hours? By days? I put in data by month and for example the actual value is 200, I got the prediction is 1.2 or something like this. I am going to re-do it with daily data instead, but is this 1.2 supposed to be an hour, a day?
Okay, I changed learning rate to .1 and got way better values but they are all the same as if I am doing a simple exponential smoothing 😂😂 The data I'm working with must be horrible.
Amazing! How can i use this for raster data? Thank you!
Amazing! Congrats for great explanation, my friend. I have a question. If I wanted to forecast future temperature regarding date in the future, how could I do that? Regards, Lucas from Brazil.
I tried using this model for forcasting energy demand using 2 excel files, but the predictions are way off , can anyone help me with it?
Great video Greg. Thank you for sharing.
I have a question. Is it better to perform data normalization/standardization before we start the process? Thank you
Thank you. As long as you've preprocessed inputs in some form, you probably can't go too wrong
Thank you so much, I followed the tutorial and got my first LSTM network working. I am however having trouble with generating out of sample predictions, I feel like I have tried everything and just can’t crack it. I am using the last batch of the test set to predict the first out of sample data point, but it ends in an error. I feel like it’s a dimension thing. Any help, anyone?
Hello, thank you so much for this video, after doing
train_predictions = model1.predict(X_train1).flatten()
train_results = pd.DataFrame(data={'Train Predictions':train_predictions, 'Actuals':y_train1})
train_results
i got this error:
ValueError: All arrays must be of the same length
What should I do ?
Why are we not shuffling the train, val, and test splits? Is not it necessary in this forecasting task?
Gets kinda messy with the sequential ordering of time series data
What about packing (X,y) and doing shuffling and do train, val split.
Or is it okay/valid to shuffle like this for this type of time series forecasting?
Great video! Thanks for the great content!
very well explained video, but in terms of the results you show, it looks like your predictions are lagging behind the actual values. Meaning the model doesn't generalize well enough and relay in its prediction on the last known value. It's seen there the trend of the blue line(prediction) always one step behind the trend of the orange line(actual values).
Would it be more beneficial, when creating the train and test sets, to shuffle the data rather than taking it chronologically?
This is a time series prediction model, not a 2d prediction model. Shuffling doesn't make sense here since the results are based on previous data
Just found the reason why…but still don’t understand why…
Basically I was using a one column df…and that’s gave me this extra one.
When I used a multiple column df from which i extracted the interesting column I obtained the good shape for X and y
Hi!! Thank you for the content!!!
When I run the df to x y fonction on my data
I have this shape
(431,5,1,1) , (431,1)
My data is one column of data and index as date time. No idea we here this extra 1 is coming from…
pass the column only to the function. E.g df_to_X_y(df['y'], WINDOW_SIZE)
how can we actually test the model? without the test values
hello, any chance you have done a LSTM model for CMAPPS dataset? or anyone in the comments? would love to consult...
I realized it was the previous statement to create a problem:
model1.add(InputLayer((5, 1)))
I cancelled one parenthesis believing it was redundant and I wrote model1.add(InputLayer(5, 1))
but this causes a crash to the following line.
At the moment it is not exactly clear to me the reason. I have to study about.
In any case I can go on now.
Sorry for the inconvenience.