180 - LSTM Autoencoder for anomaly detection
Вставка
- Опубліковано 2 гру 2020
- LSTM encoder - decoder network for anomaly detection.
Just look at the reconstruction error (MAE) of the autoencoder, define a threshold value for the error and tag any data above the threshold as anomaly.
Code generated in the video can be downloaded from here:
github.com/bnsreenu/python_fo... - Наука та технологія
Your content is awesome, it's really helping me to understand more concepts about ML because you don't only stand with the theory but you moving through the practice (that's pure gold for me). Thanks for sharing all of those knowledge with us !
Thank you. I could follow your story even though I am not a data scientist. You have unique skills of explaining something complex in simple words with good enough details.
Thanks for these videos, I really love learning something from your codes and videos.
Thank you for putting this out there. I was putting off building an LSTM based auto-encoder, but you broke it down very well and pushed me to get it done.
it is very simple with Keras
Wow this was fantastic! I didn't even know what an autoencoder was before watching
How do I modify the above example to take in 3 inputs I.e. multivariate instead of univariate? I am new to this and would appreciate your great help in this.
Very clear explanation, fantastic video, thank you very much.
This was greatly educational, and surprisingly in-depth and easy to digest. Thank you a lot and good luck with your channel :)
You're very welcome!
Your explanation is simple but clear. Thanks for you effort.
Glad it was helpful!
Hi Sreeni, Thanks for the great video. But I just curious to know that after you perform Standard Scaler transformation, how the type of train & test was in pandas data frame. It will be converted to numpy array, once you have done any transformation.
The video is interesting. I have a doubt.
1. Given the network is used to train a network where the input and output are the same, why are trainX and trainY given in the fit command.
Shouldn't it be trainX, trainX.
Same doubt here. Thanks.
I share this doubt. With model.fit(trainX, trainY), nothing works like in the video from that point. With model.fit(trainX, trainX), we are really close to the results of the video.
Yes, it should be trainX, trainX. I tried it and results improved
I'm highly inspired by your thoughts and from your tutorials. You're the best UA-camr for deep learning and medical image processing. Sir there is most promising task done by deep generative models (AAE)is generating novel drug molecules trained from existing datasets like Moses and zinc. And research contest shows that it's in the forefront in terms of application of deep learning in healthcare infact this is biggest research topic of AI in healthcare in 2020. Please make tutorial on that as well I'm waiting sir
In summary you are recommending something like VAE for generating new molecules?
@@DigitalSreeni yes sir
First of all, thank you so much sir. I have a question on how to choose the scaler? let me put it on other words, If I have a dataset but Idk which scaler should I choose, so on what basis should I choose a scaler. What is the difference between Standard scaler and minmax scaler? why only these two scalers, any alternative that can be used for anomaly detection?
Thank you for this valuable video!
Is it necessary to perform a standardization (via StandardScaler methods) as there is only one feature ?
Thanks for your good tutorials and eloquence, can we also use this architecture to diagnose video data anomalies?
Well done! Thanks for this nice video! Greetings from Germany
This is extremely helpful. Thank you very much.
Thanks alot for these videos. Just a question, should trainMAE not be calculated with trainY instead of trainX? Im a bit confused.
I think not. Because the trainMAE is based on te difference between trainX and the trainX'(value predicted by autoencoder).
If the LSTM is reconstructing the same input sequence, why do you create an X and Y? Shouldn't the input and output be both the "X"?
What's the answer?
They are not the same because the output y will be slightly different from the input X due to the encoding and decoding process errors.
But in a ideal case you will get X= Y
Y predict future? but in what time frame, 1 day or 30 days?
Hi, thank you for the teachings and videos.
Thank you . Very clear explanation !!
Thanks Sir for the videos. Do you have a tutorial on how we can use plotly that will give us at what events each anomaly corresponds ?
Thanks in advance
Hi, thanks for your video. Please, is there a way I can pull out the encoder compressed data with the original number of rows for supervised learning? I have actually tried it and the size I got was just the sample size instead of the size of the original number of rows.
The best video i have ever seen, great
Hello, the TimeDistributed wrapper is not needed for Dense layers, but I guess making it explicit makes the tutorial more understandable. Nice video
Thanks for the info!
I have a question, if I am working on a multivariate problems where i have 7 features in my data and I am using for eg. 6 features to predict 1 feature, how should I modify the code to output 1 feature since my trainX.shape[2] contains 6 features instead of 1?
Thanks for sharing Sreeni. I wanted to point out that the LSTM "units" argument is the number of hidden layer in the LSTM cell. Its not the number of LSTM cells in that particular layer (comments at 4:00 mins).
Thanks for pointing out Puneet. The terminology for LSTM is defined in a confusing way. Here it refers to horizontal arrays of LSTM layers (units).
@@DigitalSreeni On a separate note ... correct me if I am wrong - the inputs and outputs for autoencoder model should be the same right ... model.fit(input, Output ...), input and output should be same for autoencoders.
If I understand correctly, autoencoder are not able to detect reocurring patterns. If this anomalous drop would be something reocurring, is there a ways to take this into account?
Thank you very much for your content!!
I'm not sure if it is a mistake or I misunderstood something, but I noticed that after building the encoder-decoder block you are training the model as if you were to predict the labels in this line model.fit( trainX , trainY ) but afterwards you're measuring the MAE between the original data and the reconstruction in this line np.mean( np.abs( trainPredict - trainX ) ,axis=1) however this is not the error between the reconstruction and the original data but rather the error between the original data and the predicted label, isn't it? Shoudn't you measure the MAE between the original data and the TimeDistributed layer which has the same shape as the original input data?
if I wanted to use the lstm autoencoder having in input a dataset containing some text and not a temporal sequence, can it be done?
for example with a dataset containing fake news
Sir I have Timeseries dataset in which time and vibration accelerationd have been recorded. So I have to classify the faults of tool on the basis of that dataset on the basis of LSTM. so how to use it.
Thanks sir just video gave the starting point which was needed to work on (time series anomaly detection)
Great 👍
How to use LSTM (or 1D CNN) to detect contextual anomalies in timeseries?
Why did we use time distributed dense layer? why can't we use a normal dense layer, any specific reason?
Thank you for kindly sharing this.
My pleasure!
Thank you for sharing !
What i can't understand here, is the part where we create the anoamly_df.
we know that for each sequence of 30 observations, we have a single MAE.
so how can i detecte which observation of these 30 is the anomaly within a sequence ?
I think, for each of the 30 observations you have one MAE, since MAE is calculated based on 2 values: observed value and predicted value. What was predicted by the autoencoder was a vector of 30 values, trying to rebuild the observed values.
Hope I made myself understandable
Im waiting for your video on Variational Autoencoder in which you tell how to put classes on Mona Lisa, Image Super Resolution and about Style Transfer
Hi, You said that you use an undercomplete autoencoder which imply that your encoder compress the input i.e the number of features in the output of the encoder should be smaller than the number
of features on input which is not the case on your model. Can you explain why you use a latent space with dimension higher than the input?
Thank you in advance
In my example, the first LSTM layer generates 128 features and we encode it to 64, which is smaller than 128 features. Then we decode it back to 128. Therefore, the short autoencoder we have goes from 128-64-64-128. You can make it bigger /deeper if you want. Autoencoder does not necessarily mean the encoded vector is smaller than the input, it sometimes happens to be smaller than input (especially for images). In summary, autoencoder takes features from large dimension to smaller dimension and reconstructs them back.
Extremely helpul. Thanks very much
Glad it helped!
Thank you for a very informative video.
I have one question (anyone can answer it)
What advantage does autoencoders give for anomaly detection over classical ML algorithms?
It can work with unsupervised data & for anomaly, it is not needed to show the model what anomaly actually looks like beforehand.
I have a multivariate dataset with 86 dimensions, instead of 1 like in the video.
How do I compute the MAE in this case?
Can you explain how to do the same with supervised anomaly detection with labeled multivariate dataset using LSTM
can i use lstm with video analysis to detect anomaly ?
how to do the same idea of anomaly detection but not for time-series data, for example, having clients in hospital and checking their health tests?
How can we compare different models how it went when there are no labaled anomalies?
that's a great video sir. although i got two things to say one is sir , it we be a great pleasure to vide only on time distributes and the other thing sir a query .
here we set return sequences as false and then used a repeat vector so that we can stack a LSTM layers again .
but cant we just use repeatvector as True in first layer so that we can eliminate that repeatvector layer .
the thing using repeat vector is it a thing particular to autoencoder using LSTM or it is just an experimental thing tried for for better accuracy, i mean we can also try setting return sequences as true and remove repeat vector layer?
I have a doubt here in autoencoders that output is also x then here why did you trained model with trainx and trainy. instead of train
x and train x
excellent video
So inspiring! Well done. How do we get the codes please?
Thanks! But I don't understand why the model is trained to predict y, while the anomaly score is given based on MSE between y_pred and X. Shouldn't it be between y_pred and y?
I do have a similar question, I don't understand why he's training the model to predict trainY and then measuring the anomaly score between the original trainX and the predicted label instead of the reconstructed data. Maybe I misunderstood something
Can I use this method for clustering?
Sir could you please provide a video on LSTM Variational Autoencoder for multivariate time series.
I have got error in the end
y = scaler.inverse_transform(test[timesteps:].Open),
Expected 2D array, got 1D array instead
I also tried to reshape but still got a same error
so could you help me with this
Nice video... Pl tell me how to find anomalies in multivariate time series
Subs Added, thanks for the wonderful video.
Thank you too
Thank you, it looks like GE got hit hard by the 2008-2009 economic crash and maybe by Covid-19 in 2020...
Can you please make a video on hybrid Autoencoder that uses LSTM or GRU and CNN layers ?
Need to think of an application, so far I haven't explored it for any of my applications.
When I'm searching for the same data from the same time interval, I get values approximately 10 times higher than in that video. How is that possible?
For example, in the Video at 9:04 we see the table of yahoo. When I'm searching for the Values Oct 29, 2020 I find values around 60.00, in the video however I see 7.65.
Currency is in USD as in the video, what is happening?
In 2021 (after this video was made), GE decided that every 8 shares that investors own will be turned into one share. You have to divide the 'close' feature by 8.
How is it possible that timestep is 30 and the LSTM layer is 128. Shouldn't it be less than timestep to actually encode it?
I am having trouble plotting testPredict and testX. I want to see the predicted curve.
Sir y objective lens of microscope are smaller in length y can't make as size of slide that can use to focus whole slide.
Plzzzz help regard to thise like I need 100x objective lens of larger length of like slides
This is basic optics question. Why do we have different camera lenses and why not have single lens that covers a wide range? Because you compromise on quality due to many factors, optics and also chip electronics.
Firstly thanks. My question is that when input is 30*1 means 30 then how can be output 128 while in autoencoder we compress data then decode for example 30 to 15 to 10 then decode
i have problem with plotting anomalies (last task), How do I solve ValueError: Expected 2D array, got 1D array instead?
Indeed there are some errors. This should be :
#Plot anomalies
sns.lineplot(x=anomaly_df['Date'], y=scaler.inverse_transform(anomaly_df[['Close']]).flatten())
sns.scatterplot(x=anomalies['Date'], y=scaler.inverse_transform(anomalies[['Close']]).flatten(), color='r')
When I see DigitalSreeni I know I'm in good hands
Thanks for the trust. Now I am under pressure to live up to your expectations :)
how we can make differentiate between Outlier and Anomaly in this problem
As I understood anomaly detection is simply outliers detections so outliers and anomaly are the same
thank you for that 😍😍😍😍
You're welcome 😊
your video is always mean to me, besides VAE and Autoencoder could you make videos to explain about SOMs and Boltzmann (unsupervised deep learning)?
Sure, will add to my list.
thank you
Im also waiting for the video in which you will cover different types of GANs for example Style GAN, Conditional GAN or Cycle GAN.
On my list for a long time. Thanks for suggesting.
why does your script did not work in my colab environment? train loss does not reduced down to 0.3 which is much bigger value than your video. for me. every value of "trainPredict" is near -0.5 whereas trainY is distributed -1~4.
It is the same for me. In my Colab environment, the training loss is 0.4 and the validation loss is 2.4 (even after 30 epochs).Nothing in common with the 0.03 and 0.07 of the video.Why?
In all other autoencoder videos you've done .fit(x,x). Why are you doing .fit(x,y) here?
I am also curious about this
I believe it should be model.fit(trainX, trainX) instead of model.fit(trainX, trainY)
Isn’t this trained on next value regression and not reconstruction? Seems like you just mix the architectures and do next value prediction and then evaluate based on the regression error
hey hope you doing good . do you mind answering my query.
here in the first layer turn sequences is set to off and repeat vector is used to stack another LSTM layer .
is this method a standard procedure for autoencoder with LSTM of we can also try without repeatvector by setting return sequences as as true in first layer..
and do you know any tutorial on time distributed layer?
@@maaleem90 Yes, I agree. He has mixed forecast and reconstruction. This looks wrong.
@@rajarams3722 thanks brother you too got that thing. That means we are really learning it
@@rajarams3722 maaleem08 is the user name
@@rajarams3722 can we please connect over other platform so that we can have some talk coz I don't have any one in this field
Hi sir
How one feature can be 128 features... I couldn't understand here? (Input -LSTM1) @DigitalSreeni
@DigitalSreeni
@DigitalSreeni
@DigitalSreeni
Thank you very much for the tutorial.
I have a problem with sns.lineplot (row 142). I always get below error. How can I fix it?
ValueError: Expected 2D array, got 1D array instead:
array=[0.57032452 0.37515913 0.19478522 ... 0.32379982 1.23183246 0.9894165 ].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
Hello, did you manage to solve this problem? The same occurs on my own dataset
Have you found out solution?
@@gnn816
Malav Vibhakar
0 seconds ago
Have you found out solution?
@@malavvibhakar9001 I did not unfortunately. I tried out some things from stackoverflow but did not find a way.
Indeed there are some errors. This should be :
# Plot anomalies
sns.lineplot(x=anomaly_df['Date'], y=scaler.inverse_transform(anomaly_df[['Close']]).flatten())
sns.scatterplot(x=anomalies['Date'], y=scaler.inverse_transform(anomalies[['Close']]).flatten(), color='r')