Anomaly Detection using Isolation Forest - Time Series

#datascience #timeseries #isolationforest
Anomaly detection using Facebook Prophet - • Anomaly Detection mode...
Isolation forest is an unsupervised learning algorithm for anomaly detection that works on the principle of isolating anomalies
Isolation Forest builds an ensemble of “Isolation Trees for the data set, and anomalies are the points that have shorter average path length
The Isolation Forest ‘isolates’ observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature.

[@elizavetapudim9677]

Thank you for this tutorial! I've found a lot of useful tricks in code. But one suggestion is it would be great to make code more reproducible, maybe by adding some random_state.

[@saravanannatarajan6515]

Thanks Nice explanation, you are well deserved for more subscribers.

[@Abbekej]

Thank you, what a lifesaver!!

[@bastianabaleiv]

Good material, thank you!

[@murselkoyuncu]

Hello, I am giving my bachelors degree final project, and I am using isolation Forrest somewhere why we are choosing negative ones as anomalies I mean the anomaly score which we have close to "1" ? Should be anomalies right I have read the original paper of isolation Forrest the anomaly score is calculated as s(x, n) = 2^-(E(h(x))/c(n)) which is always positive. But why we have negative anomalous scores ? The less it is, the more anomalous it is, right ?

[@guidohardinghaus7649]

Nice! When is it possible get more information, how to handle with "local outliers" and "context outlier"?

[@AIEngineeringLife]

You can check this one which has both captured but using facebook prophet- ua-cam.com/video/0wfOOl5XtcU/v-deo.html
Isolation forest will update soon

[@guidohardinghaus7649]

@@AIEngineeringLife Thanks for that, but it is possible with "Isolation Forest" to use it for "local-" and "contextoutlier"? This would be very interesting for me.

[@mohammadqureshi4114]

what is the histogram taking on Y-Axis, if the score values are on X-Axis?

[@MrPetso]

When running a similar model I get the error 'invalid type promotion'. As all except for datetime are integers, the mistake must be here. How can I solve this? Perhaps split it in separate columns for year, month, day, hour?

[@AIEngineeringLife]

Petter.. Isolation forest does not understand time element. Yes one thing you can do is extract hour or weekday or other elements from data based on seasonality in your data and then feed it to model. I have a new video coming next week as a continuation of this which shows similar stuff

[@subramanialatentview1542]

Do we get the github code?

[@aradhnasingh2157]

Sir, if we are working on real time anomaly detection does isolation forest be useful for that? Should I go with the time series smoothening techniques like Kalman filtering for real time instead? I am working on AWS enviornment.

[@AIEngineeringLife]

Yes Kalman filter is a good option and another one is autoencoders for real time detection. You can check autoencoder video in my channel

[@Ohmic96]

I can't see the lines on my plotly graph but if I hover it shows values.

[@AIEngineeringLife]

Maxwell.. maybe some jupyter or browser issue. Did u try clearing cache and try it. Ideally it must not be a problem but can you try above and check

[@robertocorti4859]

Thank you for this video, it was very helpful!
In order to improve this model, could it be useful to encode time information like the day of the week and/or the hour of the day and use them as additional features? If yes, how can I do it?

[@AIEngineeringLife]

Yes. You can take average of say previous weekdays and weekends in multiple buckets like last 3 weeks, 7 weeks and 21 weeks weekday average and so on. You can use rolling and other python functions that I have covered as seperate video

[@robertocorti4859]

​@@AIEngineeringLife Given a point x_t you would consider for it the average number of previous weekdays (or weekends) in a time window of {3 , 7 , 21,....} weeks. Is that what do you mean?

[@AIEngineeringLife]

@@robertocorti4859 yes that right and duration can be anything. Depends on business scenario like seasonality and all

[@lramakr77]

This is very helpful. Have you released the follow-up video where you cover Isolation Forest + Feature engineering? Thanks

[@AIEngineeringLife]

Nope.. Not yet done.. will do it in couple of month

[@ChakoTR]

@@AIEngineeringLife we're still waiting for that video :D

[@iswariyar9169]

Thanks for this video
Have u uploaded follow up video of isolation forest

[@AIEngineeringLife]

Nope not yet but I have done one more using Autoencoders. Have to get isolation forest done yet

[@VLM234]

@AIEngineering like isolation forest, can we apply fb prophet for anomaly detection?? It seems like, doing multivariate ts prediction using prophet and rest of the steps( handling anomaly) are same as in univariate using prophet??

[@AIEngineeringLife]

Yes we can.. Have you seen my fb prophet anomaly detection video here - ua-cam.com/video/0wfOOl5XtcU/v-deo.html

[@VLM234]

@@AIEngineeringLife yes, you have done for univariate only over there.

[@VLM234]

@@AIEngineeringLife which one is supposed to be most probably best(I know it's data subjective) out of isolation forest, prophet, pycaret, autoencoder or there is any else?

[@AIEngineeringLife]

@@VLM234 .. Isolation forest is time unaware so you need to engineer features to make it time aware. pycaret uses isolation forest and other models in it. fb prophet is time aware but if features are high better to use isolation forest and engineer time aware features

[@akhileshsingh569]

Can isolation forest works with categorical data also

[@AIEngineeringLife]

Yes encode it if these variables are significant to determine anomalies

[@anuragsharma6519]

You didn't explain Isolation forest.

[@AIEngineeringLife]

Anurag.. I will do it in my follow up video where I cover Isolation Forest + Feature engineering

[@alokpatel4527]

how can I update it with streaming data?

[@AIEngineeringLife]

You can just save the model as pickle file and add it to streaming pipeline