Hi there! If you want to stay up to date with the latest machine learning and big data analysis tutorials please subscribe here: ua-cam.com/users/decisionforest Also drop your ideas for future videos, let us know what topics you're interested in! 👇🏻
Great!!. Easy to understand. After you get the anomalies, how you are going to identify from the dataset which are labeled as 0. Please add that code as well.
That was a concise and clear tutorial! It would have been lovely to see some metrics on a test data set and also a visiaul on what those outliers looked like :)
Hi, really enjoyed this tutorial. But, may I know if there are any way to output some sort of score for the accuracy of the model to use it for hyperparameter tuning? Much appreciated thanks.
thank you for the video, very insightful. my question pertains to the contamination parameter you set -- what if we don't have any expectation around what this would be? I'm working on a very high dimensional multi class classification problem and am uncertain about how I'd go about setting this parameter.
Thank you Brent, glad it was helpful. Regarding the contamination, you set that based on your expected percentage of anomalies within that dataset. Assuming you have knowledge of expected anomaly rates for that dataset of course. If you have no such information I would suggest you go for a lower value in the range .001 to .01 and analyse the results. And tweak it based on the relevancy of the findings.
Thank you for the helpful video. My question is that the method expects that some of the data points are anomalies. What if I want to check the data has anomalies or not? I mean I don't want just some percentage of the data to be necessarily classified as anomaly.
Thanks for the wonderful video. Could you please do one on supervised anomaly detection perhaps using one-class SVM with gridSearchCV or RamdomSearchCV
@DecisionForest Thanks a ton for your valuable video. Your Jupyter notebook link is not working anymore. Can you please update with working source url. That would be highly appreciated 🙏
Very good point, and normally you don’t even have the target of course otherwise you wouldn’t need this in the first place. Since this dataset had it I kept it to see how the Isolation Forest performs and how many it identifies correctly, even if partially helped (partially helped because we know that’s the target, the algorithm sees it just as another binary feature). The isfraud feature is just one feature given to an unsupervised learning algorithm therefore the algorithm doesn’t output anomalies solely on this. But it does influence the outcome slightly and in a real world scenario you’d not use the fraud features of course, you wouldn’t have it anyway.
Thank you for the detailed video! You use the terms outliers and anomalies interchangeably but they're different. Can you comment on why it's used interchangeably in this video? Is Isolation Forest used for outlier detection or anomaly detection? Hope this can be answered!
Firstly thanks for the tutorial, it was very informative. My question is related to the visualization of results. Is there a way we can can effectively visualize the results of multidimensional data?
Thanks mate! From my current knowledge the Isolation Forest cannot have feature importance implemented due to the random nature of the splits. I have another video that might be helpful with this: ua-cam.com/video/i42UzP3mT58/v-deo.html. Let me know if that helps.
Hello, I am writing the thesis of the university on this topic, but I don't find much on the internet, could you tell me how do I get a graph printed that underlines the isolation of anomalies in phyton?
Well you can plot the normal values vs anomalies per each categorical feature. This would be a nice way to show how different values have more of a predictive capability. Something like here: ua-cam.com/video/i42UzP3mT58/v-deo.html . Hope this helps.
Sir, i have doubt. Suppose we have credit card data.. First we label those data with anamoly detection... Then we go for preprocessing and model building like random forest ,am i right.. So anamoly detection is used to label the outliers correctly.. This is what i understood from this video... Is my understanding is correct sir??
Good question and hope I understood it correctly. We only have one model, the Isolation Forest algorithm that uses a Random Forest approach to isolate the records from one another. And it considers the records with the shorter paths as anomalies as it took less time to isolate them. So we do the label encoding prior to building the isolation forest (not a random forest), that leads to getting the possible outliers in the dataset.
Hey there, you have some mistakes in your video. You said that IForest can work with categorical attributes but that's wrong, it only deals with numerical. The way you handled missing values is wrong. Contrary to your other answers, label encoding is actually very wrong. By turning categorical attributes to numeric - the way you did it - it simply does not work for isolation forest. When it performs the random splits, the frequency of these classes simply introduces bias as the algorithm is forced to isolate the most common classes first. This is wrong and against the underlying principles of IForest as the most frequent classes in theory point to inliers, not outliers, and these should be isolated deeper into the trees. This also makes your classes ordinal which is also wrong. You should have used one hot encoding. Finally, I'm not sure if you understand what the contamination parameter means. If you already have labelled instances, you can simply use the percentage of these as your contamination value. Training the algorithm with the labels is simply your biggest mistake. If you wanted to compare your predictions to the actual anomalies, you could just ignore that column for the training
@@DecisionForest I actually have started watching your channel recently. In the past few days, you have a lot of videos regularly on your channel. I didn't know it took 15 months haha great work
I haven’t as I believe that if the content is useful people will eventually find it 😊 And if they like it then it sets up the connection better in the long term.
Hiii !! I have a question and i can't find anything online. Can an isolation forest works well with a massive number of 0 values and a lot of columns? Great video anyway! Thanks, Veronica
Hi Veronica, thanks for the support! To answer your question it will work but not great as there isn’t much information in those features unfortunately. Depends of course on the size of the dataset as well.
PCA summarizes the variance in the dataset so it wouldn’t negatively affect the results but it wouldn’t help either. The best best is to try with the full range of columns and then see if you obtain any relevant insights.
What do you mean, is the explanation similar to how you present it as well? The picture is just a simple representation of a decision tree in the isolation forest algorithm
I think it's not correct to use label encoder in categorical data. More details: datascience.stackexchange.com/questions/36006/categorical-data-for-sklearns-isolation-forrest
Hi Carlos, label encoding encodes the target labels with numeric value from 0 to n-1 and it's required for the Isolation Forest algorithm. As I agree that for many algorithms the ordinal features that we get impose false orders of magnitude, tree-based algorithms aren't influenced by this. The way the Isolation Forest algorithm works is that it isolates observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature. So the Isolation Forest doesn't care about the orders of magnitude.
@@DecisionForest Thanks for the explanation I'm gonna read more about the theory behind how isolation forest really works to understand your comment properly, it's my first time using this algorithm
How can you give the class feature(isFraud) as input to the model?, the model will tell you outliers solely based on this target feature. Your logic is completely wrong.
Very good question, and normally you don’t even have the target of course otherwise you wouldn’t need this in the first place. Since this dataset had it I kept it to see how the Isolation Forest performs and how many it identifies correctly, even if partially helped (partially helped because we know that’s the target, the algorithm sees it just as another binary feature). The isfraud feature is just one feature given to an unsupervised learning algorithm therefore the algorithm doesn’t output anomalies solely on this. But it does influence the outcome slightly and in a real world scenario you’d not use the fraud features of course, you wouldn’t have it anyway.
@@DecisionForest sorry I didn't understand what you mean by based on data type, I was trying to use this algorithm and trying to apply to each column so that I can get 0's and 1's column for each column, can you suggest the best way to detect column wise outliers in any dataset pls?
@@gtownmunda no problem, well fr continuous variables you can simply use a boxplot or histograms. Categorical variables don't have outliers. So for every column you can just explore the data type and check for anomalies with statistical methods.
@@DecisionForest thank you for your response, one last question! What is the issue with using isolation forest? I was thinking of iterating though all the columns and fixing the model on each column separately and getting output for each column one by one
@@gtownmunda anytime, it's good to ask questions. Unsupervised learning algos analyse all of the features, as they check for patterns between them, between multiple features. Analysing one feature is not a case for unsupervised machine learning algorithms as you can easily do that with normal EDA.
Do not waste your time. His explanation was as clear as mud. I define anyone to define an isolation forest after listening to the first 5 minues of this video.
Hi there! If you want to stay up to date with the latest machine learning and big data analysis tutorials please subscribe here:
ua-cam.com/users/decisionforest
Also drop your ideas for future videos, let us know what topics you're interested in! 👇🏻
Great!!. Easy to understand. After you get the anomalies, how you are going to identify from the dataset which are labeled as 0. Please add that code as well.
Thank you, glad you found it helpful. To do that you simply filter the dataframe with the condition target == 0 , in our example df[df.iforest == 0]
That was a concise and clear tutorial! It would have been lovely to see some metrics on a test data set and also a visiaul on what those outliers looked like :)
Hi, really enjoyed this tutorial. But, may I know if there are any way to output some sort of score for the accuracy of the model to use it for hyperparameter tuning? Much appreciated thanks.
AUC is usually used to measure the model accuracy.
thank you for the video, very insightful. my question pertains to the contamination parameter you set -- what if we don't have any expectation around what this would be? I'm working on a very high dimensional multi class classification problem and am uncertain about how I'd go about setting this parameter.
Thank you Brent, glad it was helpful. Regarding the contamination, you set that based on your expected percentage of anomalies within that dataset. Assuming you have knowledge of expected anomaly rates for that dataset of course. If you have no such information I would suggest you go for a lower value in the range .001 to .01 and analyse the results. And tweak it based on the relevancy of the findings.
@@DecisionForest can we do hyperparameter tunning? I think by adjusting the contamination value
Thank you for the helpful video. My question is that the method expects that some of the data points are anomalies. What if I want to check the data has anomalies or not? I mean I don't want just some percentage of the data to be necessarily classified as anomaly.
Great video? Can we see which tuples are the outliers?
Thanks, this was good.
How does one see the actual rows or records which are outliers after running iforest?
Hi, Thanks for the wonderful explanation, How to verify model performance and can we do hyper parameter tuning.Thanks
Thanks for the wonderful video. Could you please do one on supervised anomaly detection perhaps using one-class SVM with gridSearchCV or RamdomSearchCV
Thanks Erone, great suggestion.
@DecisionForest Thanks a ton for your valuable video. Your Jupyter notebook link is not working anymore. Can you please update with working source url. That would be highly appreciated 🙏
Great, approachable intro! Thank you
Thanks Roberto! Glad you found it useful.
please how would you deploy a trained isolation forest model to an android application ?
You didn't remove the target (isFraud) so your model is actually just using that information
Very good point, and normally you don’t even have the target of course otherwise you wouldn’t need this in the first place. Since this dataset had it I kept it to see how the Isolation Forest performs and how many it identifies correctly, even if partially helped (partially helped because we know that’s the target, the algorithm sees it just as another binary feature). The isfraud feature is just one feature given to an unsupervised learning algorithm therefore the algorithm doesn’t output anomalies solely on this. But it does influence the outcome slightly and in a real world scenario you’d not use the fraud features of course, you wouldn’t have it anyway.
Thank you for the detailed video! You use the terms outliers and anomalies interchangeably but they're different. Can you comment on why it's used interchangeably in this video? Is Isolation Forest used for outlier detection or anomaly detection? Hope this can be answered!
Thank you Ankit. For sure, I might use them interchangeably for convenience as the topic is Anomaly Detection with IF.
Anomaly points inside data are also anomalies. While outliers only refer to outlying points.
Thanks for the good job here. i have a question, how do you extract the inlier set to continue with the various machine learning algorithms?
Well explained, thank you!
Thanks Sebastian, glad it was useful!
But how do we evaluate our model that how it is perfoming,like how did we get accuracy measure ???
Check this video out, hope it will help: ua-cam.com/video/i42UzP3mT58/v-deo.html
Firstly thanks for the tutorial, it was very informative. My question is related to the visualization of results. Is there a way we can can effectively visualize the results of multidimensional data?
Umap
what is the importance of this dataset? why was this chosen??
Hey great video, but is there a way to find which feature contributed most to the row classified as an outlier?
Thanks mate! From my current knowledge the Isolation Forest cannot have feature importance implemented due to the random nature of the splits. I have another video that might be helpful with this: ua-cam.com/video/i42UzP3mT58/v-deo.html. Let me know if that helps.
Hello, I am writing the thesis of the university on this topic, but I don't find much on the internet, could you tell me how do I get a graph printed that underlines the isolation of anomalies in phyton?
Well you can plot the normal values vs anomalies per each categorical feature. This would be a nice way to show how different values have more of a predictive capability. Something like here: ua-cam.com/video/i42UzP3mT58/v-deo.html . Hope this helps.
Sir, i have doubt.
Suppose we have credit card data..
First we label those data with anamoly detection... Then we go for preprocessing and model building like random forest ,am i right..
So anamoly detection is used to label the outliers correctly..
This is what i understood from this video...
Is my understanding is correct sir??
Good question and hope I understood it correctly. We only have one model, the Isolation Forest algorithm that uses a Random Forest approach to isolate the records from one another. And it considers the records with the shorter paths as anomalies as it took less time to isolate them. So we do the label encoding prior to building the isolation forest (not a random forest), that leads to getting the possible outliers in the dataset.
Would this apply to detecting anomalies in it assets?
Hi
Much thanks for the video
Can you please provide the code or github link from which can down load along with data set .
This will be very helpful.
I am getting error while executing this commands. Can you please provide the inpyb file for this code?
Hey there, you have some mistakes in your video.
You said that IForest can work with categorical attributes but that's wrong, it only deals with numerical.
The way you handled missing values is wrong.
Contrary to your other answers, label encoding is actually very wrong. By turning categorical attributes to numeric - the way you did it - it simply does not work for isolation forest. When it performs the random splits, the frequency of these classes simply introduces bias as the algorithm is forced to isolate the most common classes first. This is wrong and against the underlying principles of IForest as the most frequent classes in theory point to inliers, not outliers, and these should be isolated deeper into the trees. This also makes your classes ordinal which is also wrong. You should have used one hot encoding.
Finally, I'm not sure if you understand what the contamination parameter means. If you already have labelled instances, you can simply use the percentage of these as your contamination value. Training the algorithm with the labels is simply your biggest mistake. If you wanted to compare your predictions to the actual anomalies, you could just ignore that column for the training
Is there an error metricthat we should be looking for?
Great tutorial. Thanks!!
Thank you, glad you found it useful.
How do you get a Data Frame with the rows that have been classified as outliers from the model?
Simply create a filter where the target == 1
Well explained!
Thank you Ketan!
Congratulations your channel got monetized so early :))
Thank you so much! Yees, it took only 15 months haha
@@DecisionForest I actually have started watching your channel recently. In the past few days, you have a lot of videos regularly on your channel. I didn't know it took 15 months haha great work
Have you ever tried to google ad sense to promote your videos to get views?
Yes in the past months I did post more often. It varies though as there are weeks in which I have a lot more work and cannot record as often.
I haven’t as I believe that if the content is useful people will eventually find it 😊 And if they like it then it sets up the connection better in the long term.
Hiii !!
I have a question and i can't find anything online.
Can an isolation forest works well with a massive number of 0 values and a lot of columns?
Great video anyway!
Thanks,
Veronica
Hi Veronica, thanks for the support! To answer your question it will work but not great as there isn’t much information in those features unfortunately. Depends of course on the size of the dataset as well.
@@DecisionForest Ok thanks, I expected that.
The dataset is big, could PCA be a valid solution? Probably I'll lose to much information
Thank you a lot
PCA summarizes the variance in the dataset so it wouldn’t negatively affect the results but it wouldn’t help either.
The best best is to try with the full range of columns and then see if you obtain any relevant insights.
if there's a cat, there's a like
hii can i have the jupyter notebook code?
Any chance we get the source code
Picture (at 3:00 minutes) looks very familiar to me....how did you get a written copy of my course ?
What do you mean, is the explanation similar to how you present it as well? The picture is just a simple representation of a decision tree in the isolation forest algorithm
@@DecisionForest its funny, I drew it exactly the same way ...anyway, liked your video
@@mfavaits great minds think alike :) Glad you enjoyed it, trying to do my best with this channel
I think it's not correct to use label encoder in categorical data.
More details:
datascience.stackexchange.com/questions/36006/categorical-data-for-sklearns-isolation-forrest
Hi Carlos, label encoding encodes the target labels with numeric value from 0 to n-1 and it's required for the Isolation Forest algorithm. As I agree that for many algorithms the ordinal features that we get impose false orders of magnitude, tree-based algorithms aren't influenced by this.
The way the Isolation Forest algorithm works is that it isolates observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature. So the Isolation Forest doesn't care about the orders of magnitude.
@@DecisionForest Thanks for the explanation I'm gonna read more about the theory behind how isolation forest really works to understand your comment properly, it's my first time using this algorithm
Happy to help. 👍🏻
King
How can you give the class feature(isFraud) as input to the model?, the model will tell you outliers solely based on this target feature. Your logic is completely wrong.
Very good question, and normally you don’t even have the target of course otherwise you wouldn’t need this in the first place. Since this dataset had it I kept it to see how the Isolation Forest performs and how many it identifies correctly, even if partially helped (partially helped because we know that’s the target, the algorithm sees it just as another binary feature). The isfraud feature is just one feature given to an unsupervised learning algorithm therefore the algorithm doesn’t output anomalies solely on this. But it does influence the outcome slightly and in a real world scenario you’d not use the fraud features of course, you wouldn’t have it anyway.
Is it univariate or multivariate based analysis for labeling the data to be outlier or not?
Can I get column wise outliers by using this? Can you help pls
For column wise outliers you don't need this, you can just explore outliers based on data type.
@@DecisionForest sorry I didn't understand what you mean by based on data type, I was trying to use this algorithm and trying to apply to each column so that I can get 0's and 1's column for each column, can you suggest the best way to detect column wise outliers in any dataset pls?
@@gtownmunda no problem, well fr continuous variables you can simply use a boxplot or histograms. Categorical variables don't have outliers. So for every column you can just explore the data type and check for anomalies with statistical methods.
@@DecisionForest thank you for your response, one last question!
What is the issue with using isolation forest? I was thinking of iterating though all the columns and fixing the model on each column separately and getting output for each column one by one
@@gtownmunda anytime, it's good to ask questions. Unsupervised learning algos analyse all of the features, as they check for patterns between them, between multiple features. Analysing one feature is not a case for unsupervised machine learning algorithms as you can easily do that with normal EDA.
Link for the notebook?
Will add the link in the description as soon as possible.
Can i get the source code and data?
Of course, the link is in the description.
Do not waste your time. His explanation was as clear as mud. I define anyone to define an isolation forest after listening to the first 5 minues of this video.
Hi Patti, thanks for the comment. Could you please clarify what you didn’t understand? As a lot of other people found it helpful.