Thank You so much for explaining most of the concepts in simplest way.When you explain anything by far anyone can say you really want us ( your students ) to learn more . One of the best teachers for sure. Thankyou and best of luck for further success on UA-cam.
Thanx bro mujhe ye sb pta tha i mean pure theory and formulas but implement kese krte h python me ye tumne bahut easy way me explain kia thanx man really appreciate that and me wait krunga agle video ka jisme real data p kre thanx for that because me R p ye kr leta hu but i am not good in python aur tum bahut sahi tarike se smjhate ho thnx for that🙏
at 14:13 point 4 and point 5 are wrong..however in code you have implemented correctly. A suggestion please "try to keep formulas and notes clean, because if that is ruined whole plot gets off". Though you have done a good job by far.
Could you please arrange the playlist videos in 1st to last manner? Just I am getting confused that which video I should watch after particular video. By the way, you explain very well👍🏻💥
Standard Normal distribution can be applied only to a normally distributed column...SND applied to a non-normally distributed column may give you a mean of 0 and and SD = 1 but the nature of distribution is still non-normal. Z score cannot be directly tied to the 99.7% of ND in such a case. The interpretation of Z score is therefore subject to the kind of the distribution structure of the column which remains unaltered before and after the Standardization process.
Sir - I appreciate your effort of making all these beautiful videos. It makes life easy for all of us. But I need to make one comment on this video, at the beginning you said when the weight(x) is increasing height(y) is also increasing but in real world if you think it's not possible right ? As per my thought process may be you wanted to say when the height(x) is increasing weight(y) is also in a increasing trend. In most of the cases, when a person's height increases automatically weight also get increased but it's not the reverse. I hope i am making sense here. Thank you once again :)
Hi Krish, its always great to learn from you... can anyone please tell when to use what? Or which amonthe the methds is best? or is it case wise and if yes what are the cases?
If we have many outliers, in that case, is it good to calculate(or detect by frequency)mode over mean and median ?? And then replace them with mode (mostly we do it for categorical variables) is this a good way?????
This was really explained very well. I have a doubt regarding multipication by 1.5 , can you please elaborate how this 1.5 came or what is the method by which we can get this value in any other dataset.
Here outliers is exactly coming in Z score when I'm putting the value of threshold=1 and not coming when putting the value of threshold=3 WHY????? Btw thaxx alot very outstanding playlist for STATISTICS.....Gtr work
Sir can you make a video on fitness based recommendation system. and can you tell me how to collect data set of different people and recommend them food and exercise according to that data set. plz reply sir....
I have few doubts... why threshold value is considered as 3 I have seen other examples they also considering as 3 only ? is it always 3 ? why to only multiply with 1.5 for lower and upper bound values ... any specific reason? Does iQR works on categorical columns also ... I know that zscore doesn't work for categorical values?
Answering your doubts--- 1. it is not really 3 actually it is 3*sd where sd is 1(since by using z-score we make the normal distribution with mean=0 and sd =1) and yes, we always take the 3rd sd. that is all the values which are beyond 3rd sd are outliers. because until we reach 3rd sd 99.7% values are covered. which seems pretty enough. 2. As krish stated 1.5 is a statistical figure and drawn out of experiments so basically we have to use it. 3. It is very obvious ....using such statistics on categorical data doesn't make any sense. If you still not get it. just tell me how exactly are you going to sort categorical values at the first place😅😂😂🤣
I am facing a unique issue , following this methdology for IQR: Mine lower value of column is coming . Below is the pseudo Code: Q1 = df[' shares'].quantile(0.25) Q3 = df[' shares'].quantile(0.75) IQR = Q3 - Q1 low = Q1 - 1.5 * IQR ------>>>>>>>>> this value is coming extreme negative which is not even present in columns The problem maybe is IQR value > Q1. How to solve ? Shall I change the % age?
Hi Krish, I have a doubt regarding the parameter inside the function , when defining the function ...it is given as data. It should be the dataset right ?
Sir. Firstly a brilliant teaching and mentor. I have some doubt in calculating upper bound and lower bound you showed. You are saying anything beyond 1.5iqr will be outlier, then why we are doing low= q1-1.5iqr and high=q3+1.5iqr. cant we straight away get outliers by comapring if number is less than or greater than 1.5iqr. I did not understand meaning of q1-1.5iqr as lower bound and similarly for upper bound. How?
@@Joshua75623 First the values are sorted in increasing order. Q1 is the 25th percentile, Q3 is the 75th percentile. IQR is the length of the box. so values > Q3+1.5times IQR and values < Q1+1.5times IQR indicates the outlier.
yea that happens....cuz mathematically proven: 1.7 is the accurate measure for the length of whiskers but it's taken as 1.5 for "symmetry". So you may check with 1.7 instead of 1.5 and observe the same result for z-score and iqr
Z-Score & Outliers - explained in very easy way. Thank you..
Thank You so much for explaining most of the concepts in simplest way.When you explain anything by far anyone can say you really want us ( your students ) to learn more . One of the best teachers for sure. Thankyou and best of luck for further success on UA-cam.
Thanx bro mujhe ye sb pta tha i mean pure theory and formulas but implement kese krte h python me ye tumne bahut easy way me explain kia thanx man really appreciate that and me wait krunga agle video ka jisme real data p kre thanx for that because me R p ye kr leta hu but i am not good in python aur tum bahut sahi tarike se smjhate ho thnx for that🙏
at 14:13 point 4 and point 5 are wrong..however in code you have implemented correctly. A suggestion please "try to keep formulas and notes clean, because if that is ruined whole plot gets off". Though you have done a good job by far.
Best!!! Thankyou! This channel is a saviour in DS doubts
Wonderfully explained. Won't forget now. Thank you
You continue this hardwork ,surely you will get success one day on UA-cam.
Could you please arrange the playlist videos in 1st to last manner?
Just I am getting confused that which video I should watch after particular video.
By the way, you explain very well👍🏻💥
Thank you so much ...Concept cleared perfectly:)
Awesome explanation Krish.Thanks
Smoothly explained
Standard Normal distribution can be applied only to a normally distributed column...SND applied to a non-normally distributed column may give you a mean of 0 and and SD = 1 but the nature of distribution is still non-normal. Z score cannot be directly tied to the 99.7% of ND in such a case. The interpretation of Z score is therefore subject to the kind of the distribution structure of the column which remains unaltered before and after the Standardization process.
Nice explanation..
Thanks Krish
Thank you, sir
Finding Outliers with help of Z Score Using Pandas:
Just some modifications
outliers=[]
def detect_outliers(df):
threshold=3 # 3 standard deviations
# mean=np.mean(df)
#std=np.std(df)
z_score=(df-df.mean())/df.std()
df['out']=z_score[z_score>threshold]
df.dropna(inplace=True)
return df.iloc[:,0].values.tolist()
detect_outliers(d)
Sir please upload you kaggle competition of house price with feature selection. Waiting for your vedio
Sir - I appreciate your effort of making all these beautiful videos. It makes life easy for all of us. But I need to make one comment on this video, at the beginning you said when the weight(x) is increasing height(y) is also increasing but in real world if you think it's not possible right ?
As per my thought process may be you wanted to say when the height(x) is increasing weight(y) is also in a increasing trend.
In most of the cases, when a person's height increases automatically weight also get increased but it's not the reverse.
I hope i am making sense here.
Thank you once again :)
Hi Krish, its always great to learn from you... can anyone please tell when to use what?
Or which amonthe the methds is best? or is it case wise and if yes what are the cases?
Simply awesome,great work
If we have many outliers, in that case, is it good to calculate(or detect by frequency)mode over mean and median ?? And then replace them with mode (mostly we do it for categorical variables) is this a good way?????
Thanks a Lot Sir, Really helpful.
super!!!
well explained sir!!!!!
can the value of inter quantile would be range in negative(-)
I am not able to find the link of the next video which you have told related to outlier handling in a dataset from kaggle.
I also not getting the video
This was really explained very well. I have a doubt regarding multipication by 1.5 , can you please elaborate how this 1.5 came or what is the method by which we can get this value in any other dataset.
Now i can breifly explain why only 1.5 and the reason is really amazing and really informative.
Can you explain me ?
@@KumarGolu2001 It is just a proof done by stats people , 1.5 gave the right bound values to detect outliers.
we can use any one of the technique 1.Z-score or 2.IQR to find outliers??
generally when data is skewed IQR better if roughly normal then z score better
Here outliers is exactly coming in Z score when I'm putting the value of threshold=1 and not coming when putting the value of threshold=3 WHY?????
Btw thaxx alot very outstanding playlist for STATISTICS.....Gtr work
Sir please make video on p value
Sir how is this method scalable when we have 50s or 100s of dimension /features in a dataset
IQR method can be used when the data is huge
Is the detect_outlier using z score program working fine for everyone?? It is not returning correct output for a diff dataset used.
perfectly explained krish but How to detect outliers for multidimensional data
Sir can you make a video on fitness based recommendation system.
and can you tell me how to collect data set of different people and recommend them food and exercise according to that data set.
plz reply sir....
actually that is very good idea
i hope you learned that
i'm gonna try this once i learned this
amazing tutorial, thank you
In the detect outlier function "y" is not defined anywhere please change it to i
I have few doubts...
why threshold value is considered as 3 I have seen other examples they also considering as 3 only ? is it always 3 ?
why to only multiply with 1.5 for lower and upper bound values ... any specific reason?
Does iQR works on categorical columns also ... I know that zscore doesn't work for categorical values?
Answering your doubts---
1. it is not really 3 actually it is 3*sd where sd is 1(since by using z-score we make the normal distribution with mean=0 and sd =1) and yes, we always take the 3rd sd. that is all the values which are beyond 3rd sd are outliers. because until we reach 3rd sd 99.7% values are covered. which seems pretty enough.
2. As krish stated 1.5 is a statistical figure and drawn out of experiments so basically we have to use it.
3. It is very obvious ....using such statistics on categorical data doesn't make any sense. If you still not get it. just tell me how exactly are you going to sort categorical values at the first place😅😂😂🤣
can categorical data (encoded) have outliers?
Here is a nice explanation about how to detect outliers on categorical data .
@@lokeshkaturi4040 can you please share timestamp in this video
how can mean be equal to zero in real-time scenario as weight cannot be assigned a negative value please help
sir why you have taken threshold as 3 as you said it should fall in 3rd standard deviation ,so it should not be (3*std)?
So which one should we use for removing the outliers??
Exploring all the ways is the best thing.
If there are more number of features (like X1 X2 X3 and so on) then how can we find out using Scotter plot
I am facing a unique issue , following this methdology for IQR:
Mine lower value of column is coming . Below is the pseudo Code:
Q1 = df[' shares'].quantile(0.25)
Q3 = df[' shares'].quantile(0.75)
IQR = Q3 - Q1
low = Q1 - 1.5 * IQR ------>>>>>>>>> this value is coming extreme negative which is not even present in columns
The problem maybe is IQR value > Q1.
How to solve ? Shall I change the % age?
Nice
Hi Krish,
I have a doubt regarding the parameter inside the function , when defining the function ...it is given as data. It should be the dataset right ?
data is just the parameter name , can be any label
sir please upload video on preprocessing data using sklearn library
Sir. Firstly a brilliant teaching and mentor. I have some doubt in calculating upper bound and lower bound you showed. You are saying anything beyond 1.5iqr will be outlier, then why we are doing low= q1-1.5iqr and high=q3+1.5iqr. cant we straight away get outliers by comapring if number is less than or greater than 1.5iqr. I did not understand meaning of q1-1.5iqr as lower bound and similarly for upper bound. How?
Oh I understood . I read the text later. Thankyou so much for sharing your knowledge
Akansha, did you got the answer for your question? Even Im getting the same doubt
@@Joshua75623 First the values are sorted in increasing order. Q1 is the 25th percentile, Q3 is the 75th percentile. IQR is the length of the box. so values > Q3+1.5times IQR and values < Q1+1.5times IQR indicates the outlier.
Damn good dude. Thank you so much.
In your code, after are np.abs(z_score) > threshold: Why are you appending y?? Shouldn't you append i?? Please correct it as it is misleading.
i couldn't find the next video which you talk about in the last....
Post the video of box plot technique
@nayazi
ua-cam.com/video/CVdcr_MC2KU/v-deo.html Watch this video from 6.25
plt.boxplot(x=dataset)
Hi.. How to work with outliers.. I don't want to remove outliers from my data. Is there a way which I can do that?
thanks alot... sir..
Sir i got error name error
Nameerror: name mean is not defined
@krish why it is only 1.5 . Why not 1 or 2. Why only 1.5
it's proven mathematically comparing with the standard normal distribution ....actually it's 1.7 but it's taken to be 1.5 for 'symmetry'
It is more appropriate to call it Quartile than calling them Quantile.
how he has set threshold as 3?
Your video is great but not systemically managed
❤💫
Z-score isn't give output for more than 3 outliers
How to detect and remove from each column in one shot
Sir, I tried finding the outliers in a dataset using both methods(Z-Score,IQR) and the answers are different. Can you please explain why?
yea that happens....cuz mathematically proven: 1.7 is the accurate measure for the length of whiskers but it's taken as 1.5 for "symmetry".
So you may check with 1.7 instead of 1.5 and observe the same result for z-score and iqr