Clear, to the point, with real life examples. I've been learning pandas and I decided to do a recap on math. The fact that you provided examples in pandas is the happiest coincidence I've come across this week. Thank you!
Thank you Sir, for making such kind of beginners friendly videos. I really enjoyed and learned a lot. Please make make more such kind of videos so that we can understand easily. ❤️
Thank you very much for detailed and nice explanation. Have a question, Do we need to remove outlier all time? What if the salary range is constant not like unusual high salary(Elan musk as per mentioned use case)?
percentile_95=df.price.quantile(0.95) sir i do by this approach, it give 350 something if i increase value of quantile outlierss comes max gap that's why i remove all values upon this condition
Your way of teaching is incredible, I love your videos. Whenever anyone ask me from where you learn all this then, I share link of ur channel to my juniors.
Sir Your quartile calculation seems to be wrong. The formula for the rank of 25th percentile is 25/100*(7+1) which is 2. This is universally accepted. It means the value should be 5000 only. I really don’t know how pandas is also doing the same mistake
In the median example at minute 2:40 , shouldn't we order the values first before guessing about which value is the median? shouldn't the values be like that: 4,000 < 5,000 < 6,000 < 7,000 < 7,500 < 8,000 < 8,000 < 10 million so, the median would be the average of 7,000 and 7,500 which is 7,250
Sir , how is the median of the data points 7500 , since the median has to be the average of Tao's and Sofia's income so it will be (7000+7500)/2 = 7250 right.. So I meant after arranging in ascending order
Hi sir, thanks a lot for your extraordinary teaching, I have learned lot and did my homework by following your machine learning tutorial. Sir, Can you do for a video about Generative Adversarial Network (GAN) for regression prediction?
I'm near about 50 . I have completed MCA from IGNOU and Digital marketing from NIIT imperia. I worked as a software developer and now im a digital marketer. If I want to change my career in data science after learning this field, can i get a job in data science field?
Again Great Video Sir. I have a silly doubt. As you said we cant take average to fill null value if outlier have very large value like Elon musk(10 million$) and now we are going to take Median to fill na values.but nan values itself present at the middle of datapoints .So how we gonna calculate median if nan value is present at those points. median=(nan+nan)/2 ?
i am a jr. data analyst with less than a year experience if i apply for jobs is it expected of be to be able to code advance python funcions? cause now i feel that i am just able to understand code by debugging it but if i try to write similar code i am not able to but i k what function does what and if i a problem statement is given i will be able to identify what thing we should be doing to achieve the result but i am unable to implement it. please give your opinions on this. cause coming from non-it i am always havinng a sens eof insecurity that i dont know python enough.
Hi! in time = 2:44 for the median you take Tao and Prem, but they must be first sorted and Prem it is not counted in the median, but Sofia do. So m=(Tao + Sofia)/2?
Nice video. I would like to suggest a change. 100th percentile doesn't exist, only 99th. In your example, Musk would have to be earning higher than himself to be the 100th percentile.
One very basic question - Should the outlier removal be applied on labels (values to be predicted) as well if outliers exist on such data labels as well ?
am I just scatter brain, or did you not include the link to video where you explain how to use iqr to remove outliers? I only see a link to a playlist, but none of them seem to be on that particular topic? EDIT: okay, seems you explained it later in this video, but it really sounded like you had a link for us...
Hi , I have a dataset where 3 columns are independent categorical features and 5 dependent features that are 10th ,25th, 50th ,75th , 90th percentile of annual wage. How can I get values (annual wage ,which is missing) from the 5 percentile columns ?
Consider my data points: 100 100 100 100 here the 50th percentile which is 100 is kinda misleading right? because 2 more 100 values are present in the right side of median. SO.. 100% of the data values are equals to 50th percentile. Can you please explain where I am confused??
For suppose the data is like this 4,4,6,7,40,100,110,120,1300...in this case taking median doesn't make sense right ....same for mean outlier 1300 involved...and for mode also 4,4 just repeating 4 for 2 times doesn't make sense right... What to do in this case please any one answer me ...could we find solution from this video..
Taking mode of 4 is perfectly ok because you are looking for a value that is most frequently occurring and 4 is that value. It really depends on what problem you are trying to solve here. Can you suggest what type of dataset this is? You just made up the values and are generally curious about such distribution?
@@codebasics I just take it as an example...but just for repeating 4 for 2 times blindly we can't take 4 for filling the missing value right because it is far less than other higher values
Hi, I'm a bit confused with the solution of the exercise. To me, the outlier is not simply removed by percentile, we should exclude the line with 365 availability and 0 reviews + 0 availability and 0 reviews because those lists are just "ghost" lists that no one actually rent them or just the data is not accurate. If we go further down, we should probably clean the data by review date also, I see some of them are with 2011 date, but if we are analyzing the average of this/recent year then there should be a cut off of the latest year we can use. Please let me know your thoughts. Thanks.
Totally agreed with your thoughts here. Percentile is just one of the ways, using common sense simple logic is totally a legit way of treating outliers
@@codebasics Thank you for replying to me so quickly, so if I apply what I said in the post first and then apply percentile, is that going to be right, or let's say with better accuracy? Also, how do we measure the accuracy? should the mean be close to the 50% percentile? how do we know our analysis is good or bad? Thank you so much!
Subscribing to this channel proved to be lot more helpful than enrolling into college for graduation
Mean, median, mode and percentile are also known as 'Measures of Central Tendency'.
Yeah bro I listened them from Khan Academy
You are the best teacher and have the best content on data analysis. NO need to go any channel.
I am happy this was helpful to you.
Thank you so much Sir you're a good teacher and you're different from others because of the practice you demonstrate
Clear, to the point, with real life examples. I've been learning pandas and I decided to do a recap on math. The fact that you provided examples in pandas is the happiest coincidence I've come across this week. Thank you!
As a. Beginner, I should say this is the best.
2:45 When you added an extra value you did not sort them in ascending order (7000,7500,8000) instead of (7000,8000,7500).
The best video to understand the concept of removing outliers
Your content and examples are great😃.
Please don't stop making such easily explained content.
Sr please make one video for freshers on real life data science project, your teaching skills are so simple everyone can understand very easily
Thank you Sir, for making such kind of beginners friendly videos. I really enjoyed and learned a lot. Please make make more such kind of videos so that we can understand easily. ❤️
Tell me Your insta id bro...plz
Before exploring the Codebasics channel. I never had an interest in Math & Stat. Thanks, Bro. Love & Respect from Pakistan
I am happy this was helpful to you.
This series are masterpiece. Thank you.
Yeah . So true ...*uck education system
Thank you very much for detailed and nice explanation.
Have a question, Do we need to remove outlier all time? What if the salary range is constant not like unusual high salary(Elan musk as per mentioned use case)?
I have no words to say, really awesome series!
percentile_95=df.price.quantile(0.95) sir i do by this approach, it give 350 something if i increase value of quantile outlierss comes max gap that's why i remove all values upon this condition
Your way of teaching is incredible, I love your videos. Whenever anyone ask me from where you learn all this then, I share link of ur channel to my juniors.
Thanks for sharing! I am happy this was helpful to you.
Ultimate Explanation🎉 Got a good idea on using mean and medain
you earned a sub man!!! what an explanation
Sir your way of teaching is very awesome
Sir
Your quartile calculation seems to be wrong. The formula for the rank of 25th percentile is 25/100*(7+1) which is 2. This is universally accepted. It means the value should be 5000 only. I really don’t know how pandas is also doing the same mistake
In the median example at minute 2:40 , shouldn't we order the values first before guessing about which value is the median?
shouldn't the values be like that: 4,000 < 5,000 < 6,000 < 7,000 < 7,500 < 8,000 < 8,000 < 10 million
so, the median would be the average of 7,000 and 7,500 which is 7,250
Sir upload real life data science project 👍😁
On UA-cam search for "codebasics data science project", you will find my videos please watch it
Sir , how is the median of the data points 7500 , since the median has to be the average of Tao's and Sofia's income so it will be (7000+7500)/2 = 7250 right.. So I meant after arranging in ascending order
Hi sir, thanks a lot for your extraordinary teaching, I have learned lot and did my homework by following your machine learning tutorial. Sir, Can you do for a video about Generative Adversarial Network (GAN) for regression prediction?
furthermore to learn, this was even a enjoyable video, thanks a lot sir.
Exactly what I wanted a mentor 👍🏻❤️🙂.
While calculating the median( when data values are even) we need to sort data values in ascending order.
Im hoping that by end of covering all playlist ill become master at data science and the following know of ML,DL,LLM
why is using median better than leaving musk out and getting the average of the rest? is it compulsory for all data to be used?
I'm near about 50 . I have completed MCA from IGNOU and Digital marketing from NIIT imperia. I worked as a software developer and now im a digital marketer. If I want to change my career in data science after learning this field, can i get a job in data science field?
Beautiful explanation
Glad it was helpful!
Only one word loved your explination
2:43 there should be sorted values and median will be equals to (7000+7500)/2
At 3:02 adding prem to the dataset is disturbing the ascending sorting order. So the median should really be 7000+7500 / 2 = 7250.
Again Great Video Sir. I have a silly doubt. As you said we cant take average to fill null value if outlier have very large value like Elon musk(10 million$) and now we are going to take Median to fill na values.but nan values itself present at the middle of datapoints .So how we gonna calculate median if nan value is present at those points. median=(nan+nan)/2 ?
maybe you can take the median of non-null values and fill up
I think for taking median of dataset first we have to rearrange data to ascending order that will shift position of Nan value!!
@@shutterup24-7 yes thats the first step
i am a jr. data analyst with less than a year experience if i apply for jobs is it expected of be to be able to code advance python funcions? cause now i feel that i am just able to understand code by debugging it but if i try to write similar code i am not able to but i k what function does what and if i a problem statement is given i will be able to identify what thing we should be doing to achieve the result but i am unable to implement it. please give your opinions on this. cause coming from non-it i am always havinng a sens eof insecurity that i dont know python enough.
Hi! in time = 2:44 for the median you take Tao and Prem, but they must be first sorted and Prem it is not counted in the median, but Sofia do. So m=(Tao + Sofia)/2?
Nice video. I would like to suggest a change. 100th percentile doesn't exist, only 99th. In your example, Musk would have to be earning higher than himself to be the 100th percentile.
Thanks for making this video its very helpful
Sir there could have been possibility that sofia's income would really high then median will not work well?
Very informative video.
Example of Mode is lit 😀
Really great no raatta
One very basic question - Should the outlier removal be applied on labels (values to be predicted) as well if outliers exist on such data labels as well ?
No
Sir apney last maen outlier ko remove kaisey kia ?
What is the difference between average and mean?
Sir what if the data is missing from or below 25% ,75% then how to find The Average.please reply
do you have any full course on data analysis?
Why don't we fill missing values with mode?
Mode is the one that appears most but why we use mean and median most of the time?
Sir I want to know which language is very important? Should we start with Java or python
Thank you that was very informative content.
Sir can u explain the steps to become a data analyst and skills required for that
On UA-cam search for "codebasics learn data analyst skills", you will find my videos please watch it
@@codebasics tq sir
One of the best tutorial ❤️🔥
Glad it was helpful!
Should we first learn pandas then attempt exercises?
Sir, could you please add the assignment link?
Hi Sir,can we use multiple median for multiple NaN data like you did in sofia;s case?
am I just scatter brain, or did you not include the link to video where you explain how to use iqr to remove outliers? I only see a link to a playlist, but none of them seem to be on that particular topic?
EDIT: okay, seems you explained it later in this video, but it really sounded like you had a link for us...
mimosvera, you are right I forgot to include a link but I just added it now. Please check video description
Thanks for such a nice tutorial
Glad it was helpful!
Hi , I have a dataset where 3 columns are independent categorical features and 5 dependent features that are 10th ,25th, 50th ,75th , 90th percentile of annual wage. How can I get values (annual wage ,which is missing) from the 5 percentile columns ?
Is the amount of statistics required for data science and data analytics the same?
what is the difference between Linear Quantile and Midpoint quantile ??
links to softwares used?
Awesome learning 🆗😎👍
Glad you enjoyed it
Is careerera a good institute to join as a beginner.
Im final yr ECE student.
So, How to identify there is an outlier in the dataset? please calrify
Consider my data points: 100 100 100 100
here the 50th percentile which is 100 is kinda misleading right? because 2 more 100 values are present in the right side of median. SO.. 100% of the data values are equals to 50th percentile. Can you please explain where I am confused??
I am not sure still, you can double-check with someone else too .In your case, you should consider mode as your measure and ignore mean or median.
It is good if you should have taught why not median and mode in some cases
in the example at 3:00 u havent sort data in ascending order for median
Sir, I need your suggestion. Can you help me ?
great sir 🥰
What is the difference between 0.99 and 0.999 quantile range as in exercise 0.999 is used?
in case of even n.of data point you have not sorted them so median is wrong
why cannt we use trimmed mean?
How can i code in Jupyter, just like you.
so clear, thx.
Glad it helped!
Power Bi KO Course Kaha Cha Hola?
Sandeep Jain sir GFG samjhne wale haath uthao😅😅
For suppose the data is like this
4,4,6,7,40,100,110,120,1300...in this case taking median doesn't make sense right ....same for mean outlier 1300 involved...and for mode also 4,4 just repeating 4 for 2 times doesn't make sense right... What to do in this case please any one answer me ...could we find solution from this video..
Taking mode of 4 is perfectly ok because you are looking for a value that is most frequently occurring and 4 is that value. It really depends on what problem you are trying to solve here. Can you suggest what type of dataset this is? You just made up the values and are generally curious about such distribution?
@@codebasics I just take it as an example...but just for repeating 4 for 2 times blindly we can't take 4 for filling the missing value right because it is far less than other higher values
how do you know that your data has oulier
nice
why 0.999 in the exercise ?
removing elon from twitter as an outlier is also great
Sir tamari sathe contact kai rite kari saku?
2:44 median will be 725
Hi, I'm a bit confused with the solution of the exercise. To me, the outlier is not simply removed by percentile, we should exclude the line with 365 availability and 0 reviews + 0 availability and 0 reviews because those lists are just "ghost" lists that no one actually rent them or just the data is not accurate. If we go further down, we should probably clean the data by review date also, I see some of them are with 2011 date, but if we are analyzing the average of this/recent year then there should be a cut off of the latest year we can use. Please let me know your thoughts. Thanks.
Totally agreed with your thoughts here. Percentile is just one of the ways, using common sense simple logic is totally a legit way of treating outliers
@@codebasics Thank you for replying to me so quickly, so if I apply what I said in the post first and then apply percentile, is that going to be right, or let's say with better accuracy? Also, how do we measure the accuracy? should the mean be close to the 50% percentile? how do we know our analysis is good or bad? Thank you so much!
🙏🏻
The video is great but i didnt like the exercise because there is more in it than it has been covered in the video
Legend
The funniest part is if Elon Musk lives in our town😂😂
Ha ha.. yes he is my neighbor ☺️🧐
Sorry sir , you miss one part in the video first we have to sort the nos. ( When the count of no is even (while finding median )
1:55
I didn't get percentile in first glance
This is legend . Go to hell teachers and education system...
You really wish musk to be your neighbour it seems
Your medin answer is totally wrong
16:04 df.income.iloc[3] =Nan will work too
what is the difference between Linear Quantile and Midpoint quantile ??