Me trying to understand data analysis with python couple of days ago now U actually make it simpler and beginners friendly, more unction to function sir
Sir I'm very impressed to see your such amazing video.. Though I am very weak in programming but now I feel like that i should start my programming journey again cause i have someone like u who can explains anything in very simple way
21:30 Median of the passenger age travelling in each Pclass can be calculated using below code instead of looking at boxplot and guessing the number. df[df['Pclass']==1]['Age'].median() df[df['Pclass']==2]['Age'].median() df[df['Pclass']==3]['Age'].median()
Both imputing and dropping missing values (NaN) is not a good practice with real world data. The ideal way is to derive a new field indicating missing values. 1 for missing else 0. because, sometimes missing value can be a new information in itself. just sharing some learning from my job :)
sir i really liked your video.. but according to road map video, you asked us to watch python 1-24 lectures first..in this eda concept, you have mentioned some new words like get_dummies, and few other new words.. stuck with the last 10 mins explaination.. else everything is really clear and understandable.. thanks for all the efforts...
It is basically one - hot encoding.. Encoding techniques are used to convert categorical data into numerical data Since it is applied on 'Embarked' column ua-cam.com/video/OTPz5plKb40/v-deo.html
very simple and lucid videos. it encourages me to practice along not getting into too many details.. at the beginning..Superb and stay blessed👏 one ques: at timestamp 27.09, from the code without mentioning dtype=int, the ouptut displays bollean value in integer form. but in my case it shows as 'true' or 'False' unless dtype is specifically mentioned as int. IS this something to do with python updates?
Krish - Thank you for the EDA, Throw some light on Story Telling - If you had to conclude the EDA, Theorotically, In lay man terms - we must do the story telling- Correct me If I am wrong .
there is another null left in embarked column in 831st entry. it still shows in the heatmap, while in the video this doesn't show.(25:07) and if I continue this path, do I apply the same method of removing age nulls(defining a class) or should I just replace the average value directly by redefining the index of the null(as it is just a single cell)?
Great video Sir, I just have two doubts that why did you not use get_dummies on "Pclass" as it was also categorical data.. and second why did you not normalize the "Fare" and "Age" Columns as their values are might over power the results?
If you type "train.info( )" you will see thae dtypes of all the columns. I don't know if this might help or not but get_dummies( ) can be used for objects only i think as they do not represent any numerical value for the system to compute get_dummies( ) changes indicates those objects into numerical values. Please correct me if i am wrong as i am also confused about this if you agree or have a different insight on this please tell me so.
hey @Krish! Should we do this data visualization for each and every column? or we do it after feature selection? if we are supposed t do for each column, wouldn't the code get to big and complex for data with hundreds or thousands of features?
Thanks a lot Kris. EDA was well explained. I could not understand the last part starting from confusion matrix and how to read the final result of the analysis?
I like the video, but how did you know exactly the graphical representation to use, i mean why countplot why not jointplot? Why line plot not boxplot? I hope you really understand my questions sir
i have a question why is he not using SimpleImputer class from scikit learn instead of finding the realtion to make the nan values having some values we can easily do it through sklearn module and also why isnt he using label encoder for binary values ???
20:20 hey, uhmm.. 50% percentile gives us MEDIAN of the age of people with 1st class... So we are using MEDIAN value instead of MEAN right? Very helpful video for me to understand EDA
You're right, 50%ile is the median. I think you should check out the definition of median and percentiles on this page - www.statisticshowto.com/probability-and-statistics/percentiles-rank-range/#:~:text=The%2050th%20percentile%20is%20generally,quartiles%20is%20the%20interquartile%20range. That should clear your doubt.
@Krish Naik what is that test size =0.30 why did u use that .from beginiinng of video everything was very good but in the end i couldn't understand x train ytrain test size whats that accuracy 0.7190 etc. please tell me sir else your efforts will go waste ...
Sir, what is the need to visualise the data in this problem. You haven't use any analysis extracted from the visualisation to get help out in data cleaning.
I understood till splitting training and output data. But From there Logistical regression application and confusion matrix application is very difficult to understand. I found theoretical explanation of Logistic regression but the code and explanation of syntax and its application videos is not found. Could anyone help with links to understand these two concepts. Thankss
Actually you should reset the laptop because if any file found in name of panda means error willl be encountered and in the other case you should download and upload in jupyter notebook and in that jupyter notebook you should copy the path...
Sir, I didn't get why you compensated the missing value of age with the average age of Pclass? Can't we simply replace the NaN values with the median values of the age column as: train['Age']=train['Age'].fillna(train['Age'].median())
In practical reality, every person has an age value but that data is missing for some people in the titanic dataset. Our goal is not just to fill in any random age where the age is missing but to fill in an educated guess/ estimate of the missing age of a person so that it can be a close representative of the true ages of those people. Of course, like you mentioned, the median of the entire age column could be used as an estimate but would that be a good representative value for ALL missing ages? Some people would have ages far above or below the median age. So on further exploration we notice that the median age for each Passenger Class is different, which would mean that in reality, people from a certain p-class would more likely be of a certain age, than someone who belongs to another p-class. And this difference is considerable (37 vs 29 vs 24). So by using using p-class to estimate age, we're just making a more educated guess for missing age values. You could of course go several steps further and consider other factors (like maybe SibSp, Parch etc.) in order to get a higher probability age value.
A very good about EDA but one thing i must mention that you didnt even touch the outliers concept. Its the major part of EDA and honestly i take this video only for outliers . But didnt find .
for me lbfgs is failed to converge on the local minima . How to fix it. i believe more categories need to be labeled like Pclass and standard scaler is required for age and fare .
Why do we need to get dummy variables for binary class variables like Sex and Embark, and why didn't we treat the variable pclass with One-hot-encoding, is it because we are treating it as ordinal, but wouldn't it cause problems with linear-regression and DNN algorithms to apply over it? Let me know Sir. Thanks.
I am getting key error after executing the following code: sns.distplot(train['Age'].dropna(),kde=False,color='darkred',bins=40) Any suggestion/idea as to what is to be done to stop getting this error?
Hi Can anyone help me with difference beyween EDA of this Titanic Dataset and EDA of Housing Price Prediction. Both follow a Different Steps. Iam quite Confused. Will Really appreciate any help.
After so much of struggle with my LMS, I was finally able to understand entire EDA in within 30 minutes. Thank you.🙏👍
Is it the inmovidu one?
What ia LMS
@@ytg6663 learning machine shorts
@@ONE-THING-2RAY Where it is?
@@ytg6663 Learning Management System
Doing a job that of True Guru, Ekalavyas are all around and raring for such knowledge-impartation. Thanks much Krish.
You are awesome sir! Not only are you a great mentor, but also a great motivator. Thanks for all the great work you have been doing. Stay blessed!
I am learning this for data analyst but not sure what more should I learn to get job asap.. if you can help please we can connect on instagram
Me trying to understand data analysis with python couple of days ago now
U actually make it simpler and beginners friendly, more unction to function sir
What a beautiful video for a beginner who is just getting his hands on data science.
Loved the video; in fact, the entire playlist gives an amazing approach to the intricacies of Machine Learning. Thank you, Sir.
You have actually cleared the EDA concept for me, Thanks a lot !!
why 0 and 1 is taken in cols as the indexing of the column is 2 and 5 then why 0 and 1 is taken can you clear
Krish, This material is FIRST CLASS. Appreciate it very much.
Sir I'm very impressed to see your such amazing video.. Though I am very weak in programming but now I feel like that i should start my programming journey again cause i have someone like u who can explains anything in very simple way
21:30 Median of the passenger age travelling in each Pclass can be calculated using below code instead of looking at boxplot and guessing the number.
df[df['Pclass']==1]['Age'].median()
df[df['Pclass']==2]['Age'].median()
df[df['Pclass']==3]['Age'].median()
good one brother i was thinking the same y to guess it when we can actually calculate it,....
There is a error comes when I want to use sns.countplot. And the error is "could not interpret input 'survived' "
@@tusharmahuri2439 bro copy the heads from the data set and not just type, the language is case sensitive
it is 'Survived' and not 'survived'
You are amazing brother. Your videos are helping me gain confidence in ML. Keep up the good work
This is one of the best data set being used to understand how to fix the nulls. Great Job and thank you .
Thanks a lot Sir... You've expailed it in a great way... Love from Pakistan
3:37 Add hahahaha Great learning Exp love you brother
Very helpful..... U did a lot of hard-work for us.... Thnk u so much sir🙌🙌🙏🙏..... And ur way of teaching is very good that is form basic
Both imputing and dropping missing values (NaN) is not a good practice with real world data. The ideal way is to derive a new field indicating missing values. 1 for missing else 0. because, sometimes missing value can be a new information in itself.
just sharing some learning from my job :)
Hi please do you mind sharing how to do that here. Or can I reach you via email?
Yes, it depends upon the dataset and problem you want to solve. In this case, dropping the null value is the best possible option in my opinion.
sir i really liked your video.. but according to road map video, you asked us to watch python 1-24 lectures first..in this eda concept, you have mentioned some new words like get_dummies, and few other new words.. stuck with the last 10 mins explaination.. else everything is really clear and understandable.. thanks for all the efforts...
Get dummy are use in pandas
It is basically one - hot encoding..
Encoding techniques are used to convert categorical data into numerical data
Since it is applied on 'Embarked' column
ua-cam.com/video/OTPz5plKb40/v-deo.html
loved your video , far better than the uni teachers :P
Thank you for providing knowledge in a simple way.
awesome explain ...........I really can't stop myself to comment on this video...……...on of the grt video on data visualization
Awesome tutorial on Exploratory Data Analysis ❤️❤️
Nice work Mr. Krish...... It's really helpful
I have seen some of your videos, excellent work. I really appreciate your work Mr. Krish Naik.
@Krish You are doing an amazing job.
This video is amazing. Thanks so much for sharing your wealth of knowledge.
Can u please make a video on treating the outliers, this will help us a lot in solving the problems
this was really one of the most usefull stuff avialable !!!!!!!!!!!!!!!
Very nicely explained. Awesome
this is beyond amazing....amazing place to learn and to revise the impn techniques
great video :)
i have a suggestion
we can drop PassengerId to increase the accuracy score because it doesn't contribute to the dependent variable
@naveen rawat
There is a error comes when I want to use sns.countplot. And the error is "could not interpret input 'survived' "
@@tusharmahuri2439show me the line of code
very simple and lucid videos. it encourages me to practice along not getting into too many details.. at the beginning..Superb and stay blessed👏
one ques: at timestamp 27.09, from the code without mentioning dtype=int, the ouptut displays bollean value in integer form. but in my case it shows as 'true' or 'False' unless dtype is specifically mentioned as int. IS this something to do with python updates?
Great work sir, learning a lot from your videos, please upload more videos on EDA..
one note, in boxplot the middle line inside the box is median value, not the mean value
Krish - Thank you for the EDA,
Throw some light on Story Telling - If you had to conclude the EDA, Theorotically, In lay man terms - we must do the story telling- Correct me If I am wrong .
Thanks a lot for the very detailed lesson Sir.. that was really fruitful and helped me complete one of my project. Thanks a ton..
wonderful explaination
Very nice one thank you very much for sharing valuable information
Thankyou sir it is very helpful 😊.
Superb explanations..
And interesting to learning
there is another null left in embarked column in 831st entry. it still shows in the heatmap, while in the video this doesn't show.(25:07)
and if I continue this path, do I apply the same method of removing age nulls(defining a class) or should I just replace the average value directly by redefining the index of the null(as it is just a single cell)?
Great one Krish. Basically covers most of the things a beginner needs to understand.
Great to understand. thanks alot
@Krish : Arrange your Complete ML playlist videos into a roadmap playlist, from start to end : to data scientist
Great work sir!!👍🏻👍🏻
3:35 the add🫠💀
great job sir, please do make more such videos for practising for beginners .
Sir play list is best
But please share the link from which u downloaded dataset fir every vedio
So that we can do what u explained in vegio
Thank you Krish.
Is there a part 2 and 3 for this video, about feature engineering on the same dataset?
that notification in the 3:39 part 🤣🤣😂😂
sir how you get to know the age age has relation with pclass (how and which analysis you did?)
@Vinayak sharma you can relate any column with any other column.
You could do a heat map of all features and get their correlation according to which you can know which feature is dependent on what
kind of fantastic video bro, but it needs 2-3x watch for crystal clear understanding.
Great video Sir, I just have two doubts that why did you not use get_dummies on "Pclass" as it was also categorical data.. and second why did you not normalize the "Fare" and "Age" Columns as their values are might over power the results?
Same doubt bro
If you type "train.info( )" you will see thae dtypes of all the columns. I don't know if this might help or not but get_dummies( ) can be used for objects only i think as they do not represent any numerical value for the system to compute get_dummies( ) changes indicates those objects into numerical values. Please correct me if i am wrong as i am also confused about this if you agree or have a different insight on this please tell me so.
Hi Krish,
Please create some more videos on EDA, it will be helpful.
Pretty nice explanation !
Another great video very useful one bro like NLP.. 📍
Hi, Enable auto subtitle, It helps a lot.
Thank you.
hey @Krish! Should we do this data visualization for each and every column? or we do it after feature selection? if we are supposed t do for each column, wouldn't the code get to big and complex for data with hundreds or thousands of features?
It was a resourceful video.
But why EDA is done before train-test split ?
sir im confuse coz we are predicting survival so it is 0 and 1 which means means its a categorical data and we r solving with regression
Thanks for the detailed video. Really helpful :)
Really helpful, Thank you soo much.
Thanks a lot Kris. EDA was well explained. I could not understand the last part starting from confusion matrix and how to read the final result of the analysis?
I didnt understood why categorical features disappeared in training data for logistic regression
when i try to apply my functinon (23:20)it is showing unexpected EOF while parsing
Please upload video related time series analysis
I like the video, but how did you know exactly the graphical representation to use, i mean why countplot why not jointplot? Why line plot not boxplot?
I hope you really understand my questions sir
16:15 now we have displot() ---- [without t]
One correction Sir-- In the boxplot, them middle line is the median(50% percentile). Thank you
loving the playlist :)))))
My doubt is
When u are apply that 'Age' and 'PClass' apply function ,but in that what is the use of axis=1. Could u plz explain that.
i have a question why is he not using SimpleImputer class from scikit learn
instead of finding the realtion to make the nan values having some values
we can easily do it through sklearn module
and also why isnt he using label encoder for binary values ???
20:20 hey, uhmm.. 50% percentile gives us MEDIAN of the age of people with 1st class... So we are using MEDIAN value instead of MEAN right?
Very helpful video for me to understand EDA
You're right, 50%ile is the median. I think you should check out the definition of median and percentiles on this page - www.statisticshowto.com/probability-and-statistics/percentiles-rank-range/#:~:text=The%2050th%20percentile%20is%20generally,quartiles%20is%20the%20interquartile%20range.
That should clear your doubt.
very helpful for beginners
Instead of mayplot lib and seaborn can we use powerbi
Hi krish,
You didn't drop the passenger ID column before fit the logistic regression model cause it doesn't contain any information.
@Krish Naik what is that test size =0.30 why did u use that .from beginiinng of video everything was very good but in the end i couldn't understand x train ytrain test size whats that accuracy 0.7190 etc. please tell me sir else your efforts will go waste ...
Are you sure that is average in boxplot near 20th mintue? Because when we talk about percentile then 50%ile should be median.
Thanks Krish
wish..... Jack and Rose could also see this data analysis
Sir, what is the need to visualise the data in this problem. You haven't use any analysis extracted from the visualisation to get help out in data cleaning.
@krish Naik we drop the age column then how again age column occur
I understood till splitting training and output data. But From there Logistical regression application and confusion matrix application is very difficult to understand. I found theoretical explanation of Logistic regression but the code and explanation of syntax and its application videos is not found. Could anyone help with links to understand these two concepts. Thankss
The middle line in box plot is not average but it's a median.
How do you know for one kind of result, which plot to use exactly?
Everytime, I import data it shows error "file not found"
import pandas as pd
data=pd.read_csv('C:\Users\Siddhi Singh\Desktop\Iris.csv')
print(data)
Actually you should reset the laptop because if any file found in name of panda means error willl be encountered and in the other case you should download and upload in jupyter notebook and in that jupyter notebook you should copy the path...
U can try using Google collab
@Krish Naik : Hi Krish, could you please explain why Age assigned cols[0] and Pclass cols[1],??I have not understood this
Sir, I didn't get why you compensated the missing value of age with the average age of Pclass?
Can't we simply replace the NaN values with the median values of the age column as: train['Age']=train['Age'].fillna(train['Age'].median())
In practical reality, every person has an age value but that data is missing for some people in the titanic dataset. Our goal is not just to fill in any random age where the age is missing but to fill in an educated guess/ estimate of the missing age of a person so that it can be a close representative of the true ages of those people. Of course, like you mentioned, the median of the entire age column could be used as an estimate but would that be a good representative value for ALL missing ages? Some people would have ages far above or below the median age. So on further exploration we notice that the median age for each Passenger Class is different, which would mean that in reality, people from a certain p-class would more likely be of a certain age, than someone who belongs to another p-class. And this difference is considerable (37 vs 29 vs 24). So by using using p-class to estimate age, we're just making a more educated guess for missing age values. You could of course go several steps further and consider other factors (like maybe SibSp, Parch etc.) in order to get a higher probability age value.
Sir, we can only use seaborn for inbuilt datasets available in seaborn? After data cleaning i am unable to use seaborn please help me
There is a error comes when I want to use sns.countplot. And the error is "could not interpret input 'survived' "
A very good about EDA but one thing i must mention that you didnt even touch the outliers concept. Its the major part of EDA and honestly i take this video only for outliers . But didnt find .
totally unrelated to the topic but how does your taskbar look like that
for me lbfgs is failed to converge on the local minima . How to fix it. i believe more categories need to be labeled like Pclass and standard scaler is required for age and fare .
Why do we need to get dummy variables for binary class variables like Sex and Embark, and why didn't we treat the variable pclass with One-hot-encoding, is it because we are treating it as ordinal, but wouldn't it cause problems with linear-regression and DNN algorithms to apply over it? Let me know Sir. Thanks.
I am getting key error after executing the following code:
sns.distplot(train['Age'].dropna(),kde=False,color='darkred',bins=40)
Any suggestion/idea as to what is to be done to stop getting this error?
Me to facing same error... Have you eliminate this error
why did you decide to analyse age with respect to Pclass in the missing value stage ?
Hi Can anyone help me with difference beyween EDA of this Titanic Dataset and EDA of Housing Price Prediction. Both follow a Different Steps. Iam quite Confused. Will Really appreciate any help.
Best explanation
Krish. Can you explain while data cleaning, why the passenger class is compared with Age and not any other columns. Big doubt of mine
did you get the answer?
Sir, how to apply median polish algorithm on image dataset using python