Logistic Regression with R: Categorical Response Variable at Two Levels (2018)
Вставка
- Опубліковано 7 вер 2024
- Provides an example of student college application for carrying out logistic regression analysis with R. This is among top 10 must know machine learning methods.
R code and data files: github.com/bkr...
Machine Learning videos: goo.gl/WHHqWP
Includes,
- use of a categorical binary output variable
- data partition
- logistic regression model
- prediction
- equation for prediction
- misclassification errors for training and test data
- confusion matrix for training and test data
- goodness-of-fit test
Machine Learning videos: goo.gl/WHHqWP
Becoming Data Scientist: goo.gl/JWyyQc
Introductory R Videos: goo.gl/NZ55SJ
Deep Learning with TensorFlow: goo.gl/5VtSuC
Image Analysis & Classification: goo.gl/Md3fMi
Text mining: goo.gl/7FJGmd
Data Visualization: goo.gl/Q7Q2A8
Playlist: goo.gl/iwbhnE
R is a free software environment for statistical computing and graphics, and is widely used by both academia and industry. R software works on both Windows and Mac-OS. It was ranked no. 1 in a KDnuggets poll on top languages for analytics, data mining, and data science. RStudio is a user friendly environment for R that has become popular.
2:46 two way table of factor variables 3:23 Data Partition 5:21 logistic regression Model,8:36 prediction 10:05 probability calculation 17:32 error test data 14:24 Interpretation of coefficinet,18:28 goodness of fit 15:03 error training data 16:04 confusion matrix
Thanks!
No words to express my gratitude for you. Found your channel days before submitting the project and you saved me !
Great to hear!
Finally I've got a perfect video on this topic.
Thanks for comments!
DR. Bharatendra Rai that video was amazing!! I hope you continue to post more videos like this! seriously amazing!!!!!!!!!!!!!!!
Thanks for comments!
Extremely crisp and accurate! Hope you get many more views! By far the best on this topic...
Thanks for the comments!
the best teaching of logistic regression!!!! Thanks a lot
Most welcome!
Thank you Mr. Dr. Bharatendra - your stuff and method are on top of youtube, greets from Europe!
Most welcome!
Thanks mate. Been struggling to find a practical person to show how to do this. Very clear and well thought out. Thank you.
You're very welcome!
Thank you for helping save my grades for this module!!!!! I might just watch all your videos because they're so helpful!!!!
Glad to hear it!
Thank you so much. Excellent video. I was really thinking I would fail my assignment until I found this.
You're very welcome!
Thanks. You gave a clear and concise explanation and a bonus was that it was in R which I am learning.
You're very welcome!
You may also find this useful:
ua-cam.com/play/PL34t5iLfZddvv-L5iFFpd_P1jy_7ElWMG.html
Wonderful video.. I was struggling to calculate the probablity from estimate in notebook but you made it quite simple.
Thanks a lot
Thanks for comments!
Thanks you very much Dr. Bharatendra. I was looking to solve some of my doubts and I finally solved them. Thanks for sharing your knowledge. I wish I could have the opportunity to help you in some occasion. Thanks for all, great job.
Thanks for your comments and feedback!
Best tutorial on logistic regression. Thank you so much for sharing.
You're very welcome!
Thank you Dr Rai for the wonderful explanation👍 God bless you 🙏
Welcome!
Excellent video. Just few things to mention. In glm result, residual deviance is greater than residual degree of freedom that means the data has overdispersion. Better to use quasibinomial function rather than binomial. Other wise p value would show false significance level.
Second thing to mention backward variable selection without montecarlo permutation has type2 error therefore better to use it cautiously or use Information theoretic approach proposed by Burnham etal with model weight as a criterion.
Thanks for this beautiful video sir
Although you created seed and resample which can reduce the error but it is extremely difficult to find proper seed size without understanding model weight (wi). Thanks
Thanks for the feedback and comments!
Very informative video and explain it in a manner that easy to understand
I have a question though , what is the difference between logistic regression and multinomial logistic regression ?
response variable has more than 2 levels in multinomial. See this for details:
ua-cam.com/video/ftjNuPkPQB4/v-deo.html
Thanks man, again other amazing job, u r the teacher we all want at univ.
Thanks for comments!
Amazing video to understand the logistic regression concepts thoroughly !!!
Thanks for comments!
Excellent video Sir. You are a great statistician and expert in R. Thank you for the video Sir
Thanks!
Thank you for sharing, very helpful
You are so welcome!
Very well explained!
Thanks for comments!
Thank you Dr. Bharatendra Rai. Can you explain more why Rank 1 is not included in the model, please?
For factor independent variables, we covert them to dummy variables. For more detailed coverage see:
ua-cam.com/video/s23CMIjfwHk/v-deo.html
thanks so much. very precise and concise explanations. Thank you Sir.
You are very welcome!
This was amazing... you explain everything step by step nicely :-)
Thanks for your feedback!
Very clear explanation. Understand all things
Thanks for comments!
@@bkrai sir, I am working on project on Real estate and banking model to predict prizes of house, could you plz help me on that?
I love your videos .... concise and to the point. Superb .... keep it up
Thanks for comments!
Love You Sir Very Useful videos
Thanks for comments! For recent Python video, see this link:
ua-cam.com/video/mKb5hRJmtCU/v-deo.html
You are just amazing 👏. You made my life easier with the codes.
Happy to hear that!
Hi Bharatendra, I saw your linear regression video also. The explanation on results was fantastic. I got to learn new things. One query - when to use linear and when to use logistic regression? Thanks
When y variable is factor, logistic is used. For numeric y linear regression is used.
thanks :)
Amazing explanation, loved the way you went through with the code and how to proceed step by step.
I have a doubt with the pvalue calculation at the end. Can you explain a bit more the "with" command you used ? i couldn't understand the parameters used in that, interpretation of p-value is fine, but would like to know the use of the command so i can employ that in some places as well.
Thanks
you can run ?with in the console, it will give you all details and also examples.
wow wow wow wow wow wow wow wow
thank you so much sir
Thanks for comments!
Great explaination, sir, can you upload a video of logistic regression with more than 10 varaibles. it would be great help.
The process will work same with any number of variables.
Thank you sir for the wonderful video.
Sir, I have a doubt that I'm not getting value while running on the test dataset. Could you please help me out of this error.
It is showing ' all arguments must have same length '
Check your data again.
If you have two or more categorical variables which are strings, how do you decide which one to make a factor of 0 or 1. Like how do you assign them specific factors ?
Sir can you explain "goodness of fit test". What is df.null-df.residual, lower tail & why it is 'F'?
Thank You
When in RStudio, you can run ?glm. This will provide you with more details.
Excellent explanation... please make a video of Boosted Regression Tree model with R. Thank you sir.
Thanks for comments and suggestion! I've added it to my list.
Thank you!
You are welcome!
Excellent Video! Could you please guide how to fit panel logistic regression in R. I want to make confusion matrix / ROC curve using pglm library but could not find fitting probabilities in pglm library
Great Explanation.Thank you Sir!
Thanks for comments!
Thank you very much!
Your videos create high value.
Kind regards from Karlsruhe
Jonathan
Thanks for your comments!
Hi there, thanks for this. I don't like how R displays the results for factors with more than 2 level - is there any way to get output like SPSS (which supplies a single odds ratio, 95% CI and p-value for each variable in the model). I have tried both the logistic.display and exp() commands but they do not provide an overall value like this. Any ideas?
You can use the output and customize it.
Very clear explanation . Thank you . Do you have any more videos on logit regression ?
Thanks for comments! The link below also has multinomial logistic regression and other regression based methods.
ua-cam.com/play/PL34t5iLfZddtKNwFNic3HWNV2qMsQ9AjD.html
Professor, can you please comment on why in your previous video on logistic regression, you trained the model and predicted on the same data without splitting.
Just wanted to show mainly how to run logistic regression. But after getting feedback created this on which is more complete.
Sir plz make on Monte Carlo simulation R
Thanks for suggestion!
1.) Suppose we have categorical fields in our data. Is it mandatory to always change to numeric factors ?
2.) If the answer for question 1 is correct, then what if we have too many unique values in each category columns?
Let us take for example : I have a dataset of 100,000 records. There are a few columns with categorical data in it. Each of these categorical columns may have 1000 or more unique values. So if I convert them into factors, then "labels = c(1:1000 or more)".
Is this ok to do it this way?
3.) Is there a way to not convert categorical data into numeric values and still use them in the machine learning model?
4.) How do we deal with Date fields?
5.) The conversion of categorical variables into dummy variables --> should we do this in all cases or is this something we need to consider only if the unique values in the categorical fields are limited to a lesser number?
1. No , it is not mandotary to change. You can set family parameter as “binomial”.
2.Answered in no.1
3.Answered in no.1
4.Convert it into factor variables
5.Try to consider it in all cases.
Very well explained
Glad it was helpful!
I tried doing this on my model , splitted my dataset into training and testing data but after that m running into a problem with "OBJECT IS NOT A MATRIX" while m building the model.
Make sure data has data frame format.
Sir
Outstanding 👍✍️
Thanks!
sir can you please provide the code for testing accuracy of this example. I'm a new learner & i find it pretty interesting & simple by the way you teach.
It's in the description.
can anyone help? I got about NSP, but in the regression in appears only 2 rows which is suspect and pathological, but in my regression there is 4 lines like that. I think that it is suspect pathological and the other 2 what can it be?
For response more than 2 levels, you need to apply multinomial logistic. Here is the link:
ua-cam.com/play/PL34t5iLfZddvv-L5iFFpd_P1jy_7ElWMG.html
This is an awesome video Sir...thanks for uploading this!!
Thanks for comments!
Perfect Sir
Thanks!
Thank you for the video Sir.
If I were running a logistic regression with categorical predictor variables, should I change them to factors?
Yes.
Sir, thanks a lot!
Most welcome!
Sir when I uploaded a data set then it doesn't take all data ..it is leaving a few rows.
Please tell me how I can upload a dataset.
Thank you
How many rows your original data has?
Thank you sir keep up the good work ;)
You are welcome!
sir how can i use one data set for training and another different dataset(having similar variables like training set) for testing?
Hi,
Dear Dr.Bharatendra Rai
What are the best models for fitting the binary data? I know that the logistic regression model is one of the models.
What is the other model to make a comparison with the logistic model to find the best model?
I would be grateful if you could assist me with this.
I look forward to hearing from you soon
Best regards,
You can use tree based methods for comparison, especially random forest and extreme gradient boosting. See this link for details:
ua-cam.com/video/hCLKMiZBTrU/v-deo.html
@@bkrai Thank you, Prof.
Are these methods (tree-based methods) can be used for regression or classification? Since my concern is to do regression ( predict disease status). As I think that these methods are used only for classification. Kindly confirm. Best regards
It does both regression or classification. I have included examples for both regression and classification.
@@bkrai Thank you, Prof.
Excellent video Dr.!, I just have one question: Why it is necessary to do the data partition for the estimation?
It can help to avoid over fitting which happens when results are good with training data, but not so good on test data.
How is it possible to train and predict using the same "train" dataset? Doesn't it defeat the purpose of training the model using the "train" dataset and then testing the model using the "test" model?
Comparing it with test results helps to assess if there is over-fitting or not.
Thanks for your share. It is very helpfull
You are welcome!
Thanks a lot for the video, it helped me a lot. Would be great if you could plot these results from your video or at least write me please how would you do it.
Let me know what exactly you are looking to plot. Here results are simply summarized in the form of a confusion matrix.
I enjoyed your video, thank you! Can I get some clarity on why you used the "train" dataset in your prediction instead of "test"? dataset: ## p1
After 'train', I also use 'test'. Note that if you get good results with 'train' but not with 'test', it will suggest over-fitting problem.
@@bkrai Thanks for your response. Appreciated
Welcome!
thank you for great lecture
Most welcome!
Excellent explanation! How do you deal with ordinal and nominal categorical variables?
If response variable is ordinal, refer to this:
ua-cam.com/video/qkivJzjyHoA/v-deo.html
Hello Sir! Why did you choose rank as factor and not as ordered?
You are right, ordinal will be more correct.
why is the candidate 4 has probability of .129 and has a classication of being admitted 1?
That's a incorrect classification. This candidate was in reality accepted but the model predicts that the candidate should not be accepted.
Great video it really helped a lot. I have a question though can I use the same model if one of my categorical variables has in the two-way table that equal zero? If not is there any alternative? How can I solve this?
Let's say your categorical variable has 10 levels and the last one has frequency below 5. You can combine last two levels into one and then do the analysis.
Thank you very much! That might work :)
Sir, do we change intergers into factors if the variables are categorical even in Multinomial Logistic Regression or it is done only in Logistic Regression?
For Multinomial Logistic Regression you can refer to this:
ua-cam.com/video/S2rZp4L_nXo/v-deo.html
@@bkrai Sir actually I came to this video after watching your video on Multinomial Logistic Regression. But now I am confused if we should always change all categorical variables into factors or it just happens in logistic regression. Because in Multinomial Regression you changed only response variable into a factor.
For response variable I would say yes. But for others you can go case by case.
@@bkrai Thank you Sir
You are welcome!
Why was it important to xtab to check if any of the ranks have zero values? Why cant the model just do it's best with the data it has?
Predictions will not be stable for those categories where sample size is less than 5.
Great as always! What do you do when you have so many rows and variables that your computer can't compute the vector in R?
You can take a sample.
HI Sir Great video's and easy to learn topics. I have small doubt don't mind. before dividing data into train and test . we need to do null values removable , finding outliers, scaling, EDA, then sampling .... could you please please share if any video on linear regression or logistic with combination of these steps. because we need to check all above conditions to predict best output . I am bit confusion on finding outliers(or remove outliers) and null values removable and scaling (min max or z-score) . Please please share any video it will helpful to us. Thanks in advance .
sir, please make a video on K- Fold cross validation.
Thanks for the suggestion, I've added it to my list.
excellent work
Thanks for comments!
rank 2 was not significant why didn't you have created dummy variables for rank and remove rank 2 from your model? Please answer
Rank is only one variable. It can only be in or out as a whole.
Why is your convention to break the training and test data with sampling and replacement... wouldn't it be more indicative of the accuracy of your model if you used either sampling without replacement, or just split the data 80/20?
Note that the random sample stored in "ind" are simply 1s and 2s. They are not yet from the data that we are using. So when we do sampling 'with replacement', we are randomly picking 1s and 2s from 1 and 2. We are not doing it from our data.
@@bkrai I think I understand... you wanted to get a random training and test set, so you did that by generating 1's and 2's randomly, and then dividing up the original data accordingly. You could have just split the data 80/20, but then you risk having some sort of bias in the order, so you generate a random sample instead.
You are absolutely correct!
In Logistic regression, how to check the linear relationship between the logit of outcome and each predictor values
That's not needed.
@@bkrai Linear relationship between the logit of outcome and each predictor values.
If this condition is not met, logistic regression is invalid
log〖𝑝/(1−𝑝)〗=𝑏0+𝑏1 ∗𝑋
I read in almost many article. If possible can you explain for this case study
One quick question: model 1 : a is the output variable, b and c are covariates, and both have significant p value. Model 2 : same output variable, b and d are covariates, and both have significant p values after we run the summary command. Finally Model 3: same output variable and all three b, c, d are covariates. Here if we see that only b and c are significant, but d doesn't have a significant p value , - then how do you interpret the result ? Can we say that adding covariate d doesn't add value to the model , even though it was significant in the previous bivariate scenario? Thank you.
Check relationship between c and d, that may help clarify.
You sir are a god send. i love you.
Thanks for your comments!
Absolutely brilliant thank you
Thanks for comments!
Dr. Bharatendra Rai could you please make videos on Machine learning using python
Thanks for the suggestion, I'll work on this.
Hi Sir,
i like your video and very useful for learning thanks for making video,
i have a question can you make a video on credit card data set i want to analyze customer who will better spend on category like online, dinning ,etc, can you predict dinning spend ,
thanks i hope you will post video soon i am waiting your video
Thanks for the suggestion! Send me a link to this dataset.
@@bkrai data.world/ash018/credit-card-transaction. This is data set I want to analysis customer who will spend more on dining
Sir i need robust regression using r..can u please post the next video for robust regression
Thanks for the suggestion, I've added it to my list.
What about the case when we have a lot of independent variables that have zero as a response or missing values?
For missing values refer to this link:
ua-cam.com/video/An7nPLJ0fsg/v-deo.html
hi sir. do you have a code for cross validation? thank you.
refer to this for various ways to use CV:
ua-cam.com/video/GmkHvDs0GG8/v-deo.html
Sir in my data the rank value is not displayed..wht is the reason!!
Rank should have values 1. W. 3 and 4.
Hi Sir, I am analyzing the data based on the traffic survey. I have Age, Gender, TripDist, TravelMode, TravelTime, DepartureTime, LectureTime information. What is the meaning of factor and margin in regression Modelling. can you help me in that. Thanks in advance
'factor' is another name for a categorical or qualitative variable.
Excellent explanation sr
Thanks!
Hello , Before u remove gre residual deviance was 369.99 when u rerun the model without gre it became 371.81 I mean increased , PLease in this case we should not keep gre even its not significant ? or the value change is negligible
That change is negligible. When a variable is not statistically significant, we should remove it.
@@bkrai thank you Boss
welcome!
@@bkrai if we have only one Y result ( not 2 as this example ) which Family type we must choose ?
If Y has only one value then that doesn't need a classification model.
Can u plz explain . what would xtab does ?
It is for cross tabulation or for making a 2-way table shown in the example.
Thanks for this video sir...
Kindly tell how can we increase the accuracy of this model...as error rate is quite high..
You can try other methods to improve accuracy:
ua-cam.com/play/PL34t5iLfZddsQ0NzMFszGduj3jE8UFm4O.html
Could you please explain set.seed(1234) what does it do and what is the importance of value we give in it, what if i give 100 or 250 please explain
Also please explain me last line what have we done here
# Godness of fit test
with(mymodel,pchisq(null.deviance-deviance,df.null-df.residual,lower.tail = F))
set.seed is used for getting same samples in training and test data. You can use any number you like and with that number you will get same samples in training and test data. And last line is to check if the model is statistically significant or not.
Could you please explain the importance of "xtabs" command in logistic regression? You said we should not get zero. Could you explain more on this.
Key idea is to have sufficient number of samples in each cell. If there are too few or zero samples, then the prediction model may not be stable or consistent.
@@bkrai Thank you for this video. It is very concise and understandable. I'd like expand on this question slightly. If you did have a zero value in the xtab, what would have been the appropriate course of action?
Thank sir
You are welcome!
Hi Sir , i have a retail train data set where i need to predict if a store should be opened or not in a respective location. I removed NAs from the train set , trying to apply glm function ( store~. , data=train, family='binomial' ) .. even after waiting 5-10 min i dont get any output .. the data set consist of character , int columns.
You will have to look at the structure of your data and make sure response variable is of factor type.
Brilliant video professor. I have question... so it is always that you convert categorical integer variables into factor variables before performing logistic regression? At the other places, like the algorithm XGB, I haven't seen you convert 'Admit' variable into a factor variable, why is it so? Thanks.
Different methods require data to be prepared in certain way. For example, XGB and neural networks require response to have numeric format.
Thanks professor for the quick response. Really appreciate. 😀
Thanks!
RANK2 -0.570976 0.358273 -1.594 0.111005 is not significant in the first model, why it not removed from the model?
Since rank is a single variable with 4 different levels, we need to keep it in model even if just one of the levels comes significant.
@@bkrai Thank you
welcome!
Great video. rank is a factor variable and looks like logistic regression has auto converted that in to dummy variables internally (from the summary model). Is there a way to find which algorithms auto converts categorical variable to dummy variables automatically and the ones one has to convert manually? Thank you for your help
Many algorithms do not need conversion of categorical variables to dummy variables. However, when using regression-based methods, R does so automatically.
Hi sir,
Please explain the use of type='response' in line number 23
Thanks
The type="response" option tells R to output probabilities of the form P(Y = 1|X), as opposed to other information such as the logit.
I have a question sir..Should i check multicollinearity (Vif) while performing the logistic regression? If any of the variable's vif value is greater than 2 then i will remove this variable from my model. Can i do that?
Yes, you should be able to do it.