Thanks sooooooo much! This is the only video I found explained the details of generating QQ plot and also make the concept so clear and easy to understand!
@@jameswhitaker4357 My stats courses were all pretty terrible, so you never really know what you're going to get. I had to teach myself statistics, and these videos are how I taught myself.
@@statquest That's what I'm going through right now! I've been using your videos and a "Intro to Statistical Learning with Applications in R" textbook which has helped a lot. I think when I saw terms like "heteroscedasticity" or the crazy formulas I would get scared and put off the studying, until I took a course that required knowing it LOL. And luckily most of these statistical tests and concepts are now pretty easy to perform in programming. Cheers!
I really appreciate from your very easy way explanation. I faced with so difficult and rough terminologies that I could not even understand the meaning of them.
Hi!!! Great video!!!! It was very helpful to understand Q-Q Plots!!!! But just one question, how do you calculate the quantiles for your dataset?? I mean, the first observation of your dataset is 0.6, but I don't understand why, since the first observation leaves 0 observations on one of its sides. Should the quantile be 0? In the video where you explain how to calculate quantiles, you explained that the quantile for each observation is calculated dividing the number of observations that this value leaves below between the total number of observations... So, for the first point... 0/15 = 0. Why 0.6??
I think I see the confusion here. The x and y-axes on the QQ-plot (on the right side) are labeled "Normal Quantiles" and "Data Quantiles". This is a little misleading - what we are plotting are the values at each quantile, not the quantile name itself. So if the first quantile is called "quantile 0", but it represents -1.5 in the normal distribution and 0.6 in the data, then we draw a dot at -1.5, 0.6 to represent the first quantile. Does that make sense?
@@statquest Perfectly. I understood this after watching some more videos. I would suggest you to clarify this if you make a new version!! As I already told you, congratulations for your videos and of course, your quick reply!! You explain really well, and the videos are perfect (easy to follow and to understand). I'm doing my Ph.D and it is really helpful people like you. Thanks a lot.
@@statquest Thank you very much for all your videos! They help me a lot. Just a follow-up question on this: how can we decide where to start as smallest quantile value in the theoretical distribution? Like you mentioned, "quantile 0" value in the sample distribution is 0.6, but how can it represent -1.5 in the normal distribution? My confusion is normal distribution doesn't technically have "quantile 0" value because it's infinity on the both tails.
@@Fan-fb4tz On the left side the first quantile is defined for the first point of 15 data points, meaning that 1/15 of the data is equal to or less than that point. Thus, we find the corresponding point on the normal curve such that 1/15th of the area under the curve is to the left of it.
@@statquest strictly speaking, the 15 lines(15 data points) divide the whole data into 16 equal groups or parts，So corresponding to normal distribution should be divided into 16 bins so that every bin has the same probability of 1/16 ,right?
Step 3 is not very clear. How do you put the lines on the normal distribution. How do you start putting the lines ? and How about the distance between each two lines ?
The quantiles for the normal distribution divide it so that the area under the curve between two lines is equal for all of the divisions. Since the normal distribution isn't as tall on the edges, there is more space between lines then in the middle, where the distribution is tall. Thus, the spacing between lines makes the area under the curve between the middle two lines is the same as the area under the curve between lines on the edges.
Thanks! It is easy to see that in the case of Uniform distribution. How about the starting point ? I think it's randomly that you started by -1.5 but I can start from -2 or -1 or ..
The starting point is defined by the need for each unit between lines to have the same exact area under the curve. To understand what this means, imagine you had to divide a normal distribution with a single line so that 50% of the area under the curve was on the left side of the line and 50% of the area under the curve was on the right side of the line. Where would you draw that line? Well, there is only one choice - right down the middle of the normal curve. If you drew it anywhere else, there would either be more area under the curve on the left side or the right side. Now imagine you had to divide the area under the curve into 4 equal amounts. Again, there is is only one option - you put a line in middle, and then you put another line so that the area under the curve on the left side is divided in half and then a third line so that the area under the curve on the left side is divided in half. Any other locations for those lines will result in the areas under the curve not being equal to each other. Thus, in this example, we have no choice about where to put the lines - they have to be put in the one configuration that makes the area under the curve between every pair of lines equal.
And again, thank you for another amazing video ! A little question : most of the points have to fit in the straight line for the data to be considered as normally distributed and at 4:15 you said it is not the case. Althought the intersection points are really close to the line, it does not matter, most of the point have to be strictly ON the line, right ? The fact that other intersection points are close or far from the line does not give any relevant information ?
I'm not sure I understand your question. For more details on how to interpret QQ-plots, see: stats.stackexchange.com/questions/101274/how-to-interpret-a-qq-plot
I always wondered how statisticians choose a distribution to which to fit the data when eyeballing it is insufficient. Now I know the answer: QQ-plots. Thank you for this!
@@bharathkumar5870 I'm not sure I understand your question. Are you asking, "why don't we just create a histogram with the data and see if the histogram looks like a normal distribution"? If so, histograms can be very tricky in terms of selecting the correct bin size. In contrast, with a q-q plot we don't have to worry about optimizing a bin size or anything else.
is there a statistical test we can do to determine how far away the dots are allowed to deviate, rather than just eyeballing it? Or is eyeballing good enough? I.e. a stat test that could say 'the chance of these 2 distributions being the same is less than X%
The "K-S Test" is what you want. However, it is very strict and tends to reject the null too easily. It's one of the few statistical tests where a large p-value (suggesting no difference) is more convincing than a small one. en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
@@statquest How were the lines drawn? Least Squares? Maybe doing R^2 calculations can provide an idea? Still trying to grasp my statistics a bit better :( :)
Hi Josh, can we use percentiles in place of quantiles to plot QQ plot ? If so, in case of percentiles we can only have upto hundred percentile no matter how big our data is then how to have a definitive answer whether or not the 2 datasets have similar distributions as mention in the video at 6:30 ?
The terms "quantiles" and "percentiles" are often used interchangeably, and in this case you can swap out quantiles for percentiles. And you can have as many percentiles as you want - however, the largest percentile is always 100. For example, you could have the 0.5 percentile, or the 1.23 percentile.
Dear Josh, thank you for this informative video. I have a one question: we are dividing the gene expression data into fifteen and also dividing the normal curve into fifteen. So for example lets take the 3rd data point: 1.9. It is the 3/15 percentile which is 0.2 So in the normal curve when we look at the z-table it should have been -0.84. In your calculations, I am observing that it is -0.89 which is actually 0.1867 percentile in the z-table. Am I missing something?
If we want 15 equal sized portions of the normal curve, then, we actually need 16 slices, the extra slice is at positive infinity. This makes it so that the area under the curve to the left of the first vertical bar in the graph is equal to the area under the curve between the 1st and 2nd vertical bar which is equal to the area under the curve between the 2nd and 3rd vertical bar (etc. etc. etc.) So when we do 3/16, we get 0.1875, and the normal quantile for that is -0.8871
thank you very much for this video. Please a have a few questions to ask. 1. From your previous video on quantile and percentile the first line was 0% quantile, why does it have a value of 0.6 in this video? 2. How are you getting the values for the x- axis, and why did it have to range from -2 to 2? Thank you
1) In this video we are plotting the actual values on the y-axis, rather than their quantiles. 2) The values come from a standard normal distribution (a "standard" normal distribution is a normal distribution with mean = 0 and standard deviation = 1). There are excel functions that will generate the x-axis coordinates from a standard normal distribution for you.
@@statquest quick question to 2:04 in your video: if we have 15 data points and we divide the dataset into 15 quantiles, then shouldn't the smallest quantile be 0.06666 so around 0.07? because in your video you are saying that it is 0.7, which would mean, that 70% of all data is covered by just one datapoint. Thank you for your video :)
I know it may sound dumb but i just got it when i understood that theoretical quantiles were the quantiles of a normal standard distribution or Z-value.
Hi, thanks for video but could you please add in the description wtf is gene expression and what mean x and y on the 0:37 graph. For the time being I see 15 data points and no idea why they are shown in this way, thank you. Maybe some simpler example instead of gene expression?
OK. I added this to the description: NOTE: The data in this video are measures of gene expression. If "gene expression" doesn't mean anything to you, just imagine that the data represents how tall a bunch of people are, or how much they weigh. Then consider the y-axis to be the height or weight of the people, and the x-axis just represents all of the data you collected on a single day. In this case, all of the data were collected on the same day, so they form a single column.
thanks for the explanation, can you clarify this please?: if we have 15 quantiles, then I thought you should plot 14 red lines in the normal distribution and the 15th line should reside in +infinite. and a little question: is the straight line generated by linear regression?
Thanks for the great explanation as always! So QQ is just a way to plot and visualize the similarity of two distributions? Are there any other scenarios when these can be used? Thanks!!
Hi Josh, Amazing Video there :) Just want to understand the intuition behind the working of QQ plots ? Is it the fact that quantiles of every normal distribution are just scaled up values of a standard normal distribution and that is why we expect a straight line ?
@@statquest Thanks for the response ;) Would really appreciate if you could make something on the same or share some content that could explain the intuition behind QQ Plots.
What's the approach for determining which distribution has the best fit for the data? Would the r-squared of the data against the straight line be a suitable measure for how well the distribution describes the data?
This is a good question, and, to be honest, I'm not sure what the answer is. I like your idea, but it may oversimplify the problem. i.e. you could get a high R^squared value, but still have some real obvious problems if you looked at it visually.
That's true, and this is something I may have misstated in another comment. I think the standard thing to do is draw line that connects the first and third quantiles.
@@statquest Hi Josh, what do the first and third quartiles refer to specifically in terms of the (x,y) coordinates? Are you referring to the theoretical quantiles or the sample quantiles (or both) for the (x,y) coordinates the line?
@@jacksonh634 I believe it depends on what you have on the x and y axes. If you have sample quantiles on the y, and theoretical quantiles on the x, then you mix. If you have sample quantiles on the x and y, then you use sample quantiles for both.
You can get a computer program to figure this out for you. For example, the R programming language has a function that will give you all the quantiles. Presumably MS Excel has a similar function.
Two questions: 1. How does having more points at the middle make the histograms narrower and the opposite at the ends? If the histograms have more width doesn't that mean that more data points can be placed in that histogram ? 2. the last example you took 4 data points and said that the distribution has 4 quartiles. Did you say such because there are 4 points ?
1. You might want to review the concept of the normal distribution. ua-cam.com/video/rzFX5NWojp0/v-deo.html The hight of the curve indicates the likelihood of observing a point there. So the wider parts have lower likelihoods of observing points and the taller have a higher likelihood. To make each region have an equal probability of observing a point, we have to have wider regions where the likelihoods are lower and narrower regions where the likelihoods are higher. 2. yes.
The name of the channel should be "Dumbing Down Probability for Dummies". I don't know whether I like the intro song better or that simple explanation.
Personally, I like to think that rather than dumbing down the material, I bring people up so that they can understand the tools and techniques in data analysis. I think "dumbing down" suggests watering down the content, as if I am presenting a simplified version of how statistics really works. That's not what happens in my videos. This is the real deal. It's just explained in a way that is relatively easy to understand and that brings people up.
Mr. Starmer: I am trying to understand the values 0.6, 1.1, 1.9 etc. that you have for the points on the y-axis. Are these the "raw" or "observed" data or are these numbers derived from some calculation? And, these points have a corresponding values on the x-axis (-1.5, -1.2, -0.89 etc.). I am totally lost as to how these values were derived. I am trying to understand this video in the context of linear regression where I have seen "sample quantiles" plotted on y-axis against "theoretical quantiles" on y-axis but it is not clear as to how these values have been derived? I apologize if my question doesn't make sense - I am not sure as to how to word the question purely because of my ignorance of the topic. Thank you for any direction you may give.
Those are raw measurement values. The corresponding x-axis values come from asking a computer program (like 'R', or even excel) to give us the quantiles.
5:30 I am confused... I thought that quartiles are three (not four) values, which divide the dataset into four equal numbers of data points. In your example you say that four data points are quartiles? 🤔
Hi, I understand how the quantile points are plotted wrt observed vs theoretical distributions, what I don't understand is what determines the slope of the straight line. While this is fairly intutitive for a normal distribution, for say the Weibull distribution I am unclear how the slope of the striaght line is used to determine whether the observed vs theoretical quantiles are a good fit for a given distribution. Any ideas?
This video explained how to compare a discrete sample to a continuous normal, and to a discrete but smaller sample. What if we wanted to compare continuous sample vs continuous theoretical, is there an analogous qqplot for that? Or the term "continuous sample" is an oxymoron since "sample" means discrete already no matter sampled from theoretically discrete or continuous distribution?
In the example, the "discrete" values come from a continuous distribution, so there is no need to do anything special that is not described in this video.
Hi, thanks for these videos! I'm confused by the logic of what happens at 1:20, when you split the data using 15 lines that occur at the points themselves. Wouldn't the intuitive thing to do be to cut the _space_ so each point resides in its own piece, rather than to use the data (i.e., the demarcations) as the cut points? In other words, you need only 14 lines to slice this data space into portions that each contain a single point. When this is then repeated on the standard normal distribution curve, wouldn't we then want to slice that curve into n equal pieces using n-1 cuts (not make n cuts resulting in n+1 equal pieces)? Edit to add one more question. I feel like I mostly get it, but I'm fuzzy on the intuition that underpins why this all works. Is it enough for the line to be straight, or must it also be at a 45-degree angle? I see how we're graphically connecting the data to the standard normal distribution, but I'm not certain as to what the intuitive connection is. It feels like it has something to do with calculus---that we're trying to compare the rates of change between quantiles for both our data and the data in a known distribution to ensure that they are constant and in lockstep.
We put the lines on the points, because those are the known values we are working with. And the line only needs to be straight - the angle doesn't matter because it's a function of the scale that the data are on.
If, by visual inspection, we suspect that the distribution is D, how do we get the quantiles of D? I can imagine that, if the CDF of D is inversible (like the CDF of the exponential distribution, or a sigmoid) then we can compute quantiles. But what about quite arbitrary distributions ?
On invertible CDFs... Every CDF is nondecreasing, so there's a theorem in introductory real analysis that says there exists an inverse. There might not be a pretty way to write it down, but it exists, and at the very least I would guess that it could probably be estimated using numerical methods.
Josh, Great video... very helpful. It looks like you might have a slight error when comparing the 2 dataset distributions however. I could be wrong but I think your second plot is incorrect on the chart.
By the way, the long term plan is to correct/update these videos. Just like textbooks have "new and revised editions", I'd like to have new and revised additions of these videos - so your feedback is helpful and appreciated. I hope that once the channel grows, youtube will give me some options for how to release revised videos - right now I have no options, but I'm also relatively small potatoes. So I can't fix the video right now, but one day I will.
@@statquest No problem. Your viewers would certainly appreciate your easy to understand videos. I just wanted to check I understood and maybe other users will find the note in the comments useful.
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
I've been studying statistics for 4 years and this channel does the best job of explaining concepts than anyone I've been taught by.
Thank you!
that intro alone, made me forget my hate for statistics and instantly fall in love with it
Hooray!
Thanks sooooooo much! This is the only video I found explained the details of generating QQ plot and also make the concept so clear and easy to understand!
Thank you very much! :)
Haven't seen the video yet, but that intro earned you a subscription
It made me think that the whole video was going to be a song lol. Very interesting nonethless
I'am not suscribed for the plots, but for the music!
That intro hit me hard xD
Not gonna lie, stats is my super weak spot. You've helped me a lot in my Data Models course and interpreting my results. +1
Happy to help!
@@statquest Thank you! I'm just kicking myself for not taking more stats courses at this point!
@@jameswhitaker4357 My stats courses were all pretty terrible, so you never really know what you're going to get. I had to teach myself statistics, and these videos are how I taught myself.
@@statquest That's what I'm going through right now! I've been using your videos and a "Intro to Statistical Learning with Applications in R" textbook which has helped a lot. I think when I saw terms like "heteroscedasticity" or the crazy formulas I would get scared and put off the studying, until I took a course that required knowing it LOL. And luckily most of these statistical tests and concepts are now pretty easy to perform in programming. Cheers!
@@jameswhitaker4357 I actually wrote a little about heteroscedasticity. Maybe I should record it.
I was waiting for the "BAAM" all video long, got just a couple of great "HOORAY!".
Thank you for the awesome channel Josh!
You made me laugh. :)
Couldn’t have asked for more clear explanation, thanks!
Glad to help!
Triple BAMM! Serious man your channel is pure art. Thanks
Thank you!
I really appreciate from your very easy way explanation.
I faced with so difficult and rough terminologies that I could not even understand the meaning of them.
🤔🤔🤔🤔🤔 well I thought that q-q plot was difficult but thanks to you I got it now. thanks and keep it up!!!
This is very very cool, more likely to learn on UA-cam than in a classroom. Grazie
Glad it was helpful!
Explained like a pro.
Tripple BAM!!!
Thank you! :)
You had my like at the beginning with the jingle. Thanks for explaining this so well!!
Glad you liked it!
I could have better grades if i had faculties like you...thank you Josh!!
Thanks!
Thanks! It helps.
Wow!!! Thank you very much for your support! BAM! :)
Nice video. Explained everything in just under 7 mins. Awesome.
😄👍👍
bam!
@@statquest bam indeed. 😁
Best intro by far so far
Hooray!!!! :)
Best intro song, it can be used as a 'mnemonic' for what QQ plots are used for =)
Can you do a video on normality tests like shapiro wilk and anderson darling? If not anytime soon, can you share link to some good materials?
You made it very clear man !!! Great doing
Glad to hear that!
phenomenal explanation and really cool intro music man!
Thanks!
Thanks for all the videos! Great music BTW. Also I'm looking forward to rockin' my new SQ hoodie!
TRIPLE BAM! Thank you for your support!
How beautiful and simple is that explaination 🥳
Thank you!
Thank you for your help! Greetings from Brazil.
Muito obrigado! :)
wow, now I can clearly understand it ! thanks alot !
Hooray!!! :)
You simply saved my life
Bam!
very nicely explained. it was a tricky concept until this video! thanks!
Hooray! I'm glad the video helped. :)
Awesome video - thank you SO much for saving my sanity.
Thanks!
so clear, so good , so nuce thank you , Josh
Thanks!
Awesome video. Explained so clearly. Really helped me a lot!
Hooray! :)
I just noticed when you said "please subscribe" at the end of the video, the subscribe button lit up:)
bam! :)
Hi!!! Great video!!!! It was very helpful to understand Q-Q Plots!!!! But just one question, how do you calculate the quantiles for your dataset?? I mean, the first observation of your dataset is 0.6, but I don't understand why, since the first observation leaves 0 observations on one of its sides. Should the quantile be 0? In the video where you explain how to calculate quantiles, you explained that the quantile for each observation is calculated dividing the number of observations that this value leaves below between the total number of observations... So, for the first point... 0/15 = 0. Why 0.6??
I think I see the confusion here. The x and y-axes on the QQ-plot (on the right side) are labeled "Normal Quantiles" and "Data Quantiles". This is a little misleading - what we are plotting are the values at each quantile, not the quantile name itself. So if the first quantile is called "quantile 0", but it represents -1.5 in the normal distribution and 0.6 in the data, then we draw a dot at -1.5, 0.6 to represent the first quantile. Does that make sense?
@@statquest Perfectly. I understood this after watching some more videos. I would suggest you to clarify this if you make a new version!! As I already told you, congratulations for your videos and of course, your quick reply!! You explain really well, and the videos are perfect (easy to follow and to understand). I'm doing my Ph.D and it is really helpful people like you. Thanks a lot.
@@statquest Thank you very much for all your videos! They help me a lot. Just a follow-up question on this: how can we decide where to start as smallest quantile value in the theoretical distribution? Like you mentioned, "quantile 0" value in the sample distribution is 0.6, but how can it represent -1.5 in the normal distribution? My confusion is normal distribution doesn't technically have "quantile 0" value because it's infinity on the both tails.
@@Fan-fb4tz On the left side the first quantile is defined for the first point of 15 data points, meaning that 1/15 of the data is equal to or less than that point. Thus, we find the corresponding point on the normal curve such that 1/15th of the area under the curve is to the left of it.
@@statquest strictly speaking, the 15 lines(15 data points) divide the whole data into 16 equal groups or parts，So corresponding to normal distribution should be divided into 16 bins so that every bin has the same probability of 1/16 ,right?
Legendary explanation! Fantastic!
Thank you!
Thanks for this! Finally I understand
Hooray!
4:35 I think there is an extra quantile drawn on the Uniform distribution
You are correct. Thanks for spotting that.
Very well explained, thanks so much
Awesomely explained! Good job!
Thank you! :)
Step 3 is not very clear. How do you put the lines on the normal distribution. How do you start putting the lines ? and How about the distance between each two lines ?
The quantiles for the normal distribution divide it so that the area under the curve between two lines is equal for all of the divisions. Since the normal distribution isn't as tall on the edges, there is more space between lines then in the middle, where the distribution is tall. Thus, the spacing between lines makes the area under the curve between the middle two lines is the same as the area under the curve between lines on the edges.
Thanks! It is easy to see that in the case of Uniform distribution. How about the starting point ? I think it's randomly that you started by -1.5 but I can start from -2 or -1 or ..
The starting point is defined by the need for each unit between lines to have the same exact area under the curve. To understand what this means, imagine you had to divide a normal distribution with a single line so that 50% of the area under the curve was on the left side of the line and 50% of the area under the curve was on the right side of the line. Where would you draw that line? Well, there is only one choice - right down the middle of the normal curve. If you drew it anywhere else, there would either be more area under the curve on the left side or the right side. Now imagine you had to divide the area under the curve into 4 equal amounts. Again, there is is only one option - you put a line in middle, and then you put another line so that the area under the curve on the left side is divided in half and then a third line so that the area under the curve on the left side is divided in half. Any other locations for those lines will result in the areas under the curve not being equal to each other. Thus, in this example, we have no choice about where to put the lines - they have to be put in the one configuration that makes the area under the curve between every pair of lines equal.
Perfect! got it!
Hooray!!! :)
And again, thank you for another amazing video ! A little question : most of the points have to fit in the straight line for the data to be considered as normally distributed and at 4:15 you said it is not the case. Althought the intersection points are really close to the line, it does not matter, most of the point have to be strictly ON the line, right ? The fact that other intersection points are close or far from the line does not give any relevant information ?
I'm not sure I understand your question. For more details on how to interpret QQ-plots, see: stats.stackexchange.com/questions/101274/how-to-interpret-a-qq-plot
@@statquest ok thank you !
great video. a video on the intuition on why q-q plot works might be interesting.
I'll keep that in mind.
It just saved me, the person who did this => you're the best
Thank you! :)
Helpful and easy to undderstand.
Thanks!
Really excellent presentation, Josh. ⭐️
Thank you! :)
I always wondered how statisticians choose a distribution to which to fit the data when eyeballing it is insufficient. Now I know the answer: QQ-plots. Thank you for this!
bam! :)
Thank you for the video! It was short and easy to understand :)
Thanks! :)
Thank you! Awesome explanation
Thank you! :)
i have a doubt...why to use this method,instead just plot the points and see if it forms a bell curve....correct me
@@bharathkumar5870 I'm not sure I understand your question. Are you asking, "why don't we just create a histogram with the data and see if the histogram looks like a normal distribution"? If so, histograms can be very tricky in terms of selecting the correct bin size. In contrast, with a q-q plot we don't have to worry about optimizing a bin size or anything else.
@@statquest thank you sir ..u cleared my doubt. Different bins give different distributions😀
How helpful! Thanks a lot for your amazing videos
Thanks!
Your videos are awesome, thank you so much!
Glad you like them!
is there a statistical test we can do to determine how far away the dots are allowed to deviate, rather than just eyeballing it? Or is eyeballing good enough? I.e. a stat test that could say 'the chance of these 2 distributions being the same is less than X%
The "K-S Test" is what you want. However, it is very strict and tends to reject the null too easily. It's one of the few statistical tests where a large p-value (suggesting no difference) is more convincing than a small one. en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
@@statquest How were the lines drawn? Least Squares? Maybe doing R^2 calculations can provide an idea?
Still trying to grasp my statistics a bit better :( :)
Thanks a lot for the wonderful explanation!
Thank you!
Very well explained !
thank you!
very clear with great examples!
Thanks!
such an awesome video, thanks!
Glad you liked it!
Thanks for saving my life
Hooray! :)
Best Explanation ever!!! 🎉🎉🎉
Thanks!
Understand it now - thank you!
Hooray! :)
Hi Josh, can we use percentiles in place of quantiles to plot QQ plot ? If so, in case of percentiles we can only have upto hundred percentile no matter how big our data is then how to have a definitive answer whether or not the 2 datasets have similar distributions as mention in the video at 6:30 ?
The terms "quantiles" and "percentiles" are often used interchangeably, and in this case you can swap out quantiles for percentiles. And you can have as many percentiles as you want - however, the largest percentile is always 100. For example, you could have the 0.5 percentile, or the 1.23 percentile.
@@statquest Thanks Josh.
Dear Josh, thank you for this informative video. I have a one question: we are dividing the gene expression data into fifteen and also dividing the normal curve into fifteen. So for example lets take the 3rd data point: 1.9. It is the 3/15 percentile which is 0.2 So in the normal curve when we look at the z-table it should have been -0.84. In your calculations, I am observing that it is -0.89 which is actually 0.1867 percentile in the z-table. Am I missing something?
If we want 15 equal sized portions of the normal curve, then, we actually need 16 slices, the extra slice is at positive infinity. This makes it so that the area under the curve to the left of the first vertical bar in the graph is equal to the area under the curve between the 1st and 2nd vertical bar which is equal to the area under the curve between the 2nd and 3rd vertical bar (etc. etc. etc.) So when we do 3/16, we get 0.1875, and the normal quantile for that is -0.8871
@@statquest thank you very much! Now it seems clear
thank you very much for this video. Please a have a few questions to ask.
1. From your previous video on quantile and percentile the first line was 0% quantile, why does it have a value of 0.6 in this video?
2. How are you getting the values for the x- axis, and why did it have to range from -2 to 2? Thank you
1) In this video we are plotting the actual values on the y-axis, rather than their quantiles.
2) The values come from a standard normal distribution (a "standard" normal distribution is a normal distribution with mean = 0 and standard deviation = 1). There are excel functions that will generate the x-axis coordinates from a standard normal distribution for you.
du hast zerfetzt bro, danke
Bitte!!!
@@statquest quick question to 2:04 in your video:
if we have 15 data points and we divide the dataset into 15 quantiles, then shouldn't the smallest quantile be 0.06666 so around 0.07? because in your video you are saying that it is 0.7, which would mean, that 70% of all data is covered by just one datapoint. Thank you for your video :)
I know it may sound dumb but i just got it when i understood that theoretical quantiles were the quantiles of a normal standard distribution or Z-value.
bam! :)
i love statquest!
Hooray! :)
It helps a lot. Thanks!
Glad it helped!
Crytal clear explanation
Thanks! :)
It's a party time with Josh Starmer and his quantiles! 😆🤘 Party on, Wayne!
:)
Great explanation, have a nice day :)
Helped a lot! Thank you :D
Hooray! :)
QBUS?
It's so clear! Thanks a lot for your video.
Hi, thanks for video but could you please add in the description wtf is gene expression and what mean x and y on the 0:37 graph. For the time being I see 15 data points and no idea why they are shown in this way, thank you. Maybe some simpler example instead of gene expression?
OK. I added this to the description:
NOTE: The data in this video are measures of gene expression. If "gene expression" doesn't mean anything to you, just imagine that the data represents how tall a bunch of people are, or how much they weigh. Then consider the y-axis to be the height or weight of the people, and the x-axis just represents all of the data you collected on a single day. In this case, all of the data were collected on the same day, so they form a single column.
@@statquest thanks for your effort!
Best intro 🔥
bam!
Great explanation, thanks!
Thank you! :)
thanks for the explanation, can you clarify this please?: if we have 15 quantiles, then I thought you should plot 14 red lines in the normal distribution and the 15th line should reside in +infinite. and a little question: is the straight line generated by linear regression?
Plotting a line at infinity would be hard to do and you can fit the line with regression.
Your videos are cool and concise , thank you .
Thanks for the great explanation as always! So QQ is just a way to plot and visualize the similarity of two distributions? Are there any other scenarios when these can be used? Thanks!!
QQ is mostly used to check tail conditions. Density plots and cumulative plots are the best way to check distribution symmetry.
Hi Josh, Amazing Video there :) Just want to understand the intuition behind the working of QQ plots ? Is it the fact that quantiles of every normal distribution are just scaled up values of a standard normal distribution and that is why we expect a straight line ?
Pretty much
@@statquest Thanks for the response ;) Would really appreciate if you could make something on the same or share some content that could explain the intuition behind QQ Plots.
What's the approach for determining which distribution has the best fit for the data? Would the r-squared of the data against the straight line be a suitable measure for how well the distribution describes the data?
This is a good question, and, to be honest, I'm not sure what the answer is. I like your idea, but it may oversimplify the problem. i.e. you could get a high R^squared value, but still have some real obvious problems if you looked at it visually.
Sweet, its not that difficult to grasp anymore! Thanks
one confusion is drawing the straightline. you should draw the data quantiles's line plot
That's true, and this is something I may have misstated in another comment. I think the standard thing to do is draw line that connects the first and third quantiles.
@@statquest Hi Josh, what do the first and third quartiles refer to specifically in terms of the (x,y) coordinates? Are you referring to the theoretical quantiles or the sample quantiles (or both) for the (x,y) coordinates the line?
@@jacksonh634 I believe it depends on what you have on the x and y axes. If you have sample quantiles on the y, and theoretical quantiles on the x, then you mix. If you have sample quantiles on the x and y, then you use sample quantiles for both.
For all the other fans, the chords are Bm, D. You're welcome :)
That's awesome! :)
How do you know that the smallest quartile on the normal distribution is at -1.5?
You can get a computer program to figure this out for you. For example, the R programming language has a function that will give you all the quantiles. Presumably MS Excel has a similar function.
Actuary studies is something else bro
Noted!
Hoorray!
Thx for the video :)
BAM! :)
Two questions:
1. How does having more points at the middle make the histograms narrower and the opposite at the ends? If the histograms have more width doesn't that mean that more data points can be placed in that histogram ?
2. the last example you took 4 data points and said that the distribution has 4 quartiles. Did you say such because there are 4 points ?
1. You might want to review the concept of the normal distribution. ua-cam.com/video/rzFX5NWojp0/v-deo.html The hight of the curve indicates the likelihood of observing a point there. So the wider parts have lower likelihoods of observing points and the taller have a higher likelihood. To make each region have an equal probability of observing a point, we have to have wider regions where the likelihoods are lower and narrower regions where the likelihoods are higher.
2. yes.
Do I need to matter the exact size size or probability when dividing the contribution? Or just need to only make sure the sizes are equal?
Just equal sizes.
This was super helpful!
Thank you! :)
what a nice lecture!
Thanks! :)
The name of the channel should be "Dumbing Down Probability for Dummies". I don't know whether I like the intro song better or that simple explanation.
Personally, I like to think that rather than dumbing down the material, I bring people up so that they can understand the tools and techniques in data analysis. I think "dumbing down" suggests watering down the content, as if I am presenting a simplified version of how statistics really works. That's not what happens in my videos. This is the real deal. It's just explained in a way that is relatively easy to understand and that brings people up.
@@statquest Understood. The way you teach not only makes it comprehensible but also ensures it sticks to the head!
thanks , awesone , this is useful in measuring gene expression effect
Mr. Starmer: I am trying to understand the values 0.6, 1.1, 1.9 etc. that you have for the points on the y-axis. Are these the "raw" or "observed" data or are these numbers derived from some calculation? And, these points have a corresponding values on the x-axis (-1.5, -1.2, -0.89 etc.). I am totally lost as to how these values were derived. I am trying to understand this video in the context of linear regression where I have seen "sample quantiles" plotted on y-axis against "theoretical quantiles" on y-axis but it is not clear as to how these values have been derived? I apologize if my question doesn't make sense - I am not sure as to how to word the question purely because of my ignorance of the topic. Thank you for any direction you may give.
Those are raw measurement values. The corresponding x-axis values come from asking a computer program (like 'R', or even excel) to give us the quantiles.
the song alone is worth it
bam! :)
5:30
I am confused... I thought that quartiles are three (not four) values, which divide the dataset into four equal numbers of data points.
In your example you say that four data points are quartiles? 🤔
Oops. That's a mistake. Quartiles divide the data into 4 parts.
Hi, I understand how the quantile points are plotted wrt observed vs theoretical distributions, what I don't understand is what determines the slope of the straight line. While this is fairly intutitive for a normal distribution, for say the Weibull distribution I am unclear how the slope of the striaght line is used to determine whether the observed vs theoretical quantiles are a good fit for a given distribution. Any ideas?
I came here for the same question and left with no answer LOL
What is the slope of the red straight line we use? is it 1 or something else?
It's just a regression line, or best fitting line. The slope (and intercept) can be anything. Does that make sense?
@@statquest Yes! perfect sense! thank you!
This video explained how to compare a discrete sample to a continuous normal, and to a discrete but smaller sample. What if we wanted to compare continuous sample vs continuous theoretical, is there an analogous qqplot for that? Or the term "continuous sample" is an oxymoron since "sample" means discrete already no matter sampled from theoretically discrete or continuous distribution?
In the example, the "discrete" values come from a continuous distribution, so there is no need to do anything special that is not described in this video.
precious help, thank you
Thanks!
Hi, thanks for these videos! I'm confused by the logic of what happens at 1:20, when you split the data using 15 lines that occur at the points themselves. Wouldn't the intuitive thing to do be to cut the _space_ so each point resides in its own piece, rather than to use the data (i.e., the demarcations) as the cut points? In other words, you need only 14 lines to slice this data space into portions that each contain a single point. When this is then repeated on the standard normal distribution curve, wouldn't we then want to slice that curve into n equal pieces using n-1 cuts (not make n cuts resulting in n+1 equal pieces)?
Edit to add one more question. I feel like I mostly get it, but I'm fuzzy on the intuition that underpins why this all works. Is it enough for the line to be straight, or must it also be at a 45-degree angle? I see how we're graphically connecting the data to the standard normal distribution, but I'm not certain as to what the intuitive connection is. It feels like it has something to do with calculus---that we're trying to compare the rates of change between quantiles for both our data and the data in a known distribution to ensure that they are constant and in lockstep.
We put the lines on the points, because those are the known values we are working with. And the line only needs to be straight - the angle doesn't matter because it's a function of the scale that the data are on.
If, by visual inspection, we suspect that the distribution is D, how do we get the quantiles of D? I can imagine that, if the CDF of D is inversible (like the CDF of the exponential distribution, or a sigmoid) then we can compute quantiles. But what about quite arbitrary distributions ?
I'm not sure. Pretty much all of the standard distributions can be inverted.
On invertible CDFs... Every CDF is nondecreasing, so there's a theorem in introductory real analysis that says there exists an inverse. There might not be a pretty way to write it down, but it exists, and at the very least I would guess that it could probably be estimated using numerical methods.
Josh, Great video... very helpful. It looks like you might have a slight error when comparing the 2 dataset distributions however. I could be wrong but I think your second plot is incorrect on the chart.
I think I know what you're talking about. The second point should be at 5.1 on the x-axis but is only at 4.1. Is that it? That's a typo.
By the way, the long term plan is to correct/update these videos. Just like textbooks have "new and revised editions", I'd like to have new and revised additions of these videos - so your feedback is helpful and appreciated. I hope that once the channel grows, youtube will give me some options for how to release revised videos - right now I have no options, but I'm also relatively small potatoes. So I can't fix the video right now, but one day I will.
@@statquest Yes that's it.
@@statquest No problem. Your viewers would certainly appreciate your easy to understand videos. I just wanted to check I understood and maybe other users will find the note in the comments useful.
At 1:34, how do you determine which mean value should be used to create the normal distribution quantiles with?
You can use any mean the only difference is you'll have different labels on the x-axis will have different values.