Quantile-Quantile Plots (QQ plots), Clearly Explained!!!
Вставка
- Опубліковано 6 вер 2024
- Quantile-Quantile (QQ) plots are used to determine if data can be approximated by a statistical distribution. For example, you might collect some data and wonder if it is normally distributed. A QQ plot will help you answer that question. You can also use QQ plots to compare to different datasets that you collected to determine if their distributions are comparable. This video shows you how to do both things.
NOTE: The data in this video are measures of gene expression. If "gene expression" doesn't mean anything to you, just imagine that the data represents how tall a bunch of people are, or how much they weigh. Then consider the y-axis to be the height or weight of the people, and the x-axis just represents all of the data you collected on a single day. In this case, all of the data were collected on the same day, so they form a single column.
For a complete index of all the StatQuest videos, check out:
statquest.org/...
If you'd like to support StatQuest, please consider...
Buying The StatQuest Illustrated Guide to Machine Learning!!!
PDF - statquest.gumr...
Paperback - www.amazon.com...
Kindle eBook - www.amazon.com...
Patreon: / statquest
...or...
UA-cam Membership: / @statquest
...a cool StatQuest t-shirt or sweatshirt:
shop.spreadshi...
...buying one or two of my songs (or go large and get a whole album!)
joshuastarmer....
...or just donating to StatQuest!
www.paypal.me/...
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
Corrections:
4:35 The Uniform Distribution has one extra quantile
5:30 I should have said that Quartiles divide the data into 4 parts.
#statquest #quantile #qqplot
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
that intro alone, made me forget my hate for statistics and instantly fall in love with it
Hooray!
Thanks sooooooo much! This is the only video I found explained the details of generating QQ plot and also make the concept so clear and easy to understand!
Thank you very much! :)
Haven't seen the video yet, but that intro earned you a subscription
It made me think that the whole video was going to be a song lol. Very interesting nonethless
I'am not suscribed for the plots, but for the music!
That intro hit me hard xD
I was waiting for the "BAAM" all video long, got just a couple of great "HOORAY!".
Thank you for the awesome channel Josh!
You made me laugh. :)
🤔🤔🤔🤔🤔 well I thought that q-q plot was difficult but thanks to you I got it now. thanks and keep it up!!!
Not gonna lie, stats is my super weak spot. You've helped me a lot in my Data Models course and interpreting my results. +1
Happy to help!
@@statquest Thank you! I'm just kicking myself for not taking more stats courses at this point!
@@jameswhitaker4357 My stats courses were all pretty terrible, so you never really know what you're going to get. I had to teach myself statistics, and these videos are how I taught myself.
@@statquest That's what I'm going through right now! I've been using your videos and a "Intro to Statistical Learning with Applications in R" textbook which has helped a lot. I think when I saw terms like "heteroscedasticity" or the crazy formulas I would get scared and put off the studying, until I took a course that required knowing it LOL. And luckily most of these statistical tests and concepts are now pretty easy to perform in programming. Cheers!
@@jameswhitaker4357 I actually wrote a little about heteroscedasticity. Maybe I should record it.
Couldn’t have asked for more clear explanation, thanks!
Glad to help!
Nice video. Explained everything in just under 7 mins. Awesome.
😄👍👍
bam!
@@statquest bam indeed. 😁
I just noticed when you said "please subscribe" at the end of the video, the subscribe button lit up:)
bam! :)
Can you do a video on normality tests like shapiro wilk and anderson darling? If not anytime soon, can you share link to some good materials?
I really appreciate from your very easy way explanation.
I faced with so difficult and rough terminologies that I could not even understand the meaning of them.
This is very very cool, more likely to learn on UA-cam than in a classroom. Grazie
Glad it was helpful!
I could have better grades if i had faculties like you...thank you Josh!!
Thanks!
You had my like at the beginning with the jingle. Thanks for explaining this so well!!
Glad you liked it!
Triple BAMM! Serious man your channel is pure art. Thanks
Thank you!
Thank you for your help! Greetings from Brazil.
Muito obrigado! :)
Best intro song, it can be used as a 'mnemonic' for what QQ plots are used for =)
It's a party time with Josh Starmer and his quantiles! 😆🤘 Party on, Wayne!
:)
Explained like a pro.
Tripple BAM!!!
Thank you! :)
Hi!!! Great video!!!! It was very helpful to understand Q-Q Plots!!!! But just one question, how do you calculate the quantiles for your dataset?? I mean, the first observation of your dataset is 0.6, but I don't understand why, since the first observation leaves 0 observations on one of its sides. Should the quantile be 0? In the video where you explain how to calculate quantiles, you explained that the quantile for each observation is calculated dividing the number of observations that this value leaves below between the total number of observations... So, for the first point... 0/15 = 0. Why 0.6??
I think I see the confusion here. The x and y-axes on the QQ-plot (on the right side) are labeled "Normal Quantiles" and "Data Quantiles". This is a little misleading - what we are plotting are the values at each quantile, not the quantile name itself. So if the first quantile is called "quantile 0", but it represents -1.5 in the normal distribution and 0.6 in the data, then we draw a dot at -1.5, 0.6 to represent the first quantile. Does that make sense?
@@statquest Perfectly. I understood this after watching some more videos. I would suggest you to clarify this if you make a new version!! As I already told you, congratulations for your videos and of course, your quick reply!! You explain really well, and the videos are perfect (easy to follow and to understand). I'm doing my Ph.D and it is really helpful people like you. Thanks a lot.
@@statquest Thank you very much for all your videos! They help me a lot. Just a follow-up question on this: how can we decide where to start as smallest quantile value in the theoretical distribution? Like you mentioned, "quantile 0" value in the sample distribution is 0.6, but how can it represent -1.5 in the normal distribution? My confusion is normal distribution doesn't technically have "quantile 0" value because it's infinity on the both tails.
@@Fan-fb4tz On the left side the first quantile is defined for the first point of 15 data points, meaning that 1/15 of the data is equal to or less than that point. Thus, we find the corresponding point on the normal curve such that 1/15th of the area under the curve is to the left of it.
@@statquest strictly speaking, the 15 lines(15 data points) divide the whole data into 16 equal groups or parts，So corresponding to normal distribution should be divided into 16 bins so that every bin has the same probability of 1/16 ,right?
Step 3 is not very clear. How do you put the lines on the normal distribution. How do you start putting the lines ? and How about the distance between each two lines ?
The quantiles for the normal distribution divide it so that the area under the curve between two lines is equal for all of the divisions. Since the normal distribution isn't as tall on the edges, there is more space between lines then in the middle, where the distribution is tall. Thus, the spacing between lines makes the area under the curve between the middle two lines is the same as the area under the curve between lines on the edges.
Thanks! It is easy to see that in the case of Uniform distribution. How about the starting point ? I think it's randomly that you started by -1.5 but I can start from -2 or -1 or ..
The starting point is defined by the need for each unit between lines to have the same exact area under the curve. To understand what this means, imagine you had to divide a normal distribution with a single line so that 50% of the area under the curve was on the left side of the line and 50% of the area under the curve was on the right side of the line. Where would you draw that line? Well, there is only one choice - right down the middle of the normal curve. If you drew it anywhere else, there would either be more area under the curve on the left side or the right side. Now imagine you had to divide the area under the curve into 4 equal amounts. Again, there is is only one option - you put a line in middle, and then you put another line so that the area under the curve on the left side is divided in half and then a third line so that the area under the curve on the left side is divided in half. Any other locations for those lines will result in the areas under the curve not being equal to each other. Thus, in this example, we have no choice about where to put the lines - they have to be put in the one configuration that makes the area under the curve between every pair of lines equal.
Perfect! got it!
Hooray!!! :)
THANK YOU!!!!!!
Thanks!
great video. a video on the intuition on why q-q plot works might be interesting.
I'll keep that in mind.
Thanks for all the videos! Great music BTW. Also I'm looking forward to rockin' my new SQ hoodie!
TRIPLE BAM! Thank you for your support!
thank you very much for this video. Please a have a few questions to ask.
1. From your previous video on quantile and percentile the first line was 0% quantile, why does it have a value of 0.6 in this video?
2. How are you getting the values for the x- axis, and why did it have to range from -2 to 2? Thank you
1) In this video we are plotting the actual values on the y-axis, rather than their quantiles.
2) The values come from a standard normal distribution (a "standard" normal distribution is a normal distribution with mean = 0 and standard deviation = 1). There are excel functions that will generate the x-axis coordinates from a standard normal distribution for you.
You simply saved my life
Bam!
How beautiful and simple is that explaination 🥳
Thank you!
You made it very clear man !!! Great doing
Glad to hear that!
very nicely explained. it was a tricky concept until this video! thanks!
Hooray! I'm glad the video helped. :)
Awesome video - thank you SO much for saving my sanity.
Thanks!
Awesome video. Explained so clearly. Really helped me a lot!
Hooray! :)
phenomenal explanation and really cool intro music man!
Thanks!
wow, now I can clearly understand it ! thanks alot !
Hooray!!! :)
is there a statistical test we can do to determine how far away the dots are allowed to deviate, rather than just eyeballing it? Or is eyeballing good enough? I.e. a stat test that could say 'the chance of these 2 distributions being the same is less than X%
The "K-S Test" is what you want. However, it is very strict and tends to reject the null too easily. It's one of the few statistical tests where a large p-value (suggesting no difference) is more convincing than a small one. en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
@@statquest How were the lines drawn? Least Squares? Maybe doing R^2 calculations can provide an idea?
Still trying to grasp my statistics a bit better :( :)
Best Explanation ever!!! 🎉🎉🎉
Thanks!
I always wondered how statisticians choose a distribution to which to fit the data when eyeballing it is insufficient. Now I know the answer: QQ-plots. Thank you for this!
bam! :)
Dear Josh, thank you for this informative video. I have a one question: we are dividing the gene expression data into fifteen and also dividing the normal curve into fifteen. So for example lets take the 3rd data point: 1.9. It is the 3/15 percentile which is 0.2 So in the normal curve when we look at the z-table it should have been -0.84. In your calculations, I am observing that it is -0.89 which is actually 0.1867 percentile in the z-table. Am I missing something?
If we want 15 equal sized portions of the normal curve, then, we actually need 16 slices, the extra slice is at positive infinity. This makes it so that the area under the curve to the left of the first vertical bar in the graph is equal to the area under the curve between the 1st and 2nd vertical bar which is equal to the area under the curve between the 2nd and 3rd vertical bar (etc. etc. etc.) So when we do 3/16, we get 0.1875, and the normal quantile for that is -0.8871
@@statquest thank you very much! Now it seems clear
Very well explained !
thank you!
Two questions:
1. How does having more points at the middle make the histograms narrower and the opposite at the ends? If the histograms have more width doesn't that mean that more data points can be placed in that histogram ?
2. the last example you took 4 data points and said that the distribution has 4 quartiles. Did you say such because there are 4 points ?
1. You might want to review the concept of the normal distribution. ua-cam.com/video/rzFX5NWojp0/v-deo.html The hight of the curve indicates the likelihood of observing a point there. So the wider parts have lower likelihoods of observing points and the taller have a higher likelihood. To make each region have an equal probability of observing a point, we have to have wider regions where the likelihoods are lower and narrower regions where the likelihoods are higher.
2. yes.
Helpful and easy to undderstand.
Thanks!
Best intro by far so far
Hooray!!!! :)
Very well explained, thanks so much
so clear, so good , so nuce thank you , Josh
Thanks!
Thank you for the video! It was short and easy to understand :)
Thanks! :)
Thanks for this! Finally I understand
Hooray!
How do you know that the smallest quartile on the normal distribution is at -1.5?
You can get a computer program to figure this out for you. For example, the R programming language has a function that will give you all the quantiles. Presumably MS Excel has a similar function.
Crytal clear explanation
Thanks! :)
thanks for the explanation, can you clarify this please?: if we have 15 quantiles, then I thought you should plot 14 red lines in the normal distribution and the 15th line should reside in +infinite. and a little question: is the straight line generated by linear regression?
Plotting a line at infinity would be hard to do and you can fit the line with regression.
Really excellent presentation, Josh. ⭐️
Thank you! :)
Legendary explanation! Fantastic!
Thank you!
And again, thank you for another amazing video ! A little question : most of the points have to fit in the straight line for the data to be considered as normally distributed and at 4:15 you said it is not the case. Althought the intersection points are really close to the line, it does not matter, most of the point have to be strictly ON the line, right ? The fact that other intersection points are close or far from the line does not give any relevant information ?
I'm not sure I understand your question. For more details on how to interpret QQ-plots, see: stats.stackexchange.com/questions/101274/how-to-interpret-a-qq-plot
@@statquest ok thank you !
The name of the channel should be "Dumbing Down Probability for Dummies". I don't know whether I like the intro song better or that simple explanation.
Personally, I like to think that rather than dumbing down the material, I bring people up so that they can understand the tools and techniques in data analysis. I think "dumbing down" suggests watering down the content, as if I am presenting a simplified version of how statistics really works. That's not what happens in my videos. This is the real deal. It's just explained in a way that is relatively easy to understand and that brings people up.
@@statquest Understood. The way you teach not only makes it comprehensible but also ensures it sticks to the head!
It just saved me, the person who did this => you're the best
Thank you! :)
Hi Josh, Amazing Video there :) Just want to understand the intuition behind the working of QQ plots ? Is it the fact that quantiles of every normal distribution are just scaled up values of a standard normal distribution and that is why we expect a straight line ?
Pretty much
@@statquest Thanks for the response ;) Would really appreciate if you could make something on the same or share some content that could explain the intuition behind QQ Plots.
What's the approach for determining which distribution has the best fit for the data? Would the r-squared of the data against the straight line be a suitable measure for how well the distribution describes the data?
This is a good question, and, to be honest, I'm not sure what the answer is. I like your idea, but it may oversimplify the problem. i.e. you could get a high R^squared value, but still have some real obvious problems if you looked at it visually.
I know it may sound dumb but i just got it when i understood that theoretical quantiles were the quantiles of a normal standard distribution or Z-value.
bam! :)
Thanks for the great explanation as always! So QQ is just a way to plot and visualize the similarity of two distributions? Are there any other scenarios when these can be used? Thanks!!
QQ is mostly used to check tail conditions. Density plots and cumulative plots are the best way to check distribution symmetry.
How helpful! Thanks a lot for your amazing videos
Thanks!
Hi, thanks for video but could you please add in the description wtf is gene expression and what mean x and y on the 0:37 graph. For the time being I see 15 data points and no idea why they are shown in this way, thank you. Maybe some simpler example instead of gene expression?
OK. I added this to the description:
NOTE: The data in this video are measures of gene expression. If "gene expression" doesn't mean anything to you, just imagine that the data represents how tall a bunch of people are, or how much they weigh. Then consider the y-axis to be the height or weight of the people, and the x-axis just represents all of the data you collected on a single day. In this case, all of the data were collected on the same day, so they form a single column.
@@statquest thanks for your effort!
Awesomely explained! Good job!
Thank you! :)
4:35 I think there is an extra quantile drawn on the Uniform distribution
You are correct. Thanks for spotting that.
Actuary studies is something else bro
Noted!
u saved my life
bam! :)
very clear with great examples!
Thanks!
very nice video. What's the difference between normal and uniform distribution? I thought both are same
Normal Distribution: en.wikipedia.org/wiki/Normal_distribution
Uniform Distribution: en.wikipedia.org/wiki/Uniform_distribution_(continuous)
If, by visual inspection, we suspect that the distribution is D, how do we get the quantiles of D? I can imagine that, if the CDF of D is inversible (like the CDF of the exponential distribution, or a sigmoid) then we can compute quantiles. But what about quite arbitrary distributions ?
I'm not sure. Pretty much all of the standard distributions can be inverted.
the song alone is worth it
bam! :)
Thanks for saving my life
Hooray! :)
Your videos are awesome, thank you so much!
Glad you like them!
i love statquest!
Hooray! :)
Great explanation, have a nice day :)
such an awesome video, thanks!
Glad you liked it!
Best intro 🔥
bam!
Understand it now - thank you!
Hooray! :)
It's so clear! Thanks a lot for your video.
Thanks a lot for the wonderful explanation!
Thank you!
Hi Josh, can we use percentiles in place of quantiles to plot QQ plot ? If so, in case of percentiles we can only have upto hundred percentile no matter how big our data is then how to have a definitive answer whether or not the 2 datasets have similar distributions as mention in the video at 6:30 ?
The terms "quantiles" and "percentiles" are often used interchangeably, and in this case you can swap out quantiles for percentiles. And you can have as many percentiles as you want - however, the largest percentile is always 100. For example, you could have the 0.5 percentile, or the 1.23 percentile.
@@statquest Thanks Josh.
5:30
I am confused... I thought that quartiles are three (not four) values, which divide the dataset into four equal numbers of data points.
In your example you say that four data points are quartiles? 🤔
Oops. That's a mistake. Quartiles divide the data into 4 parts.
It helps a lot. Thanks!
Glad it helped!
Nice lecture, but how do u identify boundary conditions, like - 1.5, for normal distribution?
This is explained at the very start of the video at 0:38. We have 15 data points, so our data have 15 quantiles. We then divide the normal distribution into 15 quantiles. Each quantile should have an equal probability - thus, with the normal distribution, the quantiles on the edge are relatively far apart, to compensate for the relatively low probability of observing a value out there. In the middle of the curve, the quantiles are close together since there is a higher probability of observing a value there. Since each quantile has to have the same probability, then there is only one way to configure the 15 lines that we draw. If that last part doesn't make sense, then just imagine we only had one quantile - so we needed to divide the normal distribution into two equal parts. Where would we put the line? Well, there's no choice involved here because there is only one location for that line - right in the middle. Similarly, when we have to divide the normal distribution into 15 equal parts, there isn't a choice about where to put the lines, there is only one option.
Hi, I understand how the quantile points are plotted wrt observed vs theoretical distributions, what I don't understand is what determines the slope of the straight line. While this is fairly intutitive for a normal distribution, for say the Weibull distribution I am unclear how the slope of the striaght line is used to determine whether the observed vs theoretical quantiles are a good fit for a given distribution. Any ideas?
I came here for the same question and left with no answer LOL
Thanks for the videos, if you're still looking for ideas how about k-l divergence?
Your videos are cool and concise , thank you .
thanks , awesone , this is useful in measuring gene expression effect
QQ is really handy
Great explanation, thanks!
Thank you! :)
Thanks for explanation :))))) , one question , how we determine the 4 quartiles in first dataset for last case?
We divide the data into 4 equal sized pieces.
@@statquest oo Tnx...
It's hard to understand Step 3. How should I add same number of quantiles? Where exactly to draw the lines?
In R, you simply call the qnorm() function and it tells you where the draw the lines. For example, for this video, I used the call: qnorm(1:16/16, mean=0, sd=1) to see where to draw the lines. In MS Excel, I believe you can use the NORM.S.INV function. For details, see: www.statology.org/q-q-plot-excel/
This video explained how to compare a discrete sample to a continuous normal, and to a discrete but smaller sample. What if we wanted to compare continuous sample vs continuous theoretical, is there an analogous qqplot for that? Or the term "continuous sample" is an oxymoron since "sample" means discrete already no matter sampled from theoretically discrete or continuous distribution?
In the example, the "discrete" values come from a continuous distribution, so there is no need to do anything special that is not described in this video.
Josh, Great video... very helpful. It looks like you might have a slight error when comparing the 2 dataset distributions however. I could be wrong but I think your second plot is incorrect on the chart.
I think I know what you're talking about. The second point should be at 5.1 on the x-axis but is only at 4.1. Is that it? That's a typo.
By the way, the long term plan is to correct/update these videos. Just like textbooks have "new and revised editions", I'd like to have new and revised additions of these videos - so your feedback is helpful and appreciated. I hope that once the channel grows, youtube will give me some options for how to release revised videos - right now I have no options, but I'm also relatively small potatoes. So I can't fix the video right now, but one day I will.
@@statquest Yes that's it.
@@statquest No problem. Your viewers would certainly appreciate your easy to understand videos. I just wanted to check I understood and maybe other users will find the note in the comments useful.
Do I need to matter the exact size size or probability when dividing the contribution? Or just need to only make sure the sizes are equal?
Just equal sizes.
Your video does explain the concept well Thanks.
But, I can't do a plot from your video... I will not know how to start with a data set at hand.
Can you explain step by step... on how to generate QQ plot?
To actually draw one, see: www.r-bloggers.com/2021/06/qq-plots-in-r-quantile-quantile-plots-quick-start-guide/
Hello, the video was amazing and I was able to get an idea of QQ plots. I do have a question though. How do we draw the normal distribution and uniform distribution? Is it just random?
The normal and uniform distributions are well defined by equations. So we just plug numbers into them to get the values out.
In the last example, shouldn't we have drawn a straight line goes through the origin ??
No, because the scale of the x and y-axes is based on the original scale of the data.
Sweet, its not that difficult to grasp anymore! Thanks
In the example with the normal distribution you have 15 lines dividing the probability, but in the example with the uniform distribution you have 16 lines dividing the probability. Or, in other words, in the first example the last line leaves a probability of 14/15 to its left side, but in the second example the last line leaves a probability of 1 to its left side.
¿?
That last line on the uniform distribution might be a typo. The idea is that you divide the distribution into the same number of equally sized pieces (where size is defined by probability) as you have data.
Thank you! Awesome explanation
Thank you! :)
i have a doubt...why to use this method,instead just plot the points and see if it forms a bell curve....correct me
@@bharathkumar5870 I'm not sure I understand your question. Are you asking, "why don't we just create a histogram with the data and see if the histogram looks like a normal distribution"? If so, histograms can be very tricky in terms of selecting the correct bin size. In contrast, with a q-q plot we don't have to worry about optimizing a bin size or anything else.
@@statquest thank you sir ..u cleared my doubt. Different bins give different distributions😀
What a great song :)
:)
Excellent comparison with Sample and normal distribution ....Subscribed..:) I have a dought can we take any normal distribution mean N(0,1) OR N(0,4) ?
Any normal distribution will do, no matter what the standard deviation is, because the quantiles represent the same thing and the x-axis will be scaled accordingly.
@@statquest I have a sample question related to Percentile calculations...can u share a mail I will share my understanding ?