Absolutely excellent but I like to suggest a minor correction at 1:54. The linear regression DOES account for the uncertainly of the line; the linear regression prediction interval would produce intervals that widen as you move away from the data just like the Bayesian ones. (Many textbooks give the approximate formula for prediction intervals that don't widen but the actual non-approximate formula will give widening bands.)
My postgrad supervisor literally told me to watch this a few times just so I can explain it clearly to Human Sciences people in my research proposal. Thanks for all your effort making it!
Great balance between technical depth and intuition for me right now. I love how you say that multiplication "is like", but still is not. This gives intuition, but provides a warning for the day when we have come further in our understanding. 🤗
the first time I watched this a month or so a go I had no idea what was happening. However I have recently needed to use a GP and after a lot of reading up on them and coming back to this video, I can appreciate it a lot more with some understanding:)
Yea my topics require some prerequisite 😅 but with a little getting used to on the notation and basics of probability/stats, I think it should be fairly digestible. Glad you got something out of it.
Great reference video, I'm sure I will come back to it again and again. The level of detail in all the simulations you do is just incredible. Do you make all your animations in manim?
Thanks brother! And i don’t use manim actually. I like representing data with Altair, which is like a better version of matplotlib. So I have a small library which turns Altair pics into vids.
I just discovered your videos yesterday and now they're popping up on my YT home screen and I feel a bit like a little boy in a toyshop. How have these high quality fantastic tutorials evaded me for so long, when I spend so much time looking at technical content on UA-cam? Seriously impressive! I'll definitely be one to share your videos when the opportunity arises.
I will have to watch your video several times to understand (if I can) everything but undoubtly your video is professional and very very well done !! congratulations
holy sh*t, you have unlocked the secret of GP and Bayesian stuff... I have struggled so hard to understand what is even GP as it is so abstract. Thank you so much for your great work!
It looks like you have optimized the hyperparameters of making an awesome video. So concise, but still a sprinkle of humor here and there. Awesome visualizations, so appreciated.
I guess this is the first time I am commenting in UA-cam! This tutorial is one of the best that I've ever seen on GPs and even math. The visual morph changes of the GP corresponding to different hyperparasite are fascinating. Wish you the best!
Literally the GOAT. So clear and concise, and the pace is perfect! A humble suggestion: adding a quick explanation of what GP does would be ideal (also, what are these sample prior, etc.) Dimension-wise, it would make more sense to me.
Absolutely perfect! I heard of GPs and was wondering what they were exactly, wanted a bit of intuition of how and why they work, how to use them, just as a quick intro or motivation before learning them later on. This video answered all of this in a duration that is absolutely perfect: not too long so that it can be watched "leisurely", and not too short so that you still give enough information that I don't have the impression that I learnt nothing. Didn't know your channel, will definitely check the rest out!
Phenomenal video. I genuinely can't thank you enough for how accessible this was. I'm sure I'll come back and reference it, or your other work, as I continue preparing for my upcoming internship working on physics-informed neural networks.
You are the best man! Thank you for your videos, you 're helping a lot of students, because your explanations are so clear and intuitive. I Hope the best for you.
Great video! Just dived into GPs by learning about their application in system identification techniques. In fact I'm learning for my examn right now and looked for a video that nicely sums up this topic and gives some intuition. This video matches my needs 100%, thank you very much.
Thank you! I've been researching paper dedicated to the gaussian approach to time series prediction(as a task in a lab), and I really struggled with it. But after your video, everything has been sorted out in my head, and i finally have understood it!
As I wrote you on LinkedIn, this is probably the best video on GPs out there! I know it takes a long time to put together something of such high quality, but I hope I will see more of your videos in the future! 😊
Really grateful for this video. Got the gist of it but will have to pull out my old friend pen and paper and work through the math of the GP assumtion. Thanks for the neat definitions :)
Thank you - I'm getting a little better over time, but it's a work in progress. If you love what I'm doing, one thing that would be *huge* for me, is if you tell anyone you think might be interested. This channel is pretty small and it'll be easier to work on it if it gets a little more attention : )
I would be really helped by putting variable definitions on screen while they're in use! I find myself forgetting what f and f* are for example as I mull it over and watch the explanation. Amazing video! I'm a fan.
Thanks Graham! It's always a balance thinking about what does/doesn't go on screen. More recently, I'm biasing towards *less* on screen, b/c I've gotten feedback that what's on screen can be overwhelming. But, if you have some question about what may be confusing, ask here and I may be able to help
@@Mutual_Information I'm thinking what was hard for me is that everything was defined and then they were used? the viewer needs to remember what each things means before they can give it the context, and context allows us to combine things and save on short term memory?
I did understand just a few things, but still I watched this video till the very end - the production value is insane! And maybe I’ll need GPs in the future? :D You definitely deserve much more subscribers, your videos are great!
I built a model years ago that I never realized is perhaps a GP model. I only learned about GP models a weeks ago. It doesn't use any real-valued data; only binary vectors. The similarity kernel is Hamming distance. Other than that, it's basically what he described here.
Excellent video, thanks. I have a question at the 1:20 mark. The distribution that you show, ‘p(y|x)’, does not look Gaussian. Am I missing something? Can a GP predict a non-Gaussian distribution?
Oh I see the confusion. That's not a GP model. That's just some non-parametric model to show the type of thing we're going for. It's there to draw a contrast when we start making assumptions. We assume the normal distribution at some point.. and that's what gives you the p(y|x) gaussian.. but in the general case, that's not necessarily true. But this isn't very clear in the vid, - sorry about the confusion :/
Your observation of the product of two normally distributed variables is true for the following reason: given independent scalar random variables X,Y, we have Var(XY) = Var(X)Var(Y) + Var(X) (E(Y))^2 + Var(Y) (E(X))^2. Given two multivariate random normals U,V with mean zero, we may choose to work in a basis (possibly different for the two distributions) where the covariance matrices are diagonal. The all components of each vector are independent and so Cov(U) Cov(V) = Cov(UV) by working element-wise. Since this is true in one basis, it must therefore be true in every basis.
I plan on making one. It’s a very interesting idea. In the meantime, there is already an excellent explanation : ua-cam.com/video/i7LjDvsLWCg/v-deo.html
Great video!! I have a question about the graphs at 9:25. Shouldn't the heat map for the for the x vs x' look so that the K is highest at the origin (0,0) and fades moving to the other corners instead of what's shown? I might still not fully understand it. Thanks!
7:01 shouldn’t it be the other way around? If most ys are deemed different from an x, then the GP would sample closer ys and therefore the functions shouldn’t wiggle too much. Or am I missing something?
Awesome video!. Only one question. Minute 09:20. A linear kernel does not imply that the realizations of the random process must be linear, does it?. Thanks!!
Thanks Jesus. And regarding your Q, in the broader model, no a linear kernel doesn't imply the realizations need to be linear, since there is a noise component in the overall kernel. That allows points along a sample to be different in a nonlinear way.
It's very simple to extend to multi-dimensional inputs in fact. If x, x' are vectors, as long as K(x, x') returns a single number, then you can apply everything exactly as you see here. The visualizations will be a little trickier, but the whole idea still works.
thank you for your understandable video! I'm still wander what is the point of "similar y for similar x", is it make sure the function is smooth, or other usage? looking forward to your reply!
The goal of the problem is predict y for a given incoming x.. and we can learn to do this by observing many pairs of (x_i, y_i)'s. So we make an assumption: "If x1 is similar to x2 (that is K(x1, x2) is large/positive)), then we expect y1 and y2 to be close". With that assumption, we can form a prediction for y when given an x.. and that basically is formed by determining "which y value would best work with our similar-y's for similar x's given the x's and y's observed?" and then you can form your prediction that way. The GP does all this hard work for you and allows for noise and whatnot.
This is something I'm working on! I'd like to make code samples available alongside my videos. They aren't currently available b/c the modeling code is intertwined with the animation code, so it would make for a terribly difficult to decipher code if released as-is. My plan is.. once my video production workflow is a little more streamlined, I'll pair these video with code snippets.
Hey, great video! In practice in my machine learning class we did both GPR and GPC I found it very difficult to scale it to more than 10k samples. It seems that despite the advantages it has, it is not useful for a lot of practical problems. Can you maybe show show video on how to invert a matrix with less than O(n^3) complexity and which software someone could use for GPR/GPC for larger data?
Yea, so that's a big component of GP research. Getting the cost down. A dominate approach are inducing point methods, where you try to summarize a large dataset with a smaller data set of "inducing points". It's a popular approach, but introduces another source of uncertainty. In my experience, I tend to use GPs with smaller datasets.
Sir i am using matlab regreession learner toolbox i understood that kernel functions tries to find how similar or far apart 2 input data points are,can you tell what are basis functions then there are 3 basis functions zero,constant and linear?
Hi, great video for GP! I have quite new to this topic, is it possible to use GP to model multiple output? e.g. my "input" data is time, and the output data is 2D coordinates (x(t), y(t)). If it is possible, how to setup the covariance function? Thanks so much
@@Mutual_Information Hi, thanks for the reply 😃 Yes I need to obtain the uncertainty as well. I think if x(t) and y(t) are "independent", then fitting independent GPR with uncertainty bounds are applicable. However, if they are dependent (like coordinates of an ant moving along a circle), then x and y are correlated. What do you think?
Hi, I am trying to implement a GPR myself and am struggling with writing code for the parameter updating process. I am using gradient descent to maximize the log density function, w.r.t. the terms: sigma_n, sigma_f, etc. My output predictions fit decently to the data, but only works for univariate input. Also, my noise prediction is just constant or zero throughout, despite the evident noise present. Please advise me on how to approach this. Might it be possible to host a virtual meeting with someone who can help?
The best video about GP I have ever seen! Thank you for sharing. I would like to reproduce the graphs that you created in a script, but unfortunately I cannot see any code about it on you github page! It is possible to access to those scripts? with the examples that you produced?
Thank Matteo - I appreciate it! Unfortunately, the code for this one was heavily intertwined with the animation code, so I didn't make it public. But I wasn't doing anything you can't learn from reading the GPyTorch docs
Gaussian Processes for Machine Learning by Rasmussen and Williams is excellent. It's pretty technical at points but it's a super thorough coverage of GPs.
So 23'47" is an upper bound on the time needed to explain your monitor backgrouds! :-) But what is the underlying data and GP's hyperparameters? And also, where does the "process" in "GP" come from?
Ah! you speak to a trade-off I was concerned about. In the original script, I actually listed the hyperparameter values of the fitted models.. but it made the explanation wordy and put a good deal extra notation on screen. Removing it was a push towards simplicity.. but yea, the cost is, I create room for questions. Also, regarding the datasets. For the first dataset presented, that's just data generated according to the linear regression assumptions, where the coefficient is small (so it's a week signal). For all other datasets, I used a GP itself to generate the data. And "Process" comes from the fact that the technical definition of a Gaussian Process is it's a type of stochastic process. Specifically, a GP is an infinite collection of random variables where any finite subset has a multivariate normal. "Process" refers to the "infinite collection of RVs". It's a pretty theoretical idea and I didn't think it was *crucial* to understanding the ML models using GPs, so I avoided it. Just thought that discussing an infinite collection of RVs could be avoided without sacrificing much.
This is cool stuff! There is something I want to understand from the similarity heat map of the linear kernel. If the function samples are dissimilar as they get further apart (according to the lines), should the heat map not be brighter at (0,0) and fade as it approach (10,10)? I am trying to get the picture in my head.
Thanks! I think you're thinking about it from a difficult angle. It's not function *samples* that are similar/dissimilar, it's specific *inputs* across samples. So, for the linear kernel, for inputs that are very similar (like input=0, so heatmap is high, which means similar), the outputs are virtually the same spot. For inputs closer to 10, the inputs are dissimilar and outputs are far apart. Since all function samples are lines, this will manifest as two lines which intersection at input=0 but are far apart at 10. Make sense?
@@Mutual_Information I do not totally understand. It is not your explanation that is bad. I simply need to get to know GP better. Thank you for trying to explain!
Literally the best explanation on the internet for GP Regression Models. He's not trying to be cool, but genuinely trying to explain the concepts
Thank you my man. And yes, my risk of being cool is zero lol
Agreed, I've been trying to understand GPs for a task at work and this is the easiest to understand explanation I've found, liked and subbed!
Absolutely excellent but I like to suggest a minor correction at 1:54. The linear regression DOES account for the uncertainly of the line; the linear regression prediction interval would produce intervals that widen as you move away from the data just like the Bayesian ones. (Many textbooks give the approximate formula for prediction intervals that don't widen but the actual non-approximate formula will give widening bands.)
*of mean zero with a standard deviation of 4.20
My postgrad supervisor literally told me to watch this a few times just so I can explain it clearly to Human Sciences people in my research proposal. Thanks for all your effort making it!
Great balance between technical depth and intuition for me right now. I love how you say that multiplication "is like", but still is not. This gives intuition, but provides a warning for the day when we have come further in our understanding. 🤗
Ha yea glad these details aren’t unnoticed. It’s a careful game making sure I never say anything *technically* wrong.
Yes sir sometimes to make a point you need to recontextualize the matter to specific to make things easier to understand@@Mutual_Information
I know how hard it is to explain this topic, so simply and comprehensively. I am extremely thankful for your efforts.
I learn each time I rewatched the video. So much better than sitting lectures where you only listen once.
the first time I watched this a month or so a go I had no idea what was happening. However I have recently needed to use a GP and after a lot of reading up on them and coming back to this video, I can appreciate it a lot more with some understanding:)
Yea my topics require some prerequisite 😅 but with a little getting used to on the notation and basics of probability/stats, I think it should be fairly digestible. Glad you got something out of it.
Great reference video, I'm sure I will come back to it again and again. The level of detail in all the simulations you do is just incredible. Do you make all your animations in manim?
Thanks brother! And i don’t use manim actually. I like representing data with Altair, which is like a better version of matplotlib. So I have a small library which turns Altair pics into vids.
Awesome tutorial! Where could I find more information on GPs for graphs and varying length strings?
I just discovered your videos yesterday and now they're popping up on my YT home screen and I feel a bit like a little boy in a toyshop. How have these high quality fantastic tutorials evaded me for so long, when I spend so much time looking at technical content on UA-cam? Seriously impressive! I'll definitely be one to share your videos when the opportunity arises.
Much appreciated! I got some really cool stuff coming in May. If you like this stuff, you'll *love* what's coming. Thanks again!
Really helpful. GPs finally clicked watching and working through this video. Great balance of accessibility + technical details. Thanks bro
I will have to watch your video several times to understand (if I can) everything but undoubtly your video is professional and very very well done !! congratulations
Thanks Xavier!
I read the distill article and came back to watch the whole video again for the second time. Now it's crystal clear! Thanks so much!!
Distill is an epic educational source :)
@@Mutual_Information It's sad that they're in hiatus since last year :(
Hopefully they'll come back some day
The production value of this is insane
holy sh*t, you have unlocked the secret of GP and Bayesian stuff... I have struggled so hard to understand what is even GP as it is so abstract. Thank you so much for your great work!
Happy to help my friend ;)
It looks like you have optimized the hyperparameters of making an awesome video. So concise, but still a sprinkle of humor here and there. Awesome visualizations, so appreciated.
haha I thought that was gonna be a nitpick, pleasantly surprised - thank you!
I guess this is the first time I am commenting in UA-cam! This tutorial is one of the best that I've ever seen on GPs and even math. The visual morph changes of the GP corresponding to different hyperparasite are fascinating. Wish you the best!
Thanks
The way you motivate the problem really adds insights for understanding.
Great video. I've seen GPs mentioned a few times in papers and always glossed over it. Thanks for the great explanation!
Literally the GOAT. So clear and concise, and the pace is perfect! A humble suggestion: adding a quick explanation of what GP does would be ideal (also, what are these sample prior, etc.) Dimension-wise, it would make more sense to me.
Absolutely perfect! I heard of GPs and was wondering what they were exactly, wanted a bit of intuition of how and why they work, how to use them, just as a quick intro or motivation before learning them later on.
This video answered all of this in a duration that is absolutely perfect: not too long so that it can be watched "leisurely", and not too short so that you still give enough information that I don't have the impression that I learnt nothing.
Didn't know your channel, will definitely check the rest out!
Thanks a lot! That's exactly what I'm going for. Relatively short and dense with useful info. Glad it worked for you.
In min 3:00 I saw a smile coming out of my mouth, just how happy I was when I was listening to you!
This is a masterpiece work! Really thank you :)
Thank you very much - glad it's getting some love :)
Truly amazing how you turned such a complex topic into an accessible explanation, thanks a lot!
Glad you liked it :)
Phenomenal video. I genuinely can't thank you enough for how accessible this was. I'm sure I'll come back and reference it, or your other work, as I continue preparing for my upcoming internship working on physics-informed neural networks.
That's awesome! Glad it helped
The best explanation of Kernel so far!
Agreed! 😎
You are the best man! Thank you for your videos, you 're helping a lot of students, because your explanations are so clear and intuitive.
I Hope the best for you.
Thank you - you improved my Friday
Great video! Just dived into GPs by learning about their application in system identification techniques. In fact I'm learning for my examn right now and looked for a video that nicely sums up this topic and gives some intuition. This video matches my needs 100%, thank you very much.
Excellent - you're the ideal viewer!
Thank you! I've been researching paper dedicated to the gaussian approach to time series prediction(as a task in a lab), and I really struggled with it. But after your video, everything has been sorted out in my head, and i finally have understood it!
Exactly what I've hoped to do - happy it helped!
Perfect! - research, delivery, production, duration, pictorial intuitiveness, mathematical rigor, naïve friendly 👏🏽
Thank you Sourav - I'm trying!
Props for explaining such a complex model in a friendly way
Man, you have some beautiful explanations, and the way you explain the details is somehow very simple to understand, thank you so much!
Glad you liked it!
What a smooth way to explain such complex math , thank you
Such a clear and intuitive explanation of GPs! Great work!. Excellent video on this topic. Brief and elegant explanations!.
Thank you - that’s what I’m going for!
What the hell that's a great channel I'm so glad I found you. Production quality is spot on, thank you for taking such care !
Happy to have you! Welcome!
By far this is the best video I have seen on this subject! Thank you very much!
As I wrote you on LinkedIn, this is probably the best video on GPs out there! I know it takes a long time to put together something of such high quality, but I hope I will see more of your videos in the future! 😊
thanks, means a lot - and it's coming. This one has been taking awhile, but it'll be out soon :)
Really grateful for this video. Got the gist of it but will have to pull out my old friend pen and paper and work through the math of the GP assumtion. Thanks for the neat definitions :)
A great video! Thank's. I used GP at work many years ago and enjoyed the framework a lot.
Dude has named his channel mutual information so when we look for the concept of mutual information, all his videos will pop up 🤣 genius!
Thanks a lot for this vid man it literally saved my life, you're really one hell of a teacher
Thank you - I'm getting a little better over time, but it's a work in progress.
If you love what I'm doing, one thing that would be *huge* for me, is if you tell anyone you think might be interested. This channel is pretty small and it'll be easier to work on it if it gets a little more attention : )
@Mutual Information best of luck man 🫡
Such a clear and intuitive explanation!
Another great video! Love seeing each one come up
Excellent video on this topic. Brief and elegant explanations!
Such a clear and intuitive explanation of GPs! Great work!
Thanks Minh :)
The visualizations are the catch. Just excellent 😊
Damn, Charlie Cox really has the best yt channel for learning ML out there!
Best video for GP I have seen! Thank you so much!
This is truly a great explanation that helps me to connect all the dots together!! Thanks a lot!!!!
I'm glad it help. When I was studying GPs, these are the ideas that floated in my head - happy to share htem.
Quite literally
Badum tssch
Love the level you've pitched this video at.
I would be really helped by putting variable definitions on screen while they're in use! I find myself forgetting what f and f* are for example as I mull it over and watch the explanation. Amazing video! I'm a fan.
Thanks Graham! It's always a balance thinking about what does/doesn't go on screen. More recently, I'm biasing towards *less* on screen, b/c I've gotten feedback that what's on screen can be overwhelming.
But, if you have some question about what may be confusing, ask here and I may be able to help
@@Mutual_Information I'm thinking what was hard for me is that everything was defined and then they were used? the viewer needs to remember what each things means before they can give it the context, and context allows us to combine things and save on short term memory?
a really really hard-core video... thanks D.J
A really good explanation! Though I wasn't able to understand everything, I would keep coming to this video until I do. ;D
Thank you. Happy to answer any questions too
I did understand just a few things, but still I watched this video till the very end - the production value is insane! And maybe I’ll need GPs in the future? :D
You definitely deserve much more subscribers, your videos are great!
Super quality content! Thank you so much: I subscribed and I hope your number of subscribers increases more and more to motivate you to keep going!
Thank you - I hope so too!
Now you make me love math again. Thanks.
Brilliant video! loved the graphics.
What an excellent video! Just thinking about the amount of effort that must have gone into this gives me anxiety
You are a legend.
I built a model years ago that I never realized is perhaps a GP model. I only learned about GP models a weeks ago. It doesn't use any real-valued data; only binary vectors. The similarity kernel is Hamming distance. Other than that, it's basically what he described here.
Amazing video on GP's .
Excellent explanations and visualizations. Helped me a lot, thank you!
This video is really nice. Thank you so much for creating this content material.
Straight forward and explained well thank you
Excellent video, thanks. I have a question at the 1:20 mark. The distribution that you show, ‘p(y|x)’, does not look Gaussian. Am I missing something? Can a GP predict a non-Gaussian distribution?
Oh I see the confusion. That's not a GP model. That's just some non-parametric model to show the type of thing we're going for. It's there to draw a contrast when we start making assumptions. We assume the normal distribution at some point.. and that's what gives you the p(y|x) gaussian.. but in the general case, that's not necessarily true. But this isn't very clear in the vid, - sorry about the confusion :/
@@Mutual_Information got it. Thanks
Your observation of the product of two normally distributed variables is true for the following reason: given independent scalar random variables X,Y, we have Var(XY) = Var(X)Var(Y) + Var(X) (E(Y))^2 + Var(Y) (E(X))^2. Given two multivariate random normals U,V with mean zero, we may choose to work in a basis (possibly different for the two distributions) where the covariance matrices are diagonal. The all components of each vector are independent and so Cov(U) Cov(V) = Cov(UV) by working element-wise. Since this is true in one basis, it must therefore be true in every basis.
What an Explanation! Become fan in seconds.
Love these vids. Can you do a video about normalizing flows in the future?
I plan on making one. It’s a very interesting idea. In the meantime, there is already an excellent explanation : ua-cam.com/video/i7LjDvsLWCg/v-deo.html
Awesome Explanation. Thank you.
Great video!! I have a question about the graphs at 9:25. Shouldn't the heat map for the for the x vs x' look so that the K is highest at the origin (0,0) and fades moving to the other corners instead of what's shown? I might still not fully understand it. Thanks!
The heatmap just shows v*x*x'. I think the constant v here is .5. At the origin, it's .5*0*0 = 0. In the top right, it's .5*10*10 = 50.
Nice visualizations man. Just discovered your channel.
So far beyond my abilities. Like Frankenstein's monster, I am soothed by its music.
13:08 isnt X a d*n matric meaning that each row represents a feature and each column is a datapoint x?
Not in this case. Here, one row provides all the features for one example.
7:01 shouldn’t it be the other way around? If most ys are deemed different from an x, then the GP would sample closer ys and therefore the functions shouldn’t wiggle too much. Or am I missing something?
Awesome video!. Only one question. Minute 09:20. A linear kernel does not imply that the realizations of the random process must be linear, does it?. Thanks!!
Thanks Jesus. And regarding your Q, in the broader model, no a linear kernel doesn't imply the realizations need to be linear, since there is a noise component in the overall kernel. That allows points along a sample to be different in a nonlinear way.
Regarding the modeling example shown at min 10, how would i go about doing this process but for multidimensional inputs?
It's very simple to extend to multi-dimensional inputs in fact. If x, x' are vectors, as long as K(x, x') returns a single number, then you can apply everything exactly as you see here. The visualizations will be a little trickier, but the whole idea still works.
Another great video. Keep up the good work!
Really good explanation, the animations help so much. Thank you, I really appreciate it.
You're very welcome - Glad to hear it's landing as intended!
Super fricking impresed! Bravo
you might want to look into probabilistic numeric, cheers great video you made there!
A truly fantastic explanation to them! The visuals were instructive and well presented, thank you for making this!
and thank you for watching ;)
Keep this up! It really helps
thank you for your understandable video!
I'm still wander what is the point of "similar y for similar x", is it make sure the function is smooth, or other usage?
looking forward to your reply!
The goal of the problem is predict y for a given incoming x.. and we can learn to do this by observing many pairs of (x_i, y_i)'s. So we make an assumption: "If x1 is similar to x2 (that is K(x1, x2) is large/positive)), then we expect y1 and y2 to be close". With that assumption, we can form a prediction for y when given an x.. and that basically is formed by determining "which y value would best work with our similar-y's for similar x's given the x's and y's observed?" and then you can form your prediction that way. The GP does all this hard work for you and allows for noise and whatnot.
Wooow! Excellent quality video!
Not sure if others are also interested, but I think a coding example with GPyTorch could be interesting.
This is something I'm working on! I'd like to make code samples available alongside my videos. They aren't currently available b/c the modeling code is intertwined with the animation code, so it would make for a terribly difficult to decipher code if released as-is. My plan is.. once my video production workflow is a little more streamlined, I'll pair these video with code snippets.
Hey, great video! In practice in my machine learning class we did both GPR and GPC I found it very difficult to scale it to more than 10k samples. It seems that despite the advantages it has, it is not useful for a lot of practical problems. Can you maybe show show video on how to invert a matrix with less than O(n^3) complexity and which software someone could use for GPR/GPC for larger data?
Yea, so that's a big component of GP research. Getting the cost down. A dominate approach are inducing point methods, where you try to summarize a large dataset with a smaller data set of "inducing points". It's a popular approach, but introduces another source of uncertainty.
In my experience, I tend to use GPs with smaller datasets.
Sir i am using matlab regreession learner toolbox i understood that kernel functions tries to find how similar or far apart 2 input data points are,can you tell what are basis functions then there are 3 basis functions zero,constant and linear?
Great video. Thank you 😊
Hi, great video for GP! I have quite new to this topic, is it possible to use GP to model multiple output? e.g. my "input" data is time, and the output data is 2D coordinates (x(t), y(t)). If it is possible, how to setup the covariance function? Thanks so much
Thank you. Do you care about uncertainty in your output?
@@Mutual_Information Hi, thanks for the reply 😃 Yes I need to obtain the uncertainty as well. I think if x(t) and y(t) are "independent", then fitting independent GPR with uncertainty bounds are applicable. However, if they are dependent (like coordinates of an ant moving along a circle), then x and y are correlated. What do you think?
Hi,
I am trying to implement a GPR myself and am struggling with writing code for the parameter updating process. I am using gradient descent to maximize the log density function, w.r.t. the terms: sigma_n, sigma_f, etc.
My output predictions fit decently to the data, but only works for univariate input. Also, my noise prediction is just constant or zero throughout, despite the evident noise present. Please advise me on how to approach this.
Might it be possible to host a virtual meeting with someone who can help?
The best video about GP I have ever seen! Thank you for sharing. I would like to reproduce the graphs that you created in a script, but unfortunately I cannot see any code about it on you github page! It is possible to access to those scripts? with the examples that you produced?
Thank Matteo - I appreciate it!
Unfortunately, the code for this one was heavily intertwined with the animation code, so I didn't make it public. But I wasn't doing anything you can't learn from reading the GPyTorch docs
Great video ! ❤
This is excellent!
I am not sure I understood why hyperparameters are optimized on the prior. What if the priors all suck by the posteriors are really good ?
Are there any good textbooks that go over this?
Gaussian Processes for Machine Learning by Rasmussen and Williams is excellent. It's pretty technical at points but it's a super thorough coverage of GPs.
Great video - I subbed!
Are Gaussian processes any good for predicting your love life? The problem is similar whys for similar exs!
So 23'47" is an upper bound on the time needed to explain your monitor backgrouds! :-)
But what is the underlying data and GP's hyperparameters?
And also, where does the "process" in "GP" come from?
Ah! you speak to a trade-off I was concerned about. In the original script, I actually listed the hyperparameter values of the fitted models.. but it made the explanation wordy and put a good deal extra notation on screen. Removing it was a push towards simplicity.. but yea, the cost is, I create room for questions.
Also, regarding the datasets. For the first dataset presented, that's just data generated according to the linear regression assumptions, where the coefficient is small (so it's a week signal). For all other datasets, I used a GP itself to generate the data.
And "Process" comes from the fact that the technical definition of a Gaussian Process is it's a type of stochastic process. Specifically, a GP is an infinite collection of random variables where any finite subset has a multivariate normal. "Process" refers to the "infinite collection of RVs". It's a pretty theoretical idea and I didn't think it was *crucial* to understanding the ML models using GPs, so I avoided it. Just thought that discussing an infinite collection of RVs could be avoided without sacrificing much.
@@Mutual_Information Thanks a lot for these further details! ;-)
This is cool stuff! There is something I want to understand from the similarity heat map of the linear kernel. If the function samples are dissimilar as they get further apart (according to the lines), should the heat map not be brighter at (0,0) and fade as it approach (10,10)? I am trying to get the picture in my head.
Thanks! I think you're thinking about it from a difficult angle. It's not function *samples* that are similar/dissimilar, it's specific *inputs* across samples. So, for the linear kernel, for inputs that are very similar (like input=0, so heatmap is high, which means similar), the outputs are virtually the same spot. For inputs closer to 10, the inputs are dissimilar and outputs are far apart. Since all function samples are lines, this will manifest as two lines which intersection at input=0 but are far apart at 10. Make sense?
@@Mutual_Information I do not totally understand. It is not your explanation that is bad. I simply need to get to know GP better. Thank you for trying to explain!
This is brilliant. Thank you.
any link to download wallpapers ? 😀
Yea, here: github.com/Duane321/mutual_information/tree/main/computer_background