Model flexibility is how well we can fit the model on training data, if each and every point on the training data is met by the curve, we are overfitting the model , that means the model need to have higher polynomial degree to reach every point. Higher polynomial degree means more flexibility.
At 6:01, you mention that LOOCV has a low bias and a high variance, but do not prove that. Is there a paper where this formally proved? Is the distribution of LOOCV known? Moreover, consider the mean squared error (MSE), i. e. the weighted L2 error of the surrogate model (with estimated parameters) and the true function. Each new sample produce another MSE. Then the MSE is a random variable which has some distribution. Why would the variance of LOOCV be greater than the variance of the MSE?
LOOCV test error is based off one observation. In other words, we are comparing the training model's average for the observation values vs the observation we left out. The one observation would likely be different than our predicted average. For an extreme example, consider if the observation that was left out was an outlier, then it would produce quite a large test error as it will be very different than the prediction. However, when we use less folds (like K =5 or K = 10), the test set will consist of more observations and the K-fold cross validation error will be the average of those test observation residuals (distance away from the predicted average). The process of using more observations in the test set and taking the average of the residuals reduces variance.
@@samadams6174 Here, the variance refers to the variance of MSE computed from each validation set. In LOOCV variance of MSE coming from each fold (k=n) is higher than the variance of MSE coming from less fold (k=5 or k=100).
Cross validation? More like "Outstanding lectures; thanks for the great information!" 🙏
Hello, I have a question - How do we define/quantify model flexibility?
Model flexibility is how well we can fit the model on training data, if each and every point on the training data is met by the curve, we are overfitting the model , that means the model need to have higher polynomial degree to reach every point. Higher polynomial degree means more flexibility.
Thanks @@venkatanarendrakotyada8167
At 6:01, you mention that LOOCV has a low bias and a high variance, but do not prove that. Is there a paper where this formally proved? Is the distribution of LOOCV known? Moreover, consider the mean squared error (MSE), i. e. the weighted L2 error of the surrogate model (with estimated parameters) and the true function. Each new sample produce another MSE. Then the MSE is a random variable which has some distribution. Why would the variance of LOOCV be greater than the variance of the MSE?
LOOCV test error is based off one observation. In other words, we are comparing the training model's average for the observation values vs the observation we left out. The one observation would likely be different than our predicted average. For an extreme example, consider if the observation that was left out was an outlier, then it would produce quite a large test error as it will be very different than the prediction. However, when we use less folds (like K =5 or K = 10), the test set will consist of more observations and the K-fold cross validation error will be the average of those test observation residuals (distance away from the predicted average). The process of using more observations in the test set and taking the average of the residuals reduces variance.
@@samadams6174 Here, the variance refers to the variance of MSE computed from each validation set. In LOOCV variance of MSE coming from each fold (k=n) is higher than the variance of MSE coming from less fold (k=5 or k=100).