MFML 069 - Model validation done right

Many ML engineers think they're doing validation when in reality they're shooting themselves in the foot. Here's what you should know in order to get it right.
Be sure to check out the rest of the MFML course playlist here: bit.ly/machinefriend
If you prefer to learn in bulk, you can get access to longer chunks of the course by signing up for the newsletter at decision.substack.com
Don't forget to hit subscribe+notify! If you found this useful or enjoyable, the best way to say thank you is by sharing it.

[@Abdullahkbc]

Hi Cassie, I am just in the middle of the validation of my model. When I was looking at the test set today, I was feeling that something is wrong. Now I watched your video and understood what was wrong. I will give up looking at the test data. But my test data was labeled manually and I am not fully sure how accurate it is. Your claim is totally correct but under the ideal condition which is not always the case. But thanks again for your clear explanation.

[@kozyrkov]

Comments like yours make my day. Best of luck for an excellent fit!

[@SornGeorge]

So as soon as you read the dataset, you split into Training - Debugging - Validating (60-20-20) and only look at the Training dataset for EDA and Data Prep/Wrangling (and model training of course)?

[@BLASBP]

If your data has spatial or temporal structure, random partition breaks such structure, and the model cannot learn from it. There really are no silver bullets for the validation issue.