170 - An Abbreviated Episode

Hilary spills a smoothie and coffee and Roger starts teaching again.

Show Notes:
Support us on Patreon
Roger on Twitter
Hilary on Twitter
List of NSSD Fellows
Get the Not So Standard Deviations book
Subscribe to the podcast on Apple Podcasts
Subscribe to the podcast on Google Play
Find past episodes
Contact us at nssdeviations @ gmail.com
Podcast art by Jessica Crowell

[@porktom]

In minute 20, you talk about how you might be able to compare analyst quality by their ability to intuitively evaluate counterfactuals on their models/analyses. Good analysts need to efficiently explore their hypothesis space and a lot of time can be lost when they don't have strong experience doing so. Do you think analysts can be misled by their intuition? What are some good methods to fight against past experience creating false confidence? Do you think that using past experience to drive analysis can be dangerous?
I feel that educated "guessing" to drive model development has caused a lot of issues in my own work. As an example, when you build a model, you might spend a lot of time fine tuning parameters, and your past experience might be helpful, but, there's a danger in coming to believe that you know how manipulating those parameters will effect outcomes. That's why hyper parameter searching methods exist. I've personally made the mistake of using intuition to take shortcuts in model development, when a better approach would be to write more code to use evidence to drive that search (like grid search).