Is pre-registration pointless? My talk to the Royal Society
Вставка
- Опубліковано 7 бер 2024
- Link to my replication crisis video: • Ethics in Statistics P...
Here's the paper this was based on: osf.io/preprints/psyarxiv/5vfq6
Come take a class with me! Sign up at simplistics.net
My Multivariate playlist: • Multivariate Statistics
And here's a paper I wrote about my eight step approach to data analysis: psyarxiv.com/r8g7c/
Undergraduate curriculum playlist (GLM-based approach): ua-cam.com/users/playlist?list...
Graduate curriculum playlist (also GLM-based approach): ua-cam.com/users/playlist?list...
Exonerating EDA paper: psyarxiv.com/5vfq6/
Download JASP (and visual modeling module): www.jasp-stat.org
Lots of great points in this video, and I agree with the thesis that pre-registration is not de facto necessary.
Interesting ideas, I always thought preregistration was essential... but you've given me food for thought. Thanks!!
I might do a follow-up video and say why it's still a good thing. My biggest point is that it's not essential for controlling probabilities. But there are other advantages to PR!
So is the primary issue with abuse that people are using exploratory methods to find something "interesting" (small p-value) using various hypothesis tests and then running with it? They're not formulating a model based on theory and they're not collecting new data to confirm what their exploratory research "revealed"?
Exactly!
@@QuantPsychgotcha. I remember doing this kinda thing when I was just starting to learning R and statistics and doing multiple linear regression on various out of the box R datasets, "star gazing" at the various predictors when doing something like lm(mpg ~ ., data=cars). I was basically just throwing random predictors into a model trying to push the R^2 up as close to 1 regardless of whether having the included predictors in the model really made any sense. Very much looking forward to your course. I'm self taught, which is great, but it leaves gaps and I'm left never being sure if I really understand something. Given that statistics is a very sharp double edged sword, I'm really afraid of becoming "that guy", if you know what I mean.
If you look at Gelman and others' (ArXiV 2011.01808) "Bayesian Workflow", which I consider required reading for Bayesian statistics, it is clear that their approach is largely exploratory.
Really enjoyed hearing your take around 8:30 about how pre-registration won't ultimately solve the problem. I saw so much p-hacking and exploratory analysis -> interesting correlation -> paper written that I became demotivated and ended up stumbling over the PhD dissertation finish line disillusioned with the whole field in general.
The real solution is significantly improving the average researchers' knowledge of statistics and expecting much more rigor, skepticism, and critique when papers are being reviewed, but that is a lot to ask of professors who review papers for free and oftentimes with the 'you approve my paper and I approve yours' going on. Even more than education, we have to have incentive structures that deincentivize publishing crappy work, but that is hard to create.
Agreed! Have you looked into registered reports? Those seem quite promising in at least shifting the incentive structure.
I learned a lot, especially on when pre-registration would be most useful. It's interesting how NIH requires pre-registration, though is every study funded within a "mature science" or are they still developing models within the study field?
Yeah, it might not make sense to have that requirement.
Do you have any suggestions for where to point people who reflexively run tests on all their exploratory analysis to get them started on doing more model building?
My UA-cam channel! And my textbook!
Even in "exploratory mode", isn't pre-registration useful also to make sure your entire work can be replicated? Like, that someone can check your exact instruments and premises and criteria, eventually in more detail than what's in the final paper? (Or do you draw a strong line between open science practices and pre-registration? I usually think of them both together)
I probably should have put this in the video, but yes preregistration is still useful. My big point is that from a statistical POV, it doesn't serve a purpose. But that doesn't mean it has other advantages.
obvious to statisticians, notsomuch to everyone else. So much YES in that statement.
Two quick questions. Isn't the concept of preregistration akin to establishing clear hypotheses in an experimental research before running the experiment (and gathering data)? Yes, some people 'cheat' and hypothesize after the fact, but what prevents a person from "preregistering" AFTER "fooling around" with the data first behind the curtain, discovering something theoretically interesting, then pretending (and "preregistering") the hypotheses as if they had existed prior to the data analyses? Not sure if I make sense.
That makes sense. It's not designed to prevent cheaters from cheating. It's designed for honest people from drifting from their original hypothesis. (And it's to satisfy reviewers). Make sense?
@@QuantPsych Yes sir! Makes sense. Thanks for the speedy reply!
There are some mathematical and logical issues that preclude finding a uniquely-best model for a phenomena based on data. With that in view, I doubt it is possible to have a final confirmation of a model.
Probably. That did come up in the meeting. One discussion we had is that if you have a truly mature theory, you've have a LOT of opportunities to refine your analytic decisions. Hopefully these decisions are theory-driven so that when you are ready for pure confirmation, the "uniquely-best" is almost a given. If you look at the cognitive modeling literature, they are there with many of their models.
If you look at Gelman and others' (ArXiV 2011.01808) "Bayesian Workflow", which I consider required reading for Bayesian statistics, it is clear that their approach is largely exploratory.
Is there an echo in here?
Ha! Looks like spam to me :)