The Flaws of Academic Statistics: the Null Ritual
Вставка
- Опубліковано 4 жов 2024
- notrelated.xyz
Nearly every academic paper published since the 1960s has used statistics known to be faulty.
That sounds extreme, but it's actually not even controversial in the statistical literature.
In the 1950s, Ronald Fisher invented a statistical technique to solve the philsophical Problem of Induction. Neyman and Pearson developed a technique for statistical quality control in factories. Yet somehow, these two techniques were confused and merged into the Null Ritual of today, which is the neurotic pattern that every paper in many disicplines have to follow.
The Null Ritual is one of the clearest examples of academic consensus so far off the tracks that scholar treat the techniques in textbooks were religious devotion, rather than with critical awareness of what they are actually supposed to be. The end result? Nearly every field is rife with misuse of numbers, publication bias, misunderstandings and fake conclusions.
Gigerenzer, Gerd. Mindless statistics. The Journal of Socio-Economics. 2004.
Fisher, Ronald. Statistical Methods and Scientific Induction. Journal of the Royal Statistical Society 17.1 (69-78). 1955.
Neyman, Jerzy. "Inductive Behavior" as a Basic Concept of Philosophy of Science. Revue de l'Institut International de Statistique 25 (7-22). 1957.
Halpin, Peter and Henderikus Stam. Inductive Inference or Inductive Behavior: Fisher and Neyman: Pearson Approaches to Statistical Testing in Psychological Research (1940-1960). The American Journal of Psychology 119.4 (635-653). 2006.
Perezgonzalez, Jose. Fisher, Neyman-Pearson or NHST? A tutorial for teaching data testing. Frontiers in Psychology 6. 2015.
Ioannidis, John. Why Most Published Research Findings Are False: Author's Reply to Goodman and Greenland. PLoS 4.6. 2007.
Francis, Gregory. Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology 57.5 (153-169). 2013.
Fanelli, Daniele. "Positive" Results Increase Down the Hierarchy of the Sciences. PLoS 5.4. 2010.
Gigerenzer, Gerd. Replication, statistical consistency, and publication bias. 2015.
Taleb, Nassim. A Short Note on P-Value Hacking.
Is this what you needed the manga guide to statistics for?
I had only 1 class in college discuss this in passing, which is terrible. This is an important topic for everyone to know about, especially since certain ideologues use Science and Stats™ to try to convince and sway academics and the general population in everything from academic publications to news articles where such works are commonly sited. I implore everyone to look into epistemological anarchism, The Replication Crisis, data dredging, and metascience to get a broader perspective on this issue and how decayed academia has become. Thanks for another great video, Luke!
🐧💻👍🏽
I seriously doubt there are "ideologues" that are hell bent on proving that women who are ovulating wear more pink and red, or that some gene/set of genes is associated with some disease.
Frequentist methods work if you are only interested in observing large scale effects. For example did countries that locked down early when Covid broke out have lower rates of transmission. Other non-frequentist methods are more appropriate if you want to understand the actual dynamics of something.
Of course the methods described in this video are unfortunately the rule. But also in science there is a considerable effort to use Bayesian methods, methods of quantifying dynamics in terms of self similarity etc.
@@bbbbrrrzzt5166 Anyone without research experience has no idea what "gene being associated with a disease" means, hence such news are almost always misinterpreted. And without the ability to critically asses the contemporary literature you are blind. Which makes almost every person who voices their's opinion on the matter ignorant. I don't think it's necesairylly bad that laymen aren't great at highly specialized difficult technical subjects, but they discuss them too frequenlty compared to their ability
Bring back Not Related if you care about your non-brainlet base
Alessandro Fadini seconded!!
@@javaman4584 fisted
Turkey on Christmas eve: I can't sleep, dude. What if we get eaten tomorrow?
Another turkey: Relax. On any given day we turkeys have a 1/365 chance of being eaten.
Also turkey: If we do die, let's be dry and put them in a food coma and take up useful space in the fridge.
even though i'm a collapsed-skull brainlet, listening to Not Related makes me feel smart; thanks, Luke!
In my university these concepts were explained as two separate things in two different classes. statistics and quality control.
Makes sense. Neyman and Pearson created this interpretation of hypothesis testing when working for industry, in quality control.
Today I was listening to your old potcast all day long in the train. I was sad that there where no potcast of the last year and was afraid that they wouldnt come back. Super happy to see one again. Hope you can make more :)
Luke when will the “how to pick up girls” videos are coming up?
You should be waiting for his "how to speak English properly" videos instead
Just go outside it won’t hurt. Not saying that will solve anything directly, but you gotta be out there for women to realize you’re available. I mean actually budget and do something social to do on a schedule that’s easy to follow. Once you do it enough you feel comfortable in the space and more willing to open up. When people realize you’re always around, they also feel more willing to open up. You don’t just expect things to happen because you deserve them because that’s a turn off for most women. Is any of this helpful?
"Lift with the legs and not with the back" there you go. Solved.
Go to church.
Probably never considering recent youtube bans for similar content. Even rsd mostly purged all their channes on yt.
You understand that statistics are messed-up when you hear statements such as “We never accept the null hypothesis... We just fail to reject it.”
Would love for the podcast to return
Unfortunately this ritualizing isn't unique to academics nor statistics. People end up having blind faith in all kinds of mathematical tools they don't understand (machine learning, financial modeling, etc). Applying statistics, as well as any other kind of mathematical modeling of real world processes, is necessarily very subjective as we must make simplifying assumptions about the world to make sense of it. Always triggers me when the results are presented as fact.
I found about the "Not related!" podcast about 2 months ago, fell in love with it and thought that it would never be brought back. Keep up the good work, genuinely excited for the new episodes
I have been waiting for another long form video for so long.
Appreciate you creating this and sharing your references.
Quite new to the channel. I enjoyed this very much and would indeed be quite happy about another episode
btw, i use arch
Plan 9 anyone?
Thank you for being my primer into this subject a couple years ago. I am now writing a small paper in my philosophy class closely related(ironic) to this topic.
There are ways to overcome this problem. But researchers need to know about them. There are techniques that correct multiple comparisons, which essentially is the crux of the matter here, but it’s not widely taught. It's mostly taught at Master's level.
this isn't closely related to what you're talking about but it reminded me of some of the gremlins that exist in deductive reasoning as well. axiom of the excluded middle is a fun one. if you reject it you lose a huge chunk of mathematics but accepting it leads to some profoundly unintuitive results like proving that there must exist logical statements which are simultaneously true and not provable.
i guess its more of a cultural parallel. the grandest aspiration of western science since the enlightenment had been to construct a "theory of everything" and it took until the 20th century for this dream to die. seems like most of the academic world in the early to mid 20th century was fixated on establishing whether or not anything they were doing had epistemological basis. idk if you know all about hilbert's program already, but essentially hilbert was a mathematician/logician who wanted to come up with a finite set of axioms to serve as the foundation for all of mathematics. essentially, he wanted to be able to take any theorem and "decompile" it in some sense to an immensely complicated but (importantly) finite string of simple logical operations. long story short, this is _provably_ impossible. like i said, having theories of everything fall apart spectacularly wasn't unique to math but you couldn't get around this one with a "we'll figure it out some day". it was a cold deductive fact. i feel like a lot of where science is today stems from a struggle to reconceptualize what science is actually _doing_ if not revealing the objective properties of the universe as we once thought it did.
@Thomas B things are only "true", in the rational sense, in relation to an axiomatic context, and the choice of these axioms is subjective, mutable, and arbitrary. if you are looking to say something more objective through rational analysis, then you're barking up the wrong tree.
FINALLY! More of not related please!
Now this is the content I truly want
More on black swans and the turkey problem 👍
Yes not related is back! Been waiting for this
Induction is making a generalization (out of a specific case).
Deduction is making a specialization (out of general case).
Luke, I believe you are going to have +1mi views soon...
Very good content everywhere, in your pages, codes, videos, and mostly ideas.
A linguistic which knows computing and is a wizard on linux, woah.
I'm using your setup on my arch distro. Thanks for that too.
Ayy, he's back
I was looking for this episode and couldn't find it
Thank you Mr Luke
Finally the podcast is back
Bring back the podcast luke! We need it!
I agree with you that statistics and data should always be published and that p-hacking is a reak problem.
However, i don't think using the fisher method is any better than what is being done now in the so called "null ritual". It is still possible to bias your p-value. Also, you mention that type 2 errors are less costly than type 1 errors. However, type 2 errors or False Negatives can be very costly. Imagine you are diagnosing patients with cancer. Not diagnosing a truely sick patient (false negative) is much worse than falsely diagnosing a healthy one.
Lastly, you mention the curse of dimensionality and how in GWAS studies it is super easy to find false correlations because of the amount of genes that are involved.
The thing is that in these studies p-values are adjusted depending on how many genes were involved so that overall, much less genes are found to be statistically significant. So i dont really get the point here.
As someone who does a lot of tech writing you captured my exact issue with this Luke's conclusion. He makes it seem like there is a massive crisis where the bulk of people in academia don't know the flaws inherent in the null hypothosis, and blindly follow it, but my first hand experience is that researchers and academics very much DO know the limitations, it's just that the "null ritual" often proves to be a better model than alternatives and a great deal of work goes into ensuring that the p-value adjusts to the kind of data sets they are using.
The issue is that hard declarations in findings are rarely made by the researchers themselves, and more so that their findings and models become codified only in the minds of the general public after the careful filtration of information through the Pleb filter.
@@iannordin5250 it appears that applying the null ritual requires 1. binary outcomes, and 2. costs with which to weight the outcomes. I think almost no scientific application of the null ritual has the latter; principled Bayesian inference seems both easier to explain and sounder.
Boomer makes video of a stream with a video feed, then proceeds to just use 1 picture for the whole video instead of the video feed.
needs to be a pure podcast mode
pretty good explanation of concepts behind using suckless
Isn't this what Derek from Veritasium was talking about in his video "Most Published Research is wrong?"?
Hello Luke, I am fairly new to the Linux scene, getting into it last year.
You are constantly talking about bash scripting. I have heard there are benefits to using other scripting languages, like Perl. I’m wondering if you have any reasons for not using Perl, on your channel. Thanks!
There are three types of lies in the world: lies, damn lies, and statistics.
I would like to see Bayesian inference as the standard. Frequentist statistics has its place, but it is like driving a Model T Ford compared to the Lamborghini of Bayesian inference: Markov Chain Monte Carlo. p values need to die in peer reviewed literature. Immediately.
I tried increasing my pp-value through natural means but so far the growth has been more of a null ritual :(
Finally a podcast! Take care, Luke
You encoded the other podcasts as MP3 instead of OPUS. That's pretty cringe bro
You got me. I was going to switch mid season, but I figured I'd finish this season off in mp3. It's easier to do tagging automation for mp3.
That's true for a statistics class (that you'll be taught "wrong"), but once you understand multivariate calculus and beyond you can see in a more general way what assumptions may be broken, ect.
Real math is good, however just trying to use it for an end goal can be destructive.
Ty mr smith, lets hope this is back for good
True, like HBO's True Detective.
The academic/statistic branche as a whole should introspect themselves. Specifically users of software like SPSS (since 1968 and bought by IBM Software for Business Analytics since oktober 2010) and Wolfram (since 23 June 1988).
The filosophical problem regarding induction is hidden in the esotericism of thermo-dynamic laws.
Btw.
This year is a Leap Year. These years are the most integer. But the 366th day doesn't add another month, oke.
Btw2.
Bobby Fischer also was a controversial chess player, a prodigy (1972 -1975 World Champ) and enigma at the same time, even before GNU Chess (since 1984). Even the ELO-rating isn't perfect.
What's the main point?
That people don't know the precise meaning of p-value?
That it would be better to publish the p-value's value rather than saying "the p-value was lower than some confidence threshold"?
That often people "build rituals" around things that they don't know well?
That sometimes studies are published even if the sample size is N=3, or that N is chosen so that you can claim certain statistical conclusions?
I liked the episode, don't get me wrong, but I'm finding it a bit confused.
I know it's an old post but the answer is yes to all of them and you even missed a few. There are many problems in academia today. Some which Luke didn't even get the time to mention.
This is an hour long podcast and not 3 minutes for a reason.
If you add more on eduction and conduction this video will be able to watch and truly understand Rick and Morty.
Great explanation Luke
Better approaches will follow general rules like 'no models are true, some models are useful' and explicitly state that there is always a role for judgement and theoretical presumptions in those judgements. Maybe I came across that because I did econometrics and the problems are clearer there.
Not Related is finally back!!! Hoooray
Bring it back!
This was surprisingly interesting. Would love to know more about that stuff. Can you recommend a channel telling more about the history of statistics?
you should recursively read the papers cited until all knowledge converges into your mind-meat
@@Willwantstobeawesome At the end of that, your brain will have so much knowledge it'll collapse into a black hole, much like the brainlet wojack in the video
Read Taleb's Technical Incerto Collection
I agree that much of probability you see in research contains basic errors, aside from being statistically dubious, some of the work I've been doing lately suggests that nature and humans(unconsciously) do use something like a p value to determine various things, but the significance is ridiculously high, like in the billions or trillions
Sounds like we should all just embrace Bayesian statistics
Still waiting for Gnu/Gardening
very energetic intro
If any physicist presents a (((P-value))) below roughly 10^{-7}, he is laughed out of the room. I've been in colloquia when in happens.
We don't use those because there have been many ephemeral 4-sigma results in practice.
Bayesian statistics are based, and your acceptable odds ratios are probably too low.
Bring the podcast back!
I have a small P value :(
I don't see why should there be anything wrong about using 5% threshold and what is wrong about having to adopt or reject a hypothesis. It's obvious, that people have to often make a decision which is digital in nature, like proceed with drug adoption or not. Of course, it is always better to study (especially such) problems in detail, and do it based on a statistics that would include meaningful sources of information. One cannot simply neglect that procedure, because 5% is set arbitrarily. You can set it lower, and you can look at the p-value itself, but at some moment one has to translate it into an operational decision. There are also other methods, that would attach your beliefs into the whole analysis, but at the end they need to have some sort of decision process too. I think, Luke attaches to much importance to people doing what they have no idea about it and deifying the whole process, which obviously sucks.
*Watching entirety of 5 minute ad*
*Luke Smith liked that*
Provided he actually gets a slice of the dough.
Thonkpad question: My X220 won't boot past grub but installation goes fine. Anyone have thoughts?
@@MantasXVIII | Sometimes reports 'cannot read outside hd0' or similar. That seems to suggest that it's looking at wrong disk or SATA connector is bad, but a) I booted through GRUB just fine, and b) there's no other drive. The drive worked fine in my desktop as boot drive, 1yr/old ssd. idk
I had a similar issue when I was installing Arch on a new SSD. I honestly don't know what the issue was, but I was previously using the linux-lts kernel since I was following a YT tutorial. I re-installed Arch using the vanilla kernel and some firmware recommendations as prescribed by the archwiki installation guide and I had no issues after that. Also my previous issues with drivers were fixed as well.
this kid is really special!!
Do bring it back
I like this channel but is anyone else triggered by the slightly distorted audio?
Edit: Apparently it was only in the beginning.
Maybe I don't see things from non-learned people, but my hunch is unless you can grasp why the normal curve is what it is and how to *carefully examine hypothesis*
Your intuition probably beats trying to cite studies
And I'm not talking binary option models, I mean as the most natural average as averages approach infinity
You're like Varg with the voice of Dan Barrett
The hype is real
This kinda reminds me of an idea by Carl Jung. Jung was saying that theories, especially political theories, that are formed from statistical analysis always form a paradox, in-fact, it forms numerous. But the one particularly concerning Jung is the conflict between the collective unconscious and the individual. When the individual, in consequential essence, is just a single number within a greater statistic, the uniqueness of the individual dies, and in coming time, the forming archetype would die also
TALEB MEMES INCOMING!
great papers, how'd you find about them?
on the web
So, dies everytime you see a non-black bird dies it help proove a raven us black (logical induction sucks, but is necessary at tines)
*does
Yeah, p-hacking is almost encouraged
The sample size p-hacking is well known, but the Null change up is interesting. I think we should demand proof (some kind of block chain or encryption) of there assumptions and hypothesis were layed out before starting the expiriment.
link to the video version?
@LukeSmith I really want yo meet you man, turn you on to sone of the higher more abstract mathematical uses. A huge oroblem is people who go into the sciences "hating math"
I guarantee, if you hate math you haven't done enough. I blame instructors and credit a great one at a community College for my live of Mathematics.
Lovely!
Luke's comments w.r.t open sourcing R/python/STATA/SPSS/Matlab scripts and data sets to promote reproducibility @41:00 things are improving very slowly only not in softest social sciences... for example: dataverse.harvard.edu/dataverse/harvardopendata
Most scientists hoard their datasets to exploit it fully for their own publications and prestige. Taleb is right about academia, its full of rentseeking careerists.
Hilarious, these fools just averaged together two entirely different theories in order to come up with a system on how to average things together. At least its consistent.
but my n > 30, so idc.
Michelson & Morley.
Very interesting podcast, thanks for the upload. Does anybody have any recommendations for aspiring researchers, who are forced to participate in the ritual if they don't wanna be eaten by big p-hacking sharks?
46:02 Great, now Luke is better than me even in burping
*AcaDAYUMia* intensifies 1:07
bUt ItS A SoFt SCienCe
bring back the wadacst
based
i heckin' luv youtube!
bring it back bring it BACCKKKKKk
I was looking for this lol
*stonks ↘️*
Aaw shit, here we go again
thanks based npr
yay
acadamia
sronk
First
no not the p values
Bring it back !