Inside the Sausage Factory: How the USMLE Passing Standard is Set
Вставка
- Опубліковано 27 сер 2024
- How is the passing score for the USMLE determined? And why does the minimum passing score keep increasing over time?
In this video, I’ll answer both questions - and provide an overview of the Angoff standard setting method; the Angoff modifications used by the National Board of Medical Examiners; the differences between norm-referenced and criterion-referenced exams; and even review some questions from the original NBME exam.
NOTES:
1. I mentioned a couple of previous Sheriff of Sodium videos…
On rising USMLE performance:
A Brief Update on USMLE Score Creep
• A Brief Update on USML...
-On the appropriate context for the USMLE Step 1 pass rate:
FACT POLICE: The USMLE Step 1 Pass Rate
• FACT POLICE: The USMLE...
2. Here is the NBME’s survey on the “ideal” fail rate for the USMLE:
Margolis J, Clauser BE, Winward M, Dillon GF. Validity evidence for USMLE
examination cut scores: results of a large-scale survey. Acad Med 2010; 85 (10): S93-S97.
pubmed.ncbi.nl...
3. There’s also a vintage post on the USMLE pass rate on the Sheriff of Sodium website:
thesheriffofso...
_
MUSIC:
‘Clutch,’ by Nicolas Major.
Wake up, babe. New sheriff dropped
Its literally 4am here and im on it hahahaha
i'm up babe
Your brain explains things in a way that my brain likes.
That classic NBME actually sounds oddly fun. It's not your thing, but a video filming someone attempt one of those old NBME exams would be awesome content.
Yes please make the video about old medical school practices and exams
That OG exam is insane.
I was a little concerned that this might make me less angry about the score ratcheting (err, "setting") process, but nope, it's exactly what I suspected. Giving them access to real performance data is so transparently outcome motivated. All they've done is get craftier about how they keep the fail rate high. Of course they still "supervise" the process to make sure these experts don't go out of bounds.
Medical school professors like to get the “right” answer, too.
If that test was administered now people would fail it. I think you should cover how we went from such a comprehensive test to mcq test
I’m gonna add it to the list.
Yes please , make that video of how the medical licensing exam has changed
It’s going on the list.
cannot wait to watch this when i have a free hour. i LOVE your content (not even pre med or anything, but you are so engaging)
I would love to see a video explaining the evolution of the exams. As always, you put out the highest quality content on the platform! Amazing!
Awesome video as always. You can’t get this sort of analysis anywhere else. Love the part on the older exam content, it sounds so wack.
wojak faces? you're speaking my language, sheriff.
I really felt that nothing else captured the essence of the minimally-competent examinee quite as well.
I was worried about keeping my steak dinner. Masterpiece, as always.
Don’t you worry about that. (And thank you for watching!)
If failures decrease, profits decrease. USMLE retakes and all the NBME practice exams are more money. Inaccurate predictions provided on the practice NBME exams like the CBSSA and CCSSA will increase confidence in weaker performers, they will take the test and fail. The guarantee of profit is padded through multiple measures. I would love to see a profit chart laid over the Mean/Minimum pass chart. Are there complementary troughs to the peaks you pointed out?
❤🔥truth
Agreed , nbme is a monopoly with a "non profit" disguise that takes advantage of medical students because , well there is no other way , how do you justify IMG exam fees , its almost double of what a US MD grad has to pay, the playing field is not equal and its near extorsion.
based
Not gonna lie - as an elderly millennial, I had to Google this. But thank you.
Hey! Loved the video. Small correction: when you give the example that a 700 in a 500 mean 100 SD exam is 95th percentile, you are forgetting that you should only account for one tail of the curve: a 700 is 97.5th percentile, and a 300 is a 2.5th percentile
You are right - thank you!
It would be interesting to also look at whether the difficulty of the questions have increased over time. Comparing older prep material with new feels like there's a noticable difference.
I agree with most of this except that they’re missing badly with their finger on the scale.
State legislators granting autonomy and prescribing authority to naturopaths, PAs and now non-licensed IMGs suggests the policy winds favor individuals deciding for themselves who is qualified and who is not.
Not that anyone really knows how to do that, but their trust in the free market is absolute.
Very insightful video!! Please do share how we went from the comprehensive, multi-day style to what we have now. Thanks for the effort you put into your videos!
The graph with the memes 😭 lmao
not only the min. passing score is increasing, also the questions are getting harder.. so, a double whammy!
Please do a history video!
Love this video!!
history lesson another time would be good!
44:46 You mean the decision was made in these years? If overall performance is improving every year, the nadir points should actually represent the years the minimum passing scores were increased? Sorry if I’m confusing.
Yes, the red arrows highlight a year when the decision was made to raise the passing score - so the difference between the average and the passing score drops a little bit immediately thereafter. What’s interesting to me the slope of each of those segments before the decision gets made. The steeper the slope, the more rapidly they had to step in and increase the passing standard.
You didn’t mention that a certain % of questions are “test” questions that the nbme puts in with the real exams to see how they perform before making them real scored questions for the future. How do “test” questions factor in? Wouldn’t they be ways for nbme to see how well they perform and throw out the bad ones before they are official?
You know, for a while, I’ve had it in my head that I might do a whole video on USMLE experimental items… but now I’m gonna officially add it to the list. Stay tuned.
@@sheriffofsodium please
How much do you estimate the USMLE cheating scandal affected the national mean score/average? Do continuously rising student averages indicate better and better students, easier and easier questions, or more and more unscrupulous activity over time due to mounted pressure?
The typical mean that the USMLE reports is for first-time test-takers from MD-granting medical schools in the United States. So far, all of the examinees whose scores have been invalidated have been IMGs - so they aren’t included in those figures.
I actually have a whole video on what’s behind #scorecreep:
A Brief Update on USMLE Score Creep
ua-cam.com/video/1gKKAZ5aO8E/v-deo.html
Sheriff don't miss
20:20 thank me later
Bilateral cellulitis :D