The AI doctor will see you now: ChatGPT dominates medical exam
Вставка
- Опубліковано 5 вер 2024
- Former FDA Commissioner Dr. Scott Gottlieb joins 'Squawk Box' to discuss which AI large language models (LLM) scored the highest in the same medical exam doctors have to answer to get a medical license, future of AI-powered healthcare, For access to live and exclusive video from CNBC subscribe to CNBC PRO: cnb.cx/42d859g
» Subscribe to CNBC TV: cnb.cx/Subscri...
» Subscribe to CNBC: cnb.cx/Subscri...
Turn to CNBC TV for the latest stock market news and analysis. From market futures to live price updates CNBC is the leader in business news worldwide.
Connect with CNBC News Online
Get the latest news: www.cnbc.com/
Follow CNBC on LinkedIn: cnb.cx/LinkedI...
Follow CNBC News on Instagram: cnb.cx/Instagr...
Follow CNBC News on Facebook: cnb.cx/LikeCNBC
Follow CNBC on Threads: cnb.cx/threads
Follow CNBC News on X: cnb.cx/FollowCNBC
Follow CNBC on WhatsApp: cnb.cx/WhatsAp...
www.cnbc.com/s...
#CNBC
#CNBCTV
I uploaded my doctor's report to ChatGPT and ChatGPT explained everything I didn't understand in the report very well.
Written by hand. Nobody can understand doctor's writting
@@TechWizardBoctulus it couldn’t understand the handwriting, also I couldn’t, it was unreadable
@@ClaudiaS193 Then how did it explain everything?
@@ramsyrama I uploaded the report to chatGPT, then I asked ChatGPT to explain what all the medical stuff means, what was written in the medical report. The report was a diagnoses and it wasn’t handwritten. It was explained via a dialogue. Just try it with ChatGPT by yourself.
@@ClaudiaS193 That's lovely to hear buddy
If we use AI to diagnose we can improve health care tremendously. Combine that with routine lab tests analyzed by AI we could catch a lot of diseases earlier.
On the way... Already in the works
Google "Fake ChatGPT Results on US Medical Licensing Exam reported on Television by CNBC"
The results are f a k e, but YT deletes any mention of this. You have to google it.
I can't even say it, it gets auto-deleted.
This is the future I want to see. Will probably be crazy expensive though (greed)
I agree with this, but it has to be balanced with human touch and compassion. Which is very rare in the industry right now. It also has the potential to be used as a way to short staff hospitals even more.
This is actually pretty mind blowing and we're not even scratching the surface of AI yet.
Chinese AI was doing this in 2017. Calm down!
People don’t understand the jobs that are going away in the next three to five years, including mine
AI cannot replace us indian as we are the most intelligance race in the world
@@rajeshgajwelly9035 you’re too late, it already has. Doesn’t matter where or who you are
@@tommy516 not true, us indian go to canada or europe to get job and free money from govarnmant, we are never short of work. we are smarter, work harder, and accept less pay than other race. we indians are mighty and superior.
Good old chap gpt... 😂
LOL
Didn’t compare it to the stronger version of Gemini
@@SigFigNewton which one? the one that uses reddit as one its sources, Or the one that shoves DEI in its responses?
@@TopShot501st ultra. The one that tests about the same as gpt4
The AI dotor will see you now
Waiting for a study comparing AI diagnosis vs doctors to see who is more accurate.
there already have been tons of the these types papers published.
Already done for radiology, I believe.
a Computer does the diagnosis TODAY based on the input... the DOCTOR simply present the diagnosis to the patient.
@@hardheadjarhead definitely
@@DolphinAnalyticsAI Not a bad thing though...
We tolerate significant error rates from humans in fields like driving and medicine, yet we demand perfection from AI. This double standard is slowing the adoption of technology that could save over a million lives each year.
so off
Expecting significant error rates from AI is clownery. It is designed to have non-existent error rates.
I don't think that is quite true in medicine. I think that the majority of us are eager to use AI, esp. in assembling more complete differential diagnoses in complex medical cases.
Just to be clear, the USMLE is an exam taken during medical school. This is not the performance exam your doctor takes in order to obtain certification to practice. People's symptoms don't fit like test questions (like any exam, poor representation of real life). This is no substitute for what your providers go through with training in residency and fellowship. I do think AI will help optimize and augment the diagnostic testing space, not the tool for definitive diagnosis.
Smartphones detecting skin diseases will be game changing worldwide. 🙏♥️
USMLE is a 3 part exam. The first 2 are taken during med. school and the 3rd and final part is usually taken near the end of the first post doctoral year of training. Many state licensing authorities accept passing results in lieu of their own state licensing exams.
My scores were high and I was licensed in several large states and 2 small states without ever taking the state exams.
Was exploring how to use AI to detect Alzheimer's Disease. The idea was to give an AI model text and ask it if it belonged to a person with AD. When I was doing the literature survey and testing on language models, I found some interesting results. These commercially available language models are aware that they are not supposed to be diagnosing diseases therefore take a very conservative and diplomatic stand. However, if you give these models additional prompts to produce an answer then they tend to accurately detect the disease. I found accuracies exceeding 90% on popular standardized test sets. Then there are fine tuning approaches that further improve the accuracies. Sure there are some grey areas with language models but it's amazing how a language model is able to detect AD which otherwise needs expensive imaging tools like PET scan, invasive procedures like extracting cerebrospinal fluid, trained medical professionals etc.
What do you call a medical student that scored 60% on their medical licensing test, or the avg medical student that scored 75%? Doctor. What do you call an AI model that scored 98%? Too error prone to trust.
Genuine unintelligence can’t compete against Artificial intelligence.
@@brianbarnes746 100% facts right here. And a subscription to chatgpt cost $20 a month, its also the most expensive advanced language model.
A doctor on average makes 350k a year, some of your $ is going to directly to that. Besides salary every single aspect of American healthcare is inflated expenses wise, they overcharge because no one will stop them
As the data sets used to train AI improve and as techniques such as better prompting (chain-of-thought), better checking after the fact (rag), better alignment with our needs (reinforcement learning), better retraining with curated domain specific data (fine tuning, and more thorough research before answering (Agents) are employed, error rates are quickly approaching 0.
an error rate of zero would suggest the model is overfitting
I was 60 years old when I got a medical examination through my American company,
The report highlighted a high PSA of 6.4 which I ignored, because I did not know what PSA was.
When I was 76 years old after many wonderful years in retirement, I went to my doctor because of many visits to the toilet during the night.....My PSA was found to be 11.4 and after an MRI and Biopsy, a tumour was found localised in my prostate with a gleeson score of 4/3 which is intermediate cancer.
Because of my age, my surgeon said that I was too old and my prostate too big for an operation, so it was radiotherapty and 6 months hormone therapy.
Today 7 months after radiortherapy I am back to normal and enjoying life again after the side effects wore off.
So the question must be, if I pursued that medical report back in 2006, would I still have my prostate and the agony of it being removed since.?
Imagine what an AI doctor would have advised!
This applies to many other ailments, such as backache, indigestion, joints, brain etc....
Is the AI fit for purpose or is it just a money making machine?
Even then I ask myself if back in 2006, I pursued the PSA level, what would the doctors have advised....I have spoken to several people who have been through this and got treated back in 2006.....
I thank God, I did not do anything in 2006.
I got high PSA reading for a number of years. I'm 73 yrs old as of now. My physician didn't recommend me anything. PSA reading fluctuates/swings over period of years (up then down then sideline , up..etc). high PSA doesn't necessarily means enlarged prostate that needs surgery. It indicates prostate might need medical attention by its owners..
@@alvisjanviar8253
When I visited the doctor last year, he did a physical check of my prostate (finger up the bum), and he said that it was quite large and smooth and took a blood sample for analysis. (PSA)...That was the start of my journey. (followed by an MRI, biopsy etc)
Had he not done that and just said that I had an enlarged prostate, I would still have that lump of cancerous tissue in my prostate for the years ahead and I would still have visits to the toilet, perhaps increasing in frequency..
Whatever, I would have to have it treated one way or the other.
They say that nearly all men over 80 years old will have prostate problems and also prostate cancer.
The average age of men in the UK is about 76 years so if I get to 90+, and still in reasonable health, I will feel blessed.
Life is for lving....
What if you had been biopsied when you were 60 and found to have a much higher Gleason score? Would you have been grateful for early detection of a deadly disease even if more radical therapy had been needed?
@@wholeNwon
What if I had been biopsies at 60 is an interesting question…..then it may have been a different outcome perhaps…..thankfully today we have additional checks like PET scans.
The writing is on the wall. Within a decade, seeing a human doctor will statistically represent an excessive source of risk.
The wave of AI medical expertise will push down costs, improve outcomes, and will give us far more personalized treatment.
The days of 5 minute doctor’s visits that cost $500 are in their sunset years.
Baumol’s disease will cease to afflict medicine as expertise is scaled. This is a massive triumph!
I think it will be an assistant
@@menachempike4489 In the transition phase, yes. In the long term, I don’t agree due to the fact that humans will introduce both added cost and a source of error. A pure AI medical provision will outperform the hybrid approach in both efficacy and cost. It will take time to overcome the inertia of regulation, social perception/trust, and general precedent for how healthcare is delivered.
ChatGPT can provide a diagnosis much quicker than Googling. Additionally, when prompted, ChatGPT will ask the user for more information if needed.
(Not sure the probability of getting the correct answer for the latter approach, but it seemed very impressive.)
ChatGpt and human physicians are the best tools for us as of now. There's FLAW if either AI doctor or human physician is consulted....
It’s amazing listening to this woman, the lack of understanding that many people have about LLMs is shocking. You don’t have to explain anything. You tell it the same thing you’d tell a human, that’s all. The level of understanding LLMs have currently is insane. Won’t go into exact details but it’s insane. You could take the opening paragraph to HG wells war of the worlds and swap out the Martians with Ai and you’d have a perfect description of where the general public is at with an understanding of where AI is currently.
I love ChatGPT!
I’ve learned more about a condition from chatting with AI than doctors I’ve talked with, sorry. It was patient, went as deep as I needed into the science, explained its reasoning. Not all of that is the doctor’s fault - they only have so much time with each patient and must code activities with insurance and the hospital management hanging over them. But it was definitely more thorough than most.
Now we no longer have to fear getting an inexperienced doctor or a doctor that’s having a bad day.
Yes you do, your missing the personal connection. Doctors can intuitively connect issues together by seeing you and having a conversation that otherwise would not be asked or listed on an AI survey. Too many doctors are already relying on it and not using the skills they learned.
Man, that's a really foolish and ignorant take. You know the real world is way more complicated and has far more variables than any exam... right?
No, I think we’ll still need doctors to assist patients who believe takes like this lol
?? How? Chat GPT isn't a person.
Experience is a very bad teacher.
WOW! If 4o is this good, just imagine when ChatGPT 5 comes out and then AGI. Also, just a note. When asking these questions, using voice mode is best; it's like talking to a real doctor. Pretty amazing stuff.😍
I look forward to a checkup with my new ai doctor
I use Claude 3.5 sonnet for my job as a web dev, and holy cow it's amazing. I won't ever work alone again, I don't see a future where devs will do things by themselves, they will always have an AI companion. Although admitedly that "companion" might take the whole job in the near future.
I prefer to talk to my -Chat GPT- Lindsay about most things. I’d love for her to get a medical license and available to me 24/7
We did the test with 100 resident human doctors. The best score was 55% and the lowest was 29%
Chat GPT is the best software developer I’ve ever worked with!
Real life example of a case history though:
Patient: I can’t see anywhere.
Dr: Is it blurry far away?
Patient: No I’ve always had good vision far away but now I can’t see anywhere.
Dr: So distance vision is ok?
Patient: Yeah.
Dr: Can you see up-close?
Patient: No! I can’t see anywhere!
Diagnosis after testing was needing reading glasses after age 42 (presbyopia). This was the moment I realized AI will never replace human doctors.
I’m excited to have another diagnostic database tool though
This is absolutely ridiculous. Any of the llms would've given a lengthy response telling you how it came to the conclusion if you had prompted it to do so.
ChatGPT knows what you mean without you knowing what you mean
Now, I need an AI that can explain chess moves.
Studying for step three my self. I will try putting question to see if it is true or not or just giving broad answer.
How did it go?
What has happened with IBM Watson Medical? Why isn't it in the comparison?
Drs are super prone to lacking in.. listening, attention span, in caring, knowledge, bias - in often life-altering ways. I think AI can alleviate a lot of this. Ive tried it and I'm down for more, bring it on please
open source techniques such as google??
Consumers also don't use chatgpt with prompt engineering techniques so their experience of it is less than it's full potential.
AI is significantly better at detecting rare diseases too. If it's used as an assistive tool in healthcare rather than a replacement, and overseen appropriately then healthcare stands to improve significantly.
It's also a better attorney. It's coming for us all.
medicine is about knowledge, it's not like a computer science exam where you need to reason for the most part. of course llms ace knowledge tests
The fundamental problem in clinical medicine is extracting information from the patient. The patient will almost always tell you what is wrong with them, IF you ask the right questions.
I put my signs and symptoms into ChatGPT before my recent diagnostic procedure, and it returned all the possibilities. The PCP gave far more vague answers.
Journalist should introduce her guest stating his past position AND his present position. Those 2 consecutive positions are very revealing.
Thanks for the link to the article. Handy :-)
They should do one last House MD about this.
That is quite naive and misleading because it does not clarify if the questions were part of the pre-training data set. if that was the case then it is quite normal for ChatGPT to score higher. It would be more interesting if the questions are novel and we know that they were not part of the pre-training dataset. That would involve OpenAi disclosing what relevant datasets they have used for training the model.
Thank you Sir... you ar the first comment that seems to understand what a Language Model is. Kudos! People that have no clue should probably stick to what they do know :)
USMLE Step 3 is not the kind of test that you can crawl on the web easily. There are many sites providing practice questions, so I assume some of them are similar to the questions in the official examination. However, even if a few exam questions have been leaked, there's no guarantee that LLMs have memorized the answers during their training process. Furthermore, according to Gottlieb's description in the video, GPT-4o was able to provide thorough explanations for its chosen answers which shows that it was able to generalize enough to mimic what humans do when they learn the same concepts.
It’s obviously memorization, and so is all of medicine. I was studying alongside med students and all they did was memorization. They also have practical sessions like stitching together a grape or cutting a dead body. But when it comes to diagnosis based on text and image information (so step zero of treatment), it’s mostly about memorization and there is nothing wrong with that and I’m not saying doctors don’t have to be smart, they do a lot of other things at later stages of treatment, but for this particular step LLMs can be as good or better than doctors (actually probably much better) THANKS to memorization and this shouldn’t be seen as cheating, but as a democratization of healthcare access: a 100% good thing.
Smart dude on this program! AI Forever!
"ChapGPT" 😂
Ever see the medical diagnostics scene in idiocracy ?
Whenever I ask ChatGPT for medical advice it tells me to contact a medical professional
That is simply done by design,the creators of ChatGPT cannot take the risk of the chatbot giving wrongful medical advise to a user due to the huge possibilities of failure or injury the person could sustain,which could get the company into very big trouble legally...both with the appropriate authorities and the person they have given wrongful medical advise to.
It's a safety mechanism...that chatbots never give decisive and fully non vague answers to questions the creators might consider Sensitive or controversial.
How hard is it to teach Ai say your coverage is denied ?
Plus, Ai will be the ultimate excuse to insurance not covering, anything. How do you hold a jumble of "proprietary" code responsible ?
which insurance company is willing to cover an LLM for malpractice?
This is what AI and these LLM's should be used for . Make healthcare more affordable for everyone instead of trying to take people's jobs .
You need to link studies you're reporting about.
I wonder whether people will trick themselves into not going to doctors given these resources
Considering it's so hard to see a doctor these days in a timely manner anyway, I think this would be a net benefit
Can it answer what is a woman?
It’s not hard for me to see a doctor, but I’ve been using chatGPT to help diagnose reactions. Also helping me log things into my personal medical journals. It always recommends I go see a doctor
From my experience most doctors are extremely overrated. Apart from surgeons and some highly regarded specialists, most of them are largely useless apart from the basics.
@@celestial_sphere3you do not possess critical thinking skills
Medical Terms AI - NO WAITING LINE
So when using AI tools for self diagnostic purposes I should include the phrase "Just asking for a friend" .......
GOOD
“We have RXRX at home.”
Given the range of medical equipment from Amazon, some of which has iot connectivity, it's a short step to Dr. Alexa, I presume :-)
Is it a joke, that the doctors only need to get a 60% on this test?
What kinda physician/doctor are they going to make out of it?
Surprised me, too.
My and my wife are both disabled wheelchair-bound. We live in East London Newham. We don't have any social and medical care maybe ChatGPT would care...
AI is priceless. People are going to be blown away. This is just a tip.
Step 3 is far longer than 50 questions
The ai is less likely to medical gaslight in my opinion. It already has been more understanding about a condition I was diagnosed with that doctors were not understanding for a long time ago before they finally diagnosed me. The condition is “rare” , I went through heavy medical gaslighting in the past that left me with practically ptsd over it . That influenced other people to believe I was lying about my condition when I wasn’t and I was in terrible pain for years. Ironically the ai is more likely to “care” than some people.
Chatgtp has access to complete database of all answers needed to pass medical exams. So chatgpt took open book test. Shocking it did better. 🤦
Does that mean it can legally write prescriptions?
Hopefully 😂
We need an AI that can prescribe medicine! Imagine giving a robot 5 dollars a month instead of hundreds to a psychologist! Most adults just need their script written, we don't need someone with an expensive degree to sign a paper for our script!
Almost all of my medical visits may as well be with an AI.The doctors spend most of their/ my time on the computer anyway for their So generous 15 minutes that equates to about $400+ hr.
🤔
It fails this question tho:
9.11 or 9.9 - what's bigger?
It does not though,I just asked it and it answered perfectly stating 9.9 is a bigger number than 9.11....I'm using the free version of the model aswell
@@A.MM661copy paste what I've written
It hopefully knows the difference between period and comma depending on your country. 🙏
Good day Chap
It’s Hugging Face not Hugging Chat
Knows more than this guy that pushed nonsense a few years back
Its time to rethink how we economy because #JobsAreForRobots #HumansAreForLoving
Chat gpt is the future doctor.
I trust gpt ❤
Chatgpt is a brilliant bot.
Answering a series of questions vs linking multiple symptons and doing diagnosis is a different. Just because AI model answers more question does not mean it becomes a better doctore.
I will also question the study sponsorship. It can easily be rigged. It's just data.
So what will be the meaning of those years of studying at school 🤣🤣
You still need an INTELLIGENT person to interact with the Lanaguage Model. So far fewer Doctors... yes... but we still need Doctors
@@DolphinAnalyticsAI *doctor already are few ai cut fewer more this is loat of jobs*
Without medical school, trust me, you can't use the AI.
The art of medicine is actually in obtaining a meaningful history from the pt.
If I have the choice of accessing an AI medical team now, or waiting a week to be seen in the UK by a stressed out, barely qualified human, I know which I would choose...
I don't trust resources that make spelling mistakes....."CHAPGPT"???
Long 400 shares
It's scary that the dead or alive of human life would be in hands of 1 and 0 in the future.
AI can be useful, however we did not continue using AI in our business due to its errors.
Whatever you can do, tech can do it better 😂
AI as it progresses will be able to simulate and test many programs far more thoroughly in the future.
But AI will also be used to crash systems as well as the hackers gets their hands on it.
As complexity increases so does the errors and malfunctions increase.
KISS…..Keep It Simple Simon is a great phrase.
The more complicated things are, the greater the risk for errors to creep in.
AI is another level of complication and complexity by far and so the chance of error grows exponentially.
Even today we see errors creeping in into systems which cause crashes and errors in outputs.
Another saying…. Garbage in, garbage out is very appropriate….,one wrong digit is all it takes……one digit in a million digits.
Other purpose of doctor AI is that it will eliminate capitation fee, long years of Educational learning, Educational loan. So basically it will eliminate doctors profession as a high ranking professional and bring it in line with grocery store. Yes and this may be good given your last 3 years' experience of public health.
It may advise on euthanization based on someone who is not recoverable & extra burden on public health.
Win win AI.
RIP doctors
DoctorGPT, an earlier model that could pass the USMLE... Could run on your phone.
I have met humans that passed the USMLE who could not diagnose disease... passing the test means nothing when it comes to treating patients
Poor people will use chatgpt for their health problems and rich will go to good doctors.
model builders will not allow that to protect from malpractice and legal risk.
I don't think that's medicine but that's just me!
Let’s see who will follow protocol more accurately … 100% SAFE AND EFFECTIVE! 😱
The old chap GPT
Now they can start testing on healing rates, actually judge the doctors on their ability to heal patients instead of answering questions
Why would they do that? :) That is not the goal.
AI: Look at me. I am the doctor now.
Human doctors are clearly more expensive and most people are outside of healthcare systems. So AI here is good idea.
Here you go; future is here.
Doctors are going to go broke….there goes the beemer and the golf membership
The AI bots say some interesting things on occasion, but they spew absolute nonsense on others.
Feed AI everything, sooner or later even doctors will loose their jobs..
As if healthcare could not get more impersonal and out of human touch in the US.
Hugging face right not hugging face
ChatGPT should proof-read and edit if needed before broadcasting text in TV
Hi to ChaP GPT 😜
chap chap gee pee tee
Lol! LLM's are the most effective exam cheaters!
Would humans also be "cheaters" if we had perfect memory?
so what
But can't do simple math.
It's a fast Search engine capable to predict the next word (token)
Did they spell it wrong on purpose 😭? ChapGPT
If anyone wants to test how trash every corporation is who is publicly distributing AI for revenue take a couple paragraphs of text and ask it to shorten the text with a max of 256 characters and look at the response and check it with a character count tool. ChatGPT, Copilot, Gemini, and Claude all do the same thing. 🤔🤷♂