Artificial intelligence (AI) has shown promise in various fields, but a recent study suggests that AI in Emergency Room setting may not be ready as yet. The study, conducted by researchers at Washington State University’s Elson S. Floyd College of Medicine, examined the performance of OpenAI’s ChatGPT program in assessing patients with chest pain.
Inconsistent Conclusions:
The researchers found that ChatGPT provided inconsistent conclusions when presented with simulated cases of patients with chest pain. Despite being fed the exact same patient data, the AI returned different heart risk assessment levels, ranging from low to high risk. Lead researcher Dr. Thomas Heston noted that such inconsistency is concerning, especially in medical emergencies where treatment decisions rely on accurate and consistent information.
Comparison with Traditional Methods:
Furthermore, the AI failed to perform as well as traditional methods used by doctors to assess a patient’s heart risk. Commonly used checklists like TIMI and HEART, which consider symptoms, health history, and age, outperformed ChatGPT in providing reliable assessments.
Complexity of Emergency Assessments:
Assessing patients with chest pain in the ER is crucial, as serious cases require immediate attention while lower-risk cases may necessitate observation or outpatient care. While doctors rely on established checklists for assessment, AI programs like ChatGPT have the potential to analyze complex medical situations quickly and thoroughly due to their ability to evaluate billions of variables.
Study Methodology:
The researchers fed ChatGPT thousands of simulated cases, varying in complexity and data sets. Despite its ability to pass medical exams in earlier research, the AI struggled to provide consistent assessments when presented with randomized health readings.
Randomness and Variation:
The study attributed ChatGPT’s inconsistent performance to the randomness built into its software, designed to simulate natural language responses. However, such randomness is not conducive to healthcare settings where treatment decisions require a single, consistent answer.
“ChatGPT was not acting in a consistent manner. Given the exact same data, ChatGPT would give a score of low risk, then next time an intermediate risk, and occasionally, it would go as far as giving a high risk.”
Dr. Thomas Heston, Associate Professor, Elson S. Floyd College of Medicine.
Future Potential of AI in the ER:
While the study highlights the limitations of AI in emergency settings, Dr. Heston believes that AI still holds promise in the ER. For instance, AI could assist by quickly providing pertinent facts from a patient’s medical record or offering multiple diagnoses in complex cases. However, further research is needed to ensure AI’s reliability and consistency in high-stakes clinical situations.
Conclusion:
The study underscores the importance of cautious integration and thorough research when implementing AI in healthcare, especially in emergency settings. While AI shows potential in aiding clinical decision-making, ensuring its accuracy and reliability is paramount to safeguarding patient care and outcomes in critical situations.
