In a pilot study that will be presented at the European Emergency Medicine Congress, ChatGPT, an artificial intelligence chatbot, was found to be as effective as a trained doctor in suggesting probable diagnoses for patients being evaluated in emergency medicine departments. This is a significant development that demonstrates the potential of AI-powered assistants to aid in healthcare.

Although the researchers say that more work is needed, their findings suggest that the technology could potentially support doctors working in emergency medicine, which could lead to shorter waiting times for patients. The study was conducted by Dr. Hidde ten Berg from the Department of Emergency Medicine and Dr. Steef Kurstjens from the Department of Clinical Chemistry and Hematology, both at Jeroen Bosch Hospital in 's-Hertogenbosch, the Netherlands.

Dr ten Berg told the Congress: “Like a lot of people, we have been trying out ChatGPT and we were intrigued to see how well it worked for examining some complex diagnostic cases. So, we set up a study to assess how well the chatbot worked compared to doctors with a collection of emergency medicine cases from daily practice.”

The Annals of Emergency Medicine has published a research study that provides anonymized data on 30 patients who received treatment at the emergency department of Jeroen Bosch Hospital in 2022. The research team entered the physicians' notes on patients' symptoms, signs, and physical examinations into two versions of ChatGPT (the free 3.5 version and the subscriber 4.0 version). Additionally, they provided the chatbot with the results of lab tests, including blood and urine analysis. For each case, the researchers compared the chatbot's shortlist of probable diagnoses with the shortlist made by emergency medicine doctors and the actual diagnosis of the patient.

During the study, it was discovered that ChatGPT's shortlists had a significant overlap of about 60% with those of the doctors. In 87% of the cases, doctors correctly diagnosed the disease within their top five likely diagnoses, compared to 97% and 87% of ChatGPT versions 3.5 and 4.0, respectively. Dr. Ten Berg acknowledged ChatGPT's ability to provide a list of possible diagnoses and suggest the most probable option. The overlap of the shortlisted diagnoses with those of the doctors indicates that ChatGPT can suggest medical diagnoses like a human doctor.

“For example, we included a case of a patient presenting with joint pain that was alleviated with painkillers, but redness, joint pain, and swelling always recurred. In the previous days, the patient had a fever and sore throat. A few times there was a discoloration of the fingertips. Based on physical exam and tests, doctors suspected rheumatic fever. However, ChatGPT correctly diagnosed vasculitis.

“It’s vital to remember that ChatGPT is not a medical device and there are privacy concerns when using ChatGPT with medical data. However, there is potential here for saving time and reducing waiting times in the emergency department. The benefit of using artificial intelligence could be in supporting doctors with less experience, or it could help in spotting rare diseases.”

Professor Youri Yordanov from the St. Antoine Hospital emergency department (APHP Paris), France, is Chair of the EUSEM 2023 abstract committee and was not involved in the research. He said: “We are a long way from using ChatGPT in the clinic, but it’s vital that we explore new technology and consider how it could be used to help doctors and their patients. People who need to go to the emergency department want to be seen as quickly as possible and to have their problems correctly diagnosed and treated. I look forward to more research in this area and hope that it might ultimately support the work of busy health professionals.”