ChatGPT Health Fails to Recognise Medical Emergencies

Written by Amelia Palazzo – Law Clerk

In recent years, the emergence of artificial intelligence (AI) and chatbot technology, including ChatGPT, has transformed numerous aspects of daily life, particularly the medical and legal landscape.

Recently, ChatGPT has been scrutinised for its failure to provide individuals with advice to seek urgent medical care, which has led to an increase in unnecessary harm, and in some cases, death. This raises potential challenges and ethical concerns surrounding the use of ChatGPT in medical emergencies.

Background

In January 2026, OpenAI launched ChatGPT Health, which has been designed to help users take a more active role in understanding and managing their health and wellness. This new feature is targeted towards providing health and medical guidance directly to the public and aims to function as a “first point of contact” for symptom guidance.

The first independent safety evaluation of ChatGPT Health has found that more than 40 million people reportedly use ChatGPT for health-related advice every day, of which more than half of those cases are under-triaged, meaning that users fail to receive appropriate medical advice and do not receive the necessary urgent care that they require.

Lead author of the study and evaluation, Dr Ashwin Ramaswamy, and his colleagues, created 60 realistic patient scenarios, covering a range of health and medical conditions, including mild illnesses and emergencies. Three independent doctors were then required to review each scenario and agree on the level of care required in the circumstances.

Dr Ramaswamy and his colleagues then asked ChatGPT Health for advice on each scenario under different circumstances / conditions. The results and recommendations generated from ChatGPT Health, were then compared with those of the independent doctors.

Results

By way of summary, the study revealed the following results:

• Although ChatGPT Health performed well in a “textbook situation” such as a stroke or allergic reaction, it failed to perform in other situations. For instance, the platform was presented with an asthma scenario in which it advised the hypothetical patient to wait rather than seeking medical intervention or treatment.
• In 51.6% of cases where someone was required to go to the hospital immediately, ChatGPT Health advised the user to stay home or book a medical appointment with their GP.
• When the platform was presented with someone experiencing respiratory failure or diabetic ketoacidosis, there was a 50/50 chance of ChatGPT advising that no action or medical treatment was required.
• 64.8% of individuals who were safe and exhibited no severe symptoms, were advised to seek immediate medical treatment.
• ChatGPT Health was nearly 12 times more likely to downplay symptoms when the “patient” advised it that a “friend” had already suggested it was nothing serious or worth acting upon.
• The platform was presented with a scenario, involving a 27-year-old male patient who advised that he intended to commit self-harm. When the patient described his symptoms to the platform, the crisis intervention banner linking to mental health and suicide support services appeared. However, when the researchers added normal lab results, including the same words and severity, the banner disappeared.

This has raised significant concerns regarding the lack of consistency and crisis protections in place, during medical emergencies. In some circumstances, it has also raised the prospects of legal liability, due to an increase in legal action pursued against technology and AI companies in relation to suicide and self-harm associated with advice provided by AI chatbot technology.

Key Takeaways

• As ChatGPT Health continues to develop, it is important to bear in mind that the quality and nature of advice provided is not always correct or appropriate in certain circumstances. It is important that you always verify your sources and consult your medical practitioner or emergency services, if any medical concerns arise.
• It is important that users of ChatGPT Health exercise extreme caution when relying upon information provided by the platform. ChatGPT Health should be used as a preliminary source of information and / or education, and not to replace professional and tailored medical advice.

Our team is constantly keeping a close watch on these developments. If you have sustained injury as a result of using AI chatbot technology, or you require assistance with a personal injury matter, we are here to help.

The contents of this publication are for reference purposes only. This publication does not constitute legal and / or medical advice and should not be relied upon as legal and / or medical advice. Specific legal and / or medical advice should always be sought separately before taking any action based on this publication.