Cross-Sectional Evaluation of Medical Disinformation Safeguards in Consumer-Facing Large Language Model Platforms

Natansh D. Modi; Cyril A. Alex; Abdulhalim A. Awaty; Bradley D. Menz; Stephen Bacchi; Kacper T. Gradon; Jessica M. Logan; Andrew Rowland; Lisa Kalisch Ellett; Ross A. McKinnon; Michael D. Wiese; Michael J. Sorich; Ashley M. Hopkins

doi:10.2196/89831

Cross-Sectional Evaluation of Medical Disinformation Safeguards in Consumer-Facing Large Language Model Platforms

Natansh D. Modi
, Cyril A. Alex
, Abdulhalim A. Awaty
, Bradley D. Menz
, Stephen Bacchi
, Kacper T. Gradon
, Jessica M. Logan
, Andrew Rowland
, Lisa Kalisch Ellett
, Ross A. McKinnon
, Michael D. Wiese
, Michael J. Sorich
, Ashley M. Hopkins

Research output: Contribution to journal › Article › peer-review

Abstract

This cross-sectional evaluation of six consumer-facing large language model platforms found significant heterogeneity in safeguard performance against the generation of health disinformation, with Claude and ChatGPT demonstrating complete resistance across all prompt types, while Copilot, Meta AI, Grok, and Gemini exhibited substantial vulnerabilities.

Original language	English
Article number	e89831
Number of pages	5
Journal	JMIR Infodemiology
Volume	6
DOIs	https://doi.org/10.2196/89831
Publication status	Published - 20 Apr 2026

Keywords

disinformation
epidemiology
large language models
misinformation
public health

Access to Document

10.2196/89831Licence: CC BY

Final published versionFinal published version, 312 KBLicence: CC BY

Cite this

Modi, N. D., Alex, C. A., Awaty, A. A., Menz, B. D., Bacchi, S., Gradon, K. T., Logan, J. M., Rowland, A., Ellett, L. K., McKinnon, R. A., Wiese, M. D., Sorich, M. J., & Hopkins, A. M. (2026). Cross-Sectional Evaluation of Medical Disinformation Safeguards in Consumer-Facing Large Language Model Platforms. JMIR Infodemiology, 6, Article e89831. https://doi.org/10.2196/89831

@article{e140abeb8e054d20b6d866bd53e57b57,

title = "Cross-Sectional Evaluation of Medical Disinformation Safeguards in Consumer-Facing Large Language Model Platforms",

abstract = "This cross-sectional evaluation of six consumer-facing large language model platforms found significant heterogeneity in safeguard performance against the generation of health disinformation, with Claude and ChatGPT demonstrating complete resistance across all prompt types, while Copilot, Meta AI, Grok, and Gemini exhibited substantial vulnerabilities.",

keywords = "disinformation, epidemiology, large language models, misinformation, public health",

author = "Modi, \{Natansh D.\} and Alex, \{Cyril A.\} and Awaty, \{Abdulhalim A.\} and Menz, \{Bradley D.\} and Stephen Bacchi and Gradon, \{Kacper T.\} and Logan, \{Jessica M.\} and Andrew Rowland and Ellett, \{Lisa Kalisch\} and McKinnon, \{Ross A.\} and Wiese, \{Michael D.\} and Sorich, \{Michael J.\} and Hopkins, \{Ashley M.\}",

year = "2026",

month = apr,

day = "20",

doi = "10.2196/89831",

language = "English",

volume = "6",

journal = "JMIR Infodemiology",

issn = "2564-1891",

publisher = "JMIR Publications Inc.",

}

TY - JOUR

T1 - Cross-Sectional Evaluation of Medical Disinformation Safeguards in Consumer-Facing Large Language Model Platforms

AU - Modi, Natansh D.

AU - Alex, Cyril A.

AU - Awaty, Abdulhalim A.

AU - Menz, Bradley D.

AU - Bacchi, Stephen

AU - Gradon, Kacper T.

AU - Logan, Jessica M.

AU - Rowland, Andrew

AU - Ellett, Lisa Kalisch

AU - McKinnon, Ross A.

AU - Wiese, Michael D.

AU - Sorich, Michael J.

AU - Hopkins, Ashley M.

PY - 2026/4/20

Y1 - 2026/4/20

N2 - This cross-sectional evaluation of six consumer-facing large language model platforms found significant heterogeneity in safeguard performance against the generation of health disinformation, with Claude and ChatGPT demonstrating complete resistance across all prompt types, while Copilot, Meta AI, Grok, and Gemini exhibited substantial vulnerabilities.

AB - This cross-sectional evaluation of six consumer-facing large language model platforms found significant heterogeneity in safeguard performance against the generation of health disinformation, with Claude and ChatGPT demonstrating complete resistance across all prompt types, while Copilot, Meta AI, Grok, and Gemini exhibited substantial vulnerabilities.

KW - disinformation

KW - epidemiology

KW - large language models

KW - misinformation

KW - public health

UR - https://www.scopus.com/pages/publications/105038974336

UR - http://purl.org/au-research/grants/NHMRC/APP2008119

UR - http://purl.org/au-research/grants/NHMRC/APP2030913

U2 - 10.2196/89831

DO - 10.2196/89831

M3 - Article

AN - SCOPUS:105038974336

SN - 2564-1891

VL - 6

JO - JMIR Infodemiology

JF - JMIR Infodemiology

M1 - e89831

ER -

Cross-Sectional Evaluation of Medical Disinformation Safeguards in Consumer-Facing Large Language Model Platforms

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this