A white rabbit runs by
ChatGPT, an artificial intelligence (AI) chatbot released in November
2022, has caused controversy within the world of
academia1. The number of articles on the subject
continues to increase almost exponentially. A general theme is: To what
extent can an AI platform write a reliable scientific text?
In our department – Clinical Pharmacology at Odense University Hospital
– we decided to approach the question in a practical manner by testing
ChatGPT in a routine task. In our Drug Information Centre (DIC) we
answer clinical drug-related questions from healthcare professionals.
The foundation of our answers is evidence-based
medicine2. We follow a standardized procedure to
ensure scientific quality and reproducibility: From a focused clinical
question, we perform a thorough literature search and write a
mini-review based on the available evidence with an accompanying
interpretation of the clinical relevance. The work is reviewed by the
department academic staff on a bi-weekly conference before the report is
sent to the inquiring health care professional.
We dedicate a significant portion of our work-day to writing and
reviewing evidence. The emergence of ChatGPT made us wonder if it could
serve as a valid review- or writing assistant – or even replace us.
Since our product (the mini-review) is purely text, we can directly
compare our answers to ChatGPTs text-based output.
When we asked ChatGPT to write a scientific review on a pharmacological
topic, our initial discovery was, that although the answer on
surface-level looks convincing, some of the provided references were
utter nonsense. In other words: when prompted to write a scientific
review, ChatGPT will provide a scientific-looking article including a
list of references with an obscure mix of relevant and real articles,
real but irrelevant articles, and articles that are complete bogus.
This finding is echoed in the rapidly exploding literature about
ChatGPTs capability to produce reliable scientific texts3. Likewise, it is openly stated by the creators of
ChatGPT that ChatGPT “sometimes writes plausible-sounding but
incorrect answers ” 4.
Intrigued by this, we decided follow the white rabbit…