Vaccination hesitancy: agreement between WHO and ChatGPT-4.0 or Gemini Advanced
Keywords:
ChatGPT, Gemini, AI, WHO, Vaccine, Vaccine HesitancyAbstract
Background. An increasing number of individuals use online Artificial Intelligence (AI) - based chatbots to retrieve information on health-related topics. This study aims to evaluate the accuracy in answering vaccine-related answers of the currently most commonly used, advanced chatbots - ChatGPT-4.0 and Google Gemini Advanced.
Methods. We compared the answers provided by the World Health Organization (WHO) to 38 open questions on vaccination myths and misconception, with the answers created by ChatGPT-4.0 and Gemini Advanced. Responses were considered as “appropriate”, if the information provided was coherent and not in contrast to current WHO recommendations or to drug regulatory indications.
Results and Conclusions. The rate of agreement between WHO answers and Chat-GPT-4.0 or Gemini Advanced was very high, as both provided 36 (94.7%) appropriate responses. The few discrepancies between WHO and AI-chatbots answers could not be considered “harmful”, and both chatbots often invited the user to check reliable sources, such as CDC or the WHO websites, or to contact a local healthcare professional. In their current versions, both AI-chatbots may already be powerful instrument to support the traditional communication tools in primary prevention, with the potential to improve health literacy, medication adherence, and vaccine hesitancy and concerns. Given the rapid evolution of AI-based systems, further studies are strongly needed to monitor their accuracy and reliability over time.
References
1. McMullan M. Patients using the internet to obtain health information: How this affects the patient–health professional relationship. Patient Educ Couns. 2006 Oct;63(1-2):24-8. doi: 10.1016/j.pec.2005.10.006. Epub 2006 Jan 6. PMID:
16406474.
2. Wimble M. Understanding health and health-related be- havior of users of internet health information. Telemed J E Health. 2016 Oct;22(10):809-815. doi: 10.1089/ tmj.2015.0267. Epub 2016 Apr 5. PMID: 27045569.
3. Blatchford A. Searching for online news content: The chal- lenges and decisions. Commun Res Pract. 2020;6(2):143-
156. https://doi.org/10.1080/22041451.2019.1676864.
4. Diviani N, van den Putte B, Giani S, van Weert JCM. Low health literacy and evaluation of online health information: 395 A systematic review of the literature. J Med Internet Res. 2015 May 7;17(5):e112. doi: 10.2196/jmir.4018. PMID: 25953147; PMCID: PMC4468598.
5. Välimäki M, Nenonen H, Koivunen M, Suhonen R. Pa- tients’ perceptions of internet usage and their opportunity to obtain health information. Med Inform Internet Med. 2007 Dec;32(4):305-14. doi: 10.1080/14639230701819792. PMID: 18072007.
6. Dwivedi YK, Hughes L, Ismagilova, E, Aarts G, Coombs C, Crick T, et al. Artificial intelligence (ai): Multidiscipli- nary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. Int J Inform Manag. 2021 Apr;57:101994. https://doi.org/10.1016/j. ijinfomgt.2019.08.002.
7. Mahowald K, Ivanova AA, Blank IA, Kanwisher N, Tenen- baum JB, Fedorenko E. Dissociating language and thought in large language models: A cognitive perspective. arXiv. 2023 Jan 18: abs/2301.06627. https://doi.org/10.48550/ arXiv.2301.06627.
8. Ayoub NF, Lee YJ, Grimm D, Divi V. Head-to-head comparison of chatgpt versus google search for medical knowledge acquisition. Otolaryngol Head Neck Surg. 2024 Jun;170(6):1484-1491. doi: 10.1002/ohn.465. Epub 2023 Aug 2. PMID: 37529853.
9. Boucher EM, Harake NR, Ward HE, Stoeckl SE, Var- gas J, Minkel J, et al. Artificially intelligent chatbots in digital mental health interventions: A review. Expert Rev Med Devices. 2021 Dec;18(suppl 1):37-49. doi: 10.1080/17434440.2021.2013200. Epub 2021 Dec 31. PMID: 34872429.
10. Xu L, Sanders L, Li K, Chow JCL. Chatbot for health care and oncology applications using artificial intelligence and machine learning: Systematic review. JMIR Cancer. 2021 Nov 29;7(4):e27850. doi: 10.2196/27850. PMID: 34847056; PMCID: PMC8669585.
11. Monteith S, Glenn T, Geddes JR, Whybrow PC, Achtyes E, Bauer M. Artificial intelligence and increasing misinfor-ma- tion. Br J Psychiatry. 2024 Feb;224(2):33-35. doi: 10.1192/ bjp.2023.136. PMID: 37881016.
12. Torun C, Sarmis A, Oguz A. Is chatgpt an accurate and reliable source of information for patients with vaccine and statin hesitancy? Medeni Med J. 2024 Mar 21;39(1):1-7. doi: 10.4274/MMJ.galenos.2024.03154. PMID: 38511678; PMCID: PMC10961658.
13. Deiana G, Dettori M, Arghittu A, Azara A, Gabutti G, Castiglia P. Artificial intelligence and public health: Evalu- ating chatgpt responses to vaccination myths and miscon- ceptions. Vaccines (Basel). 2023 Jul 7;11(7):1217. doi: 10.3390/vaccines11071217. PMID: 37515033; PMCID: PMC10386180.
14. Barlas T, Altinova AE, Akturk M, Toruner FB. Credibility of chatgpt in the assessment of obesity in type 2 diabetes ac-cording to the guidelines. Int J Obes (Lond). 2024 Feb;48(2):271-275. doi: 10.1038/s41366-023-01410-5. Epub 2023 Nov 11. PMID: 37951982.
15. Whiles BB, Bird VG, Canales BK, DiBianco JM, Terry RS. Caution! AI bot has entered the patient chat: Chat- 396 M. Fiore et al. GPT has limitations in providing accurate urologic healthcare advice. Urology. 2023 Oct;180:278-284. doi: 10.1016/j.urology.2023.07.010. Epub 2023 Jul 17. PMID:
37467806.
16. World Health Organization (WHO). Vaccines and immunization: What is vaccination? World Health Organization. 23 April 2024. Available from: https://www.who.int/news-room/questions-and-answers/item/vaccines-and-immunization-what-is-vaccination [Last accessed: 2024 Jul 2].
17. World Health Organization (WHO). Vaccines and immunization: Myths and misconceptions. 19 October 2020. Available from: https://www.who.int/news-room/questions-and-answers/item/vaccines-and-immunization-myths-and-misconceptions [Last accessed: 2024 Jul 2].
18. World Health Organization (WHO). Coronavirus disease (COVID-19): Vaccines. 5 December 2023. Available from: https://www.who.int/news-room/questions-and-answers/item/coronavirus-disease-(covid-19)-vaccines [Last accessed: 2024 Jul 2].
19. OpenAI. Introducing ChatGPT. OpenAI 2022. Available from: https://openai.com/blog/chatgpt [Last accessed: 2024 Jul 6].
20. Bard becomes Gemini: Try Ultra 1.0 and a new mobile app today. Google. Available from: https://blog.google/products/gemini/bard-gemini-advanced-app [Last accessed: 2024 Jul 6].
21. U.S. Food & Drug. Available from: https://www.fda.gov/ [Last accessed: 2024 Jul 6].
22. Marinho, A.K.B.B. Vaccination in children with immune-mediated disorders. J Pediatr (Rio J). 2023 Mar-Apr;99 Suppl 1(Suppl 1):S62-S69. doi: 10.1016/j.jped.2022.11.008. Epub 2022 Dec 21. PMID: 36566017; PMCID: PMC10066448.
23. Which countries include hepatitis B birth dose vaccines in their vaccination schedules? Our World in Data. Available from: https://ourworldindata.org/grapher/hepatitis-b-birth-dose-vaccine-immunization-schedule [Last accessed: 2024 Jul 6].
24. US Food & Drug Administration (FDA). Moderna COVID-19 Vaccine. 2023. Available from: https://www.fda.gov/vaccines-blood-biologics/coronavirus-covid-19-cber-regulated-biologics/moderna-covid-19-vaccine#:~:text=%2DCoV%2D2. [Last accessed: 2024 Jul 6].
25. Johnson SB, King AJ, Warner EL, Aneja S, Kann BH, Bylund CL. Using chatgpt to evaluate cancer myths and misconceptions: Artificial intelligence and cancer information. JNCI Cancer Spectr. 2023 Mar 1;7(2):pkad015. doi: 10.1093/jncics/pkad015. PMID: 36929393; PMCID: PMC10020140.
26. Huh S. Are chatgpt’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: A descriptive study. J Educ Eval Health Prof. 2023;20:1. doi: 10.3352/jeehp.2023.20.1. Epub 2023 Jan 11. PMID: 36627845; PMCID: PMC9905868.
27. Wang J, Wang Y, Li Y, Ma M, Xie Y, Zhang Y, et al. Bibliometric and visual analyses of vaccine literacy research from 1982 to 2023. Hum Vaccin Immunother. 2024 Jun 19; 20(1):2363019. doi: 10.1080/21645515.2024.2363019. PMCID: PMC11188838.
28. Aggarwal A, Tam CC, Wu D, Li X, Qiao S. Artificial intelligence–based chatbots for promoting health behavioral changes: Systematic review. J Med Internet Res. 2023 Feb 24;25:e40789. doi: 10.2196/40789. PMID: 36826990; PMCID: PMC10007007.
29. Brian W, Aline CG, Stefan G, Nina RS. Artificial intelligence (AI) and global health: How can ai contribute to health in resource-poor settings? BMJ Glob Health. 2018 Aug 29;3(4):e000798. doi: 10.1136/bmjgh-2018-000798. PMID: 30233828; PMCID: PMC6135465.
30. Roumeliotis KI, Tselikas ND. Chatgpt and open-AI models: A preliminary review. Future Internet 2023 May;15(6):192. doi: 10.3390/fi15060192.
31. Lechien JR, Maniaci A, Gengler I, Hans S, Chiesa-Estomba CM, Vaira LA. Validity and reliability of an instrument evaluating the performance of intelligent chatbot: The Artificial Intelligence Performance Instrument (AIPI). Eur Arch Otorhinolaryngol. 2024 Apr;281(4):2063-2079. doi: 10.1007/s00405-023-08219-y. Epub 2023 Sep 12. PMID: 37698703.
32. Norori N, Hu Q, Aellen FM, Faraci FD, Tzovara A. Addressing bias in big data and AI for health care: A call for open science. Patterns (N Y). 2021 Oct 8;2(10):100347. doi: 10.1016/j.patter.2021.100347. PMID: 34693373; PMCID: PMC8515002.
33. Parikh RB, Teeple S, Navathe AS. Addressing bias in artificial intelligence in health care. JAMA. 2019 Dec 24;322(24):2377-2378. doi: 10.1001/jama.2019.18058. PMID: 31755905.
34. Yao R, Zhang W, Evans R, Cao G, Rui T, Shen L. Inequities in health care services caused by the adoption of digital health technologies: Scoping review. J Med Internet Res. 2022 Mar 21;24(3):e34144. doi: 10.2196/34144. PMID: 35311682; PMCID: PMC8981004.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Matteo Fiore, Alessandro Bianconi, Cecilia Acuti Martellucci, Annalisa Rosso, Enrico Zauli, Maria Elena Flacco, Lamberto Manzoli (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Transfer of Copyright and Permission to Reproduce Parts of Published Papers.
Authors retain the copyright for their published work. No formal permission will be required to reproduce parts (tables or illustrations) of published papers, provided the source is quoted appropriately and reproduction has no commercial intent. Reproductions with commercial intent will require written permission and payment of royalties.