Dr. Google’s Advice on First Aid: Evaluation of the Search Engine’s Question-Answering System Responses to Queries Seeking Help in Health Emergencies

Abstract

Introduction:

The ever-growing penetration of internet and mobile technologies into society suggests that people will increasingly use web searches to seek health-related information, including advice on first aid in medical emergencies. When a bystander is incompetent in first aid and has no immediate support from Emergency Medical Services (EMS), as it happens in low-resource settings or in disasters, instructions found online could be the sole driver for administering first aid before arrival of professional help.

Study Objective:

The aim of this study was to evaluate quality of advice on first aid generated by a web search engine’s question-answering system (QAS) in response to search queries concerning provision of help in common health emergencies.

Methods:

In December 2022-January 2023, an English-language search was carried out in Google with ten queries based on the keyword combinations (what to do OR how to help) AND (bleeding OR chest pain OR choking OR not breathing OR seizure). The search engine’s QAS responses (up to 11 per search query) were evaluated for compliance with the International Federation of Red Cross First Aid Guidelines 2020 using the pre-developed checklists.

Results:

Out of 98 QAS items generated by Google, 67.3% (n = 66) were excluded, mainly because the QAS answers did not address original queries. Eligible unique QAS responses (n = 27) showed poor coverage of the guideline-compliant instructions on first aid. Mean percentage of QAS responses providing a first aid instruction with complete adherence to the guidelines varied from 0.0 for choking to 19.5 for seizure. Only three (11.1%) QAS responses contained an explicit instruction to access EMS, while 66.7% (n = 18) included directions either contradictory to the guidelines and potentially harmful (eg, use of home remedies in chest pain) or inapplicable for an untrained person (eg, use of tourniquet in bleeding).

Conclusion:

Although the search engine’s QAS responds to user’s inquiries concerning assistance in health emergencies, the QAS-generated answers, as a rule, omit potentially life-saving evidence-based instructions on first aid and oftentimes give advices noncompliant with current guidelines or inadequate for untrained people, and thus create risks for causing harm to a victim.

Introduction

Over the last decade, the quantity of internet users world-wide almost doubled. ¹ In some regions like Northern Europe, the internet penetration rate is currently as high as 98%. With the increasing availability and accessibility of digital technologies, the internet has become a powerful and influential tool for the general public to retrieve health information and inform health-related decisions. ² The number of people looking for health information online steadily rises, and in some countries (eg, Finland, Netherlands, or Norway), eight out of ten adult individuals are searching the web for health-related topics. ³

Lack of confidence with first aid and related fear of making mistakes are well-known as predominant factors impeding willingness of laypeople to intervene in a health emergency. ^4,5 To overcome these barriers in real-life critical situations, people may turn online to look for advice on how to give help.

Most internet users start their search for health information through a search engine. ² Question-answering systems (QAS) embedded in major search engines help searchers to get a quick snapshot of information on a topic without scrolling and having to read a large number of search results. Special machine-learning algorithms determine search patterns from enormous volumes of queries, and resulting patterns are utilized to pick the best excerpt from existing web content and provide prompt and simple answers to fulfill users’ information needs. ⁶ An eye-catching question answering block is shown on top of search results page above all other results. Hence, it is possible that a user will presume information provided therein as first-priority and limit evaluation of the search results to the question answering block contents. ⁷ This could be especially true in time-restricted settings, including health emergencies.

Although major search engines use algorithms to prioritize high-quality sources in their search results, a number of studies have shown that health-related information found online, including information on first aid, is commonly non-evidence-based and low in quality. ^8–10 Incorrect, incomplete, or irrelevant information on first aid may produce inappropriate and harmful actions, potentially affecting health outcomes.

There is a paucity of research exploring the ability of search engines’ QAS to respond to users’ inquiries related to first aid. ^11,12 Better understanding of this issue could help to create a framework for improving access of laypeople to reliable informational support in emergency circumstances. The aim of this study was to analyze search engine’s QAS responses to questions concerning first aid in common adult health emergencies, in particular to determine the quality of provided first aid instructions and to evaluate web sources of the information.

Methods

Description of the QAS

Featured Snippet (FS) and People Also Ask (PAA) sections are typical components of the Google Search engine’s (Google LLC; Mountain View, California USA) search results page. The FS section is a featured answer to a query that is automatically extracted from a Google-indexed webpage and is shown at the top of the search results page. These FS are usually displayed in the form of short text paragraph, list, or table, followed with a link to a source webpage, and occasionally may contain images or present as video snippets. ¹³ The PAA section is shown below a FS, under the heading “People Also Ask.” It contains several (usually up to four) questions related to a searcher’s initial query. When expanded, the questions reveal short answers presented in the format similar to a FS and are supplemented with a source webpage link. Clicking on the questions generates new related questions that drop down below. Both FS and PAA sections are usually outlined above all other search results and are presented in a larger font. The exact way how the search engine generates FS and PAA items is undisclosed. Whereas Google describes PAA section as simple as “questions people commonly search on Google,” presumably Google’s artificial intelligence algorithm chooses FS and PAA content from external websites based on how closely the information corresponds to users’ questions, and considering Google’s own measure of source authority and its ranking in the search results. ¹⁴

Search and Analysis Methodology

An English-language Google search was conducted from December 28, 2022 through January 9, 2023 with the following keyword combinations: (what to do OR how to help) AND (bleeding OR chest pain OR choking OR not breathing OR seizure). In total, ten queries were generated. The search was carried out on a personal computer with Windows 10 operating system (Microsoft Corporation; Redmond, Washington USA) using Google Chrome (Google LLC; Mountain View, California USA) web browser in Incognito mode. In this mode, the browser does not track user’s data (ie, browsing history or cookies), and thereby an influence of prior user’s web activity on search results is avoided. Search region was set as the United Kingdom.

For each query, a list of PAA questions was expanded until ten sequential questions were generated (it was considered unlikely that a user will review more in case of emergency). Full content of a FS and related PAA items was collected, including question and answer text (with images where available) and a link to source webpage. Consequently, the maximum number of collected QAS items per query was 11 (one FS and ten PAA items). Items that did not address the original query straightforward and questions concerning first aid exclusively for children were omitted from further evaluation as non-eligible.

After removing duplicates, all unique eligible QAS responses providing instructions on first aid for adults were evaluated for quality (in terms of completeness and correctness) using checklists (found in the complete Dataset ¹⁵ ) pre-developed by the authors based on the International Federation of Red Cross and Red Crescent Societies’ (IFRC; Geneva, Switzerland) International First Aid, Resuscitation, and Education Guidelines 2020. ¹⁶ Compliance of instructions provided in a QAS response with the guidelines was rated for each item of the respective checklist as complete (where checklist item wording was satisfied completely), incomplete (where checklist item wording was satisfied in part), or absent (where QAS response omitted corresponding first aid instruction). When QAS responses were presented as video snippets, content of the videos was also considered for the quality evaluation. However, videos longer than one minute were excluded (it was considered unlikely that a user will watch them in case of emergency).

Additionally, source webpages of eligible QAS responses were categorized as Academic, Charity/Non-Profit Entity, Commercial, Government, or Media Outlet, and content of the source webpages (including text, images, and videos ≤one minute) was evaluated for quality of first aid instructions using the same checklists, with a focus on source article text (ie, additional articles on the same webpage, other webpage sections [eg, frequently asked questions], links to external webpages, and resources were omitted from the evaluation).

All evaluations were carried out by the two authors independently. Obtained results were compared, and any disagreements were resolved by consensus. Data were analyzed using descriptive statistics, and Wilcoxon signed-rank test was utilized to determine statistical differences.

Results

The search conducted with ten queries following different keyword combinations produced a total of 98 QAS items (Figure 1). Of these, 67.3% (n = 66) were excluded, mainly because the QAS-generated questions and/or answers did not address original queries. Consequently, 32 QAS items were considered eligible, of which 27 were unique. The number of eligible QAS items per search query varied from zero (how to help not breathing) to six (what to do [how to help] bleeding; Figure 1). Only one (3.1%) of the eligible QAS answers contained a figure showing first aid maneuvers (back blows and abdominal thrusts for choking), whereas all other responses were presented exclusively as text.

Figure 1. Flow of Selection of QAS Items for the Analysis.

Abbreviation: QAS, question-answering system.

Evaluation of the quality of eligible unique QAS responses in terms of the guideline’s adherence showed poor coverage of the guidelines-compliant instructions on first aid (Table 1 and Dataset ¹⁵ ). In particular: for choking, only one of five unique QAS responses instructed to encourage the person to cough to clear the blockage, and no QAS answers recommended to start cardiopulmonary resuscitation (CPR) if the person who choked became unresponsive; for chest pain, no QAS responses suggested to encourage the person to refrain from physical activity or to help the person to take their antianginal medication; for not breathing, most QAS answers did not provide clear instructions on how to give chest compressions and neither of them indicated the need to use an automated external defibrillator, if available; for bleeding, instructions on how to correctly stop bleeding while taking precautions to reduce the risk of cross-infection were commonly lacking or incomplete; for seizure, all QAS responses omitted some important directions for protection of the person from harm during the convulsions, and no QAS answers suggested to check the person’s breathing once the seizure has ended and to commence CPR if required. Out of the 27 unique QAS responses, only three (11.1%) contained an explicit instruction to access Emergency Medical Services (EMS) in a critical situation, all suggesting to call the United States’ national emergency telephone number 9-1-1. Alongside this, when a QAS answer described the guidelines-recommended action on first aid, the instruction was commonly judged as partially compliant with the guidelines because of omission of relevant details on the first aid technique. Mean percentage of QAS responses providing a first aid instruction with complete adherence to the guidelines varied from 0.0 for choking to 19.5 for seizure, and mean percentage of QAS responses providing a first aid instruction with at least partial adherence to the guidelines varied from 10.0 for chest pain to 25.0 for not breathing (Table 1).

Table 1. Distribution of QAS Answers and Source Articles in Terms of Availability of the Guideline-Compliant Instructions on First Aid

Abbreviations: AED, automated external defibrillator; CPR, cardiopulmonary resuscitation; EMS, Emergency Medical Services; QAS, question-answering system.

^a Significant difference (P <.05) between QAS answers and source articles.

Of the unique QAS responses, 66.7% (n = 18) contained instructions to perform redundant actions which either contradicted the guidelines and could cause additional harm (eg, a suggestion to use home remedies like cold pack, baking soda, or hot drinks for chest pain, or to roll the person onto their side during seizures) or could be recommended for trained first aid providers but not for an untrained bystander (eg, an instruction to perform rescue breaths for a suspected cardiac arrest [not breathing] victim or to apply a tourniquet for bleeding; Dataset ¹⁵ ).

Commercial websites were most commonly selected by the search engine algorithm as a source for generating the QAS answers, followed by websites of government and academic entities. Distribution of the unique QAS responses according to the source website category is shown in Table 2.

Table 2. Web Sources of Content for Search Engine-Generated Unique QAS Responses (n = 27)

Abbreviation: QAS, question-answering system.

Evaluation of the content of source articles showed better coverage of the guidelines-compliant instructions in comparison with respective QAS answers. Mean percentage of source articles providing a first aid instruction with complete adherence to the guidelines varied from 16.7 for chest pain to 47.6 for bleeding; mean percentage of source articles providing a first aid instruction with at least partial adherence to the guidelines varied from 26.7 for chest pain to 60.0 for choking (Table 1 and Dataset ¹⁵ ). In spite of the higher quality of content compared with QAS responses, some source articles omitted potentially life-saving instructions on first aid, contained instructions to perform actions applicable for trained first aid providers only, while not being clearly indicated as such, or included potentially harmful recommendations contradictory with current guidelines on first aid.

An instruction to call a particular telephone number to activate EMS was provided in 81.5% (n = 22) of the source articles. Of these, 63.6% (n = 14) suggested the United States’ national emergency number 9-1-1, 31.8% (n = 7) instructed to call the United Kingdom’s national emergency number 9-9-9, and 4.5% (n = 1) recommended to call 1-1-1 — the national emergency number for New Zealand.

Discussion

The internet forms a backbone of modern information society. As of January 2023, there were 5.2 billion internet users world-wide, which is 64.4% of the global population, ¹⁷ and the number of smartphone users surpassed 6.5 billion people (81.3% of the world’s population). ¹⁸ The ever-growing availability and penetration of the internet and connected mobile devices suggest that people will increasingly use web search to seek health-related information, including advice on first aid in medical emergencies. This seems particularly likely when a person facing an emergency is incompetent in first aid and has no immediate support from trained bystanders or EMS, for example in resource-limited settings where EMS arrival could be substantially delayed and dispatch pre-arrival instructions are unavailable, or in mass-casualty incidents and disasters. In such cases, provision of first aid following instructions found online could represent the only way to mitigate risks for victim’s health or life before professional help will come. In order to fit the purpose of urgent informational support, the first aid advice should be easy to find, openly available and accessible, clearly understandable, concise, and evidence-based.

This study focused on evaluation of ability of the Google search engine’s QAS to answer queries concerning delivery of first aid. Google was selected as the world’s most popular search engine holding approximately 92.9% of the total search engine market share world-wide. ¹⁹ The study results show that the QAS responds to user’s enquiries concerning assistance in common health emergencies by providing short textual instructions on first aid within the FS and PAA sections. Being shown above all other search results and in larger font, the QAS responses instantly catch the eye. When using a search by voice option, Google sometimes reads instructions from a FS aloud. It seems likely that in an emergency situation, a person will instinctively accept and follow these prominent instructions, rather than go beyond them to find an answer by exploring other results or provided links.

Although the Google search engine’s QAS represents a hypothetical means for providing instant instructions on first aid for untrained people in absence of alternative assistance, the QAS commonly generates irrelevant results, which very often do not contain any advice on first aid. When the QAS provides instructions on first aid, quality of the guidance is generally very poor. Although generated from reputable sources, the QAS responses as a rule omit relevant life-saving instructions (including an instruction to call for EMS) and frequently give redundant directions contradictory with current first aid guidelines or inappropriate for an untrained person (like application of tourniquet in bleeding or administration of rescue breathing in cardiac arrest). Alongside this, when the QAS gives a guidelines-consistent instruction, it oftentimes does not provide sufficient details on how to perform the first aid maneuver. Visual aids (figures or short videos), which could help to clearly understand the first aid technique, are almost always absent. The quality issues make it unlikely that the QAS advice would trigger correct and timely first aid intervention.

The analysis of web articles used by Google as a source for the QAS responses demonstrated considerably higher quality of the content in terms of coverage of the guidelines-compliant instructions on first aid. A probable explanation for the difference in quality between the QAS answers and the source articles is that Google’s algorithm generates a QAS response from a source webpage by automatically extracting one literal quotation most closely resembling the user’s inquiry, whereas the algorithm cannot analyze the whole webpage to generate a concise selection of relevant first aid instructions suitable for untrained people. As a consequence, the QAS responses are commonly missing principal first aid instructions while emphasizing nonessential ones.

Notwithstanding the higher quality of source articles compared with QAS responses, in many cases, the source content was also omitting relevant evidence-based instructions on first aid. This supports earlier studies which found that information on first aid presented online on websites, ^8,9 in mobile apps, ²⁰ videos, ²¹ or within online courses curricula ¹⁰ is largely of poor quality. It is also worthy of note that EMS telephone numbers suggested by the source articles and the QAS responses in most cases were inconsistent with the search region that represents another important limitation to the use of the search engine as a means of informational support in a real-life emergency.

Findings of the current study are generally in-line with previous research that tested voice-based virtual digital assistants to determine their ability to respond to user’s requests for first aid information. TW Bickmore, et al showed that Alexa (Amazon; Seattle, Washington USA), Google Assistant (Google LLC; Mountain View, California USA), and Siri (Apple Inc.; Cupertino, California USA) conversational assistants failed in responding to laypeople’s health-related inquiries, including questions concerning assistance in medical emergencies more than one-half of the time, and in approximately 30% cases, they recommended subjects to take potentially harmful or fatal actions. ¹¹ According to the research by C Picard, et al, ¹² Alexa and Google Home (Google LLC; Mountain View, California USA) demonstrated high rates of first-aid-related verbal query recognition (98% and 92%, respectively), but quality of responses was generally low (only 19% and 56% responses, respectively, were either congruent with first aid guidelines or deemed helpful by reviewers), whereas low query recognition rate by Cortana (Microsoft Corp.; Redmond, Washington USA) and Siri (19% and 23%, respectively) prohibited their analysis.

Overall, current evidence indicates that laypeople should not rely on artificial intelligence of search engines’ QAS when seeking advice on first aid. It is important for the general public to understand the limitations and potential risks of using an online search instead of activating EMS in real-life emergencies. Owners and developers of search engines should recognize the level of responsibility for satisfying and managing the public’s health information needs, including possible harm from providing inaccurate, incomplete, and non-evidence-based instructions on first aid. A reasonable approach to tackle the quality issue could be to stop using the automated generation of QAS responses on first aid based on different web sources and to seek for partnership with trusted research organizations like the International Liaison Committee on Resuscitation in order to develop a uniform framework and a knowledge base for online informational support of laypeople in a broad spectrum of health emergencies. The expert-developed knowledge base could serve as the sole evidence-based source of approved and reliable QAS responses, which should be adopted for untrained people, supplemented with clear audio-visual aids, and be periodically updated to guarantee alignment with up-to-date guidelines on first aid. Whereas content of the first aid instructions should be standardized to guarantee high quality of the informational support, artificial intelligence algorithms of search engines could be utilized to adopt the QAS responses to different consumers and settings, including translation into languages, suggestion of proper telephone number of EMS based on user’s geographic location, supplementation of a response with information on location of nearest automated external defibrillator, or an option to summon trained first responders where such technologies are available.

More research is required to explore first aid information-seeking behaviors and preferences among the general public, to better understand perspectives and implications of using search engines’ QAS as a means of urgent assistance in health crises, as well as to design and develop uniform and reliable mechanisms for quality control and quality assurance of the digital informational support.

Limitations

This study has limitations. Firstly, generalizability of the results is limited to the search conducted in English for the United Kingdom search region. Secondly, some searchers may utilize search engines other than Google, or may leave out QAS responses when reviewing search results. Further, there are multiple ways to formulate search queries, and queries may be formulated with intent other than obtaining information on how to help a person in real-life emergency. Finally, reproducibility of the results may be affected due to dynamic nature of the internet and modifications of the search engine algorithms.

Conclusions

The Google search engine’s QAS in its current performance is impracticable for informational support of laypeople on first aid in real-life health emergencies. The QAS-generated responses, as a rule, omit potentially life-saving evidence-based instructions on first aid and oftentimes give redundant directions noncompliant with current guidelines or inapplicable for an untrained person that create risks for causing harm to a victim.