We are pleased to present this information retrieval issue. Our goal is to highlight the variety of tasks and organizational structures of information retrieval within heath technology assessment (HTA). This special issue was planned and organized by the HTAi Information Retrieval Group (IRG). The choice of publications in this issue reflects the versatility and high competence in information retrieval methodology that is needed as an information specialist. Furthermore, it provides insights into the daily challenges faced by information specialists.

The history of European health technology assessment (HTA) goes back more than 30 years. Almost as old as HTA agencies themselves is the desire to achieve European collaboration. This gained further impetus with the establishment of the European Network of Health Technology Assessment (EUnetHTA) in 2006. In this context, the field of information management faced specific challenges. Although these services are an integral part of HTA and information specialists play a key role here, this field is often not adequately represented in the HTA agencies within EUnetHTA. Furthermore, the organization of HTA production, including the types of HTAs produced, as well as funding, varies considerably. In order to meet these different conditions, information specialists have created various products and defined processes. With the EUnetHTA guideline, a common methodological understanding for the production of rapid Relative Effectiveness Assessments now exists. Furthermore, the Standard Operating Procedures map the complex information retrieval processes within EUnetHTA in a hands-on manner. The newly established Information Specialist Network (ISN) will in future ensure that information specialists are involved in all EUnetHTA assessments and that the methods are applied consistently in all assessments. In addition, the steering committee of the ISN manages enquiries and can be contacted to discuss methodological issues. Major barriers such as heterogeneity in the daily work of the EUnetHTA members can only be overcome through more collaboration and training.

Objectives: This study was designed to assess the sensitivity of three Ovid MEDLINE search filters developed to identify studies reporting health state utility values (HSUVs), to improve the performance of the best performing filter, and to validate resulting search filters.

Methods: Three quasi-gold standard sets (QGS1, QGS2, QGS3) of relevant studies were harvested from reviews of studies reporting HSUVs. The performance of three initial filters was assessed by measuring their relative recall of studies in QGS1. The best performing filter was then developed further using QGS2. This resulted in three final search filters (FSF1, FSF2, and FSF3), which were validated using QGS3.

Results: FSF1 (sensitivity maximizing) retrieved 132/139 records (sensitivity: 95 percent) in the QGS3 validation set. FSF1 had a number needed to read (NNR) of 842. FSF2 (balancing sensitivity and precision) retrieved 128/139 records (sensitivity: 92 percent) with a NNR of 502. FSF3 (precision maximizing) retrieved 123/139 records (sensitivity: 88 percent) with a NNR of 383.

Conclusions: We have developed and validated a search filter (FSF1) to identify studies reporting HSUVs with high sensitivity (95 percent) and two other search filters (FSF2 and FSF3) with reasonably high sensitivity (92 percent and 88 percent) but greater precision, resulting in a lower NNR. These seem to be the first validated filters available for HSUVs. The availability of filters with a range of sensitivity and precision options enables researchers to choose the filter which is most appropriate to the resources available for their specific research.

Objectives: Economic evaluation (EE) is an accepted element of decision making and priority setting in healthcare. As the number of published EEs grows, so does the number of systematic reviews (SRs) of EEs. Although search methodology makes an important contribution to SR quality, search methods in reviews of EEs have not been evaluated in detail. We investigated the resources used to identify studies in recent, published SRs of EEs, and assessed whether the resources reflected recommendations.

Methods: We searched MEDLINE for SRs of EEs published since January 2013 and extracted the following from eligible reviews: databases searched, health technology assessment (HTA) sources searched, supplementary search techniques used. Results were compared against the minimum search resources recommended by National Institute for Health and Care Excellence (NICE) (MEDLINE, Embase, NHS EED, EconLit) for economic evidence for single technology appraisals, and resource types suggested in the summary of current best evidence from SuRe Info (economic databases, general databases, HTA databases, HTA agency Web pages, gray literature).

Results: Sixty-five SRs met the inclusion criteria; data were extracted from forty-two. Five reviews (12 percent) met or exceeded the NICE recommended resources. Nine reviews (21 percent) searched at least four of the five types of resource recommended by SuRe Info. Five reviews (12 percent) searched all five. Twenty-three reviews (55 percent) did not meet the NICE recommendations or four of five of the SuRe Info recommended resource types. Search reporting was frequently unclear or incorrect.

Conclusions: Searches conducted for the majority of recently published SRs of EEs do not meet two published approaches.

Background: The most effective sources to search to identify adverse effects data for medical devices are currently unknown.

Methods: The included studies from a systematic review of the safety of recombinant human bone morphogenetic protein-2 (rhBMP-2) for spinal fusion were used for analysis. For each source searched, a record was made for each relevant publication of whether it was retrieved by the search strategy used and whether it was available in the database but not retrieved. To account for multiple publications of the same study, a record was made of the relevant studies identified. The sensitivity, precision, and number needed to read were calculated as well as the minimum combination of sources to identify all the publications or studies.

Results: There were eighty-two publications (forty-nine studies) included in the systematic review. Only one article was available in a database searched but not retrieved by our search strategy. Science Citation Index (SCI) and EMBASE both achieved the highest sensitivity (62 percent), followed closely by MEDLINE/PubMED (56 percent). With the search strategies used, the minimum combination of sources needed to identify all the publications was SCI, EMBASE, CENTRAL, and either MEDLINE or PubMED, in addition to reference checking, contacting authors and an automated current awareness service. In relation to identifying all the relevant studies, the minimum combination of studies was similar with the exclusion of CENTRAL.

Conclusions: To identify all the relevant publications or studies included in this case study systematic review, several different sources needed to be searched.

Background: Information retrieval (IR) in health technology assessment (HTA) calls for transparency and reproducibility, but common practice in the documentation and presentation of this process is inadequate in fulfilling this demand.

Objectives: Our objective is to promote good IR practice by presenting the conceptualization of retrieval and transcription readable to non-information specialists, and reporting of effectively processed search strategies.

Methods: We performed a comprehensive database search (04/2010) to synthesize the current state-of-the-art. We then developed graphical and tabular presentation methods and tested their feasibility on existing research questions and defined recommendations.

Results: No generally accepted standard of reporting of IR in HTA exists. We, therefore, developed templates for presenting the retrieval conceptualization, database selection, and additional hand-searching as well as for presenting search histories of complex and lengthy search strategies. No single template fits all conceptualizations, but some can be applied to most processes. Database interface providers report queries as entered, not as they are actually processed. In PubMed®, the huge difference between entered and processed query is shown in “Details.” Quality control and evaluation of search strategies using a validated tool such as the PRESS checklist is suboptimal when only entry-query based search histories are applied.

Conclusions: Moving toward an internationally accepted IR reporting standard calls for advances in common reporting practices. Comprehensive, process-based reporting and presentation would make IR more understandable to others than information specialists and facilitate quality control.

Objectives and Methods: Health technology assessment (HTA) often requires the identification and review of economic evaluations and models. This study surveys the available specific and general resources to search to identify economic evaluations. It also provides information on efficient searching of those resources and comments on the current evidence-base.

Results: Published checklists recommend searching for economic evaluations in specific information resources which collect economic evaluations such as NHS EED and HEED, followed by top-up searches of large biomedical bibliographic databases (such as MEDLINE and EMBASE). Other resources such as the HTA and DARE databases can yield reports of economic evaluations. Searches within NHS EED and HEED can be made more efficient by using database-specific search options. Searches within large biomedical databases such as MEDLINE and EMBASE require the use of economic search terms called search filters. Search filters are highly sensitive, retrieving most economic evaluations, but suffer from low precision returning many irrelevant records which need to be assessed.

Conclusions: It is relatively easy to identify rapidly a high proportion of economic evaluations but more research is required to improve the efficiency of this process. There are key high yield resources to search but more evidence is required on their overlap and unique contribution to searches. The value of other resources, particularly those providing access to gray literature, should be explored. Research into efficient retrieval requires clear definitions of economic evaluations to allow comparison across studies.

Objectives: The aim of this study was to analyze the information needs of Spanish health technology assessment (HTA) agencies and units to facilitate access to the resources they require to substantiate their reports.

Methods: A questionnaire was designed and distributed among HTA bodies to ascertain the actual situation of subscriptions to information resources and what information specialists from these bodies considered would be the ideal subscription situation. Their information needs were then studied, and the resources that best met these needs were put forward. Following this definition, a subscriptions policy was adopted with suppliers and publishers.

Results: The survey showed that HTA bodies share a minimum of core subscriptions that includes open sources (MEDLINE, DARE) and sources that the government subscribes to for the health community (ISI Web of Science, Cochrane Library Plus). There was no common approach to determining which databases to subscribe to (UpToDate, EMBASE, Ovid EBMR, CINAHL, or ECRI). After identifying the information needs, a list of resources was proposed that would best cover these needs and, of these, subscription to the following was proposed: Scopus, Ovid EBMR, Clinical Evidence, DynaMed, ECRI, and Hayes.

Conclusions: There are differences in the way that HTA agencies and units access the different resources of biomedical information. Combined subscription to several resources for documentation services was suggested as a way of resolving these differences.

Background: Comprehensive health technology assessments (HTAs) include thorough reflections on ethical issues associated with health technologies, their use, and value-based decisions in the assessment process. As methods of information retrieval for effectiveness assessments are not applicable to information retrieval on ethical issues, a specific methodological approach is necessary.

Objectives: In the absence of existing adapted methods, our objective was to develop a methodological approach for the systematic retrieval of information on ethical issues related to health technologies.

Method and Results: A literature search was conducted to verify the non-existence of published comprehensive methodological approaches for the information retrieval on ethical issues for HTAs, and resulted in no hits. We, therefore, developed a step-by-step workflow following the workflow of information retrieval for effectiveness assessments: Step 1: Translation of the search question using the PICO scheme and additional components. Step 2: Concept building by modeling and linking search components. Step 3: Identification of synonyms in all relevant languages. Step 4: Selection of relevant information sources. Step 5: Design of search strategies for bibliographic databases. Step 6: Execution of search strategies and information seeking, including hand-searching. Step 7: Saving of retrieval results and standardized reporting of the process and results. Step 8: Final quality check and calculation of precision and recall.

Conclusions: Systematic searching for information on ethical issues related to health technologies can be performed following the common retrieval workflow for effectiveness assessments, but should be performed separately applying adapted procedures and search terms on ethical issues relevant to the research question.

Objectives: The aim of this study was to assess systematically the scope of evidence and purposes for which evidence is used in decision-analytic models of cost-effectiveness and to assess the implications for search methods.

Methods: A content analysis of published reports of models was undertaken. Details of cited sources were extracted and categorized according to three dimensions; type of information provided by the evidence, type of source from which the evidence was drawn and type of modeling activity supported by the evidence. The analysis was used to generate a classification of evidence. Relationships within and between the categories within the classification were sought and the implications for searching considered.

Results: The classification generated fourteen types of information, seven types of sources of evidence and five modeling activities supported by evidence. A broad range of evidence was identified drawn from a diverse range of sources including both research-based and non–research-based sources. The use of evidence was not restricted to the population of model parameters but was used to inform the development of the modeling framework and to justify the analytical and methodological approach.

Conclusions: Decision-analytic models use evidence to support all aspects of model development. The classification of evidence defines in depth the role of evidence in modeling. It can be used to inform the systematic identification of evidence.

Objectives: The aim of this study was to examine location of systematic reviews of test accuracy in five specialist review databases: York CRD's DARE and HTA databases, Medion (University of Maastricht), C-EBLM (International Federation of Clinical Chemistry), and the ARIF in-house database (University of Birmingham).

Methods: Searches were limited to the period 1996–2006. Test accuracy reviews were located using in-house diagnostic search filters and with help from database producers where databases were not confined to test accuracy reviews. References were coded according to disease area, review purpose, and test application. Ease of use, volume, overlap, and content of databases was noted.

Results: A large degree of overlap existed between databases. Medion contained the largest number (n = 672) and the largest number of unique (n = 328) test accuracy references. A combination of three databases identified only 76% of test reviews. All databases were rated as easy to search but varied with respect to timeliness and compatibility with reference management software. Most reviews evaluated test accuracy (85%) but the HTA database had a larger proportion of cost-effectiveness and screening reviews and C-EBLM more reviews addressing early test development. Most reviews were conducted in secondary care settings.

Conclusions: Specialist review databases offer an essential addition to general bibliographic databases where application of diagnostic method filters can compromise search sensitivity. Important differences exist between databases in terms of ease of use and content. Our findings raise the question whether the current balance of research setting, in particular the predominance of research on tests used in secondary care, matches the needs of decision makers.

Objectives: The aim of this study was to demonstrate the best way of identifying all relevant published health economic evaluation studies, which have increased in number rapidly in the past few decades. Nevertheless, health technology assessment projects are often faced with a scarcity of relevant studies.

Methods: Six bibliographic databases were searched using various individually adapted strategies. The particular example involves the cost-effectiveness of diagnosing gastroesophageal reflux disease. Inclusion and exclusion criteria were formulated.

Results: After irrelevant studies and duplicates had been excluded, sixty-eight abstracts were reviewed. We chose forty-one of them as relevant for full-text review, which identified fourteen papers as having met the inclusion criteria. Most of the relevant studies were identified by searching the National Health Service Economic Evaluation Database (NHS EED) and PubMed databases.

Conclusions: A search in NHS EED, by means of the Cochrane Library or the Center for Reviews and Dissimination, along with a supplementary search in PubMed, is generally an appropriate, cost-effective strategy. However, because “cost-effectiveness” is not consistently indexed with Medical Subject Heading terms in PubMed, all economic search terms need to be used to fully identify the relevant references.

Objectives: The aim of this study was to evaluate the success of search strategies in retrieving key documents for a technology assessment report (TAR) on a social care topic.

Methods: This study measured the differential yield of relevant studies from various information sources and evaluated strategies in different databases, with particular reference to capturing randomized controlled trials (RCTs) as a study design.

Results: A combination of four major databases would have found all thirty-two key references. One database alone would have found 78 percent, with another two each locating 59 percent. Sixteen percent of the trials were unique references. In non–health care databases, more sensitive search strategies would have resulted in a higher yield of relevant studies, in part due to inconsistent indexing and in part to attempts to restrict searches to RCTs. Although additional terms could be used to increase the sensitivity of the original strategies, this raises the question of trading off time against exhaustiveness, given the greater number of irrelevant references likely to be retrieved.

Conclusions: A successful search for evidence on this social care topic would be possible using a combination of MEDLINE, EMBASE, the Cochrane Library and PsycINFO, supplemented by only limited use of supplementary databases. In areas such as social care where evidence-based research is not yet well established, attempts to replicate searches based on study design do not seem to be advisable, although this may be an area for future research.

Objectives: The aim of the study was to investigate the feasibility and effectiveness of searching selected databases to identify information required to populate a decision-analytic model.

Methods: Methods of searching for information to populate a decision-analytic model were piloted using a case study of prophylactic antibiotics to prevent recurrent urinary tract infections in children. This study explored how the information requirements for a decision-analytic model could be developed into searchable questions and how search strategies could be derived to answer these questions. The study also assessed the usefulness of three published search filters and explored which resources might produce relevant information for the various model parameters.

Results: Based on the data requirements for this case study, 42 questions were developed for searching. These questions related to baseline event rates, health-related quality of life and outcomes, relative treatment effects, resource use and unit costs, and antibiotic resistance. A total of 1,237 records were assessed by the modeler, and of these, 48 were found to be relevant to the model. Search precision ranged from 0 percent to 38 percent, and no single database proved the most useful for all the questions.

Conclusions: The process of conducting specific searches to address each of the model questions provided information that was useful in populating the case study model. The most appropriate resources to search were dependent on the question, and multiple database searching using focused search strategies may prove more effective in finding relevant data than thorough searches of a single database.