Objectives: No guidelines exist in the approach that Evidence Review Groups (ERGs) should take to appraise search methodologies in the manufacturer's submission (MS) in Single Technology Appraisals (STA). As a result, ERGs are left to appraise searches using their own approach. This study investigates the limitations of manufacturers' search methodologies as critiqued by ERGs in published STA reports and to provide a recommended checklist.

Methods: Limitations from search critiques in 83 ERG reports published in the NIHR Web site between 2006 and May 2011 were extracted. The limitations were grouped into themes. Comparisons were made between limitations reported in the clinical effectiveness versus cost-effectiveness searches.

Results: Twelve themes were identified, six relating to the search strategy, source, limits, filters, translation, reporting, and missing studies. The search strategy theme contained the most limitations. Missing studies were frequently found by the ERG group in the clinical effectiveness searches. The omission of searches by manufacturers for unpublished and ongoing trials was frequently reported by the ERG. By contrast, failure of the manufacturer to report strategies was the most common limitation in the cost-effectiveness searches. Themes with the most frequent limitations in both types of searches are search strategy, reporting and source.

Conclusions: It is recommended that a checklist that has reporting, source and search strategy elements be used in the appraisal of manufacturer's searches during the STA process.

Objectives: A recent study showed that estimates of cost-effectiveness submitted to National Institute for Health and Clinical Excellence (NICE) by manufacturers had significantly lower incremental cost-effectiveness ratios (ICERs) than those submitted by university-based Assessment Groups. This study extends that analysis.

Methods: Data were abstracted from relevant NICE documentation for thirty-two of eighty-two possible appraisals.

Results: The results from the analysis showed that sources of the difference in ICERs appear to be the effectiveness estimates relating to the comparator technology and the cost estimates relating to the technology under evaluation. That is, manufacturers estimated lower average benefits for the comparator technology and lower costs relating to the technology under evaluation compared with estimates submitted by the Assessment Groups.

Conclusions: These findings may be particularly important, given the introduction of the “Single Technology Appraisal.” Considerable difficulties were encountered when undertaking this study, highlighting, above all else, the complexity of explaining why results from economic evaluations purporting to answer the same question diverge.