Objectives: The aim of this study was to examine potential associations between the professional background and experience of expert clinicians and their opinions about the clinical utility of interventional procedures.

Methods: A retrospective survey of expert clinician characteristics and their opinions was conducted. Information was collected on expert clinical adviser self-declared “operator,” “researcher,” and conflict of interest status. Associations were sought between expert clinical adviser characteristics and their opinions on whether procedures were “established,” “efficacious,” and “safe.” The setting was the Interventional Procedures Programme of the UK's National Institute for Health and Clinical Excellence (NICE). A total of 598 expert clinician questionnaires relating to 182 different interventional procedures were analyzed.

Results: Expert clinical advisers with operative experience of procedures were significantly more likely to consider them as established (odds ratio [OR] 3.93; 95 percent confidence interval [CI], 2.43 to 6.36; p < .001), efficacious (OR 1.76; 95 percent CI, 1.00 to 3.08; p = .049), and safe (OR 2.28; 95 percent CI, 1.43 to 3.65; p = .001). Once adjusted for other characteristics, there was no association between either researcher or conflict of interest status and opinions about the clinical utility of procedures.

Conclusions: Expert clinical advisers are an important source of information for decision makers producing guidance about the use of procedures, especially when published evidence is sparse or of poor quality. This study suggests that those who are operators, but not those who are researchers or declare a conflict of interest, are more likely to have a favorable opinion of a procedure's clinical utility. Use of expert clinical advisers with a variety of experience and backgrounds seems a reasonable approach to obtaining authoritative opinions about interventional procedures, to supplement and help interpret evidence from published data.

Objectives: In a resource-constrained reality, physicians are facing polar demands—those of healthcare managements to adopt cost-conscious behaviors and those of ethical standards that obligate physicians to consider only their patients' best interests. In our study, we aimed to determine the attitudes, practices, and knowledge of healthcare costs among Israeli physicians.

Methods: A questionnaire was developed and mailed to a representative sample of physicians in Israel. The overall response rate was 51 percent. The study reviewed self-reported levels of cost consciousness in practice, attitudes, obstacles related to cost containment, and knowledge of the costs of medical resources.

Results: Forty-two percent of the physicians reported high levels of cost consciousness in their daily practice; 70 percent reported greater current cost consciousness in comparison to 5 years ago; 76 percent of the responses legitimized institutional demands for cost containment. Although 83 percent of the physicians that responded expressed the belief that economic thinking was inherently the role of management, only 39 percent thought it was part of the physician's role. It was found that predominant predictors of agreement to cost consciousness concepts were employment by a community health plan, a managerial position, participation in health economics seminars, and male gender.

Conclusions: Cost consciousness among physicians is related to a broad array of parameters. Interventions must emphasize the benefits of evidence-based medicine as an anchor for both cost containment and quality care, as well as providing assistance to physicians in accepting economic decision-making as part of their professional role.

Objectives and Methods: Many authors have argued that ethical, legal, and social issues (“ELSIs”) should be explicitly integrated into health technology assessment (HTA), yet doing so poses challenges. This discussion may be particularly salient for technologies viewed as ethically complex, such as genetic screening. Here we provide a brief overview of contemporary discussions of the issues from the HTA literature. We then describe key existing policy evaluation frameworks in the fields of disease screening and public health genomics. Finally, we map the insights from the HTA literature to the policy evaluation frameworks, with discussion of the implications for HTA in genetic screening.

Results and Conclusions: A critical discussion in the HTA literature considers the definition of ELSIs in HTA, highlighting the importance of thinking beyond ELSIs as impacts of technology. Existing HTA guidance on integrating ELSIs relates to three broad approaches: literature synthesis, involvement of experts, and consideration of stakeholder values. The thirteen key policy evaluation frameworks relating to disease screening and public health genomics identified a range of ELSIs relevant to genetic screening. Beyond straightforward impacts of screening, these ELSIs require consideration of factors such as the social and political context surrounding policy decisions. The three broad approaches to addressing ELSIs described above are apparent in the screening/genomics literatures. In integrating these findings we suggest that the method chosen for addressing ELSIs in HTA for genetic screening may determine which ELSIs are prioritized; and that an important challenge is the lack of guidance for evaluating such methods.

Objectives: The aim of this study was to examine the impact of the multi-hundred million dollar investment by the federal government in the developing Patient Outcomes Research Teams (PORTs) in over a dozen major academic medical centers in the United States throughout the 1990s. The objective of the PORTs was to reduce unnecessary clinical variation in medical treatment.

Methods: Using an economic derivation of welfare loss attributable to medical practice variation and hospital admission claims data for 2 million elderly patients generalizable to the nation, we estimate the change in welfare between 1991 and 2000, the period within which the PORTs were designed and executed and their results disseminated.

Results: Our results show inpatient admission types targeted by the PORTs did have less welfare loss relative to their total expenditure by 2000, but that there was not a net decrease in the welfare loss for all hospital admissions affected by the PORT.

Conclusions: We conclude that PORTs may have had favorable effects on welfare, most likely by reducing variation in clinical care, but that causality cannot be proved, and the effects were not equal across all conditions targeted by PORTs. This research provides a methodological template that may be used to evaluate the impact of patient safety research on welfare loss and on variation in medical treatment in both hospital and ambulatory settings.

Objectives: The purpose of this study is to compare the approaches used for valuing family caregiver and care recipient time devoted to providing and receiving care.

Methods: Valuation approaches were operationalized within a cohort of cystic fibrosis care recipients (n = 110). Base-case analyses, grounded in human capital theory, applied earnings estimates to caregiving time to impute the market value of time lost from labor. Unpaid labor and leisure time was valued with a replacement cost (homemaker's wage rate). Total time costs were computed and sensitivity analyses were conducted to describe the effects of alternative valuation methods on total costs.

Results: The mean time cost per care recipient–caregiver dyad over 28 days was $2,026CAD. The majority (76 percent) of time costs were due to losses from unpaid labor and leisure time. Varying the valuation of paid labor time did not result in significantly different total time costs (p = .0877). However, varying the method of valuing unpaid labor and leisure time did significantly affect total costs (p < .0001).

Conclusions: Care recipients and caregivers primarily lost time from unpaid labor and leisure in the treatment of cystic fibrosis. Moreover, when the above losses were aggregated, the method of valuation greatly influenced overall results. The findings clearly indicate that omitting caregiver and unpaid labor and leisure costs may result in an inaccurate assessment of ambulatory and home-based healthcare programs.

Objectives: The aim of this study was to provide a comprehensive summary and interpretation of the current evidence on the use and validity of statistical methods to conduct indirect comparisons of treatment effects.

Methods: A narrative review was conducted.

Results: Well-conducted methodological studies provide good evidence that adjusted indirect comparisons can lead to results similar to those from direct comparisons. The internal validity of several statistical methods to conduct indirect comparisons, therefore, has been established. Meta-regression, logistic regression, or adjusted indirect comparisons should be the methods of first choice. Unadjusted indirect comparisons are always unacceptable. Deciding whether to combine direct and indirect evidence will be a matter of informed judgment based on the similarities and dissimilarities of populations and interventions. Unverifiable assumptions with respect to the similarity of compared studies and low power are serious limitations of indirect comparisons.

Conclusions: In the absence of sufficient head-to-head evidence, adjusted indirect comparisons can be viewed as additional analytical tools to determine the comparative efficacy and effectiveness of competing interventions. Researchers who use indirect comparisons need to keep the limitations in mind.

Objectives: The study aims at evaluating the economic evidence related to testing for genetic variants of the drug-metabolizing enzyme, TPMT. Detecting TPMT genetic variants before the administration of azathioprine (AZA) has the potential to prevent serious and costly adverse drug reactions (ADRs), such as neutropenia. In particular, our analysis concentrated on assessing the reliability of data on costs of neutropenia and performing the tests, the two main cost categories that could inform an economic evaluation of TPMT pharmacogenetic testing.

Methods: A systematic literature review was performed to gather evidence on the costs of testing and neutropenia. Articles were critically appraised for their comprehensiveness and quality. To better estimate costs of TPMT tests, a small-scale survey of European diagnostic laboratories was conducted.

Results: Only seven articles were retrieved specifying the costs associated with the management and treatment of AZA-induced neutropenia. Most of these studies are based on theoretical modeling reconstructed with key-informants or on very few cases of ADRs, and either the methodology for cost calculation is not specified or costs are based on national cost databases and tariffs. After critical appraisal of these studies, we considered €2,116 as the most reliable estimate for the cost of a case of neutropenia. Literature review accompanied by the survey of several diagnostic laboratories also provided an estimate (€68) for TPMT testing. Based on these values, the net cost per prevented case of neutropenia equals to €5,300.

Conclusions: Solid economic considerations related to TPMT pharmacogenetic testing are still limited by underreporting of ADRs and high level of approximation related to cost data. Ad hoc observational studies and the ADR recording process embedded in pharmacovigilance systems, established across Europe, should represent more reliable sources of cost data in the future.

From the heydays of HTA in the 1970s, it has been argued that ethics should be a part of HTA. Despite more than 30 years with repeated intentions, only few HTA reports include ethical analysis, and there is little agreement on methods for integrating ethics. This poses the question of why it is so important to integrate ethics in HTA? The article analyzes ten arguments for making ethics part of HTA. The validity of the arguments depend on what we mean by “integrating,” “ethics,” and “HTA.” Some of the counterarguments explain why it has taken so long to integrate ethics in HTA and why there are so many ethical approaches. Nevertheless, some of the arguments for making ethics part of HTA appear to be compelling. Health care is a moral endeavor, and the vast potential of technology poses complex moral challenges. A thorough assessment of technology would include reflection on these moral aspects. Ethics provides such a moral reflection. Health technology is a way to improve the life of human individuals. This involves questions of what “the good life” is, and hence ethical issues. Trying to ignore such questions may inflict with the moral foundation of health care: to help people. Additionally, HTA is an evaluation, and as such also a reflection on values. Hence, there is a profound affinity between HTA and ethics. Accordingly, ethics cannot be “integrated” in HTA as ethics is already a constitutive part of HTA. However, ethics can be acknowledged and emphasized.

Objectives: The aim of this study was to assess the evidence that reuse of medical devices marketed for single use only (SUDs) is safe, effective and cost-effective, and to consider the use and health services impact of this practice in Canada.

Methods: A systematic review was performed of studies that reported clinical or economic outcomes following reuse of SUDs in humans. Direct costs of adverse health events associated with SUD reuse and indications of budget impact were obtained using data for devices for laparoscopic cholecystectomy and coronary angioplasty. Legal and ethical issues were reviewed, drawing on material relevant to Canada. Data on current reuse of SUDs were obtained through a survey of Canadian acute care hospitals.

Results: Studies of variable quality suggested that SUD reuse could be safe and effective, and would give cost savings, if there were no adverse events. Eliminating reuse of SUDs for laparoscopic cholecystectomy and coronary angioplasty would add less than 0.1 percent to costs of the procedures over 1 year. Adverse health events associated with device reuse create liability risks; patients should be informed of any known or foreseeable risks of reuse. Most of the 28 percent (111/398) of acute hospitals that reprocess SUDs do so in-house. Some do not have a written policy or an incident reporting mechanism.

Conclusions: There is insufficient evidence to establish the safety, efficacy and cost-effectiveness of reusing SUDs. Legal and ethical issues require attention to minimize liability and maintain patient safety and trust. Some hospitals that reprocess SUDs do not have adequate documentation. These findings do not support the reuse of SUDs in Canadian hospitals.

Background: Home-fortification is a new strategy of adding micronutrients including zinc and iron to home-made foods. Zinc supplementation may prevent morbidity and mortality related to diarrheal illnesses, and iron supplementation may improve cognitive development, in children.

Objectives: To project clinical and economic effects of home-fortification in children in an urban slum of Karachi, Pakistan.

Methods: This is a cost benefit analysis of 5,000 simulated male and female infants (6–12 months) assigned to micronutrients or placebo for 4 months and followed for 55 years. We linked the effect of zinc on longitudinal prevalence of diarrhea to mortality, and the effect of iron on hemoglobin to IQ scores and lifetime earnings. Cost estimates were based on volumes of resource utilization from the Pakistan Sprinkles Diarrhea study. Main outcome was incremental benefit defined as the gain in lifetime earnings after accounting for the incremental costs of micronutrients over placebo (societal perspective).

Results: Our model projected that the reduction in diarrhea and improvement in hemoglobin concentrations through home-fortification was associated with reduced child mortality, higher IQ scores, and higher earnings. The present value of incremental benefit was $106 (95 percent probability interval = $17 to $193) U.S. dollars, which corresponds to $464.79 ($74.54 to $846.27) international dollars using a purchasing power parity exchange rate.

Conclusions: Home-fortification appears to improve clinical outcomes at a reasonable cost, and may actually be cost beneficial when lifetime earnings are considered.

Objectives: The aim of this study was to explore the needs and requirements of decision makers in our regional healthcare system for health technology assessment (HTA) products to support portfolio development planning for a new HTA agency in Madrid, Spain.

Methods: A Delphi study was conducted during 2003. Questionnaires were developed based on a review of products and services offered by other agency members of the International Network of Agencies for Health Technology Assessment, and included preference and prioritization questions to evaluate twenty-two different products and services. The initial Delphi panel involved eighty-seven experts from twenty-one public hospitals, eleven primary healthcare centers, six private hospitals, and eight departments of the Regional Ministry of Health of the Community of Madrid.

Results: The global participation rate was 83.9 percent. Ten of the twenty-two possible products were rated of high interest by more than 80 percent of respondents. Important differences in preferences and priorities were detected across different settings. Public hospitals and primary healthcare centers shared a more “micro” perspective, preferring classic technology-centered HTA products, whereas private hospitals and Ministry representatives demanded more “macro” products and services such as organizational model and information system assessments.

Conclusions: The high participation rate supports the representativeness of the results for our regional context. The strategic development of an HTA portfolio based on decision makers’ needs and requirements as identified in this type of exercise should help achieve a better impact on policy development and decision making.

Background: Hyperbaric oxygen therapy (HBOT) has been proposed as an adjunct to standard methods of care for diabetic foot ulcers (DFU). Its use may decrease the risk of infection and lower extremity amputations (LEAs). As part of a Canadian assessment, we estimated the cost-effectiveness and budget impact of HBOT in this application.

Methods: We developed a decision model comparing adjunctive HBOT with standard care alone. The population was a 65-year-old cohort with DFU. The time horizon was 12 years taken from a Ministry of Health perspective. The health states were a healed wound with or without a minor LEA, an unhealed wound with no related surgery, and a major LEA. Efficacy data were based on outcomes reported in studies included in a literature review. Cost and capacity needs for treating DFU patients in Canada were estimated using prevalence data from the literature, and cost and utilization data from government records.

Results: The 12-year cost for patients receiving HBOT was CND$40,695 compared with CND$49,786 for standard care alone. Outcomes were 3.64 quality-adjusted life-years (QALYs) for those receiving HBOT and 3.01 QALYs for controls. Estimated cost to treat all prevalent DFU cases in Canada was CND$14.4–19.7 million/year over 4 years. If seven-person HBOT chambers were used, a further nineteen to thirty-five machines would be required nationally.

Conclusions: Adjunctive HBOT for DFU is cost-effective compared with standard care. Additional HBOT capacity would be needed if it were to be adopted as the standard of care throughout Canada.

Objectives: The aim of this study was to summarize the current evidence for the cost-effectiveness of primarily human papillomavirus (HPV) -based cervical cancer screening in settings with already established Papanicolaou test (Pap) programs. Emphasis was placed on the German situation with annual Pap screening.

Methods: Medical, economic, and health technology assessment (HTA) databases were systematically searched for cost-effectiveness studies comparing HPV to Pap screening. Study data were extracted, standardized, and summarized in cost-effectiveness plots contrasting HPV strategies to Pap screening with 1-, 2-, 3-, and 5-years interval. For each Pap setting, the likelihood of cost-effective HPV screening was assessed depending on willingness-to-pay.

Results: We reviewed twelve decision-analytic cost-effectiveness models. Study results showed wide variation due to methodical heterogeneity. Data synthesis revealed that the cost-effectiveness of HPV screening depends on the interval of the established Pap screening strategy. In comparison with Pap screening every 2 years, only 25 percent of the HPV-based screening strategies were cost-effective. However, in comparison with Pap screening every 1, 3, or 5 years, 83 percent, 55 percent, and 92 percent of HPV screening strategies were cost-effective, respectively. Results for settings with annual Pap screening are based on models assuming 100 percent screening coverage.

Conclusions: The introduction of HPV-based screening programs is cost-effective if the screening interval of the established Pap program exceeds 2 years. In settings with biennial Pap screening, introduction of HPV-based screening is unlikely to be cost-effective. Results also suggest cost-effectiveness of HPV-based screening in settings with annual Pap screening; however, this finding should be confirmed under realistic screening adherence assumptions.

Objectives: To examine the production of Health Technology Assessments (HTAs) with economic evaluations (EEs) conducted by Canadian HTA agencies.

Methods: This research used a three-step approach: (i) the Web sites of five Canadian organizations promoting HTA were searched to identify HTA reports with EEs; (ii) HTA agencies were surveyed to verify that our information was complete with respect to HTA activities and to describe the factors that influence the HTA process in Canada (i.e., selection of HTA topics, execution, dissemination of results and future trends); (iii) HTAs with EEs were appraised in terms of study design, retrieval of clinical and economic evidence, resource utilization and costing, effectiveness measures, treatment of uncertainty as well as presence of a budget impact analysis (BIA), and policy recommendations.

Results: Two hundred forty-nine HTA reports were identified of which 19 percent included EEs (n = 48). Decision analytic techniques were used in approximately 75 percent of the forty-eight EEs and probabilistic sensitivity analyses were commonly used by all agencies to deal with parameter uncertainty. BIAs or policy recommendations were given in 50 percent of the evaluations. Differences between agencies were observed in terms of selection of topics, focus of assessment and production of HTA (e.g., in-house activities). Major barriers to the conduct of HTAs with EEs were capacity, a lack of interest by decision makers and a lack of robust clinical information.

Conclusions: The results of this research point to the need for increased HTA training, collaboration, evidence synthesis, and use of pragmatic “real world” evaluations.

Objectives: The aims of this study were to assess the 1-year cost-effectiveness of a new combined immunosuppressive and anti-infectious regimen in kidney transplantation to prevent both rejection and infectious complications.

Methods: Patients (pts) transplanted from January 2000 to March 2003 (Group A) and treated with a conventional protocol were compared with pts submitted to a combined regimen including universal cytomegalovirus (CMV) prophylaxis between April 2003 and July 2005 (Group B). Costs were computed from the hospital accounting system for hospital stays, and official tariffs for outpatient visits. Patients with incomplete costs data were excluded from analysis.

Results: Fifty-three patients were analyzed in Group A, and 60 in Group B. Baseline characteristics including CMV serostatus were not significantly different between the two groups. Over 12 months after transplantation, acute rejections decreased from 41.5 percent in Group A to 6.7 percent in Group B (p < .001), and CMV infections from 47 percent to 15 percent (p < .001). Overall, readmissions decreased from 68 percent to 55 percent (p = .160), and average hospital days from 28 ± 19 to 20 ± 11 days (p < .007). The average number of outpatient visits decreased from 49 ± 10 to 39 ± 8 (p < .001). Average 1-year immunosuppressive and CMV prophylaxis costs (per patient) increased from CHF20,402 ± 7,273 to 27,375 ± 6,063 (p < .001), graft rejection costs decreased from CHF4,595 ± 10,182 to 650 ± 3,167 (p = .005), CMV treatment costs from CHF2,270 ± 6,161 to 101 ± 326 (p = .008), and outpatient visits costs from CHF8,466 ± 1’721 to 6,749 ± 1,159 (p < .001). Altogether, 1-year treatment costs decreased from CHF39’957 ± 16,573 to 36,204 ± 6,901 (p = .115).

Conclusions: The new combined regimen administered in Group B was significantly more effective, and its additional costs were more than offset by savings associated with complications avoidance.

Objectives: The aim of this study was to assess whether publications of importance for improving the health system and its technologies are highly cited intrascientifically.

Methods: Bibliometric assessment of the 596 publications used as sources in the fifty SBU Alerts from 2001to 2004 from the Swedish Council on Technology Assessment in Health Care was carried out using the Thomson Scientific citation indexes. Normalized citation scores were calculated for all included studies. Additional factors such as the time and place of the research, subject categories, and journal source were analyzed.

Results: On average, the sources in SBU Alert have been cited eight times more than the world average consistently during the time period and across research areas. Articles used as its scientific basis are often published in a few, high impact, general medical journals. However, many of the articles are published in field-specific journals with low impact factors. Most articles used in SBU Alert are published by authors based in the United States or the United Kingdom. However, Swedish, Danish, and Dutch publications are overrepresented in its science base, whereas Japanese, Taiwanese, Indian, and Russian publications are underrepresented.

Conclusions: Publications used as sources in a Swedish system for identification and early assessment of new methods in health care are also highly cited within the scientific community. This finding increases the appropriateness of using bibliometric indicators in evaluations of clinical research and suggests that decision makers through SBU Alert are getting scientifically sound advice.

Objectives: Multidisciplinary and multifactorial interventions seem to be effective in preventing falls. We aimed to assess the cost-effectiveness of a multidisciplinary fall prevention program compared with usual Dutch healthcare in community-dwelling people 65 years of age or older who experienced a fall.

Methods: Cost-effectiveness and cost-utility analysis were performed from a societal perspective. Falls and healthcare utilization were continuously measured for 12 months. Daily functioning and quality of life were measured at baseline, after 4 and 12 months. Bootstrap analyses were performed to estimate uncertainty of the findings and sensitivity analysis to assess the generalizability of assumptions made.

Results: One hundred sixty-six participants were randomly allocated to the experimental group and 167 to the control group. The overall response rate was 74 percent. Healthcare and patient and family costs of both groups were comparable. Our analyses showed no effect of the intervention program on falls, daily functioning, or quality of life measures.

Conclusions: The multidisciplinary intervention program to prevent falls was not cost-effective compared with usual care in the Netherlands. Notwithstanding our findings, however, falls still have an important impact on society and individuals in terms of costs and effects. Economic evaluations studying promising interventions to prevent falls, therefore, remain necessary.

Objectives: Information technology is a rapidly expanding branch of science which has affected other sciences. One example of using information technology in medicine is the Electronic Medical Records system. One medical university in Iran decided to introduce such system in its hospital. This study was designed to identify the factors which influence the quality of medical documentation when paper-based records are replaced with electronic records.

Methods: A set of 300 electronic medical records was randomly selected and evaluated against eleven checklists in terms of documentation of medical information, availability, accuracy and ease of use. To get the opinion of the care-providers on the electronic medical records system, ten physicians and ten nurses were interviewed by using of semi-structured guidelines. The results were also compared with a prior study with 300 paper-based medical records.

Results: The quality of documentation of the medical records was improved in areas where nurses were involved, but those parts which needed physicians' involvement were actually worse. High workloads, shortage of bedside hardware and lack of software features were prominent influential factors in the quality of documentation. The results also indicate that the retrieval of information from the electronic medical records is easier and faster, especially in emergency situations.

Conclusions: The electronic medical records system can be a good substitute for the paper-based medical records system. However, according to this study, some factors such as low physician acceptance of the electronic medical record system, lack of administrative mechanisms (for instance supervision, neglecting physicians and/or nurses in the development and implementation phases and also continuous training), availability of hardware as well as lack of specific software features can negatively affect transition from a paper-based system to an electronic system.

In light of growing demands for public accountability, the broadening scope of health technology assessment organizations (HTAOs) activities and their increasing role in decision-making underscore the importance for them to demonstrate their performance. Based on Parson's social action theory, we propose a conceptual model that includes four functions an organization needs to balance to perform well: (i) goal attainment, (ii) production, (iii) adaptation to the environment, and (iv) culture and values maintenance. From a review of the HTA literature, we identify specific dimensions pertaining to the four functions and show how they relate to performance. We compare our model with evaluations reported in the scientific and gray literature to confirm its capacity to accommodate various evaluation designs, contexts of evaluation, and organizational models and perspectives. Our findings reveal the dimensions of performance most often assessed and other important ones that, hitherto, remain unexplored. The model provides a flexible and theoretically grounded tool to assess the performance of HTAOs.

Objectives: The increasing use of full economic evaluations has led to the development of various instruments to assess their quality. The purpose of this study was to compare the frequently used British Medical Journal (BMJ) check-list and two new instruments: the Consensus Health Economic Criteria (CHEC) list and the Quality of Health Economic Studies (QHES) instrument. The analysis was based on a practical exercise on economic evaluations of the surgical treatment of obesity.

Methods: The quality of nine selected studies was assessed independently by two health economists. To compare instruments, the Spearman rank correlation coefficient was calculated for each assessor. Moreover, the test–retest reliability for each instrument was assessed with the intraclass correlation coefficient (ICC) (3,1). Finally, the inter-rater agreement for each instrument was estimated at two levels: comparison of the total score of each article by the ICC(2,1) and comparison of results per item by kappa values.

Results: The Spearman's rank correlation coefficient between instruments was usually high (rho > 0.70). Furthermore, test–retest reliability was good for every instruments, that is, 0.98 (95 percent CI, 0.86–0.99) for the BMJ check-list, 0.97 (95 percent CI, 0.73–0.98) for the CHEC list, and 0.95 (95 percent CI, 0.75–0.99) for the QHES instrument. However, inter-rater agreement was poor (kappa < 0.40 for most items and ICC(2,1) ≤ 0.5).

Conclusions: The study shows that the results of the quality assessment of economic evaluations are not so much influenced by the instrument used but more by the assessor. Therefore, quality assessments should be performed by at least two independent experts and final scoring based on consensus.