Objectives: Evaluation is essential for the management of international projects or networks in health technology assessment (HTA). It extends beyond the normal process of project management by incorporating qualitative dimensions and provides information about a project's effectiveness and achievements. This article aimed to identify the factors that are important for the success of international HTA projects. The European network for Health Technology Assessment Joint Action (EUnetHTA JA) is presented as an exemplar.

Methods: Methods for the evaluation of international HTA projects include interviews, focus groups, questionnaires, observations and documentary review, and the key points of these approaches have been summarized. The impact and effectiveness of the EUnetHTA JA was evaluated by questionnaires of project participants and external stakeholders, and by documentary review.

Results: The response rate for the three annual questionnaires sent to project participants ranged from 86 percent to 88 percent and for external stakeholders ranged from 65 percent to 88 percent. Key factors for project success included production of deliverables according to the workplan, achievement of objectives, added value generated, effective communication, involvement of external stakeholders, workstream management and progress from the preceding EUnetHTA 2006–2008 project.

Conclusions: The experience of this project can inform the evaluation of future international HTA collaborations, such as the EUnetHTA 2nd Joint Action and HTAsiaLink. A high response rate was achieved to the self-completion questionnaires and the strategy followed is recommended for evaluation of international HTA projects. Future assessments of international HTA projects should strive to measure outcomes and impact, not just outputs and process.

Objectives: The aim of this study was to identify risk points in the different stages of the smart infusion pump implementation process to prioritize improvement measures.

Methods: Failure modes and effects analysis (FMEA) in the pediatric intensive care unit (PICU) of a General and Teaching Hospital. A multidisciplinary team was comprised of two intensive care pediatricians, two clinical pharmacists and the PICU nurse manager. FMEA was carried out before implementing CareFusion infusion smart pumps and eighteen months after to identify risk points during three different stages of the implementation process: creating a drug library; using the technology during clinical practice and analyzing the data stored using Guardrails® CQI v4.1 Event Reporter software.

Results: Several actions for improvement were taken. These included carrying out periodical reviews of the drug library, developing support documents, and including a training profile in the system so that alarms set off by real programming errors could be distinguished from those caused by incorrect use of the system. Eighteen months after the implementation, these measures had helped to reduce the likelihood of each risk point occurring and increase the likelihood of their detection.

Conclusions: Carrying out an FMEA made it possible to detect risk points in the use of smart pumps, take action to improve the tool, and adapt it to the PICU. Providing user training and support tools and continuously monitoring results helped to improve the usefulness of the drug library, increased users’ compliance with the drug library, and decreased the number of unnecessary alarms.

Objectives: Semiautomated external defibrillators (AEDs) should be considered as a means to facilitate in-hospital early defibrillation (IHED) in areas where advanced life support rescuers are not readily available. In this study, we aimed to develop a checklist and a measurement protocol to evaluate and compare AEDs by assessing factors that may affect IHED.

Methods: A clinical and technical comparison of six AEDs was performed. Technical specifications were analyzed, while an emergency team evaluated ergonomics and appropriateness for IHED at Bambino Gesù Children's Hospital. A measurement protocol was implemented, which aimed to assess the ability of defibrillators to recognize shockable and nonshockable rhythms, accuracy of delivered energy, and charging time.

Results: Designs of AEDs differed in several features which influence their appropriateness for IHED. Some units showed poor ergonomics and instructions/feedback for cardiopulmonary resuscitation. Differences between defibrillators in recognizing shockable and nonshockable rhythms emerged for polymorphic ventricular tachycardia waveforms and when the frequency and amplitude of input signals varied. Tests for accuracy revealed poor performances at low and high impedance levels for most AEDs. Notably, differences greater than 20 seconds were found in the time from power-on to “ready for discharge.”

Conclusions: The approach we used to assess AEDs allowed us to evaluate their appropriateness with respect to the organizational context, to measure their parameters, and to compare models. Results showed that ergonomics and/or performances (timing and accuracy) could be improved in each device.

Objectives: Rare diseases are often heterogeneous in their progression and response to treatment, with only a small population for study. This provides challenges for evidence generation to support HTA, so novel research methods are required.

Methods: Discussion with an expert panel was augmented with references and case studies to explore robust approaches for HTA evidence generation for rare disease treatments.

Results: Traditional RCTs can be modified using sequential, three-stage or adaptive designs to gain more power from a small patient population or to focus trial design. However, such designs need to maintain important design aspects such as randomization and blinding and be analyzed to take account of the multiple analyses performed. N-of-1 trials use within-patient randomization to test repeat periods of treatment and control until a response is clear. Such trials could be particularly valuable for rare diseases and when prospectively planned across several patients and analyzed using Bayesian techniques, a population effect can be estimated that might be of value to HTA. When the optimal outcome is unclear in a rare disease, disease specific patient reported outcomes can elucidate impacts on patients’ functioning and wellbeing. Likewise, qualitative research can be used to elicit patients’ perspectives, with just a small number of patients.

Conclusions: International consensus is needed on ways to improve evidence collection and assessment of technologies for rare diseases, which recognize the value of novel study designs and analyses in a setting where the outcomes and effects of importance are yet to be agreed.

Objectives: The current digital revolution is particularly relevant for interactions of healthcare providers with patients and the community as a whole. The growing public acceptance and distribution of new communication tools such as smart mobile phones provide the prerequisite for information and communication technology (ICT) -assisted healthcare applications. The present study aimed at identifying specifications and perceptions of different interest groups regarding future demands of ICT-supported doctor–patient communication in Austria.

Methods: German-speaking Austrian healthcare experts (n = 73; 74 percent males; mean age, 43.9 years; SD 9.4) representing medical professionals, patient advocates, and administrative personnel participated in a 2-round online Delphi process. Participants evaluated scenario-based benefits and obstacles for possible prospect introduction as well as degree of innovation, desirability, and estimated implementation dates of two medical care-related future set ups.

Results: Panelists expected the future ICT-supported doctor–patient dialogue to especially improve the three factors doctors–patient relationship, patients’ knowledge, and quality of social health care. However, lack of acceptance by doctors, data security, and monetary aspects were considered as the three most relevant barriers for ICT implementation. Furthermore, inter-group comparison regarding desirability of future scenarios showed that medical professionals tended to be more skeptical about health-related technological innovations (p < .001).

Conclusions: The findings of this survey revealed different expectations among interest groups. Thus, we suggest building taskforces and using workshops for establishing a dialogue between stakeholders to positively shape the future of ICT-supported collaboration and communication between doctors and patients.

Background: The most effective sources to search to identify adverse effects data for medical devices are currently unknown.

Methods: The included studies from a systematic review of the safety of recombinant human bone morphogenetic protein-2 (rhBMP-2) for spinal fusion were used for analysis. For each source searched, a record was made for each relevant publication of whether it was retrieved by the search strategy used and whether it was available in the database but not retrieved. To account for multiple publications of the same study, a record was made of the relevant studies identified. The sensitivity, precision, and number needed to read were calculated as well as the minimum combination of sources to identify all the publications or studies.

Results: There were eighty-two publications (forty-nine studies) included in the systematic review. Only one article was available in a database searched but not retrieved by our search strategy. Science Citation Index (SCI) and EMBASE both achieved the highest sensitivity (62 percent), followed closely by MEDLINE/PubMED (56 percent). With the search strategies used, the minimum combination of sources needed to identify all the publications was SCI, EMBASE, CENTRAL, and either MEDLINE or PubMED, in addition to reference checking, contacting authors and an automated current awareness service. In relation to identifying all the relevant studies, the minimum combination of studies was similar with the exclusion of CENTRAL.

Conclusions: To identify all the relevant publications or studies included in this case study systematic review, several different sources needed to be searched.

Objectives: The need to consider the patient perspective in health technology assessments (HTA) has been widely recognized. In July 2012, the Finnish Medicines Agency (Fimea) published a national recommendation for integrating the patient perspective into the HTAs of pharmaceuticals. The aim of this study is to describe the development of the recommendation for integrating the patient perspective into the HTA process of pharmaceuticals in Finland.

Methods: The development of the recommendation was based on a review of international recommendations and experiences of patient and public involvement in HTA. The draft recommendation was tested in two focus group discussions (n = 7 patients) and three individual interviews among diabetes patients (type 1 or 2) using long-acting insulin treatment. The recommendation was open for public consultation in April 2012 and revised according to the comments received.

Results: Patients will be involved in multiple stages of Fimea's HTA process. The recommendation includes step-by-step instructions on how to assess the patient perspective. The main focus is on qualitative interviews, which will be conducted at the beginning of the assessments to gain information, particularly on patient preferences and values, including positive and negative outcomes important to patients and ethical and social aspects of the medicine's use.

Conclusions: The recommendation will act as a tool to integrate patients’ experiences, needs and preferences into Fimea's HTAs of pharmaceuticals.

Objectives: To compare cost effectiveness models for the first-line treatment of multiple myeloma, and explore the differences between the models’ structure, parameters, assumptions and results.

Methods: Three cost effectiveness models for the treatment of multiple myeloma, were compared that had been developed to inform resource allocation in the UK for the chemotherapy regimens bortezomib, melphalan and prednisolone (BMP); and melphalan, prednisolone and thalidomide (MPT) versus melphalan and prednisolone (MP). The models used alternative approaches and assumptions to estimate the overall survival and progression-free survival for each of the interventions. Through the use of sensitivity analyses, the most influential parameters and assumptions of each of the models were identified.

Results: The models developed by the manufacturers gave conflicting results, with each manufacturer favouring their drug. The differences between the model results were determined by two parameters: the hazard ratio for overall survival for MPT vs. MP and the cost of bortezomib.

Conclusions: Using models developed for assessing treatments for multiple myeloma we demonstrated that it was feasible to compare models, which then aided decision makers in making reimbursement decisions.

Background: Polyethylene (PE) wear is a major contributor to implant loosening following total knee arthroplasty (TKA). Advanced bearings are therefore required in TKA to reduce or eliminate wear-related loosening. A recently introduced bearing that combines oxidized zirconium with highly cross-linked PE has been shown to drastically reduce wear in in vitro settings, due to its improved tribological characteristics in both tibial and femoral components. However, there are no data on its cost-effectiveness.

Data and Methods: A Markov model was developed to assess the cost-effectiveness of this low-wear bearing from a German societal perspective. The model population was derived from a registry of 75,000 patients requiring TKA. The model allocated patients to receive either a low-wear or standard articulation, and followed them until death. Revisions and re-revision were included. Input values were derived from registry databases or from published reports in the literature. Threshold analysis and probabilistic sensitivity analysis were conducted to estimate model robustness.

Results: The low-wear articulation prevented 24 (2.4 percent) revisions and 8 (0.8 percent) re-revisions. The total incremental cost-effectiveness ratio (ICER) of the low-wear articulation was EUR 16,474 per quality-adjusted life-year (QALY). For patients aged less than 55 years, an ICER of EUR 653 per QALY was observed. For patients aged over 75 years, this value was EUR 83,300. Threshold and probabilistic sensitivity analysis indicated that these findings were reasonably robust.

Conclusion: Low-wear articulations may be considered cost-effective, although the cost effectiveness is age-dependent, with the cost per QALY being significantly lower for younger people than for older people.

Objectives: The aim of this study was to investigate originator pharmaceutical companies’ practices in relation to health technology assessment (HTA) and the views and perceptions of their executives on the importance of HTA in pricing and reimbursement of medicines in Greece.

Methods: A qualitative study was performed, using individual semi-structured interviews based on an interview schedule with open-ended questions. The target population was market access departments’ executives of originator pharmaceutical companies. Our target sample consisted of sixteen executives, of whom ten agreed to participate. Saturation point was reached after eight interviews. Data were audio recorded, transcribed verbatim, and analyzed using content analysis.

Results: Participants considered HTA as a very important complementary tool for decision making in health policy, particularly in the field of pharmaceuticals and medical devices. They believed that, in Greece, HTA could be institutionalized for the reimbursement mechanism of medicines under certain conditions relating to current health policy-making attitudes and conditions pertaining in the country. They considered that there are many constraints which must be overcome as well as opportunities to be exploited.

Conclusions: Decisions in pharmaceutical policy should be scientifically substantiated and HTA should be institutionalized primarily for reimbursement decisions. Development of guidelines for conducting pharmaco-economic evaluation, change in health policy goals, recording of cost and epidemiological data, and broader participation of all stakeholders in HTA decision-making processes are suggested as prerequisites for a successful implementation of HTA in Greece.

Objectives: A mapping exercise was undertaken to determine how HTA is being described and conducted across the International Network of Agencies for Health Technology Assessment (INAHTA), with the aim of harmonizing terminologies and approaches.

Methods: Three progressive surveys were undertaken. In 2010, INAHTA agencies were asked to provide details on all of their HTA products. In 2013, additional information was sought on key methodological characteristics of five of the most common HTA product types. Subsequently, final agreement was sought on three proposed product types.

Results: Forty-five HTA agencies responded to at least one of the surveys. In 2010, twenty-one agencies reported publishing over seventy named HTA products. Core domains associated with full HTA reports were reported by a third of agencies but were labeled differently, so products were classified according to product type (n = 17). Agencies producing short, tailored products increased between 2010 and 2013, with the publication of rapid reviews doubling from 33 percent to 66 percent. In 2013, half of the agencies adapted their common HTA products from documents produced by other agencies. A consensus (>70 percent) was achieved on definitions for HTA reports, mini-HTAs, and rapid reviews.

Conclusions: The product label for an HTA is not always indicative of its content. Terminology has, therefore, been agreed to make explicit the trade-off between rigor and timeliness in three common HTA product types. An INAHTA Product Type (IPT) Mark has been created to identify each of these. It is hoped this will further facilitate HTA adaptation between agencies and reduce duplication of effort.

Objectives: This study explores the awareness and the degree of acceptance of the idea of the medical technology cryonics—the freezing of a corpse to revive it in the future—among German citizens.

Methods: Data were collected on the basis of a representatively weighted online survey of 1,000 people aged between 16 and 69 years and resident in the Federal Republic of Germany.

Results: Forty-seven percent stated that they had already heard of cryonics; 22 percent could imagine having their bodies cryonized after their deaths. For 53 percent, participation in the latest technological developments which correlated with the approval of the conceivability of cryopreservation was important. The majority of the respondents were not skeptical or cautious about innovations in the medical field.

Conclusions: The study shows that cryonics is known and accepted to a certain extent. However, a large proportion of respondents did not believe that it was desirable to use medical technology to overcome death, and fundamentally rejected a post-mortal continuation of life.

Objectives: Licensing of, and coverage decisions on, new therapies should rely on evidence from patient-relevant endpoints such as overall survival (OS). Nevertheless, evidence from surrogate endpoints may also be useful, as it may not only expedite the regulatory approval of new therapies but also inform coverage decisions. It is, therefore, essential that candidate surrogate endpoints be properly validated. However, there is no consensus on statistical methods for such validation and on how the evidence thus derived should be applied by policy makers.

Methods: We review current statistical approaches to surrogate-endpoint validation based on meta-analysis in various advanced-tumor settings. We assessed the suitability of two surrogates (progression-free survival [PFS] and time-to-progression [TTP]) using three current validation frameworks: Elston and Taylor's framework, the German Institute of Quality and Efficiency in Health Care's (IQWiG) framework and the Biomarker-Surrogacy Evaluation Schema (BSES3).

Results: A wide variety of statistical methods have been used to assess surrogacy. The strength of the association between the two surrogates and OS was generally low. The level of evidence (observation-level versus treatment-level) available varied considerably by cancer type, by evaluation tools and was not always consistent even within one specific cancer type.

Conclusions: Not in all solid tumors the treatment-level association between PFS or TTP and OS has been investigated. According to IQWiG's framework, only PFS achieved acceptable evidence of surrogacy in metastatic colorectal and ovarian cancer treated with cytotoxic agents. Our study emphasizes the challenges of surrogate-endpoint validation and the importance of building consensus on the development of evaluation frameworks.

Objectives: This study describes the health technology assessment (HTA) framework introduced by Regione Lombardia to regulate the introduction of new technologies. The study outlines the process and dimensions adopted to prioritize, assess and appraise the requests of new technologies.

Methods: The HTA framework incorporates and adapts elements from the EUnetHTA Core Model and the EVIDEM framework. It includes dimensions, topics, and issues provided by EUnetHTA Core Model to collect data and process the assessment. Decision making is instead supported by the criteria and Multi-Criteria Decision Analysis technique from the EVIDEM consortium.

Results: The HTA framework moves along three process stages: (i) prioritization of requests, (ii) assessment of prioritized technology, (iii) appraisal of technology in support of decision making. Requests received by Regione Lombardia are first prioritized according to their relevance along eight dimensions (e.g., costs, efficiency and efficacy, organizational impact, safety). Evidence about the impacts of the prioritized technologies is then collected following the issues and topics provided by EUnetHTA Core Model. Finally, the Multi-Criteria Decision Analysis technique is used to appraise the novel technology and support Regione Lombardia decision making.

Conclusions: The VTS (Valutazione delle Tecnologie Sanitarie) framework has been successfully implemented at the end of 2011. From its inception, twenty-six technologies have been processed.

Objectives: In economic evaluations of healthcare technologies, situations arise where data are not randomized and numbers are small. For this reason, obtaining reliable cost estimates of such interventions may be difficult. This study explores two approaches in obtaining cost estimates for pregnant women screened for a fetal cardiac anomaly.

Methods: Two methods to reduce selection bias in health care: regression analyses and propensity scoring methods were applied to the total mean costs of pregnancy for women who received specialist cardiac advice by means of two referral modes: telemedicine and direct referral.

Results: The observed total mean costs of pregnancy were higher for the telemedicine group than the direct referral group (4,918 versus 4,311 GBP). The regression model found that referral mode was not a significant predictor of costs and the cost difference between the two groups was reduced from 607 to 94 GBP. After applying the various propensity score methods, the groups were balanced in terms of sizes and compositions; and again the cost differences between the two groups were smaller ranging from -62 (matching “by hand”) to 333 GBP (kernel matching).

Conclusions: Regression analyses and propensity scoring methods applied to the dataset may have increased the homogeneity and reduced the variance in the adjusted costs; that is, these methods have allowed the observed selection bias to be reduced. I believe that propensity scoring methods worked better for this dataset, because after matching the two groups were similar in terms of background characteristics and the adjusted cost differences were smaller.

Objectives: The aim of this study was to compare Bayesian methods with the standard methods that are used for evidence-based policy making.

Methods: We performed a Bayesian reanalysis of the data underlying a reimbursement advice by the Dutch National Health Insurance Board (CVZ) regarding the anti-diabetic drug exenatide (an alternative to insulin). We synthesized evidence from various sources that was available when the CVZ advice was drafted: expert opinion (as elicited from internists), experimental data (from direct comparison studies), and observational data. Subsequently, the original frequentist results and the results from the Bayesian reanalysis were compared in terms of outcomes and interpretations. These results were presented in a meeting with staff from CVZ, whose opinions about the usefulness of a Bayesian approach were assessed using a questionnaire.

Results: The Bayesian approach yields outcomes that summarize different pieces of evidence, which would have been difficult to obtain otherwise. Moreover, there are conceptual differences, and the Bayesian approach allows for determining probabilities of clinically relevant differences. The staff at CVZ were fairly positive with respect to the use of Bayesian methods, although practical barriers were also seen as important.

Conclusions: The Bayesian outcomes are different and could be more suited to the informational needs of policy makers. The response from staff at CVZ provides some support for this statement, but more research at the interface of science and policy is needed.