HTA principles need rigorous review

To the Editor:

In the article “Are key principles for improved health technology assessment supported and used by health technology assessment organizations?” by Neumann et al. (3), the authors set out to evaluate the quality of processes used by health technology assessment (HTA) programs based on a set of principles they published in this journal previously (2). We believe that portions of the current article, as they relate to our own organization, are misleading or inaccurate, and welcome this opportunity to correct the inaccuracies.

In our example, the authors list the Drug Effectiveness Review Project (DERP) with the Washington State Medicaid agency in combination as an HTA organization. This is inaccurate, as there are three separate entities involved. DERP is a collaboration of public organizations (eleven state Medicaid agencies and the Canadian Agency for Drugs Technology and Health), the Center for Evidence-based Policy, and the Evidence-based Practice Center (EPC) at Oregon Health & Science University; to this extent, Washington State Medicaid agency is one of the twelve current participants. DERP's mission is to conduct systematic reviews comparing the benefits and harms of pharmaceuticals to inform policy decisions. These reports synthesize evidence on the relative benefits and harms of drugs following internationally recognized standards, but do not make recommendations. While the participating organizations of DERP work together to make decisions about specific review topic selection and scoping, they use these reviews in varying ways, depending on how the report may work within the individual organization's constraints. In the case of Washington State, these systematic reviews are used to assist the state's Pharmacy and Therapeutics Committee and the Drug Utilization Review Board in making recommendations about the Preferred Drug List and the prior authorization program for Medicaid enrollees. The recommendations of the committee are made in public with full disclosure of the evidence on which they are based, and are based solely on the comparative benefits and harms of the drugs under review.

Neumann et al. applied their principles to what is referred to as ‘Washington State Medicaid/DERP’ processes combined and found them to be lacking in several areas, although it is not clear which processes (Washington State, DERP, or the EPC's) are assessed for each element. We disagree that the DERP reports do not support principles 8, 9, 11 and 14. DERP reports do address the uncertainty surrounding estimates, as well as issues of generalizability and transferability. DERP also seeks all available information, including unpublished information, and communicates the findings of their reports directly to the decision makers who will use it. Reading any of the thirty-nine DERP reports (www.ohsu.edu/drugeffectiveness) in detail would have clarified these issues.

In conducting assessments for this article, the authors selected the groups of organizations, collected information about their processes, and applied their principles in an ad hoc way. The authors describe their process as based on review of Web sites, personal knowledge of the organizations in some cases, and discussion among themselves. We acknowledge Neumann et al. were transparent regarding the relatively casual approach they used to determine how HTA organizations comply with the “key principles for improved conduct of HTA” they developed in 2008. The publication was also forthright in identifying that the endeavor was supported by an unrestricted grant from a pharmaceutical manufacturer. We believe, however, the Journal has erred in publishing what essentially amounts to findings from a nonsystematic survey of a group of HTA organizations without explicitly labeling it as “commentary.” This is very important because within days of its being published on line, industry lobbyists were characterizing the article as a serious research document and using it as a basis for criticizing technology assessment programs in both Oregon and Washington.

To be useful, efforts to define how quality in HTA will be measured must be based on methods that are at least as rigorous as those suggested by the authors for the HTA enterprise itself, including broad-based review and testing before finalization. The development of the GRADE criteria for grading the strength of recommendations was handled is such a way (1). We believe that these recommended key principles would benefit from a similar level of scrutiny and should not be widely adopted until a rigorous and independent process has been undertaken to assure their validity.