Berenson, Pronovost & Krumholz

There is growing interest in relying more on outcome measures and less on process measures, since outcome measures better reflect what patients and providers are interested in. Yet establishing valid outcome measures poses substantial challenges—including the need to riskadjust results to account for patients’ baseline health status and risk factors, assure data validity, recognize surveillance bias, and use sufficiently large sample sizes to permit correct inferences about performance. We believe the operational challenges of moving to producing accurate and reliable outcome measures, though daunting, are worth the effort to overcome.

Patients, payers, policy-makers, and providers all care about the end results of care—not the technical approaches that providers may adopt to achieve desired outcomes, and may well vary across different organizations. Public reporting and rewards for outcomes rather than processes of care should cause provider organizations to engage in broader approaches to quality improvement activities, ideally relying on rapid-learning through root cause analysis and teamwork rather than taking on a few conveniently available process measures that are actionable but often explain little of the variation in outcomes that exemplifies U.S. health care.

However, given the inherent limitations of administrative data, which are used primarily for payment purposes, and even clinical information in electronic health records (EHRs), consideration should be given to developing a national, standardized system for outcome reporting [1]. A new outcome reporting system would not be simple or inexpensive, but current data systems may simply be insufficient to support accurate reporting of outcomes. An example is the National Health Care Safety Network system for reporting health care infections [2].

Alternatively, EHR vendors could modify their products to allow them to be used to calculate validated quality measures. By standardizing which structured data elements they include in their products and the metadata they use to describe these fields, vendors could allow for the calculation of validated quality measures, such as those collected by National Surgical Quality Improvement Program and the Society of Thoracic Surgeons. Once collected, clinical data would need to be evaluated for validity and quality. Prioritizing which measures require highly valid data and which do not may also help. It may be that for rare events, less accurate, although substantially less costly, administrative data would suffice, while for more common events and conditions, it would be more costeffective to collect clinical data from clinical records. However, the quality of EHR data is also being questioned [3].

An emphasis on measurement of outcomes, rather than care processes, need not ignore the contribution of specific processes that are associated with achieving better outcomes. In fact, achieving high reliability on process measures could be viewed as an internal tactic that providers might adopt as part of a comprehensive approach to achieve good outcomes, rather than as an end in itself [4]. Professional societies or governmental agencies could maintain a library of process measures that providers could select from to audit their own performance. But here the distinction between measures for quality improvement and for public reporting becomes important: publicly reported measures could emphasize the outcomes of interest, while measures used internally for quality improvement could emphasize the care processes that an organization is working on performing better.

A relatively small number of process measures, especially if linked with intermediate outcome measures, could serve as excellent measures for public reporting, mitigating the risks for surveillance bias, although the public would need to be educated about their clinical implications. Process measures (e.g., obtaining hemoglobin A1C levels in diabetics and properly taken blood pressure readings) could be linked to intermediate outcome measures (e.g., hemoglobin A1C level and blood pressure). The use of such measures in public reporting efforts could also educate patients and consumers about these important parameters of clinical care. However, caution should be used in using intermediary outcome measures, as demonstrated by the recent experience in which intensive treatment of patients to lower their hemoglobin A1C was recently shown not to be associated with the favorable outcomes expected. NCQA and others developed process measures favoring achievement of hemoglobin A1C levels below 7 percent. Yet, it was precisely this level that failed to show improved outcomes in three recent randomized trials, ultimately leading to the abandonment of that process measure by NCQA. In some clinical areas, process measures that assess the rate at which specific harmful medical errors occur also hold appeal. For harms that are almost entirely preventable—some of which are referred to as “never events”—risk adjustment and other statistical concerns should be unimportant. A promising avenue for supporting a movement toward reliance on outcomes is greater use of patient-reported outcomes, which are derived using tools that measure what patients are able to do and how they feel through surveys. A wide variety of patient-level instruments to measure patient-reported outcomes related to physical, mental, and social well-being have been used in clinical research, such as within the National Institutes of Health’s Patient-Reported Outcomes Measurement Information System. Extending this research application for purposes of accountability and performance improvement would require additional work to address methodological and data challenges [5].

Robert A. Berenson, MD is an institute fellow at the Urban Institute.

Peter J. Pronovost, MD, PhD is the director of the Armstrong Institute for Patient Safety and Quality at Johns Hopkins, as well as Johns Hopkins Medicine’s senior vice president for patient safety and quality.

Harlan M. Krumholz, MD, is the director of the Yale-New Haven Hospital Center for Outcomes Research and Evaluation, director of the Robert Wood Johnson Foundation Clinical Scholars program at Yale University, and the Harold H. Hines, Jr. professor of cardiology, investigative medicine, and public health.

The authors thank Lawrence Casalino, MD, PhD, chief of the Division of Outcomes and Effectiveness Research and an associate professor at Weill Cornell Medical College, and Andrea Ducas, MPH and Anne Weiss, MPP of the Robert Wood Johnson Foundation for their helpful comments on this paper. This research was funded by theRobert Wood Johnson Foundation, where the report was originally published.

Notes

1. Pronovost PJ, Miller M and Wachter RM. “The GAAP in Quality Measurement and Reporting.” Journal of the American Medical Association, 298(15):1800-1802, 2007.

2.  Farmer SA, Black B and Bonow RO. “Tension Between Quality Measurement, Public Quality Reporting, and Pay for Performance.” Journal of the American Medical Association, 309(4):349-350, 2013; Introduction to Trigger Tools for Identifying Adverse Events. Cambridge, MA: Institute for Healthcare Improvement, http://www.ihi.org/knowledge/Pages/Tools/IntrotoTriggerToolsforIdentifyingAEs.aspx (accessed April 2013).

3. Kern LM, Malhotra S, Barrón Y, et al. “Accuracy of Electronically Reported ‘Meaningful Use’ Clinical Quality Measures: A Cross-Sectional Study.” Annals of Internal Medicine, 158(2):77-83, 2013.

4. Porter ME. “What Is Value in Health Care?” New England Journal of Medicine, 363(26): 2477-83, 2010.

5. Patient-Reported Outcomes in Performance Measurement. Washington: National Quality Forum,www.qualityforum.org/Projects/n-r/PatientReported_Outcomes/Patient-Reported_Outcomes.aspx.