Influential RAND researcher Soren Mattke had this to say in support of Al Lewis and Vik Khanna’s latest post on the Wellness story “Would the Real Professor Katherine Baicker Please Stand Up?”
“Gentlemen. Great post. Like you, I am disappointed that researchers of the caliber of Kate Baicker and David Cutler do not respond to the mounting debate about their paper. They should defend or disown their work rather than hope that the debate goes away.
In my mind, their paper is a product typical of high-end academic research. Two brilliant professors spot a gap in the evidence on a hot policy topic and decide to go after it. But the actual work gets done by a graduate student in his cubicle without windows or guidance, and then hastily published.
Then the problem arises that the paper becomes hugely influential and people start having a closer look. For our paper on the PepsiCo program, we reviewed in detail the seven publications that Baicker and colleagues called “high quality evidence”. We found that five of those analyzed programs that operated over 20 years ago and most of them had severe methodologic flaws. (John P. Caloyeras, Hangsheng Liu, Ellen Exum, Megan Broderick and Soeren Mattke. Managing Manifest Diseases, But Not Health Risks, Saved PepsiCo Money Over Seven Years. Health Affairs, 33, no.1 (2014):124-131)
Unfortunately, many defenders of the industry continue to take the Baicker paper at face value, while closely scrutinizing or ignoring more nuanced and scientifically sound findings.
So I herewith support your motion!
Categories: Uncategorized
Also don’t forget that EVERY SINGLE ONE of the studies in Baicker’s article was done at a time when you were allowed to talk to employees about family history. That is a major risk factor — the biggest, other than smoking — and is no longer accessible. Right there, the entire Baicker article is rendered useless as a predictor of future wellness results. (This of course is in addition to the rest of her junk science.)
The Baicker article that appeared in Health Affairs in 2010 entitled “Workplace Wellness Programs Can Generate Savings” categorized the workplace wellness literature into 3 levels of rigor. I took the studies in the most rigorous group and examined them closer. Here is what I concluded:
Naydeck study
The strength of this study is it presented cost data for all major health care categories, which all of the others don’t. One could quibble that it underestimated the cost of the wellness program by attributing the cost of the fitness center across all 10,000 employees, not just the 1,800 participants. As recognized by the authors, a significant weakness of the study is the fact that only the participants who used the fitness center realized the net cost savings. Since this was a non-randomized study, this raises serious concerns about selection bias – that the more self-motivated and health-conscious employees were essentially the “participants.”
My other concern about this study is the real savings seem to come from reduced inpatient costs. There is an unexplained tripling of inpatient costs from 2001 to 2002, which raises doubts as to the completeness of their data collection. Maybe that had to do with the 3-year look-back period they wrote about. Perhaps later years had more complete data. The participant group ultimately experienced a moderate reduction in inpatient costs, while the nonparticipants had a massive increase in inpatient costs with a slight reduction at the end. However, actual costs are not reported, only beta values for cost curves.
Based on other cost studies, my guess is over the first three years they identified some of the chronic disease patients with the highest utilization and gradually improved their care to result in less utilization. Notice how much larger the beta values are for diabetes than participation status, particularly in Model 1. It would have been nice if the researchers reported more detail on exactly what changed with inpatient costs over the 5 years.
Aldana study
This was a case with matched control study that did not match well. The participants’ costs were much higher than the non-participants’ costs at the beginning of the study. Cost data were incomplete. No costs were attributed to the mobile wellness program. No pharmacy costs were included. No explanation is given on possible reasons inpatient costs decreased in the participant group.
Ozminkowski study
This was a pre-/post-intervention quasi-experimental design. The introduction talks about spending “. . . several million dollars evaluating the program, examining, among other outcomes, its effects on health and risk factors, health care expenditures, absenteeism, and employee attitudes.” However, in the results section (Table 3), no costs for implementing the wellness program are included. This is additionally striking, as an incentive of $500 was provided for employees who participated, yet the estimated savings after four years of $225 does not include this up front cost. Also, no pharmacy costs are reported. No explanation is given for the estimated decline in inpatient costs.
Bly study
This is an observational study of employees who participated in a wellness program compared to those who did not. Results were adjusted for a few measureable variables. This study did not include program implementation costs, pharmacy costs, or outpatient costs. The only significant difference in inpatient costs between the groups is a huge sudden increase in inpatient costs for Group 3 (the sites with no wellness program) in the last year of the study.
Leigh study
This was a randomized controlled trial of retirees, not current employees. Costs are marginally captured at best. There is no report of implementation costs. The methods section has a long disclaimer about the limitations of the claims data. The results sections states “Direct and total costs were omitted but implicitly captured by hospital days, physician visits, and sick days.” Apparently no actual costs were measured. Pharmacy and laboratory costs were not measured. In Table 2, the largest supposed cost components are indirect costs, which are not defined in the methods section, or anywhere else I could find.
The other big problem I have with this study is that knowing the mountain of studies showing how hard it is to change behavior, it is highly unlikely that a “. . . low-cost health promotion program . . .” would have substantial economic or health benefits.
Fries study
This was a randomized quasi-experimental trial of mailed health promotion material. For reasons not stated, the groups were significantly imbalanced: 29,000 in the intervention group and 1,500 in the control group. This study did not randomize well either, with significant differences in the baseline claims rates. There are many methodological difficulties encountered in the claims cost analysis, which are described in the results section that I won’t repeat here. The study does not report which costs are included in the claims data. Intervention costs were not included.
The same comment about the limits of low-intensity behavior change interventions applies here.
Shi study
This was a quasi-experimental study of four different intensities of intervention. The study is not clear on how subjects were connected to groups. The introduction states they were assigned. The discussion states there was self-selection. The uneven distribution of participants in each group suggests self-selection had a large role in the distribution between the different intervention groups. Costs were not directly measured, but estimated from hospital days, doctor visits, and sick days only. No pharmacy or lab costs were included. Intervention costs were separately reported. The cost of lost employee time to participate in behavior change classes was not included.
Synthesis
First of all, I acknowledge that a hyper-controlled RCT of different wellness approaches would be difficult to carry out. I don’t mean to be hypercritical of the hard work of these researchers. However, I also don’t want to see our national resources wasted on interventions that are not proven effective. My synthesis of these studies cross-referenced with many other sources from the medical literature is as follows:
• These studies are severely limited by their sloppy accounting – they simply do not capture all the relevant costs to assess the cost-effective of worksite wellness programs.
• The lack of pharmacy cost data is particularly troubling. In the cost-effectiveness studies of cholesterol treatment (statins), high blood pressure treatment, and even smoking cessation interventions, the cost-effectiveness ratios are all positive (i.e., the intervention costs more than the alternative). In all these studies, the cost of medication and associated physician visits and lab monitoring overwhelms any savings in reduced hospitalizations and ER visits. Therefore, I believe this is the source of the disconnect between these worksite wellness studies and the cost-effectiveness studies. The worksite studies capture some inpatient savings, but ignore the increased outpatient treatment costs to achieve those savings.
• These studies are severely limited by selection bias. Many of these studies only looked at the possible cost savings of the people who actually mailed back the questionnaire, for example. The studies were initially advertised as being random in some way, but ultimately they categorized groups by other behavioral differences that confound any possible cost savings, primarily an increased overall health consciousness in the intervention groups.
• Most of these studies are severely limited by not including intervention costs. This is like writing a business plan based on the estimated sales price of a product, and not including the cost of the raw material to make the product.
• The most rigorous study, the Naybeck study, essentially concluded that getting people to use a fitness center may reduce health care costs. I don’t disagree with that. Perhaps a more comprehensive understanding of these studies should lead employers to focus more on improving the aerobic capacity of their employees, as opposed to interventions such as disease screening.
• Based on recent comprehensive reviews of the cost-effectiveness of disease management programs, another possible explanation for some of the success for reducing inpatient costs was to identify a few employees with poorly controlled chronic diseases who were consistent high utilizers of health care resources. As shown in the disease management reviews, implementation costs are high and rarely matched with decreased costs. The net cost of disease management programs is only beneficial in the most extreme subset of patients with severe disease who are frequent resource utilizers at baseline.
Informative post that puts things in perspective – thanks! Seems like it’s time for the Wellness industry to let go of past old, flawed research and figure out how to gauge the impact of well-designed wellness programs on tough-to-measure factors like productivity and morale.
Wow! this should be mandatory reading for every workplace wellness professional and every business leader that has ever talked with one! – thanks – Jon
Just once — ONCE — I’d like to see a response from one of the true believers defending their position. When we do get a response, it’s always a non-response. You have noted that when caught in an impossible position, their response is always: “Oh, we really didn’t mean that. We meant something else altogether.” And lo and behold, that’s exactly what they do time after time when they do actually respond.
Congressional hearings are long overdue. Workers are being harmed through overdiagnosis and overtreatment, money is passing under the table to brokers and consultants to pitch vendors, lies are being told to governmental bodies.
After Kate Baicker finsihes with the arduous task of ignoring this one, it’s Ron Goetzel’s turn to ignore another one: http://insurancethoughtleadership.com/11-questions-for-ron-goetzel-on-wellness/#sthash.8fAOG7ap.C25g0avi.dpbs