This Week in Evaluating Research
So, I’m a qualitative researcher, both as an academic proclivity and professionally. I get paid to talk to people about their experiences and interpret their responses in theoretical and social contexts. I get that not everyone is into qualitative research, but I find it particularly irritating when people claim that it’s by design less rigorous and less capable of producing solid, incisive and defensible conclusions. (If you’re one of those people, here’s a handy guide to critically analyzing qualitative research. You can do it!) It’s irritating because I’m also trained in quantitative methodology (thanks, Dev!), and I know that statistical analysis can be subjective and arbitrary too. We just trust numbers more.
Actually, I think we trust numbers a lot more. Which is why articles like this are very dangerous. From a study by George Washington University researchers, we discover, “Obesity puts a drag on the wallet as well as health, especially for women.”
Doctors have long known that medical bills are higher for the obese, but that’s only a portion of the real-life costs.Oh noes, I spent half of my earnings this year on being fat? Add in the half spent on rent, I apparently did not purchase any transportation, food, medications, clothing, entertainment, or books. Would that it were so.
George Washington University researchers added in things like employee sick days, lost productivity, even the need for extra gasoline — and found the annual cost of being obese is $4,879 for a woman and $2,646 for a man.
In order to make my comps (senior thesis, to you non-Carls) adviser proud, here is a quick guide, with examples from this helpfully bad study, to evaluating quantitative research.
Step One: Know your Biases
Almost any study that’s grant- or federally-funded, or is authorized by an Institutional Review Board (hint: this is almost all of them) requires researchers to disclose any potential conflicts of interest they may have regarding the material to be studied. It’s fairly obvious why this is important: if you’re being paid by Acme Vitamins, you might be compelled to structure your data analysis in a particular way to accentuate the benefits of Acme Vitamins while obscuring the fact that they may have dangerous side effects. Or, more subtly, you may choose a particular research question that casts your benefactor in a more positive light.
So, when we read that “the report was financed by one of the manufacturers of gastric banding, a type of obesity surgery,” we ought to give that a little bit of thought. The question, “What is the cost of being obese?” is not a value-neutral question. It assumes that there is a cost to being obese, and–as I’ll get to in a moment, assumes that that cost is the direct result of obesity itself. I would find it quite surprising to see a manufacturer of gastric banding financing a study and pitching its press release to AP for world-wide release that investigated the very serious, sometimes life-threatening side effects of weight-loss surgery.
That is, these people will make more money if they are able to paint an incomplete picture of the costs of obesity and the effects of weight loss surgery on those costs. I don’t meant to say that having a particular bias immediately discredits research. Every study has some orientation, whether it’s theoretical or outcomes-oriented, or financial. But it helps to shed some light on how data was parsed. In this case, it certainly helped me to parse how the research question was asked and what data were considered relevant or irrelevant to report.
The AP, furthermore, had a little editorializing to add as well: the omnipresent headless fatty picture to set the tone.
Step Two: Correlation does not equal causation
You’ve probably heard this statement a million times. I hope so, anyway. It’s the first thing you learn in statistics, and it’s probably the most important piece of knowledge to have when confronting any quantitative study. Relationships between variables are complex. Very rarely does one variable directly and singularly dictate another.
A major study published last year found medical spending averages $1,400 more a year for the obese than normal-weight people. Tuesday’s report added mostly work-related costs — things like sick days and disability claims — related to those health problems.
Of course, we all here know that the relationship between fat and health problems is much more complicated than that, and that we really don’t know the mechanism of association between obesity and the health problems that may occur more often in people who are fat. But even beyond that, are there other variables that might influence the cost of health care for people who are fat?
For those of you playing along at home, if you answered yes, give yourself one gold star. Because we also know that fat people experience discrimination in medical settings that might make them less likely to be seen frequently, allowing medical problems to advance without treatment, costing more in the long run, and perhaps contributing to the need for more sick days. People who are poor are also more likely to be obese, and also less likely to get frequent comprehensive and preventative care, because such care is unfairly allocated. For some people, weight is influenced by underlying medical conditions, like thyroid disorders, poly-cystic ovarian syndrome (and other endocrine and glandular disorders), and conditions that make movement painful (like lupus, arthritis, or MS). It may also be influenced by medications needed to control and treat illnesses not related to weight. That is, some people are fat because of expensive medical conditions, and their health care would be expensive whether they were fat or not.
So, does being overweight cause higher health care costs? Perhaps, but the study doesn’t determine or outline the mechanism of this causation. And we have plenty of evidence that suggests that discrimination against fat people (which I hope I don’t need to argue is not caused by being fat) affects how they seek care, and that weight is affected by underlying conditions that may require more (and more expensive) care. We don’t have in this study a supported statement of causation, and we’re definitely missing these confounding variables. Even though it’s politically expedient, assuming causation is lazy. Show your work!
Step Three: Look out for strawmen and other histrionics
“We’re paying a very high price as a society for obesity, and why don’t we think about it as a problem of enormous magnitude to our economy?” [Dr. Kevin Schulman, a professor of medicine and health economist at Duke University who wasn’t involved in the new report] asks. “We’re creating obesity and we need to do a man-on-the-moon effort to solve this before those poor kids in elementary school become diabetic middle-aged people.”Just count ’em up, folks! Won’t someone think of the children! Nobody takes this problem seriously! We’re ruining the economy! Cost to society! MAN ON THE MOON EFFORT!
Even though there is a growing contingent of people who are thinking more critically about the social bases of objections to fat people, I think it’s hardly true that people broadly don’t think of obesity as a “problem of enormous magnitude to our economy.” Without launching into full-on Foucault-times, in his concept of biopower, he demonstrates how these arguments about “our economy” (or “our society” or “our people” and so on) are constructed and employed to act upon and normalize the bodies of people we see as deficient or abnormal.
We see dollar signs and we see headless fatties, and miss the analytic step in the middle: How and why does obesity appear to cost us? What is the mechanism of causation? Who does it actually cost? To draw a parallel to my own research, I’m sure you’re quite familiar with public outcry over “Welfare queens,” who are costing us MILLIONS OF DOLLARS because they have children WE HAVE TO PAY FOR! In actuality, child-related tax breaks for the middle and upper classes are a substantially greater expense to the taxpayer than any welfare-related expenditures, but they seem worthy because we like wealthy, professional (usually White) people having babies. The assumptions of the reader are just as important as the assumptions of the authors, and commentators know that it doesn’t take much to stimulate the FAT IS WRONG lobe we’ve all been conditioned to develop. (That’s actual science. [Actually, it’s not.])
So, the next time you come across another study about how the fatties are ruining America–it will not be long, I promise–take a few seconds to examine how the author’s arguments are possible. What are their unstated assumptions, and how do those influence the questions they ask, the data they collect, and the conclusions they produce? What leaps in logic do they rely upon? And what social norms do they draw from or reinforce?
People take numbers really seriously; we really like the term “statistically significant.” But calculations are made in contexts, and understanding those contexts are what will make us better able, as activists and allies, to challenge the assumptions they create.