Impact Stories
November 2, 2022 – Liberalism

In the social sciences in particular, intellectual openness is paramount

In the social sciences in particular, intellectual openness is paramount

By Ilana Redstone

Many instructors of social science research methods caution students that correlation doesn’t equal causation. In other words, just because two factors are related to one another doesn’t mean that one caused the other. As important as this reminder is, it doesn’t go far enough. Rather than passively bearing this instruction in mind, students can learn to actively seek multiple possible explanations

Thinking through all possible explanations is a way of resisting the pull of causal claims. Especially in the social sciences, such claims are nearly equal in their immense appeal and their extreme difficulty to substantiate. Avoiding drawing them requires a willingness to be open, intellectually humble, and, as we’ll see, cautious with the conclusions we draw. 

When it comes to social science research, our inability to make causal claims is due, in part, to the way data are collected. Social scientists often rely on studies that solicit information about respondents’ lives. This information might include how much school a person completed, whether they are married, whether they regularly seek health care, or how many friends they have. 

The challenge is that the answers partly depend on a series of factors that we can’t usually see or measure. How much education someone completes, for instance, is affected by a combination of components that the individual controls (like how much she studies or whether she attends class) and others that she doesn’t (like how well-funded her public school is). 

The presence of unmeasurable factors makes it difficult to pinpoint causes. To take another example, some research suggests that marriage is good for health, especially for men. That claim makes sense if we think, for instance, that wives might pressure their husbands to get regular check-ups or if married couples have better health insurance. But we don’t know if these factors cause better health. It’s possible that healthier men are simply more likely to marry in the first place. In other words, we cannot assume that because a man marries he will be healthier than he would have been if he had never made it to the altar. The consequences of the potentially misplaced causal claims fall on a spectrum from mild to alarming.

On the more mild end, the authors of a 2019 study concluded that more frequent baths were associated with (and likely led to) a lower risk of cardiovascular disease (CVD). A subsequent article in the Harvard Business Review rightly noted the possibility that regular bath takers may have lower stress levels and more free time to relax, which also could explain their lower rates of CVD. In this case, being encouraged to take a few more baths, even if it turns out to have no effect on CVD, probably isn’t going to hurt anyone. But what about a situation where the stakes are higher? 

Most people need little convincing that the economic effect of becoming a mother in high school is an issue that we should, as a society, care about. After all, becoming a mother during high school can make it difficult to graduate, attend college, and work a full schedule while securing childcare. In these explanations, the act of becoming a mother while in high school causes lower earnings later in life. 

Yet, in making this causal claim, we’re assuming that, in the absence of having a baby in high school, the earnings of young women who became mothers in high school would have looked the same as the earnings of women who didn’t become mothers in high school. But, what if, on average, young women who become mothers in high school are different from young women who don’t become mothers in high school in some other way that could also affect their earnings later on? What if, for instance, high-school aged women who become mothers are, on average, less oriented towards school and studying than high-school aged women who don’t become mothers? In other words, what if our assumption doesn’t hold?

If that’s the case, their earnings might have been lower than the comparison group even if they hadn’t had a child in high school. And, if so, having a child in high school may not be the (only) thing that caused their lower earnings. From an analytic standpoint, this is a mess to tease apart. 

This problem is further complicated by the fact that the second explanation — that the young women who became mothers in high school would have had lower earnings anyway — understandably sits poorly with many people, including me. It sounds callous and unsympathetic and it could lead people and policymakers to assume these young women are undeserving of help. (As in: Why should we help them? They weren’t going to succeed anyway.) 

The ethics of such a justification to do nothing are disturbing — especially since they would be built on an assumption that itself may or may not be true. 

I’m not saying that the first conclusion is necessarily wrong. There may well be a powerful causal effect of having a baby in high school on earnings later in life. What I am saying is that, for the reasons described above, it’s difficult to know for sure. And, in this case, the costs of drawing the wrong conclusions are likely more than just a few extra baths. 

Knowledge production is cumulative, meaning researchers build off of one another’s work. This is reflected in the tagline on the Google Scholar page that reads: “Stand on the shoulders of giants.” The cumulative nature of knowledge production means that overstated claims can be compounded over time and reproduced fairly easily. And they can lead governments, institutions, and people to make decisions that, if they had full information, they wouldn’t necessarily make.

The good news is that we aren’t helpless in what we can do about this problem. 

Initiatives like the University of Pennsylvania’s Adversarial Collaboration Project encourage scholars with different ideological viewpoints to work together — hopefully offsetting our tendency to gravitate towards singular explanations. The Project acknowledges that some of the most prominent social scientific findings have proven to be unreliable, and that interventions based on that knowledge are often built on shaky ground. To solve this problem, it invites scholars to allow their methods, materials, and hypotheses to be scrutinized by their peers throughout the research process.

The upshot is that the act of seeking multiple explanations doesn’t require an admission of being wrong — it simply requires a willingness to be a little less sure that the explanation we landed on is the right one. 

Even in the absence of collaborations between ideological opponents, scholars can be careful in how we talk about research. We can temper our language in a way that suggests a link, but that isn’t conclusive. We can moderate our claims. 

We can actively seek multiple possible explanations. 

Ilana Redstone is an associate professor of sociology at the University of Illinois at Urbana-Champaign. She is also the faculty director of the Mill Institute at UATX. 

This viewpoint is part of an ongoing series that discusses the principles that unlock human potential. Find links to the entire series here.