Mormons And WEIRDos: How Psychological Science Gets Fundamentally Skewed

Can a study with a high proportion of subjects from a minority religious group be considered applicable to the general population? It would be easy to argue that it cannot. If you want to make general statements about human behavior, you have to look at a pool of people across a more general demographic distribution, right?

As it happens, much of what we think we “know” about the workings of the human mind (that isn’t based on dissection) is based on close examinations of a small, privileged slice of humanity that’s a historical oddity, when measured against the history of the human race. Think about this: Can a psychology study based solely on tests of American college students really be applied to working class Americans, or farmers in the developing world -- or mothers in 18th-century China?

Give Me That Old Time Religious Affiliation Disclosure

In late October, a group of Brigham Young researchers released a paper examining how texting affects relationships in the Journal of Couple and Relationship Therapy (they found a positive text sends good vibes to both parties, but arguing over texting is a bad sign). Media outlets like the Atlantic were quick to note that about a third of the study’s 276 participants were Mormon -- casting doubt on whether the results could be applied to the general population of Americans, where only 1.7 percent self-identify as Mormon.

Lead author Lori Schade says that she and her colleagues were aware from the start that they would be working with a heavily Mormon (and heterosexual) pool of subjects.

“We’re always sensitive to the fact that that could bias our studies, so we do a lot of statistical analyses to compare means across religions,” Schade said. “We want to make sure the groups look similar.”

Shade pointed out that in general, Mormons tend to look not that much different -- statistically -- from mainstream Protestants.

“Typically, what we find is that Mormons aren’t very different from other populations unless it’s a highly specific moral issue -- something like premarital sex or drinking,” Schade said.

Still, Schade thinks that it’s important for her team, as part of an LDS-affiliated university, to be transparent about demographics and to make an effort to recruit a diverse pool of study subjects. Even so, Schade acknowledges their results do have to be interpreted within a demographic context. Mormons place a high value on keeping marriages together, so behaviors that might result in divorce in a couple of atheists or Lutherans may just lower the quality of a marriage rather than break it up.

The media likely wouldn’t have picked up on the demographics of the texting study if Schade and colleagues hadn’t made an effort to disclose the religious affiliation of their participants. This isn’t standard operating procedure in the rest of the literature -- most researchers tend to disclose only those demographic factors deemed immediately relevant to the question at hand.

“If you're doing your research at a Mormon college, [religion] is all the more obvious as a demographic factor to assess (especially when examining issues pertaining to intimacy),” New York University psychologist Susan Andersen, who studies interpersonal relationships, wrote in an email. “Such things affect results only as a function of the domain of inquiry.”

But when can a researcher safely assume that the religion of her study participants is unimportant? Some scientists argue passionately that their colleagues are too quick to assume that things like culture and religion (or lack thereof) are irrelevant.

WEIRD Science

A typical rite of passage for young men of the Etoro people of New Guinea involves fellating an older man; by ingesting the masculine virtues of an elder through his semen, he is initiated into manhood. It is very unlikely that any researcher would attempt make broad statements about the nature of masculinity based on the customs of the Etoro. But they are very likely to concoct universal laws of the human mind based on another tribe, one with rituals that would seem equally strange to an outside observer: American college students.

University of British Columbia psychologist Joseph Henrich and two colleagues lambasted a reliance on demographically skewed pools of study subjects in a 2010 paper for the journal Behavioral and Brain Sciences. When psychology research is drawn almost entirely from what they’ve dubbed “WEIRD” populations -- people from Western, Educated, Industrialized, Rich and Democratic societies -- that makes the results, well, kind of weird.

“Psychologists tend to treat one group of undergraduates as representative of humans,” Henrich wrote in an email. “Textbooks [that] refer to ‘people’ (‘people have the endowment effect’)” -- basically, the concept that we value the things we own more than identical things we don’t own -- “are based entirely on western undergrads.”

A 2008 analysis by Clark University researcher Jeffrey Arnett found that more than two-thirds of subjects in studies published in top psychology journals came from the U.S., and 96 percent hailed from Western industrialized nations that account for only 12 percent of the world’s population.

Despite this reliance on a very unique subpopulation of people, most developmental psychologists do not routinely control for the socio-demographics of their subjects, Henrich said.

“Developmental psychologists will trot out claims for the ages when various cognitive abilities develop based on the children who live around universities,” Henrich said. “The discipline seems to have a tacit agreement that this is okay (I think because doing the real work is too hard and rather inconvenient).”

Psychology and behavioral research has also largely ignored religion until very recently, according to Henrich. In the 1990s, he noted, you couldn’t even find "religion" listed in the index of a social psychology textbook.

“We see this as part of the WEIRD problem,” Henrich said. “Psychology was created by WEIRD people with WEIRD interests (like happiness and stereotyping), so things like religion, food and kinship got left out. Arguably, food, religion and kinship are three of the most important things in the lives of non-WEIRD people, but until recently there were no sustained research programs on these topics.”

The problem of trying to generalize from a thin slice of the population isn’t just found in psychology. In medicine, for example, black Americans are rarely included in clinical trials. The attempted explanations for this are legion. It may be that some African-Americans have a lower level of trust for doctors thanks to a history of medical racism (like the unethical Tuskegee experiment that allowed poor black sharecroppers in Alabama to die of syphilis without treatment, in order to study the progression of the disease). One study found that the hassle of finding transportation and taking time off from work might make it harder for lower-income African-Americans to participate in clinical trials. Another found that physicians may not be reaching out to African-American patients as effectively as they could be.

Whatever the reason, the racial skewing in trials for certain kinds of diseases may result in medicines that aren’t equally effective for all races. Just last month, researchers found that the two HPV vaccines on the market do not target the strains of the sexually transmitted virus that are more common in black women.

In that context, Schade and her colleagues are more shocking for their willingness to admit how likely it is that their results are not universal. Other psychologists do believe that there is a common kind of humanity that can be inspected by psychological science. In one 2009 paper, Utrecht University researchers Wolfgang Stroebe and Bernard Nijstad -- critiquing Arnett’s 2008 analysis -- insisted that studying any subpopulation could yield universal laws, as long as one controlled for the cultural variables that might skew the results.

But this stance assumes that the researcher is perfectly suited to understand what variations within human culture are universal. The evidence suggests otherwise, as Arnett wrote in his own critique of Stroebe and Nijstad’s critique [PDF].

For instance, Arnett says, American psychologists once floated a theory, based on studies of American adolescents, that “distancing” between parents and teenagers may have some evolutionary basis. But this theory wholly ignores trends in countries like Indonesia, Brazil and India, where adolescents typically report being closer to their parents than to their friends.

“Given human cultural diversity, how can it be justified to assume that a theory developed on the basis of research on a tiny proportion of the world’s population can ‘apply to all of humanity?'” Arnett wrote. “This is certainly a strange way to conduct science.”