How confident should we be..? A psychologist answers our question about his research
How confident can we be about the results of scientific studies? It’s not a question the media often ask when writing about climate change. Research which hasn’t been even finished yet has for example been given as conclusive proof that climate change isn’t happening, or, alternatively, is causing us to shrink. The author of a recent study, which tests two competing theories to explain how people perceive risk about climate change, discusses the issue of confidence on his blog, in response to a question we sent him.
As we wrote on Monday, the paper concludes that social groupings are a much better measure for predicting whether or not people will express concern about climate change than an individual’s level of scientific literacy.
We got in touch with lead author, Dan Kahan, to ask about one of the effects he noticed – that with increasing science literacy, people actually became more divided on the risks posed by climate change. We asked:
“From the figure (Fig.2) [the polarising effect] appears to be quite subtle, albeit in the opposite direction to that which was predicted by the SCT thesis. It would be great if you could identify to what extent/how confident we can be to say that increasing numeracy and literacy polarises risk perception about climate change”
The purpose of the research
First, he describes the theories he wanted to test in the study:
“As you know, our study investigated two hypotheses: the science comprehension thesis (SCT), which attributes public conflict over climate change to deficits in science comprehension; and the cultural cognition thesis (CCT), which asserts that conflict over climate change is a consequence of the unconscious tendency of individuals to fit the beliefs about risk to positions that dominate in their group, and which in its strongest form would say that this tendency will be reinforced or magnified by grater science comprehension, which can be used to promote such fitting.”
CCT theory identifies two distinct groups within the public: egalitarian communitarians who perceive the risk to be higher, and hierarchical individualists who perceive the risk to be lower – so according to this theory social grouping will be the most significant indicator of how concerned people are about climate change. In contrast, SCT theory suggest individuals, regardless of their social grouping, should be more concerned about climate change risk the more science literate they are.
If SCT theory is correct, then as science literacy increases, both skeptics and non-skeptics should perceive the risk to be higher. The gap in risk perception between the two groups should not get larger. But the study shows the gap does get larger: people’s views become more polarised, and that can be explained by CCT:
“The study is consistent with CCT and furnishes modest evidence that CCT in its strongest form is correct. That position would predict that cultural polarization will be greater among individuals with the greatest science comprehension. The results fit that hypothesis.”
Confidence in conclusions
Given that the results give only “modest evidence” for CCT, we wanted to know how confident we can be in drawing these conclusions. To answer our question, Dan says there is:
“only modest evidence [for CCT thesis] mainly because of the design of the study. It’s observational –correlational — only. Observed correlations that fit a hypothesis are strong in proportion to which they rule out other explanations. Maybe something else is going on that causes both increased science comprehension & increased polarization in certain people.”
There are also other reasons to be measured when making claims based on one study:
“All study results are provisional. That’s in the nature of science. Valid studies give you more evidence than you otherwise would have had to believe something. They never “settle” the issue.”
“It is never sensible (it is a misundersanding of the nature of empirical proof) to say, “this study proves this” or “this study doesn’t necessary prove that …” etc.”
Building on the research
But the research doesn’t stop here. Professor Kahan says further experiments can build on the evidence for the CCT thesis and help to rule out other explanations:
“The only way to tell [if something else is going on] is through (well designed) experiments. We are conducting some now.”
“…if the source of the effect is CCT, it should be easy to produce much more dramatic effects through properly designed experiments. So rather than try to extract more information from the effect size about how confident or not to be in the strong CCT position, it makes sense to do experiments. Again that’s what we are now doing.”
Understanding the design of this experiment helps us to know what the study can and can’t tell us. By comparing scientific literacy and numeracy to risk perception, the research can only identify trends in how risk perception changes. It can’t identify a causal link between the two.
“Nothing in our study suggests that making people more science literate or numerate causes polarization. If CCT is correct, there is something about climate change (and certain other issues) that makes people try to maximize the fit between their beliefs and positions that predominate within their groups, which themselves are impelled into opposing stances on certain facts. That thing is the cause in the practical, normative sense.”
These findings, combined with those from future research can greatly add to our understanding on what divides public perception of climate change risk. This study alone can’t explicitly ‘prove’ either theory, it can simply add more evidence to our understanding of risk perception. That too is in the nature of science.
You can read Professor Dan Kahan’s blog, which also contains references to other papers, in full on the Cultural Cognition blog .