OK, this is getting out of hand. Kahan’s latest, widely celebrated publication is just wrong.
say goodnight to the dream that education, journalism, scientific evidence, media literacy or reason can provide the tools and information that people need in order to make good decisions. It turns out that in the public realm, a lack of information isn’t the real problem. The hurdle is how our minds work, no matter how smart we think we are. We want to believe we’re rational, but reason turns out to be the ex post facto way we rationalize what our emotions already want to believe.
Yes, indeed this is what Kahan wants us to take home from his paper. But his paper doesn’t actually show anything of the sort! Oddly, his position is somewhat vindicated by the reaction to his paper, as the media themselves display motivated reasoning.
Kaplan’s summary is a good start:
In Kahan’s experiment, some people were asked to interpret a table of numbers about whether a skin cream reduced rashes, and some people were asked to interpret a different table — containing the same numbers — about whether a law banning private citizens from carrying concealed handguns reduced crime. Kahan found that when the numbers in the table conflicted with people’s positions on gun control, they couldn’t do the math right, though they could when the subject was skin cream.
This much is true enough. But then Kaplan goes on to say that
The bleakest finding was that the more advanced that people’s math skills were, the more likely it was that their political views, whether liberal or conservative, made them less able to solve the math problem.
I would like to convince you that, while literally true, it’s unimportant. It’s like saying that flashing lights are more likely to make a competent juggler drop the the ball than they are to make someone like me drop the ball. That is because I cannot juggle; your attempts to distract me cannot work on a task that I already fail at.
It’s worse than that, though. Because Kahan is not measuring what he claims he is measuring. Not at all.
Going back to Kahan’s experiment, where “some people were asked to interpret a table of numbers about whether a skin cream reduced rashes, and some people were asked to interpret a different table — containing the same numbers — about whether a law banning private citizens from carrying concealed handguns reduced crime”, the “table of numbers” was set up specifically to be confusing. The table showed ratio of results A and not-A in cases of treatments X and not-X; thus four cases. The question was whether X was indicated to be effective in causing A by these data. Thus there were four entries in the 2×2 table:
To do this correctly you have to “solve the word problem” yielding a requirement to compare the ratios (X,A/X,not-A) and (not-X,A/not-X,not-A). If the former is larger than the latter, then the treatment can be called “effective”. But there is a trick – the sample sizes for X and not-X are different.
In a typical well-designed experiment, one could just compare (X,A) and (not-X,A). But the data presented are not like that. You have to do the ratios in your head. To make matters worse, the numbers are selected to make the arithmetic less than obvious to most people.
The “correct answer” is obtained by the less numerate far less than would be indicated by chance in the case where there is no cultural bias. By chance half would be right. But the problem is set up so that less than 30% of the less numerate get the correct answer.
And even in those cases, one has to presume that the “correct answer” may not have been obtained by correct reasoning.
Now you can set up the cultural bias so that culture pulls for the right answer or pulls for the wrong answer, and both treatments are tried. Sure enough, the left-wingers uniformly are pulled toward the answer expected by (contemporary American) left-wing ideology and vice-versa for the right wingers.
Kahan’s big result is that cultural bias is more effective in drawing an incorrect answer among the more numerate. That is what is making the headlines.
There’s no surprise that cultural bias reduces reasoning. Of course it does! So do loud noises, sexual stimuli, flashing lights, being hit over the head with a mallet. The fact that people can be distracted is not news.
The purported surprise is that a larger percentage of the more competent people were distracted.
But remember, it is a trick question! The success rate of the less competent is worse than chance! And some of that success is accidental!
In other words, a cultural distraction is more effective in reducing the competence of people who can do the task than that of those who are not competent to do the task in the first place. Stop the presses.
Consider the comparable experiment with people who can juggle on one side and people who can’t juggle on the other. Set up a pair of juggling tests, one under ordinary conditions and the other in the presence of sudden, random, loud noises. The preformance of the adept will decline. The performance of those who cannot do the task under the best of circumstances will stay the same.
Can we therefore conclude that “highly dextrous people are more subject to distraction than clumsy ones”? Well, sort of, but it doesn’t really tell us very much of interest.
Can we conclude that “they applied their dexterity to the task of dropping the ball”? In this case that doesn’t even make any sense. Why should the analogous reason be relevant in Kahan’s?
Kahan defines his experiment as testing between what he calls “SCT” (his renaming of the “deficit model”) and his preferred “ICT”:
Whereas SCT attributes conflicts over decision-relevant science to deficits in science comprehension, ICT sees the public’s otherwise intact capacity to comprehend decision-relevant science as disabled by cultural and political conflict.
Again, there is the “deficit model”. It is a complete straw man. Nobody anywhere argues against the proposition that decision-relevant science can be disabled by cultural and political conflict.
Those of us who are cast as “defending the deficit model” are simply arguing against the proposition that once such conflict is put in place (often by malice) that all attempts at explanatory exposition are doomed to failure and should cease.
But here, I am arguing that Kahan completely misinterprets his data.
More numerate individuals are benefitted from forming identity-congruent beliefs just as much as less numerate individuals are, and harmed just as much from forming identity-noncongruent beliefs. But more numerate individuals have a cognitive ability that lower numeracy ones do not. ICT predicts that more numerate individuals will use that ability opportunistically in a manner geared to promoting their interest in forming and persisting in identity-protective beliefs.
No. The data do not support this at all.
The paper can be downloaded freely. Here is the link again:
Have a look at Figure 7, which admittedly is (ironically enough) a bit of a cognitive challenge.
What does this tell us?
There are sixteen curves, testing all possibilities of four conditions:
a) treatment is effective or ineffective according to made-up data presented to subjects (dashed vs solid)
b) ideological bias applies (top vs bottom graphs)
c) subject is more like a “democrat” or like a “republican” (red vs blue)
d) subject is mathematically more/less competent than average (left vs right graphs)
Kahan draws our attention to “increased polarization” of the more numerate population. The separation between the two blue curves on the lower right is wider than the separation on the lower left. The separation between the red curves is similarly widened. That is, the effectiveness of ideology in distracting the more numerate is stronger than the effectiveness of ideology in distracting the less numerate.
So much is clear from the data.
What doesn’t follow is Kahan’s conclusion that “more numerate individuals will use that ability opportunistically in a manner geared to promoting their interest in forming and persisting in identity-protective beliefs”.
I’m not saying such a thing is impossible. It just fails Occam’s razor quite horribly. There is a much simpler explanation – the less numerate people never had the required competence in the first place. Thus the ideological confound did them less damage. They had already failed to begin with.
Rather than looking horizontally across the bottom of the graph, compare vertically, within the competency groups.
You will see that in the non-ideological case, the less numerate group preformed worse than chance would indicate. In short, it is a trick question.
Of the small group that got it “right”, we can presume that a good section of those got it right by chance, either by compensating errors or guesswork. The small remainder can be considered experimental noise, and we can define the group on the left as basically incapable of solving the problem presented to them.
In this group, the republicans defer to ideology and split their vote in the ideological condition. Interestingly, the democrats do not, and indeed show a hint of improvement in their capacity. On the whole, though, we still see worse-than-chance in all conditions except the one where the facts accord with the republican bias.
On the other hand, in the competent group, we see significant decay of the actual real competence of the democrats presented with the ideological confound, and a huge decay in the actual real competence of the republicans. However, in no case was the performance WORSE than that of the less competent group.
We also see a bump in the performance of republicans when ideology favors the correct answer. This is the “erroneously correct” or “right for the wrong reasons” crowd.
But that’s the point here. The horizontal axis on the graphs is labelled “probability of correct interpretation of the data” but that isn’t what is being measured! What’s being measured is, instead, probability of answer in agreement with correct interpretation. By adding a distraction, the performance of the competent deteriorates, but the performance of the less competent improves!
This latter effect should be discounted. The graph on the lower left is essentially noise.
Kahan wants us to believe that the more competent group is using mental resources to fool themselves. Since this has a lot of similarity to subjects in my wife’s PhD thesis about divided attention under hypnosis, I am not unfamiliar with the idea. It is conceivable.
But he hasn’t shown us that at all.
He claims he has shown us that because the effect is smaller among the less mathematically competent. Leaving aside the interesting question of whether mathematical competence enhances or suppresses capacity for self-delusion, his evidence is misinterpreted.
His argument is based on the idea that ideology can disrupt a competence that doesn’t even exist. It conflates “getting the right answer” with “correct interpretation”. The data tell us nothing about “correct interpretation”, despite the claims.
People can get the test question wrong for multiple reasons including:
1) They can fall for the arithmetic trap
2) They can pay no attention to the numbers whatsoever and vote at random
3) They can get the reasoning wrong and come up with the wrong answer
4) In the ideological challenge condition, their bias can sway them to contrive complex delusional arguments in defense of their preferred outcome (Kahan’s explanation)
5) In the ideological challenge condition, their bias can sway them to pay less attention and settle for the politically congruent answer
6) In the ideologicalchallenge condition, their bias and their perception of the social context (“demand characteristics”) of the experiment can cause them to ignore the data altogether.
In the case of the right answer there is similarly:
1′) Get the reasoning right
2′) pay no attention to the numbers whatsoever and vote at random
3′) They can get the reasoning wrong and come up with the right answer anyway
4′) In the ideological confirmation condition, their bias can sway them work harder at the problem
5′) In the ideological confirmation condition, their bias can sway them to pay less attention and settle for the politically congruent answer
6′) In the ideological confirmation condition, their bias and their perception of the social context (“demand characteristics”) of the experiment can cause them to ignore the data altogether.
Jumping to the conclusion that phenomena 4 and/or 4′ are key to the shift seems woefully premature. It is clear that the proportions are certainly different in different groups and treatments. The construction of the experiment does not isolate those phenomena from the others.
To the contrary, if this tested ideological interference in both groups, both groups would have had to perform better than chance. In the present case, the performance of the weaker group improved in the ideologically weighted conditions, moving them closer to (and in one favorable case beyond) pure chance.
To really justify #4 you’d need a case where the more both groups performed significantly worse, the more competent worst of all. Conceivably you could construct such a case. But even then all you could say is that such an effect measurably exists, not that it dominates in the real world. This particular example doesn’t even show that.
It is also notable that in all cases numeracy improved performance all else equal, though in one case (Republicans under ideological stress) not much and perhaps not significantly.
Kahan’s claim is that
1) Ideology interferes with reason MORE among those with better reasoning capacity
2) This argues against the proposition that conflicts over decision-relevant science are due in large measure to deficits in science comprehension
While I have argued here that point 1 is not supported at all, it is also notable that point 1, even if demonstrated does not lead to point 2. Skill and comprehension are mangled in reaching point 2. But since point 1 is not supported, point 2 is moot on present evidence.
This all said, nobody disputes that ideology can interfere with reason. That it would not is just a straw man position. The dispute here is over his claim that he has shown that technical competence makes it worse.
The reason this is all important is not because it amounts to the celebration of weak research. This wouldn’t be the first time such a thing happened.
It’s important because people are using it to suggest that once a problem is politicized, nobody has any obligation to provide balanced evidence. That this offers a clear path for the malign manipulation of the press somehow is not raised. In this case the press isn’t just trumpeting some marginal evidence to get eyeballs. In this case they are looking for absolution. Ironically, their embrace of Kahan’s work has all the signs of motivated reasoning.