The Takshashila Survey on Scientific Bias

Recently, my colleague Pavan Srinath put out a post on testing whether someone “defers to scientific reason above and beyond ideologies”. In his post, he made three statements, which he said are all strongly backed by scientific evidence:

The core argument in climate change is that the earth’s surface warmed significantly in the 20th century due to human-linked emissions of greenhouse gases.

The argument with nuclear safety is that health risks from nuclear power generation, both chronic and acute, have been grossly exaggerated and that due to an obsession with nuclear safety for the past 6 decades, nuclear power is now safer than most other sources of energy.

The argument with genetically modified crops is that they are just as safe as other crops, both for growing and for consumption. Additionally, crop modification through targeted molecular biology techniques is in fact less genetically invasive than conventional hybridisation techniques.

All three arguments have overwhelming scientific evidence on their side, and the nature of the scientific debate is very different from the public and political discussions regarding the same.

If you were not ideologically biased and if you were scientifically aware, he said, you would be extremely likely to agree with all the three above statements. If, however, you were biased to the “left” you were likely to agree with the first statement but not with the last two. If biased to the “right”, you were likely to agree with the latter two and not with the first.

We decided to test these beliefs by putting out a survey. The survey had exactly three questions – the above three statements that Pavan mentioned in his blog, and the respondent was supposed to agree or disagree with these statements on a five-point Likert scale. The “sample” on which the survey was administered was biased – most respondents we believe were connected on Facebook or Twitter (the two avenues we used to publicize the survey) to someone in the Takshashila community. It is very likely that most of the respondents were educated urban upper-middle-class Indians (this is a guess; we didn’t ask for these data points in the survey itself).

142 people responded to the survey. Most of these responses came within a day of our putting out the survey. Here are the results of the survey:

Firstly, we will look at the individual responses to each of the three questions:

Source: Takshashila Survey
Source: Takshashila Survey

This shows that opinion in favour of global warming is fairly strong.

Source: Takshashila Survey
Source: Takshashila Survey

While a majority of the people believe that health risks from nuclear power have been exaggerated, the opinion is not as overwhelming as it is on the global warming front. There still exist a significant number of doubters of safety of nuclear energy.

 

Source: Takshashila Survey
Source: Takshashila Survey

When it comes to GM crops, however, public opinion is largely divided. As many people agree that GM crops are safe, as do people who believe they are unsafe.

Next, we will look at interactions. The next three graphs here show bilateral “heatmaps” of responses to the three questions. The greater the redness of a particular cell in this map, the greater the number of respondents who fall in that cell.

warmingnuclear

There seems to be a positive correlation between these two beliefs that are towards different ends of the political spectrum. Among people who agree that global warming exists, more people believe that nuclear power risks are exaggerated than otherwise.

warminggm

An interesting thing here is that extreme views on one issue are correlated with extreme views on another. Note that people who strongly agree on global warming are more likely to strongly agree or disagree with GM Food safety, while those who merely “agree” with global warming are more likely to simply “agree” or “disagree” with GM Food safety. Also notice the large mass of people who strongly agree with global warming but are neutral about the safety of GM Foods. This indicates that there isn’t as much debate and discourse on the safety of GM foods as there should be.

nucleargm

These are the two “right wing issues”. Notice that the top left and bottom right areas are almost empty. People who agree with one of these are more likely to agree with the other.

So how many of our respondents can be classified as being “scientifically aware” based on Pavan’s metrics? Given that Pavan states that someone who is scientifically aware should agree with all three statements, we will consider someone who has “agreed” or “strongly agreed” with all three statements as being “scientifically aware”. This number comes out to 27 out of 142 respondents or about 19%.

How many of our respondents are scientifically unaware? For this we will look at people who either disagree or strongly disagree with each of the above statements. There are only 3 people among those we surveyed who can be thus classified (and one of them has given his/her name as “Troll” so we may not take that seriously).

Then, again going back to Pavan’s definitions, how many left-wingers do we have? For this we will consider people who agree with the statement on global warming and disagree with the other two. There are 17 such people.  What about right wingers? These are people who disagree with the statement on global warming but agree with the other two. There are 7 of them. There are 63 respondents who have said that they are neutral on at least one of the three questions.

There is so much more one can do with these responses. I have anonymized the responses and put up the data here for your benefit. You are free to analyze it and draw your own conclusions. However, I would encourage you to share your conclusions with the larger community by leaving a comment on this post.

2 thoughts on “The Takshashila Survey on Scientific Bias”

  1. Hi Karthik,

    Since there is an open invitation, I thought I will share my thoughts on this data. Please feel free to comment on its merit.

    To begin, I think that the Lickert scale allows for biases, though unavoidable. Hence, while I guess the questionnaire were intentionally twisted( one aligned with public opinion and two negating it ) I do believe this psychologically restricts the choice of the respondents.

    I have analysed this on a different parameter – “conformance with public opinion/propaganda”.

    To start with the findings, here it goes :

    Global warming is the obvious popular choice where there is also consensus in opinion. Overall scores :
    Global Warming : 588
    GM Crops : 424
    Nuclear Energy : 372 (Kudankulam – God save you)

    Respondent’s Category : Total
    Whatevea’ 16%
    Lazy Bums 27%
    Next Please 3%
    Rebels 11%
    Rockstars 1%
    Believers 32%
    Smartasses 9%

    Nuclear Energy & GM Crops : This either means that the topics are not popular enough, or there are no strong opinions on them.

    Definitions of respondents as follows :
    Whatevea’ : Gave scores right down the middle. What it means: Central Tendency Bias
    Lazy Bums : Stuck to one side of the lickert. What it means: Acquiescence Bias (AB)
    Next Please : Stuck in the middle overall, but few strong opinions. What it means: Don’t Know Cant Say
    Rebels : Don’t agree with public opinion and consistently so. What it means: Non-conformance/ Scientific Bias
    Rockstars : Don’t agree with public opinion, sometimes. What it means: Non-conformance/ Scientific Bias – AB
    Believers : Walk down the popularity path. What it means: Public Opinion Bias
    Smartasses : Know a few scientific facts here and there. What it means: Public Opinion Bias – AB

    Method :
    1. I first normalised data to provide similar scores on public opinion. i,e. it is assumed someone who did not agree on GM crops and gave it a 2, would give it a 4 and agree the negative.

    2. I ignored scores of people who had – a. central tendency bias (didnt bother reading the questions and marked somewhere around 3 for everything) & b. Acquiescence bias (didnt bother understanding the question and were too lazy moving away from one side of the scale) as this causes false positives and negatives.

    3. I measured alignment with extremes by comparing totals of the normalized scores.

    4. gave some fancy names to the types of respondents.

    So, there it is. Only two strong groups emerge (apart from people who spoil the show).
    Conformance with public opinion : 41 % (perceived moderate to left wing)
    Non-conformists ( probably atheists ) : 12 %. (most probably right wing )
    The rest dont know, dont know enough to make a strong opinion or just passed their time filling forms.

    Greenpeace jai ho ! Your employees have actually worked !

    Sorry, I cant post tables or graphs.

Put Comment