Eppur si muove

Things I Think About, by Peter Bowditch

Experimenting on skeptics

SkeptiCampAt SkeptiCamp Sydney on May 26 I conducted a small experiment to show that even people motivated to attend a skeptical conference can have their actions influenced by things they read. It was not intended to be a scientific experiment so no statistical analysis of the results has been or will be attempted. It was merely designed to demonstrate principles. It was done as part of a talk titled “We’re so skeptical we can’t be fooled”.

Each person in the audience was given a piece of paper. On one side was an instruction about turning the page over before being asked to do so. Half the forms had the instruction “Don’t turn over until asked to do so” and the other half said “Please don’t turn over until asked to do so”. The hypothesis was that the addition of the word “Please” would reduce the probability that the subjects would ignore the instruction. About a dozen people admitted to turning over before being asked to do so, but none of these people had the “Please” version. We can conclude from this that skeptics are more likely to do what you want them to do if you ask politely. The principle being tested here is known in skeptic circles as “Don’t be a dick” and there is no reason to suggest that it doesn’t apply to the wider population at large.

The main part of the experiment tested the psychological principle known as “Anchoring bias”. This states that when people are asked to estimate something their guess will be influenced by information provided at the time. Told that a number is incorrect, the average estimate will be higher in a group shown a high number than in a group getting a lower hint. These were the four statements, together with the numbers shown to the two groups:

High Low
The number of people receiving Newstart unemployment benefits in Australia is 3,475,321 41,293
The population of West Wyalong is 73,283 803
There are X kilometres of track in the Sydney CityRail system 11,345 253
The average consumption of marijuana by high school students in South Australia is X grams per year 4,286 2

Here are the results, with the average estimate for each group shown together with the best estimate currently available for the correct value.

Experimental   group High Low Actual
The number of people receiving Newstart unemployment benefits in Australia is 713,252 351,568 329,500
The population of West Wyalong is 32,919 6,827 2,946
There are X kilometres of track in the Sydney CityRail system 24,512 2,020 1,595
The average consumption of marijuana by high school students in South Australia is X grams per year 791 349

There was no useful estimate available for marijuana consumption in SA high schools, although figures do show a significant reduction in marijuana use in the age group over the last few years. Five subjects in the “High” group estimated use at 10 kilograms per year. As this would represent heavy individual usage and would be almost inconceivable as an average these were exclude from the results. Exclusion of these outliers did not affect the direction of the overall results with the “High” group still having a higher average estimate than that for subjects shown a low figure.

So on all four statements the people shown a higher target provided, on average, higher estimates with the opposite happening for the group shown lower clues. If this proves anything at all it is that committed skeptics are little different to non-skeptics in the way that they are influenced by information provided to them in the absence of corroborating data. I would hope that the difference is that skeptics would not take single, unsupported numbers at face value and would ask for additional evidence to back up what they are told. Isn’t this what makes us skeptics, after all?

References:

Anchoring bias in decision-making

Jobseekers receiving Newstart Allowance

West Wyalong Local Stats

CityRail Facts and Stats

Statistics on Illicit Drug Use in South Australia

You can follow me on Twitter here.

Comments are currently closed.

3 thoughts on “Experimenting on skeptics

  • skeptonomicon says:

    I’ve never really liked Anchoring bias experiments, and it does not seam to me to be a logical fallacy. The skeptics followed the most logical course. You asked them to answer a question of which they had only one piece of information, a wrong guess by someone else who may or may not have had any idea of the answer. Perhaps a good skeptic would have refused to answer the question, but you specifically asked them to provide this information. Given that that had no first hand information, the most logical thing to do is to use the known wrong guess and adjust it slightly higher or lower. Given the setup, this should yield the most likely chance of getting a correct answer.

    • 1. It isn’t a logical fallacy.

      2. The whole point was to show that skeptics are no different to non-skeptics when provided with insufficient information. As I said at the end, the difference is that skeptics don’t (or shouldn’t) accept things without additional confirming data.

      3. It was a fun thing to do at an event like SkeptiCamp.

      • skeptonomicon says:

        It sounds like it was a fun experiment, and likely taught a valuable lesson. I think you are still missing my point. I disagree “that skeptics don’t (or shouldn’t) accept things without additional confirming data”. Instead, skeptics should always use the best available data, and keep in mind the quality of the data, which is what they did.