Get together and do it again.
Two classic songs from my youth had the title "Do it again". One, by the Beach Boys, has become a staple on oldies' adult rock radio stations, and the other, by Steely Dan, can be regularly heard on smooth jazz outlets. I don't think that Brian Wilson, Walter Becker or Donald Fagen were thinking about science when they wrote these songs, but they always remind me of the importance of replication and reproducibility in the process of scientific research.
Anomalous or unexpected results will always be with us while ever scientific research is being done. It may seem strange to say that unexpected results are expected but that's just a quirk of the English language. The research projects which I undertook while studying cognitive psychology usually contained the dreaded "p<.05" somewhere in a discussion of the results. I know that some physicists ridicule this and other similar findings in the social sciences and medicine and hold it up as evidence that these things aren't real sciences, but probabilities like this in results merely indicate that we are dealing with things which are variable or difficult to measure precisely and recognise that the results might not be right. It can sometimes be hard to explain to people that it simply means that if a large number of similar trials were conducted, we would expect this result more than 95% of the time. It is this uncertainty which forces us to repeat experiments, sometimes many times and preferably by different researchers, to see whether the probability of error holds up.
One of the problems with pseudoscience research is that the researchers are prepared to accept the first set of favourable results that come along and to claim that the theories have been proven correct. A classic example of this was the purported confirmation by Margaret Ennis and others of the work of Jacques Benveniste which apparently showed that the effect of an active agent could survive dilution beyond the point when any active ingredient is left in the mixture. Ennis claimed to be highly skeptical of the idea of homeopathy, and this is used as further evidence to support that what she found must be correct as confirmation bias can be ruled out. There were obvious flaws in methodologies used by both Benveniste and Ennis and when better controlled and managed experiments were carried out the magic effect disappeared. Homeopaths claim that this does not show that there is any problem with the research because conditions were not replicated exactly. Scientists respond that exact replication using the faulty methods of the original research would actually prove nothing, because one of the reasons for replication is to attempt to remove possible areas of error.
In psychic research the classic method is to keep going until you find what looks like evidence for what you are trying to prove, and then stop. I was declared the person with the strongest psychic powers in a room once because I correctly predicted seven coin tosses in a row. To a researcher in the paranormal this might be all the evidence necessary to state that I had some kind of predictive power. The real fact was that starting with about 120 people in the room and eliminating people whenever they guessed incorrectly the most likely outcome would be for someone to get six or seven correct guesses in a row. This is an experiment which can be repeated with any sufficiently large group of people and almost always produce a similar result, but doesn't say anything about the psychic or paranormal powers of the last man standing.
The ability to replicate results by itself is not sufficient evidence to accept anomalous results without further question or investigation. Each attempted replication should try to remove possible ways that previous results might have been mistaken. This is particularly important when investigating things which seem to be at the limits of possibility.
Repeatability of experiments and findings is something which has long been of interest to philosophers of science. It is essentially what David Hume was saying when discussing miracles – a miracle is something which cannot be repeated because it defies the laws of nature. Charles Broad gave this a name in 1949 when he used the term Basic Limiting Principles, essentially the laws of nature which limit what is possible.
When skeptics criticise pseudoscience research they often try to avoid a priori statements of what is possible and demand that further research be done to show that some phenomenon does or does not exist. Perhaps it would be better to not even bother to do research into the possibility of the impossible.
To return to popular music, one of the early hits for the Eagles was "Take it to the limit". We should certainly do that, but perhaps we don't need to go any further.
This article was published as the Naked Skeptic column in the June 2015 edition of Australasian Science
