Publication in the peer-reviewed literature
Medical quacks and pseudoscientists don't like being told that they would have more credibility if their work was published in peer-reviewed journals. Often they will attack the peer review process itself and try to pretend that because it is not perfect it is not useful. Of course it is not perfect, because it is an invention and construction of humans not gods, but it is still better than the alternative of being able to say, claim and publish anything at all. All that this imperfection means, however, is that you can be a bit more confident when you read something in a peer-reviewed journal because you know that more than one person has read the paper before publication. Errors can still get through, though, as can be shown by The Lancet's publication of Dr Andrew Wakefield's nonsensical vaccine fiction and Social Text being fooled by Alan Sokal's postmodernist hoax. (The case of Nature publishing Dr Jacques Benveniste's memory of water paper was a special case, because the editor was trying to make a point about bad science.) Sometimes the flaw in the research just doesn't get seen by the reviewers, and I saw a case of this a few years back.
When I was studying perception at university we had a guest lecturer who told us about his latest research. He couldn't wait to tell us all about his amazing discovery, and was very proud that his paper had passed all the checks and reviews and was about to be published in a prestigious journal. He had shown that the sense of smell diminishes with age, and that older people could not smell as well as young people could. The experimental method had been to expose people of various ages to the smell of broccoli and ask them to identify it. Older people were much less able to identify the smell, so he claimed that this showed that they could not smell as well.
Any questions? I was the first, and I said something like: "My grandfather owned fruit and vegetable shops, my uncle ran a wholesale vegetable distribution business in the largest farm produce market in Australia, my parents ran fruit and vegetable shops and for the first five years of my life I lived over one of the shops. I never saw broccoli until I was about 15 years old. Is it possible that the older folk couldn't identify broccoli not because they couldn't smell it but because it was a smell that they had not experienced when young and therefore simply could not recognise"?
He had never thought of this possibility! The result was amazing. He actually crumbled in front of the class. It looked like his face was going to fall off. All he could think of was the letters to the editor in the following issue of the journal which was about to publish his work.
In another example of not thinking out the research properly (although this one was caught before much time had been wasted) I was talking to someone who was working on a Master's thesis based on the discriminatory hiring practices of the university. The problem seemed to be that while 55% of undergraduates were women, only 11% of the tenured staff were women. I pointed out that undergraduates were not the pool from which academic staff were hired. The proportion of women dropped only slightly (to between 45% and 50%) for people doing Masters degrees, but only 10% of doctoral candidates were women. As all academic staff were expected to hold doctorates it looked like the university was doing better than the average. My suggestion was to research ways that women in their late twenties with young children could be encouraged and assisted to continue their education and that this would be a much better than any form of affirmative action. She seemed to understand what I was saying, but you can never tell with ideologues.
This article was first published in July, 2005. In the spirit of recycling, a version was published on the Yahoo! 7 News Blog on May 19, 2010
