Trick or Treatment—The Undeniable Facts about Alternative Medicine (Electronic book text)

Because poorer-quality trials are more likely to produce misleading results, researchers have developed techniques for assessing quality and weeding out those trials that should not be taken seriously. For example, the Oxford quality scoring system, developed in 1996 by Alejandro Jadad and his colleagues at Oxford University, can be used to give a trial a score between 0 points (very poor) and 5 points (rigorous). The system works by awarding or deducting points according to what appears in the published version of the trial. So if the research paper confirms that there was randomization of patients then it receives a point, yet this point can be deducted if the randomization appears to have been inadequate. Or the trial can score a point if the paper describes how the researchers dealt with the data from patients who dropped out from the trial. If the researchers have thought about this in detail and bothered to describe it in their research paper, then it is a good indication of their overall level of rigour.

Critics pointed out that sixty-eight out of the eighty-nine trials in Linde’s meta-analysis scored only 3 or less on the Oxford scale, which meant that three-quarters of the trials were substandard. Moreover, critics pointed out that restricting the meta-analysis to the higher-quality trials (4 or 5 points) drastically reduced the apparent efficacy of homeopathy. In fact, the conclusion of the twenty-one higher-quality trials was that homeopathy offered either a small benefit for patients or no benefit at all. Despite the amount of data available from these twenty-one trials, it was still not possible to distinguish between these two possibilities.

In time, Linde and his colleagues agreed that their critics had a valid point, and in 1999 they published a follow-up paper that re-examined the same data with a special emphasis on the quality of the individual trials. Linde wrote: ‘We conclude that in the study set investigated, there was clear evidence that studies with better methodological quality tended to yield less positive results.’ Then, referring back to the original meta-analysis, he stressed: ‘It seems, therefore, likely that our meta-analysis at least over-estimated the effects of homeopathic treatments.’

Linde’s original 1997 paper had supported homeopathy, yet his revised 1999 paper was much more equivocal. His re-analysis of his own meta-analysis obviously disappointed the alternative medicine community, yet it was also frustrating for the medical establishment. Everyone was dissatisfied because Linde was neither able to claim that homeopathy was effective, nor was he able to dismiss it as a mere placebo.

Despite the lack of clear-cut evidence in either direction, the public was increasingly turning towards homeopathy, either consulting practitioners or buying over-the-counter remedies. This gave researchers a renewed sense of urgency to test the therapy via larger, more rigorously conducted trials. Hence, homeopathy was subjected to a much higher level of scrutiny from the late 1990s onwards.

This eventually prompted Dr Aijing Shang and his colleagues at the University of Berne, Switzerland, to undertake a fresh meta-analysis of all the trials published up to January 2003. The medical research group at Berne, which is led by Professor Mathias Eggers, has a world-wide reputation for excellence and the Swiss government had provided the team with adequate funding for a fully rigorous meta-analysis. Hopes were high that Shang would at last be able to deliver a reliable conclusion. Indeed, after two centuries of bitter dispute between homeopaths and mainstream medics, Shang’s meta-analysis was destined to decide, at last, who was right and who was wrong.

Shang was ruthless in his demand for quality, which meant that his meta-analysis included only those trials with large numbers of participants, decent blinding and proper randomization. In the end, he was left with only eight homeopathy trials. After studying the data from these eight trials — the best available trials on homeopathy — his meta-analysis reached its momentous conclusion. On average, homeopathy was only very marginally more effective than placebo. So, did this tiny marginal average benefit suggest that homeopathy actually healed patients?

Before answering this question, it is important to realize that the results of every scientific analysis are always associated with a level of uncertainty. For example, analysing the age of the Earth gives a result of 4,550 million years, and the error on this age is give or take 30 million years. The uncertainty associated with Shang’s estimate on the efficacy of homeopathy was such that his conclusion was wholly compatible with the judgement that homeopathy acted as nothing more than a placebo. In fact, the most sensible interpretation of the meta-analysis was that homeopathy was indeed nothing more than a placebo.