The Australian Report -evidence of serious procedural and scientific misconduct in producing this report

The Australian report
In March 2015, the Australian National Health and Medical Research Council (NHMRC) published an Information Paper on homeopathy, commonly referred to as ‘The Australian Report’.1
This document concludes that “…there are no health conditions for which there is reliable evidence that homeopathy is effective”.
This report triggered headlines around the world suggesting NHMRC had found that homeopathy doesn’t work for any condition.3
Australian Report key facts
An extensive detailed investigation by Gerry Dendrinos of the Australian Homeopathic Association (AHA) into NHMRC’s conduct, combined with an in-depth scientific analysis of the report by HRI, revealed evidence of serious procedural and scientific misconduct in producing this report:

• NHMRC did the homeopathy review twice, producing two reports, one in July 2012 and the one released to the public in March 2015.
• The existence of the first report has never been disclosed to the public – it was only discovered by AHA through Freedom of Information (FOI) requests.
• NHMRC say they rejected the first report because it was poor quality despite it being undertaken by a reputable scientist and author of NHMRC’s own guidelines on how to conduct evidence reviews.
• FOI requests have revealed that a member of NHMRC’s expert committee overseeing the review process – Professor Fred Mendelsohn – confirmed the first review to be high quality saying,  “I am impressed by the rigor, thoroughness and systematic approach given to this evaluation of the published reviews of efficacy and side effects of homeopathy [….] Overall, a lot of excellent work has gone into this review and the results are presented in a systematic, unbiased and convincing manner.” 
• NHMRC said the results of the report published in 2015 were based on a “rigorous assessment of over 1800 studies”. In fact results were based on only 176 studies.
• NHMRC used a method that has never been used in any other review, before or since. NHMRC decided that for trials to be ‘reliable’ they had to have at least 150 participants and reach an unusually high threshold for quality. This is despite the fact that NHMRC itself routinely conducts studies with less than 150 participants.
• These unprecedented and arbitrary rules meant the results of 171 of the trials were completely disregarded as being ‘unreliable’ leaving only 5 trials NHMRC considered to be ‘reliable’. As they assessed all 5 of these trials as negative, this explains how NHMRC could conclude that there was no ‘reliable’ evidence.
• Professor Peter Brooks, Chair of the NHMRC committee that conducted the 2015 review, initially failed to declare that he was a member of the anti-homeopathy lobby group ‘Friends of Science in Medicine’.
• In violation of NHMRC’s own guidelines there was not one homeopathy expert on the committee.

 
Complaint submitted to Commonwealth Ombudsman
In August 2016 HRI’s in-depth scientific analysis was used as part of a submission of complaint to the Commonwealth Ombudsman brought by Complementary Medicines Australia, Australian Homoeopathic Association and Australian Traditional Medicine Society.
Download the Executive Summary of the Ombudsman submission here.
 
Rachel Roberts, Chief Executive of HRI says “NHMRC’s review is just bad science. Decision-makers and the scientific community rely on these kinds of reports and need to trust their accuracy. This is not about anyone’s personal opinion as to whether homeopathy works or not. It is about the importance of evidence being reported objectively, whatever it says, and the NHMRC did not do that.”
As the complaint is ongoing, our full analysis – some 60 pages – cannot be shared as yet, but HRI’s data provided details demonstrating the following scientific failures by NHMRC which necessitate retraction of the Australian Report:

• Use of an inappropriate scientific method
• Failure to use standardised, accepted methods
• Failure to obtain sufficiently accurate data to perform a meaningful review
• Failure to conduct an effective preliminary and public consultation
• Significant post-hoc changes to the research protocol
• Impact of NHMRC’s unusual method on the review results
• Further evidence of bias and misreporting
• Poor reporting – lack of clarity, inconsistencies and errors
• Evidence that this was a case of deliberate bias, not scientific error.

Dr Alex Tournier, Executive Director of HRI, explains: “The inaccuracies in NHMRC’s report are so extreme that we decided to work with the AHA to conduct a thorough investigation to fully uncover exactly what went on.”
The complainants are now waiting to hear back from the Ombudsman regarding their submission. As NHMRC’s inaccurate Homeopathy Review has had a significant impact on the field of homeopathy research worldwide, HRI will share any news regarding the complaint as the case progresses.
Roberts says: “The public has a right to know that there are high quality studies showing homeopathy works for some medical conditions, such as hay fever, sinusitis and diarrhoea in children – information that was lost only due to NHMRC’s mishandling of the evidence. If the evidence on conventional medicine was treated this way there would be an outcry – and rightly so. NHMRC’s job was to accurately summarise the body of evidence for homeopathy for the public, a task in which they categorically failed.”
The missing first report
NHMRC’s investigation into Homeopathy ran from 2010 to 2015. NHMRC initially worked with an external contractor – from April 2012-August 2012 – to produce a review of the evidence on Homeopathy to inform the Australian public.
The report produced was called ‘A Systematic Review of the Evidence on the Effectiveness of Homeopathy’.
This review, paid for by Australian tax payers, was never made public and NHMRC continue to refuse to release it, despite repeated Freedom of Information requests.
After terminating the contract with the first review team, a second external contractor – OptumInsight – was hired to do the Homeopathy review again from December 2012-March 2015.
HRI is not alone in criticising the accuracy of NHMRC’s findings
FOI requests have brought to light that two independent experts also raised concerns over the conclusions of the 2015 report during peer review, prior to final publication. The Australasian Cochrane Centre commented that for some conditions, “…. ‘no reliable evidence’ does not seem an accurate reflection of the body of evidence”; a second expert felt “uncertain of the definitive nature of the Report’s conclusions”. NHMRC chose not to act on this feedback and did not amend their conclusions.
The real story behind the headlines
The Information Paper is designed for the general public and aims to provide a summary of the findings of a review of systematic reviews, carried out by NHMRC to assess the evidence base for effectiveness of homeopathy in humans.2
Confusing ‘lack of evidence of effect’ with ‘evidence of a lack of effect’
The Report’s conclusion that there is ‘no reliable evidence’ that homeopathy works, has been widely misunderstood, with people believing it meant that the NHMRC found that homeopathy does not work for the conditions tested, which would be a completely different result.
This misunderstanding triggered widespread media coverage, propagating the inaccurate story that the NHMRC found homeopathy to be no better than placebo for all the conditions.
In fact, NHMRC concluded that homeopathy worked no better than placebo for only 13 of the 61 health conditions they investigated.
HRI’s in-depth scientific analysis of the Australian Report revealed multiple examples of bias and misreporting which explain how NHMRC arrived at such a definitive and negative position, at odds with the conclusions of the majority of other more academically rigorous systematic reviews and meta-analyses on homeopathy.
Most crucially, NHMRC’s findings hinge primarily on their definition of reliable evidence: for a trial to be deemed ‘reliable’ it had to have at least 150 participants and a quality score of 5/5 on the Jadad scale (or equivalent on other scales). Trials that failed to meet either of these criteria were dismissed as being of ‘insufficient quality and/or size to warrant further consideration of their findings’.
Setting such a high quality threshold is very unusual, but the N=150 minimum sample size criterion is arbitrary, without scientific justification, and unprecedented in evidence reviews.
Out of 176 individual studies the NHMRC included in the homeopathy review, only 5 trials met their definition of ‘reliable‘, none of which, according to their analysis, demonstrated effectiveness of homeopathy. This explains why NHMRC concluded there is ‘no reliable evidence’ that homeopathy is effective.
For context it is worth remembering that if it was indeed the case that ‘no reliable evidence’ existed for homeopathy, it would put homeopathy in the same evidence category (“unknown effectiveness”) as 46% of conventional treatments used in the NHS, but in fact this conclusion is inaccurate.
Contrary to NHMRC’s findings, there are ‘good quality, well-designed studies with enough participants for a meaningful result‘ (to use NHMRC’s description of a reliable study) which show that certain homeopathic treatments are effective for certain conditions such as hay fever, sinusitis, upper respiratory tract infections, diarrhoea in children and lower back pain. The fact that the results of such studies were unjustifiably dismissed means that NHMRC have misled the public by misreporting the evidence for effectiveness of homeopathy.
Find out more on our Australian Report FAQs page
https://www.hri-research.org/resources/homeopathy-the-debate/the-australian-report-on-homeopathy/