Why this vaxed v. unvaxed study is not valid: Update: Study retracted AGAIN.

Update: This study has been retracted for the second time. 

 

For the last few days, people opposed to vaccines have been posting a link to a study called Pilot comparative study on the health of vaccinated and unvaccinated 6- to 12- year old U.S. children. The lead author is Jackson State, MS, University professor, Anthony R. Mawson. This study is not valid and here is why.

First of all, I need to explain what is meant by validity and reliability, with regards to science.  The University of California, Davis, has a very good synopsis. “In order for research data to be of value and of use, they must be both reliable and valid.” Reliability refers to how well the findings of the study can be repeated. If a study was done in a manner that is objective and well-executed, then other scientists should be able to repeat (or replicate) it and get the same findings. Validity refers to the believability of the research.  How well do the findings answer the study hypothesis.  There is internal validity, which refers to how well the procedures in the study measured what they were supposed to measure. And, there is external validity, which refers to how well the findings can be generalized.

So, in an ideal study of children’s health, we would not need to take the researcher’s word for anything. The data would be reliable because all claims would be verified. For example, if the the study claims that 5% of children got colds twice a year or more, it would be reliable data if the researchers used the children’s medical records to determine how many colds they had a year. We would know that the data had been compiled by the children’s healthcare providers and analyzed by the researchers. Nothing would be left to interpretation.

But, if we just ask parents, how many colds a year do you think your child has had, those answers are not necessarily reliable because parents don’t always know the difference between a cold and influenza or allergies. And, they would not be basing their answers on data they collected but rather memories. Memories are notoriously inaccurate.

That brings us to the Mawson study.  First of all, you need to know that there was an attempt to publish this study last year but the methods the study used and the fact that there were only two peer reviewers ( one being a chiropractor) caused alarm in the scientific community. The journal pulled the study before publication.  Many of us found out this was happening from Retraction Watch, a very interesting source to follow if you like reading about how science works and how studies are monitored.  Based solely upon the abstract, the study was criticized by many, including Respectful Insolence blog.

I must take a moment to point out that I homeschool one of my children so I am not biased in any way towards homeschooling. 

At Respectful Insolence blog, ORAC (aka Dr David Gorski, oncologist) rightfully criticized the methodology of the study as well as the fact that a chiropractor was used to peer review an epidemiology study. Chiropractors are not the peers of epidemiologists. ORAC also noted that this study was funded by Generation Rescue, a notoriously antivax group.

These are problems. Real problems. So, the original journal, Frontiers, took note and pulled the study.

Now, months later, the study has been published in a pay-to-publish journal online called Open Access Text. Reputable scientists don’t pay to publish their studies. Journals like Pediatrics or Vaccines or The Lancet don’t require authors to pay and they are considered far more respectable when it comes to considering authors for professorship positions. Scientists know these facts. They know that publishing in a predatory journal is not a good career move.

So, what happened after this study was pulled by Frontiers? It was submitted to Open Access Text, a predatory, pay-to-publish online journal, and published this week. And it is being spammed everywhere as a valid study.

It is not valid and here is why.

One: It was funded by two known antivax groups, Generation Rescue, Inc., and the Children’s Medical Safety Research Institute (CMSRI).  Both are well know to be opposed to vaccines. CMSRI is funded by the Dwoskin Foundation, who are big money behind a lot of antivax operations. This does not negate the results, by any means, but it does beg the question – what was the motivation for the study. By the same token, I would look very skeptically at any study published by a pharmaceutical company.

Two: Read the introduction. The authors went into the study assuming vaccines cause grave harm. ” The aims of this study were 1) to compare vaccinated and unvaccinated children on a broad range of health outcomes, including acute and chronic conditions, medication and health service utilization, and 2) to determine whether an association found between vaccination and NDDs, if any, remained significant after adjustment for other measured factors.”  That is serious bias.

Three: The study design was flawed. “The study was designed as a cross-sectional survey of homeschooling mothers on their vaccinated and unvaccinated biological children ages 6 to 12. As contact information on homeschool families was unavailable, there was no defined population or sampling frame from which a randomized study could be carried out, and from which response rates could be determined. However, the object of our pilot study was not to obtain a representative sample of homeschool children but a convenience sample of unvaccinated children of sufficient size to test for significant differences in outcomes between the groups.”  Right from the start, Mawson, et al, admit that they aren’t really able to do a good, quality study.  “A number of homeschool mothers volunteered to assist NHERI promote the study to their wide circles of homeschool contacts.”   This is also problematic. They had participants promoting the study to their own friends. How did they account for bias? They did not.

Four: Methods were flawed. The authors categorized the children as unvaccinated, partially vaccinated, or fully vaccinated based only on word of the mothers. They did not consult medical records. Mothers were then asked to indicate which illnesses their child had had but no medical records were consulted. This data was analyzed statistically but how can they analyze data they have not verified as accurate? They purposely did not use medical records because they said that would have led to low participation.

Five: The limitations. Oh my, the limitations. “We did not set out to test a specific hypothesis about the association between vaccination and health.”  So, this was not even science.

So, what does all this mean? It means we cannot validate the information the mothers gave is accurate or real. It means none of the data in this study means anything, because no one would ever be able to completely replicate it. They would never be able to go back in and find all the same anonymous mothers and guarantee the same answers from them. This kind of survey does not add anything of value to the body of literature on children’s health. Honestly, I could have done better as a freshman in college, in my introduction to research methods and statistical analysis class.

If you want a real, valid, reliable study on vaccinated versus unvaccinated, the KIGGS study is the place to go. Because the researchers used not only a parent survey but also a “standardized, computer-assisted personal interview (CAPI) of the accompanying parent by a doctor,” this data can be verified as authentic. That is reliability. This study could be repeated. Children’s vaccination status was documented. “The questions about diseases were followed by data collection on the basis of medical records in the vaccination card, about data concerning the administered vaccinations and the timing of the vaccination”  So, everything was verified. KIGGS is everything this new study is not. There is no reason whatsoever to think this new study is anything but bunk.

Remember, always verify claims and always think for yourself,

Kathy

Updated2: Other bloggers have been tackling this study and since their blogs are just as good as mine, I would like to share. Please check them out.

 

KidNurse: THE TRUTH ABOUT VACCINATED VS UNVACCINATED

Respectful Insolence: A boatload of fail: Were two horrendously bad zombie “vaxed/antivaxed” studies retracted—again?

Respectful Insolence: The Mawson “vaxed/unvaxed” study retraction: The antivaccine movement reacts with tears of unfathomable sadness

Respectful Insolence: The check must have finally cleared, or: Mawson’s incompetent “vaxed/unvaxed” study is back online

Snopes: ‘First Ever’ Study Comparing Vaccinated and Unvaccinated Children Shows Harm from Vaccines?

Science Based Medicine: Two (now retracted) studies purporting to show that vaccinated children are sicker than unvaccinated children show nothing of the sort

I Speak of Dreams: About Those “Homeschooled, Unvaccinated Children are Healthier” Studies.