Facebook

It might sound odd for a neuroscientist who studies autism in mice to empathize with anti-vaxxers, but I do. I agree with the solid research by sociologists that vaccine skeptics are only trying to make the best, most informed decisions for their children's health outcomes. Anti-vaxxers think of themselves as experts, much like scientists do. And like scientists, they rely on information that research produces to try to understand health. That's why when science produces bad information and communicates it poorly, the outcomes can become dangerous.

That bad science problem doesn't just apply to anti-vaxxers. Scientists like me rely on these studies, too. The design of my experiments, how I interpret my results, and ultimately how working with laboratory mice turns into cures for humans all originate from the same studies that anti-vaxxers cite. I understand their frustrations with poor science and communication because I feel the same frustrations in my daily life, too.

That's why I'd like to tell you the story of how just one example of bad science and its communication rippled through the news and gave thousands of anti-vaxxers ammunition to make absurd claims about autism and health.

How do we know who has autism?

To know if the rate of autism increasing, we have to know how many people have it in the first place. The prevalence, or how many people have the disease at any given time, is quite difficult to know because autism is pretty hard to measure. It’s not like we can strap a cuff to every person to measure it in the same way we can with blood pressure. Looking at medical records isn't perfect either because of the rarity, variation, and recent discovery of the illness. And while there are validated tests to diagnose autism, they are costly and time consuming, which makes testing all 320 million Americans impossible.

To minimize some of these obstacles while trying figure out the prevalence of autism in the United States, the Center for Disease Control (CDC) takes tiny samples of the population and uses a series of tests to diagnose autism for this smaller group. They then correct the sample by race, gender, age, sex, and other factors to make the group represent the population of the United States as best as they can. Using statistics and these corrections, they report a prevalence for the whole United States using the data from this tiny group. Currently, the CDC reports a prevalence of 1.5%, which was calculated from a sample of 346,978 Americans.

Is the prevalence of autism increasing?

By comparing all of the studies that take snapshots of autism prevalence, we can begin to piece together if autism is increasing, decreasing, or staying constant over time. Autism exists outside of the United States, too, and determining the prevalence in these places can give us more data points to determine autism rates.

In 2011, a group of researchers lead by Young Shin Kim (then at Yale) published a report on the first autism prevalence study ever performed in South Korea. Their finding was a prevalence of 2.64%. At the time, this was the highest prevalence ever reported anywhere in the world. Their discovery made headlines from NPR to Wired Magazine, all of them pointing to it as an example of increasing autism rates. To date, the result is the first thing you see when you search for the South Korean autism rate on Google.

The problem is, they were wrong.

How did they come up with 2.64%?

First, Kim and her team targeted all 55,266 elementary school students ages 7 to 12 in one area in South Korea: Goyang City. Goyang has two different types of schools, "regular" schools and special education schools. The special education schools had already identified individuals at risk for autism, so the team focused on regular schools to determine how many of their 36,592 students could be at risk too.

Downtown street on Ilsan, Goyang, South Korea. — Goyang City, South Korea, where the study was conducted.

Photo by Nagyman via Wikimedia Commons

Because conducting an authoritative test for autism is costly and time-consuming, the team used a two-step process to test these students. First, they asked parents and teachers to administer a simple screener test called the Autism Spectrum Screening Questionnaire (ASSQ), which you can administer yourself. Of the 36,592 regular school students, the test showed that 1,742 of them were at risk.

To see if any of these students who tested at-risk truly had autism, the researchers next administered a more authoritative, in-person test to consenting students who screened positive from the regular schools. They also administered these tests to consenting students in the special education schools to confirm their prior diagnoses.

The Autism Diagnostic Observation Schedule (ADOS), an example of a gold standard autism test.

In total, 201 study participants were diagnosed with an autism spectrum disorder. By correcting this sample for South Korea using similar methods to the CDC, they arrived at an overall prevalence of 2.64%.

Why 2.64% should be alarming

You should always double-check your math. The study suggested that the prevalence of autism was 2.64%, which should translate into 1,459 students in Goyang City having autism (55,266 total students × 2.64% = 1,459). But the researchers exhaustively tested every child they could in Goyang City, and only found 201 people with autism, which is just 0.36% of the population. So, either the prevalence estimate they calculated was off by 2.28% (2.64% − 0.36% = 2.28%) or they missed 1,258 students (1,459 − 201 = 1,258) with autism in their study.

Jungball Middle School (정발중학교) in Ilsandong-gu, Goyangsi, South Korea — The team focused on South Korea's "regular" schools to determine how many of their 36,592 students could be at risk.

Photo by Tjrdbsjung via Wikimedia Commons

Most bad science is easily recognizable when you dig into how the study was conducted. But the problem with this study wasn't how the researchers collected their data, it was how they interpreted it. What Kim's group did to collect this data in a country that had never been tested before was impressive. But by cutting a few corners in their analysis, they turned their findings into a blockbuster paper that made headlines for the wrong reasons.

To explain the gaps between what they actually found and what they theoretically should have seen, we're going to have to dig into the assumptions they made about the data.

What went wrong?

If you administered the ASSQ yourself, you might have noticed something odd (here's an alternative, called the AQ, that you can take yourself if you don't have a child). The AQ or ASSQ may very well have suggested that you or your child are at-risk for autism, even if you aren't. In other words, it may have given you a false-positive result. When I took the AQ, it suggested that I am at-risk for autism, which is a common result for most scientists. But I do not have autism.

If that happened for you, there's no reason to be alarmed. These screeners are not perfect tests. It wouldn't be unexpected if I tested at-risk for autism on a screener, but then was not diagnosed as having autism in a follow-up interview with a clinician. That two-step process is what this study used, but the researchers missed a critical part of the process when they applied their results from one city to all of Korea.

To determine the prevalence for all of South Korea, and not just the group tested, we need to know how well a screener performs. A test with 100% sensitivity gives zero false-negatives, which means it is great at telling you who is not positive.

As my result and lots of published science shows, the ASSQ is not 100% sensitive. In other words, not everyone who tests positive on the ASSQ is actually at-risk and not everyone who tests negative doesn't have autism. Yet for the purpose of this study, the authors reported that there were no false-positives. Which means that in their analysis, they calculated their statistics without factoring in any error for the ASSQ.

In their study, 1,742 students screened at-risk for autism using the ASSQ. But because the test isn't perfect, the actual number of at-risk students in Goyang City could be much higher or lower. By not factoring this problem into their calculation, they don't know exactly how many students actually are at risk for autism. And that means they don't know the inverse, the exact number of students who don’t have autism, either.

Not making these corrections sets you up for some pretty large error. If the authors had assumed a sensitivity estimate of 80% for the ASSQ instead of 100%, and not changed anything else about the study, their reported prevalence would have changed by more than 20%. In reality, the at-risk population could actually have been much higher or lower than they assumed just in Gouyang City. When you multiply that out to the entire South Korean population of 50 million people, you magnify that error.

The authors made another questionable decision by assuming a 0% nonresponse bias. In other words, they didn't account for how likely or unlikely a person would be to participate in the study in the first place. That assumption is unlikely to be true given that autism is highly stigmatized in South Korea. If you were a parent and your child screened positive for autism, you'd understandably have a huge bias toward removing your child from the study before an official diagnosis could be administered. The authors were aware of this problem and even used native clinicians in attempt to decrease it, yet they chose not to calculate how the stigma affected their results.

In both the sensitivity and nonresponse bias cases, and a few more technical ones, this study made bad assumptions that skewed their statistics. The methods they used to collect data were robust, but because of these omissions, their result of how many people had autism in South Korea wasn't useful at all.

To make matters even more confusing, their reported statistical prevalence was actually a range of anywhere from 1.91% to 3.37%. The 2.64% number that made headlines was just the average of this range. That means the prevalence reported by this study could realistically be anywhere between slightly higher than the current U.S. prevalence, on the low end, to almost triple the highest prevalence ever reported anywhere in the world. Not particularly specific, right?

What is the actual prevalence of autism in South Korea?

I have no idea. And you shouldn’t believe anyone who tells you that they absolutely do.

The best we can do without replicating the South Korea study, which took six years, is to re-analyze the data that they generated using more realistic assumptions. Peter Pantelis and Dan Kennedy from Indiana University did exactly this in 201 6 by using a monte carlo simulation, a test that looks at all possible combinations of data, how risky they are, and how plausible each outcome is.

Their result was even higher and the error was even bigger: an average prevalence of 3.3%, but a range between 2.0% and 5.4%—more than double the error margin reported in the original study.

So, is the prevalence of autism actually increasing?

Truthfully, I would argue that we're not good enough at determining prevalences yet, which means citing studies like this to make an argument that the prevalence of autism is increasing is flawed from the start. And when you add in the fact that many factors about autism often change, even the very definition of the illness, comparing prevalences across different points in time to determine a growth rate is more guesswork than it is science.

One of the authors of this study once told me, “the world is flat when it comes to understanding the brain.” As someone who is making a living on trying to understand the brain, that's far more frustrating than it is profound. And I can't imagine how angering it must be for people trying to decide on how best to keep their children safe, or understand their illness.

We all have good intentions, scientists and vaccine skeptics alike. But when there's so much at stake, all parties need to be extra cautious about the data and the claims they make about it. That means anti-vaxxers should make more of an effort to understand the science behind the headlines. No more making claims that autism is increasing 97% based on cherry-picked, error-prone studies. But it also means that researchers need to be more careful about how they interpret and communicate data. Eventually, we'll understand how to measure autism. But until we do, we have to learn to better deal with and communicate uncertainty, and trust the process of science to eventually get things right.

I empathize with anti-vaxxers. Here's how flawed autism science can lead to the wrong conclusions.