Now that you’ve learned that data is all around us and is constantly being analyzed, you may be feeling a bit down, thinking that humanity ...
Now that you’ve learned that data is all around us and is constantly being analyzed, you may be feeling a bit down, thinking that humanity is just a collection of data points. You may be asking yourself where free will comes into play if your internet browser knows so much about you that it can predict where you’re going to make your next purchase. Despite swimming in a sea of data, we need to use our brains to make sure we interpret and use that data correctly. It’s tempting to think that numbers don’t lie, but analysis is crucial. People make mistakes in data analysis every step of the way usually unintentionally, but sometimes intentionally.
One of the most consequential misrepresentations of data in recent memory is the 1998 study claiming that vaccines cause autism. Andrew Wakefield and colleagues surmised in a Lancet article that the measles, mumps, and rubella (MMR) vaccine led to the development of autism in their small study. Despite the almost immediate refutation of the study by the scientific community, parents around the world latched on to the headline. More than two decades later, pediatricians are still fighting concerned parents’ misconceptions, having to convince them that these critical vaccines are harmless. The impact and backlash of this small study was so significant that one 2011 headline reads: “The MMR vaccine and autism: Sensation, refutation, retraction, and fraud.”
So what was wrong with Wakefield’s study? Wakefield and his colleagues made several key omissions along the way, some of which may have been intentional. According to the Children’s Hospital of Philadelphia (CHOP), the study had two main flaws. The first is that administration of the MMR vaccine and diagnosis of autism are typically made around the same time (within the first few years of a child’s life). Thus the fact that eight of the study’s twelve participants were diagnosed with autism soon after they had received their MMR vaccine could have been purely coincidental. After all, ninety percent of children in the United Kingdom received their MMR vaccines at that time, but ninety percent of children were not being diagnosed with autism. In order to prove that the vaccine caused autism, the study would have needed a control group of children who had not received the vaccine.
The second major flaw, according to CHOP, was in reporting the timeline of events. According to Wakefield and colleagues, the vaccine caused gut inflammation, which then caused symptoms of autism. Upon further examination of the data, though, critics found that the eight children diagnosed with autism complained of intestinal symptoms after their autism diagnosis had been made. In other words, autism and gut inflammation may have been related, but the timeline of symptoms didn’t support the thesis that gut inflammation caused autism.
That wasn’t the end of Wakefield’s story, however. Wakefield and colleagues published a second study in 2002 claiming that the measles virus was found in intestinal biopsies of children with autism. This time they did include a control group that showed that this result was much more common in children with autism than in those without. Critics found even more flaws in this study, including the lack of distinction between the natural measles virus (which was still circulating in the UK at that time) and the virus from the vaccine.
Exposing Fraud
OKAY, SO MAYBE WAKEFIELD and his colleagues made a few mistakes and published some sloppy research. It happens quite often in the scientific community, believe it or not. It turns out, though, that Wakefield was being funded by a group of lawyers working on behalf of parents bringing lawsuits against vaccine companies. As the headline read, what began with a simple retraction by the Lancet turned into Wakefield being accused of ethical violations and eventually all-out fraud. Wakefield was removed from Britain’s General Medical Council, the register of licensed physicians in the UK. The first line of Wikipedia’s entry on Wakefield reads: “Andrew Jeremy Wakefield (born September 3, 1956) is a British fraudster, discredited academic, anti-vaccine activist, and former physician.”
Wakefield’s fraud wasn’t immediately recognized by the Lancet or by others in the medical community. What brought Wakefield’s financial conflicts and the extent of his fraud to light was a dogged journalist named Brian Deer who published several articles exposing Wakefield. This begs the question of how Wakefield’s study poorly designed, incorrectly analyzed, and fraudulently reported could have been published in the first place.
A 2011 editorial in the British Medical Journal examined Deer’s writing on Wakefield to try to expose the flaws in the system. It wasn’t just one bad apple (Wakefield), the authors argued, but a systems failure that allowed such a flawed piece of research to gain traction. The authors argue that publishing bad data has major consequences: “Science is our best way of knowing. When work presented as science is shown to be corrupt, it not only discredits that work and its authors, but it also discredits science.” Not to mention the medical consequences: Clusters of measles outbreaks persist as skeptical parents refuse to vaccinate their children. Unfortunately, this isn’t the only time medical researchers have failed to serve the population they are entrusted with far from it. The Tuskegee syphilis experiments also reveal how widely flawed the system of medical research and analysis can be.
Validity, Reliability, and the SAT
Does this mean you shouldn’t trust anything? That you should disregard any advice a medical professional gives you, question all science you read, and live solely by what you deem to be true or untrue? No that would be the pendulum swinging too far in the opposite direction.
The middle ground is that each of us needs to have a discerning eye, whether we are home watching the news or researching a vaccine in a lab. Now that you know a few ways that data analyses can go wrong, make sure what you hear meets certain standards. When you hear results of a study, here are some questions you can ask yourself:
- Is the source unbiased? Are there financial or other reasons to think the source might not be neutral?
- How big was the study? Are the results reproducible on a large scale?
- Does the conclusion seem supported by the data?
- Was there a control group, and were standard scientific procedures followed?
The two big terms we need to think about with data are reliability and validity. Reliability means how well the results of a data analysis (a medical study, for example) can be reproduced. If another study was done on the same topic, would we get the same results? If not, this tells us there may be a flaw in the study’s design, perhaps in the sample or population defined. Validity means how valid the results are, or if the data actually proves the conclusion. Believe it or not, reliability and validity are not always there in scientific studies. They’re not always there in things that we, as a society, have accepted as fact.
Let’s look at some data that is widely accepted: SAT scores. SAT scores are usually considered reliable, meaning a student would get the same or similar score if they took the test multiple times because they are so carefully researched. The SAT uses sample questions and passages to gather data before including anything new in the test. But are the results of the SAT valid? Do they tell us what they purport to tell us how college ready a student is?
The validity of SAT scores has been hotly debated in recent years, particularly since the Covid-19 pandemic began. Pre-pandemic, most colleges in the United States required standardized test scores to accompany an application. As of 2024, only four percent of colleges require them for admission. This doesn’t mean that we have reached a consensus about test scores not being a valid indicator of college readiness. It means, rather, that colleges are recognizing the debate and the effects of the pandemic and choosing not to require them.
Why is the validity of SAT scores so hotly debated? For one, higher scores correlate with higher income brackets. Does this mean that wealthier kids will necessarily do better in college? Many argue that it means the test questions are biased toward more privileged students, who have also had more opportunities for private tutors and test prep classes. The organization FairTest argues that the schools not requiring tests “recognize that standardized test scores do not measure academic ‘merit.’
What they do assess quite accurately is family wealth, but that should not be the criteria for getting into college.”
On the other side of the argument is the College Board, who argues that SAT scores are “strongly predictive of college performance” and student retention. The College Board makes the SAT, though, so it clearly has a vested interest in it being used. If you’re using the questions listed above to analyze information, that should raise some red flags. The next step would be to look for other sources and see what they say. In this case, the jury is still out. At least one writer at Forbes comes down in favor of the validity of SAT scores: “6 Arguments Against The SAT And Why They Don’t Hold Up” reads a 2020 headline.
What we need to do in this case is read many sources (or even look at the data ourselves if possible) and come to our own conclusions. A PBS article titled “Views of Authorities on Intelligence & Testing” compiled viewpoints of many different academic leaders on this issue. They pretty much all concurred that the SAT does not measure intelligence or even scholastic aptitude, as the decades-old name of the test suggests. Most do believe, however, that there is a correlation between higher test scores and performance in college. But there is no consensus on whether that matters. That lack of consensus may be why more and more colleges are giving up the standardized test requirement.
The post isn’t to disregard all studies or to believe everything you hear. As with most things in life, the answer lies somewhere in the middle. Data can tell us a lot, but it can also be misreported and misrepresented, both unintentionally and deliberately. Ask questions, look at the numbers yourself, and don’t believe everything you hear until you’ve fact-checked it or gotten it from a source you know is reputable.
