http://www.ageofautism.com/2008/12/a-message-to-au.html#more
As a person who is in the beginnings of her research career, I wish to say that though my understanding of Statistics and Research Methodology is not yet at its zenith, even at this early stage in my learning I am familiar enough with data analysis to state very affirmatively that a layperson's view of "science" and that of the scientist, him- or herself, are frequently light years apart. Ideally, the scientist is trained to view data with a skeptical eye, aware of the fact that, while numbers don't ever lie, sheer probability, human error, and interpretation are all too frequent companions to design and analysis. In addition to being skeptical and critical of the results of research, it is also important to note that scientists are trained to never base a conclusion on the results of a single study due to the extreme chance of error inevitably present in all forms of research. In science, theories are based upon the replication of results, not of a single study. Most laypeople do not seem to grasp this concept because we are so used to being bombarded by reports through the media of single studies and anecdotal evidence. A layperson's view of science is one which has been mutated and shaped by the media.
You might ask why it is that scientists are made to be so distrustful of data. Aside from the risk of a poor research design falsifying study results, part of that reason is also found in the statistical analyses frequently used to analyze the data collected in a quantitative study. There are a group of statistical analyses called "Analysis of Variance"; what these do are analyze trends within sets of data to discern the probability of a relationship between these variables. Most scientists don't do these calculations by hand but set up data tables in statistical programs, like SPSS, which do the analyses for the investigator and spit out the results. One of these results which are given with the Analyses of Variance is something called a p-value, which tells the investigator whether the results of the analysis are deemed "significant" or not. A p-value is based upon the "alpha level," which is a value the researcher sets at the beginning of the test; an alpha level is the percentage risk the researcher is willing to take that the results were achieved by chance and chance alone, as opposed to being a result of a true relationship between the variables. So by setting an alpha level of 0.05, you are stating that you are willing to take a 5% risk that your results are due to chance alone. 0.05 tends to be the upper level that most scientists will take, although this tends to vary by the type of research being done. In the Behavioral Sciences, 0.05 alpha is common, while 0.01 (or 1% chance) is seen often in something like the Biological Sciences.
This is one reason-- mathematics-- that scientists distrust the reliability of data. Think of what this is saying: if I set my alpha level at 0.05, I am accepting the fact that I am taking a 5% risk of getting a false positive and that there is really no relationship at all in the variables I'm studying. What the alpha level does is set the amount of variation the sample is allowed to contain in order to calculate whether the relationship is deemed "significant" or not. An alpha level of 0.05 allows for greater variation than does an alpha level of 0.01. By setting my alpha at 5%, I am accepting that, according to probability, if I were to replicate this exact study 20 times, 1 out of these 20 studies would give me false results. --And that's assuming the design in my study is flawless and without human error.
Now you may wonder why in the world I've begun giving a mini-lecture on data analysis when my title implies I'm writing some sort of editorial in response to Katie Wright's post on the Age of Autism blog. I've written the above to illustrate in greater detail the differences between a layperson's grasp of Science and that of the scientist. Not only in this day and age has every Tom, Dick, Harry, and Harriet become armchair psychologists, but they have become armchair scientists as well. There's not too many careers out there where you can be treated as a qualified professional either by attending Harvard OR the University of Google. As someone who has been both on the receiving end of poor health care and someone who is going into the "treating" end of the fields, I have to say how presumptuous it is for every Tom Cruise out there to profess his expertise in interpreting research without the knowledge and training to do so!
By saying that, I am also not implying that every professional out there sincerely knows what they're doing either. Any person who's been on the receiving end of ridiculously ignorant patient care knows full well that a degree hanging on a doctor's wall simply signifies he's gone through medical school. But if I were to read up on car mechanics despite not having the training and experience to work in that field, would you want me fixing YOUR car??? Well, no offense, but I don't want Jenny McCarthy interpreting MY data either.
Perhaps you tell me a car is not the same thing as a child and that parents of autistic children would be offended by such an analogy. Well, I hope you're right, considering if you destroy the engine of my car I can just go and buy a new car. A child, however, is precious and irreplaceable. And it's for this reason that it is vitally important not to leap to any single conclusion concerning the results of research because of the potentially dire effects it might have.
Example: I have a son (I don't actually, this is for the sake of argument). My son has been diagnosed as autistic. I don't know too much about Jenny McCarthy but, through word of mouth, I hear that she's a huge proponent of the gluten-free/casein-free diet and she feels it can cure autism. I'm a cautious parent, but since this isn't anything involving medications and just a change in diet, I figure it's worth the risk to cure my son. Rather than taking my son in to his doctor to run tests on his antibody teeters or to have a tissue sample taken to verify the presence of gluten or casein antibodies (since I'm not a doctor and have little idea what antibodies are other than people seem to mention them when talking about the immune system), I do the if-it-works-then-it's-a-diagnosis method. Unfortunately for my son, his food sensory issues are so extreme that all he will eat are variations of sandwiches. Now that he's gluten-free, however, this portion of nutrition has been removed from his diet. After several weeks of this new diet regime, my son is doing more poorly even though I try to supplement his diet with vitamins. He still has stool problems, his behavior is worse, and, while he was thin before, he's lost even more weight. I finally take him into the doctor, some tests are run, and I am told that, rather than being intolerant of gluten or casein, my son is intolerant to the proteins in milk.
Why did I just give this hypothetical story? Solely to illustrate two things: Generalization and Assumption. Jenny McCarthy made the mistake of generalizing that autism can be cured by a GF/CF diet, without regards to thorough research. And I have made the assumption she's right and have harmed the health of my son in the meantime. Did I do this purposefully? Vindictively? No, not at all. I did what I felt was best for my son. And not being a doctor or a researcher, I wasn't fully aware of the potential dangers inherent in making these assumptions.
Katie Wright is a mother, not a scientist. I can't for one moment imagine her crusade is one of vindictiveness or for lack of caring for her son. In fact, her fervor implies the opposite. As I said though, she is acting as a mother, not a scientist, and she is making the same mistakes that I've illustrated above: she has assumed that aspects of reported research are absolutely true regardless of research design or replication, and she has generalized her assumptions to indicate that all of autism is caused by vaccinations.
I don't know enough about the underbelly of politics going on within the Autism Speaks organization. Frankly, I'm not so sure I'd want to know. And the same thing goes for the CDC, the FDA, and the pharmaceutical companies. I think Katie Wright is right in saying this whole situation isn't just about helping autistic people, their families, and doing the research. It's just as much politics and business as anything else.
What is my take on the whole Vaccine Theory of Autism? I have a fairly precise idea which is neither here nor there. Given the amount of research coming out nowadays on the immune system in autism (and I'm not talking about research put out by the Autism Research Institute, I'm talking about WELL-DESIGNED research), I think it's fairly clear that effects of the immune system are somehow involved in a portion of cases of autism. Now, to what extent, and whether the immune system plays some role in the level of severity, I could only speculate. What role do vaccinations potentially play in severity of autistic expression? Again, unknown. And that is largely because politics (the CDC) has tried to barely touch that with a 40 foot pole, and has solely focused on black-and-white all-or-nothing-at-all research designs: do vaccinations cause autism? Not whether vaccinations might play some role in severity of autistic expression.
To some extent, I can understand the government's and medical community's hesitation in doing the research. Vaccinations have been lifesaving for many people. What do you think would happen if a set of research studies came out with strong evidence that vaccinations increase severity of the autistic phenotype? There'd be mass panic, parents would refuse to get their children inoculated, and certain childhood diseases would likely begin to reappear in force and create larger problems than autism. Already, with the mild panic that's already been induced, some childhood diseases are making a slight comeback.
At the same time, however, the parents groups seem to run wild with anecdote. Like Stalin said, "The death of one man is a tragedy, the death of millions is a statistic." And how right he was. Listen to a single person's hardluck story and we're in tears; hear about the innumerable holocausts in our human history, and it barely turns our attention.
Why do you think, in statistics, GROUPS of participants are used more often rather than relying upon single case studies? Because, while statistics aren't so moving, numbers don't lie-- and the more numbers you have, the more potentially representative it is.
But emotion is infectious: a single person can move us far beyond any number, even if that number is based upon hundreds, or thousands, or even millions. It's because we're human: we're designed to be socially-influenced creatures. We haven't evolved to be natural statisticians.
But the only way for laypeople to interpret the results that are disseminated to the public, without coming to inaccurate conclusions, is in fact by being good skeptical statisticians. Unfortunately, we must constantly caution ourselves that emotion can run away with us, leading us down false paths. It can cause us to hurt the ones we are trying to help.