Changing Your Mind: The Fallacy of Hypothesis Testing

 
The Edge has asked people to describe whether they have changed their mind about anything and if so, why? It's the Annual Question for 2008.

Some of the replies are worth discussing. For example, Irene Pepperbreg has changed her mind about the meaning of the scientific method [The Fallacy of Hypothesis Testing]. I think she makes some good points, notably ....

Third, I've learned that the scientific community's emphasis on hypothesis-based research leads too many scientists to devise experiments to prove, rather than test, their hypotheses. Many journal submissions lack any discussion of alternative competing hypotheses: Researchers don't seem to realize that collecting data that are consistent with their original hypothesis doesn't mean that it is unconditionally true. Alternatively, they buy into the fallacy that absence of evidence for something is always evidence of its absence.

I'm all for rigor in scientific research — but let's emphasize the gathering of knowledge rather than the proving of a point.

I think this is a serious problem in science today. There are too many papers being published without any serious discussion of competing explanations. There are too many papers that fail to critically examine their own basic assumptions or the possible flaws in their experiments.

There may be a reason for this behavior—scientists don't want to draw attention to possible flaws in their work for fear that the granting agency will find out—but that doesn't excuse it. Scientific rigor demands that you present both sides of a scientific debate in a fair and unbiased manner. The failure to address the arguments of your opponents is nothing less than failing to be a good scientist.

Similarly, the failure to recognize the possible flaws in one's own explanation is the mark of a bad scientist.

While Irene Pepperbreg may be right about the flaws in today's method of doing science, I'm not prepared to throw out the baby with the bathwater. Hypothesis-based science is still important. You just have to form the right hypotheses and put your work in context. The problem, in my opinion, isn't that hypothesis testing is a fallacy: the problem is that it's not being done properly.

---