Hypothesis testing isn’t the only valid use of science

By Geoffrey Hart

Previously published as: Hart, G. 2023. Hypothesis testing isn’t the only valid use of science. World Translation Services. https://www.worldts.com/english-writing/eigo-ronbun86/index.html

One of the identifying characteristics of modern science is that it is based so strongly on hypothesis testing. When you design a study, you are expected to create a research hypothesis that can be either confirmed or rejected based on your results. An entire body of scientific methods (statistics and experimental design) has been developed to support the testing of hypotheses. This approach has become the standard that is used in most branches of basic science for a simple reason: it works very well. Using this approach helps us to decide what we’re going to study and focus on how we’re going to study it. Most journals require a formal statement of your research hypotheses towards the end of the Introduction, and a description in the Conclusions section of whether your results confirmed your hypothesis.

Some researchers are so focused on the idea of testable hypotheses that they consider a study to be non-scientific if it is not based on testable hypotheses. Wolfgang Pauli famously dismissed a paper written by a colleague for this reason: “That is not only not right, it is not even wrong!” (This is often abbreviated as “you’re not even wrong”.) Pauli suggested that if the conclusion cannot be judged by the formal criteria for statistical hypothesis testing, the research isn’t science.

However, even though designing experiments to test a hypothesis produces important results, it’s not the only valid form of research. There are many reasons you might not want to base your research on a formal research hypothesis. These include:

You don’t know enough about a phenomenon to form a hypothesis. In this case, the purpose of the research is not to test a prediction, but rather to improve our understanding of how a study system operates. That is, the goal is to understand the system well enough to generate hypotheses that will lead to hypothesis testing in the future.
Your goal is to obtain raw data to support modeling. For example, it might be necessary to obtain several years of climatic data to provide inputs for a climate model and a separate dataset for validating that model. Although your hypotheses could be that climate varies over time and that the variations are sufficiently deterministic to support process-based modeling, these seem like very weak hypotheses.
Your goal is to parameterize an existing successful model. Although your hypothesis might be that you can calculate the necessary parameters for your region and that they will differ from the values for other regions, that also seems weak. The important point is that you are trying to support someone else’s hypothesis (that a system can be modeled mathematically), not that you are testing your own hypothesis.
Your goal is to collect raw data for a system that has not been previously studied. For example, your goal may be to measure soil organic carbon contents in samples from different soil types in a region where soil carbon has not been previously studied.
You may want to document something, such as the behavior of an animal, to support other researchers who will perform hypothesis-based research.

The distinction I’m trying to make here relates to the type of research that long ago would have been done by a “naturalist”, a name that derives from the phrase “natural history” (the study of nature’s story). This approach is based on observation rather than experimentation. Experimentation is generally based on predicting what will happen when you disrupt or perturb a system, but you can’t say whether and how a system will respond if you don’t know its initial state to provide a comparison. In purely observational research, the goal is to develop a body of knowledge that can subsequently be used for other purposes, such as identifying interesting questions to ask and generating testable hypotheses to answer those questions. Critics of observation sometimes point to its lack of rigor, but their criticisms ignore the effort that goes into planning a successful observational study. They also neglect the essential role of pure data collection, undirected by a hypothesis. For example, even a hard-core experimentalist would find it hard to suggest that the enormous bodies of satellite image data that are now available to researchers are not valuable, even though that data is collected to support experimentation, and does not itself represent a hypothesis.

Note: Researchers who emphasize the need to design research hypotheses often forget why this can be problematic. The problem results from confirmation bias: by focusing on proving your hypothesis, it becomes easy to ignore or overlook data that contradict your hypothesis. Many important breakthroughs result from disproving a hypothesis by observing a fact that doesn’t fit with the hypothesis.

When you begin to plan a study, it’s important to understand whether you will focus primarily on observation, with the goal of supporting the goals I listed earlier, or experimentation, with the goal of testing a hypothesis. When your goal is observation, you can use the goals I’ve listed at the start of this article to justify writing and submitting a paper that is not based on hypothesis testing to a journal, but you’re unlikely to persuade the editor of a journal to review your paper if that journal requires formal research hypotheses. Thus, you should spend some time thinking whether there might be a better place to publish your research (i.e., an observational journal). You’ll have more luck if you choose a different journal with different preferences. For example, the American Naturalist includes a category of article named “Natural History Miscellany”, which they describe as “[submissions] that enlighten our understanding of the natural history of a species in important ways and, because of their novelty, will be appreciated broadly”.

In this context, I can offer a personal example. Last year, I was standing in my garden, enjoying a favorite shrub (Salix cinerea), when I noticed that the branches seemed to be vibrating. When I looked closer, I saw a cloud of small bees surrounding the willow. This was early enough in the Canadian spring that few other flowers were available to support insects such as bees that require flowers to survive. Belatedly, I remembered the obvious: that willows are flowering plants, although their flowers are small and easy to miss. This led to the well-known, but sometimes forgotten, realization that animals of all sizes, from bees to bison, have evolved to depend on different foods at different times of year. And sure enough, based on this observation, I formed the obvious hypothesis (well supported by my research) that the bees would look for different foods at different times of year, changing their diet as one set of flowers disappeared and another appeared. This is not a shocking observation, but it’s nonetheless a good illustration of the role of observation, unsupported by hypothesis testing, in science.