Designing effective research

by Geoffrey Hart

Previously published as: Hart, G. 2018. Designing effective research. <https://worldts.com/english-writing/eigo-ronbun55/index.html>

One of the biggest problems I encounter in the journal manuscripts that I edit relates to the experimental design. Often, authors obtain results that are not statistically significant, even though it seems likely from the relative magnitudes of the numbers that the differences between treatments are large and significant. When we study biological systems, there is always a risk that high variability will make it difficult to detect significant differences. But there are proven techniques that can reduce that risk, and more researchers should use these techniques. In this article, I’ll remind you of some of the most important ones.

Control the experimental conditions

One problem with field research is that many factors are beyond our control, such as the weather, and can affect the outcome of experimental manipulations. This is why many researchers prefer laboratory research: it lets them control these variables. But even in laboratory research, living organisms can be highly variable, and their environmental conditions can affect how they respond to experimental manipulation; consider, for example, the physiological differences between mice raised in enriched versus impoverished environments or plants grown in different controlled-environment chambers or greenhouses. When it’s necessary to limit variability in the field or the lab, the first step is to review the research literature to identify the factors that can affect your research results. Understanding these factors lets you choose experimental conditions that produce the smallest possible differences in these factors, thereby minimizing their impact on the results. It also lets you perform stratified sampling, in which each group of organisms with similar properties or conditions forms a “stratum” that has reduced internal variation.

Sample sizes

Field research can be difficult because of the time and resource requirements. When I did plant physiology research many years ago, the measurements took long enough that by the time I reached the final sample plant, it was time to return and re-measure the first plant; it would not have been possible to increase the sample size. Time and budget constraints force many researchers to use a small sample size; though this may be required by their limited time and money, it often results in a lack of significant differences between treatments. If there is no inherent problem with an experimental system that would create high variation (e.g., a strongly skewed or bimodal frequency distribution that results from some phenomenon), increasing the sample size will usually help. But how much of an increase is necessary?

The best way to estimate the required sample size is to examine sample sizes in published research that is similar to your proposed study. Studies where treatments appeared to produce different results, but no significant difference, suggest that you should use a larger sample size than those researchers. If you prefer a more mathematical approach, you can try to predict the required sample size using the reported variances. Wikipedia provides a good overview of this subject ("Sample size determination"). You can also start by determining the required “statistical power” (the probability that you will correctly reject the null hypothesis) of your design.

It may not be economically or physically possible to use that sample size. In that case, you must find an effective compromise, such as reducing the number of treatments that you will compare or finding ways to control experimental conditions to reduce the natural variation.

Triangulation

In land surveying, “triangulation” means examining the position of a point by examining it from two or more angles and using basic geometry to calculate the position—something that would be difficult if you work from only one position. In research, triangulation means confirming a finding by obtaining evidence from multiple directions. For example, to measure gene expression, you can either quantify levels of the protein encoded by the gene or levels of the gene’s RNA transcripts. In ecology, you can measure the volume of carbon sequestered by vegetation using either species-specific allometric equations or destructive sampling. In both examples, the results of the two measurements will be correlated but different. If one comparison fails to produce a significant result, the other comparison may provide the desired result. Even if it doesn’t, detecting the same overall trend in both comparisons makes it more likely that a difference exists and that you’ll be able to find ways to demonstrate that difference in future research.

Confirming that you can analyze your data

Sometimes an experiment seems to be well designed, but it becomes difficult to perform statistical significance tests for the data. A simple example might be when you design an experiment based on the assumption that the data will be normally distributed, when in fact the data follows a highly skewed or non-normal distribution or even a multimodal or binomial distribution. If the research literature reveals such a problem, ask a statistician for advice on how to modify your experimental design to handle such data. You can sometimes still find a way to analyze data collected using a statistically inefficient design, but it’s better to start with an efficient design that produces data that’s easy to analyze. With a little knowledge of the likely variance structure of the phenomenon or organism that you’re studying, and the likely frequency distribution of the results, a statistician can help you design an experiment that maximizes your ability to detect significant results.

Summary

Most students learn these concepts early in their university career, but a surprising number forget to use them when they design their own research. I hope that this article will help you in your own research, and will help you train your graduate students to design more effective experiments.

For more information on designing effective research, see my book Writing for Science Journals.