The Experimental Design Assistant (EDA) online tool developed by the National Centre for the Replacement, Refinement, and Reduction of Animals in Research (NC3Rs) can help researchers improve their experimental design and conduct research that’s more likely to be relevant and reliable, an expert said.
Esther Pearl, program manager for experimental design at NC3Rs, said during a recent webinar sponsored by the Office of Laboratory Animal Welfare that the tool can benefit both researchers and institutional animal care and use committees (IACUCs) as they strive to improve research involving animals.[1]
Studies have shown that the vast majority of research—some 82% to 89%—is wasted, Pearl said. “They found this was mostly down to issues with experimental design, incomplete reporting or studies that were just not published at all. So, if we flip that around, that turns into about 10% to 15% of research that’s funded and generated is actually usable. So that’s really bad news.”
This has additional implications for animal research, Pearl said, since the harms to the animals are weighed against the likely benefits to science and society. “If the findings of our research aren’t reliable or that research is not reported in enough detail, then these benefits can’t be realized, and the research is simply unethical. So, these studies are a waste of animals.”
In addition, she pointed out that “entire programs of clinical work” then could be “based on the findings of unreliable animal research,” leading to clinical studies using treatments that are not likely to be effective. “That can also delay the discovery of new treatments while we chase things that end up being a dead end,” she said. “So, improving the rigor and transparency of animal research will improve the entire drug development process.”
Reducing research waste by improving how experiments are designed and reported is highly consistent with the 3Rs (replacement, reduction and refinement), Pearl said.
The EDA was launched to help researchers with in vivo experimental design. Meanwhile, the Animal Research: Reporting of In Vivo Experiments guidelines aid in reporting animal-based research and can work in conjunction with the EDA, she said.[2] NIH recommends using the EDA to prepare grant applications, Pearl said.
Research Improvements Slow
There are multiple causes of reliability and reproducibility issues with biomedical research, Pearl said, and they typically fall into six “buckets”: omitting null results, data dredging, weak experimental design, underpowered studies, errors and underspecified methods. Researchers have made some progress in improving experimental design over the last few years, but that progress “has been a bit slow,” she said.
If experiments are not rigorously designed, the results are unlikely to be reliable. Aspects of good experimental design include randomization, masking/blinding, using both sexes, appropriate sample size and preplanned statistical analysis, she said.
Randomization is vital to minimize selection bias. In addition, it also helps to meet one of the key assumptions of statistical analysis methods: that different groups should be drawn from the same background population using random sampling, Pearl said.
“How the randomization sequence is generated is really important. Haphazard—which is what happens when a human tries to create what they think is a random sequence— is not actually random,” she said, adding that humans actually can’t be trusted to generate a randomization sequence. True random sequences can be generated in different ways, including computer-generated, rolling a dice and flipping a coin, she said.
Masking, also known as blinding, is essential to reducing bias. It can be used at different steps in an experiment, including during the allocation and intervention, in housing and welfare management, while assessing the experiment’s outcome and when doing data processing and statistical analysis, Pearl said.
Sometimes, it’s not possible to mask during a particular stage of an experiment; in this case, researchers should be especially careful to conduct some of the other stages masked.
Another way to improve experiments is to reduce sex bias, Pearl said. In animal experiments, most research is still conducted on males rather than on females or both sexes, and this is a problem because it’s not necessarily possible to extrapolate findings from males to females.
“This is a real missed opportunity for us to collect really crucial information that could inform the design of clinical trials or provide warnings on adverse effects that are sex-specific,” she said, noting that, in about half of animal studies where both sexes are used, the researchers discover a difference between the sexes.
“So, in most cases, researchers should be including both sexes in their protocol, and it should be clear when you are reviewing a protocol what the researchers are trying to do,” Pearl explained. “Are they including both sexes to ensure the results are generalizable, or are they directly comparing the sexes? And if they are directly comparing sexes, do they have adequate numbers to make this comparison? Will they need to include more animals?”
If the researcher is not directly comparing the sexes, they may not need to increase animal numbers. “They could just do a power calculation for their different groups and just make sure half of each group is male and half female. And if the researcher is not sure about this, then you can suggest they consult a statistician, and that will help come up with a number that will give a nice, reliable result.”
