Uncaging Validity in Preclinical Research

High attrition rates in drug development bedevil drug developers, ethicists, health care professionals, and patients alike.  Increasingly, many commentators are suggesting the attrition problem partly relates to prevalent methodological flaws in the conduct and reporting of preclinical studies.

Preclinical efficacy studies involve administering a putative drug to animals (usually mice or rats) that model the disease experienced by humans.  The outcome sought in these laboratory experiments is efficacy, making them analogous to Phase 2 or 3 clinical trials.

However, that’s where the similarities end.  Unlike trials, preclinical efficacy studies employ a limited repertoire of methodological practices aimed at reducing threats to clinical generalization.  These quality-control measures, including randomization, blinding and the performance of a power calculation, are standard in the clinical realm.

This mismatch in scientific rigor hasn’t gone unnoticed, and numerous commentators have urged better design and reporting of preclinical studies.   With this in mind, the STREAM research group sought to systematize current initiatives aimed at improving the conduct of preclinical studies.  The results of this effort are reported in the July issue of PLoS Medicine.

In brief, we identified 26 guideline documents, extracted their recommendations, and classified each according to the particular validity type – internal, construct, or external – that the recommendation was aimed at addressing.   We also identified practices that were most commonly recommended, and used these to create a STREAM checklist for designing and reviewing preclinical studies.

We found that guidelines mainly focused on practices aimed at shoring up internal validity and, to a lesser extent, construct validity.  Relatively few guidelines addressed threats to external validity.  Additionally, we noted a preponderance of guidance on preclinical neurological and cerebrovascular research; oddly, none addressed cancer drug development, an area with perhaps the highest rate of attrition.

So what’s next?  We believe the consensus recommendations identified in our review provide a starting point for developing preclinical guidelines in realms like cancer drug development.  We also think our paper identifies some gaps in the guidance literature – for example, a relative paucity of guidelines on the conduct of preclinical systematic reviews.  Finally, we suggest our checklist may be helpful for investigators, IRB members, and funding bodies charged with designing, executing, and evaluating preclinical evidence.

Commentaries and lay accounts of our findings can be found in PLoS Medicine, CBC News, McGill Newsroom and Genetic Engineering & Biotechnology News.

2013 August 5 | Leave a comment | Tags: cancer efficacy preclinical studies translational research validity

@Manual{stream2013-300,
    title = {Uncaging Validity in Preclinical Research},
    journal = {STREAM research},
    author = {Valerie Henderson},
    address = {Montreal, Canada},
    date = 2013,
    month = aug,
    day = 5,
    url = {http://www.translationalethics.com/2013/08/05/uncaging-validity-in-preclinical-research/}
}

Hypothesis Generator

Is good medical research directed at testing hypotheses? Or is there a competing model of good medical research that sees hypothesis generating research as a valuable end? In an intriguing essay appearing in the August 21, 2009 issue of Cell, Maureen O’Malley and co-authors show how current funding mechanisms at agencies like NIH and NSF center their model of scientific merit around the testing of hypotheses (e.g. does molecule X cause phenomenon Y? does drug A outperform drug B?). However, as the authors (and others) point out, many areas of research are not based on such “tightly bounded spheres of inquiry.” They suggest that a “more complete representation of the iterative, interdisciplinary, and multidimensional relationships between various modes of scientific investigation could improve funding agency guidelines.”

2010 February 9 | Leave a comment | Tags: funding validity value

@Manual{stream2010-74,
    title = {Hypothesis Generator},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2010,
    month = feb,
    day = 9,
    url = {http://www.translationalethics.com/2010/02/09/hypothesis-generator/}
}

The Problem with Models

Chicago in plastic and balsa. If only animal models were as convincing as the one pictured above from the Museum of Science and Industry. 

2008 October 10 | Leave a comment | Tags: animal models methodology neurodegenerative diseases validity

@Manual{stream2008-131,
    title = {The Problem with Models},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2008,
    month = oct,
    day = 10,
    url = {http://www.translationalethics.com/2008/10/10/the-problem-with-models/}
}

Masks and Random Thoughts on Preclinical Research Validity

Epidemiologists and biostatisticians have evolved numerous ways of reducing bias in clinical trials. Randomization of patients, and masking them to their treatment allocation are two. Another is masking clinicians who assess their outcomes.

2008 February 28 | 2 comments | Tags: validity

@Manual{stream2008-175,
    title = {Masks and Random Thoughts on Preclinical Research Validity},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2008,
    month = feb,
    day = 28,
    url = {http://www.translationalethics.com/2008/02/28/masks-and-random-thoughts-on-preclinical-research-validity/}
}