Uncaging Validity in Preclinical Research

High attrition rates in drug development bedevil drug developers, ethicists, health care professionals, and patients alike.  Increasingly, many commentators are suggesting the attrition problem partly relates to prevalent methodological flaws in the conduct and reporting of preclinical studies.

Preclinical efficacy studies involve administering a putative drug to animals (usually mice or rats) that model the disease experienced by humans.  The outcome sought in these laboratory experiments is efficacy, making them analogous to Phase 2 or 3 clinical trials.

However, that’s where the similarities end.  Unlike trials, preclinical efficacy studies employ a limited repertoire of methodological practices aimed at reducing threats to clinical generalization.  These quality-control measures, including randomization, blinding and the performance of a power calculation, are standard in the clinical realm.

This mismatch in scientific rigor hasn’t gone unnoticed, and numerous commentators have urged better design and reporting of preclinical studies.   With this in mind, the STREAM research group sought to systematize current initiatives aimed at improving the conduct of preclinical studies.  The results of this effort are reported in the July issue of PLoS Medicine.

In brief, we identified 26 guideline documents, extracted their recommendations, and classified each according to the particular validity type – internal, construct, or external – that the recommendation was aimed at addressing.   We also identified practices that were most commonly recommended, and used these to create a STREAM checklist for designing and reviewing preclinical studies.

We found that guidelines mainly focused on practices aimed at shoring up internal validity and, to a lesser extent, construct validity.  Relatively few guidelines addressed threats to external validity.  Additionally, we noted a preponderance of guidance on preclinical neurological and cerebrovascular research; oddly, none addressed cancer drug development, an area with perhaps the highest rate of attrition.

So what’s next?  We believe the consensus recommendations identified in our review provide a starting point for developing preclinical guidelines in realms like cancer drug development.  We also think our paper identifies some gaps in the guidance literature – for example, a relative paucity of guidelines on the conduct of preclinical systematic reviews.  Finally, we suggest our checklist may be helpful for investigators, IRB members, and funding bodies charged with designing, executing, and evaluating preclinical evidence.

Commentaries and lay accounts of our findings can be found in PLoS Medicine, CBC News, McGill Newsroom and Genetic Engineering & Biotechnology News.

2013 August 5 | Leave a comment | Tags: cancer efficacy preclinical studies translational research validity

@Manual{stream2013-300,
    title = {Uncaging Validity in Preclinical Research},
    journal = {STREAM research},
    author = {Valerie Henderson},
    address = {Montreal, Canada},
    date = 2013,
    month = aug,
    day = 5,
    url = {http://www.translationalethics.com/2013/08/05/uncaging-validity-in-preclinical-research/}
}

Icarus, again: Adversity in another Gene Transfer Trial

Two weeks ago brought good news and bad news for gene transfer. First the good news. New England Journal of Medicine beatified a new gene transfer strategy for Wiskott-Aldrich Syndrome (WAS). WAS is a primary immunodeficiency that primarily affects boys. It is thus in the same family of disorders that have been, in varying degrees, successfully addressed using retroviral gene transfer. Like other immunodeficiencies, this represents relatively low hanging fruit for an approach like gene transfer, because scientists can access and target stem cells, and because corrected cells should be at a selective advantage for survival compared with uncorrected cells.

The NEJM article reported clinical, functional, and molecular outcomes for two boys in a trial based in Germany. Briefly the two boys were given a type of chemotherapy (in order to make space for genetically corrected cells), and then transplanted with “corrected” blood stem cells. The corrected blood stem cells contained a viral vector similar to those used in previous gene transfer trials of primary immune deficiency. The team saw: 1) stable levels of genetically corrected stem cells that expressed the WAS protein (indicating the genetically modified cells “took,” and produced WAS; 2) recovery of the function of a variety of immune cells; 3) reduction of disease symptoms, including improvement of eczema, and reduced severity of infections.

The article exhaustively ruled out events that have occurred in other, similar gene transfer trials in which children developed leukemias from the vector. Now the bad news. The same day NEJM published the results, American Society of Gene and Cell Therapy (the largest professional society devoted to gene transfer) released a statement saying that the German team just announced “a serious adverse event in a gene therapy trial for Wiskott-Aldrich syndrome (WAS)”- one of the ten children in the German trial developed a leukemia.

And so continues the saga of gene transfer: three steps forward, one back. (photo credit: vk-red 2009)

2010 November 29 | Leave a comment | Tags: cancer gene transfer insertional mutagenesis primary immune deficiencies

@Manual{stream2010-58,
    title = {Icarus, again: Adversity in another Gene Transfer Trial},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2010,
    month = nov,
    day = 29,
    url = {http://www.translationalethics.com/2010/11/29/icarus-again-adversity-in-another-gene-transfer-trial/}
}

Accelerated Approval: Safe at Any Speed?

Drug regulatory authorities like the FDA have a mandate to protect public health by requiring and evaluating evidence of safety and efficacy before licensing new drugs for commercial sale. But for decades now, patient advocates have argued that FDA bureaucracy kills by keeping promising drugs from the IV’s of terminal patients. In response to these criticisms, FDA and others have created new pathways for drug approval whereby drugs can be partially approved for sale on the basis of smaller, Phase 2 trials using surrogate endpoints (tumor shrinkage) instead of survival– provided drug companies confirm efficacy in subsequent trials.

2009 September 9 | Leave a comment | Tags: accelerated approval cancer

@Manual{stream2009-85,
    title = {Accelerated Approval: Safe at Any Speed?},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2009,
    month = sep,
    day = 9,
    url = {http://www.translationalethics.com/2009/09/09/accelerated-approval-safe-at-any-speed/}
}

Help Wanted, Part 2

So, what are some of the intriguing ethical questions of Kolata’s August 2d article? Here is one: when researchers conduct studies and ethics committees review protocols, resource allocation is an important consideration. If, as Kolata alleges, mediocre trials siphon eligible patients away from good trials, then there is a case to be made that IRBs and investigators need to ponder carefully the effects proposed trials will have on other studies- even when proposed trials have a favorable direct benefit-risk balance for volunteers who enter them.

2009 August 9 | Leave a comment | Tags: cancer clinical trials IRBs recruitment value

@Manual{stream2009-88,
    title = {Help Wanted, Part 2},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2009,
    month = aug,
    day = 9,
    url = {http://www.translationalethics.com/2009/08/09/help-wanted-part-2/}
}

Help Wanted- For the War on Cancer

Earlier this week (Aug 2), Gina Kolata of the NYTimes ran a fascinating story about challenges recruiting patients to cancer clinical trials. The story contains interesting facts, credible claims, analysis, and unfortunately, some misleading conjectures. The problem of patient recruitment also invites some hard headed ethical analysis.

2009 August 6 | Leave a comment | Tags: cancer clinical trials recruitment value

@Manual{stream2009-90,
    title = {Help Wanted- For the War on Cancer},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2009,
    month = aug,
    day = 6,
    url = {http://www.translationalethics.com/2009/08/06/help-wanted-for-the-war-on-cancer/}
}

Basic Science and Pharmaceutical Productivity

One of the great paradoxes of contemporary medicine has been a seeming inverse relationship between investment in basic science and registration of novel new drugs by regulatory agencies. The delay between basic science discoveries and clinical applications can be very long; many promising drug candidates are called on the basis of laboratory discovery, but few are chosen.

2009 May 13 | Leave a comment | Tags: attrition basic science cancer drug development kinase inhibitors pharmaceutical productivity Phase 1

@Manual{stream2009-102,
    title = {Basic Science and Pharmaceutical Productivity},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2009,
    month = may,
    day = 13,
    url = {http://www.translationalethics.com/2009/05/13/basic-science-and-pharmaceutical-productivity/}
}

Yellow Light on Gene Transfer Studies

Among the greatest heartbreaks in the field of gene transfer have been problems encountered in trials involving a rare, hereditary immune disorder, X-SCID (known popularly as “Bubble Boy” syndrome).  As is well known, a team of researchers based in Paris– and then in London– successfully reversed severe immunodeficiencies in 20 or so children using retroviral gene transfer starting around year 2000.  Shortly thereafter, however, the Paris team began observing rare leukemic disorders that were causally related to the gene transfer. To date, the Paris team has reported 4 cases of leukemia, with one leading to death. The London team has reported one leukemia.

2009 May 12 | Leave a comment | Tags: cancer gene transfer insertional mutagenesis lentiviral vectors RAC retroviral vectors SCID

@Manual{stream2009-103,
    title = {Yellow Light on Gene Transfer Studies},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2009,
    month = may,
    day = 12,
    url = {http://www.translationalethics.com/2009/05/12/yellow-light-on-gene-transfer-studies/}
}

Cancer, Low and Middle-Income Countries, and Translational Research

In the October 20, 2008 issue of Journal of Clinical Oncology, oncologist Zeba Aziz describes morning rounds in a cancer ward in Lahore, Pakistan. The first patient earns $20 a month and requires a combination therapy costing $10K. In the second case, a father can only pay 15% of the treatment needed by his daughter. The third case involves a single mother, who showed a significant tumor response on the first cycle of treatment but can not afford subsequent rounds.

2008 October 21 | Leave a comment | Tags: cancer LMICs

@Manual{stream2008-128,
    title = {Cancer, Low and Middle-Income Countries, and Translational Research},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2008,
    month = oct,
    day = 21,
    url = {http://www.translationalethics.com/2008/10/21/cancer-low-and-middle-income-countries-and-translational-research/}
}

In the Dark?

Most cancer patients who enter phase 1 clinical trials are motivated by the prospect of controlling their cancer. Increasingly, however, such studies, in the words of one ethicist, “take without giving in return” by involving biopsy procedures in which tissue is collected before and during the study in order to gauge whether a new drug is having intended biological effects. Such procedures are potentially burdensome, and offer no direct benefits for patients.

2008 September 11 | Leave a comment | Tags: biopsy cancer risk

@Manual{stream2008-137,
    title = {In the Dark?},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2008,
    month = sep,
    day = 11,
    url = {http://www.translationalethics.com/2008/09/11/in-the-dark/}
}

Are Phase 1 Volunteers Vulnerable?

In the January 2008 issue of Archives of Internal Medicine, an NIH team led by Christine Grady reports the results of a study (Participants in Phase 1 Oncology Research Trials: Are They Vulnerable?) surveying the demographics of patients who participate in phase 1 cancer studies. They report that these volunteers are overwhelmingly white, have pretty good physical functioning, earn above average incomes, have above average educational attainment, and do not lack for health insurance coverage. The authors then suggest that phase 1 volunteers “do not fit into any of the categories of vulnerable populations addressed by specific regulations.”

2008 July 3 | Leave a comment | Tags: cancer Phase 1 vulnerability

@Manual{stream2008-145,
    title = {Are Phase 1 Volunteers Vulnerable?},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2008,
    month = jul,
    day = 3,
    url = {http://www.translationalethics.com/2008/07/03/are-phase-1-volunteers-vulnerable/}
}