Thursday, November 14, 2013
The FDA, statistics and fundamentalism
Lemtrada had various features that made it hard - if not impossible - to design a proper double-blind test of its effectiveness. [Both Lemtrada and the counter-factual drug Rebif have substantial and identifying side-effects. The patient would know which drug they were on.]
Instead Sanofi/Genzyme designed a test in which the patients were not blind as to which drug they were taking however the people who rated their MS symptoms were blind. [This is the so-called "rater-blind" test.]
On Friday the FDA staff put out an advisory note on Lemtrada which (a) noted the bizarrely nasty side-effects of the drug, and (b) noted some anomalies in the statistics in which Sanofi/Genzyme presented to support the application for approval. This staff note was widely reported (see Bloomberg for an example) and it was thought that the drug may not be approved.
Safety was no surprise. The statistical anomalies were.
Today the FDA Panel heard the case. The best coverage came from Twitter - and I am grateful to Sarah Karlin for her wonderful Twitter account. The panel voted on many things, however I want to highlight just a couple:
Question 1: are the trials adequate and well-controlled?
Vote: 11 no, 6 yes. one abstained.
Question 2: Has the applicant provided substantial evidence of effectiveness of alemtuzumab [Lemtrada] for the treatment of patients with relapsing forms of MS?
Vote: 12 yes, 6 no.
On question #2, after the panel voted, the FDA staff asserted that if you voted no on question #1, you can't vote yes on #2.
The committee chair said they voted yes based on gut feeling which I am not sure is adequate - but the FDA staff assertion is wrong and I believe incompetent. It is perfectly acceptable to say that the trials are not "well-controlled" but there is substantial evidence of effectiveness. I present to you, yet again, the famous "parachute paper".
In that paper Gordon Smith and Jill Pell observe (with appropriate irony) that there has never been a well controlled double-blind study of parachutes as a device to "prevent death and major trauma related to gravitational challenge". Indeed because an adequate and well-controlled test is a requirement for approval parachutes are not approvable.
This is bunk.
We know why a parachute works.
We understand the physics well.
A single person who jumps out of an airplane and lands unharmed with a parachute is a convincing test that parachutes are effective.
Bluntly, you only need double-blind stats when you don't understand the physics.
The FDA staff have (as parodied in the parachute paper) raised their statistical mystique to a cult - as if it were the only valid form of epistemology. But there are other ways of knowing, they are more valid (as per the parachute example) and there is a deep underlying incompetence in the FDA cult of double-blind statistics and in the assertion highlighted above.
We are living in an age where it is possible to have deep knowledge as to how the body works. We know for instance there is a gene which codes for a protein which is necessary in the metabolism of certain selective serotonin reuptake inhibitor drugs and the presence or absence of that gene has effects. Very small sets of statistics that confirm the effects would be valid epistemology just like a very small set of statistics about people who jump out of planes with parachutes would enable you to conclude that parachutes are a good idea if you want to jump out of planes.
I read the statistics paper that the FDA staff put out on Lemtrada. It was not fun reading. It also convinced me that we should be circumspect about both our knowledge and our methods of determining what we know.
Circumspection about epistemological process however is hard for a fund manager. It is even harder for a government department.
PS. I wrote that an interest in epistemology would be used as a criteria in assessing job applicants at Bronte Capital. I was not kidding.
Disclosure: there is a traded contingent value right whose value depends on approval for and sales of Lemtrada [ticker: GCVRZ]. We own a small quantity. We also roughly doubled the position after the FDA staff paper was published.
The content contained in this blog represents the opinions of Mr. Hempton. You should assume Mr. Hempton and his affiliates have positions in the securities discussed in this blog, and such beneficial ownership can create a conflict of interest regarding the objectivity of this blog. Statements in the blog are not guarantees of future performance and are subject to certain risks, uncertainties and other factors. Certain information in this blog concerning economic trends and performance is based on or derived from information provided by third-party sources. Mr. Hempton does not guarantee the accuracy of such information and has not independently verified the accuracy or completeness of such information or the assumptions on which such information is based. Such information may change after it is posted and Mr. Hempton is not obligated to, and may not, update it. The commentary in this blog in no way constitutes a solicitation of business, an offer of a security or a solicitation to purchase a security, or investment advice. In fact, it should not be relied upon in making investment decisions, ever. It is intended solely for the entertainment of the reader, and the author. In particular this blog is not directed for investment purposes at US Persons.