In a stunning turn of events, Theranos, the company founded by Stanford University dropout Elizabeth Holmes and hailed as having one of the top 10 medical and technological innovations in 2013,1 has come under investigation by federal prosecutors, the Centers for Medicare and Medicaid Services (CMS), and the Securities and Exchange Commission. This is a story of science versus hype, and a seemingly disruptive technology that has yet to deliver on its promises. Although readers may be acquainted with more than a decades’ worth of news stories that have recounted the highs and lows of Theranos’ prospects (Table), herein we present what you may not know about the science (or lack thereof) which lies at the heart of the company’s imbroglio.
A PubMed search for peer-reviewed articles reveals two articles coauthored by Theranos,2,3 which contain no descriptions of the technology. A search with the U.S. Patent and Trademark Office reveals a number of microfluidics patents under the name of Ms. Holmes4-7 ; however, without independent evaluation, the accuracy, precision, and specificity of these new laboratory tests are unknown to anyone outside of the company. It is also unclear whether tests using this new technology are actually occurring. Early reports describe individuals receiving standard venipunctures,8 and a review of the CMS inspection report of Theranos’ Newark laboratory finds descriptions of standard laboratory equipment for testing of various analytes.9
Digging Deeper into Theranos’ Laboratory Testing
Seeking clarity about the performance of “low-volume” laboratory tests versus standard blood draws in the arena of direct-to-consumer testing, Drs. Brian Kidd and Joel Dudley and colleagues from the Icahn School of Medicine at Mount Sinai in New York City conducted a study of 60 healthy adults, comparing 22 common clinical laboratory analytes (complete blood count and differential, lipid panel, high-sensitivity CRP, serum phosphate, serum uric acid, and total bilirubin).10 The samples were collected in Phoenix during a five-day period, with all blood drawn from each patient in a single day within a 6.5-hour window. The study design allowed for a total of 14 samples per patient (four separate blood draws with the first and fourth draws divided into six tubes each and sent to LabCorp (Burlington, NC) and Quest Diagnostics (Madison, NJ); second and third blood draws were each collected from two separate retail locations and sent to Theranos). Thus, LabCorp and Quest Diagnostics had six replicates from each patient, and Theranos had two replicates. Testing of samples at LabCorp was performed at Accupath Diagnostics Lab in Phoenix, Arizona; Quest Diagnostics samples were tested at Sonora Quest in Tempe, Arizona; and lipid panels were tested at Quest Diagnostics Nichols Institute in San Juan Capistrano, California. Theranos samples were collected and processed on site in Phoenix before shipping to Newark, California, for testing. All samples were shipped to facilities within 9.5 hours, which is typical for reference laboratories.
Does it matter where laboratory tests are performed? Yes! To clarify, for clinical laboratory testing, there are preanalytic, analytic, and postanalytic variables. The authors sought to minimize preanalytic variables to focus primarily on analytic test variables. They in turn found missing data rates of 2.2 percent (Theranos), 0.2 percent (LabCorp), and 0 percent (Quest Diagnostics) based on 2,640 (Theranos) and 7,920 (LabCorp and Quest) possible measurements. The investigators calculated that the odds for Theranos missing a measurement due to a technical error, versus other laboratories, is 12.5:1, based on the missing data rates quoted above (95% CI, 6.9-22.4; p=1.5 × 10–22). The percentage of measurements outside their normal reference range was 12.2 percent (Theranos), 8.3 percent (LabCorp), and 7.5 percent (Quest). Theranos tests flagged outside their normal range 1.6 times more often than the other laboratories. Mean corpuscular hemoglobin concentration, lymphocyte count, and cholesterol (total, high-density lipoprotein, low-density lipoprotein) are examples of tests that more often flagged results outside of their reference ranges. Notably, 15 of 22 laboratory measurements showed significant differences between Theranos and the other two clinical services (p<0.002).
Regulations on clinical laboratory testing are among the most strenuous of any industry and are meant to control variability in testing, to ensure the highest level of patient care. CMS regulates clinical laboratory testing based on guidelines outlined in the Clinical Laboratory Improvement Amendments (CLIA).11,12 These guidelines include mandatory participation in proficiency testing from a CMS-approved program using homogenous samples distributed to laboratories.12 CLIA, or an accrediting agency, determine acceptable criteria, including the total analytic error (TAE; method bias + total imprecision) for each analyte.12 In the article by Dr. Brian A. Kidd and colleagues, the authors state that a TAE of +/– 10 percent would be exceeded by Theranos for cholesterol measurements once instrument imprecision is considered. Increased abnormal tests have consequences, including extra testing (and costs), extra visits to hospitals and clinics, and increased health-care service, which can potentially harm patients.13
Numerous questions remain about why testing by Theranos showed such unexpected variability in healthy adults. A detailed examination of the CMS inspection of the Newark laboratory, performed on November 20, 2015, reveals some potential answers.9 This 121-page document outlines numerous deficiencies that highlight a basic lack of understanding of clinical laboratory testing. Failed freezer temperatures; expired reagents; no training documentation; failure of the lab director to sign, date, or approve procedures prior to use; and proficiency testing failures with no investigation for multiple analytes are just a few of the findings.9 The fact that the original laboratory director was a dermatologist with no background in laboratory medicine should serve as a cautionary tale.14 While one might think that Theranos’ board of directors could lend insight into the business of laboratory testing, a review of its members reveals an impressive group of politicians, military figures, and accomplished individuals who have served on the board, but not a single laboratory medicine physician or clinical pathologist. This was addressed in April 2016 when a scientific and medical advisory board was added to Theranos, with a well-qualified group of laboratory professionals.15
The (Unknown) Science Behind Theranos
The science behind the technology driving the laboratory testing at Theranos remains elusive. Naturally, this has raised concern among physicians, given the potential effects of such untested technologies on patient safety.13 In the absence of any form of peer review of the science behind Theranos, it has become very difficult for pathologists to provide an informed opinion to clinicians and patients about the validity of the test results generated by the company. Publicly available sources such as newspaper articles and patent applications are no substitute for peer-reviewed scientific literature.
The publicly available facts regarding the testing pipeline at Theranos describe the use of a finger-stick method to obtain the patient’s blood sample into a patented collection device known as the “nanotainer.”7,16 The sample is then loaded onto a proprietary analysis machine code-named “Edison,”17 of which no specific information is publicly available. It is also believed that the sample is analyzed internally on the machine based on assay principles that also currently remain unknown.13 The data obtained are then wirelessly transmitted to a secure database with a portal interface from which the physician and the patient can then retrieve results.13
Table. Theranos Timeline
| 2003 | Company founded by Elizabeth Holmes |
| 2004 | Theranos raised $6.9M |
| 2010 | Theranos raised $45M from single investor |
| 2012 | FDA launches investigation after complaint from Department of Defense official evaluating Theranos technology. |
| 2013 | Walgreens and Theranos announce partnership. Theranos technology hailed as one of the top 10 medical and technological innovations in 2013.1 |
| 2014 | Theranos raised $400M, valued at $9B with Holmes owning 50 percent of company15 |
| 2015 | Theranos lobbies for and wins approval in Arizona for a law allowing patients to order their own blood testing. |
| October 2015 | Herpes simplex virus type 1 test approved by U.S. Food and Drug Administration (FDA; 261 other tests remain unapproved). |
| October 2015 | FDA calls the nanotainer an "uncleared" medical device.17 |
| November 20, 2015 | Centers for Medicare and Medicaid Services (CMS) inspects Newark, California, laboratory and finds numerous deficiencies in laboratory testing. These deficiencies could have posed "immediate jeopardy to patient health and safety."9 |
| February 12, 2016 | Theranos responds to CMS inspection. |
| February 29, 2016 | Walgreens looks to cut ties with Theranos. |
| March 28, 2016 | Kidd BA et al. article in the Journal of Clinical Investigation available online |
| April 7, 2016 | Theranos adds medical board.14 |
| April 13, 2016 | News surfaces of CMS proposing a two-year ban for Holmes from owning and/or operating a laboratory, suspending MediCare/Medicaid payments and revoking Clinical Laboratory Improvement Ammendments license from Newark, California, laboratory in a March 18 letter. Theranos had 10 days to respond; the response was not made public. No comment from CMS. |
| April 19, 2016 | Theranos investigated by federal prosecutors, by CMS, and by the Securities and Exchange Commission |
| July 8, 2016 | Federal regulators bar Elizabeth Holmes from owning or operating a medical laboratory for at least two years; certification of Newark, California laboratory revoked (effective September 5, 2016); laboratory also prohibited from taking Medicare and Medicaid payments. |
| August 1, 2016 | Elizabeth Holmes presents at the American Association of Clinical Chemistry, highlighting the "miniLab" as the new iteration of Theranos' miniaturization technology. |
| 2003 | Company founded by Elizabeth Holmes |
| 2004 | Theranos raised $6.9M |
| 2010 | Theranos raised $45M from single investor |
| 2012 | FDA launches investigation after complaint from Department of Defense official evaluating Theranos technology. |
| 2013 | Walgreens and Theranos announce partnership. Theranos technology hailed as one of the top 10 medical and technological innovations in 2013.1 |
| 2014 | Theranos raised $400M, valued at $9B with Holmes owning 50 percent of company15 |
| 2015 | Theranos lobbies for and wins approval in Arizona for a law allowing patients to order their own blood testing. |
| October 2015 | Herpes simplex virus type 1 test approved by U.S. Food and Drug Administration (FDA; 261 other tests remain unapproved). |
| October 2015 | FDA calls the nanotainer an "uncleared" medical device.17 |
| November 20, 2015 | Centers for Medicare and Medicaid Services (CMS) inspects Newark, California, laboratory and finds numerous deficiencies in laboratory testing. These deficiencies could have posed "immediate jeopardy to patient health and safety."9 |
| February 12, 2016 | Theranos responds to CMS inspection. |
| February 29, 2016 | Walgreens looks to cut ties with Theranos. |
| March 28, 2016 | Kidd BA et al. article in the Journal of Clinical Investigation available online |
| April 7, 2016 | Theranos adds medical board.14 |
| April 13, 2016 | News surfaces of CMS proposing a two-year ban for Holmes from owning and/or operating a laboratory, suspending MediCare/Medicaid payments and revoking Clinical Laboratory Improvement Ammendments license from Newark, California, laboratory in a March 18 letter. Theranos had 10 days to respond; the response was not made public. No comment from CMS. |
| April 19, 2016 | Theranos investigated by federal prosecutors, by CMS, and by the Securities and Exchange Commission |
| July 8, 2016 | Federal regulators bar Elizabeth Holmes from owning or operating a medical laboratory for at least two years; certification of Newark, California laboratory revoked (effective September 5, 2016); laboratory also prohibited from taking Medicare and Medicaid payments. |
| August 1, 2016 | Elizabeth Holmes presents at the American Association of Clinical Chemistry, highlighting the "miniLab" as the new iteration of Theranos' miniaturization technology. |
The main advertising claim of Theranos is the stated fear of the founder regarding venipuncture needles used in routine laboratory testing.16 The company patented the “nanotainer,” a 0.5-inch container device capable of storing one to two drops of blood.16 This was advertised as a revolutionary method to obtain blood samples in a painless fashion to overcome the popular fear of venipuncture.16 Additionally, the low volumes of blood required to test samples were also advertised as a major selling point of the technology (“A tiny drop is all we need”). One popular news report claimed that Theranos needed 1/100 to 1/1,000 of the volume of regular laboratory testing.16 In the same article from 2014, it was reported that Theranos performed as many as 70 different laboratory tests from a single draw of 25 to 50 µL of blood obtained from the nanotainer.16 Subsequent news articles claimed a different number (approximately 30 tests) could be run on this blood volume. In October 2015, the U.S. Food and Drug Administration (FDA) called the nanotainer an “uncleared medical device” (Table).18 Subsequently, Theranos claimed to have stopped using the device for all of its tests except for the single FDA-cleared herpes virus test.17 The current status of the nanotainer in routine Theranos testing methodology remains unknown.
A cursory evaluation of the testing menu on the Theranos website shows availability of approximately 240 tests. However, there are no references to the exact collection method used in testing on the website. A review of the offered tests reveals that the testing menu at Theranos falls mostly into four different categories, including clinical chemistry, immunology, DNA testing, and cell-based assays. Current point-of-care testing (POCT) methods use finger-stick–based techniques to obtain the necessary blood samples to enable POCT testing in clinics and homes. However, POCT tests represent a limited subset among the menu of tests available through routine laboratory testing needed in day-to-day clinical practice. A majority of laboratory testing uses venipuncture-based access to obtain the required blood samples. The test validation performed by laboratory scientists also relies on the use of venipuncture-based sampling methods to ensure repeatable results.
Pertinent to this issue of sample collection (venipuncture vs. finger stick), a recent article by Dr. Meaghan M. Bond and colleagues19 examined the issue of drop-to-drop variation in the cellular components of blood. The authors analyzed the drop-to-drop variability of standard laboratory tests including hemoglobin, WBC count, WBC differential, and platelet count in six successive drops of blood collected from a single finger stick from 11 different donors.19 For example, the authors observed a significant difference in the coefficient of variation of the platelet counts obtained from finger stick compared with venipuncture (19% vs. 4.8%).19 This study illustrates the drop-to-drop variability of the finger-stick method for easy-to-perform laboratory tests. To our knowledge, no such analysis of the impact of finger-stick sampling on the other classes of tests offered on the Theranos menu (clinical chemistry, immunology, and DNA-based testing) has been performed. We believe a thorough peer-reviewed scientific analysis of finger stick sample reliability for analyte detection is critically important before it can be offered to patients in routine practice.
In the current discussion, we have focused on the preanalytic components of the testing process such as sample collection and reliability. Equally pertinent are the analytic and post-analytic components of any laboratory testing process. In the absence of reliable peer-reviewed data, it is difficult to determine the efficacy of the microfluidic-based testing approach that Theranos claims to use.
The idea of a “lab-on-a-chip” is not a new one. Groups around the world have been working throughout the past two decades to develop such technologies, with varying degrees of success. A problem inherent to many microfluidic-based approaches is the lack of a scalable standardization of the microfluidic manufacturing processes, as well as the lack of reusability of the microfluidic chips. These two factors increase the costs of microfluidic technology for routine clinical implementation. Laboratory medicine surely could benefit from innovative approaches such as microfluidic-based testing. However, these technologies should be implemented in clinical testing only after extensive peer review of the testing methodology. A previous College of American Pathologists study estimated that 70 percent of a clinical decision is directly correlated with the laboratory results provided, illustrating the seriousness with which laboratory testing needs to be taken. As this edition of The Hematologist goes to press, Elizabeth Holmes gave a highly anticipated invited presentation at the annual meeting of the American Association of Clinical Chemistry on August 1, 2016. At this first scientific conference for the Theranos CEO, healthy skepticism abounded as to whether substantive data would be presented. AACC President Patricia Jones echoed this in her introduction of Elizabeth Holmes: “We're all aware that there have been some questions about whether we will see any science here today and the viability of Theranos' technology.” In fact, the presentation lacked any details regarding the original “Edison” testing platform. Instead, a new device, the “miniLab” was introduced to the public, with preliminary information provided about the inner components of the machine and an overview of data on precision and comparison studies. In the authors' opinion, there is nothing unique about the spectrophotometer, luminometer, and basic hematology technology at the heart of the new miniLab device, which represents a miniaturization of various detection modules used in a standard clinical laboratory. The isothermal amplification used for the molecular biology assays and the slide-based cytometer used for assessing lymphocyte subsets are newer technologies that have not been robustly tested in the clinical laboratory. While data presented appear to be robust and follow appropriate guidelines, the performance characteristics of this device at microfluidic volumes in real-world clinical applications are still uncertain and require validation. Thus, the aforementioned criticisms still stand and must be addressed in a thorough, transparent, and comprehensive manner. Theranos seems to be taking steps to redress multiple shortcomings of their original testing model, including the need for mandatory scientific peer-review. The introduction of a new scientific and medical advisory board,14 should also provide much-needed guidance. The ongoing travails of Theranos serve as an important case study in the need for physicians to have a firm understanding of the scientific principles of laboratory testing in order to provide the best level of health care for patients.
References
Competing Interests
Dr. George and Dr. Gullapalli indicated no relevant conflicts of interest. Editor-in-Chief Dr. Jason Gotlib collaborated with Theranos on a pilot study whose results were published in 2014 (see reference 2). He received no funding.
