A casual reader of recent media reports might be led to believe that forensic science lacks any scientific credibility. However, this narrative is completely inaccurate and at odds with the scientific excellence that exists throughout the forensic science community. Forensic disciplines are grounded in diverse sciences such as chemistry, biology, and physics, and every forensic discipline practiced in an accredited forensic laboratory must demonstrate that it is reliable, accurate, and fit for its intended use.
There are a lot of subjective and undefined terms in this paragraph. To date, a number of commissions, panels and other bodies staffed with actual scientists have reviewed some of the most common fields of forensics and found them lacking in scientific merit. These include the National Academy of Sciences, the Texas Forensic Science Commission, the President’s Council of Advisors on Science and Technology and the National Commission on Forensic Science. To say that several forensic disciplines are “grounded in diverse sciences” tells us little about whether the actual methods of analysis that the practitioners use to reach their conclusions are scientifically sound and reliable. One important test of scientific merit, for example, is repeatability. If a method of forensic analysis is scientifically reliable, two trained analysts using the same experiment with the same evidence should reliably get the same result. In a scientifically proven field such as DNA testing, two trained scientists testing the same blood, hair, semen, skin cells or other biological material will reliably produce the same DNA profile. The same can’t always be said of blood-spatter analysts, tool-mark analysts or bite-mark analysts.  One could say, for example, that blood-spatter analysis is “grounded in” physics — the physics of what a liquid such as blood does as it is forced from the body, travels through the air and strikes a wall or floor. But simply referencing some bit of scientific knowledge doesn’t mean that the end product of your analysis will be scientifically reliable. Astrologers, for example, utilize some principles and knowledge from astronomy, which is a legitimate field of science. This doesn’t mean that astrologers’ predictions are “grounded in science.” Just last week, the New York Times ran an editorial on this, based on a two-part investigation into a conviction based on blood-spatter evidence, written by Pamela Colloff.
Joe Bryan was convicted on the word of a detective named Robert Thorman, who testified before the jury as an expert in what is known as bloodstain-pattern analysis . . . People like Detective Thorman got certified as bloodstain-pattern analysts after taking a weeklong course that now costs as little as a few hundred dollars.
Pamela Colloff, who wrote the articles on the Times Magazine/ProPublica investigation of Joe Bryan’s case, enrolled in one of these courses, where the instructor told her, “We’re not really going to focus on the math and physics; it just kind of bogs things down.” Ms. Colloff passed the final exam, as did everyone in the class.
Thanks in part to such dubious standards, the interpretation of bloodstain evidence has become notoriously ambiguous. The same patterns can, like a Rorschach test, be read in very different ways; some trials feature two bloodstain “experts,” one on each side, who testify to opposite conclusions. A 2009 report by the National Academy of Sciences found that “the opinions of bloodstain-pattern analysts are more subjective than scientific,” and, “The uncertainties associated with bloodstain pattern analysis are enormous.”
And yet judges in many states have accepted these experts’ testimony as scientifically valid — not because of any concrete evidence that it is, but because other courts have accepted it before.
This of course is how courts routinely adjudicate challenges to the scientific validity of expert testimony. They look to see what other courts have done. This means that once a scientifically dubious field enters the criminal justice system, it becomes really difficult to reverse the process.
Let’s get back to Isenberg and Oien.
Accreditation and quality assurance systems assure the public that accredited organizations are competent and their results can be relied upon. Many groups — such as the National Commission on Forensic Science, the National Academy of Sciences, the President’s Council of Advisors on Science and Technology (PCAST), and the Department of Justice (DOJ)— recognize that accreditation is critically important. In fact, in December 2015, the Attorney General directed that all DOJ forensic laboratories must obtain or maintain accreditation.
About that 2015 directive. It has a pretty big loophole. It only states that federal prosecutors use accredited crime labs “when practicable.” As Frontline reported at the time, if finding an accredited lab would result in too much of a delay or too great an expense, federal prosecutors can continue to use unaccredited labs.
Accreditation is an external assessment of a laboratory’s technical competence to perform specific types of testing. Accreditation demonstrates that a laboratory is performing its work correctly and consistent with appropriate standards. To maintain this recognition, a laboratory is periodically reevaluated to ensure its ongoing compliance with accreditation requirements. Laboratory accreditation is internationally regarded as a reliable indicator of technical competence, and it provides credibility and public confidence in laboratory operations. An accredited laboratory’s quality assurance system must include written standard operating procedures, proficiency testing, training programs, processes for technical review of reports, testimony monitoring, and many other requirements.
All else being equal, some accreditation is certainly better than no accreditation. But accreditation is merely a baseline. It doesn’t ensure competency. It doesn’t ensure that a crime lab is operating within the parameters of sound science. In the end, an accreditation is only as valuable as the rigor, reputation and thoroughness of the accrediting organization. And as I’ll get to in a moment, the history of these groups leaves a lot to be desired. Accrediting bodies also typically exist within the existing culture of forensics. Typically, when we talk about accreditation, we aren’t talking about outside scientists who are assessing the scientific credibility and rigor of a crime lab and its analysts. These accrediting organizations usually begin their evaluations of crime labs from the perspective that even the more subjective and controversial fields of forensics, when applied according to the prevailing standards of those fields, are legitimate and reliable. In other words, accrediting organizations evaluate subjective fields such as blood-spatter, shoe-print, or hair-fiber analysis on their own respective terms. If you’re meeting the minimum standards put forth by the blood-spatter, shoe-print, or hair-fiber analysis community, you’re probably going to be accredited. Accreditation does not mean that these fields as a whole are scientific, credible or reliable.
According to the Bureau of Justice Statistics, 88 percent of the 409 publicly-funded forensic crime labs in the United States are accredited. Unaccredited labs are often very small — less than ten people — and offer services in a limited number of disciplines. In addition to forensic laboratories, laboratories performing other types of tests are accredited according to the same international standard. This includes environmental labs checking for levels of lead in groundwater, chemistry labs preparing chemicals for consumer use, or food labs ensuring the safety of our food supply.
This is a strange comparison. The fact that a crime lab is subjected to similar standards as labs that perform other types of tests says nothing about the validity of the crime lab’s analysis.
But let’s look at that 88 percent figure. It seems impressive. But it seems impressive only until you start to look at the seemingly endless parade of crime lab scandals we’ve seen over the past 15 to 20 years, and you begin to notice how many of those labs . . . were already accredited.
Here’s a quick list of such cases, which isn’t remotely comprehensive:
  • Last October, Massachusetts officials fired the head of the Office of Alcohol Testing within the Massachusetts State Police Crime Laboratory after discovering that analysts at the lab routinely withheld exculpatory information about blood-alcohol tests from defense attorneys. The revelation could affect thousands of drunk-driving tests dating back to 2011. On its website, the Massachusetts State Police Crime Laboratory describes itself as “fully accredited.”
  • In 2016, police officials in Austin shut down the DNA division of the city’s police-run crime lab indefinitely after discovering that analysts were using analytical tools that were badly outdated. The lab also had problems with cross-contamination. Despite multiple red flags about the lab dating back to 2008, the lab was repeatedly accredited by the appropriate bodies within the field of forensics. It took an inspection by the Texas Forensic Science Commission — an oversight body from outside the forensics world — to finally bring these problems to light.
  • In 2013, the New York City Medical Examiner’s Office underwent a review of 800 rape cases after investigators found 26 instances in which DNA evidence was mishandled or overlooked by an analyst at the lab. The cases spanned 2001 to 2011. In fact, the same year that the city’s Medical Examiner’s Office announced this review, it also boasted in a news release that its DNA lab received “perfect scores” from an international accrediting organization.
  • In their various incarnations, the crime labs in Houston have been the scene of multiple scandals, including 2014 revelations that an analyst had tampered with evidence; a 2005 report finding incompetence, cheating on proficiency tests, and even possible perjury and faking of test results; and the shuttering of the fingerprint lab in 2009 after an audit showed exceptionally high error rates. (This is not at all a comprehensive list of the crime lab problems in Houston.) Through all of this, the Harris County and Houston labs have boasted of accreditation from both national and international forensic organizations. The American Society of Crime Laboratory Directors, for example, accredited the Houston PD lab in 2006. Incredibly, when a 2008 investigation into cheating on proficiency tests led to the closing of the lab’s DNA testing unit, the resignation of the head of that unit and the suspension of two other analysts, the head of the organization that accredited the lab said his group had no plans to revoke the lab’s accreditation.
  • A 2013 audit of the crime lab in St. Paul, Minn., found widespread ineptitude, including “sloppy documentation, dirty equipment, faulty techniques and ignorance of basic scientific procedures.” The report recommended that the lab cease operations until it could be overhauled from the ground up. Yet in its 2007, 2008, 2009 and even 2012 annual reports, the Minnesota Department of Public Safety touted the St. Paul crime lab’s accreditation by the American Society of Crime Lab Directors. In 2009, three years before the scandal broke, crime lab director Frank Dolejsi boasted that the accreditation “is an affirmation of the quality of all aspects of the [crime lab’s] forensic science services.” After the audit, state officials vowed to win back public trust by promising that the St. Paul crime lab would — you guessed it — get itself accredited.
  • In 2014, a crime lab analyst in Florida was arrested for stealing seized drugs and tampering with evidence, possibly tainting up to 2,600 cases from at least 80 different law enforcement agencies. His lab was accredited.
  • In 2012, an audit of the North Carolina crime lab done after an exoneration found that analysts in the serology unit had been withholding exculpatory evidence from defense attorneys for at least 16 years. The audit found that analysts routinely overstated their findings and that training manuals were adamantly pro-prosecution, to the point of referring to defense experts as “whores.” The manuals were likely pro-prosecution because the analysts themselves reported to prosecutors, who wrote their year-end reviews and determined raises and promotions. The North Carolina lab had been accredited since 1988.
These are just a few examples. In 2013, the ABA Journal reviewed dozens of crime lab scandals across the country. The report found a lot of problems in both accredited and non-accredited labs. Of course, there’s Isenberg and Oien’s own FBI crime lab, which is arguably the most accredited and respected crime lab in the world. But the FBI lab, too, has been no stranger to scandal. Just a few years ago, the agency admitted that it’s hair-fiber analysts had overstated their findings nearly every time they testified. This occurred over a period spanning decades and implicated thousands of convictions. Those analysts then trained analysts at state and local labs, potentially corrupting untold thousands of more cases. And that’s just one scandal of several. There’s the Brandon Mayfield debacle. There’s the FBI’s long-held and since disproved claim about “compositional bullet lead analysis,” which also affected thousands of criminal cases. That’s quite a record from one of the most respected labs in the world. Back to Isenberg and Oien.
Forensic examiners must complete extensive training to be qualified to perform casework in accredited laboratories. Training programs can be one to two years, or longer, and require examiners to perform analyses on samples with a known correct answer. The examiner must also demonstrate a thorough understanding of the science behind the method employed and an understanding of lab policies, procedures, legal rules, evidence handling, etc. The examiner must undergo oral examinations, mock trials, and competency tests for which the correct answer is known. In addition, all examiners must demonstrate competency to apply the processes accurately and reliably before they are assigned actual cases. Once qualified to conduct casework in an accredited laboratory, every examiner undergoes continual competency monitoring through proficiency tests administered at least once per year.
This is difficult to square with the examples above, or with the other scandals at accredited labs far too numerous to list in their entirety. The most common credentialing agency for crime labs is the aforementioned American Society of Crime Laboratory Directors (ASCLD). Isenberg and Oien even cite this organization’s website in laying out the allegedly rigorous process to becoming an accredited forensic examiner. But while ASCLD may claim that these are the criteria crime labs need to meet to earn and keep their credentials, those requirements don’t appear to be all that tightly enforced. Labs that don’t meet them still get accredited. And previously accredited labs that don’t meet them don’t seem to lose their status.
If the accreditation process is so rigorous and demanding, critics wonder, then why have so few labs been sanctioned? ASCLD/LAB’s website lists the status of all accredited labs and shows that no lab’s accreditation is currently revoked or suspended; there are also no labs on probation. And [ASCLD Executive Director Ralph] Keaton says he can count on one hand the number of labs whose accreditation has ever been revoked or suspended, though he says it would probably take two hands to count the number of labs that have ever been placed on probation.
Keaton says that has a lot to do with the overall quality of accredited labs. But critics say it has more to do with the chummy nature of the inspection process, which creates a tendency to “go along to get along” among inspectors, and the agency’s own interest in keeping labs accredited. . . .
New York City criminal defense lawyer Marvin Schechter, a member of the committee that produced the NAS report, is one of ASCLD/LAB’s biggest critics. Schechter, also a member of the New York State Commission on Forensic Science, wrote a lengthy memo to his fellow commissioners in 2011 recommending that they look for a new accreditor. He characterized ASCLD/LAB as an organization more interested in protecting its members’ images than in promoting accountability.
“In fact, ASCLD/LAB could more properly be described as a product service organization,” Schechter wrote, “which sells for a fee a ‘seal of approval’ covering diverse laboratory systems, which laboratories can utilize to bolster their credibility through in-court testimony by technicians, plus ancillary services such as protection from outside inquiry, shielding of internal activities and, where necessary, especially in the event of public condemnation, a spokesperson to buffer the laboratory from media inquiry.”
The links to the pages listing revoked/suspended labs and labs on probation now go to a site called the ANSI-ANQ National Accreditation Board, which apparently merged with ASLD’s accrediting arm. But neither website appears to lists any crime labs on probation, currently in suspension, or whose accreditation has been revoked. If even egregious incompetence and misconduct rarely if ever results in a revocation or suspension, what does accreditation really mean?
More from Isenberg and Oien: 
Testimony monitoring is also a requirement for accredited laboratories. The Federal Bureau of Investigation (FBI) Laboratory requires that examiners request a transcript for each testimony provided. FBI examiners also must follow approved standards for scientific testimony and reports, which document the acceptable range of conclusions expressed in both laboratory reports and testimony. The DOJ is developing similar documents called Uniform Language for Testimony and Reports, as well as a testimony-monitoring framework, which will apply to all DOJ laboratories. The purpose of these testimony-monitoring activities is to prevent examiner testimony from exceeding scientific limitations.
This sounds great at first. But it’s all dependent on the answers to several important questions. First, who determines the “acceptable range of conclusions” for lab reports and testimony? Is it someone who already practices in that particular field? What does “approved standards for scientific testimony” mean? Who is setting those standards? Who is approving them? Who decides if an analyst has “exceeded scientific limitations?” The entire point of the National Commission on Forensic Science empaneled by President Barack Obama was to bring in actual scientists to make such determinations. Attorney General Jeff Sessions allowed that commission’s charter to expire shortly after taking office. (It’s worth noting that while Obama deserves credit for setting up the commission, his record in this area was mixed at best.) Last August, Deputy Attorney General Rod J. Rosenstein announced a new forensics review and monitoring process at DOJ. He also revealed that the new system would be overseen by a man named Ted Hunt. Ted Hunt is not a scientist. He is a longtime prosecutor. His supporters point out that he was part of the very Forensic Science Commission that Obama empaneled and that reformers lament was allowed to expire. But Hunt was often a voice of dissent on the commission. Here’s a look at Hunt’s record on the commission from Mother Jones:
In March 2016, the commission recommended that then-Attorney General Loretta Lynch direct forensic experts and attorneys working on behalf of the Justice Department to stop using the phrase “to a reasonable degree of scientific certainty.” The phrase is commonly used on witness stands and in lab reports and gives juries and judges a sense of factuality, but it is subjective and lacks any agreed-upon meaning across the sciences. Hunt was one of two commission members who opposed the recommendation, which Lynch adopted last September.
Lynch also adopted a recommendation by the commission requiring forensic testing labs that work with the department and its attorneys to publicize their internal procedures, from equipment maintenance to estimations of uncertainty, in order to foster transparency, trust, and best practices in the industry. Hunt was one of four commissioners who opposed it.
Last September, when the commission released a document supporting stricter accreditation standards for forensic labs, Hunt voted against it. And when the commission recommended that the National Institute of Standards and Technology conduct scientific evaluations of the “technical merit of test methods and practices used in forensic science disciplines,” he opposed that, too. At its final meeting, when members had already been informed that the group would be coming to an end, several commissioners pushed for a resolution encouraging experts to use more quantitative language to convey the accuracy of forensic testimony. The resolution narrowly failed, with Hunt among the nays.
So DOJ did away with a transparent, external commission mostly populated and led by scientists, and replaced it with an internal commission led by a career prosecutor who has opposed efforts to increase transparency, increase accountability, increase scientific accuracy and strengthen the requirements for accreditation. This is likely why Hunt’s appointment was hailed by ASCLD and forensics groups such as the International Association for Identification and the American Academy of Forensic Sciences, and why the new DOJ venture has been viewed with skepticism by groups such as the Innocence Project. In the remainder of their essay, Isenberg and Oien take aim at the report by the President’s Council of Advisors on Science and Technology (PCAST) issued in the waning months of the Obama administration. That report was perhaps the most strident and explicitly critical report on forensics to date. The authors of and researchers for the report were scientists, not forensics practitioners or lawyers (though a panel of judges and attorneys did contribute in an advisory role).
The essay’s discussion of the PCAST report is a bit technical to delve into here with too much detail, but in general, the PCAST authors posited that because most pattern-matching fields of forensics are so subjective — an analyst basically just “eyeballs” two samples and comes up with language to describe how similar or different they are — the only way to determine if these fields are reliable is what’s known as a “black box” test. We can’t evaluate a particular analyst’s process for determining matches because so much of it is done in his or her head. Instead, we have to look at results. That means administering competency tests. Give analysts a sample from a case in which the culprit is known, give them samples from multiple suspects, and record who gets it right and who doesn’t. As you might guess, many forensic analysts and the professional groups they belong to are reluctant to submit to such tests. Most of these fields have already been accepted by the courts. They have little to gain, and a lot to lose. In their essay, Isenberg and Oien dispute the notion that black-box tests are the only real way to measure the reliability of a given field. They argue that relying too much on such tests “ignores much peer-reviewed research, overlooks critical aspects of many studies, and fails to acknowledge the empirical value of these studies.” But these studies are, again, typically reviewed only by other practitioners from the same field, using the same procedures and standards within the field. Here again, you’re evaluating the legitimacy of an entire field based on criteria that already accepts that field’s legitimacy. In conclusion, Isenberg and Oien write:
Science continuously evolves and is built upon observation, study, and experience that spans hundreds of years. The justice system would not be well served by the exclusion of reliable forensic methods and techniques that provide valuable information to a wide range of stakeholders.
Of course, whether some of these fields are reliable is precisely what’s in dispute. But at issue here is how we evaluate new or questionable expertise for use in the courtroom. What do we do going forward? I think the answer is that we only allow this sort of analysis and expertise after it has been subjected to the rigors of scientific inquiry. Before we let a new drug hit the market, we require its manufacturer to prove that the drug is safe and effective. Those standards are enforced by scientists with expertise in pharmaceuticals. When a drug slips through and causes harm, we pull it from the market. It isn’t a perfect system, and there are interesting debates at its parameters. But in general, we test new technology before we put it to everyday use. Contrast this to forensics. The decision whether to allow a new field of forensics into court is made by a judge, not a scientist, or even a fellow practitioner. Judges typically look for guidance on these questions not from scientists, but from other judges. The briefs in such challenges are written by lawyers. Judges then tend to err on the side of letting evidence in, on the assumption that our adversarial system will sort it out. (In his speech last year, Rosenstein touted this flaw as a feature.) Even once we discover that a field is scientifically suspect, it’s difficult to get the courts to even acknowledge it, much less stop it from being used again, much less correct the cases that may have already been tainted.
Like the other defenders of these fields, Isenberg and Oien say we should simply trust the internal procedures at FBI and DOJ to get forensics right. Even at face value, that’s a difficult argument to accept. FBI agents and federal prosecutors aren’t evaluated for their allegiance to scientific principles. They’re evaluated on their ability to close cases and win convictions. Therefore, there’s a strong incentive for them to sacrifice sound science for expedience. That incentive will affect even the most conscientious and fair-minded at DOJ. But the case Isenberg and Oien are making is even harder to accept given the history of these disciplines and the ongoing onslaught of forensics and crime lab scandals. “Just trust us” hasn’t worked in the past. Why should we think it would work now?"