A
casual reader of recent media reports might be led to believe that
forensic science lacks any scientific credibility. However, this
narrative is completely inaccurate and at odds with the scientific
excellence that exists throughout the forensic science community.
Forensic disciplines are grounded in diverse sciences such as chemistry,
biology, and physics, and every forensic discipline practiced in an
accredited forensic laboratory must demonstrate that it is reliable,
accurate, and fit for its intended use.
There
are a lot of subjective and undefined terms in this paragraph. To
date, a number of commissions, panels and other bodies staffed with
actual scientists have reviewed some of the most common fields of
forensics and found them lacking in scientific merit. These include the National Academy of Sciences, the Texas Forensic Science Commission, the President’s Council of Advisors on Science and Technology and the National Commission on Forensic Science. To
say that several forensic disciplines are “grounded in diverse
sciences” tells us little about whether the actual methods of
analysis that the practitioners use to reach their conclusions are
scientifically sound and reliable. One important test of scientific
merit, for example, is repeatability. If a method of forensic analysis
is scientifically reliable, two trained analysts using the same
experiment with the same evidence should reliably get the same
result. In a scientifically proven field such as DNA testing, two
trained scientists testing the same blood, hair, semen, skin cells or
other biological material will reliably produce the same DNA profile.
The same can’t always be said of blood-spatter analysts, tool-mark
analysts or bite-mark analysts. One
could say, for example, that blood-spatter analysis is “grounded in”
physics — the physics of what a liquid such as blood does as it is
forced from the body, travels through the air and strikes a wall or
floor. But simply referencing some bit of scientific knowledge doesn’t
mean that the end product of your analysis will be scientifically
reliable. Astrologers, for example, utilize some principles and
knowledge from astronomy, which is a legitimate field of science. This
doesn’t mean that astrologers’ predictions are “grounded in science.”
Just last week, the New York Times
ran an editorial on this, based on
a two-part investigation into a conviction based on blood-spatter evidence, written by Pamela Colloff.
Joe
Bryan was convicted on the word of a detective named Robert Thorman,
who testified before the jury as an expert in what is known as
bloodstain-pattern analysis . . . People like Detective Thorman got
certified as bloodstain-pattern analysts after taking a weeklong course
that now costs as little as a few hundred dollars.
Pamela
Colloff, who wrote the articles on the Times Magazine/ProPublica
investigation of Joe Bryan’s case, enrolled in one of these courses,
where the instructor told her, “We’re not really going to focus on the
math and physics; it just kind of bogs things down.” Ms. Colloff passed
the final exam, as did everyone in the class.
Thanks
in part to such dubious standards, the interpretation of bloodstain
evidence has become notoriously ambiguous. The same patterns can, like a
Rorschach test, be read in very different ways; some trials feature two
bloodstain “experts,” one on each side, who testify to opposite
conclusions. A 2009 report by the National Academy of Sciences found
that “the opinions of bloodstain-pattern analysts are more subjective
than scientific,” and, “The uncertainties associated with bloodstain
pattern analysis are enormous.”
And yet judges
in many states have accepted these experts’ testimony as scientifically
valid — not because of any concrete evidence that it is, but because
other courts have accepted it before.
This
of course is how courts routinely adjudicate challenges to the
scientific validity of expert testimony. They look to see what other
courts have done. This means that once a scientifically dubious field
enters the criminal justice system, it becomes really difficult to
reverse the process.
Let’s get back to Isenberg and Oien.
Accreditation
and quality assurance systems assure the public that accredited
organizations are competent and their results can be relied upon. Many
groups — such as the National Commission on Forensic Science, the
National Academy of Sciences, the President’s Council of Advisors on
Science and Technology (PCAST), and the Department of Justice (DOJ)—
recognize that accreditation is critically important. In fact, in
December 2015, the Attorney General directed that all DOJ forensic
laboratories must obtain or maintain accreditation.
About that 2015 directive. It has a pretty big loophole. It only states that federal prosecutors use accredited crime labs “when practicable.” As Frontline reported at the time,
if finding an accredited lab would result in too much of a delay or too
great an expense, federal prosecutors can continue to use unaccredited
labs.
Accreditation
is an external assessment of a laboratory’s technical competence to
perform specific types of testing. Accreditation demonstrates that a
laboratory is performing its work correctly and consistent with
appropriate standards. To maintain this recognition, a laboratory is
periodically reevaluated to ensure its ongoing compliance with
accreditation requirements. Laboratory accreditation is internationally
regarded as a reliable indicator of technical competence, and it
provides credibility and public confidence in laboratory operations. An
accredited laboratory’s quality assurance system must include written
standard operating procedures, proficiency testing, training programs,
processes for technical review of reports, testimony monitoring, and
many other requirements.
All
else being equal, some accreditation is certainly better than no
accreditation. But accreditation is merely a baseline. It doesn’t ensure
competency. It doesn’t ensure that a crime lab is operating within the
parameters of sound science. In the end, an accreditation is only as
valuable as the rigor, reputation and thoroughness of the accrediting
organization. And as I’ll get to in a moment, the history of these
groups leaves a lot to be desired. Accrediting
bodies also typically exist within the existing culture of forensics.
Typically, when we talk about accreditation, we aren’t talking about
outside scientists who are assessing the scientific credibility and
rigor of a crime lab and its analysts. These accrediting organizations
usually begin their evaluations of crime labs from the perspective that
even the more subjective and controversial fields of forensics, when
applied according to the prevailing standards of those fields, are
legitimate and reliable. In other words, accrediting organizations
evaluate subjective fields such as blood-spatter, shoe-print, or
hair-fiber analysis on their own respective terms. If you’re meeting the
minimum standards put forth by the blood-spatter, shoe-print, or
hair-fiber analysis community, you’re probably going to be accredited.
Accreditation does not mean that these fields as a whole are scientific, credible or reliable.
According
to the Bureau of Justice Statistics, 88 percent of the 409
publicly-funded forensic crime labs in the United States are accredited.
Unaccredited labs are often very small — less than ten people — and
offer services in a limited number of disciplines. In addition to
forensic laboratories, laboratories performing other types of tests are
accredited according to the same international standard. This includes
environmental labs checking for levels of lead in groundwater, chemistry
labs preparing chemicals for consumer use, or food labs ensuring the
safety of our food supply.
This
is a strange comparison. The fact that a crime lab is subjected to
similar standards as labs that perform other types of tests says nothing
about the validity of the crime lab’s analysis.
But let’s look at that 88 percent figure. It seems
impressive. But it seems impressive only until you start to look at the
seemingly endless parade of crime lab scandals we’ve seen over the past
15 to 20 years, and you begin to notice how many of those labs . . . were already accredited.
Here’s a quick list of such cases, which isn’t remotely comprehensive:
- Last October,
Massachusetts officials fired the head of the Office of Alcohol Testing
within the Massachusetts State Police Crime Laboratory after
discovering that analysts at the lab routinely withheld exculpatory
information about blood-alcohol tests from defense attorneys. The
revelation could affect thousands of drunk-driving tests dating back to
2011. On its website, the Massachusetts State Police Crime Laboratory describes itself as “fully accredited.”
- In 2016,
police officials in Austin shut down the DNA division of the city’s
police-run crime lab indefinitely after discovering that analysts were
using analytical tools that were badly outdated. The lab also had
problems with cross-contamination. Despite multiple red flags about the
lab dating back to 2008, the lab was repeatedly accredited by the
appropriate bodies within the field of forensics. It took an inspection
by the Texas Forensic Science Commission — an oversight body from
outside the forensics world — to finally bring these problems to light.
- In 2013,
the New York City Medical Examiner’s Office underwent a review of 800
rape cases after investigators found 26 instances in which DNA evidence
was mishandled or overlooked by an analyst at the lab. The cases spanned
2001 to 2011. In fact, the same year that the city’s Medical Examiner’s
Office announced this review, it also boasted in a news release that its DNA lab received “perfect scores” from an international accrediting organization.
- In their various incarnations, the crime labs in Houston have been the scene of multiple scandals, including 2014 revelations that an analyst had tampered with evidence; a 2005 report finding incompetence, cheating on proficiency tests, and even possible perjury and faking of test results; and the shuttering of the fingerprint lab
in 2009 after an audit showed exceptionally high error rates. (This is
not at all a comprehensive list of the crime lab problems in Houston.)
Through all of this, the Harris County and Houston labs have boasted of
accreditation from both national and international forensic
organizations. The
American Society of Crime Laboratory Directors, for example, accredited
the Houston PD lab in 2006. Incredibly, when a 2008 investigation into
cheating on proficiency tests led to the closing of the lab’s DNA
testing unit, the resignation of the head of that unit and the
suspension of two other analysts, the head of the organization that
accredited the lab said his group had no plans to revoke the lab’s accreditation.
- A 2013 audit
of the crime lab in St. Paul, Minn., found widespread ineptitude,
including “sloppy documentation, dirty equipment, faulty techniques and
ignorance of basic scientific procedures.” The report recommended that
the lab cease operations until it could be overhauled from the ground
up. Yet in its 2007, 2008, 2009 and even 2012
annual reports, the Minnesota Department of Public Safety touted the
St. Paul crime lab’s accreditation by the American Society of Crime Lab
Directors. In 2009,
three years before the scandal broke, crime lab director Frank Dolejsi
boasted that the accreditation “is an affirmation of the quality of all
aspects of the [crime lab’s] forensic science services.” After the
audit, state officials vowed to win back public trust by promising that
the St. Paul crime lab would — you guessed it — get itself accredited.
- In
2014, a crime lab analyst in Florida was arrested for stealing seized
drugs and tampering with evidence, possibly tainting up to 2,600
cases from at least 80 different law enforcement agencies. His lab was
accredited.
- In 2012, an audit of the North Carolina crime lab
done after an exoneration found that analysts in the serology unit had
been withholding exculpatory evidence from defense attorneys for at
least 16 years. The audit found that analysts routinely overstated their
findings and that training manuals were adamantly pro-prosecution, to
the point of referring to defense experts as “whores.” The manuals were
likely pro-prosecution because the analysts themselves reported to
prosecutors, who wrote their year-end reviews and determined raises and
promotions. The North Carolina lab had been accredited since 1988.
These are just a few examples. In 2013, the ABA Journal
reviewed dozens of crime lab scandals across the country.
The report found a lot of problems in both accredited and
non-accredited labs. Of course, there’s Isenberg and Oien’s own FBI
crime lab, which is arguably the most accredited and respected crime lab
in the world. But the FBI lab, too, has been no stranger to scandal.
Just a few years ago,
the agency admitted
that it’s hair-fiber analysts had overstated their findings nearly
every time they testified. This occurred over a period spanning decades
and implicated thousands of convictions. Those analysts then trained
analysts at state and local labs, potentially corrupting untold
thousands of more cases. And that’s just one scandal of several. There’s
the Brandon Mayfield debacle. There’s the FBI’s long-held and since disproved claim about
“compositional bullet lead analysis,” which also affected thousands of criminal cases. That’s quite a record from one of the most respected labs in the world. Back to Isenberg and Oien.
Forensic
examiners must complete extensive training to be qualified to perform
casework in accredited laboratories. Training programs can be one to two
years, or longer, and require examiners to perform analyses on samples
with a known correct answer. The examiner must also demonstrate a
thorough understanding of the science behind the method employed and an
understanding of lab policies, procedures, legal rules, evidence
handling, etc. The examiner must undergo oral examinations, mock trials,
and competency tests for which the correct answer is known. In
addition, all examiners must demonstrate competency to apply the
processes accurately and reliably before they are assigned actual cases.
Once qualified to conduct casework in an accredited laboratory, every
examiner undergoes continual competency monitoring through proficiency
tests administered at least once per year.
This
is difficult to square with the examples above, or with the other
scandals at accredited labs far too numerous to list in their entirety. The
most common credentialing agency for crime labs is the aforementioned
American Society of Crime Laboratory Directors (ASCLD). Isenberg and
Oien even cite this organization’s website in laying out the allegedly
rigorous process to becoming an accredited forensic examiner. But while
ASCLD may claim that these are the criteria crime labs need to meet to
earn and keep their credentials, those requirements don’t appear to be
all that tightly enforced. Labs that don’t meet them still get
accredited. And previously accredited labs that don’t meet them don’t
seem to lose their status.
If
the accreditation process is so rigorous and demanding, critics wonder,
then why have so few labs been sanctioned? ASCLD/LAB’s website lists
the status of all accredited labs and shows that no lab’s accreditation
is currently revoked or suspended; there are also no labs on probation.
And [ASCLD Executive Director Ralph] Keaton says he can count on one
hand the number of labs whose accreditation has ever been revoked or
suspended, though he says it would probably take two hands to count the
number of labs that have ever been placed on probation.
Keaton
says that has a lot to do with the overall quality of accredited labs.
But critics say it has more to do with the chummy nature of the
inspection process, which creates a tendency to “go along to get along”
among inspectors, and the agency’s own interest in keeping labs
accredited. . . .
New
York City criminal defense lawyer Marvin Schechter, a member of the
committee that produced the NAS report, is one of ASCLD/LAB’s biggest
critics. Schechter, also a member of the New York State Commission on
Forensic Science, wrote a lengthy memo to his fellow commissioners in
2011 recommending that they look for a new accreditor. He characterized
ASCLD/LAB as an organization more interested in protecting its members’
images than in promoting accountability.
“In
fact, ASCLD/LAB could more properly be described as a product service
organization,” Schechter wrote, “which sells for a fee a ‘seal of
approval’ covering diverse laboratory systems, which laboratories can
utilize to bolster their credibility through in-court testimony by
technicians, plus ancillary services such as protection from outside
inquiry, shielding of internal activities and, where necessary,
especially in the event of public condemnation, a spokesperson to buffer
the laboratory from media inquiry.”
The links to the pages listing revoked/suspended labs and labs on probation now go to a site
called the ANSI-ANQ National Accreditation Board, which apparently
merged with ASLD’s accrediting arm. But neither website appears to lists
any crime labs on probation, currently in suspension, or whose
accreditation has been revoked. If even
egregious incompetence and misconduct rarely if ever results in a
revocation or suspension, what does accreditation really mean?
More from Isenberg and Oien:
Testimony
monitoring is also a requirement for accredited laboratories. The
Federal Bureau of Investigation (FBI) Laboratory requires that examiners
request a transcript for each testimony provided. FBI examiners also
must follow approved standards for scientific testimony and reports,
which document the acceptable range of conclusions expressed in both
laboratory reports and testimony. The DOJ is developing similar
documents called Uniform Language for Testimony and Reports, as well as a
testimony-monitoring framework, which will apply to all DOJ
laboratories. The purpose of these testimony-monitoring activities is to
prevent examiner testimony from exceeding scientific limitations.
This
sounds great at first. But it’s all dependent on the answers to several
important questions. First, who determines the “acceptable range of
conclusions” for lab reports and testimony? Is it someone who
already practices in that particular field? What does “approved
standards for scientific testimony” mean? Who is setting those
standards? Who is approving them? Who decides if an analyst has
“exceeded scientific limitations?” The entire
point of the National Commission on Forensic Science empaneled by
President Barack Obama was to bring in actual scientists to make such
determinations. Attorney General Jeff Sessions allowed that commission’s
charter to expire shortly after taking office. (It’s worth noting that
while Obama deserves credit for setting up the commission, his record in
this area
was mixed at best.) Last August, Deputy Attorney General Rod J. Rosenstein
announced
a new forensics review and monitoring process at DOJ. He also
revealed that the new system would be overseen by a man named Ted Hunt. Ted
Hunt is not a scientist. He is a longtime prosecutor. His supporters
point out that he was part of the very Forensic Science Commission that
Obama empaneled and that reformers lament was allowed to expire. But
Hunt was often a voice of dissent on the commission. Here’s a look at
Hunt’s record on the commission
from Mother Jones:
In March 2016, the commission
recommended that
then-Attorney General Loretta Lynch direct forensic experts and
attorneys working on behalf of the Justice Department to stop using the
phrase “to a reasonable degree of scientific certainty.” The phrase is
commonly used on witness stands and in lab reports and gives juries and
judges a sense of factuality, but it is subjective and lacks any
agreed-upon
meaning across the sciences. Hunt was one of two commission members who
opposed the recommendation, which Lynch
adopted last September.
Lynch also adopted a
recommendation by
the commission requiring forensic testing labs that work with the
department and its attorneys to publicize their internal procedures,
from equipment maintenance to estimations of uncertainty, in order to
foster transparency, trust, and best practices in the industry. Hunt was
one of four commissioners who
opposed it.
Last September, when the commission released a
document supporting stricter accreditation standards for forensic labs, Hunt
voted against it. And when the commission
recommended that
the National Institute of Standards and Technology conduct scientific
evaluations of the “technical merit of test methods and practices used
in forensic science disciplines,” he
opposed that,
too. At its final meeting, when members had already been informed that
the group would be coming to an end, several commissioners pushed for a
resolution encouraging experts to use more quantitative language to
convey the accuracy of forensic testimony. The resolution narrowly
failed, with Hunt among the nays.
So
DOJ did away with a transparent, external commission mostly populated
and led by scientists, and replaced it with an internal commission led
by a career prosecutor who has opposed efforts to increase transparency,
increase accountability, increase scientific accuracy and strengthen
the requirements for accreditation. This is likely why Hunt’s
appointment was
hailed by ASCLD and
forensics groups
such as the International Association for Identification and the
American Academy of Forensic Sciences, and why the new DOJ venture has
been viewed with skepticism by
groups such as the Innocence Project. In
the remainder of their essay, Isenberg and Oien take aim at the report
by the President’s Council of Advisors on Science and Technology (PCAST)
issued in the waning months of the Obama administration. That report
was perhaps the most strident and explicitly critical report on
forensics to date. The authors of and researchers for the report were
scientists, not forensics practitioners or lawyers (though a panel of
judges and attorneys did contribute in an advisory role).
The
essay’s discussion of the PCAST report is a bit technical to delve into
here with too much detail, but in general, the PCAST authors posited
that because most pattern-matching fields of forensics are so subjective
— an analyst basically just “eyeballs” two samples and comes up with
language to describe how similar or different they are — the only way to
determine if these fields are reliable is what’s known as a “black box”
test. We can’t evaluate a particular analyst’s process for determining
matches because so much of it is done in his or her head. Instead, we
have to look at results. That means administering competency tests. Give
analysts a sample from a case in which the culprit is known, give them
samples from multiple suspects, and record who gets it right and who
doesn’t. As you might guess, many forensic analysts and the professional
groups they belong to are reluctant to submit to such tests. Most of
these fields have already been accepted by the courts. They have little
to gain, and a lot to lose. In their essay,
Isenberg and Oien dispute the notion that black-box tests are the only
real way to measure the reliability of a given field. They argue that
relying too much on such tests “ignores much peer-reviewed research,
overlooks critical aspects of many studies, and fails to acknowledge the
empirical value of these studies.” But these studies are, again,
typically reviewed only by other practitioners from the same field,
using the same procedures and standards within the field. Here again,
you’re evaluating the legitimacy of an entire field based on criteria
that already accepts that field’s legitimacy. In conclusion, Isenberg and Oien write:
Science
continuously evolves and is built upon observation, study, and
experience that spans hundreds of years. The justice system would not be
well served by the exclusion of reliable forensic methods and
techniques that provide valuable information to a wide range of
stakeholders.
Of course, whether some of these fields are reliable is precisely what’s in dispute. But at
issue here is how we evaluate new or questionable expertise for use in
the courtroom. What do we do going forward? I think the answer is that
we only allow this sort of analysis and expertise after it has
been subjected to the rigors of scientific inquiry. Before we let a new
drug hit the market, we require its manufacturer to prove that the drug
is safe and effective. Those standards are enforced by scientists with
expertise in pharmaceuticals. When a drug slips through and causes harm,
we pull it from the market. It isn’t a perfect system, and there are
interesting debates at its parameters. But in general, we test new
technology before we put it to everyday use. Contrast
this to forensics. The decision whether to allow a new field of
forensics into court is made by a judge, not a scientist, or even a
fellow practitioner. Judges typically look for guidance on these
questions not from scientists, but from other judges. The briefs in such
challenges are written by lawyers. Judges then tend to err on the side
of letting evidence in, on the assumption that our adversarial system
will sort it out. (In his speech last year, Rosenstein touted this flaw
as a feature.) Even once we discover that a field is scientifically
suspect, it’s difficult to get the courts to even acknowledge it, much
less stop it from being used again, much less correct the cases that may
have already been tainted.
Like the other
defenders of these fields, Isenberg and Oien say we should simply trust
the internal procedures at FBI and DOJ to get forensics right. Even at
face value, that’s a difficult argument to accept. FBI agents and
federal prosecutors aren’t evaluated for their allegiance to scientific
principles. They’re evaluated on their ability to close cases and win
convictions. Therefore, there’s a strong incentive for them to sacrifice
sound science for expedience. That incentive will affect even the most
conscientious and fair-minded at DOJ. But the
case Isenberg and Oien are making is even harder to accept given
the history of these disciplines and the ongoing onslaught of forensics
and crime lab scandals. “Just trust us” hasn’t worked in the past.
Why should we think it would work now?"
