QUOTE ONE OF THE DAY: "Brandon Garrett, a Duke University law professor who studies the reliability of forensic science, agreed the study’s results cast serious doubt on the accuracy of jeans identifications, similar to the problems earlier research found in hair fiber and tool mark evidence. “This is one of many studies uncovering non-trivial error rates for forensic techniques,” Garrett said. “Any lawyer or any judge in a case involving this discipline should, at minimum, hear about the error rates. Many people assume that these techniques are perfect.”
-------------------------------------------------------------------------
QUOTE TWO OF THE DAY: "Studies have begun to raise doubts about the unit’s methods. FBI examiners often analyze low-quality images from security cameras and “it is reasonable to expect that the reliability of this technique may degrade under real-world imaging conditions,” the authors wrote. They argue that all image pattern analysis should undergo validation tests, performed by researchers independent of the FBI and other forensic laboratories. “Mistakes in these identifications are costly, resulting in an innocent person being accused or sentenced and a guilty person walking free.”
-------------------------------------------------------------------------
PASSAGE OF THE DAY: "The FBI’s issues with image analysis echo earlier controversies over other forensic techniques. The bureau’s lab technicians and scientists had long testified in court that they could determine what fingertip left a print and which scalp grew a hair “to the exclusion of all others.” Research and exonerations by DNA analysis have repeatedly disproved those claims, and the U.S. Department of Justice no longer permits its forensic scientists to make such unequivocal statements. ProPublica found that examiners on the Forensic Audio, Video and Image Analysis Unit, based at the FBI Lab in Quantico, Virginia, continue to use similarly flawed methods and to testify to the precision of these methods, according to a review of court records and examiners’ written reports and published articles.
--------------------------------------------------------------
STORY: "A Key FBI Photo Analysis Method Has Serious Flaws, Study Says," by Justice Reporter Ryan Gabrielson, published by ProPublica on February 25, 2020.
SUB-HEADING: "After ProPublica’s reporting last year,
scientists at UC Berkeley tested one of the FBI Lab’s photo analysis
techniques, identifying bluejeans by the pattern on their seams, and
found flaws that challenge the method’s reliability."
GIST: "A
study published this week casts doubt on the reliability of a technique
the FBI Laboratory has used for decades to identify criminals by
purporting to match their bluejeans with those photographed in
surveillance images, potentially undermining evidence used to win
numerous convictions. The FBI’s method, used
principally in bank robbery cases, matches denim pants by the light and
dark patches along their seams, called wear marks. An FBI examiner’s scientific journal article
on bluejeans identification in 1999 argued that wear marks create,
effectively, a barcode that is unique on every pair. That article
provided a legal foundation for the FBI to use an array of similar
techniques to assert matches for clothes, vehicles, human faces and skin
features. After a ProPublica investigation raised questions about the technique,
Hany Farid, a University of California, Berkeley, computer science
professor and leading forensic image analyst, and Sophie Nightingale, a
postdoctoral researcher in image science, tested the bureau’s method and
found several serious flaws. Their study, published this week
in the journal Proceedings of the National Academy of Sciences, is the
first known independent research on the technique’s reliability, even
though the courts have allowed bluejeans identifications as trial
evidence for years. The new study determined
that seams on different pairs of bluejeans are often highly similar.
Separately, multiple pictures of the same pant seam, taken under varying
conditions, can appear starkly different from one another. Taken together, the
authors write, these deficiencies show “identification based on denim
jeans should be used with extreme caution, if at all.” The FBI declined to comment on the study. In its articles last year,
ProPublica revealed that FBI examiners have tied defendants to crimes in
thousands of cases over the past half-century by using crime-scene
pictures in unproven ways and, at times, have given jurors baseless
statistics to say the risk of errors in their analyses was extremely
low. In several cases, the FBI’s most prominent image examiner contradicted the original conclusions and results in his lab reports when presenting evidence to criminal courts, FBI records and legal filings show. The FBI’s issues with
image analysis echo earlier controversies over other forensic
techniques. The bureau’s lab technicians and scientists had long
testified in court that they could determine what fingertip left a print
and which scalp grew a hair “to the exclusion of all others.” Research
and exonerations by DNA analysis have repeatedly disproved those claims,
and the U.S. Department of Justice no longer permits its forensic
scientists to make such unequivocal statements. ProPublica found that
examiners on the Forensic Audio, Video and Image Analysis Unit, based at
the FBI Lab in Quantico, Virginia, continue to use similarly flawed
methods and to testify to the precision of these methods, according to a
review of court records and examiners’ written reports and published
articles. At ProPublica’s request, several statisticians and forensic
science experts reviewed the unit’s methods. The experts identified
numerous instances of examiners overstating their techniques’ precision
and said some of their assertions defied logic. In response to ProPublica’s
reporting, Nightingale and Farid said they decided to test the FBI’s
photo comparison techniques, starting with bluejeans identification.
The researchers purchased
100 pairs of jeans from local second-hand stores and collected images of
more than 100 additional pairs of jeans through Mechanical Turk, the
Amazon service that provides workers to complete tasks. The researchers
used four high-resolution pictures of the seams on each pant leg. They documented wear marks
in the same manner FBI examiners do. But the researchers used what is
known as signal analysis to digitally convert the patterns into numeric
values and calculate how similar the jeans in different images were to
each other. The authors were
consistently able to mark the same features, suggesting the first step
in the bureau’s process works as intended.
But then the analysis
measured wear mark patterns and found the FBI Lab’s method struggled to
match images of the same pant seam, which were frequently no more
similar to one another than to seams from different pairs. Nightingale and Farid
hypothesize that denim jeans are too flexible, as the material easily
stretches and shrinks, changing how wear marks appear, even moment to
moment. The technique failed to
correctly match images of the same bluejeans in most cases unless they
allowed for a high rate of false positives. When inaccurate matches were
limited to one in 10,000, it identified less than 30% of the true
matches. Ultimately, comparing
bluejeans seams is relatively useless, Farid said. “If you’re willing to
tolerate that only one in four times this will be useful, OK, fine, use
the analysis.” Brandon Garrett, a Duke
University law professor who studies the reliability of forensic
science, agreed the study’s results cast serious doubt on the accuracy
of jeans identifications, similar to the problems earlier research found
in hair fiber and tool mark evidence.\ “This is one of many
studies uncovering non-trivial error rates for forensic techniques,”
Garrett said. “Any lawyer or any judge in a case involving this
discipline should, at minimum, hear about the error rates. Many people
assume that these techniques are perfect.” The error rates found in
the study are probably the best-case scenario, the researchers said.
Every image used in the study was taken in a controlled setting, under
good lighting and with the pant seams flattened against a hard surface. The bureau’s image unit has linked
defendants to crime photographs for decades using unproven techniques
and baseless statistics. Studies have begun to raise doubts about the
unit’s methods. FBI examiners often
analyze low-quality images from security cameras and “it is reasonable
to expect that the reliability of this technique may degrade under
real-world imaging conditions,” the authors wrote. They argue that all image
pattern analysis should undergo validation tests, performed by
researchers independent of the FBI and other forensic laboratories.
“Mistakes in these identifications are costly, resulting in an innocent
person being accused or sentenced and a guilty person walking free.” While further research is
critical, Garrett argued that alone isn’t sufficient. He said this study
and scores of others make clear the federal government should regulate
the work of forensic scientists in the same manner they do clinical
laboratories, setting rules and constantly testing their accuracy. “We’ve known about the need for national regulation for over a decade now,” Garrett said, “and we haven’t seen it.""
The entire story can be read at:
PUBLISHER'S NOTE: I am monitoring this case/issue. Keep your eye on the Charles Smith Blog for reports on developments. The Toronto Star, my previous employer for more than twenty incredible years, has put considerable effort into exposing the harm caused by Dr. Charles Smith and his protectors - and into pushing for reform of Ontario's forensic pediatric pathology system. The Star has a "topic" section which focuses on recent stories related to Dr. Charles Smith. It can be found at: http://www.thestar.com/topic/charlessmith. Information on "The Charles Smith Blog Award"- and its nomination process - can be found at: http://smithforensic.blogspot.com/2011/05/charles-smith-blog-award-nominations.html Please send any comments or information on other cases and issues of interest to the readers of this blog to: hlevy15@gmail.com. Harold Levy: Publisher: The Charles Smith Blog;
-----------------------------------------------------------------
FINAL WORD: (Applicable to all of our wrongful conviction cases): "Whenever there is a wrongful conviction, it exposes errors in our criminal legal system, and we hope that this case — and lessons from it — can prevent future injustices."
Lawyer Radha Natarajan:
Executive Director: New England Innocence Project;
--------------------------------------------------------------------------------
