PASSAGE OF THE DAY: "Our Board has been, and the intention is to always be, made up of individuals who represent the spectrum of the criminal justice system as well as the citizens of Houston. Our Board has included retired judges, prosecutors, defense attorneys, scientists, citizens, retired police officers and uniquely, exonerees. Those exonerees make up another set of names to repeat as a reminder of the consequences of this work: Anthony Robinson (16), Anthony Graves (17), and Anna Vasquez (18). All have served on our Board. Collectively this group has lost more than 40 y of life imprisoned for crimes they did not commit. Imagine for a moment what it takes to have experienced that nightmare and then be willing to serve on the board of a crime lab."
———————————————————-
PASSAGE TWO OF THE DAY: " Four big lessons have educated me to some realities. First, evidence quality hamstrings labs before they even start. Second, the lab is only one piece; we must start thinking about the whole connected criminal justice system. Third, the databases at the heart of the most sought-after conclusions are vulnerable. Fourth, labs do pretty good with “positive” and “negative” as broad conclusions, but the sticky range of “inconclusive” may just be the challenge to solve. We would not solve that one without addressing the others."
—————————————————————------
PASSAGE THREE OF THE DAY: Once a lab breaks it takes decades to piece back together. Forensic labs are immense pressure organizations that are a chokepoint in the entire criminal justice system. Their stability is a fleeting ephemeral thing. One bad administrator or one too-thin budget can leave a legacy of destruction that will take decades to repair. This comes from the reality that labs all over the country are not a few percentage points away from being adequately resourced; they are multipliers away."
----------------------------------------------------------------------
Abstract: "Houston TX experienced a widely known failure of its police forensic laboratory. This gave rise to the Houston Forensic Science Center (HFSC) as a separate entity to provide forensic services to the City of Houston. HFSC is a very large forensic laboratory and has made significant progress at remediating the past failures and improving public trust in forensic testing. HFSC has a large and robust blind testing program, which has provided many insights into the challenges forensic laboratories face. HFSC’s journey from a notoriously failed lab to a model also gives perspective to the resource challenges faced by all labs in the country. Challenges for labs include the pervasive reality of poor-quality evidence. Also that forensic laboratories are necessarily part of a much wider system of interdependent functions in criminal justice making blind testing something in which all parts have a role. This interconnectedness also highlights the need for an array of oversight and regulatory frameworks to function properly. The major essential databases in forensics need to be a part of blind testing programs and work is needed to ensure that the results from these databases are indeed producing correct results and those results are being correctly used. Last, laboratory reports of “inconclusive” results are a significant challenge for laboratories and the system to better understand when these results are appropriate, necessary and most importantly correctly used by the rest of the system."
GIST: 'When I mention the terms “crime lab” and “forensics,” most people immediately look at me with recognition, interest, and excitement. This makes my wife’s eyes roll when she takes me out and someone asks me what I do. I run a crime lab…a really big crime lab.
My wife’s eyes roll because she knows how the conversation will go. I will emphasize just how desperately difficult it is to run crime labs and how terrifyingly precarious our criminal justice system is. The hapless person asking usually runs away depressed and wanting to hide under the covers. I have been the abrupt end to many dinner parties.
I head the Houston Forensic Science Center (HFSC), the crime laboratory that serves the City of Houston. Formerly, it was the Houston Police Department (HPD) Crime Laboratory, which a March 2003 New York Times headline famously described as, "possibly the worst crime lab in the country (1). It was bad.
That headline appeared 20 y ago. Since then, the City of Houston made a radical change in response to that headline; it created HFSC. In 2024, HFSC will celebrate its 10-y anniversary. Many now point to us as the example of how crime labs should run. Flattering as it is to have made real progress, much work is still to do. Everyone has an opinion about us. Whatever your opinion, we are a really big example of the real life of crime labs in this country.
I have been running forensic laboratories for almost 30 y now. I have run military labs, commercial labs, research labs, and government labs. I have been involved with labs that run millions of samples and ones that have only a handful of employees. I have helped run accreditation programs and manufactured proficiency testing materials. No way around it, I am a nerd with all the nerd letters to show for it.
I have seen awful, horrible behavior and practice in all kinds and versions of labs.
I have also seen extraordinary dedication and professionalism. All the attention on us in Houston has forced me to think about how we have managed to make progress from that damning New York Times headline until now. Every day, I am faced with what it means in practice to try to remediate failures and rebuild public trust. It is messy, smelly, sticky and most of all, hard.\
What Happened in Houston?
The History of Houston and Forensics.
Many authors have written far more extensive and elegant histories of the HFSC. There are books, articles, and law reviews (e.g., refs. 2–5).
The short version is that in 2002, Houston could no longer ignore decades of underfunding and mismanaging its police crime lab. Media highlighted the lab’s infamously leaky roof that destroyed and contaminated evidence, the lab’s massive backlogs of sexual assault evidence and reports of fraud and negligence by lab employees (e.g., refs. 6–9). The lab and its continued antics were fodder for near daily media attention for a decade.
Numerous audits rapidly revealed the jaw dropping extent of the problems. This was topped off by what is commonly referred to as the Bromwich Report (10), named for its author, former Justice Department Inspector General Michael Bromwich. This series of reports and audit effort is still recognized as the single largest and most expansive audit of forensic operations in US history.
Four men were exonerated following wrongful convictions clearly linked to laboratory failures and misconduct. Again, many have written far more about the details, but those men’s names bear repeating as a reminder of the pain caused and eternal vigilance needed: George Rodriguez (11), Josiah Sutton (12), Gary Alvin Richard (13), and Ronald Gene Taylor (14). Together they lost more than 50 y of life imprisoned for crimes they did not commit.
During the decade from 2002 to 2012, HPD made progress in improving the lab. They listened to criticism and made incremental changes. I wish that it were more well known that HPD officials, including Tim Oettmeier (now retired), Troy Finner (the current Chief of HPD), and Matt Slinkard (the current Executive Chief of HPD), have spent most of their professional law enforcement careers addressing crime lab issues.
They not only did not give up on the lab, they created the solution that “fixed” the lab by concluding they needed to do something radically different. They chose to heed the 2009 National Academy of Sciences Report (15) and separate the lab from HPD. This created HFSC’s curious structure as a crime lab that is separate from HPD, the District Attorney and, in some ways, even the City of Houston.
The structure is a Local Government Corporation that is a semi-corporate government entity. HFSC is governed by a nine-member, noncompensated citizen Board of Directors. I often quip that I work with the Chief of Police, the Harris County DA and the Mayor of Houston. I am certain I regularly annoy all three of them more than a little. But I do not work for any of them. I work for the HFSC Board of Directors.
Our Board has been, and the intention is to always be, made up of individuals who represent the spectrum of the criminal justice system as well as the citizens of Houston. Our Board has included retired judges, prosecutors, defense attorneys, scientists, citizens, retired police officers and uniquely, exonerees.
Those exonerees make up another set of names to repeat as a reminder of the consequences of this work: Anthony Robinson (16), Anthony Graves (17), and Anna Vasquez (18). All have served on our Board. Collectively this group has lost more than 40 y of life imprisoned for crimes they did not commit. Imagine for a moment what it takes to have experienced that nightmare and then be willing to serve on the board of a crime lab.
Houston’s Mayor appointed HFSC’s first Board in 2012. They spent 2 y creating the formation documents and negotiating what components of HPD’s forensic operations would become part of HFSC. Then, on April 3, 2014, HFSC took over responsibility for operations.
HFSC by the numbers.
At its inception, HFSC was closer to a start-up business than anything else. Everything had to be built from the ground up. Yes, there were 85 City of Houston employees and 48 classified HPD officers who worked in the HPD lab at the time of the transition. There was lab space that had been the former crime lab under HPD (the roof at least no longer leaked).
But everything else, including standard operating procedures, human resource policies, financial policies, finance systems, benefits, payroll, contracts, supplies, retirement plans, security access systems, IT systems, and even bottled water (the pipes in the building were all unacceptably full of lead and all the drinking fountains had long ago been removed) had to be built from scratch.
I find that many discuss the science activity in the laboratory when discussing laboratory independence. Few think about or acknowledge the vast complexity independence entails for all the supporting apparatus to make those science decisions in the lab possible.
Obviously, I am a proponent of positioning labs to have a more equal footing with other aspects of the criminal justice system, but that comes with all the overhead functions that support employees. This has serious advantages.
Procurement and hiring are far more efficient for us than most government entities. However, all of this had to be built, negotiated, contracted, approved, implemented, budgeted, and managed. That was before we even got to the science-y stuff.
HFSC’s first Chief Executive Officer, Dan Garner, oversaw much of this original building—hiring HR directors and Chief Financial Officers and navigating the delicate initial moves of City employees and classified officers all of whom were unionized. Nothing at all was easy in this and it had very little to do with science.
I came as the first Chief Operating Officer in February of 2015. While the original Board, with the help of subject matter experts on the Technical Advisory Group, had made a lot of progress on rewriting procedures and restructuring laboratory operation, I think it is safe to say when I came is when the serious work of how the lab operated started.
You need do little but Google my name with “Houston” or “forensic” to see the tumultuous effort that this took for years. We are a very public entity. Almost every step (and certainly every misstep) is chronicled in the press.
A perspective of the level of difficulty and change the organization experienced can be encapsulated in some statistics. As of March 2023, the organization now is roughly 212 employees with three remaining civilian City employees in our management. So, in 9 y, the organization has increased the number of employees by about 60% with a near-complete alteration from classified officers and City employees to HFSC corporate employees. Nearly 460 people have worked for HFSC at some point with an average 4.7-y tenure with HFSC.
The remarkable group of people who work for HFSC has managed to create a laboratory that now regularly gains headlines about the progress we have made and is often called a “model” (19–21). This group of amazingly dedicated people has accomplished that scientific and practice turn around all while everything from payroll to water fountains and most of their colleagues have changed.
What blind testing has taught me about forensic labs?
Dozens and dozens of remarkable professionals have made countless improvements to both quality and efficiency of HFSC. These accomplishments range from getting every component of the organization accredited to ISO 17025 standards (22), including accrediting disciplines that are not required under Texas law to be accredited (latent prints, multimedia, and crime scene) (23), to building data visualization tools for managing production and quality metrics.
We have built all the networks and IT services, replaced the Laboratory Information Management System (LIMS), been the first to run our major systems in the cloud, and even replaced the building.
Even with all that, the HFSC program that gets the most attention is our blind quality control system. This has not always been the case.
For years no one seemed to pay any attention to it. This has surprised me since the 2009 NAS report (15), the 2014 PCAST report (24), and even ISO 17,025 (22) accreditation standards all encourage the use of blind testing. Only in the past couple of years have people started to recognize and wrestle with what blind testing is, should be, and can do.
We have published several papers now with data collected from our blinds (25–27). It is tougher to find a publication home for the more subtle lessons and anecdotes that have forced me to have to think about many aspects of what crime labs are and how we attempt to do something better.
I find that criticisms of forensic practice largely revolve around the science-y stuff. We cannot ignore these issues, but frankly they are the 10% challenge.
Four big lessons have educated me to some realities. First, evidence quality hamstrings labs before they even start. Second, the lab is only one piece; we must start thinking about the whole connected criminal justice system. Third, the databases at the heart of the most sought-after conclusions are vulnerable. Fourth, labs do pretty good with “positive” and “negative” as broad conclusions, but the sticky range of “inconclusive” may just be the challenge to solve. We would not solve that one without addressing the others.
The mechanics of HFSC’s blind testing program.
There are many discussions about blinded testing and versions of sequential unmasking (28, 29). HFSC began implementing what is best described as a blind proficiency system starting in 2015. Blind proficiency means that our quality division inserts constructed materials with a known and intended result into the workflow routinely and continuously in such a way that production staff are unaware which things are real cases and which are tests.
We did not implement any of this as a research project. The point has been a routine and continuous mechanism to check that the laboratory as a system is performing as we expect from receipt to reporting. To date, this is internal to just HFSC, making it very difficult to extrapolate any conclusions beyond HFSC.
We are very strong proponents of testing to the point of failure. We have routinely endeavored to make blinds not only representative of actual cases, but also “hard”. For instance, we have often made firearm blind cases using the same make and model of gun so that the evidence has all the same class characteristics making any subsequent comparisons far more difficult (27).
We have also often introduced known errors into the blind cases to see if systems recognize the errors and react accordingly. Sometimes this has been inadvertent, but we have always gained immense information about our systems and others from all these instances. We also pass our blind materials through to outsource labs (commercial labs we contract to do portions of work) when the case merits outsource work.
From the start of the program, I have kept an offhand bet with the staff. If they find a case that turns out to be a blind, I have a Starbucks card in it for them. If they think something is a blind and it turns out not to be, they owe me a buck. Most of the staff having somewhat fiendish senses of humor, they often pay me in pennies.
I have done this for two reasons. One of the most recurrent reasons I have heard from other labs hesitant to perform blind testing is that they fear staff will feel that blind testing is oppressive, intrusive or unethical. I cannot say if this whimsical bet has made the difference, but I have not had many indications our staff has ever felt that blind testing is invasive or otherwise problematic.
The second reason is that to me it is intuitive that knowing there are blinded materials in the workflow will cause people to look for them. There is an odd incentive to find them. This means keeping samples blinded will always be an arms race between our quality division and those handling, analyzing and reporting results. The system then needs to understand wherever possible what the “tells” are and work to further improve materials.
Staff members have not disappointed in providing all kinds of remarkable and often amusing anecdotes of what triggers them to think something is a blind.
The only area of work we do not have blinds is in crime scene. We have contemplated how to do blinds here, but I think it obvious that there are serious limitations to fully blinding crime scenes. No one ever seems to volunteer to be a victim in a blind homicide scene.
Fig. 1 provides the total number of blinds we have inserted as of March 2023 into each of the divisions broken out by those that have been completed, are still in process, were discovered or were thought to be blind but were not. Fig. 2 has the same data presented as percentages, which makes it easier to see the proportion of blinds discovered and wrongly guessed. We have targeted 5% of our casework output as the rate of blinded tests. The differences in the total numbers result from the differences in caseloads and the length of time we have run blinds in each discipline.
In these figures, there are two columns of “blind verifications” in latent prints and in firearms. In these disciplines the common workflow is to have an examiner render their opinion and then a second examiner take the same case and render a second opinion or “verify” the case. We do 100% verification of inclusive results, exclusive results, and unsuitable conclusions.
That high percentage is not common in laboratories widely. But this system of verifications is open between the two examiners, the second examiner is privy to the conclusions and work of the first. In about 5% of cases, we blind the second examiner to the first examiner’s work and conclusions and then look at the concordance (or lack of concordance) between the two examinations.
Blinding involves multiple components of our organization and other organizations to make this work. Our client services/case management (CS/CM) division manages records and evidence movement. The group is intended to be a front door and a firewall between the analyst and investigators and lawyers.
Like the quality division, they are an integral part of blind testing to effectively emulate actual case materials. To be extra clear, even after years of trying to establish some filters between analysts and the outside, CS/CM is a very porous filter.
Altogether, we have inserted more than 3,000 blinds into our system over the years. While I ask every audience I speak with to tell me about other blind programs, I have found very few efforts in crime labs. The best I can tell is that we have run about 3,000 more blinds than any other crime lab or system of labs in the world. This is a very unusual thing in crime labs. I struggle to see how we ever broadly repair public trust in forensics without aggressive blind testing done to the point of failure.
Our quality division, which includes eight staff members, manages the blind program. They manufacture test materials and manage contracts for manufacturing materials (specifically toxicology samples). They comb thrift shops and pawn shops for materials to mimic cases.
They hunt for used tools, old clothing items and burner phones to turn into cases. We also utilize drug and gun items from prior seized materials that have been authorized for destruction through collaborations with both HPD and the Harris County DA.
Quality division staff interact with HPD (our overwhelming source of submissions) to manage the submission process of blind cases, which necessarily involves law enforcement as Agency Case Numbers and other information starts with HPD. Interaction with HPD includes the other end in managing reports to HPD and the District Attorney and ultimately reviewing the results and process against the expected results.
Bias control at HFSC.
Potentially biasing information is a very valid concern with serious potential to contaminate laboratory conclusions. Our blind proficiency system is one of the many tools we have attempted to implement to control this potential. I think it worth adding a little perspective to just how daunting this challenge is.
I hear many scenarios posed of the law enforcement agent willfully trying to sway the result of a lab. I cannot discount this happening, but in my experience this is far from common. Far more often overwhelmed officers and prosecutors juggle far too many tragic cases and fall prey to influences they are subjected to.
I have the misfortune of having some personal understanding of this. When I was a child, my grandfather was murdered and my grandmother savagely beaten into a coma (20). I remember the jagged, emotional calls to investigators and prosecutors furiously demanding why more was not being done.
Of course, investigators are going to try to persuade others they depend on for answers to provide those answers as quickly as possible. They are human in their response to pressures like those awful calls from my family. In that attempt to persuade, that is what you do. You appeal to emotion. You try to humanize the need.
This is what I see regularly. Not a knowing endeavor by an investigator or prosecutor to get a particular desired outcome. Rather a very human response to try and get ANY answer quickly. But this usually comes with dangerous information.
A major reason for creating CS/CM in 2015 was to insulate the analysts as best we can from those calls. We also created an online request portal (we call it “Where’s my Result?” as an acknowledgment of everyone’s frustration in waiting for what they need). This portal helps sanitize and direct what information is provided with drop downs and directed questions.
There is information we need to correctly prioritize and to understand CODIS eligibility that is often contaminating but still essential. We have managers and supervisors within each division to also help manage calls from prosecutors and investigators and further sanitize what is passed on to analysts.
All this is imperfect. What we cannot do. We are a long way from anything that looks like sequential unmasking.
An enormous hurdle here is that our LIMS has no means of managing what information an analyst can or cannot see, much less tailoring that information to some point in the process. This is not just a flaw in our system. It is a fundamental structure of all the commercially available LIMS. None of them have any utility for this, nor any development plans anytime soon. In fact, attempting to disguise blinds in our systems is made far more difficult because of this inability. We must go to extraordinary lengths to emulate the data that we cannot hide.
Often troubling data are inherently and indelibly part of the packaging of the evidence. A classic for us is in seized drugs. The HPD required 8x11 envelope that most of our drug cases are submitted in has a form preprinted on the outside with submission information. This includes the suspect’s gender and race. Fairly obvious domain irrelevant information.
But short of someone unpackaging all the evidence and only providing a subsample to the analyst, we have no way to remove this. That level of intervention would add all kinds of risks of sample misidentification, physical contamination, and chain of custody complications. Which risk is worse?
Even our attempts to intercede in direct communications with analysts are often fraught. We try to keep analyst cell numbers insulated and push calls through CS/CM. Investigators dutifully trying to progress their case, often look for ways around this. We once had an analyst on vacation have a prosecutor track down their spouse’s personal cell number to then call and harangue.
We are still at the point of trying to resist where possible direct interaction of analyst with investigator. We regularly and broadly train about bias. We try to control the flow of information. But even we are a long way from sequential unmasking and reducing domain irrelevant information that leaks in from packaging and documentation. Many layers are yet to be tackled.
The tragedy of evidence quality.
We have many anecdotes that have come out of the blind program that range from simple to hilarious.
All of them have been educational about how to run blinds and about our own systems. We have made many improvements to our systems and procedures based on results from blinds.
The single most common way staff have spotted blinds or what has raised their suspicions something might be a blind has nothing to do with the scientific nature of the materials. It is “I could read the handwriting and everything was spelled right, no way that is real.”
This takes all forms. “The evidence was too pretty” or “the packaging was way too clean” are common comments. Sometimes the comments point to subtle practice tell tales. Latent print cards are a common item we receive. These are quite literally a note card on which a tape lift of a powdered surface is placed. Fig. 3 is an image of a latent lift from a case guessed to be a blind. One of the tells was that the “tape had a serrated edge – who uses a tape dispenser in the field?!?”
Our staff members know the reality of field collections. Often tape is torn with teeth, car keys or maybe a pocketknife. Other comments about this case were the “handwriting was super pretty”. I think this is illustrative to those who do not see a lot of real-world evidence what “super pretty” handwriting looks like to us. My bet is most will see this as simply legible.
=
This was in fact a real case. I got more pennies. While amusing, anecdotes like this mean our quality group has to go to great lengths to disguise blinds by writing with their nondominant hand, backing their cars over mock evidence, and harvesting dirt, leaves, and grime from their backyards to add into mock cases. In other words, they must make the evidence submission WORSE to keep it from being identified as a blind.
This is the lesson. Labs all over the country routinely receive evidence that is illegible, missing information, unsuitable, damaged, imperfect, wrong, misidentified, incorrectly sealed, inappropriately packaged, or not packaged at such a rate that correct evidence sticks out.
Evidence comes from the real world and will never be clean nor designed to be reproducible like a research project. Odds are good it is going to be decayed, smelly, sticky, foul and unusual even when it is correct. Those issues stack on top of the flaws that are overwhelmingly human.
Most of the evidence received by labs is collected by overwhelmed law enforcement officers who have limited if any training in collecting forensic evidence.
Rarely does anyone have the time and resources to design collection tools and packaging to make collection robust and easy for the wide skill variability of those who must do the collection.
Rarely are the resources available to train more individuals that necessarily must be a part of collecting physical forensic evidence.
Labs could do analyses that everyone viewed as perfect and we would still have the same problems because the quality of the evidence received by labs is so often poor. We can do a lot with remarkably degraded materials; poor-quality evidence does not always equate to invalid evidence or invalid analysis.
But poor-quality evidence most certainly equates to a loss of efficiency. Labs are often the entity that must document and report the condition of the evidence. That takes time and resources that are then not available for anything else. Evidence quality affects the entire system and is a chronic insidious drain on resources at all levels.
The interconnectedness of all things.
Other authors have more completely written about the constellation of efforts that have occurred in Texas that all contribute to the improvement of forensic science practice in the state (3). Our experience with blinds clearly confirms that labs are only one piece of the system.
HFSC cannot do most everything we do without the array of laws and institutions that have grown up around forensics over the past 20 y.
We are completely dependent on participation from HPD, Harris County District Attorney, the City of Houston and the Texas Forensic Science Commission (TFSC) to name a few. Many of those are directly involved in how we create and manage blinds.
To me, one anecdote encapsulates the connections. We have a very strict policy that we will not completely consume evidence in testing without a court order.
In toxicology we have often sent components of testing to outside commercial labs. Our DUI kit used to contain only two gray top tubes. In one case, we had screened the sample and had a presumptive positive for drugs that needed a confirmatory test we did not have available at the time, so the sample was packaged up and shipped out. The tube broke during shipping.
This then left us with only one tube that once opened, affects the alcohol concentration. Thus, we view this as consuming the evidence. Toxicology staff dutifully started the process to seek a consumption order for the remaining tube. As the process evolved for the court order, the prosecutors, officers and judge became rapidly confused when there did not seem to be a defendant.
Everyone was rapidly headed down the path to investigating where this case had gone, what had gone wrong in the arrest, and we were rapidly preparing to disclose the circumstances to TFSC. The sample was a blind.
Once we explained the situation to everyone the dust began to settle. I think several years later most can laugh about the situation. I took the view that this was a good demonstration that blinds are well and truly blind, at least in toxicology. The toxicology staff had no questions about what needed to be done with the sample and were extremely anxious regarding the irregularities about the court records, not the sample.
The whole episode graphically illustrated that blinding is not simply isolated to the lab.
For blind testing of ANY sort to work, more than just the laboratory must participate. Packaging of materials starts with law enforcement agencies. Reporting stretches out to the courts. For us, attempting to make blind tests work is completely dependent on the cooperation of HPD. This will be the case for ANY laboratory trying to do this.
Beyond just the exercise of quality systems, our experience with blinds illustrates the interconnectedness of forensic laboratories to the rest of the system. There is no single owner of forensics as an enterprise across the country or within any state. Law enforcement, laboratory, prosecution, defense, accreditation, oversight, and advocacy all have significant interests in lab results and operations, but none have exclusive rights.
This means that laboratory structure best serves everyone when it is in better parity with other components of the criminal justice system. But that can only work when effective oversight mechanisms exist that all can trust. Without that oversight and without the framework of legal structures described by Casarez and Thompson (30), lab independence and lab functioning is compromised.
Texas has an enormous advantage with the TFSC and its herculean efforts in moving forensic practice in the state forward. This, however, is rare in the country. Most states do not have the advantage of a TFSC.
The necessity to include the “Big” databases, CODIS, AFIS, and NIBIN.
In my experience in labs, we can be on solid ground scientifically and still end up wrong because something else along the way breaks. A sample is misidentified. A report goes to the wrong place. More likely, a report does not get where it is expected to go. A vital component of the most valuable results is yet impossible to blind test.
That is the big three databases. I do not think I can overstate how vital these systems are to the criminal justice system. Blind testing has illustrated to us how little we all know about how these systems are truly functioning.
CODIS, or the Combined DNA Index System, allows for the comparison of evidentiary profiles between cases and known offender profiles (31). AFIS, or the Automated Fingerprint Identification System, has algorithms to search evidentiary prints against an assortment of datasets of prints (32).
NIBIN, or the National Integrated Ballistic Information System, is subtly different in that it allows for the linkage of evidence between crimes but does not seek to link a gun to an individual (33, 34).
All these systems are complex interactions of local, state, and national levels. All these systems are an integral part of investigations and often THE result of most interest to investigators. A DNA profile is powerful, but it is the ability to link that profile to a potential individual with a name that is the real goal.
All these databases are an integral part of the workflow in DNA, latent prints and firearms. So much so that to conduct blind testing, we have had to wrestle with how to manage the interaction with the databases. Otherwise, a blind case is easily recognizable making the whole exercise moot.
I will focus on CODIS, but all the databases have similar challenges. CODIS is managed by the FBI, and there are laws, regulations, rules, and agreements that create a strict framework for how labs and others must interact with the database, what is eligible, and what is off limits (31).
All this framework is necessary and good because this is extraordinarily sensitive information with significant impacts for individuals. We should all be wary of efforts to circumvent this framework or reduce oversight!
Because of this framework, we have either had to construct blind cases that are not CODIS eligible or that we ensure profiles from a blind case are not “uploaded” to CODIS or entered into parts of the system off limits. We can only do so many ineligible cases because we do not get that many in actual case work. They become easily identifiable.
We have a specific unit that handles our CODIS work, and this group is “in on the deal” and part of how we manage appropriate interaction with CODIS. This means however, we cannot really test our CODIS handling as we would like. It sure would be helpful to have the FBI’s support but thus far it has not been forthcoming.
As with everything in this article, I have a blind story to illustrate the challenge.
We purchased blood from a tissue supply company. Most researchers are familiar with companies that supply biological fluids and tissues for research purposes. We used this blood to devise a case that was a burglary with blood stains left behind at the scene that the property owner could state did not belong to them and were new after the break in. That made the potential profile CODIS eligible. This was submitted to the lab, processed as normal, a profile developed, and data reviewed.\
Unfortunately, our quality division did not get this profile communicated to our CODIS unit as part of a blind test. Our CODIS unit did exactly what they were supposed to do and uploaded the profile as they would any other. This is not how it was supposed to work for a blind.
In this case the profile “hit” to a 2000 sexual assault case and to an offender profile in Tennessee. In other words, the blood we purchased was from an individual who was a convicted offender with a profile in CODIS and that profile was associated with a case from 2000.
We realized that we had uploaded an inappropriate profile, removed the profile, notified everyone from the local agencies to the States of Texas and Tennessee and the national level of CODIS. We also disclosed this to TFSC as a nonconformance (35, 36). We remediated the errors and have changed policies and procedures to prevent this in the future.
The part that is intriguing about this though is that we, HFSC, are the ones that noted the error. Why did we not hear about this error from Tennessee or other layers of CODIS administration? A CODIS hit in Texas to a 21-y-old case with a known offender with no previous history of burglaries should have raised flags with experienced investigators, yet we heard nothing.
We reached out to the Tennessee Bureau of Investigation (TBI).
We learned that the complainant in the original sexual assault offense had not wanted to pursue charges. Years later, the sexual assault kit was tested in a backlog reduction program, but this was long past the statute of limitations.
The offender was entered into CODIS years ago because of convictions for forgery and this entry was the profile associated with the 2000 sexual assault, again long after the statute of limitations had expired.
When TBI got our curious hit, there was nothing they could do and appropriately closed it out. In this case everything worked, but TBI does not have the resources to follow up with us and ask: “Hey, that was weird. Are you sure about that?”
Labs all over the country notice how few “reference swabs”, or known samples, we get after notifying investigators of a CODIS hit. A CODIS hit is an investigative lead and cannot be used as evidence in trial.
For that, a lab must conduct another analysis on a known swab from the suspect. But we get surprisingly few of these required swabs for follow-up testing. Why?
This could be for any number of perfectly logical and appropriate reasons. The case may have been closed as in Tennessee, so that a swab was not needed. But it could also be because the reports do not make it where they should. Or that recipients do not know what to do with them. Or that there are break downs in how information is entered. Or that swabs are collected and do not make it to labs. All of which are not somehow nefarious or incompetent, but rather the natural outcome of overtaxed, large and complex systems.
More concerningly, these outcomes could be CODIS hits used at trial without the required reference, or nothing is ever done at all with a CODIS hit. We do not know.
Our one-off accidental blind test of the CODIS system demonstrated a host of correct expected performance in one database. We see hints that something is not ideal. Only direct and routine testing will explain what is happening, identify weaknesses to remediate and give information if there are uglier problems or not.
I fully appreciate the sensitivity of these systems and the caution required in how we go about testing them. But that caution to me means greater urgency to aggressively test and ensure the entire system is performing as we all hope and need.
I have discussed, as have others, with the FBI and ATF about actively participating in efforts by labs to implement blind testing to no avail yet. Concerns largely center on protecting the validity of the data, not introducing “bad data” and questions of just how valuable the exercise is.
Data integrity is a very fair concern. One that should be solvable in database management. As to the value of the exercise, I guess I keep trying to explain the value.
The conundrum of “Inconclusive”.
Our paper about results from blind tests in firearms has attracted a fair bit of attention. This is primarily because we report a roughly 40% “false inconclusive” rate (27). This kind of percentage and the word “false” causes eyebrows to shoot up, and everyone begins to have an opinion.
Inconclusive is a term getting more and more attention recently particularly for firearms and latent prints.
The wider reality is there is a version of inconclusive in every discipline. There are many valid reasons to report inconclusive, but the language used to communicate the concept is highly variable and a very serious challenge for those using the reports.
Inconclusive can take the form of usable evidence, but upon examination there is not enough detail or sensitivity to conclude a source attribution or “identification” or “exclusion”. In other words, an analyst cannot conclude that a DNA profile originated from the defendant or that there is no way the profile could be the defendant.
I think it is intuitive to most people that these decisions fall along a continuum with uncertainty. But this topic has sparked a robust discussion. OK, it is pretty much a bar fight.
Our blind data have put us squarely in this fray. Logically, if examiners are not privy to being tested, the potential cognitive biases that may alter the willingness to conclude or not conclude are more controlled.
Comments and criticisms of what are known as the Ames studies (37, 38), two among multiple black box studies designed to help determine the error rate in firearms, often center on the nonblinded nature of the study.
Blinded testing is often offered as a means of better determining error rates required in Daubert (39). Lots of people are interested in our blind results.
Many will argue that there is a clear and obvious choice; a conclusion is either correct, or it is not. Sitting in the middle of this with all the demands of a practitioner lab, I see very little obvious. \
Inconclusive is a conclusion. It will probably always be a needed conclusion. It has implications for the rest of the system just as an inclusive or exclusive conclusion. We are tasked with having to decide something with imperfect evidence and limited or no ability to repeatedly sample to improve statistical power.
We often have one piece of priceless evidence that comes to us smelly and broken. While we endeavor to be nondestructive in testing, most testing necessarily consumes some of the evidence. We know that regardless the conclusion, what we do will have existential consequences for someone. Yet, we must do SOMETHING with the situation.
It is very easy to judge with the benefit of hindsight and additional information not available at the time of testing. It is a far more difficult challenge to render any conclusion at the time. I do not wish to absolve egregious actions that have most certainly occurred in labs. I am trying to say that “inconclusive” may sometimes be the “correct” and appropriate answer given the quality of the evidence and the available data.
But just what do we do with “inconclusive”? I think critics of how this has been measured and reported in black box studies have a point that “inconclusive” is not completely “right” when a ground truth is known (e.g., refs. 40–42). But viewing it simplistically as the same kind of error as a false positive or false negative is also not the correct view.
Laboratories should be very interested in reducing or narrowing the window of inconclusive conclusions with improved technologies and procedures. Researchers need to appreciate the gap between their ideal and the gummy world we live in.
The lab is only one part of an interconnected dependency of the system. Other components of the system must refrain from misusing an appropriately inconclusive result. All too often the appropriately inconclusive result from the lab becomes a tacitly inculpatory (mostly) or exculpatory (less often) piece of evidence at trial. This is a misrepresentation of what the laboratory reported.
How does this happen? Labs usually find themselves called to testify by the state and are thus directly examined by prosecution. The first set of questions has the frame of “this inconclusive result means you cannot exclude the defendant, correct?”
The analyst has little choice but to answer “correct”. If lucky, defense counsel on cross examination can try to rebut this asking, “inconclusive also means you cannot include my client, correct?” The analyst then has the chance to answer yes. But the first framing is what the jury heard. Likely the prosecutor will revisit this point in closing arguments outside of the presence of the laboratory.
If only this were as simple as others using the lab’s appropriate inconclusive result correctly. A growing body of data, including ours, suggests that inconclusive results mask more exclusionary evidence than associative evidence (27).
In Baldwin et al (37) the data suggest that a same-source comparison had a likelihood ratio of an inconclusive result of 1% while a different-source comparison was a likelihood ratio of 33.7% of an inconclusive result. Monson et al (43) also found that different-source set comparisons were more likely to produce an inconclusive than same-source set comparisons. Guyll et al (44) report inconclusive decisions were rendered six times more frequently for different-source than same-source comparisons.
If this were entirely about evidence quality or a random effect, we would expect a more even distribution of inconclusive results between same-source sets and different-source sets. More than just simply treating “inconclusive” as the neutral no-answer it can be, inconclusive may be more likely to be exclusive than inclusive.
How is the rest of the system supposed to interpret “inconclusive”?
We need vigorous debate about how to handle inconclusive and even more on how to explain the term inconclusive to everyone including juries. We need improved techniques to reduce the instances of inconclusive conclusions. What is not productive or helpful to anyone is an insistence that there is either “no issue with inconclusive so just leave us alone” or that inconclusive is somehow by definition always wrong.
All conclusions have existential consequences for someone in the system. No conclusion, and inconclusive is a conclusion, should be taken lightly.
What do I wish more people understood about crime labs?
Blind testing has taught me that labs face monumental challenges we need help to solve.
All these challenges are made harder with the daunting lack of resources for crime laboratories.
I point out regularly that I run a very well-funded lab compared to most, yet we are far from adequately funded. We struggle to make the lab work. Imagine the challenges that face a small lab serving rural communities.
Blind testing is an essential tool for labs. It is a tool with huge benefits and challenges to accomplish it for a giant lab. Imagine now that small lab serving rural communities where the lab director is also the quality manager and certifying scientist and plumber, how in the world will this work?
Texas may soon have some answers for that. We are actively discussing and hope to soon start a more statewide effort. But it will involve having a TFSC, researchers to help, larger labs that can help the smaller labs, and resources. Above all, it takes the will to make it happen. I hope for others it does not take a catastrophe like Houston to help drive that will for their states.
Once a lab breaks it takes decades to piece back together. Forensic labs are immense pressure organizations that are a chokepoint in the entire criminal justice system. Their stability is a fleeting ephemeral thing.
One bad administrator or one too-thin budget can leave a legacy of destruction that will take decades to repair. This comes from the reality that labs all over the country are not a few percentage points away from being adequately resourced; they are multipliers away.
Labs function because of heroic nerds who turn themselves inside out daily to desperately hold the frayed ends together. I cannot overstate the honor and privilege for me to work alongside remarkable and dedicated people who somehow keep coming back to produce results for victims and defendants they will never meet. They do this for small pay, long hours, and endless criticism.
These scientists are rare to find, hard to train, and easy to damage. Life in crime labs means a life of 24/7 nightmares. Everything we do is the remains of someone’s worst day, be they victim or defendant. This is hard on people and their families. This is not unique to labs but is unique to those throughout the criminal justice system.
Wrong answers from a lab cause all kinds of devastation. Houston has more than a little experience with this. But the right answer too late, is just as devastating. When labs are an afterthought in the budget, the damage is catastrophic. Lab errors cost millions. Backlogs cost millions. Yes, labs are crazy expensive. Far more expensive than anyone likes. But adequate support and resources for labs are far less expensive than failing our citizens.""
The entire commentary can be read at:
https://www.pnas.org/doi/10.1073/pnas.2303592120
PUBLISHER'S NOTE: I am monitoring this case/issue/resource. Keep your eye on the Charles Smith Blog for reports on developments. The Toronto Star, my previous employer for more than twenty incredible years, has put considerable effort into exposing the harm caused by Dr. Charles Smith and his protectors - and into pushing for reform of Ontario's forensic pediatric pathology system. The Star has a "topic" section which focuses on recent stories related to Dr. Charles Smith. It can be found at: http://www.thestar.com/topic/charlessmith. Information on "The Charles Smith Blog Award"- and its nomination process - can be found at: http://smithforensic.blogspot.com/2011/05/charles-smith-blog-award-nominations.html Please send any comments or information on other cases and issues of interest to the readers of this blog to: hlevy15@gmail.com. Harold Levy: Publisher: The Charles Smith Blog;
SEE BREAKDOWN OF SOME OF THE ON-GOING INTERNATIONAL CASES (OUTSIDE OF THE CONTINENTAL USA) THAT I AM FOLLOWING ON THIS BLOG, AT THE LINK BELOW: HL
https://www.blogger.com/blog/post/edit/120008354894645705/47049136857587929
FINAL WORD: (Applicable to all of our wrongful conviction cases): "Whenever there is a wrongful conviction, it exposes errors in our criminal legal system, and we hope that this case — and lessons from it — can prevent future injustices.
Lawyer Radha Natarajan;
Executive Director: New England Innocence Project;
—————————————————————————————————
FINAL, FINAL WORD: "Since its inception, the Innocence Project has pushed the criminal legal system to confront and correct the laws and policies that cause and contribute to wrongful convictions. They never shied away from the hard cases — the ones involving eyewitness identifications, confessions, and bite marks. Instead, in the course of presenting scientific evidence of innocence, they've exposed the unreliability of evidence that was, for centuries, deemed untouchable." So true!
Christina Swarns: Executive Director: The Innocence Project;
------------------------------------------------------------------
YET ANOTHER FINAL WORD:
David Hammond, one of Broadwater’s attorneys who sought his exoneration, told the Syracuse Post-Standard, “Sprinkle some junk science onto a faulty identification, and it’s the perfect recipe for a wrongful conviction.”
https://deadline.com/2021/11/alice-sebold-lucky-rape-conviction-overturned-anthony-broadwater-1234880143/
——————————————-------
