(a) Agencies should ensure the ongoing use of quality controls for forensics work, including accreditation, certification, blind testing of forensic analysts, and routine audits of cases.
(b) When errors in testing or analyzing forensic evidence are detected, agencies should correct the errors and notify the affected parties of the errors.
Comment:
a. Accreditation. The accreditation process involves having a professional scientist periodically evaluate whether an agency’s forensic crime laboratory meets scientific standards. Accreditation is an important step in ensuring that a laboratory meets minimal standards with respect to the procedures and management systems it adopted. In some but not most states, accreditation is required by law. However, accreditation does not ensure that a laboratory uses foundationally valid methods. Nor does accreditation ensure that a laboratory performs reliable and consistent casework—typically it involves review of procedures and protocols but not casework. Thus, although all forensic laboratories should be accredited, additional steps are warranted.
b. Certification. Another form of quality assurance is a requirement that individual forensic practitioners be certified to conduct all of the types of examinations they perform. This requires that a certification examination be developed and made available by government agencies or accrediting or professional organizations. Many certifications currently in use do not provide sufficient assurance that accurate work is done by an expert. The certification process should be based on rigorous standards, including regular and stringent tests of proficiency. Professionals should not be permitted to evaluate evidence based solely on their experience and judgment, i.e., without independent certification.
c. Quality controls. All agencies should adopt quality controls designed to ensure that equipment is calibrated properly and operates accurately, and that individual professionals assess evidence in a valid and reliable manner. The term “quality assurance” is sometimes used to refer to this broader concept, while the term “quality controls” is sometimes used to refer more narrowly to the acts of running control samples and calibrating equipment. We use the broader concept here in referring not only to equipment maintenance but also to evaluating staff performance. Such controls include supervision of the procedures used when handling evidence. For example, there should be procedures for (a) ensuring against contamination, (b) ensuring that a proper chain of custody is kept, and (c) carefully documenting steps taken in analyzing evidence. Such controls also can include blind testing, as discussed in Comment d, and reviewing the work in actual cases by selecting cases at random, as discussed in Comment e.
d. Blind testing. Forensic methods require that a human examiner exercise judgment. It is therefore important that agencies regularly assess how well their human examiners utilize their experience, skill, and judgment in making decisions. Blind testing, in which the person being tested does not know that a test is occurring, is one way to ensure quality control. Such testing should reflect the challenges of forensic casework, it should be conducted in a manner such that analysts do not know that they are being tested, and it should utilize standard testing procedures. It is not sufficient simply to examine whether professionals follow procedures; testing should examine the accuracy of their work.
e. Routine audits. If the process for supervising forensic staff does not include checks on the quality of their casework then there is no way to know how well they are performing. Agencies can take measures to test evidence regularly, either by human examiners or by an automated process. Ideally, agencies would select cases at random so that staff would not know in advance which work would be reviewed. Routine audits can be used to ensure that processes are working correctly and that errors are averted.
f. Duty to correct errors. When forensic testing and reporting errors occur, some standards organizations require only that some type of undefined action be taken to address the consequences of errors. Given the consequences of forensic errors, that requirement is too vague. Instead, policy and practice should require that agencies take some action to correct an error. That action should include providing notice of the error to the relevant lawyers and court.
Reporters’ Notes
1. Absence of quality controls. In the past, quality controls in crime laboratories have been inadequate, if not sorely lacking. The National Academy of Sciences summarized the state of affairs: “Forensic science facilities exhibit wide variability in capacity, oversight, staffing, certification, and accreditation across federal and state jurisdictions.” Nat’l Research Council, Comm’n on Identifying the Needs of the Forensic Scis. Cmty., Strengthening Forensic Science in the United States: A Path Forward 14 (2009) (“2009 NRC Report”). There is a lack of “rigorous mandatory certification and accreditation programs, adherence to robust performance standards, and effective oversight.” Id. at 6. The result has been a series of quality-control crises, in which entire laboratories have been shut down or audited in a range of jurisdictions around the country due to lack of standardization, failure to disclose probative results, rampant errors, and outright fraud. A number of labs have had to conduct substantial retrospective audits due to such quality-control failures. Id. at 44-45. Wrongful convictions have resulted from those failures. Id. Too many laboratories “lacked quality control measures that would have detected the questionable evidence.” Id.
2. Accreditation. Accreditation is an essential part of quality control. Accreditation means that the laboratory adheres to an established set of standards for quality and for procedures. An accredited laboratory has in place a management system that defines the various processes by which it operates, monitors that activity, and responds to deviations from acceptable practices in a routine and thoughtful manner. Id. at 195. Accreditation also requires testing of individual examiners. The National Commission on Forensic Science strongly recommends that all forensic-science service providers (FSSPs) become accredited. Doing so can promote compliance with industry best practices, promote standardization, and improve the quality of services provided. Nat’l Comm’n on Forensic Sci., Universal Accreditation 1 (2016), https://www.justice.gov/archives/ncfs/page/file/624026/download. The American Bar Association similarly has recommended that “[c]rime laboratories and medical examiner officers should be accredited, examiners should be certified, and procedures should be standardized and published to ensure the validity, reliability, and timely analysis of forensic evidence.” 2009 NRC Report, supra, at 194.
Accreditation has become far more common among crime laboratories in the United States. Sometimes that is voluntary; sometimes it is mandated by the state. Nearly nine in 10 (88 percent) of the nation’s 409 publicly funded forensic-crime laboratories were accredited by a professional science organization in 2014, which was up from 82 percent in 2009 and 70 percent in 2002. Matthew R. Durose et al., Bureau of Justice Statistics, U.S. Dep’t of Justice, Census of Publicly Funded Forensic Crime Laboratories, 2014 7 (2016). Eighty-three percent of crime labs were accredited under an international standard in 2014. International accreditation programs are based on the International Organization for Standardization (ISO) and have more rigorous requirements than non-international standards. Id. at 1-2. Since 2009, the proportion of crime labs with an ISO-based-accreditation standard increased from 27 percent to 83 percent. Id.
3. Certification. Sound quality control also requires that forensic practitioners be certified in all categories of testing in which examinations are performed, provided a certification examination is available. Nat’l Comm’n on Forensic Sci., Views of the Commission: Certification of Forensic Science Practitioners 2 (2016). The certification of individuals complements the accreditation of laboratories. In other realms of science and technology, professionals, including nurses, physicians, professional engineers, and some laboratorians, typically must be certified before they can practice. The same should be true for forensic scientists who practice and testify. 2009 NRC Report, supra, at 208. The professional forensic-science community supports the concept of certification, including the Technical Working Group on Forensic Science Education and the International Association for Identification (IAI). Id.at 209. Analyst certification is more common today. In 2014, 72 percent of crime labs employed at least one externally certified analyst. The proportion of crime labs employing one or more analysts with external certification increased from 60 percent in 2009 to 72 percent in 2014 (up 20 percent). Durose et al., supra, at 6. Seventy-eight percent of municipal crime labs employed one or more externally certified analysts in 2014, compared to 70 percent of state and 63 percent of federal crime labs. Id. Between 2009 and 2014, state-operated crime labs experienced the largest increase in the proportion of labs employing one or more externally certified analysts. Id. at 5-6.
4. Testing. Blind testing can provide an integral part of an effective quality-assurance program. It is one of many measures used by laboratories to monitor performance and to identify areas in which improvement may be needed. A testing program is a method of verifying that the laboratory’s technical procedures are valid and that the quality of work is being maintained. There are several types of tests, sometimes called proficiency tests. The primary distinction among them is whether the examiner is aware that he or she is being tested (an open or declared test) or does not realize that the sample presented for analysis is a test sample and not a real case (a blind test). Tests can be generated externally, by another laboratory (sometimes called an interlaboratory test), or internally. Another type of testing involves random case reanalysis, in which an examiner’s completed prior casework is randomly selected for reanalysis by a supervisor or another examiner. 2009 NRC Report, supra, at 207. Blind testing can determine the performance of individual analysts, monitor laboratories’ continuing performance, and identify problems in laboratories and initiate remedial actions. Those may be related to, for example, individual staff performance or systemic issues, such as the calibration of instrumentation, or the effectiveness and comparability of new tests or measurement methods. Id. at 207.
In 2014, 98 percent of crime labs conducted testing of staff, which was similar to 2009 (97 percent) and 2002 (97 percent). As in previous years, nearly all (95 percent of) crime labs evaluated the technical competence of employees through declared or open examinations. The percentage of crime labs that conducted random case reanalysis in 2014 (35 percent) was similar to that reported in 2009 (34 percent), but a decrease from 2002 (54 percent). The proportion of crime labs conducting blind testing decreased from 27 percent in 2002 to 10 percent in both 2009 and 2014. Durose et al., supra, at 4. All U.S. Department of Justice Forensic Service Providers are required to participate in a testing program applicable to the area(s) in which they conduct forensic analyses. Nat’l Comm’n on Forensic Sci., Recommendation to the Attorney General: Proficiency Testing (2016).
Much of the existing testing is not rigorous enough. Although many forensic-science disciplines have engaged in testing for the past several decades, several courts have noted that testing in some disciplines is not sufficient. 2009 NRC Report, supra, at 206 (citing court cases). It is important for tests to be representative of the challenges of forensic casework. It is equally important for test takers to utilize standard operating procedures when performing testing. Test results can be a valuable tool in guiding new research. Test providers should be willing to share their data in the aggregate. They also should strive to collect demographic data and method/process information, and should employ standard report wording to enable a meaningful review of the population’s results as an indicator of the strength of the proficiency test or the competence of the forensic community as it relates to that test (e.g., methodology or technology used). Nat’l Comm’n on Forensic Sci., Recommendation to the Attorney General: Proficiency Testing (2016). Moreover, some organizations have not viewed it as their responsibility to correct all errors, including by notifying legal actors that errors were made. Errors include inaccurate results, failures to follow procedures, and nonconformities, as well as misconduct by staff. The quality-control process is intended to help a lab identify problems. In response to errors, one important international standard, ISO 17025, recommends only that the lab “address the consequences.” Int’l Org. for Standardization, ISO 17025:2017, 8.7.1.a (2017). An agency, however, should do far more than address consequences of an error in some undefined way. An agency should adopt an explicit rule that staff must remediate the error, nonconformity, or misconduct. That must include notifying counsel and the court in order to ensure justice.