An agency should not use an algorithm or profile to direct police resources to a particular location, to identify potential targets for further investigation or surveillance, or to assess the risk of harm that individuals may pose to others unless:
- (a) the algorithm or profile has been shown to perform with acceptable accuracy, and it is sufficiently transparent that agency officials can explain the factors on which it relies and how they are weighted;
- (b) the agency has provided adequate training to all agency officials who are authorized to use the algorithm or profile in interpreting the results and has provided appropriate guidance on the limits on the permissible inferences that may be drawn;
- (c) any use of protected characteristics, such as race or ethnicity, is consistent with § 1.11(a)–(b);
- (d) the agency avoids the use of inputs, such as police-initiated contacts, or arrests and convictions for low-level offenses, that may reflect prior discriminatory enforcement practices and therefore could exacerbate racial or other disparities; and
- (e) the agency routinely audits both the algorithm and the underlying data to minimize inaccuracy and bias, and makes appropriate changes in response to any issues identified by the audits.
Comment:
a. Profiles and algorithms. Algorithms and profiles play an important role in guiding government decisionmaking. Predictive algorithms work by drawing on a variety of data sources—from incident and arrest records, to various socioeconomic indicators, to weather and time of day—to identify patterns associated with a heightened risk of criminal activity. Policing agencies use predictive algorithms to pinpoint areas in which crime is more likely to occur, or to identify individuals who are most likely to become victims or perpetrators. Agencies use this information to guide officer deployment and to develop strategies to mitigate any foreseeable risks. Algorithms also have been used to assign “threat scores” to homes and businesses, so as to inform officers and dispatchers about the potential risks that an officer may face when responding to a call for service at a particular location. Other actors within the criminal-justice system use similar technologies to inform bail determinations and sentencing, as well as decisions about probation or parole. In addition, agencies may rely on algorithms and profiles in retrospective investigations, for example, by using bank or tax records to identify suspicious patterns of activity that may be consistent with various financial crimes.
Agencies and officers also sometimes rely on more informal profiles to guide investigative activities. For example, as part of drug-interdiction efforts, agencies develop profiles of vehicles or persons who are most likely to be involved in drug trafficking, relying on various factors including vehicle descriptions, travel routes, and behavioral cues. And, of course, long before the advent of “predictive policing” technologies, agencies used past crime data to predict future trends. Although less sophisticated than modern algorithmic technologies, the underlying goal of using statistical methods to identify persons or locations that are more likely to be involved in criminal activity is essentially the same.
b. Animating concerns. The use of algorithms and profiles raises a number of very weighty concerns, particularly around reliability and the potential for racial bias. Although many use the phrase “predictive policing,” the reality is that algorithms do not “predict” behavior. Rather, they are a risk-forecasting tool. The concern is that officials may attach more significance to the prediction than is in fact warranted. This is particularly the case when algorithms are used not only to guide deployment decisions but also to inform the actions that officers subsequently take. One potential problem with using “threat scores” for particular addresses is that officers may interpret a designation of “medium” or “high” risk to signify a serious risk of harm, and thus be more likely to resort to force, despite the fact that the magnitude of the risk that they face may be quite small.
Another serious concern with predictive algorithms is that, depending on the data that are used, algorithms may rely inappropriately on race or other protected characteristics, or they may exacerbate existing racial disparities within the criminal-justice system. And, importantly, algorithms need not explicitly rely on race in order to have these effects. Arrest data, for example, are shaped in large part by how officers have been deployed and how they have exercised their enforcement discretion. Studies consistently show that although rates of illegal drug use are roughly comparable across demographic groups, Black and Latinx people have been arrested for drug offenses at a disproportionately high rate. The same is true for a variety of other offenses. If arrest data are used as part of a predictive algorithm—either to inform where officers are deployed or which individuals ought to be targeted for greater scrutiny—these disparities will be reinforced.
Compounding these concerns, public officials often lack a sufficient understanding of how a particular algorithm was developed and the various inputs that were used. Some companies that sell predictive-policing software to public agencies consider their algorithms to be proprietary and thus provide only limited information about how they work. In some cases, the algorithm may be a black box even to the developers themselves. This can occur when an algorithm is developed through certain forms of “machine learning” whereby a software program deploys complex statistical analysis to fit a predictive algorithm to whatever data it is provided. Developers know what data the resulting algorithm has access to but may not know how it uses the data to calculate its predictions. This makes it even more difficult for officials to identify potential sources of bias or, in the absence of rigorous validation and assessment, to assess the algorithm’s efficacy.
Finally, algorithms only are as good as the data on which they are built. When databases contain errors, the results are inevitably skewed. Depending on how the outputs are used, the errors can lead to potentially tragic results for members of the public and officers alike.
Similar concerns, of course, have been expressed by courts and commentators about more informal profiles, such as the drug-courier profile. As a number of judges have pointed out, officials have at one time or another cited any number of behaviors to signify drug-trafficking activity. Individuals have been found to be suspicious when traveling alone or as a pair, carrying too much luggage or too little, walking too slowly or too quickly, appearing too nervous or too calm. Even more concerning is that some of these profiles have relied explicitly on race—for example, the fact that a person is Black and traveling from a particular city—despite the fact that there are likely thousands if not hundreds of thousands of people who match that vague description and have nothing to do with the narcotics trade. What little evidence there is suggests that these profiles not only result in racial disparities but also have little predictive value. In some cases, officers have testified to having approached several hundred people for every eventual arrest.
c. Regulating the use of algorithms and profiles. The various safeguards described in this Section are intended to ensure that algorithms and profiles—to the extent that they are used at all—are used in a manner that maximizes any utility of such tools while minimizing or avoiding altogether the risk of harm.
A threshold requirement for using algorithms and profiles is that public officials—including both high-level agency officials who make the decision to acquire a particular tool and the agency officials who are authorized to use it—have at least a basic understanding of the inputs that the algorithm or profile uses and the limits of any inferences that may be drawn. This enables agency heads to take responsibility both for the tools they are using and for the various inputs and inferences on which they rely. This means that agencies generally should avoid the use of algorithms that are developed using machine-learning techniques that render the algorithms entirely opaque, even to developers themselves. This is particularly important if an algorithm is to be used as a basis for any sort of adverse action against a member of the public. Under these circumstances, it is essential that the government be able to explain the factors on which the algorithm relies and the reason why they are relevant to the determination at issue.
Agencies also should take steps to verify that the algorithm or profile has been evaluated in some manner and has been shown to perform with acceptable accuracy. The rigor with which this assessment occurs may depend in part on the strength of the interests at stake. For example, an algorithm that is used to inform arrest decisions or bail determinations may require much more rigorous testing before it is used than an algorithm that is used to direct patrol.
In addition, agencies should avoid using tools that simply categorize a particular person or location as “low” or “high” risk without specifying what the classification denotes. And they should insist that third-party developers disclose the inputs used to develop an algorithm and provide the agency with enough information to assess on an ongoing basis the efficacy of the predictive tool and any disparate impact it may have. To the extent that an algorithm is used as part of the justification for a search or seizure, or is used to inform a later determination as part of the criminal process, the same information about the reliability of the algorithm, as well as the inputs used, should be made available to the defendant and to the court.
Guarding against the possibility of racial bias or unintended racial disparities in algorithmic or profile-driven policing presents a distinct set of challenges, and the right approach may be context-specific. When it comes to more rudimentary “profiles,” the answer is relatively straightforward: consistent with § 1.11, race and other protected characteristics may be used only as part of a specific and relatively detailed suspect description. See § 1.11. When it comes to predictive algorithms, however, there is a risk that by ignoring race entirely, developers may in fact exacerbate existing racial disparities, particularly in the context of so-called “person-based” predictive policing. For example, if Black individuals are disproportionately likely to be arrested because of their race, or because they live in neighborhoods with a higher police presence, then a past history of arrest may be a much weaker predictor of future criminal activity than it would be for a white individual living in another part of town. One approach, advocated in subsection (d) is to avoid using inputs—such as arrests and convictions for low-level offenses—that are highly influenced by discretionary enforcement practices. Even still, it may not be possible to eliminate racial disparities entirely. Given this, agencies should monitor carefully stop, arrest, and use-of-force data to ensure that the introduction of any predictive policing technology is not exacerbating racial disparities.
Finally, agencies should routinely audit both the algorithm and the underlying data and make all changes that are needed to ensure accuracy and reliability. In particular, agencies should take steps to verify that any databases on which algorithms or profiles rely are up to date. And they should consider avoiding using inputs such as gang affiliations that have been shown to have a high error rate.
Reporter’s Notes
1. Algorithms and profiles, generally. For decades, police officials have relied on data (however rudimentary) about past crimes and offenders to try to “predict” where crime is likely to occur or who might be responsible for it. In the 1880s, London police officials assembled one of the first known offender profiles in an attempt to identify the serial killer known as “Jack the Ripper.” Brian Forst, Errors of Justice: Nature, Sources, and Remedies 77 (2004). In the early 1970s, the Drug Enforcement Administration developed its first “drug courier profile” to help identify potential drug traffickers at major airports. See United States v. Mendenhall, 446 U.S. 544, 562 (1980); see also Philip S. Greene & Brian W. Wice, The D.E.A. Drug Courier Profile: History and Analysis, 22 So. Tex. L.J. 261, 269-270 (1982). Similarly, officers and commanders have long relied on their experience and intuition to identify problem areas in the neighborhoods for which they were responsible. See Anthony A. Braga & David L. Weisburd, Policing Problem Places: Crime Hot Spots and Effective Prevention 35 (2010). And by the 1970s, many policing agencies engaged in at least some basic form of “crime mapping” to identify problem areas and deploy resources. See, e.g., Keith Harries, National Institute of Justice, Mapping Crime: Principle and Practice 4 (1999); Bureau of Justice Assistance, COMPSTAT: Its Origins, Evolution, and Future in Law Enforcement Agencies 3-7 (2013). Both profiling and predictive policing are, in at least some respects, about as old as policing itself.
Modern technology, however, has transformed “predictive” policing and profiling in fundamental ways. In place of pushpins on a map, agencies now rely on “place-based” predictive-policing algorithms to analyze large volumes of crime and other data to identify specific times and places where crime is most likely to occur, to assign “threat scores” to particular addresses or locations, and to identify potential crime patterns. See Andrew Guthrie Ferguson, The Rise Of Big Data Policing: Surveillance, Race, And The Future Of Law Enforcement, 81 (2017); Walter L. Perry et al., RAND Corp., Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations xiv (2013). “Person-based” predictive-policing systems, in contrast, augment the role traditionally played by suspect profiling by generating “heat lists” of potential victims or offenders and “mapping” social networks to identify potential connections between members of criminal organizations. See David Robinson & Logan Koepke, Upturn, Stuck in a Pattern: Early Evidence on “Predictive Policing” and Civil Rights 18 (2016); Andrew G. Ferguson, Policing Predictive Policing, 94 Wash U. L. Rev. 1115, 1118 (2017).
Although the underlying technologies and techniques vary, most predictive-policing systems have several features in common. Many predictive-policing algorithms are developed using a technique known as “machine learning” whereby a software system “trains” itself to see patterns in historical data, such as crime or arrest data. See Robinson & Koepke, supra, at 3; Perry, supra, at 8. In particular, the system learns which geographical characteristics (in the case of place-based systems) or personal characteristics (in the case of person-based systems)—and which combinations of such characteristics—correlate most closely with criminal activity. The system then can use the patterns gleaned from the past data to make inferences about future likelihoods of criminal activity committed at particular locations or by particular people. Michael L. Rich, Machine Learning, Automated Suspicion Algorithms, and the Fourth Amendment, 164 U. Pa. L. Rev. 871 (2016). This “training” process of finding patterns in past data can happen with varying degrees of human involvement, but generally machine-learning systems find patterns without humans instructing them on where to look. Harry Surden, Artificial Intelligence and Law: An Overview, 35 Ga. St. U. L. Rev. 1305, 1311-1312 (2019).
Proponents of these new technologies have emphasized their potential to help agencies make better use of their limited resources by identifying the specific people and places that ought to be the focus of police attention. See Albert Meijer & Martijn Wessels, Predictive Policing: Review of Benefits and Drawbacks, 42 Int’l J. Pub. Admin. 1031, 1033-1034 (2019); see, e.g., Cameron Albert-Deitch, Predictive Policing Crime Prevention Software Successful for APD, Atlanta Mag. (Nov. 10, 2014), https://www.atlantamagazine.com/news-culture-articles/predictive-policing-crime-prevention-software-successful-for-apd/ (reporting the Atlanta Police Department’s early hopes in predictive policing after a pilot program showed crime reductions). They also have touted the potential to reduce racial disparities in policing by relying on “objective” algorithms that are less prone to bias than individual officers. See Justin Jouvenal, Police Are Using Software To Predict Crime. Is It a ‘Holy Grail’ or Biased Against Minorities?, Wash. Post (Nov. 17, 2016); see also National Institute of Justice, U.S. Dep’t of Just., Predictive Policing Symposiums 3-4 (2010) (noting the potential benefits of predictive policing in reducing crime and in “enhancing police legitimacy in the community”).
2. Animating concerns. Despite their purported benefits, existing evidence regarding the accuracy, reliability, and overall utility of predictive-policing systems is mixed at best. Although one study in Los Angeles did indeed find that patrols directed by a predictive-policing algorithm yielded greater reductions in crime than patrols directed by more traditional hotspot mapping, other studies have not found a statistically significant effect on crime. Compare Mohler et al., Randomized Controlled Field Trials of Predictive Policing, 110 J. Am. Stat. Ass’n 1399, 1400 (2015) (“In the three divisions in Los Angeles, police patrols using ETAS forecasts led to an average 7.4% reduction in crime volume as a function of patrol time[.]”), with Priscilla Hunt et al., RAND Corp., Evaluation of the Shreveport Predictive Policing Experiment xiii (2014) (finding that Shreveport, Louisiana’s predictive-policing pilot project found “no statistical evidence that crime was reduced more in the experimental districts than in the control districts”); Jessica Saunders, et al., Predictions Put into Practice: A Quasi-Experimental Evaluation of Chicago’s Predictive Policing Pilot, 12 J. Experimental Criminology 347, 362 (2016) (finding that Chicago Police Department’s Strategic Subjects List, used to predict individuals who might be the future victims of gun violence, did not significantly contribute to a reduction in monthly homicides). Doubts and a lack of conclusive evidence about predictive policing’s efficacy have led police departments to slow their adoption of the new technology or abandon it altogether. See Ashley Murray & Kate Giammarise, Pittsburgh Suspends Policing Program That Used Algorithms to Predict Crime ‘Hot Spots’, Pittsburgh Post-Gazette (June 23, 2020) (announcing the suspension of Pittsburgh’s predictive-policing program due to “potential racial bias”); Avi Asher-Schapiro, In a U.S. First, California City Set to Ban Predictive Policing, Reuters (June 17, 2020), https://www.reuters.com/article/us-usa-police-tech-trfn/in-a-u-s-first-california-city-set-to-ban-predictive-policing-idUSKBN23O31A (reporting on the Santa Cruz mayor’s decision to ban the use of a major predictive-policing technology due to racial bias and efficacy concerns);Leila Miller, LAPD Will End Controversial Program That Aimed to Predict Where Crimes Would Occur, L.A. Times (Apr. 21, 2020), https://www.latimes.com/california/story/2020-04-21/lapd-ends-predictive-policing-program (reporting the end of LAPD’s longtime use of a popular place-based predictive-policing tool due to cost concerns); Jeremy Gorner & Annie Sweeney, For Years Chicago Police Rated the Risk of Tens of Thousands Being Caught Up in Violence. That Controversial Effort Has Quietly Been Ended., Chi. Trib. (Jan. 24, 2020), https://www.chicagotribune.com/news/criminal-justice/ct-chicago-police-strategic-subject-list-ended-20200125-spn4kjmrxrh4tmktdjckhtox4i-story.html (announcing Chicago’s abandonment of its “Strategic Subjects List” program due to efficacy concerns); Mark Puente, LAPD to Scrap Some Crime Data Programs After Criticism, L.A. Times(Apr. 5, 2019), https://www.latimes.com/local/lanow/la-me-lapd-predictive-policing-big-data-20190405-story.html (reporting the end of LAPD’s “chronic offender” list, citing the department’s inability to determine its effectiveness and a lack of effective oversight).
Beyond their overall efficacy, these new technologies and programs have the potential to exacerbate some of the problems endemic with more rudimentary practices and generate entirely new ones. A chief concern with both rudimentary profiles and modern predictive algorithms is racial bias. Commentators have long criticized the explicit use of race as part of the so-called “drug-courier profile.” See, e.g., United States v. Weaver, 966 F.2d 391, 397 (8th Cir. 1992) (Arnold, C.J., dissenting) (“Use of race as a factor simply reinforces the kind of stereotyping that lies behind drug-courier profiles.”); Sheri Lynn Johnson, Race and the Decision to Detain a Suspect, 93 Yale L.J.214, 243 (1983) (noting that using race as a factor in street-crime detention decisions “disadvantages blacks as a group”). There also is evidence to suggest that officers and agencies are more likely to characterize predominantly Black neighborhoods as “high crime areas” as compared with predominantly white neighborhoods, despite having similar underlying crime rates. See Ben Grunwald & Jeffrey Fagan, The End of Intuition-Based High-Crime Areas, 107 Calif. L. Rev. 345, 388-389 (2019).
Although algorithms can be programmed to avoid express reliance on race and other protected characteristics, their use nevertheless may exacerbate existing racial disparities. In particular, when algorithms are trained on historical data that itself was the product of biased or unequal policing, the “predictions” that they generate inevitably reflect these past biases. See Andrew D. Selbst, Disparate Impact in Big Data Policing, 52 Ga. L. Rev. 109, 116 (2017). As Sandra Mayson explains, “if the thing that we undertake to predict—say arrest—happened more frequently to black people than to white people in the past data, then a predictive analysis will project it to happen more frequently to black people than to white people in the future.” Sandra G. Mayson, Bias In, Bias Out, 128 Yale L.J. 2218, 2224 (2019). Studies consistently show that in many jurisdictions, Blacks and Latinx people are stopped, searched, and arrested at disproportionate rates. See, e.g., Ojmarrh Mitchell & Michael S. Caudy, Examining Racial Disparities in Drug Arrests, 32 Just. Q. 288, 309 (2015) (noting substantial racial disparities in arrests for drug possession despite comparable rates of use). Algorithms trained on data that reflect these sorts of enforcement practices will “over-predict” Black and Latinx individuals, and the neighborhoods they live in, as crime risks. See Mayson, supra at 224; see also Rashida Richardson et al., Dirty Data, Bad Predictions: How Civil Rights Violations Impact Police Data, Predictive Policing Systems, and Justice, 94 N.Y.U. L. Rev. Online 192, 203-205 (2019) (questioning the predictive value of algorithms trained on historical data from departments that have been found to engage in discriminatory policing). When agencies rely on those predictions to make policing decisions, they risk creating a “feedback loop”: officers focus more heavily on minority neighborhoods improperly labeled “high risk” and make more arrests in those neighborhoods, resulting in crime data that reflects an increasingly skewed version of reality. See Andrew G. Ferguson, Policing Predictive Policing, 94 Wash. U. L. Rev. 1115, 1183 (2017); Cathy O’Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (2016).
A related concern is with accuracy and reliability. Though algorithms predict future crime risk by learning from available criminal-justice data, that data often is incomplete and unrepresentative of the true incidence of crime in a community. See Lynn Langton et al., Bureau of Justice Statistics, Department of Justice, Victimizations Not Reported to the Police, 2006–2010 1 (2012) (noting that between 2006 and 2010, nearly half of all violent victimizations were not reported to the police). Errors in police databases can skew predictions further. See Walter L. Perry et al., RAND Corp., Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations 119-122 (2013) (describing various systematic biases in crime data—for example, the tendency for property owners to report overnight burglaries in the morning, thereby inadvertently creating the perception of an early-morning crime spree); Office of the Inspector General, City of Chicago, Chicago Police Department’s “Gang Database” 44 (Apr. 11, 2019) (expressing serious concern about the accuracy of Chicago’s gang database given the absence of any quality-control mechanisms).
Finally, there are concerns that agencies and officers may assign more weight or creditability to the information generated by policing algorithms than is in fact warranted. Because “risk” assessments often are relative, an individual or neighborhood may be labeled as “high” risk even when the risk of crime is low in absolute terms. See e.g., John Logan Koepke & David G. Robinson, Danger Ahead: Risk Assessment and the Future of Bail Reform, 93 Wash. L. Rev. 1725, 1779 n.223 (2018) (“[T]hough an individual may be assessed as ‘high risk,’ the rate of reoffense for the ‘high risk’ group may resemble a probability that is rather low, not ‘high.’”); Jessica Saunders et al., Predictions Put into Practice: A Quasi-Experimental Evaluation of Chicago’s Predictive Policing Pilot, 12 J. Experimental Criminology 347, 351 (2016) (“[W]hile the models can identify increased risk, the overall risk can still be very low. Indeed, a very high risk person for homicide might have a risk rate of 1% per year.”); Melissa Hamilton, Adventures in Risk: Predicting Violent and Sexual Recidivism in Sentencing Law, 47 Ariz. St. L.J. 1, 22 (2015) (noting that risk labels do not convey the actual probability that an individual is dangerous and may mistakenly lead decisionmakers to believe that “high risk” means “more likely than not”). Officers also may attach undue weight to the apparently “objective” algorithm despite its limitations. See Alexander Babuta & Marion Oswald, Royal United Services Institute, Data Analytics and Algorithmic Bias in Policing 15 (2019) (“A factor which may arise during deployment [of a predictive algorithm] . . . is the risk of automation bias, the tendency to over-rely on automated outputs and discount other correct and relevant information”); Robert Brauneis & Ellen P. Goodman, Algorithmic Transparency for the Smart City, 20 Yale J. L. & Tech. 103, 126 (2018) (“When the ‘machine says so,’ it can be difficult for rushed and over-extended human decision makers to resist the edict.”).
All of these problems are exacerbated by the lack of transparency around algorithms provided by third-party vendors, which often prevents agencies from understanding the systems they use. Id. at 152-153 (highlighting many vendors’ reluctance to disclose key information about their systems to agencies, including “design choices, data selection, factor weighting, and validation designs,” and how this reluctance stands in the way of public oversight); Elizabeth E. Joh, The Undue Influence of Surveillance Technology Companies on Policing, 91 N.Y.U. L. Rev. Online 101, 119, 124-125 (2017) (“When police departments agree to purchase or contract for big data tools, they typically bargain for the results, but not the proprietary algorithms that produce them.”); Rebecca Wexler, Life, Liberty, and Trade Secrets: Intellectual Property in the Criminal Justice System, 70 Stan. L. Rev. 1343, 1367 (2018) (noting assertions of “trade secrecy” by police departments to avoid disclosing information about predictive policing tools and other law enforcement technology).
3. Appropriate safeguards. The safeguards outlined in subsections (a) through (e) are designed to address the set of concerns outlined here. Subsections (a) and (b) address the problems of reliability and interpretation by ensuring that agencies take steps to understand how a particular algorithm was developed, the basis for its predictions, and the limitations on what it can in fact accomplish. A number of agencies have taken important steps in this direction. For example, the Chicago Police Department worked with researchers from the RAND Institute to evaluate the department’s person-based predictive system. The evaluation ultimately prompted the department to abandon the program after the study cast serious doubt on its effectiveness. Office of Inspector Gen., City of Chi., Advisory Concerning the Chicago Police Department’s Predictive Risk Models 2, 4 (2020). A number of jurisdictions require police departments to prepare impact statements when adopting new surveillance technologies. A similar framework could be used to encourage agencies to articulate the benefits and potential harms of any predictive tools. See, e.g., Andrew D. Selbst, Disparate Impact in Big Data Policing, 52 Ga. L. Rev. 109, 173 (2017) (emphasizing the value of an algorithmic impact-statement process in requiring agencies to consider “all reasonable alternatives” before adopting a new process or tool); Dillon Reisman et al., AI Now Institute, Algorithmic Impact Assessments: A Practical Framework for Public Agency Accountability 16 (2018) (same). Finally, jurisdictions that rely on risk assessment in the context of bail or early-release determinations have adopted a variety of approaches to address these sorts of concerns, including training for judges and administrators, as well as efforts to better communicate the significance and limitations of various risk scores. See, e.g., Brandon L. Garrett, Federal Criminal Risk Assessment, 41 Cardozo L. Rev. 101, 124-125 (2019) (highlighting the federal FIRST STEP Act’s requirement, codified at 18 U.S.C. § 3632(f), that Bureau of Prison employees receive initial training and continuing education when using the new recidivism risk-assessment system mandated by the act).
Subsections (c) and (d) are designed to reduce the risk that the use of predictive algorithms will exacerbate existing biases in criminal-justice decisionmaking. Many agencies now have policies in place to prohibit the use of race unless it is part of a specific suspect description; and for more rudimentary “profiling” this approach ensures that agencies do not inappropriately take race into account. See, e.g., Seattle Police Department Manual § 5.140.3 (2019) (prohibiting the use of race and other protected characteristics unless part of a specific suspect description); Los Angeles Police Department Policy § 345 (same); see also § 1.11 (citing additional policies). For algorithm-based “predictive” technologies, however, simply avoiding the use of race is not sufficient because, as discussed above, ostensibly “race neutral” variables like stops and arrests may produce racial disparities if they themselves are the product of racially disparate enforcement practices. A number of developers have tried to address this concern by avoiding the use of certain inputs entirely. For example, the developers of PredPol, a place-based predictive-policing system, use reported-crime-incident data to train their prediction systems instead of arrest records, since victim crime reports may be less impacted by discretionary enforcement practices. See P. Jeffrey Brantingham et al., Does Predictive Policing Lead to Biased Arrests? Results from a Randomized Controlled Trial, 5 Statistics & Pub. Pol’y 1, 2 (2018). Others have cautioned, however, that omitting certain variables may impose accuracy costs that themselves may disproportionately impact communities of color. See Sandra G. Mayson, Bias In, Bias Out, 128 Yale L.J. 2218, 2265 (2019). In response, some have suggested that a better approach might to develop algorithms in a way that affirmatively takes race into account in order to adjust for past disparities. See, e.g., Jon Kleinberg et al., Algorithmic Fairness, 108 AEA Papers & Procs. 22-27 (2018) (demonstrating in the context of college admissions that including race as a variable in machine-learning algorithms can both improve the accuracy of predictive models and increase the diversity of entering classes). It is possible, however, that this approach might violate modern equal-protection principles. See Aziz Z. Huq, Racial Equity in Algorithmic Criminal Justice, 68 Duke L.J. 1043, 1133 (2019) (speculating that using race as an explicit factor in algorithmic decisions, even to combat inequities, “runs headlong into the anticlassification rule of equal protection doctrine”); but see Crystal Yang & Will Dobbie, Working Paper, Equal Protection Under Algorithms: A New Statistical and Legal Framework (October 2019), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3462379 (suggesting there may be ways to incorporate race to adjust for potential biases in ways that are consistent with equal protection principles). It may be that in some circumstances, agencies may need to simply forego efforts to “predict” certain outputs to begin with or to adjust the manner in which predictions are used in order to mitigate the potential for racially disparate harm. Mayson, Bias in, Bias Out, supra, at 2225-2226. At the very least, agencies should carefully monitor demographic data on stops and arrests to ensure that the introduction of any predictive technologies does not produce racially disparate effects.
Finally, agencies should conduct periodic audits of any predictive algorithms or technologies that they employ and make the changes indicated by the audit results. In recent years, a number of agencies have conducted comprehensive audits of various algorithm-based programs and concluded that they were considerably less effective than expected. See e.g., Jessica Saunders, Predictions Put into Practice: A Quasi-Experimental Evaluation of Chicago’s Predictive Policing Pilot, 12 J. Experimental Criminology 347 (2016) (finding that Chicago Police Department’s “Strategic Subjects List” did not significantly contribute to a reduction in homicides). Several of these audits also pointed to precisely the sorts of concerns with bias or reliability described above. See, e.g., Mark Smith, Los Angeles Police Commission, Review of Selected Los Angeles Police Department Data-Driven Policing Strategies 29-30 (2019) (citing various problems with the data collected for the LAPD’s PredPol and LASER systems); Office of the Inspector General, City of Chicago, Chicago Police Department’s “Gang Database” 44 (Apr. 11, 2019) (raising serious concerns about the lack of quality controls around the department’s gang database). A number of jurisdictions already have ordinances in place requiring the sorts of periodic audits urged here. See, e.g., Cambridge, Mass., Municipal Code ch. 2.128 (2018) (requiring the police to evaluate the effectiveness of surveillance technologies, both prior to adoption and on a continuing basis once a technology is in use); Seattle, Or., Municipal Code § 14.18.060 (2017) (requiring the Inspector General for Public Safety and City Auditor to conduct annual reviews of all surveillance technologies).