Redesigning Arrest-Related Deaths Program Using Machine Learning Arrest | RTI Innovation Advisors
When criminal justice researchers and law enforcement agencies needed a single, official way to more accurately collect data and measure the number of arrest-related deaths (ARD) in the United States, RTI International developed the first reliable process to do so. The Bureau of Justice Statistics worked with our Center for Data Science to redesign the ARD program using a hybrid approach: identify potential arrest-related deaths through online media reports (Phase 1), and follow-up with a survey of law enforcement agencies and medical examiner/coroners’ offices to confirm identified deaths, verify facts about the decedents, and find others not identified through media review (Phase 2).
As the project got underway, media alerts revealed over 1,000,000 relevant news stories to review per month. To reduce human labor without reducing the accuracy of identified decedents, we developed an automated coding and classification pipeline of online media data using text analytics and machine learning techniques. The pipeline identified potential cases that could be verified by human coders to form a comprehensive and timely count of ARDs.
Maintaining accuracy while reducing human burden
Our machine learning-driven pipeline reduced the number of new articles needing manual review by over 99 percent. To achieve this, tens of thousands of articles were ingested nightly and run through processing steps including deduplication, named entity extraction, and relevance classification via machine learning. We worked with our client to define the most important accuracy metric (in this case, at least 95 percent of decedents retained) and implemented checks to ensure our output remained within the acceptable amount of error.
While we focused the majority of manual review on the reduced set of articles produced by our pipeline to maximize efficiency, we also labeled a one percent random sample of all articles monthly. We used this to continually validate our model. Periodically we compared the model with human coders by examining a list of cases where the two strongly disagreed. Finally, we merged our results with lists of decedents from multiple external sources to confirm our false-negative rate. We identified the cause of any missing decedents and tweaked parameters as necessary to ensure such mistakes would not be repeated.
Our methodology was successful in standardizing data collection so that the Justice Department no longer needs to rely on voluntary reporting by local law enforcement. The BJS published our technical report, and our method for counting ARDs — as reported in The Guardian — is the most comprehensive official effort so far to accurately record the number of deaths at the hands of American law enforcement and provide the “national, consistent data” described by the U.S. Attorney General. Numerous media outlets covered the story including fivethirtyeight.com who included it among the Best Data Stories of 2016 and The Measure of Everyday Life podcast.
Originally published at https://rtiinnovationadvisors.org.
