Thomson Reuters Foundation reported on August 9 that a group of the U.S. House of Representatives has asked the Justice Department to explore the use of so-called “artificial intelligence” (AI) technology to monitor the telephone communication of imprisoned people in order to prevent violent crime and suicide.
This is not a hypothetical job: Leo technologies, a company “built by the police for the police”, has provided a service that automatically monitors the phone calls of detainees and their relatives.
As linguists who study the development and application of speech recognition and other language technologies, including their ways of working (or not working) in different language types, we want to clearly and strongly point out that this is a terrible idea both technically and morally.
We oppose large-scale monitoring in any way, especially in the case of vulnerable groups without their consent or unable to opt out. Even if it can be shown that such surveillance is in the best interests of detainees and their communities – we do not believe it is possible – attempts to automate the process will expand the potential harm.
We urge Congress and the Department of justice to abandon this path and avoid incorporating automatic forecasting into our legal system.
Using a “keyword and phrase based” approach (such as Leo technologies’ description of its products), the main hypothetical benefit of this technology for prisoners – Suicide Prevention – is not feasible. Even Facebook’s suicide prevention program is under review by legal and ethical scholars, finding that keywords are an ineffective method because it does not consider context. In addition, humans often regard the output of computer programs as “goals”, so they make decisions based on wrong information without knowing that it is wrong.
Even if the ability to prevent suicide is specific and demonstrable, it is not the case, and it also has great potential harm.
Automatic transcription is a key part of these products. The effectiveness of speech recognition system depends on the close matching between its training data and the input received in the deployment environment. For most modern speech recognition systems, this means that the farther a person’s voice is from the newscaster standard, the less efficient the system is in correctly transcribing their words.
Such systems not only undoubtedly output unreliable information (although it seems very objective), but also fail more frequently for those who most often fail in the American judicial system.
A 2020 study, including Leo technologies’ Amazon service for voice transcription, confirmed earlier findings that African American English users have about twice the word error rate as white users. Considering that the incarceration rate of African Americans is five times that of whites, these tools are very unsuitable for their application and may increase the already unacceptable racial differences.
This surveillance includes not only the prisoners, but also the people who talk to them, which is an unnecessary violation of privacy. Adding so-called “artificial intelligence” will only make the situation worse: machines can not even accurately transcribe warm and comfortable family language, but also bring false “objectivity” light to inaccurate transcripts. Should those whose relatives are imprisoned bear the responsibility of defending allegations of false records based on what they say? This invasion of privacy is particularly annoying, as imprisoned persons and their families usually have to pay high fees for telephone calls in the first place.
We urge Congress and the Department of justice to abandon this path and avoid incorporating automatic forecasting into our legal system. Leo technologies claims to “transform the law enforcement paradigm from reactive to predictive”, which seems to be incompatible with a judicial system that must prove guilty.
Finally, we urge all concerned to remain highly skeptical of the “Ai” application. This is especially true when it has a real impact on people’s lives, especially when these people are as vulnerable as prisoners.