Recently, Apple released Three new features designed to ensure the safety of childrenOne of them, labeled “Communication Security in Messages,” will scan iMessages of people under the age of 13 to identify and obscure pornographic images, and alert parents when children open or send messages containing such images. At first glance, this sounds like a good way to reduce the risk of young people being exploited by adult predators. But this may do more harm than good.
Although we hope that all parents want their children to be safe, this is not a reality for many children. LGBTQ+ youth especially Are at high risk of parental violence and abuse, The probability of being homeless is twice that of others, accounting for 30% of the foster care system.In addition, they are More likely to send explicit pictures As Apple is trying to detect and report, part of the reason is the lack of sex education. Reporting children’s texting behavior to parents can reveal their sexual preferences, which can lead to violence and even homelessness.
Since the technology on which this feature is based is unlikely to be particularly accurate in detecting harmful and explicit images, these hazards have been amplified. Apple will, it says, Using “machine learning on the device to analyze image attachments and determine whether the photos are pornographic.” All photos sent or received by Apple accounts held by persons under 18 years old will be scanned if this account is associated with a designated parent If the account is linked, a parent notification will be sent.
It is not clear how effective the algorithm is, nor what it will detect.Some pornographic content detection algorithms mark content based on Skin display percentageFor example, the algorithm might tag a picture of a mother and daughter on the beach in a bathing suit. If two young people send each other a photo of a celebrity in revealing clothes, their parents may be notified.
Computer vision is a well-known problem. Existing algorithms (such as those used for face detection) have known deviations, including the following facts: Often fail to detect non-white faces. The risk of Apple’s system inaccuracy is particularly high because Most academically published nude detection algorithms Receive adult image training. Apple did not provide transparency regarding the algorithms they used, so we don’t know how effective it is, especially when it comes to detecting images taken by young people themselves—probably the most worrying.
The accuracy of these algorithms is worrying because they may be inconsistent with the expectations of young people.When we overzealously declare that behavior is “bad” or “dangerous”—even when sharing swimsuit photos among teenagers—we Fuzzy young people’s detection ability When something really harmful happens to them.
In fact, even with this feature, we are teaching young people They have no right to privacy. Deprive young people of their right to privacy and consent, on the contrary UNICEF evidence-based guide Recommendations to prevent online and offline child sexual exploitation and abuse.In addition, this feature not only has the risk of causing harm, but also Opened the door to a wider intrusion into our private conversations, Including government invasion.
When designing technology to ensure that young people are safe online, we need to do better. This first requires potential victims to participate in the design of the security system.With more and more sports Design justice It is recommended that involving the people most affected by technology is an effective way to prevent injuries and design more effective solutions.So far, young people have not yet become part of the conversations between technology companies Or researcher Yes have. they need.
We must also remember that technology cannot solve social problems alone. It is important to focus resources and energy on preventing harmful situations first. For example, by following UNICEF guidelines and research-based recommendations, expanding comprehensive, consent-based sex education programs to help young people safely understand and develop their sexual behavior.
This is an opinion and analysis article; opinions expressed Author or author Not necessarily those Scientific american.