Apple and Google are being pushed into a more direct role in the fight against AI abuse after San Francisco ordered both companies to remove dozens of so-called nudify apps from their app stores. The apps use AI to alter images and create non-consensual intimate deepfakes, a category regulators are increasingly treating as sexual abuse infrastructure rather than ordinary consumer software.
The order from San Francisco City Attorney David Chiu’s office gives the companies 28 days to respond and warns that they could face civil penalties under California law. The city argues that Apple and Google have been repeatedly notified about apps that sold non-consensual intimate-image tools and processed payments through their platforms.
Apple says such apps are forbidden on the App Store and that it has removed some of the apps identified while reviewing others for policy violations. Google says the Play Store apps named in the letter have been suspended and that it has also restricted related search terms. The dispute now centres on whether removing individual apps after reports is enough, or whether app stores must proactively prevent this category from returning under new names.
The app-store angle matters because Apple and Google control the main distribution rails for mobile software. If a harmful AI tool is available through an official store, users may assume it has passed some level of review. That trust is exactly why regulators are now asking whether the platform operators share responsibility when abuse-enabling apps are distributed and monetised through their systems.
This is different from a random website hosting harmful tools. App stores review apps, process payments, collect fees, manage developer accounts and control search visibility. That gives them more power to police abuse, but it also gives regulators a clearer target when harmful apps keep appearing.
The same broader problem has been visible in other AI abuse stories, including the recent case where xAI sued a Grok user accused of creating illegal deepfake child abuse images. The common thread is that generative AI has made visual abuse easier to produce, harder to detect quickly and more damaging when it spreads.
California has already criminalised knowingly facilitating or recklessly aiding the creation of non-consensual intimate deepfakes. A 2025 state law also gave victims a civil path against third parties that facilitate this kind of abuse. San Francisco is now using that legal direction to pressure two of the world’s most powerful app-store operators.
That matters because many AI safety problems have been handled through platform policy rather than law. A company removes an app, updates a guideline, blocks a keyword or bans an account. Regulators are now signalling that, for some categories of AI harm, policy cleanup may not be enough.
The next question is enforcement. If app stores remove dozens of apps but similar tools reappear under new branding, regulators may demand stronger developer screening, payment monitoring, search controls and automated detection. That will not be easy, but the alternative is a recurring whack-a-mole cycle where victims carry most of the burden.
For users, the message is clear: AI-generated intimate-image abuse is not a harmless prank or a grey-area editing trick. It can cause real harm to victims, especially women, girls and public-facing creators whose photos are easier to find online.
For developers, the order is another warning that AI apps will be judged by use case, not just by technical capability. A model that can edit images may be legal and useful in one product context, while a product designed around non-consensual sexual manipulation can create serious legal exposure.
For Apple and Google, the case is a test of whether their app-store power comes with a stronger duty to prevent AI abuse at distribution level. The more generative AI apps flood mobile stores, the harder it will be for platform owners to argue that they are only neutral marketplaces.
The bigger shift is that AI safety is moving from model labs to every layer of technology distribution: app stores, payment processors, social platforms, cloud providers and device makers. The companies that profit from access will increasingly be expected to help stop abuse before it reaches users.