When AI Breaks Trust, a Button Won’t Fix It
India’s digital arrest epidemic as a case study in AI-era governance failure
The digital arrest epidemic is less a story about banking fraud and more an early stress test of how governments respond when AI makes authority indistinguishable from performance. In recent months, India has woken up to the scale of a crisis hiding in plain sight. According to official estimates, scammers have defrauded citizens of nearly ₹3,000 crore by impersonating police officers, regulators, and enforcement agencies keeping victims on video calls for hours, threatening arrest, and coercing them into transferring money “for verification.”
These scams work because the victims are human. And they are now supercharged by AI. Voice cloning, deepfake video, and automated scripting have dramatically lowered the cost of simulating authority. What once required a trained con artist now runs at scale, on demand, in multiple languages, with synthetic faces that blink and nod with the cadence of a real official.
In response, the Ministry of Home Affairs (MHA), along with the RBI, is evaluating two headline interventions: an emergency Kill Switch inside banking apps, and a form of fraud insurance or shared risk pool across banks.
On paper, both sound sensible, but they rest on a fragile assumption that victims are still acting with agency. My concern is that these tools may arrive after trust has already collapsed. And once trust fails, buttons and insurance clauses don’t help much.
Any defence that relies on the victim “doing the right thing” at the moment of maximum cognitive pressure is structurally fragile. This is not a UX problem. It is a governance problem.
The Policy Response on the Table
Let’s start with what the government is actually proposing. The Kill Switch is an emergency button embedded inside banking and UPI apps. If a user suspects fraud, they can trigger it to instantly freeze outgoing transactions, stopping the “layering” process where stolen funds are split across dozens of mule accounts and disappear within minutes.
Technically, this is elegant. Behaviorally, it is far less convincing. In a digital arrest scam, victims are kept in a state of intense, manufactured fear. They are told they are being monitored. They are warned not to hang up, not to touch other apps, not to “tamper with evidence.” In that psychological state, expecting someone to calmly locate and press a Kill Switch assumes clarity that simply does not exist.
If someone says they are police, our first instinct is to obey, not to experiment. That instinct is exactly what scammers exploit and exactly what AI-generated authority makes worse. When the face on screen is photorealistic, the voice indistinguishable from a real officer, and the “case number” auto-generated, the psychological override is near-total.
A Kill Switch assumes the victim realises they are under attack before compliance. Digital arrests augmented by AI impersonation work precisely because that realisation comes too late.
The second proposal, a pooled insurance mechanism where banks and insurers jointly absorb losses from digital fraud, is a more structural shift. Until now, most cyber insurance excludes first-party fraud where the victim authorises the transaction themselves under manipulation. That exclusion is increasingly untenable. But insurance introduces its own governance risks: moral hazard for customers, weakened prevention incentives for banks, and a slow, contested claims process that in India’s grievance redressal system could easily stretch into years of litigation.
Scammers Are Not Opportunists, They Are Adversarial Systems
There is an assumption quietly embedded in the Kill Switch proposal that deserves scrutiny: that scammers will treat it as a deterrent rather than a design constraint. They will not. This is where the AI governance framing becomes essential.
In machine learning, we distinguish between a model that is robust under normal inputs and one that has been adversarially tested exposed to inputs specifically designed to find its failure modes. Scam operations function like adversarial systems. Every new safeguard introduced becomes, for them, a new signal to optimise against. The moment a Kill Switch becomes widely known, it will be absorbed into scam scripts not as a threat, but as a prop.
One likely adaptation is the pre-emptive uninstall. Digital arrest scams already begin with isolation: victims are instructed not to speak to anyone, not to hang up, not to verify. Adding “please uninstall all banking apps so criminals cannot track your phone” is a trivial extension of that script. By the time the victim realises something is wrong, the Kill Switch no longer exists on the device.
Another adaptation builds on remote-access tactics already in use. Screen-sharing and remote-control apps are now routine in high-value scams, framed as “verification” or “security checks.” Once a scammer has remote access, the Kill Switch becomes irrelevant or worse, weaponisable. A scammer could trigger a freeze on a secondary account in front of the victim: “See, we have secured your funds.” The safety feature becomes the deception.
There is also a subtler risk. The existence of a real Kill Switch creates a new vocabulary for scammers to exploit.
“Your account has been temporarily frozen.”
“We have activated the emergency security lock.”
“This is a Supreme Court-mandated hold.”
These phrases already circulate in scam calls. Official infrastructure doesn’t eliminate them; it lends them plausibility.
And if banking systems become harder to exploit, scammers will simply pivot. We are already seeing increased demands for cryptocurrency, physical gold, and gift cards assets that sit entirely outside the banking kill zone. A frozen account offers no protection if the victim is convinced to courier gold or transfer crypto. At that point, recovery is functionally impossible.
We are not dealing with opportunists. We are dealing with adaptive systems that treat every new policy as a new optimisation target. Our governance frameworks are not yet built for that.
What Other Countries Have Learned
India is not alone in grappling with this, and the international evidence carries a consistent lesson: the earlier the intervention in the scam chain, the more effective it is.
The UK mandated reimbursement for authorised push payment fraud. The result has been mixed: safety improved, but banks became extremely aggressive in blocking suspicious transfers, sometimes freezing legitimate payments. Safety and friction turned out to be harder to separate than expected.
Singapore’s approach is more instructive. Alongside a Kill Switch, banks introduced a “Money Lock” feature allowing users to quarantine a portion of their savings so it cannot be transferred digitally at all. To access it, you must go to an ATM or branch. This works not because it is smarter software, but because it introduces physical friction that a remote actor human or AI-assisted cannot override. It is, in effect, governance by design rather than governance by feature.
Australia found that telco-level interventions blocking scam calls before they reach users are often more effective than banking-side tools. The upstream metaphor holds: stop the attack vector, not just its consequences.
The pattern is consistent. Once a victim is psychologically captured, downstream tools struggle. Effective governance intervenes before cognition collapses, not after.
The Psychological Lock We Keep Ignoring
This is where policy discussions repeatedly fall short, and where the AI and education lens matters most.
A digital arrest is not a technical failure. It is a cognitive hijack. The victim’s decision-making is overridden by fear, authority, and urgency three levers that AI-generated impersonation can now apply with unprecedented precision and scale. Teaching people what button to press assumes they are still thinking like autonomous users. They are not. Their cognition has been captured.
In my training sessions, the most effective shift came not from showing people a Kill Switch or explaining fraud statistics, but from pre-exposure walking participants through realistic scam scenarios before they encountered them. Once people experience how authority is simulated, how urgency is manufactured, and how scripts unfold, they develop a cognitive pause that did not exist before. Not always. But earlier, and more often.
This is, in essence, adversarial training for humans. It is the same principle that makes red-teaming effective for AI systems: you cannot build robustness through reaction alone. You build it through rehearsal, through prior exposure to the exact patterns that will be used to exploit you. A model that has never seen adversarial inputs will fail gracefully under normal conditions and catastrophically under attack. So will a person.
Yet behavioral infrastructure this kind of sustained, realistic, pre-exposure education remains chronically underfunded and under-institutionalised. It is slow. It resists dashboards. It cannot be announced in a press release or demonstrated in a pilot. Governments, understandably, reach for the deployable instead.
The AI Governance Problem at the Heart of This
There is a deeper structural issue here that the digital arrest epidemic makes visible, and that anyone working at the intersection of AI, governance, and education should recognise.
We are building AI systems that can simulate authority at scale systems capable of generating convincing voices, faces, documents, and interactions in real time. We have not built the civic infrastructure to help people navigate a world where authority can be synthesised on demand. That gap is not a bug in a banking app. It is a governance failure.
The Kill Switch treats this as a transactional problem: user gets confused, user presses button, transaction stops. But the actual failure mode is epistemic. The victim does not know who to trust, because AI has made trust signals faces, voices, credentials, case numbers trivially replicable. No button resolves that. Only infrastructure does.
Consider the parallel to AI liability debates. In discussions around the EU AI Act and similar frameworks, one of the central tensions is between voluntary commitments and mandatory structural safeguards. Voluntary commitments like a Kill Switch that users may or may not press depend on user behaviour holding under pressure. Mandatory structural safeguards like cooling-off periods, escrow holds, or telco-level blocking do not. They protect by design, not by hope.
The same logic applies here. Insurance pools, if poorly structured, replicate the moral hazard problem that unregulated AI development creates: if the cost of harm is externalized, incentives to prevent harm weaken. Liability frameworks that mandate incident reporting, tiered accountability, and upstream intervention are harder to build but far more durable.
What Structural Governance Actually Looks Like
If we accept that victims under digital arrest are not acting freely, the obvious question follows: why are we designing systems that require them to act at all? The most effective interventions do not depend on victims recognising a scam. They assume confusion as the baseline and protect accordingly.
Mandatory cooling-off periods for large transfers to new beneficiaries. Default transaction holds that require time, not judgment. Physical-world friction like Singapore’s Money Lock that remote actors cannot override. Telco-level blocking that prevents AI-generated scam contact from reaching users in the first place. Behavioral red-flag detection built into transaction systems that triggers human review automatically, without waiting for the user to panic and press a button.
These are blunt instruments. They will frustrate legitimate users. They require coordination across banks, telecom providers, regulators, and law enforcement. They are expensive and slow to build. But they share a critical property: they do not rely on the victim’s cognition holding under adversarial pressure.
That property matters more than elegance. If behavior is hard to change at scale and it is then systems must be designed to fail safely when behavior breaks down.
The Education Imperative
None of this means behavioral education is irrelevant. It means it must be redesigned.
The current model awareness campaigns, warning banners, fraud statistics in press releases is information-transfer pedagogy. It assumes that if people know about scams, they will recognise them. The evidence does not support this. Knowledge and behaviour under stress are different systems.
Effective education, in this context, looks more like simulation than instruction. It exposes people to the experience of being targeted, not just the concept of it. It builds the muscle memory of pausing, verifying, and breaking the urgency loop before that loop is triggered in a high-stakes real-world encounter. It teaches people to recognise not just the content of a scam, but the structure: the manufactured time pressure, the simulated authority, the isolation from trusted others.
This is harder to scale than a leaflet. It requires educators who understand both social engineering and cognitive science. It requires institutions schools, community organisations, banks, workplaces to treat digital literacy not as a one-time module but as ongoing civic infrastructure. And it requires honest acknowledgment that even well-trained people will sometimes fail, which is why structural safeguards must exist in parallel.
Digital literacy in the age of AI is not about knowing what a scam looks like. It is about building the capacity to pause when your cognition is under attack. That is a different skill, and it requires different pedagogy.
The Real Question
The Kill Switch and fraud insurance are not useless. They will help in edge cases. They will look reassuring in press releases. They will create the appearance of action. But they are reactive tools aimed at victims who have already lost agency, and they are being deployed into an environment where AI has permanently lowered the cost of the attack.
Digital arrests succeed not because India lacks buttons or insurance pools, but because we have not built the infrastructure behavioral, institutional, structural to help people navigate a world where authority is synthetic and trust signals are unreliable by default.
From a policy perspective, the appeal of deployable technical fixes is understandable. A button is tangible. It can be announced, piloted, and measured. Behavioral and structural change is slow, messy, and resistant to dashboards. But the instinct to reach for the demonstrable over the durable is exactly the pattern that produces governance frameworks that look responsive and perform poorly.
The most important thing the digital arrest epidemic is telling us is not that we need better banking features. It is that we need a new model of civic infrastructure for an AI-mediated world one that assumes adversarial conditions as the baseline, invests in human resilience before the moment of attack, and builds friction into systems rather than relying on individuals to supply it under pressure.
Digital arrest scams are, in miniature, the governance problem of the AI age: how do you protect people from systems specifically designed to collapse their judgment? Technical features help at the margins. What scales is infrastructure the slow, unglamorous work of building societies that can recognise simulated authority, pause under pressure, and demand friction where it matters.
That is not a product feature. It is a civic capability. And until policy treats it as one, the numbers will keep rising.
References:
UK Mandatory APP Fraud Reimbursement
The official regulator page is the clearest starting point:
https://www.psr.org.uk/information-for-consumers/app-fraud-reimbursement-protections/
For a detailed policy breakdown, the Freshfields briefing is excellent:
https://www.freshfields.com/en/our-thinking/briefings/2024/09/authorised-push-payment-fraud-a-new-mandatory-reimbursement-regime-for-uk-psps
Singapore — Money Lock
The official Singapore government MoneySense page explains it plainly:
https://www.moneysense.gov.sg/scams/moneylock/
The MAS parliamentary reply has the adoption data (61,000 accounts, S$5.4 billion locked):
https://www.mas.gov.sg/news/parliamentary-replies/2024/oral-reply-to-parliamentary-question-on-money-lock
The Ministry of Digital Development page gives the broader anti-scam architecture context:
https://www.mddi.gov.sg/media-centre/press-releases/measures-to-protect-singaporeans-against-online-scams/
Australia — Telco-level blocking
The ACMA (Australian Communications and Media Authority) is the primary body. Their combating scams page:
https://www.acma.gov.au/combating-phone-scams
For the scale of what’s been blocked (2.3 billion scam calls since 2020):
https://www.acma.gov.au/publications/2025-02/report/action-scams-spam-and-telemarketing-october-december-2024
And the new legislative framework passed in February 2025:
https://www.twobirds.com/en/insights/2025/australia/explainer-australias-new-scam-prevention-framework