Humans Caught 143 of 1,053 Dangerous Commands.

A frontier lab ran a controlled study on whether human approval prompts work. They caught 13.6% of planted dangerous commands. A classifier caught 89%. On Friday, Anthropic makes the classifier the default and retires the prompt. I have to update something I published last week.

I argued that human approval steps are receipts rather than controls and cited research putting intervention success between 9% and 26%. Anthropic just ran the experiment properly and got 13.6%—inside my range, at the bottom of it. The argument was right. The remedy I proposed was not the one they chose.

This is the most operationally significant announcement of the month for anyone running automations, and it deserves more scrutiny than the coverage has given it—including scrutiny of the conclusion I find most flattering to my own position.

What Changes on Friday

Starting 14 August, auto mode becomes the default for new Claude Code sessions on Pro, Max and Team plans, replacing the prompt that asked users to approve each eligible action.

The Mechanism

• Tool calls route through a transcript classifier running on Sonnet 4.6, which judges each proposed action against the user's stated instructions before it executes.
• Shell commands, web fetches, external tools, and out-of-project file operations all pass through it.
• Claude proceeds autonomously unless it classifies an action as irreversible, destructive, or aimed outside your environment — in which case it still asks.
• After three consecutive denials, Claude Code reverts to manual approval.
• New safety features include prompt injection screening and customisable hard deny rules.
• Enterprise, API, Bedrock, Google Cloud and Microsoft Foundry remain opt-in for now, with the default expected to roll out there over the following month.

Anthropic has also stopped charging Pro, Max and Team plans for the tokens the classifier consumes, as of 7 August. That matters — a safety layer you pay for by the token is a safety layer people turn off.

The Study, and Why It Is Better Than Mine

Last week I made an argument about human approval steps using research from oversight literature and practitioner frameworks. It was, I said at the time, one citation layer removed from the primary studies.

Anthropic has now done something better: a controlled experiment, on the specific mechanism, at scale.

The design, as reported: 1,053 paid professional testers worked in a purpose-built environment. They knew they were being observed but not what was being measured. Mid-session, an obviously harmful command replaced a normal permission request.

Human reviewers blocked 143 of the planted dangerous commands. The classifier blocked 937.

13.6% against 89%.

Two supporting figures make the picture worse. Human performance reportedly fell to around 5% after 50 prompts — that is approval fatigue with a decay curve attached. And in production, Anthropic's data shows users accept 97% of permission requests and decline 3%.

Approve, approve, approve — ninety-seven times out of a hundred. That is not a control operating at a low detection rate. That is a reflex with a button attached, and everyone involved knew it before the study was run.

The Number That Actually Proves the Point

Here is the finding I think is most important, and it has been almost entirely overlooked.

When Claude presents a full plan rather than an individual permission request, the rejection rate jumps from 3% to 39%.

Thirteenfold. Same humans, same product, same day. The only variable is whether they are being asked to evaluate a decision or to clear an item.

That is the strongest single piece of evidence I have seen for something I argued last week: human review is not inherently theatre. It becomes theatre under specific conditions — high volume, low stakes per item, no time, confirmatory framing. Move the same person upstream to review a plan, and they engage properly.

Anthropic's own data now demonstrates both halves of that. The rubber stamp is real, and the fix is to change what you ask the human to look at, not to remove them.

Who Is Already Running It

Anthropic named Adobe, Nuro, Gusto and Garner Health as production users of auto mode, and reports that teams using it ship roughly 25% more pull requests. The Decoder's summary is blunt about the direction: the change means AI handles even more of the development process on its own.

Worth holding both halves of that. Named enterprise references are meaningful evidence that this survives contact with real codebases. They are also, necessarily, the customers willing to be named — which is a selected sample, and the fuller account of the rollout is worth reading before you generalise from four logos.

The Guardrails Were Already Gone

Two more figures explain why this was inevitable rather than radical.

By June 2026, roughly a quarter of interactive Claude Code sessions were starting in bypass mode, which removed approval prompts entirely. And as of the same month, about half of all active command-line users had written a Bash allow-rule.

So a large share of users had already disabled or hollowed out the control. Anthropic is not removing a functioning safeguard. It is replacing one that a substantial minority had already routed around, which is a materially different decision and a considerably more defensible one.

Now the Scrutiny

I have been positive so far because the evidence is strong. Here is what should temper it, and I want to give this proper weight because the flattering reading is the easy one.

The Classifier Misses Things

Anthropic's own engineering results show the deployed classifier missed 17% of 52 real cases where Claude acted beyond what the user had authorised.

That is a small sample, and 83% is far better than 13.6%. But it is not the 89% headline figure, and it comes from real deployment rather than a study environment. The gap between benchmark performance and production performance is the oldest story in this field.

The classifier also never sees Claude's reasoning or the tool results — it judges the proposed action against stated instructions, nothing more.

The Study Environment Was Not Their Codebase

The testers worked in an environment built for the study, not on their own repositories, and knew they were being evaluated. Implicator's assessment is direct: what is confirmed is the blocked-command counts; what is implied but not proven is that a classifier reviewing agent actions is safer than a human reviewing them.

That distinction is fair. A person reviewing a command in an unfamiliar sandbox has less context than a person reviewing a command against a codebase they know intimately. The 13.6% may be a floor rather than a fair reading of engaged human review.

Though I would note the counter-argument, which is that the study conditions were more favourable to the human than production is. In production there is no researcher watching and the volume is far higher.

Both Outside Evaluations Were Invited

Third-party red-teaming reportedly cut the classifier's miss rate from 12% to 7%. Trajectory Labs found Claude Code's auto mode blocked all 72 prompt injection attacks attempted, while GPT-5.6 Sol on Codex allowed 5.83% to succeed.

Those are good results, and external evaluation is genuinely better than none. But both were run at the invitation of the company whose product they assessed. The question worth watching, as Implicator puts it, is whether anyone Anthropic did not ask repeats them.

I would apply the same standard here that I applied to a vendor vulnerability statistic last week: directionally credible, not board-deck material until replicated independently.

The Commercial Interest Is Not Hidden

Teams using auto mode reportedly ship around 25% more pull requests. Longer autonomous runs also mean higher total token usage.

Anthropic has a genuine safety case here and a genuine commercial incentive pointing the same direction. Those can both be true. It does mean the study finding conveniently supports the product decision, and you should hold it slightly more loosely than you would a result that cost the company something.

Set against that: waiving the classifier's token cost is a real expense voluntarily absorbed, and it is the single strongest signal that the safety framing is sincere.

What This Means for Your Automations

Most readers here do not use Claude Code. The pattern still lands, because every automation platform will follow this within a year.

1. The Approval Prompt Is Dying Industry-Wide

A frontier lab has now published numbers showing per-action human approval detects roughly one dangerous action in seven, decaying toward one in twenty as the session runs. That evidence does not stay contained to one product.

Expect your workflow tools, your agent platforms and your integration vendors to move the same way. When they do, the question is not whether to accept it — it is whether you have anything else in place.

2. Classifier Review Is Not Authority Scoping

This is the point I would most want you to take away, and the coverage has entirely missed it.

A classifier checking each action is still review. It is dramatically better review — faster, unfatigued, consistent — but it operates at the same point in the pipeline as the human prompt it replaces, and it fails the same way when it fails: something proceeds that should not have.

Authority scoping is different in kind. A credential that cannot write cannot be talked into writing by a prompt injection or waved through by a classifier having a bad day. A rate limit does not have a 17% miss rate.

So the correct response to Friday is not "good, the classifier has it covered." It is to treat the classifier as a much better second line, and to make sure you have a first line that is structural.

Replacing a 13.6% human filter with an 89% machine filter is a large improvement to a control that sits in the wrong place. The thing that actually contains an agent is what it was never permitted to do.

3. Review Plans, Not Actions

The 3%-to-39% finding is the most actionable number in the whole release, and it costs nothing to apply.

Wherever you currently have a human approving individual AI outputs or actions, restructure it so they approve the plan instead — once, upfront, with the whole shape visible. Anthropic's data says the same person rejects thirteen times more often when you do.

This is exactly the specification-before-generation move I argued for in the verification piece, now with a controlled number behind it rather than an inference.

4. Set Your Hard Deny Rules Before Friday

If your team uses Claude Code on Pro, Max or Team plans, administrators have until 14 August to set a different default. After that, every new session starts with the classifier in charge.

Whether or not you keep auto mode — and on this evidence keeping it is defensible — write your hard deny rules now. Anything touching production credentials, anything that deletes, anything that reaches outside the project, anything that spends money. Those are the categories where an 83% catch rate is not good enough.

5. Watch the Enterprise Lag as Free Telemetry

Enterprise and API customers stay opt-in for now. That means consumer and Team users spend a month surfacing the failure modes before the default reaches larger deployments.

If you run anything at scale, that is a free preview. Watch what breaks for the early cohort in the next four weeks before you flip your own default.

The Thing That Happened the Same Week

One item belongs alongside this, because the contrast is instructive.

On 7 August — the same day OpenAI said it could not rule out critical cyber capabilities in Astra — an offensive-security skill router for AI coding agents topped GitHub Trending with more than 20,000 stars. Its design instructs the agent to rewrite its own global rules and to assume every target is authorised.

Nothing about it exploits a vulnerability. The architecture simply permits an agent to edit its own constraints, and that capability shipped as a convenience feature.

So in a single week: one lab replaced a failing human control with a machine one and published the numbers; another paused a model over capability it could not bound; and twenty thousand developers starred a tool that tells an agent to ignore its rulebook.

Guardrails at the vendor layer do very little if the layer above them is designed to remove them. That is not an argument against Friday's change. It is a reminder that 88.4% of organisations reported agent security incidents in a year when every major vendor had guardrails, and that the configuration layer is where the outcome is decided.

The Broader Read

I want to be clear that I think Anthropic has made the right call, on evidence, and has been unusually transparent about numbers that make its own prior product look bad.

A company publishing "our approval prompt caught 13.6% of dangerous commands" is publishing an indictment of a feature it shipped and defended for two years. That takes something, and it is the second time in a week a frontier lab has absorbed a real cost to say something true — OpenAI pausing Astra being the other.

What I would resist is the conclusion that oversight has been solved by moving it to a machine. It has been improved by roughly six-fold at the same point in the pipeline. The 17% production miss rate, the invited evaluations, and the 20,000-star repo all point at the same unglamorous truth: the control that holds is the one built into what the system is permitted to do, not the one checking what it proposes to do.

Gartner still projects $206 billion of AI agent spending this year. Almost none of it will be spent on authority scoping, because authority scoping is not a product anyone sells. It is an afternoon of unglamorous configuration that no vendor can do for you.

Which is, once again, the systems problem. Anthropic just fixed a large piece of theirs and showed everyone the working. Yours is still yours.

Frequently Asked Questions

Do I need to do anything before 14 August?

If your team uses Claude Code on Pro, Max or Team plans, administrators can set a different default before Friday; after that every new session starts in auto mode. Users who previously chose a different default may get a one-time prompt. Regardless of which way you go, set your hard deny rules now.

Is the classifier actually safer than me approving things?

On the published evidence, substantially — 89% against 13.6% in a controlled study, with human performance decaying to around 5% after fifty prompts. The honest caveats are that the study ran in a purpose-built environment rather than testers' own codebases, and that the deployed classifier missed 17% of 52 real cases where Claude exceeded authorisation. Better, not solved.

Does this apply to Enterprise or API usage?

Not yet. Auto mode remains opt-in on Claude Enterprise, the API, Amazon Bedrock, Google Cloud's agent platform and Microsoft Foundry, with the default expected to extend there over the following month. Larger deployments get a preview of the failure modes first.

Will this cost me more in tokens?

The classifier itself is not billed on Pro, Max and Team plans as of 7 August. But longer autonomous runs mean higher total usage, so your overall consumption will likely rise even though the safety layer is free. Budget for the second effect rather than the first.

Should I do the same thing in my own automations?

The plan-review finding transfers immediately and costs nothing — restructure human checkpoints to approve plans rather than individual actions, since Anthropic's own data shows rejection rates jump from 3% to 39% when you do. Building your own classifier is a much larger undertaking and probably not where your next afternoon is best spent. Scope your credentials first.

What is the single most important takeaway?

That a machine reviewing actions is a better version of the wrong control. It is worth having, and it is not a substitute for limiting what your automations are permitted to do in the first place.

Related Reading

The Five Eyes AI Agent Security Guide — governance-first architecture, and why authority scoping outperforms review at any detection rate.

You Don't Need an Agent, You Need a Rule — the four-category classification, and the case for constraints that cannot be talked around.

The Automation Ratio Is the Only Metric That Predicts Survival — the metric that only means something once you know who, or what, could have corrected the output.

AvePoint: 88.4% of Organisations Hit by Agent Security Incidents — recorded in a year when every major vendor already shipped guardrails.

Bending Spoons: $2.57M Revenue Per Employee — what high AI leverage looks like when the checking layer is machine-run by design.

74% of AI Agent Deployments Get Rolled Back — the failure rate that looks different once you assume the approval step was never working.

Stop Chasing the Biggest Model — task-model matching, including matching a Sonnet-class classifier to a supervision job.

About the Author

Hamza Baig is the founder of Hexona Systems, an AI automation agency serving clients across six continents, and the AI Automation Institute, a community of more than 40,000 entrepreneurs building with AI.

He has been featured in the GHL Top 50, Yahoo Finance and Brainz Magazine, and writes regularly on automation architecture, agent governance and the operational realities of AI deployment.

Read more analysis on the Hamza Automates blog, including the no-code automation workflow guide, or get in touch to discuss an automation build.

Follow @hamza_automates on Instagram for daily automation breakdowns.

Note: study figures are Anthropic's own, reported via contemporaneous coverage, and the two third-party evaluations cited were conducted at Anthropic's invitation. Product behavior described reflects announced plans as of 10 August 2026 and may change before or after the 14 August rollout.