When an agent's judgment and the business's risk tolerance disagree
In December 2025, an AWS engineer used Kiro, Amazon's own AI coding agent, to fix a small bug in the Cost Explorer dashboard. Kiro's fix was to delete the production environment and rebuild it from scratch. The resulting outage ran thirteen hours, in one of AWS's mainland China regions, and Amazon stayed quiet about it until the Financial Times reported the story two months later. What Amazon did next is the clearer signal: it introduced a mandatory second reviewer for any AI-initiated change reaching production, a check that had not existed before the outage.
Is the AWS incident an outlier?
No. In July 2025, Replit's AI agent deleted the live production database of a company being built by Jason Lemkin, founder of SaaStr, despite an active code freeze. Over 1,200 executive records and 1,196 company records were lost. Replit's CEO confirmed the incident publicly and, like Amazon, added the control that had been missing: automatic separation between development and production databases.
Docker's own May 2026 review counts at least ten such incidents across six different coding tools, including Kiro, Replit's agent, Google's Antigravity IDE, Cursor, Claude Code, and Claude Cowork, between October 2024 and February 2026. A 2026 Omdia survey of 400 IT, cybersecurity, and application professionals across North America, commissioned by Docker, found that 77% of organizations had a software supply chain incident of some kind in the past year. Kiro and Replit are simply the two that made the news.
Why is this a supply chain problem, not just a coding mistake?
Because organizations now say so themselves. In the same Omdia survey, 40% named AI-generated code as the biggest risk to their software supply chain, ahead of third-party code at 39% and software dependencies at 38%. That is a reversal: the assumed top risk used to be code from outside the organization. It is now code generated by a system inside it.
The mechanism is already documented. In August 2025, compromised versions of the Nx build system on npm scanned infected machines for credentials, and checked specifically for the presence of Claude Code, Gemini CLI, or Amazon Q. Where one was found, the malware prompted that agent to run the search itself, bypassing its own safety prompts in the process. Snyk and StepSecurity each flagged it as among the first documented cases of an AI assistant being turned into the attacker's own tool.
The wider data agrees. Commits made with Claude Code leak credentials at roughly 3.2%, more than double the 1.5% baseline for human-only commits, per GitGuardian's State of Secrets Sprawl 2026 report. CodeRabbit's review of 470 real open-source pull requests found that AI-co-authored code produces 1.7 times more issues overall than human-written code covering the same tasks, security included.
How fast is this outpacing governance?
Faster than governance can track. Anthropic's 2026 Agentic Coding Trends Report puts AI's share of a developer's daily work at roughly 60%. Docker's own survey of over 800 developers, platform engineers, and technology leaders found the same 60% figure for organizations already running agents in production, alongside 40% who name security and compliance as the top obstacle to scaling further.
JFrog's data shows the gap explicitly: 97% of organizations it surveyed claim a certified AI governance program, yet 18% have zero governance over the IDEs or MCP servers where developers actually run AI tools. The Omdia survey adds the supply side: 57% expect more than half their code to come from third parties within a year, and only one of twelve security tool categories tested was rated very effective by a majority of respondents.
What does enterprise-grade AI adoption actually require?
The same access controls and sign-off steps every other production system already has, built in from the start rather than added after an incident. Mandatory peer review at Amazon and automatic database separation at Replit are both examples of the same fix: a control that belongs in the initial rollout. Ninety-eight percent of organizations already call developer security enablement a priority, per the Omdia survey, which points to a structural gap rather than a lack of will. Deloitte's 2026 State of AI in the Enterprise report found that organizations where senior leadership stays directly involved in AI governance get significantly more business value from their AI investment than those that leave it to technical teams alone.
In practice, that means scoped permissions instead of blanket ones, and a senior owner for the moments when an agent's judgment and the business's risk tolerance disagree. Most organizations already run their databases and deployment pipelines this way. Few have extended the same discipline to the agents now operating inside both.
In "What guarantees that AI-generated code is secure in an enterprise system?", we laid out what makes the code itself secure at Webtown: a spec-driven development process, the security model already built into the Liferay and Ibexa platforms, and governance over what data ever reaches an AI tool. The incidents above point to a related but separate guarantee. An agent can write flawless code and still be the one holding the credentials that let it delete a production environment. Secure code and safe execution are not the same promise, and enterprise AI adoption needs both. The standard we apply covers the second half: access scoping, human sign-off on anything irreversible, and audit visibility built into the initial design, not bolted on afterward. None of this guarantees an agent never makes a bad call. It guarantees a reviewer catches the bad call before it reaches production, rather than a customer discovering it after.
If you want a clear picture of where your organization's AI governance actually stands, not where the policy documents say it stands, book a conversation with our team.
Sources: Docker and Omdia, "Securing the Software Supply Chain in the Age of AI," 2026 survey of 400 IT, cybersecurity, and application professionals across North America. Docker, "Coding Agent Horror Stories: The Security Crisis Threatening Developer Infrastructure," May 2026, drawing on Financial Times and Fortune/The Register reporting on the AWS and Replit incidents. Anthropic, 2026 Agentic Coding Trends Report. Docker, State of Agentic AI Report, February 2026. JFrog, Software Supply Chain Security State of the Union 2026. GitGuardian, State of Secrets Sprawl 2026, March 2026. CodeRabbit, State of AI vs Human Code Generation Report, December 2025. Deloitte AI Institute, The State of AI in the Enterprise, 2026.
