Nine Seconds to Lose a Database

It may not feel like it, but we are still in the early innings of AI coding.

In a 2026 New Relic survey of 200 US technology leaders, 67% said AI now drives over half of their teams' weekly code. The same share had suffered a production failure tied to AI-generated code in the past six months. 

AI already writes most of the code, while the practices for governing it are still being invented. That gap is widening. The research group METR found that the length of software tasks leading AI models can complete on their own has doubled about every four months since 2023. 

Safety is not keeping pace. In Veracode's 2026 tests, AI-written code compiled almost every time, yet only 56% passed security checks, a rate that has barely moved in a year. 

Governance moves at the speed of organizations, so the gap between what agents can do and what companies can oversee grows every few months.

An instruction in a prompt is a request. It is not a requirement, and it is not a control.

In April, that gap cost PocketOS its production database and backups. PocketOS is a small software company that helps car rental businesses manage reservations, payments and vehicle fleets. The Guardian reported that a PocketOS coding agent had instructions forbidding destructive actions without human approval. But the agent assumed it was working in a test environment and deleted everything in nine seconds. 

The lesson: An instruction in a prompt is a request. It is not a requirement, and it is not a control.

Governed autonomy

In software development, granting autonomy to agents works when the work is contained and easy to undo. For example: a prototype or a refactor on an unmerged branch, where a mistake simply gets thrown away. 

Mission critical, production-grade code is a different matter. When code moves money, controls who sees customer data or keeps a paid service running, mistakes reach customers. Some mistakes cannot be undone. In these cases, fully autonomous delivery is never acceptable.

Before an agent can act responsibly, it needs three kinds of knowledge: the system, the rules and the intent.

Most organizations are not yet equipped to govern such mission-critical work. In Black Duck's 2026 survey of 831 engineers, 97% of teams used AI coding tools. Only 30% fully governed that use, and 52% called manual code review a bottleneck. 

Agents should handle what can be checked against explicit rules. Accountable humans should own business intent, consequential tradeoffs and decisions that are hard to reverse. That division of labor only works if the agent understands the full context in which it operates. 

Need to know

Before an agent can act responsibly, it needs three kinds of knowledge: the system, the rules and the intent. Together, these three forms of context explain what exists, what is permitted and what needs to be accomplished.

The system. Mature software contains years of accumulated decisions, many undocumented. Which service owns a permission check? What functionality already exists? An agent working from a prompt alone must fill those gaps with assumptions that can produce broken dependencies or security failures.

The rules. Architecture standards, security policies and compliance obligations often live in scattered documents or people's heads. Agents need rules expressed in forms that software can evaluate and enforce. A policy embedded in a prompt is not the same as a control.

The intent. Why is a change needed? Which business requirement does it serve? Who requested it? Without that knowledge, an agent can produce technically correct work that is wrong for the business.

Accountability in practice

At AppFaktors, an  enterprise AI company that governs AI-driven software delivery, we've been working to put these principles into practice. Earlier this year, a wireless connectivity provider asked us for a change that sounded small: a new kind of user who could see one customer's data and nothing else. 

The change touched 38 files, including seven interface pages. A single missed permission check could expose one customer's devices, locations and reports to another, a serious breach of trust for a company whose business depends on reliability and security.

With agents, accountability means being able to explain any change during normal work, not reconstructing it after a failure.

We applied Application Context Engineering (ACE), our approach to giving coding agents the application's architecture, specifications and machine-checkable policies as they work. The provider's team answered the agent's questions and approved its design before any code was written. Automated checks tested each step against those policies.

That change and a larger one, copying a live customer's data into an anonymized demo account, reached the development environment in about seven hours of hands-on build time. 

We asked OpenAI's Codex and Anthropic's Claude to estimate the same work for one experienced developer using no AI tools. Their most optimistic estimates ranged from 4.5 to 10 working days for the user role and 12 to 18 working days for the demo account. Combined, that’s roughly three to five and a half weeks of development time.

To be clear, neither feature went straight to production. The provider owned the release decision along with every change traced to its requirement, the policies it was checked against and the people who approved it. 

The lesson: With agents, accountability means being able to explain any change during normal work, not reconstructing it after a failure.

From pilot to production

Moving AI from pilot to production remains a major challenge for enterprises. In KPMG’s Q1 2026 AI Pulse survey of U.S. companies with at least $1 billion in revenue, 65% reported difficulty scaling AI use cases, up from 33% the previous quarter. Meanwhile, 63% now require human validation of agent output, up from 22% a year earlier.

That helps explain the growing demand for forward-deployed engineers (FDEs), who work directly with clients to turn AI experiments into production systems. According to data reported by Business Insider, FDE job postings surged 729% in the year ending April 2026.

FDEs bring something AI agents still lack: engineering judgment informed by business context. Too often, however, that knowledge remains in the engineer’s head or notes. When the engagement ends, what remains is documentation that quickly becomes outdated.

The question is not simply how much work agents can perform, but how much authority they can safely exercise.

We believe there is a better model: Engineers supply the judgment, while the application retains the context. With ACE, engineers approve the design, and the system preserves the decisions, rules and approvals that shaped it. That knowledge stays current as the application evolves, allowing the next team to build on what came before rather than start from scratch.

Factory scale

The need to preserve context becomes even more urgent as enterprises scale from individual agents to AI-powered software factories. In a 2026 OutSystems survey, 94% of IT leaders expressed concern about agent sprawl. When teams rely on their own prompts and informal rules, mistakes can spread across everything their agents produce. At factory scale, context, policies and accountability must become shared infrastructure.

The question is not simply how much work agents can perform, but how much authority they can safely exercise. Can they access the knowledge they need? Are policies enforced as work happens? Can every change be traced to a business requirement and a person who approved it? The path forward is to start with bounded tasks, enforce clear controls and expand autonomy only when the evidence supports it.

Meanwhile, the race toward artificial general intelligence continues. Models will become more capable whether enterprise governance keeps pace or not. The widely reported OpenAI/Hugging Face incident in July illustrates the risks: two models being tested in an isolated environment found a flaw in a software installer, reached the open internet and accessed Hugging Face's production database in search of answers to their test.

The lesson is not that AI development should stop. It is that capability alone cannot justify autonomy. The wireless provider example offers a more practical measure of progress: work estimated to take weeks reached development in about a working day, with a clear record of what was built, why it was built and who authorized it.

The goal is not to remove people from the software development process. It is to make human judgment count at scale.

Lisa Wolfe is Vice President of AI Strategy at AppFaktors, where she focuses on governance and enterprise adoption of AI-driven software development. A computer scientist, author and veteran technology executive, she previously held leadership roles at Hewlett-Packard and ServiceNow. Her work spans AI, enterprise software, product strategy and technology marketing.

Sources

1.    New Relic (June 2026). 2026 AI Code Report. Survey of 200 U.S. technology leaders.

2.    METR (January 2026). Time Horizon 1.1. Research on the growing length of software tasks AI models can complete autonomously.

3.    Veracode (July 2026). 2026 GenAI Code Security Report. Analysis of security vulnerabilities in AI-generated code.

4.    The Guardian (April 29, 2026). Report on the PocketOS database deletion. Account of an AI coding agent deleting a company's production database and backups.

5.    Black Duck (June 2026). The State of AI-Powered Software Development. Survey of 831 software engineers and DevOps professionals.

6.    KPMG (Q1 2026). AI Quarterly Pulse Survey. Survey of 237 U.S. business leaders at companies with annual revenue of at least $1 billion.

7.    Business Insider (2026). Interview with KPMG forward-deployed engineer Justin Johnsen. On the importance of business context in enterprise AI deployment.

8.    Business Insider (May 2026). Job postings for this tech role have grown more than 700% in the last year. Indeed data showing a 729% increase in forward-deployed engineer job postings from April 2025 to April 2026.

9.    OutSystems (April 2026). What 1,900 IT leaders know about scaling agentic AI. Global survey covering AI adoption, governance and agent sprawl.

10. AppFaktors (October 2026). Wireless connectivity provider case study. Customer implementation results and development-time estimates obtained from OpenAI Codex and Anthropic Claude.

11. TechCrunch (July 21, 2026). OpenAI says Hugging Face was breached by its pre-release models. Report on AI models escaping a test environment and accessing Hugging Face's production database.