Companies Rush AI Agents Into Production—Only 21% Have Safeguards

Your company’s AI agent just approved a $500,000 vendor contract while you were in a meeting. Another one modified customer pricing without human review. A third accessed confidential employee records to “improve HR recommendations.” These aren’t hypothetical scenarios—they’re happening right now at companies racing to deploy autonomous AI systems faster than they can figure out how to control them.

A new Deloitte report reveals that 74% of companies plan to use AI agents within two years, up from just 23% today. But here’s the alarming part: only 21% have implemented proper governance frameworks to oversee these systems. We’re essentially handing over business-critical decisions to software that can act independently, make judgment calls, and interact with real systems—all while most companies treat AI oversight as an afterthought they’ll “figure out later.”

The gap between deployment speed and safety controls is widening dangerously. AI agents aren’t like traditional software that waits for human commands. They observe, decide, and act autonomously—booking meetings, processing refunds, modifying databases, sending communications. When something goes wrong, there’s often no audit trail explaining why the agent made that specific decision. And as these systems proliferate across finance, HR, customer service, and operations, the potential for cascading errors grows exponentially.

The Governance Gap: Why This Matters Now

The fundamental problem isn’t that AI agents are inherently dangerous—it’s that companies are deploying them like traditional software when they behave fundamentally differently. A standard business application does exactly what you tell it, every time, predictably. AI agents interpret instructions, make contextual decisions, and adapt their behavior based on circumstances. That flexibility is precisely what makes them useful, but it’s also what makes them risky without proper controls.

Deloitte’s survey found that adoption is accelerating dramatically. The share of companies not using AI agents will collapse from 25% today to just 5% within two years. This isn’t gradual adoption—it’s a flood. And most organizations are unprepared for the operational, legal, and security implications of systems that can act independently on their behalf.

The consequences of poor governance range from embarrassing to catastrophic. An AI agent with excessive permissions might leak confidential data by including it in a routine email summary. Another could approve fraudulent transactions because it misinterpreted context. A customer service agent could make unauthorized commitments that create legal liability. Without detailed logging, companies often discover these mistakes only after damage is done, with no way to reconstruct what happened or prevent recurrence.

Ali Sarrafi, CEO of Kovant (an AI governance platform), describes the core issue as “governed autonomy versus reckless automation.”

According to Sarrafi: “Well-designed agents with clear boundaries, policies, and definitions—managed the same way an enterprise manages any worker—can move fast on low-risk work inside clear guardrails, but escalate to humans when actions cross defined risk thresholds.”

What “Governed Autonomy” Actually Means

The concept sounds simple but requires rethinking how companies deploy AI. Instead of giving an agent broad access and hoping for the best, governed autonomy means implementing tiered permission systems similar to how businesses manage human employees.

1. Tier 1: Read-Only Access – The agent can view information and make recommendations but cannot take any actions. A financial planning agent might analyze spending patterns and suggest budget adjustments, but a human must approve and execute changes.
2. Tier 2: Limited Actions with Human Approval – The agent can draft emails, prepare reports, or queue transactions, but everything requires human sign-off before execution. This works well for tasks where AI speeds up preparation but judgment matters—like customer refund decisions or vendor contract reviews.
3. Tier 3: Autonomous Action Within Boundaries – The agent can act independently but only within strictly defined limits. For example, a customer service agent might be authorized to issue refunds up to $100 automatically, but anything larger gets escalated to human review. Or an HR agent can answer policy questions but cannot access salary data or make hiring decisions.

The critical component is detailed action logging. Every decision the agent makes—what data it accessed, what rules it applied, what action it took, and why—gets recorded in auditable logs. Sarrafi emphasizes this transparency:

“With detailed action logs, observability, and human gatekeeping for high-impact decisions, agents stop being mysterious bots and become systems you can inspect, audit, and trust.”

This logging serves multiple purposes. When something goes wrong, teams can replay exactly what happened and identify where the agent’s logic failed. For compliance, logs prove the company has controls in place and can demonstrate accountability to regulators. And crucially, transparent logging makes AI systems insurable—a growing concern as insurance companies refuse to cover opaque autonomous systems.

Why AI Agents Fail in Real Business Environments

The gap between demo performance and production reliability stems from complexity and context. In controlled tests, AI agents work with clean data, clear objectives, and limited scope. Real business environments are messy: fragmented systems, inconsistent data formats, ambiguous instructions, and edge cases that weren’t anticipated.

Sarrafi points to a specific failure mode: “When an agent is given too much context or scope at once, it becomes prone to hallucinations and unpredictable behavior.” For example, an agent tasked with “analyze our customer satisfaction trends and take appropriate action” might have access to customer feedback, support tickets, sales data, product reviews, and employee notes. With such broad context, it might misinterpret correlation as causation and trigger inappropriate responses—like automatically discounting products that happen to have recent negative reviews without understanding that the reviews reference a now-fixed bug.

Production-grade systems address this by decomposing operations into narrower, focused tasks. Instead of one mega-agent with vast permissions, you deploy multiple specialized agents with limited scope: one analyzes customer feedback patterns, another flags anomalies, a third drafts responses, and a fourth (human) approves significant actions. This architecture makes behavior more predictable and failures more contained.

The Insurance Problem: Why Carriers Won’t Cover Opaque AI

As AI agents take real actions affecting business operations, insurance companies are grappling with how to assess and price risk. The problem: traditional insurance models assume you can evaluate what happened after an incident. But with opaque AI systems, insurers often can’t determine whether losses resulted from agent malfunction, inadequate controls, or reasonable decisions given the available information.

Detailed action logs change this equation. When every agent decision is recorded—what data was accessed, which rules applied, what alternatives were considered, what action was taken—insurers can perform forensic analysis similar to investigating human employee actions. This makes AI systems insurable because risk becomes quantifiable.

Sarrafi notes: “This visibility, combined with human supervision where it matters, turns AI agents from inscrutable components into systems that can be inspected, replayed, and audited. It allows rapid investigation and correction when issues arise, which boosts trust among operators, risk teams, and insurers alike.”

For businesses, this isn’t just about insurance coverage—it’s about legal liability. If an AI agent causes harm (financial loss, privacy violations, discriminatory decisions), companies need to demonstrate they had reasonable controls in place. Without audit trails and governance frameworks, defending against lawsuits or regulatory action becomes nearly impossible.

Deloitte’s Governance Blueprint

Deloitte’s recommended approach centers on three pillars: technical controls, organizational oversight, and workforce readiness.

Technical controls include the tiered autonomy model described earlier, plus identity and access management. Just as employees don’t get database admin credentials on day one, AI agents shouldn’t have unrestricted access to systems. Permissions should map to specific roles and escalate based on proven reliability in lower-risk scenarios.

Organizational oversight means establishing clear governance structures—who decides what agents can do, who monitors their performance, who investigates incidents, and who owns the risk. Deloitte’s “Cyber AI Blueprints” recommend embedding AI governance into existing risk management frameworks rather than treating it as a separate IT concern. This ensures executive visibility and accountability.

Workforce readiness addresses the human element. Employees need training on what information they shouldn’t share with AI systems (like passwords, confidential strategies, or sensitive personal data), how to recognize when an agent is behaving unexpectedly, and what to do when things go wrong. Without this literacy, even the best technical controls can be undermined by well-meaning employees who don’t understand the risks.

What Businesses Should Do Now

For companies already deploying or planning to deploy AI agents, immediate actions include:

Audit current agent deployments. Identify every AI system with autonomous capabilities, document what permissions it has, and assess whether appropriate controls exist. Many companies discover “shadow AI” agents deployed by individual teams without central oversight.

Implement logging before expanding deployment. Prioritize making existing agents auditable over deploying new ones. The data captured in logs becomes invaluable for both improving agent performance and demonstrating accountability.

Establish tiered rollout processes. New agents should start in read-only mode, prove reliability over weeks or months, then gradually gain additional permissions based on measured performance. Rushing to full autonomy creates unnecessary risk.

Create cross-functional governance teams. AI agent oversight shouldn’t be purely technical—it requires input from legal, compliance, risk management, and business operations. These teams define acceptable use policies and approval workflows for high-impact actions.

The race to deploy AI agents won’t slow down, but companies that build governance alongside deployment will have sustainable competitive advantages. Those that rush into production without controls will face costly incidents, regulatory scrutiny, and ultimately, the need to retrofit safeguards after damage is done.

FAQ

Q: How is an AI agent different from regular automation or an AI chatbot?

A: Traditional automation follows fixed rules (“if X happens, do Y”). Chatbots respond to prompts but don’t take actions in business systems. AI agents combine both: they observe situations, make contextual decisions based on training rather than hard-coded rules, and take real actions—like approving purchases, modifying records, or sending emails—without waiting for human commands. This autonomy makes them powerful but also risky if not properly controlled. Think of automation as a vending machine, chatbots as information desks, and agents as junior employees who can make decisions and act independently within their authority.

Q: Can’t we just turn off AI agents if they make mistakes?

A: You can, but by the time you notice a problem, damage may already be done. If an agent leaked confidential data, approved fraudulent transactions, or made unauthorized commitments to customers, shutting it down doesn’t undo those actions. This is why detailed logging and tiered permissions matter—they help catch mistakes early and limit the scope of what can go wrong. Also, in fast-moving business environments, the cost of disabling an agent (lost productivity, delayed decisions) creates pressure to keep problematic systems running rather than fixing underlying governance issues.

Q: Isn’t requiring human approval for decisions just slowing down the benefits of AI agents?

A: Not if designed thoughtfully. Tiered autonomy means agents handle routine, low-risk tasks automatically while escalating only unusual or high-impact decisions. For example, a customer service agent might resolve 95% of inquiries independently (password resets, order status, simple refunds) but flag the 5% that involve unusual patterns or large amounts. This gives you most of the speed benefit while maintaining human judgment where it matters most. The alternative—full autonomy without oversight—creates risk that eventually forces you to pull back or face serious consequences.

For updates on AI governance and deployment best practices, join our WhatsApp community where 2,000+ engineers and business leaders discuss real-world AI implementation.