What Actually Breaks When You Put an AI Agent in Production

Most AI agent content stops at the moment the agent works. You wire up a model, give it some tools, watch it handle a task correctly, and the demo ends there.

Production is a different problem. The agent stops being a thing you watch and becomes a thing that runs while you sleep, on inputs nobody reviewed, with a budget attached to it.

I have written before about choosing between an AI agent and traditional automation. This post is about what happens after you have already chosen the agent, deployed it, and started paying for it.

TL;DR

• Agents rarely fail loudly. They usually produce a plausible wrong result that no error handler catches.
• Models will invent values for parameters you did not supply, and the more capable models are only more likely to ask instead of guess.
• Uncapped retry loops are the single most expensive failure mode, because every retry looks like a legitimate request.
• Non-determinism makes conventional unit tests useless. You need an evaluation set with thresholds, not assertions.
• Every write action an agent can take needs an idempotency key before it goes anywhere near production.

Failure 1: The agent invents the parameters you forgot to require

This is the failure mode that surprises people most, because it is documented behavior rather than a bug.

When a model calls a tool but the context does not contain enough information to fill a required parameter, it does not necessarily stop and ask. Anthropic’s tool use documentation is explicit about this: given a get_weather tool and the prompt “What’s the weather?” with no location, the model may return a call with location set to a value you never provided, along with a unit parameter it also filled in on its own.

The documentation notes that more capable models are more likely to recognize the gap and ask for the missing value, while smaller and faster models are more likely to infer something reasonable. It also states plainly that this behavior “is not guaranteed.”

That last sentence is the part that matters in production. The failure is probabilistic. It will pass your testing and appear three weeks later on a real customer record.

What this looks like when it goes wrong

Consider a support triage agent that receives a message with no explicit account reference. If it infers an account from context that happens to be nearby in the conversation, calls update_ticket_priority against the wrong account, and returns a confident summary saying the ticket was handled, nothing in the system objects.

Nothing errors. The logs show a successful tool call. The metric that would catch it — “how often did the agent act on an inferred identifier?” — is one almost nobody has built.

How to constrain it

Two mechanisms exist for this specific problem, and both belong in the request rather than the prompt:

• Strict schema conformance. Adding strict: true to a custom tool definition forces tool calls to match your schema exactly, rather than approximately.
• Validation before execution, not after. The tool handler in your application should reject a call whose parameters were not present in the source input. The model proposes; your code decides whether the proposal is grounded.

The general rule: a parameter that identifies who or what an action applies to should never be inferrable. It should be passed deterministically or the workflow should stop.

Failure 2: The retry loop that bills you for silence

Deterministic workflows fail visibly. A webhook returns 500, the run stops, an alert fires.

Agent loops fail expensively instead. When a tool returns an error, the model may reason about the error, adjust its approach, and call the tool again. That is usually the behavior you want. It is also an unbounded loop wearing a helpful disguise.

Consider what a stuck run actually costs. Using Claude API pricing, Claude Sonnet 5 is billed at $2 per million input tokens and $10 per million output tokens. A normal agent turn might use a few thousand tokens. But an agent that retries a failing tool twenty times does not pay twenty times a few thousand tokens — it pays for a conversation history that grows with every attempt, because each retry resends the entire accumulated transcript.

That is quadratic growth on your most expensive dimension, produced by a system that is reporting no errors.

📊 Cost note: Tool definitions themselves are billed as input tokens on every single call. The tools parameter, the tool use system prompt (286 to 675 tokens depending on model), and every tool_result block all count. An agent with twelve tools defined pays for those twelve schemas on every turn of every loop.

The controls that actually work

• Hard iteration cap per run. Not a guideline in the prompt. A counter in your loop that terminates the run.
• Token budget per run. Terminate when a single run exceeds a threshold, and alert on it. This catches loops that stay under the iteration cap but grow context aggressively.
• Prompt caching for the stable prefix. Cache reads cost 0.1x the base input price. If your system prompt and tool definitions are stable across calls, caching turns your largest repeated cost into a tenth of itself.
• Distinguish retryable from terminal errors. A 429 is worth retrying. A 422 validation error means the model’s approach is wrong and retrying it will produce the same result more expensively.

⚠️ Warning: Do not set the iteration cap based on what your agent needs on a good day. Set it based on what you are willing to pay for a bad one.

Failure 3: Retries that duplicate real side effects

The previous failure costs money. This one costs trust.

Picture an agent that calls send_invoice. The call times out at the network layer. The agent, seeing a failure, calls send_invoice again. But the first call had already succeeded — the timeout was on the response, not the request.

The customer now has two invoices, and the agent’s log shows one failure and one success.

This is not an AI problem. It is a distributed systems problem that AI agents encounter far more often, because agents retry autonomously and frequently, whereas a deterministic workflow retries only where an engineer explicitly wrote a retry.

The rule

Every tool that produces a side effect needs an idempotency key generated before the first attempt and reused across retries. The categories that matter:

Tool type	Retry safety	Requirement
Read, search, fetch	Safe	None
Update to a known state	Usually safe	Version check or conditional update
Create, send, charge, delete	Unsafe	Idempotency key, non-negotiable

💡 Pro tip: Generate the idempotency key from the intent (for example, a hash of invoice ID plus recipient plus period), not from a random value per attempt. A random key regenerated on retry provides no protection at all.

Failure 4: You cannot unit test what does not repeat

Conventional automation has a comfortable testing property: the same input produces the same output, so assertEqual is a meaningful statement.

An agent breaks that assumption. Two runs on identical input may produce different phrasing, different tool call ordering, or different reasoning paths that both arrive at acceptable results. An exact-match assertion on agent output is either flaky or so loose it tests nothing.

The replacement is not “test less.” It is a different instrument.

Build an evaluation set, not a test case

1. Collect real inputs. Thirty to fifty actual cases beat a thousand synthetic ones. Include the ambiguous, the malformed, and the adversarial — those are where the failures live.
2. Label the acceptable outcome, not the exact output. For a classification agent this is the correct category. For an action agent it is the correct tool with correctly grounded parameters.
3. Measure four things per run: accuracy against the labels, escalation rate, p95 latency, and cost per task.
4. Set thresholds that gate deployment. A prompt change that improves accuracy by two points while tripling cost is not obviously an improvement, and you cannot see that tradeoff without measuring both.
5. Re-run the set on every change to the prompt, the model version, the tool schemas, or the retrieved context. All four are inputs to behavior.

✅ Quick win: If you have an agent in production with no evaluation set, start by logging every input and output for one week. That log is your evaluation set. You do not need to construct one from scratch.

Failure 5: Observability designed for the wrong system

Standard application logging records that a request happened and whether it succeeded. For an agent, both facts are nearly useless, because the interesting failures are successful requests with wrong content.

To debug an agent failure a week after it happened, you need the full decision trail:

• The original input, exactly as received.
• The context that was retrieved or assembled, and from where.
• The resolved prompt actually sent, including the tool definitions in effect.
• Every tool_use block: which tool, which parameters, in what order.
• Every tool_result, including errors.
• The final action taken and by whose authority — autonomous, or approved by a person.
• Token counts and cost for the run.

Without the resolved prompt and the tool parameters, you are guessing. With them, most agent failures become obvious within minutes.

⚠️ Warning: This log contains customer data and possibly credentials in tool arguments. Redact at write time, set a retention policy, and keep it out of any store you would not point an auditor at. An agent trace log is one of the most sensitive artifacts your system produces.

Failure 6: The silent quality drift

The failures above announce themselves eventually. This one does not.

Nothing in your stack is static. Model versions get deprecated and replaced. Your knowledge base gets new documents. An upstream API changes a field name. A tool description gets edited by someone improving clarity. Each change is individually reasonable, and each one changes agent behavior.

Without a periodic re-run of your evaluation set, quality drifts downward at a rate nobody notices until someone escalates a complaint that turns out to be three months old.

The fix is unglamorous and effective: run the evaluation set on a schedule, not only on deploy. Weekly is enough for most systems. Alert on threshold breach.

The production checklist

Before an agent handles real work unattended:

• Hard iteration cap and per-run token budget, enforced in code.
• Idempotency keys on every write, send, charge, and delete tool.
• Strict schema validation, plus grounding checks on identifying parameters.
• Least-privilege tool access — the agent holds only what this task requires.
• Human approval gate on high-impact or low-confidence actions.
• Full decision-trail logging, with redaction and retention policies.
• Evaluation set with accuracy, escalation, latency, and cost thresholds.
• Scheduled evaluation re-runs, not just deploy-time runs.
• A documented rollback path — you should be able to disable the agent and fall back to the manual process in minutes.

➡️ Next step: If your agent fails more than three of these, the correct move is usually not to fix all nine. It is to narrow the agent’s scope until the remaining surface is one you can actually control, then expand.

What this means for how you design agents

Every failure above points the same direction: the model is the least reliable component in the system, so it should hold the least authority.

That is the principle underneath the hybrid architecture I described in how AI agents fit into no-code automation — let the model interpret, and let deterministic code decide and execute. The failures in this post are what it costs when that boundary is missing.

It is also the design constraint I keep returning to while building Noderan. The interesting engineering in an automation platform is not the part that calls a model. It is the caps, the idempotency, the approval gates, and the traces — the machinery that makes a probabilistic component safe to run unattended.

An agent that works in a demo is a prototype. An agent with a budget cap, an idempotency key, an evaluation set, and a rollback path is a system.

Frequently Asked Questions

Why do AI agents fail in production but work in demos?

A demo runs a handful of clean inputs once, usually with someone watching. Production runs messy inputs continuously, unattended. Failure modes with a low per-run probability — an invented parameter, a retry loop, a partially completed multi-step action — become frequent enough to matter once the run count is high enough.

What is the most expensive AI agent failure mode?

Uncapped retry and reasoning loops. Each retry resends the accumulated conversation history, so cost grows faster than the retry count, and because every retry is a well-formed API call, nothing in your error monitoring fires.

How do you test a non-deterministic AI agent?

Not with exact-match assertions. Build a fixed evaluation set of real inputs labeled with acceptable outcomes, then measure accuracy, escalation rate, latency, and cost across runs. Gate deployment on thresholds rather than on a single passing run.

Should an AI agent be allowed to retry a failed action?

Only for read-only or idempotent operations. Any tool that writes, sends, charges, or deletes needs an idempotency key derived from the intent, so a retry after an ambiguous timeout cannot produce a duplicate side effect.

How much does it cost to run an AI agent?

It depends on model, context size, and loop length rather than on request count alone. Tool definitions and the tool use system prompt are billed as input tokens on every call, and conversation history is resent each turn. Prompt caching on a stable prefix reduces repeated input cost to 0.1x the base rate.

Key Takeaways

• Agent failures are usually silent and plausible, not loud and obviously broken.
• Models can invent parameters you did not supply; ground identifying values in code, not in the prompt.
• Cap iterations and token budget per run — a stuck loop reports no errors while it bills you.
• Idempotency keys are mandatory on any tool with a side effect.
• Replace unit tests with an evaluation set and thresholds; re-run it on a schedule.
• Log the full decision trail, redacted, or expect to debug by guessing.

Related Reading

• AI Agent or Traditional Automation? How to Choose the Right Approach — the decision framework that comes before this post.
• How AI Agents Fit Into No-Code Automation — the hybrid architecture these controls protect.
• What Is Noderan? — the platform I am building around these constraints.

Sources

• Claude API pricing — token rates, prompt caching multipliers, and tool use token overhead. Retrieved 2 September 2026.
• Tool use with Claude — documented parameter inference behavior and strict tool use. Retrieved 2 September 2026.

I write about AI agents, workflow automation, product development, and technical SEO at omerbatutandogan.com.