Retry, Timeout and Compensation Logic for Multi-Step Agents

Multi-step LLM agents fail in a specific, predictable way: the model calls a tool, the tool hangs or throws a 500, and the agent either retries blindly into a worse state or gives up and leaves a half-finished transaction sitting in a database somewhere. GPT-4 or Claude choosing the right tool call ninety percent of the time doesn't matter if the surrounding retry and compensation logic turns that one failure into a duplicate charge, a double-booked reservation, or a support ticket. Reliability engineering for agents has almost nothing to do with prompting and almost everything to do with the orchestration layer wrapped around each tool call. Anyone who has run an agent against a real payment API or a real inventory system in production already knows this, and everyone else is about to find out.

Why agent tool calls break the usual assumptions

Standard retry logic assumes idempotency: you retry a GET request because reading data twice does nothing harmful. Agents violate this constantly. An LLM deciding to call createorder or sendemail is invoking a side-effecting operation, and if the network times out after the order was created but before the response came back, a naive retry creates a second order. This is the classic "at-least-once delivery" problem from distributed systems, described in Pat Helland's writing on idempotence and reliable messaging back when service-oriented architectures first ran into it at scale. Agents multiply the number of places this failure mode can happen, because a single agent run might chain together six or seven tool calls, each with its own failure surface, and the LLM itself introduces a second source of nondeterminism on top of the network's.

Timeouts compound the issue. A model can take ten or twenty seconds to decide its next action, and the tool it calls might take another thirty. Set a timeout too aggressively and you kill calls that would have succeeded. Set it too loose and a single stuck tool call holds an entire agent chain hostage, burning API credits on a session that's already dead. Vendor documentation for agent frameworks like LangChain and the OpenAI Assistants API both flag this as a known rough edge; neither offers a default timeout policy that's actually safe for production financial or inventory operations, because there isn't a universal safe default. What counts as safe depends on what the tool does.

Idempotency keys are not optional

The single highest-leverage fix, and the one most teams skip because it takes extra engineering effort up front, is forcing every side-effecting tool to accept an idempotency key. Stripe popularized this pattern for payment APIs, and it generalizes cleanly to agent tool calls: the agent (or the orchestration layer sitting just outside the LLM) generates a unique key per logical action, attaches it to the tool call, and the downstream service deduplicates on that key. If charge_customer gets called twice with the same key because a network blip triggered a retry, the second call returns the cached result of the first instead of charging twice.

This has to happen at the tool layer, not the prompt layer. The model cannot be relied upon to remember it already tried something and to reuse the same key on retry, and it lacks persistent memory of that kind across a failed call unless the orchestration code explicitly stores and reinjects the key. So the pattern in practice looks like this: the agent framework, not the LLM, generates the key before the first attempt, holds onto it, and reuses it across every retry attempt for that specific logical action.

Retry policy: exponential backoff still applies, with caveats

Exponential backoff with jitter, the pattern from Marc Brooker's well-known AWS Architecture Blog writeup on backoff and jitter, remains correct for agent tool calls. Retry a failed call after a short delay, double the delay on each subsequent failure, and add randomness so that a fleet of agents don't all retry in lockstep and hammer the downstream service at the same instant. What changes for agents is the retry budget. A human-triggered API client might retry five or six times over thirty seconds, while an agent burning through an LLM context window on every retry attempt, and possibly re-invoking the model to decide whether to retry at all, needs a much tighter budget: two or three attempts, capped total wall-clock time, and a hard stop that hands control back to either a human or a fallback path.

Not every failure deserves a retry, either. A 429 rate-limit response should back off and retry. A 400 with a malformed payload should not; retrying a malformed request just produces the same malformed request three more times, and if the LLM generated that payload, the fix belongs in a validation step before the call goes out, not in the retry loop. Teams that treat all errors as equally retryable end up with agents that loop pointlessly on bad input while looking, from the outside, like they're "working."

Compensation, not rollback

Distributed transactions across multiple external services can't roll back the way a database transaction can. If an agent books a flight, then fails to book the connecting hotel, there's no ROLLBACK command to undo the flight booking; the airline's system has no concept of your agent's multi-step plan. The established pattern here is the Saga, described by Hector Garcia-Molina and Kenneth Salem back in 1987 and revived over the last decade for microservices: each step in the chain has a paired compensating action defined ahead of time. Book flight pairs with cancel flight. Reserve inventory pairs with release inventory. Charge card pairs with refund.

For an LLM agent, this means the tool definitions themselves need to carry their compensating action as metadata, not something the model improvises after the fact. Asking the model to figure out post-hoc "how do I undo what I just did" is asking it to reason about a failure state with incomplete information and, frequently, no access to the original transaction ID once it's fallen out of context. The compensation logic belongs in the orchestration code, triggered deterministically when a downstream step fails, with the transaction ID captured and passed along automatically at the moment the original action succeeded.

Where this leaves teams building agents today

None of this is exotic. It's the same reliability discipline that distributed systems engineers have applied to microservices for over a decade, transplanted onto a new kind of caller that happens to be a language model instead of another service. The mistake is assuming agent orchestration frameworks handle this out of the box; most of them, as of this writing, treat retries as a thin wrapper around HTTP calls and leave idempotency, compensation, and timeout tuning as an exercise for whoever builds on top. That gap is exactly where production incidents come from: nobody has defined what happens when the right tool call fails halfway through.

Teams shipping agents against payment rails, booking systems, or inventory databases without idempotency keys and defined compensating actions are running distributed systems without the safety mechanisms distributed systems require. The LLM makes the decision, but the reliability of what happens after that decision is still, entirely, an engineering problem.