Retries, Timeouts, and Failure Handling

Tools fail. A database times out, an API returns a 503, a search endpoint rate-limits you. In a normal service you catch the error, retry or bail, and move on. Inside an agent loop you have a second option that a plain service does not: you can hand the error back to the model and let it adapt.

That extra option changes how you think about failure. This article covers when to retry in code, when to time out, how to tell a transient failure from a permanent one, how to keep a retry from double-charging a customer, and where the line sits between your code recovering and the model recovering.

A failing tool is different from a failing service

When a REST endpoint fails, the caller is another piece of code. It follows a fixed policy: retry three times, then return a 500. The caller cannot think its way around the problem.

An agent’s caller is a language model. If you feed the error back as an observation, the model can read it and change course. A ProductNotFound error might make it search by a different name. A permission denied might make it try a read-only tool instead. The model is a general reasoner sitting in the loop, and a clear error message is information it can act on.

So the first rule is not about retries at all: when a tool fails, return the error to the model as a normal tool result, not as a crash. A raised exception that escapes the loop kills the run. An error string in the tool result keeps the loop alive and gives the model a chance to recover.

def run_tool(call, tools):
    try:
        return str(tools[call.name](**call.arguments))
    except Exception as e:
        # The model reads this like any other observation.
        return f"ERROR: {type(e).__name__}: {e}"

The error text is a prompt now. ERROR: ConnectionTimeout: upstream took >5s tells the model something it can use; ERROR: 500 tells it almost nothing. Write tool errors the way you would write a helpful log line.

Retries with backoff, and timeouts

Not every failure is worth bothering the model with. A one-off network blip does not need the model to reason about anything; it needs a retry. Handling those in code is cheaper and faster than spending a whole model turn on them.

Two mechanisms cover most of it:

Retry with backoff for transient failures. Try again after a short wait, and grow the wait each time (for example 1s, 2s, 4s). Add a little random jitter so a fleet of agents does not retry in lockstep. Cap the attempts; three is a sane default.
Timeouts so a hung tool cannot stall the whole loop. A tool that never returns is worse than one that fails, because the agent sits frozen and the step limit never trips. Every tool that touches the network needs a timeout.

Timeouts matter more in an agent than in a normal request handler. The loop is sequential: each tool call blocks the next model turn. One tool hanging for 60 seconds freezes the entire agent, and the user is watching a spinner the whole time.

Why add jitter to backoff?

If an upstream service hiccups, every agent that hit it fails at the same moment. Plain exponential backoff makes them all wait the same 1s, then the same 2s, so they retry in a synchronized wave and hammer the recovering service at exactly the same instants.

Jitter means adding randomness to each wait, so instead of everyone waiting exactly 2s, each waits somewhere between 1 and 3. The retries spread out over time, and the recovering service sees a smooth trickle instead of repeated spikes. This is the same “thundering herd” problem you already guard against in ordinary backend code; agents just make it easier to fan out.

Transient versus permanent failures

Retrying only helps if the thing that failed might succeed next time. That is the whole distinction, and it decides your response.

A transient failure is temporary: a timeout, a 503 Service Unavailable, a 429 Too Many Requests, a dropped connection. Nothing about the request was wrong; the world was briefly busy. Retry these, with backoff.

A permanent failure will fail identically no matter how many times you retry: a 404 Not Found, a 401 Unauthorized, a malformed argument, a validation error. The request itself is the problem. Retrying wastes time and money and delays the moment the model could have done something useful.

flowchart TB
    ERR[Tool raised an error] --> Q{Would retrying<br/>change the result?}
    Q -->|"yes: timeout, 503, 429"| RETRY[Retry with backoff in code]
    Q -->|"no: 404, 401, bad input"| MODEL[Return error to the model]
    RETRY --> STILL{Still failing<br/>after N tries?}
    STILL -->|yes| MODEL
    STILL -->|no| OK[Return result]

The trap is retrying a permanent failure. If the model called a tool with a bad argument, retrying the same call three times just burns three attempts to arrive at the same error. Send that one straight back to the model so it can fix the argument. Retry the transient ones in code; route the permanent ones to the model.

Idempotency: don’t double-charge on a retry

Retries introduce a sharp edge. Suppose the tool did its work but the response got lost on the way back. Your code sees a timeout, assumes failure, and retries. Now the action ran twice.

For a read, that is harmless. For anything with a side effect, charging a card, sending an email, inserting a row, it is a real bug. A retry that double-charges a customer is much worse than the original timeout.

The fix is idempotency: design the tool so that running it twice with the same input has the same effect as running it once. The standard technique is an idempotency key, a unique token the caller sends with the request. The server records the key and, if it sees the same key again, returns the original result instead of doing the work a second time.

def charge_card(amount, idempotency_key):
    if payments.already_processed(idempotency_key):
        return payments.result_for(idempotency_key)  # no second charge
    return payments.charge(amount, key=idempotency_key)

Generate the key once per logical action, before the first attempt, and reuse it across every retry of that action. If you generate a fresh key on each retry, the server sees each attempt as new and the protection is gone. Any tool that writes or spends money should be idempotent before you ever put it behind a retry.

Let the model recover, or handle it in code?

Both layers can respond to failure. Splitting them well is the core design decision, and the rule is short: handle the mechanical stuff in code, hand the judgment calls to the model.

Code owns the failures where the right response is fixed and needs no reasoning:

Transient errors, retried with backoff.
Timeouts on hung tools.
The step and budget limits that stop a runaway loop.

The model owns the failures where the right response depends on the goal:

A permanent error where a different tool or argument might work.
An empty result that suggests a different search.
An ambiguous state where the next move depends on what the task is actually trying to do.

The failure you should never allow is the one that escapes both layers and kills the run. If code cannot fix it and you did not pass it to the model, the agent crashes on a problem a human would have shrugged off. Route every failure to one layer or the other.

flowchart LR
    F[Tool fails] --> D{Fixed response<br/>or judgment?}
    D -->|fixed| C[Code: retry,<br/>timeout, limit]
    D -->|judgment| M[Model: adapt<br/>via error text]
    C --> L[Loop continues]
    M --> L

Wrapping a tool call with a timeout and a retry

Here is the shape in one place. It retries transient errors with backoff, enforces a timeout, sends permanent errors straight to the model, and never lets an exception escape.

import time, random

TRANSIENT = (TimeoutError, ConnectionError)

def call_with_retry(fn, *args, retries=3, timeout=5.0, **kwargs):
    for attempt in range(retries):
        try:
            return run_with_timeout(fn, timeout, *args, **kwargs)
        except TRANSIENT as e:
            if attempt == retries - 1:
                return f"ERROR: {type(e).__name__} after {retries} tries: {e}"
            wait = (2 ** attempt) + random.random()  # backoff + jitter
            time.sleep(wait)
        except Exception as e:
            # Permanent: retrying won't help. Let the model decide.
            return f"ERROR: {type(e).__name__}: {e}"

run_with_timeout is whatever your stack gives you: a concurrent.futures future with future.result(timeout=...), an asyncio.wait_for, or a client-level request timeout. The point is that transient errors get a few quiet retries, permanent errors and exhausted retries come back as readable text, and nothing bubbles up to crash the loop.

Common beginner mistakes

Letting exceptions escape the loop: an unhandled tool error kills the whole run. Catch it and return the message as a tool result instead.
Retrying permanent failures: a 404 or a bad argument fails the same way every time. Retrying just wastes attempts before the model gets its chance.
No timeout on a network tool: one hung call freezes the sequential loop indefinitely, and the step limit never fires.
Retrying a non-idempotent write: without an idempotency key, a retried charge or insert runs twice.
Opaque error strings: ERROR: 500 gives the model nothing to reason about. Say what actually went wrong.
New idempotency key per attempt: generate the key once per action and reuse it, or the retry looks like a brand-new request.

Questions you will face in production

“Should I retry in code or just let the model try again?” Retry in code for transient, mechanical failures where the response is fixed and no reasoning is needed: a timeout, a 503, a rate limit. Let the model handle failures where the right next step depends on the goal, like a wrong argument or an empty result. Code retries are cheaper and faster; model retries cost a full turn but can actually change the approach.

“How many times should I retry?” Three attempts with exponential backoff and jitter is a good default for transient errors. More than that and you are usually delaying an inevitable failure while the user waits. Pair it with an overall time or cost budget for the whole loop so a chain of slow retries cannot blow past what you are willing to spend.

“What if a tool succeeds but the response is lost?” This is exactly why idempotency matters. From your side it looks like a failure, so you retry, but the action already ran. Make any tool with side effects idempotent using a key generated once per action, so the second attempt is a safe no-op that returns the original result.

What to remember

A failing tool is not a failing service: you can hand the error to the model and it can adapt.
Never let a tool exception escape the loop; return it as a readable tool result.
Retry transient errors in code with backoff and jitter; time out every network tool.
Send permanent errors straight to the model; retrying them only wastes attempts.
Make any tool with side effects idempotent before you put it behind a retry.
Code handles the mechanical failures, the model handles the judgment calls, and nothing should crash the run.

What to study next

Failure handling keeps the loop alive; the next step is controlling what it is allowed to do when it recovers. Read Guardrails and Human in the Loop for how to put approval gates and hard limits around risky tool calls. For the retry patterns one layer down, at the LLM API itself, see your first LLM integration, where the same backoff logic applies to the model call.