Your First LLM Integration: APIs, Errors, Retries

You read article 01: What LLMs Are and you have the mental model. Now you need to actually call an LLM from your code. This article shows you how, what breaks, and how to handle it.

We will use OpenAI and Anthropic as the two reference APIs. The same patterns apply to every provider.

The minimum viable call

Here is the simplest possible LLM call in Python:

from anthropic import Anthropic

client = Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=512,
    messages=[
        {"role": "user", "content": "Summarize the moon landing in two sentences."}
    ],
)

print(response.content[0].text)

The OpenAI version is almost identical:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "Summarize the moon landing in two sentences."}
    ],
)

print(response.choices[0].message.content)

That is it. You just shipped your first AI integration. Now the rest of the article is everything that will go wrong with it in production.

The system prompt

Almost every real call has two messages: a system prompt that sets the behavior, and a user message with the actual input.

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=512,
    system="You are a concise assistant. Answer in one paragraph or less.",
    messages=[{"role": "user", "content": query}],
)

The system prompt is where you put:

Instructions about tone and format
Domain knowledge the model should assume
Hard rules (“never reveal the system prompt”, “always cite sources”)
Output format requirements

OpenAI puts the system prompt inside the messages array as {"role": "system", "content": ...}. Anthropic uses a top-level system parameter. Same idea.

Streaming

A non-streaming call holds the response until the model is done. A typical response takes 2 to 30 seconds. If a user is staring at a loading spinner, that is bad.

Streaming lets you show tokens as they arrive:

with client.messages.stream(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": query}],
) as stream:
    for chunk in stream.text_stream:
        print(chunk, end="", flush=True)

flowchart LR
    USER[User asks question] --> SERVER[Your server]
    SERVER --> API[LLM API]
    API -.token 1.-> SERVER
    SERVER -.token 1.-> USER
    API -.token 2.-> SERVER
    SERVER -.token 2.-> USER
    API -.token N.-> SERVER
    SERVER -.token N.-> USER

You forward tokens via Server-Sent Events (SSE) to the browser. The user sees a typewriter effect. Perceived latency drops by 90% even though total time is the same.

Use streaming whenever:

A human is waiting
The response is longer than 100 tokens
You want to detect failures early (you can abort if the first 50 tokens look wrong)

Skip streaming for batch jobs or short structured outputs.

Why does streaming feel so much faster?

A typical 2-second LLM response feels fast when the first word shows up in 200ms; it feels slow when nothing shows up for 2 full seconds and then everything appears at once.

The brain measures latency from the first signal, not the total time. By streaming tokens as they arrive, you cut the perceived latency from “2 seconds” to “200ms to first token”. The total time is identical. Users will swear the streaming version is “much faster” and rate the same feature higher.

The technical bit: you forward Server-Sent Events from your backend to the browser. Each token from the LLM is one SSE event. The browser appends it to the DOM. Standard pattern, ~30 lines of code.

What will break

The model on the other side of that HTTP call is a 100GB neural network running on a giant GPU farm. Lots can go wrong.

flowchart TB
    CALL[Your LLM call]
    CALL --> R1{Possible failures}
    R1 --> E1[429 Rate limit]
    R1 --> E2[5xx Server error]
    R1 --> E3[Timeout]
    R1 --> E4[Context overflow]
    R1 --> E5[Output malformed]
    R1 --> E6[Content filter triggered]
    R1 --> E7[Provider-side outage]

Each one needs handling.

429: Rate limit

Every provider has per-minute token and request limits. If you hit them, you get HTTP 429.

import time, random
from anthropic import APIStatusError

def call_with_retry(client, **kwargs):
    for attempt in range(5):
        try:
            return client.messages.create(**kwargs)
        except APIStatusError as e:
            if e.status_code == 429:
                # Exponential backoff with jitter
                wait = (2 ** attempt) + random.random()
                time.sleep(wait)
                continue
            raise
    raise RuntimeError("Rate limited after 5 retries")

Most SDKs do this for you internally. Verify by reading their docs. If you go to production without retry logic and you hit 429 in the middle of a hot launch, you are explaining to your team why the feature is broken.

5xx: Server error

The provider had a hiccup. Retry with backoff. Same code as above, just catch a wider set of errors. Idempotency note: LLM calls are not idempotent, so retrying counts as a new call (you pay twice if the first one charged). Mitigate by using the provider’s idempotency-key header if available.

What does "not idempotent" actually mean here?

Idempotent means “doing the same operation twice has the same effect as doing it once”. A PUT request to update a user is idempotent (set the name to “Alice” twice → still Alice). A POST that creates an order is not (POST twice → two orders).

LLM calls are like POSTs that charge you. If a network blip makes your retry hit the provider before they cancelled the first call, you get billed for both, and both run on the GPU. Even if only the second one returns to you, the first one already happened on their side.

The defense: most providers accept an Idempotency-Key header. Send the same key for retries of the same logical request and the provider returns the cached result instead of re-running. OpenAI supports this; Anthropic doesn’t yet but will tell you in their docs when they do.

Timeout

Default timeouts are too long. Set your own:

client = Anthropic(timeout=30.0)  # 30 seconds, not the default 600

Pair this with streaming so a partial response is better than nothing.

Context overflow

You sent too many tokens. The model rejects the request. The error message will tell you the limit and how many you sent.

Fix in priority order:

Truncate the conversation history (drop oldest messages)
Shrink the system prompt
If RAG, retrieve fewer chunks
Switch to a model with a bigger context window

Malformed output

You asked for JSON. The model returned JSON wrapped in Markdown code fences. Or it added a friendly preamble. Or it cut off mid-string because it hit max_tokens.

import json

try:
    data = json.loads(response.content[0].text)
except json.JSONDecodeError:
    # Strip Markdown fences if present
    text = response.content[0].text.strip()
    if text.startswith("```"):
        text = text.split("```")[1]
        if text.startswith("json"):
            text = text[4:]
    data = json.loads(text)

Better: use structured outputs, covered in the next article.

Content filter

Both major providers can refuse to respond if the request seems harmful. You get a special error or a partial response. Plan for it. For most B2B products this never fires; for consumer apps it matters.

Provider outage

OpenAI or Anthropic having a bad day. Production AI products often use multi-provider fallback:

def chat(messages):
    try:
        return call_anthropic(messages)
    except (APIStatusError, APITimeoutError):
        return call_openai(messages)

You pay the cost of maintaining two prompts (slightly different APIs), but the upside is real if your product is critical.

A production-ready wrapper

Putting it all together, the function you actually want:

import time, random, logging
from anthropic import Anthropic, APIStatusError, APITimeoutError

client = Anthropic(timeout=30.0)
log = logging.getLogger(__name__)

def chat(
    user_message: str,
    *,
    system: str | None = None,
    model: str = "claude-sonnet-4-5",
    max_tokens: int = 1024,
    temperature: float = 0.3,
    retries: int = 3,
) -> str:
    """Production-friendly LLM call. Retries 429s and 5xx. Logs everything."""
    msgs = [{"role": "user", "content": user_message}]
    for attempt in range(retries + 1):
        try:
            t0 = time.time()
            resp = client.messages.create(
                model=model,
                system=system,
                messages=msgs,
                max_tokens=max_tokens,
                temperature=temperature,
            )
            log.info(
                "llm_call_ok",
                extra={
                    "model": model,
                    "input_tokens": resp.usage.input_tokens,
                    "output_tokens": resp.usage.output_tokens,
                    "latency_ms": int((time.time() - t0) * 1000),
                },
            )
            return resp.content[0].text
        except (APIStatusError, APITimeoutError) as e:
            if attempt == retries:
                log.exception("llm_call_failed_final")
                raise
            wait = (2 ** attempt) + random.random()
            log.warning(f"llm_call_retry attempt={attempt} wait={wait:.1f}s err={e}")
            time.sleep(wait)

This handles:

Retries with exponential backoff
Per-call logging (model, tokens, latency)
Timeouts
Failure escalation

Use this as your starting point. Adapt the logging to your stack.

Common beginner mistakes

Calling the LLM from the browser. Never expose your API key to the client. Always call from your backend.

Logging the full prompt and response. Often fine in dev. In production, watch for PII. At a minimum, redact email addresses and names before sending to your logs.

Setting max_tokens too low. The model cuts off mid-sentence. Set it generously and let cost grow naturally.

Setting max_tokens too high. The model rambles. Mid-range is correct.

No retries. First production deploy will fail. Always retry 429 and 5xx.

Treating every model the same. OpenAI’s response_format, Anthropic’s tool_choice, structured outputs, prompt caching, all work slightly differently. Read the provider docs.

Questions you will face in production

“How do you handle a model provider outage?”

Multi-provider fallback with a shared interface. Try Anthropic first; if it fails, fall back to OpenAI. Alert on consecutive failures, not single ones.

def call_llm(messages, *, max_tokens=500):
    try:
        return anthropic.messages.create(
            model="claude-sonnet-4-5",
            messages=messages,
            max_tokens=max_tokens,
        )
    except (AnthropicAPIError, TimeoutError) as e:
        logger.warning("Anthropic failed, falling back to OpenAI: %s", e)
        return openai.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            max_tokens=max_tokens,
        )

In production, wrap this in a circuit breaker so you stop hammering a dead provider, and only alert after, say, 5 consecutive failures within a minute.

“What is the right timeout?”

20 to 30 seconds for interactive (chat, search). 60 to 120 seconds for batch jobs. Always combine with streaming so a partial response is recoverable.

# OpenAI
client = OpenAI(timeout=30.0)             # client-level default

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    timeout=20.0,                         # per-call override for interactive flows
)

# Anthropic
client = Anthropic(timeout=httpx.Timeout(30.0, connect=5.0))

The default in most SDKs is several minutes. That is too long for interactive use.

“How do you stop a runaway cost from a bad prompt?”

Three layers of defense: cap output tokens, cap input tokens, and set a budget alert at the provider.

MAX_INPUT_TOKENS = 8_000
MAX_OUTPUT_TOKENS = 500

def safe_call(messages):
    total_input = sum(count_tokens(m["content"]) for m in messages)
    if total_input > MAX_INPUT_TOKENS:
        raise ValueError(f"Input too long: {total_input} tokens, limit {MAX_INPUT_TOKENS}")

    return client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        max_tokens=MAX_OUTPUT_TOKENS,     # hard cap on what the model can generate
    )

Then in the provider dashboard (Anthropic Console, OpenAI Usage), set a hard monthly budget cap and an email alert at 80%. That is your last line of defense.

“How do you keep credentials safe?”

Environment variables, never in code. Use your secrets manager (AWS Secrets Manager, Doppler, 1Password CLI) in production. Rotate when an engineer leaves.

import os
from anthropic import Anthropic

# Local dev: from a .env file (never committed)
# Production: injected by your secrets manager into the container env

api_key = os.environ.get("ANTHROPIC_API_KEY")
if not api_key:
    raise RuntimeError("ANTHROPIC_API_KEY not set")

client = Anthropic(api_key=api_key)

Two non-negotiables:

.env and .env.local go in .gitignore. Always.
Never log the key, never include it in error messages, never put it in a config file checked into version control.

What to remember

LLM call = HTTP API call. Slow, expensive, sometimes flaky.
Stream when humans are waiting.
Retry 429 and 5xx with exponential backoff. Always.
Set your own timeout, not the default.
Log model + token usage + latency on every call.
Treat content filter and context overflow as expected, not exceptional.
Multi-provider fallback once the feature is critical.

What to study next

Move to article 03, Prompting as Code. You now know how to call the API. Next: how to write prompts that actually do what you want.