State and Memory

An LLM call has no memory. Every request starts from scratch, so if your agent is going to plan across ten steps, something has to carry what happened between them. That something is state, and once tasks run long enough, managing it becomes one of the harder parts of building an agent.

This article covers where an agent’s state actually lives, why the context window forces you to prune it, and when you need memory that outlives a single run.

An LLM call is stateless

The model does not remember your last request. Each call to the API is a pure function: text in, text out, nothing persists on the model’s side. Ask a question, then ask a follow-up in a separate call, and it has no idea the first exchange happened.

So how does a chat feel like it remembers? The client resends the whole conversation every time. The “memory” is not in the model. It is in the list of messages you keep on your side and pass in on every call.

For an agent, this list is the working state. The agent loop appends to it on every turn: the model’s response, the tool it asked for, the result of that tool. Each pass, you send the growing list back so the model sees everything it has done so far.

flowchart LR
    U[User goal] --> H[Message<br/>history]
    H --> M[Model call]
    M --> T[Tool result]
    T --> H
    M --> A([Answer])

The history is the agent. Lose it between turns and the agent forgets its own plan mid-task.

The context window is finite

That growing list has a hard ceiling. Every model has a context window: the maximum number of tokens it can read in one call, prompt and response combined. Send more than that and the call fails.

On a short task this never comes up. On a long one it always does. A research agent that reads twenty pages, or a coding agent that pastes in file after file, piles up tool results until the history no longer fits. Even before the hard limit, long histories cost more (you pay per token on every call) and the model tracks a bloated context worse than a tight one.

So on any long-running task you cannot just keep appending. You have to actively manage what stays in the history and what gets dropped.

Managing history: truncation and summarization

Two moves keep a history inside the window. You will use both.

Truncation drops old messages. The simplest version keeps the system prompt, the original goal, and the last N turns, and throws away everything older. It is cheap and easy, and it works when only recent context matters. The risk: you drop a fact the model still needs, like a decision made early in the task, and it starts contradicting itself.

Summarization (also called compaction) replaces old turns with a short summary. Instead of deleting the first fifteen tool calls, you ask the model to compress them into a paragraph (“Searched the docs, found the config lives in settings.py, confirmed the bug is in the parser”), then continue with that summary standing in for the raw history. You keep the meaning and pay far fewer tokens.

Compaction costs an extra model call and can lose detail, so trigger it on a threshold, not every turn: when the history crosses some fraction of the window, compact the oldest turns and keep going.

flowchart TB
    C{History over<br/>threshold?}
    C -->|no| K[Keep appending]
    C -->|yes| S[Summarize<br/>oldest turns]
    S --> R[Replace old turns<br/>with summary]
    R --> K

A useful default: keep the system prompt and goal pinned, always keep the last few turns verbatim (the model needs recent detail to act), and compact the middle.

A history that fits

Here is the shape of compaction in the loop. The model itself does the summarizing:

def maybe_compact(messages, model, max_tokens=6000, keep_recent=4):
    if count_tokens(messages) < max_tokens:
        return messages

    pinned = messages[:1]          # system prompt / goal
    recent = messages[-keep_recent:]
    stale = messages[1:-keep_recent]

    summary = model.call([
        {"role": "user",
         "content": "Summarize these agent steps in 5 lines, "
                    "keeping decisions and facts:\n" + render(stale)},
    ]).text

    return pinned + [{"role": "system", "content": f"Earlier: {summary}"}] + recent

Call maybe_compact at the top of each loop iteration before sending to the model. On short tasks it is a no-op; on long ones it quietly holds the history inside budget without dropping the decisions that matter.

Short-term versus long-term memory

Everything so far is short-term memory: the working state of one run. When the task ends, the message list is discarded. That is the right default. Most agents should start each run clean.

But some things should outlive a run. If a user tells your assistant “I deploy to us-east-1” today, they do not want to repeat it next time. That fact belongs in long-term memory: a store that persists across separate runs and conversations.

The two solve different problems, so keep them separate:

Reach for long-term memory only when persistence earns its keep. It adds a store to run, a retrieval step, and a new way to be wrong (stale or irrelevant recalled facts). A stateless agent that starts fresh each run is simpler and often enough.

Long-term memory needs retrieval

Long-term memory has a catch the short-term kind does not. Working state is small enough to send in full. Long-term memory grows without bound, and you cannot paste a user’s entire history into every prompt: it would not fit, and most of it is irrelevant to the current task.

So long-term memory is two operations: write a fact to the store, and retrieve only the facts relevant to right now. Retrieval is the hard half. Keyword search misses paraphrases (“prod region” will not match “us-east-1”). What you want is search by meaning, which is exactly what embeddings and vector search give you.

You store each memory as an embedding, a vector that captures its meaning. At query time you embed the current situation, find the nearest stored vectors, and inject just those few memories into the prompt. It is the same retrieval you would build for RAG; the full treatment is in the embeddings deep dive.

flowchart LR
    F[New fact] --> E1[Embed] --> V[(Vector<br/>store)]
    Q[Current task] --> E2[Embed] --> N[Find nearest] --> V
    N --> P[Inject top matches<br/>into prompt]

The pattern is worth saying plainly: short-term memory is the whole history in the prompt; long-term memory is a big external store you query for the few relevant pieces and inject on demand.

Common beginner mistakes

• Rebuilding the message list each turn: append to one list and carry it; do not reconstruct history from scratch and lose earlier turns.
• No plan for the context limit: it works in the demo, then the first long task blows the window. Decide your truncation or compaction strategy before you ship.
• Truncating the goal: never drop the original task or system prompt when trimming; pin them.
• Compacting away decisions: a summary that keeps chit-chat but loses “we chose Postgres” makes the agent contradict itself.
• Long-term memory when you needed none: adding a vector store to an agent that could just start fresh is complexity you will maintain forever.
• Never expiring stored facts: a preference from a year ago can be wrong now; give long-term memory a way to go stale.

Questions you will face in production

“How do I know when to compact?” Track the token count of the history and trigger on a threshold, commonly somewhere around half to three-quarters of the window, not on a fixed turn count. Tasks vary wildly in how fast they fill context, so watch the tokens, not the turns.

“Should I persist the full conversation to a database?” Logging raw runs for debugging is fine and useful. That is different from long-term memory, which is curated facts you deliberately retrieve later. Do not confuse an audit log with a memory store; dumping whole transcripts back into future prompts recalls mostly noise.

“Won’t summarizing lose important detail?” It can, which is why you keep the most recent turns verbatim and only compact older ones. Tell the summarizer explicitly to preserve decisions, identifiers, and unresolved questions. Detail you might still need should stay raw; only the settled past gets compressed.

What to remember

• An LLM call is stateless; the agent’s working memory is the message history you resend each turn
• The context window is finite, so long tasks force you to manage that history
• Truncation drops old turns; summarization (compaction) replaces them with a short recap
• Pin the goal and system prompt, keep recent turns verbatim, compact the middle
• Short-term memory is one run’s history; long-term memory persists across runs
• Long-term memory needs retrieval, and retrieval by meaning is embeddings plus vector search

What to study next

State is what lets an agent think across steps. The next thing that breaks a long run is not memory but the world being unreliable: tools time out, APIs return errors, calls hang. Read retries, timeouts, and failure handling for how to keep the loop alive when the things it calls fall over. If the embeddings side was new, the embeddings deep dive covers the retrieval half of long-term memory in full.

Further reading

• Anthropic: Building effective agents. Practical patterns for agent design, including where state and memory fit.
• Lost in the Middle: How Language Models Use Long Contexts. The evidence that models use information buried in long inputs worse than short ones.
• Anthropic: Contextual retrieval. How retrieval-augmented context is built, the mechanism behind long-term memory recall.

Where this article comes from. This is a synthesis of common practice in AI engineering as of 2026, not a citation of any single paper. The sources above are where the mechanics come from. If you find an error or have a better source for a claim, the article gets fixed within a day, send me a note.