What long-term memory actually takes in a production LLM application

At 10:47 p.m., I was standing in my kitchen with one hand on a cold mug of coffee, testing a support assistant on my laptop. The conversation looked fine until the assistant confidently recommended a refund policy that had changed six weeks earlier.

That was not a model failure. The model had answered from a memory system that stored an old policy, retrieved it without checking its age, and offered no way to correct the record. In a production LLM application, long-term memory is not a larger context window. It is a data system with rules for capture, retrieval, correction, privacy, and cost.

I have built multi-model AI products and spent years shipping systems where a stale record can generate a support ticket, a billing dispute, or a compliance flag. The hard part is not making a model remember. The hard part is deciding what deserves to survive the current conversation, how strongly it should be trusted, and when it should disappear.

Long-term memory is a product decision before it is a database decision

A language model does not remember a customer in the human sense. It receives tokens during a request. Those tokens might come from the current conversation, a summary, a database row, a retrieved document, or a tool result. The application decides what enters that prompt and what stays outside it.

That distinction matters. A context window is temporary working space. Long-term memory is information your application stores and may retrieve later. The two systems interact, but they are not the same thing.

The usual advice is simple: store every conversation, embed it, and retrieve the most similar passages later. That approach is attractive because it can be built quickly with PostgreSQL and pgvector, Pinecone, Weaviate, or another vector store. It is also how teams end up with a memory attic full of objects nobody can identify.

Here's the part most people miss: similarity is not truth. A semantically similar message can be old, sarcastic, incomplete, private, or specific to a situation that no longer applies. If a customer says, "I hate pineapple on pizza," that may be a useful preference. If they say, "Pretend I am a chef who hates pineapple," storing it as a permanent food preference is a bug.

A production memory feature needs an explicit answer to four questions:

• What information can be saved?
• Who or what can authorize the save?
• When should the information be retrieved?
• How can a person inspect, correct, or delete it?

A useful memory architecture separates at least three things:

• Working memory for the current request and active conversation.
• Durable facts such as a confirmed account preference or project decision.
• Source material such as documents, tickets, and event records that can be retrieved without becoming personal memory.

An educational assistant illustrates the difference. A confirmed reading preference is different from a single confusing answer during a Tuesday session. Treating both as equal creates a tutor that grows more certain as it grows more wrong.

Capture only memories that have a job

Memory capture should begin with a use case, not a storage mechanism. If a memory cannot change a future decision, response, permission check, or workflow, it probably does not belong in durable memory.

Consider an assistant for a hotel support desk. It might store that a guest requested a ground-floor room because of limited mobility, because that fact can affect a future reservation. It should not store every sentence from the conversation, including the guest's passing complaint about airport traffic.

An assistant for an engineering team might store that the payments service runs PostgreSQL 16 and that schema changes require a migration review. It should not convert a speculative suggestion from a design meeting into an architectural rule.

I use a capture test that asks whether the proposed memory is:

• Specific enough to act on.
• Stable beyond the current exchange.
• Useful in a defined future situation.
• Safe to retain for the stated purpose.
• Traceable to a source and capture time.

That last item gets ignored most often. Do not store a bare sentence like "User prefers concise answers." Store a record with an owner, source, timestamp, confidence, scope, and expiration policy. A simple relational shape might look like this:

```text

memory_id

subject_id

memory_type

content

source_event_id

created_at

updated_at

confidence

scope

expires_at

status

```

The `scope` field matters more than many teams expect. A person can prefer concise answers for customer support and detailed answers for technical design work. A global preference inferred from one context can quietly damage every other interaction.

Capture can use an LLM, but the model should propose a memory rather than silently write it. A small classifier or structured extraction prompt can return a candidate record. Application code then rejects records that lack a type, source, owner, or expiration date.

The mistake I keep watching teams make is letting the model write directly to the memory store because the demo feels smoother that way. Add a write gate. Log every accepted and rejected candidate. Give the user a visible way to say, "Do not remember that."

Memory should earn its place. A bigger pile is not better recall.

Retrieval is a ranking problem with permissions attached

Once information exists, the application has to decide what enters the next prompt. This is where many LLM memory designs stop at vector similarity. A query arrives, the system creates an embedding, and the vector database returns the nearest records.

That handles one signal. Production retrieval needs several.

A useful retrieval score combines semantic similarity with recency, source reliability, scope, user confirmation, and business permissions. The exact formula will vary, but the fields should be visible in logs. If a stale billing policy outranks a current policy document, you need to know why.

Consider a support assistant answering a refund question at 2 a.m. in Dublin. It finds three records:

• A customer message from January saying a manager approved a refund.
• A help-center document from March describing the old refund window.
• A policy record updated in August with the current rule.

The January message may be relevant to the customer's case. The March document is semantically close but obsolete. The August record is the current policy. A similarity-only retriever can surface any of them first.

Your retrieval pipeline should filter before ranking where possible. Check tenant, user, data classification, document status, and permission boundaries before the model sees a result. A customer from one account must never retrieve another account's support history because the text happens to be similar.

Then rank the remaining candidates using signals such as:

• Semantic match to the current request.
• Recency and explicit expiration.
• Confirmed versus inferred status.
• Authority of the source.
• Scope of the memory.

Do not assume the model will resolve conflicts reliably when you drop five contradictory facts into its context. Resolve simple conflicts in application code. If two records describe a preference, prefer the later confirmed value over the older inferred one. If two policy documents disagree, route the conflict to a source-of-truth rule or a human review path.

Your context budget is the amount of prompt space and attention you can spend on retrieved information before the response becomes slower, more expensive, or less reliable. A long context window does not remove this budget. A model can accept a large prompt and still miss the right detail. (And yes, I have watched that happen in a live demo, which is a special kind of pain.)

Measure retrieval, not just final answer quality. Track hit rate, stale-memory rate, unauthorized retrieval attempts, tokens added per request, and the percentage of responses that cite or depend on memory. If you cannot see those numbers, you are tuning by anecdote.

Summaries are useful, but they are lossy compression

Conversation summaries are often the first memory feature teams ship. After a session reaches a length threshold, an LLM compresses it into a short paragraph. The next session receives the summary instead of the full transcript.

That works when the summary has a narrow purpose. It becomes risky when teams treat the summary as a complete record.

In 2023, the Generative Agents research project used a memory stream, retrieval, and reflection to support believable behavior in a simulated town. The work showed why memory systems need more than storage: the system has to decide which past events matter for the current situation. It also showed the danger of letting remembered events accumulate without a clear product boundary.

A summary can lose negation, uncertainty, timing, or ownership. "The team discussed moving the launch to October" can collapse into "The launch moved to October." That single change can send a project assistant in the wrong direction for weeks.

Use typed summaries rather than one free-form paragraph. For a project assistant, store separate fields for:

• Decisions that were approved.
• Open questions that still need an answer.
• Actions with owners and due dates.
• Constraints that came from a source document.
• Facts that remain uncertain.

For a tutoring product, a session summary might record that a learner solved two-step subtraction with visual support but needed help reading the word problem. That is more useful than a polished paragraph about the session. It gives the next interaction a testable starting point.

Summaries also need provenance. Store the source conversation range or event IDs used to create the summary. If a user challenges a remembered fact, the application should show where it came from and replace it without rebuilding the entire history.

A summary is not a transcript. It is an opinion about what matters. Treat it that way.

Run a summary check against the source before saving it. Ask a separate model, rule set, or evaluator to flag changed dates, dropped negation, invented decisions, and missing owners. Keep the original records available under the product's retention policy.

And set an expiration date for temporary summaries. A project status from last quarter should not keep shaping a response after the project closes.

Privacy, deletion, and correction are part of the feature

Long-term memory creates a second data surface. Your team already has a user database, logs, analytics events, support records, and document storage. A memory layer adds another place where personal data can hide.

That shifts the design question from "Can the model remember this?" to "Can the product explain, control, and remove this?"

The European Union's General Data Protection Regulation includes rights related to access, correction, and erasure. California's Consumer Privacy Act also gives consumers rights over personal information. The exact duties depend on your company, users, and processing activities, so involve counsel early. The engineering point is simpler: deletion cannot mean removing one vector row while leaving the same fact sitting in a transcript, summary, cache, backup, or analytics event.

Build a memory inventory before launch. For each memory type, identify:

• The system of record.
• Copies and derived forms.
• Retention period.
• Deletion owner.
• Access rules.
• Audit events.

Use stable IDs so a deletion request can fan out across stores. If you use embeddings, remember that the vector is derived personal data when it can be linked to a person. Deleting the visible text but retaining the searchable representation is not a clean deletion design.

Correction needs the same care. Suppose Priya tells a personal finance assistant, "I no longer work at Acme." The application should not merely append a new fact and leave the old employer active. It should mark the previous record superseded, preserve the audit trail where allowed, and retrieve only the current value.

Do not put sensitive information into a memory system just because the model can use it. Health details, financial information, precise location, authentication data, and private family information need a clear purpose and stronger controls. Often the right answer is not to store them at all.

Give people a memory control panel that shows plain-language records, not embedding IDs. Let them view, edit, delete, and disable memory. This is not a nice extra. It is how you discover that your inferred preference is nonsense before a customer does.

Reliability comes from an operational loop

Memory failures rarely appear as a clean database error. The service returns 200, the model writes fluent prose, and the user gets an answer built on the wrong past.

That makes observability part of the memory design, not an afterthought. Instrument the path from capture to retrieval to response. Record which memories were considered, which passed filters, which entered the prompt, and whether the model claimed to use them.

Keep sensitive text out of general logs. Log IDs, types, scores, timestamps, and redacted snippets where possible. Store the full audit record under separate access controls.

A practical production dashboard can track:

• Memory write acceptance and rejection rates.
• Retrieval hit rate on labeled test queries.
• Stale or expired memories returned.
• Average memory tokens per request.
• User corrections and deletions.
• Responses that conflict with current source documents.

Build a memory test set from real failure patterns. Include the Dublin refund case, the old employer case, a changed hotel room preference, and a project decision that was discussed but never approved. Test exact retrieval, conflict handling, tenant isolation, and deletion propagation.

Then test the absence of memory. Ask what happens when no relevant record exists. A safe assistant should say it does not know or ask a targeted question. It should not fill the gap with a plausible invention drawn from an unrelated memory.

The conventional startup advice is to ship the memory feature quickly and improve it from user feedback. For LLM applications, that often creates a hidden dependency that becomes hard to unwind. Once users trust the assistant to remember, changing its behavior can feel like data loss even when the original system was poorly designed.

So the deeper lesson is to ship the controls with the memory: source tracking, expiration, correction, deletion, retrieval logs, and evaluation cases. The model call is the easy part.

A production plan that starts small

You do not need a human-like memory system on day one. Start with one memory type and one future task it supports.

For example, a customer support assistant might begin with confirmed account preferences. Do not store open-ended conversation history as durable memory. Do not infer sensitive traits. Do not let the model write without a structured proposal and application validation.

A sensible rollout looks like this:

• Define one memory type and its allowed fields.
• Store source, scope, timestamp, confidence, and expiration.
• Add a user-facing view and delete action.
• Filter permissions before semantic retrieval.
• Create a small labeled test set from real conversations.
• Track cost, stale retrieval, correction, and deletion behavior.
• Expand only after the first memory type holds up under failure testing.

Use PostgreSQL for relational memory when you need transactions, ownership, audit records, and clear joins. Add pgvector when semantic lookup helps. Use object storage for long transcripts where appropriate, but keep the metadata and deletion links in a system you can query. A dedicated vector database can make sense at higher scale, though it does not solve capture or policy decisions for you.

Keep memory writes asynchronous when the user does not need them before the response. This reduces response latency and gives you room to validate candidates. For facts that affect the immediate answer, use an explicit confirmation path instead of assuming the write succeeded.

The second half of the work is product language. Say "I saved that preference" only when the system actually saved it. Say "I found an older note" when the result may be stale. Let the user ask, "What do you remember about me?" and answer from the same records your application uses.

If you are hiring an AI product or applied-AI leader, listen for this level of detail. A candidate who talks only about vector search has not described a memory product. A candidate who asks about retention, conflict rules, correction paths, tenant boundaries, and operating metrics is thinking about the system that actually reaches production.

Long-term memory is not a personality feature. It is a controlled data pipeline attached to a model.

FAQ

What is long-term memory in an LLM application?

Long-term memory is information an application stores beyond the current conversation and retrieves for later requests. It can include confirmed preferences, project decisions, or structured user facts, but it should have a source, scope, retention rule, and correction path.

Is a larger context window the same as LLM memory?

No. A context window is temporary input available during one model request. LLM memory persists outside that request and requires storage, retrieval, access control, expiration, and deletion behavior.

Should I store every conversation in a vector database?

Usually not. Full transcripts can be useful as source records, but storing every message as durable personal memory creates stale, duplicated, and misleading retrieval results. Start with a narrow memory type tied to a defined product task.

Which database is best for AI memory?

The right choice depends on the data and workload. PostgreSQL with pgvector is often a strong starting point because it combines structured records, permissions, transactions, and vector search. A dedicated vector database may help at larger retrieval scale, but neither choice replaces memory policy.

How do I test memory quality?

Create labeled cases for correct retrieval, stale facts, conflicting facts, missing memory, unauthorized access, correction, and deletion. Track retrieval quality and system behavior separately from the final answer, because a fluent answer can still come from the wrong record.

Your next step is simple: pick one memory type, write down its source and expiration rules, and build the delete path before adding more retrieval. The model is not the memory system. The controls around the model are what make memory safe to ship.