Not All Memory is the Same

Memory is a complex subject, and an active research area. Our understanding of how our own memory works is still growing, and that as a metaphor is an imperfect match for AI agents which are wired differently to us (literally!).

When people talk about memory for agents, there are a number of different things they might mean:

• Facts: "This user is a subscriber to our Pro plan", "This project uses Go 1.26".
• Preferences: "Jon prefers aisle seats", "Peter prefers it when I talk like a pirate".
• Procedures: "After I'm done writing code, I must run the end-to-end test suite and check the linter passes".
• History: "Last time I tried to refactor these files, it failed because there were subtle runtime bugs that I didn't check for".

We categorise these into three different kinds of memory: factual, procedural and episodic.

Factual memory

This is the most straightforward kind of memory - it consists of remembered things that are true (or believed to be true). It includes information about concrete entities, properties of them, and preferences.

For example:

• The production app uses Next.js.
• Peter doesn't like me to commit or push code for him.
• The team uses Github Issues for ticket tracking.
• The coding standards prohibit throwing exceptions.

These are typically very useful snippets of information, but also tend to be tightly scoped and useful in particular circumstances. For example, the agent might find it useful to know that it's working with Next.js when planning an approach to a piece of work, but it doesn't need to know about the coding standards until it goes to implement that. Recall of this memory can hence be a tricky problem.

An important point to note is that this information can become stale over time. Just because a fact is true today doesn't mean it will be tomorrow; over time we will discover new information that invalidates some of our old memories. Hence the memory system needs to be able to account for this - old memories must sometimes be invalidated and forgotten.

Procedural memory

Procedural memory covers workflows, habits and conventions. It's the learned knowledge that tells the agent how it should act and how it should achieve the goals it's given. This takes the form of things like:

• After making code changes, make sure to run the linters via make lint to check for any errors.
• If the answer to a user's query exists in the documentation, always provide as specific a hyperlink as possible to it so they don't have to search.
• When iterating over a map, make sure to sort the keys first in any case where deterministic ordering matters.

There's an underlying "personality" from the original model, which is then heavily shaped by the system prompt; procedural memories augment that with more specific processes it should follow. These are very often relevant in a particular case or situation, where the memory can go into significantly more detail without unnecessarily polluting context for tasks that don't require it.

These can also be be invalidated over time. Typically that happens less frequently than factual memories, but it is still possible¹.

Episodic memory

The final type of memory we'll discuss here is episodic memory, which is recollection of events, interactions and outcomes. This is critical to how we build memory, because it's where the others arise from. Individual episodes capture a particular part of an interaction and can be grouped together with other, similar episodes to reinforce a particular lesson.

An agent can accrue a large volume of transcripts and hence a large number of potential episodes over time. This means that an important job of the memory system is to decide which parts of these transcripts form episodes and which of those continue to be useful in the future.

With Volary, it's possible for agents to recall specific episodes, but they also provide a critical foundation which other memories can be derived from.

Organisation

Given these pretty different kinds of memory, they have to be organised in a way that allows them to be recalled. The human brain is (mostly) extremely good at this, allowing us to recall a huge amount of information throughout our lives. The equivalent for agents is more structured but ideally retains the same ability to link a memory to a present action based on a small amount of information. This means simple text search or similarity comparisons aren't enough;memories need to be recallable based on many different points of similarity, just as we can recall a place based on a smell or taste.

There is a lot of detail in how memories are best organised, which I'll talk more about in the next article in this series. In the meantime you can discuss memory, recall or anything you like in our Slack community.