Key points
- Anthropic's Pokémon Red test: an older Claude model made 413 note files and won one badge; a newer one kept few files and won four.
- Anthropic's legal test: a Claude-built wiki scored 87 percent versus 73 percent for search alone on questions spanning multiple documents.
- An arXiv paper finds deleting facts labeled stale can cut accuracy when the labeling errs, and suggests keeping memories by default without validation.
- Anthropic's Dreaming, launched in May as a research preview, reviews up to 100 past sessions and leaves the original memory store untouched.
Why it matters for marketers: If you build or buy agents that remember brand rules, campaigns or customer context, memory design matters: curated notes plus review beat hoarding, and automatic deletion carries its own risk.
Anthropic published research on how Claude keeps knowledge across sessions by writing notes to files it can read later. The company says the hard part is selection: every note is a bet on what will matter. In its Pokémon Red test, an older Sonnet model over-recorded and finished with one badge, while a newer model kept a few files of plans and tested rules and earned four. Anthropic's comparison covers a single playthrough per model, and all tests come from the company.
Anthropic also describes Dreaming, a process that reviews earlier session transcripts during idle time and checks notes against them. In one case it caught a wrong note that had trapped Claude three times. Anthropic also reports Claude built a linked wiki from about 350,000 internal documents, which beat search alone on multi-document legal questions but only matched it on simple lookups. The company notes costs: extra compute, delayed results and ongoing upkeep.
A separate arXiv paper by Samarth Vinayaka looks at the opposite risk: memory systems that delete facts labeled stale. The author reports that when the labeling is wrong, deleting can hurt accuracy more than keeping bad information. The paper recommends validating the risk on matched data, or defaulting to keeping memories when that is not possible.
For marketing teams, this bears on agents that carry brand guidelines, campaign history and customer details between sessions. Ask vendors what gets saved, who can correct it, and whether old entries are deleted or merely reorganized. Anthropic also warns against loading too many memories into every session, since a bad note then spreads to all of them. Watch for independent tests beyond these vendor and single-author results.
Sources
This brief is our own summary. Read the full reporting at the source.
