Not sure it's explained here but from the graph (fig. 10) it looks like wiki + search beats dreaming, and that dreaming adds only marginal gains to the wiki + search configuration.
I'm assuming search is much cheaper than dreaming (though maybe it depends on the time scale?) so that sounds pretty important.
So far I've turned off automatic memory creation in every harness I've used. Claude Code sometimes creates some anyway, and I probably delete 95% of them. The memories it creates are almost always relevant only to the specific task that it is currently working on. If it's a task I'll need to do again, then yeah, maybe that's useful. But that isn't the case most of the time, so I delete them to avoid cluttering context.
So the main improvement to memories I'd like to see is better judgment on which ones to create.
The parts about "dreaming" make me think of how agent transcripts contain a ton of information, which I think is currently under-utilized. I'll repeat advice I've seen here a few times: ingest your own transcripts into duckdb (claude code will build that pipeline in about 10 minutes) and do some data analysis on them. Some ideas if you don't know where to start: look at tool call failures to see if you can prevent common ones, check out if you have large cache misses and what causes them, look at various efficiency metrics over time (tokens per $, lines of code per tokens, etc.) with main thread vs sub-agents, etc. I started with this sort of things, and ended up following some threads to interesting insights. For example, I found indications that I tended to make Opus 5.5 / Sol 6.1 divide their work into units that were too small. They have the capacity to work on more stuff at once than older models, and I wasn't really taking advantage of that.
> If it's a task I'll need to do again, then yeah, maybe that's useful. But that isn't the case most of the time, so I delete them to avoid cluttering context.
It would be useful if they could scope them properly. Then it would be useful to keep around. But I haven’t seen memory systems be able to do that.
> look at tool call failures to see if you can prevent common ones
I often see tool call failures because the agent invokes the tool with wrong parameter names, e.g. for file creation it provides "file_content", but the tool expects "file_text" instead. My colleague told me to put this in a pre-prompt file, but I don't understand why the models make these kinds of mistakes - the model should have the available tools with their parameter names in context, should it not?
I've also disabled memory in Claude for finding it more annoying than helpful.
However, I have it on for ChatGPT (use it mostly for on the go queries via mobile) and their memory experience is significantly better. I haven't enabled Claude's memory in a long time so things may have changed though.
> So far I've turned off automatic memory creation in every harness I've used.
I also never use memories. I don't see the usefulness above project-specific prompting and a good knowledge of your own codebase. Am I missing something?
> Claude Code sometimes creates some anyway, and I probably delete 95% of them.
The thing you're asking it for might be intent, which is something the technology fundamentally lacks.
Which.. is something that has been discussed very extensively, which makes one wonder why Anthropic still keeps trying to push that square peg through a round hole.
So the main improvement to memories I'd like to see is better judgment on which ones to create.
Well you have good instincts, then! It's probably not obvious on the first read-through, but Figure 4 shows the qualitative strides they've made on exactly this goal.
I started with [proactive post-hoc analysis], and ended up following some threads to interesting insights.
Can definitely echo that experience, glad I'm not alone. I think this actually speaks indirectly to your initial complaint/observation, too: Anthropic designs the [memory/tendency-to-reflect] systems to cover a huge breadth of usecases and preferences and they help a lot compared to [statelessness/never reflecting], but are distractingly inferior to alternatives that the user takes an active part in shaping and using.
I partly say this because I also finally pulled the trigger on deprecating the last ClaudeCode memories on my system in favor of a personal system[1] recently, and the issues I noticed were things a casual user/'vibecoder' might appreciate, but any engineer working fulltime on multiple projects via multiple harnesses would find counterintuitive. E.g.:
- The memories were hidden away and clearly not written for human eyes; useful for low-stakes ledgers, frustrating when the right kind of errant note can derail a longrun 2 weeks later.
- The memories seemed to err towards the side of remembering everything and recording incidental details; useful for remembering the user is gluten-free, but not for tracking spec changes over time.
- The memories could be specific to a subagent or the project or the whole system; an obvious & elegant system for toy examples, but a terribly unweildy one for anyone working at scale over the span of months.
[1]: if curious, a somewhat-outdated version of it is here: https://gitlab.com/doering-ai/libs/fleet TL;DR: Friendship ended with ClaudeCode memory files; my new best friends are `BULLETIN.md` for transitory alerts, `campaigns/[slug]/LEDGER.md` for basic episodic memory, and `reports/[date]-[slug].md` for writeups polished specifically for posterity.
I have a skill called memory-prune whose sole purpose is to take an automatically created memory, assess whether it deserves to live, get stored in code comments, a rule, an AGENTS.md file in the project, the global AGENTS.md/CLAUDE.md file, or (and this is the most often picked option) not at all and deleted.
Hmm, this is kind of like memory consolidation for recall. I wonder if it would be equally/more effective if you kept the memory in place and made automated pointers to the memory items (recall pointers) in the codebase files that inject the memory into agent context if the pointers ever appear in tool results.
I’d be interested if you could share this skill please. Always looking to improve memory management which so often gets unwieldy when left to its own devices
Their progress toward cerulean chart is selling me pretty hard on no memory.
The biggest thing I want to avoid with a long running agent is volatility. Anything that pushes it into one direction or the other that quickly is not desirable in the average case. At least not to me.
In that chart, the "no memory" run didn't make it either. It's not much to go by but my read here is that poorly implemented memory makes things worse, while properly implemented memory makes things better. (...in Pokemon.)
Curiously this seems to apply to humans as well. The "dreaming" is making it question its own assumptions about difficulties it has had in the past. That sounds like a very helpful therapeutic process.
(Not sure of the word dreaming is entirely suitable though: it was cross referencing its notes against the actual history! Very rigorous "dreamer" lol)
Great article -- still have some sections to read, but it's just too much to read. Emotionally, that is.
Part of me feels the pang of regret and shame seeing so many little details I researched and dreamed[1] (ha!) of prototyping ~2.5y ago expressed and validated so cleanly. Obviously A) I never did get the prototype working nor publish the underlying ideas and B) this blog post (paper? ...plog post?) is building on it at a scale and with quality I could never dream of (fig. 3 is especially striking), not to mention all the new ideas mixed in. Still... sharing cause it helps, I think, and because I expect y'all can relate; if you're old enough, it's very similar to the feeling you get when you think about how much Apple stock you could've bought in the 90s.
Anyway, more materially:
1. Pretty interesting to see Opus 5.5, Sonnet 5.5, and Opus 5 tied on the basic memory test (aforementioned fig. 3); the Opus comparison alone seems to imply either a hard plateau or that 5.5's huge improvements were accomplished by making good practices more likely, but the Sonnet detail throws that for a loop... I suppose the plateau explanation isn't contradicted, so maybe that's right (for this benchmark).
2. Fig. 4 is gonna do rounds on Reddit and Twitter, my god. Pretty stellar, intuitive depiction of the Mythos step-change that enabled true agentic engineering for the first time this summer! The highlighting is also just the right amount of facetious; love it.
3. Pretty bummed to see no followup philosophy references. We know you have them, Anthropic nerds! Do not be ashamed of the soft and slippery.
4. I am now reminded that I knew they were trying dreaming at scale, as I'm actually on the waitlist for it! And that backs up my overall takeway, actually: I can't help but feel like they're either A) vastly underestimating the amount of CogSci literature that could help them go beyond the "dreaming is when you think in your offtime" paradigm, or B) not sharing their real learnings.
Today's RSI announcement and the final sentence of this seminal plog post being a pitch for enterprise customers supports the latter, I think. That's ostensibly a good thing from a societal stability perspective, but I can't help but feel left out -- not to mention paralyzed by fear of the known unknown...
That "I never got to finish" feeling is the absolute worst. I've been struggling with it for years, and I feel like understand ya here.
It absolutely sucks you feel that, and I hope you find some comfort soon, however it may come.
If you're looking for a "so what now?" Moment, there's only one thing I can say that I've been proselytizing a bit: share everything. All the time. And get people to look at it and be your hype person; and if you're not getting that in your friend ground or workplace or community, find a new one to add to.
The only way I've found to beat that feeling of "why is it i have the good ideas years ahead but can't do anything with it" is to roll more dice. That's it. These larger companies just essentially have infinitely many die and they're rolling them infinitely faster.
You've done nothing wrong. You're just with some other folks like us that don't have the same die rolling power.
Don't give up on your ideas. Don't stop experimenting. Make your own tools because it's for you, and maybe your dice will come up that you get the other person in the world that sees it and goes "yes!" And you get all the fancy posts and stars and attention and such. Because that feeling of validation - not just popularity, but acceptance and being seen and heard for one's craft - is one everyone seeks and needs in life. It's not a weakness. It's being a human.
Thank you for sharing your experience with internet strangers.
> The only way I've found to beat that feeling of "why is it i have the good ideas years ahead but can't do anything with it" is to roll more dice. That's it. These larger companies just essentially have infinitely many die and they're rolling them infinitely faster.
Many of us have ideas, ideas are cheap. Executing a complex idea well from thought to completion is much more rare.
I've gotta say, that is by far the most wholesome, empathetic reply I've ever received on this forum, and possibly on any forum. So on a personal level, thank you. Really. Especially because lamenting that I could've scooped Anthropic is just undeniably hubristic; my reasoning for not sharing partial results at the time was mostly about AI safety, which is silly in hindsight for betraying the exact same sin.
The (penti-)closing statement is also pretty damn poignant, and very true. I daresay your thoughts on publishing deserve to be published!
Substantively, I absolutely agree in theory -- I even recently set up my Substack with this in mind, despite that being an insane thing to do when frantically job searching! I (/Claude...) just found a brief exchange[1] I had w/ @simonw on this very topic almost exactly two years ago below a heartfelt blog post[2] about someone's personal reasons for struggling with this, even. Reading it now mostly serves to remind me of the roadblocks, though:
1. I was quibbling with his advice to "force" yourself to share by reflex (and the author's framing of the failure to do so as "laziness"), and two more years of reading about, discussing, and managing AD[H]D symptoms have only strengthened my convictions there. Charitably, getting into the habit of publishing is an investment, and my metaconscious/'inner critic' has plenty of quite plausible arguments as to why it wouldn't be worth the squeeze[3]; less charitably, I'd note that the relatable blogger in question only posted twice over the following 12 months.
2. That exchange took place just as A) reasoning models were changing everything, B) artificial articles outnumbered human ones for the first time.[4] In the intervening years we've seen AI turn from niche to vitriolic, and at least a dozen variations of "If you didn't care enough to write it yourself, I don't care enough to read it" posts hit the front page of HN. Obviously we all agree and are talking about human writing/ideating here, but the overall vibe is still undeniably... tense, let's say[5].
I still appreciate your comment and agree, mostly because the anxieties presumably lessen considerably with exposure and practice. In other words, I'd get used to it.
In fact, I just queried the corpus that I ironically have yet to publish the pipeline for, and I've written 6,280 real comments (>50 words of original text) and 1,788 long ones (>100 words) since that day on Reddit, Bluesky, and here. Some of that might be real differences --say, ostensible anonymity on Reddit and lower expectations for mere comments-- but the only factor that accounts for a spread of 6,280-0 is habit and comfort, I think.
As usual (55 times since November 11th, 2024): apologies for the rant, y'all.
This article is a lot of validation for what I am building. I'll definitely be revisiting the article a few times to take in the ideas. Memory systems in AIs are extremely important and imho you can do much better than simply training the base model for memory management.
So after reading the article, I asked a Claude Code session I'm working in at the moment about its memory and whether it needs to "dream" to reorganize itself because I honestly wasn't sure what it was doing about it.
It explained that unlike Chat, each project in Claude Code creates memory files in ~/.claude/projects/[project path]/memory/ with notes about both the project and my impromptu directives or instructions, and a MEMORY.md index file that's loaded for every new session. It told me that my current project - which has been going on for a while - has 61 files in its memory, so it suggested that now in fact would be a good time to clean them up. Perfect.
It gave some examples of things things it has memorized: Apparently I once told it to always capitalize the first letter after a colon. I don't remember doing that, but hey, that's basic English grammar, dammit, so I can see myself getting pissy about it. It had saved that bit of petulance on my part in the memory.
That's great, but it was saved only in the memory folder of that project. This is good to know! I hadn't realized that stuff wasn't saved globally. Or rather, I vaguely remember turning that option off at some point and now I'm realizing that means I need to do some maintenance manually if I don't want to have to constantly remind Claude to capitalize properly, (and to always refer to me as Mr. Big.)
Definitely a good article for me to read right now.
> Apparently I once told it to always capitalize the first letter after a colon. I don't remember doing that, but hey, that's basic English grammar, dammit, so I can see myself getting pissy about it.
It's definitely not correct UK English grammar. TIL that the US uses some very odd colon rules.
Ugg. “dreaming”. More anthropomorphizing we didn’t need. And reading comments which drop the term unironically just makes my skin crawl worse than all the LLM written posts on HN combined.
Such a bullshit article, as much of Anthropic "research" is these days.
> By “agent memory,” we mean knowledge that Claude gains from experience—including interactions with users and tasks it’s been assigned—and that it retains between sessions, saved as context it can read later.
That Claude rarely, if ever, actually "remembers". Since these are just random markdown files randomly injected into the tiny context.
"I'm sorry, I'll make a note of it" and will proceed to write a 15000th note with the same content it's already written dozens of times before
> We compared the memory folders from one run each of Claude Sonnet 3.7 and Claude Fable 5, when each was about 10,000 steps into the game.
What a useless experiment
> That general guidance was enough for Opus 4.7 and Opus 5 to use memory effectively
Define "effectively"
> In this case, prompting Claude to read its memory files before answering—and then to explicitly test its theories—nearly doubled its benchmark score
Then it's not "memory". It's just telling the model to read a text/spec file before implementing something
> Claude doesn’t load all of its auto memory into every session’s context window. It loads only the first 200 lines or 25 KB of MEMORY.md
So, random injection of random things into context that depend on whether the short ToC entries have any relevance on what Claude is doing, and which will be ignored.
And you have to explicitly tell the agent to read specific files otherwise it will ignore them.
Reading such a passionate take as someone who has looked up is a strange experience, I must admit. Kinda like how the Bernard twist in S01 of Westworld felt. I'll try to respond as substantively as I can, regardless:
That Claude rarely, if ever, actually "remembers". Since these are just random markdown files randomly injected into the tiny context.
Complaining about your anecdotal experiences with Claude(Code, hopefully?) isn't a serious criticism, the markdown files are optimized to not be random, and calling 1,000,000 tokens of context "tiny" seems like a plainly unreasonable bar. How much of War and Peace can you hold in your head at once and quote flawlessly, and is it more than 133%?
What a useless experiment
The experiment is testing their fixes to the problem you lamented just one sentence prior.
Define "effectively"
They do exactly that in figure 3, as mentioned at the end of the sentence you're quoting. Opus 5 used that advice to raise its benchmark score by 46.8%, and Opus 4.7 used it for a gain of 82.7%. The specific benchmark looks far from trivial, too: https://github.com/pgasawa/continual-learning-bench/tree/mai...
Then it's not "memory". It's just telling the model to read a text/spec file before implementing something
It fits the colloquial definition, and there is no specific scientific definition in CogSci, CogPsych, Neuro, CogNeuro, NeuroPsych, nor philosophy. There's lots of useful ones offered in different contexts, but they differ considerably.[1] Regardless, it's well accepted industry jargon at this point: it doesn't matter in the same way that it doesn't matter than databases can never truly be "queried" because they can't engage in linguistic discourse.
So, random injection of random things into context that depend on whether the short ToC entries have any relevance on what Claude is doing
Isn't this inevitably, undeniably true of any possible memory system, just from, say, a big-O perspective? If you have to persist every item in a list to decide which one is relevant enough to load into the same pool of memory, there's no point in the second step anymore.
What is this bloated article about?
This article is very plausibly wrong, misguided, or even philosophically bankrupt, but I don't see any support for the claim that its topic is unclear. It's discussing the efficacy of proactive state management during idle periods for longrunning LLM-backed agents.
---
[1]: For example, did you know that modern CogSci is generally convinced that memories are illusions that the brain creates through a process that merely consults ineffable underlying representations (usually), and not an object retrieved from storage? That definition would also contradict Anthropic's wording here, but only in a way that weakens your criticism rather than supports it.
> Reading such a passionate take as someone who has looked up is a strange experience, I must admit.
There's nothing passionate. Calling bullshit for what it is is not passionate, and we need more of calling out.
I would say that you are much more passionate about this and have a prior bias that makes you see something that's not there in Anthropic's marketing materials
> Complaining about your anecdotal experiences with Claude(Code, hopefully?) isn't a serious criticism
Dismissing actual real-life user experience that anyone who's ever dealt with Claude Code as "anecdotal experience" is exactly why these self-congratulatory articles exist. Even if they mention exactly these issues in passing.
> and calling 1,000,000 tokens of context "tiny" seems like a plainly unreasonable bar.
It is both tiny and easily polluted. Its training data will easily override any and all data in the context even early on in the session. And yes, that million token context is still incapable of maintaining and recalling any of the "memories" with any consistency. Which the article itself acknowledges.
> The experiment is testing their fixes to the problem you lamented just one sentence prior.
These experiments are useless because they invent a problem and valiantly find solutions for it never measuring whether it has any actual impact in real life.
So what if new way of writing summaries is marginally better than the old way? It doesn't matter if the only way to make Claude reliably use them is to explicitly tell it to read them?
> It fits the colloquial definition, and there is no specific scientific definition in CogSci, CogPsych, Neuro, CogNeuro, NeuroPsych, nor philosophy
Here are your biases showing. Anthropic calls random markdown files "memories" and you immediately take them at their word.
These are not memories. These are objectively random markdown files whose only existence is assured by random
harness injections and whatever claude retains in its current context from the first 200 lines of an injected ToC.
Yes, it's an accepted industry jargon, but you don't have to behave that it does something else than what it does.
Or that explicitly telling Claude "read a specific markdown file in your memory" is in any way, shape, or form different from "read a specific markdown file in specs directory".
Of course it's the only way to make Claude actually read and act on it.
> It's discussing the efficacy of proactive state management during idle periods for longrunning LLM-backed agents.
No. No it doesn't. It considers "components of agent memory" and focuses solely on "how we can write markdown files", and spends probably less than 1% on "how we can make sure the agent actually uses them". Even the conclusion completely ignores it despite this:
--- start quote ---
Injected memories use up valuable tokens, and an incorrect note can be carried into every session. Loading a note also doesn’t guarantee that Claude acts on it: our documentation on Claude Code warns that longer instruction files reduce adherence [1 000 000 tokens of context is huge lol - troupo]
...
It loads only the first 200 lines or 25 KB of MEMORY.md [but we sure love the sound of our voice deacribing how well it can write the hundreds of tiny files – troupo]
Another day, another Anthropic marketing article trying to deceive people into believing Claude is conscious. Absolutely insane anybody takes this company seriously when they describe "writing notes into an .md file to be re-injected into context later" as ""dreaming"".
Dreaming is being used here as an analogy for a post-hoc conflict resolution process for memory (which is part of the theorized purpose of dreaming in the human brain), rather than used literally. I think it's the best word to describe what it does and it is very important to a good memory system, as memory conflicts are one of the most likely things pulling down their performance in the long run. Not really a misnomer, imo, and i don't think they are trying to use the word to suggest that Claude is conscious.
> post-hoc conflict resolution process for memory (which is part of the theorized purpose of dreaming in the human brain)
That’s not accurate. One leading hypothesis is that the experience of dreaming is a side effect of memory processing and consolidation[1]. Using the name of a side effect to describe the process that produces it is misleading, and at best it goes well beyond what’s actually known.
[Edit: and the side effect in question, dreaming, is closely tied to consciousness in humans. Using an “analogy” with those connotations is an unfortunate mistake.]
[1] “The process of memory reactivation and consolidation in the sleeping brain appears to influence conscious experience during sleep, contributing to dream content recalled on awakening.” — https://pmc.ncbi.nlm.nih.gov/articles/PMC4704085/
Not sure it's explained here but from the graph (fig. 10) it looks like wiki + search beats dreaming, and that dreaming adds only marginal gains to the wiki + search configuration.
I'm assuming search is much cheaper than dreaming (though maybe it depends on the time scale?) so that sounds pretty important.
So far I've turned off automatic memory creation in every harness I've used. Claude Code sometimes creates some anyway, and I probably delete 95% of them. The memories it creates are almost always relevant only to the specific task that it is currently working on. If it's a task I'll need to do again, then yeah, maybe that's useful. But that isn't the case most of the time, so I delete them to avoid cluttering context.
So the main improvement to memories I'd like to see is better judgment on which ones to create.
The parts about "dreaming" make me think of how agent transcripts contain a ton of information, which I think is currently under-utilized. I'll repeat advice I've seen here a few times: ingest your own transcripts into duckdb (claude code will build that pipeline in about 10 minutes) and do some data analysis on them. Some ideas if you don't know where to start: look at tool call failures to see if you can prevent common ones, check out if you have large cache misses and what causes them, look at various efficiency metrics over time (tokens per $, lines of code per tokens, etc.) with main thread vs sub-agents, etc. I started with this sort of things, and ended up following some threads to interesting insights. For example, I found indications that I tended to make Opus 5.5 / Sol 6.1 divide their work into units that were too small. They have the capacity to work on more stuff at once than older models, and I wasn't really taking advantage of that.
I give the same advice constantly. Turning off memory has been one of the best ways I've found to reduce "WTF per minute" when using agents.
> If it's a task I'll need to do again, then yeah, maybe that's useful. But that isn't the case most of the time, so I delete them to avoid cluttering context.
It would be useful if they could scope them properly. Then it would be useful to keep around. But I haven’t seen memory systems be able to do that.
> look at tool call failures to see if you can prevent common ones
I often see tool call failures because the agent invokes the tool with wrong parameter names, e.g. for file creation it provides "file_content", but the tool expects "file_text" instead. My colleague told me to put this in a pre-prompt file, but I don't understand why the models make these kinds of mistakes - the model should have the available tools with their parameter names in context, should it not?
I've also disabled memory in Claude for finding it more annoying than helpful.
However, I have it on for ChatGPT (use it mostly for on the go queries via mobile) and their memory experience is significantly better. I haven't enabled Claude's memory in a long time so things may have changed though.
And somewhat open source.
>I found indications that I tended to make Opus 5.5 / Sol 6.1 divide their work into units that were too small.
How did you determine this?
Do you mean that you were giving them tasks that were too small or you were telling them to split the tasks up or something?
> So far I've turned off automatic memory creation in every harness I've used.
I also never use memories. I don't see the usefulness above project-specific prompting and a good knowledge of your own codebase. Am I missing something?
> Claude Code sometimes creates some anyway, and I probably delete 95% of them.
The thing you're asking it for might be intent, which is something the technology fundamentally lacks.
Which.. is something that has been discussed very extensively, which makes one wonder why Anthropic still keeps trying to push that square peg through a round hole.
I just periodically ask it to curate all memory, and ask me questions to do so.
And in the Claude Desktop UI I manually curate the memories.
Works well enough, I don’t like the concept of dreaming as it quickly leads to memory bloat and/or outdated stuff.
Well you have good instincts, then! It's probably not obvious on the first read-through, but Figure 4 shows the qualitative strides they've made on exactly this goal.
Can definitely echo that experience, glad I'm not alone. I think this actually speaks indirectly to your initial complaint/observation, too: Anthropic designs the [memory/tendency-to-reflect] systems to cover a huge breadth of usecases and preferences and they help a lot compared to [statelessness/never reflecting], but are distractingly inferior to alternatives that the user takes an active part in shaping and using.
I partly say this because I also finally pulled the trigger on deprecating the last ClaudeCode memories on my system in favor of a personal system[1] recently, and the issues I noticed were things a casual user/'vibecoder' might appreciate, but any engineer working fulltime on multiple projects via multiple harnesses would find counterintuitive. E.g.:
- The memories were hidden away and clearly not written for human eyes; useful for low-stakes ledgers, frustrating when the right kind of errant note can derail a longrun 2 weeks later.
- The memories seemed to err towards the side of remembering everything and recording incidental details; useful for remembering the user is gluten-free, but not for tracking spec changes over time.
- The memories could be specific to a subagent or the project or the whole system; an obvious & elegant system for toy examples, but a terribly unweildy one for anyone working at scale over the span of months.
[1]: if curious, a somewhat-outdated version of it is here: https://gitlab.com/doering-ai/libs/fleet TL;DR: Friendship ended with ClaudeCode memory files; my new best friends are `BULLETIN.md` for transitory alerts, `campaigns/[slug]/LEDGER.md` for basic episodic memory, and `reports/[date]-[slug].md` for writeups polished specifically for posterity.
I have a skill called memory-prune whose sole purpose is to take an automatically created memory, assess whether it deserves to live, get stored in code comments, a rule, an AGENTS.md file in the project, the global AGENTS.md/CLAUDE.md file, or (and this is the most often picked option) not at all and deleted.
Hmm, this is kind of like memory consolidation for recall. I wonder if it would be equally/more effective if you kept the memory in place and made automated pointers to the memory items (recall pointers) in the codebase files that inject the memory into agent context if the pointers ever appear in tool results.
I’d be interested if you could share this skill please. Always looking to improve memory management which so often gets unwieldy when left to its own devices
Their progress toward cerulean chart is selling me pretty hard on no memory.
The biggest thing I want to avoid with a long running agent is volatility. Anything that pushes it into one direction or the other that quickly is not desirable in the average case. At least not to me.
In that chart, the "no memory" run didn't make it either. It's not much to go by but my read here is that poorly implemented memory makes things worse, while properly implemented memory makes things better. (...in Pokemon.)
Curiously this seems to apply to humans as well. The "dreaming" is making it question its own assumptions about difficulties it has had in the past. That sounds like a very helpful therapeutic process.
(Not sure of the word dreaming is entirely suitable though: it was cross referencing its notes against the actual history! Very rigorous "dreamer" lol)
Great article -- still have some sections to read, but it's just too much to read. Emotionally, that is.
Part of me feels the pang of regret and shame seeing so many little details I researched and dreamed[1] (ha!) of prototyping ~2.5y ago expressed and validated so cleanly. Obviously A) I never did get the prototype working nor publish the underlying ideas and B) this blog post (paper? ...plog post?) is building on it at a scale and with quality I could never dream of (fig. 3 is especially striking), not to mention all the new ideas mixed in. Still... sharing cause it helps, I think, and because I expect y'all can relate; if you're old enough, it's very similar to the feeling you get when you think about how much Apple stock you could've bought in the 90s.
Anyway, more materially:
1. Pretty interesting to see Opus 5.5, Sonnet 5.5, and Opus 5 tied on the basic memory test (aforementioned fig. 3); the Opus comparison alone seems to imply either a hard plateau or that 5.5's huge improvements were accomplished by making good practices more likely, but the Sonnet detail throws that for a loop... I suppose the plateau explanation isn't contradicted, so maybe that's right (for this benchmark).
2. Fig. 4 is gonna do rounds on Reddit and Twitter, my god. Pretty stellar, intuitive depiction of the Mythos step-change that enabled true agentic engineering for the first time this summer! The highlighting is also just the right amount of facetious; love it.
3. Pretty bummed to see no followup philosophy references. We know you have them, Anthropic nerds! Do not be ashamed of the soft and slippery.
4. I am now reminded that I knew they were trying dreaming at scale, as I'm actually on the waitlist for it! And that backs up my overall takeway, actually: I can't help but feel like they're either A) vastly underestimating the amount of CogSci literature that could help them go beyond the "dreaming is when you think in your offtime" paradigm, or B) not sharing their real learnings.
Today's RSI announcement and the final sentence of this seminal plog post being a pitch for enterprise customers supports the latter, I think. That's ostensibly a good thing from a societal stability perspective, but I can't help but feel left out -- not to mention paralyzed by fear of the known unknown...
I'm still reading it, but to your first part:
That "I never got to finish" feeling is the absolute worst. I've been struggling with it for years, and I feel like understand ya here.
It absolutely sucks you feel that, and I hope you find some comfort soon, however it may come.
If you're looking for a "so what now?" Moment, there's only one thing I can say that I've been proselytizing a bit: share everything. All the time. And get people to look at it and be your hype person; and if you're not getting that in your friend ground or workplace or community, find a new one to add to.
The only way I've found to beat that feeling of "why is it i have the good ideas years ahead but can't do anything with it" is to roll more dice. That's it. These larger companies just essentially have infinitely many die and they're rolling them infinitely faster.
You've done nothing wrong. You're just with some other folks like us that don't have the same die rolling power.
Don't give up on your ideas. Don't stop experimenting. Make your own tools because it's for you, and maybe your dice will come up that you get the other person in the world that sees it and goes "yes!" And you get all the fancy posts and stars and attention and such. Because that feeling of validation - not just popularity, but acceptance and being seen and heard for one's craft - is one everyone seeks and needs in life. It's not a weakness. It's being a human.
Thank you for sharing your experience with internet strangers.
> The only way I've found to beat that feeling of "why is it i have the good ideas years ahead but can't do anything with it" is to roll more dice. That's it. These larger companies just essentially have infinitely many die and they're rolling them infinitely faster.
Many of us have ideas, ideas are cheap. Executing a complex idea well from thought to completion is much more rare.
2 replies →
I've gotta say, that is by far the most wholesome, empathetic reply I've ever received on this forum, and possibly on any forum. So on a personal level, thank you. Really. Especially because lamenting that I could've scooped Anthropic is just undeniably hubristic; my reasoning for not sharing partial results at the time was mostly about AI safety, which is silly in hindsight for betraying the exact same sin.
The (penti-)closing statement is also pretty damn poignant, and very true. I daresay your thoughts on publishing deserve to be published!
Substantively, I absolutely agree in theory -- I even recently set up my Substack with this in mind, despite that being an insane thing to do when frantically job searching! I (/Claude...) just found a brief exchange[1] I had w/ @simonw on this very topic almost exactly two years ago below a heartfelt blog post[2] about someone's personal reasons for struggling with this, even. Reading it now mostly serves to remind me of the roadblocks, though:
1. I was quibbling with his advice to "force" yourself to share by reflex (and the author's framing of the failure to do so as "laziness"), and two more years of reading about, discussing, and managing AD[H]D symptoms have only strengthened my convictions there. Charitably, getting into the habit of publishing is an investment, and my metaconscious/'inner critic' has plenty of quite plausible arguments as to why it wouldn't be worth the squeeze[3]; less charitably, I'd note that the relatable blogger in question only posted twice over the following 12 months.
2. That exchange took place just as A) reasoning models were changing everything, B) artificial articles outnumbered human ones for the first time.[4] In the intervening years we've seen AI turn from niche to vitriolic, and at least a dozen variations of "If you didn't care enough to write it yourself, I don't care enough to read it" posts hit the front page of HN. Obviously we all agree and are talking about human writing/ideating here, but the overall vibe is still undeniably... tense, let's say[5].
I still appreciate your comment and agree, mostly because the anxieties presumably lessen considerably with exposure and practice. In other words, I'd get used to it.
In fact, I just queried the corpus that I ironically have yet to publish the pipeline for, and I've written 6,280 real comments (>50 words of original text) and 1,788 long ones (>100 words) since that day on Reddit, Bluesky, and here. Some of that might be real differences --say, ostensible anonymity on Reddit and lower expectations for mere comments-- but the only factor that accounts for a spread of 6,280-0 is habit and comfort, I think.
As usual (55 times since November 11th, 2024): apologies for the rant, y'all.
---
[1]: https://arxiv.org/abs/2606.12073
What was today's RSI announcement?
I struggle to see anything "seminal" in the article: https://news.ycombinator.com/item?id=50402734
This article is a lot of validation for what I am building. I'll definitely be revisiting the article a few times to take in the ideas. Memory systems in AIs are extremely important and imho you can do much better than simply training the base model for memory management.
So after reading the article, I asked a Claude Code session I'm working in at the moment about its memory and whether it needs to "dream" to reorganize itself because I honestly wasn't sure what it was doing about it.
It explained that unlike Chat, each project in Claude Code creates memory files in ~/.claude/projects/[project path]/memory/ with notes about both the project and my impromptu directives or instructions, and a MEMORY.md index file that's loaded for every new session. It told me that my current project - which has been going on for a while - has 61 files in its memory, so it suggested that now in fact would be a good time to clean them up. Perfect.
It gave some examples of things things it has memorized: Apparently I once told it to always capitalize the first letter after a colon. I don't remember doing that, but hey, that's basic English grammar, dammit, so I can see myself getting pissy about it. It had saved that bit of petulance on my part in the memory.
That's great, but it was saved only in the memory folder of that project. This is good to know! I hadn't realized that stuff wasn't saved globally. Or rather, I vaguely remember turning that option off at some point and now I'm realizing that means I need to do some maintenance manually if I don't want to have to constantly remind Claude to capitalize properly, (and to always refer to me as Mr. Big.)
Definitely a good article for me to read right now.
> Apparently I once told it to always capitalize the first letter after a colon. I don't remember doing that, but hey, that's basic English grammar, dammit, so I can see myself getting pissy about it.
It's definitely not correct UK English grammar. TIL that the US uses some very odd colon rules.
yeah this is not a rule in the US either :(
[flagged]
Ugg. “dreaming”. More anthropomorphizing we didn’t need. And reading comments which drop the term unironically just makes my skin crawl worse than all the LLM written posts on HN combined.
They're not even using holographic projections and lower dimensional manifolds ..
Looks like a promising way to bind customers.
Shh... tiny agents is dreaming...
Such a bullshit article, as much of Anthropic "research" is these days.
> By “agent memory,” we mean knowledge that Claude gains from experience—including interactions with users and tasks it’s been assigned—and that it retains between sessions, saved as context it can read later.
That Claude rarely, if ever, actually "remembers". Since these are just random markdown files randomly injected into the tiny context.
"I'm sorry, I'll make a note of it" and will proceed to write a 15000th note with the same content it's already written dozens of times before
> We compared the memory folders from one run each of Claude Sonnet 3.7 and Claude Fable 5, when each was about 10,000 steps into the game.
What a useless experiment
> That general guidance was enough for Opus 4.7 and Opus 5 to use memory effectively
Define "effectively"
> In this case, prompting Claude to read its memory files before answering—and then to explicitly test its theories—nearly doubled its benchmark score
Then it's not "memory". It's just telling the model to read a text/spec file before implementing something
> Claude doesn’t load all of its auto memory into every session’s context window. It loads only the first 200 lines or 25 KB of MEMORY.md
So, random injection of random things into context that depend on whether the short ToC entries have any relevance on what Claude is doing, and which will be ignored.
And you have to explicitly tell the agent to read specific files otherwise it will ignore them.
What is this bloated article about?
Exactly my experience with claude-code. It keeps apologising for not using the memory and keep storing the same memory again and again.
Reading such a passionate take as someone who has looked up is a strange experience, I must admit. Kinda like how the Bernard twist in S01 of Westworld felt. I'll try to respond as substantively as I can, regardless:
Complaining about your anecdotal experiences with Claude(Code, hopefully?) isn't a serious criticism, the markdown files are optimized to not be random, and calling 1,000,000 tokens of context "tiny" seems like a plainly unreasonable bar. How much of War and Peace can you hold in your head at once and quote flawlessly, and is it more than 133%?
The experiment is testing their fixes to the problem you lamented just one sentence prior.
They do exactly that in figure 3, as mentioned at the end of the sentence you're quoting. Opus 5 used that advice to raise its benchmark score by 46.8%, and Opus 4.7 used it for a gain of 82.7%. The specific benchmark looks far from trivial, too: https://github.com/pgasawa/continual-learning-bench/tree/mai...
It fits the colloquial definition, and there is no specific scientific definition in CogSci, CogPsych, Neuro, CogNeuro, NeuroPsych, nor philosophy. There's lots of useful ones offered in different contexts, but they differ considerably.[1] Regardless, it's well accepted industry jargon at this point: it doesn't matter in the same way that it doesn't matter than databases can never truly be "queried" because they can't engage in linguistic discourse.
Isn't this inevitably, undeniably true of any possible memory system, just from, say, a big-O perspective? If you have to persist every item in a list to decide which one is relevant enough to load into the same pool of memory, there's no point in the second step anymore.
This article is very plausibly wrong, misguided, or even philosophically bankrupt, but I don't see any support for the claim that its topic is unclear. It's discussing the efficacy of proactive state management during idle periods for longrunning LLM-backed agents.
---
[1]: For example, did you know that modern CogSci is generally convinced that memories are illusions that the brain creates through a process that merely consults ineffable underlying representations (usually), and not an object retrieved from storage? That definition would also contradict Anthropic's wording here, but only in a way that weakens your criticism rather than supports it.
> Reading such a passionate take as someone who has looked up is a strange experience, I must admit.
There's nothing passionate. Calling bullshit for what it is is not passionate, and we need more of calling out.
I would say that you are much more passionate about this and have a prior bias that makes you see something that's not there in Anthropic's marketing materials
> Complaining about your anecdotal experiences with Claude(Code, hopefully?) isn't a serious criticism
Dismissing actual real-life user experience that anyone who's ever dealt with Claude Code as "anecdotal experience" is exactly why these self-congratulatory articles exist. Even if they mention exactly these issues in passing.
> and calling 1,000,000 tokens of context "tiny" seems like a plainly unreasonable bar.
It is both tiny and easily polluted. Its training data will easily override any and all data in the context even early on in the session. And yes, that million token context is still incapable of maintaining and recalling any of the "memories" with any consistency. Which the article itself acknowledges.
> The experiment is testing their fixes to the problem you lamented just one sentence prior.
These experiments are useless because they invent a problem and valiantly find solutions for it never measuring whether it has any actual impact in real life.
So what if new way of writing summaries is marginally better than the old way? It doesn't matter if the only way to make Claude reliably use them is to explicitly tell it to read them?
> It fits the colloquial definition, and there is no specific scientific definition in CogSci, CogPsych, Neuro, CogNeuro, NeuroPsych, nor philosophy
Here are your biases showing. Anthropic calls random markdown files "memories" and you immediately take them at their word.
These are not memories. These are objectively random markdown files whose only existence is assured by random harness injections and whatever claude retains in its current context from the first 200 lines of an injected ToC.
Yes, it's an accepted industry jargon, but you don't have to behave that it does something else than what it does.
Or that explicitly telling Claude "read a specific markdown file in your memory" is in any way, shape, or form different from "read a specific markdown file in specs directory".
Of course it's the only way to make Claude actually read and act on it.
> It's discussing the efficacy of proactive state management during idle periods for longrunning LLM-backed agents.
No. No it doesn't. It considers "components of agent memory" and focuses solely on "how we can write markdown files", and spends probably less than 1% on "how we can make sure the agent actually uses them". Even the conclusion completely ignores it despite this:
--- start quote ---
Injected memories use up valuable tokens, and an incorrect note can be carried into every session. Loading a note also doesn’t guarantee that Claude acts on it: our documentation on Claude Code warns that longer instruction files reduce adherence [1 000 000 tokens of context is huge lol - troupo]
...
It loads only the first 200 lines or 25 KB of MEMORY.md [but we sure love the sound of our voice deacribing how well it can write the hundreds of tiny files – troupo]
--- end quote ---
[flagged]
Another day, another Anthropic marketing article trying to deceive people into believing Claude is conscious. Absolutely insane anybody takes this company seriously when they describe "writing notes into an .md file to be re-injected into context later" as ""dreaming"".
Dreaming is being used here as an analogy for a post-hoc conflict resolution process for memory (which is part of the theorized purpose of dreaming in the human brain), rather than used literally. I think it's the best word to describe what it does and it is very important to a good memory system, as memory conflicts are one of the most likely things pulling down their performance in the long run. Not really a misnomer, imo, and i don't think they are trying to use the word to suggest that Claude is conscious.
> post-hoc conflict resolution process for memory (which is part of the theorized purpose of dreaming in the human brain)
That’s not accurate. One leading hypothesis is that the experience of dreaming is a side effect of memory processing and consolidation[1]. Using the name of a side effect to describe the process that produces it is misleading, and at best it goes well beyond what’s actually known.
[Edit: and the side effect in question, dreaming, is closely tied to consciousness in humans. Using an “analogy” with those connotations is an unfortunate mistake.]
[1] “The process of memory reactivation and consolidation in the sleeping brain appears to influence conscious experience during sleep, contributing to dream content recalled on awakening.” — https://pmc.ncbi.nlm.nih.gov/articles/PMC4704085/
It is incredibly unscientific. Superstitious, in fact.
I wouldn’t be particularly surprised if Anthropic were to reconstitute itself as a religion.
Think of the tax benefits!