Great article -- still have some sections to read, but it's just too much to read. Emotionally, that is.
Part of me feels the pang of regret and shame seeing so many little details I researched and dreamed[1] (ha!) of prototyping ~2.5y ago expressed and validated so cleanly. Obviously A) I never did get the prototype working nor publish the underlying ideas and B) this blog post (paper? ...plog post?) is building on it at a scale and with quality I could never dream of (fig. 3 is especially striking), not to mention all the new ideas mixed in. Still... sharing cause it helps, I think, and because I expect y'all can relate; if you're old enough, it's very similar to the feeling you get when you think about how much Apple stock you could've bought in the 90s.
Anyway, more materially:
1. Pretty interesting to see Opus 5.5, Sonnet 5.5, and Opus 5 tied on the basic memory test (aforementioned fig. 3); the Opus comparison alone seems to imply either a hard plateau or that 5.5's huge improvements were accomplished by making good practices more likely, but the Sonnet detail throws that for a loop... I suppose the plateau explanation isn't contradicted, so maybe that's right (for this benchmark).
2. Fig. 4 is gonna do rounds on Reddit and Twitter, my god. Pretty stellar, intuitive depiction of the Mythos step-change that enabled true agentic engineering for the first time this summer! The highlighting is also just the right amount of facetious; love it.
3. Pretty bummed to see no followup philosophy references. We know you have them, Anthropic nerds! Do not be ashamed of the soft and slippery.
4. I am now reminded that I knew they were trying dreaming at scale, as I'm actually on the waitlist for it! And that backs up my overall takeway, actually: I can't help but feel like they're either A) vastly underestimating the amount of CogSci literature that could help them go beyond the "dreaming is when you think in your offtime" paradigm, or B) not sharing their real learnings.
Today's RSI announcement and the final sentence of this seminal plog post being a pitch for enterprise customers supports the latter, I think. That's ostensibly a good thing from a societal stability perspective, but I can't help but feel left out -- not to mention paralyzed by fear of the known unknown...
That "I never got to finish" feeling is the absolute worst. I've been struggling with it for years, and I feel like understand ya here.
It absolutely sucks you feel that, and I hope you find some comfort soon, however it may come.
If you're looking for a "so what now?" Moment, there's only one thing I can say that I've been proselytizing a bit: share everything. All the time. And get people to look at it and be your hype person; and if you're not getting that in your friend ground or workplace or community, find a new one to add to.
The only way I've found to beat that feeling of "why is it i have the good ideas years ahead but can't do anything with it" is to roll more dice. That's it. These larger companies just essentially have infinitely many die and they're rolling them infinitely faster.
You've done nothing wrong. You're just with some other folks like us that don't have the same die rolling power.
Don't give up on your ideas. Don't stop experimenting. Make your own tools because it's for you, and maybe your dice will come up that you get the other person in the world that sees it and goes "yes!" And you get all the fancy posts and stars and attention and such. Because that feeling of validation - not just popularity, but acceptance and being seen and heard for one's craft - is one everyone seeks and needs in life. It's not a weakness. It's being a human.
Thank you for sharing your experience with internet strangers.
Such a bullshit article, as much of Anthropic "research" is these days.
> By “agent memory,” we mean knowledge that Claude gains from experience—including interactions with users and tasks it’s been assigned—and that it retains between sessions, saved as context it can read later.
That Claude rarely, if ever, actually "remembers". Since these are just random markdown files randomly injected into the tiny context.
"I'm sorry, I'll make a note of it" and will proceed to write a 15000th note with the same content it's already written dozens of times before
> We compared the memory folders from one run each of Claude Sonnet 3.7 and Claude Fable 5, when each was about 10,000 steps into the game.
What a useless experiment
> That general guidance was enough for Opus 4.7 and Opus 5 to use memory effectively
Define "effectively"
> In this case, prompting Claude to read its memory files before answering—and then to explicitly test its theories—nearly doubled its benchmark score
Then it's not "memory". It's just telling the model to read a text/spec file before implementing something
> Claude doesn’t load all of its auto memory into every session’s context window. It loads only the first 200 lines or 25 KB of MEMORY.md
So, random injection of random things into context that depend on whether the short ToC entries have any televance on what Claude is doing, and which will be ignored.
And you have to explicitly tell the agemt to read specific files otherwise it will ignore them.
Part of me feels the pang of regret and shame seeing so many little details I researched and dreamed[1] (ha!) of prototyping ~2.5y ago expressed and validated so cleanly. Obviously A) I never did get the prototype working nor publish the underlying ideas and B) this blog post (paper? ...plog post?) is building on it at a scale and with quality I could never dream of (fig. 3 is especially striking), not to mention all the new ideas mixed in. Still... sharing cause it helps, I think, and because I expect y'all can relate; if you're old enough, it's very similar to the feeling you get when you think about how much Apple stock you could've bought in the 90s.
Anyway, more materially:
1. Pretty interesting to see Opus 5.5, Sonnet 5.5, and Opus 5 tied on the basic memory test (aforementioned fig. 3); the Opus comparison alone seems to imply either a hard plateau or that 5.5's huge improvements were accomplished by making good practices more likely, but the Sonnet detail throws that for a loop... I suppose the plateau explanation isn't contradicted, so maybe that's right (for this benchmark).
2. Fig. 4 is gonna do rounds on Reddit and Twitter, my god. Pretty stellar, intuitive depiction of the Mythos step-change that enabled true agentic engineering for the first time this summer! The highlighting is also just the right amount of facetious; love it.
3. Pretty bummed to see no followup philosophy references. We know you have them, Anthropic nerds! Do not be ashamed of the soft and slippery.
4. I am now reminded that I knew they were trying dreaming at scale, as I'm actually on the waitlist for it! And that backs up my overall takeway, actually: I can't help but feel like they're either A) vastly underestimating the amount of CogSci literature that could help them go beyond the "dreaming is when you think in your offtime" paradigm, or B) not sharing their real learnings.
Today's RSI announcement and the final sentence of this seminal plog post being a pitch for enterprise customers supports the latter, I think. That's ostensibly a good thing from a societal stability perspective, but I can't help but feel left out -- not to mention paralyzed by fear of the known unknown...
That "I never got to finish" feeling is the absolute worst. I've been struggling with it for years, and I feel like understand ya here.
It absolutely sucks you feel that, and I hope you find some comfort soon, however it may come.
If you're looking for a "so what now?" Moment, there's only one thing I can say that I've been proselytizing a bit: share everything. All the time. And get people to look at it and be your hype person; and if you're not getting that in your friend ground or workplace or community, find a new one to add to.
The only way I've found to beat that feeling of "why is it i have the good ideas years ahead but can't do anything with it" is to roll more dice. That's it. These larger companies just essentially have infinitely many die and they're rolling them infinitely faster.
You've done nothing wrong. You're just with some other folks like us that don't have the same die rolling power.
Don't give up on your ideas. Don't stop experimenting. Make your own tools because it's for you, and maybe your dice will come up that you get the other person in the world that sees it and goes "yes!" And you get all the fancy posts and stars and attention and such. Because that feeling of validation - not just popularity, but acceptance and being seen and heard for one's craft - is one everyone seeks and needs in life. It's not a weakness. It's being a human.
Thank you for sharing your experience with internet strangers.
> By “agent memory,” we mean knowledge that Claude gains from experience—including interactions with users and tasks it’s been assigned—and that it retains between sessions, saved as context it can read later.
That Claude rarely, if ever, actually "remembers". Since these are just random markdown files randomly injected into the tiny context.
"I'm sorry, I'll make a note of it" and will proceed to write a 15000th note with the same content it's already written dozens of times before
> We compared the memory folders from one run each of Claude Sonnet 3.7 and Claude Fable 5, when each was about 10,000 steps into the game.
What a useless experiment
> That general guidance was enough for Opus 4.7 and Opus 5 to use memory effectively
Define "effectively"
> In this case, prompting Claude to read its memory files before answering—and then to explicitly test its theories—nearly doubled its benchmark score
Then it's not "memory". It's just telling the model to read a text/spec file before implementing something
> Claude doesn’t load all of its auto memory into every session’s context window. It loads only the first 200 lines or 25 KB of MEMORY.md
So, random injection of random things into context that depend on whether the short ToC entries have any televance on what Claude is doing, and which will be ignored.
And you have to explicitly tell the agemt to read specific files otherwise it will ignore them.
What is this bloated article about?