One Line of Code, 41% Better Memory: When One AI Agent Optimizes Another

I used Claude Code (Opus 4.6) to systematically optimize Lerim’s memory agents — an AI agent improving another AI agent’s prompts, tool descriptions, and DSPy signatures through 24 automated experiments across two rounds. Here’s what happened.

Written March 31, 2026. Epilogue added April 9, 2026. This is a field report from a specific experiment on a specific snapshot of Lerim. The codebase has moved on significantly since — the judge, the evals, the memory architecture, and the agents themselves have all been rebuilt to be simpler, in large part because of what this experiment taught me. The numbers and lessons below are real, but don’t try to reproduce the specific code changes — they live in an older commit. For what changed and why, jump to What came next.

Introduction

Coding agents forget everything between sessions. The architectural decision you explained to Claude yesterday? Gone. The debugging workaround Cursor discovered last week? Lost.

Lerim fixes this. It watches coding sessions, extracts decisions and learnings, and makes those memories available across every future session with any agent. Under the hood, it’s an agentic system with several tools that manages the memory for you in the background.

But here’s the thing that kept me up at night: how good is the extraction, really? How accurate is the dedup? I knew there was room to improve — I just didn’t know where.

So I set up an experiment: I pointed Claude Code (Opus 4.6) at Lerim’s codebase, gave it an eval harness and a golden dataset, and told it to optimize. An AI agent improving another AI agent’s code, measuring its own impact, keeping what works, reverting what doesn’t.

The result: 41% improvement in composite quality score from 14 experiments over one evening. And the single biggest win was changing one line of code.

The System Being Optimized

Lerim’s architecture has several layers, each with tunable text that an optimizer can modify:

The optimization surface includes: system prompts that guide the agent, DSPy signature docstrings that instruct the extraction LLM, tool descriptions that influence tool selection, Pydantic schema field descriptions that shape output format, the harness, and post-extraction filters that gate quality.

That’s a lot of knobs. The question is: which ones actually matter?

Agent Optimizing Agent

This is where it gets interesting. I adapted Karpathy’s AutoResearch pattern — but instead of an AI agent optimizing a training script, I had Claude Code optimizing Lerim’s agents (harness and the prompts and tools). An AI agent reading another AI agent’s prompts and harness and tools, modifying them, measuring the impact, and deciding whether to keep or revert.

The setup follows AutoResearch’s 3-file pattern:

1. program.md — I wrote the directives: which files Claude Code can modify, what the eval metric is, what constraints to follow (“don’t overfit to the golden cases”)
2. Modifiable code — Lerim’s prompts, tool descriptions, DSPy signatures, schema definitions, and the whole harness. Claude Code edits these.
3. Immutable evaluator — A composite score (extraction quality, search relevance, dedup accuracy, maintain precision) measured against 15 golden test cases. Claude Code cannot touch the evaluator.

The loop runs ~30 minutes per experiment: Claude Code modifies a prompt or signature, the evaluator runs Lerim’s full pipeline against golden cases, scores the output, and Claude Code decides whether the change improved things.

The evaluator combines LLM-based quality assessment (is each extracted memory atomic and actionable?) with deterministic metrics (does the extracted title match golden data? does dedup correctly identify duplicates?). The composite score weights extraction quality, search relevance, dedup accuracy, and maintain precision — a single number that tells the optimizer whether a change helped or hurt.

This is reminiscent of what MiniMax demonstrated with their M2.7 self-evolution work — where the model participated in its own RL research workflow. The difference here is cross-model: Claude Opus 4.6 optimizes a system that runs on MiniMax M2.5. The optimizer and the optimized are different models from different companies, which means the improvements are genuine capability gains, not self-reinforcing biases.

Two Levels of Evaluation

I measured improvements at two levels to make sure component-level gains hold up in practice:

Component-level — Each dimension tested independently against golden cases. Fast (~15 min per experiment), used for the optimization loop. This is where the 14 experiments ran.

End-to-end (lifecycle) — Sequential syncs of 3 sessions with a maintain cycle afterward. Tests how memories accumulate, how dedup works across sessions, and whether maintenance decisions improve. Slower (~7 min), used to validate that component gains translate to real-world improvement.

Round 1: Component-Level Results

I ran 14 experiments over 4.5 hours. Seven were kept, seven were discarded.

The dedup accuracy improvement is the most dramatic: from 0.28 to 0.72. Before optimization, the agent classified most candidates as “add” even when nearly identical memories already existed. After optimization, it correctly identifies duplicates and updates existing memories instead.

End-to-End Validation

After the component-level optimization, I ran the full lifecycle eval on both the original and optimized code to verify the gains hold up end-to-end:

The biggest surprise: maintain improved by 29% even though I never directly optimized the maintain prompt. Better extraction quality (from ChainOfThought) produced higher-quality memories, which the maintain judge scored more favorably. Component-level optimization cascaded into end-to-end improvement.

What Claude Code Changed (and What It Reverted)

Half of the experiments failed. Here’s what I learned from both the wins and the losses:

The biggest win: one line of code. Experiment 010 changed dspy.Predict(MemoryExtractSignature) to dspy.ChainOfThought(MemoryExtractSignature). That’s it. One import swap. This gives the extraction LLM a reasoning scratchpad before producing structured output. The result: better-quality candidates with more consistent titles and richer bodies. This single change accounted for nearly half the total improvement.

What’s remarkable is that Claude Code found this — it read the DSPy extraction pipeline, noticed it was using bare Predict, hypothesized that ChainOfThought would improve reasoning quality for a complex task, made the change, and confirmed the improvement. An AI agent reading another AI system’s code, understanding the design tradeoff, and making the right call.

Schema field descriptions matter more than prompts. The MemoryCandidate model’s title field had a 4-word description: “Short memory title.” Claude Code changed it to describe the expected format: start with a verb or noun phrase, max 10 words, self-contained. The LLM immediately started producing more consistent, descriptive titles — which cascaded into better dedup accuracy because title matching improved.

Explicit thresholds beat vague language. The sync prompt said “top_similarity very high” for no_op classification. Claude Code replaced it with concrete thresholds (0.7 for no_op, 0.4 for the update/add boundary). Classification consistency improved immediately.

What failed: ChainOfThought for summarization (hurt), tool description changes (affected multiple flows unpredictably), restrictive extraction rules (too cautious), and body format guidance (changed output enough to break fuzzy matching). Seven experiments reverted.

Per-Dimension Progression

The dedup accuracy chart (bottom-left) tells the real story. It’s volatile — swinging between 0.17 and 0.72 across experiments, sometimes with identical code. This variance comes from LLM non-determinism in the classification step. The optimization shifted the distribution upward so even the “bad” runs are better than the old baseline.

Key Takeaways

1. Agent-optimizing-agent works. Claude Code systematically improved Lerim’s memory quality by reading prompts, hypothesizing changes, and measuring impact. The keep/revert discipline caught regressions that intuition would have missed (7 of 14 experiments were correctly discarded).

2. Schema-level changes outperform prompt-level changes. Field descriptions in the Pydantic model (20 words controlling output format) had more impact than 50 lines of prompt engineering. The LLM’s output specification matters more than its instruction.

3. One module change cascades through the entire system. ChainOfThought for extraction improved extraction quality, which produced better candidates, which made dedup easier, which improved dedup accuracy. The same change in summarization disrupted this cascade. Understanding these second-order effects is why the eval harness is essential.

4. 50% failure rate is normal and valuable. Seven experiments failed. Each failure narrowed the search space — “don’t change tool descriptions in isolation” is knowledge that prevents future mistakes. The experiment log is as valuable as the kept changes.

Round 2: The Quality Gap

41% improvement. Seven kept experiments. A clear success — or so I thought.

Then I compared Lerim’s actual output to Claude Code’s auto-memory, and something was off.

Claude Code’s ~/.claude/ had 15 curated memory files. Each one was atomic, actionable, and context-independent — things like “user prefers tabs,” “always explain before coding,” “never drop Codex support.” Every memory would change how an agent behaves in a future session.

Lerim had 50+ active memories. Many were research dumps (“7 trace format investigation notes”), code-derivable facts (“the CTE uses ROW_NUMBER() OVER PARTITION BY repo_path”), or implementation details that were just describing what the code does. High eval scores, low practical value.

The eval was measuring the wrong thing. It rewarded recall (did you find everything?) but didn’t penalize over-extraction (did you extract garbage?). A system that extracts everything scores well on completeness but produces memories no agent would ever use.

So I recalibrated:

• Added a quality_alignment dimension to the judge: is each memory atomic, actionable, context-independent, structured with WHY + HOW TO APPLY?
• Reweighted the composite: precision (25%) + quality_alignment (25%) now account for half the score. Completeness dropped to 15%.
• Built a 327-case golden dataset across 20 categories — including 70 negative cases (sessions where the correct answer is zero memories) and 50 mixed cases (1-2 real decisions buried in implementation noise).
• Created a tiered eval strategy: 30-case fast eval (~15 min) for each experiment, full 327-case eval (~4 hours) at checkpoints to catch overfitting.

Then I ran /autoresearch again — same skill, different program.md, different objective.

Round 2 Results: Extraction Quality

10 experiments. 3 kept, 7 discarded. Extraction improved from 0.819 to 0.847 (+3.4%).

Smaller gains than Round 1 — but targeted at what actually matters: the quality of extracted memories.

What Worked

Quality criteria in the extraction signature (exp021, +2.2%). Adding a “QUALITY BAR” section to the DSPy signature — atomic, actionable, context-independent, structured, durable — gave the extraction LLM a clear standard to evaluate against. This is positive guidance: “here’s what good looks like.”

Body structure WHY + HOW TO APPLY (exp022, +0.7%). Changing the schema’s body field description from “Include WHY, alternatives, context” to “Lead with rule/fact, then WHY, then HOW TO APPLY” produced more structured, actionable memory bodies. The LLM’s output specification matters more than its instruction — same lesson from Round 1.

Positive body example (exp029). Adding one good example with the WHY/HOW TO APPLY structure reinforced the pattern without regression.

What Failed — and What It Taught

Restrictive rules backfire. Every. Single. Time. This is now confirmed across 5 separate experiments spanning two rounds:

Five separate restrictive-rule experiments across both rounds all regressed. The pattern is unambiguous.

Tell the LLM what good looks like. Never tell it what to avoid. Negative examples, skip lists, “don’t do X” directives — they all reduce recall without improving precision. The LLM becomes cautious and starts missing real decisions.

The no_op default is sacred (exp024). Softening “default to no_op when uncertain” to “only no_op when same specific facts” crashed maintain from 1.000 to 0.667 — the first time maintain ever regressed. The conservative dedup default prevents memory store flooding. Weakening it is catastrophic.

Sync agent changes are dangerous (exp023). Changing dedup classification language in the sync agent’s signature affected extraction quality (-0.019) even though the change targeted dedup. The sync agent controls the entire flow — touching it has unpredictable second-order effects.

The AutoResearch Skill

After two rounds, I packaged the entire pattern into a reusable Claude Code skill. Now I can run /autoresearch 100 and the agent starts the optimization loop autonomously — reads program.md for the objective, iterates hypothesize-edit-eval-keep/revert, and logs everything.

The key insight: separate the method from the objective. The skill encodes the AutoResearch pattern (how to optimize). The program.md encodes the objective (what to optimize). Change the objective, run the same skill. Round 1 optimized for pipeline quality. Round 2 optimized for extraction precision. Same skill, different program.md.

Two Rounds, One Pattern

The diminishing returns are telling. Round 1 found low-hanging fruit — the extraction pipeline was using a basic Predict call instead of ChainOfThought. Round 2 squeezed out smaller gains by teaching the LLM what quality looks like. The prompt is now near its ceiling for MiniMax-M2.5.

What came next

A week after writing this post, I tore most of the system apart.

That sounds worse than it is. The experiment’s numbers are real and the lessons held up. But writing the post forced me to look at Lerim’s extracted memories next to Claude Code’s ~/.claude/ and admit that even after +41%, the output still didn’t feel like the memories I actually wanted in my sessions. The experiment had taught me how to optimize — it hadn’t taught me what good looked like at a level the judge could reliably measure. Round 2 was the first pass at that realization. The refactor that followed was the second.

Here’s what changed between writing this post and publishing it, and why:

The judge was rebuilt as a DSPy-native, 2-layer metric. The original composite score weighted extraction/search/dedup/maintain against 327 golden cases. The new metric is simpler and more honest: a deterministic layer (25%) checks structure, format, and content heuristics; an LLM judge layer (75%) scores quality (split into worth_keeping, type_correct, well_structured), recall, and precision.

Memory architecture v2: flat directory, 4-type schema. The decisions//learnings/ split in the original architecture diagram is gone. Memories now live in a flat directory with a frontmatter type of user, feedback, project, or reference. The split had turned out to be an artifact of early thinking about “what kinds of things coding agents should remember” rather than a distinction the extractor could reliably make. Collapsing it simplified dedup, eliminated a whole class of misclassifications, and let the judge focus on content quality instead of directory placement.

The extraction and summarization pipelines were deleted. The original diagram shows Extract Pipeline and Summarize Pipeline as siblings under the Lead Agent. Both are now handled by the agent itself via its tools — there is no separate pipeline. Working on designing the agent and tools more carefully now and will do another rounds of experiment soon and publish the results.

Optimization is the main theme of Lerim. The harness, prompts, and model weights. The new design is completely based on that.

Lerim is open source and free. Try it

Follow the journey on X