Context Compaction in LLM Agents: Part 2 - Learning to Compact

Introduction

In Part 1, we looked at the fundamentals of context management and compaction. We covered three families of external strategies: semantic compression (asking the LLM to compress its own text), loss-aware pruning (dropping tokens that least increase the model’s loss), and dynamic summarization (maintaining a living summary that evolves over time).

These methods work. They are deployed in production today. But they share a common limitation: they are all external to the model’s reasoning process. A fixed threshold decides when to summarize. A perplexity score decides what to prune. A heuristic decides what is “important.” The model itself never gets a say.

This matters because the model is the one entity that actually knows what it needs. A pruning step that drops a seemingly redundant sentence might remove the exact detail the model needs two turns later. A summarizer triggered at a fixed token count might fire right in the middle of a multi-step derivation, cutting off a chain of thought mid-search.

So the natural question is: what if the model itself decides when, what, and how to compact?

That is exactly what the most exciting recent work does. In this post, we look at three approaches that move compaction from an external heuristic to a model-level capability.

Inference-Time Self-Compaction: SelfCompact

The first approach is the simplest and most immediately deployable. SelfCompact (Li et al., Jun 2026) gives the model two things at inference time, with no fine-tuning required:

1. A compaction tool the model can invoke to summarize accumulated context
2. A lightweight rubric specifying when to fire and when to suppress

The rubric is the key innovation. Without it, the tool alone is worse than useless. Models invoke it at unhelpful moments or not at all. The rubric provides decision boundaries:

Fire when:

• A sub-task has resolved (the agent finished searching, found the answer, completed the tool call)
• The trajectory is converging (the model is repeating similar steps)

Suppress when:

• Mid-derivation (the model is in the middle of a multi-step calculation)
• The model is stuck (compacting will not help it get unstuck)

The results are striking. On competitive math benchmarks, SelfCompact improves over a no-summarization baseline by up to 18.1 points while using 30-70% fewer tokens per question than fixed-interval summarization. On agentic search tasks, the improvement is 5-9 points.

But the most interesting finding is what the authors call the meta-cognitive gap. Left to their own devices, models cannot reliably tell when their own context is rotting. They do not have a built-in sense of “this context is getting stale.” But a lightweight external rubric closes that gap. This reframes “when to compact” not as something the model must learn through training, but as a capability that scaffolds can supply right now.

The figure above, from the SelfCompact paper, illustrates the core setup. The model is given both a compaction tool and a rubric that defines when to use it. The rubric distinguishes between trajectory states where compaction is safe (a sub-task has resolved, the trajectory is converging) and states where it should be suppressed (mid-derivation, or when the model is stuck). The results show that this combination matches or exceeds fixed-interval summarization at a fraction of the token cost.

The practical takeaway: if you are running agents today and using fixed-interval summarization, you can likely get significant improvements by adding a simple rubric that tells the model when compaction is safe.

RL-Trained Compaction: CompactionRL

SelfCompact works without training. But what if you could train the compaction decision directly? That is what CompactionRL (Jul 2026) does.

The core idea: frame compaction as a reinforcement learning problem. The model (or a separate summarization agent) receives a reward that includes both task success and token budget adherence. Over training, it learns not just what to summarize, but what information is actually needed downstream.

CompactionRL uses a split architecture: a summary agent produces the compaction, and a separate execution agent works with the compacted context. This separation matters because it mirrors how production agents already work. Claude Code, Codex, and similar tools already have distinct phases. The summary agent learns to produce summaries that are useful to the execution agent, not summaries that look good to a human evaluator.

This is an important shift. Most summarization methods optimize for human-readable quality. But what the execution agent needs is not a readable summary. It needs the specific facts, constraints, and partial results that will help it complete the task. CompactionRL optimizes for exactly that.

The RL framing also solves a problem that plagues heuristic methods: the budget-quality tradeoff. With a fixed compression ratio, you are guessing. With RL, the model learns the right ratio for different types of content. A long exploration phase might need aggressive compaction, while a critical tool-call result might need to be preserved verbatim.

CompactionRL represents the direction production agents are heading. The compaction policy becomes part of the model’s learned behavior, not an external scaffold. The model learns what to keep by experiencing the consequences of keeping or discarding different types of information across many episodes.

The figure above, from the CompactionRL paper, shows the split architecture in action. A separate summary agent produces the compaction, and the execution agent works with the compacted context. The reward signal flows back based on both task success and token budget adherence, so the summary agent learns over time what information the execution agent actually needs downstream.

Agent-Specific Learned Compaction: ACON

Both SelfCompact and CompactionRL are designed for general text contexts. But agent traces are not general text. They are sequences of tool calls, observations, and reasoning steps. An action-observation pair (“I searched for X, here are the results”) has very different importance characteristics than a paragraph of prose.

ACON (Microsoft, Oct 2025, code) is built specifically for this setting. It learns a policy for what to keep versus compress in agent trajectories, taking into account the structure of tool-call traces.

The key insight is that agent context has heterogeneous value density. A failed tool call that revealed a dead end might be critical context for the next turn. A successful tool call that returned a large file might only need its summary. A planning step that set the agent’s strategy might need to be preserved verbatim. A redundant exploration of the same code path might be safely dropped entirely.

ACON’s learned policy captures these distinctions. Rather than treating all tokens equally, it recognizes that a tool-call boundary is a natural compression point, that observations can often be summarized more aggressively than reasoning, and that failed attempts carry information that successful ones do not.

On long-horizon agentic tasks, this agent-specific approach outperforms general-purpose compression. The learned policy adapts to the workflow: a coding agent that reads lots of files needs different compaction than a research agent that executes lots of searches.

The figure above, from the ACON paper, illustrates how the learned policy handles agent trajectories. Unlike general-purpose compression that treats all tokens equally, ACON recognizes that tool-call boundaries are natural compression points, that observations can be summarized more aggressively than reasoning steps, and that failed attempts carry different information than successful ones. This structure-aware approach gives better results on long-horizon agentic benchmarks.

Comparison and Takeaways

Method	Training required	What gets compressed	Decision mechanism	Inspectability
SelfCompact	None	Accumulated context	Model + rubric	High (you see the rubric and the invocations)
CompactionRL	RL training loop	Context under budget	Learned policy	Medium (learned behavior, harder to inspect)
ACON	Learned policy	Agent trajectory (tool calls + observations)	Agent-specific learned policy	Medium

If you are building agents today:

• SelfCompact-style rubrics are the easiest win. No training, no infrastructure changes. Just add a tool and a set of rules for when to use it. The rubric does not need to be sophisticated. Even simple rules like “compact after a sub-task resolves” outperform fixed-interval summarization.
• CompactionRL is the future for production agents. If you have the infrastructure to run RL training loops, learning the compaction policy directly gives you a model that understands what its execution phase actually needs.
• ACON is the right choice if your agent traces are dominated by tool calls. The agent-specific structure awareness makes a real difference on long-horizon workflows.

All three of these methods share a limitation: they optimize within a single run. They make the current conversation or agent episode more efficient. But what happens when the run is over? The next run starts from scratch, re-debating decisions that were already made, rediscovering constraints that were already found.

That is the problem the next wave of work tackles: compiling context across runs, so the next agent starts with operating memory instead of a blank slate. We will cover that in Part 3.