Token Cost Engineering in Agent Loops — Prompt Caching and State Pruning

The agent pipeline runs a ReAct loop: Thought → Action → Observation, repeated until the agent has enough information to answer. Every iteration sends the system prompt again — and every iteration appends new tool results to the conversation history. Two things grow: the stable prefix sent identically on every call, and the accumulating context that is different every call. This post covers one fix for each.

---

The 5× asymmetry: why output tokens cost more

Every LLM pricing table has two numbers. On Sonnet 4.6: $3.00 and $15.00 per million tokens. Output is 5× more expensive than input.

This reflects how inference works at the hardware level. Input tokens are processed in a single forward pass — the attention mechanism reads all input tokens in parallel. Output tokens are generated one at a time, each requiring a full pass through the model and an update to the KV cache. Output is memory-bandwidth bound. Input is compute-bound. Memory bandwidth is the scarcer resource at scale.

Model	Input (per 1M tokens)	Output (per 1M tokens)	Multiplier
claude-sonnet-4-6	$3.00	$15.00	5×
claude-haiku-4-5	$1.00	$5.00	5×
gpt-4.1	$2.00	$8.00	4×

Token splits across system types

The 5× gap matters differently depending on what your system produces.

Classification (99.8% input): 500-token document → 1-token label (“P0”, “CRITICAL”). Output cost is negligible. Longer prompts or more examples add to the dominant input cost.

RAG query (~84% input): system prompt + 4 retrieved chunks + user query = ~1,100 tokens. Response: ~200 tokens. Even at 84% input by volume, that 200-token output costs as much as 1,000 input tokens.

Summarization (~50% input): 2,000-token document → 2,000-token summary. Equal split by token count, but the output half carries 83% of the cost. Reducing output length has 5× the leverage of reducing input.

Chatbot with growing history (~85% input by turn 20): each response gets appended to the conversation and re-sent on the next turn. The output from turn 19 is billed at $15/M on generation, then at $3/M again when it’s included in turn 20’s input. History accumulation, not per-turn generation, becomes the cost driver.

Agent loop with compounding (~74% apparent input): each ReAct iteration sends the accumulated conversation history. Output tokens from iteration N-1 are re-billed as input at iteration N. A 400-token observation at iteration 2 gets included in the input for iterations 3, 4, 5, 6, 7, 8, 9, 10 — nine more times at input rate.

For a 10-iteration FixGen run with I=2,400 average input tokens, O=400 output tokens:

10 × 2,400 × $3/M  =  $72,000/M   (base input per iteration)
10 ×   400 × $15/M =  $60,000/M   (output per iteration)
400 × 9 later iters × $3/M = $10,800/M  (compounding: prior outputs re-billed as input)
                              ──────────
                              $142,800/M ≈ $0.143/run (before optimization)

The two optimizations in this post target different parts of that formula. Prompt caching cuts the stable input terms — harness docs billed at 10% instead of 100%. State pruning cuts the compounding term — stale tool results truncated before they accumulate across iterations.

When to prioritize each lever

Input-heavy (>90% of cost from input): caching stable prefixes is the highest-leverage move. Shaving output length has minimal impact.

Output-heavy (>30% of cost from output): reduce output verbosity — shorter tool responses, tighter answer formats. Caching has no effect on output cost.

Growing history in multi-turn or agent contexts: state pruning. The longer the agent runs, the more stale context accumulates. Pruning truncates old tool results before they compound across iterations.

---

The problem: stable tokens billed at full price every iteration

Every agent in the pipeline inherits from BaseAgent, which loads a shared harness at startup:

# app/agents/base.py
@staticmethod
def _load_harness_docs() -> str:
    """Load AGENTS.md and CONSTRAINTS.md from the configured harness path."""
    docs: list[str] = []
    for filename in ("AGENTS.md", "CONSTRAINTS.md"):
        fp = p / filename
        if fp.exists():
            docs.append(f"=== {filename} ===\n{fp.read_text()}")
    return "\n\n".join(docs)

Every LLM call uses _with_harness(agent_system_prompt) which prepends them:

def _with_harness(self, system: str) -> str | list:
    docs = getattr(self, "_harness_docs", "")
    if not docs:
        return system
    return [
        {"type": "text", "text": docs, "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": f"\n\n---\n\n{system}"},
    ]

DiagnosisAgent runs up to 15 ReAct iterations per incident (raised from the original default of 10 after finding a real case that needed the extra room). FixGenerationAgent runs up to 20. The harness docs are sent — and billed — on every single one.

---

The fix: one content block, one cache_control flag

Anthropic’s prompt caching marks a content block with cache_control: {"type": "ephemeral"}. On the first call, the marked prefix is written to the cache (5-minute TTL). Every subsequent call that sends the same prefix reads from cache at 10% of the regular input token price.

The change shown above to _with_harness does exactly this: instead of a plain string, it returns a list of content blocks. The harness docs are the first block with cache_control. The agent-specific system prompt is the second block, never cached (it changes between agent types).

Both the Anthropic Python SDK and LiteLLM accept list[dict] for the system parameter. No other call sites needed to change — the if system: checks in LiteLLMProvider.complete() handle both types.

---

Tracking the savings

LLMResponse carries cache metadata:

@dataclass
class LLMResponse:
    content: str
    input_tokens: int
    output_tokens: int
    provider: str
    model: str
    cost_usd: float
    cache_read_input_tokens: int = 0    # tokens served from cache (billed at 10%)
    cache_creation_input_tokens: int = 0  # tokens written to cache (billed at 125%)

LiteLLMProvider extracts them from LiteLLM’s Anthropic response:

details = getattr(usage, "prompt_tokens_details", None)
if details:
    cache_read = getattr(details, "cached_tokens", 0) or 0

The gateway applies correct billing:

# Cached tokens are billed at 10% — compute actual cost accordingly
billed_input = raw.input_tokens - raw.cache_read_input_tokens
cost = self._compute_cost(model, billed_input, raw.output_tokens)
cost += self._compute_cost(model, raw.cache_read_input_tokens, 0) * 0.1

And logs cache activity on every call:

[gateway] task=diagnosis model=claude-sonnet-4-6 in=2043 out=312 cache_read=1858 cache_write=0 cost=0.006234 latency=2341ms

---

The measured result

Baseline vs cached — 6 iterations of a diagnosis loop, same system prompt, same user message, with the exact per-iteration cache activity captured this time (not just the 6-call aggregate):

Metric                            Without cache      With cache       Delta
────────────────────────────────────────────────────────────────────────────
Total input tokens                        9,822           9,822
Cache read tokens                             –           7,015    (71% of input)
Cache write tokens                            –           1,403
Total cost, as this repo tracks it (USD)  $0.158106    $0.138626    -$0.0195 (12.3%)
Total cost, real Anthropic billing (USD)  $0.158106    $0.139678    -$0.0184 (11.7%)
Avg latency (ms)                          32,580          32,143
First-call latency (ms)                   38,269          34,855    (cache write)

That second cost row exists because of a real bug this measurement surfaced: Anthropic genuinely charges a 1.25× premium on cache-write tokens (verified directly, not assumed) — but this repo’s own LLMGateway.complete() never applies that premium anywhere in its billing code, so every cost figure this app has ever tracked for a call that includes a cache write has been a small undercount of the real bill. The 1,403 write tokens above should have cost 25% more than they were charged (~$0.001 more here). Corrected for that, the real savings is 11.7%, not 12.3% — the effect is real either way, just slightly smaller than what this app’s own dashboard would show you.

Per-iteration, this time captured directly rather than only summed:

With cache:
  Iteration 1:  input=1,447  cache_write=1,403  cache_read=0
  Iteration 2:  input=1,523  cache_write=0      cache_read=1,403
  Iteration 3:  input=1,599  cache_write=0      cache_read=1,403
  Iteration 4:  input=1,675  cache_write=0      cache_read=1,403
  Iteration 5:  input=1,751  cache_write=0      cache_read=1,403
  Iteration 6:  input=1,827  cache_write=0      cache_read=1,403

This is the clean, textbook version of the mechanic: the cache is written once, on the first call. Every one of the 5 remaining calls reads that exact same fixed-size block in full — 5 × 1,403 = 7,015, exactly matching the reported total. No partial hits, no missed reads, no discrepancy — when the harness content is genuinely stable across the loop, the math is just “(remaining iterations) × (cache block size).”

The ~12% cost reduction closely reproduces the original finding (13%) on different content, a different day, real independent API calls — a solid confirmation that the cost-savings effect itself is real and repeatable, even though the exact token counts differ (this run’s harness stand-in is smaller than the original’s).

The latency comparison, though, came out differently this time: average latency was slightly lower with caching (32,580ms → 32,143ms), and even the first call — the one paying the cache-write cost — came out faster (38,269ms → 34,855ms), not slower. That’s the opposite direction from the earlier result. Six real API calls per run, each taking 30+ seconds with real network and server variance, isn’t a large enough sample to draw a confident conclusion about latency either way — getting the opposite result on an independent re-run is itself the useful signal here: it means the original “caching costs a bit of latency” claim was more likely measurement noise than a real, reproducible effect. The cost savings reproduced closely; the latency direction did not.

---

Why 71% and not 100%

The cached block here is 1,403 tokens (this run’s harness-doc stand-in). Each iteration adds new conversation turns (the growing ReAct context), which are never part of the cached block. So the cache’s share of each call’s total input naturally shrinks as the conversation grows, even though the cached block itself never changes size:

Iteration 2:  1,403 / 1,523 total = 92.1% of that call's input is cached
Iteration 3:  1,403 / 1,599 total = 87.7%
Iteration 4:  1,403 / 1,675 total = 83.8%
Iteration 5:  1,403 / 1,751 total = 80.1%
Iteration 6:  1,403 / 1,827 total = 76.8%

Averaged across all 6 calls — including the first, which has a 0% hit rate since nothing was cached yet — that comes out to the reported 71%. The absolute number of tokens saved per call stays exactly constant (1,403, every single read); it’s the percentage that drifts down over a longer-running loop, purely because the denominator (total input, including the ever-growing conversation tail) keeps growing while the numerator doesn’t.

This is expected and fine. The cache saves the same number of tokens per iteration regardless of conversation length — it’s the ratio that changes.

---

Why the 5-minute TTL matters for agents

Anthropic’s cache TTL is 5 minutes. A single ReAct loop usually completes in under 2 minutes — all iterations share the same cached prefix with no expiry risk.

Where TTL matters: between incidents, not within them. If the next incident starts more than 5 minutes after the previous one, the cache is cold and the first iteration pays the write cost again. For the pipeline polling CloudWatch every 5 minutes, the cache will frequently be cold at the start of each incident. The savings come from within-incident reuse, not cross-incident.

For higher-throughput pipelines running many concurrent incidents, cross-incident cache sharing is possible if incidents start within the TTL window. In this pipeline, incident volume is low enough that within-incident savings are the primary benefit.

---

State pruning: cutting the compounding history

Prompt caching handles the stable prefix — harness docs, system prompt, tool definitions. It cannot help with the growing conversation history, because history changes on every iteration.

In FixGenerationAgent, the largest per-iteration growth comes from read_file tool results. Each call returns the full source file — typically 1,500–2,000 tokens. By iteration 5, the agent has read several files, and every subsequent iteration re-sends all of those file contents as part of the conversation history.

The agent needed those file contents to reason about what to change. By iteration 8, either the edit has been applied or the file is no longer relevant. The full text is dead weight billed on every remaining iteration.

The fix: _prune_tool_results()

def _prune_tool_results(
    messages: list[dict],
    keep_last: int = 2,
    threshold: int = 500,
) -> tuple[list[dict], int]:
    """Truncate large tool results that are no longer recent.

    Keeps the last `keep_last` tool messages intact so the model retains
    recent context. Earlier tool results longer than `threshold` chars are
    replaced with a short stub — the model can re-call read_file if it
    needs the content again.

    Returns (pruned_messages, chars_removed).
    """
    tool_indices = [i for i, m in enumerate(messages) if m.get("role") == "tool"]
    prune_set = set(tool_indices[:-keep_last]) if len(tool_indices) > keep_last else set()

    pruned_messages: list[dict] = []
    chars_removed = 0
    for i, msg in enumerate(messages):
        if i in prune_set:
            content = msg.get("content", "")
            if isinstance(content, str) and len(content) > threshold:
                chars_removed += len(content) - threshold
                msg = {**msg, "content": f"[pruned — {len(content)} chars — re-call read_file if needed]"}
        pruned_messages.append(msg)
    return pruned_messages, chars_removed

Called every 3 iterations in the FixGen loop:

if iteration > 0 and iteration % 3 == 0:
    messages, pruned = _prune_tool_results(messages)

threshold=500 characters catches read_file results (1,500–2,000 tokens each) while preserving short tool confirmations like {"status": "ok", "path": "..."} (~50 chars — kept intact). keep_last=2 retains the two most recent oversized results — the agent needs recent file state to reason about the next edit. Everything older is replaced with a short stub.

Cost impact

Measured directly, not estimated: the real _prune_tool_results() function, run against a simulated 10-iteration FixGen loop with 3 real-sized file reads (tokenized with tiktoken, priced with the actual LLMGateway._compute_cost, Sonnet’s real $3/M input rate) — no live API calls needed, since this only isolates the cost of carrying growing tool-result history, not the model’s own reasoning.

Total input-token cost from carrying tool-result history across 10 iterations:
  without pruning: $0.102597
  with pruning:    $0.048453
  saved:           $0.054144  (52.8%)

The absolute dollar saving ($0.054) landed almost exactly on the original back-of-envelope estimate — that part of the math held up. The percentage is real this time too, but scoped precisely: 52.8% reduction in the cost of resending stale tool results specifically — the exact mechanism pruning targets — not “52.8% off the whole FixGen step,” which also includes the model’s own reasoning/output tokens on every call, untouched by pruning. The real, whole-pipeline percentage (if you want one) still needs to come from the live /agents/pr-stats endpoint, not from this isolated calculation.

Prompt caching and state pruning attack different parts of the cost with no overlap:

Prompt caching:  stable harness prefix (1,403 tokens, measured) — billed at 10% instead of 100%
State pruning:   stale tool results (18,000 tokens) — removed from later iterations entirely

---

Impact on pipeline cost

Per-incident cost comes from the full pipeline: triage (Haiku), diagnosis (Sonnet), fix generation (Sonnet, up to 20 iterations), self-critique (Haiku), code review (GPT-4.1). (Live average cost per incident is tracked at GET /agents/pr-stats and should be pulled fresh rather than cited as a fixed number here — it moves as real incident volume and mix changes.)

Prompt caching (Anthropic calls only):

• Diagnosis (6 iterations measured, reproduced independently on a second real run): ~12% reduction on that step (11.7% once corrected for a real cache-write billing gap this measurement surfaced — see “The measured result” above)
• FixGeneration (10–20 iterations): scales with iteration count — more iterations means more cache-read calls amortizing the same fixed write cost
• Triage and self-critique (Haiku): benefit from caching, but shorter runs mean smaller absolute savings
• CodeReview (GPT-4.1): not affected — OpenAI’s caching mechanism is different and not implemented here

State pruning (FixGeneration only):

• $0.054 saved per run on a simulated 10-iteration loop with 3 file reads — measured directly against the real _prune_tool_results() function, not a live model run — a real 52.8% reduction in the cost of carrying stale tool-result history specifically (not the whole FixGen step, which also includes untouched model-reasoning cost).
• No overlap with prompt caching: caching targets stable harness tokens, pruning targets stale tool results that have grown into dead weight

Combined, on the FixGen step specifically (the dominant cost component for most incidents), these two changes stack to a meaningfully lower cost than either alone — the exact percentage, for either the FixGen step or the full pipeline, depends on the current incident mix and should be measured against the live endpoint rather than assumed.

---

Summary

Prompt caching — three logical changes:

• _with_harness() — return content blocks with cache_control instead of a plain string
• LLMResponse — add cache_read_input_tokens and cache_creation_input_tokens fields
• LLMGateway.complete() — bill cached tokens at 10%, log cache activity

No agent code changed. No prompt changed. No behavior changed. The ReAct loop sees the same system prompt content; only the billing changes.

State pruning — one function, one call site:

• _prune_tool_results() — replace oversized tool results older than keep_last=2 with a stub
• Called every 3 iterations in the FixGen agentic loop

No model changed. No prompts changed. The agent still reasons over the same events — it just stops carrying full file contents past the point where they’re useful.

Together, these target the two independent cost drivers in agent loops: the stable prefix billed redundantly on every iteration, and the growing history carrying data that has already served its purpose.

---

What doesn’t get cached

The agent-specific system prompt (the second content block in the list) is never cached — it’s small (~80 tokens for FixGen, ~50 for Diagnosis) and different for each agent type. Not worth the complexity of a separate cache entry.

The growing conversation history (ReAct turns) is not cached — it changes on every iteration. Anthropic supports caching messages in addition to the system prompt, but caching a growing conversation is complex: you’d need to checkpoint at specific message boundaries and re-cache when the checkpoint changes. The harness docs cache already captures the highest-value stable prefix; the incremental gain from conversation caching doesn’t justify the complexity.

Tool definitions are also not cached here. They’re static for a given agent, and caching them alongside the system prompt would require restructuring how complete_with_tools passes tools. That’s a natural follow-on if token costs become a bottleneck.