← Home

agent-platform

A 7-agent pipeline, orchestrated by a plain-Python IncidentLoop (no LLM call of its own), that detects production crashes, grounds the diagnosis against the actual repo, generates a patch, validates it in a Docker sandbox, and opens a PR — with a human approval gate before any merge, escalating urgency for HIGH/CRITICAL severity. When the diagnosis can't confidently name a file, ErrorClarityAgent adds targeted logging instead of guessing; when review requests changes, MergeDecisionAgent decides whether the fix ships anyway or gets regenerated.

Source on GitHub →

Pipeline

Every production error flows through this graph. Sandbox failure regenerates the fix up to 3× before any GitHub noise. Below the confidence threshold with no file identified, ErrorClarityAgent adds logging instead of generating a fix. A REQUEST_CHANGES review doesn't automatically block merge — MergeDecisionAgent decides whether it ships anyway or goes back for a refix. HIGH/CRITICAL incidents stop at the approval gate; a merged PR fires MonitorGenerationAgent to draft new CloudWatch alarms (dry-run in production today).

flowchart TB A([CloudWatch Logs<br/>poll · 5 min]) --> D{Dedup gate} D -- duplicate --> X([drop]) D -- new --> E[TriageAgent · Haiku] E -- noise --> X E -- duplicate --> X E -- real, P0–P3 --> F[DiagnosisAgent · DeepSeek-V4.1-Flash] F --> Q{Confidence ≥ 70%?} Q -- yes --> G[FixGenerationAgent · Sonnet] Q -- no, file unknown --> EC[ErrorClarityAgent] Q -- no, file known --> L[Human approval] G --> SC[Self-critique · Haiku] SC -- LIKELY WRONG --> G SC -- pass --> H{Sandbox<br/>npm test} H -- fail · retry 3× --> G H -- pass --> J[Open GitHub PR] J --> I[CodeReviewAgent · GPT-5.5] EC -- adds logging --> I EC -- flag only, no PR --> L I --> K{Review verdict} K -- approved --> L K -- REQUEST_CHANGES --> MD[MergeDecisionAgent · Haiku] MD -- merge_now --> L MD -- refix_first --> G L --> M[Merge] M -. webhook .-> MG[MonitorGenerationAgent<br/>dry-run] classDef event fill:#0b0d10,stroke:#2f343b,color:#e6e8eb classDef agent fill:#13161a,stroke:#6ee7b7,color:#e6e8eb classDef gate fill:#0b0d10,stroke:#23272d,color:#9aa1a9 class A,X event class E,F,G,SC,I,EC,MD,MG agent class D,H,K,Q gate
End-to-end pipeline — from detected event to merged fix PR.

Deployment topology

Cloudflare-fronted ALB sits in front of ECS Fargate. Postgres (RDS) is the source of truth for incident and approval state; pgvector backs the RAG index. Diagnosis calls DeepSeek-V4.1-Flash through Together. CI/CD is OIDC — no secrets in the repo.

flowchart LR U([User · Interviewer]) --> CF[Cloudflare<br/>edge] CF --> ALB[AWS ALB<br/>:443] ALB --> ECS[ECS Fargate task<br/>FastAPI + Uvicorn :8000] ECS --> DB[(Postgres · RDS<br/>incidents + approvals)] ECS --> PG[(pgvector<br/>RAG index)] ECS --> S3[(S3<br/>artefacts)] GH[GitHub Actions OIDC] -. on push to main .-> ECR[ECR] ECR -. ECS pull .-> ECS CWLOGS[(CloudWatch Logs<br/>target app)] -. poll + webhook .-> ECS classDef ext fill:#0b0d10,stroke:#23272d,color:#9aa1a9 classDef compute fill:#13161a,stroke:#6ee7b7,color:#e6e8eb classDef store fill:#13161a,stroke:#2f343b,color:#e6e8eb class U,CF,GH ext class ALB,ECS compute class DB,PG,S3,ECR,CWLOGS store
AWS ECS Fargate · Cloudflare · GitHub Actions OIDC.

The agents

TriageAgent

ReAct + typed wrapper

Classifies events as real / noise / duplicate. P0–P3 severity. Haiku for cost. Noise and duplicates terminate here — nothing downstream ever sees them.

DiagnosisAgent

ReAct + grounding gate

Names the file and function behind an error, on DeepSeek-V4.1-Flash with a harness evolved by the optimizer agent. Explores the repo with grep, file reads, symbol verification (local checkout first; a failed lookup says so instead of reporting "not found") and a tree-sitter call graph, and submits through a grounding gate that rejects any diagnosis not backed by code it actually read. A diagnosis that ends with no accepted answer is retried once.

ErrorClarityAgent

Tool loop + scope guard

Runs when diagnosis confidence falls below 70% and no file was identified. Doesn't guess — reads exact code lines via GitHub and adds targeted logging (never a behavior fix) so the next diagnosis has real data to work with. A scope guard rejects any proposed change that isn't net-new observability.

FixGenerationAgent

Direct LLM + critique + sandbox

Writes the patch (Sonnet) via an internal tool-use loop. Haiku self-critique runs before the sandbox — a LIKELY WRONG verdict triggers a retry, not a hard stop. Validates in a Docker clone of the real repo (3× retries), checks BlastRadiusGuard (protected paths, size caps), opens a GitHub Issue + PR.

CodeReviewAgent

Direct LLM · cross-provider

Reviews whichever PR got opened (GPT-5.5 via litellm) — enforced cross-provider at startup so the reviewer can't share blind spots with the generator. Applies different criteria depending on the source: root-cause/symptom-fix checks for a fix PR, secrets/PII and behavior-change checks for a clarity PR.

MergeDecisionAgent

Single-call classifier

Fires only when code review comes back REQUEST_CHANGES. One Haiku call classifies the feedback as merge_now (ship despite non-blocking nitpicks) or refix_first — narrower than a general auto-merge decision.

MonitorGenerationAgent

Direct LLM · dry-run

Fires on GitHub PR merge (webhook), reads the diff, and drafts CloudWatch alarm configs — roughly one per 75 lines changed. Dry-run only in production today; provisioning is gated behind a flag for controlled rollout.

Self-improving harness

An optimizer agent evolves DiagnosisAgent's prompts, tool descriptions, settings and control flow, using GEPA-style proposals from failing traces and RRSI-style regularization: a noise band measured on the unchanged harness, cost-aware acceptance, and an LLM leakage critic. Whole repos are held out, and the final harness is tested against simply running the original twice. The first full run, on Sonnet 5, rejected every edit as noise. After a power analysis, a 7.4-hour run on DeepSeek-V4.1-Flash accepted two changes.

Held-out localization

78.8% → 89.9%

66 cases from unseen repos · +11.1pp, 95% CI +6.1 to +16.2

vs running twice

Tie, at ~60% of the cost

best-of-2 on the original harness: 89.9%

vs Claude Sonnet 5

89.9% vs 79.5%

same held-out cases · 4.7× lower cost per diagnosis

Regression gate. Every PR that changes diagnosis replays 434 SWE-bench Verified cases and compares each with main's own measured pass rate, a required check on main. It catches a 3.5-point localization drop about 85% of the time at a 5% false-alarm rate; the old fixed-threshold gate caught under 10%.

Read the 7-part series →

How the metrics are computed

Tech stack

Agent runtime Anthropic SDK · Claude Sonnet 4.6 (fix) / Haiku 4.5 (triage, critique) · DeepSeek-V4.1-Flash via Together (diagnosis) · GPT-5.5 (review) · routed through litellm
Web framework FastAPI · WebSocket streaming · Pydantic v2
Storage Postgres (RDS) for incident/approval state · pgvector for RAG
RAG Function-boundary chunking · hybrid search (α=0.7 vector + 0.3 lexical, min_score=0.45) · cross-encoder rerank for incidents · document chunk registry
Sandbox Docker Compose · Jest · mongodb-memory-server
Tracing Langfuse — every LLM call + tool execution as nested spans
Self-improvement Optimizer agent: GEPA-style proposals · RRSI-style regularization (noise-calibrated acceptance, LLM leakage critic) · checkpointed, budget-capped, tripwired long-running loop
Evals SWE-bench Verified (500) · repo-disjoint evolve/held-out split · pass@k / pass^k · paired regression gate over 434 cases, required on main
Resilience Circuit breakers · schema validation at handoffs · context checkpointing
Cost Prompt caching (cache_control: ephemeral · 77% hit rate · 13% reduction on diagnosis) · state pruning (~18K tokens removed per FixGen run) · diagnosis on DeepSeek at 4.7× lower cost per diagnosis than Sonnet
Deploy AWS ECS Fargate · ALB · Cloudflare · GitHub Actions OIDC
Frontend React + Vite · WebSocket dashboard · served from same container

Design decisions worth defending

  1. Poll the log destination, not the server — and keep a webhook as a second path. A background poller runs filter_log_events against CloudWatch Logs on an interval, so detection doesn't add traffic to the production container. A CloudWatch-alarm webhook (SNS) is also wired in for lower-latency notification — detection isn't a single mechanism, it's both running concurrently.
  2. Retrieval is tuned per use case, not one-size-fits-all. Code retrieval uses hybrid search (70% vector + 30% lexical, min_score=0.45) — upgraded from pure vector after both agents were found to be passing low-relevance chunks to the LLM regardless of score. Incident retrieval uses two-stage retrieval: vector search at min_score=0.80 for recall, cross-encoder reranking (rerank_incidents) for precision.
  3. Hard blocks are deterministic, soft hints are LLM-shaped. Dedup is a hard block (drop the event); regression context is a soft hint (prompt injection). Mixing the two created a four-failure-mode bug.
  4. Ground every symbol against the repo. DiagnosisAgent verifies symbols against the local checkout first (definitions before references), and a failed lookup returns VERIFY_ERROR instead of pretending the symbol doesn't exist. That rule came from a GitHub token typo that made the tool answer "not found" on 87 of 87 calls, which the optimizer then tried to work around with a prompt edit.
  5. Checking an answer isn't the same as requiring one be built from real data. DiagnosisAgent's grounding guard used to run only after the model had already committed to an answer — enough tool calls to pass a count-based gate, but not necessarily calls to anything that reads code, let the model fabricate a cited file and quote a snippet that didn't exist. Now it's required to call an actual code-reading tool before it's allowed to answer at all, and a grounding failure flags the human-facing narrative directly instead of only nulling structured fields silently.
  6. Don't guess when you can't ground. If DiagnosisAgent's confidence falls below 70% and it can't name a file, FixGenerationAgent never runs on a hunch. ErrorClarityAgent takes over instead — it reads exact code lines and adds targeted logging, and a scope guard rejects any proposed change that isn't observability. The next diagnosis gets real data instead of a fabricated one.
  7. Sandbox before PR. Every fix runs in a Docker container against the real test suite. If tests fail, regenerate up to 3× before opening any GitHub noise.
  8. Cross-provider review. Fix generation uses Claude Sonnet; code review uses GPT-5.5. Enforced at startup — the gateway raises on boot if both resolve to the same provider. Different model families have different blind spots; the reviewer is structurally incapable of being lenient with its own output.
  9. A REQUEST_CHANGES review doesn't mean start over. MergeDecisionAgent (one Haiku call) classifies review feedback as merge_now — ship despite non-blocking nitpicks — or refix_first. Narrower than a general auto-merge decision: it only fires in response to that one review outcome, and it's the only thing standing between a correct fix and an unnecessary regeneration loop.
  10. Approval gate before every merge. All PRs require human approval; HIGH/CRITICAL incidents escalate immediately. Rejections are logged as RLHF preference pairs.
  11. Monitor the pipeline, not just the target app. Three background loops watch the system watching the system: a threshold monitor polls infra health (memory, ALB 5xx rate, ECS running-vs-desired) with a 30-min per-resource cooldown; a drift detector alerts if the 2-day auto-fix approval rate drops >15pp off a 7-day baseline or below a 50% floor; and validate_models_live() confirms every configured Anthropic model ID still resolves against the live API on every startup — logging loudly rather than raising, closing the exact gap that once broke agents silently for months.
  12. Two orthogonal controls on token cost. Prompt caching marks the stable harness prefix with cache_control: ephemeral — 77% cache hit rate, 13% cost reduction on the diagnosis step. State pruning removes stale read_file results from the FixGen conversation history every 3 iterations (~18K tokens per 10-iteration run, 29% reduction on that step). Neither interferes with the other: caching targets what is always the same, pruning targets what has already served its purpose.
  13. A harness change has to beat measured noise. The same harness run twice moves by several points, so the optimizer measures that noise on the unchanged harness and rejects any gain inside it unless the change is also clearly cheaper. The first full run rejected every edit, correctly. The fix wasn't loosening the rules: a power analysis showed 51 cases couldn't see a 5-point effect, so the next run used 138 cases the model actually fails.
  14. Re-validate harness changes on every model you ship. The evolved prompt edit helped DeepSeek and did nothing for Sonnet (−2.3pp, within noise), while the retry change helped both. So the evolved harness shipped together with the model it was evolved on.
  15. Gate on each case's own history, not a fixed threshold. Cases that always pass barely move under a mild regression, so the old gate (fail if more than 6 of 56 always-pass cases break) would have missed most real 3–5 point drops. The new gate compares every one of 434 cases with main's measured pass rate on that case.