agent-platform

Pipeline

Every production error flows through this graph. Sandbox failure regenerates the fix up to 3× before any GitHub noise. Below the confidence threshold with no file identified, ErrorClarityAgent adds logging instead of generating a fix. A REQUEST_CHANGES review doesn't automatically block merge — MergeDecisionAgent decides whether it ships anyway or goes back for a refix. HIGH/CRITICAL incidents stop at the approval gate; a merged PR fires MonitorGenerationAgent to draft new CloudWatch alarms (dry-run in production today).

Deployment topology

Cloudflare-fronted ALB sits in front of ECS Fargate. SQLite is the source of truth for incident and approval state; pgvector backs the RAG index. CI/CD is OIDC — no secrets in the repo.

The agents

How the metrics are computed

• Agent pipeline — ~6 min — wall-clock from detected event to GitHub PR creation. Measured by scripts/measure_mttr.py against the live demo DB. Covers triage → diagnosis → fix generation → sandbox → PR open; excludes human review and CI time.
• Avg MTTD — 75.8h — wall-clock from when an error actually first occurred in the target app's logs to the pipeline detecting it, across resolved incidents. Bounded by the polling interval on one side; on the other, largely a function of how often a given error recurs before crossing the alerting threshold, not a wait added by the agent pipeline itself.
• Avg MTTR — 32 min — wall-clock from detected event to merged fix PR, including human review and CI time. Measured across resolved incidents in the live demo DB. The ~6 min agent pipeline is the automated portion; the remainder is async human approval and CI.
• CI pass rate — 100% — fraction of generated patches that pass the Docker sandbox test suite on the first run. The sandbox clones the target repo, runs a baseline (pre-patch), applies the fix, and only fails the run if the fix introduces new test failures beyond the baseline.
• Avg confidence — 56% — DiagnosisAgent's self-reported confidence, averaged across every diagnosis — including the lower-confidence ones that escalated to ErrorClarityAgent or human approval rather than auto-merging. The 11 PRs that did merge each individually cleared the 70% threshold; this average isn't gated the same way.
• Avg cost / incident — $0.7611 — total Anthropic + OpenAI token spend across triage, diagnosis, fix generation, and review for one incident, averaged over resolved incidents in the live demo DB.

Tech stack

Design decisions worth defending

1. Poll the log destination, not the server — and keep a webhook as a second path. A background poller runs filter_log_events against CloudWatch Logs on an interval, so detection doesn't add traffic to the production container. A CloudWatch-alarm webhook (SNS) is also wired in for lower-latency notification — detection isn't a single mechanism, it's both running concurrently.
2. RAG finds candidates. The live store confirms truth. pgvector holds index-time snapshots; current incident state lives in SQLite. Blocking decisions always re-read the live store — the post explains why. Code retrieval uses hybrid search (70% vector + 30% lexical, min_score=0.45) — upgraded from pure vector after both agents were found to be passing low-relevance chunks to the LLM regardless of score. Incident retrieval uses two-stage retrieval: vector search at min_score=0.80 for recall, cross-encoder reranking (rerank_incidents) for precision.
3. Hard blocks are deterministic, soft hints are LLM-shaped. Dedup is a hard block (drop the event); regression context is a soft hint (prompt injection). Mixing the two created a four-failure-mode bug — walked through here.
4. Ground every symbol against the repo. DiagnosisAgent runs verify_symbol_in_repo via GitHub Code Search; a server-side guard re-checks every named function in the parsed output and rejects fabricated camelCase identifiers.
5. Checking an answer isn't the same as requiring one be built from real data. DiagnosisAgent's grounding guard used to run only after the model had already committed to an answer — enough tool calls to pass a count-based gate, but not necessarily calls to anything that reads code, let the model fabricate a cited file and quote a snippet that didn't exist. Now it's required to call an actual code-reading tool before it's allowed to answer at all, and a grounding failure flags the human-facing narrative directly instead of only nulling structured fields silently — the full story.
6. Don't guess when you can't ground. If DiagnosisAgent's confidence falls below 70% and it can't name a file, FixGenerationAgent never runs on a hunch. ErrorClarityAgent takes over instead — it reads exact code lines and adds targeted logging, and a scope guard rejects any proposed change that isn't observability. The next diagnosis gets real data instead of a fabricated one.
7. Sandbox before PR. Every fix runs in a Docker container against the real test suite. If tests fail, regenerate up to 3× before opening any GitHub noise.
8. Cross-provider review. Fix generation uses Claude Sonnet; code review uses GPT-4.1. Enforced at startup — the gateway raises on boot if both resolve to the same provider. Different model families have different blind spots; the reviewer is structurally incapable of being lenient with its own output.
9. A REQUEST_CHANGES review doesn't mean start over. MergeDecisionAgent (one Haiku call) classifies review feedback as merge_now — ship despite non-blocking nitpicks — or refix_first. Narrower than a general auto-merge decision: it only fires in response to that one review outcome, and it's the only thing standing between a correct fix and an unnecessary regeneration loop.
10. Approval gate before every merge. All PRs require human approval; HIGH/CRITICAL incidents escalate immediately. Rejections are logged as RLHF preference pairs.
11. Monitor the pipeline, not just the target app. Three background loops watch the system watching the system: a threshold monitor polls infra health (memory, ALB 5xx rate, ECS running-vs-desired) with a 30-min per-resource cooldown; a drift detector alerts if the 2-day auto-fix approval rate drops >15pp off a 7-day baseline or below a 50% floor; and validate_models_live() confirms every configured Anthropic model ID still resolves against the live API on every startup — logging loudly rather than raising, closing the exact gap that once broke agents silently for months.
12. Two orthogonal controls on token cost. Prompt caching marks the stable harness prefix with cache_control: ephemeral — 77% cache hit rate, 13% cost reduction on the diagnosis step. State pruning removes stale read_file results from the FixGen conversation history every 3 iterations (~18K tokens per 10-iteration run, 29% reduction on that step). Neither interferes with the other: caching targets what is always the same, pruning targets what has already served its purpose.