Circle K Cuts Incident Triage from 45–60 Minutes to Under 15

When an alert fires, the engineer usually isn't fixing the system yet. First they open metrics, hunt for the right lines in the logs, check container status, read the service documentation and look at recent code changes. At Circle K, this triage took 45–60 minutes. The company automated exactly this part of the work: an agent collects context from several sources and sends the on-call engineer in Slack a probable cause of the failure, the evidence and next steps.

According to Circle K's internal metrics, average initial triage time dropped to under 15 minutes — a reduction of 67–75%. Simple cases are handled in 5–8 minutes. This is not an independent experiment but a write-up published by AWS about the rollout of its own service, so the figures should be read as one team's result, not a promise for any infrastructure.

How the triage works

A signal from Grafana travels through Amazon SNS and SQS into Lambda. In the first version, two specialized agents worked in parallel. One checked the state of ECS tasks, load balancers, metrics and CloudWatch logs. The other looked for application errors and traces in OpenSearch, and also pulled the service card with its dependencies from the knowledge base.

A third agent then turned to GitHub: it found the repository of the affected service and studied the code, infrastructure files and recent commits. A final model call merged the results into a short report: probable cause, supporting facts, service owner and recommended actions.

Imagine a spike in HTTP 5xx responses. The infrastructure check finds container restarts, the logs show a refused connection to PostgreSQL, and the repository analysis reveals that a new connection is created for every request. If a marketing campaign started shortly before the failure, the system links the load growth to connection pool exhaustion. The on-call engineer gets not a scatter of messages but a verifiable hypothesis and fix options: reuse connections, add RDS Proxy, or adjust scaling.

What changed as load grew

When the volume of logs and context grew, individual Bedrock Flows nodes began running short on time. The team first strengthened filtering and reduced the context passed along, and then moved the logic to Amazon Bedrock AgentCore Runtime. The three agents were merged into a single process with persistent state, and access to OpenSearch, the knowledge base and GitHub was centralized through AgentCore Gateway.

Three decisions matter in practice. First, the agent can refine its queries during the investigation without losing the context it has already found. Second, OpenTelemetry traces show step durations and bottlenecks. Third, tools get limited, centrally managed access to data.

What you can replicate without tying yourself to AWS

The main idea here is not the choice of model or full autonomy. Start by automating repeatable fact gathering, leaving the decision and the fix to the engineer. Split sources into infrastructure, application logs, documentation and code; give each tool minimal permissions; require the final report to cite observable data.

Circle K separately notes that it is more useful to pre-filter logs and give the model the relevant lines than to make it sift through huge volumes on its own. But such a filter can hide an important signal, and a model can convincingly connect unrelated events. So the agent's output should remain a hypothesis: with timestamps, sources and a way to quickly verify each conclusion. The best first stage of rollout is shadow mode on incident history, measuring time, accuracy of causes and the share of false recommendations, and only then connecting to the live on-call channel.