AI Post-incident Platform

At a large enterprise, incident response already had a defined rhythm: PagerDuty page, Slack channel, Google Meet, triage team. The weak spot was everything after recovery. Incidents were tracked in a spreadsheet, reports were inconsistent, and the organization had no usable memory once the incident closed. My team and I built an AI post-incident platform that fit the existing workflow, generated structured reports from the artifacts teams already produced, and made past incidents discoverable when the next incident hit.

Problem

The team handling incidents had process, but not a system. Resolution details lived across Slack threads, call notes, and a spreadsheet row that usually captured only the minimum needed to close the loop. There were no consistent post-mortems, no structured record of root cause, and no easy way to see what had already been tried during a similar failure.

That gap mattered in two places. Engineers lost time re-learning the same lessons during new incidents. Leadership had no reliable way to review patterns across incidents without asking people to reconstruct context by hand. The organization had incident activity, but not incident memory.

Approach

The first design constraint was cultural, not technical. We could not start by asking incident responders to adopt a new workflow. The platform had to plug into the path they already used and prove value before it tried to change behavior.

That led to a zero-friction integration model. When PagerDuty opened an incident, it already created a Slack channel and pulled the right people in. The AI Post-Incident Platform joined that channel automatically, collected the full thread, and paired it with the Google Meet transcript captured through PagerDuty Scribe. From there, the platform generated a structured report: root cause analysis, timeline, actions taken, follow-ups, and links to the relevant repos, services, docs, and logs.

The second part of the product was retrieval. A report that exists but cannot be found is still operationally dead. We built a searchable interface for incident history and added a RAG-based chatbot on top of Turbopuffer embeddings so engineers and leaders could query prior incidents in natural language instead of scanning old channels by hand.

For high-stakes operational windows we added views that grouped related incidents and surfaced periodic AI summaries so teams could reason about system-wide impact instead of treating each page as an isolated event.

Hard Parts

Adoption came before process change

The product worked because it respected the incident culture that already existed. Trying to enforce a new ritual on day one would have failed. Meeting the current workflow first let the platform prove that a higher-quality incident record could appear without adding work for responders. That credibility made later process conversations possible.

Parallel AI processing created a new systems problem

Our first instinct was to treat report generation as a single end-to-end generation step. That path hit a latency wall quickly. Breaking the work into parallel pipelines over sections of the report improved speed, but it introduced a new problem: each sub-agent needed enough shared context to stay consistent without blowing up token usage or conflicting with the others. The architecture moved from "call a model" to "orchestrate partial views of the same incident."

Observability lagged behind the agent complexity

Once the AI pipeline split into multiple steps, debugging got harder. We looked at durable workflow and trace-first runtimes to make execution and failures legible, but infrastructure constraints inside the company made a clean adoption hard. The result worked, but understanding why a given report was good, slow, or wrong still took too much manual digging.

Outcome

The clearest validation came during the organization's busiest operational period and the months after. We launched the platform as a proof of concept, before doing real scalability hardening, and it still stayed reliable under that load. It has since processed hundreds of incidents, which forced the ingestion, summarization, and retrieval flows through a real production workload instead of a staged demo.

The product also pulled demand from both ends of the organization. Leadership started using it for reporting and trend analysis. Engineering teams asked for deeper integrations, including an API they could use for automated incident investigation and post-incident code fixes. That was the point where the project stopped looking like a single internal tool and started looking like a platform.

What I would do differently

We shipped the first version as a proof of concept, which was the right call for adoption, but it also meant some technical debt was intentional. If I were doing it again, I would keep the zero-friction rollout and add a narrow round of scalability hardening earlier. Peak load proved the platform could hold up under real pressure, but that reliability came from a simpler architecture and some luck, not from explicit capacity work.

I would also separate the "make this useful fast" decisions from the "make this easy to operate" decisions more aggressively. The product earned trust quickly; the harder part was keeping agent behavior, latency, and debugging legible as the workflow grew.

What I would do the same

What I would do next

• AI
• Incident management
• Internal tools
• Platform engineering