30 sec3 min coredeep dive optional
oncall-kit is Anthropic’s open starter kit that mines your team’s incident history into triage playbooks for a Claude-assisted on-call, with a human signing off at every phase before anything goes near production.
- Setup reads 30 to 90 days of pager history, alert threads, and postmortems, then drafts one playbook per failure class, every claim tagged with how many incidents support it.
- Nothing activates without passing five hard-stop human gates, and the validation gate demands a 70% pass rate plus zero harmful answers on blind-replayed incidents.
- The agent is read-only by construction: it proposes fixes, even paste-ready canary ramp plans, but a human runs every deploy and there is no allowlist to widen.
- Apache 2.0, 77 stars, one frozen v1.0 commit from August 6, 2026: you fork it and make it yours, which the README says is the intended use.
Most AI on-call demos start with a heroic agent and end with an incident nobody can explain. Anthropic open sourced oncall-kit on August 3, 2026 as the working counterargument: the kit behind its CI on-call blog post, built so that Claude earns incident trust the way a new engineer does, by reading your history first and touching nothing without permission.
- 77 stars
- 4 forks
- Python
- Apache 2.0
What oncall-kit actually does
The kit puts Claude into a team’s Slack incident channel as a read-only assistant. When an alert fires, Claude classifies the symptom, loads the matching playbook, and checks a living file called lessons.md for known causes before posting a first-pass diagnosis. The rule that makes this trustworthy: every claim in the diagnosis links to the log line or dashboard behind it, so a human can cross-examine the reasoning instead of trusting a summary.
The playbooks themselves are the interesting artifact. Setup mines your own resolved incidents, pager history and alert threads and postmortems, and clusters them into failure classes, one reference file per class. Each claim carries a provenance tag such as seen 2 times, unverified, so the team can see at a glance which guidance is battle-tested and which is a first draft. Humans review every file before it counts.
Claude is never the detector in this design. Your existing alerting and paging stay primary, humans declare and close every incident, and the agent’s single autonomous state change is a staleness flag, which requires abandonment evidence from three separate signals before it posts with receipts attached. Everything else it does is gather evidence, propose, verify, and communicate.
The loop closes with two quiet disciplines. Success per incident is measured, not vibes: metrics returning to baseline within the hour is the test, and Claude polls until it can confirm or deny. And the weekly handoff skill writes the Monday morning document, what happened, what it means, what to do next, so the knowledge one on-call engineer earned does not evaporate at rotation change. Nothing here is exotic; the kit’s bet is that boring, verified loops are what an on-call actually needs.
The five gates: how setup refuses to go fast
The anti-demo of this kit is the setup flow, which is engineered to be slow in exactly the places an agent integration usually cuts corners. Each phase ends at a hard stop that waits for a human.
| Phase | What happens | Human sign-off on |
|---|---|---|
| 0 Discover | Enumerates connections, probes read access, generates STACK.md | Scope of what it can see |
| 1 Mine | Clusters 30 to 90 days of incidents into playbook drafts, seeds lessons.md | Every drafted file |
| 2 Interview | Asks about 15 minutes of questions only humans can answer: paging thresholds, severity norms, deploy windows | The ONCALL.md policy file |
| 3 Validate | Replays 5 to 10 held-out past incidents blind and grades each | At least 70% pass, zero harmful |
| 4 Install | Emits paste-ready routine messages, smallest scope first | Every paste into Slack |
The validation gate is the one worth stealing regardless of whether you adopt the kit. Incidents deliberately held back during mining get replayed in fresh context, graded pass, warn, miss, or harmful, and the bar is a floor plus a hard veto: plenty of passes cannot buy back a single harmful answer. The flow cannot reach activation before the fourth gate clears, and paging stays off until a shadow period has run its course.
Try it in ten minutes, connecting nothing
The demo path needs zero connections and zero admin rights. Clone the repository, open Claude Code inside it, and paste the single instruction from test-fixtures/RUNBOOK.md. Setup then runs end to end on 48 synthetic incidents from an imaginary team, covering mining, drafting, all five gates, and the graded validation.
git clone https://github.com/anthropics/oncall-kit
cd oncall-kit
claude
# paste the demo instruction from test-fixtures/RUNBOOK.mdThat same fixture works as a regression suite, a pattern more agent projects should copy: edit a skill, replay the demo, and diff what changed against the answer key. For real adoption you need a Claude Tag (Claude in Slack) on a Team or Enterprise plan, an org owner to attach read-only connections for each capability, and then the phrase @Claude run the oncall-setup skill in your incident channel.
Vendor neutrality is a file, not a promise
Skills in the kit may never name a specific tool. They reference capabilities, a metrics source, a log store, a pager, a code host, and a generated file called STACK.md binds each capability to whatever your team actually runs. Swapping Grafana for Datadog changes one mapping in one file and touches zero skills. The README goes further: a skill that names a vendor is classified as a bug in your fork, which turns neutrality from a design intention into an enforceable review rule.
The same discipline shows up in the memory design. lessons.md and paging-log.md live in git, where changes get reviewed; the README states plainly that load-bearing facts belong in files rather than channel memory, because files outrank chat history when the two disagree. Recurring lessons get promoted into playbooks through human-merged pull requests, so the knowledge base compounds under review rather than drifting in a channel scrollback.
The catch: a frozen v1.0 you are meant to fork
Honesty section, because the repo deserves the scrutiny. The entire repository is one commit dated August 6, 2026, tagged v1.0 in the commit message, with no releases since. The README labels it a reference implementation, says it is not maintained, and does not accept contributions. If you were hoping for a product, this is not that; it is a well-documented blueprint with a worked example.
Read as a blueprint, that is less of a problem than it sounds. CLAUDE.md, the repo-wide safety rules, is the only piece the README suggests refreshing from upstream now and then, because that is where the hard constraints live. The rest of the kit becomes your team’s code the moment you fork it, and divergence is the point. Between the fictional 48-incident demo, a filled-in non-CI example under examples/, and paste-ready setup requests down to a suggested pilot spend limit, the kit is unusually complete for something explicitly unmaintained.
How does the validation gate actually grade an agent?
Mining holds back 5 to 10 resolved incidents as a blind set, never used for drafting. In phase 3, the agent replays each one in a fresh context, with no memory of having mined it, and produces the diagnosis it would have posted at the time. A grader marks each replay pass, warn, miss, or harmful, and the gate arithmetic is deliberately asymmetric: at least 70 percent pass-plus-warn, and zero harmful answers, or the flow sends you back to mining rather than onward to install.
The zero-harmful veto is the part worth copying into any agent evaluation. Volume metrics can be gamed by a confidently wrong agent that is usually harmless; a single harmful answer, meaning guidance that would have made the incident worse, cannot be averaged away. Paired with the provenance tags on playbook claims, the evaluation measures two different things: whether the agent’s knowledge base is grounded, and whether acting on it is safe.
The grading table has a second life as an organizational artifact. The README suggests it doubles as the pitch for teammates deciding whether to adopt, which is smart: the strongest argument for an agent on-call is not a demo, it is a table showing how the agent would have performed on incidents your team remembers. Re-validation is quarterly, and sooner after infrastructure changes, so the table stays honest as the system drifts.
It helps to be precise about the four marks. Pass means the replayed diagnosis would have been actionable at the time. Warn means usable but with gaps a human should see flagged. Miss means the diagnosis would not have helped, which costs points against the 70 percent floor. Harmful means it would have made things worse, and that mark vetoes the gate outright regardless of the other grades. The asymmetry is the point: an agent that is usefully right ninety percent of the time and dangerously wrong once is more dangerous than one that is uniformly mediocre.
What the gate cannot catch is a playbook that is consistently right for the wrong reason on your historical distribution. Holdouts come from the same 30 to 90 days as the training set, so a novel failure class still lands on a human cold. That is the residual risk the shadow period exists to price, and it is why graduation is a review, not an anniversary.
What does a team’s first month with the kit actually look like?
Week one is gates and shadow mode. Setup runs its five phases with humans at every stop, then the agent goes live in the least risky way possible: diagnoses go to a review-only channel, and the human rotation keeps running every incident the way it always has. Two weeks of shadow triggers a mandatory review of the diagnosis quality, not an automatic graduation. Paging, the moment where a wrong agent opinion costs real money, gets its own go or no-go after that review.
Setup honesty extends to what it cannot see. Phase 0 records gaps as well as connections: if a team has no structured log store, STACK.md says what the agent loses without one, and the routing tree proposals come with an alert-coverage report that flags blind spots. Mining also surfaces organizational findings that have nothing to do with agents, including routing conflicts where two teams own the same alert and bus-factor concentration where one responder dominates the history. Those reports are worth running the kit for even if the agent part never graduates.
Once live, the rhythm settles into three loops. Per incident: Claude triages, proposes, verifies recovery by polling metrics until they return to baseline, and appends the lesson to lessons.md. Weekly: the handoff skill drafts the Monday morning document, what happened, what it means, what to do, plus an on-call health check and promotion pull requests for recurring lessons. Quarterly: replay validation runs again, especially after infrastructure changes that could invalidate mined playbooks.
The compounding mechanism is deliberately boring. Each closed incident writes its lesson into lessons.md, which is how the next investigation begins from a smarter base, and patterns that repeat get promoted from log entries into playbook files through reviewed pull requests. The README credits the fix-the-playbook principle, when a diagnosis is wrong you amend the playbook that produced it rather than just correcting the answer, to Anthropic’s code-migration-kit work, and it is the detail that turns a demo into a system.
Cost of attention stays bounded by design. The optional weather skill, a standing status report, checks first whether a report is even due and exits if not, so a quiet week produces near-zero channel noise and a bad week produces a handful of posts. For a team wondering whether an agent on-call means an agent that talks all day, the answer here is no: silence is the common case, and every post that does land carries evidence links.
Is read-only a real constraint or a marketing line?
Read the constraint as a capability list and it holds up. The agent’s complete set of outputs is: messages in the channel, log entries, an optional status page, proposed pull requests, and pages. It cannot write to any monitored system, cannot deploy, cannot flip a flag, and the README states there is no allowlist to widen, which forecloses the most common permission-creep path in agent integrations. Even its canary ramp plan, a genuinely useful artifact with flag name, percentage steps, hold times, and an abort metric, is emitted as paste-ready text for a human to run by hand.
The state-machine design enforces the same boundary. Incident transitions between active, stale, mitigated, and resolved are human actions in every case except one: the agent may raise a staleness flag, and only after gathering abandonment evidence from three separate signals, posting that evidence alongside. That exception is a nudge, never a closure, and it is the right one to grant because staleness neglect is a real failure mode of on-call rotations that costs nothing to automate.
What read-only does not cover is influence. An agent that posts persuasively wrong diagnoses into an incident channel can still cause damage through the humans who act on them, which is exactly why the validation gate’s harmful-answer veto and the evidence-link requirement exist. The constraint that matters is not that the agent cannot touch production, it is that every claim arrives with its receipts attached and every playbook earns its trust on held-out incidents first.
The maintenance split keeps the memory trustworthy over time. Claude appends to lessons.md freely, because an apprentice that refuses to write learns nothing; humans prune it periodically, because a lessons file nobody curates becomes a superstition archive. Promotion follows the same split: a recurring pattern starts as appended lines, earns its keep across incidents, then becomes a pull request against a playbook file that a reviewer merges or rejects. The agent proposes the codification, the team owns the canon.
There is a quiet reason this shape ages well. Because every constraint lives in reviewed files rather than in a hosted product’s configuration, an audit of “what is the agent allowed to do” is a git log, not a support ticket. When an incident review asks who permitted the agent to see the pager, the answer has an author and a date, and tightening the boundary is a pull request rather than a renegotiation with a vendor console.
For teams comparing this with policy-side tooling, the layers compose cleanly. Platform policy governs what any agent may call and for how long; this kit governs what one specific agent knows and says inside one workflow. A team could run both, and the kit’s fork-and-own model means the safety rules in CLAUDE.md become part of your reviewed codebase rather than a vendor promise.