Self-healing
Known fixes run automatically and are verified; new ones are analysed, then approved by engineers.
The self-healing architecture
Walk it: known failures take the middle row, new failures the top row. Switch clouds to see each provider's services.
Point at a component to light its links; select it for what it does here and the practice that makes it right on AWS.
The self-healing loop
A loop, not a script: each pass fixes a problem with an approved action, or teaches the system a new one.
Detect
Watchers and alerts catch crashes, out-of-memory kills, failed scheduling and error spikes.
Five levels of automation
Teams climb one level at a time; each relies on the one below being reliable.
Manual detection
Users or on-call engineers notice the problem.
- Incidents reported by users
- Diagnosis depends on who is on call
Who does what in the loop
Engineers set the rules and decide on anything new. Automation runs what is approved. AI gathers evidence and drafts.
- Set alert rules and SLOs
- Watchers, alert rules, anomaly detection
- Summarise noisy signals into one incident
- Confirm the cause
- Correlate events, logs, metrics, changes
- Search runbooks and past incidents
- Approve new or risky fixes
- Run approved runbooks
- Draft the fix and change record
- Handle escalations
- Health checks, automatic rollback
- Report what happened
- Turn approved fixes into rules
- Blameless post-mortems
- Store rules as code
- Point out repeated fixes
An action runs unattended only after engineers approved it for that kind of failure; every action is logged, verified and reversible.