Classify incidentes, coordinate updates, e transformar o resposta em follow-up work. — Claude Skill

name: incident-response description: incidente triagem, cascade prevention, e postmortem methodology. usar when handling production incidentes, designing resilience patterns, ou conducting chaos engenharia exercises. keywords:

• incidente resposta
  • outage
  • postmortem
  • triage
  • incident
  • response

incidente resposta

Structured incidente management a partir de detection through postmortem, com resilience patterns para preventing e containing cascading failures.

When para usar

• Production incidente in progress (outage, degradation, dados loss)
• Designing circuit breakers, bulkheads, ou fallback strategies
• Conducting ou planning chaos engenharia exercises
• Writing ou reviewing postmortem documents
• Establishing on-call procedures e escalamento paths

Avoid when:

• o issue is a development-time bug com no production impacto
• Designing general system architecture (usar system-design instead)

Quick Reference

incidente resposta workflow

Phase 1: detetar

• Alert fires ou utilizador relatório received
• Confirm o issue is real (não a false positive)
• Identify affected services e utilizador impacto scope

Phase 2: triagem

• Classify severidade (P0-P3)
• atribuir incidente commander
• Open communication channel (war room, Slack channel)
• Begin status página updates

Phase 3: Contain

• Stop o bleeding: rollback, feature assinalar, traffic shift
• Prevent cascade: circuit breakers, load shedding, bulkhead isolation
• Communicate: stakeholder updates cada 15 minutes para P0/P1

Phase 4: Resolve

• Implement fix (minimal viable fix primeiro)
• validar in staging if time permits
• Deploy com monitoring e rollback plano pronto
• Confirm recovery com métricas returning para baseline

Phase 5: Postmortem

• Document timeline dentro de 48 hours
• Conduct blameless rever com todos participants
• Identify root cause e contributing factors
• atribuir ação items com responsáveis e deadlines
• Update runbooks e alerting based on lessons learned

severidade Framework

Output

• incidente timeline e severidade classification
• Containment ações taken
• Postmortem document com ação items
• Updated runbooks e alerting rules

Common Mistakes

• Skipping severidade classification e treating everything as P0
• Making changes sem a rollback plano
• Forgetting para communicate status para stakeholders
• Writing postmortems that atribuir blame instead de identifying systemic issues
• não following up on postmortem ação items

triagem Framework

severidade classification, cascade prevention, communication protocols, e escalamento paths para production incidentes. usar during active incidentes ou when establishing incidente resposta procedures.

severidade Classification (P0-P3)

P0 -- Critical

Definition: Complete service outage, active dados loss, ou security breach affecting todos utilizadores.

Examples:

• Production database unreachable
• Authentication service completely down
• Active dados corruption ou loss
• Security breach com confirmed exfiltration
• Payment processing halted

P1 -- High

Definition: Major feature broken ou significant degradation affecting a large subset de utilizadores.

Examples:

• Payment processing failing para one region
• pesquisar functionality returning errors para 20%+ de queries
• API latency 10x above normal
• Mobile app crash on lançar para specific OS version

P2 -- Medium

Definition: Degraded performance ou partial feature loss com workarounds disponível.

Examples:

• Elevated latency (2-3x normal) on non-critical endpoints
• Background job processing delayed
• Non-critical third-party integration down
• relatório generation slow but functional

P3 -- Low

Definition: Minor issue com minimal utilizador impacto. Workaround exists ou issue is cosmetic.

Examples:

• UI rendering glitch in edge case
• Non-critical cron job failed (will retry)
• Slow dashboard load para internal tool
• Minor logging error that does não affect functionality

severidade decisão Tree

Is dados being lost ou corrupted?
├─ Yes → P0
└─ No
Is there a security breach?
├─ Yes → P0
└─ No
Is o primary service completely down?
├─ Yes → P0
└─ No
Is a major feature broken para many utilizadores?
├─ Yes → P1
└─ No
Is performance significantly degraded?
├─ Yes → P2
└─ No → P3

Cascade Prevention

Circuit Breakers

Automatically stop calling a failing dependency para prevent cascading failure.

Implementation checklist:

• cada external dependency has a circuit breaker
• Failure thresholds are tuned per dependency (não one-size-fits-todos)
• OPEN declarar returns a meaningful fallback (cached dados, degraded resposta, error)
• HALF-OPEN probes are lightweight (health verificar, não full pedido)
• Circuit breaker declarar is observable (métricas, dashboard)
• Alerts fire when a circuit breaker opens

Configuration template:

Dependency: [service nomear]
Failure threshold: [N] failures in [T] seconds
Reset timeout: [T] seconds
Fallback: [cached resposta | error mensagem | degraded mode]

Bulkhead Isolation

Partition resources so failure in one area cannot exhaust resources para another.

Patterns:

• Thread pool isolation: Separate thread pools per dependency
• Connection pool isolation: Dedicated connection pools per downstream service
• processo isolation: Critical e non-critical workloads in separate processos
• Infrastructure isolation: Separate clusters para critical vs batch workloads

checklist:

• Critical path dependencies have dedicated resource pools
• Non-critical background work cannot starve critical pedido handling
• Resource limits are set per pool (max connections, max threads)
• Pool exhaustion gatilhos alerts, não silent queuing

Load Shedding

Intentionally drop low-prioridade work para preserve capacity para high-prioridade traffic.

prioridade tiers:

Implementation:

• usar pedido prioridade headers ou path-based classification
• Return 503 com Retry-depois header para shed pedidos
• monitorizar shed taxa as a métrica (shedding > 0 is an alert)

Graceful Degradation Strategies

Communication Protocols

Status página Updates

template para status página entry:

[TIMESTAMP] - [STATUS: Investigating | Identified | Monitoring | Resolved]

impacto: [brief description de utilizador-visible impacto]
atual status: [What we know e what we're doing]
próximo update: [When para expect o próximo update]

Update cadence:

• P0: cada 15 minutes until resolved
• P1: cada 30 minutes until resolved
• P2: At start e resolução
• P3: At resolução only

Stakeholder Notification template

Subject: [P0/P1] [Service] - [brief impacto description]

severidade: P[0-3]
Start time: [ISO 8601 timestamp]
impacto: [Who is affected e how]
atual status: [What we know]
ações taken: [What we've done so far]
ETA: [If known, otherwise "investigating"]
próximo update: [When]
incidente commander: [nomear]
War room: [Link/channel]

Internal Communication Rules

1. One fonte de truth: todos updates go through o incidente channel, não DMs
2. Facts, não speculation: Share what você know, assinalar what você suspect
3. Timestamp everything: cada ação e observation gets a timestamp
4. No blame: Focus on what happened, não who caused it
5. claro handoffs: When rotating, explicitly entregar off contexto

escalamento Paths

escalamento gatilhos

escalamento checklist

• Primary on-call paged e acknowledged
• If no acknowledgment in 5 min, secondary on-call paged
• incidente commander assigned
• Relevant equipa leads notified
• Status página updated
• cliente suporte briefed com talking points
• executivo stakeholders notified (P0/P1 only)

On-Call Responsibilities

During incidente:

• Acknowledge página dentro de 5 minutes
• Assess severidade e open incidente channel
• Begin investigation e document findings in real time
• Coordinate com other equipas as needed
• Provide status updates at o obrigatório cadence

depois incidente:

• Ensure monitoring confirms resolução
• Draft incidente timeline
• Schedule postmortem if obrigatório
• Update runbooks com qualquer novo learnings
• entregar off para próximo on-call if shift ends during incidente

Postmortem Patterns

Blameless postmortem structure, root cause analysis techniques, ação item tracking, e chaos engenharia patterns. usar depois incidente resolução ou when designing resilience testing programs.

Blameless Postmortem Structure

Core Principles

1. No blame: Focus on systems, processos, e conditions -- não individuals
2. Assume good intent: Everyone involved was doing their best com o information disponível
3. Learn, don't punish: o goal is prevention, não accountability
4. Share widely: Postmortems are organizational learning, não equipa shame

Postmortem Document template

# incidente Postmortem: [Title]

**Date:** [incidente date]
**severidade:** P[0-3]
**Duration:** [Start time] para [End time] ([total duration])
**incidente Commander:** [nomear]
**Author:** [nomear]
**Status:** [Draft | rever | Final]

## Summary

[1-2 sentence description de what happened e o utilizador impacto]

## impacto

- **utilizadores affected:** [Number ou percentage]
- **Duration:** [How long utilizadores experienced o issue]
- **Revenue impacto:** [If applicable]
- **dados impacto:** [qualquer dados loss ou corruption]
- **SLA impacto:** [qualquer SLA violations]

## Timeline

todos times in [timezone].

| Time | evento |
|------|-------|
| HH:MM | [primeiro sinal / alert fired] |
| HH:MM | [On-call acknowledged] |
| HH:MM | [severidade classified as P_] |
| HH:MM | [Key investigation finding] |
| HH:MM | [mitigação applied] |
| HH:MM | [Issue confirmed resolved] |
| HH:MM | [Monitoring confirmed stable] |

## Root Cause

[Detailed description do root cause. What condition ou change led para o failure?]

## Contributing Factors

- [Factor 1: e.g., em falta monitoring para this failure mode]
- [Factor 2: e.g., deployment during high-traffic period]
- [Factor 3: e.g., no automatizado rollback configured]

## What Went Well

- [Thing 1: e.g., alert fired dentro de 2 minutes de impacto]
- [Thing 2: e.g., equipa coordinated effectively in war room]
- [Thing 3: e.g., rollback was smooth e fast]

## What Went Poorly

- [Thing 1: e.g., took 20 minutes para identify o failing service]
- [Thing 2: e.g., no runbook existed para this failure mode]
- [Thing 3: e.g., status página was não updated para 30 minutes]

## ação Items

| ID | ação | responsável | prioridade | Due Date | Status |
|----|--------|-------|----------|----------|--------|
| 1 | [ação description] | [nomear] | P1 | [Date] | Open |
| 2 | [ação description] | [nomear] | P2 | [Date] | Open |

## Lessons Learned

[Key takeaways that deve inform future design, processo, ou tooling decisões]

Postmortem Meeting Facilitation

antes o meeting:

• Draft o postmortem document e share 24 hours in advance
• todos participants rever o timeline para precisão
• incidente commander prepara o root cause analysis

During o meeting (60-90 min):

1. Timeline rever (15 min): Walk through eventos, correct errors, fill lacunas
2. Root cause discussion (20 min): Apply 5 Whys ou fishbone analysis
3. Contributing factors (15 min): What made o incidente worse ou harder para resolve?
4. What went well (10 min): Reinforce effective practices
5. ação items (20 min): Define concrete, assignable, time-bounded ações

depois o meeting:

• Finalize o document dentro de 24 hours
• Distribute para o broader organization
• Enter ação items no tracking system
• Schedule follow-up rever para ação item completion

Root Cause Analysis Techniques

5 Whys

Repeatedly ask "why" para drill past symptoms para o underlying cause.

Example:

Problem: utilizadores received duplicate order confirmation emails.

Why 1: o email service sent o confirmation twice.
Why 2: o order completion evento was published twice.
Why 3: o order service retried depois a timeout.
Why 4: o mensagem broker acknowledged slowly under load.
Why 5: o broker's disk was 95% full, causing write delays.

Root cause: No disk usage monitoring ou alerting on o mensagem broker.
ação: Add disk usage alerting at 80% threshold + auto-scaling.

Guidelines:

• Stop when você reach a systemic cause você can fix (processo, tooling, design)
• Do não stop at "human error" -- ask why o system allowed o error
• Some incidentes have multiple root causes; run 5 Whys para each branch
• Answers deve be factual, não speculative

Fishbone Diagram (Ishikawa)

Categorize contributing factors across standard dimensions.

┌─ People: On-call unfamiliar com service
├─ processo: No rollback runbook existed
Duplicate emails ───├─ Technology: No idempotency on email sends
├─ Environment: Broker disk at 95%
├─ Monitoring: No disk usage alerts
└─ External: Upstream traffic spike

Standard categories:

• People: Knowledge lacunas, staffing, communication
• processo: em falta runbooks, pouco claro procedures, aprovação estrangulamentos
• Technology: bugs, em falta features, architectural lacunas
• Environment: Infrastructure, capacity, configuration
• Monitoring: em falta alerts, incorrect thresholds, observability lacunas
• External: Third-party outages, traffic spikes, attacks

Fault Tree Analysis

Work backward a partir do failure para identify todos possible causes.

Top evento: Service outage
├── AND: Load balancer failure
│ ├── OR: Config error
│ └── OR: Health verificar misconfigured
└── AND: No failover triggered
├── OR: Failover não configured
└── OR: Failover health verificar also failed

When para usar: Complex incidentes com multiple interacting failures where 5 Whys is insufficient.

ação Item Tracking

ação Item qualidade Criteria

cada ação item devem be:

• Specific: claro description de what para do (não "improve monitoring")
• Assignable: One responsável, não a equipa
• Time-bounded: Due date, não "when we get para it"
• Verifiable: claro definition de done
• Prioritized: P1 (antes próximo on-call rotation), P2 (this sprint), P3 (this quarter)

ação Item Categories

Tracking e Follow-Up

• Enter todos ação items no equipa's issue tracker immediately
• Tag com ¤KEEP0¤ e incidente ID para traceability
• rever open postmortem ação items semanal in equipa standup
• Escalate overdue P1 items para engenharia manager
• Close ação items only when verified complete (não just "code merged")

ação Item Anti-Patterns

Chaos engenharia Patterns

Fault Injection

Intentionally introduce failures para verify resilience.

Common fault types:

Fault Injection checklist

• Hypothesis defined: "We believe [X] will happen when [fault]"
• Blast radius limited (single instance, canary, staging)
• Rollback mechanism pronto (kill switch para o experiment)
• Monitoring in place para observe o effect
• equipa is aware o experiment is running
• Abort criteria defined (stop if real utilizador impacto exceeds N%)

Game Days

Structured exercises where equipas practice incidente resposta against simulated failures.

Game Day Planning template:

## Game Day: [Title]

**Date:** [Date e time]
**Duration:** [Expected duration]
**Facilitator:** [nomear]
**Participants:** [equipa members]

### cenário
[Description do simulated incidente]

### objetivos
- [ ] validar alerting deteta o failure dentro de [N] minutes
- [ ] validar equipa can triagem para correct severidade
- [ ] validar mitigação can be applied dentro de [N] minutes
- [ ] validar communication protocols are followed

### Ground Rules
- This is practice, não evaluation
- Facilitator controls o cenário progression
- Anyone can call "stop" if real production impacto is detected
- Document todos observations in real time

### Debrief Questions
1. Did alerts fire as expected?
2. Was o right equipa engaged quickly enough?
3. Were runbooks adequate?
4. What would we do differently in a real incidente?

Game day cadence:

• trimestral para critical services
• depois major architecture changes
• When onboarding novo on-call engineers
• depois qualquer P0 incidente (test o fixes)

Resilience Testing Methodology

Resilience Maturity Levels

Resilience Testing checklist

para each critical service, validar:

• Single instance failure: Service recovers when one instance dies
• Dependency timeout: Service handles slow dependencies gracefully
• Dependency outage: Service degrades (não crashes) when dependency is down
• Network partition: Service handles split-brain cenários
• Load spike: Service sheds load ou scales under 3x normal traffic
• Disk full: Service alerts e degrades antes crashing
• Configuration error: Service fails fast com claro error on bad config
• Rollback: Previous version can be deployed dentro de 5 minutes
• dados corruption: Backup restore has been tested dentro do último quarter

Steady-declarar Hypothesis

antes running qualquer chaos experiment, define what "normal" looks like:

Steady declarar:
- pedido success taxa > 99.9%
- p99 latency < 200ms
- Error taxa < 0.1%
- No alerts firing

Experiment: Kill 1 de 3 service instances

Hypothesis: Steady declarar métricas remain dentro de 10% de baseline
dentro de 60 seconds do fault injection.

Abort if: Error taxa > 5% para more than 30 seconds.