Metrics to measure Reliability

• MTTF (Mean Time To Failure):
  • Time from the begining of deployment till the first failure is reported
• MTBF (Mean Time Between Failures):
  • Time between Two Failures
• MTTR (Mean Time To Repair):
  • Time Taken to repair (resolve the incident)

Post-Mortem

• Blameless
• Every Person involved in Incident Respose will have to prepare this document
• Initial Draft of this document will/can be from Incident Commander
• In Post-mortem unlike RCA (Root Cause Analysis), document can speak about multiple factors contribute the Error.
• All the actions taken, what can be done to eliminate the same problem from reocurrance/
• Sample Postmortem Doc

Ansible and Chef

• Enable Debug Logs
• Integrate Ansible/Chef Logs to Monitoring System
• Production Deployments should be Canary

Docker, Kubernetes

• Enable Logging Drivers
• Change K8s deployment strategy to be Canary

Monitoring

• Make monitoring Systems Observations