[RCA]: GovCloud Userconsole Not Accessible 

2021-03-24

Summary

Impact

Inability to login to GovCloud cloud console/create new user registrations for 20 hours 30 minutes

Root Cause(s)

1. During release operations a critical database migration was not run. As a result, access to user login and registration was lost.

2. The impact duration was longer than anticipated for the following reasons:

   1. Time-to-detect was 20 hours:

      1. Monitoring and alerting for logins in GovCloud is in need of improvement
      2. Alerts for our monitoring clusters being unavailable indirectly brought attention to the incident.

   2. Time-to-respond was 27 minutes once the issue was detected.

   3. Time-to-remediate was 1 hour 18 minutes

3. Operator error was the primary cause, however the lack of monitoring parity with other production environments is the primary reason for longer TTD.

Resolution Action Items

Running DB migrations restored access without requiring any further restarts.

Timeline

Action Items

Future Work

• Mitigate failure: reduce likelihood of operator error by improving release automation in GovCloud
• Mitigate impact: improve monitoring of login and registration failures for GovCloud