PDX Edge Firewall outage

Incident Report for The Linux Foundation

Resolved

This incident has been resolved.

Posted May 04, 2018 - 21:51 UTC

Monitoring

All firewall services have been restored. A timeout caused the cluster to become unhealthy, but also blocked the failover event from succeeding. The split-brain has been resolved and we are monitoring for further issues.

Posted May 04, 2018 - 18:47 UTC

Identified

The firewall cluster is in a split-brain, with services failing to start properly on one of the nodes. We resolving the split to bring services back online as quickly as possible.

Posted May 04, 2018 - 18:29 UTC

Investigating

We are investigating a service failure on our PDX edge firewall systems and are working to restore services as quickly as possible. This affects all services in our Portland location.

Posted May 04, 2018 - 18:10 UTC