<b>Event Summary (times in PDT):</b></br>Date of incident - July 21, 2017</br>Time event began - 09:41 AM</br>Project services restored - 10:18 AM</br>Final resolution - 10:40 AM</br>

<b>Causes</b></br>Root cause - Vexxhost had an issue with their Ceph cluster during the upgrade of one the storage nodes which resulted in blocked I/O for a period of time.

There was no action that LF staff could take to aid in the resolution of the Vexxhost issues besides asking the service provider for regular updates so that we could share them with our projects.</br>

<b>Effects</b></br>Downtime for projects: This outage primarily affects those collaborative projects using Jenkins and Nexus CI systems. The following project systems were down during this time:

<ul>

<li>AGL - Jenkins</li>

<li>EdgeX - Jenkins and Nexus</li>

<li>Hyperledger - Jenkins and Nexus</li>

<li>FD.io - Gerrit, Jenkins, JIRA, Nexus, OpenGrok and Sonar</li>

<li>ONAP - Jenkins and Nexus</li>

</ul>

<b>Timeline</b></br>From Vexxhost:

At around 12PM ET, we were conducting maintenance which involvedupgrading our Ceph cluster from 10.2.3 to 10.2.7. This is typicallynot service impacting and we've done it quite a few times across thelifetime of the 10.2.X release with no issues.

As per the Ceph documentation, it recommends that the Ceph monitorsare upgraded first then the Ceph OSDs. This is the order that we'vealways done which has always worked, which was previously tested inthe testing and staging environments.

Upon upgrading all 3 monitors, all OSDs started flapping which meantthat it was the equivalent of the OSD services going up and down.This resulted in the placement groups (which you could call 'buckets'that contain data) go out of active state and not go back into it,which meant that no I/O would be read or written to the cluster.

Upon investigation, we discovered that there was a CRC error which wasoccurring on all OSDs. In the 10.2.4 release, a flag in the osdmap(a map which includes all the OSDs of the cluster, shared across allOSDs) called require_jewel_osds. With the monitors running 10.2.7and the OSDs running 10.2.3, the CRC errors were coming up because theOSD was not able to properly validate the osdmap.

However, this is just a warning but not an error. In our staging andtest environments, we did not see this causing any problems. However,in an environment which was much larger such as our production, thesewarning messages were being logged to the monitor processes whichmeant that they were receiving thousands of messages per second in acomponent that is not designed to handle that much messages.

Once we investigated the issue, we realized there is tuneable which bydefault is set to true to log all messages to the monitor. We alreadyhave centralized logging in place so it was unnecessary so once wepushed out the config to disable it, the monitors are able to do theirjobs again and placement groups started going into active mode again.

At the moment, the cluster is currently undergoing a deep scrubbecause of the fact that placement groups went down and back up. Thisis not affecting on the customer side and on any performance and weexpect the cluster to go back in a healthy state by end of the day ET.

In order to avoid this from happening again, we're going to addsimulated large number of OSDs during upgrades as part of our CI whichwill help us catch issues like this before happening in the future.The service was backup and running at around 1:30 PM ET and by 3:00 PMET, the cluster was in a state that was not causing any slow requests.

We apologize once again for the inconvenience, we know that you dependand rely on our service and we're going to be doing our best movingforward to avoid this from happening again in the future. We aim tobe fully transparent with our customers and please let me know if youhave any other questions at all.

We have received an update from our provider:

They had an issue with their Ceph cluster during the upgrade of one the storage nodes which resulted in blocked I/O for a period of time.Everything should be accessible once again now. They will be updating with a post mortem shortly regarding what happened.

The outage appears to be resolved. We are waiting for an update from Vexxhost on the cause of the outage and will update as soon as we have the information.

Our access to Vexxhost VMs appears to be coming back online. We are continuing to monitor the situation and will provide an update as soon as we have more information. Services should be coming back online at this time.

The incident is ongoing and we are waiting for more information from our provider. We will update again at 10:30am (PDT) or sooner if we learn any additional information.