Service Outage on 15 November
24th November 2016
Last week on 15 November there was a power failure at the datacentre where our servers are located (Everest datacentre in Reading, UK). This power failure was widely reported as it affected a number of high-profile internet suppliers. Power failures should not normally result in a disruption to services, as there are multiple redundant systems put in place by the datacentre owners. However, on this occasion, the redundant systems failed.
The power failure caused disruption to customer websites and email services. Our phone lines were also disrupted because the telephony supplier has its systems based in the same datacentre. During the failure, updates were posted to our Twitter account, and, as soon as services had stabalised again, affected customers were contacted by email. We would like to reiterate our apology for the disruption.
Whilst the exact cause of the failure is still being investigated by the datacentre's engineers, we have received an interim statement from our supplier explaining the overall sequence of events and general source of the failure. Below the statement there is a summary in less technical language.
Reading [datacentre] experienced a regional power failure which took the A & B power feeds out from two separate sub-stations at the same time. Ultimately, this regional power failure resulted in problems transitioning from our dual mains supply to backup generator power and as a result not all of the Data Centre successfully transferred to backup power.
This failure to transfer to generator power meant that some of the Data Centre continued on UPS supply and services remained operational. Ultimately the generators were not able to transfer the power load and the timeline of events is as follows:
- 08:12:02 - Primary and secondary high voltage feeds from the grid fail
- 08:12:17 - Start command to emergency diesel generators is automatically sent
- 08:12:56 - Generators have started, synchronised with each other, and providing power to site
- 08:13:31 - Generators have excitation failure as load is stepped on, due to reasons to be fully investigated.
UPS continued to feed the Data Centre and you will have experienced issued from approximately 08:39 GMT.
Our backup generators designed to carry double the total Data Centre power requirement and are tested for load on a monthly basis. We have a series of engineers on site currently investigating why there was a failure in the process of generators accepting the power load and this issue is being resolved as a priority. The engineers have focused on settings in the synchronizing controls and a fix has been implemented. The process of testing this fix is currently being coordinated and thorough testing will be undertaken.
We will be receiving a full ‘Reason For Outage’ report from the engineers and this will provide greater detail.
Mains power was restored to the site around 10:15 GMT and our engineers immediately began the process of restoring full functionality across all devices. Unfortunately for some Servers this process required manual intervention and as a result you have experienced a delay in your services coming online. Please accept our apologies for this delay.
Abhinav Mathur, Commercial Manager, Namesco
To summarise the statement in less technical language, the datacentre has two power feeds from separate National Grid substations. Both of these substations stopped supplying power to the datacentre at the same time due to a regional power cut. The UPSs (Uninterruptable Power Supplies – basically really big batteries) successfully supplied power for the few minutes it took the diesel generators to start. However, once the generators started supplying power, they failed after about half a minute (cause currently unknown).
The UPSs, which are designed to provide power for long enough for the generators to start (i.e. a few minutes), were able to continue supplying power for about half an hour, but eventually ran out. At that point, there was no longer any power supply, and this continued until mains power was restored by National Grid later that morning. Unfortunately our servers – along with many others in the datacentre – did not come back online when power was restored and required manual intervention. Due to the number of servers involved, it took datacentre staff a while to bring all affected servers back online.
We will of course post any additional relevant information we receive.
