We are currently experiencing an outage on all servers located in Kent. This includes primary email, all websites hosted on web2.gbdns.net and web3.gbdns.net.
Update: 00:01 - Service has been restored we are awaiting an RFO.
Update: 11:55 - RFO Below
We suffered a loss of connectivity to the F25 data-centre on the Kent Science Park at around 23:20hrs for just over half an hour. The problems leading up to the loss of service started at around 22:00hrs when the Kent Science Park campus started receiving an Over-Voltage from the national grid of 275V-AC.
The UPS equipment in the F25 data-centre where customer services are located is capable of dealing with an Over-Voltage and has been pulling the voltage down to 230V for the last three hours without issue.
The UPS equipment located in building B300 where the fibre ring terminates with our connectivity provider is also capable of dealing with an over-voltage, but not exceeding 260V-AC. As a result their UPS system switched to battery followed shortly by generator at around 22:00hrs. Unfortunately at 23:20 they suffered a catastrophic failure of a UPS which took out connectivity. Power was restored by isolating the failed UPS from the cluster around half an hour later which restored service.
The campus is still receiving a Over-Voltage of 275V-AC as I write this announcement. As a result KSP should be considered at risk while the national grid are working to restore the correct incoming voltage.
Going forward direct fibre to the F25 Data-centre is currently being dug, which will terminate in Telehouse North. Once this is completed we will no longer have electrical reliance on building B300.
We must apologise for any inconvenience caused this evening and will ensure we do everything possible to prevent a reoccurrence.
An update will be issued once incoming power voltage has returned to normal.
Update: 13:00
Incoming voltage was restored to 230V by the national grid at around 7am this morning. As a result building B300 is now running from mains, so our at-risk period has ended.
In the short-term we have been assured by our connectivity provider that they are currently undergoing investigation with the help of their UPS provider to discover how one of their UPS units failed shortly after their generator started.
Additionally we have been given an estimated completion time of approximately 8 weeks from NEOs who are installing the secondary circuit to Telehouse North. This will remove our electrical reliance on the ring which terminates in B300. This will also give diversity from our Telehouse East core.
