Lightning Strikes Kill Customer Data Stored On Google Compute Engine (GCE) Servers

Google has reported four successive lightning strikes have resulted in a small fraction of permanent data loss for its Google Compute Engine (GCE) disks, which allow customers to run cloud-based virtual machines. The incident last Thursday caused power loss to capacity stored in Belgium, resulting in loss for 0.000001 percent of disk space. In a report Tuesday, Google explained some recently written data was on storage systems less equipped for power failures due to extended backup battery drain. “This outage is wholly Google's responsibility,” the company said.

It's not the first time GCE has had issues. In February, GCE experienced a global outage that lasted for nearly two hours affecting businesses that depend on GCE for their day-to-day operations. GCE is seen as a competitor to Amazon AWS and Microsoft Azure for dominance of the cloud, but instances like these will shake consumer confidence in the GCE brand as they look for the most stable cloud services possible. To be sure, AWS and Azure have also had their share of outages.

In a statement to the BBC, project manager Justin Gale, who works for lightning protection service Orion, said: “The cabling alone can be struck anything up to a kilometer away, bring [the shock] back to the data center and fuse everything that's in it.” Gale explained power or phone cables at a distance from a building can still cause disruptions if struck by lightning.

“We would like to take this opportunity to highlight an important reminder for our customers: GCE instances and persistent disks within a zone exist in a single Google data center and are therefore unavoidably vulnerable to data center-scale disasters,” Google said. The company recommends using GCE snapshots and Google Cloud Storage to avoid a similar instance reoccurring.

Google said it is currently undertaking a storage hardware upgrade program that will avoid similar incidents happening in the future, with most of its persistent disk storage already upgraded. The company also identified the need for improvements to response procedures for system engineers, to make sure employees are better equipped to handle future lightning strikes.