War damage to data centers exceeded what AWS services are designed to withstand.

Half a year after Iranian drone strikes knocked out multiple Amazon data centers, the US tech company has acknowledged the permanent loss of some customer data that was hosted in Bahrain and the United Arab Emirates.

Amazon Web Services was “unable to restore access to the resources and data” hosted in some of the war-damaged data centers, according to an AWS dashboard update posted on September 15. The development was first reported by Reuters and suggests that the Iranian strikes inflicted catastrophic damage on the data centers.

Customer data was irretrievably lost in one of three AWS availability zones in the United Arab Emirates region, specifically the mec1-az2 availability zone. Each availability zone is serviced by one or more Amazon data centers.

you are viewing a single comment's thread
view the rest of the comments
[–] 10 points 11 hours ago (3 children)

I've been working on Disaster Recovery for my company and the amount of sleep I'm losing on this is insane in general.

Our devs just write their own IaC stuff so like, they can provision their own databases and shit without us knowing with no backups or customer keys and stuff to back up properly.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 11 points 11 hours ago

    It's incredibly hard to reign that behavior in once it's entrenched as well.

    You basically gotta take away privileges from everyone and it becomes a huge fight if you don't have organizational support. Usually you have to wait for something to explode spectacularly for that to happen.

  • source
  • parent
  • [–] 5 points 9 hours ago

    Fellow SRE here.

    The number of times I've heard "we have replication" followed by "we've tested it...on paper" make me cry.

    Trying to convince a CEO that the only way to know for sure that their backups work is to do an actual disaster test is harder than trying to convince my parents that I don't need to eat more.

  • source
  • parent
  • [–] 5 points 11 hours ago*

    Our devs just write their own IaC stuff so like, they can provision their own databases and shit without us knowing with no backups or customer keys and stuff to back up properly.

    That'd give me nightmares too on top of the DR, hopefully at the minimum they can't spin up resources in anything but a dev/test environment.

    Best of luck to you, governance is super important, if it makes you feel any better you're definitely not alone.

  • source
  • parent