War damage to data centers exceeded what AWS services are designed to withstand.

Half a year after Iranian drone strikes knocked out multiple Amazon data centers, the US tech company has acknowledged the permanent loss of some customer data that was hosted in Bahrain and the United Arab Emirates.

Amazon Web Services was “unable to restore access to the resources and data” hosted in some of the war-damaged data centers, according to an AWS dashboard update posted on September 15. The development was first reported by Reuters and suggests that the Iranian strikes inflicted catastrophic damage on the data centers.

Customer data was irretrievably lost in one of three AWS availability zones in the United Arab Emirates region, specifically the mec1-az2 availability zone. Each availability zone is serviced by one or more Amazon data centers.

you are viewing a single comment's thread
view the rest of the comments
[–] 56 points 1 day ago (13 children)

Anyone in IT who has ever been near a backup or had to do a DR exercise knows full well not to have a backup in the same geographic area as your primary workload. Usually it’s for natural disasters, but it very much applies to war too.

This is just as much, if not more, on the companies that didn’t have geographicly dispersed backups.

  • source
  • hideshow 13 child comments
  • [–] 6 points 1 day ago (9 children)

    I think pretty much every software dev has used AWS at some time, and when you set up new services, it specifically warns you to use multiple geographic areas for your services. It doesn't say "war" or something, only that it's good for redundancy, IIRC.

    So anyway, yes, I follow instructions, but I'm always surprised that you have software people and systems admins who can't even follow basic instructions. I don't know what percentage of people it is, but it feels like over 50% who can't follow even the simplest instructions.

  • source
  • parent
  • hideshow 9 child comments
  • [–] 6 points 23 hours ago (8 children)

    I'm not surprised at all. It costs more to use more regions, both due to literal cost and in time to set it up.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 2 points 23 hours ago (7 children)

    The extra time to set up in more regions is pretty trivial, at least in my experience. It's either an option when setting up the service, or it's something that you need to cover in any case while documenting how to set up the service. But I suspect the devs I'm complaining don't read basic instructions also don't write basic instructions.

    As for the monetary cost, that's a business decision. The devs I am complaining about are not making business decisions. They're just ignoring basic instructions.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 1 point 22 hours ago (6 children)

    I haven't used aws for ages but "an option when setting up the service" does not encompass my recollection of it: you can set up (say) S3 in multiple regions, but that doesn't mean your application can actually make use of multiple regions. It doesn't mean your vlans actually enable anything to talk between the regions.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 1 point 22 hours ago (5 children)

    I haven't used it for a while, either, but it was just an option sitting there when using the AWS UI to set up RDS, IIRC.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 1 point 22 hours ago (4 children)

    Right, but how is your application going to handle it if you enable it?

  • source
  • parent
  • hideshow 4 child comments
  • [–] 1 point 22 hours ago (3 children)

    You have to set up your application to use the service, regardless, so I would say that you just have to follow the instructions.

    I am not talking about fixing after a service has been set up for only one region, but about following the instructions when first setting it up.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 22 hours ago (2 children)

    Yes, but the application setup is more complicated if it can talk to more than one region.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 21 hours ago (1 child)

    The application setup to have the ability to talk to more than one region is the normal setup that you get when you follow instructions. You're not saving time to set it up a different way. You're simply creating technical debt for the person who has to come around and set it up in the standard way in the future. And it probably takes you longer to set up services locked into a single region since for the other way, you just have to follow instructions.

  • source
  • parent
  • hideshow 1 child comment