War damage to data centers exceeded what AWS services are designed to withstand.

Half a year after Iranian drone strikes knocked out multiple Amazon data centers, the US tech company has acknowledged the permanent loss of some customer data that was hosted in Bahrain and the United Arab Emirates.

Amazon Web Services was “unable to restore access to the resources and data” hosted in some of the war-damaged data centers, according to an AWS dashboard update posted on September 15. The development was first reported by Reuters and suggests that the Iranian strikes inflicted catastrophic damage on the data centers.

Customer data was irretrievably lost in one of three AWS availability zones in the United Arab Emirates region, specifically the mec1-az2 availability zone. Each availability zone is serviced by one or more Amazon data centers.

top 50 comments

sorted by: hot top controversial new old
[–] 1 point 23 minutes ago

kinda surprising it didn't happen earlier

  • source
  • [–] 42 points 13 hours ago (2 children)

    "We don't need our own backups. The cloud takes care of everything."

  • source
  • hideshow 2 child comments
  • load more comments (1 reply)
    [–] 20 points 14 hours ago (3 children)

    Do they have a Go Fund Me or something I can donate to or something to keep them rolling?

  • source
  • hideshow 3 child comments
  • [–] 9 points 13 hours ago (2 children)

    Neither Amazon or Islamic Republic of Iran need your money, friend. Buy yourself a nice hat in stead.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 6 points 11 hours ago

    So like... the opposite of a data breach, the headlines I am used to reading about "customer data"? Hmmm... Ok.

  • source
  • [–] 150 points 20 hours ago*

    Me: "please delete my data"
    Amazon: "Lol no"
    Iran "I got you babe"

  • source
  • [–] 201 points 21 hours ago (1 child)

    The only way to get them to delete your data.

  • source
  • hideshow 1 child comment
  • [–] 8 points 12 hours ago

    Oh no, not amazons marketing data!

  • source
  • [–] 46 points 18 hours ago (5 children)

    At this point, I wouldn't be surprised if a bombing on US territory was cheered if it was an AWS data center (or Meta, xAI, etc). Hell, Iran could probably even find US citizens for sabotage at this point if they just ask, lol.

  • source
  • hideshow 5 child comments
  • [–] 5 points 12 hours ago

    I expect Iran already hires Americans for unorganized sabotage, I know both Iran and Russia utilises Telegram to hire teenagers "torpedoes" as they're called among mobsters.

    It's a fully gangster technique, but it's effective. One example would be when the Israeli embassy in Copenhagen had two kids attempt to throw a grenade their way in 2024 - It's proven to be related to Foxtrot (the largest Swedish organised crime syndicate), which Iran has influence on through one of the leaders living in Iran

  • source
  • parent
  • load more comments (4 replies)
    [–] 157 points 21 hours ago (5 children)

    the cloud is much more secure because your data is decentralized

    • Every major cloud provider when advertising their services over on-prem soloutions.
  • source
  • hideshow 5 child comments
  • [–] 62 points 21 hours ago* (2 children)

    * But only if you pay for the most redundant storage, which cost all the money.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 6 points 12 hours ago
    [–] 55 points 21 hours ago (6 children)

    Never thought I'd be a cheerleader for Iran against my own country. But please, Iran, keep making America better!

  • source
  • hideshow 6 child comments
  • load more comments (6 replies)
    [–] 52 points 21 hours ago (12 children)

    Anyone in IT who has ever been near a backup or had to do a DR exercise knows full well not to have a backup in the same geographic area as your primary workload. Usually it’s for natural disasters, but it very much applies to war too.

    This is just as much, if not more, on the companies that didn’t have geographicly dispersed backups.

  • source
  • hideshow 12 child comments
  • [–] 5 points 15 hours ago (9 children)

    I think pretty much every software dev has used AWS at some time, and when you set up new services, it specifically warns you to use multiple geographic areas for your services. It doesn't say "war" or something, only that it's good for redundancy, IIRC.

    So anyway, yes, I follow instructions, but I'm always surprised that you have software people and systems admins who can't even follow basic instructions. I don't know what percentage of people it is, but it feels like over 50% who can't follow even the simplest instructions.

  • source
  • parent
  • hideshow 9 child comments
  • [–] 5 points 14 hours ago (8 children)

    I'm not surprised at all. It costs more to use more regions, both due to literal cost and in time to set it up.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 2 points 14 hours ago (7 children)

    The extra time to set up in more regions is pretty trivial, at least in my experience. It's either an option when setting up the service, or it's something that you need to cover in any case while documenting how to set up the service. But I suspect the devs I'm complaining don't read basic instructions also don't write basic instructions.

    As for the monetary cost, that's a business decision. The devs I am complaining about are not making business decisions. They're just ignoring basic instructions.

  • source
  • parent
  • hideshow 7 child comments
  • load more comments (7 replies)
  • [–] 35 points 19 hours ago (4 children)

    Speaking as a member of a company who has just recently realized that we can have products in regions other than us-east-1, this is not surprising. If UE1 went down the economy would collapse overnight

  • source
  • hideshow 4 child comments
  • [–] 11 points 18 hours ago (3 children)

    I've been working on Disaster Recovery for my company and the amount of sleep I'm losing on this is insane in general.

    Our devs just write their own IaC stuff so like, they can provision their own databases and shit without us knowing with no backups or customer keys and stuff to back up properly.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 11 points 18 hours ago

    It's incredibly hard to reign that behavior in once it's entrenched as well.

    You basically gotta take away privileges from everyone and it becomes a huge fight if you don't have organizational support. Usually you have to wait for something to explode spectacularly for that to happen.

  • source
  • parent
  • [–] 5 points 16 hours ago

    Fellow SRE here.

    The number of times I've heard "we have replication" followed by "we've tested it...on paper" make me cry.

    Trying to convince a CEO that the only way to know for sure that their backups work is to do an actual disaster test is harder than trying to convince my parents that I don't need to eat more.

  • source
  • parent
  • [–] 5 points 18 hours ago*

    Our devs just write their own IaC stuff so like, they can provision their own databases and shit without us knowing with no backups or customer keys and stuff to back up properly.

    That'd give me nightmares too on top of the DR, hopefully at the minimum they can't spin up resources in anything but a dev/test environment.

    Best of luck to you, governance is super important, if it makes you feel any better you're definitely not alone.

  • source
  • parent
  • [–] 11 points 16 hours ago
    [–] 35 points 20 hours ago (1 child)

    War damage to data centers exceeded what AWS services are designed to withstand.

    Well good thing it isn't a war then

  • source
  • hideshow 1 child comment
  • [–] 2 points 10 hours ago

    They are intentionally designed to withstand a little bit of "war damage"?

    Like an Amazon data warehouse construction guy needs to specifically add a little bit of extra cement or some shit to meet a war damage protection quota?

  • source
  • parent
  • [–] 5 points 14 hours ago

    good. more. do more.

  • source
  • [–] 11 points 18 hours ago

    AWS makes so much cash that they won't even leave the region

  • source
  • [–] 8 points 17 hours ago

    Do it again. Harder!

  • source
  • [–] 18 points 21 hours ago

    Iran out here doing everyone a favor

  • source
  • [–] 17 points 21 hours ago

    2+1 isn't this building and the neighboring building. No back up no compassion. (Kein Backup kein Mitleid)

  • source
  • [–] 3 points 16 hours ago

    Based based based based

  • source
  • [–] 11 points 21 hours ago
    [–] 9 points 20 hours ago

    Someone needs to shine a laser at bezos' noggin for a better permanent loss.

  • source
  • [–] 8 points 20 hours ago

    Making me wanna Inshallah up

  • source
  • load more comments
    view more: next ›