291
I do it in production (thelemmy.club)
submitted 13 hours ago by CallMeAl@piefed.zip to c/linuxmemes@lemmy.world
you are viewing a single comment's thread
view the rest of the comments
[-] flambonkscious@sh.itjust.works 5 points 8 hours ago

That sounds like a fun frequent task to introduce!

Get your BoFH on!

[-] kungen@feddit.nu 3 points 8 hours ago

We don't usually do it for fun, but I'm sure my teammate who has petitioned deploying a "chaosmonkey"-thingy to our prod clusters would love it 😁

[-] flambonkscious@sh.itjust.works 2 points 7 hours ago

I really wish we could get to that level of maturity (cries in healthcare)

Not sure if we should take the approach of biting the bullet and forcing it on ourselves or slowly growing to it (under a hostile govt that sacked ~30% of the countries ICT team)

[-] Peffse@lemmy.world 1 points 3 hours ago

back when my team had proper coverage, we used to do chaosmonkey style tests for hosted health systems. We'd schedule an event with the hospital IT staff. Say... 1 hour on a specific day near midnight. The local IT staff would put out a notice that for the hour the system may be inaccessible. Then we'd start a test suite, as ungracefully as possible. We'd run a abort shutdown command on one server, see how RHCS would handle it. Bring it back up stable. Silo a database instance. See how that handled. Yank out the Red1 cable from the switch, see how the network balanced the new traffic over Red2/Blue. That kind of stuff. We caught a few problems that way.

The clients hated it, but it really made the system more bulletproof for when a new recruit ran a command in the wrong window and took down an unrelated resource.

[-] pivot_root@lemmy.world 3 points 8 hours ago* (last edited 8 hours ago)

Your teammate has the right approach here. People will quickly learn to stop relying on the dev environment in production deployments when the environment becomes unreliable.

this post was submitted on 30 Jul 2026
291 points (97.7% liked)

linuxmemes

32259 readers
1673 users here now

Hint: :q!


Sister communities:


Community rules (click to expand)

1. Follow the site-wide rules

2. Be civil
  • Understand the difference between a joke and an insult.
  • Do not harrass or attack users for any reason. This includes using blanket terms, like "every user of thing".
  • Don't get baited into back-and-forth insults. We are not animals.
  • Leave remarks of "peasantry" to the PCMR community. If you dislike an OS/service/application, attack the thing you dislike, not the individuals who use it. Some people may not have a choice.
  • Bigotry of any kind will not be tolerated. This is an LGBTQ+-friendly community -- if that is a problem for you, you should leave.
  • 3. Post Linux-related content
  • Including Unix and BSD.
  • Non-Linux content is acceptable as long as it makes a reference to Linux. For example, the poorly made mockery of sudo in Windows.
  • No porn, no politics, no trolling or ragebaiting.
  • Don't come looking for advice, this is not the right community.
  • 4. No recent reposts
  • Everybody uses Arch btw, can't quit Vim, <loves / tolerates / hates> systemd, and wants to interject for a moment. You can stop now.
  • 5. πŸ‡¬πŸ‡§ Language/язык/Sprache
  • This is primarily an English-speaking community. πŸ‡¬πŸ‡§πŸ‡¦πŸ‡ΊπŸ‡ΊπŸ‡Έ
  • Comments written in other languages are allowed.
  • The substance of a post should be comprehensible for people who only speak English.
  • Titles and post bodies written in other languages will be allowed, but only as long as the above rule is observed.
  • 6. (NEW!) Regarding public figuresWe all have our opinions, and certain public figures can be divisive. Keep in mind that this is a community for memes and light-hearted fun, not for airing grievances or leveling accusations.
  • Keep discussions polite and free of disparagement.
  • We are never in possession of all of the facts. Defamatory comments will not be tolerated.
  • Discussions that get too heated will be locked and offending comments removed.
  • Β 

    Please report posts and comments that break these rules!


    Important: never execute code or follow advice that you don't understand or can't verify, especially here. The word of the day is credibility. This is a meme community -- even the most helpful comments might just be shitposts that can damage your system. Be aware, be smart, don't remove France.

    founded 3 years ago
    MODERATORS