all 29 comments

sorted by: hot top controversial new old
[–] 105 points 2 years ago* (7 children)

If you can fuck up a database in prod you have a systems problem caused by your boss. Getting fired for that shit would be a blessing because that company sucks ass.

  • source
  • hideshow 7 child comments
  • [–] 51 points 2 years ago (5 children)

    What if you're the one that was in charge of adding safe guards?

  • source
  • parent
  • hideshow 5 child comments
  • [–] 5 points 2 years ago (2 children)

    If you are adding guardrails to production... It's the same story.

    Boss should purchase enough equipment to have a staging environment. Don't touch prod, redeploy everything on a secondary, with the new guardrails, read only export from prod, and cutover services to the secondary when complete.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 10 points 2 years ago

    Small companies often allow devs access to prod DBs. It doesn't change the fact that it's a catastrophically stupid decision, but you often can't do anything about it.

    And of course, when they inevitably fuck up the blame will be on the IT team for not implementing necessary restrictions.

    Frequent snapshots ftmfw.

  • source
  • parent
  • [–] 85 points 2 years ago (4 children)

    I always run my queries in a script that will automatically rollback if the number of rows changed isn't one. If I have to change multiple rows I should probably ask myself what am I doing.

  • source
  • hideshow 4 child comments
  • [–] 36 points 2 years ago* (1 child)

    I always start a session with disabling auto commit (note, I could add it to my settings, but then it would backfire that one time my settings don't execute, so I'm making it a habit to type it out every time, first thing I connect)

    BTW: what kind of genius decides that auto commit should be enabled by default?

  • source
  • parent
  • hideshow 1 child comment
  • [–] 61 points 2 years ago (6 children)

    Don't you people have a development environment?

  • source
  • hideshow 6 child comments
  • [–] 92 points 2 years ago (3 children)

    The P in Prod stands for "It'll be Pfine"

  • source
  • parent
  • hideshow 3 child comments
  • [+] 45 points 2 years ago* (last edited 9 months ago)
    [–] 18 points 2 years ago

    A few months back I crashed a db in prod. I detached it and when I tried to reattach it simply refused, saying it was corrupted or some shit.
    Lucky me we have a backup solution.
    Unfortunately it was being upgraded, with difficulties.
    That was a long day.

  • source
  • [–] 17 points 2 years ago

    The famous onosecond

  • source
  • [–] 15 points 2 years ago

    emails

    Yeah, don't hire that guy.

  • source
  • [–] 11 points 2 years ago

    Makes me think of what happened to gitlab

  • source
  • [–] 6 points 2 years ago (3 children)

    I have several times insisted that a migration be done via an ad hoc endpoint, because I'm a jerk, but also it's much easier then to test, and no one has to yolo connect directly to prod.

  • source
  • hideshow 3 child comments
  • [–] 1 point 2 years ago (2 children)

    Endpoint? Why the fuck is a migration using an endpoint, if you want testability a script will do just fine

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 2 years ago (1 child)

    Because I didn't want someone to yolo connect to production, and we don't have infrastructure in place for running arbitrary scripts against production. An http endpoint takes very little time to write, and let's you take advantage of ci/cd/test infrastructure that's already in place.

    This was for a larger more complicated change. Smaller ones can go in as regular data migrations in source control, but those still go through code review and get deployed to dev before going out.

  • source
  • parent
  • hideshow 1 child comment