173
What's the worst way you ever broke production? (lemm.ee)
submitted 2 years ago by to c/asklemmy@lemmy.world
103 comments fedilink hide all child comments
 

Fess up. You know it was you.

you are viewing a single comment's thread
view the rest of the comments
[–] 19 points 2 years ago (1 child)

UPDATE ON articles SET status = 0 WHERE body LIKE '%...%'

On master production server, running myisam, against a text column, millions of rows.

This causes queries to stack because table locks

Rather than waiting for the query to finish. a slave was promoted to master.

Lesson: don't trust mysqladmin to not do something bad.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] 2 points 2 years ago

    Table locks can be a real pain. You know you need to do the change, but the system is constantly running queries towards it. Now days it's a bit easier with algorithm=inplace and lock=none, but in the good old days you were on your own. Your only friend was luck. Large migrations like that still gives me shivers

  • permalink
  • fedilink
  • source
  • parent