10
Another ddos attack... again. :) (lemmy.today)
submitted 9 months ago by [M] to c/lemmytoday@lemmy.today
23 comments fedilink hide all child comments
 

The attackers found another slow db query to exploit, so had to put some temporary measures in place.

Will post more details later!

This time they actually took down the instance for several hours.

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 1 point 9 months ago (10 children)

Was it like that before the attack as well?

I can look into it and make sure we are federating properly. During an attack or after, it wont work, but it should federate as soon as the attack is stopped.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 10 child comments
    •
  • [–] 2 points 9 months ago* (7 children)

    I've noticed issues with federation tonight, but not before this attack. I just made a bunch of comments and posts and was wondering why none of them were gaining any traction. I checked from other instances and the posts do not appear on them.

    Thanks for all you do!

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] [S] 2 points 9 months ago (5 children)

    Send me a link to a comment or post and what instance you are looking on. :) Thank you!

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 1 point 9 months ago* (last edited 9 months ago) (4 children)

    Sure! For example, I made a bunch of cross posts with this post as the OG: https://lemmy.today/post/42851505 (posted to lemmy.world, but I can't see this post if I navigate to the instance)

    Another example, a cross post of the above to programming.dev: https://lemmy.today/post/42851619 (cannot be found in the instance)

    Then a few comments which haven't seemed to federate:

    I made more than that though, and I'm pretty sure none of them have federated.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] [S] 3 points 9 months ago (2 children)

    I think I figured it out. Has to do with my mitigation of the attack yesterday.

    Now new comments and posts are federating to other instances again, but the old comments and posts, im not sure they will federate.

    Attackers were hammering a certain endpoint to cause the database to be 100% busy in their attack. So i blocked that endpoint, thinking it would not have an effect on federation. Chat gpt also agreed that it wouldnt.

    Chat gpt was wrong. :) So I changed from blocking it to caching it, which is at least going to help prevent further attacks to a degree.

    Thanks for letting me know!

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 9 months ago* (1 child)

    Thanks for the fix! I've reposted some of the content and it seems that federation is working on these. Hopefully I don't end up spamming by accident!

    Edit: it seems like I could have just edited the posts and forced federation that way. TIL!

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 2 points 9 months ago*

    Im just looking at new comments or posts in those instances and it all seems to be delayed from the entire fediverse, not just lemmy.today:

    https://programming.dev/c/opensource?dataType=Post&sort=New (last comment 13 hours ago, last post 17 hours ago)

    https://lemmy.world/c/foss?dataType=Comment&sort=New (last comment 18 days ago, last post 18 days ago)

    https://lemmy.zip/c/technology?dataType=Post&sort=New (last comment 5 hours ago, last post 2 hours ago)

    Lets wait and see when these instance communities gets new posts and comments and see if the ones from lemmy.today are still missing. Will keep an eye on them. :)

  • permalink
  • fedilink
  • source
  • parent
  • [–] [S] 1 point 9 months ago* (last edited 9 months ago)

    I see some comments not being federated yet, here is one example:

    https://lemmy.today/post/42824004 (112 comments, last comment 1 hour ago)

    https://lemmy.world/post/39592788 ( 108 comments, last comment 3 hours ago)

    https://sh.itjust.works/post/50825191 (105 comments, last comment 3 hours ago)

    In this example, lemmy.world and sh.itjust.works has less comments than lemmy.today on the same post.

    Normally comments show up right away so perhaps this is a sign of a problem. Will keep looking at it during the day.

  • permalink
  • fedilink
  • source
  • parent