Yeah, those gigabyte sized access logs is something we have seen for weeks.

Anubis has also gotten largely ineffective, and yesterday I had some time to sample some of the access log IPs and it is clear that this is a coordinated scraping attack from the Huawei ASN that additionally employs residential proxies mostly in Pakistan, Vietnam and Brazil.

I don't really have a good idea how to deal with it, but it is a complete PITA.

All hail flamingo-cant!

Well done! We knew you would come through, honest!

On the scraping.... That's crazy. Is there anything that can be done to protect things?

Phew! I was worried for a bit.

I notice that feddit.org use Anubis for bot mitigation. Is this something we do too?

Yikes sounds like a nightmare. I hope you managed to get some sleep!

Those logs are insane! 1.6GB compressed?! Any idea what the load/throughput was like during that time?

Thank you for your efforts to fix it!

Just wondering, is there a backup place where things like this can be communicated? Blue sky or mastodon for example?

See this is why I hate working in the IT department, because it involves using computers, and they suck.

What's the database underneath lemmy? Is there a transactional option so that it fails safe in the event of unexpected shut down?

Also thank you very much for fixing.

Awesome work, flamingos-cant. Appreciate all you do for us.

Thanks for all your hard work!

Thank you for fixing it and keeping things going :)

Thanks Flamingos!

Glad to see we are back. Those access logs are crazy.

I’m sure you will fine a lot of Chrome/12X, Chrome/13X and Chrome/14X user agents there. Seems to be the favourites of attackers/scrapers recently.

On another note, do the server has a profile somewhere like mastodon or something we could check in case it goes down in future.

Thank you so much for the work you do!

Great job. Although I think my eyes have been widened as to how big the fediverse actually is... I found all sorts of stuff that I still don't understand, communities I didn't know existed, tools that I don't know how to use and a lot of communities that have warnings i can't get past. Along with a whole heap of other stuff that I don't know how to get into.

Thanks for fixing it

Do you have anything in place for detecting and mitigating DOS attacks? I wonder if you were to grep the logs by IP address you'd see the offending IP addresses. Though if you're using a load balancer or some other kind of proxy and don't have IP address forwarding configured, all requests might come through as the same IP address.

some data got duplicated in random tables that shouldn’t be

I'd also check that there's plenty of disk space spare, given that the log files are so large, as I've seen environments start to behave unusually once disk space has ran out, including SSH locking up.

Rotate your logs!!!