lemmy.world (the normal design) shows the first page without javascript.
The buttons to switch pages don't work without javascript in the new design, but it still shows more than old.lemmy.world currently does.
old.lemmy.world used to have working page switch buttons without javascript.
if you set a cookie with the name jwt it'll bypass the challenge. this is currently needed due to ddos by criminal ai crawler operators
Thank you. Will javascript or this cookie be needed forever now?
we'll have to see. when you're logged in you'll have that cookie anyway.
due to the structure of the page it's a very attractive crawler target for those that don't care about robots.txt and pretend they're real browsers. they're hitting it from a range of different countries that even our initial attempts of limiting the challenge to certain countries was not useful.
even after implementing this challenge we're still getting lots of requests on this domain that all fail the challenge, 42k challenges issued within the last 24h and only 371 (0.89%) solved.
mlmym also doesn't seem to be that efficient with its api call usage per page load, so it would likely also need some investigation there if that can be optimized to reduce the server load from mlmym pages compared to other clients.
criminal ai crawler operators are killing the (public) js-free web.
Thank you for your in depth explanation.
due to the structure of the page it’s a very attractive crawler target
Would you say the old "mlmym" design is more attractive than the new design?
I don't have the historical comparison to fully understand the chart. Are you saying most of the blue requests are bots? Can you estimate how much it would cost to serve all the bot requests, and enable js-free old.lemmy.world? How much a prestige old web tax would be?
Would you say the old “mlmym” design is more attractive than the new design?
anything that is plain html and doesn't need js is more attractive for crawlers than things that aren't plain html.
historical comparison
this doesn't provide much historical context, it's just for the last 7 days
Are you saying most of the blue requests are bots?
one way or another, yes. we definitely don't have that many legitimate users trying to access it and then stopping when they get a cloudflare challenge.
Can you estimate how much it would cost to serve all the bot requests
currently no. this was a quick fix implemented when it got to the point that we couldn't handle the traffic anymore and lemmy.world was getting outages from the load caused by these criminals. the amount of crawler traffic we see also gets spikes here and there, so what might be enough today might not be enough tomorrow. they just don't care about anything but themselves.
Welcome to the official Lemmy.world Support community! Post your issues or questions about Lemmy.world here.
This community is for issues related to the Lemmy World instance only. For Lemmy software requests or bug reports, please go to the Lemmy github page.
This community is subject to the rules defined here for lemmy.world.
To open a support ticket
You can also DM https://lemmy.world/u/lwreport or email [email protected] (PGP Supported) if you need to reach our directly to the admin team.
