373
When you're high level, people notice you. Shock! (ttrpg.network)
submitted 2 years ago by to c/rpgmemes@ttrpg.network
51 comments fedilink hide all child comments
 

This comic follows on from the Previous comic which will almost certainly provide context.

You might not wanna be famous, but when you're level 10, every organization within a mile is watching what you're doing.

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 40 points 2 years ago (2 children)

most of these AI scrapers don't respect robots.txt, so I'm not sure that really helps much, but... we have tried doing all of these things.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 27 points 2 years ago (1 child)

    Someone on lemmy suggested to create a dummy endpoint that normal people won't be able to navigate to, and disallow it in robots.txt

    Then when somebody crawls it you know they are ignoring robots.txt, and you ip ban them

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 16 points 2 years ago (3 children)

    That's pretty clever.

    I think that these AI scrapers might be smart enough that this doesn't really work though - at least if I were designing them I'd have them all come from dynamic IPs and not have any of them bother hitting the same target more than once. These things are very dedicated to acquiring content without consent, and if they're capable of causing problems for (say) Reddit, I'm not sure my little website is going to have much luck deterring them.

    Honestly a better strategy might be to just glaze everything I draw.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 8 points 2 years ago (1 child)

    I am not sure if it costs money, but you could implement captchas.

    Or use cloudflare to do that bot detecting for you.

    Worst case you make it so you need to create an account to see content.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 5 points 2 years ago

    Well, we are already using cloudflare, that's one of the other reasons why the site is so slow... I don't think the other two suggestions prevent a scraper from requesting the information from the server... I think they'd just make it more arduous for real people to access the content.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 2 years ago (1 child)

    Instead of a tech solution, why not a legal one? Place somewhere in the website that refusal to follow your robots.txt is agreement to pay you x amount of money for your content. Then combine that with the dummy page solution the other person brought up so you can record the IP address, then take them to court so they pay you. Has potential to bring you a really really nice chunk of money.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •