this post was submitted on 05 Jul 2023
807 points (98.4% liked)
World News
32378 readers
521 users here now
News from around the world!
Rules:
-
Please only post links to actual news sources, no tabloid sites, etc
-
No NSFW content
-
No hate speech, bigotry, propaganda, etc
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
How does Pinterest get around this then? They pollute image searches like crazy, and require you to login to see anything. At least they did, I blocked them from searches so maybe it's different now.
Easy - detect if you're getting accessed by a search crawler or a human. Serve a full page or just a login request.
So how can a user pretend to be a web crawler?
This browser addon can spoof useragents:https://add0n.com/useragent-switcher.html