this post was submitted on 05 Jul 2023
807 points (98.4% liked)

World News

32378 readers
521 users here now

News from around the world!

Rules:

founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 79 points 1 year ago (11 children)

How does Pinterest get around this then? They pollute image searches like crazy, and require you to login to see anything. At least they did, I blocked them from searches so maybe it's different now.

[–] [email protected] 14 points 1 year ago (6 children)

Easy - detect if you're getting accessed by a search crawler or a human. Serve a full page or just a login request.

[–] [email protected] 11 points 1 year ago (5 children)

So how can a user pretend to be a web crawler?

[–] [email protected] 22 points 1 year ago* (last edited 1 year ago)

This browser addon can spoof useragents:https://add0n.com/useragent-switcher.html

load more comments (4 replies)
load more comments (4 replies)
load more comments (8 replies)