this post was submitted on 21 Aug 2024
321 points (100.0% liked)

196

16725 readers
2369 users here now

Be sure to follow the rule before you head out.

Rule: You must post before you leave.

^other^ ^rules^

founded 2 years ago
MODERATORS
 
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 70 points 4 months ago (3 children)

I am confused, does this mean Reddit is not going to be searchable on search engines anymore?

[–] [email protected] 66 points 4 months ago (4 children)

oh no, Reddit is like, the only way to have google still be useful.

[–] [email protected] 54 points 4 months ago

Funnily enough, google is also the only way to have Reddit be useful.

Their own search function has been nothing but garbage.

[–] [email protected] 43 points 4 months ago (2 children)

That's the catch, Google made a deal with Reddit and remains the only search engine allowed to access its data for indexing. It cuts off every other search engine

[–] [email protected] 27 points 4 months ago (1 children)

Tell me that there is an anti trust suit over this.

[–] [email protected] 26 points 4 months ago

There's a suit over google in general so this may well be part of it

[–] [email protected] 3 points 4 months ago (1 children)

really? ddg will show me reddit links, did they have to make a webscraper or something

[–] [email protected] 4 points 4 months ago

There's a cutoff date, anything indexed before the robots.txt was changed stays in the index

[–] [email protected] 31 points 4 months ago (1 children)

We fucked the internet. It’s proprietary now.

[–] [email protected] 11 points 4 months ago* (last edited 4 months ago) (1 children)
[–] [email protected] 8 points 4 months ago (1 children)
[–] [email protected] 2 points 3 months ago

cat5-o-nine-tails

[–] [email protected] 9 points 4 months ago (1 children)

Good news! Google paid up and still has access I'm pretty sure.

[–] [email protected] 1 points 4 months ago (1 children)

That's bad news, that means the internet is dying

[–] [email protected] 1 points 4 months ago (1 children)

Sorry, the /s was sort of implied.

[–] [email protected] 2 points 4 months ago

Ah, sorry. I have trouble with that sometimes :P

[–] [email protected] 9 points 4 months ago (1 children)

Perhaps, likely depends on the crawler though

[–] [email protected] 12 points 4 months ago

Yeah i dont think ignoring robots.txt is even illegal. They can ofcourse just block your crawlers IP but that would be a cat and mouse game that they would lose in the end.