59
DuckDuckGo is censoring certain news outlets in it's "duck.ai" service's search (hexbear.net)
submitted 2 months ago by to c/chapotraphouse@hexbear.net
17 comments fedilink hide all child comments
 

I was messing around with duck.ai because it doesn't need an account and I like to see how censored western AIs are. Specifically around the Iran war.

I was going to ask it to read Aljazeera to get their coverage of the war and then compare what it summarizes to the same thing from other news sites. To see if it changes what it says based on which news org I have it pull the info from. But then I ran into an interesting hiccup.

I tried this with multiple models and as soon as you specify to search for Aljazeera + specific news event it will get a 404 error. And the same for CNN.

I thought this might be the sites themselves blocking AI crawlers. So I tried it in deepseek. It had no issue pulling both Aljazeera and CNN. I also researched exa (the search plugin AIs use) and it does advertise a feature where you can block certain URLs from showing up.

It does seem possible to force it to see them. I was able to get it to access Aljazeera.com on the backend and read from their own documents to find articles but in the websearch tool itself there is no result coming up from either CNN or Aljazeera. Which is odd right?

I did some research. I thought maybe it had something to do with the Zionists. This smells of them. And the founder of duckduckgo does have ties to the founder of an "Israeli" tech firm (they seem to be cousins) and even said in 2013, I think, that he wanted to visit "Israel" in an interview he did with the Times of "Israel". So this might explain the Aljazeera part. It is banned in "Israel" after all. But why CNN? That's an American company. It tends to follow the state departments messaging.

I tried some more sites. Trying to think of any that might be getting blocked on purpose.

RT - not working Aljazeera - not working CNN - not working NBC - working ABC - working FOX - working QQ news - working CGTN - not working

After trying these I tried AP next. Then it told me "duckduckgo is temporarily unavailable". I refreshed. Same thing. It was acting like the site was not working. Until i changed my IP and deleted browser data. Then suddenly it worked again. Interesting huh? And AP is also not working btw.

So it blocks certain news sites, and if you keep trying to access them too often it blocks you from using the service by pretending to be offline I guess?

Would love to see if anyone can replicate this behavior. If you do it be careful not to let it trick you. What it will do is you'll ask for coverage from a specific site and it will find that site has nothing (404 error) and then it'll just make things up from other sites, and act like it did what you asked. You have to specifically tell it not to include results from any website other than the one you requested. And watch the thinking so you can see as it tries the searches and they fail.

you are viewing a single comment's thread
view the rest of the comments
[–] 10 points 2 months ago* (4 children)

it could be that Al Jazeera is blocking high-traffic or abusive crawlers... i just found out a couple weeks ago that Meta's AI crawler had been hammering our little site so hard for days that it was essentially DOS'd for upwards of 5 - 10 minutes at a time, over and over again day and night, just spamming invalid urls trying to learn every combination of possible pages that it could conceive of that might possibly exist on our site, 99.9% of which did not exist.

Cloudflare (blech) allows you to see and block those crawlers and as soon as I put that in place the site came back to life almost immediately. DDG's crawler was in that list and so were several chinese AI crawlers, but only Meta's wasn't following any sort of rules so that one got blocked.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] [S] 5 points 2 months ago (3 children)

    Well the thing is DeepSeek is much more well known than DDG's AI. So why would they block DDG but not Deepseek if that was the case? And it's not just Aljazeera. The same pattern emerges for all the other News sites I listed. They don't work on DDG. They work on Deepseek.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 13 points 2 months ago* (1 child)

    I just looked at Al Jazeera's robots.txt and it contains:

    # Disallow Rules

    User-agent: anthropic-ai

    Disallow: /

    User-agent: ChatGPT-User

    Disallow: /

    User-agent: ClaudeBot

    Disallow: /

    User-agent: Claude-Web

    Disallow: /

    User-agent: cohere-ai

    Disallow: /

    User-agent: GPTBot

    Disallow: /

    User-agent: PerplexityBot

    Disallow: /

    User-agent: Bytespider

    Disallow: /

    It looks like duck.ai uses ChatGPT on the backend by default, so if it's behaving it won't scrape Al Jazeera's site. You could try switching it to one of the non-GPT, non-Claude models and see if it works better.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 2 points 2 months ago*

    Mistral doesn't use websearch and their Gemma model when asked will attempt to search, get a 404, and then just throws "Gemma 4 31B is temporarily unavailable. Please switch to a different model or try again later."

    What Gemma is thinking before it throws this error:

    The Claude model they have just says:

    But this is specifically the search tool. If you tell it something else it can access their site:

    It is only the search function, which has a built in method to censor which sites it can search from, that has the issue. I don't know if the ReadDocument tool would somehow be unblocked while searches would be blocked. But it strikes me as odd that the searches don't even show up. Like the search tool itself throws a 404 error when searching for these things. It's not just having no results or showing the URLs but being unable to access them due to the site blocking it from reading the content. It acts as if that URL simply does not exist and it's an invalid query.

    So either Aljazeera, RT, CGTN, AP, CNN, and probably more I didn't check, have all specifically blocked the search function used by DDG's various AI models, and also all have not blocked Deepseek, or it is something on DDG's end causing the fail. And considering that the more I tried to do it they eventually just had the entire site pretend to be down, and that Gemma instead of telling me it can't do the search pretends the model is down? It seems very fishy.

    What do you think? If you know a bit about how these work on the backend do you think this behavior lines up with the websites doing the blocking? I don't know enough to say for sure, but it doesn't seem like it at first glance to me.

    Edit: To clarify when the Claude model says it gets other results but not aljazeera results it is lying. You can see it do searches and when it searches for Al Jazeera it gets back a 404. But then it tries a more general search and gets other things back. And it then acts like these were a single search when talking about it. It confused me at first so wanted to point that out.

  • permalink
  • fedilink
  • source
  • parent