The conservative think tank The Heritage Foundation wrote that the initial 2022 iteration of KOSA did not go far enough, as the bill did not explicitly list transgender health care as a harm.[27][28] The inclusion of the phrase "consistent with evidence-informed medical information"[29] could be used by attorneys general to cherry-pick anti-trans sources as justification since there is no definition of what "evidence-based medical information" can include.[30] Senator Blackburn, co-author of the bill, has argued that some education about racism and the civil rights movement overlaps with critical race theory, which she labels a "dangerous ideology" that can inflict "mental and emotional damage" upon children.[31] She has also explicitly stated that the bill will be used to censor content involving the transgender community.[32] EFF columnist Jason Kelly states that in the framework provided by the bill, that KOSA could be used to censor education about racism in schools since it could be claimed that it impacts mental health.[33]

In September 2023, a video from the Family Policy Alliance showed Blackburn saying that there should be a priority to "protecting minor children from the transgender [sic] in this culture", alongside her promotion for KOSA, stating "This would put a duty of care and responsibility on the social media platforms, and this is where children are being indoctrinated."[34] This drew criticism from LGBT advocacy groups, fearing that the bill would allow LGBT information for minors to be censored. A spokesperson for Blackburn stated that KOSA was not intended to censor LGBT information.[34] To address these concerns, the bill's language was altered so that the "duty of care" only focused on the product design features that influenced minors' behavior with the platforms, and not the content. As a result, several LGBTQ groups, including GLAAD and GLSEN, dropped their opposition to the bill.[35] However, the EFF, Fight for the Future, and the American Civil Liberties Union found the revisions far from adequate, arguing that LGBTQ content could still be suppressed by targeting any design feature that makes that content available.[26][36][37][38]

If you use Bing, DuckDuckGo, Mojeek, Qwant or any other alternative search engine that doesn’t rely on Google’s indexing and search Reddit by using “site:reddit.com,” you will not see any results from the last week. DuckDuckGo is currently turning up seven links when searching Reddit, but provides no data on where the links go or why, instead only saying that “We would like to show you a description here but the site won't allow us.” Older results will still show up, but these search engines are no longer able to “crawl” Reddit, meaning that Google is the only search engine that will turn up results from Reddit going forward. Searching for Reddit still works on Kagi, an independent, paid search engine that buys part of its search index from Google.

The news shows how Google’s near monopoly on search is now actively hindering other companies’ ability to compete at a time when Google is facing increasing criticism over the quality of its search results. And while neither Reddit or Google responded to a request for comment, it appears that the exclusion of other search engines is the result of a multi-million dollar deal that gives Google the right to scrape Reddit for data to train its AI products.

“They’re [Reddit] killing everything for search but Google,” Colin Hayhurst, CEO of the search engine Mojeek told me on a call. Hayhurst tried contacting Reddit via email when Mojeek noticed it was blocked from crawling the site in early June, but said he has not heard back.

“It's never happened to us before,” he said. “Because this happens to us, we get blocked, usually because of ignorance or stupidity or whatever, and when we contact the site you certainly can get that resolved, but we've never had no reply from anybody before.”

As Jason wrote yesterday, there’s been a huge increase in the number of websites that are trying to block bots that AI companies use to scrape them for training data by updating their robots.txt file. Robots.txt is a text file which instructs bots whether they are or are not allowed to access a website. Googlebot, for example, is the crawler or “spider” that Google uses to index the web for search results. Websites with a robots.txt file can make an exception to give Googlebot access, and not other bots, so they can appear in search results that can generate a lot of traffic. Recently Google also introduced Google-Extended, a bot which crawls the web specifically to improve its Gemini apps, so websites can allow Googlebot to crawl but block the crawler Google uses to power its generative AI products.

Robots.txt files are just instructions, which crawlers can and have ignored, but according to Hayhurst Reddit is also actively blocking its crawler.

Reddit has been upset about AI companies scraping the site to train large language models, and has taken public and aggressive steps to stop them from continuing to do so. Last year, Reddit broke a lot of third party apps beloved by the Reddit community when it started charging to access its API, making many of those third party apps too expensive to operate. Earlier this year, Reddit announced that it signed a $60 million with Google, allowing it to license Reddit content to train its AI products.

Reddit’s robots.txt used to include a bunch of jokes, like forbidding the robot Bender from Futurama from scraping it (User-Agent: bender, Disallow: /my_shiny_metal_ass) and specific pages that search engines are and are not allowed to access. “/r*.rss/” was allowed, while “/login” was not allowed.

Today, Reddit’s robots.txt is much simpler and more strict. In addition to a few links to Reddit’s new “public content policies,” the file simply includes the following instruction:

User-agent: *
Disallow: /

Which basically means: no user-agent (bot) should scrape any part of the site. “Reddit believes in an open internet, but not the misuse of public content,” the updated robots.txt file says.

“Unfortunately, we’ve seen an uptick in obviously commercial entities who scrape Reddit and argue that they are not bound by our terms or policies,” Reddit said in June. “Worse, they hide behind robots.txt and say that they can use Reddit content for any use case they want. While we will continue to do what we can to find and proactively block these bad actors, we need to do more to protect Redditors’ contributions. In the next few weeks, we’ll be updating our robots.txt instructions to be as clear as possible: if you are using an automated agent to access Reddit, you need to abide by our terms and policies, and you need to talk to us.”

Reddit appears to have updated its robots.txt file around June 25, after Mojeek’s Hayhurst noticed its crawler was getting blocked. That announcement said that “good faith actors – like researchers and organizations such as the Internet Archive – will continue to have access to Reddit content for non-commercial use,” and that “We are selective about who we work with and trust with large-scale access to Reddit content.” It also links to a guide on accessing Reddit data which plainly states Reddit considers “Search or website ads” as a “commercial purpose” and that no one can use Reddit data without permission or paying a fee.

Google did not respond to a request for comment, but its announcement of the company’s deal with Reddit points out not only how valuable Reddit is for training AI, but what many of us already know: As Google Search gets increasingly worse in turning up relevant search results, one of the best ways to still get them is to add “Reddit” to your search queries, directing Google to a site where real humans have been writing advice and recommendations for almost two decades. There are a lot of ways to illustrate how useful Reddit can be, but I’m not going to do better than this video:

https://www.youtube.com/watch?v=tcJcw55zIcc

The fact that Google is the only search engine that leads users to that information now, and that it is apparently the result of a $60 million deal around AI training data, is another example of the unintended consequences of the indiscriminate scraping of the entire internet in order to power generative AI tools.

“We've always crawled respectfully and we've done it for 20 years. We're verified on Cloudflare, we don't train AI, we're like genuine, traditional genuine searching, we don't do ‘answer engine’ stuff,” Hayhurst said. “Answer engine” is Perplexity’s name for its AI-powered search engine. “The whole point about Mojeek, our proposition is that we don't do any tracking. But people also use us because we provide a completely different set of results.”

Reddit’s deal with Google, Hayhurst said, makes it harder to offer these alternative ways of searching the web.

“It's part of a wider trend, isn't it?” he said. “It concerns us greatly. The web has been gradually killed and eroded. I don't want to make too much of a generalization, but this didn't help the small guys.”

To be sure, not all of the measures are identical. In some states, Republicans want to scrap age verification requirements for employers. In other states, they want minors to be able to serve alcohol. A Washington Post report last year noted some state GOP officials also eyed proposals to allow kids as young as 14 to “work certain jobs in meatpacking plants and shield businesses from civil liability if a child laborer is sickened.”

Evidently, related efforts are underway again this year, and this NoLa.com report out of Louisiana stood out as especially striking.

A Louisiana House committee voted Thursday to repeal a law requiring employers to give child workers lunch breaks and to cut unemployment benefits — part of a push by Republicans to remove constraints on employers and reduce aid for injured and unemployed workers.

Evidently, now that Republicans control all of the levers of power in Louisiana again — former Democratic Gov. John Bel Edwards stepped down after two terms earlier this year — Republican Gov. Jeff Landry directed officials to “reform” the state’s business environment.

Some GOP officials are taking that directive quite seriously — to the point that they’re advancing a plan to scrap requirements for lunch breaks for child workers.

According to the local report, the proposal is being championed by Republican state Rep. Roger Wilder, who owns some smoothie franchises across the region, and who said many of his child employees want to work without lunch breaks.

“The wording is ‘We’re here to harm children.’ Give me a break,” he said. “These are young adults.”

For the record, the kids might very well be young, but they’re minors, not adults.

The reporting comes with some caveats: The bill this week passed a state legislative committee, and it’s headed to the state House floor in Louisiana, but it hasn’t yet become law. There’s still time, in other words, for the measure to be amended and/or defeated. But the bigger picture is nevertheless unmistakable: In contemporary politics, Republican governance, especially at the state level, is increasingly invested in rolling back child-labor safeguards.

As for why officials in so many states are “reforming” their child-labor laws — an issue that appeared to be relatively settled until quite recently — there’s reason to believe the sudden flurry is not coincidental.

The Washington Post last year highlighted the role of something called the Foundation for Government Accountability, which is taking the lead on “maneuvering” these changes through state legislatures.

The Florida-based think tank and its lobbying arm, the Opportunity Solutions Project, have found remarkable success among Republicans to relax regulations that prevent children from working long hours in dangerous conditions. And they are gaining traction at a time the Biden administration is scrambling to enforce existing labor protections for children. ... Since 2016, the FGA’s Opportunity Solutions Project has hired 115 lobbyists across the country with a presence in 22 states, according to the nonpartisan political watchdog group OpenSecrets.

With this in mind, there’s no reason to expect these measures to disappear anytime soon.

I argue that this is an argument for excluding some of the most vulnerable in our community, and we should treat Twitter as an inaccessible space for organising.

It’s the people staying on Twitter to the exclusion of other networks that is causing the problem.

We MUST do better than this.

@emilygorcenski

@katiefenn but our enemies don’t organize with Twitter, they just use it as the conduit to project their organization. The conclusions of that study were already determined when it was commissioned. The channel must be severed.

@emilygorcenski That’s true. It shows that they were organised ahead of time when it came out. They’d briefed sympathetic and influential people, and made sure that critics were spending the whole day reading the damn thing while they were giving interviews.

In fact, I think the fact that the earliest criticisms were based off of an easily refuted, leaked press briefing really hurt us.

We should be doing the same - organise off the platform, and channel it through Twitter.