Again, this is an expression of your ignorance, not reality.

OLMo 2 was trained on Wikipedia and other fully public forums. Its training sources and data is fully accessible and open.

Decided to check out the first example quickly. It's hard to dig through the information, but following the chain of sources:

In the first stage, which covers over 90% of the total pretraining budget, we use the OLMo-Mix-1124, a collection of approximately 3.9 trillion tokens sourced from DCLM, Dolma, Starcoder, and Proof Pile II.

As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl

So, it's using data from Common Crawl. What data, exactly? That'd be harder to dig up. DCLM has a repository, but they don't make an effort to point out how, or if, they're filtering the data.

What I can quickly find is information from Common Crawl itself, which is the ultimate source of data. On that I can immediately see only two things:

1. They have an opt-out list, and from what I understand, they're scraping websites by default, providing mechanisms for you to block them or opt-out... If you're even aware they exist.
2. According to their stats page, they have 313423 pages scraped from github.com. I doubt github has that many pages of non-user-generated content, so the question is... What data is going in there, who wrote it, and did anybody agree to that? What about other websites from those top domains, such as blogspot.com, wordpress.org, readthedocs.io? I doubt they got permission to use the 17175161 pages of content from blogspot they've scraped.

So yeah, maybe there's a "good" model out there I'd actually accept, but I've seen "open" models being released, and I can't possibly check all of them, just checking the websites for one of the sources for one of the models took 20 minutes here, if I wanted to verify this properly I'd have to setup the tooling to query the terabytes of data for this source, and all others, and even there I'm not sure if I'd find answers.