They're not both true, though. It's actually perfectly fine for a new dataset to contain AI generated content. Especially when it's mixed in with non-AI-generated content. It can even be better in some circumstances, that's what "synthetic data" is all about.

The various experiments demonstrating model collapse have to go out of their way to make it happen, by deliberately recycling model outputs over and over without using any of the methods that real-world AI trainers use to ensure that it doesn't happen. As I said, real-world AI trainers are actually quite knowledgeable about this stuff, model collapse isn't some surprising new development that they're helpless in the face of. It's just another factor to include in the criteria for curating training data sets. It's already a "solved" problem.

The reason these articles keep coming around is that there are a lot of people that don't want it to be a solved problem, and love clicking on headlines that say it isn't. I guess if it makes them feel better they can go ahead and keep doing that, but supposedly this is a technology community and I would expect there to be some interest in the underlying truth of the matter.

Also, what do you mean by synthetic data? If it's made by AI, that's how collapse happens.

But that's exactly my point. Synthetic data is made by AI, but it doesn't cause collapse. The people who keep repeating this "AI fed on AI inevitably dies!" Headline are ignorant of the way this is actually working, of the details that actually matter when it comes to what causes model collapse.

If people want to oppose AI and wish for its downfall, fine, that's their opinion. But they should do so based on actual real data, not an imaginary story they pass around among themselves. Model collapse isn't a real threat to the continuing development of AI. At worst, it's just another checkbox that AI trainers need to check off on their "am I ready to start this training run?" Checklist, alongside "have I paid my electricity bill?"

The problem with curated data is that you have to, well, curate it, and that's hard to do at scale.

It was, before we had AI. Turns out that that's another aspect of synthetic data creation that can be greatly assisted by automation.

For example, the Nemotron-4 AI family that NVIDIA released a few months back is specifically intended for creating synthetic data for LLM training. It consists of two LLMs, Nemotron-4 Instruct (which generates the training data) and Nemotron-4 Reward (which curates it). It's not a fully automated process yet but the requirement for human labor is drastically reduced.

the only way to guarantee training data isn't from its own model is to make it yourself

But that guarantee isn't needed. AI-generated data isn't a magical poison pill that kills anything that tries to train on it. Bad data is bad, of course, but that's true whether it's AI-generated or not. The same process of filtering good training data from bad training data can work on either.