Actually that is how copyright law could work.

On the ingest/training side I think the courts have been mostly consistent with your view and sided with the AI companies, though some lawyers are still opening new cases so at least some of them aren't so sure.

The problem is the defense the AI companies have given is still a potential problem for software. They state that even as the ingest of works allow the model to generate material that would violate the copyright of trained data, that it's the fault of the operator, not the AI company. So even if a user oblivious to the original material manages to get a model to create a knock-off of a copyrighted work, it is still the user's fault, not the AI company.

This implication to my knowledge has not been tested in court. I haven't seen cases where someone found and sued an AI user for plausibly oblivious copyright violation. Now I cannot imagine that the consequences can be very punitive if they can show the AI models will produce such a violation without the user indicating any awareness of the source material, but they may still demand it be ripped out, which depending on age of code could be a huge mess and a great loss of functionality of other things that depend on the offending code. Even if the particular case could be won on arguments about whether it meets the criteria of an infringement in that case, fighting the case is hard enough.

The question is whether the risk of accidental copyright violation and the risk of such a violation even being noticed is significant enough. Currently in the GenAI hype world where tons of money is seen to be had from fueling the phenomenon, that risk appears low. Question is if the hype dies down and the money suckers want more money, will they at least start making copyright claims against folks focusing on how the AI defense was to throw the users under the bus. Again, open source projects themselves are probably low on the list of targets, as financial penalties seem unlikely, but it can't be ruled out.