They only take from the statistical distributions of words in the context of preceding words (which is why they never say "the the" etc, why the grammar is nearly always correct). But that doesn't mean that whole sentences are lifted from the source material. There are near infinite paths through those word distributions, and many have never been produced by humans, so LLMs do produce sentences that have never been uttered before.

They couldn't produce new conceptual context spaces in the way that humans can sometimes, but they can produce new combinations within existing context spaces.

Nah, I'm still giving that one to the blockchain. LLMs are going to be useful for a while, but Ethereum still hasn't figured out a real use, and they're the only ones that haven't given up and moved fully into coin gambling.

You are speaking of "model collapse", I take it? That doesn't happen in the real world with properly generated and curated synthetic data. Model collapse has only been demonstrated in highly artificial circumstances where many generations of model were "bred" exclusively on the outputs of previous generations, without the sort of curation and blend of additional new data that real-world models are trained with.

There is no sign that we are at "the peak" of AI development yet.