Curated data should improve the results as well. Just jamming all the trash data is why the models they keep jamming into everything have such trash results.

the LLM's dataset uses only public domain and openly licensed material.

I'm curious about the specifics of all this. Probably the most well-known "openly licensed" sort of licenses (aside from licenses specifically intended only for software) are the Creative Commons family of licenses, all of which require attribution. So then the question would become "if you've used any of my CC-licensed content in training this model, am I attributed somewhere?" If so, surely the list is extremely long. Or maybe Creative Commons wasn't "openly"-enough licensed and they excluded all CC-licensed content from the training set.

Also, the public domain is definitely strongly biased toward very old content. You'd think a lot of the answers you got from that LLM would be based on some very outdated information. Maybe they specifically limited it to (or at least adjusted weights or something to make it prefer) recent materials in the public domain.

But then the article also says:

It performed about as well as Meta's similarly sized Llama 2-7B from 2023.

On top of all this, I have to say that the LLM sphere really is just scams piled on top of scams, so it's fairly probable either that it doesn't perform anywhere near as well as Llama 2-7B and they're just lying or that actually Llama 2-7B (and indeed all LLMs as well) is just total shit too.

You can but the results are going to be essentially unusable compared to SOTA. You are still giving the big AI companies a massive monopoly just so the big copyright companies can make even more money.

Only one solution is good for us. In both cases, the ones that actually created the data get screwed though.