There have been a number of papers recently ("TinyStories" and "Textbooks are all you need" are two that come to mind) showing that higher quality training data for LLMs leads to significantly better models. So yes, this is technically true.
post
replies:
all 11 comments