The issue I have with this idea, is that the LLM doesn’t have an understanding of the algorithms, formulas, proofs or theorems. It “knows” they exist, but not what they actually mean. It’s like having a toddler pick out associations on a word board and hoping they happen to relate to each other.

Yes, there is more to an LLM, and I can’t really think of a good analogy. My point is that the associations being made aren’t based in mathematical knowledge, it’s based on “do these things relate to each other in my training data.”

LLMs specifically are incapable of “new thoughts.”