Closely related is a thought I had after responding to yet another paper that says hallucinations can be fixed:

I'm starting to suspect that mathematics is not an emergent skill of language models. Formally, given a fixed set of hard mathematical questions, it doesn't appear that increasing training data necessarily improves the model's ability to generate valid proofs answering those questions. There could be a sharp divide between memetically-trained models which only know cultural concepts and models like Gödel machines or genetic evolution which easily generate proofs but have no cultural awareness whatsoever.

Every time I hear a moderate AI argument (e.g. AI will be an aid for searching literature or writing code), it's like, "Look, it's impressive that the AI managed to do this. Sure, it took about three dozen prompts over five hours, made me waste another five hours because it generated some completely incorrect nonsense that I had to verify, produced an answer that was much lower quality than if I had just searched it up myself, and boiled two lakes in the process. You should acknowledge that there is something there, even if it did take a trillion dollars of hardware and power to grind the entire internet and all books and scientific papers into a viscous paste. Your objections are invalid because I'm sure things are gonna improve because Progress."

I am doubly annoyed when I turn my back and they switch back to spouting nonsense about exponential curves and how AI is gonna be smarter than humans at literally everything.

Wouldn't f(x) = x^2 + 1 be a counterexample to "any entire (differentiable everywhere) function that is never zero must be constant"? Or are some terms defined differently in complex analysis than in the math I learned?