LLMs are terrible at math.

Mathematician chiming in: they're not nearly as bad as they used to be. Straight up to differential equations and linear algebra, they're pretty accurate. The writing is on the wall for some serious problems in the very near future, because basically every student in college right now is just offloading their math abilities to the machines.

Some of the seniors are good. Most of the rest of them are not.

It might even be one of those things that they can't fix, simply due to how LLMs operate.

For high-level math, there are some flaws, yes. They often come up with wrong answers if you ask questions about problems that there aren't very many sources on. There are hundreds of thousands, and maybe millions of differential equations papers... but there are only tens of thousands of Algebraic Topology papers. It's simply a smaller training set.

However, they're still giving right answers most of the time. I've only had one situation where someone in a group said "bUt cHatGPT sAiD..." and I had to give them the bad news that it was wrong.

In any case, we've had calculators for a long time, yet children are still taught math in school because society recognizes that understanding as crucial.

I think that the fundamental difference is that the tools replace different things. A calculator makes it so that you don't need to algorithmically process 5 sig-fig numbers without error, several times per problem, in order to have your answer be "correct". You still have to understand that you do, in fact, need to multiple the length of the field by the width to find the area of a rectangular field.

LLMs are replacing the need to actually understand anything about how a rectangle works.