Which has lead to a whole new issue popping up where the AI chooses not to use the tool, because it doesn't trust it. Now this is anthropomorphizing a lot, in reality it's a reward misalignment issue, but still. It's called tool aversion and can be an issue. The LLM would even make up a reason the tool won't work, or tell you it did use the tool when it didn't actually.

In order to fix this a technique called cryptographic receipts is used. This adds an expected hash output based on the tool and the input. The tool outputs the correct hash, but if the LLM didn't call the tool and made up some BS, the hash isn't there. On the router level the missing or incorrect hash is easily detected and an exception raised in order to not present the result to the user (and hopefully correct the issue).

Another whole issue with using tools for certain stuff is it requires the situation to be properly evaluated for the tool to be called in the first place. So a user might hear the marketing say: Our old AI used to mess up maths, but our latest and greatest model is super duper smart and can do maths. The user then goes to verify this is the case with some simple tests, and sees it's correct. Then follows up with some harder math problems and the output is still correct. However the user doesn't know and isn't informed the AI used the tool, the user is lead to believe the AI is smart and can do maths now. So in situations where the detection doesn't work, or the LLM doesn't use the tool for whatever reason, the user is fed a nonsense response. Which the user will fully trust, given what he knows and tested himself.

It's so annoying these AI companies continually do shit like this. Lie to users, keep them in the dark and overpromise. Users relate the AI to human intelligence. And for humans, if you can do hard math problems, the circumstances don't really matter. We grasp the basic concepts and can execute to get a result and know how to double check. For these AI tools, it might get a PhD level math question totally right, faster and better than a human. And then get a simple calculus problem, someone in primary school could solve, wrong. For users this makes no sense.