Okay, i think there is quite a misunderstanding here.

Some older versions of LLMs (chatgpt3.5-turbo-instruct) can play chess relatively well (around 1750 Elo) : here is a link to an article studying that.

Some points :

• it is of course way worse than almost any algorithm designed for chess
• one of the reason we cannot get these result back (at least not that good, here is a link to a blog post of someone making recent LLMs chatbots better at chess) could be that we do not have access to pure completion mode on models trained on selected data (where they could purposefully choose only good chess matches), and those are now hidden behind a chatbot layer instead.
• it seems to reveal that models have a somehow accurate representation of the chess board when predicting chess moves
• it seems to have a quite unique feat that is : if you feed them a prompt that say they play as a very good player, and then the beginning of a game with a blatant bad move (giving away a queen for example), they sometimes play the entire game with moves that purposefully give away pieces, as if they guess that the only reason they would lose a piece that easily is by purposefully losing them. It has close to zero utility, but it's interesting anyway.