There's no real reason to expect it to play chess.

There is. All the general media is calling these LLMs AI and AIs have been playing chess and winning for decades.