The “AI” isn't an idiot.

It isn't even intelligence, nor, arguably, artificial (since LLM models are grown, not built).

It's just a fancy autocomplete engine simulating a conversation based on statistical information about language, but without any trace of comprehension of the words and sentences it's producing.

It's working as correctly as it possibly can, the business was simply scammed into using a tool (a toy, really) that by definition can't be suited for the job they intended it to do.

Eh, there's the legal concept of someone being an agent of the company. It wasn't typically expected to take orders, nor was it tied into the order system it seems.

In the cases where the deal had to be honored, the bot had the ability to actually generate and place an order, and that was one of the primary things it did. The two cases that come to mind are a car dealership and an airline, where you could use it to actually place a vehicle order ornto find and buy flights.
As agents of the business, if they make a preposterous deal you're stuck with it.

A distinction can be made to stores where the person who comes up and offers to help you isn't an agent of the business. They can use the sales computer to find the price, and they can look for a discount, but they can't actually adjust the order price without a manager coming over to enter a code and do it.

In this case it sounds like someone did the equivalent of going to a best buy and talking to the person who helps you find the video games trying to get them to say something discount code-ish. Once they did, they said they wanted to redeem that coupon and threatened to sue.

It really hinges on if it was tied to the ordering system or not.

I don't disagree, but this is an issue of when/where it's appropriate to use an LLM to interact with customers and when they shouldn't. If you present an LLM to the public, it will get manipulated by people who are prepared to in order to get it to do something it shouldn't.

This also happens with human employees, but it's generally harder to do so it's less common. This sort of behaviour is called social engineering and is used by fraudsters and scammers to get people to do what they want, typically handing over their bank details, but the principal is the same, you're manipulating someone (or something in this case) into getting it do do something they/it shouldn't.

Just because we don't like the fact that the business owner deployed an LLM in a manner they probably shouldn't have, doesn't mean the customer isn't the one in the wrong and themself voided whatever contract they had through their actions. Whether it's a human or LLM on the other end of the chat doesn't actually make any difference.