Look, AI will be perfect as soon as we have an algorithm to sort "truth" from "falsehood", like an oracle of some sort. They'll probably have that in GPT-5, right?

If the companies wanted to produce an LLM that didn’t output toxic waste, they could just not put toxic waste into it.

The article title and that part remind me of this quote from Charles Babbage in 1864:

On two occasions I have been asked, — "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" In one case a member of the Upper, and in the other a member of the Lower, House put this question. I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

It feels as if Babbage had already interacted with today's AI pushers.

The really annoying thing is, the people behind AI surely ought to know all this already. I remember just a few years ago when DALL-E mini came out, and they'd purposefully not trained it on pictures of human faces so you couldn't use it to generate pictures of human faces -- they'd come out all garbled. What's changed isn't that they don't know this stuff -- it's that the temptation of money means they don't care anymore

The chatbot “security” model is fundamentally stupid:

1. Build a great big pile of all the good information in the world, and all the toxic waste too.
2. Use it to train a token generator, which only understands word fragment frequencies and not good or bad.
3. Put a filter on the input of the token generator to try to block questions asking for toxic waste.
4. Fail to block the toxic waste. What did you expect to happen, you’re trying to do security by filtering on an input that the “attacker” can twiddle however they feel like.

Output filters work similarly, and fail similarly.

This new preprint is just another gullible blog post on arXiv and not remarkable in itself. But this one was picked up by an equally gullible newspaper. “Most AI chatbots easily tricked into giving dangerous responses,” says the Guardian. [Guardian, archive]

The Guardian’s framing buys into the LLM vendors’ bad excuses. “Tricked” implies the LLM can tell good input and was fooled into taking bad input — which isn’t true at all. It has no idea what any of this input means.

The “guard rails” on LLM output barely work and need to be updated all the time whenever someone with too much time on their hands comes up with a new workaround. It’s a fundamentally insecure system.