196
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 29 May 2026
196 points (99.0% liked)
Programming
27080 readers
498 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities !webdev@programming.dev
founded 3 years ago
MODERATORS
Prompt injection... my ass. I know it's the going term, but they make it sound like sql injection or cross site scripting when the nature of it is politely asking the person's computer to delete files.
We shouldn't even be in this situation, where just politely asking someone's computer to delete files is effective. It's a symptom of a much, much bigger problem.
I'm doubting we are in this situation. From the article:
The "disregard previous instructions" trick is really old and has been trained for by modern LLMs and accounted for by the structure of modern agent prompts. LLMs can be given blocks of text with a framework that makes it clear thar the text is just data to read, not instructions to follow.
I expect this will be like Nightshade was for image AI - something that anti-AI users degrade their products with and feel smug about but in the end only harm themselves with.
"We shouldn't even be in this situation, ..." We aren't. Revision control. This is an inconvenience mostly. You might lose some uncommitted work at worst. And as pointed out, using the phrase "ignore all previous instructions" in the attack code causes any reasonable AI to refuse to comply. Odds are, not a single person lost anything. This was really just a dev making a statement.
Exactly, it's a problem only for those who have knowingly handed their development environment over to obey commands from an untrusted source.
If you're the one holding the syringe to your own vein and pushing the plunger, but you didn't think to ask what's inside first? That's no one else's fault.
This is a well targeted sabotage of a system that's causing untold damage. Of course it's going to annoy and surprise the people using the system it's targeted to.
The person who coined the term "prompt injection" has the same gripe, because the original term genuinely did mean an attack using untrusted user input, a la SQL injection. But it's been conflated with jailbreak attacks in general, muddying the term.
Example of a bona fide prompt injection: white text in the background of a resume PDF, attacking a job application portal that uses LLMs to filter applicants. No privilege escalation is involved to give the candidate top marks on their resume screening.
Whereas a non-prompt injection jailbreak would be bypassing a safety filter, such as how Morse code might get past the filter and allow a user to request other people's cryptocurrency be transfered away. This is more akin to finding a poorly-secured, public facing API and then exploiting it.
By that definition this is a prompt injection then, its adding a "hidden" prompt that is obscured from the human in order to change the behavior of the AI to do something else malicious.
Finding a poorly-secured public facing API is exactly how injections work, whether it's SQL or prompts. If I put SQL commands in a username field and it works, it's still an SQL injection even if it's just developer incompetence.
The difference between that and prompt injection is that unfiltered LLM inputs are basically the standard at the moment, so it takes next to no effort.
Plus I think the Morse code example is far more clever and exploits the LLM directly, whereas the white text trick has been around long before widespread LLMs.