23
Phishing / spam detection with local LLMs? (lemmy.ca)
submitted 10 hours ago by to c/selfhosted@lemmy.world
12 comments fedilink hide all child comments
 

I've tried giving screenshots of phishing emails to a local Qwen instance and so far it always correctly detected it as scam, even points out the exact elements that it based its judgement on. Sending screenshots to it ad-hoc isn't too scalable for family and friends. I'd like to be able to either forward emails for screening, or perhaps have it screen everything from a mailbox.

Has anyone done anything like this? Is there anything self-hostable that does this?

you are viewing a single comment's thread
view the rest of the comments
[–] 26 points 10 hours ago (3 children)

I'd be pretty concerned about prompt injection risks with feeding a LLM unsanitized data. You definitely need a good harness around it..

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] [S] 10 points 10 hours ago (2 children)

    Good point. It'll have to have no access to the internet or anything local outside of its container. Just text in, text out.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 6 points 6 hours ago

    You could (and probably should) use a system-one style inference system for spam classification. Much cheaper and the structured output means it's impossible to go rogue and curl some malware or whatever. It can absolutely misclassify but its output is programmatically structured and just ranks a pre-selected set of output tokens.

    In your case that's

    Spam

    Not_spam

  • permalink
  • fedilink
  • source
  • parent
  • [–] 8 points 10 hours ago

    Yeah if it's just a basic input with a function call for spam or not spam the risk is low. What's the worst case outcome, it tricks it into saying no it isn't a scam and you have to delete it manually? Hahaha

  • permalink
  • fedilink
  • source
  • parent