oh my god the maitai guy’s actually getting torn apart in the comments
Yeah some of you guys are very good at hacking things. We expected this to get broken eventually, but didn't anticipate how many people would be trying for the bounty, and their persistence. Our logs show over 2000 "saves" before 1 got through. We'll keep trying to get better, and things like this game give us an idea on how to improve.
after it’s pointed out 2000 near-misses before a complete failure is ridiculously awful for anything internet-facing:
Maitai helps LLMs adhere to the expectations given to them. With that said, there are multiple layers to consider when dealing with sensitive data with chatbots, right? First off, you'd probably want to make sure you authenticate the individual on the other end of the convo, then compartmentalize what data the LLM has access to for only that authenticated user. Maitai would be just 1 part of a comprehensive solution.
so uh, what exactly is your product for, then? admit it, this shit just regexed for the secret string on output, that’s why the pirate poem thing worked
e: dear god
We're using Maitai's structured output in prod (Benchify, YC S24) and it's awesome. OpenAI interface for all the models. Super consistent. And they've fixed bugs around escaping characters that OpenAI didn't fix yet.