200
As we all predicted: The fearmongering surrounding Anthropic Mythos was just marketing after all. (www.theregister.com)
submitted 5 months ago by to c/fuck_ai@lemmy.world
8 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 5 months ago (1 child)

It's not unknown if it's effective...

someone from Mozilla put their name on a blog post saying they fixes 22 vulnerabilities and an upcoming patch fixes 200+

https://blog.mozilla.org/en/privacy-security/ai-security-zero-day-vulnerabilities/

This isn't anthropic, they have no incentive to lie about how good the product is. They're committing limited development resources to these fixes so would want them to be real problems.

It's also definitely more than 0, the original mythos post disclosed a OpenBSD vulnerability that was patched and they disclosed more as hashes for unpatched vulnerabilities, basically signatures of either descriptions or exploit implementations, so when they fix and disclose the plain text ones we know the original post was telling the truth. The signatures are difficult to forge.

We also have a ton of 3rd party researchers looking at this stuff if they did forge those signatures and I haven't seen any whistleblowers with access to the model saying it garbage. If you have any sources let me know, I would be interested in reading that.

My point is not to white knight for anthropic. They're flaunting IP rights and driving up energy prices for personal profit, but that when you take a position and say something it should be for actual reasons, not just "I hate this company"

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 5 months ago* (last edited 5 months ago)

    Hmmh. Yes. I mean they throw a lot of numbers around. A 10 fold performance increase would be impressive. But they're not really transparent, are they? How many "vulnerabilities" from the one category compare to the "security-sensitive bugs" from the other category? We know Opus found 7 medium and 14 high classified vulnerabilities and then for Mythos they give one total number. I also don't know whether the tooling and pipeline changed, or the focus... Or how many resources they allocated in February compared to March... So I think it's really hard to tell whether the Mythos model is like 10x better or a hundred times better or whatever compared to what they had two months ago. I tend to question claims like that if they come from a profit-oriented company and they don't provide us with enough details to fact-check the claim. I mean I read 271 vulnerabilities. And then I go to check in the list of CVEs fixed in Firefox 150 and I just see the name "Claude" mentioned three times. Dunno why that is. But it was 22 times in 148 when they used Opus.

    And then they didn't compare it with anything. For all we know a random Chinese (or Google) model might find 20 or 200 or 2000 bugs in Firefox. But that'd be important info if we want to hype Anthropic in specific. But as far as I know Mozilla only has that one cooperation with one company. So there's no reference.

    The info I have is all from the theregister article above, they said there's wildly different perspectives and counts out there. And they wrote who said it. I mean I don't have access anyway. All I can do is wait and see what effect it has on some software projects.

    But I think what I'd really like to know is the details on all of this. Someone dissect some example findings... Some numbers on the allocated resources. Like if they spend a million dollar electricity bill to find some medium vulnerabilities or whatever the numbers are.

  • permalink
  • fedilink
  • source
  • parent