Of course (thelemmy.club)
submitted 1 day ago* by to c/linuxmemes@lemmy.world
 
Of course
you are viewing a single comment's thread
view the rest of the comments
[–] 43 points 15 hours ago (21 children)

Tinfoil hat time:

A year ago, I did some reverse engineering. I figured a pattern matching engine has to be good matching patterns, so I threw the messed-up output of Ghidra into Claude to clean it up and to give me an overview of what the fuck happens in a certain part.

I was absolutely surprised. Sure, it got some things wrong, but it knew way more than it should have been able to guess from that short bit of fucked up, optimised not-quite-C. Including guessing context that it should not be able to guess. This was a firmware with not-yet-released functions that, to my knowledge, shouldn't exist outside of this bit of binary firmware. At least not for us non-NDA mortals.

At the time, Mistral had a new-ish model as well, so I figured I'd give it a try. It worked as expected, cleaned up things, gave variables reasonable names, made educated guesses and so on, but didn't act as if it had background info. And I was using Sonnet 4.x, so the small Claude which shouldn't be that much larger.

Since then, I have the conspiracy theory that the Claude people threw a bunch of proprietary sources in the training, too. I have no idea how, maybe aggressively training on user inputs? Anyway, this does not help to allalleviate my suspicions.

  • source
  • hideshow 21 child comments
  • [–] 4 points 4 hours ago*

    They're using gig workers by the hundreds of thousands, if not millions to do "data annotation". You can actually make quite a bit if you've got higher credentials like Phd, masters, or a very good resume.

    My theory is they've been using professionals to inadvertently leak proprietary methodologies. They could be doing so by creating problems for them to solve that lead people to inadvertently expose solutions based on their actual jobs.

    It's lowkey corporate espionage.

  • source
  • parent
  • [–] 2 points 4 hours ago (2 children)

    Oh it learned enough from code that it has developed some basic, and hear me out of this, artificial verbal intelligence. I know I know, not just pattern matching, but actual artificial intelligence at a low level.

    Please don't downvote me (sorry don't remember if this is fuckai or not) but I'm just saying it's one possible explanation to this phenomenon you describe.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 3 hours ago (1 child)

    i would err towards it ingesting proprietary code before this, it's not wrong but AI spits out copyrighted and should-be private stuff all the time.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 1 hour ago

    There are studies and tools like LiCoEval and others that investigate LLM memorization. It does happen but it's not all the time, and it's very infrequent for real world (complex) examples. Mostly this concerns textbook examples or standard boilerplate code - from what I read. And I obviously don't know the specific example mentioned. This is also a problem that can be fixed.

    But I'm really just half mocking the very careful language used by the comment I replied to.

  • source
  • parent
  • [–] 2 points 5 hours ago (2 children)

    I threw about 10 short phrased meeting notes (4-5 words each) into an LLM to get some document formatting started with titles and sections etc. It built a full scope document and inferred a ton of information about the project and product. It was surprising how much it could infer, and suggested very sensible testing criteria and alternate solutions should the main product path not pan out.

    I can see why people start believing it "thinks", because it was remarkable what it "understood" about a vague new product. Had I sent the info out to stake holders, they would have assumed this person was a key team member that was involved from day 1.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 hours ago* (last edited 2 hours ago) (1 child)

    "Intelligence", as a word has always been ill defined. What's it mean? It's mostly been used to create prejudices via standardized tests, the bell curve, and IQ tests.

    I don't believe it's "intelligent" as we don't really know how to define that. But I do think it's really really really good at writing reasonable seeming statements in various languages using probability.

    Coding and programming, are just languages. So it will likely be able to get pretty damn good at it, just as it has with the English language.

    I think with LLMs accurate regurgitation of facts, is the colary to good code.

    But I don't believe that means it "thinks" (which requires introspection, and an internal experience). It does multi-agent computing, and this might even simulate some aspects of thinking... But it's not thinking. It's computing.

    It won't be able to do any more than that, until it consistently gets data from having a body (and something like glial thought-interactions/organs/flaws/survival).

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 2 hours ago*

    By the way, this also means if you could download a mind to a computer - it would slowly lose it's humanity as it lived longer and longer interfacing with computer systems rather than human systems/organs/experiencs.

    Eventually that "uploaded human" would become a Clankr, and stop understanding human life, the will to survive/think. Eventually it would just compute (not think/care or have flaws).

    Many sci-fi stories are based on the man-to-technology-to-cold-rational-evil pipeline... Less have gone in the other direction, even though it often produces very popular characters: Spock, Data, the Terminator, Pinnoccio.

  • source
  • parent
  • [–] 17 points 10 hours ago* (last edited 8 hours ago) (1 child)

    I specifically asked our CEO about this just to cover my ass. It seems insane to trust these guys at all. Here in the USA, small businesses have de facto no legal protection from large businesses.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 13 points 10 hours ago*

    Absolutely. Especially if you have company info / IP that someone else would pay a lot for. This is a question for lawyers and managers to negotiate with the lawyers at OpenAI, Anthropic etc, not for us little people to decide. I'm not throwing company data in an online service that is not cleared for us to use, and I don't even work with spicy stuff. Back in my IT days we would have called that a data breach, which would have lead to something from an unpleasant hour in the boss' office to a promotion to customer.

  • source
  • parent
  • [–] 29 points 14 hours ago (8 children)

    Since then, I have the conspiracy theory that the Claude people threw a bunch of proprietary sources in the training, too.

    An enormous amount of people use this technology for privat company data without batting an eye; not just software developers, but business people for reports, etc.

    They all throw their confidential data in there without even considering the implications, just "trusting" Anthropic. claude code bots are running on tens of thousands of proprietary code bases that used to be private, and of course they are using all this data to train their model.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 8 points 13 hours ago (6 children)

    Then people are surprised that claude code gets better at coding

  • source
  • parent
  • hideshow 6 child comments
  • [–] 8 points 11 hours ago (5 children)

    Better at replicating closed source software.

    It surely gets better at coding in general, after all there are smart people at anthropic with near limitless budgets, but my conspiracy thesis is that it gets a lot better at replicating stuff it has seen before, but really shouldn't have.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 4 points 10 hours ago (4 children)

    Meta had an explicit higher price for using their model without retention of prompts for training. This all but confirms your theory.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 2 points 10 hours ago (3 children)

    Wait... wait... you believe Facebook?

  • source
  • parent
  • hideshow 3 child comments
  • [–] 3 points 9 hours ago (2 children)

    Oh, facebook will steal the data even if they say they won't. What's interesting is they put a price on blatant prompt copying.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 7 points 14 hours ago

    I'm sure it's all but officially confirmed, that all the people upload to it goes to training data unsanitised. And Microslop for sure uploaded entirety of their codebase to all the bots. This is at least part of the reason why claude is so surprisingly OK at reverse-engineering all the big stuff.

  • source
  • parent