144
AI still doesn't work very well, businesses are faking it, and a reckoning is coming (www.theregister.com)
submitted 5 months ago by [M] to c/technology@piefed.social
18 comments fedilink hide all child comments
 

a recent attempt to rewrite SQLite in Rust using AI. "It passed all the unit tests, the shape of the code looks right," he said. It's 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It's a dumpster fire. Throw it away. All that money you spent on it is worthless."

Insurers, he said, are already lobbying state-level insurance regulators to win a carve-out in business insurance liability policies so they are not obligated to cover AI-related workflows. "That kills the whole system," Deeks said. Smiley added: "The question here is if it's all so great, why are the insurance underwriters going to great lengths to prohibit coverage for these things? They're generally pretty good at risk profiling."

all 18 comments

sorted by: hot top controversial new old
[–] 30 points 5 months ago (2 children)

Wow.

2000x worse, huh?

I mean... I'm impressed that it runs, passes unti tests, and 'works', but is also that much worse.

That's a kind of achievement.

Not a useful kind, but... impressively bad.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] 15 points 5 months ago (2 children)

    Also it’s probably incredibly difficult to optimize a huge LLM-generated codebase since there are no human authors who know it intimately to begin with.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 8 points 5 months ago*

    Having a stable set of individuals with a deep understanding of 'how things work' is so totally anathema to the modern paradigm of 'every coder is is a contractor, basically'.

    Everybody wants to do software development, but doesn't want to foster software developers.

    So, they try and build machine god to replace us, and as most of us predicted... didn't work out so well, but goddamnit, they'll burn a trillion dollars before they let their ego take a hit.

    ... oh well, I guess.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 7 points 5 months ago (3 children)

    The 2000x difference is for more complex workloads. It has ok performance for very simple queries.

    So not quite as bad as the headline number suggests. But still very bad and not a viable alternative.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 7 points 5 months ago*

    I mean, on the one hand, its SQLite.

    On the other hand...

    ... arguably the entire point of a database language is to efficiently handle complex workloads.

    And then when you remember that... this was a project, in development, that cost time, money, energy, made RAM prices go up by maybe ¢22 per GB all on its own...

    This is an insane negative return on investment.

    Like imagine if you paid the same amount of money to ... people, a contracted firm, and they handed you this.

    You'd potentially be firing them or suing them for breach of contract, blacklisting them as far and wide as you could.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 3 points 5 months ago (1 child)

    I couldn't find if they were able to fix the identified bugs, seems like an important detail. How far does a month of LLM plus a month of talent get you?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 5 points 5 months ago (1 child)

    They probably dont care. They did this to generate headlines about how capable their AI is. It has served its purpose. So long as all of the investors only saw the propaganda articles the line will only go up and they can abandon this project.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 5 months ago*

    The propaganda articles about how the LLM missed critical logic and that it performs worse than SQLite?

    I'm less less interested in the extreme skepticism or hype.

    The project is an impressive demonstration from a pure technical perspective. I couldn't imagine 5 years ago a model being able to rewrite such a complex project.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 18 points 5 months ago (1 child)

    Reminds me of the Claude fluff article that mentions that reverts have only increased 0.04%.

    It made big noise that the number of Pull Requests has doubled though.

    Logically because you have one PR from an LLM, then another by a human to fix the LLM slop.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
    • [–] 4 points 5 months ago (2 children)

    Tbh I see myself using AI for shits and giggles. (nothing helpful)
    I try not to use it alot due to the ethics it comes with it.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] 6 points 5 months ago*

    In fairness to LLMs, (which I run locally) I've been able to use them for like, bits of code that are roughly 200 lines or less.

    Or like, feed it a code base and say hey, make sure all the comments are formatted the same way.

    But uh, for... trying to engineer an entire system?

    Nope nope nope, they get very confused, very fast, as overall conplexity increases.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 3 points 5 months ago

    @rimu How does it get worse at all?! SQLite has quite a few vtables, and Rust has monomorphisation, so it should be possible to do better on benchmarks.

    (I have occasionally thought about rewriting SQLite in Rust before regaining my senses.)

  • permalink
  • fedilink
  • source
    •
    [–] 2 points 5 months ago

    There are some things it can do well, like collecting / organizing certain data out of large documents. Sort of like a recursive-multi-google operation

  • permalink
  • fedilink
  • source
    •
    [–] 1 point 5 months ago

    @rimu "Fake it 'till you make it" does not work with artefacts. It may help you form habits, but if it's not your attitude that is the problem, but rather the tool, then "Fake it 'till you make it" *will lead to the emperors new clothes* .

  • permalink
  • fedilink
  • source
    •