you are viewing a single comment's thread
view the rest of the comments
[–] 10 points 1 day ago (2 children)

I know, but even regarding bug catching, there have been a LOT of bogus bug reports "discovered by AI" that could not be replicated in any way, shape or form. Granted, I've seen one or two articles saying that AI helped find real software bugs, but what's the ratio there, 1:500 for useful/useless AI bug reports?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 19 hours ago

    Granted, I’ve seen one or two articles saying that AI helped find real software bugs, but what’s the ratio there, 1:500 for useful/useless AI bug reports?

    It's worth distinguishing between LLM generated bug-reports submitted by randos, and developers themselves using LLMs to look for bugs in their software.

    My own experience, with my own code-bases, is that the LLM I use is extremely good at finding real flaws in my code. It'll also nitpick a lot, but it has has become better at triaging issues than it used to be, so that kind of thing is easy to ignore.

    Because of that, having an LLM take a look at my code before I push it has become another QC step, along with running tests, linters, and formatters. The end result is higher quality software

  • source
  • parent
  • [–] -4 points 1 day ago*

    bogus bug reports “discovered by AI” that could not be replicated in any way, shape or form

    And we NEED to be more serious about use of secure verifiable identities for people so we can properly filter inputs from internet contributions. Anonymous sources should be de-prioritized to just above the known consistently bad actors. That's nothing about AI or LLMs, that's just the world continuing to be far too trusting of basically anonymous contributions.

    If you want to submit a bunch of AI slop under your secure ID, be prepared for that ID to be de-prioritized beneath anonymous contributions.

    If you want submissions from your secure ID to be taken seriously, at least task your LLM with building a proof-of-concept replication of the bugs in the reports your other LLM session creates before submitting it for humans to spend their time evaluating.

    We have a human test department overseas that gets a little eager to pump up their metrics once in a while, we haven't resorted to penalizing them (yet) but when they send an "observation" with no way to reproduce it, no log files to back it up, we tend to ignore that observation until the review meeting and then in the review meeting ask "have you seen this again? do you have any way to reproduce it? do you have screen shots or log files?" and when all those things are missing, the observation is closed with a "cannot reproduce" tag, waiting for another tester to see it again and produce more evidence.

  • source
  • parent