213
AI Software Development – What Does The Data Say? (codemanship.wordpress.com)
submitted 2 weeks ago* (last edited 2 weeks ago) by to c/programming@programming.dev
67 comments fedilink hide all child comments
 

studies show a clear trend – output is up (more code, more commits, bigger diffs), but outcomes don’t reflect that trend. If anything, the average team is taking longer to ship worse software

you are viewing a single comment's thread
view the rest of the comments
[–] 16 points 2 weeks ago (3 children)

This aligns with my experience, largely. Of course it's still my job to maximize LLM effectiveness within my organization. Which is a delicate balancing act to protect my teams from overeager executive leadership looking for huge gains.

My own summary is that AI can be an accelerator, but the harder you lean into it, the worse outcomes will be. No matter how much code is written, you still need actual human minds to understand it and they can only handle so much volume before getting overwhelmed.

Also, if AI gives you 20% productivity gains, but that 20% goes into playing with AI trying to get more, you haven't really gained anything. Usage needs to be standardized rather than developers constantly negotiating with AI trying to coax out better outcomes.

  • permalink
  • fedilink
  • source
  • hideshow 6 child comments
    •
  • [–] 6 points 2 weeks ago

    The impression I've gotten, fooling around with it at home and talking to friends in tech and hearing from actual users online is:

    A good developer can develop faster with it. Giving it small, discrete tasks for first drafts or throw away code (like bash scripting) can work well.

    It's better google. If you are trying to figure out if a function that will do a thing exists, or are trying to figure out what architecture would work best in a given situation, it can be helpful. But in these cases, it should be used carefully - dont ask it to do your work for you, ask it to give you options, pros and cons, and sources. But in this regard, it can do a lot to help an experienced developer become more productive faster in a stack or tool they are unfamiliar with.

    Vibe coding is a real thing, and it can work. For internal tools in a small company, a non-technical person can create a mostly functional piece of software to get a job done. My expectation is that over time, these people will become real developers, as they end up dealing with bugs and edge cases in the vibe code they created.

    At the top end of ai-for-software-development, there is some sort of something with automated iterative looping and verification, where a developer can translate a set of requirements into code, and then a collection of ai agents iteratively develop the code until it works as expected. This is what the tech bros seem really hyped on, and it does seem to work... but at the same time, my feeling is that this is how you get multimillion dollar AI bills. And presuming this is how big tech is developing their products - it seems prone to making inefficient, buggy code, so I don't think it will be worth it long term.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 4 points 2 weeks ago

    This is a tale older than AI. Most of the AI productivity pushes I struggle to get adopted fail not because of AI bad or its too hard to do. They fail because of a broken CI/CD pipeline. They fail because some team thinks their process is sacred and unique.

  • permalink
  • fedilink
  • source
  • parent