I am in charge of policy related to Ai at work, but under different circumstances than most. The people above me asked the question, can we use Ai in a way that adds value to the product? My response was let's test out a couple of options and make policy based on the results. We gave everyone access to Copilot as we use JetBrains products and I asked every individual what their results were after a month. Four people found that they were able to work faster, the rest said it slowed them down and the hallucinations outright fucked their shit up.

At first I couldn't get a grasp on how this sanely work for a whole company implemented in a one size fits all manner. Then 2 of the people who used LLMs that were Jr.s left the company. When they had to transfer ownership of the internal projects to other people. I realized the mess inexperience devs with LLMs could do. They left behind untouchable code. That's when I figured it out, the other two people were Sr.s and you need to know exactly what you are doing to get a net positive from LLM coding.

All three of us were doing slight variations of the same thing, which was very telling of what is going to happen to software over the next year. We would never touch an agent as they just inject flaws and bugs into the code base. We all found Tab complete to be useful enough to leave on, but noted it was worse that worthless quite frequently. What we were all doing was using it as a cracked out search engine through the chat features. Rarely would we accept offered changes directly, but we would manually type sections adapting them to fit better in the code, the way we all used stackoverflow 5 years ago.

I am not saying inexperienced dev can't use LLMs effectively. You just have to know exactly what you are doing. Now the others come to me talking about MCP servers, agentic coding, whatever new snake oil is being sold to us. Now I am trying to convince these people, who initially rejected LLMs, that they are being gaslit by salesmen into thinking that LLMs are useful.