Eventually, yes.
LLM's require expertise to supervise. Expertise requires practice to built and maintain. If you have the LLM do things, you're getting less practice in, which slowly degrades your expertise, which in turn degrades your ability to supervise the LLM.