I've noticed this in my own llm programming. You really have to review everything it does, because sometimes I'll be like "test this program with that input" and it will literally write a python program that doesn't do anything except print out a bunch of statements that say "test successful".

It also gets stuck in loops a lot so you have to keep an eye on it while it works. Pretty much if anything takes any longer than expected you just want to cancel and try again. It also keeps putting emojis in every single file no matter how many times I tell it not to. It's like it's taunting me.