This is so fucked up. The AI company has the perfect answer and yet it rolls the die to recreate the same thing by chance. What are they expecting, really?

You're still correct. The thing about LLMs is that they're statistical models that output one of the most likely responses, from the list of most likely responses. It still has some randomness though. You can tune this, but no randomness is shit, and too much randomness sometimes generates stupid garbage. With a large enough output, you're statistically likely, with any randomness, to generate some garbage. A compiler is sufficiently large and complex that it's going to end up creating garbage somewhere, even if it's trained on these compilers specifically.