Because weight x = y isn't really a satisfactory answer to the "why" in question though. To really answer that you need to refer to the pretraining and context, which in a lot of cases today, and likely forever, since there's no reason for anyone to cold start model training ever again, is necessarily opaque. A lot of the best why answers are going to be given by the model itself (which... obviously circular). Trust and attribution are just really hard in this paradigm. Not necessarily so different from the challenge of trusting/attributing human output, except that humans are more diverse and in general difficult to manipulate.

AI is not deterministic at all, only if you keep the seed values for generating random numbers the same all over the pipeline.

It's literally a huge stochastic model.

I had the misfortune of working a lot with claude skills for the past few months. I run the exact same skill, on the exact same codebase and it doesn't yield the same results. Sometimes it straight up ignores parts my prompt. Other times it ignores the skill file entirely. And then I ask Claude, why did it do X thing that it wasn't supposed to do? The answer is "I apologize, I should have blabla". It is extremely unreliable and unpredictable. Now I'm stuck with a feature that I'm responsible for that occasionally goes rogue. Fun times.

Nearly all evaluation steps for an LLM I produce some level of entropy. Yes you can evaluate an LLM model deterministically, but we don't. We even tune the levels to get good results. Model cards have sampling parameters supplied to give "the best" results. For example, Qwen 3.8 flash next gives

We recommend using the following sets of sampling parameters for generation:

Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Replaying the seed will only get you the same result using a particular engine and a particular random number generator. Run it on llama.cpp Vs Strata and you can get different results. Different libc, different results.

A compiler is systematically deterministic. No calls to rand()

You make a good point about capability. However, I'm trying to say that in practical capability, I do know exactly what compiler flags were enabled (it's whatever the Makefile passed, which is whatever configure determined, the whole thing is checked in as source code), but the exact model+seed isn't being recorded for this Linux changes.

To make the compiler analogy, we should be saving model, seed, prompt, etc and then the LLM output becomes the equivalent of the compiled code. I think that's an extremely defensible argument, while "bet you can't write assembly!" is not.