frisbird comments on Stop Thinking of LLMs as Next-Token Predictors

[–] frisbird@lemmy.ml 4 points 1 month ago (3 children)

The whole thing with R’s in strawberry hasn’t been true for a while now.

I don't know why you think this is true. Transformers literally do not have any concept of algorithms (like the ones humans use to count). You have to add a "skill" - a deterministic implementation of an algorithm - and then you need to use RL to tune the parameters until the transformer outputs the tokens required to pipe the output of the skill to the user instead of producing a normal text output.

Turns out you can use RL to get the model to do basic calculation

Absolutely not. That's just not how transformers work. It's a technical impossibility. Maybe you're talking about non-transformer LLMs, like Mamba, but very few people are using Mamba-based LLMs outside of the research field. When we move beyond transformers, yes, counting is something they can do because they're effectively FSMs.

Notably, this is the exact same problem humans have.

Not really. It might have similarities, but I would never say it's the exact same problem.

The way our brains work is also stochastic

Sure.

and we struggle to do complex math in our heads

But for different reasons. Transformers because they are state destroying. Humans because they have limited and volatile working memory.

But of course, we can reinforce train ourselves to get better at it.

We can reinforce train ourselves to get better at math in so many different ways. We can memorize facts. We have metacognition and can often assess when we do or don't know a fact. We can develop cognitive behavioral algorithms that model physical systems. We can develop cognitive behavioral algorithms that model abstract systems. We can combine all of these together in nested, iterative, and recursive structures. It has only a little bit to do with reinforcing the association between abstract symbolic tokens.

And what we typically do is use an external aid like pen and paper to work through problems, which is basically no different from an LLM harness

It's certainly different because pen and paper are most often used to enhance working memory, but the fact-based reasoning and the algorithmic state machines are encoded in our brain which is impossible for a transformer LLM. It remains to be seen what Mamba-likes are truly capable of, but they certainly are addressing many of the critiques I'm raising.

If you hook up an LLM to REPL in octave, then it can do math quite well all of a sudden. [...] Understanding the limitations of LLMs and how to use them effectively requires moving past reductive thinking

It also requires understanding how things actually work. Hooking up the LLM to a REPL and then iteratively fine tuning it changes the model from outputting a stochastic answer via next-token prediction to outputting a stochastic algorithm via next-token prediction (that will answer the question for you). The LLM did not get better at doing math. It was reweighted to answer questions in the form of "here's a solution that will answer your question for you since I cannot answer you because I have no ability to do math", and within that structure you will STILL get hallucinations and can still fuck the model up by posing sufficiently complex or misdirecting word problems. But the corpus for converting word problems to algorithms is massive, and combined with fine-tuning AND ensemble sampling (which increases your real inference cost in multiples) you're going to produce decent algorithms from math word problems relatively consistently.

Which is the same way it gets better at coding and yet still can't actually solve complex problems in design space, constantly has to use ensemble sampling, and constantly has to be told to re-roll the dice whenever the test fails. And that behavior is so costly under the hood that it's eye watering.

While token generation is the base operation, focusing on that is like trying to understand the brain by looking at individual neuron firings.

Not really. Watching individual neuron firings would be equivalent to watching individual parameter weights and the outputs of each step of the transformer. Token generation is literally the entire functioning of transformers.

What’s actually interesting in both cases are the high level patterns that end up being produced which I’d argue are substrate independent

Yeah, patterns are, by definition, substrate independent. But transformers only maintain high level patterns on a per-token basis. High level patterns can and do emerge from weighted parameter space, and in many surprising ways, but they are fundamentally limited in transformers because transformers are, at base, next-token predictors so even though we get emergent high-level patterns that can, for example, sort lists, we STILL get hallucinations specifically because the high-level patterns are ephemeral on a per-token basis. Reinforcement learning and back propagation can only go so far in fitting the parameter space before it results in contention with other outcomes.

Meanwhile, a combination of an LLM with a harness can be seen as a type of a neurosymbolic system. The neural network generates novel patterns, while the symbolic engine provides the rails for it to function within.

Yes. Inference -> fitness check -> iterate. Agentic retry. It's incredibly expensive precisely because it uses next-token predictors to generate an answer with an already-known fitness algorithm and then just re-runs inference until the answer passes the fitness test. It's an automated human-in-the-loop system where a human says "No, that's not right, try again" and we all know how quickly we run of free tokens when we do that, and we've also all had the experience of the damn thing never getting anywhere near close to the solution after a dozen attempts at reprompting.

Yes, modern transformer harnesses do a TON of work and actually make these parrots useful instead of novelties. But it doesn't change the fact that they are fundamentally statistically weighted parameter-space stochastic next-token predictors, no matter how much you add to them.

Instead of arguing against the technical reality, why not focus on the truth about the harnesses - they add a ton of value and make next-token prediction much more useful in some contexts, especially contexts like producing working code.

permalink

fedilink

source

parent

hideshow 3 child comments

[–] yogthos@lemmy.ml [S] 1 point 1 month ago (2 children)

I don’t know why you think this is true.

I mean you can just try it with DeepSeek or any other large model yourself. This is literally a solved problem now.

Absolutely not.

Evidently you need to read up on how reasoning chains work.

Not really. It might have similarities, but I would never say it’s the exact same problem.

It literally is the same problem. Your brains didn't evolve to do formal logic natively. We emulate it exactly the way the LLM does.

But for different reasons. Transformers because they are state destroying. Humans because they have limited and volatile working memory.

No, for the same fundamental reasons. It's got nothing to do with state being destroyed either. It has to do with the fact that stochastic systems aren't a good fit for doing symbolic logic.

It’s certainly different because pen and paper are most often used to enhance working memory, but the fact-based reasoning and the algorithmic state machines are encoded in our brain which is impossible for a transformer LLM.

Fact based reasoning is something our brains are famously terrible at doing actually. That's why we use tools like computers in the first place. Our brains can be trained to express patterns of formal logic, and an artificial neural network can be trained to do the same thing. That's why modern LLMs can reliably tell you the number of R's in strawberry.

It also requires understanding how things actually work.

It doesn't, that's the whole beauty of genetic algorithms. All you have to do is specify your selection pressures and your goal criteria, and the system evolves a solution to fit the shape your desire. The LLM doesn't need to get better at doing math, the stochastic approach means it converges on a solution given the right environmental pressures. And that's why hallucinations don't matter, they get weeded out by the attempts being tested against the environment.

Which is the same way it gets better at coding and yet still can’t actually solve complex problems in design space, constantly has to use ensemble sampling, and constantly has to be told to re-roll the dice whenever the test fails. And that behavior is so costly under the hood that it’s eye watering.

I can tell you haven't actually worked with these tools recently.

Not really. Watching individual neuron firings would be equivalent to watching individual parameter weights and the outputs of each step of the transformer. Token generation is literally the entire functioning of transformers.

As a communist, I expect you to understand the concept of quantity transforming into quality.

Yeah, patterns are, by definition, substrate independent. But transformers only maintain high level patterns on a per-token basis. High level patterns can and do emerge from weighted parameter space, and in many surprising ways, but they are fundamentally limited in transformers because transformers are, at base, next-token predictors so even though we get emergent high-level patterns that can, for example, sort lists, we STILL get hallucinations specifically because the high-level patterns are ephemeral on a per-token basis.

Do explain how this is different from saying that human brains are fundamentally limited in that neurons are just next state predictors.

Yes. Inference -> fitness check -> iterate. Agentic retry. It’s incredibly expensive precisely because it uses next-token predictors to generate an answer with an already-known fitness algorithm and then just re-runs inference until the answer passes the fitness test.

Except it's not incredibly expensive because the system works on the principle of gradient dissent. It isn't just producing a random value each turn, it produces a plausible value within the context which is precisely what allows it to quickly converge on a solution.

Yes, modern transformer harnesses do a TON of work and actually make these parrots useful instead of novelties. But it doesn’t change the fact that they are fundamentally statistically weighted parameter-space stochastic next-token predictors, no matter how much you add to them.

Exactly the way the neurons in your brain are stochastic next state predictors.

Instead of arguing against the technical reality, why not focus on the truth about the harnesses - they add a ton of value and make next-token prediction much more useful in some contexts, especially contexts like producing working code.

I would urge you to spend a bit of time actually learning about the technical reality instead of continuing to argue here.

permalink

fedilink

source

parent

hideshow 2 child comments

[–] frisbird@lemmy.ml 4 points 1 month ago (1 child)

Yog, you know I respect you, but we have fundamentally different understandings here and I think yours is wrong.

An individual neuron is not a next state predictor. An individual neuron is analogous to a NN parameter. It's not exactly like a NN parameter, it's more complex, which allows animal neural networks to do a lot more than the NNs we have built, but it's an analogy. It's more appropriate to say that a NN parameter is attempting to be analogous to an animal neuron. It's not there yet, and no amount of training will get it there, because of structural differences.

The insistence that LLMs are exactly like human brains is reductive. Calling an LLM a next token predictor is not reductive. Adding a harness doesn't make LLMs not next token predictors. And the difference been an animal brain and an LLM Is not the quality of the harness nor the amount of training.

You're right. My understanding of the use of ensemble methods was outdated. Everyone has moved over to mixture of experts routers, a non-emergent property designed and engineered into the transformer. They still generate the next token, but they use a hard coded algorithm to generate a bunch together and then weight the results.

Yes, reasoning models are doing math. But the didn't learn to do math better. Reasoning models uses a hidden text buffer where the LLM produces intermediary tokens that are statistically most likely to produce next tokens that will pass the fitness requirements. Through RL, reasoning models began to statistically weight patterns of tokens that resulted in rerunning the "math" multiple times, because it produced more and more context that constrained the probability space. This is not like how human reasoning works in the vast majority of cases, and calling it "exactly like" human reasoning is begging the question.

As for human analog, the "number of letters" test that most models have gotten better at (but still hallucinate on sometimes), also succeed on nonsense words that are not in their training set. And they work because they are engineered, hard-coded, to break down words into individual characters. That is not an emergent property of the NN like it is in humans.

I recognize that changes in quantity can induce changes in quality, and the emergent functional structures in energized parameter space of LLMs is exactly that, and that some of those things are analogous to what happens in animal neurology. But it's not an exact match, as you are saying, and it's not all the same structures.

When we force humans into next token predictors, they are surprisingly convergent. 70% of times that someone is asked to pick a number between 1 and 10, they pick 7. LLMs exhibit that behavior too, but at 90%. The reason LLMs do it is because they are trained on data that constantly conjoins text about guessing a random number and the curious human behavior of guessing 7. They produce similar, analogous behavior but for completely different reasons.

Back to the article, the claim that reinforcement learning makes them "not next token generators" or that somehow calling them "next token generators" is reductive just doesn't follow. NNs have been producing emergent behavior long behavior LLMs and transformers. The emergent behavior is worth point out to people who think that the output is entirely random (which is reductive). But the fact that transformers are fundamentally next token generators is constantly reinforced by all the ways that frontier model engineers have to design their transformer algorithms - everything becomes about producing tokens in a sequence that constraints the probability space.

And the point about transformers having ephemeral state on a per token basis, that's how transformers work, even with the mixture of experts routers. That's exactly why they give transformers scratch space for "reasoning", because they are trying to capture the value of the emergent NN structures in the token stream its generating. Contrast this with Mamba models that have a persistent working memory directly in their models and don't rely exclusively on the session history of token stream for this response to probabilistically re-energize the same emergent structures. This is why I don't think transformer-based "reasoning chains" are analogous to human reasoning chains. They might be analogous to a very small subset of limited human mental behaviors, but that's not the same thing.

permalink

fedilink

source

parent

hideshow 1 child comment

[–] yogthos@lemmy.ml [S] 2 points 1 month ago

Right, I'm not saying they are equivalent or that LLMs work exactly like the brain. We completely agree that LLMs on their own do not do what brains do. I'm talking about the general principles here. My key point is that a lot of simple behaviors can combine together to produce complex emergent behaviors.

The real question is whether high level patterns human brains produce, which we agree are substrate agnostic, can be expressed within an artificial system. The underlying mechanics of the two systems don't have to be the same. Similarly how a combustion engine and an electric one both achieve the same end result in a very different way, it seems very likely to me that the patterns the brain produces can also be supported by a completely different substrate. I've seen no evidence to suggest that there is something special about biological substrate that makes it a unique platform in this regard.

The reason why saying LLM is a next token predictor is reductive is because this view ignores all the complexity associated with the token being produced. It's similar to the way we use language a single word at a time, and you could say my brain is the next word predictor. Yet, each word exists within a context, and it's a token associated with a web of ideas in my head at any one time. Ignoring all that and focusing solely on the word I produced is reductive and completely misses how my brain actually arrived at that word.

This is not like how human reasoning works in the vast majority of cases, and calling it “exactly like” human reasoning is begging the question.

What evidence is there that this is not how human reasoning works, pretty much all the evidence that I've seen strongly suggests that human reasoning is very much a stochastic process. Again, the algorithm is almost certainly different, but the principles appears to be very much the same.

As for human analog, the “number of letters” test that most models have gotten better at (but still hallucinate on sometimes), also succeed on nonsense words that are not in their training set. And they work because they are engineered, hard-coded, to break down words into individual characters. That is not an emergent property of the NN like it is in humans.

Humans hallucinate all the time and produce nonsense statements regularly. And just because something is engineered doesn't mean it works on a different principle. It's entirely possible for evolution to come up with a similar solution to one that's consciously designed. The selection pressure is thermodynamics in both cases. Either a system evolves towards a certain state being moulded by energy gradients, or it is designed towards that state because the designer is trying to minimize energy usage and maximize performance. So, whether something is emergent or not is not really a relevant point here.

And you're right, transformer is just a piece of a much bigger puzzle. In my opinion, active memory formation and dynamic learning are likely the next steps. The real elephant in the room is that we need to delineate training and inference phases. That's the clearest sign of the limitations of the LLM approach. A human brain can learn efficiently on the fly from few shot examples, and to integrate new information continuously. The current approach is not efficient enough to allow for that, and the fact that the model needs a huge numbers of samples to identify invariant patterns within them itself suggests that there is something wrong with the approach.

There is some research being done in this direction already such as this paper, there are ideas for creating an equivalent of short term memory that's used in the active context, and then these memories get consolidated into longer term memory layers. So, you could have a base model that is pretrained, and then have active learning on top of that which might get a lot closer to having the plasticity our brains exhibit. Another really interesting approach is MemOS which would allow packaging up skills into composable building blocks that can be shared. So, a model could learn a task, package it up and then another model can reuse it.

It's also possible that the whole transformer approach is a dead end entirely. For example, SpkingBrain looks like a promising alternative.

So, to sum up, I think LLMs exhibit a lot of similar behaviors to biological brains, which suggests they likely operate on similar underlying principles. But I don't think they're equivalent to brains, and at best they mimic what specialized brain structures are doing and you would need to build out other structures to get closer to mimicking what the brain does as a whole. But it's also possible that they're just a dead end entirely, and that this approach is not efficient enough to do on the fly learning which is a key aspect of general intelligence.

permalink

fedilink

source

parent