this post was submitted on 20 Dec 2024
12 points (92.9% liked)

LocalLLaMA

2292 readers
3 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 2 years ago
MODERATORS
[email protected]
[email protected]
[email protected]
12
Can you fine-tune on localized steering of an LLM? (lemmy.dbzer0.com)
submitted 2 days ago* (last edited 2 days ago) by [email protected] to c/[email protected]
18 comments fedilink hide all child comments
 

I want to fine tune an LLM to "steer" it in the right direction. I have plenty of training examples in which I stop the generation early and correct the output to go in the right direction, and then resume generation.

Basically, for my dataset doing 100 "steers" on a single task is much cheaper than having to correct 100 full generations completely, and I think each of these "steer" operations has value and could be used for training.

So maybe I'm looking for some kind of localized DPO. Does anyone know if something like this exists?

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 1 points 2 days ago* (last edited 2 days ago) (1 children)

Can SFT be used on partial generations? What I mean by a "steer" is a correction to only a portion, and not even the end, of model output.

For example, a "bad" partial output might be:

<assistant> Here are four examples:
1. High-quality example 1
2. Low-quality example 2

and the "steer" might be:

<assistant> Here are four examples:
1. High-quality example 1
2. High-quality example 2

but the full response will eventually be:

<assistant> Here are four examples:
1. High-quality example 1
2. High-quality example 2
3. High-quality example 3
4. High-quality example 4

The corrections don't include the full output.

[–] [email protected] 2 points 2 days ago (1 children)

I do not know what SFT means. So I can't comment on that, I'm afraid.

Models only predict the distribution of the next token. So "partial response" vs "full response" is a consequence repeated inference untill the stop token is reached. It's mostly unrelated to the model parameters.

For training, it makes no difference.

[–] [email protected] 1 points 2 days ago (1 children)

The article you linked to uses SFT (supervised fine tuning, a specific training technique) as its alignment strategy. There are other ways to fine-tune a model.

I guess I'm wondering if you can train on these partial responses without needing the full rest of the output, without the stop token, or if you need full examples as the article hints to.

[–] [email protected] 1 points 2 days ago (1 children)

I was unaware of that acronym, thank you. It does make me wonder: is there unsupervised training of LLMs?

Yes, you can train without the stop token. The stop token is just that: another token. I do not expect the model to unlearn the usage of the stop token from training on a few 100 new examples.

[–] [email protected] 1 points 2 days ago (1 children)

Unsupervised training happens during the pre-training phase when you dump all kinds of quality documents and it learns the relationship between tokens

[–] [email protected] 1 points 2 days ago (1 children)

Could you perhaps share a reference for this? I'm eager to learn as I don't quite understand.

I've always trained LLM supervised: predict token N+1 based on tokens 1 to N.

[–] [email protected] 1 points 2 days ago (1 children)

This pre-training was done by Meta. It's what Llama-3.1-405B is (in contrast to Llama-3.1-405B-Instruct). https://huggingface.co/meta-llama/Llama-3.1-405B

Training Data

Overview: Llama 3.1 was pretrained on ~15 trillion tokens of data from publicly available sources. The fine-tuning data includes publicly available instruction datasets, as well as over 25M synthetically generated examples.

[–] [email protected] 1 points 2 days ago* (last edited 2 days ago) (1 children)

Oh I see the origin of my confusion. The terminology "supervised learning" got repurposed.

It's all supervised learning if the model is learning the relationship between input and expected output (using supervised learning as described in (1)). The methodology of "pre-training" is the same as that of "supervised fine tuning".

There's no unsupervised learning happening, as described in (2)

[–] [email protected] 1 points 2 days ago* (last edited 2 days ago) (1 children)

No, it's unsupervised. In pre-training, the text data isn't structured at all. It's books, documents, online sources, all put together.

Supervised learning uses data with "ground truth" labels.

[–] [email protected] 1 points 2 days ago (1 children)

Have you worked with (variational) auto-encoders? I think they're a great example of what I would call unsupervised learning.

Supervised learning uses data with "ground truth" labels.

What are "ground truth" labels?

[–] [email protected] 1 points 2 days ago (1 children)

Ground truth labels are just prescriptive labels that we recognize as being true. The main thing that distinguishes unsupervised from supervised is that in unsupervised learning, what is "good" is learned from the unstructured data itself. In supervised learning, what is "good" is learned from some external input, like "good" human-provided examples.

[–] [email protected] 1 points 2 days ago (1 children)

Would you call token (N+1), given tokens (1 to N) as a ground truth?

[–] [email protected] 1 points 2 days ago (1 children)

No, in that case there's no labelling required. That would be unsupervised learning.

https://en.wikipedia.org/wiki/Unsupervised_learning

Conceptually, unsupervised learning divides into the aspects of data, training, algorithm, and downstream applications. Typically, the dataset is harvested cheaply "in the wild", such as massive text corpus obtained by web crawling, with only minor filtering (such as Common Crawl). This compares favorably to supervised learning, where the dataset (such as the ImageNet1000) is typically constructed manually, which is much more expensive.

[–] [email protected] 2 points 2 days ago (1 children)

So supervised vs unsupervised, according to you, is a property of the dataset?

[–] [email protected] 0 points 2 days ago (1 children)

Sorry, I really don't care to continue talking about the difference between supervised and unsupervised learning. It's a pattern used to describe how you are doing ML. It's not a property of a dataset (you wouldn't call Dataset A "unsupervised"). Read the Wikipedia articles for more details.

[–] [email protected] 1 points 2 days ago* (last edited 1 day ago)

It's alright :)