LocalLLaMA

2292 readers

2 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 2 years ago

MODERATORS

[email protected]

Can you fine-tune on localized steering of an LLM? (lemmy.dbzer0.com)

submitted 2 days ago* (last edited 2 days ago) by [email protected] to c/[email protected]

18 comments fedilink hide all child comments

I want to fine tune an LLM to "steer" it in the right direction. I have plenty of training examples in which I stop the generation early and correct the output to go in the right direction, and then resume generation.

Basically, for my dataset doing 100 "steers" on a single task is much cheaper than having to correct 100 full generations completely, and I think each of these "steer" operations has value and could be used for training.

So maybe I'm looking for some kind of localized DPO. Does anyone know if something like this exists?

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 2 points 1 day ago

I dont know what you mean with steering?

Do you want a given output structure, like json or toml?
Do you want to align the model, with your dataset of question and answer pairs?

First of all, have you tried giving the model multiple examples of input output pairs in the context, this already helps the model a lot to output the correct format.

Second you can force a specific output structure by using a regex or grammar: https://python.langchain.com/docs/integrations/chat/outlines/#constrained-generation https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md

And third, in case you want to train a model to respond differently and the previous steps were not good enough, you can fine-tune. I can recommend this project to you, as it teaches how to fine-tune a model: https://github.com/huggingface/smol-course

Depending on the size of the model, that you want to fine-tune and the amount of compute that you have available you can either train by updating all parameters like ORPO or you can train via PEFT (LoRA)