13
Have you tuned a model? (lemmy.world)
submitted 4 months ago by to c/fosai@lemmy.world
15 comments fedilink hide all child comments
 

Apologies if this seems like a survey post. I’m just learning about tuning and want to get a lay of the land. I don’t think I have the money to tune locally so might have to rent some VRAM, but curious how much better tuning is vs something like RAG.

What model? What was your use case? What tuning tool did you use? What is hardware setup? How large was your training set and how did you create it? How effective was the model as tasks pre- and post-tuning?

Thanks!

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 2 months ago (1 child)

Personally, I've adjusted dozens of sampler values, written middlewares, llama-server scripts and configuration loading mechanisms, openai api compatible HTTP proxies, and even a python3 API for accessing context information and being able to switch models on the fly. I've even created a local model benchmark performance script.

But besides running some scripts which others have made to tune a model specifically with specific input parameters, not really. Honestly, I have a lot to learn.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] [S] 1 point 2 months ago (1 child)

    I’ve heard a lot about Lora like Unsloth. Have you used anything like it?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 2 months ago (1 child)

    lora, yes. mostly custom scripts downloaded off of hugging face to automatically handle a lot of complicated stuff I'm not totally sure of how it actually works under the hood to be honest

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 2 points 2 months ago* (1 child)

    Did you notice a big improvement in the tasks you were tuning for? What is your hardware setup and how long did it take?

    Thanks!

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 months ago (1 child)

    unfortunately, i did not notice much of a difference with model tuning. it took a pretty decent chunk of time. For my most powerful pc, which is what I run most models (the lower end machines with worse gpus run embedded text models) I got a fairly powerful machine with a single 4090. I have had better luck just downloading differently tuned variants of the same model from others

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 2 points 2 months ago (1 child)

    Bummer. Do you think it was the training data or just nature of fine tuning? Something else? What were you tuning it for if you don’t mind my asking?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 months ago (1 child)

    just the nature of them being quite old models without proper tool calling functionality. What actually DID help was setting up middleware and custom python servers/clients with proper json mapping to enable the proper tools to be selected. so, literally zero model tuning required in the end.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •