8
[AIT] what is your selfhosted local LLM stack? (lemmy.blahaj.zone)
submitted 1 day ago by to c/selfhosted@lemmy.world
37 comments fedilink hide all child comments
 

What tools and projects do you use for your local LLM chats and agents?

I'm trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.

you are viewing a single comment's thread
view the rest of the comments
[–] 0 points 9 hours ago* (3 children)

If OP is doing orbital dynamics in their day job, they probably live in California. JPL, Raytheon, LLNL, LBL, and Berkeley are all here.

Even if they don't have rooftop solar, most of our grid is renewable. Their local GPUs have a decent chance of being more environmentally friendly than charging your cell phone, depending on where you live.

And using it for science and programming is really different than using ChatGPT to write you an email. I'm home right now, but I have two agents at work running debug and trial experiments, and they'll be running it all night until I get in in the morning. I'll review their work and steer them all day while I'm in the office; and while those two completely different projects are running, I'll be personally working on a third. They are enormous productivity multipliers when you have a well-defined and deterministic target.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 8 hours ago (2 children)

    My grievance isn't so much with current household models with PCs sitting there running off solar (that's starting to get very bloody specific though). But how did you get to a point where you could get a local model on your PC. All the pillaging of content, hundreds of Billions to Trillion(s) of dollars spend on a product that didn't renumerate anyone who created their learning data.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 7 hours ago

    Unpopular opinion time: I'm torn as to where that theft line is.

    If a turn of phrase in Lord of the Rings inspires something new in me, do I owe royalties to the Tolkien estate?

    The stories on FanFiction.net are derivative by definition — is it theft? I love "Harry Potter and the Natural 20", a 4th-wall breaking D&D and Potter universe mashup story (seriously, it's hilarious). Does that story owe royalties to Wizards of the Coast and (ugh) JKR?

    Maybe those are super transformative. What about opinion pieces where the author redigests information from all over, many paid writings? Are they selling the original authors' product, even though restated?

    Maybe it's ok if they paid for the subs. But what if it came from free library books?

    My wall has a bunch of art from artists at SDCC. I'm looking at Spider Gwen; Thor, Iron Man, and Mjolnir Cap charging into battle; Kaladin and Syl before the Stormfather; and a map of all the places the Gaang went in ATLA. Those artists all ingested the original work to make those derivatives; were they not allowed to make them if they borrowed their friends' copy?

    It feels kinda wrong to vacuum up public data to train models, kinda? But I don't know that I can fairly call it immoral without condemning all these other things I think aren't immoral. I kind of want to, but that'd be really inconsistent, and that's profoundly frustrating in its own way.

    It's messy and complicated and I think the gut response of most people isn't actually a reasoned one.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 4 points 8 hours ago

    We involuntarily contributed to their creation, the least we get to do is enjoy the fruits of that theft. More should be given, financially reimbursement through taxation, but until that happens, free use of the technology is consolation.

    Besides, following through with your analogy, if a baby is born from rape - does that baby deserve hate for who their father is?

  • permalink
  • fedilink
  • source
  • parent