9
[AIT] what is your selfhosted local LLM stack? (lemmy.blahaj.zone)
submitted 1 day ago by to c/selfhosted@lemmy.world
39 comments fedilink hide all child comments
 

What tools and projects do you use for your local LLM chats and agents?

I'm trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.

you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 19 hours ago (6 children)

I'm sure OPs RTX 3060 will have a tolerable effect on the school children and the rainforest.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 5 points 18 hours ago (5 children)

    @FauxLiving @YeahToast "I'm sure my driving to work will doom the world" same mentality.

    Also, it's not so much use of a model as the creation of the model. Training is expensive in terms of resources and energy. Using the model and normalizing AI usage leads to more AI usage and more AI normalization, which leads to new models being trained..

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 1 point 15 hours ago (4 children)

    Put it into perspective:

    • Everyone doing their laundry uses more water than all data centers combined. Golf courses also use vastly more water.
    • Data centers use about 1-2% of global electricity.

    People driving cars pollutes more as does global shipping, regular old consumerism, etc. The environmental impact of data centers is negligible in the grand scheme of things.

    Now that AI is making open source software the most secure software ever written (the hard way, hehe), not using it to scan your stuff is now considered worst practices and basically idiotic negligence. No company that uses software or anything with firmware can afford to ignore it anymore.

    There's no way out of it. Accept the new AI reality and push for good regulations. Example: If we made it so that all AI data centers had to be powered via renewable energy on site that would allay most of the energy use/environmental concerns.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 3 hours ago

    I don't buy it. If Datacentres were the powersipping butterflies you suggest, why all the new fossil power plants everywhere? I would probably not have given a shit about the datacentres, if not for all the new fossil pollution that is part and parcel of the construction.

    Lies, damn lies, and statistics.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 14 hours ago* (last edited 14 hours ago) (2 children)

    @riskable Justify using an unethical technology however you want, but it's still the same basic argument that drivers often make (which is what I was pointing out). There is always some worse impact to the environment, and that is why we don't need to change our behavior.

    Ethics are for suckers, I guess?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 13 hours ago (1 child)

    AI isn't inherently unethical. The very premise of saying such a thing is flawed.

    Is it unethical to use AI to find new drugs and medical treatments? To use it for physics research? What about regular research? Writing code?

    Where do you draw the line?

    Also, AI was used to design the current crop of fusion reactors and the full potential of that is only just getting started. If it turns out that AI solves all the issues and generates a working fusion reactor design, it'll make every one of your arguments moot. Because that one achievement will be worth everything. It would "save the world" so to speak (from an energy/emissions standpoint).

    Chatting with an LLM is less impact than driving a car. Think about that for a second before you say using it is unethical. It's literally more negative for the world for someone to drive a car every day to work than it is to ask an AI to find something on the Internet or write a program in Python.

    Or perhaps your underlying complaint is more about capitalism than it is about AI?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 12 hours ago

    How AI was trained is unethical. There was no consent, limited renumeration to individuals for using their work. If I drive to work, I provide value to people . Having AI scrape and summarize searches (incorrectly at times) has no benefit. Forcing all work meetings to use AI, has no benefit.

  • permalink
  • fedilink
  • source
  • parent