2219
Save The Planet (lazysoci.al)
submitted 1 year ago by to c/microblogmemes@lemmy.world
288 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 12 points 1 year ago* (9 children)

I know she's exaggerating but this post yet again underscores how nobody understands that it is training AI which is computationally expensive. Deployment of an AI model is a comparable power draw to running a high-end videogame. How can people hope to fight back against things they don't understand?

  • permalink
  • fedilink
  • source
  • hideshow 14 child comments
    •
  • [–] 25 points 1 year ago (2 children)

    I mean, continued use of AI encourages the training of new models. If nobody used the image generators, they wouldn't keep trying to make better ones.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
    • load more comments (1 reply)
  • [–] 20 points 1 year ago (22 children)

    It's closer to running 8 high-end video games at once. Sure, from a scale perspective it's further removed from training, but it's still fairly expensive.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 25 child comments
    •
  • [–] 2 points 1 year ago (4 children)

    really depends. You can locally host an LLM on a typical gaming computer.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] 5 points 1 year ago* (last edited 1 year ago)

    Well that's sort of half right. Yes you can run the smaller models locally, but usually it's the bigger models that we want to use. It would also be very slow on a typical gaming computer and even a high end gaming computer. To make it go faster not only is the hardware used in datacenters more optimised for the task, it's also a lot faster. This is both a speed increase per unit as well as more units being used than you would normally find in a gaming PC.

    Now these things aren't magic, the basic technology is the same, so where does the speed come from? The answer is raw power, these things run insane amounts of power through them, with specialised cooling systems to keep them cool. This comes at the cost of efficiency.

    So whilst running a model is much cheaper compared to training a model, it is far from free. And whilst you can run a smaller model on your home PC, it isn't directly comparable to how it's used in the datacenter. So the use of AI is still very power hungry, even when not counting the training.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 1 year ago* (last edited 1 year ago)

    Not at all. Not even close.

    Image generation is usually batched and takes seconds, so 700W (a single H100 SXM) for a few seconds for a batch of a few images to multiple users. Maybe more for the absolute biggest (but SFW, no porn) models.

    LLM generation takes more VRAM, but is MUCH more compute-light. Typically one has banks of 8 GPUs in multiple servers serving many, many users at once. Even my lowly RTX 3090 can serve 8+ users in parallel with TabbyAPI (and modestly sized model) before becoming more compute bound.

    So in a nutshell, imagegen (on an 80GB H100) is probably more like 1/4-1/8 of a video game at once (not 8 at once), and only for a few seconds.

    Text generation is similarly efficient, if not more. Responses take longer (many seconds, except on special hardware like Cerebras CS-2s), but it parallelized over dozens of users per GPU.

    This is excluding more specialized hardware like Google's TPUs, Huawei NPUs, Cerebras CS-2s and so on. These are clocked far more efficiently than Nvidia/AMD GPUs.

    ...The worst are probably video generation models. These are extremely compute intense and take a long time (at the moment), so you are burning like a few minutes of gaming time per output.

    ollama/sd-web-ui are terrible analogs for all this because they are single user, and relatively unoptimized.

  • permalink
  • fedilink
  • source
  • parent
    • load more comments (19 replies)
  • [–] 6 points 1 year ago (12 children)

    Right, but that's kind of like saying "I don't kill babies" while you use a product made from murdered baby souls. Yes you weren't the one who did it, but your continued use of it caused the babies too be killed.

    There is no ethical consumption under capitalism and all that, but I feel like here is a line were crossing. This fruit is hanging so low it's brushing the grass.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 13 child comments
    •
    • load more comments (11 replies)
  • [–] 1 point 1 year ago (2 children)

    How about, fuck AI, end story.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
    • load more comments (4 replies)