134
The real energy use of agentic AI / Agents use about 600x more energy than simple AI prompts (www.theclimatebrink.com)
submitted 2 months ago by [M] to c/climate@slrpnk.net
13 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 19 points 2 months ago* (1 child)

God I hate these fake science calculation posts.

His prompts were each using 2.9 million token. That's a massive amount, he was basically purposefully using complex tasks that needed massive amounts of data parsing. We're talking about like 1 percent of users that are using 2.9 million per prompt.

The method used to calculate doesn't include batching. These companies aren't running one request per gpu here. Even the authors of the method admit it over estimates by 4 to 20x.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] 9 points 2 months ago

    2.9 million per prompt

    Hmmmm wait a moment

    The max context window of most of the big players is 1 million right now. Author almost certainly means per turn. But the whole context gets included with every prompt sent to the API. If you're using like 200k context, which is not unheard of with a large codebase and without compacting often, 2-3 million input tokens per turn to the agent is actually not THAT much. But most of it would be cached tokens, which are cheaper and presumably more energy efficient than fresh input tokens.

    E.g I just had a debugging session with 6 turns, shows me about 580k input tokens, but over 20 million cached input tokens, ~60k output tokens. But it also took like an hour with all the test runs and everything. 6 prompts to a chatbot takes like a minute or two in comparison.

  • permalink
  • fedilink
  • source
  • parent