607
Companies Are Throttling Employees’ AI Use Because It’s Too Expensive (www.404media.co)
submitted 2 months ago by to c/technology@lemmy.world
143 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 11 points 2 months ago (25 children)

I don't really understand how people are using so many tokens. At work I haven't even hit $200 I spend per month. Wtf are people doing with these things that burns so many tokens?

  • permalink
  • fedilink
  • source
  • hideshow 25 child comments
    •
  • [–] 18 points 2 months ago*

    If you run "agentic coding harness" or any kind of goal oriented loop then tokens goe brrrr.

    And LLM sellers are pushing for that (duh), as they managed to convince people to use infinite monkeys typewriting until they make Hamled.

    (Type made on purpose)

  • permalink
  • fedilink
  • source
  • parent
  • [–] 5 points 2 months ago (2 children)

    I tried Warp terminal because now that's bankrolled by openai's magic infinite money you can use your own openai api keys without a subscription. So I put one from my work account. I do a git commit (manually) and then it comes a prompt under it "push it, open a PR and switch to main?". I click yes, it used one million tokens for that... (And it took about a minute because it did like 20 requests, so there was no time saving at all vs doing it manually)

    I wget something, it comes a prompt under it "now compare the hash?". Boom, another 500k tokens

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 2 months ago (1 child)

    That's completely insane. At best it would be useful for the pr title and message, but the rest of that is waste.

    These are the kinds of things I just ask in chat. "Whata the cli command to compare hashes again?"

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 5 points 2 months ago (1 child)

    If you click the most expensive model and then click max/fast mode, the same task can easily cost 10 or 20x of the cheaper models

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 2 months ago

    I watched two colleagues this week and both had Opus 4.8 1M max thinking. No matter which task. It’s also slow as fuck. I work almost all day with GPT-5.4 low thinking and get good results… but faster and cheaper.

    I guess good model selection and promoting will be what sets devs apart in the near future. Once that bubble bursts a bit more and prices increase further that will be an interesting reckoning. Also for companies who basically taunted their employees into tokenmaxxing.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 4 points 2 months ago

    I am not able to use the tokens provided by a Claude Max account either.
    But if someone tries to be clever and have 10 employees use a single Max account, they probably run into the limits often. And if the response is to let them just buy API-prized tokens instead of getting more accounts, that gets very expensive very fast. The single-user accounts are subsidized. The extra token prices are not.
    Actual business accounts are prohibitively expensive. And at least Anthropic terminates subsidized accounts when they see extensive use.

    Real token prices are insane. Most businesses couldn't afford them. And eventually the VC capital will dry up. The cheap AI bubble will burst. And then the market is in for a real sticker shock.
    Better be prepared to switch to local inference for as many use cases as possible.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 4 points 2 months ago (4 children)

    I read they were automating everything whether it needed AI or not just to get credit for using AI.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 6 points 2 months ago (3 children)

    I had to push back on that at work. Most of the problems presented were easily solvable via conventional methods. Only one task was a legitimate use of AI. There are some others, but the pressure to consider AI for every task is a little bananas

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 2 months ago* (2 children)

    My boss was talking about using AI agents for CI/CD processes. Like, I get using them to build CI/CD processes, but involving AI agents in the actual build process is ridiculously stupid. A representative from Microsoft specifically said in a training session to not use them that way so it's obviously not only my stupid ass boss.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 2 months ago (3 children)

    I've heard "loops" will burn a lot of tokens. Haven't tried it myself. A person could also spool up multiple loops to work on multiple branches at the same time.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 3 points 2 months ago* (1 child)

    I am not convinced yet of letting agents completely unattended. Watching them work makes review easier for me. If I let the agent just produce some result it needed half an hour (or more) for, it’s very likely so convoluted that I can at best skim over it and then go „yeah yeah ok, it’s probably fine “.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 4 points 2 months ago*

    If I let the agent just produce some result it needed half an hour (or more) for, it’s very likely so convoluted that I can at best skim over it and then go „yeah yeah ok, it’s probably fine “.

    I am seeing the first job ads for senior software developers which can debug the resulting mess. A lot of it will be just unmaintenable. They will get 20 years of technical debt with ten times the speed and ten times the volume.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 3 points 2 months ago

    Agent loops for SWE burn a LOT of tokens.

    I'm unfortunately temporarily disabled and can't use my hands for another 2 months. So I've leaned heavily into AI based workflows to keep my job in the meantime.

    Aside from the nightmare of keeping quality high, not atrophying skills, and avoiding a lack of domain knowledge. It works reasonably okay.

    Token usage is insane though. A productive day might cost a few hundred dollars in tokens all things considered. Quality is expensive as well, a good 1/4 of that are automated systems that exist to identify defects, quality, coherence....etc issues early.

  • permalink
  • fedilink
  • source
  • parent
  • [–] [S] 2 points 2 months ago (1 child)

    What are you using? Which product? I got 2k last month and was told to use cheaper models indeed

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 0 points 2 months ago

    I generally use sonnet 4.6, switching to opus 4.6 for more complex stuff. I try to stick with medium thinking, but will use max for stuff I am not super specific about in my prompt (or obscure errors).

    I use them through a GitHub copilot enterprise license, via the plugin for jetbrains.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 0 points 2 months ago (1 child)

    Our company pays by token usage.

    Claude opus 4.5 costs about 25$ per one million input tokens.

    Well I manage to get to about 50 million input tokens per day regularly on the agent. Not everyday, but at least 7 per month. So I am alone cost the company about 2000$ extra on top of my salary.

    Well they fixed it by implementing some great caching for the tokens and using sonnet instead of opus I can save some money too. Also gemini flash is much cheaper and similar performant. So you can fix it so you don't burn money on ai

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •