973
Managers (thelemmy.club)
submitted 4 months ago* by to c/whitepeopletwitter@sh.itjust.works
180 comments fedilink hide all child comments
 
Managers
you are viewing a single comment's thread
view the rest of the comments
[–] 8 points 4 months ago (7 children)

Pretty sure these AI companies are running at a cost, and due to AI Scaling Laws you hit the accuracy limit a lot sooner with a smaller model so it would probably be both worse and more expensive.

I could see how you might think speedrunning bankruptcy is similar to being "ahead of the curve" in this economy, though.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] 7 points 4 months ago

    There's a big difference between training a model, running a model, and running a model at scale.

    A small, self hosted setup will have lower accuracy and queries per second, and it will have a cost, but the cost will be no more than playing a videogame. You'll still have something surprisingly accurate and responsive for some tasks, like being a wiki interface or something.

    Remember that some of these models can run on a standard smartphone, and all the hoopla when people found that chrome was downloading models onto people's devices.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 7 points 4 months ago* (last edited 4 months ago) (5 children)

    No that's not how this works. Inference is cheap and efficient. AI companies are bankrupting themselves with training costs that they need to recoup back by selling inference. Open-weight models have already been trained.

    Also, going big in terms of model size shows diminishing marginal returns on accuracy, not efficiency of scale. Smaller models are way more efficient and consistently catch up to the largest models, which is why today's SOTA 27 billion parameter model competes with yesterday's SOTA 500+ billion parameter model.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 4 months ago (1 child)

    AI companies are bankrupting themselves with training costs that they need to recoup back by selling inference.

    I think they hit a wall in actual returns on performance with pretraining, years ago. Then they started scaling up on post-training/reinforcement learning to continue improvement, but that might be hitting a plateau as well. More recently it looks like they're relying more heavily on scaling up on inference, which is a significant problem for their long term business models.

    If they're not able to cheaply deliver inference (and charge at a premium), how will they be able to sustain their businesses?

    It seems that the most recent, largest models are using a lot more tokens to accomplish the same tasks, so even as token cost drops the actual cost of using the latest models seems to be going up with time (even as performance improves).

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 4 months ago*

    If they’re not able to cheaply deliver inference (and charge at a premium), how will they be able to sustain their businesses?

    I definitely agree that they have a big problem on their hands, and are in deep deep trouble. They are in a position where they must sell a service that is very cheap in order to pay for up front costs that were very expensive.

    This is also why the release of Deepseek was such a devastating blow to US AI companies. It proved that:

    1. they don't really have a moat that would lock users into their service, or secret special knowledge that prevents other companies from training competitive models. They're in a race to the bottom

    2. Deepseek was not only able to train a model of the same caliber, but they were able to do it at a tiny fraction of the cost that US AI companies spent on training US models. Because they spent so much less on training, it means that Deepseek is able to undercut the US companies and offer inference at a much lower price

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 4 months ago (2 children)

    Inference is cheap and efficient.

    Tell that to all the Github users that are screaming about the new token based billing. In reality inference on these massive models with big context windows is expensive, but was subsidized so hard, that nobody has an accurate feeling for the cost.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 4 months ago (1 child)

    No, it is cheap and efficient. It is relative, and the comparison is to model training. But yeah, its not free

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 4 months ago* (last edited 4 months ago)

    Sure it's much much cheaper than training, but importantly those companies are not recouping anything with inference because it is still more expensive than what they are selling it for.

    They are double bankrupting themselves.

    At work we run inference for a research project with an open weights model in the public cloud another part of my company provides and we pay around 25$ a day for a VM with a single L40s. It's both slow - despite not even serving concurrent users - and kind of bad in its outputs.

    Edit: Interference -> Inference, arguing on the internet after waking up first thing in the morning might not have been the best idea

  • permalink
  • fedilink
  • source
  • parent