Definitely. We do this at my job. But a server big enough to provide a comparable service would cost at least $30k. And you likely need more than one. And you still need someone to maintain it.

Not at the moment. It's definitely cheaper to buy LLMs as a service than running an in-house server hall. Anthropic and OpenAI are running at a loss, you know.

Now, risking to lose your data to the service provider is another thing.

Look at it this way, if I pay Open AI $1k a month, their people make sure it works they month. If I let Joe from finance spend $10k on computer parts, Joe needs to ensure that it runs or I need to hire someone who is knowledgeable on local models. If I'm a simple firm, $12k a year might be cheaper because someone else makes sure it works and I can fire Joe when the fancy autocorrect loses a bunch of clients by falsifying work. Then I don't have to spend even bigger bucks hiring a new Joe because no one knows how to work the damn server he replaced the break room with.

It depends, sometimes the private API models are better at a specific task to the point it wouldn't be worth using one that is worse. (Really it's the harnesses around them tbh)

Overall sovereign AI is where it's at. If you have good enough tooling in place

We have a few dgx sparks in the office that are gathering dust right now because nobody wants to bother with them (and because IT doesn't allow them on the network because Linux is scary so they're a pain to get set up).

Matching the level of productivity of frontier models on the local models is possible to a limited degree (depending on the exact task), but takes getting used to. For the local models you have to lean into the fact that tokens are 'free' so its fine to burn a lot of them. (Whereas with frontier models the recommendation is to take a higher model for complex tasks because a lower would take more tries to get it right)