Is the Future of AI Local? : technology@lemmygrad.ml

Is the Future of AI Local? (tombedor.dev)

submitted 5 months ago by yogthos@lemmygrad.ml to c/technology@lemmygrad.ml

24 comments fedilink hide all child comments

all 24 comments

sorted by: hot top controversial new old

[–] ksynwa@lemmygrad.ml 7 points 5 months ago (21 children)

Does anyone use local LLMs? I don't use LLMs myself, just the occassional shooting shit with z.ai, but in mainstream discussion local LLMs are almost never brought up except as a potential hedge against the AI bubble bursting by people who has used local models for less than five minutes in their entire lives.

The hardware requirements make local models unlikely. Everyone who talks about trying (not using) local models seems to have macbook pros. If the bubble bursts, then the future will probably be large open source models that can be vendored by anyone.

hideshow 21 child comments

[–] yogthos@lemmygrad.ml [S] 7 points 5 months ago (19 children)

I run local models on a macbook pro incidentally. A 32bln param model can do a lot of useful stuff I find. The progress on making the models smaller and faster has been very rapid, and I fully expect that we'll get to a point where you'd be able to run the equivalent of current frontier models on a local machine within a few years. On top of that, we see things like ASIC chips being developed that implement the model in hardware. These could become similar to GPU chips you just plug in your computer.

The tech industry has gone through many cycles of going from mainframe to personal computer over the years. As new tech appears, it requires a huge amount of computing power to run initially. But over time people figure out how to optimize it, hardware matures, and it becomes possible to run this stuff locally. I don't see why this tech should be any different.

parent

hideshow 19 child comments

[–] Assian_Candor@hexbear.net 7 points 5 months ago* (9 children)

I was surprised to learn I could run SAM2 locally on an upgraded t470 with 32GB of RAM with acceptable inference times. It made me question the wisdom of shelling out $20 a month for Claude. Especially as usage ramps up an old GPU would probably pay for itself pretty quickly and open up a universe of high quality models for orchestration

Also in the case of SAM2 the utility of custom tuning can't be overstated

parent

hideshow 9 child comments

[–] yogthos@lemmygrad.ml [S] 6 points 5 months ago (8 children)

And another aspect is that, at least in the realm of coding, we're trying to get these models to write code in a way humans do it. But I'd argue that it's not really an optimal approach because models have different strength. The biggest limitation they have is that they struggle with large contexts, but if given a small and focused task, even small models can handle it well. So, we could move to structuring programs out of small isolated components that can be reasoned about independently. There are already tools like workflow engines that do this sort of stuff, they just never caught on with human coders because they require more ceremony. But I think that viewing a program as a state graph would be a really nice way for humans to be able to tell whether the semantics are correct, and then the LLM could implement each node in the graph as a small isolated task that can be verified fairly easily.

parent

hideshow 8 child comments

[–] Assian_Candor@hexbear.net 3 points 5 months ago (7 children)

Love this approach. It's a unique way of thinking about this and I havent seen it elsewhere.

Ran across this yesterday not sure if you've seen it. A bit esoteric but the author argues for using go instead of Python for LLM coding. Its stuck with me to the point i am thinking of doing a full refactor on a project of mine even though I don't know go as well. But the bloat of the monorepo from having all these python dependencies is definitely food for thought. Your construct I think takes it a step further which is simplifying the code base for machines

https://lifelog.my/episode/why-i-vibe-in-go-not-rust-or-python

parent

hideshow 7 child comments

[–] yogthos@lemmygrad.ml [S] 6 points 5 months ago (6 children)

Right, languages can help us provide a lot of guard rails, and Go is a pretty good candidate being a fairly simple language with types keeping the code on track. I've played a bit with LLMs writing it, and results seem pretty decent overall. But then there's the whole architecture layer on top of that, and that seems to be an area that's largely unexplored right now.

I think the key is focusing on the contract. The human has to be able to tell that the code is doing what's intended, and the agent needs clear requirements and fixed context to work in. Breaking the program up into small isolated steps seems like a good approach for getting both these things. You can review the overall logic of the application by examining the graph visually, and then you can check the logic of each step independently without needing a lot of context for what's happening around it.

I've actually been playing a bit with the idea a bit. Here's an example of what this looks like in practice. The graph is just a data structure showing how different steps connect to each other:

and each node is a small bit of code with a spec around its input/output that the LLM has to follow:

It's been a fun experiment to play with so far.

parent

hideshow 6 child comments

[–] chesmotorcycle@lemmygrad.ml 2 points 5 months ago (5 children)

Have you been able to incorporate this approach to an existing project? I would imagine a lot of refactoring would be needed, but then again it might work for stitching in new components as well.

parent

hideshow 5 child comments

[–] yogthos@lemmygrad.ml [S] 4 points 5 months ago (4 children)

I ended up incorporating this pattern into a project at one of my jobs a while back, and fell in love with it ever since. I was working at a startup, and their requirements kept changing every few months. It became really painful to add and remove features after a while, and I was like I need to come up with a way to stay sane doing it. So, I started thinking about ways to compartmentalize things, and settled on this idea. It mades things a lot more manageable because I could come back to some code I've written months ago, and just look at the data flow and then figure out how it needs to be rearranged without having to relearn the details of the codebase.

It is a bit of effort to move existing logic over to this pattern though because you really have to make sure each component is context free. If you have an app with a bunch of shared state already, pulling that apart is a big effort that's hard to justify.

parent

hideshow 4 child comments

[–] chesmotorcycle@lemmygrad.ml 3 points 5 months ago (3 children)

Nice! Glad to hear this has been battle tested, haha. I'm part of a big team on my current project, and it's just been released (well, disabled by a feature flag for now). May not be the ideal candidate for trying this out, but here's hoping I can when the next thing comes up.

parent

hideshow 3 child comments

[–] yogthos@lemmygrad.ml [S] 4 points 5 months ago (2 children)

Yeah, give it a shot if you get the opportunity. Also worth mentioning that there are off the shelf engines you could use too. For example, there are js libraries like flowcraft that implement the workflow part for you.

parent

hideshow 2 child comments

[–] chesmotorcycle@lemmygrad.ml 4 points 5 months ago (1 child)

Ok, if there are existing libraries that handle the workflow bit, this will be an even easier sell. Cheers, tovarish!

parent

hideshow 1 child comment

[–] yogthos@lemmygrad.ml [S] 4 points 5 months ago

O7

parent

[–] CriticalResist8@lemmygrad.ml 5 points 5 months ago (8 children)

that ASIC chip prototype is pretty impressive. You can try it on https://chatjimmy.ai/ without an account, ask it to write something big like an essay or guide - literally the longest you'll wait is to get connected to the API, but the answer appears instantly.

Only limitation right now is they put a small llama 8b model on their chip, but it's a prototype and proof of concept of course. I'm sure soon China will print a full Deepseek model on such a chip lol.

Right now there isn't much interest in making AI more efficient to run but yeah there's no reason we won't find advances there. China is already doing a lot to squeeze models into smaller hardware.

I don't run LLMs locally because what I'm limited to is not great (especially context size is limited) but the way things are going we will definitely start to see open options open up, I think. If only because academia requires it.

parent

hideshow 8 child comments

[–] yogthos@lemmygrad.ml [S] 4 points 5 months ago (2 children)

That's what I'm thinking too. There's no reason why you couldn't make a chip like this for a full blown Deepseek model, and then when new models come out you just print new chips for them. The really nice part is that their approach doesn't need DRAM either because the state of each transistor acts as memory, it just needs a bit of SRAM which we don't have a shortage of.

I'm fully convinced that the whole AI as a service business model is going to be very short lived. Ultimately, nobody really likes their data going out to some company, and to have to pay subscription fees to use the models. If we start getting these kinds of specialized chips, they're going to be a game changer.

parent

hideshow 2 child comments

[–] CriticalResist8@lemmygrad.ml 3 points 5 months ago (1 child)

I could however totally see an economy where the chips themselves while cheap to produce cost a premium based on model and number of parameters.

Because the tech is certainly impressive and they have proof of concept. I don't know how scalable this is for them (or others), but it clearly works and shows immediate advantages. If it could integrate with existing consumer hardware, like say a PCI card you plug the chip into and switch them out when you want to change the model, anybody could easily have this at home.

But with capitalism we'd probably have to settle for DRM'd chips that self-destruct after X many tokens generated lol.

parent

hideshow 1 child comment

[–] yogthos@lemmygrad.ml [S] 2 points 5 months ago

that's disgustingly plausible scenario

parent

[–] chesmotorcycle@lemmygrad.ml 2 points 5 months ago (4 children)

Impressive speed! I only tried a couple small coding questions, but it's faster than anything I've seen.

parent

hideshow 4 child comments

[–] CriticalResist8@lemmygrad.ml 3 points 5 months ago (3 children)

Oh man you're underselling it to the rest of the website haha.

But it's tough to understate just how fast this is without seeing it. 15,749 tokens/s is what I get, and most responses from the big models might be a bit over 1000 tokens, maybe 2000 if they're stretching it (including chain of thought). Longest I got deepseek to go recently was just a bit below 5000 tokens.

But at such speeds, all of these generation lengths - 1000, 2000, 5000 - are basically done in the blink of an eye. 5000 tokens will be written in a third of a second, or just slightly above the average reaction time.

Unfortunately Jimmy seems limited to ~1000 tokens generation so we won't be able to really push it to the limits lol.

parent

hideshow 3 child comments

[–] chesmotorcycle@lemmygrad.ml 3 points 5 months ago (2 children)

Fills out modified DPRK form for Jimmy, lol

parent

hideshow 2 child comments

[–] CriticalResist8@lemmygrad.ml 3 points 5 months ago (1 child)

I hope they can scale it and not only that but that others are able to replicate it. This definitely has potential and would bypass the entire GPU/TPU problem and the layer architecture which is very inefficient.

Speed is not the end-all be-all, but it's not just the speed, it's also being able to run this fully locally. Imagine a PCI card for these chips, and you can just switch out the chip for another when you want to switch the model.

I'm just hopium-posting mind you lol, they clearly ran into bottlenecks if all they can offer is a 'tiny' llama 8b model. The micro required to etch an 800b model on that chip is magnitudes above, and at that point it might cost as much as a new CPU. But, it does leave the GPU available for other things, and lets everyone run SOTA models.

Really hope this goes somewhere, or if not that, something similar enough.

parent

hideshow 1 child comment

[–] chesmotorcycle@lemmygrad.ml 3 points 5 months ago

I think that last bit is exactly right. It doesn't have to be exactly this that catches on, but the model of massive data centers that run their chips into oblivion every 6-12 months is peak monopoly capital irrationality.

parent

[–] MasterBlaster@lemmygrad.ml 3 points 5 months ago

I use local models. To me the entire point of AI falls apart unless you can run it independently.

I just think generating videos and stuff is more exciting and sensational. But if you get any use out of LLMs its a no brained to set one up instead of paying a subscription.

parent

[–] Bronstein_Tardigrade@lemmygrad.ml 3 points 5 months ago (1 child)

You can already download and integrate a version of DeepSeek with openKylinOS for $0.

hideshow 1 child comment

[–] yogthos@lemmygrad.ml [S] 4 points 5 months ago

Sure, but it's still expensive for most people to get frontier model performance locally. I expect that in a few years the models will get optimized enough that even ones that run on modest hardware will be able to do everything a current frontier model does. And that's going to be the big game changer because there isn't going to be much demand for models as a service at that point.

parent