▲ 75 ▼ Why your local LLM feels dumber than it is (forum.level1techs.com) submitted 4 days ago* by inari@piefed.zip to c/technology@lemmy.world 103 comments fedilink hide all child comments
[–] majster@lemmy.zip 3 points 3 days ago (2 children) This really killed a wish to run local LLMs. So fiddly... permalink fedilink source hideshow 4 child comments replies: [–] frongt@lemmy.zip 5 points 3 days ago (2 children) Not any more than any other local service. Ollama makes it really easy to get started. The ministral models are small but still useful. permalink fedilink source parent hideshow 4 child comments replies: [–] brucethemoose@lemmy.world 10 points 3 days ago* (1 child) https://sleepingrobots.com/dreams/stop-using-ollama/ And thats just the tip of the iceberg. Also, while Minstral is fine, it is pretty much obsolete... permalink fedilink source parent hideshow 2 child comments replies: [–] frongt@lemmy.zip 3 points 3 days ago (1 child) Yeah, I'm just saying it's a really easy way to get started, without being overwhelmed by formats and quantizations and top-k and stuff. You run it, it goes, then you can move to something better. permalink fedilink source parent hideshow 2 child comments replies: [–] brucethemoose@lemmy.world 2 points 3 days ago* Its sleazy software, and IMO its defaults provide a really terrible 1st time experience. It convinces 1st time users that locally run LLMs are terrible, and goes out of its way to be incompatible with other stuff. I think LM Studio would be a much better "1st time" recommendation. Its not perfect either, but its lightyears better, not an active con like ollama, and its smoother to "graduate" from it to croco.cpp or llama.cpp or something. permalink fedilink source parent [–] nullroot@lemmy.world 1 point 3 days ago* Pretty funny. Most of this article was saying that ollama sucks now and they're just popular because there were one of the first. Personally I started with them, too, but constantly ran into issues with them. I've have much better experiences with jan, open web UI, and LM studio. The first two are foss, and the second properly attributes their use of open source projects, unlike ollama. Edit: nevermind, it's the link below me. It's still early in the morning, my bad permalink fedilink source parent [–] brucethemoose@lemmy.world 5 points 3 days ago* You shouldn't let that discourage you. VLLM is really finicky, but there are easier backends to use (though which one depends on your hardware). permalink fedilink source parent
[–] frongt@lemmy.zip 5 points 3 days ago (2 children) Not any more than any other local service. Ollama makes it really easy to get started. The ministral models are small but still useful. permalink fedilink source parent hideshow 4 child comments replies: [–] brucethemoose@lemmy.world 10 points 3 days ago* (1 child) https://sleepingrobots.com/dreams/stop-using-ollama/ And thats just the tip of the iceberg. Also, while Minstral is fine, it is pretty much obsolete... permalink fedilink source parent hideshow 2 child comments replies: [–] frongt@lemmy.zip 3 points 3 days ago (1 child) Yeah, I'm just saying it's a really easy way to get started, without being overwhelmed by formats and quantizations and top-k and stuff. You run it, it goes, then you can move to something better. permalink fedilink source parent hideshow 2 child comments replies: [–] brucethemoose@lemmy.world 2 points 3 days ago* Its sleazy software, and IMO its defaults provide a really terrible 1st time experience. It convinces 1st time users that locally run LLMs are terrible, and goes out of its way to be incompatible with other stuff. I think LM Studio would be a much better "1st time" recommendation. Its not perfect either, but its lightyears better, not an active con like ollama, and its smoother to "graduate" from it to croco.cpp or llama.cpp or something. permalink fedilink source parent [–] nullroot@lemmy.world 1 point 3 days ago* Pretty funny. Most of this article was saying that ollama sucks now and they're just popular because there were one of the first. Personally I started with them, too, but constantly ran into issues with them. I've have much better experiences with jan, open web UI, and LM studio. The first two are foss, and the second properly attributes their use of open source projects, unlike ollama. Edit: nevermind, it's the link below me. It's still early in the morning, my bad permalink fedilink source parent
[–] brucethemoose@lemmy.world 10 points 3 days ago* (1 child) https://sleepingrobots.com/dreams/stop-using-ollama/ And thats just the tip of the iceberg. Also, while Minstral is fine, it is pretty much obsolete... permalink fedilink source parent hideshow 2 child comments replies: [–] frongt@lemmy.zip 3 points 3 days ago (1 child) Yeah, I'm just saying it's a really easy way to get started, without being overwhelmed by formats and quantizations and top-k and stuff. You run it, it goes, then you can move to something better. permalink fedilink source parent hideshow 2 child comments replies: [–] brucethemoose@lemmy.world 2 points 3 days ago* Its sleazy software, and IMO its defaults provide a really terrible 1st time experience. It convinces 1st time users that locally run LLMs are terrible, and goes out of its way to be incompatible with other stuff. I think LM Studio would be a much better "1st time" recommendation. Its not perfect either, but its lightyears better, not an active con like ollama, and its smoother to "graduate" from it to croco.cpp or llama.cpp or something. permalink fedilink source parent
[–] frongt@lemmy.zip 3 points 3 days ago (1 child) Yeah, I'm just saying it's a really easy way to get started, without being overwhelmed by formats and quantizations and top-k and stuff. You run it, it goes, then you can move to something better. permalink fedilink source parent hideshow 2 child comments replies: [–] brucethemoose@lemmy.world 2 points 3 days ago* Its sleazy software, and IMO its defaults provide a really terrible 1st time experience. It convinces 1st time users that locally run LLMs are terrible, and goes out of its way to be incompatible with other stuff. I think LM Studio would be a much better "1st time" recommendation. Its not perfect either, but its lightyears better, not an active con like ollama, and its smoother to "graduate" from it to croco.cpp or llama.cpp or something. permalink fedilink source parent
[–] brucethemoose@lemmy.world 2 points 3 days ago* Its sleazy software, and IMO its defaults provide a really terrible 1st time experience. It convinces 1st time users that locally run LLMs are terrible, and goes out of its way to be incompatible with other stuff. I think LM Studio would be a much better "1st time" recommendation. Its not perfect either, but its lightyears better, not an active con like ollama, and its smoother to "graduate" from it to croco.cpp or llama.cpp or something. permalink fedilink source parent
[–] nullroot@lemmy.world 1 point 3 days ago* Pretty funny. Most of this article was saying that ollama sucks now and they're just popular because there were one of the first. Personally I started with them, too, but constantly ran into issues with them. I've have much better experiences with jan, open web UI, and LM studio. The first two are foss, and the second properly attributes their use of open source projects, unlike ollama. Edit: nevermind, it's the link below me. It's still early in the morning, my bad permalink fedilink source parent
[–] brucethemoose@lemmy.world 5 points 3 days ago* You shouldn't let that discourage you. VLLM is really finicky, but there are easier backends to use (though which one depends on your hardware). permalink fedilink source parent