As a voice assistant server for my home assistant setup.
Edit. I leaned it by watching a Networkchuck video on YouTube. Basically a Piper integration for “voice to text” then a Whisper intergration for “text to voice”, and openWakeWord integration for wake word.
That sounds interesting! Can you describe what software you used for that? And how powerful does the hardware has to be?
Can you tell me more about this? I’ve considered trying to build and self-host something for home automation that would essentially be a FOSS and locally run Alexa/Google Assistant. Is this what you’re doing? How exactly does Ollama fit in?
Can you describe your setup?
I would love an offline voice assist while coding, but I’m paranoid about sharing my voice with AI providers given how easy voices are to clone these days.
I've been experimenting with it for different use cases:
I only have a GeForce 1080 Ti in it, so some projects are a bit slow and I don't have the biggest models, but what really matters is the self-satisfaction I get by not using somebody else's model, or that's what I try to tell myself while I'm waiting for responses.
I currently don't. But I am ollama-curious. I would like to feed it a bunch of technical manuals and then be able to ask it to recite specs or procedures (with optional links to it's source info for sanity checking). Is this where I need to be looking/learning?
you might want to look into RAG and 'long-term memory' concepts. I've been playing around with creating a self-hosted LLM that has long-term memory (using pre-trained models), which is essentially the same thing as you're describing. Also - GPU matters. I'm using an RTX 4070 and it's noticeably slower than something like in-browser chatgpt, but I know 4070 is kinda pricey so many home users might have earlier/slower gpu's.
How have you been making those models? I have a 4070 and doing it locally has been a dependency hellscape, I’ve been tempted to rent cloud GPU time just to save the hassle.
I'm downloading pre-trained models. I had a bunch of dependency issues getting text-generation-webui to work and honestly I probably installed some useless crap in the process, but I did get it to work. LM Studio is much simpler, but less customization(or I just don't know how to use it all in lm studio). But yea, I'm just downloading pre-trained models and running them in these UI's (right now I just loaded up 'deepseek-r1-distill-qwen-7b' in LM Studio). I also have the nvidia app installed and I make sure my gpu drivers are always up to date.
I employ this technique to embellish my email communications, thereby enhancing their perceived authenticity and relatability. Admittedly, I am not particularly adept at articulating my thoughts in comprehensive, well-structured sentences. I tend to favor a more primal, straightforward cognitive style—what one might colloquially refer to as a "meat-and-potatoes" or "caveman" approach to thinking. Ha.
I haven't been able to find a model that is both performant and useful on my machines (RTX 3060 12GB and M4 Mac mini), but I am open to suggestions! I know I want to use local LLMs more, but I feel that their utility is limited on consumer hardware
The Mac Mini should support a slew of models because of the unified memory right? I’m using the Gemma3 12b model while locally developing my work project now on a laptop with a 4090M. The laptop/4090M kind of sucks tbh, employer definitely wasted their money but it wasn’t up to me.
How much ram on the mini? Gemma3 27b is like 17GB, so that should all fit in the unified memory. The 12b version is only like 8GB so I’d think that would work on your 3060.
You could probably also find some much more slimmed down models that focus on a specific thing you care about on hugging face. You don’t need a model trained on all of Shakespeare’s works if you want your local I’ll to explain code you’re reviewing.
My Mac mini (32GB) can run 12B parameter models at around 13 tokens/sec, and my 3060 can achieve roughly double. However, both machines have a hard time keeping up with larger models. I'll have to look into some special-purpose models
Did you check out Gemma 3 variants? They were quite good in my opinion.
I’m in the process of trying them out again
Phi4 has been okay for me, and I use deepseek R1 32B quantized for some coding tasks. Both are a lot for my aging m1 MacBook Pro to handle.
Lately Ive been trying deepseek 8b for document summaries and it’s pretty fast but janky.
What I’m working towards is setting up an RSS app and feeding that into a local model (freshRSS I think lets you subscribe to a combined feed) to build a newspaper of my news subscriptions, but that’s not viable until I get a computer to run as a server.
I’m currently using it to generate initial contact emails, and generate contextual responses to received replies, for a phishing project at work.
in order to prevent phishing, right? (cue anakin/padme meme)
at work, right?
Mostly to help quickly pull together and summarize / organize information from web searches done via Open WebUI
Also to edit copy or brainstorm ideas for messaging and scripts etc
Sometimes to have discussions around complex topics to ensure I understand them.
Favorite model to run locally now is easily Qwen3-30B-A3B. It can do reasoning or more quick response stuff and runs very well on my 24 GB of VRAM RTX 3090. Plus, because it has a MoE architecture with only 3B parameters active when doing inference its lightning fast.
have you tried LM Studio as an interface? I havent tried open webui yet, just lm studio and text-generation-webui, so I am not sure if I'm limiting myself by using LM studio so much (I am very much a novice to the tech and do not work in computer science, so I'm trying to balance ease of use with customization)
It sounds like we’re on similar levels of technical proficiency. I have learned a lot by reading and going down wormholes on how LLMs work and how to troubleshoot and even optimize them to an extent but I’m not a computer engineer or programmer for sure.
I started with LM studio before Ollama/Open WebUI and it does have some good features and an overall decent UI. I switched because OWUI seems to have more extensibility with tools and functions etc and I wanted something I could run as a server and use on my phone and laptop elsewhere etc. OWUI has been great for that although setting up remote access for the server on the web did take a lot of trial and error. The OWUI team also develops and updates the software very quickly so that’s great.
I’m not familiar with text-generation-WebUI but at this point I’m not really wanting for much more out of a setup than my docker stack with Ollama and OWUI
Thanks for the excellent response! I'm going to give openwebui a try and do some of that trial and error as well - best way to learn!
Communication in general. I've found I don't communicate well, or rather what I type is taken in ways I would never have considered it to be taken - and AI really helps with that. If I'm sending a high-visibility message I'll usually run it through a local LLM first to make sure there aren't things I didn't intend.
For example, one I didn't put through I was saying something like "Yes I'm available to meet up, let's meet now because I have a commitment in 20 minutes". It was taken as "He has other commitments that are apparently a higher priority" and I was called out for saying it. Using AI usually catches those nuances I don't normally get.
Analyze jira tickets
Analyze meeting transcripts from every video call I've ever been on
Basically as a second brain strictly used for retrieving knowledge. I will never use it for reasoning.
I've set it up on my home computer for testing with code generation. My main tasks are machine learning projects, simple web apps, etc. I've connected it to the continue.dev extension in vs code and tried some of the smaller code models. I'm going to do some testing to find what completion models work the best, then fiddle with chat models. Agent mode was bad with the default small models, but then again I've never been impressed with agent mode even with GitHub copilot using 4o.
A place to discuss Ollama, from basic use, extensions and addons, integrations, and using it in custom code to create agents.