What's stuck for me after trying too many things:

• Ollama (or llama.cpp server if you want control over flags) as the model server, bound to localhost only
• Open WebUI as the front-end; it also handles OpenAI-compatible APIs, so local and hosted models live in the same history
• LiteLLM in between once you have more than one backend: one endpoint, per-key budgets, and logs
• SearXNG as the search tool for anything agent-ish; keep concurrency low or upstream engines start blocking it

Biggest lesson: RAM decides what you can run, and document upload/embeddings in Open WebUI eat more memory than chat does. On small boxes, point embeddings at an API or give it 4 GB.