We already have this! It’s called a tool harness, and basically any kind of local agent framework supports it already, as well as a whole slew of locally runnable LLMs in every size.

There isn't any lightweight local LLM which give valid results, or you dedicate >90GB for it, or you permits that it connect to the online source servers, wasting a lot of bandwith. You can use locally only AIs dedicated to certain tasks, eg for summarizing documents or in some graphic apps to automate some functions or for accounting in the office, but forget an local alround Assistant.

If you want an AI for certain tasks, use search assistants, like DuckAI or Andisearch, which are good for summarize results fom Webpages without the need of big and large lengage models, being privacy friendly, or use a specific one which you can anonym without account, eg one of those from HuggingFace or one specific from Futurepedia, there you can filter what you need.