Eh, are you really using that many new models where sshing into your server / cding into your model folder, wgetting a url and configuring your interfaces every now and then is that much of an issue? I can't imagine using more than like one or two new models a month unless there's some insane string of releases or something.
Also, I mean, everyone's setup is different, but there's a significant amount of performance you're potentially leaving on the table by not using llama.cpp, potentially in the double-digit percentages. (Plus, if you have a fairly recent Nvidia setup and are willing to wait a bit for the latest models, ik_llama.cpp is a fantastic fork that I've found can get way better performance on most models than even llama.cpp.)