Introducing Quartermaster, an open source local AI platform designed for ease of use that does not sacrifice customizability

It started as a fork of [llama-swap](github.com/.../llama-swap), but I have been building it out for myself since then as a convenient tool for all my local AI needs, and by now it has drifted far enough to be its own thing. The main idea is that you point it at your models folder and it configures things for you. It reads the GGUF headers, measures how much VRAM you actually have free, and works out context length, GPU offload, CPU/MoE split and KV cache size per model. All of it stays editable per model if you disagree with what it picked. It is not only text. llama.cpp for LLMs, with the Vulkan, CUDA, ROCm or CPU build downloaded and kept updated for you, stable-diffusion.cpp for images (SD, SDXL, Flux, Qwen-Image, LoRAs, upscaling), and vLLM if you already have it set up. You can register any other backend yourself by pointing at an executable, which is how I run TTS, and how you would run a llama.cpp fork like ik\_llama. Everything sits behind one OpenAI-compatible API on one port, with a single scheduler, so models swap in and out without fighting each other for VRAM. There is also a chat playground built in with web search, and a Hugging Face browser to search for a model, pick a quant and download it straight into the models folder and much more! If you are interested, you can read more about it [here.](quartermaster-labs.github.io/.../Quartermaster) MIT licensed.

What’s your local AI setup ?

I’m gradually moving my main PC from Windows to Fedora, so I’m also using that as an excuse to rethink my whole local AI stack. I was mostly using LM Studio before, but I wanted something better suited to a proper server-style setup: running in the background, handling model load/unload cleanly, giving me more control over parameters, and especially getting as much as possible out of my VRAM. I’ve started testing llama.cpp directly and honestly I really like it so far. With Bucoid’s Qwen3.8 27B IQ4\_XS on my 16GB of VRAM, I went from around 22 tok/s with LM Studio/Unsloth to roughly 27 tok/s with llama.cpp using equivalent inference settings in my case. I’m on a 5060 Ti 16GB. For anyone else with a 16GB card who wants to try Qwen3.8 27B, Bucoid’s model is a really nice option. Since it’s IQ4\_XS rather than a 3-bit quant, it keeps more quality while still fitting this kind of hardware. Depending on the settings, you can aim for roughly 40k–80k context while staying within the VRAM limit, although you’ll also need to quantize the KV cache. It works with LM Studio or Unsloth too, even if I’ve found llama.cpp more efficient in my setup. [huggingface.co/.../Qwen3.8-27B-IQ4-XS-MTP-1...](huggingface.co/.../Qwen3.8-27B-IQ4-XS-MTP-1...) What I’m still looking for is mainly the UI side. I don’t want to replace the CLI or hide llama.cpp behind another full platform. I’d rather have a good UI that complements it, so I don’t need to keep a terminal open for every simple action, while still letting llama.cpp handle the actual inference directly without adding another layer that could hurt performance. Ideally, I’d like to be able to : load/unload or switch models easily tweak a lot of parameters, ideally with presets see what’s currently loaded and resource usage manage a few chats/tests keep llama.cpp/API running quietly in the background And if you’re using something other than llama.cpp, O...

Is everyone here coding with TUIs?