My RTX 5070 has 12GB of VRAM, which is enough for smaller local models but not enough to hold every model I wanted to try. The Windows PC with the 5070 attached also has 64GB of system RAM, so a practical solution is to let llama.cpp put as much as possible on the GPU and offload the remainder to RAM. The result is a private local-model server with seven selectable models: ...
Goal: wiring Hugging Face models into my local llm (Ollama) app, testing prompt-following, and seeing how far I can push a 12GB RTX 5070 before the system starts leaning hard on the CPU. The short version: it works, but only because the stack is allowed to spill parts of the model back to system memory. The current setup The image UI lives on my web server, but the GPU does not. The request path looks like this: ...
To be precise, this does not make the RTX appear as a local GPU on my workstation…the programs run on the gaming PC, and I control them remotely over SSH. The setup WSL2 gives the GPU to Linux via NVIDIA’s driver passthrough. nvidia-smi works inside Ubuntu with zero driver installs on the Linux side. The trick is making it reachable and keeping it alive: ...