Wsl2

My RTX 5070 has 12GB of VRAM, which is enough for smaller local models but not enough to hold every model I wanted to try. The Windows PC with the 5070 attached also has 64GB of system RAM, so a practical solution is to let llama.cpp put as much as possible on the GPU and offload the remainder to RAM. The result is a private local-model server with seven selectable models: ...

To be precise, this does not make the RTX appear as a local GPU on my workstation…the programs run on the gaming PC, and I control them remotely over SSH. The setup WSL2 gives the GPU to Linux via NVIDIA’s driver passthrough. nvidia-smi works inside Ubuntu with zero driver installs on the Linux side. The trick is making it reachable and keeping it alive: ...