Running LLMs on a single 16 GB GPU: what's realistic
By Indian Software Developers. Published , 5 min read.
Self-hosting a model is attractive when data cannot leave your infrastructure or when API costs grow with volume. On a single consumer GPU the main constraint is memory, and every decision follows from it.
Quantisation is the main lever
On an RTX 5080 with 16 GB of VRAM, a 27B-parameter model only becomes practical with aggressive quantisation. In our setup, a Q4_K_S GGUF build served through llama-server was the workable balance between fitting in memory and keeping output quality acceptable.
Choosing a server
llama.cpp's llama-server is simple and runs GGUF models well. vLLM offers higher throughput for many concurrent users but is more demanding to set up, especially on Windows, where the choice is between WSL2 and community builds for newer GPU architectures.
Fine-tuning on the same hardware
LoRA and QLoRA with Unsloth make fine-tuning small models feasible on a 16 GB card. A 1B-parameter model trains quickly enough to iterate on dataset design, which is where most of the quality comes from.