Case study, status: r&d
Local LLM inference and fine-tuning lab
A private setup for running and fine-tuning open-weight models on a single consumer GPU.
Overview
Problem
Some clients cannot send data to third-party model APIs, and API costs grow with volume. Self-hosting needs real knowledge of the trade-offs.
Objective
Find out what quality and speed are realistic on a 16 GB consumer GPU, and build a repeatable fine-tuning workflow.
Solution
- Runs llama-server with GGUF models on an RTX 5080 16 GB; Q4_K_S quantisation was the practical choice for a 27B model.
- Compared vLLM on Windows through WSL2 with a community native Blackwell build.
- Fine-tuned with Unsloth using LoRA and QLoRA, including a Llama 3.2 1B experiment on generated stock OHLCV analysis data.
- Ported an MLX fine-tuning script from macOS to Windows and CUDA.
Architecture
Client app
OpenAI-compatible endpoint
llama-server / vLLM
Quantised GGUF model
RTX 5080 16 GB
AI workflow
- 1Choose model size and quantisation to fit available VRAM
- 2Serve through an OpenAI-compatible API
- 3Generate a domain dataset for fine-tuning
- 4Train LoRA / QLoRA adapters with Unsloth
- 5Compare outputs against the base model
Technology
- llama.cpp (llama-server)
- GGUF
- vLLM
- Unsloth
- MLX
- CUDA
Want something similar built?
Send a short brief. We'll reply personally with questions or a suggested next step — usually a technical discovery call.