Local LLM inference and fine-tuning lab
A private setup for running and fine-tuning open-weight models on a single consumer GPU.
Problem
01 — OverviewSome clients cannot send data to third-party model APIs, and API costs grow with volume. Self-hosting needs real knowledge of the trade-offs.
Find out what quality and speed are realistic on a 16 GB consumer GPU, and build a repeatable fine-tuning workflow.
Solution
02 — Build- 01
Runs llama-server with GGUF models on an RTX 5080 16 GB; Q4_K_S quantisation was the practical choice for a 27B model.
- 02
Compared vLLM on Windows through WSL2 with a community native Blackwell build.
- 03
Fine-tuned with Unsloth using LoRA and QLoRA, including a Llama 3.2 1B experiment on generated stock OHLCV analysis data.
- 04
Ported an MLX fine-tuning script from macOS to Windows and CUDA.
Architecture
03 — SystemAI workflow
04 — SequenceChoose model size and quantisation to fit available VRAM
Serve through an OpenAI-compatible API
Generate a domain dataset for fine-tuning
Train LoRA / QLoRA adapters with Unsloth
Compare outputs against the base model
Technology & role
05 — DeliveryEverything — hardware setup, inference configuration, dataset generation and training.
Want something similar built?
Send a short brief. We'll reply personally with questions or a suggested next step — usually a technical discovery call.