Stable Diffusion on Consumer Hardware: The Home Image Lab

For about eighteen months I resisted running image models at all. Text generation had an obvious homelab use — summarising notes, drafting config, answering questions about my own documents — and image generation looked like a toy that would eat a GPU and produce six-fingered nightmares. Then I needed forty header illustrations for an internal wiki, priced up the hosted options, watched the per-image meter tick, and did what any sensible tinkerer does: pulled a model onto the machine already sitting under my desk and ran the whole thing offline.

That is the pitch for a home image lab. The same weights that back the paid services will run on a mid-range gaming card, the generation happens on hardware you own, and once the model is downloaded the network cable is decorative. No prompts leave the house, no images land in someone else’s training set, and the marginal cost of the ten-thousandth picture is a few pence of electricity. The catch is that image generation is fussier about hardware than a language model of comparable fame, and the failure modes are stranger. This is the guide I wish I’d had before I started.

Why run it locally at all

The privacy argument is the one people reach for first, and it holds. Every prompt you type into a hosted generator is a sentence about what you are thinking, and every image is stored, moderated, and — read the terms — frequently reusable by the provider. If you are illustrating anything remotely sensitive, or simply dislike feeding a stranger’s model for free, local generation removes the question entirely. I can verify the isolation the same way I verify it for any offline service: unplug the machine and watch it keep working.

The stronger argument, once you’re past privacy, is control. A hosted generator hides the sampler, the scheduler, the seed and the model version behind a friendly box, so when the output is wrong you have no levers. Running it yourself exposes every knob. You choose the checkpoint, pin the seed to reproduce a result exactly, swap the sampler when faces come out mushy, and load a LoRA to nudge the style without retraining anything. Image quality lives entirely in those knobs, and you can only turn them on your own machine.

Cost is the third reason and the least romantic. Hosted image APIs charge per generation, and image work is iterative — you rarely nail a composition first try, and a single decent result can be twenty rejected ones behind it. Locally, iteration is free, so you experiment like someone who isn’t being charged for curiosity, which is the only way to actually get good at prompting.

How much hardware you actually need

The honest headline: VRAM is the whole game, and everything else is secondary. The model weights and the working tensors have to fit in your graphics card’s memory, and when they don’t the software either crashes or crawls.

For the older Stable Diffusion 1.5 family, 4GB of VRAM is workable with memory-saving flags and 6–8GB is comfortable. These models produce 512×512 images natively and upscale from there. They are ancient in AI terms and still excellent for a lot of work, especially with community fine-tunes.

For SDXL, the 2023 generation that most people now treat as the baseline, budget 8GB as a floor and 12GB for an easy life. SDXL renders at 1024×1024 natively, so it wants roughly double the memory of 1.5 and rewards you with sharper, more coherent images and far better text-in-image behaviour.

For Flux.1, the Black Forest Labs models that landed in August 2024 and reset expectations for open-weight quality, the full versions are hungry — 24GB is the comfortable target, though quantised GGUF builds and the smaller schnell variant will squeeze onto 12GB with patience. This is where a consumer card starts to feel its limits, and where the quantisation tricks below earn their keep.

A used 12GB card is the sweet spot for a home lab: it runs SDXL without drama, dabbles in quantised Flux, and cost me less than three months of a hosted subscription. System RAM matters far less — 16GB is fine — and the CPU barely participates once generation starts. If you have no discrete GPU at all, generation runs on CPU, but a picture that takes four seconds on a card takes several minutes on a processor, so it’s a proof of concept rather than a workflow.

Choosing a runner

Three front ends dominate the homelab space, and the right one depends on how much you enjoy plumbing.

AUTOMATIC1111’s WebUI (usually just “A1111”) is the friendly default. It presents a single web page with prompt boxes, sliders, an extensions ecosystem, and sensible defaults, so you can generate your first image in the time it takes to read this paragraph. Most tutorials assume it.

ComfyUI exposes the generation as a node graph you wire by hand — load checkpoint, encode prompt, sample, decode, save — which looks intimidating and then becomes the reason you never go back. Because every step is a visible node, you can see exactly where VRAM goes, branch a workflow, and reproduce it precisely by saving the graph as JSON. It’s also markedly lighter on memory, which is why the Flux crowd lives there. I run it for anything repeatable.

Fooocus hides everything behind a near-magic single box tuned to make SDXL look good with no fiddling. It’s the one I point non-technical housemates at.

Here is ComfyUI as a compose service, which is how I keep it reproducible and out of my base system’s Python environment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

services:
  comfyui:
    image: yanwk/comfyui-boot:latest
    ports:
      - "8188:8188"
    volumes:
      - ./models:/root/ComfyUI/models
      - ./output:/root/ComfyUI/output
      - ./input:/root/ComfyUI/input
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

The volume layout matters more than it looks. Checkpoints are large — an SDXL model is 6–7GB and a full Flux model is over 20GB — so mounting models as a host directory means you download each one once and share it across every runner rather than baking a 30GB image. Bring it up, then drop a checkpoint into ./models/checkpoints/ and refresh the browser:

1
2
3
4
5

docker compose up -d
# fetch a checkpoint into the shared models dir
curl -L -o ./models/checkpoints/sd_xl_base_1.0.safetensors \
  https://example.com/models/sd_xl_base_1.0.safetensors
docker compose logs -f comfyui   # watch it detect the GPU on startup

After that first download the machine never needs the internet again for generation. Pull the cable and it keeps drawing.

The settings that actually change the output

Beginners fixate on the prompt and ignore the sampler, which is backwards. A handful of parameters do most of the visible work.

Sampler and steps. The sampler is the algorithm that walks noise towards an image. Euler a is a fast, forgiving default; DPM++ 2M Karras is my go-to for SDXL when I want clean detail. Steps control how many refinement passes it takes — around 20–30 is the sensible band. Below that the image is under-cooked; above it you’re mostly burning time for diminishing gains.

CFG scale governs how literally the model obeys your prompt. Low values (3–5) let it improvise and often look more natural; high values (10+) force adherence and tip into a fried, over-saturated look. For most SDXL work I sit around 6–7. Flux prefers even lower guidance and comes with its own recommended range, so read the model card.

Seed is the random starting point. Fix it and you can change one word of a prompt and see only that word’s effect, which is how you learn what a prompt token actually does. Leave it random and every generation is a fresh roll of the dice.

Resolution must match what the model was trained on or composition falls apart — SD 1.5 wants 512-ish, SDXL wants 1024-ish. Ask for a 512×512 SDXL image and you’ll get duplicated limbs and cloned faces, because you’ve dragged it far outside its training distribution. Generate at native resolution, then upscale as a second step if you want bigger.

Squeezing a big model onto a small card

When the model won’t fit, you have real options before you give up and buy a bigger card.

The first lever is precision. Running weights in half precision (fp16) rather than full precision halves the memory, and it’s the default in every modern runner for good reason — the visible quality loss is negligible. The second lever is quantisation: GGUF and NF4 builds of Flux compress the weights further, trading a sliver of fidelity for fitting a 24GB model into 12GB. The third is offloading, where the runner keeps only the active part of the model on the GPU and parks the rest in system RAM, swapping as needed. A1111’s --medvram and --lowvram flags do exactly this; they’re slower per image and they’re the difference between generating and staring at an error.

VAE handling is a common quiet memory hog. The VAE decodes the final latent into pixels and can spike VRAM right at the end of a generation, producing an out-of-memory crash on the last step of an otherwise-successful run. Enabling tiled VAE decodes the image in patches and flattens that spike, which is often the single change that lets a stubborn resolution complete.

Troubleshooting: the errors you will actually meet

CUDA out of memory. The signature failure. Lower your resolution first, then your batch size to 1, then add --medvram. If it dies specifically on the final decode step, it’s the VAE — enable tiled VAE. On NVIDIA, a generation that crept up to the limit can leave memory fragmented, so a full restart of the runner clears it when nothing else will.

Every image comes out a solid black square. This is a NaN — the maths overflowed and the tensor filled with garbage. It bites older GPUs and half-precision VAEs. The fix is almost always to force the VAE to run in full precision (--no-half-vae in A1111) or to load a known-good fp16 VAE for the model. It looks catastrophic and it’s a one-flag fix.

Bodies with extra limbs or melted faces. Usually the resolution is wrong for the model, or the CFG is cranked too high, or the checkpoint is a mismatch for the style you want. Generate at native resolution, drop CFG towards 6, and if faces are the specific problem, run an inpainting or face-restoration pass afterwards rather than expecting the base model to nail them at a distance.

Wildly slow generation on a card that should be quick. Confirm the model actually loaded onto the GPU — nvidia-smi should show VRAM occupied and the fan working while it renders. If VRAM is idle, the runner fell back to CPU, usually because the CUDA build and the driver disagree. On NVIDIA, missing or broken xformers also leaves speed on the table; installing it can roughly halve step time.

Results you can’t reproduce. You changed something you forgot about — most often the seed was random, but it can be a silent model or VAE swap. This is exactly why I moved my repeatable work to ComfyUI: the saved graph pins the checkpoint, sampler, seed and every value, so a workflow that worked in March still works in December.

Where this fits with the rest of a local-AI setup

An image lab slots neatly alongside the other offline-AI services worth running at home. If you’ve already stood up a language model for chat or document work, you have most of the mental model and half the tooling. A private front end like Open WebUI can even drive image generation through the same interface you use for text, so the whole local stack lives behind one login. And if your interest in local AI started with keeping documents private rather than making pictures, a local RAG stack for chatting with your own documents is the natural companion project — same hardware, same offline principle, different output.

Is it worth it, and who for

Build a home image lab if you generate images often enough that per-image pricing annoys you, if you care where your prompts and outputs end up, or if you simply want to understand how these models behave instead of treating them as a slot machine. The learning alone justified it for me: after a fortnight of free iteration I prompt far more deliberately than any metered service ever taught me to.

Skip it if you make a handful of images a year — a hosted generator will be quicker to reach and cheaper in total than a suitable GPU. And be honest about the hardware floor: without a discrete card with a decent slab of VRAM, you can run all of this, but you’ll spend the difference in patience.

For me it earned its rack space within the first project. The weights sit on a disk I own, the card was going spare in a gaming machine, and a wiki that needed forty illustrations got them over a quiet evening, offline, for the cost of the electricity — with the network cable on the floor where I could see it.