Proxmox GPU Passthrough for a Local LLM Box

I bought a second-hand graphics card to run language models at home, put it in the machine that already had spare PCIe slots, and then discovered that machine was my hypervisor. Which left an awkward question: does the GPU belong to the host, or to a guest?

I went with a guest. The reasoning took me about a week to arrive at and thirty seconds to explain: the hypervisor should stay boring. If I install the CUDA stack on the Proxmox host, then every kernel update is a potential hypervisor-down event, and my hypervisor also runs the things I care about — DNS, the reverse proxy, backups. Putting the driver stack inside a VM means an inference experiment gone wrong costs me one VM reboot instead of the whole house.

That’s the “why”. The “how” is where you’ll spend your evening, because PCI passthrough is one of those features that either works on the first try or fails in six distinct ways with error messages that describe symptoms rather than causes.

What passthrough actually is, and the cheaper alternative

When you pass a PCI device through, the host stops driving it. The kernel binds it to vfio-pci, a stub driver whose entire job is to hold the device open and hand its memory-mapped regions to QEMU. The guest then sees real hardware on its virtual PCIe bus and loads a real driver. No translation layer, no shared context, near-native performance.

The cost is exclusivity. That card is now gone from the host and gone from every other VM. If you want two VMs to share it, passthrough is the wrong tool — you want vGPU (enterprise licensing, Nvidia’s rules) or you want to give up on VMs entirely.

Which brings me to the alternative I should mention before you spend the evening: an LXC container with the device bind-mounted. Proxmox containers share the host kernel, so you install the Nvidia driver on the host, mount /dev/nvidia* into the container, and you’re done in twenty minutes. Multiple containers can share the card. It’s genuinely easier.

I rejected it because it puts the driver on the host, which is the thing I was trying to avoid, and because unprivileged LXC with GPU devices involves cgroup rules and idmap fiddling that I find harder to reason about than a VM boundary. If your priorities differ, take the container route with my blessing — you’ll be running Ollama before dinner.

Hardware prerequisites, honestly stated

Three things have to be true, and none of them are negotiable:

1. The CPU and chipset support IOMMU. Intel calls it VT-d, AMD calls it AMD-Vi. Consumer boards from the last decade generally have it; some cheap ones have it in silicon and omit it from the BIOS menu, which is its own kind of insult.
2. It’s enabled in the BIOS. Look for “VT-d”, “IOMMU”, “SVM Mode”, or on some boards it hides under “North Bridge configuration”. Also enable “Above 4G Decoding” and, if you see it, “Resizable BAR” — modern cards want a large BAR window and passthrough with a 256 MB aperture is a known source of misery.
3. The GPU sits in its own IOMMU group, or you can force it there. This is the one that kills most attempts.

Check the third one first, before you touch anything else. This little loop has saved me from more than one wasted evening:

1
2
3
4
5
6
7
8
9

#!/bin/bash
# iommu-groups.sh — list every IOMMU group and what's in it
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in "$g"/devices/*; do
        echo -e "\t$(lspci -nns "${d##*/}")"
    done
done

Run it. If your output is empty, IOMMU is off — go back to the BIOS. If your GPU appears in a group alongside nothing else except its own audio function, you’ve won:

1
2
3
4

IOMMU Group 14:
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3060 Ti] [10de:2489] (rev a1)
IOMMU Group 15:
	01:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)

If instead your GPU shares a group with the SATA controller, a NIC and a USB hub, your board’s PCIe root port doesn’t implement ACS, and everything downstream of it is one indivisible blob. You can pass the whole blob through (losing your disks — no), or you can apply the ACS override patch, which lies to the kernel about isolation. The override works. It also means a malicious or buggy guest could in principle DMA into a neighbour’s memory. In a home lab running my own workloads I consider that acceptable; in anything multi-tenant I would not.

Note the [10de:2489] and [10de:228b] vendor:device IDs above. Write yours down. You need them next.

Host preparation

Enable IOMMU on the kernel command line. Proxmox 8 installs with systemd-boot on UEFI/ZFS setups and GRUB elsewhere, so check which you have before editing the wrong file.

For GRUB, edit /etc/default/grub:

1

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

Then update-grub. For systemd-boot, append the same options to /etc/kernel/cmdline and run proxmox-boot-tool refresh. On AMD, amd_iommu=on is the equivalent, though it’s been the default for years — iommu=pt (passthrough mode) is the part that matters, since it skips DMA translation for host devices and cuts the overhead on everything else in the box.

Now load the vfio modules and claim the card by ID:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

cat >> /etc/modules <<'MOD'
vfio
vfio_iommu_type1
vfio_pci
MOD

# claim both functions of the card before any real driver can
echo "options vfio-pci ids=10de:2489,10de:228b disable_vga=1" > /etc/modprobe.d/vfio.conf

# keep the host's drivers off it
cat > /etc/modprobe.d/blacklist-nvidia.conf <<'BL'
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist snd_hda_intel
BL

update-initramfs -u -k all
reboot

The ids= list must contain both functions — video and audio. Miss the audio function and the card ends up split across two drivers, which produces a VM that boots, shows a display, and then hard-locks the host when the guest resets the device.

One caveat on claiming the card by ID: that vendor and device pair matches every card of that model in the box, whatever slot it sits in. If you later add a second identical card intending to keep one for the host, vfio-pci claims both of them at boot and you will lose an evening working out why the host has no display. Two cards of the same model means binding by PCI address instead, via a driver_override applied from a systemd unit that runs before the vfio modules load. With a single card the ID method is simpler and does the right thing.

After the reboot, confirm the host has genuinely let go:

1
2
3
4
5
6
7
8

# lspci -nnks 01:00
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3060 Ti] [10de:2489] (rev a1)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:4067]
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GA104 [10de:228b] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

Kernel driver in use: vfio-pci on both lines is the whole game. Anything else and you stop here and fix it, because attaching the device to a VM while the host still holds it will, at best, fail loudly.

The VM

Machine type q35 and BIOS OVMF (UEFI). SeaBIOS passthrough is possible and I have never once been glad I tried it. Give the VM an EFI disk, set the CPU type to host, and add the card as a PCI device with “All Functions” ticked and PCI-Express enabled.

The resulting VM config on disk looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# /etc/pve/qemu-server/210.conf
agent: 1
bios: ovmf
boot: order=scsi0
cores: 8
cpu: host
efidisk0: local-zfs:vm-210-disk-0,efitype=4m,size=1M
hostpci0: 0000:01:00,pcie=1
machine: q35
memory: 24576
name: llm-box
net0: virtio=BC:24:11:00:00:01,bridge=vmbr0
numa: 0
ostype: l26
scsi0: local-zfs:vm-210-disk-1,discard=on,size=120G,ssd=1
scsihw: virtio-scsi-single
sockets: 1

Two details worth noticing. hostpci0: 0000:01:00 with no function suffix passes the entire device — both functions — which is what you want. And I left x-vga=1 off deliberately: this VM never renders a desktop, so the card runs headless as a compute device and I reach the guest over SSH. Every “Code 43”-adjacent horror story I’ve read starts with someone trying to make the passed-through card the VM’s primary display.

Memory: give the VM enough that the model’s non-GPU overhead fits comfortably. 24 GB of RAM for a 12 GB card sounds generous until you watch Ollama read a 20 GB GGUF off disk into page cache before uploading layers to VRAM.

There is a second memory consequence that catches people out. A device doing DMA can write to any guest physical address at any moment, and it has no way of knowing the hypervisor moved a page out from under it, so QEMU pins the guest’s entire RAM allocation in host memory as soon as you attach a passed-through device. That 24 GB is committed for the whole life of the VM. The balloon driver stops reclaiming anything, KSM finds nothing to merge, and overcommitting the host — half the reason to run a hypervisor at all — quietly stops applying to this guest. Set the balloon to zero in the VM options so the behaviour is explicit, and size everything else on the box assuming that memory is spoken for.

Inside the guest

Debian 12, then the driver. Use the distribution’s packaged driver where you can — the .run installer from Nvidia works but reinstalls itself into your life on every kernel bump.

1
2
3
4
5
6
7

# in the guest
apt install -y linux-headers-amd64 build-essential
echo "deb http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware" \
  > /etc/apt/sources.list.d/backports.list
apt update
apt install -y -t bookworm-backports nvidia-driver firmware-misc-nonfree
reboot

Then the sanity check that matters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2        |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M  | Bus-Id          Disp.A |  Volatile Uncorr ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap  |           Memory-Usage | GPU-Util  Compute M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060 Ti      Off | 00000000:01:00.0   Off |                  N/A |
| 30%   34C    P8              11W / 200W  |      1MiB /  8192MiB   |      0%      Default |
+-----------------------------------------+------------------------+----------------------+

Note the bus ID: 00000000:01:00.0. QEMU preserved the address, which is cosmetic but pleasing. From here, the container runtime needs the NVIDIA Container Toolkit, and then Ollama sees the card with no further ceremony:

1
2
3
4
5
6
7
8
9

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
# ... repo config elided ...
apt install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
docker exec ollama ollama run llama3.1:8b "explain IOMMU groups in one sentence"

If the model loads and nvidia-smi shows VRAM filling, you’re finished. On 8 GB, an 8B model at Q4_K_M fits entirely in VRAM with room for context — see why Q4_K_M is usually the right default for the reasoning behind that choice.

Troubleshooting: the six ways this fails

“Group N is not viable” on VM start. Something else in the GPU’s IOMMU group is still bound to a host driver. Run the group script again, find the offending device, and either bind it to vfio-pci as well or accept that your board needs the ACS override. This is the most common failure and it is always the same cause.

The host goes black at boot and never comes back. You blacklisted the driver for your only GPU. If the board has no integrated graphics, the passed-through card is also the console. Boot with a rescue USB, remove /etc/modprobe.d/vfio.conf, rebuild the initramfs. Then either add a cheap second card for the host console or accept SSH-only management with a working out-of-band path — which is a good idea regardless, and one I’ve argued for at length elsewhere.

nvidia-smi reports “No devices were found” inside the guest despite lspci showing the card. Nine times out of ten the guest lacks headers matching its running kernel, so the DKMS build silently failed. dkms status tells you immediately.

VM boots once, then the second start hangs. The card didn’t reset cleanly. AMD cards are famous for this (“reset bug”); Nvidia’s consumer cards behave better since the Turing generation but can still sulk. If your card needs it, vendor-reset exists as a kernel module. The pragmatic workaround is to stop rebooting the VM casually.

Terrible throughput despite VRAM being full. Check that the slot is actually running at the width you think. lspci -vvs 01:00.0 | grep LnkSta in the guest reports the negotiated link. I once spent an hour on this before noticing I’d put the card in the chipset-fed x4 slot rather than the CPU-fed x16 one. Inference cares less about PCIe width than gaming does — the model lives in VRAM once loaded — though model load times go up noticeably.

Host locks up hard when the VM shuts down. Almost always the audio function still bound to snd_hda_intel. Check lspci -nnk again.

Is it worth it?

For a machine that exists only to serve models: no. Install Debian on bare metal, install the driver, run Ollama, go outside. The passthrough tax buys you nothing.

For a hypervisor you already run other things on — this is where it earns its keep. My inference VM has been destroyed and rebuilt four times while I worked out what I actually wanted from it, and the DNS resolver on the same physical box never noticed. That isolation is the product; the GPU is incidental. It’s the same argument for keeping a hypervisor boring that made Proxmox worth learning in the first place.

The running cost deserves a mention, since it rarely makes it into the build post. A card sitting in a VM waiting for prompts still draws power simply to exist — that 11W idle figure in the nvidia-smi output above is real, and it is 11W every hour of every day whether or not you ask it anything. Persistence mode holds the driver loaded and the card at a slightly higher floor, buying you faster first-token latency for a few more watts. Across a year the idle draw comfortably outweighs the electricity spent on actual inference, because the card spends almost all of its life doing nothing. Worth knowing before you leave it powered on out of habit.

Budget a whole evening for it. Check IOMMU groups before you buy anything, since a board with a hostile group layout will make you miserable regardless of how good the card is. And when it works, it properly works — near-native tokens per second, and a hypervisor that stays as dull as it should be.