Jellyfin Hardware Transcoding Without the Pain

The first thing to understand about transcoding is that the best transcode is the one that never happens.

A transcode means your server is decoding a video stream, re-encoding it into something else, and shipping the result — in real time, for every viewer. Direct play means the server reads bytes off a disk and puts them on a socket. One of those costs a CPU or a GPU. The other costs nothing at all and looks better, because it is the original file.

I mention this because a great many people arrive at hardware transcoding having concluded their setup is slow, when what they actually have is a client that cannot play HEVC and a subtitle track forcing a burn-in. Getting the GPU working fixes the symptom. Understanding why you are transcoding at all often removes the problem.

Before you build anything, spend five minutes finding out what your setup is currently doing. Play a file on each client you care about, open Dashboard → Activity, and read what it says. “Direct Play” means the client is receiving the original file. “Direct Stream” means the container is being remuxed and the video is untouched, which is cheap. “Transcode” tells you the codec it is converting from and to, and that string is a diagnosis: HEVC → H264 means a client codec gap, 1080p → 720p means a bitrate limit somebody set, and a transcode with no obvious reason means subtitles. Three sessions of that data will tell you whether you have a hardware problem or a library problem, and the second one is far more common and considerably cheaper to fix.

That said: you will need it eventually, someone will always watch on a browser that supports three codecs, and the difference between software and hardware is the difference between one stream and eight. Here is how to get it right in a container, which is where all the difficulty lives.

Why the Container Makes This Hard

On bare metal this is nearly automatic. Install the driver, install Jellyfin, tick the box.

In Docker there are four things that have to line up, and the failure mode when any one is wrong is identical: the option is there in the dashboard, you enable it, and playback silently falls back to software while your CPU melts. Jellyfin just quietly does the slow thing.

The four are: the device node must be passed into the container; the container’s user must have permission to open it; the driver userspace must be present inside the image; and the GPU has to actually support the codec you are asking for. Miss any and you get the same non-error.

Intel Quick Sync

Quick Sync is the sensible default for a homelab. Any Intel chip with integrated graphics from roughly the last decade has it, the encoder quality is good, and the power draw at idle is a couple of watts. My box does four simultaneous 1080p transcodes without the fan changing note.

First, confirm the render node exists on the host:

1
2
3

ls -la /dev/dri
# crw-rw---- 1 root video  226,   0 Nov 13 09:12 card0
# crw-rw---- 1 root render 226, 128 Nov 13 09:12 renderD128

renderD128 is the one you want. Note the group — render here, and on other distributions it is video, and its numeric ID varies between distributions in a way that will waste an hour of your life. Get the number:

1
2

getent group render | cut -d: -f3
# 993

That number goes into the compose file. The group name inside the container is meaningless; the kernel checks the GID.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

services:
  jellyfin:
    image: jellyfin/jellyfin:10.9.11
    user: 1000:1000
    group_add:
      - "993"          # host's render group GID — check yours
    devices:
      - /dev/dri:/dev/dri
    restart: unless-stopped
    ports:
      - "8096:8096"
    volumes:
      - jellyfin_config:/config
      - jellyfin_cache:/cache
      - /srv/media:/media:ro
    environment:
      JELLYFIN_PublishedServerUrl: https://jellyfin.mylab.local

volumes:
  jellyfin_config:
  jellyfin_cache:

Pinning the tag rather than tracking latest matters here more than usual, because a Jellyfin major upgrade changes the bundled ffmpeg and can change your transcoding behaviour overnight. The 10.10 release landed a few days ago and I have deliberately left this box on the 10.9 line until I have tested it somewhere I do not watch films.

Verify from inside the container before touching the dashboard:

1
2

docker compose exec jellyfin /usr/lib/jellyfin-ffmpeg/vainfo
# should list VAProfileH264Main, VAProfileHEVCMain, VAProfileVP9Profile0 ...

If that prints profiles, you are done and the dashboard will work. If it prints failed to initialize display, the device is not passed through. If it prints a permission error, the GID is wrong.

Then in Dashboard → Playback → Transcoding, pick Video Acceleration API (VA-API) and set the device to /dev/dri/renderD128. Tick only the codecs vainfo reported. Ticking HEVC on a chip that cannot encode it produces — you guessed it — a silent software fallback.

NVIDIA

More performance, more power, more setup. You need the container toolkit on the host, and then:

1
2
3
4

    runtime: nvidia
    environment:
      NVIDIA_VISIBLE_DEVICES: all
      NVIDIA_DRIVER_CAPABILITIES: compute,video,utility

That video capability is the one people miss. Without it NVENC is invisible inside the container and everything else looks fine.

Check it:

1

docker compose exec jellyfin nvidia-smi

The catch on consumer cards is the concurrent NVENC session limit — historically three on GeForce parts, enforced in the driver rather than the silicon. If you have four people watching, you hit a wall that has nothing to do with your hardware being capable. Quadro and datacentre cards have no such limit, which is the entire reason they cost what they cost.

Tone Mapping, or Why HDR Looks Grey

This is the most common “hardware transcoding is broken” report and it is a separate feature.

An HDR file played on an SDR display needs its colour space converted. Without tone mapping, you get the washed-out grey picture everyone recognises. Jellyfin can do this on the GPU, and it is a separate tick box from transcoding itself.

On Intel, enable tone mapping and prefer the VPP option where your chip supports it — it runs on fixed-function hardware and costs almost nothing. The OpenCL path is more accurate and needs the compute runtime present in the image. On NVIDIA the CUDA path does the same job.

The honest advice is to avoid the situation. If your library is 4K HDR and your clients are 1080p SDR, you are converting HDR to SDR on every single play. Keeping a 1080p SDR version of the handful of films that get watched on phones costs disk and saves you the entire problem. Disk is cheap; my media sits on the SnapRAID array where a second copy of forty films is noise.

The Subtitle That Ruins Everything

Here is the thing that catches everyone, and it is worth the section.

Text subtitles — SRT, WebVTT — are sent to the client and rendered there. Free.

Image-based subtitles — PGS from a Blu-ray rip, or complex styled ASS from anime releases — cannot be sent to most clients. Jellyfin has to burn them into the picture, which means decoding the video, compositing the subtitle, and re-encoding. Every time. Even if the client could otherwise direct play the file perfectly.

So the person who reports “my server transcodes everything even though the client supports HEVC” has usually just turned subtitles on. The fix is to convert your PGS tracks to SRT with OCR ahead of time and store both, at which point that film direct plays forever.

This one detail has saved me more CPU than the GPU did.

Which Silicon, Actually

The buying question, answered from having wasted money on it.

An Intel iGPU from an 8th-generation chip or newer handles H.264 and HEVC decode and encode, which covers essentially everything in a normal library. It draws two or three watts while doing it. A used mini PC with one of these costs less than a graphics card and will saturate your upload before it saturates the encoder. For the overwhelming majority of homelabs this is the whole answer, and the electricity maths makes the case louder than the performance does.

Generation matters for one thing: AV1. Decode arrived on 11th-generation Intel parts and encode on the Arc discrete cards. If your library has no AV1 in it — and in 2024 almost nobody’s does — this is a problem for later. It is coming, though, and it is the one reason to think about the future here at all.

The reason to reach for a discrete card is stream count. An iGPU does four to six 1080p transcodes comfortably and then degrades; a modern NVIDIA card does a dozen and is limited by driver policy rather than throughput. If you are serving an extended family, that is the line.

The reason to avoid a discrete card is that it idles at 10–15 watts to do nothing for twenty-three hours a day, and that adds up to real money over a year for a capability you use during two films a week.

AMD works via VA-API and is the least well-trodden path of the three. It functions. The documentation is thinner, the driver situation on older cards is fiddlier, and you will find fewer people who have hit your exact problem. If you already have one, use it. I would not buy one for this.

The unglamorous conclusion is that the correct transcoding GPU for most people is the one already soldered to the CPU they own, and the correct amount of money to spend is zero.

Troubleshooting

It says hardware acceleration is enabled and the CPU is still at 100%. Play something, then read the transcode log — Dashboard → Logs, or the file directly. The ffmpeg command line is printed in full. Look for -hwaccel vaapi and h264_vaapi. If you see libx264, it fell back. The reason is usually on the line above.

1
2

docker compose exec jellyfin sh -c 'ls -t /config/log/ | head -3'
docker compose logs jellyfin --tail=200 | grep -i -E 'vaapi|qsv|nvenc|fallback'

Permission denied on /dev/dri/renderD128. The GID. Get it from the host, put it in group_add, restart. Do not solve this by running the container as root — you are passing a kernel device into a process that parses untrusted media files, which is precisely the situation least deserving of root.

Watch the GPU actually work. On Intel, intel_gpu_top on the host shows the video engine light up during a transcode. If the render bar moves and the video bar does not, you are doing GPU-assisted decode and software encode, which is a half-configured state that fools a lot of people.

Playback starts then dies after ten seconds. The transcode cache filled or the disk is too slow. That /cache volume gets hammered — put it on an SSD, and if it is on a spinning array you will feel it. A ramdisk works if you have the memory to spare.

Everything transcodes after an upgrade. Codec support flags reset, or the bundled ffmpeg changed. Re-run vainfo, re-tick the boxes. This is the argument for the pinned tag.

Four users, four streams, one very unhappy box. Check whether they are all transcoding for the same reason. Usually they are, and usually the reason is one file with a codec nobody supports. Watch it with your monitoring — the Prometheus and Grafana setup makes this visible over time in a way that staring at top never will.

Verdict

Hardware transcoding in Jellyfin is genuinely straightforward once you know the four things that have to line up, and it is miserable until then because every failure looks the same. Pass the device, get the GID right, check vainfo before you touch the dashboard, and tick only the codecs the GPU actually has.

Who needs it: anyone with more than two simultaneous viewers, anyone who watches away from home on a variable connection, anyone with a 4K library and 1080p clients.

Who does not: a household of two watching on a TV that direct plays everything. If your dashboard says “Direct Play” for every session you have ever had, a GPU changes nothing and you have better places to spend a weekend — owning your music library, for instance, where the payoff is immediate.

The genuinely useful realisation is that most transcoding is self-inflicted. Fix your subtitle tracks, keep a compatible copy of the films that get watched on phones, and check what your clients actually support. I did all three and my four-transcode ceiling turned into a problem I no longer have.