Running Whisper Locally for Offline Transcription

I had about ninety hours of audio sitting on a NAS: recorded talks, a few interviews, voice memos I made while walking and then never listened to again. All of it effectively lost, because audio is the only common file format you cannot grep.

The obvious fix is a transcription service. The obvious objection is that some of those recordings are conversations other people were in, and uploading them to a third party is a decision I don’t get to make on my own. So: locally, then. This turned out to be one of those rare homelab projects where the local option is better than the hosted one on most axes, and the axes where it loses are the ones you can measure in advance.

Which Whisper are we talking about

Whisper is OpenAI’s speech recognition model, released with open weights in 2022. “Running Whisper” now means picking one of four implementations, and the choice matters more than the model size does.

openai/whisper — the reference PyTorch implementation. Correct, well documented, and slow. Its job is to define what the output should look like; treat it as the spec rather than the tool.

whisper.cpp — Gerganov’s C++ port, same lineage as llama.cpp, same GGML file format and the same obsession with running on hardware nobody thinks is enough. It’ll transcribe on a Raspberry Pi, which I’ve written about separately, and it has no Python dependencies at all — a genuine feature when you’re deploying to something small.

faster-whisper — a reimplementation on CTranslate2, running roughly four times faster than the reference at the same accuracy with substantially less memory. This is what I run for anything on a GPU, and what most self-hosted transcription services are quietly using underneath.

WhisperX — faster-whisper plus forced alignment for accurate word-level timestamps, plus speaker diarisation via pyannote. Heavier, more dependencies, and the only sane option when you need to know who said what.

For a batch job against a GPU, faster-whisper. For anything CPU-only or embedded, whisper.cpp. Those two cover almost everything.

Model sizes, and the one that matters

The interesting jump is small to medium, where word error rate on clear English roughly halves. Beyond that, large-v3 buys you accented speech, background noise and non-English — it is dramatically better at Danish than medium is, and roughly the same at a clearly-recorded English podcast.

There’s a distilled option too. distil-large-v3 is about half the size and around twice the speed of large-v3, with a small accuracy cost, and it’s English-only. If your material is all English and you’re transcribing hours at a time, it’s the sweet spot. My audio is mixed, so I run large-v3 and wait.

The actual setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# docker-compose.yml
services:
  whisper:
    image: ghcr.io/linuxserver/faster-whisper:latest
    container_name: whisper
    environment:
      - WHISPER_MODEL=large-v3
      - WHISPER_BEAM=5
      - WHISPER_LANG=auto
    volumes:
      - ./cache:/config
      - /srv/media/audio:/audio:ro
      - /srv/media/transcripts:/out
    ports:
      - "127.0.0.1:10300:10300"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

That container speaks the Wyoming protocol, which matters if you’re wiring it into Home Assistant’s voice pipeline later. For batch work I skip the server entirely and drive the library directly, because the useful knobs live there:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

#!/usr/bin/env python3
# transcribe.py — batch transcription with sane defaults
from faster_whisper import WhisperModel
from pathlib import Path

model = WhisperModel(
    "large-v3",
    device="cuda",
    compute_type="float16",       # int8_float16 halves VRAM, costs ~1% accuracy
    download_root="/srv/models/whisper",
)

AUDIO = Path("/srv/media/audio")
OUT = Path("/srv/media/transcripts")

for f in sorted(AUDIO.glob("**/*.m4a")):
    dest = OUT / f.relative_to(AUDIO).with_suffix(".txt")
    if dest.exists():
        continue                   # resume-safe: never redo finished work
    dest.parent.mkdir(parents=True, exist_ok=True)

    segments, info = model.transcribe(
        str(f),
        beam_size=5,
        vad_filter=True,           # the single most important setting here
        vad_parameters=dict(min_silence_duration_ms=500),
        condition_on_previous_text=False,
    )

    print(f"{f.name}: {info.language} ({info.language_probability:.2f}), "
          f"{info.duration:.0f}s")

    with dest.open("w") as fh:
        for s in segments:
            fh.write(f"[{s.start:7.2f} --> {s.end:7.2f}] {s.text.strip()}\n")

Two of those arguments are doing the heavy lifting, and both are switched off by default.

vad_filter=True runs Silero VAD over the audio first and only feeds Whisper the parts containing speech. This fixes the model’s most notorious failure, covered below, and as a bonus it’s faster because silence never reaches the model.

condition_on_previous_text=False stops each 30-second window being primed with the transcript of the last one. Conditioning improves coherence on clean audio and causes catastrophic repetition loops on messy audio — once the model starts repeating a phrase, the conditioning feeds it back and it repeats forever. I’ve had a five-minute recording produce four hundred identical lines. Turning it off costs a little contextual polish and removes the failure mode entirely.

Running it looks like this:

1
2
3
4

$ python3 transcribe.py
2024-09-14-talk.m4a: en (0.99), 3612s
2024-08-02-memo.m4a: da (0.94), 187s
2024-07-19-interview.m4a: en (0.97), 2841s

On a 12 GB consumer card, large-v3 at float16 chews through roughly 8–10 minutes of audio per minute of wall clock. Ninety hours took a night and a morning. The GPU sat at about 140 W throughout, so call it two kilowatt-hours for the lot — less than the hosted alternative would have cost, though only because I already owned the card. That distinction matters, and I’ve argued the general case for counting your own hardware honestly.

The hallucination problem, which is real

Whisper was trained on 680,000 hours scraped from the internet, an enormous fraction of which is YouTube. It learned the statistical shape of that material, including what tends to appear at the end of a video.

Feed it thirty seconds of silence or room tone and it will confidently emit:

1
2
3

[  0.00 -->  29.00] Thank you for watching!
[  0.00 -->  29.00] Subtitles by the Amara.org community
[  0.00 -->  29.00] Please subscribe to my channel.

None of it was said. The model has a strong prior that audio contains speech, so given no speech it produces the most probable speech-shaped thing it knows. My first run against a recording with a long silent gap produced a transcript that thanked me for watching, twice, in the middle of an interview.

VAD filtering fixes this almost entirely, because silence never reaches the model in the first place. What VAD doesn’t fix is the softer version — quiet background noise that isn’t quite silence gets transcribed as plausible filler. If a transcript matters, spot-check the low-confidence segments rather than trusting the whole file.

The related failure: single-word answers. Ask a question, get “Yeah”, and Whisper often renders that half-second as a full sentence it invented. Short utterances have the least acoustic evidence and the most room for the language model half to improvise.

Prepare the audio first

Whisper resamples everything to 16 kHz mono internally, so feeding it a 48 kHz stereo file means the decode happens inside the Python process, single-threaded, while the GPU waits. On a ninety-hour batch that’s a meaningful fraction of the total time spent doing something ffmpeg does better.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# normalise everything to what Whisper wants, in parallel
find /srv/media/audio -name '*.m4a' -print0 \
  | xargs -0 -P 4 -I{} sh -c '
      out="/srv/media/prepped/$(basename "{}" .m4a).wav"
      [ -f "$out" ] && exit 0
      ffmpeg -nostdin -loglevel error -i "{}" \
        -ar 16000 -ac 1 -c:a pcm_s16le \
        -af "highpass=f=80,loudnorm=I=-16:TP=-1.5:LRA=11" \
        "$out"
    '

The loudnorm filter earns its place. Voice memos recorded on a phone in a pocket sit at wildly different levels, and Whisper’s accuracy on quiet passages improves noticeably once everything is normalised to a consistent loudness. The highpass at 80 Hz strips rumble — traffic, air conditioning, the microphone bumping a table — which the VAD would otherwise sometimes classify as speech.

The -nostdin is there because ffmpeg reads stdin by default and, inside an xargs pipeline, will happily consume the file list you’re feeding it. That’s an afternoon I’m not getting back.

What to do with the text

A directory of .txt files is already better than a directory of .m4a files, and grep -ril "kubernetes" /srv/media/transcripts was genuinely enough for the first month. Two upgrades were worth the effort.

Timestamped links. Because the transcript keeps [start --> end] markers, a two-line script turns a grep hit into a seek. I keep this in ~/.local/bin:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/bin/bash
# hear <pattern> — find a phrase and play the audio from where it was said
grep -rn --include='*.txt' -i "$1" /srv/media/transcripts | while IFS= read -r hit; do
  file="${hit%%:*}"
  ts=$(sed -n "$(echo "$hit" | cut -d: -f2)p" "$file" | grep -oP '^\[\s*\K[0-9.]+')
  audio="/srv/media/audio/$(basename "$file" .txt).m4a"
  echo "$audio at ${ts}s"
  mpv --start="$ts" "$audio"
  break
done

Summaries from a local model. Feeding a 3,000-word transcript to a 7B model with a “extract decisions and open questions” prompt turns an hour-long recording into six bullet points I’ll actually read. The transcripts are already local and the model is already local, so nothing crosses the network — the same reasoning that made local transcription worth doing in the first place applies straight through to the summarisation step. Beyond that, embedding the transcripts makes them retrievable by meaning rather than by exact word, which is a whole pipeline of its own.

Diarisation, if you need names

Transcripts of conversations are much less useful when you can’t tell who’s speaking. WhisperX handles it:

1
2
3
4
5
6
7
8
9

pip install whisperx

whisperx /srv/media/audio/2024-07-19-interview.m4a \
  --model large-v3 \
  --compute_type float16 \
  --diarize \
  --min_speakers 2 --max_speakers 3 \
  --hf_token "$HF_TOKEN" \
  --output_dir /srv/media/transcripts

The --hf_token is required because pyannote’s models sit behind a gated licence acceptance on Hugging Face. You accept the terms once in a browser, then the token lets the download proceed. The models themselves run locally after that.

Set --min_speakers and --max_speakers if you know them. The clustering does a decent job unguided and a much better job when told the answer. Realistic expectation: two people in a quiet room, near-perfect. Four people around a table with one bad microphone, it’ll merge two of them and you’ll be editing.

Troubleshooting

CUDA out of memory on large-v3 with 8 GB. Use compute_type="int8_float16", which drops the model to about 5 GB with an accuracy cost small enough that I’ve never noticed it in practice. Failing that, drop to medium.

Could not load library libcudnn_ops_infer.so.8. faster-whisper needs cuDNN 8, and CTranslate2’s wheels expect it in a specific place. This is why I run it in a container built on nvidia/cuda:12.x-cudnn8-runtime rather than fighting pip on the host. It’s the single most common installation failure with this stack.

Language detected wrongly on short files. Detection uses only the first 30 seconds. A file that opens with music or throat-clearing gets classified on that. Pass language="en" explicitly when you know.

Timestamps drift over a long recording. Whisper’s native timestamps are estimates from its decoder and they wander on files over an hour. WhisperX’s forced alignment fixes this properly, and it’s the main reason to use WhisperX even when you don’t need diarisation.

Transcription is far slower than expected. Check it’s actually on the GPU — WhisperModel("large-v3", device="cuda") falls back to CPU with only a warning if CUDA is unavailable. nvidia-smi during a run settles it in seconds. If you’re passing the card through to a VM, the passthrough itself has failure modes that look identical to this from inside the guest.

Output has no punctuation. You’re on tiny or base. Punctuation is one of the first things the small models drop.

Is it worth it?

For one meeting recording: no. Use a hosted service, get your text in ninety seconds, get on with your day.

For a backlog, or for anything with other people’s voices in it, local Whisper is straightforwardly the better option. The accuracy of large-v3 matches the commercial services I compared against on clear audio and beats several of them on accented English. The material never leaves the house. And once the pipeline exists, the marginal cost of transcribing something is electricity.

What surprised me is how much the searchability changed my behaviour rather than the transcripts themselves. Ninety hours of audio was an archive I never opened. Ninety hours of text next to the audio files is something I grep about once a week, follow the timestamp, and listen to the ninety seconds that matter. The transcription is the index. The audio is still the artefact.