Self-Hosted Speech-to-Text in a Home Automation Loop

Transcribing a recording is a batch problem. Nobody minds if it takes three minutes, because nobody is standing in the kitchen waiting for it.

Voice control is a different discipline entirely. You say “turn the kitchen lights off”, and the gap between your last syllable and the click of the relay is the entire user experience. Under about a second, the house feels responsive. Over two, you say it again, because you’ve assumed it didn’t hear you — and now two commands are in flight and the lights come on again.

The whole engineering problem is that budget. Every component in the chain spends some of it, and the ones that spend the most are rarely the ones you’d expect.

What the chain actually contains

A voice command touches five stages, and understanding which are cheap is what makes the tuning tractable.

Wake word. A small model runs continuously on the microphone device, listening for one phrase. This must be cheap enough to run forever on a microcontroller and accurate enough to ignore the television.

VAD and endpointing. Once woken, something decides when you’ve stopped talking. This is the stage that quietly eats your budget, because “the human has finished” is a judgement call and the safe default is to wait.

Speech to text. Whisper, or something like it, turns audio into a string.

Intent matching. Home Assistant takes the string and works out which entity you meant and what to do to it. Local, template-based, and fast.

Text to speech. Piper says “OK” back to you. Optional and mostly cosmetic, though a house that acknowledges commands feels considerably more trustworthy than one that silently obeys.

Where the milliseconds go

Measured on my setup — a mini PC running Home Assistant and the Wyoming services, an ESP32-based microphone in the kitchen, everything on the same wired network:

Read the VAD line again. The single largest cost in a well-tuned local voice pipeline is waiting to be sure you stopped speaking. It dwarfs the transcription. People spend a weekend putting Whisper on a GPU to shave 200 ms off a stage that costs 250 ms, while 700 ms of deliberate silence sits above it untouched.

And read the large-v3 line, because it’s the counterintuitive one. Running the best model makes the assistant worse. On a constrained vocabulary — twenty light entities, five rooms, four verbs — tiny.en is already correct. There is no accuracy left for a bigger model to recover, so all it does is add two seconds.

The setup

Three Wyoming services, one compose file. Wyoming is Home Assistant’s protocol for voice components: a thin, boring wire format that lets each piece run wherever it likes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# docker-compose.yml
services:
  whisper:
    image: rhasspy/wyoming-whisper:latest
    container_name: wyoming-whisper
    command: >
      --model tiny.en
      --language en
      --beam-size 1
      --initial-prompt "turn on off lights kitchen bedroom living room set brightness"
    volumes:
      - ./whisper-data:/data
    ports:
      - "10300:10300"
    restart: unless-stopped

  piper:
    image: rhasspy/wyoming-piper:latest
    container_name: wyoming-piper
    command: --voice en_GB-alba-medium
    volumes:
      - ./piper-data:/data
    ports:
      - "10200:10200"
    restart: unless-stopped

  openwakeword:
    image: rhasspy/wyoming-openwakeword:latest
    container_name: wyoming-openwakeword
    command: --preload-model 'ok_nabu'
    volumes:
      - ./wakeword-custom:/custom
    ports:
      - "10400:10400"
    restart: unless-stopped

Three flags there are doing real work.

--beam-size 1 turns off beam search. The default of 5 explores five candidate transcriptions and picks the best, which is the right trade for a podcast and the wrong one for “lights off”. Greedy decoding is roughly three times faster and, on a two-word command, gives the identical answer.

--initial-prompt primes the decoder with vocabulary. Whisper’s decoder is a language model, and telling it which words are likely biases it towards them. Feed it your room names and the difference is immediate: “bedroom” stops being transcribed as “bed room”, and unusual entity names stop being helpfully corrected into common English words.

en_GB-alba-medium rather than a high voice. Piper’s high-quality voices sound better and take three times as long. For “OK” and “sorry, I didn’t understand that”, medium is inaudibly different and finishes before you’ve stopped listening for it.

Home Assistant’s side

The Wyoming integration discovers each service; add them by host and port. Then the Assist pipeline glues them together — Settings → Voice assistants → Add assistant, pick Whisper for STT, Piper for TTS, openWakeWord for wake word, and “Home Assistant” for the conversation agent.

The intent layer is where your commands actually become useful, and it’s plain YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# config/custom_sentences/en/lab.yaml
language: en
intents:
  LabPowerQuery:
    data:
      - sentences:
          - "how much power is the rack (using|drawing)"
          - "what's the rack (pulling|drawing)"
        response: rack_power

  HouseGoodnight:
    data:
      - sentences:
          - "(goodnight|good night)"
          - "I'm going to bed"
        response: goodnight

responses:
  intents:
    LabPowerQuery:
      rack_power: "The rack is drawing {{ states('sensor.rack_power') }} watts."
    HouseGoodnight:
      goodnight: "Goodnight. Locking up."

Then an automation on HouseGoodnight to actually turn things off. The split — sentences and response text in custom_sentences, behaviour in an automation — is worth respecting, because it means you can iterate on phrasings without touching logic.

Note the alternation syntax (goodnight|good night). Whisper will transcribe that phrase both ways depending on nothing you can control, so matching both is mandatory rather than thorough. This is the general lesson: your intent sentences have to cover what the transcriber produces, which overlaps with, but differs from, what a person says.

The satellite in the kitchen

The microphone end is an ESP32 with an I2S microphone and a small speaker, flashed with ESPHome. About £15 of parts, and the config is short enough to read in one go:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

# kitchen-satellite.yaml
esphome:
  name: kitchen-satellite

esp32:
  board: esp32dev
  framework:
    type: esp-idf

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

api:
  encryption:
    key: !secret api_key

i2s_audio:
  - id: mic_bus
    i2s_lrclk_pin: GPIO25
    i2s_bclk_pin: GPIO26

microphone:
  - platform: i2s_audio
    id: mic
    i2s_audio_id: mic_bus
    i2s_din_pin: GPIO33
    adc_type: external
    pdm: false
    channel: left
    bits_per_sample: 32bit

voice_assistant:
  microphone: mic
  use_wake_word: false        # wake word runs on the HA side
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 2.0
  on_start:
    - light.turn_on:
        id: status_led
        brightness: 60%
  on_end:
    - light.turn_off: status_led

noise_suppression_level and auto_gain are the two settings that decide whether this thing works from three metres away. Both default to off, and both should be on for any microphone that isn’t in your hand. Auto gain in particular converts “the model heard nothing” into “the model heard a command” on quiet speech, at the cost of amplifying the fridge when nobody’s talking — which the VAD then discards.

The on_start light is not decorative. An assistant that shows you it’s listening removes the single most common human failure, which is speaking before the device is ready and losing the first two words. Half of the “it never hears me” complaints I’ve debugged were people talking over the wake-word acknowledgement.

use_wake_word: false sends the wake-word job to openWakeWord on the server. The alternative, ESPHome’s micro_wake_word, runs detection on the ESP32 itself and stops the satellite streaming audio to the network until it hears its name. That’s the better architecture — less network chatter, and no audio leaves the device unless you addressed it — and it constrains you to the handful of pre-trained models available. I run server-side detection because I wanted a custom phrase; if the stock wake words suit you, take the on-device route and enjoy the quieter network.

Tuning the stage that costs the most

The VAD silence timeout lives in the Assist pipeline’s advanced settings. The default is generous because the failure mode of being too aggressive is horrible — cutting someone off mid-sentence and acting on half a command.

For short commands you can push it down hard. I run 400 ms of trailing silence, which is fine for “kitchen lights off” and occasionally truncates me if I pause mid-thought. That’s a trade I’ll take, because a 400 ms endpoint plus tiny.en puts the whole round trip at roughly 700 ms and the house feels instant.

The other lever nobody uses: put the wake word and the command in one breath. “Ok Nabu, kitchen lights off” spoken continuously endpoints faster than “Ok Nabu” … pause … “kitchen lights off”, because the VAD isn’t sitting through your pause. Once you’ve internalised that, the assistant feels twice as fast with no configuration change at all.

Troubleshooting

The wake word triggers at the television. openWakeWord has a threshold and the default is tuned for recall. Raise it. If you’re using a common phrase, expect this; a custom wake word trained on something phonetically unusual has far fewer false positives than any variation on a name people say on TV.

Commands work at the microphone, fail from across the room. Almost always the microphone rather than the software. This is the least glamorous finding of the whole project: mic array quality dominates everything downstream. A single cheap electret at three metres in a room with hard floors produces audio that no model can rescue. Two-mic beamforming at the same distance is a different world.

Short commands get transcribed as complete sentences. Whisper hallucinating over insufficient acoustic evidence — the same failure that plagues batch transcription of silence, scaled down. “Off” becomes “I’m off to the shops.” The --initial-prompt helps a lot; a min_speech_duration in the VAD helps more, by rejecting the utterance rather than guessing at it.

Everything is 3–4 seconds and you can’t find the culprit. Turn on debug logging for homeassistant.components.assist_pipeline and watch the events. Each stage logs its start and end. In my experience the answer is either the VAD timeout or a cloud-backed integration in the service call stage — a light behind a vendor cloud API can spend 1,500 ms acknowledging a command that Zigbee would have executed in 40 ms. The voice pipeline gets blamed for the lamp’s networking.

Whisper is pinned at 100% CPU and slow. tiny.en on a modern CPU handles a command in a couple hundred milliseconds; if it doesn’t, check which model actually loaded. The container downloads on first run and falls back silently if the requested name is wrong. docker logs wyoming-whisper | head shows what it settled on.

It all works, then breaks after a Home Assistant update. The Wyoming integration and the containers version independently. Pin the image tags — rhasspy/wyoming-whisper:2.4.0 rather than latest — and update deliberately. latest on a voice stack means your kitchen lights stop responding on a Tuesday for reasons you’ll spend an hour on.

Is it worth it?

Honest answer: as a voice assistant, it’s worse than the commercial ones. It doesn’t know what the weather is unless you wire that up, it can’t answer general questions, it mishears you more often, and setting it up cost me a weekend that Amazon would have charged £30 and ten minutes for.

As a control surface for a house you already automate, it’s better than any of them, for three specific reasons. It works when the internet is down, which is precisely when you most want the lights to respond. It has access to every entity in Home Assistant, including the ones no commercial assistant will ever integrate — my rack’s power draw, the state of a sensor I built myself. And the audio never leaves the building, which for an always-on microphone in a kitchen is the entire point.

Start with tiny.en, a 400 ms endpoint, and a decent microphone. Spend money on the microphone and nothing else. If you’re building this out from scratch, the broader wiring of Whisper and Piper into Home Assistant covers the parts I’ve skipped here, and the Raspberry Pi variant is worth reading if your hardware budget is smaller than mine.

The thing I did wrong for two months: I ran small.en on a GPU because bigger felt better, and my kitchen lights took two and a half seconds. Switching to tiny.en on the CPU made the house feel alive and freed the card for things that actually need it.