MQTT Essentials: The Protocol Behind Every Smart Home

If you’ve spent any time poking at a smart home, you’ve bumped into MQTT whether you meant to or not. Zigbee2MQTT, ESPHome, Tasmota, Home Assistant’s own internal bus — they all lean on it. It’s worth understanding properly, because once you do, an awful lot of “magic” stops being magic and starts being a handful of topics and a tiny broker process.

What it actually is

MQTT is a publish/subscribe messaging protocol. It was designed in 1999 by Andy Stanford-Clark of IBM and Arlen Nipper for monitoring oil pipeline telemetry over expensive, unreliable satellite links, which tells you everything about its priorities: tiny payloads, low overhead, and tolerance for flaky connections. When every byte over the satellite costs money and the connection might drop mid-sentence, you design a protocol that says as little as possible and copes gracefully when it’s cut off. Those constraints turned out to describe a battery-powered wireless sensor in a spare bedroom almost perfectly, which is why a two-decade-old industrial protocol quietly became the backbone of the consumer smart home. Devices don’t talk to each other directly. They talk to a broker — a small server that receives messages and fans them out to whoever has subscribed.

Messages are organised into topics, which look like file paths:

1
2
3

zigbee2mqtt/kitchen_motion
home/livingroom/temperature
tele/garage_door/STATE

A device publishes a value to a topic. Anything that has subscribed to that topic gets the value pushed to it. The publisher has no idea who’s listening, and frankly doesn’t care. That decoupling is the whole point — your motion sensor doesn’t need to know that three automations, a dashboard, and a logging script all want its data.

This is why MQTT scales so gracefully in a house full of gadgets. Compare it to the alternative, where every device polls every other device over HTTP: fifty sensors and fifty consumers becomes a quadratic mess of connections, and half of them are asleep to save battery when you ask. With pub/sub there’s exactly one connection per device — to the broker — and the broker handles the fan-out. A battery sensor publishes once when its reading changes and goes back to sleep; it never has to hold open a connection waiting for someone to ask. That’s the design paying off, and it’s why a protocol built for satellite links over oil pipelines turns out to be perfect for a cupboard full of Zigbee door sensors.

Running a broker

The de-facto broker is Eclipse Mosquitto. It’s a single C binary with a footprint so small you’ll forget it’s running. Here’s a sane self-hosted setup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

services:
  mosquitto:
    image: eclipse-mosquitto:2
    container_name: mosquitto
    restart: unless-stopped
    ports:
      - "1883:1883"
    volumes:
      - ./config:/mosquitto/config
      - ./data:/mosquitto/data
      - ./log:/mosquitto/log

The one trap newcomers hit: Mosquitto 2.x stopped allowing anonymous connections by default, which is correct but surprises people. You need an explicit config file at ./config/mosquitto.conf:

1
2
3
4
5

listener 1883
persistence true
persistence_location /mosquitto/data/
log_dest stdout
password_file /mosquitto/config/passwd

Then create a user:

1

docker exec -it mosquitto mosquitto_passwd -c /mosquitto/config/passwd homeassistant

Restart the container and you’ve got an authenticated broker. Don’t be the person who exposes port 1883 to the internet with allow_anonymous true — there are bots that scan for exactly that, and an open broker leaks every sensor reading in your house.

Even authenticated, an MQTT broker deserves to live somewhere isolated. This is the same argument I make for keeping smart-home gear off your main LAN entirely: a cheap sensor with firmware you’ll never update is exactly the kind of foothold you don’t want next to your NAS. I put the broker and every device that talks to it on their own segment, which I’ve written up in detail in VLAN segmentation at home. The broker is the one process that sees everything your devices say — treat it as the sensitive thing it is.

Watching the traffic

The fastest way to understand MQTT is to subscribe to the firehose and watch. The mosquitto_sub and mosquitto_pub clients are perfect for this:

1
2
3
4
5
6

# Subscribe to everything (the # wildcard matches all levels)
mosquitto_sub -h localhost -u homeassistant -P secret -t '#' -v

zigbee2mqtt/kitchen_motion {"occupancy":true,"battery":87,"linkquality":120}
home/livingroom/temperature 21.4
tele/garage_door/STATE {"POWER":"OFF"}

The -v flag prints the topic alongside the payload, which you want. There are two wildcards: + matches a single level (home/+/temperature), and # matches everything below a point (zigbee2mqtt/#). Publishing is just as direct:

1
2

mosquitto_pub -h localhost -u homeassistant -P secret \
  -t 'home/livingroom/setpoint' -m '20.5'

That single line is enough to drive a thermostat if something is subscribed to the topic. This is where MQTT stops being abstract — you can poke your whole house from a shell.

Once you can see the traffic, the ecosystem around MQTT makes a lot more sense. When you pair a Zigbee coordinator with Zigbee2MQTT, every button, bulb and sensor simply becomes a stream of topics under zigbee2mqtt/, which is why mosquitto_sub -t '#' is the single best debugging tool for a misbehaving Zigbee device — you watch the raw messages and instantly see whether the sensor is even publishing. The same is true of ESPHome sensors you build yourself: they announce themselves over MQTT (or Home Assistant’s native API, but MQTT is the lingua franca), and the firehose is where you confirm your soldering actually worked before you write a single automation.

QoS, retain, and the bits that bite

Two features cause most of the confusion, so learn them now.

Retain tells the broker to hold onto the last message on a topic and immediately deliver it to any new subscriber. This is why your dashboard shows the current temperature the instant it connects, rather than waiting for the next reading. Sensors usually retain their state; events (like “button pressed”) usually don’t, because re-delivering a stale button press on reconnect is chaos.

QoS (Quality of Service) has three levels: 0 (fire and forget), 1 (delivered at least once, possibly duplicated), and 2 (delivered exactly once, with more handshaking). For home automation, QoS 0 or 1 is almost always right. QoS 2 is rarely worth its overhead unless a duplicated message would actually cause harm.

The instinct is to reach for QoS 2 because “exactly once” sounds obviously best, but resist it. QoS 2 requires a four-part handshake for every message, which on a battery device means more radio-on time and shorter battery life for a guarantee you rarely need. A temperature reading published every thirty seconds does not care if one copy is lost — the next one arrives momentarily. The place QoS 1 or 2 earns its keep is commands: a “turn the heating off” message you genuinely want delivered even if the broker was briefly unreachable. Match the QoS to the cost of a lost message, not to which number is biggest.

Home Assistant MQTT Discovery

The feature that makes MQTT genuinely pleasant with Home Assistant is discovery. Rather than hand-configuring every entity, a device publishes a specially-formatted config message to a topic under homeassistant/, and Home Assistant automatically creates the entity. Zigbee2MQTT and ESPHome both do this for you, which is why a new Zigbee bulb often just appears in Home Assistant seconds after pairing. A discovery message looks roughly like this:

1
2

homeassistant/sensor/livingroom_temp/config
{"name":"Living Room Temperature","state_topic":"home/livingroom/temperature","unit_of_measurement":"°C","device_class":"temperature"}

That config is itself a retained message, which is how the entity survives a Home Assistant restart — the broker replays it on reconnect. Understanding this demystifies a lot of “where did that entity come from?” moments, and it’s the first place to look when an entity you expected fails to appear: subscribe to homeassistant/# and check whether the discovery message was ever actually published.

The classic footgun is a retained message you can’t get rid of. If a misconfigured device retained nonsense to a topic, it haunts you forever. Clear it by publishing an empty retained message:

1
2

mosquitto_pub -h localhost -u homeassistant -P secret \
  -t 'zigbee2mqtt/old_device' -r -n

The -r retains, the -n sends a null payload, and the broker deletes the retained entry.

Troubleshooting: when messages don’t arrive

MQTT problems are nearly always one of a small handful of things, and the fix is usually to stop guessing and subscribe to # to see reality.

A device connects, then drops, then reconnects in a loop. This is almost always duplicate client IDs. MQTT insists each connection has a unique client ID, and if two devices (or two copies of one misconfigured integration) claim the same ID, the broker kicks the older one off every time the newer one connects — producing an endless flap. Give each client a distinct ID and the loop stops dead.

Home Assistant can’t see a device that’s clearly publishing. Confirm the publish first with mosquitto_sub -t '#' -v; if the message is there, the problem is the subscription side — usually a topic typo or a wrong wildcard. Remember + is one level and # is everything below; home/#/temperature is invalid because # must be the last character. That one catches everyone once.

Nothing works after enabling authentication. The commonest cause is a password_file the broker can’t read. Mosquitto is picky about permissions on that file; if the container logs say “Error: Unable to open pwfile”, check ownership and that the path inside the container matches your config. And after editing mosquitto.conf you must actually restart the broker — it does not hot-reload.

Retained messages ghosting a deleted device. Covered above, but it’s worth repeating because it’s the single most confusing MQTT symptom: a device you removed weeks ago still shows in Home Assistant. The stale retained message is doing that. Clear it with the empty-retained-message trick and it’s gone for good.

Is it worth it / who is this for

MQTT is worth understanding for anyone running more than a couple of smart devices, and essential if you self-host Home Assistant with Zigbee or ESPHome gear. It is genuinely simple — there’s no sprawling spec to wade through, and an afternoon with mosquitto_sub teaches you most of what matters.

It’s overkill if your “smart home” is two cloud-locked plugs controlled by an app; you’ll never see the broker. But the moment you want local control, automations that survive an internet outage, and the freedom to wire any device to any logic, MQTT is the layer everything sits on. Run a broker, subscribe to # for an evening, and watch your house narrate itself. It’s the most useful hour you’ll spend in home automation.

One last piece of advice from having lived with it: resist the urge to over-engineer your topic structure on day one. A tidy, hierarchical scheme (home/<room>/<device>/<measurement>) pays off enormously once you have fifty entities, but you don’t need to design it perfectly before you begin. Start simple, watch the firehose, and refactor when the shape of your house’s data becomes obvious. The broker doesn’t care, and neither, in the end, should you.