Systemd Without Fear: Writing Your First Service Unit
Making Linux babysit your apps
Contents
You have written a small program. It runs beautifully in your terminal. Then you close the terminal, or the box reboots, or the process quietly dies at 3am, and your service is simply gone. I have lost more small self-hosted tools to a closed SSH session than I care to admit — a webhook receiver here, a scraper there, all launched from a shell that later closed and took the process with it. There is a tool already installed on virtually every modern Linux machine that solves all of this, and yet plenty of otherwise-competent people treat it like forbidden magic. That tool is systemd, and writing your first service unit is far less frightening than its reputation suggests.
What systemd and units actually are
systemd is the init system on most mainstream distributions: Debian, Ubuntu, Fedora, RHEL, Arch, openSUSE and friends. It is the very first process the kernel starts (PID 1), and its job is to bring the system up, start everything in the right order, and keep an eye on things while the machine is running.
The way you tell systemd about “things” is through units. A unit is a small text file describing something systemd should manage. There are several kinds: .service units for processes, .timer units for scheduled jobs, .socket units for socket activation, .mount units for filesystems, and a few others. For running your own application, the one you want is the humble .service.
Why bother running your script as a service
You could just launch your app in a tmux session and hope for the best. People do. But a service unit gives you several things for nothing:
- Restart on crash. If the process exits unexpectedly, systemd brings it back.
- Start on boot. No more “did someone forget to start the app after the reboot?”
- Centralised logging. Anything your app writes to stdout or stderr is captured by the journal automatically.
- Resource and security controls. You can sandbox the process, limit its memory, and run it as an unprivileged user with a single line each.
- Dependency ordering. You can say “start after the network is up” or “start after the database” and systemd sorts the sequencing out.
In short, you stop being your app’s babysitter and let the operating system do it instead. It is also the honest alternative to reaching for a container for every little thing. A container is a fine unit of deployment, but if all you have is one Python script that needs to survive a reboot, a service unit is a dozen lines of plain text with no daemon, no image registry and no networking layer to reason about. When you do run containers, incidentally, this same knowledge pays off — the tidy way to run a rootless container as a background service is a systemd unit, which is exactly how Podman prefers you do it.
The anatomy of a .service file
A service unit is an INI-style file split into sections. Here is the skeleton, with the directives you will use most often:
| |
Let us walk through the three sections.
[Unit] holds metadata and ordering. Description is the human-readable label you see in systemctl status. After= defines ordering (start this after the network target), while Wants= is a soft dependency that pulls the network target in without making your service fail if it is absent.
[Service] is the meat. ExecStart is the full command to run — always use absolute paths, because systemd does not inherit your shell’s PATH. Type=simple (the default) means “the process you start is the service; don’t wait for it to fork”. User and Group drop privileges so the app does not run as root. WorkingDirectory sets the directory the process starts in. Environment injects environment variables; you can repeat it, or point EnvironmentFile=/etc/myapp.env at a file of KEY=value lines. Restart=on-failure restarts the process if it exits non-zero, and RestartSec adds a short delay so you do not hammer a failing service in a tight loop.
[Install] tells systemd what should happen when you enable the unit. WantedBy=multi-user.target is the usual choice and roughly means “start this when the system reaches normal multi-user operation”, i.e. on boot.
A worked example
Say you have a Python app living in /opt/widget. Create the unit at /etc/systemd/system/widget.service:
| |
First, create the dedicated user so the app never runs as root:
| |
The --system flag creates a service account with no login, which is exactly what a daemon wants.
Enable, start, and check it
Whenever you create or edit a unit, systemd needs to re-read its configuration:
| |
Now start the service and have it come up on every boot in one command:
| |
The --now flag means “enable for boot and start it immediately”. Check how it is doing:
| |
You will see whether it is active (running), its PID, memory use, and the last few log lines. To restart after a code change, or stop it entirely:
| |
Reading the logs with journalctl
Because systemd captures your app’s output, you never need to wire up your own log files just to see what happened. The journal is queryable:
| |
The -p flag filters by priority (err, warning, info, and so on), -b limits to the current boot, and -f follows new entries. This alone is worth the price of admission.
Timers: a tidier cron
If you reach for cron to run something on a schedule, systemd has a cleaner alternative: a .timer paired with a .service. The service does the work once and exits; the timer says when to run it. Create backup.service (a one-shot, Type=oneshot) and then backup.timer:
| |
OnCalendar uses a readable syntax, Persistent=true runs a missed job if the machine was off at the scheduled time, and the logs land in the journal alongside everything else. Enable it with sudo systemctl enable --now backup.timer and inspect upcoming runs with systemctl list-timers. The win over cron is not just readability: a timer’s output goes to the journal automatically, so a failed nightly backup leaves a queryable record instead of vanishing into an unread /var/spool/mail entry.
Ordering: starting after things that must exist first
Real services rarely stand alone. Your app might need the database up, or a network share mounted, before it can do anything useful. systemd’s ordering directives handle this without sleep-loops or retry hacks. The distinction that trips people up is between wants and after:
Wants=postgresql.servicepulls the dependency in — enabling your unit will start Postgres too.After=postgresql.serviceorders the two — your unit starts only once Postgres has been started.
You almost always want both together, because ordering without a dependency does nothing if the dependency was never pulled in, and a dependency without ordering can race:
| |
One caveat worth internalising: After=postgresql.service means “after Postgres has been started”, not “after Postgres is ready to accept connections”. Daemons often report started before they are actually listening. For a robust setup, either use a socket-activated dependency or make your app retry its first connection with a short backoff rather than assuming the port is live the instant systemd hands over. This is the same lesson you learn wiring up any dependent service — I hit it again setting up self-hosted personal finance with Firefly III, where the app container started faster than its database could answer.
Hardening with a few free lines
systemd can sandbox your service with directives that would otherwise require considerable effort. Add these to [Service]:
| |
NoNewPrivileges=true stops the process gaining privileges via setuid binaries. ProtectSystem=strict makes the whole filesystem read-only except for a few permitted paths, and ReadWritePaths carves out the directories the app genuinely needs to write. ProtectHome=true hides /home, and PrivateTmp=true gives the service its own throwaway /tmp. Run systemd-analyze security widget.service to get a scored report of how locked-down your unit is.
Troubleshooting: when the unit won’t behave
Your first unit will misbehave. Everyone’s does. Here is how to diagnose the common failures rather than flailing.
The service won’t start and status is unhelpful. Read the journal, which carries the actual error your process printed:
| |
The -x flag adds systemd’s catalogue notes, which often name the exact cause (a missing binary, a permission denial, a bad WorkingDirectory).
ExecStart fails with “No such file or directory” for a command you know exists. systemd does not inherit your shell’s PATH, so python or node on their own won’t resolve. Use the absolute path (/usr/bin/python3), or if you genuinely need shell features like pipes or globbing, invoke a shell explicitly: ExecStart=/bin/sh -c 'exec /opt/widget/run.sh | tee -a /var/log/widget.log'.
Nothing changed after you edited the unit. You forgot sudo systemctl daemon-reload. systemd reads units into memory; your edits sit on disk unread until you reload. This is the single most common “systemd is broken” that isn’t.
The service starts, exits cleanly, and systemd marks it failed — or restarts it forever. This is a Type mismatch. Type=simple expects a long-running foreground process; if your program forks and the parent exits, systemd thinks it died. Use Type=forking for classic daemons, or better, run the program in the foreground (most modern daemons have a --foreground or --no-daemon flag). Conversely, Restart=always on a script that is meant to run once and stop will restart it in an infinite loop — use Restart=on-failure or a oneshot service.
Permission denied on a path the app clearly owns. Check your hardening directives before blaming file ownership. ProtectSystem=strict makes almost everything read-only; if the app writes to /opt/widget/data, that path must be listed in ReadWritePaths=. PrivateTmp=true gives the service its own /tmp, so anything else expecting files there won’t find them. Run systemctl show widget.service | grep -i protect to see what’s actually applied.
Secrets showing up where they shouldn’t. Values in Environment= lines are visible to anyone who can run systemctl show. Put credentials in an EnvironmentFile=/etc/widget.env owned by the service user with chmod 600, or use systemd’s LoadCredential= for the properly paranoid.
One more discipline: never edit vendor units under /lib/systemd/system (or /usr/lib/systemd/system) directly — a package update will clobber your changes. Put your own units and overrides under /etc/systemd/system, or use systemctl edit widget.service to create a drop-in that layers cleanly on top.
The verdict: is it worth it?
Emphatically yes, and the cost of entry is a dozen lines of text. A service unit is just a short INI file, and the handful of directives above cover the overwhelming majority of real-world needs. Once you have written one, the rest of systemd opens up: timers replace cron, the journal replaces log-file archaeology, and the sandboxing directives give you hardening that used to demand a great deal more effort.
Who is this for? Anyone running their own software on a Linux box they care about — a homelab service, a side project on a VPS, an internal tool at work. If your “deployment” is currently a tmux session and hope, this is the upgrade that pays for itself the first time the machine reboots and everything simply comes back. Write that first unit, run systemctl enable --now, and let Linux do the babysitting it was built for.
Related Content
