Auditd: Watching Who Did What on Your Boxes

There is a particular flavour of dread that comes from finding a file changed and having no idea who changed it. A config that used to work now doesn’t. A user account exists that you don’t remember creating. A binary’s timestamp is newer than the last time you touched the box. Ordinary logs get you partway — auth.log shows the login, the shell history shows some commands — but shell history is trivially wiped, and neither tells you that a process opened a file at a precise instant as a precise user. For that you need the kernel itself to be the witness, and on Linux the kernel’s witness is the audit subsystem, driven by auditd.

Auditd is one of those tools that sits unloved in the corner because its output looks like line noise and its rule syntax looks worse. That reputation is unfair. Underneath the intimidating surface is a genuinely powerful capability: the kernel records security-relevant events as they happen, tamper-resistantly, with the real user identity attached even after a sudo. I want to demystify it — why you would run it, how to write rules that answer real questions, and how to read the output without your eyes glazing over.

Why kernel-level auditing beats the alternatives

The reason to reach past ordinary logging is attribution that survives an attacker. Application logs are written by the application, which means anything running as that application can forge or delete them. Shell history is a plain file in the user’s home directory; the first thing a competent intruder does is unset HISTFILE or wipe it. These sources are useful and also fundamentally untrustworthy once someone hostile is on the box.

The audit subsystem lives in the kernel. Events are generated in kernel space and handed to auditd before any userland process gets a say, and the logs land in /var/log/audit/ which you can lock down so that even root leaves a trace when tampering with them. Crucially, auditd records the auid — the audit (login) user ID — which is stamped at login and does not change when the user runs sudo or su. Ordinary logs show you that “root” ran a command; auditd shows you that the person who logged in as alice ran that command as root. That distinction is the entire difference between a useless log and an accountable one.

The honest trade is that auditd tells you what happened with great precision and tells you nothing about whether it was bad. It is a flight recorder. It does not prevent anything, it does not alert on its own, and left to its default rules it records a firehose of noise. The skill is writing a small set of rules that watch the things you actually care about, so that when something goes wrong you have the recording, and the rest of the time you are not drowning.

Rules that answer real questions

Auditd rules come in two shapes. Watches monitor a file or directory for access. Syscall rules trigger on specific system calls, optionally filtered by user, architecture, or arguments. You load them with auditctl for testing and persist them in /etc/audit/rules.d/*.rules for boot. Here is a starter set with the intent spelled out, which matters more than the syntax:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# /etc/audit/rules.d/homelab.rules

# --- watch sensitive files for any change ---
-w /etc/passwd    -p wa -k identity
-w /etc/shadow    -p wa -k identity
-w /etc/sudoers   -p wa -k privilege
-w /etc/sudoers.d -p wa -k privilege
-w /etc/ssh/sshd_config -p wa -k sshd

# --- record every use of sudo/su ---
-w /usr/bin/sudo -p x -k priv_esc
-w /bin/su       -p x -k priv_esc

# --- catch new/changed cron jobs (a classic persistence trick) ---
-w /etc/crontab   -p wa -k cron
-w /etc/cron.d    -p wa -k cron

# --- log commands run as root by real people (auid) ---
-a always,exit -F arch=b64 -F euid=0 -F auid>=1000 -F auid!=unset -S execve -k root_cmd

# --- lock the config so rule changes require a reboot to take effect ---
-e 2

Read that from the top. The -w lines say “watch this path; the -p wa means alert on writes and attribute changes; tag it with a key I can search on later”. Watching /etc/passwd, /etc/shadow and /etc/sudoers covers the classic account-and-privilege tampering. Watching sshd_config catches someone loosening your SSH hardening. The cron watches catch one of the most common persistence mechanisms going, because a scheduled job is how malware survives a reboot.

The -a always,exit line is the interesting one. It says: on 64-bit systems, whenever a process with effective UID 0 (root) calls execve (runs a program), and the login user is a real human (auid>=1000) rather than a system account, log it. That single rule gives you a record of every command run as root by an actual person, attributed to who they logged in as. The -k keys throughout are how you find these events later without wading through everything.

That final -e 2 makes the configuration immutable until reboot. Once loaded, nobody — not even root — can change or delete audit rules without rebooting the machine, and the reboot itself is logged. On a box you care about, that is worth the minor inconvenience, because it means an intruder can’t quietly disable the recording that would expose them.

Reload rules with augenrules --load (which assembles /etc/audit/rules.d/*.rules into the running config) and confirm with auditctl -l.

Reading the output without despair

A raw audit record is genuinely horrible to look at:

1
2
3

type=SYSCALL msg=audit(1723209841.331:2094): arch=c000003e syscall=59
success=yes exit=0 a0=55d... ppid=1421 pid=1533 auid=1000 uid=0 gid=0
euid=0 ... comm="useradd" exe="/usr/sbin/useradd" key="root_cmd"

Nobody reads that directly, and you are not meant to. The tool for querying is ausearch, and the tool for summarising is aureport. This is where the -k keys pay off. Want every privileged command run today?

1
2
3
4
5

$ sudo ausearch -k root_cmd --start today -i
----
type=SYSCALL msg=audit(...) : arch=x86_64 syscall=execve success=yes
  exe=/usr/sbin/useradd comm=useradd
  auid=alice uid=root ...

The -i flag is the one that makes auditd usable: it interprets the raw numbers, turning auid=1000 into auid=alice, syscall=59 into syscall=execve, and the epoch timestamp into a readable date. Always pass -i. Now that record reads in plain English: alice, logged in as herself, ran useradd as root today. That is the sentence you came for.

aureport gives the bird’s-eye view — aureport -au for authentication events, aureport -x for executables that triggered rules, aureport -f for file access. When you are trying to reconstruct an incident, start with an aureport summary to spot the anomaly, then drill in with ausearch -k <key> on the relevant tag.

This is the layer that turns a suspicion into a timeline, and it complements the edge defences rather than replacing them. A CrowdSec or Fail2ban ban stops the brute-force at the door; auditd tells you what happened in the rare case something got through the door. The two answer different questions, and a serious setup wants both.

Starting from a baseline, then pruning

Writing every rule from scratch is a mistake that leads either to gaps or to exhaustion. There are well-regarded published rule sets to start from — the audit rules shipped with the CIS and STIG hardening baselines, and community sets that map rules to the MITRE ATT&CK techniques they detect. These are a much better starting point than a blank file, because someone has already thought about the syscalls that matter for privilege escalation, persistence, and log tampering.

The catch is that these baselines are written for busy multi-user servers under a compliance regime, and dropped onto a quiet homelab box they generate a torrent of events you will never read. The right move is to load a baseline, run the box under a normal day of use, and then prune — use aureport to see which keys are firing constantly, and either tighten those rules with auid and path filters or drop them. What you want at the end is a rule set small enough that every event it produces is one you would actually want to know about. A hundred noisy rules you ignore protect nothing; fifteen sharp ones you would investigate protect a great deal.

The same instinct applies to where the rules point. A homelab does not need to audit every syscall the compliance frameworks care about; it needs to watch the handful of things that would matter if this specific box were compromised. Identity files, privilege escalation, cron and systemd persistence paths, your SSH config, and real-human root commands cover the vast majority of what an intruder does to establish a foothold. Start there, add a rule only when you can name the question it answers, and your audit log stays readable.

Troubleshooting

The logs fill the disk. Overly broad rules — watching a busy directory, or logging every execve without the auid filter — generate gigabytes fast. Tune /etc/audit/auditd.conf: max_log_file (megabytes per file), num_logs (how many to rotate), and max_log_file_action = ROTATE. If auditing genuinely matters on the box, set space_left_action and admin_space_left_action thoughtfully, because the paranoid default of halting the system on a full audit disk will absolutely take your service down.

auditctl says “Operation not permitted”. You loaded the immutable flag -e 2 earlier in the session and now can’t change rules. That is the flag working. Reboot to load a new rule set. This trips everyone once.

Rules vanish after reboot. You loaded them with auditctl (runtime only) and never wrote them to /etc/audit/rules.d/. Runtime rules die on reboot by design. Put them in a .rules file and use augenrules --load.

No auid, or auid=unset (4294967295). The process wasn’t started through a PAM session that stamps the login UID — common for daemons and some container workloads. There is nothing to fix on the auditd side; it faithfully reports that there was no login user. It is a reason to be cautious about attributing container-internal actions, which have their own identity story.

Auditd and your log shipper fight. If you forward audit logs off the box (a very good idea, since local logs an attacker can reach are only half-trustworthy), use the audisp/audit-plugins dispatcher rather than tailing the file, so you don’t miss records during rotation. Shipping audit logs to a separate host means the tamper-resistance survives even a full compromise of the audited machine.

Is auditd worth running?

For a machine that matters — one holding data you would grieve, or one exposed enough to be a real target — yes, with a caveat about restraint. Auditd’s value is entirely proportional to the discipline of your rule set. A default install with the stock rules produces so much noise that you will ignore it, which is worse than not running it at all because it gives false comfort. A tight set of a dozen rules watching your identity files, your privilege escalation, your cron persistence, and your real-human root commands gives you a tamper-resistant record that answers the exact questions an incident raises, at negligible cost.

It is the last layer, and it only earns its place once the earlier ones are sound. Get your threat model straight so you know which boxes deserve this attention, harden the front door, and keep tested backups. Then add auditd to the machines that matter, ship its logs somewhere the attacker can’t reach, and sleep a little better knowing that if something ever does slip past everything else, the kernel was watching, and it wrote down a name, a time, and a command.