Linux Audit Framework: Tracking Who Did What on Your Servers

The moment a server does something you didn’t expect — a file changes, a binary appears, a user gains a privilege they shouldn’t have — the first question is always the same: who did this, and when? Ordinary logs rarely answer it. They tell you a service restarted, not that someone read /etc/shadow at 02:14. For the real answer you need the Linux Audit Framework, the kernel-level recorder that has been sitting on your box this whole time, almost certainly underused.

What auditd actually sees

The audit subsystem lives in the kernel. It taps into syscalls and security events directly, so it sees things no userspace logger can: a file being opened, a process changing its UID, a network socket binding. The userspace daemon auditd receives these events and writes them to /var/log/audit/audit.log. Because it operates below the application layer, it’s very hard for a normal process to lie to it — that’s the whole point.

1
2
3
4
5
6
7
8
9

$ sudo systemctl status auditd
● auditd.service - Security Auditing Service
     Active: active (running)
$ sudo auditctl -s
enabled 1
failure 1
pid 842
backlog 0
backlog_limit 8192

failure 1 means audit failures are logged but don’t panic the kernel; on a high-assurance box you might set failure 2 to halt instead. The backlog_limit matters under heavy load — too small and events get dropped silently, which defeats the purpose. A useful early tweak is to bump backlog_limit to something like 8192 or higher and set --backlog_wait_time so the kernel briefly pauses the offending process rather than silently discarding records; a dropped audit event is a blind spot you don’t know you have, which is the worst kind.

It’s worth being clear-eyed about what auditd is and isn’t. It is a recorder, not an alarm. Out of the box it will not tell you that something bad happened — it will faithfully write down everything you told it to watch, and leave the noticing to you or to whatever you ship the logs to. That distinction matters because the temptation, once you see how much it can capture, is to watch everything and then drown. The skill here is entirely in choosing what to record, and the rest of this piece is really about that discipline. Think of it as a flight recorder: you don’t read it every day, but when the plane does something strange you are extremely grateful it was running.

There’s also a lineage worth knowing, because it explains the archaic syntax you’re about to meet. The audit framework grew out of the requirements for government-grade security certifications, where you must be able to prove after the fact exactly who touched what. That heritage is why the rule language feels like it was designed by a committee optimising for completeness over ergonomics — because it was. You are borrowing an enterprise compliance tool for a home server, and it shows, but the capability underneath is genuinely first-rate and it’s already installed.

Writing rules that earn their keep

Out of the box auditd records almost nothing useful. You add rules. The two kinds that matter are watches (on files and directories) and syscall rules.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# /etc/audit/rules.d/hardening.rules

# watch the files that define who can do what
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k privilege
-w /etc/sudoers.d/ -p wa -k privilege

# flag every use of privilege-escalation tools
-w /usr/bin/sudo -p x -k priv_exec
-w /bin/su -p x -k priv_exec

# catch changes to the audit config itself (tamper detection)
-w /etc/audit/ -p wa -k auditconfig

# log all execve calls by real users (uid >= 1000), 64-bit
-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k userexec

# make the ruleset immutable until next reboot — must be last
-e 2

The -k tags are keys you’ll search on later, and they’re the difference between a usable audit setup and an unreadable wall of text. Load the rules and confirm:

1
2
3
4

$ sudo augenrules --load
$ sudo auditctl -l
-w /etc/shadow -p wa -k identity
-a always,exit -F arch=b64 -S execve -F auid>=1000 ...

That auid field — the audit or login UID — is the unsung hero. It’s stamped at login and follows the session even through sudo and su, so when root runs a command you can still trace it back to the human who logged in. That’s how you answer “who”, not just “what”. The value 4294967295 (that’s -1 as an unsigned 32-bit integer) means “no login UID set” — typically a daemon or a boot-time process that never went through a login — which is why the execve rule above explicitly excludes it. You care about humans; the init system running its usual business is noise.

A word on the immutable flag, because it’s the part people either skip or misuse. The -e 2 line locks the ruleset so it can’t be changed until the next reboot — not even by root. That’s the point: an intruder who gains root can’t quietly switch off the very auditing that would catch them without rebooting the box, and a reboot is itself a loud, logged, noticeable event. The catch is that it also locks you out until reboot, so leave -e 2 off while you’re still iterating on your rules and only add it once the ruleset has settled. I’ve locked myself into a bad ruleset on a remote box more than once; learn from my irritation.

A worked example: catching the thing you didn’t expect

Rules are abstract until you see one earn its keep, so here’s the shape of a real investigation. Suppose you notice an unfamiliar binary in /usr/local/bin and want to know how it got there. With the execve rule above already running, you ask ausearch when and by whom:

1
2
3
4

$ sudo ausearch -k userexec -i | grep -B2 cp
type=SYSCALL msg=audit(...): ... comm="cp" exe="/usr/bin/cp"
   success=yes auid=smarc uid=smarc
type=PATH ... name="/usr/local/bin/suspicious" nametype=CREATE

There’s your answer: the human smarc, logged in as themselves, copied the file into place — no compromise, just me being careless. Contrast that with an auid of 4294967295 or an execve chain that traces back to a web-server process, and you’d be looking at a genuine incident. The framework doesn’t interpret any of this for you; it just makes the interpretation possible, which is more than ordinary logs ever managed.

Cutting the noise before it starts

The single biggest reason people abandon auditd is that the default advice — “watch execve, watch the identity files” — generates more than they expected the moment a busy service starts up. Rather than tightening rules after the disk fills, it’s worth adding exclusions from the beginning for the processes you already know are noisy and uninteresting. auditctl supports an exclude filter type for exactly this:

1
2
3
4
5
6
7
8
9

# /etc/audit/rules.d/exclude-noise.rules

# don't bother recording routine cron wake-ups and systemd housekeeping
-a exclude,always -F msgtype=CWD
-a exclude,always -F subj_type=cron_t

# a chatty backup agent that touches thousands of files a night,
# already covered by its own application log
-a exclude,always -F exe=/usr/local/bin/backup-agent

CWD records are a common first cut: they capture the current working directory for every syscall event and add bulk without adding much forensic value on a server where working directories are predictable. The exe= exclusion is the more targeted move — if you already trust and separately log a specific, well-understood process, excluding it by path keeps the audit log focused on the events you’d actually stop and read. Be conservative here: exclude the processes you understand deeply, never a whole user or a whole directory tree, because an overbroad exclusion is a blind spot you built yourself, which is worse than one you inherited.

It’s also worth knowing what auditd is not competing with. Newer eBPF-based tools such as Falco watch similar territory — process execution, file access, network activity — but do it from userspace-friendly probes designed for real-time alerting rather than a compliance-grade audit trail, and they tend to be lighter on a busy host. The two aren’t really substitutes: Falco is closer to an alarm, auditd closer to the flight recorder mentioned earlier. Plenty of hardened setups run both, using Falco to page someone immediately and auditd to provide the immutable, government-grade record that survives to explain exactly what happened afterwards.

Reading the logs without losing your mind

Raw audit records are deliberately machine-friendly and human-hostile. Don’t grep audit.log directly; use ausearch and aureport.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# everything tagged 'identity' in the last day
$ sudo ausearch -k identity --start yesterday -i
type=SYSCALL ... a0=... comm="vim" exe="/usr/bin/vim"
   auid=smarc uid=root ... key="identity"
type=PATH ... name="/etc/shadow" nametype=NORMAL

# a summary of authentication events
$ sudo aureport -au --summary
Authentication Report
total  acct
26     smarc
3      backup-svc

The -i flag interprets numeric IDs into names. ausearch -k privilege pulls every sudoers touch; aureport -x --summary ranks executables by frequency, which is a quick way to spot something running that shouldn’t be.

Shipping it somewhere safe

A log that lives only on the box being attacked is worth little — a competent intruder clears it. Forward audit events off-host. The clean way is audisp’s remote plugin or, more commonly these days, having your log shipper tail the file and stream it to a central store you trust.

1
2
3
4
5
6

# /etc/audit/plugins.d/au-remote.conf
active = yes
direction = out
path = /sbin/audisp-remote
type = always
format = string

Pair that with off-host retention and the immutable -e 2 flag above, and tampering becomes loud rather than silent.

This matters more than it might seem, because the whole value proposition of auditd collapses if the attacker can edit the record. Local-only audit logs are security theatre against anyone competent: the first thing a serious intruder does is clean up after themselves. The point of shipping events off-box as they happen is that by the time the attacker thinks to cover their tracks, the evidence has already left the building. Where you send it is up to you — a syslog server, a managed log store, a full SIEM — but the direction of travel is what counts.

Troubleshooting: the ways this goes wrong

Auditd has a small set of failure modes that catch nearly everyone the first time.

Rules don’t survive a reboot. If you added rules with auditctl on the command line, they’re live but ephemeral — they vanish on restart. Persistent rules must live in a file under /etc/audit/rules.d/ and be loaded with augenrules --load. If your carefully crafted watches evaporate overnight, this is almost always why.

“Rule exists” or the ruleset won’t load. Once -e 2 is in effect, any attempt to change rules fails until reboot — by design. If augenrules --load complains and nothing updates, check whether you’ve already locked the config. Comment out the -e 2 line, reboot, iterate, and re-enable it only when you’re done.

The disk fills and the box behaves strangely. A too-broad ruleset — watching every open syscall on a busy server, say — can generate gigabytes of records a day and, depending on your disk_full_action, either fill /var or start halting processes. Tune max_log_file, num_logs, and the rotation settings in /etc/audit/auditd.conf, and resist the urge to watch everything. Narrow rules are not just tidier; they’re the difference between a useful tool and a self-inflicted outage.

Performance tanks under load. Every audited syscall carries a small cost, and on a hot path that adds up. If a database or web server slows after you enable auditing, your rules are almost certainly too broad. This is the same watch-the-important-thing discipline that runs through all of Linux hardening — the kind of unglamorous, keep-the-attacker-honest work I wrote about in the context of what kernel exploits keep teaching us. Auditing won’t stop a privilege-escalation bug, but it’s how you find out one was used.

The verdict

Is auditd worth setting up? For any server holding data you’d be unhappy to lose or leak — yes, and it’s free, already installed, and battle-tested. The threshold I use is simple: if the box holds something I’d have to explain the loss of — the family photo archive, the documents in a self-hosted Nextcloud, anything with other people’s data on it — it gets auditd. If it’s a throwaway box I could rebuild in ten minutes and lose nothing, I don’t bother. The honest caveat is that it is genuinely noisy and the rule syntax is archaic; a careless ruleset will bury you in records and tank performance on a busy host. Start narrow (identity files, privilege escalation, the audit config itself), ship the logs off-box, and widen only when you have a question that needs answering. This is for sysadmins who’ve ever had to reconstruct an incident after the fact — and having done that without auditd once, I will never run a serious box without it again.