Linux Audit Framework: Tracking Who Did What on Your Servers

The moment a server does something you didn’t expect — a file changes, a binary appears, a user gains a privilege they shouldn’t have — the first question is always the same: who did this, and when? Ordinary logs rarely answer it. They tell you a service restarted, not that someone read /etc/shadow at 02:14. For the real answer you need the Linux Audit Framework, the kernel-level recorder that has been sitting on your box this whole time, almost certainly underused.

1 What auditd actually sees

The audit subsystem lives in the kernel. It taps into syscalls and security events directly, so it sees things no userspace logger can: a file being opened, a process changing its UID, a network socket binding. The userspace daemon auditd receives these events and writes them to /var/log/audit/audit.log. Because it operates below the application layer, it’s very hard for a normal process to lie to it — that’s the whole point.

$ sudo systemctl status auditd
● auditd.service - Security Auditing Service
     Active: active (running)
$ sudo auditctl -s
enabled 1
failure 1
pid 842
backlog 0
backlog_limit 8192

failure 1 means audit failures are logged but don’t panic the kernel; on a high-assurance box you might set failure 2 to halt instead. The backlog_limit matters under heavy load — too small and events get dropped silently, which defeats the purpose.

2 Writing rules that earn their keep

Out of the box auditd records almost nothing useful. You add rules. The two kinds that matter are watches (on files and directories) and syscall rules.

# /etc/audit/rules.d/hardening.rules

# watch the files that define who can do what
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k privilege
-w /etc/sudoers.d/ -p wa -k privilege

# flag every use of privilege-escalation tools
-w /usr/bin/sudo -p x -k priv_exec
-w /bin/su -p x -k priv_exec

# catch changes to the audit config itself (tamper detection)
-w /etc/audit/ -p wa -k auditconfig

# log all execve calls by real users (uid >= 1000), 64-bit
-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k userexec

# make the ruleset immutable until next reboot — must be last
-e 2

The -k tags are keys you’ll search on later, and they’re the difference between a usable audit setup and an unreadable wall of text. Load the rules and confirm:

$ sudo augenrules --load
$ sudo auditctl -l
-w /etc/shadow -p wa -k identity
-a always,exit -F arch=b64 -S execve -F auid>=1000 ...

That auid field — the audit or login UID — is the unsung hero. It’s stamped at login and follows the session even through sudo and su, so when root runs a command you can still trace it back to the human who logged in. That’s how you answer “who”, not just “what”.

3 Reading the logs without losing your mind

Raw audit records are deliberately machine-friendly and human-hostile. Don’t grep audit.log directly; use ausearch and aureport.

# everything tagged 'identity' in the last day
$ sudo ausearch -k identity --start yesterday -i
type=SYSCALL ... a0=... comm="vim" exe="/usr/bin/vim"
   auid=smarc uid=root ... key="identity"
type=PATH ... name="/etc/shadow" nametype=NORMAL

# a summary of authentication events
$ sudo aureport -au --summary
Authentication Report
total  acct
26     smarc
3      backup-svc

The -i flag interprets numeric IDs into names. ausearch -k privilege pulls every sudoers touch; aureport -x --summary ranks executables by frequency, which is a quick way to spot something running that shouldn’t be.

4 Shipping it somewhere safe

A log that lives only on the box being attacked is worth little — a competent intruder clears it. Forward audit events off-host. The clean way is audisp’s remote plugin or, more commonly these days, having your log shipper tail the file and stream it to a central store you trust.

# /etc/audit/plugins.d/au-remote.conf
active = yes
direction = out
path = /sbin/audisp-remote
type = always
format = string

Pair that with off-host retention and the immutable -e 2 flag above, and tampering becomes loud rather than silent.

5 The verdict

Is auditd worth setting up? For any server holding data you’d be unhappy to lose or leak — yes, and it’s free, already installed, and battle-tested. The honest caveat is that it is genuinely noisy and the rule syntax is archaic; a careless ruleset will bury you in records and tank performance on a busy host. Start narrow (identity files, privilege escalation, the audit config itself), ship the logs off-box, and widen only when you have a question that needs answering. This is for sysadmins who’ve ever had to reconstruct an incident after the fact — and having done that without auditd once, I will never run a serious box without it again.