Reading the Tea Leaves: Hunting Intruders with journalctl and lnav

When you suspect something is wrong with a server — a sluggish response, an odd process, a vague unease — the temptation is to start poking at running state. But the running state is the present, and an intruder’s interesting work is usually in the past. The record of that past is sitting right there in your logs, already written, already timestamped. Logs are your first and cheapest forensic tool, and two utilities turn them from an overwhelming wall of text into a readable story: journalctl and lnav.

1 Logs as your first forensic tool

Before reaching for specialist forensic kits, look at what the system has been quietly recording all along: every authentication attempt, every sudo, every service start and stop, every kernel event. An attacker has to interact with the system, and most interactions leave a trace. The skill is not collecting more data — it is asking the right questions of the data you already have.

The catch is that logs are append-only history, which means they are also a target. A competent intruder will try to erase their tracks. So this article has two halves: how to read the logs, and how to make sure the logs survive someone who wants them gone.

2 journalctl essentials

On any systemd-based distribution, the journal is the central log store, and journalctl is how you query it. A handful of flags do most of the work.

• -u <unit> — filter to one service, e.g. journalctl -u ssh.service.
• --since / --until — bound a time window: --since "2026-06-01 22:00" --until "2026-06-02 02:00".
• -p <priority> — filter by severity, from emerg down to debug; -p warning shows warnings and worse.
• -f — follow live, like tail -f, for watching events as they happen.
• -k — kernel messages only (the dmesg equivalent), useful for spotting hardware tampering or odd module loads.
• -o <format> — change output; -o verbose shows every field, -o json feeds other tools, -o short-iso gives unambiguous timestamps.

Combine them freely. To see everything around the time you got suspicious:

journalctl --since "2026-06-02 00:00" --until "2026-06-02 01:30" -o short-iso

3 Finding failed SSH logins and sudo misuse

SSH is the front door, and brute-force or credential-stuffing attempts are the most common knock. Pull authentication events:

journalctl -u ssh.service --since "today" | grep -Ei "failed|invalid user"

A handful of failures is internet background noise. Hundreds from one address in a minute is a brute-force attempt. A successful login immediately following a flurry of failures — Accepted password after dozens of Failed password — is the pattern that should make your stomach drop, because it may mean the brute force worked.

sudo misuse is the other tell. Privilege escalation by an account that has no business escalating, or a barrage of authentication failure entries, is worth a hard look:

journalctl --since "today" | grep -Ei "sudo|authentication failure"
journalctl _COMM=sudo --since "yesterday"

That second form uses a journal field directly, which is more precise than grepping. Pay attention to which user ran what command, and whether it fits their normal behaviour.

4 What suspicious patterns look like

Beyond individual events, you are hunting for shapes in the data.

• Logins at odd hours from accounts that are usually nine-to-five.
• New or unexpected users appearing — useradd, usermod, or sudden additions to the sudo/wheel group.
• Source addresses from countries or networks you have no relationship with.
• Service restarts you didn’t trigger, especially of security tooling — an attacker stopping auditd or your log shipper is a giant red flag.
• Gaps in the timeline. Logs that suddenly go quiet during a window of activity may mean someone deleted entries, and absence of evidence becomes evidence.
• Cron or systemd timer changes, the classic persistence mechanism — check for new units and cron jobs that shouldn’t exist.

5 Enter lnav, the log navigator

journalctl is excellent for the journal, but real investigations span many files: web server access logs, application logs, firewall logs, plus the journal itself. Correlating across them by eye is miserable. lnav — the log navigator — is built for exactly this. Install it with sudo apt install lnav or sudo dnf install lnav, then point it at everything:

lnav /var/log/nginx/ /var/log/auth.log /var/log/syslog

lnav automatically detects common formats, parses timestamps, and merges every source into one unified, time-ordered stream. Now a request hitting your web server and the resulting application error and the firewall entry all line up chronologically, which is precisely how an incident actually unfolds. It colourises by severity, supports live tailing, and lets you jump to errors with a keypress.

To feed it the journal too, pipe it in:

journalctl -o json | lnav

6 SQL queries over your logs

lnav’s superpower is that it exposes the parsed log data as SQL tables. Press ; inside lnav and you can query your logs like a database. Want the top offending IP addresses in an access log?

SELECT c_ip, count(*) AS hits
FROM access_log
GROUP BY c_ip
ORDER BY hits DESC
LIMIT 20;

Or every request that returned a server error in a time window:

SELECT log_time, c_ip, cs_uri_stem, sc_status
FROM access_log
WHERE sc_status >= 500
ORDER BY log_time;

This turns “scroll through thousands of lines hoping to spot something” into “ask a precise question and get an answer”. Building a timeline is then trivial: filter to the suspect address or user across all merged sources, and lnav shows you, in order, every footprint they left.

7 Building a simple investigation

Put it together into a repeatable flow. Suppose failed-then-successful SSH logins tipped you off.

1. Identify the suspect address and account from the journal: journalctl -u ssh.service | grep -i accepted.
2. Establish the time window around the successful login.
3. Open everything in lnav and filter to that window and that source IP.
4. Trace what the session did: new processes, file changes, privilege escalation, outbound connections.
5. Check for persistence: new users, new SSH keys in ~/.ssh/authorized_keys, new cron jobs, new systemd units.
6. Write down the timeline — times, source, accounts, actions — because a clear timeline is what turns a hunch into a defensible conclusion.

The goal is a narrative: who got in, when, how, and what they touched.

8 Turning findings into action

Investigation is only useful if it changes something. Once you have a malicious source address, block it. fail2ban automates this going forward by watching your logs and banning addresses that trip its rules; its sshd jail catches exactly the brute-force pattern above:

sudo apt install fail2ban
sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd

For an immediate, manual block while you investigate, drop the address at the firewall:

sudo ufw deny from 203.0.113.45
# or, with nftables/iptables-based firewalls, an equivalent drop rule

Then close the door that let them in: rotate compromised credentials, remove rogue SSH keys, disable password authentication in favour of keys, and patch whatever they exploited.

9 Ship your logs off the box

Here is the part that separates an inconvenience from a disaster. If your logs live only on the machine an attacker controls, they can delete them — and a thorough intruder will. The defence is to ship logs off-box in real time, to a host the attacker does not control. Even if they wipe the local journal, the remote copy preserves the truth.

The simplest approach uses rsyslog or systemd-journald’s forwarding to send entries to a central log server as they are written. A minimal rsyslog forwarding line sends everything onward:

# /etc/rsyslog.d/90-forward.conf
*.* @@logserver.example.com:6514

The @@ requests TCP (use @ for UDP), and you should wrap it in TLS for anything crossing untrusted networks. This is the foundation of a proper SIEM pipeline, the subject of our companion article: forward everything to a place attackers cannot reach, and your logs keep telling the truth even when the original host has been compromised.

10 Conclusion

Your logs already recorded who knocked, when, and what they did next — the work is asking the right questions. journalctl filters the journal down to the failed logins, sudo misuse and odd-hours activity that betray an intruder, while lnav merges every log source into one timeline and lets you query it with SQL. Turn what you find into fail2ban rules and firewall blocks, close the entry point, and — above all — ship your logs off-box so an attacker can never quietly erase the evidence. The tea leaves are already there. Learn to read them.