Goal
Investigate suspicious SSH connection activity against an exposed test VM in a private lab network, isolate the attacker's IP addresses, extract the usernames targeted, and determine the attack timeline using native Linux CLI tools.
Environment
- Target host: Ubuntu 22.04 LTS server (
192.168.10.20) with OpenSSH server running on default port 22. - Test attacking host: Kali Linux VM (
192.168.10.85) running a simulated dictionary scan with Hydra. - Log storage:
/var/log/auth.logand systemd journal (journalctl -u ssh).
Tools
journalctl— query and filter systemd logs.grep&egrep— extract matching failure patterns.awk&cut— parse structured log columns.sort&uniq— aggregate frequency counts.
Steps Taken
1. Checking SSH Service Status and Recent Failure Events
First, I checked the SSH daemon status and filtered the authentication journal for recent failed attempts:
$ sudo journalctl -u ssh --since '1 hour ago' | grep 'Failed password' | head -n 5$ Oct 24 10:14:02 server sshd[14201]: Failed password for invalid user admin from 192.168.10.85 port 49152 ssh2$ Oct 24 10:14:03 server sshd[14203]: Failed password for invalid user test from 192.168.10.85 port 49154 ssh2$ Oct 24 10:14:04 server sshd[14205]: Failed password for root from 192.168.10.85 port 49156 ssh2$ Oct 24 10:14:05 server sshd[14207]: Failed password for invalid user guest from 192.168.10.85 port 49158 ssh2$ Oct 24 10:14:06 server sshd[14209]: Failed password for chetan from 192.168.10.85 port 49160 ssh2
2. Identifying Attacker IP Addresses and Connection Volume
To calculate how many connection attempts were made and determine the source IPs:
$ grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr$ 482 192.168.10.85$ 3 192.168.10.12
The output clearly confirmed that 192.168.10.85 generated 482 failed authentication attempts in a short window.
3. Extracting Targeted Usernames
Attackers commonly probe default and high-privilege usernames. I parsed the usernames tested by the remote IP:
$ grep 'Failed password.*192.168.10.85' /var/log/auth.log | awk -F'for ' '{print $2}' | awk '{print $1}' | sed 's/invalid//' | sort | uniq -c | sort -nr | head -n 6$ 110 root$ 95 admin$ 72 test$ 54 user$ 40 guest$ 32 chetan
4. Verifying if Any Attempt Succeeded
A critical step in triage is verifying whether the attacker ever successfully authenticated:
$ grep 'Accepted' /var/log/auth.log | grep '192.168.10.85'$
The command returned 0 records, confirming that no successful logins occurred from 192.168.10.85.
What I Saw & What It Meant
- Rapid timestamp intervals: Attempts arrived at 1-second intervals from sequential source ports (
49152,49154,49156), which points directly to automated credential stuffing rather than manual typing. - Invalid user flag: Log lines containing
invalid userindicate the account does not exist in/etc/passwd. Lines withoutinvalid user(such asrootandchetan) mean the username exists on the system, indicating the attacker was testing passwords against valid local accounts.
Problem Encountered
Raw /var/log/auth.log format can vary slightly depending on whether the username exists or contains spaces. Standard awk '{print $9}' resulted in column misalignment between Failed password for invalid user X (6 tokens) versus Failed password for root (4 tokens).
Fix
I adjusted the awk pattern to scan for the keyword for and grab the token immediately following it, standardizing the extraction across both existing and non-existent usernames.
Preventive Controls Implemented
Following the investigation, I tested two immediate controls on the test server:
- Disable SSH password authentication: Switched to Ed25519 SSH key-based authentication only in
/etc/ssh/sshd_config.d/50-cloud-init.conf:PasswordAuthentication no PermitRootLogin no - Fail2ban jail: Configured a local
jail.localwithmaxretry = 5andbantime = 1hto automatically drop repeated failed connections viaiptables/nftables.
What I Learned
- Reading native auth logs directly with standard Unix text filters provides fast, reliable incident triage without needing a heavy SIEM for single-host troubleshooting.
- Distinction between
invalid userand existing usernames in SSH logs is crucial for assessing whether an attacker has successfully performed username enumeration. - Key-only SSH authentication removes the threat of password guessing entirely.