SOC & SIEM Log Monitoring Lab

Lab Overview

This lab documents the design and testing of a centralized security logging and detection environment. The goal is to simulate how a Security Operations Center (SOC) collects authentication logs from endpoints, indexes events in a SIEM, and configures alert logic to detect suspicious activity.

Status Note: This lab is currently in active progress. The log collection pipeline (Linux endpoints forwarding to Splunk via Universal Forwarder and Syslog) and authentication brute-force correlation searches are operational; Suricata network IDS sensor integration is in progress.

Architecture & Data Flow

[ Endpoint: Ubuntu Server ]               [ Endpoint: Debian Workstation ]
  - Auth logs (/var/log/auth.log)           - Systemd Journal & Audit logs
  - Splunk Universal Forwarder (UF)         - Rsyslog daemon
           |                                          |
           | Port 9997 (Encrypted Stream)             | Port 514 (Syslog UDP)
           v                                          v
+-------------------------------------------------------------------------+
|                  SIEM Indexer & Search Head (Splunk)                    |
|                                                                         |
|  +--------------------+   +---------------------+   +----------------+  |
|  | Inputs Collector   |-->| Index: `os_linux`   |-->| Correlation    |  |
|  | (9997 & 514)       |   | Parsing & Extraction|   | Search Engine  |  |
|  +--------------------+   +---------------------+   +-------+--------+  |
+-------------------------------------------------------------|-----------+
                                                              v
                                              +-------------------------------+
                                              | SOC Analyst Alert & Dashboard |
                                              | - Brute-force threshold alert |
                                              | - Triage checklist            |
                                              +-------------------------------+

Setup & Configuration

1. Central SIEM Deployment

Deployed a dedicated Splunk instance in a virtual container with dedicated log indexes:

• index=os_linux: For Linux authentication (/var/log/auth.log), sudo actions, and secure shell events.
• index=network: Reserved for firewall drop events and future IDS alert logs.

2. Universal Forwarder Deployment on Endpoints

Configured the Splunk Universal Forwarder on the Ubuntu target to monitor authentication logs without manual polling:

# /opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///var/log/auth.log]
disabled = false
index = os_linux
sourcetype = linux_secure

3. Correlation Rule: SSH Brute-Force Detection

Configured a scheduled search to trigger when more than 5 failed authentication attempts occur from a single IP address within a 5-minute rolling window:

index=os_linux sourcetype=linux_secure "Failed password"
| stats count by src_ip, user
| where count >= 5
| eval alert_level="Medium", event_desc="Potential SSH Brute-Force Attempt"

Simulated Attack & SOC Triage Workflow

To validate the alert pipeline, I generated controlled failed SSH requests from an external test IP (192.168.10.85):

$ index=os_linux sourcetype=linux_secure 'Failed password' | stats count by src_ip, user | sort -count
$ src_ip           user        count
$ 192.168.10.85    root        42
$ 192.168.10.85    admin       28
$ 192.168.10.85    guest       15
$ [ALERT TRIGGERED] Rule 'SSH_BruteForce_Threshold' triggered at 2024-09-12 16:32:00

SOC Analyst Triage Checklist Applied:

1. Validate Event Volume: Check whether attempts are isolated or distributed. 85 failed attempts within 3 minutes confirms an automated scan.
2. Account Status Verification: Verify whether any targeted usernames exist on the host. root exists; admin and guest do not.
3. Check for Success (Accepted): Run query index=os_linux sourcetype=linux_secure src_ip=192.168.10.85 "Accepted password". Result returned 0 matches, confirming no breach occurred.
4. Remediation Action: Add source IP to local firewall drop list and verify fail2ban jailed the source IP.

Real Challenges Encountered

Timestamp Mismatches in Distributed VMs

• Problem: Events forwarded from the endpoint appeared 10 minutes in the past on the SIEM timeline, confusing correlation searches.
• Cause: The endpoint VM's clock had drifted after being paused and resumed in the hypervisor.
• Fix: Installed and configured chrony on all endpoints and the SIEM host to synchronize clocks against local NTP servers before running event simulations.

What's Next for This Lab

1. Add Windows Event Log forwarding (WinEventLog:Security Event IDs 4624, 4625 for logon tracking).
2. Integrate Suricata IDS event logs (eve.json) into the network index to correlate network-level port scans with host-level log activity.