Wireshark: Investigating TCP Traffic and Handshakes

Objective

Understand how TCP three-way handshakes, connection terminations, and transmission errors appear during packet captures, and learn to identify abnormal traffic patterns using Wireshark display filters.

Lab Setup

The capture was conducted within a local virtualized subnet (192.168.10.0/24):

• Client machine: Ubuntu 22.04 LTS (192.168.10.15)
• Target server: Debian 12 running Nginx and SSH (192.168.10.50)
• Analyzer: Wireshark 4.2 on the monitoring interface

Traffic Capture & Analysis

1. The Standard Three-Way Handshake

A normal HTTP request to http://192.168.10.50 produced standard connection establishment:

$ 192.168.10.15 -> 192.168.10.50 [SYN] Seq=0 Win=64240 Len=0 MSS=1460
$ 192.168.10.50 -> 192.168.10.15 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0
$ 192.168.10.15 -> 192.168.10.50 [ACK] Seq=1 Ack=1 Win=64240 Len=0

Key observations:

• Sequence numbers start relative to 0 in Wireshark for readability.
• The client advertises Maximum Segment Size (MSS) of 1460 bytes.
• Once ACK is received, data payload transmission starts immediately.

2. Identifying Closed Ports (RST, ACK)

When attempting to connect to an inactive port (port 8080), the host responds with RST, ACK to reset the connection attempt:

$ 192.168.10.15 -> 192.168.10.50 TCP 66 54322 -> 8080 [SYN]
$ 192.168.10.50 -> 192.168.10.15 TCP 54 8080 -> 54322 [RST, ACK] Seq=1 Ack=1

This confirms port 8080 is not listening and no firewall is silently dropping (dropping would result in timeout rather than an immediate RST).

Essential Wireshark Display Filters

The following filters were most effective during the analysis:

• tcp.flags.syn == 1 && tcp.flags.ack == 0 — Isolate initial connection requests.
• tcp.flags.reset == 1 — Find dropped or refused connections.
• tcp.analysis.retransmission — Identify packet loss and network congestion.
• ip.addr == 192.168.10.50 && tcp.port == 80 — Filter traffic for a specific service.

Key Takeaways

1. Wireshark's TCP stream following (Follow > TCP Stream) provides clear application-layer payloads, while packet-level inspection exposes timing and flag anomalies.
2. Silent packet drops vs active RST responses are the primary indicator of firewall drop policy vs inactive service.
3. Keeping relative sequence numbering enabled makes debugging handshakes significantly easier.