How to Configure Linux TCP SYN Cookies to Prevent SYN Floods

The Mechanics of a SYN Flood

The Transmission Control Protocol (TCP) requires a three-way handshake to establish a reliable connection between a client and a server.

1. The client sends a SYN (Synchronize) packet.
2. The Linux server allocates memory for the connection, places it in a “half-open” queue, and replies with a SYN-ACK packet.
3. The client sends a final ACK packet, and the connection is fully established.

A SYN Flood is a devastating Denial of Service (DoS) attack that exploits this architecture. An attacker sends thousands of spoofed SYN packets per second. The Linux server faithfully allocates memory for every single one and replies with a SYN-ACK. However, the attacker never sends the final ACK. Within seconds, the Linux server’s half-open queue fills up completely. The kernel is now mathematically unable to accept any new, legitimate connections, and the web server effectively drops off the internet.

To protect your Linux server from this attack without requiring expensive hardware firewalls, you must enable TCP SYN Cookies in the kernel using sysctl.

How SYN Cookies Defeat the Attack

When SYN Cookies are enabled, the Linux kernel fundamentally changes how it handles the three-way handshake during a flood.

If the half-open queue is full and a new SYN packet arrives, the kernel refuses to allocate any memory. Instead, it cryptographically hashes the client’s IP, port, and timestamp, encodes that hash into the sequence number of the SYN-ACK packet, and sends it back to the client. It then completely forgets about the connection.

If the client is legitimate, it will reply with an ACK containing that specific sequence number. The Linux kernel will decode the hash, verify its authenticity, and instantly allocate the memory to establish the connection. The attacker’s spoofed packets are ignored, and legitimate traffic continues to flow effortlessly.

Enabling SYN Cookies via sysctl

Modern Linux distributions usually have SYN cookies compiled into the kernel, but they are not always aggressively enabled or tuned.

You can check the current status using:

sysctl -a | grep syncookies

If net.ipv4.tcp_syncookies = 0, you are highly vulnerable.

To enable the protection immediately in RAM (without rebooting), run:

sudo sysctl -w net.ipv4.tcp_syncookies=1

Tuning the Backlog Queue

SYN Cookies only activate when the half-open queue is completely full. If the queue is too small, SYN Cookies trigger constantly, which requires CPU overhead for the cryptographic hashing. You should increase the queue size so the server can handle normal traffic bursts without resorting to cookies.

# Increase the maximum number of half-open connections
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=4096

# Increase the core network device backlog (how many packets can queue before the CPU processes them)
sudo sysctl -w net.core.netdev_max_backlog=4096

Making the Protection Permanent

Changes made via the -w flag are lost if the server reboots. To permanently harden the kernel against SYN floods, you must append these settings to the system configuration file.

Open /etc/sysctl.conf using a text editor with root privileges:

sudo nano /etc/sysctl.conf

Add the following directives to the bottom of the file:

# TCP SYN Flood Protection
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 4096

Save and close the file. To force the kernel to read the file immediately, execute:

sudo sysctl -p

Your Linux server is now immune to basic TCP exhaustion attacks. It will aggressively allocate memory for normal traffic, but instantly switch to the cryptographic, zero-memory SYN Cookie defense mechanism the millisecond a flood is detected.