Architectural Foundation of VRRP and High-Availability Clustering
Enterprise infrastructure services—such as reverse proxies, API gateways, database clusters, and DNS caches—must maintain continuous availability without presenting single points of failure (SPOF). Keepalived is a routing and system daemon designed for Linux environments that implements the Virtual Router Redundancy Protocol (VRRP, RFC 5798). It provides active-passive or active-active failover capabilities by binding a shared Virtual IP (Floating IP) dynamically across multiple physical or virtual server nodes.
Under the VRRP protocol architecture, participating cluster nodes belong to a common Virtual Router Identifier (virtual_router_id). One node operates in the MASTER state with the highest operational priority, binding the floating virtual IP to its local network interface and periodically broadcasting VRRP advertisement packets (IP protocol 112 to multicast address 224.0.0.18). The remaining nodes listen in the BACKUP state. If the MASTER fails to dispatch heartbeats within a configured advertisement interval (typically three missed intervals), the highest-priority BACKUP automatically assumes the MASTER role, attaches the VIP, and dispatches Gratuitous ARP (GARP) frames across the local network to refresh upstream switch Content Addressable Memory (CAM) tables instantaneously.
| Cluster State | Network Interface Action | VRRP Advertisement Role | Gratuitous ARP Behaviour | Traffic Handling |
|---|---|---|---|---|
| MASTER | Virtual Floating IP bound (aliased) | Transmits heartbeats (Interval: 1s) | Broadcasts GARP on state entry | Receives and processes 100% of incoming VIP traffic |
| BACKUP | Virtual Floating IP dormant (detached) | Listens for MASTER heartbeats | Silent | Idle standby; ready to preempt upon timeout |
| FAULT | Virtual Floating IP dropped | Ceases all VRRP transmission | Silent | Node isolated following healthcheck script failure |
Configuring Kernel Prerequisites on Ubuntu Server
Before installing Keepalived, the Linux kernel must be instructed to permit local services (such as Nginx or HAProxy) to bind to an IP address that is not yet locally assigned to the physical interface. This avoids service initialisation crashes on the BACKUP node prior to failover.
# Append non-local IP binding and IP forwarding to sysctl configuration
sudo tee /etc/sysctl.d/60-keepalived.conf << 'EOF'
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
EOF
# Apply the kernel sysctl parameters immediately
sudo sysctl --system
# Verify parameters are active
sysctl net.ipv4.ip_nonlocal_bind
Installing and Securing Keepalived
Deploy Keepalived across both target Ubuntu Server nodes (Node 1: 192.168.1.11, Node 2: 192.168.1.12, Floating VIP: 192.168.1.50):
# Update APT cache and install keepalived
sudo apt update && sudo apt install -y keepalived
# Verify installed binary version
keepalived --version
Opening Firewall Ports for VRRP Multicast Traffic
If Uncomplicated Firewall (UFW) is active, you must permit IP protocol 112 (vrrp) communication between cluster peers to prevent catastrophic split-brain scenarios:
# On Node 1 (192.168.1.11): Allow VRRP traffic from Node 2 (192.168.1.12)
sudo ufw allow from 192.168.1.12 to any proto vrrp
# On Node 2 (192.168.1.12): Allow VRRP traffic from Node 1 (192.168.1.11)
sudo ufw allow from 192.168.1.11 to any proto vrrp
# Reload firewall
sudo ufw reload
Configuring Master and Backup Keepalived Nodes
The core configuration file resides at /etc/keepalived/keepalived.conf. Replace eth0 with your primary network interface name (identify via ip link).
Step 1: Configuration for Node 1 (Primary MASTER)
sudo tee /etc/keepalived/keepalived.conf << 'EOF'
global_defs {
router_id lb-node-01
enable_script_security
script_user root
}
# Healthcheck script tracking frontend proxy service availability
vrrp_script check_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
fall 2
rise 2
}
vrrp_instance VI_STATIC {
state MASTER
interface eth0
virtual_router_id 51
priority 101
advert_int 1
authentication {
auth_type PASS
auth_pass Secr3tClust3rP@ss
}
virtual_ipaddress {
192.168.1.50/24 dev eth0 label eth0:vip
}
track_script {
check_haproxy
}
}
EOF
Step 2: Configuration for Node 2 (Secondary BACKUP)
sudo tee /etc/keepalived/keepalived.conf << 'EOF'
global_defs {
router_id lb-node-02
enable_script_security
script_user root
}
vrrp_script check_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
fall 2
rise 2
}
vrrp_instance VI_STATIC {
state BACKUP
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass Secr3tClust3rP@ss
}
virtual_ipaddress {
192.168.1.50/24 dev eth0 label eth0:vip
}
track_script {
check_haproxy
}
}
EOF
Starting Keepalived and Testing Failover
Start and enable the daemon on both servers:
# Start Keepalived on Node 1 then Node 2
sudo systemctl enable --now keepalived
# Inspect local interface on Node 1 to verify floating VIP attachment
ip addr show dev eth0
You should see 192.168.1.50/24 listed alongside the physical IP address on Node 1. On Node 2, the floating IP will remain absent while Node 1 is healthy.
Simulating Failover
To simulate an outage, stop Keepalived or terminate the tracked service on Node 1:
# On Node 1: Stop Keepalived
sudo systemctl stop keepalived
# On Node 2: Check IP address immediately
ip addr show dev eth0
Node 2 immediately binds 192.168.1.50 and emits Gratuitous ARP broadcasts. Inspect logs via journalctl -u keepalived -n 50 --no-pager to observe the transition: Entering MASTER STATE.
Troubleshooting Keepalived and VRRP Failures
| Observed Anomaly | Underlying Root Cause | Technical Verification | Remediation Protocol |
|---|---|---|---|
| Split-Brain (Both nodes claim VIP simultaneously) | Firewall blocking IP protocol 112 (VRRP multicast) or mismatched virtual_router_id / auth_pass |
Inspect ip addr: both nodes display VIP; run tcpdump -i eth0 proto 112 |
Ensure UFW permits VRRP packets between node IPs; verify identical passwords and IDs |
| Healthcheck Script Fails with “Permission Denied” | Keepalived script security blocks unprivileged execution or root script ownership rules violated | Logs show: Script check_haproxy failed to execute: Exit status 1 |
Add enable_script_security and script_user root inside global_defs |
| Stale ARP on Gateway Router after Failover | Upstream router/switch drops or ignores Gratuitous ARP frames | Clients cannot ping VIP for minutes until ARP cache expires | Configure garp_master_refresh 5 and garp_master_repeat 3 in keepalived.conf |
| VIP Flapping / Rapid State Oscillations | Weight adjustment causes priority inversion, or tracked script latency exceeds interval | Logs show alternating Entering MASTER STATE and Entering BACKUP STATE |
Increase script interval (e.g., 3s); add nopreempt to prevent thrashing when master recovers |