How to Configure IPsec Transport Mode Rules in Windows 11 Using PowerShell

In high-security enterprise environments, securing network traffic between Windows endpoints, domain controllers, and internal application tiers often requires cryptographic protection directly at the IP layer. While Virtual Private Networks (VPNs) tunnel traffic across disparate subnets, IPsec Transport Mode encrypts and authenticates packet payloads end-to-end between two direct endpoints without encapsulating the original IP headers. Windows 11 provides comprehensive IPsec management through Windows Defender Firewall with Advanced Security and PowerShell cmdlets.

Understanding IPsec Transport Mode vs Tunnel Mode

IPsec operates in two distinct operational modes:

• Transport Mode: Encrypts or signs only the IP payload (the TCP or UDP segment and application data). The original IP packet header remains intact and visible, allowing standard routing across internal subnets without adding GRE or IP-in-IP headers. Ideal for host-to-host server isolation and client-to-server encryption.
• Tunnel Mode: Encapsulates the entire original IP packet (header and payload) inside a completely new outer IP header. Typically used for gateway-to-gateway site-to-site VPNs.

Checking Existing IPsec Connection Security Rules in PowerShell

Connection Security Rules (CSR) define how Windows 11 negotiates IPsec security associations (IKE/AuthIP):

1. Right-click the Start button on Windows 11 and select Terminal (Admin) or PowerShell (Admin).
2. To inspect all existing Connection Security Rules:

   Get-NetIPsecRule

3. To inspect active IPsec main mode and quick mode cryptographic associations:

   Get-NetIPsecMainModeCryptoSet
   Get-NetIPsecQuickModeCryptoSet

Creating an IPsec Quick Mode Cryptographic Suite

Before applying a transport rule, define the encryption algorithms used to protect data packets (Quick Mode):

New-NetIPsecQuickModeCryptoSet -DisplayName "Enterprise-AES-GCM" `
    -CryptoProposal "AESGCM256" `
    -Name "Enterprise-AES-GCM"

AES-GCM-256 provides authenticated hardware-accelerated encryption with minimal CPU overhead on modern processors.

Configuring an IPsec Transport Mode Connection Security Rule

To enforce IPsec Transport Mode encryption for all traffic destined for a sensitive database or application server (e.g., 192.168.10.50):

New-NetIPsecRule -DisplayName "Secure Transport to DB01" `
    -RuleId "Secure-Transport-DB01" `
    -LocalAddress Any `
    -RemoteAddress 192.168.10.50 `
    -Mode Transport `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -QuickModeCryptoSet "Enterprise-AES-GCM" `
    -Authentication Kerberos

In this rule:

• -Mode Transport: Explicitly specifies Transport Mode (not Tunnel).
• -InboundSecurity Require / -OutboundSecurity Require: Mandates that all incoming and outgoing packets must be cryptographically protected.
• -Authentication Kerberos: Uses active Active Directory domain credentials to authenticate endpoints seamlessly.

Auditing Active IPsec Security Associations (SAs)

After communicating with the remote server, verify that the IPsec Transport Mode association has been negotiated successfully:

Get-NetIPsecQuickModeSA

The output displays the local and remote endpoints, the active proposal (e.g., AES-GCM-256), and total protected bytes transmitted.

Testing and Troubleshooting IPsec Connectivity

If connections fail or time out:

1. Test raw port reachability:

   Test-NetConnection -ComputerName 192.168.10.50 -Port 443

2. Inspect Windows Defender Firewall operational logs for IKE negotiation failures:

   Get-WinEvent -LogName "Microsoft-Windows-Windows Firewall With Advanced Security/ConnectionSecurity" -MaxEvents 20

3. To temporarily soften the rule to allow unencrypted traffic if negotiation fails during rollout:

   Set-NetIPsecRule -DisplayName "Secure Transport to DB01" -InboundSecurity Request -OutboundSecurity Request

Enforcing IPsec Transport Mode via PowerShell provides robust, zero-trust host isolation across enterprise Windows 11 endpoints without requiring complex VPN overlays.