How to Configure IPsec Connection Security Rules in Windows 11 Using PowerShell

In zero-trust enterprise network architectures, perimeter firewalls alone cannot protect against lateral movement once an attacker compromises an endpoint on the internal local area network (LAN). Securing east-west traffic between workstations, domain controllers, and database servers requires network-layer cryptographic verification. The Windows operating system includes native Internet Protocol Security (IPsec) support integrated directly into Windows Defender Firewall with Advanced Security.

IPsec Connection Security Rules authenticate communicating endpoints before data transfer begins and optionally encrypt network payloads using standard cryptographic suites (such as AES-GCM or AES-CBC with SHA-256). In Windows 11 and Windows Server, administrators can configure, manage, and audit these rules programmatically using the NetSecurity PowerShell module. Leveraging cmdlets such as New-NetIPsecRule, Get-NetIPsecRule, and New-NetIPsecMainModeCryptoProposal enables scalable deployment of domain isolation and server-to-server encrypted transport without relying on graphical MMC snap-ins.

Understanding IPsec Modes and Authentication in Windows

Windows IPsec operates through two distinct cryptographic phases:

• Main Mode (IKE / AuthIP): Establishes a secure management channel between the two communicating computers. The endpoints exchange identity certificates, Kerberos tickets, or pre-shared keys (PSKs) and negotiate Diffie-Hellman cryptographic parameters to create a Master Key.
• Quick Mode (Transport Mode): Negotiates the security parameters for the actual user traffic. It specifies whether packets require Encapsulating Security Payload (ESP) encryption, authentication headers (AH), or both, before encapsulating the TCP/UDP data.

Connection Security Rules define how hosts communicate securely, whereas standard Firewall Rules specify what traffic (ports/protocols) is permitted or blocked.

Prerequisites for PowerShell IPsec Configuration

To configure IPsec rules, open PowerShell with elevated administrative privileges. Ensure that the IKEEXT (IKE and AuthIP IPsec Keying Modules) service and PolicyAgent (IPsec Policy Agent) service are running:

Get-Service -Name IKEEXT, PolicyAgent | Set-Service -StartupType Automatic -PassThru | Start-Service

Creating a Pre-Shared Key Authentication Proposal

While Active Directory environments typically rely on Kerberos v5 or Computer Certificates for Main Mode authentication, test environments or isolated server pairs often use Pre-Shared Keys (PSKs).

Create a Main Mode authentication proposal using New-NetIPsecAuthProposal:

$authProposal = New-NetIPsecAuthProposal -Machine -PresharedKey "ContosoSecureKey2026!"

To enforce domain Kerberos authentication in an Active Directory environment instead:

$authProposal = New-NetIPsecAuthProposal -Machine -Kerberos

Configuring an IPsec Transport Mode Rule with Encryption

To enforce end-to-end encryption for all traffic exchanged between your Windows 11 workstation (e.g., 192.168.1.50) and a secure database server (e.g., 192.168.1.100):

New-NetIPsecRule -DisplayName "Secure-DB-Transport" `
    -Description "Requires ESP AES-256 encryption between workstation and database" `
    -LocalAddress "192.168.1.50" `
    -RemoteAddress "192.168.1.100" `
    -Protocol TCP `
    -RemotePort 1433 `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Phase1AuthSet $authProposal `
    -Encryption AES256

Parameters explained:

• InboundSecurity / OutboundSecurity: Setting both to Require ensures that packets without valid IPsec security associations (SAs) are dropped immediately. Setting them to Request allows fallback to cleartext if the remote peer does not support IPsec.
• Encryption: Enforces AES-256 encryption for the packet payload.

Configuring Subnet-Wide Domain Isolation Rules

To implement domain isolation across an entire administrative subnet (10.10.0.0/16) requesting authentication for all protocols:

New-NetIPsecRule -DisplayName "Corporate-Subnet-Isolation" `
    -RemoteAddress "10.10.0.0/16" `
    -InboundSecurity Request `
    -OutboundSecurity Request `
    -Phase1AuthSet (New-NetIPsecAuthProposal -Machine -Kerberos)

This allows non-domain systems to communicate in cleartext while automatically negotiating secure, authenticated channels between domain-joined systems.

Auditing and Inspecting Active IPsec Rules

To review all configured Connection Security Rules on the local machine:

Get-NetIPsecRule | Select-Object DisplayName, InboundSecurity, OutboundSecurity, RemoteAddress, Enabled | Format-Table -AutoSize

To view the active security associations (Main Mode and Quick Mode SAs) negotiated by the kernel with remote endpoints:

# Inspect Main Mode authenticated peers
Get-NetIPsecMainModeSA | Select-Object PeerAddress, LocalAddress, AuthenticationMethod, CipherAlgorithm

# Inspect active Quick Mode encrypted tunnels
Get-NetIPsecQuickModeSA | Select-Object PeerAddress, LocalAddress, EncapsulationMode, EncryptionAlgorithm

Testing and Troubleshooting IPsec Connectivity

If network connections fail after enabling a rule, verify whether Main Mode negotiation succeeded using Windows Security Event Logs. Event ID 4656 records successful Main Mode authentication, while Event ID 4950 flags negotiation failures.

To temporarily disable a rule without deleting its configuration during troubleshooting:

Disable-NetIPsecRule -DisplayName "Secure-DB-Transport"

To re-enable it once testing is complete:

Enable-NetIPsecRule -DisplayName "Secure-DB-Transport"

Deleting IPsec Rules

To permanently remove an IPsec Connection Security Rule:

Remove-NetIPsecRule -DisplayName "Secure-DB-Transport" -Confirm:$false

Configuring IPsec Connection Security Rules via PowerShell provides Windows 11 systems with automated, cryptographic network isolation that safeguards critical data in transit across modern networks.