How to Configure IPsec Main Mode Rules in Windows 11 Using PowerShell

In secure enterprise networks and zero-trust domain environments, Internet Protocol Security (IPsec) encrypts and authenticates network traffic flowing between endpoints. IPsec communication is established in two distinct phases: Phase 1 (Main Mode) and Phase 2 (Quick Mode). While Quick Mode negotiates specific data packet encryption algorithms and port filters, Main Mode establishes the primary, authenticated cryptographic channel using Internet Key Exchange (IKEv1/IKEv2) or Authenticated IP (AuthIP). In Windows 11, administrators can configure granular Phase 1 Main Mode rules directly through PowerShell NetSecurity cmdlets.

In this guide, you will learn how to configure Main Mode cryptographic sets, define computer and user authentication requirements, create Main Mode security rules, and audit active Phase 1 associations in Windows 11 using PowerShell.

Understanding Main Mode (Phase 1) vs. Quick Mode (Phase 2)

The IPsec negotiation lifecycle separates identity verification from transport encryption:

• Main Mode (Phase 1): Authenticates the identity of the two communicating computers (and optionally the active user accounts) using Kerberos v5, pre-shared keys, or X.509 digital certificates. It performs a Diffie-Hellman (DH) key exchange to establish an encrypted master secret that protects subsequent negotiations.
• Quick Mode (Phase 2): Operates inside the secure tunnel created by Main Mode. It negotiates Security Associations (SAs) for Encapsulating Security Payload (ESP) or Authentication Header (AH) to protect individual application TCP/UDP connections.

Without properly aligned Main Mode rules, endpoints cannot establish Phase 1 associations, causing connection timeouts or unencrypted cleartext fallbacks.

Creating a Custom Main Mode Cryptographic Set

Main Mode rules evaluate cryptographic proposals (encryption algorithms, integrity hashing, and Diffie-Hellman key exchange groups). To define modern, enterprise-grade cryptography conforming to NIST recommendations, create a cryptographic set using New-NetIPSecMainModeCryptoSet:

Launch an elevated PowerShell console (Run as Administrator) and execute:

New-NetIPSecMainModeCryptoSet -DisplayName "EnterpriseMainModeCrypto" -Proposal @(
    "AES256-SHA384-DH24",
    "AES256-SHA256-DH14",
    "AES128-SHA256-DH14"
)

In this proposal array:

• AES256: 256-bit Advanced Encryption Standard for Phase 1 privacy.
• SHA384 / SHA256: Secure Hash Algorithm for payload integrity and hashing.
• DH24 / DH14: 2048-bit (DH14) and 2048-bit with 256-bit subgroup (DH24) Diffie-Hellman modular exponentiation groups, mitigating downgrade attacks.

Verify that the cryptographic set is registered:

Get-NetIPSecMainModeCryptoSet -DisplayName "EnterpriseMainModeCrypto"

Defining Phase 1 Authentication Credentials

Next, define which credentials Windows uses to authenticate remote peers during Phase 1 using New-NetIPSecPhase1AuthSet:

For domain-joined environments utilizing Kerberos v5 computer authentication:

New-NetIPSecPhase1AuthSet -DisplayName "DomainKerberosAuth" -Proposal @(
    "ComputerKerberos"
)

For hybrid or workgroup networks requiring computer certificates issued by an internal Certificate Authority (CA):

New-NetIPSecPhase1AuthSet -DisplayName "DomainCertAuth" -Proposal @(
    "ComputerCert:Thumbprint=abcdef0123456789abcdef0123456789abcdef01"
)

Creating the Main Mode Rule

Now, bind the cryptographic set and authentication set to a network endpoint or subnet using New-NetIPSecMainModeRule.

For example, to require Main Mode Phase 1 negotiation for all traffic communicating with the corporate server subnet 10.20.0.0/16:

New-NetIPSecMainModeRule -DisplayName "Secure-CorpServers-Phase1" `
    -RemoteAddress "10.20.0.0/16" `
    -MainModeCryptoSet "EnterpriseMainModeCrypto" `
    -Phase1AuthSet "DomainKerberosAuth" `
    -Profile "Domain", "Private"

Parameters include:

• -RemoteAddress: The destination IPv4/IPv6 network or specific server IP address.
• -Profile: Applies the rule when connected to Domain or Private network profiles.

Verify the created rule in PowerShell:

Get-NetIPSecMainModeRule -DisplayName "Secure-CorpServers-Phase1" | Format-List

Auditing Active Main Mode Security Associations (SAs)

Once a connection begins communicating with the remote subnet, inspect the active Phase 1 Security Associations established by the Windows kernel using Get-NetIPSecMainModeSA:

Get-NetIPSecMainModeSA | Format-Table LocalAddress, RemoteAddress, SelectedCipherAlgorithm, SelectedHashAlgorithm, KeyExchangeGroup, LifetimeSeconds -AutoSize

Review the output to ensure that the agreed SelectedCipherAlgorithm (e.g., AES256) and KeyExchangeGroup (e.g., DH24) match your configured proposal.

Troubleshooting Main Mode Negotiation Failures

If Phase 1 fails to establish, traffic may fail to connect or pass unencrypted:

• Audit Windows Filtering Platform Events: Inspect the Windows Security log for event ID 4652 (an IPsec Main Mode negotiation failed) using Get-WinEvent:

  Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4652} -MaxEvents 5 | Format-List

  Look for failure codes such as Negotiation timed out (firewall blocking UDP port 500/4500) or No proposal chosen (mismatched Diffie-Hellman groups).
• Verify UDP Ports 500 and 4500: Ensure intermediate hardware routers permit IKE (UDP 500) and IPsec NAT-Traversal (UDP 4500) packets.
• Clock Skew on Kerberos: Ensure local system time is within 5 minutes of the domain controller clock; clock drift triggers Kerberos ticket validation errors.

Removing Main Mode Rules and Cryptographic Sets

To dismantle an IPsec Phase 1 configuration when re-architecting network policies:

# Remove the Main Mode rule
Remove-NetIPSecMainModeRule -DisplayName "Secure-CorpServers-Phase1" -Confirm:$false

# Remove the associated cryptographic and authentication sets
Remove-NetIPSecMainModeCryptoSet -DisplayName "EnterpriseMainModeCrypto" -Confirm:$false
Remove-NetIPSecPhase1AuthSet -DisplayName "DomainKerberosAuth" -Confirm:$false

The rules are purged from the Windows kernel filtering platform immediately without requiring an operating system restart.