How to Configure IPsec Main Mode Cryptographic Sets in Windows 11 Using PowerShell

In secure network architectures, IPsec protects sensitive communication channels through a two-phase negotiation process. While Phase 2 (Quick Mode) negotiates payload encryption, Phase 1 (Main Mode) is responsible for mutual authentication and establishing the initial authenticated and encrypted management tunnel via Internet Key Exchange (IKEv1/IKEv2) or Authenticated IP (AuthIP). In Windows 11, the NetSecurity module in PowerShell provides dedicated cmdlets to create and enforce custom Main Mode cryptographic sets, enabling system administrators to mandate modern Diffie-Hellman groups and strong encryption while barring weak legacy ciphers.

Understanding Main Mode Cryptographic Proposals

During Phase 1 negotiation, the two connecting hosts establish a Main Mode Security Association (MM SA). The Main Mode cryptographic set defines the specific list of cryptographic suites that Windows will propose or accept during this initial handshake.

Every Main Mode proposal must define three essential cryptographic components:

• Encryption Algorithm: Protects the Phase 1 identity exchange and credential payloads (for example, AES-CBC-256 or AES-CBC-128).
• Hash Algorithm: Provides cryptographic integrity and generates pseudo-random key material (for example, SHA-256 or SHA-384).
• Diffie-Hellman Key Exchange Group: Determines the mathematical strength of the initial shared secret negotiation. Windows 11 supports Group 14 (2048-bit MODP), Group 19 (256-bit Random ECP), Group 20 (384-bit Random ECP), and Group 24 (2048-bit MODP with 256-bit prime order subgroup).

By defining explicit Main Mode sets, organizations ensure that endpoints never fall back to insecure legacy standards such as DES, 3DES, MD5, SHA-1, or Diffie-Hellman Group 1 and Group 2.

Creating Main Mode Cryptographic Proposals

Before grouping proposals into a set, individual cryptographic configurations are created using the New-NetIPSecMainModeCryptoProposal cmdlet. Each proposal establishes an algorithm combination along with SA lifetime limits.

Open an elevated PowerShell window by right-clicking Start and selecting Terminal (Admin). To create a modern, high-security proposal utilizing AES-256 encryption, SHA-384 hashing, and Elliptic Curve Diffie-Hellman Group 20, execute:

$PrimaryProposal = New-NetIPSecMainModeCryptoProposal -Encryption AES256 -Hash SHA384 -KeyGroup ECP384 -LifetimeMinutes 480 -MaxMinutes 0

If your enterprise connects to interoperable edge firewalls that require 256-bit elliptic curves or 2048-bit modular exponentiation, define an alternate proposal as an approved fallback:

$FallbackProposal = New-NetIPSecMainModeCryptoProposal -Encryption AES256 -Hash SHA256 -KeyGroup ECP256 -LifetimeMinutes 480 -MaxMinutes 0

In these commands, -LifetimeMinutes 480 enforces an 8-hour Main Mode SA lifetime, ensuring regular re-keying while reducing cryptographic overhead on high-throughput connections.

Assembling the Main Mode Cryptographic Set

Once your proposals are defined in variables, register them as a persistent cryptographic set using the New-NetIPSecMainModeCryptoSet cmdlet. Proposals are evaluated by the Windows Filtering Platform in the order they are passed.

To create the persistent set containing your primary and fallback proposals, run:

New-NetIPSecMainModeCryptoSet -DisplayName "Enterprise-MainMode-Strict" -Name "Enterprise-MainMode-Strict" -Proposal $PrimaryProposal, $FallbackProposal

Windows 11 registers the set in its local security policy store, making it immediately available to connection security rules and Main Mode rules.

Auditing and Inspecting Configured Sets

To inspect all configured Main Mode sets on the system and verify their properties, use Get-NetIPSecMainModeCryptoSet:

Get-NetIPSecMainModeCryptoSet | Select-Object Name, DisplayName

To inspect the individual proposals inside your newly created set, expand the nested Proposal property:

(Get-NetIPSecMainModeCryptoSet -Name "Enterprise-MainMode-Strict").Proposal | Format-Table Encryption, Hash, KeyGroup, LifetimeMinutes

The resulting table confirms the exact algorithms and key exchange groups configured in the set, ensuring full compliance with corporate security standards.

Binding the Set to Main Mode Rules

A Main Mode cryptographic set must be bound to a Main Mode rule (or connection security rule) to take effect. In Windows 11, Main Mode rules determine which endpoints authenticate using specific credentials and cryptographic parameters.

To create a new Main Mode rule that binds your cryptographic set to all communications with a secure server subnet (for example, 10.200.0.0/16), execute:

New-NetIPSecMainModeRule -DisplayName "Secure-Subnet-Phase1" -Name "Secure-Subnet-Phase1" `
    -LocalAddress Any -RemoteAddress 10.200.0.0/16 `
    -MainModeCryptoSet "Enterprise-MainMode-Strict"

If you already have an existing Main Mode rule named "Branch-Office-IKE" and wish to enforce this new cryptographic set, update it using Set-NetIPSecMainModeRule:

Set-NetIPSecMainModeRule -DisplayName "Branch-Office-IKE" -MainModeCryptoSet "Enterprise-MainMode-Strict"

Now, any Phase 1 IKEv2 or AuthIP negotiation initiated towards the specified remote addresses will strictly mandate the cipher suites defined within your custom set.

Verifying Live Main Mode Security Associations

Once a connection security rule triggers an IPsec handshake with a target host, Windows establishes the Phase 1 SA. You can inspect live Main Mode SAs in real time to verify that the desired algorithms were negotiated successfully.

Query the active Main Mode Security Associations:

Get-NetIPSecMainModeSA | Format-List LocalEndpoint, RemoteEndpoint, CipherSuite, KeyExchangeGroup

The output will show AES-256 alongside your configured key exchange group (such as ECP384 or ECP256), confirming that Phase 1 tunnel encryption was established using your designated cryptographic proposal.

Modifying and Decommissioning Main Mode Sets

If corporate cryptographic guidelines mandate changes—such as deprecating 256-bit curves in favour of 384-bit curves across all nodes—you can update the set dynamically using Set-NetIPSecMainModeCryptoSet.

To enforce Group 20 exclusively, generate a single proposal and apply it:

$StrictOnly = New-NetIPSecMainModeCryptoProposal -Encryption AES256 -Hash SHA384 -KeyGroup ECP384 -LifetimeMinutes 240
Set-NetIPSecMainModeCryptoSet -Name "Enterprise-MainMode-Strict" -Proposal $StrictOnly

If a set is retired and no longer referenced by active rules, remove it cleanly using Remove-NetIPSecMainModeCryptoSet:

Remove-NetIPSecMainModeCryptoSet -Name "Enterprise-MainMode-Strict" -Confirm:$false

Using PowerShell to configure and enforce Main Mode cryptographic sets provides a repeatable, auditable method for hardening network encryption and eliminating legacy cryptographic vulnerabilities across Windows 11 enterprise endpoints.