How to Configure IPsec Certificate Authentication Rules in Windows 11 Using PowerShell

In secure enterprise networks, establishing IPsec communication requires negotiating security associations based on authenticated machine credentials. While pre-shared keys (PSKs) and Kerberos are common in domain environments, zero-trust architectures and remote device management require certificate-based authentication using X.509 digital certificates. Windows Defender Firewall with Advanced Security allows administrators to specify exact certificate criteria to ensure that only devices issued certificates by trusted Enterprise Certificate Authorities (CAs) can establish connections. PowerShell provides full automation for configuring IPsec certificate authentication rules in Windows 11.

How IPsec Certificate Authentication Functions

During Main Mode (Phase 1 / IKEv2 or AuthIP) negotiations, the two communicating hosts exchange and validate digital certificates issued by a Public Key Infrastructure (PKI):

• CA Validation: The negotiating host verifies that the peer’s certificate chains up to a trusted root or intermediate Certification Authority stored in the local Windows certificate store.
• EKU Validation: Ensures the certificate contains the appropriate Enhanced Key Usage extension, typically IP security IKE intermediate (OID 1.3.6.1.5.5.8.2.2) or Client Authentication (OID 1.3.6.1.5.5.7.3.2).
• Subject Name Matching: Restricts authentication to specific Distinguished Names (DNs) or User Principal Names (UPNs).

Inspecting Local Machine Certificates in PowerShell

Before configuring firewall authentication sets, verify that a suitable machine certificate exists in the Windows 11 Local Machine store:

1. Right-click the Start button on Windows 11 and select Terminal (Admin) or PowerShell (Admin).
2. Query the local machine certificate repository:

   Get-ChildItem -Path Cert:\LocalMachine\My | Select-Object Subject, Issuer, Thumbprint, NotAfter

3. Locate the certificate issued by your enterprise Root or Subordinate CA. Copy its Root CA Distinguished Name (e.g., CN=Enterprise-Root-CA, DC=corp, DC=example, DC=com).

Creating an IPsec Authentication Proposal Using Certificates

Windows 11 manages IPsec authentication through Phase 1 authentication sets:

# Define the Certificate Authentication Proposal
$rootCA = "CN=Enterprise-Root-CA, DC=corp, DC=example, DC=com"

New-NetIPsecPhase1AuthSet -DisplayName "Enterprise-Cert-Auth" `
    -Name "Enterprise-Cert-Auth" `
    -Proposal (New-NetIPsecAuthProposal -Machine -Cert -Authority $rootCA -AuthorityType Root)

In this cmdlet:

• -Machine: Specifies computer-level authentication (rather than user credentials).
• -Cert: Enforces X.509 certificate validation.
• -Authority: Sets the required CA Distinguished Name that signed the peer certificate.
• -AuthorityType Root: Mandates validation against the trusted root authority.

Configuring an IPsec Connection Security Rule with Certificate Auth

Now link the certificate authentication set to an active Connection Security Rule protecting traffic to sensitive internal subnets:

New-NetIPsecRule -DisplayName "Secure Data Traffic via Certs" `
    -RuleId "Secure-Data-Certs" `
    -LocalAddress Any `
    -RemoteAddress 10.200.0.0/16 `
    -Mode Transport `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Phase1AuthSet "Enterprise-Cert-Auth" `
    -QuickModeCryptoSet "Enterprise-AES-GCM"

This mandates that all inbound and outbound communication with the 10.200.0.0/16 subnet must be encrypted using AES-GCM and authenticated using validated enterprise certificates.

Auditing Negotiated Certificate SAs

To verify that active IPsec associations successfully authenticated via digital certificates:

Get-NetIPsecMainModeSA | Select-Object LocalAddress, RemoteAddress, PeerId, SelectedProposal

Look for the PeerId field, which displays the certificate subject name negotiated with the remote peer.

Troubleshooting Certificate Validation Failures

If negotiation stalls or fails:

1. Inspect Windows Filtering Platform security audit events:

   Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4653, 4654} -MaxEvents 10

2. Verify that the root certificate is installed in Cert:\LocalMachine\Root and intermediate certificates reside in Cert:\LocalMachine\CA.
3. Ensure Certificate Revocation Lists (CRLs) or OCSP responders are accessible across the network; offline CRL checks cause handshake timeouts.

Using PowerShell to automate IPsec certificate authentication provides robust cryptographic mutual authentication across enterprise Windows 11 endpoints.