How to Configure macOS System Extensions (EndpointSecurity API) to Bypass Kext Restrictions

For over a decade, macOS security and monitoring software relied heavily on Kernel Extensions (kexts) to intercept system calls, monitor file activity, and filter network traffic. However, with the introduction of macOS Catalina and the subsequent transition to Apple Silicon, Apple deprecated kexts in favour of System Extensions. System Extensions run entirely in user space, dramatically improving kernel stability and security.

The crown jewel of this new architecture is the EndpointSecurity (ES) API, a C API that allows security products to monitor and block system events in real-time. This guide explains how macOS administrators and developers can configure, deploy, and manage EndpointSecurity System Extensions effectively.

The Architecture of the EndpointSecurity API

Unlike kexts, which run in Ring 0 (kernel space) and can cause kernel panics if they crash, EndpointSecurity clients run in user space as System Extensions. The macOS kernel securely passes system events (such as file executions, process forks, and mounting of volumes) to the ES client via an XPC connection.

The ES API provides two types of event subscriptions:

• Notify Events: The client is informed that an event occurred (e.g., ES_EVENT_TYPE_NOTIFY_EXEC). The client can log the event but cannot stop it.
• Auth Events: The client intercepts the event before it executes (e.g., ES_EVENT_TYPE_AUTH_EXEC). The kernel suspends the action until the ES client returns an authorization decision (allow or deny).

Prerequisites for EndpointSecurity Development

Because the ES API is incredibly powerful (allowing a program to freeze the entire OS by denying or stalling auth events), Apple strictly controls its usage. You cannot simply compile an ES program and run it on a production Mac.

You must possess an Apple Developer Account and explicitly request the com.apple.developer.endpoint-security.client entitlement from Apple. If you are an enterprise administrator deploying a third-party security tool (like CrowdStrike, SentinelOne, or Jamf Protect), the vendor has already secured this entitlement, but you must configure MDM payloads to approve the extension.

Step 1: Approving System Extensions via MDM (For Administrators)

If you are deploying a third-party EndpointSecurity tool, macOS will prompt the user to manually approve the System Extension in System Settings, followed by a prompt to grant Full Disk Access. In an enterprise environment, this manual interaction is unacceptable.

You must deploy a SystemExtensionPolicy payload via your Mobile Device Management (MDM) solution.

Create a mobileconfig profile with the following keys:

<key>PayloadContent</key>
<array>
    <dict>
        <key>PayloadType</key>
        <string>com.apple.system-extension-policy</string>
        <key>AllowUserOverrides</key>
        <false/>
        <key>AllowedSystemExtensions</key>
        <dict>
            <key>TEAMID1234</key>
            <array>
                <string>com.securityvendor.agent.extension</string>
            </array>
        </dict>
    </dict>
</array>

Replace TEAMID1234 with the vendor’s 10-character Apple Team ID, and com.securityvendor.agent.extension with the specific bundle identifier of the System Extension.

Step 2: Granting Full Disk Access (FDA) via MDM

An EndpointSecurity client cannot monitor file events globally unless it has Full Disk Access (FDA). You must deploy a Privacy Preferences Policy Control (PPPC) profile to automatically grant this.

<key>PayloadType</key>
<string>com.apple.TCC.configuration-profile-policy</string>
<key>Services</key>
<dict>
    <key>SystemPolicyAllFiles</key>
    <array>
        <dict>
            <key>Identifier</key>
            <string>com.securityvendor.agent.extension</string>
            <key>IdentifierType</key>
            <string>bundleID</string>
            <key>Allowed</key>
            <true/>
        </dict>
    </array>
</dict>

Step 3: Managing the Extension Locally (For Developers/Testing)

If you are developing or testing an ES client locally, you can manage the System Extension using the systemextensionsctl command-line utility.

To view all currently loaded System Extensions on your Mac:

systemextensionsctl list

You will see a list of extensions categorized by state (e.g., [activated enabled] or [terminated waiting to uninstall]).

Important Note on Uninstallation: A System Extension cannot be uninstalled simply by dragging the host application to the Trash. The host application must explicitly call OSSystemExtensionRequest.activationRequest(forExtensionWithIdentifier:queue:) to deactivate it, or you must disable System Integrity Protection (SIP) to manually remove it during development.

Step 4: Disabling SIP for Local ES Debugging

If you are compiling your own ES tool but Apple has not yet granted you the official entitlement, you can still test your code by disabling System Integrity Protection (SIP). This disables signature and entitlement enforcement.

1. Boot your Mac into Recovery Mode (Hold Command+R on Intel, or hold the Power Button on Apple Silicon).
2. Open Terminal from the Utilities menu.
3. Execute: csrutil disable
4. Execute: csrutil authenticated-root disable
5. Reboot normally.

Warning: Never disable SIP on a production machine. This is strictly for compiling and testing unsigned EndpointSecurity clients in a sandbox environment.

Conclusion

The EndpointSecurity API represents a massive leap forward in macOS security architecture. By moving critical event monitoring out of the kernel (kexts) and into user space, Apple has ensured that third-party security tools can no longer trigger kernel panics. Understanding how to approve, provision FDA, and manage these System Extensions via MDM is now a mandatory skill for any enterprise macOS administrator.