How to Use macOS Endpoint Security (ES) Framework to Intercept and Block Unauthorized Process Executions

Securing enterprise macOS deployments has historically relied on kernel extensions (kexts). Antivirus vendors, Data Loss Prevention (DLP) agents, and zero-trust security tools loaded code directly into the macOS kernel to monitor file system access and intercept malicious processes. However, a single bug in a kext could trigger a fatal kernel panic, bringing down the entire operating system. Starting with macOS Catalina (10.15), Apple deprecated kexts for security software and introduced the Endpoint Security (ES) framework. This powerful, user-space C API allows security developers to deeply monitor, intercept, and definitively block unauthorized process executions and file mutations without ever touching the fragile macOS kernel.

The Architecture of the Endpoint Security Framework

The ES framework operates via a specialized system daemon (endpointsecurityd). When an application utilizes the ES API, it establishes a high-performance IPC (Inter-Process Communication) channel with this daemon.

The architecture provides two distinct models of operation:

1. Notification Mode (AUTH/NOTIFY): The security agent asks the OS to simply notify it after an event has occurred (e.g., a file was opened, a process spawned). This is highly performant and used for logging, auditing, and telemetry (similar to EDR agents).
2. Authorization Mode (AUTH/BLOCK): The security agent registers a callback and asks the OS to pause an event before it executes. The macOS kernel halts the process and waits for the user-space security agent to return an explicit “Allow” or “Deny” verdict. If the agent returns “Deny”, the kernel mathematically blocks the system call, returning an EPERM (Operation not permitted) error to the calling process.

Provisioning Entitlements

Because the ES API is incredibly powerful—capable of instantly bricking the operating system if misconfigured—Apple strictly controls access. You cannot simply compile an ES application and run it.

To utilize the framework, your application must possess the com.apple.developer.endpoint-security.client entitlement. To obtain this, you must hold a paid Apple Developer account and explicitly request access to the Endpoint Security entitlement via a rigorous Apple review process. Once granted, you must sign your binary with this entitlement using your Developer ID certificate.

Initializing the ES Client

Developing with the ES framework requires writing native C or C++ code, as the API relies heavily on low-level structs and function pointers.

The first step is to create a new client and define the global message handler block. This block will be invoked asynchronously by the OS whenever a monitored event occurs.

#include <EndpointSecurity/EndpointSecurity.h>

es_client_t *client = NULL;

es_new_client_result_t result = es_new_client(&client, ^(es_client_t *c, const es_message_t *msg) {
    // 1. Check if the message requires authorization
    if (msg->action_type == ES_ACTION_TYPE_AUTH) {
        
        // 2. Identify the specific event type (e.g., Process Execution)
        if (msg->event_type == ES_EVENT_TYPE_AUTH_EXEC) {
            
            // 3. Extract the path of the executable attempting to launch
            const char *exec_path = msg->event.exec.target->executable->path.data;
            
            // 4. Implement your security logic (e.g., block the execution of 'nc' or 'nmap')
            if (strcmp(exec_path, "/usr/bin/nc") == 0) {
                // Deny the execution
                es_respond_auth_result(c, msg, ES_AUTH_RESULT_DENY, false);
                printf("Blocked unauthorized execution of netcat.\n");
            } else {
                // Allow the execution
                es_respond_auth_result(c, msg, ES_AUTH_RESULT_ALLOW, false);
            }
        }
    }
});

if (result != ES_NEW_CLIENT_RESULT_SUCCESS) {
    printf("Failed to create Endpoint Security client.\n");
    exit(1);
}

Subscribing to Authentication Events

Merely creating the client is insufficient. You must explicitly tell the OS which specific events you want to monitor. Subscribing to every available event will rapidly overwhelm your agent and cause systemic UI lag. You must be surgically precise.

To subscribe to process execution authorization events (ES_EVENT_TYPE_AUTH_EXEC), pass an array of events to es_subscribe:

es_event_type_t events[] = { ES_EVENT_TYPE_AUTH_EXEC };
es_return_t sub_result = es_subscribe(client, events, 1);

if (sub_result != ES_RETURN_SUCCESS) {
    printf("Failed to subscribe to execution events.\n");
}

Handling the Deadline Clause

When operating in Authorization mode, the macOS kernel does not wait indefinitely for your user-space agent to respond. If your code hangs, attempts to perform a slow DNS lookup, or crashes, the entire operating system would freeze waiting for the verdict.

To prevent this, the ES framework enforces a strict deadline. You can check msg->deadline to see exactly how much time remains. If your agent fails to respond with es_respond_auth_result() before the deadline expires, the macOS kernel will autonomously kill your security agent process via a SIGKILL signal, ensuring the operating system remains responsive.