How to Configure Apple Endpoint Security Framework (ESF) for Custom Mac Malware Threat Hunting

Historically, building Endpoint Detection and Response (EDR) solutions or custom malware hunting tools for macOS required writing deeply intrusive Kernel Extensions (Kexts). These Kexts were notoriously unstable, frequently causing kernel panics, and posed massive security risks if exploited. Apple has entirely deprecated Kexts for security software.

The modern, architecturally sound replacement is the Endpoint Security Framework (ESF). Operating entirely in user space, ESF provides a C API that allows developers and security teams to monitor critical system events—such as process executions, file creations, and kernel module loads—in real-time, directly from the macOS kernel.

This guide explores the architecture of the Endpoint Security Framework and how to configure a rudimentary ESF client to hunt for malicious macOS behavior.

Understanding Endpoint Security Architecture

ESF operates via a publish-subscribe model. The macOS kernel (specifically the EndpointSecurity.kext subsystem) acts as the publisher. Your user-space application (the System Extension) acts as the subscriber.

There are two types of events your client can subscribe to:

1. Notify Events: The kernel informs your application that an action occurred (e.g., “A process was executed”). This is purely for telemetry and logging.
2. Auth Events: The kernel pauses the execution of an action and asks your application for permission. Your application must explicitly reply with an ES_AUTH_RESULT_ALLOW or ES_AUTH_RESULT_DENY. If you deny it, the kernel blocks the action.

To prevent malicious actors from using ESF to build rootkits, Apple strictly regulates who can use the API. To run an ESF client outside of development mode, your application must possess a special Entitlement (com.apple.developer.endpoint-security.client), which Apple only grants to verified security vendors.

Step 1: Initialising the ESF Client in C

To interact with the framework, you write a daemon in C, Objective-C, or Swift. The daemon must run as root.

First, you include the framework and define the client initialization block. When an event occurs, the kernel invokes this block.

#include <EndpointSecurity/EndpointSecurity.h>
#include <stdio.h>

int main(int argc, const char * argv[]) {
    es_client_t *client = NULL;
    
    // Create the client and define the callback block
    es_new_client_result_t result = es_new_client(&client, ^(es_client_t *c, const es_message_t *msg) {
        
        // Ensure we are processing a process execution event
        if (msg->event_type == ES_EVENT_TYPE_NOTIFY_EXEC) {
            
            // Extract the executable path
            const char *exec_path = msg->event->exec.target->executable->path.data;
            pid_t pid = audit_token_to_pid(msg->process->audit_token);
            
            printf("Process Executed: PID %d | Path: %s\\n", pid, exec_path);
        }
    });

    if (result != ES_NEW_CLIENT_RESULT_SUCCESS) {
        printf("Failed to create Endpoint Security client. Error: %d\\n", result);
        return 1;
    }
    // ... continues below

Step 2: Subscribing to Events

Simply creating the client does not instruct the kernel to send data. You must explicitly subscribe to the specific events you want to monitor. Subscribing to too many events (especially Auth events) can severely degrade system performance.

To hunt for malware, monitoring process executions (ES_EVENT_TYPE_NOTIFY_EXEC) and file creations (ES_EVENT_TYPE_NOTIFY_CREATE) are critical.

    // Define the events to subscribe to
    es_event_type_t events[] = {
        ES_EVENT_TYPE_NOTIFY_EXEC
    };
    
    // Subscribe the client to the kernel
    if (es_subscribe(client, events, 1) != ES_RETURN_SUCCESS) {
        printf("Failed to subscribe to events.\\n");
        return 1;
    }

    printf("Endpoint Security Client Active. Monitoring executions...\\n");
    
    // Keep the daemon running
    dispatch_main();
    return 0;
}

Step 3: Compiling and Bypassing Entitlements for Testing

Because you likely do not have the Apple-granted ESF entitlement for production code, you can only run this tool by disabling System Integrity Protection (SIP) during development.

Reboot your Mac into Recovery Mode (Command+R, or hold the Power Button on Apple Silicon) and run:

csrutil disable

Reboot normally. Compile the C code using clang, linking against the Endpoint Security framework:

clang -framework EndpointSecurity -o es_hunter es_hunter.c

Execute the binary as root:

sudo ./es_hunter

As you open applications or run terminal commands, you will see a real-time stream of every single executable binary launching on the macOS system, directly fed from the kernel.

Step 4: Threat Hunting Logic (Muting and Filtering)

A raw feed of EXEC events generates massive noise. A robust threat hunting tool must filter out legitimate Apple binaries. ESF provides the es_mute_process API to silence events from trusted processes.

For example, if you do not want to see events generated by the macOS Spotlight indexer (mdworker), you can extract its audit token and mute it. The kernel will then stop sending telemetry for that specific process, saving immense CPU cycles in user space.

A typical custom EDR logic flow involves:

1. Receive ES_EVENT_TYPE_AUTH_EXEC.
2. Check the binary path against a list of known Apple directories (e.g., /System/Library/).
3. Check the cryptographic signature of the binary using the macOS Security framework.
4. If the signature is invalid or matches a known malware hash, return ES_AUTH_RESULT_DENY to kill the process before it launches.

Conclusion

The Apple Endpoint Security Framework provides security engineers with unprecedented, safe, and stable visibility into the macOS kernel. By migrating threat hunting and EDR capabilities from unstable kernel extensions into highly structured user-space System Extensions, organizations can achieve deep forensic telemetry and active prevention without risking system crashes.