How to Trace Mount and Unmount System Calls in Linux Using mountsnoop

In modern Linux environments, storage administration and container orchestration rely heavily on filesystem mount operations. Containers, systemd service sandboxes, snaps, and automated backup daemons continuously create, pivot, and tear down filesystem mounts across mount namespaces. When an unexpected storage mount occurs, an encrypted volume fails to mount, or unmount operations return EBUSY (device or resource busy), traditional commands like df or mount only provide static snapshots of the current state without revealing which process initiated the operation or the exact flags supplied.

The BPF Compiler Collection (BCC) provides mountsnoop, an extended Berkeley Packet Filter (eBPF) tracing utility. By attaching probes to the kernel’s mount(), umount(), and move_mount() system calls, mountsnoop captures every mount and unmount event in real time with near-zero performance overhead, logging the invoking process, PID, mount parameters, filesystem types, and return codes.

Understanding mountsnoop Architecture and eBPF Probing

Whenever a user-space process issues a mount operation, execution transitions into kernel space via system calls such as sys_mount or the modern VFS system call sys_move_mount. Under standard logging configurations, these invocations leave no trace in system journals or audit logs unless explicit audit rules are configured.

mountsnoop operates by attaching kprobes and tracepoints to:

• mount and umount entry points: Capturing the source device path, target directory mount point, filesystem type (such as ext4, xfs, btrfs, or overlay), and binary mount flags (such as MS_RDONLY, MS_NOSUID, MS_NODEV, or MS_BIND).
• System call return probes (kretprobes): Recording the completion status (such as 0 for success, or negative errno codes such as -16 EBUSY, -2 ENOENT, or -13 EACCES).
• Process context: Recording the executable command name (COMM), Process ID (PID), and Mount Namespace ID.

Installing BCC Tools on Linux Distributions

To run mountsnoop, ensure your system has the BCC tools package installed. On modern Linux distributions with kernel 5.4 or later, BCC tools run out of the box.

On Ubuntu and Debian systems, install the package via apt:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

Note: On Ubuntu, BCC binaries are typically installed with a -bpfcc suffix (e.g. mountsnoop-bpfcc).

On Red Hat Enterprise Linux, Rocky Linux, or Fedora:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Arch Linux:

sudo pacman -S bcc bcc-tools linux-headers

Running mountsnoop to Monitor Mount Operations in Real Time

Launch mountsnoop with root privileges in an interactive terminal:

sudo mountsnoop-bpfcc

(If installed in standard paths on RHEL/Fedora, execute sudo /usr/share/bcc/tools/mountsnoop.)

The tool outputs columnar data as events occur:

COMM             PID     TID     MNT_NS      CALL
systemd          1       1       4026531840  mount("none", "/run/user/1000", "tmpfs", MS_NODEV|MS_NOSUID, "") = 0
dockerd          1284    1290    4026531840  mount("overlay", "/var/lib/docker/overlay2/abc/merged", "overlay", 0, "lowerdir=...") = 0
umount           4812    4812    4026531840  umount2("/mnt/backup", 0) = 0

Key columns explained:

• COMM: Name of the application or binary performing the system call.
• PID / TID: Process ID and Thread ID responsible for the call.
• MNT_NS: The inode number of the process’s mount namespace, crucial for distinguishing host mounts from containerised namespaces.
• CALL: The full system call invocation, arguments, options string, and the return value.

Filtering Mount Events and Tracking Failures

In high-throughput environments running Kubernetes or Docker, hundreds of short-lived ephemeral volumes are created and torn down every minute. mountsnoop supports command-line switches to filter and structure trace output:

Displaying Timestamps

Include precise timestamps to correlate mount activity with application logs or system alerts:

sudo mountsnoop-bpfcc -t

Output displays elapsed seconds since trace initiation or system boot, providing chronological context for race conditions.

Tracking Namespace Isolation

To identify whether a mount event originated inside a container or on the root host system, inspect the namespace column (MNT_NS). You can determine the root host mount namespace ID using ls -l /proc/1/ns/mnt:

readlink /proc/1/ns/mnt

If the MNT_NS value in mountsnoop differs from PID 1’s namespace inode, the mount call occurred inside an isolated container, sandbox, or chroot environment.

Isolating Failed Mount Calls

When storage mounts fail intermittently during system boot or automated script execution, filter output to reveal non-zero return codes:

sudo mountsnoop-bpfcc | grep -v "= 0"

This quickly surfaces errors such as:

• = -16 (EBUSY): The device or mount target is currently open or held by another process.
• = -2 (ENOENT): The target mount directory path does not exist.
• = -13 (EACCES): Insufficient privileges or filesystem permission conflict.
• = -22 (EINVAL): Invalid mount options or unsupported filesystem type.

Diagnosing Container and Systemd Mount Leaks

To trace dynamic storage orchestration in containerised hosts, launch mountsnoop while starting a container:

# Terminal 1: Run trace
sudo mountsnoop-bpfcc

# Terminal 2: Run test container
docker run --rm -it alpine sh -c "echo Hello"

You will observe dockerd or containerd establishing the multi-layer overlay filesystem, creating read-only binds (MS_BIND|MS_RDONLY), masking sensitive paths like /proc/sys with tmpfs or read-only remounts, and subsequently calling umount2(..., MNT_DETACH) upon container termination.

Using the bpftrace Implementation for Lightweight Profiling

If full Python-based BCC tools are unavailable, modern Linux distributions include a lightweight mountsnoop.bt script powered by bpftrace:

sudo bpftrace /usr/sbin/mountsnoop.bt

Or trace mount calls directly using a one-line bpftrace command:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_mount { printf("%-16s %-6d %s -> %s\n", comm, pid, str(args->dev_name), str(args->dir_name)); }'

This provides an immediate, low-overhead stream of device-to-directory mappings suitable for resource-constrained edge systems.