NFS Flamegraph

A lightweight, eBPF-based tracing tool designed to generate FlameGraphs for NFS client activity.

Note: The flamegraph above was captured using the load scripts in nfs-traffic/ to simulate I/O against the arch/ directory of a Linux kernel repository hosted on an AWS EFS share.

Overview

This tool hooks directly into the Linux kernel's NFS IO operations (nfs_file_read and nfs_file_write) to trace exactly which files and deep directory trees are driving read and write operations. It natively captures the transferred byte counts and hierarchical directory paths, dynamically formatting the data into Brendan Gregg's folded stack format so that FlameGraphs can be easily generated using the FlameGraph tool.

Motivation

Standard NFS monitoring tools (like nfsstat or nfsiostat) typically provide high-level, aggregate metrics. When an NFS share experiences heavy I/O load, identifying the specific file or directory tree causing the workload can be difficult. This tool provides a low-overhead tracing capability to identify exact file access patterns and map them visually.

How the bpftrace script achieves this

Tracing dynamic file paths in eBPF typically hits a major kernel barrier:

• The 512-Byte eBPF Stack Limit: Manually storing multiple long directory names inside the BPF kernel stack quickly crashes the script due to the rigid memory limit.

To address this limitation, we are using an asynchronous Thread ID (TID) streaming technique:

• The script hooks onto NFS operations and iteratively walks up the active dentry tree pointers ($dentry->d_parent).
• Instead of allocating strings to BPF maps, the script outputs the raw name of each directory segment over a printf stream, uniquely tagged with the active Thread ID (tid). Under the hood, bpftrace seamlessly pushes these printf events into a lockless BPF Ring Buffer (or Perf Buffer on older kernels), providing a highly efficient, high-throughput stream from kernel-space to user-space without bloating the BPF stack.
• A user-space awk pipeline asynchronously reads this continuous stream. It buffers the segments by tid to reassemble the full tree paths (up to 16 folder levels deep) natively in user-space, and aggregates the byte counters to emit the final FlameGraph format.

Usage

Ensure you have Brendan Gregg's FlameGraph repository downloaded.

chmod +x nfs_monitor.sh

# Trace activity across all NFS mounts for 15 seconds
./nfs_monitor.sh 15

# Once completed, convert the folded trace directly into a FlameGraph
cat trace.folded | flamegraph.pl > nfs_io_flamegraph.svg