A filesystem and disk I/O deep dive using blktrace and debugfs

SeaGL 2017 Oct 6 at 16:30

Programs like iostat and iotop can tell you all sorts of interesting things about what’s happening on your block devices. You can answer questions like: “How busy are my disks?” “What’s my average latency?” “How many blocks per second am I writing?” “Which processes are doing the most disk I/O?”

What do you do when that information isn’t detailed enough? While it’s great to have statistics on how your disks are performing and which processes are responsible for generating most of the I/O, how do you answer a question like: “Which files are most actively being written to or read from and by whom?”

This question can be answered by blktrace, a block layer IO tracing mechanism which, among other things, provides detailed information about every logical block address (LBA) that is read and written from your disk. In this talk I’ll show you how to run blktrace and how to interpret some of it’s cryptic output. Using some arithmetic and debugfs I’ll show you how to go from an LBA, to a disk partition, to a filesystem block, to an inode, and finally to a filename.

In this talk we’ll also cover some fun things about the GNU/Linux ‘ext’ family of filesystems. “What’s an inode?” “How are files broken up and organized into multiple disk blocks?” We’ll explore these concepts using the debugfs tool, which we’ll use to help us read file contents straight off the disk using dd. How cool is that?

Presenters