Performance problems reading files with holes

In the last filesystem session of LSFMM 2017, Anna Schumaker presented some performance problems she has found when working with sparse files. There are some major differences between ext4, XFS, and Btrfs when accessing files with holes.

NFSv4.2 adds a READ_PLUS command that incorporates some optimizations for reading sparse files. READ_PLUS returns both data and holes, but the latter are represented by a range rather than a series of blocks of zeroes. She has been implementing that but ran into some performance problems.

When using the SEEK_DATA and SEEK_HOLE functionality on a file that has an even mix of data and holes, ext4 and XFS perform reasonably well (though XFS is a bit worse than ext4), but Btrfs is completely unusable, she said. It takes roughly twice as long to read the sparse file than it does to read a non-sparse file of the same size on Btrfs.

Several theories were offered as to what might be going wrong. It would seem that caching probably plays a role. Ted Ts'o noted that SEEK_HOLE is an ioctl() command that is implemented by the filesystems. Both ext4 and XFS cache the hole information; ext4 used to do it badly. He has not looked at Btrfs to see what it does, but suggested the problem might lie in this area.

Ts'o thought a quick microbenchmark might help point to the problem. Jan Kara, though, was puzzled by the results reported. He said he needed more information; it was generally agreed by the attendees that more data was needed to shed some light on the problem.

Trond Myklebust pointed out that the fact that only Btrfs exhibits the problem would seem to indicate that there isn't an NFS implementation bug. Jeff Layton suggested writing a small user-space program that exhibits the behavior. If that were to get posted to the mailing lists, it should be pretty easy for the filesystem developers to reproduce the problem and diagnose what is going on.