Improving RAID performance with filesystem and RAID parameters
Bringing LVM into the Mix
When you create an XFS filesystem directly on a software RAID device, mkfs.xfs is smart enough to work out the chunk and stripe sizes for you and configure the XFS filesystem optimally. Unfortunately, if you are using a hardware RAID card or using LVM between XFS and the software RAID, it is up to you to make sure the filesystem is created optimally.
I created both a hand-aligned XFS and a XFS filesystem with default parameters to test what difference, if any, this made to performance. Many folks use LVM on top of a RAID device for improved system management and flexibility. Being able to store many filesystems on a single RAID device is very convenient. I retested the ext3 configurations to see whether using the extra LVM layer sapped much performance.
Figure 5 shows the performance of ext3 and XFS on the same RAID device. For the "raw" configuration, the filesystem is created directly on the RAID device. In the "lvm" configuration, an LVM physical volume is created on the RAID device, and an LVM logical volume is created on the LVM physical volume. The filesystem is then created on the LVM logical volume.
Notice that the gains in read performance resulting from alignment of the ext3 filesystem are gone and that read performance across all filesystems is identical. Aligning the stripe on the XFS filesystem actually made write performance worse than with no alignment attempt. LVM itself appears to be limiting block read performance, so attempts to align the filesystem are not resulting in any performance differences.
Interestingly, though, block output for XFS was greatly improved by using LVM. XFS attempts to delay writes to files for as long as possible. The delayed writes might be the reason that XFS is faster at writing with the use of LVM – if LVM maintains a write cache of its own, then perhaps data can be streamed out to disk from that cache more effectively when it is presented in large chunks by the filesystem. Notice that the read performance is significantly reduced when LVM is used.
Figure 6 shows the block rewrite bonnie++ benchmark. Here you can see that the gains in correctly aligning the ext3 filesystem have been preserved to some degree, with the aligned ext3 filesystem being almost 1MB/sec faster than the unaligned ext3 on LVM. The differences for correct alignment of ext3 are less when using LVM, however: 1MB/sec for LVM compared with about 2.5MB/sec on raw RAID.
Although differences of a megabyte per second might not seem so important, in the worst case, this means the aligned ext3 on raw RAID would be more than 15% faster than unaligned ext3 on LVM. When you consider that this is running on the same machine and same disks, 15% free performance is something you might like to have.
Because it can be somewhat difficult to compare two three-dimensional graphs by eye, I decided to graph not the two original datasets, but the difference between the aligned and non-aligned datasets. In Figure 7, I'm showing you the (ext3aligned – ext3) IOzone random read performance for LVM. Although it is not completely in favor of the ext3aligned filesystem, you can see significant performance peaks where the ext3aligned filesystem is much faster in the 256 to 4,096KB file size range (to the left of the figure).
Figure 8 shows the difference of random write performance for ext3 with and without alignment on LVM. Notice that the peaks and troughs are in the same places, although the extremes are not as pronounced. The dips in performance for the ext3align filesystem are almost eradicated.
Figure 9 shows XFS with alignment and without alignment when created on LVM for random read. Notice that the graph is much less busy for XFS compared with ext3 (shown in Figure 7). Random reads of smaller files are much faster on a properly aligned XFS filesystem, as the peaks on the left side of Figure 9 show. For smaller writes to 4MB files, an aligned XFS filesystem is slightly slower than an XFS filesystem created with the default mkfs.xfs arguments (difference graph not shown). The aligned XFS is faster for the larger writes to files of 16 to 256MB.
Digging a Deeper Hole
The Linux kernel has supported the ext4 filesystem since v2.6.28. Performance improvements over its successor, ext3, include 48-bit block addressing.
One downside of specifying the RAID stripe size at mkfs time is what happens when you add an extra disk to an existing RAID. All of a sudden, you have four data disks on your RAID 5 instead of three, and the stripe alignment that the filesystem is using is incorrect. Luckily, ext4 has a mount option to allow you to set the stripe size at mount time, so if you add a disk to a RAID or extend an LVM volume, you can tell ext4 what you've done so the filesystem can optimize future activity to fit the modified disk configuration.
Unfortunately, specifying the sunit and swidth options when mounting an XFS filesystem is only effective when the filesystem is created directly on a RAID device. So if you use LVM on your RAID and add a disk, you can't tell XFS about the change at mount time. This is once again a case in which XFS does not play well with LVM.
When using LVM, metadata is stored at various abstraction layers. For example, pvcreate has various options for setting the size of metadata and data alignment. So far, I have not found tweaking pvcreate to have the same effect as setting filesystem stripe parameters, but that question is open to further investigation.
As this study shows, you can get 10% to 20% extra performance if you align your ext3 filesystem to the RAID array chunk and stripe size. Although the differences were not as significant for XFS, proper alignment of XFS can make a large difference to the performance of file creation and deletion.
- If you are using the ext3 or ext4 filesystem, you should always use -E stripe-width=x,stride=y to tell the filesystem about your RAID configuration.
- If you are using XFS directly on the RAID device, the system should detect your RAID configuration for you automatically.
- If you are using XFS on an LVM partition created on a RAID, use -d sunit= x,swidth=y at mkfs.xfs time to tell it about your RAID configuration.
- If you are using another filesystem, check that it has detected your RAID configuration – in particular, check the stripe and chunk size – at mkfs time.
The actual benefits that you reap by taking these steps will vary depending on your configuration, but regardless of the numbers, aligning your filesystem parameters with the RAID system is a good strategy for achieving better performance, especially on ext3 and ext4 filesystems.
- OSDL's Linux RAID: http://linux-raid.osdl.org/index.php/Linux_Raid
- "A Case For Redundant Arrays of Inexpensive Disks (RAID)," by David A. Patterson, Garth Gibson, and Randy H. Katz. http://www-2.cs.cmu.edu/~garth/RAIDpaper/Patterson88.pdf
Kernel king admits his tone has alienated volunteers, but says the demands of the process require directness.
New flaw in an old encryption scheme leaves the experts scrambling to disable SSL 3
Lennart Poettering wants to change the way Linux developers talk to each other.
Enterprise giant frees itself from ink and home PCs (and visa versa).
Mozilla’s product think tank sinks silently into history.
TODO group will focus on open source tools in large-scale environments.
New tool will look like GParted but support a wider range of storage technologies.
New public key pinning feature will help prevent man-in-the-middle attacks.
Carnegie Mellon researchers say 3 million pages could fall down the phishing hole in the next year.
The US government rolls new best-practice rules for protecting SSH.