Improving RAID performance with filesystem and RAID parameters
Bringing LVM into the Mix
When you create an XFS filesystem directly on a software RAID device, mkfs.xfs is smart enough to work out the chunk and stripe sizes for you and configure the XFS filesystem optimally. Unfortunately, if you are using a hardware RAID card or using LVM between XFS and the software RAID, it is up to you to make sure the filesystem is created optimally.
I created both a hand-aligned XFS and a XFS filesystem with default parameters to test what difference, if any, this made to performance. Many folks use LVM on top of a RAID device for improved system management and flexibility. Being able to store many filesystems on a single RAID device is very convenient. I retested the ext3 configurations to see whether using the extra LVM layer sapped much performance.
Figure 5 shows the performance of ext3 and XFS on the same RAID device. For the "raw" configuration, the filesystem is created directly on the RAID device. In the "lvm" configuration, an LVM physical volume is created on the RAID device, and an LVM logical volume is created on the LVM physical volume. The filesystem is then created on the LVM logical volume.
Notice that the gains in read performance resulting from alignment of the ext3 filesystem are gone and that read performance across all filesystems is identical. Aligning the stripe on the XFS filesystem actually made write performance worse than with no alignment attempt. LVM itself appears to be limiting block read performance, so attempts to align the filesystem are not resulting in any performance differences.
Interestingly, though, block output for XFS was greatly improved by using LVM. XFS attempts to delay writes to files for as long as possible. The delayed writes might be the reason that XFS is faster at writing with the use of LVM – if LVM maintains a write cache of its own, then perhaps data can be streamed out to disk from that cache more effectively when it is presented in large chunks by the filesystem. Notice that the read performance is significantly reduced when LVM is used.
Figure 6 shows the block rewrite bonnie++ benchmark. Here you can see that the gains in correctly aligning the ext3 filesystem have been preserved to some degree, with the aligned ext3 filesystem being almost 1MB/sec faster than the unaligned ext3 on LVM. The differences for correct alignment of ext3 are less when using LVM, however: 1MB/sec for LVM compared with about 2.5MB/sec on raw RAID.
Although differences of a megabyte per second might not seem so important, in the worst case, this means the aligned ext3 on raw RAID would be more than 15% faster than unaligned ext3 on LVM. When you consider that this is running on the same machine and same disks, 15% free performance is something you might like to have.
Because it can be somewhat difficult to compare two three-dimensional graphs by eye, I decided to graph not the two original datasets, but the difference between the aligned and non-aligned datasets. In Figure 7, I'm showing you the (ext3aligned – ext3) IOzone random read performance for LVM. Although it is not completely in favor of the ext3aligned filesystem, you can see significant performance peaks where the ext3aligned filesystem is much faster in the 256 to 4,096KB file size range (to the left of the figure).
Figure 8 shows the difference of random write performance for ext3 with and without alignment on LVM. Notice that the peaks and troughs are in the same places, although the extremes are not as pronounced. The dips in performance for the ext3align filesystem are almost eradicated.
Figure 9 shows XFS with alignment and without alignment when created on LVM for random read. Notice that the graph is much less busy for XFS compared with ext3 (shown in Figure 7). Random reads of smaller files are much faster on a properly aligned XFS filesystem, as the peaks on the left side of Figure 9 show. For smaller writes to 4MB files, an aligned XFS filesystem is slightly slower than an XFS filesystem created with the default mkfs.xfs arguments (difference graph not shown). The aligned XFS is faster for the larger writes to files of 16 to 256MB.
Digging a Deeper Hole
The Linux kernel has supported the ext4 filesystem since v2.6.28. Performance improvements over its successor, ext3, include 48-bit block addressing.
One downside of specifying the RAID stripe size at mkfs time is what happens when you add an extra disk to an existing RAID. All of a sudden, you have four data disks on your RAID 5 instead of three, and the stripe alignment that the filesystem is using is incorrect. Luckily, ext4 has a mount option to allow you to set the stripe size at mount time, so if you add a disk to a RAID or extend an LVM volume, you can tell ext4 what you've done so the filesystem can optimize future activity to fit the modified disk configuration.
Unfortunately, specifying the sunit and swidth options when mounting an XFS filesystem is only effective when the filesystem is created directly on a RAID device. So if you use LVM on your RAID and add a disk, you can't tell XFS about the change at mount time. This is once again a case in which XFS does not play well with LVM.
When using LVM, metadata is stored at various abstraction layers. For example, pvcreate has various options for setting the size of metadata and data alignment. So far, I have not found tweaking pvcreate to have the same effect as setting filesystem stripe parameters, but that question is open to further investigation.
Wrap Up
As this study shows, you can get 10% to 20% extra performance if you align your ext3 filesystem to the RAID array chunk and stripe size. Although the differences were not as significant for XFS, proper alignment of XFS can make a large difference to the performance of file creation and deletion.
In summary:
- If you are using the ext3 or ext4 filesystem, you should always use -E stripe-width=x,stride=y to tell the filesystem about your RAID configuration.
- If you are using XFS directly on the RAID device, the system should detect your RAID configuration for you automatically.
- If you are using XFS on an LVM partition created on a RAID, use -d sunit= x,swidth=y at mkfs.xfs time to tell it about your RAID configuration.
- If you are using another filesystem, check that it has detected your RAID configuration – in particular, check the stripe and chunk size – at mkfs time.
The actual benefits that you reap by taking these steps will vary depending on your configuration, but regardless of the numbers, aligning your filesystem parameters with the RAID system is a good strategy for achieving better performance, especially on ext3 and ext4 filesystems.
Infos
- OSDL's Linux RAID: http://linux-raid.osdl.org/index.php/Linux_Raid
- "A Case For Redundant Arrays of Inexpensive Disks (RAID)," by David A. Patterson, Garth Gibson, and Randy H. Katz. http://www-2.cs.cmu.edu/~garth/RAIDpaper/Patterson88.pdf
« Previous 1 2
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.
-
DebConf24 to be Held in South Korea
Busan will be the location of the latest DebConf running July 28 through August 4
-
Fedora Unleashes Atomic Desktops
Fedora has combined its solid distribution with rpm-ostree system to make it possible to deliver a new family of Fedora spins, called Fedora Atomic Desktops.
-
Bootloader Vulnerability Affects Nearly All Linux Distributions
The developers of shim have released a version to fix numerous security flaws, including one that could enable remote control execution of malicious code under certain circumstances.