ZFS RAID-Z on Linux – Guide to Data Protection and Performance

Setting up ZFS RAID-Z on Linux can significantly enhance your system’s storage capabilities by providing both reliability and scalability. Whether you are a beginner or an experienced Linux administrator, configuring ZFS RAID-Z presents a strong solution for managing large volumes of data, complete with built-in redundancy and error correction. In this guide, we will walk you through the essential steps to set up and configure ZFS RAID-Z on Linux, ensuring that you maximize the performance and efficiency of your storage system from the start.

Introduction

ZFS RAID-Z on Linux offers a highly reliable and efficient storage solution, providing strong data integrity, fault tolerance, and advanced features that traditional RAID configurations do not have. ZFS can detect and correct data corruption in real-time, ensuring your data remains safe even in the event of hardware failures. Additionally, ZFS RAID-Z reduces storage waste by implementing effective data compression and deduplication. With its seamless integration into Linux systems, ZFS is suitable for both personal and enterprise-level storage needs, offering flexible scalability, easy management, and enhanced performance.

What is the difference between RAID-Z1, RAID-Z2, and RAID-Z3?

RAID-Z1 employs single parity, RAID-Z2 employs double parity, and RAID-Z3 employs triple parity. The main difference lies in the level of data redundancy and fault tolerance, with RAID-Z3 providing the highest protection against disk failures.

Can ZFS RAID-Z be used for both data storage and backup?

Yes, ZFS RAID-Z can serve both as a high-performance data storage solution and as a backup system, thanks to its efficient snapshot and cloning capabilities that allow easy data replication and recovery.

Is ZFS RAID-Z compatible with all Linux distributions?

ZFS RAID-Z is compatible with many Linux distributions, although it may require the installation of additional packages like zfsutils-linux. It’s most commonly used on distributions such as Ubuntu, Debian, and Fedora.

How do I manage ZFS RAID-Z on Linux?

ZFS RAID-Z is managed using the zfs and zpool command-line utilities. These tools allow you to create pools, configure devices, take snapshots, and perform various administrative tasks.

Does ZFS RAID-Z on Linux support data encryption?

Yes, ZFS provides built-in encryption for datasets, enabling users to secure sensitive data at rest with strong cryptographic protection. This adds an extra layer of security to your storage system.

Whether you’re handling large-scale data or personal backups, ZFS RAID-Z is a solid and cost-effective choice for reliable storage management.

ZFS RAID-Z General Use Cases

I was using RAID-10 on my home server, but recently, I decided to switch to ZFS RAID-Z because I consider it much more versatile. You can find some more common use cases for ZFS RAID-Z below:

• Data Centers and Enterprise Storage Solutions
  • Scenario: Large organizations require high availability and fault tolerance for critical data storage.
  • Why ZFS RAID-Z?: ZFS RAID-Z offers strong redundancy with parity protection, ensuring data integrity and preventing data loss due to disk failures. It is ideal for environments that need to maintain uptime and reliability without risking data corruption.
• Backup and Archival Systems
  • Scenario: Companies or individuals need to store and protect large volumes of backup data.
  • Why ZFS RAID-Z?: ZFS’s built-in snapshots and compression capabilities make it a perfect choice for efficient, long-term data storage. Snapshots provide point-in-time copies of the data that can be easily restored, while compression helps reduce storage costs.
• Media and Content Creation
  • Scenario: Creative professionals (e.g., video editors and graphic designers) need large, fast, and reliable storage for video files and media assets.
  • Why ZFS RAID-Z?: ZFS RAID-Z offers large, scalable storage with high throughput and reliability, making it ideal for storing high-resolution video files, large databases, and media libraries. Its ability to handle big data and ensure no data corruption makes it an excellent fit for the media industry.
• Home NAS (Network Attached Storage)
  • Scenario: Tech enthusiasts or small businesses want to set up a personal storage solution for media, backups, or file sharing.
  • Why ZFS RAID-Z?: With ZFS RAID-Z, home users can easily create reliable, fault-tolerant storage systems with minimal complexity. It offers data integrity, simple expansion, and efficient storage management, making it great for a home NAS setup.
• Cloud Storage and Private Cloud Solutions
  • Scenario: Businesses or individuals looking to set up their own cloud storage systems require scalable and fault-tolerant storage.
  • Why ZFS RAID-Z?: ZFS RAID-Z’s scalability and data integrity features are well-suited for private cloud environments. Features like compression and snapshots can effectively reduce storage overhead and improve management for cloud-based storage solutions.

These use cases highlight ZFS RAID-Z’s versatility and why it’s an essential tool in various industries, from enterprise data centers to personal NAS setups.

Disk Recommendations for ZFS

Choosing the right disk types is essential when building a ZFS RAID-Z system to achieve the desired balance of performance, reliability, and cost. Below are the most suitable disk types for a ZFS RAID-Z system:

The table above can assist you in determining the best drive type for your use case. To help you make a selection, here are what I consider key factors when choosing disks for ZFS:

• Storage Capacity
  • Choose drives of the same size to maximize usable storage and avoid imbalance in the RAID-Z1 pool.
• Reliability
  • Prioritize drives with higher Mean Time Between Failures (MTBF)/Annualized Failure Rate (AFR) and workload ratings.
• Compatibility
  • Ensure firmware and rotational speeds are consistent across all drives to minimize performance bottlenecks.
• Cost vs. Performance
  • You can balance your budget with your storage needs. If you go by the book, NAS HDDs are an excellent option for budget builds; for enterprise or demanding workloads, SAS drives or SSDs are better. IMHO, this is bullshit, a managerial/marketing spin to make you spend more money. A good consumer-grade HDD/SSD will satisfy any need.

By carefully selecting the right disk type, you can build a reliable, efficient, and tailored ZFS system for your specific needs.

NVMe Drives and ZFS

Using NVMe drives in a ZFS system can deliver exceptional performance compared to traditional HDDs or SATA SSDs. However, the suitability of NVMe drives depends on your specific use case, as they are generally more expensive and may not always justify their cost for specific applications. Here’s a breakdown of when and why NVMe drives are a good choice for ZFS:

• High Performance
  • NVMe drives leverage PCIe lanes for significantly faster read/write speeds and lower latency compared to SATA SSDs or HDDs.
  • Ideal for workloads that involve frequent random reads and writes, such as virtualization, databases, or high-performance file systems.
• Reduced Latency
  • NVMe drives excel in latency-sensitive applications, making them suitable for ZFS systems handling real-time analytics or transactional workloads.
• Compact Form Factor
  • NVMe drives are small, making them perfect for space-constrained setups like rack-mounted servers.
• Advanced Features
  • Support for high queue depths and better parallelism ensures efficient handling of large numbers of simultaneous I/O operations.

An NVMe-only ZFS system can be significantly more expensive than an HDD or even SSD-based solution, especially if you require high capacity. Ensure that the cost is justified for your needs. In light of this, consider the following:

• Cost
  • NVMe drives are significantly more expensive per TB than SATA SSDs or HDDs. You can use them when performance justifies the cost.
• Write Endurance
  • Choose NVMe drives with high endurance (measured in TBW or DWPD) to handle ZFS’s metadata and write amplification.
  • Enterprise-grade NVMe drives are preferable for durability.
• Cooling
  • NVMe drives can generate considerable heat under heavy loads, so adequate cooling is essential.
• Limited Capacity
  • NVMe drives typically have smaller capacities compared to HDDs, making them less cost-effective for bulk storage.
• Controller Bottlenecks
  • Ensure your system has sufficient PCIe lanes to avoid bandwidth bottlenecks, especially in multi-drive setups.

NVMe drives can enhance the performance of ZFS, making them most suitable for high-performance environments where speed and low latency are essential. Traditional HDDs or SSDs might offer a better price-to-performance ratio for general-purpose or cost-sensitive systems. If you decide to use NVMe drives, consider integrating them as a cache layer (L2ARC or ZIL) alongside traditional drives to effectively balance cost and performance.

Build Prerequisites

To get started, you’ll need a computer with Linux installed. I will be using Debian Linux as my primary system. If you’re interested in learning how to install and configure Debian on your computer, please look at my articles below, which provide detailed instructions:

Debian 11 Server – Minimal Installation Guide

Follow this guide for a Debian 11 Server minimal installation, providing a solid foundation for any server setup or project you want to build.

Read More!

Debian 11 Server – Initial Customization Guide

Discover introductory steps to streamline performance, security, and administration in our Debian Server Initial Customization guide.

Read More!

To successfully assemble ZFS RAID-Z on Linux, I recommend you get at least five identical hard drives. You may consider adding additional drives for L2ARC and/or ZIL (SLOG), but they are very niche-specific, and I recommend using them only if you’re 100% sure you need them.

ZFS RAID-Z on Linux Configuration

Once the operating system is installed and all required hard disks are connected, you can assemble the ZFS RAID-Z1. I will assume that all your disks are empty and unformatted. If they are not, you must first erase all the data from the disks you plan to use for your RAID-Z1. You can refer to my article on how to secure erase disks on Linux for guidance below:

Secure Erase on Linux – Safely Wipe Drives for Data Security

Secure erase on Linux is crucial for data security. Learn the safest methods to wipe drives and protect sensitive information with this guide.

Read More!

Required Tools

First, let’s make sure you have all the necessary tools installed. The following command-line interface (CLI) tools are required: lsscsi, lsblk, zfsutils-linux, zfs-dkms, and zfsnap. If your disks are not empty, you may also need hdparm or dd. To check whether these commands are installed, execute the following in the console:

for tool in lsscsi util-linux zfsutils-linux zfs-dkms zfsnap zfs-zed; do echo "Checking for $tool..."; dpkg -l | grep -q "^ii.*$tool" && echo "$tool is installed." || echo "$tool is not installed."; done

There is a good chance that anything zfs will not be installed on your system, especially if you followed our prerequisite articles. To install a ZFS related packages, please execute the following in the console:

DEBIAN_FRONTEND=noninteractive apt install -y zfsutils-linux zfs-dkms zfsnap zfs-zed

Raid-Z1 Pool Setup

Before creating the pool, you must check if the system recognizes the disks you intend to use for ZFS. You can do that with the lsscsi command:

lsscsi
...
[0:0:0:0]    disk    QEMU     QEMU HARDDISK    2.5+  /dev/sda
[0:0:0:1]    disk    QEMU     QEMU HARDDISK    2.5+  /dev/sdb
[0:0:0:2]    disk    QEMU     QEMU HARDDISK    2.5+  /dev/sdc
[0:0:0:3]    disk    QEMU     QEMU HARDDISK    2.5+  /dev/sdd
[0:0:0:4]    disk    QEMU     QEMU HARDDISK    2.5+  /dev/sde
[0:0:0:5]    disk    QEMU     QEMU HARDDISK    2.5+  /dev/sdf

lsscsi command will show you all the disks recognized by the system but will not show if any are already in use. You can check that with the lsblk command:

lsblk
...
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   20G  0 disk
|-sda1   8:1    0 1023M  0 part /boot
|-sda2   8:2    0    2M  0 part
|-sda3   8:3    0  128M  0 part /boot/efi
|-sda4   8:4    0    1G  0 part [SWAP]
`-sda5   8:5    0 17.9G  0 part /
sdb      8:16   0   10G  0 disk
sdc      8:32   0   10G  0 disk
sdd      8:48   0   10G  0 disk
sde      8:64   0   10G  0 disk
sdf      8:80   0   10G  0 disk

The example above shows that the sda disk has partitions, indicating that the operating system is installed on it. Therefore, it cannot be used for ZFS pool. All the other disks (sdb – sdf) are free to use, so let’s create our RAID-Z1 pool:

zpool create -m /mnt/zfs-data data raidz /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

Yup, it is as simple as that. If you execute the zfs list now, you will notice the pool is already mounted at /mnt/zfs-data:

zfs list
...
NAME   USED  AVAIL     REFER  MOUNTPOINT
data   177K  38.3G     35.1K  /mnt/zfs-data

The size of the created pool will be the total capacity of all disks minus one, as one disk is reserved for parity. This configuration allows for the failure of only one disk without losing any data. When one drive fails, the RAID-Z1 array is considered degraded, and you cannot afford to lose another drive before the entire array is compromised, which would result in the loss of all your data. To mitigate this risk, you can opt for RAID-Z2 or even RAID-Z3. In RAID-Z2, two disks are reserved for parity, while RAID-Z3 uses three disks for parity.

RAID-Z Pool Optimization

In this section, we will discuss some options that can improve your ZFS pool performance. The first one to look at is atime.

Access Time – atime

In modern operating systems, each filesystem may have hundreds or thousands of files being read at any time. The “atime” (access time) flag requires the kernel to update the access time for each file and commit that updated time to disk with every access operation. This can be quite costly, particularly when the same files are accessed frequently. While the operating system is likely to cache the file contents, reducing the load on I/O operations, updating the access time on disk still incurs overhead.

In the example above, a RAID-Z1 pool was created with default options, which also means it was mounted with default options (rw,xattr,noacl):

mount |grep data
...
data on /mnt/zfs-data type zfs (rw,xattr,noacl)

To check if the atime flag is enabled, you can execute the following in the console:

zfs get all data | grep time
...
data  atime                 on                     default
data  relatime              off                    default

As you can observe, the atime flag is set to “on.” To disable it, you can execute the following in the console:

zfs set atime=off data

And if you check the ZFS pool and mount now, you will notice the atime is disabled:

mount |grep data
...
data on /mnt/zfs-data type zfs (rw,noatime,xattr,noacl)

zfs get all data | grep time
...
data  atime                 off                    local
data  relatime              off                    default

Disabling the atime feature is an effective way to enhance I/O performance, especially in filesystems with many small files accessed regularly.

Checksum – fletcher4

End-to-end checksums are a key feature of ZFS, and fletcher4 will be the default on most systems/setups. Checksums are used for data to metadata integrity verification and self-healing (scrub, mirror, raidz, raid-z2, raid-z3) of the volume. When the ZFS kernel module is loaded, a “microbenchmark” is executed to determine the optimal algorithm for checksums. You can observe the result of such a benchmark in the /proc/spl/kstat/zfs/ directory:

cat /proc/spl/kstat/zfs/fletcher_4_bench
...
0 0 0x01 -1 0 1794134436 59035942531399
implementation   native         byteswap
scalar           6427047419     5670993967
superscalar      9341593805     7504818105
superscalar4     8653253331     6743226916
sse2             13169208452    9066876881
ssse3            12154114828    11755687763
avx2             24998955968    17605505750
avx512f          38661028010    12326399623
avx512bw         36454245733    32353502346
fastest          avx512f        avx512bw

The default algorithm chosen for the checksum is marked “fastest,” and it depends on which optimizations (flags) your CPU has. If you are interested in those, you can check them by executing either of two commands in the console:

cat /proc/cpuinfo | grep flags
lscpu | grep -i flags

For most use cases, fletcher4 is fine unless you have a large amount of data (like 100s of terabytes) and deduplication is a concern. In that case, you may consider changing the checksum algorithm to sha256. In any case, completely turning off the checksum (to improve CPU performance) is not recommended, and I advise you not to do it.

Compression – zstd

Compression is disabled by default on ZFS. One of the main arguments against using on-the-fly compression is that it requires CPU resources during read and write operations, which can introduce latency. Ideally, we want the system to minimize interactions with the drives because they are relatively slow.

Compression can significantly reduce the number of blocks the drive needs to access, leading to faster performance at the expense of CPU time. When dealing with moderately compressible files, compression effectively increases the “areal density” of the drive platters. This means that the drive heads have to move over a smaller area to retrieve the same amount of data.

So yes, enable compression, especially if you are using mechanical drives. You can use LZ4 compression, which has a good balance between compression and decompression speed, although I recommend you consider zstd. To enable compression, execute the following in the console:

zfs set compression=zstd data

You can tune the zstd compression further by specifying the zstd level using zstd-N, where N is the integer from 1 (fastest) to 19 (best compression ratio). Otherwise, the default is zstd-3.

Optional Settings – autoexpand, autoreplace & listsnapshots

The autoexpand option ” enables ” the pool’s expansion when a smaller disk is replaced with a larger one. These features will allow you to increase the pool size without exporting and importing the pool or rebooting the system. You can enable the option by executing the following in the console:

zpool set autoexpand=on data

The autoreplace option governs the automatic replacement of devices. When enabled, any new device detected in the same physical location as a previously belonging device in the pool is automatically formatted and replaced. If not enabled,  device replacement must be initiated using the zpool replace command (i.e., in case of disk failure). You can enable the option by executing the following in the console:

zpool set autoreplace=on data

The listsnapshots option will display available snapshots when executing the zfs list command. It is disabled by default and you can enable it by executing the following in the console:

zpool set listsnapshots=on data

Otherwise, you must use the zfs list -t filesystem,snapshot command to display the snapshots. I will discuss snapshots more later in this guide.

Optional Features for ZFS RAID-Z on Linux

Above in this guide, I mentioned L2ARC and ZIL (SLOG) and how you can consider adding additional drives to your computer. I also mentioned that they are very niche-specific and that you should be absolutely sure you need them. If, in any case, you do need them, I will show you here how you can do it.

Adding L2ARC Drive

ZFS includes several features designed to enhance performance for data read operations that are frequently accessed. One key feature is the Adaptive Replacement Cache (ARC), which utilizes the server’s RAM. When a system receives read requests, ZFS uses the ARC to fulfill those requests quickly. Once the ARC reaches its maximum capacity, ZFS allocates L2ARC drives within a ZFS pool (if available) to handle the read requests that exceed the ARC’s capacity. This reduces the use of slower hard drives and, therefore, increases system performance.

To add an L2ARC disk to your ZFS pool, you’ll need to make sure your system has a spare disk already available. Once you ensure that, you can add an L2ARC disk by executing the following in the console:

zpool add data cache /dev/sdg

You can observe the “cache” disk is added by executing the zpool status command:

zpool status
...
  pool: data
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sdf     ONLINE       0     0     0
            sde     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
        cache
          sdg       ONLINE       0     0     0

errors: No known data errors

Please consider the following factors before adding the L2ARC disk to your pool. Maintaining an L2ARC incurs overhead in the primary ARC, which may lead to decreased performance on low-memory servers. In such cases, allocating additional memory to the primary ARC could be more beneficial than using the L2ARC. This is likely why some experts recommend only adding an L2ARC to servers with at least 32GB to 64GB of RAM. Additionally, larger L2ARC disks require more memory to manage effectively.

Redundancy is also a factor here, so it is advisable not to use a single device for L2ARC. Sure, if the L2ARC device fails, you will lose whatever is currently on it without compromising the data pool, but better be safe than sorry and use redundant devices (i.e., zpool add data cache mirror /dev/sdg /dev/sdh).

If you decide to use L2ARC, ensure your disk is much faster than the existing data storage drives. I would consider enterprise-grade NVMe drives.

Adding SLOG Drive

Another ZFS performance feature is the ZFS intent log (ZIL). The ZIL writes synchronous transactions to disk in special pre-allocated space so ZFS can confirm that data is on non-volatile storage, providing lower latency for synchronous writes. The data is then written to disk in a normal transaction group (TXG), which allows the ZIL space to be reused for further writing.

You can enhance ZIL performance by utilizing a dedicated virtual device (vdev) known as a Separate Intent Log (SLOG). A SLOG moves the ZIL to dedicated SSDs rather than using a portion of the data disks for its operation. This configuration can lead to significant improvements due to SSDs’ lower latency compared to traditional data disks.

To add an SLOG disk to your ZFS pool, you must ensure your system has an already available spare disk. Once you ensure that, you can add an SLOG disk by executing the following in the console:

zpool add data log /dev/sdh

Adding an SLOG device might benefit the storage environment with heavy synchronous writes. If you think you should use an SLOG drive, also make sure it is much faster than existing data storage drives. The drive doesn’t have to be big, and usually devices in the 16GB to 32GB range will suffice. As with L2ARC, consider the redundancy here also (i.e., zpool add data log mirror /dev/sdh /dev/sdi).

ZFS Maintenance

In this section, I will explain how to monitor your ZFS pool for degraded or over-capacity volumes and disk errors. Additionally, I will demonstrate how to create regular snapshots of the ZFS pool.

Monitoring

ZFS includes a useful feature known as the ZFS Event Daemon (ZED). This daemon monitors ZFS events in real-time and responds to specific conditions or issues within the ZFS storage system. As part of the ZFS on Linux ecosystem, ZED plays a vital role in automating and managing ZFS pools and datasets by reacting to various events and errors. Below, you can observe some key features of ZFS ZED:

• Event Monitoring
  • ZED listens for ZFS events generated by the ZFS Event Logging System (ZFS Event Stream or ZEVENTS).
  • It captures events like disk failures, pool errors, scrub completion, or resilver progress.
• Automated Responses
  • Based on the type of event, ZED can trigger pre-configured actions, such as sending notifications, replacing failed devices, or generating logs for further analysis.
• Notifications
  • Sends email alerts or logs messages to syslog/journald for critical events (e.g., disk failures or pool degradation).
• Diagnostics and Logging
  • Helps with proactive diagnostics by logging detailed event data, allowing administrators to troubleshoot and take corrective actions quickly.
• Hot Spare Management
  • Automatically replaces a failed disk with a hot spare, if configured.
• Customizable Event Handlers
  • Allows administrators to define custom actions by editing ZED’s event configuration files.

ZED will handle common events such as disk failures, checksum errors, resilvering and scrub events, and pool state changes. ZFS ZED is configured using the /etc/zfs/zed.d/ directory, which contains a series of event handler scripts and configuration files. These scripts determine how ZED responds to specific ZFS events.

The main configuration file can be found at /etc/zfs/zed.d/zed.rc, where you can set various options, including the email address to which event/alert notifications will be sent. Additionally, you can create or modify scripts to define custom responses to events in /etc/zfs/zed.d/ directory.

The service should be running and enabled, and you can check the status by executing the following in the console:

systemctl status zfs-zed.service

ZFS ZED is an indispensable tool for ZFS administrators. It helps automate responses to storage events and provides timely notifications. Properly configuring ZED can improve the reliability, maintainability, and overall performance of ZFS pools in production environments.

Snapshots

ZFS snapshots are read-only copies of a file system or volume. They can be created in seconds and initially do not consume any additional disk space within the pool. As data is added or deleted, the active volume changes, causing the snapshot to consume disk space by continuing to reference the old backup copies of the data. ZFS snapshots provide an efficient way to create multiple time-stamped copies of an entire file system without requiring double the storage space.

ZFS can create snapshots with the zfSnap script, which comes with the zfsnap package. While you can create snapshots manually by executing the zfSnap script, the best way to do it is to create a cron job. To do so, execute the following in the console:

crontab -e

Once the file is open, paste the following content at the bottom of the file:

0 3 * * 0 /usr/sbin/zfSnap -d -s -S -a 1m data

In the example above, a snapshot will be created every Sunday at 3 AM, and it is set to expire in one month. You can also execute the same command in the console whenever you want. To list the snapshots, you can execute the following and observe the results:

zfs list -t filesystem,snapshot
...
NAME                           USED  AVAIL     REFER  MOUNTPOINT
data                           685K  38.3G     35.1K  /mnt/zfs-data
data@2025-01-20_10.50.17--1m     0B      -     35.1K  -

As you can see from the example, a snapshot is created for the data volume, followed by the date and time of creation (after @), and it is set to expire in one month (–1m). In a month, you will notice four complete snapshots of your data.

If you need to roll back and restore the snapshot, you can do that by executing the following:

zfs rollback data@2025-01-20_10.50.17--1m

ZFS RAID-Z on Linux – Videos

Below, you will find complementary videos on YouTube and Asciinema. The YouTube video demonstrates the whole ZFS RAID-Z1 pool setup and all the commands used. On Asciinema, there is the same video; only you can pause it and copy/paste the commands used directly from the video.

YouTube

Asciinema

Conclusion

ZFS RAID-Z on Linux offers a powerful, reliable, and feature-rich solution for managing and protecting your data. With advanced capabilities like data integrity verification, efficient storage utilization, and robust fault tolerance, it’s an excellent choice for personal and professional storage needs. Whether you’re building a home server or managing enterprise-level storage, ZFS RAID-Z ensures your data remains secure and accessible. By understanding its setup and operations, you can harness its full potential and enjoy a worry-free storage experience.