ZFS Filesystem Best Practices#

• Author: Grzegorz "SinusPL" Surmann
• Email: zfs [at] funil [dot] de
• Since: June 2011

This guide is a practical ZFS quick reference. It focuses on everyday administration, safe defaults, common pool layouts, snapshots, backups, monitoring, recovery, and examples you can adapt.

Naming convention used in every example:

• Pools are named pool0, pool1, pool2, and so on.
• Filesystem datasets are named volume0, volume1, volume2, and so on.
• Block devices created with zfs create -V are also called volumes here, but the examples explicitly call them zvols when they are block devices.
• Disk identifiers are examples only. Prefer stable paths such as /dev/disk/by-id/... instead of /dev/sdX.

Warning: Many ZFS commands can destroy data. Read every command before using it, replace the device names with the correct ones, and keep tested backups.

ZFS is a copy-on-write storage system that combines a filesystem, volume manager, software RAID, checksumming, snapshots, replication, compression, quotas, and optional native encryption into one coherent design. Its main strength is data integrity: every block is checksummed, redundant pools can repair bad copies automatically, snapshots make point-in-time recovery cheap, and zfs send / zfs receive can replicate exact dataset history to another pool or host. ZFS also makes everyday administration cleaner because storage is organized into pools, vdevs, datasets, zvols, and properties instead of separate RAID, partition, LVM, and filesystem layers. Its limits matter just as much: ZFS is not a backup; it cannot save data after too many devices in the same vdev fail, make an unsafe topology safe after the fact, protect against bad commands or destroyed snapshots, or guarantee portability after newer pool features are enabled. Good ZFS systems are planned around redundancy, stable disk identifiers, free space, regular scrubs, tested backups, and datasets designed for the workload.

Table Of Contents#

• Core Ideas
• Platform Notes
• Platform Appendices
  • Linux And OpenZFS Packages
  • Debian
  • Ubuntu
  • FreeBSD
  • TrueNAS
  • Proxmox VE
• What ZFS Is Not
• Golden Rules
• Command Risk Matrix
• Recommended Pool Layouts
  • Choosing Mirror Or RAIDZ
  • Mirror Vdevs
  • Convert A Single-Disk Pool To A Mirror
  • RAIDZ1
  • RAIDZ2
  • RAIDZ3
  • What Not To Do
  • Practical Capacity Math
• New Disk Burn-In And Acceptance Test
• Pool Creation Checklist
• What To Document After Creating A Pool
• Dataset Design
• Common Dataset Properties
  • Property Glossary
  • Property Inheritance
  • Compression
  • Access Time
  • Extended Attributes And ACLs
  • Record Size
  • Quotas And Reservations
  • Mountpoints
• Everyday Commands
• Snapshots
  • Snapshot Holds
  • Compare Snapshots With zfs diff
• Snapshot Retention
• Pool Checkpoints
• Backups With ZFS Send And Receive
  • Replication Safety
  • Resume Interrupted Replication
  • ZFS Bookmarks For Replication
  • Inspect Send Streams With zstreamdump
  • Redacted Sends
• Scrubs
• Pool Initialization
• Disk Replacement
  • When To Replace A Disk
  • Hot Spares
• Expanding Pools
  • Add Another Mirror Vdev
  • Add A RAIDZ Vdev
  • Expand An Existing RAIDZ Vdev
• Import And Export
  • Import Using A Specific Device Directory
  • Import By Pool ID Or Temporary Name
  • Hostid And Active Pool Import Warnings
• Pool Feature Flags And Compatibility
• Encryption
  • Encrypted Dataset Recovery Checklist
• Zvols
• Virtual Machines
• Databases
• Media And Archive Storage
• Shares
• Delegation
• ZFS For Containers
• Cache, Log, And Special Vdevs
  • Special Small Blocks On Special Vdevs
• TRIM
• Monitoring
  • Command History For Auditing
  • Reading zpool status
  • Annotated zpool status Examples
  • Interpreting ZFS Errors
  • Events And Alerting
• SMART Checks
• Automation Examples
• Performance Basics
  • Performance Troubleshooting Decision Tree
• Security And Permissions
• Boot Pools
• Disaster Recovery
  • Recovery Decision Tree
  • Symptom-Based Recovery Runbook
  • Minimum Safe Recovery Bundle
  • Incident Evidence Form
  • Failed Pool Recovery Triage
  • Clone Failing Disks Before Recovery Attempts
  • Physical Pool Inspection With zdb
  • Pool Rewind Recovery With -F
  • Recover A Destroyed Pool Entry
  • Handling Permanent Data Errors
  • Ransomware Or Mass Deletion Recovery
  • Recovery Practice Lab
  • Restore Drill Templates
• Community FAQ: Top 20 Recurring ZFS Questions
• Common Mistakes
• Example Build: General Home Or Small Server
• Example Build: Backup Pool
• Example Build: VM Pool
• Quick Reference
• Maintenance Schedule
• Final Best Practices Checklist
• References
• Disclaimer

Core Ideas#

ZFS combines a volume manager and a filesystem. Instead of creating a hardware RAID array and then putting a filesystem on top, you normally give ZFS direct access to the disks and let it manage redundancy, checksums, repair, snapshots, compression, and replication.

Important terms:

• Pool: top-level storage container, for example pool0.
• Vdev: one redundancy group inside a pool, such as one mirror or one RAIDZ2 group.
• Dataset: a ZFS filesystem, for example pool0/volume0.
• Zvol: a ZFS block device, for example pool0/volume2.
• Snapshot: read-only point-in-time copy, for example pool0/volume0@daily-2026-05-14.
• Clone: writable dataset based on a snapshot.
• Scrub: online checksum verification and repair.
• Resilver: rebuild after replacing or adding redundancy to a device.

Platform Notes#

This guide is OpenZFS-focused. Command syntax is mostly portable, but defaults and integration details vary by operating system and appliance.

Check your platform documentation for:

• Package names and service names.
• Default mount behavior and dataset mount ordering.
• ACL support, especially acltype, aclmode, and SMB/NFS integration.
• Bootloader support before enabling new pool feature flags on boot pools.
• Location of compatibility profiles and cache files.
• Native encryption support and key-loading service behavior.
• Distribution-specific root-on-ZFS tooling and boot environment workflows.

Examples in this guide use Linux-style paths such as /dev/disk/by-id/. On FreeBSD, illumos, appliances, or rescue media, the stable device directory and service commands may differ.

Platform Appendices#

These notes are deliberately conservative. Use them to avoid the most common cross-platform mistakes, then verify exact commands against your operating system or appliance documentation.

Linux And OpenZFS Packages#

Linux systems usually provide OpenZFS as distribution packages, DKMS modules, or prebuilt kernel modules. Service names, initramfs integration, boot pool support, and default timers vary by distribution.

Operational notes:

• Use /dev/disk/by-id/ or another stable persistent path for pool creation and imports.
• Check the distribution's ZFS package documentation before enabling new feature flags on a boot pool.
• After changing root-on-ZFS, encryption, or boot pool properties, rebuild the initramfs or boot configuration if your distribution requires it.
• On systemd-based hosts, check whether packaged scrub, trim, import, mount, and zed units already exist before creating custom units.
• Keep kernel and ZFS packages compatible; avoid rebooting into a kernel that has no matching ZFS module.

Useful checks:

zfs version
modinfo zfs
systemctl list-unit-files 'zfs*'
systemctl status zfs-import-cache.service
systemctl status zfs-mount.service
systemctl status zfs-zed.service

Debian#

Debian deserves separate attention because ZFS packages are normally installed from contrib, and newer OpenZFS versions may come from backports. Kernel headers, DKMS, and package pinning matter more than on distributions that ship prebuilt ZFS modules for the running kernel.

Operational notes:

• Enable the Debian repository components required by your release, normally including contrib, before installing ZFS packages.
• Consider Debian backports when you need a newer OpenZFS release, but pin deliberately and document why the backport is used.
• Install matching kernel headers before or with zfs-dkms; a kernel without matching headers can leave the ZFS module unbuilt after upgrades.
• Keep at least one known-working kernel installed until the new kernel boots and loads ZFS successfully.
• Root-on-ZFS on Debian should follow a current Debian-specific root-on-ZFS guide, not a generic Linux guide.
• After kernel, initramfs, bootloader, encryption, or root dataset changes, rebuild and test the boot path before treating the host as recovered.

Useful checks:

apt-cache policy zfsutils-linux zfs-dkms
apt-cache policy linux-image-amd64 linux-headers-amd64
dkms status
modprobe zfs
zfs version
systemctl list-unit-files 'zfs*'

Example package install for a normal non-root pool, after repositories are correct for your Debian release:

apt update
apt install linux-headers-amd64 zfs-dkms zfsutils-linux

Example backports sanity check, replacing trixie-backports with your actual release backports suite:

apt-cache policy -t trixie-backports zfsutils-linux zfs-dkms

Best practice:

• Do not mix random third-party ZFS packages with Debian packages unless you have a clear rollback plan.
• Check dkms status after every kernel upgrade.
• Before rebooting a remote Debian ZFS host, confirm that the new kernel has a built ZFS module and that an older working kernel remains available.
• For rescue media, verify that it can import the pool feature set used by the Debian host.

Ubuntu#

Ubuntu also uses zfsutils-linux, but its packaging and kernel integration differ from Debian. OpenZFS documentation notes that Ubuntu includes ZFS support in its default Linux kernel packages, while the command-line tools are installed with zfsutils-linux after the required repository component, commonly universe, is available.

Operational notes:

• Enable universe if zfsutils-linux is not visible to APT.
• Install zfsutils-linux for command-line administration tools.
• Do not assume Debian DKMS instructions apply directly to Ubuntu; check whether the running Ubuntu kernel already provides the ZFS module package path for that release.
• For root-on-ZFS, follow Ubuntu release-specific guidance and verify bootloader, initramfs, encrypted root, and feature flag compatibility before upgrades.
• Ubuntu desktop/server releases, HWE kernels, and cloud images can differ; check the running kernel and package source before troubleshooting module load failures.
• If Secure Boot or custom kernels are involved, verify module loading and signing behavior before relying on ZFS at boot.

Useful checks:

apt-cache policy zfsutils-linux
apt-cache policy linux-generic linux-image-generic
modinfo zfs
zfs version
systemctl list-unit-files 'zfs*'
systemctl status zfs-import-cache.service
systemctl status zfs-zed.service

Install tools after repositories are correct:

apt update
apt install zfsutils-linux

Best practice:

• Keep release upgrades conservative on boot pools and test rescue media before enabling new pool features.
• Confirm zpool import behavior after major Ubuntu upgrades, especially on encrypted, root-on-ZFS, or multi-pool hosts.
• Check Ubuntu package versions before copying commands between LTS releases.
• Treat Ubuntu installer-created layouts as release-specific; document dataset names, boot pool layout, and recovery commands immediately after installation.

FreeBSD#

FreeBSD integrates ZFS deeply, including root-on-ZFS and boot environments on many installations. Device names, service management, loader behavior, and periodic jobs differ from Linux examples.

Operational notes:

• Expect disk paths such as /dev/gpt/..., /dev/diskid/..., /dev/ada0, /dev/da0, or GEOM labels instead of Linux /dev/disk/by-id/....
• Use FreeBSD boot environment tooling where available instead of hand-rolling root dataset rollback.
• Check loader and bootcode support before enabling new feature flags on boot pools.
• Review FreeBSD periodic and service configuration before adding duplicate scrub or snapshot automation.
• Use FreeBSD ACL and jail documentation when mixing ZFS datasets with jails, SMB, or NFS exports.

Useful checks:

zpool status -gLPv pool0
zpool get bootfs pool0
zfs list -o name,mountpoint,canmount
service zfs onestatus
sysrc zfs_enable

TrueNAS#

TrueNAS is an appliance platform. It exposes ZFS through a web interface, middleware, alerts, tasks, replication jobs, sharing services, and API workflows. Shell commands are useful for inspection, but persistent configuration should normally be changed through the UI or supported API.

Operational notes:

• Prefer the TrueNAS UI or API for pools, datasets, shares, snapshots, replication tasks, alerts, scrub schedules, and SMART tests.
• Do not manually edit middleware-managed configuration files unless TrueNAS documentation or support tells you to.
• Use shell commands for read-only investigation, evidence capture, and support bundles.
• Be careful importing pools created elsewhere; feature flags, encryption, ACLs, and share metadata may not match the appliance version.
• For recovery, export shares or stop clients first, then use read-only inspection before changing pool state.

Useful checks:

zpool status -gLPv pool0
zpool events -v
zfs list -r pool0
zfs list -t snapshot -r pool0

Proxmox VE#

Proxmox VE often uses ZFS for host storage, VM disks, container root filesystems, replication, and sometimes root-on-ZFS. Proxmox also tracks storage definitions and guest disk ownership outside plain ZFS metadata.

Operational notes:

• Prefer Proxmox storage configuration tools, UI, and documented CLI workflows for VM and container storage.
• Do not rename, destroy, or receive over VM backing datasets or zvols without checking Proxmox storage metadata and guest configuration.
• For VM storage, mirrors usually give better latency and rebuild behavior than wide RAIDZ.
• Check bootloader and Proxmox release notes before upgrading feature flags on root pools.
• Coordinate ZFS snapshots with guest-aware backups when application consistency matters.

Useful checks:

zpool status -gLPv pool0
zfs list -t filesystem,volume -r pool0
pvesm status
qm config 100
pct config 100

What ZFS Is Not#

ZFS is strong storage software, but it is not magic.

• ZFS is not a backup. Snapshots in the same pool can disappear with the pool.
• ZFS is not protection against every admin mistake. A valid destructive command can destroy valid data.
• ZFS is not a substitute for SMART monitoring, cabling checks, cooling, or replacing failing disks.
• ZFS is not a way to make a weak vdev topology safe after the fact.
• ZFS is not guaranteed portable after enabling new feature flags.
• ZFS is not recovery from losing encryption keys.
• ZFS is not immune to ransomware if the attacker can destroy snapshots or reach backup targets.

Golden Rules#

1. Use ECC RAM when possible, especially for large pools or important data.
2. Use reliable disks and monitor them. ZFS protects against many failures, not against neglect.
3. Never build ZFS on top of hardware RAID. Use HBA or IT mode so ZFS can see each disk directly.
4. Use stable disk paths from /dev/disk/by-id/.
5. Plan vdev width and redundancy before creating the pool. You cannot remove a RAIDZ vdev from a pool in normal designs.
6. Do not fill pools. Keep at least 20% free space for performance and recovery room.
7. Use snapshots, but do not treat snapshots as backups.
8. Back up to another pool, system, disk, or remote host with zfs send.
9. Scrub regularly and read the results.
10. Test restores, not only backups.
11. Set properties at creation time when possible.
12. Keep pools, operating system packages, and boot environments maintained.
13. Do not casually run zpool upgrade if the pool must remain importable on older systems.
14. Avoid mixing very different disk sizes or speeds inside one vdev.
15. Replace failing disks early. Redundancy is not a reason to wait.

Command Risk Matrix#

Use this matrix before copying commands from notes, forums, old shell history, or AI output. The safest ZFS workflow is usually inspect, document, decide, then change.

Practical rule: if a command changes topology, destroys data, rewinds transactions, overwrites a receive target, clears evidence, or changes feature compatibility, stop and write down exactly what you expect to happen before running it.

Recommended Pool Layouts#

Choosing Mirror Or RAIDZ#

Use this table as a starting point, then adjust for budget, recovery time, capacity, and workload.

Avoid RAIDZ1 for new pools with large disks or important data. It can be acceptable for disposable data, scratch pools, or tiny lab systems, but it is not a conservative default.

Mirror Vdevs#

Mirrors are usually the best general-purpose layout for home labs, small servers, VM storage, databases, and workloads with random I/O.

Example: create pool0 from two mirrored disks.

zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -O xattr=sa \
  -O acltype=posixacl \
  -m /pool0 \
  pool0 \
  mirror \
  /dev/disk/by-id/disk0 \
  /dev/disk/by-id/disk1

Example: create pool0 from two mirror vdevs.

zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool0 \
  pool0 \
  mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1 \
  mirror /dev/disk/by-id/disk2 /dev/disk/by-id/disk3

Why use mirrors:

• Fast rebuilds compared with wide RAIDZ.
• Good random read and write performance.
• Easy expansion by adding another mirror vdev.
• Simpler replacement and growth strategy.

Tradeoff:

• A two-way mirror gives 50% usable capacity.

Convert A Single-Disk Pool To A Mirror#

If pool0 was created from one disk, you can attach a second disk to turn the plain single-disk vdev into a mirror. This is useful when you started without redundancy and want to add it later.

Check the current device name:

zpool status pool0

Attach a second disk:

zpool attach pool0 /dev/disk/by-id/disk0 /dev/disk/by-id/disk1

Watch the resilver:

zpool status pool0

After the resilver completes, pool0 has mirror redundancy.

Example before:

pool0
  /dev/disk/by-id/disk0

Example after:

pool0
  mirror-0
    /dev/disk/by-id/disk0
    /dev/disk/by-id/disk1

If zpool status shows a short device name because the pool was imported that way, use the name shown there:

zpool attach pool0 sdb /dev/disk/by-id/disk1

Best practice:

• Back up first. The original single disk is still a single point of failure until the resilver completes.
• Use a new disk that is at least as large as the existing disk.
• Use stable /dev/disk/by-id/ names for the new disk when possible.
• Do not confuse attach with add. attach mirrors an existing vdev; add creates another top-level vdev.

RAIDZ1#

RAIDZ1 is single-parity RAIDZ. It is usually only acceptable for small, non-critical pools with small disks and good backups.

Example:

zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool0 \
  pool0 \
  raidz1 \
  /dev/disk/by-id/disk0 \
  /dev/disk/by-id/disk1 \
  /dev/disk/by-id/disk2

Best practice:

• Prefer mirrors or RAIDZ2 for important data.
• Avoid RAIDZ1 with large modern disks when the data matters.

RAIDZ2#

RAIDZ2 uses two parity disks per vdev. It is a good choice for larger media, archive, and backup pools.

Example:

zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool0 \
  pool0 \
  raidz2 \
  /dev/disk/by-id/disk0 \
  /dev/disk/by-id/disk1 \
  /dev/disk/by-id/disk2 \
  /dev/disk/by-id/disk3 \
  /dev/disk/by-id/disk4 \
  /dev/disk/by-id/disk5

Best practice:

• Use RAIDZ2 instead of RAIDZ1 for valuable data on large disks.
• Keep vdevs at reasonable widths. Common widths are 6, 8, 10, or 12 disks.
• Expansion normally means adding another RAIDZ2 vdev of similar width.

RAIDZ3#

RAIDZ3 uses three parity disks per vdev. It is suitable for large, slower, high-capacity archive pools where rebuild time is long and capacity matters more than write performance.

Example:

zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool0 \
  pool0 \
  raidz3 \
  /dev/disk/by-id/disk0 \
  /dev/disk/by-id/disk1 \
  /dev/disk/by-id/disk2 \
  /dev/disk/by-id/disk3 \
  /dev/disk/by-id/disk4 \
  /dev/disk/by-id/disk5 \
  /dev/disk/by-id/disk6 \
  /dev/disk/by-id/disk7

What Not To Do#

Avoid these layouts unless you fully understand the risk:

# No redundancy. Any disk failure can destroy the pool.
zpool create pool0 /dev/disk/by-id/disk0 /dev/disk/by-id/disk1

# Hardware RAID hides disks and errors from ZFS.
zpool create pool0 /dev/disk/by-id/hardware-raid-volume0

# Unstable device names can change after reboot.
zpool create pool0 /dev/sdb /dev/sdc

Practical Capacity Math#

ZFS capacity is easier to plan when you separate raw disk capacity, vdev redundancy, filesystem overhead, snapshots, reservations, and free-space headroom. Vendor disk sizes are decimal TB, while many tools display binary TiB, so a "12 TB" disk appears as about 10.9 TiB before ZFS overhead.

Rough usable capacity before metadata, snapshots, compression, and free-space headroom:

Example rough math:

6 x 12 TB RAIDZ2 raw vendor size: 72 TB
Parity equivalent: 2 disks
Rough usable before overhead: 4 x 12 TB = 48 TB
Displayed binary capacity before overhead: about 43.7 TiB
Practical planning target at 80% full: about 35 TiB used

Why real numbers differ:

• ZFS stores metadata, checksums, labels, spacemaps, and indirect blocks.
• RAIDZ padding and small-block workloads can reduce efficiency.
• Snapshots keep old blocks alive after files are modified or deleted.
• Quotas, reservations, and zvol refreservations can make free space look lower than expected.
• Compression can make logical data larger than physical pool usage.
• Keeping free space is intentional; performance and recovery get worse when a pool is nearly full.

Useful checks:

zpool list -o name,size,alloc,free,capacity,fragmentation,health
zfs list -o name,used,avail,refer,usedbysnapshots,usedbydataset,usedbychildren,usedbyrefreservation
zfs get logicalused,used,compressratio pool0/volume0

Best practice:

• Plan around 70% to 80% normal usage, not theoretical maximum capacity.
• Count snapshot retention in capacity planning.
• Do not mix very small and very large disks inside one vdev unless the lost capacity is acceptable.
• For backup pools, plan capacity for both current data and retained history.

New Disk Burn-In And Acceptance Test#

Test new or reused disks before trusting them with a pool. Burn-in does not prove a disk is perfect, but it often catches shipping damage, weak sectors, bad cables, overheating, and wrong device identification before the pool contains data.

Start with inventory and SMART data:

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0

Run a short SMART test:

smartctl -t short /dev/disk/by-id/disk0
smartctl -a /dev/disk/by-id/disk0

Run a long SMART test:

smartctl -t long /dev/disk/by-id/disk0
smartctl -a /dev/disk/by-id/disk0

For a destructive write/read test on a new empty disk, use a tool appropriate for your platform. Replace the device name only after verifying it several times.

Warning: the next example destroys data on the target device. Use it only on a new or intentionally wiped disk.

badblocks -wsv /dev/disk/by-id/disk0

For SSDs or production-like tests, a workload tool such as fio is often more useful than badblocks. Keep the test destructive only when the disk is empty and intentionally selected.

Acceptance checklist:

• Verify model, serial, size, and stable /dev/disk/by-id/ name.
• Check SMART before and after stress testing.
• Watch temperature during long tests.
• Replace suspect cables, trays, HBAs, or power paths before blaming ZFS.
• Reject disks with new reallocated, pending, or uncorrectable sectors.
• Record disk serials and intended vdev membership before zpool create.

Pool Creation Checklist#

Before creating a pool:

• Confirm the disks with lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE.
• Use /dev/disk/by-id/ paths.
• Decide mirror, RAIDZ2, or RAIDZ3 before writing data.
• Use ashift=12 for most modern disks and SSDs.
• Use compression=zstd on current OpenZFS unless you have a specific reason not to.
• Use atime=off for most server datasets on both HDDs and SSDs.
• Use acltype=posixacl and xattr=sa on Linux when POSIX ACLs or many extended attributes are needed.
• Decide whether the pool needs encryption.
• Decide mountpoints before creating many datasets.

Preview disk identifiers:

ls -l /dev/disk/by-id/

Show block devices:

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,MOUNTPOINTS

Check existing pools:

zpool list
zpool status

Before running zpool create:

• Confirm every device path points to the intended disk.
• Confirm the disks are not mounted and are not part of another active pool.
• Confirm backups exist if the disks ever held data.
• Confirm the vdev layout is the one you intend to live with.
• Save the exact command in your notes before pressing Enter.

Warning: zpool create writes labels and creates a new pool on the selected devices. A wrong device path can destroy data.

What To Document After Creating A Pool#

Write down the pool design while the details are still fresh. This makes later disk replacement, import, recovery, and audits much easier.

Record:

• Pool name, creation date, host, and operating system.
• OpenZFS version and package source.
• Pool layout, vdev type, vdev width, and redundancy level.
• Disk model, serial, size, and /dev/disk/by-id/ path for every device.
• ashift, pool compatibility setting, and enabled feature policy.
• Dataset tree, mountpoints, key properties, and snapshot policy.
• Encryption roots, key format, and offline key recovery location.
• Backup target, replication command, and retention policy.
• Rescue media version known to import the pool.
• Expected import command for recovery, preferably read-only and no-mount.

Useful capture commands:

zpool status -gLPv pool0
zpool get all pool0
zfs get -r all pool0
zpool history -l pool0

Dataset Design#

Do not put everything directly in the root of the pool. Create datasets for different data types so you can apply different snapshots, quotas, compression, record sizes, mountpoints, and backup policies.

Example dataset layout:

pool0
pool0/volume0   # general files
pool0/volume1   # user files
pool0/volume2   # media or archives
pool0/volume3   # virtual machines
pool0/volume4   # databases
pool0/volume5   # backups

Create filesystem datasets:

zfs create -o mountpoint=/pool0/volume0 pool0/volume0
zfs create -o mountpoint=/pool0/volume1 pool0/volume1
zfs create -o mountpoint=/pool0/volume2 pool0/volume2

List datasets:

zfs list

Show important properties:

zfs get compression,atime,recordsize,mountpoint,quota,reservation pool0/volume0

Common Dataset Properties#

Property Glossary#

These are the properties and settings most often mentioned in this guide.

Property Inheritance#

Most editable ZFS dataset properties inherit from the parent dataset unless a child dataset overrides them. This is useful for setting broad defaults at the pool or parent level, then changing only the datasets that need different behavior.

Set defaults on pool0:

zfs set compression=zstd pool0
zfs set atime=off pool0

Override one child dataset:

zfs set recordsize=16K pool0/volume4

Show property values and where they came from:

zfs get -r -o name,property,value,source compression,atime,recordsize pool0

Show only local overrides:

zfs get -r -s local all pool0

Remove a local override so the dataset inherits again:

zfs inherit compression pool0/volume0
zfs inherit -r atime pool0/volume1

Received properties from zfs receive can also affect behavior. To inspect them, include the received column:

zfs get -o name,property,value,received,source compression,mountpoint pool1/volume0

Best practice:

• Set common defaults on parent datasets.
• Use local overrides only when a workload needs them.
• Review source when a property does not behave as expected.
• Be careful with inherited mountpoint, sharenfs, sharesmb, quota, and reservation; they can affect many children at once.

Compression#

Use Zstandard compression by default on current OpenZFS systems. zstd is a good modern default because it usually compresses better than lz4 while still offering good performance; in OpenZFS, zstd is equivalent to zstd-3.

zfs set compression=zstd pool0
zfs set compression=zstd pool0/volume0

Use lz4 when you need the lowest CPU overhead, have older systems that must import the pool, or have latency-sensitive VM/database workloads where testing shows zstd costs too much CPU.

zfs set compression=lz4 pool0/volume3

Use stronger Zstandard levels for cold or archival datasets. Higher levels can save more space, but they cost more CPU during writes.

zfs set compression=zstd-6 pool0/volume2

Check compression:

zfs get compressratio,compression pool0/volume2

Common compression choices:

Best practice:

• Set compression before writing data.
• Changing compression affects newly written blocks only.
• To recompress old data, rewrite it or replicate it to a new dataset.
• Already-compressed media, backups, and archives may not shrink much, but ZFS will store blocks uncompressed when compression is not useful.
• Check OpenZFS feature compatibility before using zstd on pools that must be imported by older systems.

Access Time#

Disable access time updates for most server datasets. In ZFS the property is atime=off; this is the ZFS equivalent of a noatime mount option.

zfs set atime=off pool0

Why this matters: with atime=on, reading a file can also update metadata to record the access time. On HDD pools this can turn read-heavy workloads into extra random writes and seeks. On SSD pools the latency cost is usually smaller, but disabling unnecessary access-time writes still reduces metadata churn and write amplification.

If an application needs approximate access-time behavior, use atime=on with relatime=on:

zfs set atime=on pool0/volume0
zfs set relatime=on pool0/volume0

If an application really needs every read to update access time, use relatime=off, but expect more write activity:

zfs set atime=on pool0/volume0
zfs set relatime=off pool0/volume0

Best practice:

• Use atime=off for most file servers, backup datasets, media datasets, VM image datasets, and databases.
• Keep atime=on only for software that actually uses access times, such as some mail, archival, audit, or cleanup workflows.
• Do not describe this as an SSD-only or HDD-only setting. HDDs benefit more visibly, but SSDs also avoid unnecessary writes.

Extended Attributes And ACLs#

The ZFS property is xattr, not xattrs. Extended attributes are used by POSIX ACLs, SELinux labels, Samba metadata, desktop file managers, container runtimes, and some backup tools.

On Linux, a common modern setting for datasets that use POSIX ACLs or many extended attributes is:

zfs set acltype=posixacl pool0/volume0
zfs set xattr=sa pool0/volume0

xattr=sa stores extended attributes as ZFS system attributes instead of placing them in hidden directory-style objects. This can reduce extra metadata I/O, especially on HDD pools where seeks are expensive. SSD pools can also benefit, but the difference is usually less dramatic than on spinning disks.

Check current settings:

zfs get acltype,xattr pool0/volume0

Compatibility note:

• xattr=sa is best for many current Linux/OpenZFS systems.
• If a pool must be imported by older systems, appliances, or non-Linux platforms, verify support before relying on xattr=sa.
• xattr=on is the more conservative compatibility choice when you do not know where the pool will be imported.

Best practice:

• Use xattr=sa together with acltype=posixacl for Linux datasets that need POSIX ACLs, Samba ACL metadata, SELinux labels, or container metadata.
• Set it at dataset creation time when possible.
• Do not expect it to fix a bad pool layout. It reduces metadata overhead; it does not replace RAM, mirrors, or good disk latency.
• For simple media or backup datasets with no ACL or extended-attribute needs, the performance difference may be small.

Record Size#

recordsize affects filesystem datasets. It does not affect zvols.

General files:

zfs set recordsize=128K pool0/volume0

Large media and archives:

zfs set recordsize=1M pool0/volume2

Databases with small random I/O:

zfs set recordsize=16K pool0/volume4

Virtual machine image files:

zfs set recordsize=64K pool0/volume3

Best practice:

• Set recordsize before writing data.
• Changing recordsize affects newly written blocks only.
• Match database record size to the database page size when possible.

Quotas And Reservations#

A quota limits maximum dataset usage.

zfs set quota=500G pool0/volume0

A reservation guarantees space to a dataset.

zfs set reservation=100G pool0/volume1

Reference quotas limit only the dataset itself, excluding child datasets.

zfs set refquota=200G pool0/volume1

Show space usage:

zfs list -o name,used,avail,refer,quota,reservation

Mountpoints#

Set explicit mountpoints for clarity.

zfs set mountpoint=/pool0/volume0 pool0/volume0
zfs set mountpoint=/pool0/volume1 pool0/volume1

Temporarily unmount and mount:

zfs unmount pool0/volume0
zfs mount pool0/volume0

Mount all ZFS datasets:

zfs mount -a

Read-Only Datasets#

Make a dataset read-only:

zfs set readonly=on pool0/volume2

Make it writable again:

zfs set readonly=off pool0/volume2

Copies#

The copies property stores extra copies of blocks inside the same pool. It is not a replacement for redundancy or backups, but it can help protect very small, important datasets.

zfs set copies=2 pool0/volume1

Best practice:

• Use real vdev redundancy first.
• Use copies=2 only for selected important datasets, not huge media stores.

Everyday Commands#

Check Pool Health#

zpool status
zpool status pool0

Show only pools that have known problems:

zpool status -x

Short list:

zpool list

Show pool I/O:

zpool iostat -v pool0 5

Show dataset usage:

zfs list

Show snapshots:

zfs list -t snapshot

Create A Dataset#

zfs create pool0/volume0

Create with mountpoint and properties:

zfs create \
  -o mountpoint=/pool0/volume0 \
  -o compression=zstd \
  -o atime=off \
  pool0/volume0

Rename A Dataset#

zfs rename pool0/volume0 pool0/volume1

Destroy A Dataset#

Destroying a dataset deletes its data.

Before running zfs destroy:

• Confirm the dataset name with zfs list.
• Check whether snapshots or backups contain the data you expect.
• Stop services that may still be writing to the dataset.
• Prefer renaming or setting readonly=on first when you are unsure.

zfs destroy pool0/volume0

Destroy a dataset and its snapshots:

zfs destroy -r pool0/volume0

Move Files Into A Dataset#

Create the dataset:

zfs create -o mountpoint=/pool0/volume0 pool0/volume0

Copy data with preserved permissions:

rsync -aHAX --info=progress2 /source/volume0/ /pool0/volume0/

After verifying the copy, switch services or users to the new path.

See What Uses Space#

zfs list -o name,used,avail,refer,mountpoint

Show snapshots and written data:

zfs list -t filesystem,snapshot -o name,used,refer,written

Show pool allocation:

zpool list -o name,size,alloc,free,capacity,fragmentation,health

Clear A Resolved Error#

Only clear errors after understanding and fixing the cause.

zpool clear pool0

Clear one device:

zpool clear pool0 /dev/disk/by-id/disk0

Snapshots#

Snapshots are cheap, read-only points in time. They protect against accidental deletion, bad updates, and ransomware that does not have permission to destroy snapshots.

Create a snapshot:

zfs snapshot pool0/volume0@manual-2026-05-14

Create recursive snapshots:

zfs snapshot -r pool0@manual-2026-05-14

List snapshots:

zfs list -t snapshot

List snapshots for one dataset:

zfs list -t snapshot -r pool0/volume0

Destroy a snapshot:

zfs destroy pool0/volume0@manual-2026-05-14

Snapshot Naming#

Use sortable names:

pool0/volume0@hourly-2026-05-14-1300
pool0/volume0@daily-2026-05-14
pool0/volume0@weekly-2026-W20
pool0/volume0@monthly-2026-05

Avoid vague names:

pool0/volume0@new
pool0/volume0@backup
pool0/volume0@test

Snapshot Holds#

A hold protects a snapshot from accidental destruction. This is useful during recovery work, ransomware investigation, backup validation, or any time a snapshot must survive cleanup scripts.

Create a snapshot and hold it:

zfs snapshot pool0/volume0@recovery-2026-06-30
zfs hold keep pool0/volume0@recovery-2026-06-30

Apply a hold recursively to snapshots with the same name:

zfs snapshot -r pool0@recovery-2026-06-30
zfs hold -r keep pool0@recovery-2026-06-30

List holds:

zfs holds pool0/volume0@recovery-2026-06-30
zfs holds -r pool0@recovery-2026-06-30

Release a hold when the snapshot no longer needs protection:

zfs release keep pool0/volume0@recovery-2026-06-30
zfs release -r keep pool0@recovery-2026-06-30

Best practice:

• Use holds on snapshots that are part of an active recovery or legal hold.
• Use clear hold tags such as keep, incident-2026-06-30, or restore-test.
• Do not leave holds undocumented; they can prevent expected snapshot pruning.

Restore One File#

Snapshots are visible under .zfs/snapshot when enabled.

Enable snapshot directory visibility:

zfs set snapdir=visible pool0/volume0

Restore one file:

cp /pool0/volume0/.zfs/snapshot/daily-2026-05-14/example.txt /pool0/volume0/example.txt

Restore a directory:

rsync -aHAX /pool0/volume0/.zfs/snapshot/daily-2026-05-14/dir0/ /pool0/volume0/dir0/

Compare Snapshots With zfs diff#

zfs diff shows file-level changes between a snapshot and a later snapshot or the live dataset. It is useful after accidental deletes, bad sync jobs, ransomware, or application upgrades because it answers "what changed?" before you decide what to restore.

Compare a snapshot to the live dataset:

zfs diff pool0/volume0@daily-2026-05-14 pool0/volume0

Compare two snapshots:

zfs diff pool0/volume0@daily-2026-05-14 pool0/volume0@daily-2026-05-15

Use file type markers and machine-friendly output:

zfs diff -FH pool0/volume0@daily-2026-05-14 pool0/volume0@daily-2026-05-15

Common change markers:

• +: path was created.
• -: path was removed.
• M: path was modified.
• R: path was renamed.

Best practice:

• Use zfs diff before rollback when you are not sure what changed.
• Redirect output to an incident note for later review.
• For huge datasets, expect the command to take time and produce a lot of output.

Roll Back A Dataset#

Rollback returns the entire dataset to the snapshot state. Newer changes are lost.

Before running rollback:

• Use zfs diff or a clone to inspect what will be lost.
• Copy important newer files somewhere else.
• Confirm applications are stopped or can tolerate the rollback.
• Avoid recursive rollback unless every child dataset can move back in time.

zfs rollback pool0/volume0@daily-2026-05-14

Rollback and destroy newer snapshots if required:

zfs rollback -r pool0/volume0@daily-2026-05-14

Best practice:

• Prefer restoring individual files when possible.
• Use rollback only when you want the whole dataset back in time.

Clone A Snapshot#

Use a clone to inspect or test from an old state without rolling back.

zfs clone pool0/volume0@daily-2026-05-14 pool0/volume1

Destroy the clone when finished:

zfs destroy pool0/volume1

Snapshot Retention#

Example retention policy:

• Keep hourly snapshots for 24 hours.
• Keep daily snapshots for 14 days.
• Keep weekly snapshots for 8 weeks.
• Keep monthly snapshots for 12 months.

Use an existing snapshot tool where possible, such as sanoid, zrepl, syncoid, zfs-auto-snapshot, or a platform-native scheduler.

Simple manual snapshot example:

zfs snapshot -r pool0@daily-2026-05-14

Simple manual cleanup example:

Warning: destroying snapshots can permanently remove your ability to recover old versions or send incrementals based on those snapshots.

zfs destroy pool0/volume0@daily-2026-04-14

Best practice:

• Automate snapshot creation and pruning.
• Monitor snapshot space usage.
• Keep snapshots for recovery convenience, not as your only backup.

Pool Checkpoints#

A pool checkpoint is a short-term, pool-wide rewind point. It captures the entire state of pool0, including datasets, snapshots, pool properties, and vdev configuration. It is useful before risky pool-wide operations or destructive maintenance, such as a large cleanup, zfs destroy, pool feature upgrade testing on the same OpenZFS implementation, or a complicated migration step.

Check whether the checkpoint feature is available on the pool:

zpool get feature@zpool_checkpoint pool0

Create a checkpoint:

zpool checkpoint pool0

Check whether a checkpoint exists and how much space it uses:

zpool status pool0
zpool list -o name,size,alloc,free,checkpoint,health pool0

Discard a checkpoint after the maintenance succeeds:

zpool checkpoint -d pool0

Wait until checkpoint discard finishes:

zpool checkpoint -d -w pool0

Rewind to a checkpoint. The pool must be exported first, then imported with the rewind flag:

Before checkpoint rewind:

• Confirm every change after the checkpoint can be lost.
• Export the pool cleanly if possible.
• Capture zpool status, zpool history, and backup state first.
• Prefer file or dataset restore when only a small part of the pool is wrong.

zpool export pool0
zpool import --rewind-to-checkpoint pool0

Preview the checkpointed state read-only before committing to the rewind:

zpool export pool0
zpool import -o readonly=on --rewind-to-checkpoint pool0

Important limits:

• A checkpoint is not a backup. It lives inside the same pool.
• A pool can have only one active checkpoint.
• Keep checkpoints temporary. They can consume space as the live pool changes.
• Rewinding permanently loses all changes written after the checkpoint.
• Once a pool is imported with --rewind-to-checkpoint, that checkpoint is consumed and cannot be used again.
• While a checkpoint exists, some operations are blocked, including vdev remove, attach, detach, mirror split, and reguid.
• Adding a new vdev after a checkpoint is possible, but if you rewind, that vdev must be added again.
• Scrubs do not repair checkpointed data that has been freed in the current live state.
• Reservations and refreservations can become misleading while a checkpoint exists because the checkpoint may consume space they normally protect.

Best practice:

• Use snapshots for normal file and dataset recovery.
• Use checkpoints for short maintenance windows where a whole-pool rewind would be acceptable.
• Discard the checkpoint as soon as you are sure the operation succeeded.
• Do not keep a checkpoint around for routine retention.

Backups With ZFS Send And Receive#

ZFS replication is one of the strongest ZFS features. Use zfs send and zfs receive to copy snapshots exactly to another pool or host.

For a first full receive, let zfs receive create the destination dataset. If the destination already exists and has diverged, do not force it unless you are intentionally replacing or rolling it back.

Before running zfs receive -F:

• Confirm the destination dataset is expendable or already backed up.
• List destination snapshots and check for local-only changes.
• Prefer receiving into a new dataset when you are not sure.
• Remember that -F rolls back the destination to match the incoming stream.

Replication Safety#

Replication is only a backup when the destination is usable, protected, and tested. A successful send command by itself is not enough.

Before the first replication:

• Decide whether the destination should preserve source properties or use local mountpoints, quotas, and sharing settings.
• Receive with -u so datasets do not mount immediately with unexpected preserved mountpoints.
• Record the source dataset, destination dataset, first snapshot, and retention policy.
• Confirm the destination pool has enough free space for current data and retained snapshots.

During normal replication:

• Keep the source snapshot until the destination has received and verified the matching snapshot.
• Save the exact source and destination snapshot names in backup logs.
• Monitor receive_resume_token for interrupted receives.
• Use raw sends for encrypted datasets when the receiver should not see plaintext.
• Avoid zfs receive -F unless the destination rollback is intentional.

After replication:

• Inspect the destination dataset before mounting it.
• Verify mountpoints and sharing properties before exposing data to clients.
• Test file restore and full dataset restore paths.
• Keep destination snapshots protected from the same users or systems that can damage the source.
• Do not prune the only common snapshot needed for the next incremental send.

Local Backup To Another Pool#

Create a snapshot:

zfs snapshot -r pool0/volume0@backup-2026-05-14

Send it to pool1:

zfs send -R pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume0

The -u option receives the dataset without mounting it immediately. Check the received mountpoint before mounting because recursive sends can preserve source properties:

zfs get mountpoint pool1/volume0
zfs set mountpoint=/pool1/volume0 pool1/volume0

Incremental Backup#

Create the next snapshot:

zfs snapshot -r pool0/volume0@backup-2026-05-15

Send only the difference:

zfs send -R -I pool0/volume0@backup-2026-05-14 pool0/volume0@backup-2026-05-15 | zfs receive -u pool1/volume0

Remote Backup Over SSH#

Create a snapshot:

zfs snapshot -r pool0/volume0@backup-2026-05-14

Send to another host:

zfs send -R pool0/volume0@backup-2026-05-14 | ssh backup0.example.net zfs receive -u pool1/volume0

Incremental remote send:

zfs send -R -I pool0/volume0@backup-2026-05-14 pool0/volume0@backup-2026-05-15 | ssh backup0.example.net zfs receive -u pool1/volume0

Receive Into A Different Dataset Name#

zfs send -R pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume1

Raw Encrypted Sends#

For encrypted datasets, raw sends preserve encryption without exposing plaintext to the receiving system.

zfs send -w pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume0

Incremental raw send:

zfs send -w -I pool0/volume0@backup-2026-05-14 pool0/volume0@backup-2026-05-15 | zfs receive -u pool1/volume0

Resume Interrupted Replication#

Large sends can fail because of network loss, remote reboot, disk errors, or an interrupted terminal. Use resumable receives for large backups and restores so you do not need to restart from zero.

Start a receive in resumable mode:

zfs send -R pool0/volume0@backup-2026-06-30 | ssh backup0.example.net zfs receive -s -u pool1/volume0

If the receive is interrupted, check the resume token on the receiving side:

zfs get receive_resume_token pool1/volume0

Resume from the sender with the token value:

zfs send -t TOKEN | ssh backup0.example.net zfs receive -s -u pool1/volume0

If you decide to abandon the partial receive, abort it on the receiving side:

zfs receive -A pool1/volume0

Best practice:

• Use zfs receive -s for long transfers over unreliable links.
• Save the exact source and destination snapshot names in your backup logs.
• Do not destroy the source snapshot until the receive has completed and a test restore succeeds.

ZFS Bookmarks For Replication#

A bookmark records the creation point of a snapshot and can be used as the source side of a later incremental send. Bookmarks are useful when you want to delete old snapshots on the source but still keep an incremental replication anchor.

Create a snapshot and bookmark:

zfs snapshot pool0/volume0@backup-2026-06-30
zfs bookmark pool0/volume0@backup-2026-06-30 pool0/volume0#backup-2026-06-30

Use the bookmark as the incremental source:

zfs snapshot pool0/volume0@backup-2026-07-01
zfs send -i pool0/volume0#backup-2026-06-30 pool0/volume0@backup-2026-07-01 | zfs receive -u pool1/volume0

List bookmarks:

zfs list -t bookmark

Destroy a bookmark when it is no longer needed:

zfs destroy pool0/volume0#backup-2026-06-30

Best practice:

• Keep the source snapshot until you know the receiver has the matching state.
• Use bookmarks to reduce long-term source snapshot clutter, not as a replacement for real backup snapshots on the destination.
• Document which destination snapshot a bookmark corresponds to.

Inspect Send Streams With zstreamdump#

zstreamdump inspects a zfs send stream. It can show stream headers, record counts, and metadata, and it validates checksums by default. This is useful when you archive send streams as files or want to sanity-check a stream before receiving it on a recovery system.

Estimate a send first:

zfs send -nP pool0/volume0@backup-2026-05-14

Write a stream to a file:

zfs send pool0/volume0@backup-2026-05-14 > /safe-copy/pool0-volume0-2026-05-14.zfs

Inspect the stream:

zstreamdump /safe-copy/pool0-volume0-2026-05-14.zfs

Verbose inspection:

zstreamdump -v /safe-copy/pool0-volume0-2026-05-14.zfs

Best practice:

• Use zstreamdump to detect a malformed or truncated stream file early.
• Do not treat it as a restore test. A real test receive is still required.
• Store checksums for stream files using a normal file checksum tool as well.
• Avoid zstreamdump -d for normal checks because it dumps record data and can produce very large output.

Redacted Sends#

Redacted sends are an advanced replication feature. They create send streams that omit blocks not allowed by a redaction bookmark. This can be useful when a dataset has a sanitized clone and you need to replicate the allowed state without sending sensitive historical blocks from the original snapshot.

Conceptual workflow:

zfs snapshot pool0/volume0@base-2026-07-03
zfs clone pool0/volume0@base-2026-07-03 pool0/volume1

# Remove or replace sensitive data inside pool0/volume1, then snapshot it.
zfs snapshot pool0/volume1@sanitized-2026-07-03

zfs redact \
  pool0/volume0@base-2026-07-03 \
  pool0/volume0#redact-2026-07-03 \
  pool0/volume1@sanitized-2026-07-03

zfs send --redact pool0/volume0#redact-2026-07-03 \
  pool0/volume0@base-2026-07-03 | zfs receive -u pool1/volume0

Important limits:

• Redacted sends are for advanced workflows, not ordinary backups.
• The receiver gets a redacted snapshot with intentionally missing data.
• Raw sends and redacted sends cannot be combined.
• Only use this after testing on a disposable dataset with the same OpenZFS feature support.
• Keep normal full backups separately; a redacted stream is not a complete historical backup.

Best practice:

• Keep at least one backup outside the primary machine.
• Use recursive sends for datasets with children.
• Use raw sends for encrypted datasets when the receiver should not have the key.
• Use resumable receives for large transfers.
• Regularly test receiving and restoring.

Scrubs#

A scrub reads data, verifies checksums, and repairs bad copies when redundancy exists.

Start a scrub:

zpool scrub pool0

Check progress:

zpool status pool0

Stop a scrub:

zpool scrub -s pool0

Recommended schedule:

• Consumer disks: scrub every 2 to 4 weeks.
• Enterprise disks: scrub monthly or according to workload.
• Backup pools that are often offline: scrub after import and before trusting a restore.

Best practice:

• Scrub during low activity windows.
• Investigate checksum, read, or write errors.
• Do not ignore recurring errors after clearing them.

Pool Initialization#

zpool initialize writes to unallocated regions of pool devices. It is useful after creating a new pool or replacing disks because it forces writes across free regions instead of waiting for future allocations to touch them for the first time.

Initialize every eligible device in pool0:

zpool initialize pool0

Initialize one device:

zpool initialize pool0 /dev/disk/by-id/disk0

Check progress:

zpool status pool0

Suspend and resume initialization:

zpool initialize -s pool0
zpool initialize pool0

Cancel initialization:

zpool initialize -c pool0

How it differs from other maintenance:

• A scrub reads allocated data and verifies checksums.
• Initialization writes currently unallocated regions.
• TRIM tells SSDs or thin-provisioned storage that free regions are unused.
• Initialization is not a backup and does not validate application data.

Best practice:

• Run initialization after creating important pools if the extra I/O is acceptable.
• Run it during low activity windows on large HDD pools.
• Do not use it as a substitute for SMART tests, scrubs, or restore tests.

Disk Replacement#

Identify A Failing Disk#

Check status:

zpool status -v pool0

Look at disks:

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE

Check SMART data:

smartctl -a /dev/disk/by-id/disk0

Offline A Disk#

If the disk is still present and you need to replace it:

zpool offline pool0 /dev/disk/by-id/disk0

Replace A Disk#

Replace old disk with new disk:

zpool replace pool0 /dev/disk/by-id/disk0 /dev/disk/by-id/disk4

If ZFS already sees the old disk as unavailable:

zpool status pool0
# Use the unavailable device GUID shown by zpool status.
zpool replace pool0 1234567890123456789 /dev/disk/by-id/disk4

Watch resilver progress:

zpool status pool0

After a successful replacement, clear old errors if needed:

zpool clear pool0

Online A Disk#

zpool online pool0 /dev/disk/by-id/disk4

Detach A Disk From A Mirror#

Only detach from mirrors when you understand the redundancy left behind.

zpool detach pool0 /dev/disk/by-id/disk0

When To Replace A Disk#

Do not replace disks blindly. First rule out cabling, power, trays, backplanes, controller problems, and accidental device renaming. Then replace the disk when evidence points to the device itself or when redundancy risk is increasing.

Strong replacement signals:

• SMART shows current pending sectors, offline uncorrectable sectors, or new reallocated sectors.
• zpool status shows repeated or increasing READ, WRITE, or CKSUM errors for the same device.
• The disk disappears, reappears, or changes link speed under load.
• The disk repeatedly faults or becomes UNAVAIL.
• Long SMART tests fail.
• Resilver or scrub repeatedly stalls on the same device.
• Cabling, power, slots, and controller paths have been checked or swapped.

Replacement rules:

• Replace one disk at a time unless the situation requires otherwise.
• Let resilver finish before replacing another disk in the same vdev.
• Do not clear errors before recording zpool status, events, and SMART data.
• Use stable /dev/disk/by-id/ names or device GUIDs from zpool status.
• If more than one disk in the same vdev looks weak, back up or clone before stressing the pool.
• After replacement and resilver, run a scrub and check SMART again.

Hot Spares#

A hot spare is a disk assigned to a pool so it can be used as a replacement candidate when another device fails. It can reduce time spent in a degraded state, but it is not extra parity and it is not a backup. Platform behavior varies, so test whether your system and event daemon actually activate spares the way you expect.

Create a mirror pool with one spare:

zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -m /pool0 \
  pool0 \
  mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1 \
  spare /dev/disk/by-id/disk2

Add a spare to an existing pool:

zpool add pool0 spare /dev/disk/by-id/disk4

Check spare state:

zpool status pool0

Remove an unused spare:

zpool remove pool0 /dev/disk/by-id/disk4

Best practice:

• Keep spares at least as large as the devices they may replace.
• Use stable /dev/disk/by-id/ names.
• Monitor and test spare activation instead of assuming it works.
• Replace the failed original disk permanently; do not leave the pool relying on a temporary spare forever.
• A spare does not help if too many disks in the same vdev fail before resilver completes.

Expanding Pools#

Add Another Mirror Vdev#

This is a common and clean expansion method.

Before running zpool add:

• Run zpool status and confirm the existing top-level vdev layout.
• Confirm you are adding a complete redundant vdev, not a lone disk.
• Use zpool attach instead when the goal is to mirror an existing disk.
• Keep a current backup. New top-level vdevs become part of pool survival.

zpool add pool0 mirror /dev/disk/by-id/disk4 /dev/disk/by-id/disk5

Best practice:

• Add vdevs with similar redundancy and performance.
• Do not add a single disk vdev to a redundant pool.
• Use zpool attach, not zpool add, when the goal is to mirror an existing single disk.

Bad example:

# This can make the whole pool depend on one disk.
zpool add pool0 /dev/disk/by-id/disk4

Grow After Replacing All Disks#

Enable autoexpand:

zpool set autoexpand=on pool0

After every disk in a vdev has been replaced with larger disks, expand:

zpool online -e pool0 /dev/disk/by-id/disk4

Add A RAIDZ Vdev#

The traditional and widely supported way to expand a RAIDZ pool is to add another complete RAIDZ vdev.

Before adding a RAIDZ vdev:

• Confirm the new vdev has the intended parity level.
• Avoid mixing weak and strong vdevs in the same pool.
• Remember that the pool fails if any top-level vdev fails.

zpool add pool0 raidz2 \
  /dev/disk/by-id/disk6 \
  /dev/disk/by-id/disk7 \
  /dev/disk/by-id/disk8 \
  /dev/disk/by-id/disk9 \
  /dev/disk/by-id/disk10 \
  /dev/disk/by-id/disk11

Best practice:

• Keep new RAIDZ vdevs similar to existing vdevs.
• Avoid mixing a RAIDZ2 vdev with a single disk or weak vdev.

Expand An Existing RAIDZ Vdev#

Newer OpenZFS versions support RAIDZ expansion. This widens an existing RAIDZ vdev by attaching another disk to that RAIDZ vdev.

Check whether the pool advertises the feature:

zpool get feature@raidz_expansion pool0

Find the RAIDZ vdev name:

zpool status pool0

Expand a six-disk raidz2-0 vdev into a seven-disk raidz2-0 vdev:

zpool attach pool0 raidz2-0 /dev/disk/by-id/disk12

Watch progress:

zpool status pool0

Important limits:

• RAIDZ expansion needs OpenZFS support and the raidz_expansion pool feature.
• If the feature is unavailable or disabled, check your operating system and OpenZFS version before upgrading pool feature flags. New feature flags can make a pool unimportable on older systems.
• The new disk must be at least as large as the smallest disk in that RAIDZ vdev.
• Expansion keeps the same parity level. A RAIDZ1 vdev stays RAIDZ1, RAIDZ2 stays RAIDZ2, and RAIDZ3 stays RAIDZ3.
• Existing blocks keep their old data-to-parity ratio. New blocks use the wider layout after expansion.
• Expansion reads and rewrites allocated data in the vdev, so it can take a long time.
• A scrub is started after expansion to verify copied blocks.

Examples of what this can and cannot do:

OK:  6-wide RAIDZ2 -> 7-wide RAIDZ2
OK:  5-wide RAIDZ1 -> 6-wide RAIDZ1
NO:  5-wide RAIDZ1 -> 6-wide RAIDZ2
NO:  6-wide RAIDZ2 -> 7-wide RAIDZ3

To change parity level, create a new pool or new vdev with the desired RAIDZ level and move data with zfs send and zfs receive. Adding a RAIDZ2 vdev to a pool that already has a RAIDZ1 vdev does not make the old RAIDZ1 vdev safer; the pool is still limited by its weakest vdev.

Import And Export#

Export a pool before moving disks to another system:

zpool export pool0

List importable pools:

zpool import

Import by name:

zpool import pool0

Import with an alternate root:

zpool import -R /mnt pool0

Import without mounting datasets:

zpool import -N pool0

Import Using A Specific Device Directory#

The -d option tells ZFS where to search for pool devices. This is a device search path, not a mountpoint. Use -R when you want to change where datasets mount.

Import using stable long names:

zpool import -d /dev/disk/by-id pool0

Import using shorter /dev names:

zpool import -d /dev pool0

Using /dev can make zpool status output shorter, for example sdb instead of a long /dev/disk/by-id/... name. The tradeoff is that /dev/sdX names are not stable across reboots, controller changes, or disk reordering.

Search multiple directories:

zpool import -d /dev/disk/by-id -d /dev/disk/by-path pool0

List importable pools from a specific directory without importing:

zpool import -d /dev

Import with short device names but mount everything under /mnt:

zpool import -d /dev -R /mnt pool0

Import for recovery without mounting datasets:

zpool import -d /dev -N -o readonly=on pool0

Best practice:

• Prefer /dev/disk/by-id/ for normal operation because it survives device renumbering.
• Use -d /dev deliberately when short names are more important than stable names, such as quick lab work or temporary recovery.
• Use -R /mnt for recovery environments so datasets do not mount over the live system paths.

Import By Pool ID Or Temporary Name#

If multiple importable pools have the same name, import by the numeric pool ID shown by zpool import. This happens after disk moves, lab tests, backup disk rotation, or attaching old replacement disks.

List importable pools and IDs:

zpool import

Example output:

  pool: pool0
    id: 1234567890123456789
 state: ONLINE
action: The pool can be imported using its name or numeric identifier.

Import by ID:

zpool import 1234567890123456789

Import by ID under a temporary name:

zpool import 1234567890123456789 pool2

Import by ID read-only under /mnt:

zpool import -N -o readonly=on -R /mnt 1234567890123456789 pool2

Best practice:

• Import by ID when names collide.
• Use a temporary name such as pool2 for inspection or recovery.
• Keep old disks from previous pools offline unless they are intentionally part of the recovery.

Hostid And Active Pool Import Warnings#

If disks are moved without a clean zpool export, ZFS may report that the pool was last accessed by another system or may still be active. This warning exists to prevent two systems from importing and writing to the same pool at the same time, which can destroy data.

Safe checks:

zpool import -d /dev/disk/by-id
hostid

If the previous system is definitely powered off or gone and the pool was not exported cleanly, force import may be appropriate:

zpool import -d /dev/disk/by-id -f -N pool0

For recovery, combine force import with read-only and no-mount options when possible:

zpool import -d /dev/disk/by-id -f -N -o readonly=on pool0

On Linux systems that lack a persistent /etc/hostid, generate one before putting pools into regular service:

zgenhostid
hostid

Best practice:

• Never force import a pool that may still be imported by another live system.
• Give cloned VMs and rescue systems unique host IDs.
• Prefer clean zpool export pool0 before moving disks.
• Use -f only after verifying the old host is offline or the import warning is stale.
• If the problem follows a rescue boot, also check device paths and cachefile behavior before changing pool labels or topology.

Import read-only for recovery:

zpool import -o readonly=on pool0

Force import only when necessary, such as after a crashed system that cannot export the pool:

zpool import -f pool0

Best practice:

• Export cleanly before moving pools.
• Use read-only import when inspecting damaged systems.
• Avoid force import unless you know why it is needed.

Pool Feature Flags And Compatibility#

OpenZFS features are stored on disk. Enabling a new feature can make a pool unimportable on older operating systems, older rescue media, appliances, or other OpenZFS versions that do not support that feature. This is why zpool upgrade should be a deliberate compatibility decision, not routine cleanup.

Show whether pools can be upgraded:

zpool upgrade

Show features supported by the current OpenZFS version:

zpool upgrade -v

Show feature states for one pool:

zpool get all pool0 | grep 'feature@'

Check the compatibility property:

zpool get compatibility pool0

List available compatibility profiles where your platform provides them:

ls /usr/share/zfs/compatibility.d
ls /etc/zfs/compatibility.d

Enable all supported features only after checking recovery compatibility:

Before running zpool upgrade:

• Confirm every system that may import the pool supports the new features.
• Test current rescue media.
• Check appliance, bootloader, and older host compatibility.
• Document the minimum OpenZFS version required after the upgrade.
• Do not do this immediately before unrelated risky maintenance.

zpool upgrade pool0

Upgrade policy:

• Updating OS packages or OpenZFS packages is normal maintenance.
• Running zpool upgrade is an on-disk compatibility decision.
• Boot pools should stay conservative unless the bootloader and rescue media are known to support the new features.
• Backup pools that may move between systems should use compatibility profiles where available.
• Feature upgrades should be planned, documented, and tested like other storage migrations.

Best practice:

• Keep boot pools and portable backup pools conservative.
• Test rescue media before enabling new feature flags.
• Document the minimum OpenZFS version needed to import each pool.
• Do not upgrade feature flags immediately before major maintenance unless you have a rollback or restore plan.
• Use compatibility profiles where available for pools that must move between systems.

Encryption#

ZFS native encryption is per dataset. Enable it when creating a dataset.

Create an encrypted dataset with a passphrase:

zfs create \
  -o encryption=on \
  -o keyformat=passphrase \
  -o mountpoint=/pool0/volume0 \
  pool0/volume0

Load key:

zfs load-key pool0/volume0

Mount encrypted dataset:

zfs mount pool0/volume0

Unload key:

zfs unload-key pool0/volume0

Check encryption:

zfs get encryption,keyformat,keystatus pool0/volume0

Encrypted Dataset Recovery Checklist#

An encrypted pool can be perfectly healthy while encrypted datasets remain unrecoverable without their keys. ZFS native encryption protects data by design; there is no backdoor if the key or passphrase is lost.

Before an incident:

• Store passphrases or key files in an offline password manager, sealed print, or other tested recovery process.
• Document which datasets are encryption roots.
• Test key loading after reboot.
• Use raw encrypted sends when the backup host should not decrypt the data.

List encryption roots and key status:

zfs get -r encryptionroot,encryption,keyformat,keylocation,keystatus pool0

Import the pool without mounting datasets:

zpool import -N pool0

Dry-run a key load to test whether the key is correct:

zfs load-key -n pool0/volume0

Load keys recursively, then mount:

zfs load-key -r pool0/volume0
zfs mount pool0/volume0

Load all available encryption roots:

zfs load-key -a

Use a temporary key location without changing the dataset property:

zfs load-key -L file:///root/recovery-key0 pool0/volume0

Change a passphrase only after the current key is loaded:

zfs change-key pool0/volume0

Raw encrypted backup:

zfs snapshot pool0/volume0@secure-2026-06-30
zfs send -w pool0/volume0@secure-2026-06-30 | zfs receive -u pool1/volume0

Best practice:

• Test that backups can be received and mounted with the expected key process.
• Do not mix raw and non-raw incremental receives for the same encrypted replication chain.
• Keep key backups separate from the encrypted pool.
• Treat lost keys as permanent data loss for that encrypted dataset.

General best practice:

• Create encryption at dataset creation time.
• Keep recovery keys or passphrases offline.
• Use raw sends for encrypted backup when the backup host should not decrypt the data.

Zvols#

A zvol is a block device backed by ZFS. Use zvols for VM disks, iSCSI targets, or applications that need a block device.

Create a 100G zvol:

zfs create -V 100G pool0/volume0

Create a sparse 100G zvol:

zfs create -s -V 100G pool0/volume1

Set zvol block size at creation time:

zfs create -o volblocksize=16K -V 100G pool0/volume2

Find the device:

ls -l /dev/zvol/pool0/

Snapshot a zvol:

zfs snapshot pool0/volume0@before-update-2026-05-14

Best practice:

• Set volblocksize before writing data.
• Use smaller volblock sizes for databases or random I/O.
• Use larger volblock sizes for sequential workloads.
• Do not overuse sparse zvols unless you monitor free pool space carefully.
• Keep enough free space for snapshots and writes.

Virtual Machines#

For VM image files stored in a filesystem dataset:

zfs create -o mountpoint=/pool0/volume3 pool0/volume3
zfs set recordsize=64K pool0/volume3
zfs set compression=zstd pool0/volume3
zfs set atime=off pool0/volume3

For VM zvols:

zfs create -o volblocksize=16K -V 200G pool0/volume4

Best practice:

• Prefer mirrors for VM pools.
• Avoid very wide RAIDZ for heavy VM random writes.
• Keep snapshots short-lived for busy VM disks unless you need them.
• Monitor snapshot growth.
• Consider a separate dataset or zvol per VM.

Databases#

Databases often need more deliberate tuning than ordinary file storage.

Example for a PostgreSQL-like dataset:

zfs create -o mountpoint=/pool0/volume4 pool0/volume4
zfs set recordsize=8K pool0/volume4
zfs set compression=zstd pool0/volume4
zfs set atime=off pool0/volume4

Example for a MySQL or MariaDB InnoDB-like dataset:

zfs create -o mountpoint=/pool0/volume5 pool0/volume5
zfs set recordsize=16K pool0/volume5
zfs set compression=zstd pool0/volume5
zfs set atime=off pool0/volume5

InnoDB commonly uses 16K pages, so recordsize=16K is a practical starting point. If innodb_page_size is different, match recordsize to that value when possible.

Example for a database backup dataset:

zfs create -o mountpoint=/pool0/volume6 pool0/volume6
zfs set recordsize=1M pool0/volume6
zfs set compression=zstd pool0/volume6

Best practice:

• Match recordsize to the database page size when possible.
• Set recordsize before initializing or loading the database.
• Prefer mirrors for write-heavy databases.
• Coordinate database-consistent snapshots with the database.
• Do not assume a filesystem snapshot is application-consistent unless the application was flushed, paused, or designed for crash consistency.

Media And Archive Storage#

For large files:

zfs create -o mountpoint=/pool0/volume2 pool0/volume2
zfs set recordsize=1M pool0/volume2
zfs set compression=zstd pool0/volume2
zfs set atime=off pool0/volume2

For cold archives:

zfs set compression=zstd-6 pool0/volume2

Best practice:

• Use RAIDZ2 or RAIDZ3 for large archive pools.
• Use larger record sizes for large sequential files.
• Keep a second copy on another pool or system.

Shares#

ZFS can manage NFS and SMB sharing on some platforms, but exact behavior depends on the operating system. Many administrators prefer to manage shares with the native NFS or Samba configuration and let ZFS handle mountpoints.

Example dataset for sharing:

zfs create -o mountpoint=/pool0/volume0 pool0/volume0
zfs set compression=zstd pool0/volume0
zfs set atime=off pool0/volume0

Example NFS property where supported:

zfs set sharenfs=on pool0/volume0

Disable NFS sharing:

zfs set sharenfs=off pool0/volume0

Example SMB property where supported:

zfs set sharesmb=on pool0/volume1

Disable SMB sharing:

zfs set sharesmb=off pool0/volume1

Best practice:

• Use one dataset per share when permissions, quotas, or snapshots differ.
• Keep share configuration documented.
• Test permissions from a client machine.

Delegation#

ZFS delegation allows non-root users to perform selected ZFS operations.

Allow user user0 to create snapshots on pool0/volume0:

zfs allow user0 snapshot pool0/volume0

Allow user user0 to create and destroy snapshots:

zfs allow user0 snapshot,destroy pool0/volume0

View delegated permissions:

zfs allow pool0/volume0

Remove delegated permissions:

zfs unallow user0 snapshot,destroy pool0/volume0

Best practice:

• Delegate the minimum permissions needed.
• Be careful with destroy, mount, send, and receive.
• Do not delegate pool-level administration casually.

ZFS For Containers#

ZFS works well for containers when each important container, image store, or application state directory gets its own dataset. This gives clean quotas, snapshots, rollback points, send/receive backups, and optional clones for template-based deployments.

Example dataset for container state:

zfs create -o mountpoint=/var/lib/container0 pool0/volume0
zfs set compression=zstd pool0/volume0
zfs set atime=off pool0/volume0
zfs set quota=50G pool0/volume0

Use a parent dataset for many containers:

zfs create -o mountpoint=/containers pool0/volume1
zfs create -o mountpoint=/containers/container0 pool0/volume1/container0
zfs create -o mountpoint=/containers/container1 pool0/volume1/container1

Create a template and clone it:

zfs create -o mountpoint=/containers/templates/debian0 pool0/volume1/templates/debian0
zfs snapshot pool0/volume1/templates/debian0@clean-2026-07-03
zfs clone pool0/volume1/templates/debian0@clean-2026-07-03 pool0/volume1/container2
zfs set mountpoint=/containers/container2 pool0/volume1/container2

Snapshot one container before upgrades:

zfs snapshot pool0/volume1/container0@before-upgrade-2026-07-03

Best practice:

• Use one dataset per container or per important application state directory.
• Put container runtime caches, databases, and user data in separate datasets when they need different snapshot or recordsize policies.
• Use quotas to stop one container from filling the pool.
• Do not let untrusted containers manage host ZFS datasets.
• Be conservative with delegated mount, destroy, rollback, and receive permissions.
• If using a container engine's ZFS storage driver, follow that engine's documented dataset layout and do not also manage its internal datasets by hand.

Cache, Log, And Special Vdevs#

L2ARC#

L2ARC is a read cache on fast devices. It does not replace RAM.

Add an L2ARC cache device:

zpool add pool0 cache /dev/disk/by-id/ssd-cache0

Best practice:

• Add RAM first when possible.
• Use L2ARC only when the working set benefits from read caching.
• Do not expect L2ARC to improve write performance.

SLOG#

SLOG is a separate intent log device for synchronous writes. It is useful only for sync write workloads such as NFS, databases, or virtualization where sync writes matter.

Add a mirrored SLOG:

Before adding a SLOG:

• Confirm the workload uses synchronous writes.
• Use power-loss-protected devices.
• Mirror the SLOG for important pools.
• Understand that losing an active SLOG can lose recent sync writes.

zpool add pool0 log mirror /dev/disk/by-id/ssd-log0 /dev/disk/by-id/ssd-log1

Best practice:

• Use power-loss-protected SSDs.
• Mirror the SLOG for important pools.
• Do not add a cheap consumer SSD as SLOG.
• SLOG does not speed up normal asynchronous writes.

Special Vdev#

A special vdev can store metadata and optionally small blocks. It can greatly improve metadata-heavy workloads, but if it fails and is not redundant, the pool can fail.

Add a mirrored special vdev:

Before adding a special vdev:

• Confirm it is redundant and at least as reliable as the main pool.
• Confirm backups are current.
• Understand that a failed special vdev can make the pool unavailable.
• Use conservative special_small_blocks settings.

zpool add pool0 special mirror /dev/disk/by-id/ssd-special0 /dev/disk/by-id/ssd-special1

Best practice:

• Use redundancy for special vdevs.
• Treat special vdevs as critical pool members.
• Plan before adding one. Removing it may not be practical.

Special Small Blocks On Special Vdevs#

When a pool has a special vdev, metadata is stored there by default. You can also opt selected datasets into storing small file or zvol blocks on the special allocation class with special_small_blocks.

Example for a metadata-heavy small-file dataset:

zfs set special_small_blocks=16K pool0/volume0

Example for a VM or database dataset where only very small blocks should move:

zfs set special_small_blocks=4K pool0/volume3

Check the setting:

zfs get special_small_blocks pool0/volume0

Monitor special vdev usage:

zpool list -v pool0
zpool iostat -v pool0 5

Important behavior:

• The threshold applies after compression and encryption.
• Blocks smaller than or equal to the threshold can be allocated on the special class.
• Existing blocks are not moved just because the property changes; rewrite or replicate data if you need old blocks to follow the new policy.
• If the special class fills, allocations intended for it can spill back to normal data vdevs.

Best practice:

• Mirror or otherwise redundantly protect every special vdev.
• Keep the threshold conservative, such as 4K, 8K, or 16K, unless testing proves a higher value is worth the special-vdev capacity.
• Do not put large media datasets on special vdevs by accident.
• Track special vdev capacity separately from total pool capacity.

TRIM#

For SSD pools, enable autotrim if appropriate for your platform and devices.

zpool set autotrim=on pool0

Manual trim:

zpool trim pool0

Best practice:

• Enable TRIM for SSD-backed pools unless your environment has a reason not to.
• Monitor device behavior after enabling autotrim.

Monitoring#

Minimum monitoring:

zpool status
zpool list
zfs list

Useful health command:

zpool status -x

zpool status -x is the quickest daily check because it suppresses normal pool details and only reports pools with known problems. Healthy output usually says all pools are healthy. If it prints pool names, read the full status for each pool:

zpool status -v pool0

Command History For Auditing#

zpool history records ZFS administrative commands in the pool history. It is valuable during incidents because it can show when a destructive command, property change, vdev addition, replacement, import, export, or feature upgrade happened.

Show pool history:

zpool history pool0

Show long output with user, host, and zone information where available:

zpool history -l pool0

Include internal ZFS events:

zpool history -i pool0

Useful history terms to search for:

• zfs destroy
• zpool add
• zpool attach
• zpool replace
• zpool import
• zpool upgrade
• zfs set mountpoint
• zfs receive -F

Best practice:

• Save zpool history -l pool0 with incident notes before changing the pool.
• Treat it as an operational audit trail, not as a complete security log.
• Review history before assuming ZFS changed something by itself.

Reading zpool status#

Start with the state, status, action, and errors lines before looking at individual devices.

Healthy example:

  pool: pool0
 state: ONLINE
  scan: scrub repaired 0B in 02:10:15 with 0 errors
config:

        NAME        STATE     READ WRITE CKSUM
        pool0       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            disk0   ONLINE       0     0     0
            disk1   ONLINE       0     0     0

errors: No known data errors

Degraded mirror example:

  pool: pool0
 state: DEGRADED
status: One or more devices could not be used.
action: Attach the missing device and online it, or replace it.
config:

        NAME        STATE     READ WRITE CKSUM
        pool0       DEGRADED     0     0     0
          mirror-0  DEGRADED     0     0     0
            disk0   ONLINE       0     0     0
            disk1   UNAVAIL      0     0     0

Checksum error example:

        NAME        STATE     READ WRITE CKSUM
        pool0       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            disk0   ONLINE       0     0    12
            disk1   ONLINE       0     0     0

Permanent data error example:

errors: Permanent errors have been detected in the following files:

        pool0/volume0:<0x12345>
        /pool0/volume0/file0.txt

How to react:

• ONLINE with zero errors: keep monitoring and scrub on schedule.
• DEGRADED: check cabling, power, and enclosure state before replacing disks.
• UNAVAIL: identify the exact device by stable path, serial, or GUID.
• Growing READ, WRITE, or CKSUM counts: inspect SMART data and cables.
• Named permanent file errors: restore those files from backup after the pool is stable.
• Metadata or object errors without filenames: prioritize copying readable data out and restoring from backup.

Annotated zpool status Examples#

Daily healthy check with zpool status -x:

all pools are healthy

Annotation:

• This is the desired daily monitoring output.
• It does not replace scheduled scrubs, SMART tests, backup checks, or capacity monitoring.
• If automation reports anything else, run zpool status -gLPv pool0 before changing anything.

Healthy pool after a clean scrub:

  pool: pool0
 state: ONLINE
  scan: scrub repaired 0B in 02:10:15 with 0 errors on Sun Jul  5 03:10:15 2026
config:

        NAME                       STATE     READ WRITE CKSUM
        pool0                      ONLINE       0     0     0
          mirror-0                 ONLINE       0     0     0
            /dev/disk/by-id/disk0  ONLINE       0     0     0
            /dev/disk/by-id/disk1  ONLINE       0     0     0

errors: No known data errors

Annotation:

• state: ONLINE means the pool has no currently failed vdevs.
• scrub repaired 0B and 0 errors means this scrub found no data that needed repair.
• All per-device counters are zero, so there is no current ZFS evidence pointing to one device.

Degraded mirror with one missing side:

  pool: pool0
 state: DEGRADED
status: One or more devices could not be used because the label is missing or invalid.
action: Replace the device using `zpool replace`.
config:

        NAME                       STATE     READ WRITE CKSUM
        pool0                      DEGRADED     0     0     0
          mirror-0                 DEGRADED     0     0     0
            /dev/disk/by-id/disk0  ONLINE       0     0     0
            1234567890123456789    UNAVAIL      0     0     0  was /dev/disk/by-id/disk1

errors: No known data errors

Annotation:

• The pool is still importable because one mirror side remains online.
• The long number is a vdev GUID; use zpool status -gLPv pool0 and physical disk inventory before replacing anything.
• First check whether /dev/disk/by-id/disk1 disappeared because of cabling, power, enclosure, HBA, or OS naming.
• If the disk is truly failed and redundancy remains, replace it with a known-good disk and let the resilver complete.

Checksum errors on one disk:

  pool: pool0
 state: ONLINE
  scan: scrub repaired 128K in 01:40:11 with 0 errors on Sun Jul  5 03:40:11 2026
config:

        NAME                       STATE     READ WRITE CKSUM
        pool0                      ONLINE       0     0     0
          mirror-0                 ONLINE       0     0     0
            /dev/disk/by-id/disk0  ONLINE       0     0    24
            /dev/disk/by-id/disk1  ONLINE       0     0     0

errors: No known data errors

Annotation:

• ZFS repaired data from redundancy, so user data may still be intact.
• The checksum counter points to a device, path, cable, controller, RAM, or power problem that needs investigation.
• Do not clear counters until SMART data, cabling, and a follow-up scrub have been checked.

Permanent file errors:

  pool: pool0
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.
action: Restore the file in question if possible. Otherwise restore the entire pool from backup.
config:

        NAME                       STATE     READ WRITE CKSUM
        pool0                      ONLINE       0     0     0
          mirror-0                 ONLINE       0     0     0
            /dev/disk/by-id/disk0  ONLINE       0     0     0
            /dev/disk/by-id/disk1  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        /pool0/volume0/file0.txt

Annotation:

• The pool can be online while one or more files are damaged.
• Restore named files from snapshots, backup pools, or replicated copies after checking hardware health.
• If errors point to metadata or unnamed objects, prioritize copying readable data out and restoring the dataset from backup.

Resilver in progress:

  pool: pool0
 state: DEGRADED
  scan: resilver in progress since Sun Jul  5 10:00:00 2026
        512G scanned at 1.20G/s, 128G issued at 300M/s, 4.00T total
        128G resilvered, 3.12% done, 03:40:00 to go
config:

        NAME                         STATE     READ WRITE CKSUM
        pool0                        DEGRADED     0     0     0
          mirror-0                   DEGRADED     0     0     0
            /dev/disk/by-id/disk0    ONLINE       0     0     0
            replacing-1              DEGRADED     0     0     0
              /dev/disk/by-id/disk1  OFFLINE      0     0     0
              /dev/disk/by-id/disk2  ONLINE       0     0     0

Annotation:

• Do not remove more devices from the same vdev while resilvering unless the current path is clearly wrong and you have a recovery plan.
• Watch for new read, write, or checksum errors on the remaining old disk.
• After resilver completes, scrub and review SMART data before declaring the incident closed.

Interpreting ZFS Errors#

ZFS reports device and data health through pool state, vdev state, scrub results, and per-device error counters. The three common device error counters are READ, WRITE, and CKSUM.

Example status columns:

NAME                       STATE     READ WRITE CKSUM
pool0                      ONLINE       0     0     0
  mirror-0                 ONLINE       0     0     0
    /dev/disk/by-id/disk0  ONLINE       0     0     0
    /dev/disk/by-id/disk1  ONLINE       0     0     0

What the counters mean:

• READ: the device failed or struggled to return requested data.
• WRITE: the device failed or struggled to write data.
• CKSUM: data was read but did not match the checksum ZFS expected.

When not to worry much:

• zpool status -x says all pools are healthy.
• A scrub reports 0 errors.
• Old nonzero counters appear after a known, fixed event and do not increase.
• A pool is ONLINE and a transient cabling or power problem was fixed, then verified by a clean scrub.
• zpool status shows a scrub or resilver in progress; that is normal during maintenance, but it should finish.

When to investigate soon:

• Any READ, WRITE, or CKSUM counter is nonzero.
• Counters increase over time.
• A scrub repairs data.
• zpool status -x reports degraded or unhealthy pools.
• SMART reports pending sectors, reallocated sectors, media errors, or CRC errors.
• A device shows DEGRADED, FAULTED, UNAVAIL, REMOVED, or repeated online/offline transitions.

When to worry immediately:

• The pool state is DEGRADED, FAULTED, or UNAVAIL.
• A non-redundant pool or vdev has any device problem.
• A RAIDZ1 vdev has one failed disk.
• A mirror has only one remaining good side.
• zpool status -v lists permanent data errors or specific damaged files.
• Multiple devices in the same vdev show errors at the same time.

Recommended response:

zpool status -v pool0
zpool scrub pool0
zpool status pool0
smartctl -a /dev/disk/by-id/disk0
zpool events

If errors point to one disk, cable, HBA port, or enclosure slot, fix the hardware first. After the cause is fixed and a scrub is clean, clear stale counters:

zpool clear pool0

Do not clear errors just to hide them. Clearing is useful after you understand the cause, replace or repair the bad component, and verify the pool.

Watch pool I/O:

zpool iostat -v pool0 5

Watch dataset space:

zfs list -o name,used,avail,refer,mountpoint

Show pool events:

zpool events

Show detailed pool history:

zpool history pool0

Events And Alerting#

ZFS kernel events explain what happened before a pool reached its current state. They are useful during recovery because they can show whether a problem was a checksum error, I/O error, slow device, missing vdev, import failure, or configuration change.

Show recent events:

zpool events

Show full event payloads:

zpool events -v

Follow events while replacing hardware or testing a pool:

zpool events -f

Clear old events after you have documented them:

zpool events -c

Best practice:

• Enable your platform's ZFS event daemon or alerting system, often called zed on OpenZFS/Linux systems.
• Send alerts to email, chat, monitoring, or another place that someone actually reads.
• Alert on pool state changes, checksum errors, I/O errors, slow I/O, failed imports, and vdev removals.
• Keep event output with incident notes before clearing it.

Best practice:

• Configure email or alerting for pool errors.
• Monitor SMART data separately.
• Monitor free space and snapshot growth.
• Alert before the pool reaches 80% usage.
• Treat checksum errors as serious.

SMART Checks#

ZFS checks data integrity, but disk firmware still reports useful health information.

Show SMART details:

smartctl -a /dev/disk/by-id/disk0

Run a short test:

smartctl -t short /dev/disk/by-id/disk0

Run a long test:

smartctl -t long /dev/disk/by-id/disk0

Best practice:

• Schedule SMART tests.
• Track reallocated sectors, pending sectors, CRC errors, and media errors.
• Replace suspect disks before they fail completely.

Automation Examples#

Automation should make safe maintenance happen reliably. It should not hide errors, destroy snapshots blindly, or make recovery harder. Test every command manually on a non-critical dataset before scheduling it.

General automation rules:

• Prefer the platform's built-in scheduler, appliance UI, or packaged ZFS units when they exist.
• Log output somewhere outside the pool being protected when possible.
• Alert on failure instead of silently discarding command output.
• Keep snapshot pruning separate from snapshot creation until the policy is proven.
• Test a restore path after enabling snapshot or replication automation.

ZED Event Alerts#

On OpenZFS systems that ship zed, enable the event daemon and configure its notification target according to your distribution. Service names and configuration paths vary.

Linux systemd example:

systemctl enable --now zfs-zed.service
systemctl status zfs-zed.service

Common configuration path on many OpenZFS/Linux systems:

grep -E 'ZED_EMAIL|ZED_NOTIFY|ZED_SYSLOG' /etc/zfs/zed.d/zed.rc

Test by confirming events are visible:

zpool events -v

Best practice:

• Send ZFS events to a mailbox, monitoring system, or chat bridge that is actually watched.
• Alert on pool state changes, vdev removals, checksum errors, I/O errors, slow I/O, failed imports, and scrub failures.
• Do not rely on zpool status checks alone for production systems.

SMART Alerts#

SMART monitoring is separate from ZFS. ZFS may detect bad data, while SMART may show a disk becoming unreliable before ZFS sees a checksum failure.

Example smartd entry for one disk:

/dev/disk/by-id/disk0 -a -o on -S on -s (S/../.././02|L/../../7/03) -m admin@example.invalid

Useful manual checks:

smartctl -a /dev/disk/by-id/disk0
smartctl -t short /dev/disk/by-id/disk0
smartctl -t long /dev/disk/by-id/disk0

Best practice:

• Monitor every real disk, not just the pool.
• Track SATA CRC errors separately from media errors because CRC errors often point to cables, backplanes, HBAs, or power.
• Treat repeated SMART failures and repeated ZFS device errors as replacement evidence.

Scrub Timers#

Scrubs verify checksums and repair bad copies when redundancy is available. Schedule them when the system can tolerate extra I/O.

If your distribution already ships scrub timers, inspect them first:

systemctl list-timers 'zfs*scrub*'
systemctl list-unit-files 'zfs*scrub*'

Example custom systemd service:

[Unit]
Description=Scrub ZFS pool pool0

[Service]
Type=oneshot
ExecStart=/usr/sbin/zpool scrub pool0

Example custom systemd timer:

[Unit]
Description=Monthly scrub for ZFS pool pool0

[Timer]
OnCalendar=monthly
Persistent=true

[Install]
WantedBy=timers.target

Enable and inspect a custom timer after placing the files in the correct systemd unit directory for your distribution:

systemctl daemon-reload
systemctl enable --now zfs-scrub-pool0.timer
systemctl list-timers zfs-scrub-pool0.timer

Best practice:

• Do not schedule all pools to scrub at the same minute on a small host.
• Pair scrub automation with alerting that reports nonzero repair counts and failed scrubs.
• Check zpool status pool0 after the first scheduled scrub.

Simple Snapshot Cron#

For a very small system, cron can create snapshots. This is simple, but pruning, monitoring, and replication safety are easier with purpose-built tools.

Example hourly snapshot:

15 * * * * /usr/sbin/zfs snapshot -r pool0/volume0@hourly-$(date +\%Y-\%m-\%d-\%H)

Example daily snapshot:

30 2 * * * /usr/sbin/zfs snapshot -r pool0/volume0@daily-$(date +\%Y-\%m-\%d)

Best practice:

• Use a consistent snapshot naming scheme.
• Add pruning only after verifying that creation works and restore tests pass.
• Do not let snapshots grow without capacity alerts.

Sanoid And Syncoid#

Sanoid is useful when you want policy-driven snapshots and pruning. Syncoid is useful for asynchronous ZFS replication over local paths or SSH.

Minimal Sanoid-style policy example:

[pool0/volume0]
        use_template = production
        recursive = yes

[template_production]
        hourly = 24
        daily = 14
        monthly = 3
        autosnap = yes
        autoprune = yes

Run Sanoid from cron or the packaged timer according to its installation documentation:

sanoid --cron

Example Syncoid push replication:

syncoid --recursive pool0/volume0 backuphost:pool1/volume0

Best practice:

• Use a restricted SSH key for replication targets.
• Keep at least one backup destination unavailable to normal clients.
• Test receiving into a temporary dataset such as pool1/volume2 before trusting production replication.

zrepl#

zrepl is useful for scheduled snapshotting, pruning, and replication with a daemon model. It is more complex than simple cron, but it can be easier to monitor consistently across hosts.

Minimal illustrative job skeleton:

jobs:
- name: push_pool0_volume0
  type: push
  connect:
    type: ssh+stdinserver
    host: backuphost
    user: zrepl
  filesystems:
    "pool0/volume0<": true
  snapshotting:
    type: periodic
    prefix: zrepl_
    interval: 1h
  pruning:
    keep_sender:
    - type: not_replicated
    - type: last_n
      count: 24
    keep_receiver:
    - type: last_n
      count: 48

Best practice:

• Validate configuration with the tool's check command before enabling the service.
• Confirm replication creates snapshots on the receiver and that old snapshots prune as expected.
• Do a restore drill from the receiver before calling the automation complete.

Performance Basics#

Keep Free Space#

Performance usually drops as a pool gets full.

Best practice:

• Keep pools below 80% used.
• Start planning expansion before 80%.
• Avoid going above 90% except temporarily.

Check capacity:

zpool list -o name,capacity,free,fragmentation,health

Match Layout To Workload#

Good defaults:

• General file server: mirrors or RAIDZ2.
• VM storage: mirrors.
• Database storage: mirrors.
• Media archive: RAIDZ2 or RAIDZ3.
• Backup target: RAIDZ2, RAIDZ3, or mirrors depending on restore needs.

Compression#

Usually keep compression enabled:

zfs set compression=zstd pool0

Compression can increase speed by reducing disk I/O.

Sync Writes#

Check sync behavior:

zfs get sync pool0/volume0

Default is usually correct:

zfs set sync=standard pool0/volume0

Dangerous setting:

# Can lose acknowledged synchronous writes during power loss or crash.
zfs set sync=disabled pool0/volume0

Best practice:

• Keep sync=standard unless you understand the application and risk.
• Use a proper SLOG for important sync write workloads.

Deduplication#

Do not enable deduplication casually.

zfs get dedup pool0/volume0

ZFS deduplication uses a deduplication table, often called the DDT, to map block checksums to already-stored blocks. The DDT is on disk, but performance depends heavily on keeping hot DDT entries in RAM through the ARC. When the DDT does not fit in RAM, writes that should be simple sequential work can turn into many random DDT lookups from disk. The pool is not automatically corrupt, but it can feel broken: writes become very slow, deletes and snapshot destruction can take much longer, scrub or resilver work competes with DDT I/O, and imports or administrative commands may become painfully slow on small-memory systems.

Check pool-level dedup statistics:

zpool status -D pool0
zpool status -DD pool0

Use -DD when available to see how much of the DDT is resident in ARC. If the resident portion is low and the workload is writing many small or unique blocks, expect random I/O pressure and high latency.

Estimate dedup usefulness before enabling it:

zdb -S pool0

Run dedup simulation only during a quiet period or on a test system. It scans the pool and builds simulated DDT statistics, so it can be expensive on large pools.

If dedup was enabled and the DDT is now too large:

zfs set dedup=off pool0/volume0

Turning dedup off stops new writes from adding more deduplicated blocks, but it does not immediately remove existing DDT entries. Existing deduplicated blocks remain referenced until they are deleted, snapshots that reference them are destroyed, or the data is rewritten without dedup. The safest escape path is usually to create a new non-dedup dataset, copy or replicate data into it, verify the result, then retire the old dataset after snapshots no longer hold the old blocks.

Best practice:

• Leave dedup off for most systems.
• Dedup needs large amounts of RAM, fast metadata I/O, and careful planning.
• Do not enable dedup unless the measured dedup ratio is worth the performance and recovery cost.
• If a platform supports a dedicated dedup allocation class, mirror it and treat it as critical pool metadata, not as a disposable cache.
• Compression is usually the better choice.

Performance Troubleshooting Decision Tree#

Performance problems are usually caused by layout, full pools, sync writes, weak hardware, slow devices, metadata pressure, or an application pattern that does not match the dataset properties. Do not start with random tunables.

First capture a baseline:

zpool status -v pool0
zpool list -o name,size,alloc,free,capacity,fragmentation,health
zpool iostat -v pool0 5
zfs list -o name,used,avail,refer,compressratio,mountpoint -r pool0
zfs get -r compression,recordsize,volblocksize,sync,logbias,primarycache,secondarycache,dedup pool0

Decision tree:

1. If pool capacity is above 80%, free space, destroy unneeded snapshots, add capacity, or move data before tuning anything else.
2. If capacity is above 90%, treat it as an urgent operational problem because allocation, fragmentation, snapshots, and recovery all get worse.
3. If one vdev is much busier than others in zpool iostat -v, check whether the pool was expanded unevenly, whether one vdev is slower, or whether a disk is failing.
4. If latency-sensitive VMs or databases are on RAIDZ, consider moving them to mirror vdevs rather than trying to tune around random I/O limits.
5. If writes are slow only for applications that use fsync, check sync, SLOG health, power-loss protection, and application transaction settings.
6. If reads are slow after a cold boot but improve later, check ARC size, working set size, and whether the workload is cache-friendly.
7. If metadata-heavy workloads are slow, check snapshot count, directory count, small files, special vdev design, and whether a mirrored special vdev would help a new pool design.
8. If dedup is enabled, check zpool status -DD pool0 before looking elsewhere.
9. If compression is off, test compression=zstd on a representative dataset because less physical I/O often improves throughput.
10. If the pool is healthy but the application is slow, measure inside the application before changing ZFS properties.

Symptom map:

Useful measurements:

zpool iostat -rw pool0 5
zpool iostat -lq pool0 5
zpool iostat -v pool0 5
arcstat 5
iostat -x 5

Not every platform has every helper command. If arcstat or iostat -x is unavailable, use the closest platform equivalent.

Workload-specific checks:

• For databases, confirm recordsize before data is loaded and use a separate dataset such as pool0/volume0.
• For zvols, confirm volblocksize before creating the virtual disk and do not expect changing it later to rewrite existing blocks.
• For NFS, SMB, iSCSI, and VM hosts, separate network, guest, and storage measurements before blaming ZFS.
• For all workloads, compare performance during a scrub/resilver and while idle.

Safe tuning order:

1. Fix hardware errors, pool health, and capacity first.
2. Move workloads to the right layout before tuning properties.
3. Set workload properties on a new dataset or zvol, copy data in, and test.
4. Add support vdevs only after proving the bottleneck and understanding failure consequences.
5. Keep notes of before/after commands and measurements.

Security And Permissions#

Set ownership after creating a dataset:

chown -R user0:group0 /pool0/volume0

Set basic permissions:

chmod 750 /pool0/volume0

Use ACLs when needed:

zfs set acltype=posixacl pool0/volume0
zfs set xattr=sa pool0/volume0

Best practice:

• Keep one dataset per permission boundary.
• Use encryption for data at rest when needed.
• Keep backup permissions as strict as primary permissions.
• Limit who can destroy snapshots.

Boot Pools#

Boot environments vary by operating system. The safest general practices are:

• Keep boot pool layouts simple.
• Use mirrors for boot disks when uptime matters.
• Do not use exotic feature flags if the bootloader cannot read them.
• Keep a tested rescue USB or recovery environment.
• Snapshot boot environments before major upgrades where supported.
• Confirm the system can boot after disk replacement.

Check pool features:

zpool get all pool0 | less

Upgrade pool features only after checking compatibility:

zpool upgrade pool0

Disaster Recovery#

Recovery Decision Tree#

Use this as the first pass during an incident. The goal is to choose the safest path before running commands that change pool state.

Incident rules:

• Stop writes first when corruption, ransomware, or failing hardware is suspected.
• Prefer read-only and no-mount imports during investigation.
• Clone failing disks before aggressive recovery attempts.
• Copy critical data out before starting heavy repairs if more hardware looks weak.
• Do not clear errors until you have captured status, events, and SMART data.

Symptom-Based Recovery Runbook#

Start from the symptom you see. Keep the first pass read-only unless the pool is already healthy and the action is clearly routine maintenance.

Pool Is Missing From zpool list#

1. Check whether the pool is only exported or not imported:

zpool import
zpool import -d /dev/disk/by-id

1. If the pool appears, try read-only and no-mount import first:

zpool import -d /dev/disk/by-id -N -o readonly=on pool0

1. If the pool does not appear, check disk inventory and kernel logs:

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
dmesg --ctime | grep -Ei 'zfs|i/o|error|reset|ata|scsi|nvme'

1. If disks are missing at the OS level, solve cabling, power, HBA, enclosure, or driver visibility before trying ZFS recovery flags.

Pool Imports Read-Only#

1. Treat read-only import as a chance to copy data out, not as proof that repair is complete.

zpool status -gLPv pool0
zfs list -r pool0
zfs list -t snapshot -r pool0

1. Copy the most important data to another pool or host:

rsync -aHAX --info=progress2 /pool0/volume0/ /safe-copy/volume0/

1. After data is safe, decide whether to repair hardware, restore from backup, or attempt documented import recovery.

One Device Is FAULTED, DEGRADED, UNAVAIL, Or REMOVED#

1. Capture exact device identity:

zpool status -gLPv pool0
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0

1. Check whether the fault follows the disk, cable, slot, HBA port, enclosure bay, or power connector.
2. If redundancy remains and the disk is truly failed, replace one disk at a time:

zpool replace pool0 1234567890123456789 /dev/disk/by-id/disk4
zpool status pool0

1. After resilver, scrub and keep the failed disk untouched until the pool is confirmed healthy.

READ, WRITE, Or CKSUM Counters Increase#

1. Save status and events before clearing anything:

zpool status -v pool0
zpool events -v

1. Check SMART, kernel logs, cabling, power, HBA, and enclosure health.
2. If counters increase during a scrub, treat the cause as active and fix hardware before trusting the pool.
3. Clear counters only after the cause is fixed and a scrub is clean:

zpool clear pool0

zpool status Lists Permanent File Errors#

1. Record the full verbose status:

zpool status -v pool0

1. Restore named files from snapshots, a backup pool, or a replicated copy.
2. If ZFS lists metadata objects or unnamed errors, copy readable data out and restore the dataset from a clean backup.
3. Scrub again after restore and only then decide whether zpool clear pool0 is appropriate.

Pool Will Not Import#

1. Do not run random import flags; capture the import view first:

zpool import
zpool import -d /dev/disk/by-id

1. Try read-only and no-mount import:

zpool import -d /dev/disk/by-id -N -o readonly=on pool0

1. If the pool complains that it is active on another host, verify that the old host is powered off or exported before using -f.
2. If import suggests rewind, dry-run first:

zpool import -d /dev/disk/by-id -F -n pool0

1. Use -X only when backup restore is worse than accepting last-resort rewind risk.

Encryption Key Is Missing Or Dataset Will Not Mount#

1. Check which datasets need keys:

zfs get -r encryptionroot,keystatus,keylocation,mounted pool0

1. Load the key for the encryption root, not random children:

zfs load-key pool0/volume0
zfs mount pool0/volume0

1. If the key is lost, ZFS cannot recover encrypted data from checksums, redundancy, snapshots, or pool metadata.
2. Restore from an unencrypted backup or from a backup whose key is still available.

Accidental Snapshot Or Dataset Destroy#

1. Stop writes to affected datasets if recent snapshots or replication targets may still preserve data.
2. Check other snapshots, clones, replication targets, and backup pools:

zfs list -t snapshot -r pool0
zfs list -r pool1

1. If a replicated backup exists, receive into a new dataset for inspection instead of overwriting the original:

zfs send -R pool1/volume0@backup-2026-05-14 | zfs receive -u pool0/volume2

1. If the only copy was destroyed and no snapshot or backup exists, recovery from that pool is usually not realistic.

Interrupted Receive Or Replication#

1. Look for a resumable receive token:

zfs get receive_resume_token pool1/volume0

1. Resume if a valid token exists:

zfs send -t TOKEN | zfs receive -s -u pool1/volume0

1. If the destination must be discarded, confirm that losing the partially received dataset is acceptable before using destructive receive options.

Special, Dedup, Cache, Or Log Device Missing#

1. Identify which class is missing:

zpool status -gLPv pool0

1. A missing cache device is usually removable after the pool is stable.
2. A missing special or dedup vdev is critical pool metadata and may make the pool unavailable.
3. A missing separate log can require zpool import -m, but that can lose recent acknowledged synchronous writes.
4. Prefer finding or cloning the missing device before accepting data loss.

Minimum Safe Recovery Bundle#

Capture this bundle before changing pool state during an incident. It gives you enough evidence to reason about the failure, ask for help, and avoid losing device identity.

Pool health and device paths:

zpool status -gLPv pool0
zpool list -v pool0

Import view, useful when the pool is exported or will not import:

zpool import -d /dev/disk/by-id

Events and command history:

zpool events -v
zpool history -l pool0

Dataset and snapshot inventory:

zfs list -r pool0
zfs list -t snapshot -r pool0

Disk identity and SMART data:

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0

If hardware may be failing, save the output somewhere outside the suspect pool. Do not start repeated imports, scrubs, resilvers, or rewind attempts until you have considered cloning the weakest disks.

Incident Evidence Form#

Use this form for failed pools, permanent data errors, ransomware, accidental deletion, failed imports, or any case where you may ask another administrator for help. Store the completed form outside the affected pool.

Incident title:
Date and time first noticed:
Timezone:
Host name:
Operating system and version:
OpenZFS version:
Pool name:
Pool GUID:
Affected datasets:
Business impact:
Last known good backup:
Last known good scrub:
Recent maintenance:
Recent power loss or crash:
Recent disk, HBA, cable, enclosure, kernel, package, or firmware changes:
Current pool state:
Exact error message:
Commands already run:
Commands not yet run:
Disk serial numbers and slots:
Suspect disks:
SMART failures:
Kernel log errors:
Encryption keys available:
Snapshots available:
Replication target available:
Pool checkpoint present:
Data already copied out:
Next planned action:
Risk accepted by:

Attach these command outputs when available:

zpool status -gLPv pool0
zpool list -v pool0
zpool events -v
zpool history -l pool0
zfs list -r pool0
zfs list -t snapshot -r pool0
zfs get -r encryptionroot,keystatus,keylocation,mounted pool0
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0

Best practice:

• Record exact commands and outputs before trying recovery flags.
• Record disk serial numbers, not only operating system device names.
• Add photos of drive bays or labels when physical slot mapping matters.
• Keep a copy of the evidence form with the restored system documentation after the incident.

Accidental File Deletion#

1. Check snapshots:

zfs list -t snapshot -r pool0/volume0

1. Restore from .zfs/snapshot:

cp /pool0/volume0/.zfs/snapshot/daily-2026-05-14/file0.txt /pool0/volume0/file0.txt

Bad System Update#

Snapshot before the update:

zfs snapshot -r pool0@before-update-2026-05-14

Rollback one dataset if needed:

zfs rollback pool0/volume0@before-update-2026-05-14

If the risky change affects the whole pool rather than one dataset, create a pool checkpoint before starting. Rewinding to it loses all later changes, so use it only when a whole-pool undo is acceptable:

zpool checkpoint pool0

Pool Will Not Import#

List importable pools:

zpool import

Try read-only import:

zpool import -o readonly=on pool0

Try alternate root:

zpool import -R /mnt pool0

Use force only when you understand why:

zpool import -f pool0

Failed Pool Recovery Triage#

When a pool is failed, faulted, or non-importable, the first goal is to avoid making the situation worse. Do not destroy and recreate the pool, do not clear labels, do not run filesystem repair tools such as fsck, and do not repeatedly try random import flags. If the data is important and the disks may be failing, stop and make sector-level clones of the suspect disks first, for example with a recovery tool such as ddrescue, before further import attempts.

Capture the current state before changing anything:

zpool import
zpool import -d /dev/disk/by-id
zpool import -d /dev
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
dmesg --ctime | grep -Ei 'zfs|i/o|error|reset|ata|scsi|nvme'

If the pool imports, collect detailed status:

zpool status -gLPv pool0
zpool events -v
zpool history pool0

Useful options while investigating:

• -g shows vdev GUIDs, useful when device names are missing or unstable.
• -L resolves symlinks to the current real device path.
• -P prints full paths instead of shortened names.
• -v shows known permanent data errors and affected files when ZFS can name them.

Start with the least invasive import:

zpool import -d /dev/disk/by-id -N -o readonly=on pool0

Use an alternate root in a rescue environment so datasets do not mount over the live system:

zpool import -d /dev/disk/by-id -N -o readonly=on -R /mnt pool0

Use -f only if the pool appears active because it was not exported cleanly and you are sure no other system is using it:

zpool import -d /dev/disk/by-id -f -N -o readonly=on pool0

If a pool imports read-only, copy the most important data out before repair attempts:

rsync -aHAX --info=progress2 /mnt/pool0/volume0/ /safe-copy/volume0/

If the pool is DEGRADED but importable, prefer recovery over experimentation:

• Verify all cables, HBAs, enclosures, and power before replacing disks.
• If a missing disk reappears, try zpool online pool0 /dev/disk/by-id/disk0.
• If a disk is truly failed and redundancy remains, replace it with zpool replace.
• Copy critical data before starting heavy operations if more disks look weak.
• After replacement or repair, let resilver finish and then scrub.

Example replacement:

zpool status -gLPv pool0
zpool replace pool0 1234567890123456789 /dev/disk/by-id/disk4
zpool status pool0

If zpool status -v lists permanent errors in files, restore those files from backup after the pool is stable. If it lists metadata objects or does not name a file, assume affected data may not be recoverable from that pool and prioritize backup restore.

Understand which missing device class matters:

• A missing normal top-level vdev usually means the pool cannot be recovered without that vdev or a backup.
• A missing mirror side or RAIDZ member may be survivable if enough replicas remain.
• A missing special or dedup vdev is critical and can make the whole pool unavailable.
• A missing cache device should not lose pool data; remove or replace it after the pool is stable.
• A missing separate log device may require zpool import -m, but that can discard recent synchronous writes.

Clone Failing Disks Before Recovery Attempts#

If the pool failure may involve physically failing disks, make sector-level clones before running heavy recovery operations. Scrubs, resilvers, repeated imports, and full-file copies can put enough load on a marginal disk to finish it off.

When to clone first:

• SMART shows pending sectors, reallocated sectors, media errors, or repeated resets.
• dmesg shows I/O errors, link resets, timeouts, or NVMe errors.
• The disk clicks, drops offline, or disappears under load.
• Multiple disks in the same vdev are suspect.
• The pool contains data that has no tested backup.

Capture device identity:

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0
zpool import -d /dev/disk/by-id

Example ddrescue workflow:

ddrescue -f -n /dev/disk/by-id/disk0 /safe-copy/disk0.img /safe-copy/disk0.map
ddrescue -f -r3 /dev/disk/by-id/disk0 /safe-copy/disk0.img /safe-copy/disk0.map

Use the map file so the recovery can resume. Work on cloned images or cloned replacement disks when possible, and keep the original disks unchanged until the recovery is complete.

If you clone to replacement disks, attach or import using the replacement devices, not the failing originals:

zpool import -d /dev/disk/by-id -N -o readonly=on pool0

Best practice:

• Clone the weakest disks first.
• Do not run zpool clear or label operations before imaging suspect disks.
• Do not write recovered data back onto the same failing pool.
• Keep notes mapping old serial numbers to cloned images or replacement disks.

Physical Pool Inspection With zdb#

zdb is the ZFS debugger. It can read vdev labels, pool configuration, uberblocks, and some dataset or object metadata directly from devices. That makes it useful when a pool will not import cleanly, when disk names changed after moving hardware, or when you need to prove which physical devices belong to which pool. It is not fsck, it is not a routine repair command, and much of its output assumes ZFS internals.

Use zdb for inspection, not as the first recovery action. If disks may be failing, clone them first and run zdb against the clones or images. If the pool imports read-only, copy or replicate the data out before spending time on deeper metadata analysis.

Useful read-only checks:

zdb -l /dev/disk/by-id/disk0
zdb -lu /dev/disk/by-id/disk0
zdb -lll /dev/disk/by-id/disk0

What these show:

• zdb -l reads ZFS labels from one device or partition.
• zdb -lu also shows uberblocks, including transaction group history.
• zdb -lll shows every label copy, including stale or duplicate configurations.

Look for:

• The expected pool name, pool GUID, vdev GUID, and top-level vdev GUID.
• Whether all mirror or RAIDZ members agree about the same pool layout.
• Whether labels mention old device paths, old hostnames, or old pool names.
• Whether a disk has no valid ZFS labels, labels from another pool, or labels from an old destroyed pool.
• Whether only very old uberblocks remain, which can explain why rewind would lose recent writes.

Inspect an exported or non-imported pool using a specific device directory:

zdb -e -p /dev/disk/by-id -C pool0
zdb -e -p /dev/disk/by-id -d pool0

This is useful after booting rescue media, moving disks to another system, or cloning disks to image files and loop devices. The -e option tells zdb to operate on an exported pool instead of relying on the normal cache file. The -p option limits the search path, similar in spirit to importing with zpool import -d /dev/disk/by-id.

If a pool checkpoint exists, zdb can inspect the checkpointed state without rolling the pool back:

zdb -e -p /dev/disk/by-id -k -C pool0

This helps compare the current on-disk configuration with the checkpointed configuration before deciding whether zpool import --rewind-to-checkpoint is appropriate.

When to worry:

• zdb -l shows I/O errors while reading labels from an original disk. Stop and image the disk before continuing.
• Different members of the same mirror or RAIDZ vdev report different pool GUIDs or incompatible top-level vdev GUIDs.
• A disk that should be part of pool0 reports labels from pool1.
• Labels are present, but the expected vdev is missing enough members that redundancy cannot reconstruct the data.
• zdb -lu only shows old transaction groups and zpool import -F -n reports a large rewind.

When not to worry immediately:

• Old path names in labels are common after moving disks between systems.
• Old hostnames are common after migration or rescue booting.
• One bad label copy is not automatically fatal if other label copies are valid.
• A label from an old pool on an unused replacement disk matters only if you are about to reuse that disk. Clear it only after verifying backups and disk identity.

Advanced salvage options exist, but they are last-resort work. zdb -B can generate a backup stream from a numeric objset ID when normal dataset metadata is damaged but the dataset is still readable. zdb -r can copy a path or object out of a dataset in some cases. Treat these as expert recovery tools: work from cloned media, write output to a different pool such as pool1, and document every command before running it.

Avoid this pattern on original disks unless you have accepted the risk:

Before using zdb -F or zdb -X:

• Clone suspect disks first when hardware may be failing.
• Capture labels, import output, SMART data, and pool history.
• Confirm backups or recovery images exist.
• Prefer documented zpool import dry runs.
• Treat any write-capable repair attempt as potentially destructive.

zdb -F pool0
zdb -FX pool0

For ordinary recovery, prefer the documented zpool import -F -n dry run first, then an explicit import decision. zdb -F and zdb -X are deep debugging and rewind tools, not everyday pool administration commands.

Pool Rewind Recovery With -F#

zpool import -F is recovery mode for a non-importable pool. It tries to make the pool importable by discarding the last few transactions. This can recover a pool after damaged recent metadata, but any discarded transactions are lost permanently.

Always dry-run first:

zpool import -d /dev/disk/by-id -F -n pool0

If the dry run says recovery is possible and the data loss is acceptable, import with recovery mode. Use -N to avoid mounting datasets immediately:

Before importing with -F:

• Save the dry-run output.
• Confirm the estimated lost transactions are acceptable.
• Use -N so datasets do not mount automatically.
• Plan to copy important data out after import.

zpool import -d /dev/disk/by-id -F -N pool0

After a successful rewind, scrub the pool and then copy or replicate important data elsewhere:

zpool scrub pool0
zpool status pool0
zfs snapshot -r pool0/volume0@recovered-2026-05-18
zfs send -R pool0/volume0@recovered-2026-05-18 | zfs receive -u pool1/volume0

Use -X only as a last resort. It enables extreme transaction search and may roll back to a transaction group that is not guaranteed to be consistent.

Dry-run the extreme option first:

zpool import -d /dev/disk/by-id -F -X -n pool0

Extreme recovery should be reserved for cases where the alternative is restoring from backup or accepting loss:

zpool import -d /dev/disk/by-id -F -X -N pool0

If zpool import or zpool status prints a specific recovery command, prefer that exact command over guessing. If no recovery action is offered and the pool still cannot import, plan for backup restore or professional recovery rather than trying destructive commands.

If a separate log device is missing, -m may allow import by discarding the missing log device. Recent synchronous transactions can be lost.

zpool import -d /dev/disk/by-id -m -N pool0

Recover A Destroyed Pool Entry#

If a pool was destroyed with zpool destroy, it may still be listed with -D until labels are overwritten.

List destroyed pools:

zpool import -D

Import a destroyed pool read-only and without mounting datasets:

zpool import -D -f -N -o readonly=on pool0

If this works, copy the data to another pool immediately. Do not treat this as a normal undo feature; overwritten labels or reused disks can make recovery impossible.

Handling Permanent Data Errors#

Permanent errors mean ZFS could not reconstruct some data from available replicas. They are different from device counters that were corrected during a scrub.

Start with verbose status:

zpool status -v pool0

If files are listed, restore those files from a snapshot, backup pool, or replication target:

cp /pool0/volume0/.zfs/snapshot/daily-2026-06-30/file0.txt /pool0/volume0/file0.txt

Or restore from a backup dataset:

rsync -aHAX /pool1/volume0/file0.txt /pool0/volume0/file0.txt

If the error is in metadata, a directory, or an object that ZFS cannot map to a file, prioritize copying readable data out and restoring the dataset from a clean backup.

After restoring or accepting loss, scrub again:

zpool scrub pool0
zpool status -v pool0

Clear stale errors only after the scrub is clean or after you have documented and accepted unrecoverable damage:

zpool clear pool0

Some OpenZFS versions support corrective receive, which can repair data blocks from a suitable healthy send stream for the affected dataset. It cannot repair metadata and it does not fix the hardware cause of corruption:

zfs send pool1/volume0@clean-2026-06-30 | zfs receive -c pool0/volume0
zpool scrub pool0

Best practice:

• Do not clear permanent errors before recording zpool status -v.
• Restore named files from backup instead of rolling back whole datasets when possible.
• If permanent errors return after restore, suspect hardware, cabling, RAM, or controller problems.
• Treat metadata permanent errors as high risk and restore the dataset or pool from a clean backup.

Restore From Backup Pool#

Restore into a new dataset for inspection first:

zfs send -R pool1/volume0@backup-2026-05-14 | zfs receive -u pool0/volume1

After verifying the restored data, move applications or users to the restored dataset. Replacing an existing dataset should be a deliberate maintenance task, not an automatic first step.

Ransomware Or Mass Deletion Recovery#

If clients are actively deleting or rewriting files, stop the writes first. A perfect snapshot plan can still be damaged if the attacker or broken client has permission to destroy snapshots.

Immediate actions:

• Disconnect affected clients or stop the share service.
• Disable shares for the affected datasets.
• Preserve existing snapshots with holds.
• Avoid rolling back until you know which snapshot is clean.
• Restore into a new dataset first, then cut users over.

Disable ZFS-managed shares where supported:

zfs set sharenfs=off pool0/volume0
zfs set sharesmb=off pool0/volume0

Optionally make the dataset read-only while investigating:

zfs set readonly=on pool0/volume0

Snapshot the current damaged state for investigation:

zfs snapshot -r pool0@incident-2026-06-30
zfs hold -r incident pool0@incident-2026-06-30

List candidate clean snapshots:

zfs list -t snapshot -r pool0/volume0

Clone a known-good snapshot for inspection:

zfs clone pool0/volume0@daily-2026-06-29 pool0/volume1
zfs set mountpoint=/pool0/volume1 pool0/volume1

Copy known-good data into a new recovery dataset:

zfs create -o mountpoint=/pool0/volume2 pool0/volume2
rsync -aHAX --info=progress2 /pool0/volume1/ /pool0/volume2/

After validation, repoint shares or applications to the recovered dataset. Keep the incident snapshot and holds until investigation and backup verification are finished.

Recovery Practice Lab#

Practice recovery on a throwaway system before you need it. This lab uses file-backed vdevs under /tmp. Do not run it on a production host, and choose an unused pool name.

Create a small mirror pool:

mkdir -p /tmp/zfs-lab
truncate -s 512M /tmp/zfs-lab/disk0 /tmp/zfs-lab/disk1 /tmp/zfs-lab/disk2
zpool create -o ashift=12 -O compression=zstd -m /tmp/zfs-lab/mnt pool2 mirror /tmp/zfs-lab/disk0 /tmp/zfs-lab/disk1
zfs create pool2/volume0

Create a file and snapshot:

echo important-data > /tmp/zfs-lab/mnt/volume0/file0.txt
zfs snapshot pool2/volume0@before-delete

Delete and restore one file:

rm /tmp/zfs-lab/mnt/volume0/file0.txt
cp /tmp/zfs-lab/mnt/volume0/.zfs/snapshot/before-delete/file0.txt /tmp/zfs-lab/mnt/volume0/file0.txt

Simulate one failed mirror side and replace it:

zpool offline pool2 /tmp/zfs-lab/disk1
zpool status pool2
zpool replace pool2 /tmp/zfs-lab/disk1 /tmp/zfs-lab/disk2
zpool status pool2

Practice send and receive to another dataset:

zfs snapshot pool2/volume0@backup-test
zfs send pool2/volume0@backup-test | zfs receive pool2/volume1
zfs list pool2/volume1

Clean up when finished:

zpool destroy pool2
rm -rf /tmp/zfs-lab

Practice goals:

• Restore one file from .zfs/snapshot.
• Read zpool status during a degraded mirror.
• Replace a failed device and watch resilver progress.
• Send and receive a snapshot.
• Destroy the lab pool only after confirming it is the throwaway pool.

Restore Drill Templates#

Run restore drills on a lab host, temporary dataset, or isolated backup target. Do not overwrite production datasets during a drill.

Single-File Restore Drill#

Goal: prove that a user file can be restored without rolling back the whole dataset.

Setup:

zfs create -o mountpoint=/pool0/volume2 pool0/volume2
printf 'restore-drill\n' > /pool0/volume2/file0.txt
zfs snapshot pool0/volume2@drill-file-2026-07-05
cp /pool0/volume2/file0.txt /tmp/file0.before
rm /pool0/volume2/file0.txt

Restore:

cp /pool0/volume2/.zfs/snapshot/drill-file-2026-07-05/file0.txt /pool0/volume2/file0.txt
cmp /tmp/file0.before /pool0/volume2/file0.txt

Cleanup after validation, only if pool0/volume2 is the temporary drill dataset:

zfs destroy -r pool0/volume2

Pass criteria:

• The file is restored with expected contents.
• Ownership and permissions are acceptable for the application.
• No unrelated files were changed.

Fail criteria:

• The snapshot is not visible or not mounted.
• The restored file differs unexpectedly.
• The recovery process required rolling back the whole dataset.

Dataset Restore From Backup Pool Drill#

Goal: prove that a replicated dataset can be restored into a new dataset.

Restore into a temporary dataset:

zfs send -R pool1/volume0@backup-2026-07-05 | zfs receive -u pool0/volume2
zfs set mountpoint=/pool0/volume2 pool0/volume2
zfs mount pool0/volume2

Validate:

zfs list -r pool0/volume2
find /pool0/volume2 -maxdepth 2 -type f | head

Pass criteria:

• The received dataset mounts in the expected temporary location.
• Expected files, snapshots, and properties are present.
• Applications can read test data from the temporary dataset.

Fail criteria:

• The send stream fails or requires destructive receive against production data.
• Required snapshots are missing on the backup side.
• Properties such as mountpoint, readonly, recordsize, or encryption are not understood.

Ransomware Snapshot Drill#

Goal: prove that a clean snapshot can be found and restored without trusting the damaged live dataset.

Inspect candidate snapshots:

zfs list -t snapshot -r pool0/volume0
zfs diff -FH pool0/volume0@daily-2026-07-04 pool0/volume0

Clone a clean snapshot:

zfs clone pool0/volume0@daily-2026-07-04 pool0/volume2
zfs set mountpoint=/pool0/volume2 pool0/volume2

Pass criteria:

• A known-clean snapshot can be identified.
• The clone mounts without overwriting production.
• Users or applications can validate data in the clone.

Fail criteria:

• Snapshots are missing, pruned too aggressively, or writable by the attacker.
• The only restore path requires destroying current evidence.
• Backup targets were reachable and damaged by the same attacker.

Encrypted Dataset Drill#

Goal: prove that encryption keys and recovery notes work on a fresh boot or rescue environment.

Check key state:

zfs get -r encryptionroot,keystatus,keylocation,mounted pool0

Load and mount:

zfs load-key pool0/volume0
zfs mount pool0/volume0

Pass criteria:

• The documented key or passphrase unlocks the expected encryption root.
• The dataset mounts at the expected path.
• Recovery operators know where offline key backups are stored.

Fail criteria:

• The key is missing, unknown, or stored only on the failed system.
• The wrong encryption root is documented.
• The dataset imports but cannot be mounted because keys are unavailable.

Bare Import Drill#

Goal: prove that a pool can be imported on rescue media without mounting over the live system.

Import read-only and no-mount:

zpool import -d /dev/disk/by-id -N -o readonly=on -R /mnt pool0
zpool status -gLPv pool0
zfs list -r pool0
zpool export pool0

Pass criteria:

• The pool imports read-only with stable device paths.
• Datasets are visible and do not mount over the rescue environment.
• Rescue media supports the pool's enabled feature flags.

Fail criteria:

• The pool cannot import because rescue media is too old.
• Device paths are unstable or ambiguous.
• The procedure requires force import without proving the old host is offline.

Community FAQ: Top 20 Recurring ZFS Questions#

These are common ZFS questions that repeatedly appear on Reddit, Stack Overflow / Stack Exchange-style sites such as Server Fault, Super User, and Unix & Linux, plus forums and NAS communities. The answers here are kept practical and conservative.

1. What Is The Difference Between A Pool, Vdev, Dataset, And Zvol?#

pool0 is the storage pool. A vdev is a top-level redundancy group inside that pool, such as a mirror or RAIDZ2 group. pool0/volume0 is usually a filesystem dataset. A zvol is a block device created with zfs create -V, for example pool0/volume1.

zpool status pool0
zfs list
zfs list -t volume

2. Why Does One Bad Vdev Endanger The Whole Pool?#

ZFS redundancy is at the vdev level. If pool0 has two mirror vdevs, each mirror must remain healthy enough to serve data. If any top-level data vdev is lost, the whole pool can be lost. There is no extra parity layer above vdevs.

Better:

zpool create pool0 \
  mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1 \
  mirror /dev/disk/by-id/disk2 /dev/disk/by-id/disk3

Dangerous:

zpool create pool0 \
  mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1 \
  /dev/disk/by-id/disk2

3. Should ZFS Be Used On Top Of Hardware RAID?#

Normally no. ZFS works best when it sees individual disks, serial numbers, errors, latency, and flush behavior directly. Hardware RAID can hide failing drives, reorder writes, block SMART visibility, and make recovery harder.

Use an HBA or controller in IT/JBOD mode:

zpool create pool0 mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1

4. Can A Single-Disk Pool Become A Mirror Later?#

Yes. Use zpool attach, not zpool add.

zpool status pool0
zpool attach pool0 /dev/disk/by-id/disk0 /dev/disk/by-id/disk1
zpool status pool0

attach adds redundancy to an existing vdev. add creates a new top-level vdev.

5. I Accidentally Used zpool add Instead Of zpool attach. What Now?#

First, stop writing data and inspect the layout. Do not try to "balance" the mistake by adding more random disks.

zpool status pool0
zpool list -v pool0

If you added a removable top-level mirror or single-disk vdev and your OpenZFS version supports removal for that topology, zpool remove may work:

zpool remove pool0 /dev/disk/by-id/disk2

Before attempting removal:

• Make a fresh backup or confirm replication is current.
• Confirm the added vdev is not RAIDZ.
• Confirm your OpenZFS version supports removal for that exact layout.
• Expect removal to take time and create indirect mapping metadata.
• Keep monitoring with zpool status.

If the added vdev cannot be removed, the clean recovery is usually to back up, destroy and recreate the pool correctly, then restore.

6. Can RAIDZ1 Be Converted To RAIDZ2 Or RAIDZ3 In Place?#

No. RAIDZ expansion can widen a RAIDZ vdev on newer OpenZFS versions, but it does not change the parity level. RAIDZ1 stays RAIDZ1, RAIDZ2 stays RAIDZ2, and RAIDZ3 stays RAIDZ3.

To change parity level, create a new pool or new vdev and move the data:

zfs snapshot -r pool0@move-2026-05-14
zfs send -R pool0@move-2026-05-14 | zfs receive -u pool1/volume0

7. What Is The Safest Way To Expand A Pool?#

For mirrors, add another mirror vdev:

zpool add pool0 mirror /dev/disk/by-id/disk4 /dev/disk/by-id/disk5

For RAIDZ, add another complete RAIDZ vdev or use RAIDZ expansion only if your OpenZFS version supports it:

zpool get feature@raidz_expansion pool0
zpool attach pool0 raidz2-0 /dev/disk/by-id/disk12

Do not add a lone disk to a redundant pool.

8. Can A Vdev Be Removed?#

Sometimes. Top-level mirror and single-disk vdev removal may be supported on modern OpenZFS, but RAIDZ vdev removal is not a normal design path. Special vdevs are pool-critical, and removing them may be unsupported or impractical in many layouts.

Check before assuming:

zpool status pool0
zpool remove pool0 mirror-1

Plan pool topology as if top-level vdevs are permanent.

9. Why Did Deleting Files Not Free Space?#

Common causes are snapshots, clones, open deleted files, reservations, zvols, or refreservations. Start with snapshots.

zfs list -t snapshot -o name,used,refer
zfs list -o name,used,avail,refer,usedsnap,usedds,usedrefreserv

Destroy old snapshots only when they are no longer needed:

zfs destroy pool0/volume0@daily-2026-04-14

10. Why Do zpool list, zfs list, And df Show Different Space?#

They measure different layers. zpool list reports pool-level allocation. zfs list reports dataset-level space after ZFS accounting. df reports what the mounted filesystem presents to applications. Snapshots, reservations, parity, metadata, refreservations, and slop space can make the numbers differ.

Use ZFS tools first:

zpool list pool0
zfs list -o name,used,avail,refer,mountpoint
zfs get quota,reservation,refquota,refreservation pool0/volume0

11. Are Snapshots Backups?#

No. Snapshots are excellent local recovery points, but they live on the same pool. If pool0 is destroyed, stolen, overwritten, or lost, its snapshots are lost too. Replicate snapshots to another pool or host.

zfs snapshot -r pool0/volume0@daily-2026-05-14
zfs send -R pool0/volume0@daily-2026-05-14 | zfs receive -u pool1/volume0

12. How Should I Use zfs send And zfs receive?#

Use a full send first, then incremental sends. Let the first receive create the destination dataset.

zfs snapshot -r pool0/volume0@backup-2026-05-14
zfs send -R pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume0

zfs snapshot -r pool0/volume0@backup-2026-05-15
zfs send -R -I pool0/volume0@backup-2026-05-14 pool0/volume0@backup-2026-05-15 | zfs receive -u pool1/volume0

For encrypted datasets where the receiver should not decrypt data, use raw sends:

zfs send -w pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume0

13. Should Deduplication Be Enabled?#

Usually no. Deduplication needs careful memory, metadata, and workload planning. It can make writes, deletes, and pool recovery much slower. Compression is the right default for most systems.

The main risk is the deduplication table. If the DDT no longer fits well in RAM, ZFS has to read DDT entries from disk during writes and frees. On hard-disk pools this can turn normal work into heavy random I/O. Disabling dedup only affects new writes; existing deduped blocks keep their DDT entries until the old blocks and snapshots are removed or rewritten.

zfs set compression=zstd pool0/volume0
zfs get dedup pool0/volume0
zpool status -DD pool0

Only enable dedup after testing with representative data and a recovery plan.

14. What recordsize Should I Use?#

Set recordsize per dataset before writing data. Use larger records for large sequential files and smaller records for databases or VM image files.

zfs set recordsize=128K pool0/volume0  # general files
zfs set recordsize=1M pool0/volume2    # media, archives, torrents
zfs set recordsize=16K pool0/volume4   # some databases

Changing recordsize affects newly written blocks only. Existing data must be rewritten to adopt the new size.

15. Do I Need SLOG, L2ARC, Or A Special Vdev?#

Usually not at first. Add RAM, pick a good pool layout, and measure the workload before adding support vdevs.

• SLOG helps synchronous writes when backed by a fast, power-loss-protected device.
• L2ARC is a read cache and does not replace RAM.
• Special vdevs can speed metadata and small blocks, but they are critical to the pool and should be redundant.

Check sync behavior before buying a SLOG:

zfs get sync pool0/volume0
zpool iostat -v pool0 5

16. How Much RAM Does ZFS Need? Is ECC Required?#

There is no universal rule such as "1 GB RAM per 1 TB storage" for normal ZFS. More RAM improves ARC caching, metadata-heavy workloads, and dedup-heavy systems. ECC is strongly recommended for important data because ZFS can repair bad on-disk copies, but it cannot make unreliable memory reliable.

On Linux, ARC limits can be tuned through module options, but tune only after observing real memory pressure.

17. Why Are Writes Slow?#

Common causes include sync writes without a suitable SLOG, an over-wide RAIDZ vdev, SMR disks, a nearly full pool, small random writes, wrong recordsize or volblocksize, encryption CPU limits, weak controllers, bad cables, snapshots on busy zvols, or simply expecting mirror-like IOPS from RAIDZ.

Start with:

zpool status pool0
zpool iostat -v pool0 5
zfs get recordsize,volblocksize,sync,compression pool0/volume0
zpool list -o name,capacity,fragmentation,health pool0

18. What Is ashift, And Why Does Everyone Recommend ashift=12?#

ashift is the sector-size exponent used by a vdev. ashift=12 means 4096-byte sectors and is a safe default for most modern HDDs and SSDs, including many devices that report 512-byte logical sectors. It is set when the vdev is created and cannot be changed for that vdev later.

zpool create -o ashift=12 pool0 mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1

19. How Do I Import A Pool After Moving Disks?#

List importable pools:

zpool import

Import using stable names:

zpool import -d /dev/disk/by-id pool0

Import using short /dev names and mount below /mnt:

zpool import -d /dev -R /mnt pool0

For recovery, avoid mounting datasets immediately:

zpool import -d /dev -N -o readonly=on pool0

20. Why Is A Dataset Busy And Unable To Destroy, Export, Or Unmount?#

Something is still using it. Common causes are a shell with its current working directory inside the dataset, a running service, NFS or SMB sharing, a container mount, an open deleted file, a child dataset, a clone, or a snapshot hold.

Inspect before forcing anything:

zfs list -r pool0/volume0
zfs holds -r pool0/volume0
lsof +f -- /pool0/volume0
fuser -vm /pool0/volume0

Then stop the process, unshare the dataset, remove holds, or destroy dependent clones deliberately.

Common Mistakes#

Adding A Single Disk To A Redundant Pool#

Bad:

zpool add pool0 /dev/disk/by-id/disk4

This creates a new top-level vdev. If that one disk fails, the whole pool can fail even if the original vdevs are redundant.

Better:

zpool add pool0 mirror /dev/disk/by-id/disk4 /dev/disk/by-id/disk5

Assuming RAID Is Backup#

Redundancy protects against some disk failures. It does not protect against:

• Accidental deletion.
• Ransomware.
• Theft.
• Fire.
• Controller bugs.
• Admin mistakes.
• Silent application-level corruption already written to disk.

Use snapshots and backups.

Filling The Pool#

Bad:

zpool list pool0
# Capacity near 95%

Better:

zpool list -o name,capacity,free pool0

Plan expansion before the pool is full.

Ignoring Snapshots#

Snapshots can consume space when data changes.

Check snapshot space:

zfs list -t snapshot -o name,used,refer

Destroy old snapshots deliberately:

zfs destroy pool0/volume0@daily-2026-04-14

Enabling Dedup Without Planning#

Bad:

zfs set dedup=on pool0/volume0

Better:

zfs set compression=zstd pool0/volume0

Why: if the DDT grows beyond available ARC/RAM, writes and frees may need random metadata reads from disk. Turning dedup off later stops new dedup work, but old deduped blocks keep their DDT entries until those blocks are removed or rewritten.

Using Unstable Disk Names#

Bad:

zpool create pool0 mirror /dev/sdb /dev/sdc

Better:

zpool create pool0 mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1

Example Build: General Home Or Small Server#

Create a mirrored pool:

zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -O xattr=sa \
  -O acltype=posixacl \
  -m /pool0 \
  pool0 \
  mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1

Create datasets:

zfs create -o mountpoint=/pool0/volume0 pool0/volume0
zfs create -o mountpoint=/pool0/volume1 pool0/volume1
zfs create -o mountpoint=/pool0/volume2 pool0/volume2

Set properties:

zfs set recordsize=128K pool0/volume0
zfs set recordsize=128K pool0/volume1
zfs set recordsize=1M pool0/volume2
zfs set quota=500G pool0/volume1

Create initial snapshots:

zfs snapshot -r pool0@initial-2026-05-14

Check health:

zpool status pool0
zfs list

Example Build: Backup Pool#

Create pool1 as a RAIDZ2 backup pool:

zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool1 \
  pool1 \
  raidz2 \
  /dev/disk/by-id/disk0 \
  /dev/disk/by-id/disk1 \
  /dev/disk/by-id/disk2 \
  /dev/disk/by-id/disk3 \
  /dev/disk/by-id/disk4 \
  /dev/disk/by-id/disk5

Send a recursive backup from pool0/volume0. For the first full receive, pool1/volume0 should not already exist:

zfs snapshot -r pool0/volume0@backup-2026-05-14
zfs send -R pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume0

Export when using removable disks:

zpool export pool1

Example Build: VM Pool#

Create mirror-based VM pool:

zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool2 \
  pool2 \
  mirror /dev/disk/by-id/nvme0 /dev/disk/by-id/nvme1 \
  mirror /dev/disk/by-id/nvme2 /dev/disk/by-id/nvme3

Create VM dataset:

zfs create -o mountpoint=/pool2/volume0 pool2/volume0
zfs set recordsize=64K pool2/volume0

Create VM zvol:

zfs create -o volblocksize=16K -V 200G pool2/volume1

Snapshot before maintenance:

zfs snapshot pool2/volume1@before-maintenance-2026-05-14

Quick Reference#

Inventory disks before pool creation:

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
ls -l /dev/disk/by-id/
smartctl -a /dev/disk/by-id/disk0

Run SMART tests:

smartctl -t short /dev/disk/by-id/disk0
smartctl -t long /dev/disk/by-id/disk0

Check packaged ZFS automation units:

systemctl list-unit-files 'zfs*'
systemctl list-timers 'zfs*'

Check ZED event daemon on systemd systems:

systemctl status zfs-zed.service
zpool events -v

Pool health:

zpool status -x

Detailed status:

zpool status -v pool0

Command history:

zpool history -l pool0

Detailed recovery status with GUIDs and paths:

zpool status -gLPv pool0

Minimum recovery bundle:

zpool status -gLPv pool0
zpool list -v pool0
zpool events -v
zpool history -l pool0
zfs list -r pool0
zfs list -t snapshot -r pool0
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0

Performance baseline:

zpool status -v pool0
zpool list -o name,size,alloc,free,capacity,fragmentation,health
zpool iostat -v pool0 5
zfs get -r compression,recordsize,volblocksize,sync,logbias,primarycache,secondarycache,dedup pool0

Dedup table statistics and ARC residency:

zpool status -DD pool0

Inspect ZFS labels on a physical disk:

zdb -l /dev/disk/by-id/disk0

Inspect labels and uberblocks:

zdb -lu /dev/disk/by-id/disk0

Inspect an exported pool using a specific device directory:

zdb -e -p /dev/disk/by-id -C pool0

List pools:

zpool list

List datasets:

zfs list

Show property sources:

zfs get -r -o name,property,value,source compression,atime,relatime,xattr,recordsize pool0

Check ACL and extended attribute settings:

zfs get acltype,xattr pool0/volume0

Create dataset:

zfs create pool0/volume0

Create snapshot:

zfs snapshot pool0/volume0@daily-2026-05-14

List snapshots:

zfs list -t snapshot

Hold recovery snapshot:

zfs hold keep pool0/volume0@daily-2026-05-14

List snapshot holds:

zfs holds pool0/volume0@daily-2026-05-14

Compare snapshots:

zfs diff -FH pool0/volume0@daily-2026-05-14 pool0/volume0@daily-2026-05-15

Rollback:

zfs rollback pool0/volume0@daily-2026-05-14

Destroy snapshot:

zfs destroy pool0/volume0@daily-2026-05-14

Start scrub:

zpool scrub pool0

Stop scrub:

zpool scrub -s pool0

Initialize free regions:

zpool initialize pool0

Wait for background activity:

zpool wait -t scrub,resilver,initialize pool0

Export pool:

zpool export pool0

Import pool:

zpool import pool0

Import read-only without mounting:

zpool import -N -o readonly=on pool0

Import by pool ID under a temporary name:

zpool import 1234567890123456789 pool2

Force read-only no-mount import after verifying the old host is offline:

zpool import -f -N -o readonly=on pool0

Create pool checkpoint:

zpool checkpoint pool0

Discard pool checkpoint:

zpool checkpoint -d pool0

Rewind to pool checkpoint:

zpool export pool0
zpool import --rewind-to-checkpoint pool0

Check pool feature compatibility:

zpool upgrade
zpool get compatibility pool0

Document pool layout:

zpool status -gLPv pool0
zpool get all pool0
zfs get -r all pool0
zpool history -l pool0

Send snapshot:

# First full receive; pool1/volume0 should not already exist.
zfs send pool0/volume0@daily-2026-05-14 | zfs receive pool1/volume0

Incremental send:

zfs send -I pool0/volume0@daily-2026-05-14 pool0/volume0@daily-2026-05-15 | zfs receive pool1/volume0

Resume interrupted receive:

zfs get receive_resume_token pool1/volume0
zfs send -t TOKEN | zfs receive -s -u pool1/volume0

Inspect a send stream:

zstreamdump /safe-copy/pool0-volume0-2026-05-14.zfs

Set special small block threshold:

zfs set special_small_blocks=16K pool0/volume0

Add a hot spare:

zpool add pool0 spare /dev/disk/by-id/disk4

List events:

zpool events -v

Create a temporary restore target:

zfs send -R pool1/volume0@backup-2026-07-05 | zfs receive -u pool0/volume2

Load encryption key:

zfs load-key pool0/volume0

Maintenance Schedule#

Daily:

• Check alerts.
• Confirm free space is healthy.
• Confirm backups completed.

Weekly:

• Review zpool status.
• Watch for repeated device errors, failed SMART tests, and disks that drop offline.
• Review zpool events -v for new hardware or data errors.
• Review zpool history -l after maintenance or incidents.
• Confirm ZED, SMART, snapshot, scrub, and replication automation reported success.
• Review snapshot growth.
• Confirm backup replication.

Monthly:

• Run or verify scrub completion.
• Check SMART data.
• Test a small restore.
• Review the command risk matrix before planned topology, feature, or recovery changes.
• Check feature-flag compatibility before any planned zpool upgrade.
• Test loading encryption keys for encrypted recovery datasets.
• Review pool capacity trend.
• Confirm no old pool checkpoint was accidentally left behind.
• Review pool documentation after disk replacement or layout changes.

Quarterly:

• Test a full restore path.
• Practice the recovery lab on a throwaway host or VM.
• Run at least one restore drill template and record pass/fail results.
• Test import on current rescue media for critical pools.
• Verify resumable replication and backup documentation.
• Inspect one archived send stream and perform a test receive.
• Review retention policy.
• Confirm recovery media works.
• Review whether feature upgrades are needed.
• Recheck whether rescue media still supports all enabled pool features.

Final Best Practices Checklist#

• Use direct disk access, not hardware RAID.
• Use /dev/disk/by-id/ paths.
• Verify platform-specific service names, boot behavior, ACL behavior, and appliance rules before applying generic examples.
• Burn in and inventory new disks before creating important pools.
• Use mirrors for performance-sensitive workloads.
• Use RAIDZ2 or RAIDZ3 for large archive pools.
• Plan usable capacity with parity, snapshots, reservations, and free-space headroom.
• Use ashift=12 for modern disks unless you know otherwise.
• Enable compression=zstd by default.
• Use atime=off unless software needs access-time updates.
• Use xattr=sa on Linux datasets with POSIX ACLs or heavy extended attributes after checking compatibility needs.
• Create separate datasets for separate policies.
• Use property inheritance deliberately and review property sources.
• Set recordsize or volblocksize before writing data.
• Keep pools below 80% used.
• Schedule scrubs.
• Initialize new important pools when the extra I/O is acceptable.
• Monitor SMART health.
• Replace disks based on evidence, not guesswork, and resilver one disk at a time when possible.
• Read zpool status state, action, and error lines before changing anything.
• Use the command risk matrix before topology, feature, destroy, receive, import, rewind, or deep recovery commands.
• Keep annotated zpool status examples handy for incident triage.
• Snapshot automatically.
• Use zfs diff to inspect suspicious changes before rollback.
• Hold critical recovery snapshots during incidents.
• Replicate backups with zfs send.
• Treat replication as complete only after destination inspection and restore testing.
• Inspect archived send streams and still perform real test receives.
• Use resumable receives for large backup or restore streams.
• Treat redacted sends as advanced partial-copy workflows, not full backups.
• Use pool checkpoints only for short whole-pool maintenance rollback windows.
• Import read-only and without mounting when investigating damaged pools.
• Capture the minimum recovery bundle before changing pool state in an incident.
• Fill out an incident evidence form for failed imports, permanent errors, missing devices, ransomware, or major operator mistakes.
• Start recovery from the symptom-based runbook and move from inspection to change only when the path is clear.
• Force import only after verifying the pool is not active on another host.
• Clone failing disks before aggressive recovery attempts.
• Use zdb for offline inspection of labels, GUIDs, and uberblocks, not as a routine repair command.
• Test restores.
• Run restore drills and record pass/fail criteria.
• Configure ZED, SMART, scrub, snapshot, and replication automation where appropriate.
• Capture a performance baseline before tuning.
• Avoid dedup unless the dedup ratio, DDT size, RAM, and recovery plan are all acceptable.
• Use native encryption where appropriate.
• Keep encryption keys or passphrases recoverable offline.
• Configure ZFS event alerting.
• Review zpool history during incidents.
• Be careful with zpool upgrade and feature flags.
• Mirror special vdevs and keep special_small_blocks thresholds conservative.
• Use separate datasets, quotas, and snapshots for container state.
• Test hot spare behavior before relying on it.
• Replace failing disks promptly.
• Document pool layout, disk serials, and recovery steps.
• Keep platform-specific notes for boot pools, ACLs, service names, and rescue media.

References#

The official OpenZFS documentation should be preferred over forum posts when checking command syntax, feature availability, and destructive operation behavior. The community links are useful for examples and recurring questions, but always verify advice against the manual pages for your OpenZFS version.

Official OpenZFS documentation:

• OpenZFS Documentation
• OpenZFS master man pages
• ZFS concepts
• ZFS properties
• Zpool concepts
• Zpool properties
• Zpool feature flags
• OpenZFS messages

Pool administration:

• zpool-create(8)
• zpool-add(8)
• zpool-attach(8)
• zpool-detach(8)
• zpool-remove(8)
• zpool-replace(8)
• zpool-online(8)
• zpool-offline(8)
• zpool-status(8)
• zpool-list(8)
• zpool-iostat(8)
• zpool-clear(8)
• zpool-scrub(8)
• zpool-resilver(8)
• zpool-checkpoint(8)
• zpool-initialize(8)
• zpool-trim(8)
• zpool-wait(8)
• zpool-history(8)
• zpool-events(8)
• zpool-import(8)
• zpool-export(8)
• zpool-upgrade(8)
• zpool-labelclear(8)

Dataset, snapshot, and replication commands:

• zfs-create(8)
• zfs-destroy(8)
• zfs-set(8)
• zfs-get(8)
• zfs-inherit(8)
• zfs-list(8)
• zfs-mount(8)
• zfs-unmount(8)
• zfs-snapshot(8)
• zfs-hold(8)
• zfs-release(8)
• zfs-diff(8)
• zfs-clone(8)
• zfs-promote(8)
• zfs-rollback(8)
• zfs-send(8)
• zfs-receive(8)
• zfs-bookmark(8)
• zfs-redact(8)
• zfs-rewrite(8)
• zfs-load-key(8)
• zfs-unload-key(8)
• zfs-change-key(8)
• zfs-allow(8)
• zfs-unallow(8)

Debugging, recovery, and supporting OpenZFS tools:

• zdb(8)
• zstreamdump(8)
• zgenhostid(8)
• zed(8)
• zinject(8)

Platform and scheduler documentation:

• OpenZFS Getting Started: Debian
• OpenZFS Getting Started: Ubuntu
• Debian Package Search: zfsutils-linux
• Ubuntu Package Search: zfsutils-linux
• FreeBSD Handbook ZFS Chapter
• Proxmox VE Administration Guide
• Proxmox VE ZFS on Linux Wiki
• TrueNAS Documentation
• TrueNAS Storage Documentation
• systemd.timer(5)
• systemd.service(5)

Related tools mentioned in this guide:

• smartmontools
• e2fsprogs badblocks
• fio
• GNU ddrescue
• Sanoid and Syncoid
• zrepl
• zfs-auto-snapshot

Community discussion and Q&A portals:

• Reddit r/zfs
• Stack Overflow ZFS tag
• Server Fault ZFS tag
• Super User ZFS tag
• Unix & Linux Stack Exchange ZFS tag
• TrueNAS Community Forums
• Proxmox Forum ZFS tag

Disclaimer#

This guide was compiled and refined with LLM assistance, based on personal ZFS notes from the past 15 years.

• Signed: Grzegorz "SinusPL" Surmann

Copyright 2011-2026 by Grzegorz Surmann