# ZFS Filesystem Best Practices

- **Author:** Grzegorz "SinusPL" Surmann
- **Email:** zfs [at] funil [dot] de
- **Since:** June 2011

This guide is a practical ZFS quick reference. It focuses on everyday
administration, safe defaults, common pool layouts, snapshots, backups,
monitoring, recovery, and examples you can adapt.

Naming convention used in every example:

- Pools are named `pool0`, `pool1`, `pool2`, and so on.
- Filesystem datasets are named `volume0`, `volume1`, `volume2`, and so on.
- Block devices created with `zfs create -V` are also called volumes here, but the examples explicitly call them zvols when they are block devices.
- Disk identifiers are examples only. Prefer stable paths such as `/dev/disk/by-id/...` instead of `/dev/sdX`.

> Warning: Many ZFS commands can destroy data. Read every command before using
> it, replace the device names with the correct ones, and keep tested backups.

ZFS is a copy-on-write storage system that combines a filesystem, volume
manager, software RAID, checksumming, snapshots, replication, compression,
quotas, and optional native encryption into one coherent design. Its main
strength is data integrity: every block is checksummed, redundant pools can
repair bad copies automatically, snapshots make point-in-time recovery cheap,
and `zfs send` / `zfs receive` can replicate exact dataset history to another
pool or host. ZFS also makes everyday administration cleaner because storage is
organized into pools, vdevs, datasets, zvols, and properties instead of
separate RAID, partition, LVM, and filesystem layers. Its limits matter just as
much: ZFS is not a backup; it cannot save data after too many devices in the
same vdev fail, make an unsafe topology safe after the fact, protect against
bad commands or destroyed snapshots, or guarantee portability after newer pool
features are enabled. Good ZFS systems are planned around redundancy, stable
disk identifiers, free space, regular scrubs, tested backups, and datasets
designed for the workload.

## Table Of Contents

- [Core Ideas](#core-ideas)
- [Platform Notes](#platform-notes)
- [Platform Appendices](#platform-appendices)
  - [Linux And OpenZFS Packages](#linux-and-openzfs-packages)
  - [Debian](#debian)
  - [Ubuntu](#ubuntu)
  - [FreeBSD](#freebsd)
  - [TrueNAS](#truenas)
  - [Proxmox VE](#proxmox-ve)
- [What ZFS Is Not](#what-zfs-is-not)
- [Golden Rules](#golden-rules)
- [Command Risk Matrix](#command-risk-matrix)
- [Recommended Pool Layouts](#recommended-pool-layouts)
  - [Choosing Mirror Or RAIDZ](#choosing-mirror-or-raidz)
  - [Mirror Vdevs](#mirror-vdevs)
  - [Convert A Single-Disk Pool To A Mirror](#convert-a-single-disk-pool-to-a-mirror)
  - [RAIDZ1](#raidz1)
  - [RAIDZ2](#raidz2)
  - [RAIDZ3](#raidz3)
  - [What Not To Do](#what-not-to-do)
  - [Practical Capacity Math](#practical-capacity-math)
- [New Disk Burn-In And Acceptance Test](#new-disk-burn-in-and-acceptance-test)
- [Pool Creation Checklist](#pool-creation-checklist)
- [What To Document After Creating A Pool](#what-to-document-after-creating-a-pool)
- [Dataset Design](#dataset-design)
- [Common Dataset Properties](#common-dataset-properties)
  - [Property Glossary](#property-glossary)
  - [Property Inheritance](#property-inheritance)
  - [Compression](#compression)
  - [Access Time](#access-time)
  - [Extended Attributes And ACLs](#extended-attributes-and-acls)
  - [Record Size](#record-size)
  - [Quotas And Reservations](#quotas-and-reservations)
  - [Mountpoints](#mountpoints)
- [Everyday Commands](#everyday-commands)
- [Snapshots](#snapshots)
  - [Snapshot Holds](#snapshot-holds)
  - [Compare Snapshots With zfs diff](#compare-snapshots-with-zfs-diff)
- [Snapshot Retention](#snapshot-retention)
- [Pool Checkpoints](#pool-checkpoints)
- [Backups With ZFS Send And Receive](#backups-with-zfs-send-and-receive)
  - [Replication Safety](#replication-safety)
  - [Resume Interrupted Replication](#resume-interrupted-replication)
  - [ZFS Bookmarks For Replication](#zfs-bookmarks-for-replication)
  - [Inspect Send Streams With zstreamdump](#inspect-send-streams-with-zstreamdump)
  - [Redacted Sends](#redacted-sends)
- [Scrubs](#scrubs)
- [Pool Initialization](#pool-initialization)
- [Disk Replacement](#disk-replacement)
  - [When To Replace A Disk](#when-to-replace-a-disk)
  - [Hot Spares](#hot-spares)
- [Expanding Pools](#expanding-pools)
  - [Add Another Mirror Vdev](#add-another-mirror-vdev)
  - [Add A RAIDZ Vdev](#add-a-raidz-vdev)
  - [Expand An Existing RAIDZ Vdev](#expand-an-existing-raidz-vdev)
- [Import And Export](#import-and-export)
  - [Import Using A Specific Device Directory](#import-using-a-specific-device-directory)
  - [Import By Pool ID Or Temporary Name](#import-by-pool-id-or-temporary-name)
  - [Hostid And Active Pool Import Warnings](#hostid-and-active-pool-import-warnings)
- [Pool Feature Flags And Compatibility](#pool-feature-flags-and-compatibility)
- [Encryption](#encryption)
  - [Encrypted Dataset Recovery Checklist](#encrypted-dataset-recovery-checklist)
- [Zvols](#zvols)
- [Virtual Machines](#virtual-machines)
- [Databases](#databases)
- [Media And Archive Storage](#media-and-archive-storage)
- [Shares](#shares)
- [Delegation](#delegation)
- [ZFS For Containers](#zfs-for-containers)
- [Cache, Log, And Special Vdevs](#cache-log-and-special-vdevs)
  - [Special Small Blocks On Special Vdevs](#special-small-blocks-on-special-vdevs)
- [TRIM](#trim)
- [Monitoring](#monitoring)
  - [Command History For Auditing](#command-history-for-auditing)
  - [Reading zpool status](#reading-zpool-status)
  - [Annotated zpool status Examples](#annotated-zpool-status-examples)
  - [Interpreting ZFS Errors](#interpreting-zfs-errors)
  - [Events And Alerting](#events-and-alerting)
- [SMART Checks](#smart-checks)
- [Automation Examples](#automation-examples)
- [Performance Basics](#performance-basics)
  - [Performance Troubleshooting Decision Tree](#performance-troubleshooting-decision-tree)
- [Security And Permissions](#security-and-permissions)
- [Boot Pools](#boot-pools)
- [Disaster Recovery](#disaster-recovery)
  - [Recovery Decision Tree](#recovery-decision-tree)
  - [Symptom-Based Recovery Runbook](#symptom-based-recovery-runbook)
  - [Minimum Safe Recovery Bundle](#minimum-safe-recovery-bundle)
  - [Incident Evidence Form](#incident-evidence-form)
  - [Failed Pool Recovery Triage](#failed-pool-recovery-triage)
  - [Clone Failing Disks Before Recovery Attempts](#clone-failing-disks-before-recovery-attempts)
  - [Physical Pool Inspection With zdb](#physical-pool-inspection-with-zdb)
  - [Pool Rewind Recovery With -F](#pool-rewind-recovery-with-f)
  - [Recover A Destroyed Pool Entry](#recover-a-destroyed-pool-entry)
  - [Handling Permanent Data Errors](#handling-permanent-data-errors)
  - [Ransomware Or Mass Deletion Recovery](#ransomware-or-mass-deletion-recovery)
  - [Recovery Practice Lab](#recovery-practice-lab)
  - [Restore Drill Templates](#restore-drill-templates)
- [Community FAQ: Top 20 Recurring ZFS Questions](#community-faq-top-20-recurring-zfs-questions)
- [Common Mistakes](#common-mistakes)
- [Example Build: General Home Or Small Server](#example-build-general-home-or-small-server)
- [Example Build: Backup Pool](#example-build-backup-pool)
- [Example Build: VM Pool](#example-build-vm-pool)
- [Quick Reference](#quick-reference)
- [Maintenance Schedule](#maintenance-schedule)
- [Final Best Practices Checklist](#final-best-practices-checklist)
- [References](#references)
- [Disclaimer](#disclaimer)

## Core Ideas

ZFS combines a volume manager and a filesystem. Instead of creating a hardware
RAID array and then putting a filesystem on top, you normally give ZFS direct
access to the disks and let it manage redundancy, checksums, repair, snapshots,
compression, and replication.

Important terms:

- Pool: top-level storage container, for example `pool0`.
- Vdev: one redundancy group inside a pool, such as one mirror or one RAIDZ2 group.
- Dataset: a ZFS filesystem, for example `pool0/volume0`.
- Zvol: a ZFS block device, for example `pool0/volume2`.
- Snapshot: read-only point-in-time copy, for example `pool0/volume0@daily-2026-05-14`.
- Clone: writable dataset based on a snapshot.
- Scrub: online checksum verification and repair.
- Resilver: rebuild after replacing or adding redundancy to a device.

## Platform Notes

This guide is OpenZFS-focused. Command syntax is mostly portable, but defaults
and integration details vary by operating system and appliance.

Check your platform documentation for:

- Package names and service names.
- Default mount behavior and dataset mount ordering.
- ACL support, especially `acltype`, `aclmode`, and SMB/NFS integration.
- Bootloader support before enabling new pool feature flags on boot pools.
- Location of compatibility profiles and cache files.
- Native encryption support and key-loading service behavior.
- Distribution-specific root-on-ZFS tooling and boot environment workflows.

Examples in this guide use Linux-style paths such as `/dev/disk/by-id/`. On
FreeBSD, illumos, appliances, or rescue media, the stable device directory and
service commands may differ.

## Platform Appendices

These notes are deliberately conservative. Use them to avoid the most common cross-platform mistakes, then verify exact commands against your operating system or appliance documentation.

### Linux And OpenZFS Packages

Linux systems usually provide OpenZFS as distribution packages, DKMS modules, or prebuilt kernel modules. Service names, initramfs integration, boot pool support, and default timers vary by distribution.

Operational notes:

- Use `/dev/disk/by-id/` or another stable persistent path for pool creation and imports.
- Check the distribution's ZFS package documentation before enabling new feature flags on a boot pool.
- After changing root-on-ZFS, encryption, or boot pool properties, rebuild the initramfs or boot configuration if your distribution requires it.
- On systemd-based hosts, check whether packaged scrub, trim, import, mount, and `zed` units already exist before creating custom units.
- Keep kernel and ZFS packages compatible; avoid rebooting into a kernel that has no matching ZFS module.

Useful checks:

```bash
zfs version
modinfo zfs
systemctl list-unit-files 'zfs*'
systemctl status zfs-import-cache.service
systemctl status zfs-mount.service
systemctl status zfs-zed.service
```

### Debian

Debian deserves separate attention because ZFS packages are normally installed from `contrib`, and newer OpenZFS versions may come from backports. Kernel headers, DKMS, and package pinning matter more than on distributions that ship prebuilt ZFS modules for the running kernel.

Operational notes:

- Enable the Debian repository components required by your release, normally including `contrib`, before installing ZFS packages.
- Consider Debian backports when you need a newer OpenZFS release, but pin deliberately and document why the backport is used.
- Install matching kernel headers before or with `zfs-dkms`; a kernel without matching headers can leave the ZFS module unbuilt after upgrades.
- Keep at least one known-working kernel installed until the new kernel boots and loads ZFS successfully.
- Root-on-ZFS on Debian should follow a current Debian-specific root-on-ZFS guide, not a generic Linux guide.
- After kernel, initramfs, bootloader, encryption, or root dataset changes, rebuild and test the boot path before treating the host as recovered.

Useful checks:

```bash
apt-cache policy zfsutils-linux zfs-dkms
apt-cache policy linux-image-amd64 linux-headers-amd64
dkms status
modprobe zfs
zfs version
systemctl list-unit-files 'zfs*'
```

Example package install for a normal non-root pool, after repositories are correct for your Debian release:

```bash
apt update
apt install linux-headers-amd64 zfs-dkms zfsutils-linux
```

Example backports sanity check, replacing `trixie-backports` with your actual release backports suite:

```bash
apt-cache policy -t trixie-backports zfsutils-linux zfs-dkms
```

Best practice:

- Do not mix random third-party ZFS packages with Debian packages unless you have a clear rollback plan.
- Check `dkms status` after every kernel upgrade.
- Before rebooting a remote Debian ZFS host, confirm that the new kernel has a built ZFS module and that an older working kernel remains available.
- For rescue media, verify that it can import the pool feature set used by the Debian host.

### Ubuntu

Ubuntu also uses `zfsutils-linux`, but its packaging and kernel integration differ from Debian. OpenZFS documentation notes that Ubuntu includes ZFS support in its default Linux kernel packages, while the command-line tools are installed with `zfsutils-linux` after the required repository component, commonly `universe`, is available.

Operational notes:

- Enable `universe` if `zfsutils-linux` is not visible to APT.
- Install `zfsutils-linux` for command-line administration tools.
- Do not assume Debian DKMS instructions apply directly to Ubuntu; check whether the running Ubuntu kernel already provides the ZFS module package path for that release.
- For root-on-ZFS, follow Ubuntu release-specific guidance and verify bootloader, initramfs, encrypted root, and feature flag compatibility before upgrades.
- Ubuntu desktop/server releases, HWE kernels, and cloud images can differ; check the running kernel and package source before troubleshooting module load failures.
- If Secure Boot or custom kernels are involved, verify module loading and signing behavior before relying on ZFS at boot.

Useful checks:

```bash
apt-cache policy zfsutils-linux
apt-cache policy linux-generic linux-image-generic
modinfo zfs
zfs version
systemctl list-unit-files 'zfs*'
systemctl status zfs-import-cache.service
systemctl status zfs-zed.service
```

Install tools after repositories are correct:

```bash
apt update
apt install zfsutils-linux
```

Best practice:

- Keep release upgrades conservative on boot pools and test rescue media before enabling new pool features.
- Confirm `zpool import` behavior after major Ubuntu upgrades, especially on encrypted, root-on-ZFS, or multi-pool hosts.
- Check Ubuntu package versions before copying commands between LTS releases.
- Treat Ubuntu installer-created layouts as release-specific; document dataset names, boot pool layout, and recovery commands immediately after installation.

### FreeBSD

FreeBSD integrates ZFS deeply, including root-on-ZFS and boot environments on many installations. Device names, service management, loader behavior, and periodic jobs differ from Linux examples.

Operational notes:

- Expect disk paths such as `/dev/gpt/...`, `/dev/diskid/...`, `/dev/ada0`, `/dev/da0`, or GEOM labels instead of Linux `/dev/disk/by-id/...`.
- Use FreeBSD boot environment tooling where available instead of hand-rolling root dataset rollback.
- Check loader and bootcode support before enabling new feature flags on boot pools.
- Review FreeBSD periodic and service configuration before adding duplicate scrub or snapshot automation.
- Use FreeBSD ACL and jail documentation when mixing ZFS datasets with jails, SMB, or NFS exports.

Useful checks:

```bash
zpool status -gLPv pool0
zpool get bootfs pool0
zfs list -o name,mountpoint,canmount
service zfs onestatus
sysrc zfs_enable
```

### TrueNAS

TrueNAS is an appliance platform. It exposes ZFS through a web interface, middleware, alerts, tasks, replication jobs, sharing services, and API workflows. Shell commands are useful for inspection, but persistent configuration should normally be changed through the UI or supported API.

Operational notes:

- Prefer the TrueNAS UI or API for pools, datasets, shares, snapshots, replication tasks, alerts, scrub schedules, and SMART tests.
- Do not manually edit middleware-managed configuration files unless TrueNAS documentation or support tells you to.
- Use shell commands for read-only investigation, evidence capture, and support bundles.
- Be careful importing pools created elsewhere; feature flags, encryption, ACLs, and share metadata may not match the appliance version.
- For recovery, export shares or stop clients first, then use read-only inspection before changing pool state.

Useful checks:

```bash
zpool status -gLPv pool0
zpool events -v
zfs list -r pool0
zfs list -t snapshot -r pool0
```

### Proxmox VE

Proxmox VE often uses ZFS for host storage, VM disks, container root filesystems, replication, and sometimes root-on-ZFS. Proxmox also tracks storage definitions and guest disk ownership outside plain ZFS metadata.

Operational notes:

- Prefer Proxmox storage configuration tools, UI, and documented CLI workflows for VM and container storage.
- Do not rename, destroy, or receive over VM backing datasets or zvols without checking Proxmox storage metadata and guest configuration.
- For VM storage, mirrors usually give better latency and rebuild behavior than wide RAIDZ.
- Check bootloader and Proxmox release notes before upgrading feature flags on root pools.
- Coordinate ZFS snapshots with guest-aware backups when application consistency matters.

Useful checks:

```bash
zpool status -gLPv pool0
zfs list -t filesystem,volume -r pool0
pvesm status
qm config 100
pct config 100
```

## What ZFS Is Not

ZFS is strong storage software, but it is not magic.

- ZFS is not a backup. Snapshots in the same pool can disappear with the pool.
- ZFS is not protection against every admin mistake. A valid destructive command can destroy valid data.
- ZFS is not a substitute for SMART monitoring, cabling checks, cooling, or replacing failing disks.
- ZFS is not a way to make a weak vdev topology safe after the fact.
- ZFS is not guaranteed portable after enabling new feature flags.
- ZFS is not recovery from losing encryption keys.
- ZFS is not immune to ransomware if the attacker can destroy snapshots or reach backup targets.

## Golden Rules

1. Use ECC RAM when possible, especially for large pools or important data.
2. Use reliable disks and monitor them. ZFS protects against many failures, not against neglect.
3. Never build ZFS on top of hardware RAID. Use HBA or IT mode so ZFS can see each disk directly.
4. Use stable disk paths from `/dev/disk/by-id/`.
5. Plan vdev width and redundancy before creating the pool. You cannot remove a RAIDZ vdev from a pool in normal designs.
6. Do not fill pools. Keep at least 20% free space for performance and recovery room.
7. Use snapshots, but do not treat snapshots as backups.
8. Back up to another pool, system, disk, or remote host with `zfs send`.
9. Scrub regularly and read the results.
10. Test restores, not only backups.
11. Set properties at creation time when possible.
12. Keep pools, operating system packages, and boot environments maintained.
13. Do not casually run `zpool upgrade` if the pool must remain importable on older systems.
14. Avoid mixing very different disk sizes or speeds inside one vdev.
15. Replace failing disks early. Redundancy is not a reason to wait.

## Command Risk Matrix

Use this matrix before copying commands from notes, forums, old shell history, or AI output. The safest ZFS workflow is usually inspect, document, decide, then change.

| Risk Level | Examples | What It Can Change | Minimum Safe Habit |
| --- | --- | --- | --- |
| Read-only inspection | `zpool status`, `zpool list`, `zfs list`, `zfs get`, `zpool events`, `zpool history`, `zdb -l` | Nothing intended | Safe to run, but still verify pool names and device names |
| Background verification | `zpool scrub`, `zpool wait`, `smartctl -t long` | Reads most or all data and can stress weak hardware | Check SMART and cabling first when disks are already suspect |
| Routine dataset administration | `zfs create`, `zfs set`, `zfs inherit`, `zfs snapshot`, `zfs hold` | Dataset properties, snapshots, retention behavior, mount behavior | Verify dataset path and inherited properties before running recursively |
| Hardware repair with redundancy | `zpool offline`, `zpool online`, `zpool replace`, `zpool attach`, `zpool detach` | Vdev membership, resilver behavior, redundancy state | Capture `zpool status -gLPv`, verify serials, and change one device at a time |
| Pool layout change | `zpool create`, `zpool add`, `zpool remove`, special vdev addition, cache/log vdev addition | Permanent pool topology or metadata placement | Back up first, verify every device path, and understand whether the change can be undone |
| Destructive data change | `zfs destroy`, `zpool destroy`, recursive destroy, `zfs rollback`, `zfs receive -F` | Datasets, snapshots, pool labels, or newer data | Require explicit intent, list targets first, and confirm backups or snapshots outside the blast radius |
| Import override or rewind | `zpool import -f`, `zpool import -F`, `zpool import -X`, checkpoint rewind, `zpool import -m` | Ownership checks, recent transactions, or synchronous writes | Prefer read-only no-mount imports and dry runs; use only after old hosts are offline and loss is acceptable |
| Feature compatibility change | `zpool upgrade`, enabling new feature flags, changing `compatibility` | Future importability on older systems, appliances, and rescue media | Test rescue media and all target hosts before upgrading |
| Deep recovery/debugging | `zdb -F`, `zdb -X`, advanced `zdb` object extraction | Can become destructive or depend on internals | Work from cloned disks or images and prefer backup restore when possible |
| Error clearing | `zpool clear`, `zpool events -c` | Removes visible counters or event history | Capture status, events, history, and SMART data before clearing |

Practical rule: if a command changes topology, destroys data, rewinds transactions, overwrites a receive target, clears evidence, or changes feature compatibility, stop and write down exactly what you expect to happen before running it.

## Recommended Pool Layouts

### Choosing Mirror Or RAIDZ

Use this table as a starting point, then adjust for budget, recovery time,
capacity, and workload.

| Need | Prefer | Why |
| --- | --- | --- |
| VM storage, databases, containers, and random I/O | Mirror vdevs | Better IOPS, faster resilver, easier expansion |
| General small server with two disks | Two-way mirror | Simple redundancy and easy replacement |
| Important small server with three disks | Three-way mirror | Can survive two mirror-side failures |
| Large media or archive pool | RAIDZ2 or RAIDZ3 | Better capacity efficiency for mostly sequential data |
| Very large disks or long rebuild windows | RAIDZ2, RAIDZ3, or mirrors | Avoid single-parity risk during recovery |
| Backup pool with retained history | RAIDZ2 or RAIDZ3 | Capacity matters, but redundancy still matters |
| Workload with many small synchronous writes | Mirrors first, then measure SLOG need | Layout usually matters more than adding support vdevs |

Avoid RAIDZ1 for new pools with large disks or important data. It can be
acceptable for disposable data, scratch pools, or tiny lab systems, but it is
not a conservative default.

### Mirror Vdevs

Mirrors are usually the best general-purpose layout for home labs, small
servers, VM storage, databases, and workloads with random I/O.

Example: create `pool0` from two mirrored disks.

```bash
zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -O xattr=sa \
  -O acltype=posixacl \
  -m /pool0 \
  pool0 \
  mirror \
  /dev/disk/by-id/disk0 \
  /dev/disk/by-id/disk1
```

Example: create `pool0` from two mirror vdevs.

```bash
zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool0 \
  pool0 \
  mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1 \
  mirror /dev/disk/by-id/disk2 /dev/disk/by-id/disk3
```

Why use mirrors:

- Fast rebuilds compared with wide RAIDZ.
- Good random read and write performance.
- Easy expansion by adding another mirror vdev.
- Simpler replacement and growth strategy.

Tradeoff:

- A two-way mirror gives 50% usable capacity.

### Convert A Single-Disk Pool To A Mirror

If `pool0` was created from one disk, you can attach a second disk to turn the
plain single-disk vdev into a mirror. This is useful when you started without
redundancy and want to add it later.

Check the current device name:

```bash
zpool status pool0
```

Attach a second disk:

```bash
zpool attach pool0 /dev/disk/by-id/disk0 /dev/disk/by-id/disk1
```

Watch the resilver:

```bash
zpool status pool0
```

After the resilver completes, `pool0` has mirror redundancy.

Example before:

```text
pool0
  /dev/disk/by-id/disk0
```

Example after:

```text
pool0
  mirror-0
    /dev/disk/by-id/disk0
    /dev/disk/by-id/disk1
```

If `zpool status` shows a short device name because the pool was imported that
way, use the name shown there:

```bash
zpool attach pool0 sdb /dev/disk/by-id/disk1
```

Best practice:

- Back up first. The original single disk is still a single point of failure until the resilver completes.
- Use a new disk that is at least as large as the existing disk.
- Use stable `/dev/disk/by-id/` names for the new disk when possible.
- Do not confuse `attach` with `add`. `attach` mirrors an existing vdev; `add` creates another top-level vdev.

### RAIDZ1

RAIDZ1 is single-parity RAIDZ. It is usually only acceptable for small,
non-critical pools with small disks and good backups.

Example:

```bash
zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool0 \
  pool0 \
  raidz1 \
  /dev/disk/by-id/disk0 \
  /dev/disk/by-id/disk1 \
  /dev/disk/by-id/disk2
```

Best practice:

- Prefer mirrors or RAIDZ2 for important data.
- Avoid RAIDZ1 with large modern disks when the data matters.

### RAIDZ2

RAIDZ2 uses two parity disks per vdev. It is a good choice for larger media,
archive, and backup pools.

Example:

```bash
zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool0 \
  pool0 \
  raidz2 \
  /dev/disk/by-id/disk0 \
  /dev/disk/by-id/disk1 \
  /dev/disk/by-id/disk2 \
  /dev/disk/by-id/disk3 \
  /dev/disk/by-id/disk4 \
  /dev/disk/by-id/disk5
```

Best practice:

- Use RAIDZ2 instead of RAIDZ1 for valuable data on large disks.
- Keep vdevs at reasonable widths. Common widths are 6, 8, 10, or 12 disks.
- Expansion normally means adding another RAIDZ2 vdev of similar width.

### RAIDZ3

RAIDZ3 uses three parity disks per vdev. It is suitable for large, slower,
high-capacity archive pools where rebuild time is long and capacity matters
more than write performance.

Example:

```bash
zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool0 \
  pool0 \
  raidz3 \
  /dev/disk/by-id/disk0 \
  /dev/disk/by-id/disk1 \
  /dev/disk/by-id/disk2 \
  /dev/disk/by-id/disk3 \
  /dev/disk/by-id/disk4 \
  /dev/disk/by-id/disk5 \
  /dev/disk/by-id/disk6 \
  /dev/disk/by-id/disk7
```

### What Not To Do

Avoid these layouts unless you fully understand the risk:

```bash
# No redundancy. Any disk failure can destroy the pool.
zpool create pool0 /dev/disk/by-id/disk0 /dev/disk/by-id/disk1
```

```bash
# Hardware RAID hides disks and errors from ZFS.
zpool create pool0 /dev/disk/by-id/hardware-raid-volume0
```

```bash
# Unstable device names can change after reboot.
zpool create pool0 /dev/sdb /dev/sdc
```

### Practical Capacity Math

ZFS capacity is easier to plan when you separate raw disk capacity, vdev
redundancy, filesystem overhead, snapshots, reservations, and free-space
headroom. Vendor disk sizes are decimal TB, while many tools display binary
TiB, so a "12 TB" disk appears as about 10.9 TiB before ZFS overhead.

Rough usable capacity before metadata, snapshots, compression, and free-space
headroom:

| Layout | Rough Usable Capacity |
| --- | --- |
| Single disk | Size of that disk, with no redundancy |
| Two-way mirror | Size of one disk |
| Three-way mirror | Size of one disk, with two redundant copies |
| Multiple mirror vdevs | Sum of one side from each mirror |
| RAIDZ1 with `N` disks | `(N - 1) * smallest disk` |
| RAIDZ2 with `N` disks | `(N - 2) * smallest disk` |
| RAIDZ3 with `N` disks | `(N - 3) * smallest disk` |

Example rough math:

```text
6 x 12 TB RAIDZ2 raw vendor size: 72 TB
Parity equivalent: 2 disks
Rough usable before overhead: 4 x 12 TB = 48 TB
Displayed binary capacity before overhead: about 43.7 TiB
Practical planning target at 80% full: about 35 TiB used
```

Why real numbers differ:

- ZFS stores metadata, checksums, labels, spacemaps, and indirect blocks.
- RAIDZ padding and small-block workloads can reduce efficiency.
- Snapshots keep old blocks alive after files are modified or deleted.
- Quotas, reservations, and zvol refreservations can make free space look lower than expected.
- Compression can make logical data larger than physical pool usage.
- Keeping free space is intentional; performance and recovery get worse when a pool is nearly full.

Useful checks:

```bash
zpool list -o name,size,alloc,free,capacity,fragmentation,health
zfs list -o name,used,avail,refer,usedbysnapshots,usedbydataset,usedbychildren,usedbyrefreservation
zfs get logicalused,used,compressratio pool0/volume0
```

Best practice:

- Plan around 70% to 80% normal usage, not theoretical maximum capacity.
- Count snapshot retention in capacity planning.
- Do not mix very small and very large disks inside one vdev unless the lost capacity is acceptable.
- For backup pools, plan capacity for both current data and retained history.

## New Disk Burn-In And Acceptance Test

Test new or reused disks before trusting them with a pool. Burn-in does not
prove a disk is perfect, but it often catches shipping damage, weak sectors,
bad cables, overheating, and wrong device identification before the pool
contains data.

Start with inventory and SMART data:

```bash
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0
```

Run a short SMART test:

```bash
smartctl -t short /dev/disk/by-id/disk0
smartctl -a /dev/disk/by-id/disk0
```

Run a long SMART test:

```bash
smartctl -t long /dev/disk/by-id/disk0
smartctl -a /dev/disk/by-id/disk0
```

For a destructive write/read test on a new empty disk, use a tool appropriate
for your platform. Replace the device name only after verifying it several
times.

Warning: the next example destroys data on the target device. Use it only on a
new or intentionally wiped disk.

```bash
badblocks -wsv /dev/disk/by-id/disk0
```

For SSDs or production-like tests, a workload tool such as `fio` is often more
useful than `badblocks`. Keep the test destructive only when the disk is empty
and intentionally selected.

Acceptance checklist:

- Verify model, serial, size, and stable `/dev/disk/by-id/` name.
- Check SMART before and after stress testing.
- Watch temperature during long tests.
- Replace suspect cables, trays, HBAs, or power paths before blaming ZFS.
- Reject disks with new reallocated, pending, or uncorrectable sectors.
- Record disk serials and intended vdev membership before `zpool create`.

## Pool Creation Checklist

Before creating a pool:

- Confirm the disks with `lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE`.
- Use `/dev/disk/by-id/` paths.
- Decide mirror, RAIDZ2, or RAIDZ3 before writing data.
- Use `ashift=12` for most modern disks and SSDs.
- Use `compression=zstd` on current OpenZFS unless you have a specific reason not to.
- Use `atime=off` for most server datasets on both HDDs and SSDs.
- Use `acltype=posixacl` and `xattr=sa` on Linux when POSIX ACLs or many extended attributes are needed.
- Decide whether the pool needs encryption.
- Decide mountpoints before creating many datasets.

Preview disk identifiers:

```bash
ls -l /dev/disk/by-id/
```

Show block devices:

```bash
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,MOUNTPOINTS
```

Check existing pools:

```bash
zpool list
zpool status
```

Before running `zpool create`:

- Confirm every device path points to the intended disk.
- Confirm the disks are not mounted and are not part of another active pool.
- Confirm backups exist if the disks ever held data.
- Confirm the vdev layout is the one you intend to live with.
- Save the exact command in your notes before pressing Enter.

Warning: `zpool create` writes labels and creates a new pool on the selected
devices. A wrong device path can destroy data.

## What To Document After Creating A Pool

Write down the pool design while the details are still fresh. This makes later
disk replacement, import, recovery, and audits much easier.

Record:

- Pool name, creation date, host, and operating system.
- OpenZFS version and package source.
- Pool layout, vdev type, vdev width, and redundancy level.
- Disk model, serial, size, and `/dev/disk/by-id/` path for every device.
- `ashift`, pool compatibility setting, and enabled feature policy.
- Dataset tree, mountpoints, key properties, and snapshot policy.
- Encryption roots, key format, and offline key recovery location.
- Backup target, replication command, and retention policy.
- Rescue media version known to import the pool.
- Expected import command for recovery, preferably read-only and no-mount.

Useful capture commands:

```bash
zpool status -gLPv pool0
zpool get all pool0
zfs get -r all pool0
zpool history -l pool0
```

## Dataset Design

Do not put everything directly in the root of the pool. Create datasets for
different data types so you can apply different snapshots, quotas, compression,
record sizes, mountpoints, and backup policies.

Example dataset layout:

```text
pool0
pool0/volume0   # general files
pool0/volume1   # user files
pool0/volume2   # media or archives
pool0/volume3   # virtual machines
pool0/volume4   # databases
pool0/volume5   # backups
```

Create filesystem datasets:

```bash
zfs create -o mountpoint=/pool0/volume0 pool0/volume0
zfs create -o mountpoint=/pool0/volume1 pool0/volume1
zfs create -o mountpoint=/pool0/volume2 pool0/volume2
```

List datasets:

```bash
zfs list
```

Show important properties:

```bash
zfs get compression,atime,recordsize,mountpoint,quota,reservation pool0/volume0
```

## Common Dataset Properties

### Property Glossary

These are the properties and settings most often mentioned in this guide.

| Property Or Setting | What It Controls | Practical Note |
| --- | --- | --- |
| `ashift` | Vdev sector size exponent | Set at vdev creation; `ashift=12` is a safe default for most modern disks |
| `recordsize` | Maximum filesystem data block size | Set before writing data; use larger values for media, smaller values for databases only when needed |
| `volblocksize` | Zvol block size | Set when creating a zvol; match VM or database workload before writing data |
| `compression` | Transparent block compression | `zstd` is a good default on current OpenZFS |
| `atime` | Access-time updates on reads | `off` reduces metadata writes for most server datasets |
| `relatime` | Reduced access-time update behavior | Use only when software needs approximate access-time behavior |
| `xattr` | Extended attribute storage | `sa` is efficient on Linux/OpenZFS when compatible |
| `acltype` | ACL implementation | Use `posixacl` on Linux when POSIX ACLs are needed |
| `aclmode` | How chmod interacts with ACLs | Platform-sensitive; check SMB/NFS and OS behavior |
| `quota` | Maximum logical space a dataset and children can use | Good for containing users, containers, and applications |
| `refquota` | Maximum space for one dataset, excluding children | Useful when child datasets have separate limits |
| `reservation` | Guaranteed space for a dataset and children | Can make free space look lower; use deliberately |
| `refreservation` | Guaranteed space for one dataset or zvol | Common with zvols; affects visible free space |
| `mountpoint` | Where a dataset mounts | Check after replication because sends can preserve source properties |
| `canmount` | Whether a dataset can be mounted | Useful for parent datasets such as `canmount=off` |
| `readonly` | Whether writes are allowed | Useful for protection and recovery inspection |
| `copies` | Extra copies of blocks inside the same pool | Not a replacement for redundancy or backups |
| `sync` | Synchronous write behavior | Do not set `sync=disabled` unless data loss is acceptable |
| `primarycache` | ARC caching policy | Leave default unless measuring a specific workload |
| `logbias` | Hint for latency or throughput behavior | Usually leave default; relevant with sync workloads |
| `special_small_blocks` | Small blocks eligible for special vdev allocation | Use conservative thresholds and only with redundant special vdevs |
| `encryption` | Native dataset encryption | Set when creating an encryption root |
| `keyformat` | Encryption key type | Common values are `passphrase`, `hex`, and `raw` |
| `keylocation` | Where ZFS loads an encryption key from | Keep offline recovery copies of keys or passphrases |
| `dedup` | Block-level deduplication | Usually avoid unless memory, DDT size, and recovery plan are acceptable |

### Property Inheritance

Most editable ZFS dataset properties inherit from the parent dataset unless a
child dataset overrides them. This is useful for setting broad defaults at the
pool or parent level, then changing only the datasets that need different
behavior.

Set defaults on `pool0`:

```bash
zfs set compression=zstd pool0
zfs set atime=off pool0
```

Override one child dataset:

```bash
zfs set recordsize=16K pool0/volume4
```

Show property values and where they came from:

```bash
zfs get -r -o name,property,value,source compression,atime,recordsize pool0
```

Show only local overrides:

```bash
zfs get -r -s local all pool0
```

Remove a local override so the dataset inherits again:

```bash
zfs inherit compression pool0/volume0
zfs inherit -r atime pool0/volume1
```

Received properties from `zfs receive` can also affect behavior. To inspect
them, include the `received` column:

```bash
zfs get -o name,property,value,received,source compression,mountpoint pool1/volume0
```

Best practice:

- Set common defaults on parent datasets.
- Use local overrides only when a workload needs them.
- Review `source` when a property does not behave as expected.
- Be careful with inherited `mountpoint`, `sharenfs`, `sharesmb`, `quota`, and `reservation`; they can affect many children at once.

### Compression

Use Zstandard compression by default on current OpenZFS systems. `zstd` is a
good modern default because it usually compresses better than `lz4` while still
offering good performance; in OpenZFS, `zstd` is equivalent to `zstd-3`.

```bash
zfs set compression=zstd pool0
zfs set compression=zstd pool0/volume0
```

Use `lz4` when you need the lowest CPU overhead, have older systems that must
import the pool, or have latency-sensitive VM/database workloads where testing
shows `zstd` costs too much CPU.

```bash
zfs set compression=lz4 pool0/volume3
```

Use stronger Zstandard levels for cold or archival datasets. Higher levels can
save more space, but they cost more CPU during writes.

```bash
zfs set compression=zstd-6 pool0/volume2
```

Check compression:

```bash
zfs get compressratio,compression pool0/volume2
```

Common compression choices:

| Setting | Best Use | Notes |
| --- | --- | --- |
| `zstd` | New general-purpose datasets | Good default on current OpenZFS; same as `zstd-3`. |
| `zstd-1` | Faster Zstandard | Lower CPU than default `zstd`, usually less compression. |
| `zstd-6` | Cold data and archives | Better compression, more write CPU. |
| `zstd-fast` | Fast Zstandard mode | Useful when `lz4` is too light but regular `zstd` is too costly. |
| `lz4` | Old systems, weak CPUs, very low latency | Very fast and still a safe conservative fallback. |
| `gzip` / `gzip-N` | Legacy compatibility only | Usually not worth using now; `zstd` is generally better. |
| `zle` | Mostly-zero data | Compresses runs of zeros only. |
| `off` | Rare exceptions | Usually avoid disabling compression. |

Best practice:

- Set compression before writing data.
- Changing compression affects newly written blocks only.
- To recompress old data, rewrite it or replicate it to a new dataset.
- Already-compressed media, backups, and archives may not shrink much, but ZFS will store blocks uncompressed when compression is not useful.
- Check OpenZFS feature compatibility before using `zstd` on pools that must be imported by older systems.

### Access Time

Disable access time updates for most server datasets. In ZFS the property is
`atime=off`; this is the ZFS equivalent of a `noatime` mount option.

```bash
zfs set atime=off pool0
```

Why this matters: with `atime=on`, reading a file can also update metadata to
record the access time. On HDD pools this can turn read-heavy workloads into
extra random writes and seeks. On SSD pools the latency cost is usually smaller,
but disabling unnecessary access-time writes still reduces metadata churn and
write amplification.

If an application needs approximate access-time behavior, use `atime=on` with
`relatime=on`:

```bash
zfs set atime=on pool0/volume0
zfs set relatime=on pool0/volume0
```

If an application really needs every read to update access time, use
`relatime=off`, but expect more write activity:

```bash
zfs set atime=on pool0/volume0
zfs set relatime=off pool0/volume0
```

Best practice:

- Use `atime=off` for most file servers, backup datasets, media datasets, VM image datasets, and databases.
- Keep `atime=on` only for software that actually uses access times, such as some mail, archival, audit, or cleanup workflows.
- Do not describe this as an SSD-only or HDD-only setting. HDDs benefit more visibly, but SSDs also avoid unnecessary writes.

### Extended Attributes And ACLs

The ZFS property is `xattr`, not `xattrs`. Extended attributes are used by
POSIX ACLs, SELinux labels, Samba metadata, desktop file managers, container
runtimes, and some backup tools.

On Linux, a common modern setting for datasets that use POSIX ACLs or many
extended attributes is:

```bash
zfs set acltype=posixacl pool0/volume0
zfs set xattr=sa pool0/volume0
```

`xattr=sa` stores extended attributes as ZFS system attributes instead of
placing them in hidden directory-style objects. This can reduce extra metadata
I/O, especially on HDD pools where seeks are expensive. SSD pools can also
benefit, but the difference is usually less dramatic than on spinning disks.

Check current settings:

```bash
zfs get acltype,xattr pool0/volume0
```

Compatibility note:

- `xattr=sa` is best for many current Linux/OpenZFS systems.
- If a pool must be imported by older systems, appliances, or non-Linux platforms, verify support before relying on `xattr=sa`.
- `xattr=on` is the more conservative compatibility choice when you do not know where the pool will be imported.

Best practice:

- Use `xattr=sa` together with `acltype=posixacl` for Linux datasets that need POSIX ACLs, Samba ACL metadata, SELinux labels, or container metadata.
- Set it at dataset creation time when possible.
- Do not expect it to fix a bad pool layout. It reduces metadata overhead; it does not replace RAM, mirrors, or good disk latency.
- For simple media or backup datasets with no ACL or extended-attribute needs, the performance difference may be small.

### Record Size

`recordsize` affects filesystem datasets. It does not affect zvols.

General files:

```bash
zfs set recordsize=128K pool0/volume0
```

Large media and archives:

```bash
zfs set recordsize=1M pool0/volume2
```

Databases with small random I/O:

```bash
zfs set recordsize=16K pool0/volume4
```

Virtual machine image files:

```bash
zfs set recordsize=64K pool0/volume3
```

Best practice:

- Set `recordsize` before writing data.
- Changing `recordsize` affects newly written blocks only.
- Match database record size to the database page size when possible.

### Quotas And Reservations

A quota limits maximum dataset usage.

```bash
zfs set quota=500G pool0/volume0
```

A reservation guarantees space to a dataset.

```bash
zfs set reservation=100G pool0/volume1
```

Reference quotas limit only the dataset itself, excluding child datasets.

```bash
zfs set refquota=200G pool0/volume1
```

Show space usage:

```bash
zfs list -o name,used,avail,refer,quota,reservation
```

### Mountpoints

Set explicit mountpoints for clarity.

```bash
zfs set mountpoint=/pool0/volume0 pool0/volume0
zfs set mountpoint=/pool0/volume1 pool0/volume1
```

Temporarily unmount and mount:

```bash
zfs unmount pool0/volume0
zfs mount pool0/volume0
```

Mount all ZFS datasets:

```bash
zfs mount -a
```

### Read-Only Datasets

Make a dataset read-only:

```bash
zfs set readonly=on pool0/volume2
```

Make it writable again:

```bash
zfs set readonly=off pool0/volume2
```

### Copies

The `copies` property stores extra copies of blocks inside the same pool. It is
not a replacement for redundancy or backups, but it can help protect very small,
important datasets.

```bash
zfs set copies=2 pool0/volume1
```

Best practice:

- Use real vdev redundancy first.
- Use `copies=2` only for selected important datasets, not huge media stores.

## Everyday Commands

### Check Pool Health

```bash
zpool status
zpool status pool0
```

Show only pools that have known problems:

```bash
zpool status -x
```

Short list:

```bash
zpool list
```

Show pool I/O:

```bash
zpool iostat -v pool0 5
```

Show dataset usage:

```bash
zfs list
```

Show snapshots:

```bash
zfs list -t snapshot
```

### Create A Dataset

```bash
zfs create pool0/volume0
```

Create with mountpoint and properties:

```bash
zfs create \
  -o mountpoint=/pool0/volume0 \
  -o compression=zstd \
  -o atime=off \
  pool0/volume0
```

### Rename A Dataset

```bash
zfs rename pool0/volume0 pool0/volume1
```

### Destroy A Dataset

Destroying a dataset deletes its data.

Before running `zfs destroy`:

- Confirm the dataset name with `zfs list`.
- Check whether snapshots or backups contain the data you expect.
- Stop services that may still be writing to the dataset.
- Prefer renaming or setting `readonly=on` first when you are unsure.

```bash
zfs destroy pool0/volume0
```

Destroy a dataset and its snapshots:

```bash
zfs destroy -r pool0/volume0
```

### Move Files Into A Dataset

Create the dataset:

```bash
zfs create -o mountpoint=/pool0/volume0 pool0/volume0
```

Copy data with preserved permissions:

```bash
rsync -aHAX --info=progress2 /source/volume0/ /pool0/volume0/
```

After verifying the copy, switch services or users to the new path.

### See What Uses Space

```bash
zfs list -o name,used,avail,refer,mountpoint
```

Show snapshots and written data:

```bash
zfs list -t filesystem,snapshot -o name,used,refer,written
```

Show pool allocation:

```bash
zpool list -o name,size,alloc,free,capacity,fragmentation,health
```

### Clear A Resolved Error

Only clear errors after understanding and fixing the cause.

```bash
zpool clear pool0
```

Clear one device:

```bash
zpool clear pool0 /dev/disk/by-id/disk0
```

## Snapshots

Snapshots are cheap, read-only points in time. They protect against accidental
deletion, bad updates, and ransomware that does not have permission to destroy
snapshots.

Create a snapshot:

```bash
zfs snapshot pool0/volume0@manual-2026-05-14
```

Create recursive snapshots:

```bash
zfs snapshot -r pool0@manual-2026-05-14
```

List snapshots:

```bash
zfs list -t snapshot
```

List snapshots for one dataset:

```bash
zfs list -t snapshot -r pool0/volume0
```

Destroy a snapshot:

```bash
zfs destroy pool0/volume0@manual-2026-05-14
```

### Snapshot Naming

Use sortable names:

```text
pool0/volume0@hourly-2026-05-14-1300
pool0/volume0@daily-2026-05-14
pool0/volume0@weekly-2026-W20
pool0/volume0@monthly-2026-05
```

Avoid vague names:

```text
pool0/volume0@new
pool0/volume0@backup
pool0/volume0@test
```

### Snapshot Holds

A hold protects a snapshot from accidental destruction. This is useful during
recovery work, ransomware investigation, backup validation, or any time a
snapshot must survive cleanup scripts.

Create a snapshot and hold it:

```bash
zfs snapshot pool0/volume0@recovery-2026-06-30
zfs hold keep pool0/volume0@recovery-2026-06-30
```

Apply a hold recursively to snapshots with the same name:

```bash
zfs snapshot -r pool0@recovery-2026-06-30
zfs hold -r keep pool0@recovery-2026-06-30
```

List holds:

```bash
zfs holds pool0/volume0@recovery-2026-06-30
zfs holds -r pool0@recovery-2026-06-30
```

Release a hold when the snapshot no longer needs protection:

```bash
zfs release keep pool0/volume0@recovery-2026-06-30
zfs release -r keep pool0@recovery-2026-06-30
```

Best practice:

- Use holds on snapshots that are part of an active recovery or legal hold.
- Use clear hold tags such as `keep`, `incident-2026-06-30`, or `restore-test`.
- Do not leave holds undocumented; they can prevent expected snapshot pruning.

### Restore One File

Snapshots are visible under `.zfs/snapshot` when enabled.

Enable snapshot directory visibility:

```bash
zfs set snapdir=visible pool0/volume0
```

Restore one file:

```bash
cp /pool0/volume0/.zfs/snapshot/daily-2026-05-14/example.txt /pool0/volume0/example.txt
```

Restore a directory:

```bash
rsync -aHAX /pool0/volume0/.zfs/snapshot/daily-2026-05-14/dir0/ /pool0/volume0/dir0/
```

### Compare Snapshots With zfs diff

`zfs diff` shows file-level changes between a snapshot and a later snapshot or
the live dataset. It is useful after accidental deletes, bad sync jobs,
ransomware, or application upgrades because it answers "what changed?" before
you decide what to restore.

Compare a snapshot to the live dataset:

```bash
zfs diff pool0/volume0@daily-2026-05-14 pool0/volume0
```

Compare two snapshots:

```bash
zfs diff pool0/volume0@daily-2026-05-14 pool0/volume0@daily-2026-05-15
```

Use file type markers and machine-friendly output:

```bash
zfs diff -FH pool0/volume0@daily-2026-05-14 pool0/volume0@daily-2026-05-15
```

Common change markers:

- `+`: path was created.
- `-`: path was removed.
- `M`: path was modified.
- `R`: path was renamed.

Best practice:

- Use `zfs diff` before rollback when you are not sure what changed.
- Redirect output to an incident note for later review.
- For huge datasets, expect the command to take time and produce a lot of output.

### Roll Back A Dataset

Rollback returns the entire dataset to the snapshot state. Newer changes are
lost.

Before running rollback:

- Use `zfs diff` or a clone to inspect what will be lost.
- Copy important newer files somewhere else.
- Confirm applications are stopped or can tolerate the rollback.
- Avoid recursive rollback unless every child dataset can move back in time.

```bash
zfs rollback pool0/volume0@daily-2026-05-14
```

Rollback and destroy newer snapshots if required:

```bash
zfs rollback -r pool0/volume0@daily-2026-05-14
```

Best practice:

- Prefer restoring individual files when possible.
- Use rollback only when you want the whole dataset back in time.

### Clone A Snapshot

Use a clone to inspect or test from an old state without rolling back.

```bash
zfs clone pool0/volume0@daily-2026-05-14 pool0/volume1
```

Destroy the clone when finished:

```bash
zfs destroy pool0/volume1
```

## Snapshot Retention

Example retention policy:

- Keep hourly snapshots for 24 hours.
- Keep daily snapshots for 14 days.
- Keep weekly snapshots for 8 weeks.
- Keep monthly snapshots for 12 months.

Use an existing snapshot tool where possible, such as `sanoid`,
`zrepl`, `syncoid`, `zfs-auto-snapshot`, or a platform-native scheduler.

Simple manual snapshot example:

```bash
zfs snapshot -r pool0@daily-2026-05-14
```

Simple manual cleanup example:

Warning: destroying snapshots can permanently remove your ability to recover
old versions or send incrementals based on those snapshots.

```bash
zfs destroy pool0/volume0@daily-2026-04-14
```

Best practice:

- Automate snapshot creation and pruning.
- Monitor snapshot space usage.
- Keep snapshots for recovery convenience, not as your only backup.

## Pool Checkpoints

A pool checkpoint is a short-term, pool-wide rewind point. It captures the
entire state of `pool0`, including datasets, snapshots, pool properties, and
vdev configuration. It is useful before risky pool-wide operations or
destructive maintenance, such as a large cleanup, `zfs destroy`, pool feature
upgrade testing on the same OpenZFS implementation, or a complicated migration
step.

Check whether the checkpoint feature is available on the pool:

```bash
zpool get feature@zpool_checkpoint pool0
```

Create a checkpoint:

```bash
zpool checkpoint pool0
```

Check whether a checkpoint exists and how much space it uses:

```bash
zpool status pool0
zpool list -o name,size,alloc,free,checkpoint,health pool0
```

Discard a checkpoint after the maintenance succeeds:

```bash
zpool checkpoint -d pool0
```

Wait until checkpoint discard finishes:

```bash
zpool checkpoint -d -w pool0
```

Rewind to a checkpoint. The pool must be exported first, then imported with the
rewind flag:

Before checkpoint rewind:

- Confirm every change after the checkpoint can be lost.
- Export the pool cleanly if possible.
- Capture `zpool status`, `zpool history`, and backup state first.
- Prefer file or dataset restore when only a small part of the pool is wrong.

```bash
zpool export pool0
zpool import --rewind-to-checkpoint pool0
```

Preview the checkpointed state read-only before committing to the rewind:

```bash
zpool export pool0
zpool import -o readonly=on --rewind-to-checkpoint pool0
```

Important limits:

- A checkpoint is not a backup. It lives inside the same pool.
- A pool can have only one active checkpoint.
- Keep checkpoints temporary. They can consume space as the live pool changes.
- Rewinding permanently loses all changes written after the checkpoint.
- Once a pool is imported with `--rewind-to-checkpoint`, that checkpoint is consumed and cannot be used again.
- While a checkpoint exists, some operations are blocked, including vdev `remove`, `attach`, `detach`, mirror `split`, and `reguid`.
- Adding a new vdev after a checkpoint is possible, but if you rewind, that vdev must be added again.
- Scrubs do not repair checkpointed data that has been freed in the current live state.
- Reservations and refreservations can become misleading while a checkpoint exists because the checkpoint may consume space they normally protect.

Best practice:

- Use snapshots for normal file and dataset recovery.
- Use checkpoints for short maintenance windows where a whole-pool rewind would be acceptable.
- Discard the checkpoint as soon as you are sure the operation succeeded.
- Do not keep a checkpoint around for routine retention.

## Backups With ZFS Send And Receive

ZFS replication is one of the strongest ZFS features. Use `zfs send` and
`zfs receive` to copy snapshots exactly to another pool or host.

For a first full receive, let `zfs receive` create the destination dataset. If
the destination already exists and has diverged, do not force it unless you are
intentionally replacing or rolling it back.

Before running `zfs receive -F`:

- Confirm the destination dataset is expendable or already backed up.
- List destination snapshots and check for local-only changes.
- Prefer receiving into a new dataset when you are not sure.
- Remember that `-F` rolls back the destination to match the incoming stream.

### Replication Safety

Replication is only a backup when the destination is usable, protected, and
tested. A successful send command by itself is not enough.

Before the first replication:

- Decide whether the destination should preserve source properties or use local mountpoints, quotas, and sharing settings.
- Receive with `-u` so datasets do not mount immediately with unexpected preserved mountpoints.
- Record the source dataset, destination dataset, first snapshot, and retention policy.
- Confirm the destination pool has enough free space for current data and retained snapshots.

During normal replication:

- Keep the source snapshot until the destination has received and verified the matching snapshot.
- Save the exact source and destination snapshot names in backup logs.
- Monitor `receive_resume_token` for interrupted receives.
- Use raw sends for encrypted datasets when the receiver should not see plaintext.
- Avoid `zfs receive -F` unless the destination rollback is intentional.

After replication:

- Inspect the destination dataset before mounting it.
- Verify mountpoints and sharing properties before exposing data to clients.
- Test file restore and full dataset restore paths.
- Keep destination snapshots protected from the same users or systems that can damage the source.
- Do not prune the only common snapshot needed for the next incremental send.

### Local Backup To Another Pool

Create a snapshot:

```bash
zfs snapshot -r pool0/volume0@backup-2026-05-14
```

Send it to `pool1`:

```bash
zfs send -R pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume0
```

The `-u` option receives the dataset without mounting it immediately. Check the
received mountpoint before mounting because recursive sends can preserve source
properties:

```bash
zfs get mountpoint pool1/volume0
zfs set mountpoint=/pool1/volume0 pool1/volume0
```

### Incremental Backup

Create the next snapshot:

```bash
zfs snapshot -r pool0/volume0@backup-2026-05-15
```

Send only the difference:

```bash
zfs send -R -I pool0/volume0@backup-2026-05-14 pool0/volume0@backup-2026-05-15 | zfs receive -u pool1/volume0
```

### Remote Backup Over SSH

Create a snapshot:

```bash
zfs snapshot -r pool0/volume0@backup-2026-05-14
```

Send to another host:

```bash
zfs send -R pool0/volume0@backup-2026-05-14 | ssh backup0.example.net zfs receive -u pool1/volume0
```

Incremental remote send:

```bash
zfs send -R -I pool0/volume0@backup-2026-05-14 pool0/volume0@backup-2026-05-15 | ssh backup0.example.net zfs receive -u pool1/volume0
```

### Receive Into A Different Dataset Name

```bash
zfs send -R pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume1
```

### Raw Encrypted Sends

For encrypted datasets, raw sends preserve encryption without exposing plaintext
to the receiving system.

```bash
zfs send -w pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume0
```

Incremental raw send:

```bash
zfs send -w -I pool0/volume0@backup-2026-05-14 pool0/volume0@backup-2026-05-15 | zfs receive -u pool1/volume0
```

### Resume Interrupted Replication

Large sends can fail because of network loss, remote reboot, disk errors, or an
interrupted terminal. Use resumable receives for large backups and restores so
you do not need to restart from zero.

Start a receive in resumable mode:

```bash
zfs send -R pool0/volume0@backup-2026-06-30 | ssh backup0.example.net zfs receive -s -u pool1/volume0
```

If the receive is interrupted, check the resume token on the receiving side:

```bash
zfs get receive_resume_token pool1/volume0
```

Resume from the sender with the token value:

```bash
zfs send -t TOKEN | ssh backup0.example.net zfs receive -s -u pool1/volume0
```

If you decide to abandon the partial receive, abort it on the receiving side:

```bash
zfs receive -A pool1/volume0
```

Best practice:

- Use `zfs receive -s` for long transfers over unreliable links.
- Save the exact source and destination snapshot names in your backup logs.
- Do not destroy the source snapshot until the receive has completed and a test restore succeeds.

### ZFS Bookmarks For Replication

A bookmark records the creation point of a snapshot and can be used as the
source side of a later incremental send. Bookmarks are useful when you want to
delete old snapshots on the source but still keep an incremental replication
anchor.

Create a snapshot and bookmark:

```bash
zfs snapshot pool0/volume0@backup-2026-06-30
zfs bookmark pool0/volume0@backup-2026-06-30 pool0/volume0#backup-2026-06-30
```

Use the bookmark as the incremental source:

```bash
zfs snapshot pool0/volume0@backup-2026-07-01
zfs send -i pool0/volume0#backup-2026-06-30 pool0/volume0@backup-2026-07-01 | zfs receive -u pool1/volume0
```

List bookmarks:

```bash
zfs list -t bookmark
```

Destroy a bookmark when it is no longer needed:

```bash
zfs destroy pool0/volume0#backup-2026-06-30
```

Best practice:

- Keep the source snapshot until you know the receiver has the matching state.
- Use bookmarks to reduce long-term source snapshot clutter, not as a replacement for real backup snapshots on the destination.
- Document which destination snapshot a bookmark corresponds to.

### Inspect Send Streams With zstreamdump

`zstreamdump` inspects a `zfs send` stream. It can show stream headers, record
counts, and metadata, and it validates checksums by default. This is useful when
you archive send streams as files or want to sanity-check a stream before
receiving it on a recovery system.

Estimate a send first:

```bash
zfs send -nP pool0/volume0@backup-2026-05-14
```

Write a stream to a file:

```bash
zfs send pool0/volume0@backup-2026-05-14 > /safe-copy/pool0-volume0-2026-05-14.zfs
```

Inspect the stream:

```bash
zstreamdump /safe-copy/pool0-volume0-2026-05-14.zfs
```

Verbose inspection:

```bash
zstreamdump -v /safe-copy/pool0-volume0-2026-05-14.zfs
```

Best practice:

- Use `zstreamdump` to detect a malformed or truncated stream file early.
- Do not treat it as a restore test. A real test receive is still required.
- Store checksums for stream files using a normal file checksum tool as well.
- Avoid `zstreamdump -d` for normal checks because it dumps record data and can produce very large output.

### Redacted Sends

Redacted sends are an advanced replication feature. They create send streams
that omit blocks not allowed by a redaction bookmark. This can be useful when a
dataset has a sanitized clone and you need to replicate the allowed state
without sending sensitive historical blocks from the original snapshot.

Conceptual workflow:

```bash
zfs snapshot pool0/volume0@base-2026-07-03
zfs clone pool0/volume0@base-2026-07-03 pool0/volume1

# Remove or replace sensitive data inside pool0/volume1, then snapshot it.
zfs snapshot pool0/volume1@sanitized-2026-07-03

zfs redact \
  pool0/volume0@base-2026-07-03 \
  pool0/volume0#redact-2026-07-03 \
  pool0/volume1@sanitized-2026-07-03

zfs send --redact pool0/volume0#redact-2026-07-03 \
  pool0/volume0@base-2026-07-03 | zfs receive -u pool1/volume0
```

Important limits:

- Redacted sends are for advanced workflows, not ordinary backups.
- The receiver gets a redacted snapshot with intentionally missing data.
- Raw sends and redacted sends cannot be combined.
- Only use this after testing on a disposable dataset with the same OpenZFS feature support.
- Keep normal full backups separately; a redacted stream is not a complete historical backup.

Best practice:

- Keep at least one backup outside the primary machine.
- Use recursive sends for datasets with children.
- Use raw sends for encrypted datasets when the receiver should not have the key.
- Use resumable receives for large transfers.
- Regularly test receiving and restoring.

## Scrubs

A scrub reads data, verifies checksums, and repairs bad copies when redundancy
exists.

Start a scrub:

```bash
zpool scrub pool0
```

Check progress:

```bash
zpool status pool0
```

Stop a scrub:

```bash
zpool scrub -s pool0
```

Recommended schedule:

- Consumer disks: scrub every 2 to 4 weeks.
- Enterprise disks: scrub monthly or according to workload.
- Backup pools that are often offline: scrub after import and before trusting a restore.

Best practice:

- Scrub during low activity windows.
- Investigate checksum, read, or write errors.
- Do not ignore recurring errors after clearing them.

## Pool Initialization

`zpool initialize` writes to unallocated regions of pool devices. It is useful
after creating a new pool or replacing disks because it forces writes across
free regions instead of waiting for future allocations to touch them for the
first time.

Initialize every eligible device in `pool0`:

```bash
zpool initialize pool0
```

Initialize one device:

```bash
zpool initialize pool0 /dev/disk/by-id/disk0
```

Check progress:

```bash
zpool status pool0
```

Suspend and resume initialization:

```bash
zpool initialize -s pool0
zpool initialize pool0
```

Cancel initialization:

```bash
zpool initialize -c pool0
```

How it differs from other maintenance:

- A scrub reads allocated data and verifies checksums.
- Initialization writes currently unallocated regions.
- TRIM tells SSDs or thin-provisioned storage that free regions are unused.
- Initialization is not a backup and does not validate application data.

Best practice:

- Run initialization after creating important pools if the extra I/O is acceptable.
- Run it during low activity windows on large HDD pools.
- Do not use it as a substitute for SMART tests, scrubs, or restore tests.

## Disk Replacement

### Identify A Failing Disk

Check status:

```bash
zpool status -v pool0
```

Look at disks:

```bash
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
```

Check SMART data:

```bash
smartctl -a /dev/disk/by-id/disk0
```

### Offline A Disk

If the disk is still present and you need to replace it:

```bash
zpool offline pool0 /dev/disk/by-id/disk0
```

### Replace A Disk

Replace old disk with new disk:

```bash
zpool replace pool0 /dev/disk/by-id/disk0 /dev/disk/by-id/disk4
```

If ZFS already sees the old disk as unavailable:

```bash
zpool status pool0
# Use the unavailable device GUID shown by zpool status.
zpool replace pool0 1234567890123456789 /dev/disk/by-id/disk4
```

Watch resilver progress:

```bash
zpool status pool0
```

After a successful replacement, clear old errors if needed:

```bash
zpool clear pool0
```

### Online A Disk

```bash
zpool online pool0 /dev/disk/by-id/disk4
```

### Detach A Disk From A Mirror

Only detach from mirrors when you understand the redundancy left behind.

```bash
zpool detach pool0 /dev/disk/by-id/disk0
```

### When To Replace A Disk

Do not replace disks blindly. First rule out cabling, power, trays, backplanes,
controller problems, and accidental device renaming. Then replace the disk when
evidence points to the device itself or when redundancy risk is increasing.

Strong replacement signals:

- SMART shows current pending sectors, offline uncorrectable sectors, or new reallocated sectors.
- `zpool status` shows repeated or increasing `READ`, `WRITE`, or `CKSUM` errors for the same device.
- The disk disappears, reappears, or changes link speed under load.
- The disk repeatedly faults or becomes `UNAVAIL`.
- Long SMART tests fail.
- Resilver or scrub repeatedly stalls on the same device.
- Cabling, power, slots, and controller paths have been checked or swapped.

Replacement rules:

- Replace one disk at a time unless the situation requires otherwise.
- Let resilver finish before replacing another disk in the same vdev.
- Do not clear errors before recording `zpool status`, events, and SMART data.
- Use stable `/dev/disk/by-id/` names or device GUIDs from `zpool status`.
- If more than one disk in the same vdev looks weak, back up or clone before stressing the pool.
- After replacement and resilver, run a scrub and check SMART again.

### Hot Spares

A hot spare is a disk assigned to a pool so it can be used as a replacement
candidate when another device fails. It can reduce time spent in a degraded
state, but it is not extra parity and it is not a backup. Platform behavior
varies, so test whether your system and event daemon actually activate spares
the way you expect.

Create a mirror pool with one spare:

```bash
zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -m /pool0 \
  pool0 \
  mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1 \
  spare /dev/disk/by-id/disk2
```

Add a spare to an existing pool:

```bash
zpool add pool0 spare /dev/disk/by-id/disk4
```

Check spare state:

```bash
zpool status pool0
```

Remove an unused spare:

```bash
zpool remove pool0 /dev/disk/by-id/disk4
```

Best practice:

- Keep spares at least as large as the devices they may replace.
- Use stable `/dev/disk/by-id/` names.
- Monitor and test spare activation instead of assuming it works.
- Replace the failed original disk permanently; do not leave the pool relying on a temporary spare forever.
- A spare does not help if too many disks in the same vdev fail before resilver completes.

## Expanding Pools

### Add Another Mirror Vdev

This is a common and clean expansion method.

Before running `zpool add`:

- Run `zpool status` and confirm the existing top-level vdev layout.
- Confirm you are adding a complete redundant vdev, not a lone disk.
- Use `zpool attach` instead when the goal is to mirror an existing disk.
- Keep a current backup. New top-level vdevs become part of pool survival.

```bash
zpool add pool0 mirror /dev/disk/by-id/disk4 /dev/disk/by-id/disk5
```

Best practice:

- Add vdevs with similar redundancy and performance.
- Do not add a single disk vdev to a redundant pool.
- Use `zpool attach`, not `zpool add`, when the goal is to mirror an existing single disk.

Bad example:

```bash
# This can make the whole pool depend on one disk.
zpool add pool0 /dev/disk/by-id/disk4
```

### Grow After Replacing All Disks

Enable autoexpand:

```bash
zpool set autoexpand=on pool0
```

After every disk in a vdev has been replaced with larger disks, expand:

```bash
zpool online -e pool0 /dev/disk/by-id/disk4
```

### Add A RAIDZ Vdev

The traditional and widely supported way to expand a RAIDZ pool is to add
another complete RAIDZ vdev.

Before adding a RAIDZ vdev:

- Confirm the new vdev has the intended parity level.
- Avoid mixing weak and strong vdevs in the same pool.
- Remember that the pool fails if any top-level vdev fails.

```bash
zpool add pool0 raidz2 \
  /dev/disk/by-id/disk6 \
  /dev/disk/by-id/disk7 \
  /dev/disk/by-id/disk8 \
  /dev/disk/by-id/disk9 \
  /dev/disk/by-id/disk10 \
  /dev/disk/by-id/disk11
```

Best practice:

- Keep new RAIDZ vdevs similar to existing vdevs.
- Avoid mixing a RAIDZ2 vdev with a single disk or weak vdev.

### Expand An Existing RAIDZ Vdev

Newer OpenZFS versions support RAIDZ expansion. This widens an existing RAIDZ
vdev by attaching another disk to that RAIDZ vdev.

Check whether the pool advertises the feature:

```bash
zpool get feature@raidz_expansion pool0
```

Find the RAIDZ vdev name:

```bash
zpool status pool0
```

Expand a six-disk `raidz2-0` vdev into a seven-disk `raidz2-0` vdev:

```bash
zpool attach pool0 raidz2-0 /dev/disk/by-id/disk12
```

Watch progress:

```bash
zpool status pool0
```

Important limits:

- RAIDZ expansion needs OpenZFS support and the `raidz_expansion` pool feature.
- If the feature is unavailable or disabled, check your operating system and OpenZFS version before upgrading pool feature flags. New feature flags can make a pool unimportable on older systems.
- The new disk must be at least as large as the smallest disk in that RAIDZ vdev.
- Expansion keeps the same parity level. A RAIDZ1 vdev stays RAIDZ1, RAIDZ2 stays RAIDZ2, and RAIDZ3 stays RAIDZ3.
- Existing blocks keep their old data-to-parity ratio. New blocks use the wider layout after expansion.
- Expansion reads and rewrites allocated data in the vdev, so it can take a long time.
- A scrub is started after expansion to verify copied blocks.

Examples of what this can and cannot do:

```text
OK:  6-wide RAIDZ2 -> 7-wide RAIDZ2
OK:  5-wide RAIDZ1 -> 6-wide RAIDZ1
NO:  5-wide RAIDZ1 -> 6-wide RAIDZ2
NO:  6-wide RAIDZ2 -> 7-wide RAIDZ3
```

To change parity level, create a new pool or new vdev with the desired RAIDZ
level and move data with `zfs send` and `zfs receive`. Adding a RAIDZ2 vdev to
a pool that already has a RAIDZ1 vdev does not make the old RAIDZ1 vdev safer;
the pool is still limited by its weakest vdev.

## Import And Export

Export a pool before moving disks to another system:

```bash
zpool export pool0
```

List importable pools:

```bash
zpool import
```

Import by name:

```bash
zpool import pool0
```

Import with an alternate root:

```bash
zpool import -R /mnt pool0
```

Import without mounting datasets:

```bash
zpool import -N pool0
```

### Import Using A Specific Device Directory

The `-d` option tells ZFS where to search for pool devices. This is a device
search path, not a mountpoint. Use `-R` when you want to change where datasets
mount.

Import using stable long names:

```bash
zpool import -d /dev/disk/by-id pool0
```

Import using shorter `/dev` names:

```bash
zpool import -d /dev pool0
```

Using `/dev` can make `zpool status` output shorter, for example `sdb` instead
of a long `/dev/disk/by-id/...` name. The tradeoff is that `/dev/sdX` names are
not stable across reboots, controller changes, or disk reordering.

Search multiple directories:

```bash
zpool import -d /dev/disk/by-id -d /dev/disk/by-path pool0
```

List importable pools from a specific directory without importing:

```bash
zpool import -d /dev
```

Import with short device names but mount everything under `/mnt`:

```bash
zpool import -d /dev -R /mnt pool0
```

Import for recovery without mounting datasets:

```bash
zpool import -d /dev -N -o readonly=on pool0
```

Best practice:

- Prefer `/dev/disk/by-id/` for normal operation because it survives device renumbering.
- Use `-d /dev` deliberately when short names are more important than stable names, such as quick lab work or temporary recovery.
- Use `-R /mnt` for recovery environments so datasets do not mount over the live system paths.

### Import By Pool ID Or Temporary Name

If multiple importable pools have the same name, import by the numeric pool ID
shown by `zpool import`. This happens after disk moves, lab tests, backup disk
rotation, or attaching old replacement disks.

List importable pools and IDs:

```bash
zpool import
```

Example output:

```text
  pool: pool0
    id: 1234567890123456789
 state: ONLINE
action: The pool can be imported using its name or numeric identifier.
```

Import by ID:

```bash
zpool import 1234567890123456789
```

Import by ID under a temporary name:

```bash
zpool import 1234567890123456789 pool2
```

Import by ID read-only under `/mnt`:

```bash
zpool import -N -o readonly=on -R /mnt 1234567890123456789 pool2
```

Best practice:

- Import by ID when names collide.
- Use a temporary name such as `pool2` for inspection or recovery.
- Keep old disks from previous pools offline unless they are intentionally part of the recovery.

### Hostid And Active Pool Import Warnings

If disks are moved without a clean `zpool export`, ZFS may report that the pool
was last accessed by another system or may still be active. This warning exists
to prevent two systems from importing and writing to the same pool at the same
time, which can destroy data.

Safe checks:

```bash
zpool import -d /dev/disk/by-id
hostid
```

If the previous system is definitely powered off or gone and the pool was not
exported cleanly, force import may be appropriate:

```bash
zpool import -d /dev/disk/by-id -f -N pool0
```

For recovery, combine force import with read-only and no-mount options when
possible:

```bash
zpool import -d /dev/disk/by-id -f -N -o readonly=on pool0
```

On Linux systems that lack a persistent `/etc/hostid`, generate one before
putting pools into regular service:

```bash
zgenhostid
hostid
```

Best practice:

- Never force import a pool that may still be imported by another live system.
- Give cloned VMs and rescue systems unique host IDs.
- Prefer clean `zpool export pool0` before moving disks.
- Use `-f` only after verifying the old host is offline or the import warning is stale.
- If the problem follows a rescue boot, also check device paths and cachefile behavior before changing pool labels or topology.

Import read-only for recovery:

```bash
zpool import -o readonly=on pool0
```

Force import only when necessary, such as after a crashed system that cannot
export the pool:

```bash
zpool import -f pool0
```

Best practice:

- Export cleanly before moving pools.
- Use read-only import when inspecting damaged systems.
- Avoid force import unless you know why it is needed.

## Pool Feature Flags And Compatibility

OpenZFS features are stored on disk. Enabling a new feature can make a pool
unimportable on older operating systems, older rescue media, appliances, or
other OpenZFS versions that do not support that feature. This is why `zpool
upgrade` should be a deliberate compatibility decision, not routine cleanup.

Show whether pools can be upgraded:

```bash
zpool upgrade
```

Show features supported by the current OpenZFS version:

```bash
zpool upgrade -v
```

Show feature states for one pool:

```bash
zpool get all pool0 | grep 'feature@'
```

Check the compatibility property:

```bash
zpool get compatibility pool0
```

List available compatibility profiles where your platform provides them:

```bash
ls /usr/share/zfs/compatibility.d
ls /etc/zfs/compatibility.d
```

Enable all supported features only after checking recovery compatibility:

Before running `zpool upgrade`:

- Confirm every system that may import the pool supports the new features.
- Test current rescue media.
- Check appliance, bootloader, and older host compatibility.
- Document the minimum OpenZFS version required after the upgrade.
- Do not do this immediately before unrelated risky maintenance.

```bash
zpool upgrade pool0
```

Upgrade policy:

- Updating OS packages or OpenZFS packages is normal maintenance.
- Running `zpool upgrade` is an on-disk compatibility decision.
- Boot pools should stay conservative unless the bootloader and rescue media are known to support the new features.
- Backup pools that may move between systems should use compatibility profiles where available.
- Feature upgrades should be planned, documented, and tested like other storage migrations.

Best practice:

- Keep boot pools and portable backup pools conservative.
- Test rescue media before enabling new feature flags.
- Document the minimum OpenZFS version needed to import each pool.
- Do not upgrade feature flags immediately before major maintenance unless you have a rollback or restore plan.
- Use compatibility profiles where available for pools that must move between systems.

## Encryption

ZFS native encryption is per dataset. Enable it when creating a dataset.

Create an encrypted dataset with a passphrase:

```bash
zfs create \
  -o encryption=on \
  -o keyformat=passphrase \
  -o mountpoint=/pool0/volume0 \
  pool0/volume0
```

Load key:

```bash
zfs load-key pool0/volume0
```

Mount encrypted dataset:

```bash
zfs mount pool0/volume0
```

Unload key:

```bash
zfs unload-key pool0/volume0
```

Check encryption:

```bash
zfs get encryption,keyformat,keystatus pool0/volume0
```

### Encrypted Dataset Recovery Checklist

An encrypted pool can be perfectly healthy while encrypted datasets remain
unrecoverable without their keys. ZFS native encryption protects data by design;
there is no backdoor if the key or passphrase is lost.

Before an incident:

- Store passphrases or key files in an offline password manager, sealed print, or other tested recovery process.
- Document which datasets are encryption roots.
- Test key loading after reboot.
- Use raw encrypted sends when the backup host should not decrypt the data.

List encryption roots and key status:

```bash
zfs get -r encryptionroot,encryption,keyformat,keylocation,keystatus pool0
```

Import the pool without mounting datasets:

```bash
zpool import -N pool0
```

Dry-run a key load to test whether the key is correct:

```bash
zfs load-key -n pool0/volume0
```

Load keys recursively, then mount:

```bash
zfs load-key -r pool0/volume0
zfs mount pool0/volume0
```

Load all available encryption roots:

```bash
zfs load-key -a
```

Use a temporary key location without changing the dataset property:

```bash
zfs load-key -L file:///root/recovery-key0 pool0/volume0
```

Change a passphrase only after the current key is loaded:

```bash
zfs change-key pool0/volume0
```

Raw encrypted backup:

```bash
zfs snapshot pool0/volume0@secure-2026-06-30
zfs send -w pool0/volume0@secure-2026-06-30 | zfs receive -u pool1/volume0
```

Best practice:

- Test that backups can be received and mounted with the expected key process.
- Do not mix raw and non-raw incremental receives for the same encrypted replication chain.
- Keep key backups separate from the encrypted pool.
- Treat lost keys as permanent data loss for that encrypted dataset.

General best practice:

- Create encryption at dataset creation time.
- Keep recovery keys or passphrases offline.
- Use raw sends for encrypted backup when the backup host should not decrypt the data.

## Zvols

A zvol is a block device backed by ZFS. Use zvols for VM disks, iSCSI targets,
or applications that need a block device.

Create a 100G zvol:

```bash
zfs create -V 100G pool0/volume0
```

Create a sparse 100G zvol:

```bash
zfs create -s -V 100G pool0/volume1
```

Set zvol block size at creation time:

```bash
zfs create -o volblocksize=16K -V 100G pool0/volume2
```

Find the device:

```bash
ls -l /dev/zvol/pool0/
```

Snapshot a zvol:

```bash
zfs snapshot pool0/volume0@before-update-2026-05-14
```

Best practice:

- Set `volblocksize` before writing data.
- Use smaller volblock sizes for databases or random I/O.
- Use larger volblock sizes for sequential workloads.
- Do not overuse sparse zvols unless you monitor free pool space carefully.
- Keep enough free space for snapshots and writes.

## Virtual Machines

For VM image files stored in a filesystem dataset:

```bash
zfs create -o mountpoint=/pool0/volume3 pool0/volume3
zfs set recordsize=64K pool0/volume3
zfs set compression=zstd pool0/volume3
zfs set atime=off pool0/volume3
```

For VM zvols:

```bash
zfs create -o volblocksize=16K -V 200G pool0/volume4
```

Best practice:

- Prefer mirrors for VM pools.
- Avoid very wide RAIDZ for heavy VM random writes.
- Keep snapshots short-lived for busy VM disks unless you need them.
- Monitor snapshot growth.
- Consider a separate dataset or zvol per VM.

## Databases

Databases often need more deliberate tuning than ordinary file storage.

Example for a PostgreSQL-like dataset:

```bash
zfs create -o mountpoint=/pool0/volume4 pool0/volume4
zfs set recordsize=8K pool0/volume4
zfs set compression=zstd pool0/volume4
zfs set atime=off pool0/volume4
```

Example for a MySQL or MariaDB InnoDB-like dataset:

```bash
zfs create -o mountpoint=/pool0/volume5 pool0/volume5
zfs set recordsize=16K pool0/volume5
zfs set compression=zstd pool0/volume5
zfs set atime=off pool0/volume5
```

InnoDB commonly uses 16K pages, so `recordsize=16K` is a practical starting
point. If `innodb_page_size` is different, match `recordsize` to that value
when possible.

Example for a database backup dataset:

```bash
zfs create -o mountpoint=/pool0/volume6 pool0/volume6
zfs set recordsize=1M pool0/volume6
zfs set compression=zstd pool0/volume6
```

Best practice:

- Match `recordsize` to the database page size when possible.
- Set `recordsize` before initializing or loading the database.
- Prefer mirrors for write-heavy databases.
- Coordinate database-consistent snapshots with the database.
- Do not assume a filesystem snapshot is application-consistent unless the application was flushed, paused, or designed for crash consistency.

## Media And Archive Storage

For large files:

```bash
zfs create -o mountpoint=/pool0/volume2 pool0/volume2
zfs set recordsize=1M pool0/volume2
zfs set compression=zstd pool0/volume2
zfs set atime=off pool0/volume2
```

For cold archives:

```bash
zfs set compression=zstd-6 pool0/volume2
```

Best practice:

- Use RAIDZ2 or RAIDZ3 for large archive pools.
- Use larger record sizes for large sequential files.
- Keep a second copy on another pool or system.

## Shares

ZFS can manage NFS and SMB sharing on some platforms, but exact behavior depends
on the operating system. Many administrators prefer to manage shares with the
native NFS or Samba configuration and let ZFS handle mountpoints.

Example dataset for sharing:

```bash
zfs create -o mountpoint=/pool0/volume0 pool0/volume0
zfs set compression=zstd pool0/volume0
zfs set atime=off pool0/volume0
```

Example NFS property where supported:

```bash
zfs set sharenfs=on pool0/volume0
```

Disable NFS sharing:

```bash
zfs set sharenfs=off pool0/volume0
```

Example SMB property where supported:

```bash
zfs set sharesmb=on pool0/volume1
```

Disable SMB sharing:

```bash
zfs set sharesmb=off pool0/volume1
```

Best practice:

- Use one dataset per share when permissions, quotas, or snapshots differ.
- Keep share configuration documented.
- Test permissions from a client machine.

## Delegation

ZFS delegation allows non-root users to perform selected ZFS operations.

Allow user `user0` to create snapshots on `pool0/volume0`:

```bash
zfs allow user0 snapshot pool0/volume0
```

Allow user `user0` to create and destroy snapshots:

```bash
zfs allow user0 snapshot,destroy pool0/volume0
```

View delegated permissions:

```bash
zfs allow pool0/volume0
```

Remove delegated permissions:

```bash
zfs unallow user0 snapshot,destroy pool0/volume0
```

Best practice:

- Delegate the minimum permissions needed.
- Be careful with `destroy`, `mount`, `send`, and `receive`.
- Do not delegate pool-level administration casually.

## ZFS For Containers

ZFS works well for containers when each important container, image store, or
application state directory gets its own dataset. This gives clean quotas,
snapshots, rollback points, send/receive backups, and optional clones for
template-based deployments.

Example dataset for container state:

```bash
zfs create -o mountpoint=/var/lib/container0 pool0/volume0
zfs set compression=zstd pool0/volume0
zfs set atime=off pool0/volume0
zfs set quota=50G pool0/volume0
```

Use a parent dataset for many containers:

```bash
zfs create -o mountpoint=/containers pool0/volume1
zfs create -o mountpoint=/containers/container0 pool0/volume1/container0
zfs create -o mountpoint=/containers/container1 pool0/volume1/container1
```

Create a template and clone it:

```bash
zfs create -o mountpoint=/containers/templates/debian0 pool0/volume1/templates/debian0
zfs snapshot pool0/volume1/templates/debian0@clean-2026-07-03
zfs clone pool0/volume1/templates/debian0@clean-2026-07-03 pool0/volume1/container2
zfs set mountpoint=/containers/container2 pool0/volume1/container2
```

Snapshot one container before upgrades:

```bash
zfs snapshot pool0/volume1/container0@before-upgrade-2026-07-03
```

Best practice:

- Use one dataset per container or per important application state directory.
- Put container runtime caches, databases, and user data in separate datasets when they need different snapshot or recordsize policies.
- Use quotas to stop one container from filling the pool.
- Do not let untrusted containers manage host ZFS datasets.
- Be conservative with delegated `mount`, `destroy`, `rollback`, and `receive` permissions.
- If using a container engine's ZFS storage driver, follow that engine's documented dataset layout and do not also manage its internal datasets by hand.

## Cache, Log, And Special Vdevs

### L2ARC

L2ARC is a read cache on fast devices. It does not replace RAM.

Add an L2ARC cache device:

```bash
zpool add pool0 cache /dev/disk/by-id/ssd-cache0
```

Best practice:

- Add RAM first when possible.
- Use L2ARC only when the working set benefits from read caching.
- Do not expect L2ARC to improve write performance.

### SLOG

SLOG is a separate intent log device for synchronous writes. It is useful only
for sync write workloads such as NFS, databases, or virtualization where sync
writes matter.

Add a mirrored SLOG:

Before adding a SLOG:

- Confirm the workload uses synchronous writes.
- Use power-loss-protected devices.
- Mirror the SLOG for important pools.
- Understand that losing an active SLOG can lose recent sync writes.

```bash
zpool add pool0 log mirror /dev/disk/by-id/ssd-log0 /dev/disk/by-id/ssd-log1
```

Best practice:

- Use power-loss-protected SSDs.
- Mirror the SLOG for important pools.
- Do not add a cheap consumer SSD as SLOG.
- SLOG does not speed up normal asynchronous writes.

### Special Vdev

A special vdev can store metadata and optionally small blocks. It can greatly
improve metadata-heavy workloads, but if it fails and is not redundant, the pool
can fail.

Add a mirrored special vdev:

Before adding a special vdev:

- Confirm it is redundant and at least as reliable as the main pool.
- Confirm backups are current.
- Understand that a failed special vdev can make the pool unavailable.
- Use conservative `special_small_blocks` settings.

```bash
zpool add pool0 special mirror /dev/disk/by-id/ssd-special0 /dev/disk/by-id/ssd-special1
```

Best practice:

- Use redundancy for special vdevs.
- Treat special vdevs as critical pool members.
- Plan before adding one. Removing it may not be practical.

### Special Small Blocks On Special Vdevs

When a pool has a special vdev, metadata is stored there by default. You can
also opt selected datasets into storing small file or zvol blocks on the special
allocation class with `special_small_blocks`.

Example for a metadata-heavy small-file dataset:

```bash
zfs set special_small_blocks=16K pool0/volume0
```

Example for a VM or database dataset where only very small blocks should move:

```bash
zfs set special_small_blocks=4K pool0/volume3
```

Check the setting:

```bash
zfs get special_small_blocks pool0/volume0
```

Monitor special vdev usage:

```bash
zpool list -v pool0
zpool iostat -v pool0 5
```

Important behavior:

- The threshold applies after compression and encryption.
- Blocks smaller than or equal to the threshold can be allocated on the special class.
- Existing blocks are not moved just because the property changes; rewrite or replicate data if you need old blocks to follow the new policy.
- If the special class fills, allocations intended for it can spill back to normal data vdevs.

Best practice:

- Mirror or otherwise redundantly protect every special vdev.
- Keep the threshold conservative, such as `4K`, `8K`, or `16K`, unless testing proves a higher value is worth the special-vdev capacity.
- Do not put large media datasets on special vdevs by accident.
- Track special vdev capacity separately from total pool capacity.

## TRIM

For SSD pools, enable autotrim if appropriate for your platform and devices.

```bash
zpool set autotrim=on pool0
```

Manual trim:

```bash
zpool trim pool0
```

Best practice:

- Enable TRIM for SSD-backed pools unless your environment has a reason not to.
- Monitor device behavior after enabling autotrim.

## Monitoring

Minimum monitoring:

```bash
zpool status
zpool list
zfs list
```

Useful health command:

```bash
zpool status -x
```

`zpool status -x` is the quickest daily check because it suppresses normal pool
details and only reports pools with known problems. Healthy output usually says
all pools are healthy. If it prints pool names, read the full status for each
pool:

```bash
zpool status -v pool0
```

### Command History For Auditing

`zpool history` records ZFS administrative commands in the pool history. It is
valuable during incidents because it can show when a destructive command,
property change, vdev addition, replacement, import, export, or feature upgrade
happened.

Show pool history:

```bash
zpool history pool0
```

Show long output with user, host, and zone information where available:

```bash
zpool history -l pool0
```

Include internal ZFS events:

```bash
zpool history -i pool0
```

Useful history terms to search for:

- `zfs destroy`
- `zpool add`
- `zpool attach`
- `zpool replace`
- `zpool import`
- `zpool upgrade`
- `zfs set mountpoint`
- `zfs receive -F`

Best practice:

- Save `zpool history -l pool0` with incident notes before changing the pool.
- Treat it as an operational audit trail, not as a complete security log.
- Review history before assuming ZFS changed something by itself.

### Reading zpool status

Start with the `state`, `status`, `action`, and `errors` lines before looking
at individual devices.

Healthy example:

```text
  pool: pool0
 state: ONLINE
  scan: scrub repaired 0B in 02:10:15 with 0 errors
config:

        NAME        STATE     READ WRITE CKSUM
        pool0       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            disk0   ONLINE       0     0     0
            disk1   ONLINE       0     0     0

errors: No known data errors
```

Degraded mirror example:

```text
  pool: pool0
 state: DEGRADED
status: One or more devices could not be used.
action: Attach the missing device and online it, or replace it.
config:

        NAME        STATE     READ WRITE CKSUM
        pool0       DEGRADED     0     0     0
          mirror-0  DEGRADED     0     0     0
            disk0   ONLINE       0     0     0
            disk1   UNAVAIL      0     0     0
```

Checksum error example:

```text
        NAME        STATE     READ WRITE CKSUM
        pool0       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            disk0   ONLINE       0     0    12
            disk1   ONLINE       0     0     0
```

Permanent data error example:

```text
errors: Permanent errors have been detected in the following files:

        pool0/volume0:<0x12345>
        /pool0/volume0/file0.txt
```

How to react:

- `ONLINE` with zero errors: keep monitoring and scrub on schedule.
- `DEGRADED`: check cabling, power, and enclosure state before replacing disks.
- `UNAVAIL`: identify the exact device by stable path, serial, or GUID.
- Growing `READ`, `WRITE`, or `CKSUM` counts: inspect SMART data and cables.
- Named permanent file errors: restore those files from backup after the pool is stable.
- Metadata or object errors without filenames: prioritize copying readable data out and restoring from backup.

### Annotated zpool status Examples

Daily healthy check with `zpool status -x`:

```text
all pools are healthy
```

Annotation:

- This is the desired daily monitoring output.
- It does not replace scheduled scrubs, SMART tests, backup checks, or capacity monitoring.
- If automation reports anything else, run `zpool status -gLPv pool0` before changing anything.

Healthy pool after a clean scrub:

```text
  pool: pool0
 state: ONLINE
  scan: scrub repaired 0B in 02:10:15 with 0 errors on Sun Jul  5 03:10:15 2026
config:

        NAME                       STATE     READ WRITE CKSUM
        pool0                      ONLINE       0     0     0
          mirror-0                 ONLINE       0     0     0
            /dev/disk/by-id/disk0  ONLINE       0     0     0
            /dev/disk/by-id/disk1  ONLINE       0     0     0

errors: No known data errors
```

Annotation:

- `state: ONLINE` means the pool has no currently failed vdevs.
- `scrub repaired 0B` and `0 errors` means this scrub found no data that needed repair.
- All per-device counters are zero, so there is no current ZFS evidence pointing to one device.

Degraded mirror with one missing side:

```text
  pool: pool0
 state: DEGRADED
status: One or more devices could not be used because the label is missing or invalid.
action: Replace the device using `zpool replace`.
config:

        NAME                       STATE     READ WRITE CKSUM
        pool0                      DEGRADED     0     0     0
          mirror-0                 DEGRADED     0     0     0
            /dev/disk/by-id/disk0  ONLINE       0     0     0
            1234567890123456789    UNAVAIL      0     0     0  was /dev/disk/by-id/disk1

errors: No known data errors
```

Annotation:

- The pool is still importable because one mirror side remains online.
- The long number is a vdev GUID; use `zpool status -gLPv pool0` and physical disk inventory before replacing anything.
- First check whether `/dev/disk/by-id/disk1` disappeared because of cabling, power, enclosure, HBA, or OS naming.
- If the disk is truly failed and redundancy remains, replace it with a known-good disk and let the resilver complete.

Checksum errors on one disk:

```text
  pool: pool0
 state: ONLINE
  scan: scrub repaired 128K in 01:40:11 with 0 errors on Sun Jul  5 03:40:11 2026
config:

        NAME                       STATE     READ WRITE CKSUM
        pool0                      ONLINE       0     0     0
          mirror-0                 ONLINE       0     0     0
            /dev/disk/by-id/disk0  ONLINE       0     0    24
            /dev/disk/by-id/disk1  ONLINE       0     0     0

errors: No known data errors
```

Annotation:

- ZFS repaired data from redundancy, so user data may still be intact.
- The checksum counter points to a device, path, cable, controller, RAM, or power problem that needs investigation.
- Do not clear counters until SMART data, cabling, and a follow-up scrub have been checked.

Permanent file errors:

```text
  pool: pool0
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.
action: Restore the file in question if possible. Otherwise restore the entire pool from backup.
config:

        NAME                       STATE     READ WRITE CKSUM
        pool0                      ONLINE       0     0     0
          mirror-0                 ONLINE       0     0     0
            /dev/disk/by-id/disk0  ONLINE       0     0     0
            /dev/disk/by-id/disk1  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        /pool0/volume0/file0.txt
```

Annotation:

- The pool can be online while one or more files are damaged.
- Restore named files from snapshots, backup pools, or replicated copies after checking hardware health.
- If errors point to metadata or unnamed objects, prioritize copying readable data out and restoring the dataset from backup.

Resilver in progress:

```text
  pool: pool0
 state: DEGRADED
  scan: resilver in progress since Sun Jul  5 10:00:00 2026
        512G scanned at 1.20G/s, 128G issued at 300M/s, 4.00T total
        128G resilvered, 3.12% done, 03:40:00 to go
config:

        NAME                         STATE     READ WRITE CKSUM
        pool0                        DEGRADED     0     0     0
          mirror-0                   DEGRADED     0     0     0
            /dev/disk/by-id/disk0    ONLINE       0     0     0
            replacing-1              DEGRADED     0     0     0
              /dev/disk/by-id/disk1  OFFLINE      0     0     0
              /dev/disk/by-id/disk2  ONLINE       0     0     0
```

Annotation:

- Do not remove more devices from the same vdev while resilvering unless the current path is clearly wrong and you have a recovery plan.
- Watch for new read, write, or checksum errors on the remaining old disk.
- After resilver completes, scrub and review SMART data before declaring the incident closed.

### Interpreting ZFS Errors

ZFS reports device and data health through pool state, vdev state, scrub
results, and per-device error counters. The three common device error counters
are `READ`, `WRITE`, and `CKSUM`.

Example status columns:

```text
NAME                       STATE     READ WRITE CKSUM
pool0                      ONLINE       0     0     0
  mirror-0                 ONLINE       0     0     0
    /dev/disk/by-id/disk0  ONLINE       0     0     0
    /dev/disk/by-id/disk1  ONLINE       0     0     0
```

What the counters mean:

- `READ`: the device failed or struggled to return requested data.
- `WRITE`: the device failed or struggled to write data.
- `CKSUM`: data was read but did not match the checksum ZFS expected.

When not to worry much:

- `zpool status -x` says all pools are healthy.
- A scrub reports `0 errors`.
- Old nonzero counters appear after a known, fixed event and do not increase.
- A pool is `ONLINE` and a transient cabling or power problem was fixed, then verified by a clean scrub.
- `zpool status` shows a scrub or resilver in progress; that is normal during maintenance, but it should finish.

When to investigate soon:

- Any `READ`, `WRITE`, or `CKSUM` counter is nonzero.
- Counters increase over time.
- A scrub repairs data.
- `zpool status -x` reports degraded or unhealthy pools.
- SMART reports pending sectors, reallocated sectors, media errors, or CRC errors.
- A device shows `DEGRADED`, `FAULTED`, `UNAVAIL`, `REMOVED`, or repeated online/offline transitions.

When to worry immediately:

- The pool state is `DEGRADED`, `FAULTED`, or `UNAVAIL`.
- A non-redundant pool or vdev has any device problem.
- A RAIDZ1 vdev has one failed disk.
- A mirror has only one remaining good side.
- `zpool status -v` lists permanent data errors or specific damaged files.
- Multiple devices in the same vdev show errors at the same time.

Recommended response:

```bash
zpool status -v pool0
zpool scrub pool0
zpool status pool0
smartctl -a /dev/disk/by-id/disk0
zpool events
```

If errors point to one disk, cable, HBA port, or enclosure slot, fix the
hardware first. After the cause is fixed and a scrub is clean, clear stale
counters:

```bash
zpool clear pool0
```

Do not clear errors just to hide them. Clearing is useful after you understand
the cause, replace or repair the bad component, and verify the pool.

Watch pool I/O:

```bash
zpool iostat -v pool0 5
```

Watch dataset space:

```bash
zfs list -o name,used,avail,refer,mountpoint
```

Show pool events:

```bash
zpool events
```

Show detailed pool history:

```bash
zpool history pool0
```

### Events And Alerting

ZFS kernel events explain what happened before a pool reached its current
state. They are useful during recovery because they can show whether a problem
was a checksum error, I/O error, slow device, missing vdev, import failure, or
configuration change.

Show recent events:

```bash
zpool events
```

Show full event payloads:

```bash
zpool events -v
```

Follow events while replacing hardware or testing a pool:

```bash
zpool events -f
```

Clear old events after you have documented them:

```bash
zpool events -c
```

Best practice:

- Enable your platform's ZFS event daemon or alerting system, often called `zed` on OpenZFS/Linux systems.
- Send alerts to email, chat, monitoring, or another place that someone actually reads.
- Alert on pool state changes, checksum errors, I/O errors, slow I/O, failed imports, and vdev removals.
- Keep event output with incident notes before clearing it.

Best practice:

- Configure email or alerting for pool errors.
- Monitor SMART data separately.
- Monitor free space and snapshot growth.
- Alert before the pool reaches 80% usage.
- Treat checksum errors as serious.

## SMART Checks

ZFS checks data integrity, but disk firmware still reports useful health
information.

Show SMART details:

```bash
smartctl -a /dev/disk/by-id/disk0
```

Run a short test:

```bash
smartctl -t short /dev/disk/by-id/disk0
```

Run a long test:

```bash
smartctl -t long /dev/disk/by-id/disk0
```

Best practice:

- Schedule SMART tests.
- Track reallocated sectors, pending sectors, CRC errors, and media errors.
- Replace suspect disks before they fail completely.

## Automation Examples

Automation should make safe maintenance happen reliably. It should not hide errors, destroy snapshots blindly, or make recovery harder. Test every command manually on a non-critical dataset before scheduling it.

General automation rules:

- Prefer the platform's built-in scheduler, appliance UI, or packaged ZFS units when they exist.
- Log output somewhere outside the pool being protected when possible.
- Alert on failure instead of silently discarding command output.
- Keep snapshot pruning separate from snapshot creation until the policy is proven.
- Test a restore path after enabling snapshot or replication automation.

### ZED Event Alerts

On OpenZFS systems that ship `zed`, enable the event daemon and configure its notification target according to your distribution. Service names and configuration paths vary.

Linux systemd example:

```bash
systemctl enable --now zfs-zed.service
systemctl status zfs-zed.service
```

Common configuration path on many OpenZFS/Linux systems:

```bash
grep -E 'ZED_EMAIL|ZED_NOTIFY|ZED_SYSLOG' /etc/zfs/zed.d/zed.rc
```

Test by confirming events are visible:

```bash
zpool events -v
```

Best practice:

- Send ZFS events to a mailbox, monitoring system, or chat bridge that is actually watched.
- Alert on pool state changes, vdev removals, checksum errors, I/O errors, slow I/O, failed imports, and scrub failures.
- Do not rely on `zpool status` checks alone for production systems.

### SMART Alerts

SMART monitoring is separate from ZFS. ZFS may detect bad data, while SMART may show a disk becoming unreliable before ZFS sees a checksum failure.

Example `smartd` entry for one disk:

```text
/dev/disk/by-id/disk0 -a -o on -S on -s (S/../.././02|L/../../7/03) -m admin@example.invalid
```

Useful manual checks:

```bash
smartctl -a /dev/disk/by-id/disk0
smartctl -t short /dev/disk/by-id/disk0
smartctl -t long /dev/disk/by-id/disk0
```

Best practice:

- Monitor every real disk, not just the pool.
- Track SATA CRC errors separately from media errors because CRC errors often point to cables, backplanes, HBAs, or power.
- Treat repeated SMART failures and repeated ZFS device errors as replacement evidence.

### Scrub Timers

Scrubs verify checksums and repair bad copies when redundancy is available. Schedule them when the system can tolerate extra I/O.

If your distribution already ships scrub timers, inspect them first:

```bash
systemctl list-timers 'zfs*scrub*'
systemctl list-unit-files 'zfs*scrub*'
```

Example custom systemd service:

```ini
[Unit]
Description=Scrub ZFS pool pool0

[Service]
Type=oneshot
ExecStart=/usr/sbin/zpool scrub pool0
```

Example custom systemd timer:

```ini
[Unit]
Description=Monthly scrub for ZFS pool pool0

[Timer]
OnCalendar=monthly
Persistent=true

[Install]
WantedBy=timers.target
```

Enable and inspect a custom timer after placing the files in the correct systemd unit directory for your distribution:

```bash
systemctl daemon-reload
systemctl enable --now zfs-scrub-pool0.timer
systemctl list-timers zfs-scrub-pool0.timer
```

Best practice:

- Do not schedule all pools to scrub at the same minute on a small host.
- Pair scrub automation with alerting that reports nonzero repair counts and failed scrubs.
- Check `zpool status pool0` after the first scheduled scrub.

### Simple Snapshot Cron

For a very small system, cron can create snapshots. This is simple, but pruning, monitoring, and replication safety are easier with purpose-built tools.

Example hourly snapshot:

```cron
15 * * * * /usr/sbin/zfs snapshot -r pool0/volume0@hourly-$(date +\%Y-\%m-\%d-\%H)
```

Example daily snapshot:

```cron
30 2 * * * /usr/sbin/zfs snapshot -r pool0/volume0@daily-$(date +\%Y-\%m-\%d)
```

Best practice:

- Use a consistent snapshot naming scheme.
- Add pruning only after verifying that creation works and restore tests pass.
- Do not let snapshots grow without capacity alerts.

### Sanoid And Syncoid

Sanoid is useful when you want policy-driven snapshots and pruning. Syncoid is useful for asynchronous ZFS replication over local paths or SSH.

Minimal Sanoid-style policy example:

```ini
[pool0/volume0]
        use_template = production
        recursive = yes

[template_production]
        hourly = 24
        daily = 14
        monthly = 3
        autosnap = yes
        autoprune = yes
```

Run Sanoid from cron or the packaged timer according to its installation documentation:

```bash
sanoid --cron
```

Example Syncoid push replication:

```bash
syncoid --recursive pool0/volume0 backuphost:pool1/volume0
```

Best practice:

- Use a restricted SSH key for replication targets.
- Keep at least one backup destination unavailable to normal clients.
- Test receiving into a temporary dataset such as `pool1/volume2` before trusting production replication.

### zrepl

`zrepl` is useful for scheduled snapshotting, pruning, and replication with a daemon model. It is more complex than simple cron, but it can be easier to monitor consistently across hosts.

Minimal illustrative job skeleton:

```yaml
jobs:
- name: push_pool0_volume0
  type: push
  connect:
    type: ssh+stdinserver
    host: backuphost
    user: zrepl
  filesystems:
    "pool0/volume0<": true
  snapshotting:
    type: periodic
    prefix: zrepl_
    interval: 1h
  pruning:
    keep_sender:
    - type: not_replicated
    - type: last_n
      count: 24
    keep_receiver:
    - type: last_n
      count: 48
```

Best practice:

- Validate configuration with the tool's check command before enabling the service.
- Confirm replication creates snapshots on the receiver and that old snapshots prune as expected.
- Do a restore drill from the receiver before calling the automation complete.

## Performance Basics

### Keep Free Space

Performance usually drops as a pool gets full.

Best practice:

- Keep pools below 80% used.
- Start planning expansion before 80%.
- Avoid going above 90% except temporarily.

Check capacity:

```bash
zpool list -o name,capacity,free,fragmentation,health
```

### Match Layout To Workload

Good defaults:

- General file server: mirrors or RAIDZ2.
- VM storage: mirrors.
- Database storage: mirrors.
- Media archive: RAIDZ2 or RAIDZ3.
- Backup target: RAIDZ2, RAIDZ3, or mirrors depending on restore needs.

### Compression

Usually keep compression enabled:

```bash
zfs set compression=zstd pool0
```

Compression can increase speed by reducing disk I/O.

### Sync Writes

Check sync behavior:

```bash
zfs get sync pool0/volume0
```

Default is usually correct:

```bash
zfs set sync=standard pool0/volume0
```

Dangerous setting:

```bash
# Can lose acknowledged synchronous writes during power loss or crash.
zfs set sync=disabled pool0/volume0
```

Best practice:

- Keep `sync=standard` unless you understand the application and risk.
- Use a proper SLOG for important sync write workloads.

### Deduplication

Do not enable deduplication casually.

```bash
zfs get dedup pool0/volume0
```

ZFS deduplication uses a deduplication table, often called the DDT, to map
block checksums to already-stored blocks. The DDT is on disk, but performance
depends heavily on keeping hot DDT entries in RAM through the ARC. When the DDT
does not fit in RAM, writes that should be simple sequential work can turn into
many random DDT lookups from disk. The pool is not automatically corrupt, but it
can feel broken: writes become very slow, deletes and snapshot destruction can
take much longer, scrub or resilver work competes with DDT I/O, and imports or
administrative commands may become painfully slow on small-memory systems.

Check pool-level dedup statistics:

```bash
zpool status -D pool0
zpool status -DD pool0
```

Use `-DD` when available to see how much of the DDT is resident in ARC. If the
resident portion is low and the workload is writing many small or unique blocks,
expect random I/O pressure and high latency.

Estimate dedup usefulness before enabling it:

```bash
zdb -S pool0
```

Run dedup simulation only during a quiet period or on a test system. It scans
the pool and builds simulated DDT statistics, so it can be expensive on large
pools.

If dedup was enabled and the DDT is now too large:

```bash
zfs set dedup=off pool0/volume0
```

Turning dedup off stops new writes from adding more deduplicated blocks, but it
does not immediately remove existing DDT entries. Existing deduplicated blocks
remain referenced until they are deleted, snapshots that reference them are
destroyed, or the data is rewritten without dedup. The safest escape path is
usually to create a new non-dedup dataset, copy or replicate data into it, verify
the result, then retire the old dataset after snapshots no longer hold the old
blocks.

Best practice:

- Leave dedup off for most systems.
- Dedup needs large amounts of RAM, fast metadata I/O, and careful planning.
- Do not enable dedup unless the measured dedup ratio is worth the performance and recovery cost.
- If a platform supports a dedicated dedup allocation class, mirror it and treat it as critical pool metadata, not as a disposable cache.
- Compression is usually the better choice.

### Performance Troubleshooting Decision Tree

Performance problems are usually caused by layout, full pools, sync writes, weak hardware, slow devices, metadata pressure, or an application pattern that does not match the dataset properties. Do not start with random tunables.

First capture a baseline:

```bash
zpool status -v pool0
zpool list -o name,size,alloc,free,capacity,fragmentation,health
zpool iostat -v pool0 5
zfs list -o name,used,avail,refer,compressratio,mountpoint -r pool0
zfs get -r compression,recordsize,volblocksize,sync,logbias,primarycache,secondarycache,dedup pool0
```

Decision tree:

1. If pool capacity is above 80%, free space, destroy unneeded snapshots, add capacity, or move data before tuning anything else.
2. If capacity is above 90%, treat it as an urgent operational problem because allocation, fragmentation, snapshots, and recovery all get worse.
3. If one vdev is much busier than others in `zpool iostat -v`, check whether the pool was expanded unevenly, whether one vdev is slower, or whether a disk is failing.
4. If latency-sensitive VMs or databases are on RAIDZ, consider moving them to mirror vdevs rather than trying to tune around random I/O limits.
5. If writes are slow only for applications that use `fsync`, check `sync`, SLOG health, power-loss protection, and application transaction settings.
6. If reads are slow after a cold boot but improve later, check ARC size, working set size, and whether the workload is cache-friendly.
7. If metadata-heavy workloads are slow, check snapshot count, directory count, small files, special vdev design, and whether a mirrored special vdev would help a new pool design.
8. If dedup is enabled, check `zpool status -DD pool0` before looking elsewhere.
9. If compression is off, test `compression=zstd` on a representative dataset because less physical I/O often improves throughput.
10. If the pool is healthy but the application is slow, measure inside the application before changing ZFS properties.

Symptom map:

| Symptom | Likely Areas | First Checks | Common Fixes |
| --- | --- | --- | --- |
| Slow writes on every workload | Full pool, slow vdev, failing disk, sync write load | `zpool list`, `zpool iostat -v`, SMART, `zfs get sync` | Free space, fix hardware, move sync workload, add proper SLOG only when needed |
| Slow random VM I/O | RAIDZ layout, volblocksize mismatch, sync writes, host memory pressure | `zfs get volblocksize,sync`, guest workload, `zpool iostat -v` | Use mirror vdevs, create new zvols with correct volblocksize, tune guest cache safely |
| Slow database workload | Recordsize mismatch, sync writes, small random I/O, dataset sharing with noisy workloads | `zfs get recordsize,sync,logbias`, DB metrics, pool latency | Use separate dataset, set recordsize before loading data, keep `sync=standard` for durable data |
| Slow directory listing or deletes | Too many snapshots, many small files, metadata pressure, dedup | `zfs list -t snapshot`, `zpool status -DD`, application path count | Reduce snapshot count, split datasets, avoid dedup, consider special vdev in a planned design |
| Scrub or resilver very slow | Weak disk, busy pool, wide RAIDZ, SMR disks, USB enclosure, controller issues | `zpool status`, `zpool iostat -v`, SMART, kernel logs | Replace weak hardware, reduce load, avoid poor enclosures, prefer mirrors for faster rebuilds |
| Import or mount slow | Many datasets, many snapshots, dedup pressure, encryption key prompts, failing disks | `zfs list`, `zfs mount`, `zpool status -DD`, SMART | Reduce dataset/snapshot count, fix hardware, avoid dedup, document key loading |
| Replication slow | Network, SSH cipher, slow source reads, slow destination writes, snapshot count | `zpool iostat -v` on both ends, network test, send stream size | Use resumable sends, test local send speed, tune network path, avoid receiving into overloaded pools |

Useful measurements:

```bash
zpool iostat -rw pool0 5
zpool iostat -lq pool0 5
zpool iostat -v pool0 5
arcstat 5
iostat -x 5
```

Not every platform has every helper command. If `arcstat` or `iostat -x` is unavailable, use the closest platform equivalent.

Workload-specific checks:

- For databases, confirm `recordsize` before data is loaded and use a separate dataset such as `pool0/volume0`.
- For zvols, confirm `volblocksize` before creating the virtual disk and do not expect changing it later to rewrite existing blocks.
- For NFS, SMB, iSCSI, and VM hosts, separate network, guest, and storage measurements before blaming ZFS.
- For all workloads, compare performance during a scrub/resilver and while idle.

Safe tuning order:

1. Fix hardware errors, pool health, and capacity first.
2. Move workloads to the right layout before tuning properties.
3. Set workload properties on a new dataset or zvol, copy data in, and test.
4. Add support vdevs only after proving the bottleneck and understanding failure consequences.
5. Keep notes of before/after commands and measurements.

## Security And Permissions

Set ownership after creating a dataset:

```bash
chown -R user0:group0 /pool0/volume0
```

Set basic permissions:

```bash
chmod 750 /pool0/volume0
```

Use ACLs when needed:

```bash
zfs set acltype=posixacl pool0/volume0
zfs set xattr=sa pool0/volume0
```

Best practice:

- Keep one dataset per permission boundary.
- Use encryption for data at rest when needed.
- Keep backup permissions as strict as primary permissions.
- Limit who can destroy snapshots.

## Boot Pools

Boot environments vary by operating system. The safest general practices are:

- Keep boot pool layouts simple.
- Use mirrors for boot disks when uptime matters.
- Do not use exotic feature flags if the bootloader cannot read them.
- Keep a tested rescue USB or recovery environment.
- Snapshot boot environments before major upgrades where supported.
- Confirm the system can boot after disk replacement.

Check pool features:

```bash
zpool get all pool0 | less
```

Upgrade pool features only after checking compatibility:

```bash
zpool upgrade pool0
```

## Disaster Recovery

### Recovery Decision Tree

Use this as the first pass during an incident. The goal is to choose the safest
path before running commands that change pool state.

| Situation | First Safe Check | Preferred Recovery Path | Avoid |
| --- | --- | --- | --- |
| Deleted one file | `zfs list -t snapshot -r pool0/volume0` | Copy from `.zfs/snapshot` or restore from backup | Rolling back the whole dataset unnecessarily |
| Bad package or app update | `zfs list -t snapshot -r pool0` | Restore files, clone a snapshot, or rollback one dataset | Pool-wide rewind unless every later change can be lost |
| Ransomware or mass deletion | `zpool export pool0` if safe, or stop clients | Import read-only, hold snapshots, restore into new datasets | Letting clients keep writing to the pool |
| Pool is `DEGRADED` | `zpool status -gLPv pool0` | Fix cabling/power, replace failed disk, resilver, scrub | Replacing multiple disks at once without reason |
| Pool will not import | `zpool import -d /dev/disk/by-id` | Try read-only no-mount import, then documented recovery flags | Random `-f`, `-F`, `-X`, or label operations |
| Permanent file errors | `zpool status -v pool0` | Restore named files from backup or healthy send stream | Clearing errors before copying evidence |
| Interrupted backup or restore | `zfs get receive_resume_token pool1/volume0` | Resume with `zfs send -t TOKEN` | Restarting from zero when a valid token exists |
| Missing SLOG | `zpool import -d /dev/disk/by-id` | Prefer finding the SLOG; use `-m` only if loss is acceptable | Assuming recent sync writes survived |
| Missing special or dedup vdev | `zpool status -gLPv pool0` | Restore missing vdev or restore from backup | Treating it like a disposable cache device |
| Encrypted dataset unavailable | `zfs get -r keystatus,keylocation pool0` | Load correct key, mount, then restore if needed | Destroying or recreating encryption roots |
| Accidental `zpool destroy` | `zpool import -D` | Import destroyed pool read-only and copy data out | Reusing or relabeling the disks first |

Incident rules:

- Stop writes first when corruption, ransomware, or failing hardware is suspected.
- Prefer read-only and no-mount imports during investigation.
- Clone failing disks before aggressive recovery attempts.
- Copy critical data out before starting heavy repairs if more hardware looks weak.
- Do not clear errors until you have captured status, events, and SMART data.

### Symptom-Based Recovery Runbook

Start from the symptom you see. Keep the first pass read-only unless the pool is already healthy and the action is clearly routine maintenance.

#### Pool Is Missing From zpool list

1. Check whether the pool is only exported or not imported:

```bash
zpool import
zpool import -d /dev/disk/by-id
```

2. If the pool appears, try read-only and no-mount import first:

```bash
zpool import -d /dev/disk/by-id -N -o readonly=on pool0
```

3. If the pool does not appear, check disk inventory and kernel logs:

```bash
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
dmesg --ctime | grep -Ei 'zfs|i/o|error|reset|ata|scsi|nvme'
```

4. If disks are missing at the OS level, solve cabling, power, HBA, enclosure, or driver visibility before trying ZFS recovery flags.

#### Pool Imports Read-Only

1. Treat read-only import as a chance to copy data out, not as proof that repair is complete.

```bash
zpool status -gLPv pool0
zfs list -r pool0
zfs list -t snapshot -r pool0
```

2. Copy the most important data to another pool or host:

```bash
rsync -aHAX --info=progress2 /pool0/volume0/ /safe-copy/volume0/
```

3. After data is safe, decide whether to repair hardware, restore from backup, or attempt documented import recovery.

#### One Device Is FAULTED, DEGRADED, UNAVAIL, Or REMOVED

1. Capture exact device identity:

```bash
zpool status -gLPv pool0
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0
```

2. Check whether the fault follows the disk, cable, slot, HBA port, enclosure bay, or power connector.
3. If redundancy remains and the disk is truly failed, replace one disk at a time:

```bash
zpool replace pool0 1234567890123456789 /dev/disk/by-id/disk4
zpool status pool0
```

4. After resilver, scrub and keep the failed disk untouched until the pool is confirmed healthy.

#### READ, WRITE, Or CKSUM Counters Increase

1. Save status and events before clearing anything:

```bash
zpool status -v pool0
zpool events -v
```

2. Check SMART, kernel logs, cabling, power, HBA, and enclosure health.
3. If counters increase during a scrub, treat the cause as active and fix hardware before trusting the pool.
4. Clear counters only after the cause is fixed and a scrub is clean:

```bash
zpool clear pool0
```

#### zpool status Lists Permanent File Errors

1. Record the full verbose status:

```bash
zpool status -v pool0
```

2. Restore named files from snapshots, a backup pool, or a replicated copy.
3. If ZFS lists metadata objects or unnamed errors, copy readable data out and restore the dataset from a clean backup.
4. Scrub again after restore and only then decide whether `zpool clear pool0` is appropriate.

#### Pool Will Not Import

1. Do not run random import flags; capture the import view first:

```bash
zpool import
zpool import -d /dev/disk/by-id
```

2. Try read-only and no-mount import:

```bash
zpool import -d /dev/disk/by-id -N -o readonly=on pool0
```

3. If the pool complains that it is active on another host, verify that the old host is powered off or exported before using `-f`.
4. If import suggests rewind, dry-run first:

```bash
zpool import -d /dev/disk/by-id -F -n pool0
```

5. Use `-X` only when backup restore is worse than accepting last-resort rewind risk.

#### Encryption Key Is Missing Or Dataset Will Not Mount

1. Check which datasets need keys:

```bash
zfs get -r encryptionroot,keystatus,keylocation,mounted pool0
```

2. Load the key for the encryption root, not random children:

```bash
zfs load-key pool0/volume0
zfs mount pool0/volume0
```

3. If the key is lost, ZFS cannot recover encrypted data from checksums, redundancy, snapshots, or pool metadata.
4. Restore from an unencrypted backup or from a backup whose key is still available.

#### Accidental Snapshot Or Dataset Destroy

1. Stop writes to affected datasets if recent snapshots or replication targets may still preserve data.
2. Check other snapshots, clones, replication targets, and backup pools:

```bash
zfs list -t snapshot -r pool0
zfs list -r pool1
```

3. If a replicated backup exists, receive into a new dataset for inspection instead of overwriting the original:

```bash
zfs send -R pool1/volume0@backup-2026-05-14 | zfs receive -u pool0/volume2
```

4. If the only copy was destroyed and no snapshot or backup exists, recovery from that pool is usually not realistic.

#### Interrupted Receive Or Replication

1. Look for a resumable receive token:

```bash
zfs get receive_resume_token pool1/volume0
```

2. Resume if a valid token exists:

```bash
zfs send -t TOKEN | zfs receive -s -u pool1/volume0
```

3. If the destination must be discarded, confirm that losing the partially received dataset is acceptable before using destructive receive options.

#### Special, Dedup, Cache, Or Log Device Missing

1. Identify which class is missing:

```bash
zpool status -gLPv pool0
```

2. A missing cache device is usually removable after the pool is stable.
3. A missing special or dedup vdev is critical pool metadata and may make the pool unavailable.
4. A missing separate log can require `zpool import -m`, but that can lose recent acknowledged synchronous writes.
5. Prefer finding or cloning the missing device before accepting data loss.

### Minimum Safe Recovery Bundle

Capture this bundle before changing pool state during an incident. It gives you
enough evidence to reason about the failure, ask for help, and avoid losing
device identity.

Pool health and device paths:

```bash
zpool status -gLPv pool0
zpool list -v pool0
```

Import view, useful when the pool is exported or will not import:

```bash
zpool import -d /dev/disk/by-id
```

Events and command history:

```bash
zpool events -v
zpool history -l pool0
```

Dataset and snapshot inventory:

```bash
zfs list -r pool0
zfs list -t snapshot -r pool0
```

Disk identity and SMART data:

```bash
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0
```

If hardware may be failing, save the output somewhere outside the suspect pool.
Do not start repeated imports, scrubs, resilvers, or rewind attempts until you
have considered cloning the weakest disks.

### Incident Evidence Form

Use this form for failed pools, permanent data errors, ransomware, accidental deletion, failed imports, or any case where you may ask another administrator for help. Store the completed form outside the affected pool.

```text
Incident title:
Date and time first noticed:
Timezone:
Host name:
Operating system and version:
OpenZFS version:
Pool name:
Pool GUID:
Affected datasets:
Business impact:
Last known good backup:
Last known good scrub:
Recent maintenance:
Recent power loss or crash:
Recent disk, HBA, cable, enclosure, kernel, package, or firmware changes:
Current pool state:
Exact error message:
Commands already run:
Commands not yet run:
Disk serial numbers and slots:
Suspect disks:
SMART failures:
Kernel log errors:
Encryption keys available:
Snapshots available:
Replication target available:
Pool checkpoint present:
Data already copied out:
Next planned action:
Risk accepted by:
```

Attach these command outputs when available:

```bash
zpool status -gLPv pool0
zpool list -v pool0
zpool events -v
zpool history -l pool0
zfs list -r pool0
zfs list -t snapshot -r pool0
zfs get -r encryptionroot,keystatus,keylocation,mounted pool0
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0
```

Best practice:

- Record exact commands and outputs before trying recovery flags.
- Record disk serial numbers, not only operating system device names.
- Add photos of drive bays or labels when physical slot mapping matters.
- Keep a copy of the evidence form with the restored system documentation after the incident.

### Accidental File Deletion

1. Check snapshots:

```bash
zfs list -t snapshot -r pool0/volume0
```

2. Restore from `.zfs/snapshot`:

```bash
cp /pool0/volume0/.zfs/snapshot/daily-2026-05-14/file0.txt /pool0/volume0/file0.txt
```

### Bad System Update

Snapshot before the update:

```bash
zfs snapshot -r pool0@before-update-2026-05-14
```

Rollback one dataset if needed:

```bash
zfs rollback pool0/volume0@before-update-2026-05-14
```

If the risky change affects the whole pool rather than one dataset, create a
pool checkpoint before starting. Rewinding to it loses all later changes, so use
it only when a whole-pool undo is acceptable:

```bash
zpool checkpoint pool0
```

### Pool Will Not Import

List importable pools:

```bash
zpool import
```

Try read-only import:

```bash
zpool import -o readonly=on pool0
```

Try alternate root:

```bash
zpool import -R /mnt pool0
```

Use force only when you understand why:

```bash
zpool import -f pool0
```

### Failed Pool Recovery Triage

When a pool is failed, faulted, or non-importable, the first goal is to avoid
making the situation worse. Do not destroy and recreate the pool, do not clear
labels, do not run filesystem repair tools such as `fsck`, and do not repeatedly
try random import flags. If the data is important and the disks may be failing,
stop and make sector-level clones of the suspect disks first, for example with a
recovery tool such as `ddrescue`, before further import attempts.

Capture the current state before changing anything:

```bash
zpool import
zpool import -d /dev/disk/by-id
zpool import -d /dev
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
dmesg --ctime | grep -Ei 'zfs|i/o|error|reset|ata|scsi|nvme'
```

If the pool imports, collect detailed status:

```bash
zpool status -gLPv pool0
zpool events -v
zpool history pool0
```

Useful options while investigating:

- `-g` shows vdev GUIDs, useful when device names are missing or unstable.
- `-L` resolves symlinks to the current real device path.
- `-P` prints full paths instead of shortened names.
- `-v` shows known permanent data errors and affected files when ZFS can name them.

Start with the least invasive import:

```bash
zpool import -d /dev/disk/by-id -N -o readonly=on pool0
```

Use an alternate root in a rescue environment so datasets do not mount over the
live system:

```bash
zpool import -d /dev/disk/by-id -N -o readonly=on -R /mnt pool0
```

Use `-f` only if the pool appears active because it was not exported cleanly and
you are sure no other system is using it:

```bash
zpool import -d /dev/disk/by-id -f -N -o readonly=on pool0
```

If a pool imports read-only, copy the most important data out before repair
attempts:

```bash
rsync -aHAX --info=progress2 /mnt/pool0/volume0/ /safe-copy/volume0/
```

If the pool is `DEGRADED` but importable, prefer recovery over experimentation:

- Verify all cables, HBAs, enclosures, and power before replacing disks.
- If a missing disk reappears, try `zpool online pool0 /dev/disk/by-id/disk0`.
- If a disk is truly failed and redundancy remains, replace it with `zpool replace`.
- Copy critical data before starting heavy operations if more disks look weak.
- After replacement or repair, let resilver finish and then scrub.

Example replacement:

```bash
zpool status -gLPv pool0
zpool replace pool0 1234567890123456789 /dev/disk/by-id/disk4
zpool status pool0
```

If `zpool status -v` lists permanent errors in files, restore those files from
backup after the pool is stable. If it lists metadata objects or does not name a
file, assume affected data may not be recoverable from that pool and prioritize
backup restore.

Understand which missing device class matters:

- A missing normal top-level vdev usually means the pool cannot be recovered without that vdev or a backup.
- A missing mirror side or RAIDZ member may be survivable if enough replicas remain.
- A missing special or dedup vdev is critical and can make the whole pool unavailable.
- A missing cache device should not lose pool data; remove or replace it after the pool is stable.
- A missing separate log device may require `zpool import -m`, but that can discard recent synchronous writes.

### Clone Failing Disks Before Recovery Attempts

If the pool failure may involve physically failing disks, make sector-level
clones before running heavy recovery operations. Scrubs, resilvers, repeated
imports, and full-file copies can put enough load on a marginal disk to finish
it off.

When to clone first:

- SMART shows pending sectors, reallocated sectors, media errors, or repeated resets.
- `dmesg` shows I/O errors, link resets, timeouts, or NVMe errors.
- The disk clicks, drops offline, or disappears under load.
- Multiple disks in the same vdev are suspect.
- The pool contains data that has no tested backup.

Capture device identity:

```bash
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0
zpool import -d /dev/disk/by-id
```

Example `ddrescue` workflow:

```bash
ddrescue -f -n /dev/disk/by-id/disk0 /safe-copy/disk0.img /safe-copy/disk0.map
ddrescue -f -r3 /dev/disk/by-id/disk0 /safe-copy/disk0.img /safe-copy/disk0.map
```

Use the map file so the recovery can resume. Work on cloned images or cloned
replacement disks when possible, and keep the original disks unchanged until the
recovery is complete.

If you clone to replacement disks, attach or import using the replacement
devices, not the failing originals:

```bash
zpool import -d /dev/disk/by-id -N -o readonly=on pool0
```

Best practice:

- Clone the weakest disks first.
- Do not run `zpool clear` or label operations before imaging suspect disks.
- Do not write recovered data back onto the same failing pool.
- Keep notes mapping old serial numbers to cloned images or replacement disks.

### Physical Pool Inspection With zdb

`zdb` is the ZFS debugger. It can read vdev labels, pool
configuration, uberblocks, and some dataset or object metadata directly from
devices. That makes it useful when a pool will not import cleanly, when disk
names changed after moving hardware, or when you need to prove which physical
devices belong to which pool. It is not `fsck`, it is not a routine repair
command, and much of its output assumes ZFS internals.

Use `zdb` for inspection, not as the first recovery action. If disks may be
failing, clone them first and run `zdb` against the clones or images. If the
pool imports read-only, copy or replicate the data out before spending time on
deeper metadata analysis.

Useful read-only checks:

```bash
zdb -l /dev/disk/by-id/disk0
zdb -lu /dev/disk/by-id/disk0
zdb -lll /dev/disk/by-id/disk0
```

What these show:

- `zdb -l` reads ZFS labels from one device or partition.
- `zdb -lu` also shows uberblocks, including transaction group history.
- `zdb -lll` shows every label copy, including stale or duplicate configurations.

Look for:

- The expected pool name, pool GUID, vdev GUID, and top-level vdev GUID.
- Whether all mirror or RAIDZ members agree about the same pool layout.
- Whether labels mention old device paths, old hostnames, or old pool names.
- Whether a disk has no valid ZFS labels, labels from another pool, or labels from an old destroyed pool.
- Whether only very old uberblocks remain, which can explain why rewind would lose recent writes.

Inspect an exported or non-imported pool using a specific device directory:

```bash
zdb -e -p /dev/disk/by-id -C pool0
zdb -e -p /dev/disk/by-id -d pool0
```

This is useful after booting rescue media, moving disks to another system, or
cloning disks to image files and loop devices. The `-e` option tells `zdb` to
operate on an exported pool instead of relying on the normal cache file. The
`-p` option limits the search path, similar in spirit to importing with
`zpool import -d /dev/disk/by-id`.

If a pool checkpoint exists, `zdb` can inspect the checkpointed state without
rolling the pool back:

```bash
zdb -e -p /dev/disk/by-id -k -C pool0
```

This helps compare the current on-disk configuration with the checkpointed
configuration before deciding whether `zpool import --rewind-to-checkpoint`
is appropriate.

When to worry:

- `zdb -l` shows I/O errors while reading labels from an original disk. Stop and image the disk before continuing.
- Different members of the same mirror or RAIDZ vdev report different pool GUIDs or incompatible top-level vdev GUIDs.
- A disk that should be part of `pool0` reports labels from `pool1`.
- Labels are present, but the expected vdev is missing enough members that redundancy cannot reconstruct the data.
- `zdb -lu` only shows old transaction groups and `zpool import -F -n` reports a large rewind.

When not to worry immediately:

- Old path names in labels are common after moving disks between systems.
- Old hostnames are common after migration or rescue booting.
- One bad label copy is not automatically fatal if other label copies are valid.
- A label from an old pool on an unused replacement disk matters only if you are about to reuse that disk. Clear it only after verifying backups and disk identity.

Advanced salvage options exist, but they are last-resort work. `zdb -B` can
generate a backup stream from a numeric objset ID when normal dataset metadata
is damaged but the dataset is still readable. `zdb -r` can copy a path or
object out of a dataset in some cases. Treat these as expert recovery tools:
work from cloned media, write output to a different pool such as `pool1`, and
document every command before running it.

Avoid this pattern on original disks unless you have accepted the risk:

Before using `zdb -F` or `zdb -X`:

- Clone suspect disks first when hardware may be failing.
- Capture labels, import output, SMART data, and pool history.
- Confirm backups or recovery images exist.
- Prefer documented `zpool import` dry runs.
- Treat any write-capable repair attempt as potentially destructive.

```bash
zdb -F pool0
zdb -FX pool0
```

For ordinary recovery, prefer the documented `zpool import -F -n` dry run first,
then an explicit import decision. `zdb -F` and `zdb -X` are deep debugging and
rewind tools, not everyday pool administration commands.

### Pool Rewind Recovery With -F

`zpool import -F` is recovery mode for a non-importable pool. It tries to make
the pool importable by discarding the last few transactions. This can recover a
pool after damaged recent metadata, but any discarded transactions are lost
permanently.

Always dry-run first:

```bash
zpool import -d /dev/disk/by-id -F -n pool0
```

If the dry run says recovery is possible and the data loss is acceptable, import
with recovery mode. Use `-N` to avoid mounting datasets immediately:

Before importing with `-F`:

- Save the dry-run output.
- Confirm the estimated lost transactions are acceptable.
- Use `-N` so datasets do not mount automatically.
- Plan to copy important data out after import.

```bash
zpool import -d /dev/disk/by-id -F -N pool0
```

After a successful rewind, scrub the pool and then copy or replicate important
data elsewhere:

```bash
zpool scrub pool0
zpool status pool0
zfs snapshot -r pool0/volume0@recovered-2026-05-18
zfs send -R pool0/volume0@recovered-2026-05-18 | zfs receive -u pool1/volume0
```

Use `-X` only as a last resort. It enables extreme transaction search and may
roll back to a transaction group that is not guaranteed to be consistent.

Dry-run the extreme option first:

```bash
zpool import -d /dev/disk/by-id -F -X -n pool0
```

Extreme recovery should be reserved for cases where the alternative is
restoring from backup or accepting loss:

```bash
zpool import -d /dev/disk/by-id -F -X -N pool0
```

If `zpool import` or `zpool status` prints a specific recovery command, prefer
that exact command over guessing. If no recovery action is offered and the pool
still cannot import, plan for backup restore or professional recovery rather
than trying destructive commands.

If a separate log device is missing, `-m` may allow import by discarding the
missing log device. Recent synchronous transactions can be lost.

```bash
zpool import -d /dev/disk/by-id -m -N pool0
```

### Recover A Destroyed Pool Entry

If a pool was destroyed with `zpool destroy`, it may still be listed with `-D`
until labels are overwritten.

List destroyed pools:

```bash
zpool import -D
```

Import a destroyed pool read-only and without mounting datasets:

```bash
zpool import -D -f -N -o readonly=on pool0
```

If this works, copy the data to another pool immediately. Do not treat this as a
normal undo feature; overwritten labels or reused disks can make recovery
impossible.

### Handling Permanent Data Errors

Permanent errors mean ZFS could not reconstruct some data from available
replicas. They are different from device counters that were corrected during a
scrub.

Start with verbose status:

```bash
zpool status -v pool0
```

If files are listed, restore those files from a snapshot, backup pool, or
replication target:

```bash
cp /pool0/volume0/.zfs/snapshot/daily-2026-06-30/file0.txt /pool0/volume0/file0.txt
```

Or restore from a backup dataset:

```bash
rsync -aHAX /pool1/volume0/file0.txt /pool0/volume0/file0.txt
```

If the error is in metadata, a directory, or an object that ZFS cannot map to a
file, prioritize copying readable data out and restoring the dataset from a
clean backup.

After restoring or accepting loss, scrub again:

```bash
zpool scrub pool0
zpool status -v pool0
```

Clear stale errors only after the scrub is clean or after you have documented
and accepted unrecoverable damage:

```bash
zpool clear pool0
```

Some OpenZFS versions support corrective receive, which can repair data blocks
from a suitable healthy send stream for the affected dataset. It cannot repair
metadata and it does not fix the hardware cause of corruption:

```bash
zfs send pool1/volume0@clean-2026-06-30 | zfs receive -c pool0/volume0
zpool scrub pool0
```

Best practice:

- Do not clear permanent errors before recording `zpool status -v`.
- Restore named files from backup instead of rolling back whole datasets when possible.
- If permanent errors return after restore, suspect hardware, cabling, RAM, or controller problems.
- Treat metadata permanent errors as high risk and restore the dataset or pool from a clean backup.

### Restore From Backup Pool

Restore into a new dataset for inspection first:

```bash
zfs send -R pool1/volume0@backup-2026-05-14 | zfs receive -u pool0/volume1
```

After verifying the restored data, move applications or users to the restored
dataset. Replacing an existing dataset should be a deliberate maintenance task,
not an automatic first step.

### Ransomware Or Mass Deletion Recovery

If clients are actively deleting or rewriting files, stop the writes first. A
perfect snapshot plan can still be damaged if the attacker or broken client has
permission to destroy snapshots.

Immediate actions:

- Disconnect affected clients or stop the share service.
- Disable shares for the affected datasets.
- Preserve existing snapshots with holds.
- Avoid rolling back until you know which snapshot is clean.
- Restore into a new dataset first, then cut users over.

Disable ZFS-managed shares where supported:

```bash
zfs set sharenfs=off pool0/volume0
zfs set sharesmb=off pool0/volume0
```

Optionally make the dataset read-only while investigating:

```bash
zfs set readonly=on pool0/volume0
```

Snapshot the current damaged state for investigation:

```bash
zfs snapshot -r pool0@incident-2026-06-30
zfs hold -r incident pool0@incident-2026-06-30
```

List candidate clean snapshots:

```bash
zfs list -t snapshot -r pool0/volume0
```

Clone a known-good snapshot for inspection:

```bash
zfs clone pool0/volume0@daily-2026-06-29 pool0/volume1
zfs set mountpoint=/pool0/volume1 pool0/volume1
```

Copy known-good data into a new recovery dataset:

```bash
zfs create -o mountpoint=/pool0/volume2 pool0/volume2
rsync -aHAX --info=progress2 /pool0/volume1/ /pool0/volume2/
```

After validation, repoint shares or applications to the recovered dataset. Keep
the incident snapshot and holds until investigation and backup verification are
finished.

### Recovery Practice Lab

Practice recovery on a throwaway system before you need it. This lab uses
file-backed vdevs under `/tmp`. Do not run it on a production host, and choose
an unused pool name.

Create a small mirror pool:

```bash
mkdir -p /tmp/zfs-lab
truncate -s 512M /tmp/zfs-lab/disk0 /tmp/zfs-lab/disk1 /tmp/zfs-lab/disk2
zpool create -o ashift=12 -O compression=zstd -m /tmp/zfs-lab/mnt pool2 mirror /tmp/zfs-lab/disk0 /tmp/zfs-lab/disk1
zfs create pool2/volume0
```

Create a file and snapshot:

```bash
echo important-data > /tmp/zfs-lab/mnt/volume0/file0.txt
zfs snapshot pool2/volume0@before-delete
```

Delete and restore one file:

```bash
rm /tmp/zfs-lab/mnt/volume0/file0.txt
cp /tmp/zfs-lab/mnt/volume0/.zfs/snapshot/before-delete/file0.txt /tmp/zfs-lab/mnt/volume0/file0.txt
```

Simulate one failed mirror side and replace it:

```bash
zpool offline pool2 /tmp/zfs-lab/disk1
zpool status pool2
zpool replace pool2 /tmp/zfs-lab/disk1 /tmp/zfs-lab/disk2
zpool status pool2
```

Practice send and receive to another dataset:

```bash
zfs snapshot pool2/volume0@backup-test
zfs send pool2/volume0@backup-test | zfs receive pool2/volume1
zfs list pool2/volume1
```

Clean up when finished:

```bash
zpool destroy pool2
rm -rf /tmp/zfs-lab
```

Practice goals:

- Restore one file from `.zfs/snapshot`.
- Read `zpool status` during a degraded mirror.
- Replace a failed device and watch resilver progress.
- Send and receive a snapshot.
- Destroy the lab pool only after confirming it is the throwaway pool.

### Restore Drill Templates

Run restore drills on a lab host, temporary dataset, or isolated backup target. Do not overwrite production datasets during a drill.

#### Single-File Restore Drill

Goal: prove that a user file can be restored without rolling back the whole dataset.

Setup:

```bash
zfs create -o mountpoint=/pool0/volume2 pool0/volume2
printf 'restore-drill\n' > /pool0/volume2/file0.txt
zfs snapshot pool0/volume2@drill-file-2026-07-05
cp /pool0/volume2/file0.txt /tmp/file0.before
rm /pool0/volume2/file0.txt
```

Restore:

```bash
cp /pool0/volume2/.zfs/snapshot/drill-file-2026-07-05/file0.txt /pool0/volume2/file0.txt
cmp /tmp/file0.before /pool0/volume2/file0.txt
```

Cleanup after validation, only if `pool0/volume2` is the temporary drill dataset:

```bash
zfs destroy -r pool0/volume2
```

Pass criteria:

- The file is restored with expected contents.
- Ownership and permissions are acceptable for the application.
- No unrelated files were changed.

Fail criteria:

- The snapshot is not visible or not mounted.
- The restored file differs unexpectedly.
- The recovery process required rolling back the whole dataset.

#### Dataset Restore From Backup Pool Drill

Goal: prove that a replicated dataset can be restored into a new dataset.

Restore into a temporary dataset:

```bash
zfs send -R pool1/volume0@backup-2026-07-05 | zfs receive -u pool0/volume2
zfs set mountpoint=/pool0/volume2 pool0/volume2
zfs mount pool0/volume2
```

Validate:

```bash
zfs list -r pool0/volume2
find /pool0/volume2 -maxdepth 2 -type f | head
```

Pass criteria:

- The received dataset mounts in the expected temporary location.
- Expected files, snapshots, and properties are present.
- Applications can read test data from the temporary dataset.

Fail criteria:

- The send stream fails or requires destructive receive against production data.
- Required snapshots are missing on the backup side.
- Properties such as mountpoint, readonly, recordsize, or encryption are not understood.

#### Ransomware Snapshot Drill

Goal: prove that a clean snapshot can be found and restored without trusting the damaged live dataset.

Inspect candidate snapshots:

```bash
zfs list -t snapshot -r pool0/volume0
zfs diff -FH pool0/volume0@daily-2026-07-04 pool0/volume0
```

Clone a clean snapshot:

```bash
zfs clone pool0/volume0@daily-2026-07-04 pool0/volume2
zfs set mountpoint=/pool0/volume2 pool0/volume2
```

Pass criteria:

- A known-clean snapshot can be identified.
- The clone mounts without overwriting production.
- Users or applications can validate data in the clone.

Fail criteria:

- Snapshots are missing, pruned too aggressively, or writable by the attacker.
- The only restore path requires destroying current evidence.
- Backup targets were reachable and damaged by the same attacker.

#### Encrypted Dataset Drill

Goal: prove that encryption keys and recovery notes work on a fresh boot or rescue environment.

Check key state:

```bash
zfs get -r encryptionroot,keystatus,keylocation,mounted pool0
```

Load and mount:

```bash
zfs load-key pool0/volume0
zfs mount pool0/volume0
```

Pass criteria:

- The documented key or passphrase unlocks the expected encryption root.
- The dataset mounts at the expected path.
- Recovery operators know where offline key backups are stored.

Fail criteria:

- The key is missing, unknown, or stored only on the failed system.
- The wrong encryption root is documented.
- The dataset imports but cannot be mounted because keys are unavailable.

#### Bare Import Drill

Goal: prove that a pool can be imported on rescue media without mounting over the live system.

Import read-only and no-mount:

```bash
zpool import -d /dev/disk/by-id -N -o readonly=on -R /mnt pool0
zpool status -gLPv pool0
zfs list -r pool0
zpool export pool0
```

Pass criteria:

- The pool imports read-only with stable device paths.
- Datasets are visible and do not mount over the rescue environment.
- Rescue media supports the pool's enabled feature flags.

Fail criteria:

- The pool cannot import because rescue media is too old.
- Device paths are unstable or ambiguous.
- The procedure requires force import without proving the old host is offline.

## Community FAQ: Top 20 Recurring ZFS Questions

These are common ZFS questions that repeatedly appear on Reddit,
Stack Overflow / Stack Exchange-style sites such as Server Fault, Super User,
and Unix & Linux, plus forums and NAS communities. The answers here are kept
practical and conservative.

### 1. What Is The Difference Between A Pool, Vdev, Dataset, And Zvol?

`pool0` is the storage pool. A vdev is a top-level redundancy group inside that
pool, such as a mirror or RAIDZ2 group. `pool0/volume0` is usually a filesystem
dataset. A zvol is a block device created with `zfs create -V`, for example
`pool0/volume1`.

```bash
zpool status pool0
zfs list
zfs list -t volume
```

### 2. Why Does One Bad Vdev Endanger The Whole Pool?

ZFS redundancy is at the vdev level. If `pool0` has two mirror vdevs, each
mirror must remain healthy enough to serve data. If any top-level data vdev is
lost, the whole pool can be lost. There is no extra parity layer above vdevs.

Better:

```bash
zpool create pool0 \
  mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1 \
  mirror /dev/disk/by-id/disk2 /dev/disk/by-id/disk3
```

Dangerous:

```bash
zpool create pool0 \
  mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1 \
  /dev/disk/by-id/disk2
```

### 3. Should ZFS Be Used On Top Of Hardware RAID?

Normally no. ZFS works best when it sees individual disks, serial numbers,
errors, latency, and flush behavior directly. Hardware RAID can hide failing
drives, reorder writes, block SMART visibility, and make recovery harder.

Use an HBA or controller in IT/JBOD mode:

```bash
zpool create pool0 mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1
```

### 4. Can A Single-Disk Pool Become A Mirror Later?

Yes. Use `zpool attach`, not `zpool add`.

```bash
zpool status pool0
zpool attach pool0 /dev/disk/by-id/disk0 /dev/disk/by-id/disk1
zpool status pool0
```

`attach` adds redundancy to an existing vdev. `add` creates a new top-level
vdev.

### 5. I Accidentally Used `zpool add` Instead Of `zpool attach`. What Now?

First, stop writing data and inspect the layout. Do not try to "balance" the
mistake by adding more random disks.

```bash
zpool status pool0
zpool list -v pool0
```

If you added a removable top-level mirror or single-disk vdev and your OpenZFS
version supports removal for that topology, `zpool remove` may work:

```bash
zpool remove pool0 /dev/disk/by-id/disk2
```

Before attempting removal:

- Make a fresh backup or confirm replication is current.
- Confirm the added vdev is not RAIDZ.
- Confirm your OpenZFS version supports removal for that exact layout.
- Expect removal to take time and create indirect mapping metadata.
- Keep monitoring with `zpool status`.

If the added vdev cannot be removed, the clean recovery is usually to back up,
destroy and recreate the pool correctly, then restore.

### 6. Can RAIDZ1 Be Converted To RAIDZ2 Or RAIDZ3 In Place?

No. RAIDZ expansion can widen a RAIDZ vdev on newer OpenZFS versions, but it
does not change the parity level. RAIDZ1 stays RAIDZ1, RAIDZ2 stays RAIDZ2,
and RAIDZ3 stays RAIDZ3.

To change parity level, create a new pool or new vdev and move the data:

```bash
zfs snapshot -r pool0@move-2026-05-14
zfs send -R pool0@move-2026-05-14 | zfs receive -u pool1/volume0
```

### 7. What Is The Safest Way To Expand A Pool?

For mirrors, add another mirror vdev:

```bash
zpool add pool0 mirror /dev/disk/by-id/disk4 /dev/disk/by-id/disk5
```

For RAIDZ, add another complete RAIDZ vdev or use RAIDZ expansion only if your
OpenZFS version supports it:

```bash
zpool get feature@raidz_expansion pool0
zpool attach pool0 raidz2-0 /dev/disk/by-id/disk12
```

Do not add a lone disk to a redundant pool.

### 8. Can A Vdev Be Removed?

Sometimes. Top-level mirror and single-disk vdev removal may be supported on
modern OpenZFS, but RAIDZ vdev removal is not a normal design path. Special
vdevs are pool-critical, and removing them may be unsupported or impractical in
many layouts.

Check before assuming:

```bash
zpool status pool0
zpool remove pool0 mirror-1
```

Plan pool topology as if top-level vdevs are permanent.

### 9. Why Did Deleting Files Not Free Space?

Common causes are snapshots, clones, open deleted files, reservations, zvols,
or refreservations. Start with snapshots.

```bash
zfs list -t snapshot -o name,used,refer
zfs list -o name,used,avail,refer,usedsnap,usedds,usedrefreserv
```

Destroy old snapshots only when they are no longer needed:

```bash
zfs destroy pool0/volume0@daily-2026-04-14
```

### 10. Why Do `zpool list`, `zfs list`, And `df` Show Different Space?

They measure different layers. `zpool list` reports pool-level allocation.
`zfs list` reports dataset-level space after ZFS accounting. `df` reports what
the mounted filesystem presents to applications. Snapshots, reservations,
parity, metadata, refreservations, and slop space can make the numbers differ.

Use ZFS tools first:

```bash
zpool list pool0
zfs list -o name,used,avail,refer,mountpoint
zfs get quota,reservation,refquota,refreservation pool0/volume0
```

### 11. Are Snapshots Backups?

No. Snapshots are excellent local recovery points, but they live on the same
pool. If `pool0` is destroyed, stolen, overwritten, or lost, its snapshots are
lost too. Replicate snapshots to another pool or host.

```bash
zfs snapshot -r pool0/volume0@daily-2026-05-14
zfs send -R pool0/volume0@daily-2026-05-14 | zfs receive -u pool1/volume0
```

### 12. How Should I Use `zfs send` And `zfs receive`?

Use a full send first, then incremental sends. Let the first receive create the
destination dataset.

```bash
zfs snapshot -r pool0/volume0@backup-2026-05-14
zfs send -R pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume0

zfs snapshot -r pool0/volume0@backup-2026-05-15
zfs send -R -I pool0/volume0@backup-2026-05-14 pool0/volume0@backup-2026-05-15 | zfs receive -u pool1/volume0
```

For encrypted datasets where the receiver should not decrypt data, use raw
sends:

```bash
zfs send -w pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume0
```

### 13. Should Deduplication Be Enabled?

Usually no. Deduplication needs careful memory, metadata, and workload planning.
It can make writes, deletes, and pool recovery much slower. Compression is the
right default for most systems.

The main risk is the deduplication table. If the DDT no longer fits well in RAM,
ZFS has to read DDT entries from disk during writes and frees. On hard-disk
pools this can turn normal work into heavy random I/O. Disabling dedup only
affects new writes; existing deduped blocks keep their DDT entries until the old
blocks and snapshots are removed or rewritten.

```bash
zfs set compression=zstd pool0/volume0
zfs get dedup pool0/volume0
zpool status -DD pool0
```

Only enable dedup after testing with representative data and a recovery plan.

### 14. What `recordsize` Should I Use?

Set `recordsize` per dataset before writing data. Use larger records for large
sequential files and smaller records for databases or VM image files.

```bash
zfs set recordsize=128K pool0/volume0  # general files
zfs set recordsize=1M pool0/volume2    # media, archives, torrents
zfs set recordsize=16K pool0/volume4   # some databases
```

Changing `recordsize` affects newly written blocks only. Existing data must be
rewritten to adopt the new size.

### 15. Do I Need SLOG, L2ARC, Or A Special Vdev?

Usually not at first. Add RAM, pick a good pool layout, and measure the workload
before adding support vdevs.

- SLOG helps synchronous writes when backed by a fast, power-loss-protected device.
- L2ARC is a read cache and does not replace RAM.
- Special vdevs can speed metadata and small blocks, but they are critical to the pool and should be redundant.

Check sync behavior before buying a SLOG:

```bash
zfs get sync pool0/volume0
zpool iostat -v pool0 5
```

### 16. How Much RAM Does ZFS Need? Is ECC Required?

There is no universal rule such as "1 GB RAM per 1 TB storage" for normal ZFS.
More RAM improves ARC caching, metadata-heavy workloads, and dedup-heavy
systems. ECC is strongly recommended for important data because ZFS can repair
bad on-disk copies, but it cannot make unreliable memory reliable.

On Linux, ARC limits can be tuned through module options, but tune only after
observing real memory pressure.

### 17. Why Are Writes Slow?

Common causes include sync writes without a suitable SLOG, an over-wide RAIDZ
vdev, SMR disks, a nearly full pool, small random writes, wrong `recordsize` or
`volblocksize`, encryption CPU limits, weak controllers, bad cables, snapshots
on busy zvols, or simply expecting mirror-like IOPS from RAIDZ.

Start with:

```bash
zpool status pool0
zpool iostat -v pool0 5
zfs get recordsize,volblocksize,sync,compression pool0/volume0
zpool list -o name,capacity,fragmentation,health pool0
```

### 18. What Is `ashift`, And Why Does Everyone Recommend `ashift=12`?

`ashift` is the sector-size exponent used by a vdev. `ashift=12` means 4096-byte
sectors and is a safe default for most modern HDDs and SSDs, including many
devices that report 512-byte logical sectors. It is set when the vdev is
created and cannot be changed for that vdev later.

```bash
zpool create -o ashift=12 pool0 mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1
```

### 19. How Do I Import A Pool After Moving Disks?

List importable pools:

```bash
zpool import
```

Import using stable names:

```bash
zpool import -d /dev/disk/by-id pool0
```

Import using short `/dev` names and mount below `/mnt`:

```bash
zpool import -d /dev -R /mnt pool0
```

For recovery, avoid mounting datasets immediately:

```bash
zpool import -d /dev -N -o readonly=on pool0
```

### 20. Why Is A Dataset Busy And Unable To Destroy, Export, Or Unmount?

Something is still using it. Common causes are a shell with its current working
directory inside the dataset, a running service, NFS or SMB sharing, a container
mount, an open deleted file, a child dataset, a clone, or a snapshot hold.

Inspect before forcing anything:

```bash
zfs list -r pool0/volume0
zfs holds -r pool0/volume0
lsof +f -- /pool0/volume0
fuser -vm /pool0/volume0
```

Then stop the process, unshare the dataset, remove holds, or destroy dependent
clones deliberately.

## Common Mistakes

### Adding A Single Disk To A Redundant Pool

Bad:

```bash
zpool add pool0 /dev/disk/by-id/disk4
```

This creates a new top-level vdev. If that one disk fails, the whole pool can
fail even if the original vdevs are redundant.

Better:

```bash
zpool add pool0 mirror /dev/disk/by-id/disk4 /dev/disk/by-id/disk5
```

### Assuming RAID Is Backup

Redundancy protects against some disk failures. It does not protect against:

- Accidental deletion.
- Ransomware.
- Theft.
- Fire.
- Controller bugs.
- Admin mistakes.
- Silent application-level corruption already written to disk.

Use snapshots and backups.

### Filling The Pool

Bad:

```bash
zpool list pool0
# Capacity near 95%
```

Better:

```bash
zpool list -o name,capacity,free pool0
```

Plan expansion before the pool is full.

### Ignoring Snapshots

Snapshots can consume space when data changes.

Check snapshot space:

```bash
zfs list -t snapshot -o name,used,refer
```

Destroy old snapshots deliberately:

```bash
zfs destroy pool0/volume0@daily-2026-04-14
```

### Enabling Dedup Without Planning

Bad:

```bash
zfs set dedup=on pool0/volume0
```

Better:

```bash
zfs set compression=zstd pool0/volume0
```

Why: if the DDT grows beyond available ARC/RAM, writes and frees may need
random metadata reads from disk. Turning dedup off later stops new dedup work,
but old deduped blocks keep their DDT entries until those blocks are removed or
rewritten.

### Using Unstable Disk Names

Bad:

```bash
zpool create pool0 mirror /dev/sdb /dev/sdc
```

Better:

```bash
zpool create pool0 mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1
```

## Example Build: General Home Or Small Server

Create a mirrored pool:

```bash
zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -O xattr=sa \
  -O acltype=posixacl \
  -m /pool0 \
  pool0 \
  mirror /dev/disk/by-id/disk0 /dev/disk/by-id/disk1
```

Create datasets:

```bash
zfs create -o mountpoint=/pool0/volume0 pool0/volume0
zfs create -o mountpoint=/pool0/volume1 pool0/volume1
zfs create -o mountpoint=/pool0/volume2 pool0/volume2
```

Set properties:

```bash
zfs set recordsize=128K pool0/volume0
zfs set recordsize=128K pool0/volume1
zfs set recordsize=1M pool0/volume2
zfs set quota=500G pool0/volume1
```

Create initial snapshots:

```bash
zfs snapshot -r pool0@initial-2026-05-14
```

Check health:

```bash
zpool status pool0
zfs list
```

## Example Build: Backup Pool

Create `pool1` as a RAIDZ2 backup pool:

```bash
zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool1 \
  pool1 \
  raidz2 \
  /dev/disk/by-id/disk0 \
  /dev/disk/by-id/disk1 \
  /dev/disk/by-id/disk2 \
  /dev/disk/by-id/disk3 \
  /dev/disk/by-id/disk4 \
  /dev/disk/by-id/disk5
```

Send a recursive backup from `pool0/volume0`. For the first full receive,
`pool1/volume0` should not already exist:

```bash
zfs snapshot -r pool0/volume0@backup-2026-05-14
zfs send -R pool0/volume0@backup-2026-05-14 | zfs receive -u pool1/volume0
```

Export when using removable disks:

```bash
zpool export pool1
```

## Example Build: VM Pool

Create mirror-based VM pool:

```bash
zpool create \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -m /pool2 \
  pool2 \
  mirror /dev/disk/by-id/nvme0 /dev/disk/by-id/nvme1 \
  mirror /dev/disk/by-id/nvme2 /dev/disk/by-id/nvme3
```

Create VM dataset:

```bash
zfs create -o mountpoint=/pool2/volume0 pool2/volume0
zfs set recordsize=64K pool2/volume0
```

Create VM zvol:

```bash
zfs create -o volblocksize=16K -V 200G pool2/volume1
```

Snapshot before maintenance:

```bash
zfs snapshot pool2/volume1@before-maintenance-2026-05-14
```

## Quick Reference

Inventory disks before pool creation:

```bash
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
ls -l /dev/disk/by-id/
smartctl -a /dev/disk/by-id/disk0
```

Run SMART tests:

```bash
smartctl -t short /dev/disk/by-id/disk0
smartctl -t long /dev/disk/by-id/disk0
```

Check packaged ZFS automation units:

```bash
systemctl list-unit-files 'zfs*'
systemctl list-timers 'zfs*'
```

Check ZED event daemon on systemd systems:

```bash
systemctl status zfs-zed.service
zpool events -v
```

Pool health:

```bash
zpool status -x
```

Detailed status:

```bash
zpool status -v pool0
```

Command history:

```bash
zpool history -l pool0
```

Detailed recovery status with GUIDs and paths:

```bash
zpool status -gLPv pool0
```

Minimum recovery bundle:

```bash
zpool status -gLPv pool0
zpool list -v pool0
zpool events -v
zpool history -l pool0
zfs list -r pool0
zfs list -t snapshot -r pool0
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
smartctl -a /dev/disk/by-id/disk0
```

Performance baseline:

```bash
zpool status -v pool0
zpool list -o name,size,alloc,free,capacity,fragmentation,health
zpool iostat -v pool0 5
zfs get -r compression,recordsize,volblocksize,sync,logbias,primarycache,secondarycache,dedup pool0
```

Dedup table statistics and ARC residency:

```bash
zpool status -DD pool0
```

Inspect ZFS labels on a physical disk:

```bash
zdb -l /dev/disk/by-id/disk0
```

Inspect labels and uberblocks:

```bash
zdb -lu /dev/disk/by-id/disk0
```

Inspect an exported pool using a specific device directory:

```bash
zdb -e -p /dev/disk/by-id -C pool0
```

List pools:

```bash
zpool list
```

List datasets:

```bash
zfs list
```

Show property sources:

```bash
zfs get -r -o name,property,value,source compression,atime,relatime,xattr,recordsize pool0
```

Check ACL and extended attribute settings:

```bash
zfs get acltype,xattr pool0/volume0
```

Create dataset:

```bash
zfs create pool0/volume0
```

Create snapshot:

```bash
zfs snapshot pool0/volume0@daily-2026-05-14
```

List snapshots:

```bash
zfs list -t snapshot
```

Hold recovery snapshot:

```bash
zfs hold keep pool0/volume0@daily-2026-05-14
```

List snapshot holds:

```bash
zfs holds pool0/volume0@daily-2026-05-14
```

Compare snapshots:

```bash
zfs diff -FH pool0/volume0@daily-2026-05-14 pool0/volume0@daily-2026-05-15
```

Rollback:

```bash
zfs rollback pool0/volume0@daily-2026-05-14
```

Destroy snapshot:

```bash
zfs destroy pool0/volume0@daily-2026-05-14
```

Start scrub:

```bash
zpool scrub pool0
```

Stop scrub:

```bash
zpool scrub -s pool0
```

Initialize free regions:

```bash
zpool initialize pool0
```

Wait for background activity:

```bash
zpool wait -t scrub,resilver,initialize pool0
```

Export pool:

```bash
zpool export pool0
```

Import pool:

```bash
zpool import pool0
```

Import read-only without mounting:

```bash
zpool import -N -o readonly=on pool0
```

Import by pool ID under a temporary name:

```bash
zpool import 1234567890123456789 pool2
```

Force read-only no-mount import after verifying the old host is offline:

```bash
zpool import -f -N -o readonly=on pool0
```

Create pool checkpoint:

```bash
zpool checkpoint pool0
```

Discard pool checkpoint:

```bash
zpool checkpoint -d pool0
```

Rewind to pool checkpoint:

```bash
zpool export pool0
zpool import --rewind-to-checkpoint pool0
```

Check pool feature compatibility:

```bash
zpool upgrade
zpool get compatibility pool0
```

Document pool layout:

```bash
zpool status -gLPv pool0
zpool get all pool0
zfs get -r all pool0
zpool history -l pool0
```

Send snapshot:

```bash
# First full receive; pool1/volume0 should not already exist.
zfs send pool0/volume0@daily-2026-05-14 | zfs receive pool1/volume0
```

Incremental send:

```bash
zfs send -I pool0/volume0@daily-2026-05-14 pool0/volume0@daily-2026-05-15 | zfs receive pool1/volume0
```

Resume interrupted receive:

```bash
zfs get receive_resume_token pool1/volume0
zfs send -t TOKEN | zfs receive -s -u pool1/volume0
```

Inspect a send stream:

```bash
zstreamdump /safe-copy/pool0-volume0-2026-05-14.zfs
```

Set special small block threshold:

```bash
zfs set special_small_blocks=16K pool0/volume0
```

Add a hot spare:

```bash
zpool add pool0 spare /dev/disk/by-id/disk4
```

List events:

```bash
zpool events -v
```

Create a temporary restore target:

```bash
zfs send -R pool1/volume0@backup-2026-07-05 | zfs receive -u pool0/volume2
```

Load encryption key:

```bash
zfs load-key pool0/volume0
```

## Maintenance Schedule

Daily:

- Check alerts.
- Confirm free space is healthy.
- Confirm backups completed.

Weekly:

- Review `zpool status`.
- Watch for repeated device errors, failed SMART tests, and disks that drop offline.
- Review `zpool events -v` for new hardware or data errors.
- Review `zpool history -l` after maintenance or incidents.
- Confirm ZED, SMART, snapshot, scrub, and replication automation reported success.
- Review snapshot growth.
- Confirm backup replication.

Monthly:

- Run or verify scrub completion.
- Check SMART data.
- Test a small restore.
- Review the command risk matrix before planned topology, feature, or recovery changes.
- Check feature-flag compatibility before any planned `zpool upgrade`.
- Test loading encryption keys for encrypted recovery datasets.
- Review pool capacity trend.
- Confirm no old pool checkpoint was accidentally left behind.
- Review pool documentation after disk replacement or layout changes.

Quarterly:

- Test a full restore path.
- Practice the recovery lab on a throwaway host or VM.
- Run at least one restore drill template and record pass/fail results.
- Test import on current rescue media for critical pools.
- Verify resumable replication and backup documentation.
- Inspect one archived send stream and perform a test receive.
- Review retention policy.
- Confirm recovery media works.
- Review whether feature upgrades are needed.
- Recheck whether rescue media still supports all enabled pool features.

## Final Best Practices Checklist

- Use direct disk access, not hardware RAID.
- Use `/dev/disk/by-id/` paths.
- Verify platform-specific service names, boot behavior, ACL behavior, and appliance rules before applying generic examples.
- Burn in and inventory new disks before creating important pools.
- Use mirrors for performance-sensitive workloads.
- Use RAIDZ2 or RAIDZ3 for large archive pools.
- Plan usable capacity with parity, snapshots, reservations, and free-space headroom.
- Use `ashift=12` for modern disks unless you know otherwise.
- Enable `compression=zstd` by default.
- Use `atime=off` unless software needs access-time updates.
- Use `xattr=sa` on Linux datasets with POSIX ACLs or heavy extended attributes after checking compatibility needs.
- Create separate datasets for separate policies.
- Use property inheritance deliberately and review property sources.
- Set `recordsize` or `volblocksize` before writing data.
- Keep pools below 80% used.
- Schedule scrubs.
- Initialize new important pools when the extra I/O is acceptable.
- Monitor SMART health.
- Replace disks based on evidence, not guesswork, and resilver one disk at a time when possible.
- Read `zpool status` state, action, and error lines before changing anything.
- Use the command risk matrix before topology, feature, destroy, receive, import, rewind, or deep recovery commands.
- Keep annotated `zpool status` examples handy for incident triage.
- Snapshot automatically.
- Use `zfs diff` to inspect suspicious changes before rollback.
- Hold critical recovery snapshots during incidents.
- Replicate backups with `zfs send`.
- Treat replication as complete only after destination inspection and restore testing.
- Inspect archived send streams and still perform real test receives.
- Use resumable receives for large backup or restore streams.
- Treat redacted sends as advanced partial-copy workflows, not full backups.
- Use pool checkpoints only for short whole-pool maintenance rollback windows.
- Import read-only and without mounting when investigating damaged pools.
- Capture the minimum recovery bundle before changing pool state in an incident.
- Fill out an incident evidence form for failed imports, permanent errors, missing devices, ransomware, or major operator mistakes.
- Start recovery from the symptom-based runbook and move from inspection to change only when the path is clear.
- Force import only after verifying the pool is not active on another host.
- Clone failing disks before aggressive recovery attempts.
- Use `zdb` for offline inspection of labels, GUIDs, and uberblocks, not as a routine repair command.
- Test restores.
- Run restore drills and record pass/fail criteria.
- Configure ZED, SMART, scrub, snapshot, and replication automation where appropriate.
- Capture a performance baseline before tuning.
- Avoid dedup unless the dedup ratio, DDT size, RAM, and recovery plan are all acceptable.
- Use native encryption where appropriate.
- Keep encryption keys or passphrases recoverable offline.
- Configure ZFS event alerting.
- Review `zpool history` during incidents.
- Be careful with `zpool upgrade` and feature flags.
- Mirror special vdevs and keep `special_small_blocks` thresholds conservative.
- Use separate datasets, quotas, and snapshots for container state.
- Test hot spare behavior before relying on it.
- Replace failing disks promptly.
- Document pool layout, disk serials, and recovery steps.
- Keep platform-specific notes for boot pools, ACLs, service names, and rescue media.

## References

The official OpenZFS documentation should be preferred over forum posts when
checking command syntax, feature availability, and destructive operation
behavior. The community links are useful for examples and recurring questions,
but **always verify** advice against the manual pages **for your OpenZFS version**.

Official OpenZFS documentation:

- [OpenZFS Documentation](https://openzfs.github.io/openzfs-docs/)
- [OpenZFS master man pages](https://openzfs.github.io/openzfs-docs/man/master/index.html)
- [ZFS concepts](https://openzfs.github.io/openzfs-docs/man/master/7/zfsconcepts.7.html)
- [ZFS properties](https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html)
- [Zpool concepts](https://openzfs.github.io/openzfs-docs/man/master/7/zpoolconcepts.7.html)
- [Zpool properties](https://openzfs.github.io/openzfs-docs/man/master/7/zpoolprops.7.html)
- [Zpool feature flags](https://openzfs.github.io/openzfs-docs/man/master/7/zpool-features.7.html)
- [OpenZFS messages](https://openzfs.github.io/openzfs-docs/msg/index.html)

Pool administration:

- [zpool-create(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-create.8.html)
- [zpool-add(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-add.8.html)
- [zpool-attach(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-attach.8.html)
- [zpool-detach(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-detach.8.html)
- [zpool-remove(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-remove.8.html)
- [zpool-replace(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-replace.8.html)
- [zpool-online(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-online.8.html)
- [zpool-offline(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-offline.8.html)
- [zpool-status(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-status.8.html)
- [zpool-list(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-list.8.html)
- [zpool-iostat(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-iostat.8.html)
- [zpool-clear(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-clear.8.html)
- [zpool-scrub(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-scrub.8.html)
- [zpool-resilver(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-resilver.8.html)
- [zpool-checkpoint(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-checkpoint.8.html)
- [zpool-initialize(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-initialize.8.html)
- [zpool-trim(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-trim.8.html)
- [zpool-wait(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-wait.8.html)
- [zpool-history(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-history.8.html)
- [zpool-events(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-events.8.html)
- [zpool-import(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-import.8.html)
- [zpool-export(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-export.8.html)
- [zpool-upgrade(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-upgrade.8.html)
- [zpool-labelclear(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-labelclear.8.html)

Dataset, snapshot, and replication commands:

- [zfs-create(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-create.8.html)
- [zfs-destroy(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-destroy.8.html)
- [zfs-set(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-set.8.html)
- [zfs-get(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-get.8.html)
- [zfs-inherit(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-inherit.8.html)
- [zfs-list(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-list.8.html)
- [zfs-mount(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-mount.8.html)
- [zfs-unmount(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-unmount.8.html)
- [zfs-snapshot(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-snapshot.8.html)
- [zfs-hold(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-hold.8.html)
- [zfs-release(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-release.8.html)
- [zfs-diff(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-diff.8.html)
- [zfs-clone(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-clone.8.html)
- [zfs-promote(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-promote.8.html)
- [zfs-rollback(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-rollback.8.html)
- [zfs-send(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-send.8.html)
- [zfs-receive(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-receive.8.html)
- [zfs-bookmark(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-bookmark.8.html)
- [zfs-redact(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-redact.8.html)
- [zfs-rewrite(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-rewrite.8.html)
- [zfs-load-key(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-load-key.8.html)
- [zfs-unload-key(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-unload-key.8.html)
- [zfs-change-key(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-change-key.8.html)
- [zfs-allow(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-allow.8.html)
- [zfs-unallow(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-unallow.8.html)

Debugging, recovery, and supporting OpenZFS tools:

- [zdb(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zdb.8.html)
- [zstreamdump(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zstreamdump.8.html)
- [zgenhostid(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zgenhostid.8.html)
- [zed(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zed.8.html)
- [zinject(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zinject.8.html)

Platform and scheduler documentation:

- [OpenZFS Getting Started: Debian](https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/index.html)
- [OpenZFS Getting Started: Ubuntu](https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/index.html)
- [Debian Package Search: zfsutils-linux](https://packages.debian.org/search?keywords=zfsutils-linux)
- [Ubuntu Package Search: zfsutils-linux](https://packages.ubuntu.com/search?keywords=zfsutils-linux)
- [FreeBSD Handbook ZFS Chapter](https://docs.freebsd.org/en/books/handbook/zfs/)
- [Proxmox VE Administration Guide](https://pve.proxmox.com/pve-docs/pve-admin-guide.html)
- [Proxmox VE ZFS on Linux Wiki](https://pve.proxmox.com/wiki/ZFS_on_Linux)
- [TrueNAS Documentation](https://www.truenas.com/docs/)
- [TrueNAS Storage Documentation](https://www.truenas.com/docs/scale/scaletutorials/storage/)
- [systemd.timer(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html)
- [systemd.service(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html)

Related tools mentioned in this guide:

- [smartmontools](https://www.smartmontools.org/)
- [e2fsprogs badblocks](https://e2fsprogs.sourceforge.net/)
- [fio](https://fio.readthedocs.io/)
- [GNU ddrescue](https://www.gnu.org/software/ddrescue/)
- [Sanoid and Syncoid](https://github.com/jimsalterjrs/sanoid)
- [zrepl](https://zrepl.github.io/)
- [zfs-auto-snapshot](https://github.com/zfsonlinux/zfs-auto-snapshot)

Community discussion and Q&A portals:

- [Reddit r/zfs](https://www.reddit.com/r/zfs/)
- [Stack Overflow ZFS tag](https://stackoverflow.com/questions/tagged/zfs)
- [Server Fault ZFS tag](https://serverfault.com/questions/tagged/zfs)
- [Super User ZFS tag](https://superuser.com/questions/tagged/zfs)
- [Unix & Linux Stack Exchange ZFS tag](https://unix.stackexchange.com/questions/tagged/zfs)
- [TrueNAS Community Forums](https://forums.truenas.com/)
- [Proxmox Forum ZFS tag](https://forum.proxmox.com/tags/zfs/)

## Disclaimer

This guide was compiled and refined with LLM assistance, based on **personal ZFS notes from the past 15 years**.

- **Signed:** Grzegorz "SinusPL" Surmann

---
Copyright 2011-2026 by Grzegorz Surmann