@poinck I wouldn’t do a mirror and a Z2 in the same pool for fault tolerance reasons. Ideally, all vdevs in a pool should be able to tolerate the same number of faulted devices.
A pool of mirrors has three major advantages:
- Each disk can read independently, so you can get really high random read performance. In contrast, all disks in a raidz have to do the same thing at the same time. This can be a big deal for a pool of SSDs.
- Write performance of a pool correlates to the number of vdevs (with the exception of draid). A pool of mirrors typically gives you more vdevs, so it gets higher random write performance.
- If the pool has only ever contained mirrors, you can *remove* a mirror from the pool as long as you have enough capacity. The data will be evacuated to other vdevs.
ARC should prevent the disks from needing to be read often, minimizing advantage 1 at the cost of RAM. With transaction grouping, ZFS should only see random write load with sync writes, and a log vdev turns those into sequential writes on your pool devices, minimizing advantage 2.
With recent versions of ZFS, you can expand a raidz vdev a drive at a time. You could start a raidz2 with three drives (one drive of capacity), then add one drive at a time. There are some limitations with this, so research it further before relying on it.
@poinck Once you add a raidz vdev to a pool, you can’t remove the mirror. You’d be stuck with the mismatch until the whole pool is destroyed and rebuilt. This isn’t a technical problem, but it makes maintenance more painful (which two drive bays are the mirror, again?). If you want raidz, you should really start with it to save yourself future pain.
You also can’t increase the fault tolerance of a raidz vdev. That is, you can’t take a raidz1 and add a drive to make it a raidz2.
I personally don’t trust single-fault-tolerant vdevs. Resilvering takes long enough I want double fault tolerance for all spinning disks bigger than maybe 4 TB. That means raidz2 or three-member mirrors.