NVME experience?

NVME experience?

[Graham Cobb]

I am about to deploy my first btrfs filesystems on NVME. Does anyone
have any hints or advice? Initially they will be root disks, but I am
thinking about also moving home disks and other frequently used data to
NVME, but probably not backups and other cold data.

I am mostly wondering about non-functionals like typical failure modes,
life and wear, etc. Which might affect decisions like how to split
different filesystems among the drives, whether to mix NVME with other
drives (SSD or HDD), etc.

Are NVME drives just SSDs with a different interface? With similar
lifetimes (by bytes written, or another measure)? And similar typical
failure modes?

Are they better or worse in terms of firmware bugs? Error detection for
corrupted data? SMART or other indicators that they are starting to fail
and should be replaced?

I assume that they do not (in practice) suffer from "faulty cable" problems.

Anyway, I am hoping someone has experiences to share which might be useful.

Graham

Re: NVME experience?

[Zygo Blaxell]

On Sat, Jan 16, 2021 at 11:42:03PM +0000, Graham Cobb wrote:
> I am about to deploy my first btrfs filesystems on NVME. Does anyone
> have any hints or advice? Initially they will be root disks, but I am
> thinking about also moving home disks and other frequently used data to
> NVME, but probably not backups and other cold data.
> 
> I am mostly wondering about non-functionals like typical failure modes,
> life and wear, etc. Which might affect decisions like how to split
> different filesystems among the drives, whether to mix NVME with other
> drives (SSD or HDD), etc.

It doesn't really make sense to RAID a NVME device except with another
NVME device.  NVME's strength is access latency, and mirroring it with a
SATA device loses that strength (there is also more bandwidth in NVME
than SATA, if the underlying SSD is fast enough to use it).

If you need to cram one more disk into a full machine, and all you have
an unused NVME slot, it'll work, but it's wasteful.

You can use a NVME device as a dm-cache or bcache device--the low-latency
interface is ideal for caching use cases, if you have a suitably fast
and robust SSD.

> Are NVME drives just SSDs with a different interface? With similar
> lifetimes (by bytes written, or another measure)? And similar typical
> failure modes?

Pretty much.  Vendors will often sell model "XYZ123" as a SATA drive and
model "XYZ124" as NVME, with identical specs other than the interface
and different prices.  Probably most of the PCB is the same as well, as
M.2 board area can fit inside a 2.5" SATA box.  Firmware on the interface
side will be different, but firmware on the media side will be the same.

As a result, the device lifetime, failure modes, and firmware bugs are
often identical for NVME and SATA, and you have to read the data
sheets just as carefully to know what you're getting.

Like SATA disks, NVME devices can have volatile RAM caches and can
therefore (in theory) have power-loss write-cache bugs--it's

	nvme set-feature -f 6 -v 0 -s /dev/nvme...

to disable write cache on those, not 'hdparm -W0 /dev/sd...' as for SATA.
To date I haven't met an NVME device that needs this, but I have
relatively few of them compared to SATA and the industry is still young,
so I expect one will come along some day.

> Are they better or worse in terms of firmware bugs? Error detection for
> corrupted data? SMART or other indicators that they are starting to fail
> and should be replaced?

They need their own protocol for SMART and similar data, and the specific
data sets returned are different, but this is not more different than
it is between any other SATA drive models.  Up-to-date smartmontools
just works on most of the NVMEs I've tried, the rest worked a month or
two later at most.

Since they're basically the same drives, their SMART data has basically
the same usefulness on NVME and SATA within a device model family,
i.e. if the firmware self-test can't detect bad sectors in the SATA model,
it won't be able to detect them in the NVME version either.

> I assume that they do not (in practice) suffer from "faulty cable" problems.

This is a theoretical problem (like dust in the connector, bent pins,
manufacturing defect, etc) but I've never hit it in practice so I can't
say much about what it looks like.

Usually if a PCIE card isn't seated correctly it doesn't work in
obvious ways.  NVME would be similar.

NVME devices are held in their slots with screws, so you're not going to
have "loose connector" or "pulled cable" issues that are possible with
SATA (though in cable-less setups, SATA drives are held in their slots
with screws too).

> Anyway, I am hoping someone has experiences to share which might be useful.

If you want more than one or two NVME on the same consumer mainboard,
then you sometimes have to worry about having enough PCIE lanes for your
other devices, as each NVME device will permanently occupy one.  "You can
use the second NVME port _or_ two of the built-in SATA ports _or_ one
of the X16 slots" is a common trade-off as the BIOS can switch one PCIE
lane to multiple points on the mainboard but not share it between devices.

This means NVME devices are frequently limited to single-device
configurations on consumer mainboards, two devices on some newer
mainboards.

You can buy M.2 SSDs that don't have NVME interfaces (they implement
SATA in the connector, there's a B key notch instead of M key so they
will not fit in an incompatible slot).  Watch out for that.

> Graham

Fast, SLC SAS SSD's as dm-cache or bcache [Re: NVME experience?]

[Andrew Luke Nesbit]

Dear Zygo,

Thank you for this fantastic message.  I hope you -- and the rest of the 
list -- doesn't mind my jumping in mid-message...

On 17/01/2021 19:55, Zygo Blaxell wrote:
> You can use a NVME device as a dm-cache or bcache device--the low-latency
> interface is ideal for caching use cases, if you have a suitably fast
> and robust SSD.

I have about 4 enterprise grade HGST SLC 100 GB and SLC 200 GB SAS 
SSD's.  I want to use these as some kind of cache or fast tier for a NAS 
I am building.

**It has been suggested to me that I should use these SLC SAS drives 
with dm-cache, by pairing one of each of four SSD's with a corresponding 
high capacity NAS drive.**

This is for a home NAS so quietness is the major concern.  Putting the 
NAS in our living room is the only practical option _at this point in 
time_ for various reasons.

This is going to be a "pure NAS" and the chassis will be roomy.  For 
example, I already have both the following chassis and I have yet to 
choose one or the other:

-   Supermicro SC 732D4F-903B ( 
https://www.supermicro.com/en/products/chassis/tower/732/SC732D4F-903B )

-   Fractal Define XL R2 ( 
https://www.fractal-design.com/products/cases/define/define-xl-r2/black-pearl/ 
)

I think WD Red or Seagate Ironwolf drives would be best option as they 
run at 5400 RPM and keep vibration lower.  4x 8-14 TB HDD's in mirrored 
pairs (obvioulsy taken the dm-cache pair also into account).

Note to self: Investigate HDD mounting kits to absorb mechanical 
vibration.  Recommendations are welcome!

Any other suggestions would be greatly welcome!!

Andrew