Re: btrfs vs write caching firmware bugs (was: Re: BTRFS recovery not possible)

[Zygo Blaxell <ce3g8jdj@umail.furryterror.org>]

[-- Attachment #1: Type: text/plain, Size: 3427 bytes --]

On Mon, Jun 24, 2019 at 12:37:51AM -0400, Zygo Blaxell wrote:
> On Sun, Jun 23, 2019 at 10:45:50PM -0400, Remi Gauvin wrote:
> > On 2019-06-23 4:45 p.m., Zygo Blaxell wrote:
> > 
> > > 	Model Family: Western Digital Green Device Model: WDC WD20EZRX-00DC0B0 Firmware Version: 80.00A80
> > > 
> > > Change the query to 1-30 power cycles, and we get another model with
> > > the same firmware version string:
> > > 
> > > 	Model Family: Western Digital Red Device Model: WDC WD40EFRX-68WT0N0 Firmware Version: 80.00A80
> > > 
> > 
> > > 
> > > These drives have 0 power fail events between mkfs and "parent transid
> > > verify failed" events, i.e. it's not necessary to have a power failure
> > > at all for these drives to unrecoverably corrupt btrfs.  In all cases the
> > > failure occurs on the same days as "Current Pending Sector" and "Offline
> > > UNC sector" SMART events.  The WD Black firmware seems to be OK with write
> > > cache enabled most of the time (there's years in the log data without any
> > > transid-verify failures), but the WD Black will drop its write cache when
> > > it sees a UNC sector, and btrfs notices the failure a few hours later.
> > > 
> > 
> > First, thank you very much for sharing.  I've seen you mention several
> > times before problems with common consumer drives, but seeing one
> > specific identified problem firmware version is *very* valuable info.
> > 
> > I have a question about the Black Drives dropping the cache on UNC
> > error.  If a transid id error like that occurred on a BTRFS RAID 1,
> > would BTRFS find the correct metadata on the 2nd drive, or does it stop
> > dead on 1 transid failure?
> 
> Well, the 2nd drive has to have correct metadata--if you are mirroring
> a pair of disks with the same firmware bug, that's not likely to happen.

OK, I forgot the Black case is a little complicated...

I guess if you had two WD Black drives and they had all their UNC sector
events at different times, then the btrfs RAID1 repair should still
work with write cache enabled.  That seems kind of risky, though--what
if something bumps the machine and both disks get UNC sectors at once?

Alternatives in roughly decreasing order of risk:

	1.  Disable write caching on both Blacks in the pair

	2.  Replace both Blacks with drives in the 0-failure list

	3.  Replace one Black with a Seagate Firecuda or WD Red Pro
	(any other 0-failure drive will do, but these have similar
	performance specs to Black) to ensure firmware diversity

	4.  Find some Black drives with different firmware that have UNC
	sectors and see what happens with write caching during sector
	remap events:  if they behave well, enable write caching on
	all drives with matching firmware, disable if not

	5.  Leave write caching on for now, but as soon as any Black
	reports UNC sectors or reallocation events in SMART data, turn
	write caching off for the remainder of the drive's service life.

> There is a bench test that will demonstrate the transid verify self-repair
> procedure: disconnect one half of a RAID1 array, write for a while, then
> reconnect and do a scrub.  btrfs should self-repair all the metadata on
> the disconnected drive until it all matches the connected one.  Some of
> the data blocks might be hosed though (due to CRC32 collisions), so
> don't do this test on data you care about.
> 
> > 
> > 



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]