Re: How to handle a RAID5 arrawy with a failing drive?

From: Chris Murphy <lists@colorremedies.com>
To: Marc MERLIN <marc@merlins.org>
Cc: Btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: How to handle a RAID5 arrawy with a failing drive?
Date: Mon, 17 Mar 2014 11:38:27 -0600	[thread overview]
Message-ID: <788CE3D5-5E1F-48D0-A1A9-4E7DD320BED7@colorremedies.com> (raw)
In-Reply-To: <20140317161307.GJ6143@merlins.org>

On Mar 17, 2014, at 10:13 AM, Marc MERLIN <marc@merlins.org> wrote:
> 
> What's confusing is that mounting in degraded mode shows all devices:
> polgara:~# btrfs fi show
> Label: backupcopy  uuid: 7d8e1197-69e4-40d8-8d86-278d275af896
>        Total devices 10 FS bytes used 376.27GiB
>        devid    1 size 465.76GiB used 42.42GiB path /dev/dm-0
>        devid    2 size 465.76GiB used 42.40GiB path /dev/dm-1
>        devid    3 size 465.75GiB used 42.40GiB path /dev/mapper/crypt_sde1 << this is missing
>        devid    4 size 465.76GiB used 42.40GiB path /dev/dm-3
>        devid    5 size 465.76GiB used 42.40GiB path /dev/dm-4
>        devid    6 size 465.76GiB used 42.40GiB path /dev/dm-5
>        devid    7 size 465.76GiB used 42.40GiB path /dev/dm-6
>        devid    8 size 465.76GiB used 42.40GiB path /dev/mapper/crypt_sdj1
>        devid    9 size 465.76GiB used 42.40GiB path /dev/mapper/crypt_sdk1
>        devid    10 size 465.76GiB used 42.40GiB path /dev/dm-8

/dev/mapper/crypt_sde1 is completely unavailable, as in not listed by lsblk? If it's not connected and not listed by lsblk yet it's listed by btrfs fi show that's a bug.

> 
> eventually it turned into:
> BTRFS: bdev /dev/mapper/crypt_sde1 errs: wr 3891, rd 321927996, flush 0, corrupt 0, gen 0
> BTRFS: bdev /dev/mapper/crypt_sde1 errs: wr 3891, rd 321927997, flush 0, corrupt 0, gen 0
[snip]
> BTRFS: error (device dm-9) in __btrfs_free_extent:5755: errno=-5 IO failure
> BTRFS info (device dm-9): forced readonly
> BTRFS: error (device dm-9) in btrfs_run_delayed_refs:2713: errno=-5 IO failure

I think it's a lost cause at this point. Your setup is substantially more complicated than my simple setup, and I can't even get the simple setup to recover from an idealized single device raid5 failure. The only apparent way out is to mount degraded, backup, and then start over.

In your case it looks like at least two devices are reporting, or Btrfs thinks they're reporting, I/O errors. Whether this is the physical drive itself, or if it's some other layer (it looks like these are dmcrypt logical block devices).

Chris Murphy