Failed Disk RAID10 Problems

Failed Disk RAID10 Problems

[Justin Brown]

Hi,

I have a Btrfs RAID 10 (data and metadata) file system that I believe
suffered a disk failure. In my attempt to replace the disk, I think
that I've made the problem worse and need some help recovering it.

I happened to notice a lot of errors in the journal:

end_request: I/O error, dev dm-11, sector 1549378344
BTRFS: bdev /dev/mapper/Hitachi_HDS721010KLA330_GTA040PBG71HXF1 errs:
wr 759675, rd 539730, flush 23, corrupt 0, gen 0

The file system continued to work for some time, but eventually a NFS
client encountered IO errors. I figured that device was failing (It
was very old.). I attached a new drive to the hot-swappable SATA slot
on my computer, partitioned it with GPT, and ran partprobe to detect
it. Next I attempted to add a new device, which was successful.
However, something peculiar happened:

~: btrfs fi df /var/media/
Data, RAID10: total=2.33TiB, used=2.33TiB
Data, RAID6: total=72.00GiB, used=71.96GiB
System, RAID10: total=96.00MiB, used=272.00KiB
Metadata, RAID10: total=4.12GiB, used=2.60GiB

I don't know where that RAID6 file system came from, but it did not
exist over the weekend when I last checked. I attempted to run a
balance operation, but this is when the IO errors became severe, and I
cancelled it. Next, I tried to remove the failed device, thinking that
Btrfs could rebalance after that. Removing the failed device failed:

~: btrfs device delete /dev/dm-11 /var/media
ERROR: error removing the device '/dev/dm-11' - Device or resource busy

I shutdown the system and detached the failed disk. Upon reboot, I
cannot mount the filesystem:

~: mount /dev/mapper/SAMSUNG_HD103SI_499431FS734755p1 /var/media
mount: wrong fs type, bad option, bad superblock on
/dev/mapper/SAMSUNG_HD103SI_499431FS734755p1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

BTRFS: device label media devid 2 transid 44804
/dev/mapper/WDC_WD10EACS-00D6B0_WD-WCAU40229179p1
BTRFS info (device dm-10): disk space caching is enabled
BTRFS: failed to read the system array on dm-10
BTRFS: open_ctree failed

I reattached the failed disk, and I'm still getting the same mount
error as above.

Here's where the array currently stands:

Label: 'media'  uuid: 7b7afc82-f77c-44c0-b315-669ebd82f0c5
Total devices 5 FS bytes used 2.39TiB
devid    1 size 931.51GiB used 919.41GiB path
/dev/mapper/SAMSUNG_HD103SI_499431FS734755p1
devid    2 size 931.51GiB used 919.41GiB path
/dev/mapper/WDC_WD10EACS-00D6B0_WD-WCAU40229179p1
devid    3 size 1.82TiB used 1.19TiB path
/dev/mapper/WDC_WD20EFRX-68AX9N0_WD-WMC1T1268493p1
devid    4 size 931.51GiB used 920.41GiB path
/dev/mapper/WDC_WD10EARS-00Y5B1_WD-WMAV50654875p1
devid    5 size 931.51GiB used 918.50GiB path
/dev/mapper/Hitachi_HDS721010KLA330_GTA040PBG71HXF1
devid    6 size 1.82TiB used 3.41GiB path
/dev/mapper/WDC_WD20EFRX-68AX9N0_WD-WMC300239240p1

Btrfs v3.12

Devid 6 is the drive that I added earlier.

What can I do to recover this file system? I have another spare drive
that I can use if it's any help.

Thanks,
Justin

Re: Failed Disk RAID10 Problems

[Chris Murphy]

On May 28, 2014, at 12:19 AM, Justin Brown <justin.brown@fandingo.org> wrote:

> Hi,
> 
> I have a Btrfs RAID 10 (data and metadata) file system that I believe
> suffered a disk failure. In my attempt to replace the disk, I think
> that I've made the problem worse and need some help recovering it.
> 
> I happened to notice a lot of errors in the journal:
> 
> end_request: I/O error, dev dm-11, sector 1549378344
> BTRFS: bdev /dev/mapper/Hitachi_HDS721010KLA330_GTA040PBG71HXF1 errs:
> wr 759675, rd 539730, flush 23, corrupt 0, gen 0
> 
> The file system continued to work for some time, but eventually a NFS
> client encountered IO errors. I figured that device was failing (It
> was very old.). I attached a new drive to the hot-swappable SATA slot
> on my computer, partitioned it with GPT, and ran partprobe to detect
> it. Next I attempted to add a new device, which was successful.

For future reference, it should to add a device and then use btrfs device delete missing. But I've found btrfs replace start to be more reliable. It does the add, delete and balance in one step.


> ~: mount /dev/mapper/SAMSUNG_HD103SI_499431FS734755p1 /var/media
> mount: wrong fs type, bad option, bad superblock on
> /dev/mapper/SAMSUNG_HD103SI_499431FS734755p1,
>       missing codepage or helper program, or other error
> 
>       In some cases useful info is found in syslog - try
>       dmesg | tail or so.
> 
> BTRFS: device label media devid 2 transid 44804
> /dev/mapper/WDC_WD10EACS-00D6B0_WD-WCAU40229179p1
> BTRFS info (device dm-10): disk space caching is enabled
> BTRFS: failed to read the system array on dm-10
> BTRFS: open_ctree failed

I'd try in order:

mount -o degraded,ro
mount -o recovery,ro
mount -o degraded,recovery,ro

If any of those works, then update your backup before trying anything else. Whatever command above worked, try it without ro.

If a degrade option is needed then that makes me think a btrfs device delete missing won't work, but then I'm also not seeing a missing device in your btrfs fi show either. You definitely need to make sure the device producing the errors is the device that's missing and is the one you're removing.

Chris Murphy

Re: Failed Disk RAID10 Problems

[Chris Murphy]

On May 28, 2014, at 1:03 AM, Chris Murphy <lists@colorremedies.com> wrote:
> 
> For future reference, it should to add a device and then use btrfs device delete missing.

it should work (if not it's probably a bug).

Chris Murphy

Fwd: Failed Disk RAID10 Problems

[Justin Brown]

Chris,

Thanks for the tip. I was able to mount the drive as degraded and
recovery. Then, I deleted the faulty drive, leaving me with the
following array:


Label: media  uuid: 7b7afc82-f77c-44c0-b315-669ebd82f0c5

Total devices 6 FS bytes used 2.40TiB

devid    1 size 931.51GiB used 919.88GiB path
/dev/mapper/SAMSUNG_HD103SI_499431FS734755p1

devid    2 size 931.51GiB used 919.38GiB path /dev/dm-8

devid    3 size 1.82TiB used 1.19TiB path /dev/dm-6

devid    4 size 931.51GiB used 919.88GiB path /dev/dm-5

devid    5 size 0.00 used 918.38GiB path /dev/dm-11

devid    6 size 1.82TiB used 3.88GiB path /dev/dm-9


/dev/dm-11 is the failed drive. I take it that size 0 is a good sign.
I'm not really sure where to go from here. I tried rebooting the
system with the failed drive attached, and Btrfs re-adds it to the
array. Should I physically remove the drive now? Is a balance
recommended?


Thanks,

Justin

Re: Failed Disk RAID10 Problems

[Chris Murphy]

On May 28, 2014, at 12:39 PM, Justin Brown <justin.brown@fandingo.org> wrote:

> Chris,
> 
> Thanks for the tip. I was able to mount the drive as degraded and
> recovery. Then, I deleted the faulty drive, leaving me with the
> following array:
> 
> 
> Label: media  uuid: 7b7afc82-f77c-44c0-b315-669ebd82f0c5
> 
> Total devices 6 FS bytes used 2.40TiB
> 
> devid    1 size 931.51GiB used 919.88GiB path
> /dev/mapper/SAMSUNG_HD103SI_499431FS734755p1
> 
> devid    2 size 931.51GiB used 919.38GiB path /dev/dm-8
> 
> devid    3 size 1.82TiB used 1.19TiB path /dev/dm-6
> 
> devid    4 size 931.51GiB used 919.88GiB path /dev/dm-5
> 
> devid    5 size 0.00 used 918.38GiB path /dev/dm-11
> 
> devid    6 size 1.82TiB used 3.88GiB path /dev/dm-9
> 
> 
> /dev/dm-11 is the failed drive.

You deleted a faulty drive, dm-11 is a failed drive. Is there a difference between faulty drive and failed drive, or are they the same drive? And what drive is the one you said you successfully added?

I don't see how you have 6 devices raid10, with one failed and one added device. You need an even number of good drives to fix this.


> I take it that size 0 is a good sign.

Seems neither good nor bad to me, it's 0 because it's a dead drive presumably and therefore Btrfs isn't getting device information from it.

> I'm not really sure where to go from here. I tried rebooting the
> system with the failed drive attached, and Btrfs re-adds it to the
> array. Should I physically remove the drive now? Is a balance
> recommended?

No don't do anything else until someone actually understands faulty vs failed vs added drives.


Chris Murphy

Re: Failed Disk RAID10 Problems

[Chris Murphy]

On May 28, 2014, at 12:39 PM, Justin Brown <justin.brown@fandingo.org> wrote:

> Chris,
> 
> Thanks for the tip. I was able to mount the drive as degraded and
> recovery. Then, I deleted the faulty drive, leaving me with the
> following array:
> 
> 
> Label: media  uuid: 7b7afc82-f77c-44c0-b315-669ebd82f0c5
> 
> Total devices 6 FS bytes used 2.40TiB
> 
> devid    1 size 931.51GiB used 919.88GiB path
> /dev/mapper/SAMSUNG_HD103SI_499431FS734755p1
> 
> devid    2 size 931.51GiB used 919.38GiB path /dev/dm-8
> 
> devid    3 size 1.82TiB used 1.19TiB path /dev/dm-6
> 
> devid    4 size 931.51GiB used 919.88GiB path /dev/dm-5
> 
> devid    5 size 0.00 used 918.38GiB path /dev/dm-11
> 
> devid    6 size 1.82TiB used 3.88GiB path /dev/dm-9
> 
> 
> /dev/dm-11 is the failed drive. I take it that size 0 is a good sign.
> I'm not really sure where to go from here. I tried rebooting the
> system with the failed drive attached, and Btrfs re-adds it to the
> array. Should I physically remove the drive now? Is a balance
> recommended?

I'm going to guess at what I think has happened. You had a 5 device raid10. devid 5 is the failed device, but at the time you added new device devid 6, it was not considered failed by btrfs. Your first btrfs fi show does not show size 0 for devid 5. So I think btrfs made you a 6 device raid10 volume.

But now devid 5 has failed, shows up as size 0. The reason you have to mount degraded still is because you have a 6 device raid10 now, and 1 device has failed. And you can't remove the failed device because you've mounted degraded. So actually it was a mistake to add a new device first, but it's an easy mistake to make because right now btrfs really tolerates a lot of error conditions that it probably should give up on and outright fail the device.

So I think you might have to get a 7th device to fix this with btrfs replace start. You can later delete devices once you're not mounted degraded. Or you can just do a backup now while you can mount degraded, and then blow away the btrfs volume and start over.

If you have a current backups and are willing to lose data on this volume, you can try the following

1. Poweroff, remove the failed drive, boot, and do a normal mount. That probably won't work but it's worth a shot. If it doesn't work try mount -o degraded. [That might not work either, in which case stop here, I think you'll need to go with a 7th device and use 'btrfs replace start 5 /dev/newdevice7 /mp' That will explicitly replace failed device 5 with new device.]

2. Assuming mount -o degraded works, take a btrfs fi show. There should be a missing device listed. Now try btrfs device delete missing /mp and see what happens. If it at least doesn't complain, it means it's working and might take hours to replicate data that was on the missing device onto the new one. So I'd leave it alone until iotop or something like that tells you it's not busy anymore.

3. Unmount the file system. Try to mount normally (not degraded).



Chris Murphy

Re: Failed Disk RAID10 Problems

[Justin Brown]

Chris,

Thanks for the continued help. I had to put the recovery on hiatus
while I waited for new hard drives to be delivered. I never was able
to figure out how to replace the failed drive, but I did learn a lot
about how Btrfs works. The approach to doing practically all
operations with file system mounted specially was quite a surprise.

In the end, I created a Btrfs RAID5 file system with the newly
delivered drives on another system and used rsync to copy from the
degraded array. There was a little file system damage that showed up
as "csum failed" errors in the logs from the IO that was in progress
when the original failure occurred. Fortunately, it was all data that
could be recovered from other systems, and there wasn't any need to
troubleshoot the errors.

Thanks,
Justin


On Wed, May 28, 2014 at 3:40 PM, Chris Murphy <lists@colorremedies.com> wrote:
>
> On May 28, 2014, at 12:39 PM, Justin Brown <justin.brown@fandingo.org> wrote:
>
>> Chris,
>>
>> Thanks for the tip. I was able to mount the drive as degraded and
>> recovery. Then, I deleted the faulty drive, leaving me with the
>> following array:
>>
>>
>> Label: media  uuid: 7b7afc82-f77c-44c0-b315-669ebd82f0c5
>>
>> Total devices 6 FS bytes used 2.40TiB
>>
>> devid    1 size 931.51GiB used 919.88GiB path
>> /dev/mapper/SAMSUNG_HD103SI_499431FS734755p1
>>
>> devid    2 size 931.51GiB used 919.38GiB path /dev/dm-8
>>
>> devid    3 size 1.82TiB used 1.19TiB path /dev/dm-6
>>
>> devid    4 size 931.51GiB used 919.88GiB path /dev/dm-5
>>
>> devid    5 size 0.00 used 918.38GiB path /dev/dm-11
>>
>> devid    6 size 1.82TiB used 3.88GiB path /dev/dm-9
>>
>>
>> /dev/dm-11 is the failed drive. I take it that size 0 is a good sign.
>> I'm not really sure where to go from here. I tried rebooting the
>> system with the failed drive attached, and Btrfs re-adds it to the
>> array. Should I physically remove the drive now? Is a balance
>> recommended?
>
> I'm going to guess at what I think has happened. You had a 5 device raid10. devid 5 is the failed device, but at the time you added new device devid 6, it was not considered failed by btrfs. Your first btrfs fi show does not show size 0 for devid 5. So I think btrfs made you a 6 device raid10 volume.
>
> But now devid 5 has failed, shows up as size 0. The reason you have to mount degraded still is because you have a 6 device raid10 now, and 1 device has failed. And you can't remove the failed device because you've mounted degraded. So actually it was a mistake to add a new device first, but it's an easy mistake to make because right now btrfs really tolerates a lot of error conditions that it probably should give up on and outright fail the device.
>
> So I think you might have to get a 7th device to fix this with btrfs replace start. You can later delete devices once you're not mounted degraded. Or you can just do a backup now while you can mount degraded, and then blow away the btrfs volume and start over.
>
> If you have a current backups and are willing to lose data on this volume, you can try the following
>
> 1. Poweroff, remove the failed drive, boot, and do a normal mount. That probably won't work but it's worth a shot. If it doesn't work try mount -o degraded. [That might not work either, in which case stop here, I think you'll need to go with a 7th device and use 'btrfs replace start 5 /dev/newdevice7 /mp' That will explicitly replace failed device 5 with new device.]
>
> 2. Assuming mount -o degraded works, take a btrfs fi show. There should be a missing device listed. Now try btrfs device delete missing /mp and see what happens. If it at least doesn't complain, it means it's working and might take hours to replicate data that was on the missing device onto the new one. So I'd leave it alone until iotop or something like that tells you it's not busy anymore.
>
> 3. Unmount the file system. Try to mount normally (not degraded).
>
>
>
> Chris Murphy