breathe life into degraded raid10 (no space left on specific device)

breathe life into degraded raid10 (no space left on specific device)

[Mate Gabri]

Dear List,

I tried to remove a device from a 12 disk RAID10 array but it failed with a "no space left" and the system crashed. After a reset i could only mount the array in degraded mode because the device was marked as missing. I've tried a replace command but it said that it does not support RAID5/6 arrays so i went with the device add command which added the disk to the array but there was the missing device message still so i issued the "btrfs device delete missing [path]" command which was running for more than 24 hours without happening much. Also if i try to do some disk io (rsync) the system crashes after 10-15 minutes. 

If i try to mount the array as is i get this:

[   15.649539] btrfs: failed to read chunk tree on sdc
[   15.682209] btrfs: open_ctree failed

Mounting with degraded works. Now if i try to "btrfs device delete missing" i get a no space left error on /dev/sdc. I tried to do a rebalance which worked with the data chunks but with metadata it fails with the no space left error. I've tried to delete some snapshots which happened but after that some metadata rebalance started which failed too with no space left error. I've issued a whole array scrub which complained that dev id 12 is missing. A scrub on /dev/sdc finished successfuly without any error. I'm sure that /dev/sdc is broken metadata wise but i'm afraid to remove that device too. Do you have any suggestions what sould i do?

Here's some info:

uname -a:
Linux backup 3.13-0.bpo.1-amd64 #1 SMP Debian 3.13.10-1~bpo70+1 (2014-04-23) x86_64 GNU/Linux

 btrfs --version
Btrfs v3.14.1

btrfs fi show (/dev/sdl was removed and re-added)
Label: 'backup'  uuid: 667ec955-bcaa-4175-827d-3a44eaa515bb
        Total devices 13 FS bytes used 9.29TiB
        devid    1 size 2.71TiB used 1.56TiB path /dev/sda4
        devid    2 size 2.71TiB used 1.56TiB path /dev/sdb4
        devid    3 size 2.73TiB used 1.56TiB path /dev/sdc
        devid    4 size 2.73TiB used 1.56TiB path /dev/sdd
        devid    5 size 2.73TiB used 1.56TiB path /dev/sde
        devid    6 size 2.73TiB used 1.56TiB path /dev/sdf
        devid    7 size 2.73TiB used 1.56TiB path /dev/sdg
        devid    8 size 2.73TiB used 1.56TiB path /dev/sdh
        devid    9 size 2.73TiB used 1.56TiB path /dev/sdi
        devid   10 size 2.73TiB used 1.56TiB path /dev/sdj
        devid   11 size 2.73TiB used 1.56TiB path /dev/sdk
        devid   13 size 2.73TiB used 1.75GiB path /dev/sdl
        *** Some devices missing

Btrfs v3.14.1


Data, RAID10: total=8.87TiB, used=8.79TiB
System, RAID1: total=32.00MiB, used=1.11MiB
Metadata, RAID10: total=518.72GiB, used=516.31GiB

Best regards,
--
GABRI Mate
Rendszergazda
mgabri@plex.hu | +36 (30) 450 9750

Plex Online Kft. | http://plex.hu
H-1118 Budapest, Ugron Gábor u. 35.
T: +36 (1) 445 0167 | F: +36 (1) 248 3250