RE: Problems balancing BTRFS

[devel@roosoft.ltd.uk]

So been discussing this on IRC but looks like more sage advice is needed.


So quick history. A BTRFS filesystem was initially created with 2x 6Tb
and 2x1Tb drives. This was configure in a RAID 1 Meta/system   and RAID
5 data configuration

Recently 2 more 6Tb drives were used to replace the 2 1Tb ones. One was
Added to the filesystem and then the original 1Tb was deleted, the other
was just a direct replace.


The filesystem was then expanded to fill the new space with


>    btrfs fi resize 4:max /mnt/media/


The filesystem was very unbalanced and currently looks like this after
an attempt to rebalance it failed.


btrfs fi show
Label: none  uuid: 6abaa68a-2670-4d8b-8d2a-fd7321df9242
    Total devices 4 FS bytes used 2.80TiB
    devid    1 size 5.46TiB used 1.20TiB path /dev/sdb
    devid    2 size 5.46TiB used 1.20TiB path /dev/sdc
    devid    4 size 5.46TiB used 826.03GiB path /dev/sde
    devid    5 size 5.46TiB used 826.03GiB path /dev/sdd


btrfs fi usage  /mnt/media/
WARNING: RAID56 detected, not implemented
Overall:
    Device size:          21.83TiB
    Device allocated:           8.06GiB
    Device unallocated:          21.82TiB
    Device missing:             0.00B
    Used:               6.26GiB
    Free (estimated):             0.00B    (min: 8.00EiB)
    Data ratio:                  0.00
    Metadata ratio:              2.00
    Global reserve:         512.00MiB    (used: 0.00B)

Data,RAID5: Size:2.80TiB, Used:2.80TiB
   /dev/sdb       1.20TiB
   /dev/sdc       1.20TiB
   /dev/sdd     822.00GiB
   /dev/sde     822.00GiB

Metadata,RAID1: Size:4.00GiB, Used:3.13GiB
   /dev/sdd       4.00GiB
   /dev/sde       4.00GiB

System,RAID1: Size:32.00MiB, Used:256.00KiB
   /dev/sdd      32.00MiB
   /dev/sde      32.00MiB

Unallocated:
   /dev/sdb       4.26TiB
   /dev/sdc       4.26TiB
   /dev/sdd       4.65TiB
   /dev/sde       4.65TiB


A scrub is clean

btrfs scrub status /mnt/media/
UUID:             6abaa68a-2670-4d8b-8d2a-fd7321df9242
Scrub started:    Thu Nov 21 11:30:49 2019
Status:           finished
Duration:         17:20:11
Total to scrub:   2.80TiB
Rate:             47.10MiB/s
Error summary:    no errors found


A readonly fs check is clean


Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/sdb
UUID: 6abaa68a-2670-4d8b-8d2a-fd7321df9242
[1/7] checking root items                      (0:00:13 elapsed, 373111
items checked)
[2/7] checking extents                         (0:04:18 elapsed, 205334
items checked)
[3/7] checking free space cache                (0:00:37 elapsed, 1233
items checked)
[4/7] checking fs roots                        (0:00:10 elapsed, 10714
items checked)
[5/7] checking csums (without verifying data)  (0:00:02 elapsed, 414138
items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 90
items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 3079343529984 bytes used, no error found
total csum bytes: 3003340776
total tree bytes: 3362635776
total fs tree bytes: 177717248
total extent tree bytes: 35635200
btree space waste bytes: 153780830
file data blocks allocated: 3077709344768
 referenced 3077349277696



A full balance is now failing



[Fri Nov 22 11:31:27 2019] BTRFS info (device sdb): relocating block
group 8808400289792 flags data|raid5
[Fri Nov 22 11:32:07 2019] BTRFS info (device sdb): found 74 extents
[Fri Nov 22 11:32:24 2019] BTRFS info (device sdb): found 74 extents
[Fri Nov 22 11:32:43 2019] BTRFS info (device sdb): relocating block
group 8805179064320 flags data|raid5
[Fri Nov 22 11:33:24 2019] BTRFS info (device sdb): found 61 extents
[Fri Nov 22 11:33:44 2019] BTRFS info (device sdb): found 61 extents
[Fri Nov 22 11:33:52 2019] BTRFS info (device sdb): relocating block
group 8801957838848 flags data|raid5
[Fri Nov 22 11:33:54 2019] BTRFS warning (device sdb): csum failed root
-9 ino 307 off 131760128 csum 0x07436c62 expected csum 0x0001cbde mirror 1
[Fri Nov 22 11:33:54 2019] BTRFS warning (device sdb): csum failed root
-9 ino 307 off 131764224 csum 0xd009e874 expected csum 0x00000000 mirror 1
[Fri Nov 22 11:33:54 2019] BTRFS warning (device sdb): csum failed root
-9 ino 307 off 131760128 csum 0x07436c62 expected csum 0x0001cbde mirror 2
[Fri Nov 22 11:33:54 2019] BTRFS warning (device sdb): csum failed root
-9 ino 307 off 131764224 csum 0xd009e874 expected csum 0x00000000 mirror 2
[Fri Nov 22 11:33:54 2019] BTRFS warning (device sdb): csum failed root
-9 ino 307 off 131760128 csum 0x07436c62 expected csum 0x0001cbde mirror 1
[Fri Nov 22 11:33:54 2019] BTRFS warning (device sdb): csum failed root
-9 ino 307 off 131760128 csum 0x07436c62 expected csum 0x0001cbde mirror 2
[Fri Nov 22 11:34:02 2019] BTRFS info (device sdb): balance: ended with
status: -5


Any idea how to proceed from here? The drives are relatively new and
there were no issues in the 2x6Tb+ 2x1Tb


Thanks in advance.


-- 
==

Don Alex.