btrfs mirror fails to mount - corrupt leaf

btrfs mirror fails to mount - corrupt leaf

[Craig Silva]

This happened as the result of a power outage (someone knocked the plug on a power block)

The NAS is openmedivault - btrfs-progs v4.20.1 

btrfs filesystem show
Label: none  uuid: 34f2c6cc-2665-4b9b-9503-356c83066d33
Total devices 2 FS bytes used 864.48GiB
devid    1 size 3.18TiB used 2.02TiB path /dev/sda1
devid    2 size 3.18TiB used 2.02TiB path /dev/sdb1

blkid
/dev/sdb1: UUID="34f2c6cc-2665-4b9b-9503-356c83066d33" UUID_SUB="5c40e61e-04d0-4025-8c97-be4a12da719a" TYPE="btrfs" PARTLABEL="primary" PARTUUID="c0bde629-1d0c-44d9-a48a-6f4657627428"
/dev/sda1: UUID="34f2c6cc-2665-4b9b-9503-356c83066d33" UUID_SUB="05db5d7d-92e9-46d3-a7bf-655af00ccb85" TYPE="btrfs" PARTLABEL="primary" PARTUUID="3a5b7725-0be5-4935-89ae-2b67813cbc07"

btrfs-find-root /dev/sda1
incorrect offsets 5447 33559879
Superblock thinks the generation is 1443155
Superblock thinks the level is 1

All supers are valid

dmesg:

BTRFS critical (device sda1): corrupt leaf: root=2 block=546514255872 slot=205, unexpected item end, have 33559879 expect 5447
BTRFS critical (device sda1): corrupt leaf: root=2 block=546514255872 slot=205, unexpected item end, have 33559879 expect 5447
BTRFS error (device sda1): failed to read block groups: -5
BTRFS error (device sda1): open_ctree failed
BTRFS info (device sda1): disk space caching is enabled
BTRFS info (device sda1): has skinny extents
BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0
BTRFS info (device sda1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0

I checked hardware both sda and sdb with smartctl and no apparent errors.

I'm in the process of a restore (ho-hum) - but it would be nice if there was a way of fixing it

Re: btrfs mirror fails to mount - corrupt leaf

[Qu Wenruo]

On 2023/4/3 14:49, Craig Silva wrote:
> btrfs filesystem show
> Label: none  uuid: 34f2c6cc-2665-4b9b-9503-356c83066d33
> Total devices 2 FS bytes used 864.48GiB
> devid    1 size 3.18TiB used 2.02TiB path /dev/sda1
> devid    2 size 3.18TiB used 2.02TiB path /dev/sdb1
> 
> blkid
> /dev/sdb1: UUID="34f2c6cc-2665-4b9b-9503-356c83066d33" UUID_SUB="5c40e61e-04d0-4025-8c97-be4a12da719a" TYPE="btrfs" PARTLABEL="primary" PARTUUID="c0bde629-1d0c-44d9-a48a-6f4657627428"
> /dev/sda1: UUID="34f2c6cc-2665-4b9b-9503-356c83066d33" UUID_SUB="05db5d7d-92e9-46d3-a7bf-655af00ccb85" TYPE="btrfs" PARTLABEL="primary" PARTUUID="3a5b7725-0be5-4935-89ae-2b67813cbc07"
> 
> btrfs-find-root /dev/sda1
> incorrect offsets 5447 33559879
> Superblock thinks the generation is 1443155
> Superblock thinks the level is 1
> 
> All supers are valid
> 
> dmesg:
> 
> BTRFS critical (device sda1): corrupt leaf: root=2 block=546514255872 slot=205, unexpected item end, have 33559879 expect 5447
> BTRFS critical (device sda1): corrupt leaf: root=2 block=546514255872 slot=205, unexpected item end, have 33559879 expect 5447

hex(33559879) = 0x2001547
hex(    5447) = 0x0001547

So an obvious bitflip.

Please run memtest to make sure your hardware memory is working correctly.

With the hardware fixed, you need to use newer enough (your kernel 
doesn't seem new enough, as such corruption can be prevented since 
v5.11) and mount with "-o rescue=all" to backup your important data first.

Then with newer enough btrfs-progs, you may want to try "btrfs check 
--init-extent-tree", which can be dangerous, but can at least rebuild 
extent tree.

Thanks,
Qu

> BTRFS error (device sda1): failed to read block groups: -5
> BTRFS error (device sda1): open_ctree failed
> BTRFS info (device sda1): disk space caching is enabled
> BTRFS info (device sda1): has skinny extents
> BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0
> BTRFS info (device sda1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0
> 
> I checked hardware both sda and sdb with smartctl and no apparent errors.
> 
> I'm in the process of a restore (ho-hum) - but it would be nice if there was a way of fixing it

btrfs mirror fails to mount - corrupt leaf

[Craig Silva]

btrfs filesystem show
Label: none  uuid: 34f2c6cc-2665-4b9b-9503-356c83066d33
Total devices 2 FS bytes used 864.48GiB
devid    1 size 3.18TiB used 2.02TiB path /dev/sda1
devid    2 size 3.18TiB used 2.02TiB path /dev/sdb1

blkid
/dev/sdb1: UUID="34f2c6cc-2665-4b9b-9503-356c83066d33" UUID_SUB="5c40e61e-04d0-4025-8c97-be4a12da719a" TYPE="btrfs" PARTLABEL="primary" PARTUUID="c0bde629-1d0c-44d9-a48a-6f4657627428"
/dev/sda1: UUID="34f2c6cc-2665-4b9b-9503-356c83066d33" UUID_SUB="05db5d7d-92e9-46d3-a7bf-655af00ccb85" TYPE="btrfs" PARTLABEL="primary" PARTUUID="3a5b7725-0be5-4935-89ae-2b67813cbc07"

btrfs-find-root /dev/sda1
incorrect offsets 5447 33559879
Superblock thinks the generation is 1443155
Superblock thinks the level is 1

All supers are valid

dmesg:

BTRFS critical (device sda1): corrupt leaf: root=2 block=546514255872 slot=205, unexpected item end, have 33559879 expect 5447
BTRFS critical (device sda1): corrupt leaf: root=2 block=546514255872 slot=205, unexpected item end, have 33559879 expect 5447
BTRFS error (device sda1): failed to read block groups: -5
BTRFS error (device sda1): open_ctree failed
BTRFS info (device sda1): disk space caching is enabled
BTRFS info (device sda1): has skinny extents
BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0
BTRFS info (device sda1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0

I checked hardware both sda and sdb with smartctl and no apparent errors.

I'm in the process of a restore (ho-hum) - but it would be nice if there was a way of fixing it