Re: 'btrfs rescue' command (recommended by btrfs check) fails on old BTRFS RAID1 on (currently) openSUSE Leap 15.3

[Johannes Kastl <kastl@b1-systems.de>]

[-- Attachment #1.1: Type: text/plain, Size: 3668 bytes --]

Hi Qu,

TL;DR: took a while until I had all of the data backed up properly and had some 
time to test this. Unfortunately the filesystem is now no longer mountable...

Any ideas?

Johannes

On 24.04.22 at 11:21 Qu Wenruo wrote:
> On 2022/4/24 17:10, Johannes Kastl wrote:

>> So would resizing the filesystem (to 8GiB) workaround this "limitation",
>> so afterwards it could properly fix the device size?
> 
> I'm not yet sure if it's a bug in progs causing false ENOSPC, or really
> there isn't many space left.
> 
> For the former case, no matter how much free space you have, it won't help.
> 
> For the latter case, it would definitely help.

So, I deleted the partitions on both disks and re-created them with the new 
(bigger size), keeping the start sector and the btrfs signature intact.

I could then resize both disks to the same value successfully. At least, the 
commands ran without errors.

Fixing the device size fails nonetheless (see below). And I can no longer mount 
the filesystem, when I try I find this in the logs:

> [87396.889043] BTRFS error (device sdb1): super_total_bytes 15393162784768 mismatch with fs_devices total_rw_bytes 15393162788864
> [87396.889974] BTRFS error (device sdb1): failed to read chunk tree: -22
> [87396.892741] BTRFS error (device sdb1): open_ctree failed

(Don't get confused by sdb1, this is from a rescue system with only some HDDs 
attached)

Fixing the device-size on Leap 15.3:
> # btrfs filesystem show /mnt/DUMBO_BACKUP_4TB/
> Label: 'DUMBO_BACKUP_4TB'  uuid: 50651b41-bf33-47e7-8a08-afbc71ba0bf8
>         Total devices 2 FS bytes used 3.17TiB
>         devid    1 size 7.00TiB used 3.64TiB path /dev/sdd1
>         devid    2 size 7.00TiB used 3.63TiB path /dev/sdc1
> 
> # umount /mnt/DUMBO_BACKUP_4TB
> # btrfs rescue fix-device-size /dev/sdd1
> Unable to find block group for 0
> Unable to find block group for 0
> Unable to find block group for 0
> transaction.c:189: btrfs_commit_transaction: BUG_ON `ret` triggered, value -28
> btrfs(+0x51f99)[0x55edf7a43f99]
> btrfs(+0x525a9)[0x55edf7a445a9]
> btrfs(btrfs_fix_super_size+0x98)[0x55edf7a2f438]
> btrfs(btrfs_fix_device_and_super_size+0x84)[0x55edf7a2f584]
> btrfs(+0x6ceee)[0x55edf7a5eeee]
> btrfs(main+0x8e)[0x55edf7a1108e]
> /lib64/libc.so.6(__libc_start_main+0xef)[0x7f672ad962bd]
> btrfs(_start+0x2a)[0x55edf7a1128a]
> Aborted (core dumped)
> # 

I tested fixing the device-id by booting from a Tumbleweed rescue stick, running 
kernel 5.16 with btrfsprogs 5.16. This also fails, but spits out an error 
message that is a little different:

 > [...]
> Unable to find block group for 0
> Error: failed to commit current transaction: -28 (No space left on device)
> No device size related problem found
> ERROR: commit_root already set when starting transaction
> extent buffer leak: start ... len 16384

(I had to type this off of the screen)

As the mounting failed with an error related to chunks, I tried the btrfs rescue 
chunk-recover command, but that also aborts and dumps a core, even on Tumbleweed 
with kernel 5.16...

The error messages look something like this:
 > Unable to find block group for 0
 > Unable to find block group for 0
 > Unable to find block group for 0

followed by a "...BUG_ON `ret` triggered, value -28"

So this could all be related to -28 (No space left on device)?

-- 
Johannes Kastl
Linux Consultant & Trainer
Tel.: +49 (0) 151 2372 5802
Mail: kastl@b1-systems.de

B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg
http://www.b1-systems.de
GF: Ralph Dehner
Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]