RAID6 Profile Conversion "error (device sda: state A) in __btrfs_free_extent"

RAID6 Profile Conversion "error (device sda: state A) in __btrfs_free_extent"

[Norbert Morawski]

Hello People,
this is my first mail to a mailing list, so please give me constructive feedback.
I am also not a native English speaker, so please excuse my mistakes.

I have started a BTRFS profile conversion from array to "RAID6 for Data and RAID1C3 for Metadata".

Steps I did:

1. Created an BTRFS Array 6 x 8TB "This happened sequentially since I had to move the Data from a LVM"
2. Filled the Storage to 24TB out of 44TB
3. Proceeded with the RAID6 conversion -> "btrfs balance start -dconvert=raid6 -mconvert=raid1c3 /mnt/Data --background"
4. About 40% into the balance, the METADATA is full and does not extend.
5. BTRFS goes into forced Read-only mode.
6. Mounted with "skip_balance"
7. Added disk "/dev/nvme1n1" into the Array.
8. Started "btrfs scrub"
9. Resumed BTFS Balance -> Metadata extended once from 16GB to 17GB at 99.7% usage. -> Second Extend does not happen.
10. BTRFS goes into forced Read-only mode.

This is the Kernel-log:
'''
[114905.372197] BTRFS: error (device sda: state A) in find_free_extent_update_loop:4129: errno=-22 unknown
[114905.372205] BTRFS info (device sda: state EA): forced readonly
[114905.372211] BTRFS: error (device sda: state EA) in reset_balance_state:3595: errno=-22 unknown
[114905.372221] BTRFS info (device sda: state EA): balance: canceled
'''

After some troubleshooting, I mounted the FS with the option "skip_balance".
This pauses the balance process and allowed me to remove the least important files and also start the "btrfs scrub" process.

After that and another balance attempt, I managed to get more Kernel Info, and it appears that the Metadata does not want to extend.
'''
[133728.876517] BTRFS info (device sda: state A): dumping space info:
[133728.876520] BTRFS info (device sda: state A): space_info DATA has 5050423623680 free, is not full
[133728.876522] BTRFS info (device sda: state A): space_info total=30382508408832, used=25064527589376, pinned=0, reserved=0, may_use=0, readonly=267557195776 zone_unusable=0
[133728.876526] BTRFS info (device sda: state A): space_info METADATA has 3459383296 free, is full
[133728.876528] BTRFS info (device sda: state A): space_info total=32212254720, used=28193619968, pinned=21725184, reserved=32768, may_use=537165824, readonly=327680 zone_unusable=0
[133728.876532] BTRFS info (device sda: state A): space_info SYSTEM has 39206912 free, is not full
[133728.876534] BTRFS info (device sda: state A): space_info total=41943040, used=2736128, pinned=0, reserved=0, may_use=0, readonly=0 zone_unusable=0
[133728.876537] BTRFS info (device sda: state A): global_block_rsv: size 536870912 reserved 536854528
[133728.876539] BTRFS info (device sda: state A): trans_block_rsv: size 0 reserved 0
[133728.876540] BTRFS info (device sda: state A): chunk_block_rsv: size 0 reserved 0
[133728.876541] BTRFS info (device sda: state A): delayed_block_rsv: size 0 reserved 0
[133728.876542] BTRFS info (device sda: state A): delayed_refs_rsv: size 16777216 reserved 16384
[133728.876544] BTRFS: error (device sda: state A) in __btrfs_free_extent:3077: errno=-28 No space left
[133728.876550] BTRFS info (device sda: state EA): forced readonly
[133728.876552] BTRFS error (device sda: state EA): failed to run delayed ref for logical 54198081880064 num_bytes 16384 type 176 action 2 ref_mod 1: -28
[133728.876555] BTRFS: error (device sda: state EA) in btrfs_run_delayed_refs:2151: errno=-28 No space left
[133728.876560] BTRFS warning (device sda: state EA): Skipping commit of aborted transaction.
[133728.876561] BTRFS: error (device sda: state EA) in cleanup_transaction:1984: errno=-28 No space left
[133728.879119] BTRFS info (device sda: state EA): balance: ended with status: -30
'''
*Canceling the balance is not possible, FS goes into forced read-only mode.*

## Information
OS: Proxmox 8.0
Kernel: 6.2.16-3-pve
Btrfs: btrfs-progs v6.2

btrfs fi show:
'''
Label: 'Backup'  uuid: 450ead52-9143-45d7-9326-4cba501700ed
        Total devices 1 FS bytes used 313.18GiB
        devid    1 size 465.76GiB used 341.03GiB path /dev/sdh

Label: 'Data'  uuid: c2c74dd4-a471-4ac5-9321-597fb39ba992
        Total devices 7 FS bytes used 22.82TiB
        devid    1 size 7.28TiB used 5.16TiB path /dev/sda
        devid    2 size 7.28TiB used 5.47TiB path /dev/sdb
        devid    3 size 7.28TiB used 6.48TiB path /dev/sde
        devid    4 size 7.28TiB used 6.49TiB path /dev/sdd
        devid    5 size 7.28TiB used 6.49TiB path /dev/sdc
        devid    6 size 7.28TiB used 6.49TiB path /dev/sdf
        devid    7 size 953.87GiB used 110.00GiB path /dev/nvme1n1
'''
btrfs fi df /mnt/Data

'''

Data, single: total=8.65TiB, used=8.40TiB
Data, RAID6: total=15.02TiB, used=14.39TiB
System, DUP: total=8.00MiB, used=896.00KiB
System, RAID1C3: total=32.00MiB, used=1.48MiB
Metadata, DUP: total=13.00GiB, used=9.28GiB
Metadata, RAID1C3: total=17.00GiB, used=16.98GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
WARNING: Multiple block group profiles detected, see 'man btrfs(5)'
WARNING:    Data: single, raid6
WARNING:    Metadata: dup, raid1c3
WARNING:    System: dup, raid1c3

'''

I hope this helps further improving BTRFS, and maybe we even find a solution for me.

Best Regards
Norbert Morawski