[6.7 regression] [BISECTED] 28270e25c69a causes overallocation of metadata space

[6.7 regression] [BISECTED] 28270e25c69a causes overallocation of metadata space

[Ivan Shapovalov]

Hi all,

Starting from v6.7 I've noticed severe overallocation of metadata space
on both of my btrfs root filesystems (both NVMe SSDs, both use
snapshots):

```
# mount | grep -w /
rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=<...>

# btrfs fi usage /
<...>

Data,single: Size:550.00GiB, Used:497.12GiB (90.39%)
   /dev/mapper/root      550.00GiB

Metadata,DUP: Size:72.00GiB, Used:8.38GiB (11.64%)
   /dev/mapper/root      144.00GiB

<...>
```

Running a full metadata balance (`btrfs balance start -m /`) or a
"staged" balance
(e. g. `for u in {0..90..5}; do btrfs balance start -musage=$u /`)
does not help / makes the overallocation worse.

Bisect log:
```
# bad: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
# good: [ffc253263a1375a65fa6c9f62a893e9767fbebfa] Linux 6.6
git bisect start 'v6.7' 'v6.6'
# bad: [deefd5024f0772cf56052ace9a8c347dc70bcaf3] Merge tag 'vfio-v6.7-rc1' of https://github.com/awilliam/linux-vfio
git bisect bad deefd5024f0772cf56052ace9a8c347dc70bcaf3
# bad: [5a6a09e97199d6600d31383055f9d43fbbcbe86f] Merge tag 'cgroup-for-6.7' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect bad 5a6a09e97199d6600d31383055f9d43fbbcbe86f
# good: [17047fbced563cf5abe5aa546f6a92af48900b69] bcachefs: Fix incorrectly freeing btree_path in alloc path
git bisect good 17047fbced563cf5abe5aa546f6a92af48900b69
# good: [b827ac419721a106ae2fccaa40576b0594edad92] exportfs: Change bcachefs fid_type enum to avoid conflicts
git bisect good b827ac419721a106ae2fccaa40576b0594edad92
# bad: [9ab021a1b57007a22761f6f41d91eb4aae10d145] Merge tag 'x86_cache_for_6.7_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 9ab021a1b57007a22761f6f41d91eb4aae10d145
# good: [8b16da681eb0c9b9cb2f9abd0dade67559cfb48d] Merge tag 'nfsd-6.7' of git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux
git bisect good 8b16da681eb0c9b9cb2f9abd0dade67559cfb48d
# bad: [07a274a8862dba86a270ced2a4c5ff3f7a01b66a] btrfs: remove redundant root argument from btrfs_update_inode_item()
git bisect bad 07a274a8862dba86a270ced2a4c5ff3f7a01b66a
# good: [3ee56a58ad8921cb43c49d56347a8e270871844c] btrfs: reserve space for delayed refs on a per ref basis
git bisect good 3ee56a58ad8921cb43c49d56347a8e270871844c
# bad: [9f9918a8017b7925da2fad16b4ccf6a14630f03e] btrfs: sysfs: announce presence of raid-stripe-tree
git bisect bad 9f9918a8017b7925da2fad16b4ccf6a14630f03e
# bad: [87463f7e0250d471fac41e7c9c45ae21d83b5f85] btrfs: zoned: factor out DUP bg handling from btrfs_load_block_group_zone_info
git bisect bad 87463f7e0250d471fac41e7c9c45ae21d83b5f85
# bad: [50564b651d01c19ce732819c5b3c3fd60707188e] btrfs: abort transaction on generation mismatch when marking eb as dirty
git bisect bad 50564b651d01c19ce732819c5b3c3fd60707188e
# bad: [28270e25c69a2c76ea1ed0922095bffb9b9a4f98] btrfs: always reserve space for delayed refs when starting transaction
git bisect bad 28270e25c69a2c76ea1ed0922095bffb9b9a4f98
# good: [adb86dbe426f9a54843d70092819deca220a224d] btrfs: stop doing excessive space reservation for csum deletion
git bisect good adb86dbe426f9a54843d70092819deca220a224d
# first bad commit: [28270e25c69a2c76ea1ed0922095bffb9b9a4f98] btrfs: always reserve space for delayed refs when starting transaction
```

Thanks,
-- 
Ivan Shapovalov / intelfx /

Re: [6.7 regression] [BISECTED] 28270e25c69a causes overallocation of metadata space

[Filipe Manana]

On Wed, Jan 17, 2024 at 6:04 AM Ivan Shapovalov <intelfx@intelfx.name> wrote:
>
> Hi all,
>
> Starting from v6.7 I've noticed severe overallocation of metadata space
> on both of my btrfs root filesystems (both NVMe SSDs, both use
> snapshots):
>
> ```
> # mount | grep -w /
> rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=<...>
>
> # btrfs fi usage /
> <...>
>
> Data,single: Size:550.00GiB, Used:497.12GiB (90.39%)
>    /dev/mapper/root      550.00GiB
>
> Metadata,DUP: Size:72.00GiB, Used:8.38GiB (11.64%)
>    /dev/mapper/root      144.00GiB
>
> <...>
> ```
>
> Running a full metadata balance (`btrfs balance start -m /`) or a
> "staged" balance
> (e. g. `for u in {0..90..5}; do btrfs balance start -musage=$u /`)
> does not help / makes the overallocation worse.
>
> Bisect log:
> ```
> # bad: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
> # good: [ffc253263a1375a65fa6c9f62a893e9767fbebfa] Linux 6.6
> git bisect start 'v6.7' 'v6.6'
> # bad: [deefd5024f0772cf56052ace9a8c347dc70bcaf3] Merge tag 'vfio-v6.7-rc1' of https://github.com/awilliam/linux-vfio
> git bisect bad deefd5024f0772cf56052ace9a8c347dc70bcaf3
> # bad: [5a6a09e97199d6600d31383055f9d43fbbcbe86f] Merge tag 'cgroup-for-6.7' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
> git bisect bad 5a6a09e97199d6600d31383055f9d43fbbcbe86f
> # good: [17047fbced563cf5abe5aa546f6a92af48900b69] bcachefs: Fix incorrectly freeing btree_path in alloc path
> git bisect good 17047fbced563cf5abe5aa546f6a92af48900b69
> # good: [b827ac419721a106ae2fccaa40576b0594edad92] exportfs: Change bcachefs fid_type enum to avoid conflicts
> git bisect good b827ac419721a106ae2fccaa40576b0594edad92
> # bad: [9ab021a1b57007a22761f6f41d91eb4aae10d145] Merge tag 'x86_cache_for_6.7_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> git bisect bad 9ab021a1b57007a22761f6f41d91eb4aae10d145
> # good: [8b16da681eb0c9b9cb2f9abd0dade67559cfb48d] Merge tag 'nfsd-6.7' of git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux
> git bisect good 8b16da681eb0c9b9cb2f9abd0dade67559cfb48d
> # bad: [07a274a8862dba86a270ced2a4c5ff3f7a01b66a] btrfs: remove redundant root argument from btrfs_update_inode_item()
> git bisect bad 07a274a8862dba86a270ced2a4c5ff3f7a01b66a
> # good: [3ee56a58ad8921cb43c49d56347a8e270871844c] btrfs: reserve space for delayed refs on a per ref basis
> git bisect good 3ee56a58ad8921cb43c49d56347a8e270871844c
> # bad: [9f9918a8017b7925da2fad16b4ccf6a14630f03e] btrfs: sysfs: announce presence of raid-stripe-tree
> git bisect bad 9f9918a8017b7925da2fad16b4ccf6a14630f03e
> # bad: [87463f7e0250d471fac41e7c9c45ae21d83b5f85] btrfs: zoned: factor out DUP bg handling from btrfs_load_block_group_zone_info
> git bisect bad 87463f7e0250d471fac41e7c9c45ae21d83b5f85
> # bad: [50564b651d01c19ce732819c5b3c3fd60707188e] btrfs: abort transaction on generation mismatch when marking eb as dirty
> git bisect bad 50564b651d01c19ce732819c5b3c3fd60707188e
> # bad: [28270e25c69a2c76ea1ed0922095bffb9b9a4f98] btrfs: always reserve space for delayed refs when starting transaction
> git bisect bad 28270e25c69a2c76ea1ed0922095bffb9b9a4f98
> # good: [adb86dbe426f9a54843d70092819deca220a224d] btrfs: stop doing excessive space reservation for csum deletion
> git bisect good adb86dbe426f9a54843d70092819deca220a224d
> # first bad commit: [28270e25c69a2c76ea1ed0922095bffb9b9a4f98] btrfs: always reserve space for delayed refs when starting transaction

This sounds like the generally more pessimistic metadata reservation
is triggering allocation of many metadata block groups
that never get used and then unused and therefore not
reclaimed/deleted. Not something impossible to happen before that,
but much more likely due to more reserved space.

I'll send some fixes.

Did you actually run into -ENOSPC issues?

Thanks.

> ```
>
> Thanks,
> --
> Ivan Shapovalov / intelfx /
>

Re: [6.7 regression] [BISECTED] 28270e25c69a causes overallocation of metadata space

[Ivan Shapovalov]

On 2024-01-17 at 11:28 +0000, Filipe Manana wrote:
> On Wed, Jan 17, 2024 at 6:04 AM Ivan Shapovalov
> <intelfx@intelfx.name> wrote:
> > 
> > <...>
> > # first bad commit: [28270e25c69a2c76ea1ed0922095bffb9b9a4f98]
> > btrfs: always reserve space for delayed refs when starting
> > transaction
> 
> This sounds like the generally more pessimistic metadata reservation
> is triggering allocation of many metadata block groups
> that never get used and then unused and therefore not
> reclaimed/deleted. Not something impossible to happen before that,
> but much more likely due to more reserved space.
> 
> I'll send some fixes.
> 
> Did you actually run into -ENOSPC issues?

No, I did not wait around for a ENOSPC -- I noticed the issue when my
monitoring told me that there was almost no unallocated space left (on
my personal server's rootfs the situation was even worse than on the fs
above, there was around 200G of metadata logical space allocated with
only 13G used).

-- 
Ivan Shapovalov / intelfx /

Re: [6.7 regression] [BISECTED] 28270e25c69a causes overallocation of metadata space

[Linux regression tracking #adding (Thorsten Leemhuis)]

On 17.01.24 07:04, Ivan Shapovalov wrote:
> Hi all,
> 
> Starting from v6.7 I've noticed severe overallocation of metadata space
> on both of my btrfs root filesystems (both NVMe SSDs, both use
> snapshots):
> 
> ```
> # mount | grep -w /
> rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=<...>
> 
> # btrfs fi usage /
> <...>
> 
> Data,single: Size:550.00GiB, Used:497.12GiB (90.39%)
>    /dev/mapper/root      550.00GiB
> 
> Metadata,DUP: Size:72.00GiB, Used:8.38GiB (11.64%)
>    /dev/mapper/root      144.00GiB
> 
> <...>
> ```
> 
> Running a full metadata balance (`btrfs balance start -m /`) or a
> "staged" balance
> (e. g. `for u in {0..90..5}; do btrfs balance start -musage=$u /`)
> does not help / makes the overallocation worse.

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot ^introduced 28270e25c69a
#regzbot title btrfs: severe overallocation of metadata space
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.