LTP fs_fill test results in BTRFS warning (device loop0): could not allocate space for a delete; will truncate on mount warnings

LTP fs_fill test results in BTRFS warning (device loop0): could not allocate space for a delete; will truncate on mount warnings

[Rachel Sibley]

Hello,

While running LTP [1] as part of CKI [2] testing, we noticed the fs_fill 
test fails pretty
consistently with BTRFS warnings seen below, this is seen with recent 
kernel (5.2.12).
I have included the logs below for reference.

Thank you,
Rachel

* 
https://artifacts.cki-project.org/pipelines/147783/logs/aarch64_host_1_console.log
8747.237733] LTP: starting fs_fill
[ 8748.085884] EXT4-fs (loop0): mounting ext2 file system using the ext4 
subsystem
[ 8748.103546] EXT4-fs (loop0): mounted filesystem without journal. 
Opts: (null)
[ 8751.856179] EXT4-fs (loop0): mounting ext3 file system using the ext4 
subsystem
[ 8751.882784] EXT4-fs (loop0): mounted filesystem with ordered data 
mode. Opts: (null)
[ 8759.496036] EXT4-fs (loop0): mounted filesystem with ordered data 
mode. Opts: (null)
[ 8768.360525] XFS (loop0): Mounting V5 Filesystem
[ 8768.368464] XFS (loop0): Ending clean mount
[ 8775.694012] XFS (loop0): Unmounting Filesystem
[ 8776.179816] BTRFS: device fsid 3f8f3f1f-6293-40c4-91f4-bfe1d5c4451d 
devid 1 transid 5 /dev/loop0
[ 8776.190118] BTRFS info (device loop0): disk space caching is enabled
[ 8776.195513] BTRFS info (device loop0): has skinny extents
[ 8776.200927] BTRFS info (device loop0): flagging fs with big metadata 
feature
[ 8776.212841] BTRFS info (device loop0): checking UUID tree
[ 9062.793701] BTRFS warning (device loop0): could not allocate space 
for a delete; will truncate on mount
[ 9062.794100] BTRFS warning (device loop0): could not allocate space 
for a delete; will truncate on mount
<snip>
[ 9062.794172] BTRFS warning (device loop0): could not allocate space 
for a delete; will truncate on mount

fs_fill log: https://paste.fedoraproject.org/paste/368EirX-AHSsZhG4Zutm2w

[1] https://github.com/linux-test-project/ltp
[2] https://cki-project.org/

<https://projects.engineering.redhat.com/secure/ViewProfile.jspa?name=jstancek>

Re: LTP fs_fill test results in BTRFS warning (device loop0): could not allocate space for a delete; will truncate on mount warnings

[Chris Murphy]

On Fri, Sep 6, 2019 at 1:21 PM Rachel Sibley <rasibley@redhat.com> wrote:
>
> Hello,
>
> While running LTP [1] as part of CKI [2] testing, we noticed the fs_fill
> test fails pretty
> consistently with BTRFS warnings seen below, this is seen with recent
> kernel (5.2.12).
> I have included the logs below for reference.

I'm only 6 for 6, but  so far can't get it to trigger with 5.3.0-rc7


-- 
Chris Murphy

Re: LTP fs_fill test results in BTRFS warning (device loop0): could not allocate space for a delete; will truncate on mount warnings

[Chris Murphy]

On Fri, Sep 6, 2019 at 3:35 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Fri, Sep 6, 2019 at 1:21 PM Rachel Sibley <rasibley@redhat.com> wrote:
> >
> > Hello,
> >
> > While running LTP [1] as part of CKI [2] testing, we noticed the fs_fill
> > test fails pretty
> > consistently with BTRFS warnings seen below, this is seen with recent
> > kernel (5.2.12).
> > I have included the logs below for reference.
>
> I'm only 6 for 6, but  so far can't get it to trigger with 5.3.0-rc7

Also cannot reproduce, 10 for 10 with 5.2.13-200.fc30.x86_64 which was
just built in koji, same as 5.2.12 which was quickly replaced with a
single unrelated revert for touchpads.

But comparing fs_fill logs, yours has a ton more writer threads (~6x)
and ENOSPC counts. Possibly related?

-- 
Chris Murphy

Re: LTP fs_fill test results in BTRFS warning (device loop0): could not allocate space for a delete; will truncate on mount warnings

[Rachel Sibley]

On 9/6/19 5:56 PM, Chris Murphy wrote:
> On Fri, Sep 6, 2019 at 3:35 PM Chris Murphy <lists@colorremedies.com> wrote:
>> On Fri, Sep 6, 2019 at 1:21 PM Rachel Sibley <rasibley@redhat.com> wrote:
>>> Hello,
>>>
>>> While running LTP [1] as part of CKI [2] testing, we noticed the fs_fill
>>> test fails pretty
>>> consistently with BTRFS warnings seen below, this is seen with recent
>>> kernel (5.2.12).
>>> I have included the logs below for reference.
>> I'm only 6 for 6, but  so far can't get it to trigger with 5.3.0-rc7
> Also cannot reproduce, 10 for 10 with 5.2.13-200.fc30.x86_64 which was
> just built in koji, same as 5.2.12 which was quickly replaced with a
> single unrelated revert for touchpads.
>
> But comparing fs_fill logs, yours has a ton more writer threads (~6x)
> and ENOSPC counts. Possibly related?
It could be related to the higher thread/ENOSPC counts, I tried to 
manually reproduce
right now and was unsuccessful, I'll see if I can get a reproducible 
scenario. I'm traveling
the next few weeks but will review results when I get back.

Thanks for looking into it!
Rachel
>