delayed_refs has NO entry / btrfs_update_root:136: Aborting unused transaction(No space left).

delayed_refs has NO entry / btrfs_update_root:136: Aborting unused transaction(No space left).

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1104 bytes --]

(resending,... seems this hasn't gotten through to the list, when I've
sent it the first time)


Hi.

On Debian's 4.19.28-2 kernel (which includes the recent read-
corruption 
on compression fix) the following happens:

As a consequence of the bug from the "Reproducer for "compressed data +
hole data corruption bug, 2018 edition" still works on 4.20.7" mail
thread I started (trying) to verify whether any of my data was
affected.

Part of this was looking for files which actually are compressed by the
two methods Zygo mentioned (compsize and filefrag -v).

For this I used two scripts like the attached ones (yes I know, bad
performance) being fed by find path -type f -exec script {} \; .


I've did this already on one of my disks, which I btrfs checked before
(both normal and lowmem mode with no error), blockdev --setro'ed the
device and mounted it ro.

The filefrag seems to cause all kinds of errors and call traces, giving
the dmesg output attached.


Any ideas what causes that?


These days I unfortunately strongly loose trust in the stability and
integrity of btrfs :-(


Cheers,
Chris.

[-- Attachment #2: compsize-compressed --]
[-- Type: application/x-shellscript, Size: 225 bytes --]

[-- Attachment #3: filefrag-compressed --]
[-- Type: application/x-shellscript, Size: 217 bytes --]

[-- Attachment #4: dmesg.xz --]
[-- Type: application/x-xz, Size: 5424 bytes --]

Re: delayed_refs has NO entry / btrfs_update_root:136: Aborting unused transaction(No space left).

[Christoph Anton Mitterer]

Anything I should do with respect to this?

I.e. is further debug info needed for an interested developer? or can I
simply scrap that particular image (which is not an important one)?

Cheers,
Chris. 


On Sun, 2019-03-17 at 04:42 +0100, Christoph Anton Mitterer wrote:
> (resending,... seems this hasn't gotten through to the list, when
> I've
> sent it the first time)
> 
> 
> Hi.
> 
> On Debian's 4.19.28-2 kernel (which includes the recent read-
> corruption 
> on compression fix) the following happens:
> 
> As a consequence of the bug from the "Reproducer for "compressed data
> +
> hole data corruption bug, 2018 edition" still works on 4.20.7" mail
> thread I started (trying) to verify whether any of my data was
> affected.
> 
> Part of this was looking for files which actually are compressed by
> the
> two methods Zygo mentioned (compsize and filefrag -v).
> 
> For this I used two scripts like the attached ones (yes I know, bad
> performance) being fed by find path -type f -exec script {} \; .
> 
> 
> I've did this already on one of my disks, which I btrfs checked
> before
> (both normal and lowmem mode with no error), blockdev --setro'ed the
> device and mounted it ro.
> 
> The filefrag seems to cause all kinds of errors and call traces,
> giving
> the dmesg output attached.
> 
> 
> Any ideas what causes that?
> 
> 
> These days I unfortunately strongly loose trust in the stability and
> integrity of btrfs :-(
> 
> 
> Cheers,
> Chris.

Re: delayed_refs has NO entry / btrfs_update_root:136: Aborting unused transaction(No space left).

[Johannes Thumshirn]

On 20/03/2019 01:59, Christoph Anton Mitterer wrote:
> Anything I should do with respect to this?
> 
> I.e. is further debug info needed for an interested developer? or can I
> simply scrap that particular image (which is not an important one)?


OK, I can give it a shot, but please be aware I'm still relatively new
to BTRFS.

First of all, have you tried a more recent kernel than the Debian
kernels you referenced? E.g. Linus' current master or David's misc-next
branch? Just so we don't try to hunt down a bug that's already fixed.

If it's still reproducible with a more recent kernel, I think having an
image would be very valuable.

From the dmesg (I don't have the referenced kernel version handy, Greg's
4.19.27 is the closest I have) we're hitting this WARN_ON():

        WARN_ON(cur_trans != info->running_transaction);

Which means there's a mismatch between the transaction that's currently
running and the transaction the filesystem thinks is running, if I read
it correctly.

Also if you can still reproduce the bug, please activate tracing in
btrfs and send the trace output.

Byte,
	Johannes
-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: delayed_refs has NO entry / btrfs_update_root:136: Aborting unused transaction(No space left).

[Christoph Anton Mitterer]

On Wed, 2019-03-20 at 10:59 +0100, Johannes Thumshirn wrote:
> First of all, have you tried a more recent kernel than the Debian
> kernels you referenced? E.g. Linus' current master or David's misc-
> next
> branch? Just so we don't try to hunt down a bug that's already fixed.

I haven't and that's a bit difficult for me unless it's packaged by the
distro (policy reasons).


Also giving out the image is a bit problematic as it's huge (8TB).

> Also if you can still reproduce the bug, please activate tracing in
> btrfs and send the trace output.

How would I do that?


In the meantime, I think I can reproduce it with fresh images so could
you try the following:

# truncate --size 1G image
# mkfs.btrfs image

# mount -o compress image /mnt
# cd /mnt

# # create some data e.g.:
# tar xaf /usr/src/linux-source-4.19.tar.xz
# cd
# umount /mnt

# losetup -r -f image
# mount -o compress /dev/loop0 /mnt

# find /mnt -type f -exec filefrag -v {} \;


And there your kernel log will explode ;-)

The culprit seems to be the device itself being read-only i.e.
losetup's -r, respectively blockdev --setro DEVICE which I've used
previously.

If you repeat the above from the losetup point, but with -r ...
everything works fine.
Haven't checked whether -o compress actually makes a difference.


Cheers,
Chris.

Re: delayed_refs has NO entry / btrfs_update_root:136: Aborting unused transaction(No space left).

[Christoph Anton Mitterer]

On Sat, 2019-04-13 at 00:46 +0200, Christoph Anton Mitterer wrote:
> If you repeat the above from the losetup point, but with -r ...

s/with -r/without -r/

Re: delayed_refs has NO entry / btrfs_update_root:136: Aborting unused transaction(No space left).

[Johannes Thumshirn]

On Sat, Apr 13, 2019 at 12:46:28AM +0200, Christoph Anton Mitterer wrote:
> On Wed, 2019-03-20 at 10:59 +0100, Johannes Thumshirn wrote:
> > First of all, have you tried a more recent kernel than the Debian
> > kernels you referenced? E.g. Linus' current master or David's misc-
> > next
> > branch? Just so we don't try to hunt down a bug that's already fixed.
> 
> I haven't and that's a bit difficult for me unless it's packaged by the
> distro (policy reasons).
> 
> 
> Also giving out the image is a bit problematic as it's huge (8TB).
> 
> > Also if you can still reproduce the bug, please activate tracing in
> > btrfs and send the trace output.
> 
> How would I do that?
> 
> 
> In the meantime, I think I can reproduce it with fresh images so could
> you try the following:
> 
> # truncate --size 1G image
> # mkfs.btrfs image
> 
> # mount -o compress image /mnt
> # cd /mnt
> 
> # # create some data e.g.:
> # tar xaf /usr/src/linux-source-4.19.tar.xz
> # cd
> # umount /mnt
> 
> # losetup -r -f image
> # mount -o compress /dev/loop0 /mnt
> 
> # find /mnt -type f -exec filefrag -v {} \;
> 

I'll give it a try.


> 
> And there your kernel log will explode ;-)
> 
> The culprit seems to be the device itself being read-only i.e.
> losetup's -r, respectively blockdev --setro DEVICE which I've used
> previously.
> 
> If you repeat the above from the losetup point, but with -r ...
> everything works fine.
> Haven't checked whether -o compress actually makes a difference.
> 
> 
> Cheers,
> Chris.
> 
> 

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: delayed_refs has NO entry / btrfs_update_root:136: Aborting unused transaction(No space left).

[Christoph Anton Mitterer]

Hey.

Just asking... was anyone able to reproduce these errors (as described
below)?

Cheers,
Chris.


On Sat, 2019-04-13 at 00:46 +0200, Christoph Anton Mitterer wrote:
> On Wed, 2019-03-20 at 10:59 +0100, Johannes Thumshirn wrote:
> > First of all, have you tried a more recent kernel than the Debian
> > kernels you referenced? E.g. Linus' current master or David's misc-
> > next
> > branch? Just so we don't try to hunt down a bug that's already
> > fixed.
> 
> I haven't and that's a bit difficult for me unless it's packaged by
> the
> distro (policy reasons).
> 
> 
> Also giving out the image is a bit problematic as it's huge (8TB).
> 
> > Also if you can still reproduce the bug, please activate tracing in
> > btrfs and send the trace output.
> 
> How would I do that?
> 
> 
> In the meantime, I think I can reproduce it with fresh images so
> could
> you try the following:
> 
> # truncate --size 1G image
> # mkfs.btrfs image
> 
> # mount -o compress image /mnt
> # cd /mnt
> 
> # # create some data e.g.:
> # tar xaf /usr/src/linux-source-4.19.tar.xz
> # cd
> # umount /mnt
> 
> # losetup -r -f image
> # mount -o compress /dev/loop0 /mnt
> 
> # find /mnt -type f -exec filefrag -v {} \;
> 
> 
> And there your kernel log will explode ;-)
> 
> The culprit seems to be the device itself being read-only i.e.
> losetup's -r, respectively blockdev --setro DEVICE which I've used
> previously.
> 
> If you repeat the above from the losetup point, but with -r ...
> everything works fine.
> Haven't checked whether -o compress actually makes a difference.
> 
> 
> Cheers,
> Chris.
> 

Re: delayed_refs has NO entry / btrfs_update_root:136: Aborting unused transaction(No space left).

[Christoph Anton Mitterer]

Since no one seems to show any big interest in this issues, I've added
it for the records in https://bugzilla.kernel.org/show_bug.cgi?id=203621

Cheers,
Chris.

Re: Patch "Btrfs: do not start a transaction during fiemap"

[Christoph Anton Mitterer]

Hey David.

Regarding your patch "Btrfs: do not start a transaction during
fiemap"...

I assume since the blockdevice had to be set read-only in order for the
bug to happen... all these aborted transactions, etc. couldn't cause
any corruptions/etc. upon the fs,... so there's nothing further one
would need to check, right?


Thanks,
Chris.