Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
From: Alexander Wetzel <alexander.wetzel@web.de>
To: linux-btrfs@vger.kernel.org, wqu@suse.com
Subject: [BUG] BTRFS critical corrupt leaf - bisected to 496245cac57e
Date: Sat, 13 Jul 2019 22:48:55 +0200
Message-ID: <5a89e922-00af-51a9-390f-b0a6b1f6cfb6@web.de> (raw)

Hello,

After updating one of my VMs from 5.1.16 to 5.2 btrfs is acting up strange:
The system is using btrfs as root (also for /boot) and has compression 
enabled. (It's a gentoo virtual machine and not using an initrd.) 
Rebooting the system into 5.2 is able to bring up openssh, but not other 
services (like e.g. postfix).

Redirecting dmesg to a file also failed. The file was created but empty, 
even when checked immediately. Rebooting the system into the old kernel 
fully restores functionality and a btrfs srub shows no errors..

When running a bad kernel the system is partially ro, but since I'm 
using selinux in strict mode it could also by caused by selinux unable 
to read some data.

Here how a reboot via ssh looks for a broken kernel:
xar /home/alex # shutdown -r now
shutdown: warning: cannot open /var/run/shutdown.pid

But deleting the "bad" kernel and running "grub-mkconfig -o 
/boot/grub/grub.cfg" works as it should and the system is using the 
previous kernel... So it still can write some things...

I've bisected the problem to 496245cac57e (btrfs: tree-checker: Verify 
inode item)

filtering for btrfs and removing duplicate lines just shows three uniq 
error messages:
  BTRFS critical (device vda3): corrupt leaf: root=300 block=8645398528 
slot=4 ino=259223, invalid inode generation: has 139737289170944 expect 
[0, 1425224]
  BTRFS critical (device vda3): corrupt leaf: root=300 block=8645398528 
slot=4 ino=259223, invalid inode generation: has 139737289170944 expect 
[0, 1425225]
  BTRFS critical (device vda3): corrupt leaf: root=300 block=8645398528 
slot=4 ino=259223, invalid inode generation: has 139737289170944 expect 
[0, 1425227]
  BTRFS error (device vda3): block=8645398528 read time tree block 
corruption detected

All there errors are only there with commit 496245cac57e, booting a 
kernel without the patch after that just works normally again.

I tried to reproduce the issue transferring the fs with btrfs 
send/receive to another system. I was able to mount the migrated Fs with 
a 5.2 kernel and also btrfs scub was ok...

I tried to revert the commit but a simple revert is not working. So I've 
just verified that a kernel build on 496245cac57e shows the symptoms 
while using the commit before that is fine.

I've not tried more, so I still can reproduce the issue when needed and 
gather more data. (The system is now back running 5.1.16)

Now I guess I could have some corruption only detected/triggered with 
the patch and btrfs check may fix it... shall I try that next?

Here the dmesg from the affected system (the first few lines still in 
the log buffer when I checked):
[    8.963796] BTRFS critical (device vda3): corrupt leaf: root=300 
block=8645398528 slot=4 ino=259223, invalid inode generation: has 
139737289170944 expect [0, 1425224]
[    8.963799] BTRFS error (device vda3): block=8645398528 read time 
tree block corruption detected
[    8.967487] audit: type=1400 audit(1563023702.540:19): avc:  denied 
{ write } for  pid=2154 comm="cp" name="localtime" dev="vda3" 
ino=1061039 scontext=system_u:system_r:initrc_t 
tcontext=system_u:object_r:locale_t tclass=file permissive=0
[    9.023608] audit: type=1400 audit(1563023702.590:20): avc:  denied 
{ mounton } for  pid=2194 comm="mount" path="/chroot/dns/run/named" 
dev="vda3" ino=1061002 scontext=system_u:system_r:mount_t 
tcontext=system_u:object_r:named_var_run_t tclass=dir permissive=0
[    9.038235] audit: type=1400 audit(1563023702.610:21): avc:  denied 
{ getattr } for  pid=2199 comm="start-stop-daem" path="pid:[4026531836]" 
dev="nsfs" ino=4026531836 scontext=system_u:system_r:initrc_t 
tcontext=system_u:object_r:nsfs_t tclass=file permissive=0
[    9.100897] BTRFS critical (device vda3): corrupt leaf: root=300 
block=8645398528 slot=4 ino=259223, invalid inode generation: has 
139737289170944 expect [0, 1425224]
[    9.100900] BTRFS error (device vda3): block=8645398528 read time 
tree block corruption detected
[    9.137974] BTRFS critical (device vda3): corrupt leaf: root=300 
block=8645398528 slot=4 ino=259223, invalid inode generation: has 
139737289170944 expect [0, 1425224]
[    9.137976] BTRFS error (device vda3): block=8645398528 read time 
tree block corruption detected
[    9.138095] audit: type=1400 audit(1563023702.710:22): avc:  denied 
{ getattr } for  pid=2237 comm="start-stop-daem" path="pid:[4026531836]" 
dev="nsfs" ino=4026531836 scontext=system_u:system_r:initrc_t 
tcontext=system_u:object_r:nsfs_t tclass=file permissive=0
[    9.138477] BTRFS critical (device vda3): corrupt leaf: root=300 
block=8645398528 slot=4 ino=259223, invalid inode generation: has 
139737289170944 expect [0, 1425224]
[    9.138480] BTRFS error (device vda3): block=8645398528 read time 
tree block corruption detected
[    9.161866] BTRFS critical (device vda3): corrupt leaf: root=300 
block=8645398528 slot=4 ino=259223, invalid inode generation: has 
139737289170944 expect [0, 1425224]
[    9.161868] BTRFS error (device vda3): block=8645398528 read time 
tree block corruption detected
[    9.170228] BTRFS critical (device vda3): corrupt leaf: root=300 
block=8645398528 slot=4 ino=259223, invalid inode generation: has 
139737289170944 expect [0, 1425224]
[    9.170230] BTRFS error (device vda3): block=8645398528 read time 
tree block corruption detected
[    9.214491] BTRFS critical (device vda3): corrupt leaf: root=300 
block=8645398528 slot=4 ino=259223, invalid inode generation: has 
139737289170944 expect [0, 1425224]
[    9.214494] BTRFS error (device vda3): block=8645398528 read time 
tree block corruption detected


Alexander

             reply index

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-13 20:48 Alexander Wetzel [this message]
2019-07-14  1:30 ` Qu Wenruo
2019-07-14  9:25   ` Alexander Wetzel
2019-07-14  9:49     ` Qu Wenruo
2019-07-14 12:07       ` Alexander Wetzel
2019-07-14 12:51         ` Qu Wenruo
2019-07-14 15:40       ` Chris Murphy
2019-07-15  1:07         ` Qu Wenruo

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5a89e922-00af-51a9-390f-b0a6b1f6cfb6@web.de \
    --to=alexander.wetzel@web.de \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=wqu@suse.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org linux-btrfs@archiver.kernel.org
	public-inbox-index linux-btrfs


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/ public-inbox