On 2019/7/14 上午4:48, Alexander Wetzel wrote:
> Hello,
> 
> After updating one of my VMs from 5.1.16 to 5.2 btrfs is acting up strange:
> The system is using btrfs as root (also for /boot) and has compression
> enabled. (It's a gentoo virtual machine and not using an initrd.)
> Rebooting the system into 5.2 is able to bring up openssh, but not other
> services (like e.g. postfix).
> 
> Redirecting dmesg to a file also failed. The file was created but empty,
> even when checked immediately. Rebooting the system into the old kernel
> fully restores functionality and a btrfs srub shows no errors..
> 
> When running a bad kernel the system is partially ro, but since I'm
> using selinux in strict mode it could also by caused by selinux unable
> to read some data.
> 
> Here how a reboot via ssh looks for a broken kernel:
> xar /home/alex # shutdown -r now
> shutdown: warning: cannot open /var/run/shutdown.pid
> 
> But deleting the "bad" kernel and running "grub-mkconfig -o
> /boot/grub/grub.cfg" works as it should and the system is using the
> previous kernel... So it still can write some things...
> 
> I've bisected the problem to 496245cac57e (btrfs: tree-checker: Verify
> inode item)
> 
> filtering for btrfs and removing duplicate lines just shows three uniq
> error messages:
>  BTRFS critical (device vda3): corrupt leaf: root=300 block=8645398528
> slot=4 ino=259223, invalid inode generation: has 139737289170944 expect
> [0, 1425224]
>  BTRFS critical (device vda3): corrupt leaf: root=300 block=8645398528
> slot=4 ino=259223, invalid inode generation: has 139737289170944 expect
> [0, 1425225]
>  BTRFS critical (device vda3): corrupt leaf: root=300 block=8645398528
> slot=4 ino=259223, invalid inode generation: has 139737289170944 expect
> [0, 1425227]

The generation number is 0x7f171f7ba000, I see no reason why it would
make any sense.

I see no problem rejecting obviously corrupted item.

The problem is:
- Is that corrupted item?
  At least to me, it looks corrupted just from the dmesg.

- How and when this happens
  Obviously happened on some older kernel.
  V5.2 will report such problem before writing corrupted data back to
  disk, at least prevent such problem from happening.

Please provide the following dump:
 # btrfs ins dump-tree -b 8645398528 /dev/vda3

>  BTRFS error (device vda3): block=8645398528 read time tree block
> corruption detected
> 
> All there errors are only there with commit 496245cac57e, booting a
> kernel without the patch after that just works normally again.
> 
> I tried to reproduce the issue transferring the fs with btrfs
> send/receive to another system. I was able to mount the migrated Fs with
> a 5.2 kernel and also btrfs scub was ok...
> 
> I tried to revert the commit but a simple revert is not working. So I've
> just verified that a kernel build on 496245cac57e shows the symptoms
> while using the commit before that is fine.
> 
> I've not tried more, so I still can reproduce the issue when needed and
> gather more data. (The system is now back running 5.1.16)
> 
> Now I guess I could have some corruption only detected/triggered with
> the patch and btrfs check may fix it... shall I try that next?

Sorry, AFAIK btrfs doesn't check as strict as kernel tree-checker, as
corrupted data in kernel space could lead to system crash while in user
space it would only cause btrfs check to crash.

Thus I made tree-checker way picky about irregular data, it's literally
checking every member and even unused member.
The dump mentioned above should help us to determine whether btrfs check
can detect and fix it.
(I believe it shouldn't be that hard to fix in btrfs-progs)

Thanks,
Qu

> 
> Here the dmesg from the affected system (the first few lines still in
> the log buffer when I checked):
> [    8.963796] BTRFS critical (device vda3): corrupt leaf: root=300
> block=8645398528 slot=4 ino=259223, invalid inode generation: has
> 139737289170944 expect [0, 1425224]
> [    8.963799] BTRFS error (device vda3): block=8645398528 read time
> tree block corruption detected
> [    8.967487] audit: type=1400 audit(1563023702.540:19): avc:  denied {
> write } for  pid=2154 comm="cp" name="localtime" dev="vda3" ino=1061039
> scontext=system_u:system_r:initrc_t tcontext=system_u:object_r:locale_t
> tclass=file permissive=0
> [    9.023608] audit: type=1400 audit(1563023702.590:20): avc:  denied {
> mounton } for  pid=2194 comm="mount" path="/chroot/dns/run/named"
> dev="vda3" ino=1061002 scontext=system_u:system_r:mount_t
> tcontext=system_u:object_r:named_var_run_t tclass=dir permissive=0
> [    9.038235] audit: type=1400 audit(1563023702.610:21): avc:  denied {
> getattr } for  pid=2199 comm="start-stop-daem" path="pid:[4026531836]"
> dev="nsfs" ino=4026531836 scontext=system_u:system_r:initrc_t
> tcontext=system_u:object_r:nsfs_t tclass=file permissive=0
> [    9.100897] BTRFS critical (device vda3): corrupt leaf: root=300
> block=8645398528 slot=4 ino=259223, invalid inode generation: has
> 139737289170944 expect [0, 1425224]
> [    9.100900] BTRFS error (device vda3): block=8645398528 read time
> tree block corruption detected
> [    9.137974] BTRFS critical (device vda3): corrupt leaf: root=300
> block=8645398528 slot=4 ino=259223, invalid inode generation: has
> 139737289170944 expect [0, 1425224]
> [    9.137976] BTRFS error (device vda3): block=8645398528 read time
> tree block corruption detected
> [    9.138095] audit: type=1400 audit(1563023702.710:22): avc:  denied {
> getattr } for  pid=2237 comm="start-stop-daem" path="pid:[4026531836]"
> dev="nsfs" ino=4026531836 scontext=system_u:system_r:initrc_t
> tcontext=system_u:object_r:nsfs_t tclass=file permissive=0
> [    9.138477] BTRFS critical (device vda3): corrupt leaf: root=300
> block=8645398528 slot=4 ino=259223, invalid inode generation: has
> 139737289170944 expect [0, 1425224]
> [    9.138480] BTRFS error (device vda3): block=8645398528 read time
> tree block corruption detected
> [    9.161866] BTRFS critical (device vda3): corrupt leaf: root=300
> block=8645398528 slot=4 ino=259223, invalid inode generation: has
> 139737289170944 expect [0, 1425224]
> [    9.161868] BTRFS error (device vda3): block=8645398528 read time
> tree block corruption detected
> [    9.170228] BTRFS critical (device vda3): corrupt leaf: root=300
> block=8645398528 slot=4 ino=259223, invalid inode generation: has
> 139737289170944 expect [0, 1425224]
> [    9.170230] BTRFS error (device vda3): block=8645398528 read time
> tree block corruption detected
> [    9.214491] BTRFS critical (device vda3): corrupt leaf: root=300
> block=8645398528 slot=4 ino=259223, invalid inode generation: has
> 139737289170944 expect [0, 1425224]
> [    9.214494] BTRFS error (device vda3): block=8645398528 read time
> tree block corruption detected
> 
> 
> Alexander