On 2020/1/17 下午4:48, Kenneth Topp wrote:
> On 2020-01-17 03:33, Qu Wenruo wrote:
>> On 2020/1/17 下午4:21, Kenneth Topp wrote:
>>> I've been running 5.4.8 for 10 days and this popped up:
>>>
>>> [917767.192190] BTRFS error (device dm-19): block=21374679056384 write
>>> time tree block corruption detected
>>> [917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
>>> errno=-5 IO failure
>>> [917767.193438] BTRFS info (device dm-19): forced readonly
>>
>> You missed the most important line. One line before "write time tree
>> block corruption detected" is the most important line.
>>
> 
> sorry, here is the missing line.
> 
> [917767.192187] BTRFS critical (device dm-19): corrupt leaf:
> root=18446744073709551610 block=21374679056384 slot=174 ino=219743
> file_offset=1385357312, file extent end range (1460178944) g
> oes beyond start offset (1385385984) of the next file extent
> [917767.192190] BTRFS error (device dm-19): block=21374679056384 write
> time tree block corruption detected
> [917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
> errno=-5 IO failure
> [917767.193438] BTRFS info (device dm-19): forced readonly

Hi Filipe,

This looks like one of your fixed problem for log tree, but I'm not very
confident.

Is this a fixed one or a new one?

Thanks,
Qu

> 
> thanks for responding so quickly!
> 
> ken
> 
>>>
>>>
>>> This was a btrfs filesystem created on 5.4.8 with two 4kn drives, each
>>> with a single whole disk dm-crypt partition.  The fs was created with
>>> mkfs -O no-holes -d single -m raid1 and mounted with
>>> clear_cache,space_cache=v2.
>>>
>>> After that issue, I tried to update my backups, but it didn't go very
>>> well:
>>>
>>> [918052.104923] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 21374567694336 wanted 6c2341bd found 640f30e8 level 0
>>> [918052.121801] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.121818] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397271552
>>> [918052.138406] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.139080] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.139094] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397275648
>>> [918052.139768] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140158] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140171] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397279744
>>> [918052.140532] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140868] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140880] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397283840
>>> [918052.159794] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
>>> [918052.213959] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.216411] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.216418] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397271552
>>> [918052.216570] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
>>> [918052.216790] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.217037] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397271552
>>> [918052.217165] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
>>> [918062.496366] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043100192768 wanted 8bf4c1c9 found 23b8dea8 level 0
>>> [918062.504832] verify_parent_transid: 1 callbacks suppressed
>>> [918062.504836] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.504848] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 28672
>>> [918062.506174] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100241920 wanted 45126 found 45104
>>> [918062.509823] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 28672 csum 0xa0f9eaf7 expected csum 0x00000000 mirror 1
>>> [918062.513826] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100241920 wanted 45126 found 45104
>>> [918062.513838] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 49152
>>> [918062.519723] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 49152 csum 0x76421bb0 expected csum 0x00000000 mirror 1
>>> [918062.530287] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.530866] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.530874] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 65536
>>> [918062.539238] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.539560] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.539566] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 69632
>>> [918062.540020] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.540261] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.540269] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 77824
>>> [918062.540965] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.541199] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 114688
>>> [918062.541753] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 118784
>>> [918062.544496] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 65536 csum 0xdcb45a27 expected csum 0x00000000 mirror 1
>>> [918062.547421] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 114688 csum 0x1d5a0f30 expected csum 0x00000000 mirror 1
>>> [918062.549968] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 69632 csum 0x24dd186c expected csum 0x00000000 mirror 1
>>> [918062.572502] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 77824 csum 0xba341c47 expected csum 0x00000000 mirror 1
>>> [918062.575300] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 118784 csum 0x519e7044 expected csum 0x00000000 mirror 1
>>> [918062.584005] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 172032
>>> [918062.586782] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 221184
>>> [918062.587106] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 221184 csum 0xc9c50b2d expected csum 0x00000000 mirror 1
>>> [918062.590739] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 450560
>>> [918062.591255] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 450560 csum 0xd2e5e3eb expected csum 0x00000000 mirror 1
>>> [918062.601361] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043100258304 wanted 61da40d5 found 204cd10a level 0
>>> [918062.602875] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 503808 csum 0x183beedc expected csum 0x00000000 mirror 1
>>> [918085.820716] verify_parent_transid: 42 callbacks suppressed
>>> [918085.820730] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.837964] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.837992] __btrfs_lookup_bio_sums: 17 callbacks suppressed
>>> [918085.837994] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159711232
>>> [918085.838459] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.838987] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.838995] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159715328
>>> [918085.839349] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840054] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840065] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159719424
>>> [918085.840402] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840943] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840951] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159723520
>>> [918085.850438] btrfs_print_data_csum_error: 17 callbacks suppressed
>>> [918085.850442] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
>>> [918085.959941] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.960447] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.960452] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159711232
>>> [918085.960624] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
>>> [918085.961220] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159711232
>>> [918085.961341] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
>>> [918087.026984] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193855488
>>> [918087.028152] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193859584
>>> [918087.028598] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193863680
>>> [918087.029005] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193867776
>>> [918087.042810] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
>>> [918087.054812] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193904640 csum 0x279aea06 expected csum 0x00000000 mirror 1
>>> [918087.071004] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043549310976 wanted fcc4f73a found 3d3d8e83 level 0
>>> [918087.103303] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193937408 csum 0x1fedc264 expected csum 0x00000000 mirror 1
>>> [918087.118559] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
>>> [918087.131234] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
>>> [918087.132513] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
>>> [918087.135749] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
>>> [918119.990544] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043135205376 wanted 79d70254 found 93ac2b61 level 0
>>> [918120.010421] verify_parent_transid: 105 callbacks suppressed
>>> [918120.010428] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043135205376 wanted 45126 found 45104
>>> [918120.010565] __btrfs_lookup_bio_sums: 48 callbacks suppressed
>>> [918120.010567] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 678543360
>>> [918120.028733] btrfs_print_data_csum_error: 5 callbacks suppressed
>>> [918120.028735] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
>>> [918120.040591] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043135205376 wanted 45126 found 45104
>>> [918120.041097] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043135205376 wanted 45126 found 45104
>>> [918120.041112] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 678543360
>>> [918120.041440] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
>>
>> Since you haven't rebooted, all these mismatch can be caused by aborted
>> transaction.
>>
>>>
>>>
>>> Fortunately, after a reboot, everything appears healthy, no file
>>> corruption that I can observe, but I'm still running some checks.
>>
>> That's great, write time tree checker prevented corruption.
>>
>> But still, btrfs check, and scrub recommended.
>>
>> And the most important thing, please provide the missing key log line.
>>
>> Thanks,
>> Qu
>>
>>>
>>> I'm not sure what the write tree is or if this is related to the read
>>> tree issues mentioned on the mailing list, any advice/pointers would be
>>> most appreciated.  One thing I can note is that I've started to write
>>> heavily to this filesystem via nfs export (the nfs client happens to be
>>> a vm on the same machine, fwiw).
>>>