Write time tree block corruption detected

Write time tree block corruption detected

[auxsvr]

Hello,

On linux 5.3.18-lp152.75-default (openSUSE 15.2), I got the following error message:

[33573.070335] BTRFS critical (device sda2): corrupt leaf: root=3 block=531135201280 slot=0 devid=1 invalid bytes used: have 64503152640 expect [0, 64424509440]
[33573.070338] BTRFS error (device sda2): block=531135201280 write time tree block corruption detected 
[33573.071909] BTRFS: error (device sda2) in btrfs_commit_transaction:2264: errno=-5 IO failure (Error while writing out transaction) 
[33573.071911] BTRFS info (device sda2): forced readonly 
[33573.071913] BTRFS warning (device sda2): Skipping commit of aborted transaction. 
[33573.071915] BTRFS: error (device sda2) in cleanup_transaction:1823: errno=-5 IO failure 
[33573.071917] BTRFS info (device sda2): delayed_refs has NO entry 
[33577.283856] BTRFS info (device sda2): delayed_refs has NO entry

Probably related is that the filesystem had less than 1 GiB free when this occurred. Rebooting and deleting some files seemed to have resolved it for a while, until:

[ 5201.020901] BTRFS critical (device sda2): corrupt leaf: root=3 block=536671682560 slot=0 devid=1 invalid bytes used: have 64503152640 expect [0, 64424509440]
[ 5201.020904] BTRFS error (device sda2): block=536671682560 write time tree block corruption detected
[ 5201.022983] BTRFS: error (device sda2) in btrfs_commit_transaction:2264: errno=-5 IO failure (Error while writing out transaction)
[ 5201.022985] BTRFS info (device sda2): forced readonly
[ 5201.022987] BTRFS warning (device sda2): Skipping commit of aborted transaction.
[ 5201.022988] BTRFS: error (device sda2) in cleanup_transaction:1823: errno=-5 IO failure
[ 5201.022991] BTRFS info (device sda2): delayed_refs has NO entry
[ 5201.023071] BTRFS info (device sda2): delayed_refs has NO entry

after a successful scrub and balance. This time df reports 8.8GiB free, but:

# btrfs fi df /
Data, single: total=55.98GiB, used=47.18GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=2.00GiB, used=1.01GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

# btrfs fi show /
Label: none  uuid: 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
        Total devices 1 FS bytes used 48.19GiB
        devid    1 size 60.00GiB used 60.07GiB path /dev/sda2

60.07GiB used out of 60.00GiB! I recall that this is what I saw the first time too, so it might be related to the cause.
Is there anything I can do to ensure that the filesystem is not further damaged and that the issue is resolved?

Regards,
Petros

Re: Write time tree block corruption detected

[auxsvr]

On Monday, 31 May 2021 00:30:00 EEST auxsvr@gmail.com wrote:
> Hello,
> 
> On linux 5.3.18-lp152.75-default (openSUSE 15.2), I got the following error message:
> 
> [33573.070335] BTRFS critical (device sda2): corrupt leaf: root=3 block=531135201280 slot=0 devid=1 invalid bytes used: have 64503152640 expect [0, 64424509440]
> [33573.070338] BTRFS error (device sda2): block=531135201280 write time tree block corruption detected 
> [33573.071909] BTRFS: error (device sda2) in btrfs_commit_transaction:2264: errno=-5 IO failure (Error while writing out transaction) 
> [33573.071911] BTRFS info (device sda2): forced readonly 
> [33573.071913] BTRFS warning (device sda2): Skipping commit of aborted transaction. 
> [33573.071915] BTRFS: error (device sda2) in cleanup_transaction:1823: errno=-5 IO failure 
> [33573.071917] BTRFS info (device sda2): delayed_refs has NO entry 
> [33577.283856] BTRFS info (device sda2): delayed_refs has NO entry

After using btrfs check --repair /dev/sda2:

ref mismatch on [184344514560 4096] extent item 1024, found 0
owner ref check failed [184344514560 4096]
repair deleting extent record: key 184344514560 168 4096 
Repaired extent references for 184344514560
Dev extent's total-byte(61236838400) is not equal to byte-used(63451430912) in dev[1, 216, 1]

without resolving the issue, as the filesystem would still become read-only, I deleted all snapshots and there hasn't been an issue for a few hours of heavy use. However, btrfs check still returns the error regarding size mismatch and this worries me a bit. Is there an easy way to resolve this or should I recreate the filesystem?

It seems that every time free space is less than 1 GiB and I use zypper, I get a similar corruption message. Thankfully, there hasn't been any data loss so far. There's also a backup of the filesystem right after the first incident, in case it is useful for further investigation.

Regards,
Petros

Re: Write time tree block corruption detected

[auxsvr]

On Monday, 7 June 2021 14:55:10 EEST you wrote:
> It seems that every time free space is less than 1 GiB and I use zypper, I get a similar corruption message.

Correction: I just got:

[26355.330817] BTRFS critical (device sda2): corrupt leaf: root=3 block=537007243264 slot=0 devid=1 invalid bytes used: have 64503152640 expect [0, 64424509440]
[26355.330819] BTRFS error (device sda2): block=537007243264 write time tree block corruption detected

and:

# btrfs fi df /
Data, single: total=55.98GiB, used=43.04GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=2.00GiB, used=1.17GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

# btrfs fi show /
Label: none  uuid: 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
        Total devices 1 FS bytes used 44.22GiB
        devid    1 size 60.00GiB used 60.07GiB path /dev/sda2

60.07GiB used again!
 
Regards,
Petros

Re: Write time tree block corruption detected

[Qu Wenruo]

On 2021/6/8 下午10:45, auxsvr wrote:
> On Monday, 7 June 2021 14:55:10 EEST you wrote:
>> It seems that every time free space is less than 1 GiB and I use zypper, I get a similar corruption message.
>
> Correction: I just got:
>
> [26355.330817] BTRFS critical (device sda2): corrupt leaf: root=3 block=537007243264 slot=0 devid=1 invalid bytes used: have 64503152640 expect [0, 64424509440]

The line itself should explain itself, we're over allocating the device.

> [26355.330819] BTRFS error (device sda2): block=537007243264 write time tree block corruption detected

Mind to dump the device tree?

You can dump it with the following command, and the output contains
nothing confidential, but only device extent layout:

# btrfs ins dump-tree -t dev /dev/sda2

>
> and:
>
> # btrfs fi df /
> Data, single: total=55.98GiB, used=43.04GiB
> System, single: total=32.00MiB, used=16.00KiB
> Metadata, single: total=2.00GiB, used=1.17GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> # btrfs fi show /
> Label: none  uuid: 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
>          Total devices 1 FS bytes used 44.22GiB
>          devid    1 size 60.00GiB used 60.07GiB path /dev/sda2

My current guess is, there is something wrong in the past, leaving the
accounting wrong.

It leaves some ghost used bytes, thus causing the problem.

Your previous btrfs-check is able to detect that, but unfortunately we
don't have the ability to repair it in btrfs-check right now.

I'll work on repair ability soon, thus if you can keep the fs, there is
a high chance to repair it.

Thanks,
Qu

>
> 60.07GiB used again!
>
> Regards,
> Petros
>
>
>
>

Re: Write time tree block corruption detected

[Qu Wenruo]

On 2021/6/9 上午6:39, Qu Wenruo wrote:
>
>
> On 2021/6/8 下午10:45, auxsvr wrote:
>> On Monday, 7 June 2021 14:55:10 EEST you wrote:
>>> It seems that every time free space is less than 1 GiB and I use
>>> zypper, I get a similar corruption message.
>>
>> Correction: I just got:
>>
>> [26355.330817] BTRFS critical (device sda2): corrupt leaf: root=3
>> block=537007243264 slot=0 devid=1 invalid bytes used: have 64503152640
>> expect [0, 64424509440]
>
> The line itself should explain itself, we're over allocating the device.
>
>> [26355.330819] BTRFS error (device sda2): block=537007243264 write
>> time tree block corruption detected
>
> Mind to dump the device tree?
>
> You can dump it with the following command, and the output contains
> nothing confidential, but only device extent layout:
>
> # btrfs ins dump-tree -t dev /dev/sda2

Sorry, you may also need to dump the super block:

# btrfs ins dump-super /dev/sda2

One of my concern is, the device used bytes may differ between super
block and device tree.

Thanks,
Qu
>
>>
>> and:
>>
>> # btrfs fi df /
>> Data, single: total=55.98GiB, used=43.04GiB
>> System, single: total=32.00MiB, used=16.00KiB
>> Metadata, single: total=2.00GiB, used=1.17GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> # btrfs fi show /
>> Label: none  uuid: 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
>>          Total devices 1 FS bytes used 44.22GiB
>>          devid    1 size 60.00GiB used 60.07GiB path /dev/sda2
>
> My current guess is, there is something wrong in the past, leaving the
> accounting wrong.
>
> It leaves some ghost used bytes, thus causing the problem.
>
> Your previous btrfs-check is able to detect that, but unfortunately we
> don't have the ability to repair it in btrfs-check right now.
>
> I'll work on repair ability soon, thus if you can keep the fs, there is
> a high chance to repair it.
>
> Thanks,
> Qu
>
>>
>> 60.07GiB used again!
>>
>> Regards,
>> Petros
>>
>>
>>
>>

Re: Write time tree block corruption detected

[auxsvr]

On Wednesday, 9 June 2021 04:05:18 EEST Qu Wenruo wrote:
> # btrfs ins dump-tree -t dev /dev/sda2
# btrfs ins dump-tree -t dev /dev/sda2
btrfs-progs v4.19.1 
device tree key (DEV_TREE ROOT_ITEM 0) 
leaf 181304705024 items 59 free space 11984 generation 1049986 owner DEV_TREE
leaf 181304705024 flags 0x1(WRITTEN) backref revision 1
fs uuid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
chunk uuid e09a0d7a-15ec-48ef-89fa-adaa5ec52990
        item 0 key (0 PERSISTENT_ITEM 1) itemoff 16243 itemsize 40
                persistent item objectid 0 offset 1
                device stats
                write_errs 0 read_errs 0 flush_errs 0 corruption_errs 0 generation 0
        item 1 key (1 DEV_EXTENT 22020096) itemoff 16195 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 229569986560 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 2 key (1 DEV_EXTENT 1095761920) itemoff 16147 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 537007226880 length 33554432
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 3 key (1 DEV_EXTENT 2169503744) itemoff 16099 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 227422502912 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 4 key (1 DEV_EXTENT 3243245568) itemoff 16051 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 226348761088 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 5 key (1 DEV_EXTENT 4316987392) itemoff 16003 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 225275019264 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 6 key (1 DEV_EXTENT 5390729216) itemoff 15955 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 224201277440 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 7 key (1 DEV_EXTENT 6464471040) itemoff 15907 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 223127535616 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 8 key (1 DEV_EXTENT 7538212864) itemoff 15859 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 222053793792 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 9 key (1 DEV_EXTENT 8611954688) itemoff 15811 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 220980051968 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 10 key (1 DEV_EXTENT 9685696512) itemoff 15763 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 315011104768 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 11 key (1 DEV_EXTENT 10759438336) itemoff 15715 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 295851524096 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 12 key (1 DEV_EXTENT 11833180160) itemoff 15667 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 437149237248 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 13 key (1 DEV_EXTENT 12906921984) itemoff 15619 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 187694055424 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 14 key (1 DEV_EXTENT 14014218240) itemoff 15571 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 181251604480 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 15 key (1 DEV_EXTENT 15087960064) itemoff 15523 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 328164442112 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 16 key (1 DEV_EXTENT 16161701888) itemoff 15475 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 379972485120 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 17 key (1 DEV_EXTENT 17235443712) itemoff 15427 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 531168755712 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 18 key (1 DEV_EXTENT 18309185536) itemoff 15379 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 457684549632 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 19 key (1 DEV_EXTENT 19382927360) itemoff 15331 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 347860893696 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 20 key (1 DEV_EXTENT 20456669184) itemoff 15283 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 345142984704 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 21 key (1 DEV_EXTENT 21530411008) itemoff 15235 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 532242497536 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 22 key (1 DEV_EXTENT 23644340224) itemoff 15187 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 183399088128 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 23 key (1 DEV_EXTENT 24718082048) itemoff 15139 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 537040781312 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 24 key (1 DEV_EXTENT 25791823872) itemoff 15091 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 208095150080 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 25 key (1 DEV_EXTENT 26865565696) itemoff 15043 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 207021408256 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 26 key (1 DEV_EXTENT 27939307520) itemoff 14995 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 205947666432 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 27 key (1 DEV_EXTENT 29013049344) itemoff 14947 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 204873924608 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 28 key (1 DEV_EXTENT 30086791168) itemoff 14899 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 203800182784 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 29 key (1 DEV_EXTENT 31160532992) itemoff 14851 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 202726440960 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 30 key (1 DEV_EXTENT 32234274816) itemoff 14803 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 201652699136 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 31 key (1 DEV_EXTENT 33308016640) itemoff 14755 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 200578957312 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 32 key (1 DEV_EXTENT 34381758464) itemoff 14707 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 199505215488 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 33 key (1 DEV_EXTENT 35455500288) itemoff 14659 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 198431473664 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 34 key (1 DEV_EXTENT 36529242112) itemoff 14611 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 460905775104 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 35 key (1 DEV_EXTENT 37602983936) itemoff 14563 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 196283990016 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 36 key (1 DEV_EXTENT 38676725760) itemoff 14515 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 259097886720 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 37 key (1 DEV_EXTENT 39750467584) itemoff 14467 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 184472829952 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 38 key (1 DEV_EXTENT 40824209408) itemoff 14419 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 195210248192 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 39 key (1 DEV_EXTENT 41897951232) itemoff 14371 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 182325346304 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 40 key (1 DEV_EXTENT 42971693056) itemoff 14323 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 194136506368 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 41 key (1 DEV_EXTENT 44045434880) itemoff 14275 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 186620313600 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 42 key (1 DEV_EXTENT 45119176704) itemoff 14227 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 188767797248 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 43 key (1 DEV_EXTENT 46192918528) itemoff 14179 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 189841539072 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 44 key (1 DEV_EXTENT 47266660352) itemoff 14131 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 193062764544 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 45 key (1 DEV_EXTENT 48340402176) itemoff 14083 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 191989022720 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 46 key (1 DEV_EXTENT 49414144000) itemoff 14035 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 190915280896 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 47 key (1 DEV_EXTENT 50487885824) itemoff 13987 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 280606277632 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 48 key (1 DEV_EXTENT 51561627648) itemoff 13939 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 471576084480 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 49 key (1 DEV_EXTENT 52635369472) itemoff 13891 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 282753761280 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 50 key (1 DEV_EXTENT 53709111296) itemoff 13843 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 523384127488 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 51 key (1 DEV_EXTENT 54782853120) itemoff 13795 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 524625641472 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 52 key (1 DEV_EXTENT 55856594944) itemoff 13747 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 526202699776 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 53 key (1 DEV_EXTENT 56930336768) itemoff 13699 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 533316239360 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 54 key (1 DEV_EXTENT 58004078592) itemoff 13651 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 264802140160 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 55 key (1 DEV_EXTENT 59077820416) itemoff 13603 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 433558913024 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 56 key (1 DEV_EXTENT 60151562240) itemoff 13555 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 534389981184 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 57 key (1 DEV_EXTENT 61225304064) itemoff 13507 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 290303508480 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
        item 58 key (1 DEV_EXTENT 62299045888) itemoff 13459 itemsize 48
                dev extent chunk_tree 3
                chunk_objectid 256 chunk_offset 535463723008 length 1073741824
                chunk_tree_uuid 00000000-0000-0000-0000-000000000000
total bytes 64424509440
bytes used 46488170496
uuid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4

> # btrfs ins dump-super /dev/sda2
# btrfs ins dump-super /dev/sda2
superblock: bytenr=65536, device=/dev/sda2
---------------------------------------------------------
csum_type               0 (crc32c)
csum_size               4
csum                    0x5c8acbc2 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    44c67fa4-e2c4-4da5-9d07-98959ff77bc4
metadata_uuid           44c67fa4-e2c4-4da5-9d07-98959ff77bc4
label
generation              1050015
root                    181625438208
sys_array_size          97
chunk_root_generation   1047576
root_level              1
chunk_root              537007226880
chunk_root_level        0
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             64424509440
bytes_used              46488170496
sectorsize              4096
nodesize                16384
leafsize (deprecated)           16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x161
                        ( MIXED_BACKREF |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
cache_generation        1050015
uuid_tree_generation    1047664
dev_item.uuid           abc9d19a-d9d1-40a4-9bd5-f28654800249
dev_item.fsid           44c67fa4-e2c4-4da5-9d07-98959ff77bc4 [match]
dev_item.type           0
dev_item.total_bytes    64424509440
dev_item.bytes_used     63451430912
dev_item.io_align       0
dev_item.io_width       0
dev_item.sector_size    0
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0

Thanks,
Petros

Re: Write time tree block corruption detected

[Qu Wenruo]

Confirmed that the result in device tree is indeed smaller than the
superblock bytes_used.

Would you please try to compile the following branch of btrfs-progs?

https://github.com/adam900710/btrfs-progs/tree/device_size_fix

And run "./btrfs check --repair --force /dev/sda2" for the newly
compiled btrfs-progs to see if it solves your problem.

Thanks,
Qu

On 2021/6/9 下午4:04, auxsvr wrote:
> On Wednesday, 9 June 2021 04:05:18 EEST Qu Wenruo wrote:
>> # btrfs ins dump-tree -t dev /dev/sda2
> # btrfs ins dump-tree -t dev /dev/sda2
> btrfs-progs v4.19.1
> device tree key (DEV_TREE ROOT_ITEM 0)
> leaf 181304705024 items 59 free space 11984 generation 1049986 owner DEV_TREE
> leaf 181304705024 flags 0x1(WRITTEN) backref revision 1
> fs uuid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
> chunk uuid e09a0d7a-15ec-48ef-89fa-adaa5ec52990
>          item 0 key (0 PERSISTENT_ITEM 1) itemoff 16243 itemsize 40
>                  persistent item objectid 0 offset 1
>                  device stats
>                  write_errs 0 read_errs 0 flush_errs 0 corruption_errs 0 generation 0
>          item 1 key (1 DEV_EXTENT 22020096) itemoff 16195 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 229569986560 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 2 key (1 DEV_EXTENT 1095761920) itemoff 16147 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 537007226880 length 33554432
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 3 key (1 DEV_EXTENT 2169503744) itemoff 16099 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 227422502912 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 4 key (1 DEV_EXTENT 3243245568) itemoff 16051 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 226348761088 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 5 key (1 DEV_EXTENT 4316987392) itemoff 16003 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 225275019264 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 6 key (1 DEV_EXTENT 5390729216) itemoff 15955 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 224201277440 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 7 key (1 DEV_EXTENT 6464471040) itemoff 15907 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 223127535616 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 8 key (1 DEV_EXTENT 7538212864) itemoff 15859 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 222053793792 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 9 key (1 DEV_EXTENT 8611954688) itemoff 15811 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 220980051968 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 10 key (1 DEV_EXTENT 9685696512) itemoff 15763 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 315011104768 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 11 key (1 DEV_EXTENT 10759438336) itemoff 15715 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 295851524096 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 12 key (1 DEV_EXTENT 11833180160) itemoff 15667 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 437149237248 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 13 key (1 DEV_EXTENT 12906921984) itemoff 15619 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 187694055424 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 14 key (1 DEV_EXTENT 14014218240) itemoff 15571 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 181251604480 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 15 key (1 DEV_EXTENT 15087960064) itemoff 15523 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 328164442112 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 16 key (1 DEV_EXTENT 16161701888) itemoff 15475 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 379972485120 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 17 key (1 DEV_EXTENT 17235443712) itemoff 15427 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 531168755712 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 18 key (1 DEV_EXTENT 18309185536) itemoff 15379 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 457684549632 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 19 key (1 DEV_EXTENT 19382927360) itemoff 15331 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 347860893696 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 20 key (1 DEV_EXTENT 20456669184) itemoff 15283 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 345142984704 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 21 key (1 DEV_EXTENT 21530411008) itemoff 15235 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 532242497536 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 22 key (1 DEV_EXTENT 23644340224) itemoff 15187 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 183399088128 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 23 key (1 DEV_EXTENT 24718082048) itemoff 15139 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 537040781312 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 24 key (1 DEV_EXTENT 25791823872) itemoff 15091 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 208095150080 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 25 key (1 DEV_EXTENT 26865565696) itemoff 15043 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 207021408256 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 26 key (1 DEV_EXTENT 27939307520) itemoff 14995 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 205947666432 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 27 key (1 DEV_EXTENT 29013049344) itemoff 14947 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 204873924608 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 28 key (1 DEV_EXTENT 30086791168) itemoff 14899 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 203800182784 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 29 key (1 DEV_EXTENT 31160532992) itemoff 14851 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 202726440960 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 30 key (1 DEV_EXTENT 32234274816) itemoff 14803 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 201652699136 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 31 key (1 DEV_EXTENT 33308016640) itemoff 14755 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 200578957312 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 32 key (1 DEV_EXTENT 34381758464) itemoff 14707 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 199505215488 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 33 key (1 DEV_EXTENT 35455500288) itemoff 14659 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 198431473664 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 34 key (1 DEV_EXTENT 36529242112) itemoff 14611 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 460905775104 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 35 key (1 DEV_EXTENT 37602983936) itemoff 14563 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 196283990016 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 36 key (1 DEV_EXTENT 38676725760) itemoff 14515 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 259097886720 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 37 key (1 DEV_EXTENT 39750467584) itemoff 14467 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 184472829952 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 38 key (1 DEV_EXTENT 40824209408) itemoff 14419 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 195210248192 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 39 key (1 DEV_EXTENT 41897951232) itemoff 14371 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 182325346304 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 40 key (1 DEV_EXTENT 42971693056) itemoff 14323 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 194136506368 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 41 key (1 DEV_EXTENT 44045434880) itemoff 14275 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 186620313600 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 42 key (1 DEV_EXTENT 45119176704) itemoff 14227 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 188767797248 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 43 key (1 DEV_EXTENT 46192918528) itemoff 14179 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 189841539072 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 44 key (1 DEV_EXTENT 47266660352) itemoff 14131 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 193062764544 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 45 key (1 DEV_EXTENT 48340402176) itemoff 14083 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 191989022720 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 46 key (1 DEV_EXTENT 49414144000) itemoff 14035 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 190915280896 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 47 key (1 DEV_EXTENT 50487885824) itemoff 13987 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 280606277632 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 48 key (1 DEV_EXTENT 51561627648) itemoff 13939 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 471576084480 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 49 key (1 DEV_EXTENT 52635369472) itemoff 13891 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 282753761280 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 50 key (1 DEV_EXTENT 53709111296) itemoff 13843 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 523384127488 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 51 key (1 DEV_EXTENT 54782853120) itemoff 13795 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 524625641472 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 52 key (1 DEV_EXTENT 55856594944) itemoff 13747 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 526202699776 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 53 key (1 DEV_EXTENT 56930336768) itemoff 13699 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 533316239360 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 54 key (1 DEV_EXTENT 58004078592) itemoff 13651 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 264802140160 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 55 key (1 DEV_EXTENT 59077820416) itemoff 13603 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 433558913024 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 56 key (1 DEV_EXTENT 60151562240) itemoff 13555 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 534389981184 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 57 key (1 DEV_EXTENT 61225304064) itemoff 13507 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 290303508480 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>          item 58 key (1 DEV_EXTENT 62299045888) itemoff 13459 itemsize 48
>                  dev extent chunk_tree 3
>                  chunk_objectid 256 chunk_offset 535463723008 length 1073741824
>                  chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> total bytes 64424509440
> bytes used 46488170496
> uuid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
>
>> # btrfs ins dump-super /dev/sda2
> # btrfs ins dump-super /dev/sda2
> superblock: bytenr=65536, device=/dev/sda2
> ---------------------------------------------------------
> csum_type               0 (crc32c)
> csum_size               4
> csum                    0x5c8acbc2 [match]
> bytenr                  65536
> flags                   0x1
>                          ( WRITTEN )
> magic                   _BHRfS_M [match]
> fsid                    44c67fa4-e2c4-4da5-9d07-98959ff77bc4
> metadata_uuid           44c67fa4-e2c4-4da5-9d07-98959ff77bc4
> label
> generation              1050015
> root                    181625438208
> sys_array_size          97
> chunk_root_generation   1047576
> root_level              1
> chunk_root              537007226880
> chunk_root_level        0
> log_root                0
> log_root_transid        0
> log_root_level          0
> total_bytes             64424509440
> bytes_used              46488170496
> sectorsize              4096
> nodesize                16384
> leafsize (deprecated)           16384
> stripesize              4096
> root_dir                6
> num_devices             1
> compat_flags            0x0
> compat_ro_flags         0x0
> incompat_flags          0x161
>                          ( MIXED_BACKREF |
>                            BIG_METADATA |
>                            EXTENDED_IREF |
>                            SKINNY_METADATA )
> cache_generation        1050015
> uuid_tree_generation    1047664
> dev_item.uuid           abc9d19a-d9d1-40a4-9bd5-f28654800249
> dev_item.fsid           44c67fa4-e2c4-4da5-9d07-98959ff77bc4 [match]
> dev_item.type           0
> dev_item.total_bytes    64424509440
> dev_item.bytes_used     63451430912
> dev_item.io_align       0
> dev_item.io_width       0
> dev_item.sector_size    0
> dev_item.devid          1
> dev_item.dev_group      0
> dev_item.seek_speed     0
> dev_item.bandwidth      0
> dev_item.generation     0
>
> Thanks,
> Petros
>
>

Re: Write time tree block corruption detected

[auxsvr]

On Wednesday, 9 June 2021 11:32:57 EEST Qu Wenruo wrote:
> Would you please try to compile the following branch of btrfs-progs?
> 
> https://github.com/adam900710/btrfs-progs/tree/device_size_fix
> 
> And run "./btrfs check --repair --force /dev/sda2" for the newly
> compiled btrfs-progs to see if it solves your problem.

Ran the repair and everything seems to be fine so far.

Thank you,
Petros

Write time tree block corruption detected

[Spencer Collyer]

I have just received a 'write time tree block corruption detected' message on a BTRFS fs I use. As per
https://btrfs.wiki.kernel.org/index.php/Tree-checker it mentions sending a mail to this mailing list for this case.

The dmesg output from the error onwards is as follows:

[17434.620469] BTRFS error (device dm-1): block=13642806624256 write time tree block corruption detected
[17435.048842] BTRFS: error (device dm-1) in btrfs_commit_transaction:2323: errno=-5 IO failure (Error while writing out transaction)
[17435.048848] BTRFS info (device dm-1): forced readonly
[17435.048851] BTRFS warning (device dm-1): Skipping commit of aborted transaction.
[17435.048853] BTRFS: error (device dm-1) in cleanup_transaction:1894: errno=-5 IO failure

As can be seen, the fs was forced into readonly mode. I restarted the box that the fs was mounted on and am able to write to the fs again (tried 'touch'ing a non-existent file and that worked), but I'm reluctant to do any more with it until I know it is safe to do so.

Is there anything I should do to ensure the fs is OK? The page I quoted earlier said not to run 'btrfs check --repair' unless told to by a developer.

Thanks for your attention.

Spencer

Re: Write time tree block corruption detected

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1443 bytes --]



On 2020/8/12 下午9:49, Spencer Collyer wrote:
> I have just received a 'write time tree block corruption detected' message on a BTRFS fs I use. As per
> https://btrfs.wiki.kernel.org/index.php/Tree-checker it mentions sending a mail to this mailing list for this case.
> 
> The dmesg output from the error onwards is as follows:
> 
> [17434.620469] BTRFS error (device dm-1): block=13642806624256 write time tree block corruption detected

What's the line before this line?

That's the most important line, and it's unfortunately not there...

Thanks,
Qu

> [17435.048842] BTRFS: error (device dm-1) in btrfs_commit_transaction:2323: errno=-5 IO failure (Error while writing out transaction)
> [17435.048848] BTRFS info (device dm-1): forced readonly
> [17435.048851] BTRFS warning (device dm-1): Skipping commit of aborted transaction.
> [17435.048853] BTRFS: error (device dm-1) in cleanup_transaction:1894: errno=-5 IO failure
> 
> As can be seen, the fs was forced into readonly mode. I restarted the box that the fs was mounted on and am able to write to the fs again (tried 'touch'ing a non-existent file and that worked), but I'm reluctant to do any more with it until I know it is safe to do so.
> 
> Is there anything I should do to ensure the fs is OK? The page I quoted earlier said not to run 'btrfs check --repair' unless told to by a developer.
> 
> Thanks for your attention.
> 
> Spencer
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Write time tree block corruption detected

[Spencer Collyer]

On Wed, 12 Aug 2020 22:15:23 +0800, Qu Wenruo wrote:
> On 2020/8/12 下午9:49, Spencer Collyer wrote:
> > I have just received a 'write time tree block corruption detected'
> > message on a BTRFS fs I use. As per
> > https://btrfs.wiki.kernel.org/index.php/Tree-checker it mentions
> > sending a mail to this mailing list for this case.
> > 
> > The dmesg output from the error onwards is as follows:
> > 
> > [17434.620469] BTRFS error (device dm-1): block=13642806624256
> > write time tree block corruption detected  
> 
> What's the line before this line?
> 
> That's the most important line, and it's unfortunately not there...
> 
> Thanks,
> Qu

Oh, didn't realise that. Unfortunately it has gone from the dmesg ring buffer, so I can't go back to it.  If it happens again I'll take a copy of the full dmesg output so I can go back to it.

The page I mentioned previously (https://btrfs.wiki.kernel.org/index.php/Tree-checker) just mentions to report the error to this mailing list. Might be an idea to expand that line to explain what needs reporting.

Rereading the linked page, I notice that it says that if it is a write error that is prevented then the fs should still be OK, and you can run 'btrfs check --readonly' to check that. It is 'btrfs check --repair' that it says to not run unless told to by a developer. So I'm going to run the '--readonly' check and if that looks good I'll try re-running the command that caused the failure - does that seem reasonable?

Thanks for your attention,

Spencer

Re: Write time tree block corruption detected

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2186 bytes --]



On 2020/8/13 下午1:53, Spencer Collyer wrote:
> On Wed, 12 Aug 2020 22:15:23 +0800, Qu Wenruo wrote:
>> On 2020/8/12 下午9:49, Spencer Collyer wrote:
>>> I have just received a 'write time tree block corruption detected'
>>> message on a BTRFS fs I use. As per
>>> https://btrfs.wiki.kernel.org/index.php/Tree-checker it mentions
>>> sending a mail to this mailing list for this case.
>>>
>>> The dmesg output from the error onwards is as follows:
>>>
>>> [17434.620469] BTRFS error (device dm-1): block=13642806624256
>>> write time tree block corruption detected  
>>
>> What's the line before this line?
>>
>> That's the most important line, and it's unfortunately not there...
>>
>> Thanks,
>> Qu
> 
> Oh, didn't realise that. Unfortunately it has gone from the dmesg ring buffer, so I can't go back to it.  If it happens again I'll take a copy of the full dmesg output so I can go back to it.
> 
> The page I mentioned previously (https://btrfs.wiki.kernel.org/index.php/Tree-checker) just mentions to report the error to this mailing list. Might be an idea to expand that line to explain what needs reporting.
> 
> Rereading the linked page, I notice that it says that if it is a write error that is prevented then the fs should still be OK, and you can run 'btrfs check --readonly' to check that. It is 'btrfs check --repair' that it says to not run unless told to by a developer. So I'm going to run the '--readonly' check and if that looks good I'll try re-running the command that caused the failure - does that seem reasonable?

Running btrfs check is always a good idea to ensure nothing is wrong.

Write time tree checker mostly detects the following types of errors:
- Memory bit flip
  Then you'd better to at least run a memtest to ensure your memory is
  fine.

- Btrfs internal bug
  One example is the csum item overlap, which is a bug and soon get
  fixed.

Although you can definitely try the same command, I doubt if it can
reproduce the problem.

But anyway, if you can reproduce the bug, it would definitely be helpful
for us to look into.

Thanks,
Qu

> 
> Thanks for your attention,
> 
> Spencer
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

write time tree block corruption detected

[Kenneth Topp]

I've been running 5.4.8 for 10 days and this popped up:

[917767.192190] BTRFS error (device dm-19): block=21374679056384 write 
time tree block corruption detected
[917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104: 
errno=-5 IO failure
[917767.193438] BTRFS info (device dm-19): forced readonly


This was a btrfs filesystem created on 5.4.8 with two 4kn drives, each 
with a single whole disk dm-crypt partition.  The fs was created with 
mkfs -O no-holes -d single -m raid1 and mounted with 
clear_cache,space_cache=v2.

After that issue, I tried to update my backups, but it didn't go very 
well:

[918052.104923] BTRFS warning (device dm-19): dm-19 checksum verify 
failed on 21374567694336 wanted 6c2341bd found 640f30e8 level 0
[918052.121801] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918052.121818] BTRFS info (device dm-19): no csum found for inode 
219677 start 3397271552
[918052.138406] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918052.139080] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918052.139094] BTRFS info (device dm-19): no csum found for inode 
219677 start 3397275648
[918052.139768] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918052.140158] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918052.140171] BTRFS info (device dm-19): no csum found for inode 
219677 start 3397279744
[918052.140532] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918052.140868] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918052.140880] BTRFS info (device dm-19): no csum found for inode 
219677 start 3397283840
[918052.159794] BTRFS warning (device dm-19): csum failed root 5 ino 
219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
[918052.213959] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918052.216411] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918052.216418] BTRFS info (device dm-19): no csum found for inode 
219677 start 3397271552
[918052.216570] BTRFS warning (device dm-19): csum failed root 5 ino 
219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
[918052.216790] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918052.217037] BTRFS info (device dm-19): no csum found for inode 
219677 start 3397271552
[918052.217165] BTRFS warning (device dm-19): csum failed root 5 ino 
219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
[918062.496366] BTRFS warning (device dm-19): dm-19 checksum verify 
failed on 20043100192768 wanted 8bf4c1c9 found 23b8dea8 level 0
[918062.504832] verify_parent_transid: 1 callbacks suppressed
[918062.504836] BTRFS error (device dm-19): parent transid verify failed 
on 20043100192768 wanted 45126 found 45104
[918062.504848] BTRFS info (device dm-19): no csum found for inode 
5464098 start 28672
[918062.506174] BTRFS error (device dm-19): parent transid verify failed 
on 20043100241920 wanted 45126 found 45104
[918062.509823] BTRFS warning (device dm-19): csum failed root 5 ino 
5464098 off 28672 csum 0xa0f9eaf7 expected csum 0x00000000 mirror 1
[918062.513826] BTRFS error (device dm-19): parent transid verify failed 
on 20043100241920 wanted 45126 found 45104
[918062.513838] BTRFS info (device dm-19): no csum found for inode 
5464098 start 49152
[918062.519723] BTRFS warning (device dm-19): csum failed root 5 ino 
5464098 off 49152 csum 0x76421bb0 expected csum 0x00000000 mirror 1
[918062.530287] BTRFS error (device dm-19): parent transid verify failed 
on 20043100192768 wanted 45126 found 45104
[918062.530866] BTRFS error (device dm-19): parent transid verify failed 
on 20043100192768 wanted 45126 found 45104
[918062.530874] BTRFS info (device dm-19): no csum found for inode 
5464098 start 65536
[918062.539238] BTRFS error (device dm-19): parent transid verify failed 
on 20043100192768 wanted 45126 found 45104
[918062.539560] BTRFS error (device dm-19): parent transid verify failed 
on 20043100192768 wanted 45126 found 45104
[918062.539566] BTRFS info (device dm-19): no csum found for inode 
5464098 start 69632
[918062.540020] BTRFS error (device dm-19): parent transid verify failed 
on 20043100192768 wanted 45126 found 45104
[918062.540261] BTRFS error (device dm-19): parent transid verify failed 
on 20043100192768 wanted 45126 found 45104
[918062.540269] BTRFS info (device dm-19): no csum found for inode 
5464098 start 77824
[918062.540965] BTRFS error (device dm-19): parent transid verify failed 
on 20043100192768 wanted 45126 found 45104
[918062.541199] BTRFS info (device dm-19): no csum found for inode 
5464098 start 114688
[918062.541753] BTRFS info (device dm-19): no csum found for inode 
5464098 start 118784
[918062.544496] BTRFS warning (device dm-19): csum failed root 5 ino 
5464098 off 65536 csum 0xdcb45a27 expected csum 0x00000000 mirror 1
[918062.547421] BTRFS warning (device dm-19): csum failed root 5 ino 
5464098 off 114688 csum 0x1d5a0f30 expected csum 0x00000000 mirror 1
[918062.549968] BTRFS warning (device dm-19): csum failed root 5 ino 
5464098 off 69632 csum 0x24dd186c expected csum 0x00000000 mirror 1
[918062.572502] BTRFS warning (device dm-19): csum failed root 5 ino 
5464098 off 77824 csum 0xba341c47 expected csum 0x00000000 mirror 1
[918062.575300] BTRFS warning (device dm-19): csum failed root 5 ino 
5464098 off 118784 csum 0x519e7044 expected csum 0x00000000 mirror 1
[918062.584005] BTRFS info (device dm-19): no csum found for inode 
5464098 start 172032
[918062.586782] BTRFS info (device dm-19): no csum found for inode 
5464098 start 221184
[918062.587106] BTRFS warning (device dm-19): csum failed root 5 ino 
5464098 off 221184 csum 0xc9c50b2d expected csum 0x00000000 mirror 1
[918062.590739] BTRFS info (device dm-19): no csum found for inode 
5464098 start 450560
[918062.591255] BTRFS warning (device dm-19): csum failed root 5 ino 
5464098 off 450560 csum 0xd2e5e3eb expected csum 0x00000000 mirror 1
[918062.601361] BTRFS warning (device dm-19): dm-19 checksum verify 
failed on 20043100258304 wanted 61da40d5 found 204cd10a level 0
[918062.602875] BTRFS warning (device dm-19): csum failed root 5 ino 
5464098 off 503808 csum 0x183beedc expected csum 0x00000000 mirror 1
[918085.820716] verify_parent_transid: 42 callbacks suppressed
[918085.820730] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918085.837964] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918085.837992] __btrfs_lookup_bio_sums: 17 callbacks suppressed
[918085.837994] BTRFS info (device dm-19): no csum found for inode 
5468073 start 159711232
[918085.838459] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918085.838987] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918085.838995] BTRFS info (device dm-19): no csum found for inode 
5468073 start 159715328
[918085.839349] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918085.840054] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918085.840065] BTRFS info (device dm-19): no csum found for inode 
5468073 start 159719424
[918085.840402] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918085.840943] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918085.840951] BTRFS info (device dm-19): no csum found for inode 
5468073 start 159723520
[918085.850438] btrfs_print_data_csum_error: 17 callbacks suppressed
[918085.850442] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
[918085.959941] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918085.960447] BTRFS error (device dm-19): parent transid verify failed 
on 21374567694336 wanted 45126 found 45104
[918085.960452] BTRFS info (device dm-19): no csum found for inode 
5468073 start 159711232
[918085.960624] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
[918085.961220] BTRFS info (device dm-19): no csum found for inode 
5468073 start 159711232
[918085.961341] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
[918087.026984] BTRFS info (device dm-19): no csum found for inode 
5468073 start 193855488
[918087.028152] BTRFS info (device dm-19): no csum found for inode 
5468073 start 193859584
[918087.028598] BTRFS info (device dm-19): no csum found for inode 
5468073 start 193863680
[918087.029005] BTRFS info (device dm-19): no csum found for inode 
5468073 start 193867776
[918087.042810] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
[918087.054812] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 193904640 csum 0x279aea06 expected csum 0x00000000 mirror 1
[918087.071004] BTRFS warning (device dm-19): dm-19 checksum verify 
failed on 20043549310976 wanted fcc4f73a found 3d3d8e83 level 0
[918087.103303] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 193937408 csum 0x1fedc264 expected csum 0x00000000 mirror 1
[918087.118559] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
[918087.131234] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
[918087.132513] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
[918087.135749] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
[918119.990544] BTRFS warning (device dm-19): dm-19 checksum verify 
failed on 20043135205376 wanted 79d70254 found 93ac2b61 level 0
[918120.010421] verify_parent_transid: 105 callbacks suppressed
[918120.010428] BTRFS error (device dm-19): parent transid verify failed 
on 20043135205376 wanted 45126 found 45104
[918120.010565] __btrfs_lookup_bio_sums: 48 callbacks suppressed
[918120.010567] BTRFS info (device dm-19): no csum found for inode 
5468073 start 678543360
[918120.028733] btrfs_print_data_csum_error: 5 callbacks suppressed
[918120.028735] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
[918120.040591] BTRFS error (device dm-19): parent transid verify failed 
on 20043135205376 wanted 45126 found 45104
[918120.041097] BTRFS error (device dm-19): parent transid verify failed 
on 20043135205376 wanted 45126 found 45104
[918120.041112] BTRFS info (device dm-19): no csum found for inode 
5468073 start 678543360
[918120.041440] BTRFS warning (device dm-19): csum failed root 5 ino 
5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1


Fortunately, after a reboot, everything appears healthy, no file 
corruption that I can observe, but I'm still running some checks.

I'm not sure what the write tree is or if this is related to the read 
tree issues mentioned on the mailing list, any advice/pointers would be 
most appreciated.  One thing I can note is that I've started to write 
heavily to this filesystem via nfs export (the nfs client happens to be 
a vm on the same machine, fwiw).

Re: write time tree block corruption detected

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 13139 bytes --]



On 2020/1/17 下午4:21, Kenneth Topp wrote:
> I've been running 5.4.8 for 10 days and this popped up:
> 
> [917767.192190] BTRFS error (device dm-19): block=21374679056384 write
> time tree block corruption detected
> [917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
> errno=-5 IO failure
> [917767.193438] BTRFS info (device dm-19): forced readonly

You missed the most important line. One line before "write time tree
block corruption detected" is the most important line.

> 
> 
> This was a btrfs filesystem created on 5.4.8 with two 4kn drives, each
> with a single whole disk dm-crypt partition.  The fs was created with
> mkfs -O no-holes -d single -m raid1 and mounted with
> clear_cache,space_cache=v2.
> 
> After that issue, I tried to update my backups, but it didn't go very well:
> 
> [918052.104923] BTRFS warning (device dm-19): dm-19 checksum verify
> failed on 21374567694336 wanted 6c2341bd found 640f30e8 level 0
> [918052.121801] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.121818] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397271552
> [918052.138406] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.139080] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.139094] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397275648
> [918052.139768] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.140158] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.140171] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397279744
> [918052.140532] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.140868] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.140880] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397283840
> [918052.159794] BTRFS warning (device dm-19): csum failed root 5 ino
> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
> [918052.213959] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.216411] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.216418] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397271552
> [918052.216570] BTRFS warning (device dm-19): csum failed root 5 ino
> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
> [918052.216790] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.217037] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397271552
> [918052.217165] BTRFS warning (device dm-19): csum failed root 5 ino
> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
> [918062.496366] BTRFS warning (device dm-19): dm-19 checksum verify
> failed on 20043100192768 wanted 8bf4c1c9 found 23b8dea8 level 0
> [918062.504832] verify_parent_transid: 1 callbacks suppressed
> [918062.504836] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.504848] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 28672
> [918062.506174] BTRFS error (device dm-19): parent transid verify failed
> on 20043100241920 wanted 45126 found 45104
> [918062.509823] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 28672 csum 0xa0f9eaf7 expected csum 0x00000000 mirror 1
> [918062.513826] BTRFS error (device dm-19): parent transid verify failed
> on 20043100241920 wanted 45126 found 45104
> [918062.513838] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 49152
> [918062.519723] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 49152 csum 0x76421bb0 expected csum 0x00000000 mirror 1
> [918062.530287] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.530866] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.530874] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 65536
> [918062.539238] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.539560] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.539566] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 69632
> [918062.540020] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.540261] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.540269] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 77824
> [918062.540965] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.541199] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 114688
> [918062.541753] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 118784
> [918062.544496] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 65536 csum 0xdcb45a27 expected csum 0x00000000 mirror 1
> [918062.547421] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 114688 csum 0x1d5a0f30 expected csum 0x00000000 mirror 1
> [918062.549968] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 69632 csum 0x24dd186c expected csum 0x00000000 mirror 1
> [918062.572502] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 77824 csum 0xba341c47 expected csum 0x00000000 mirror 1
> [918062.575300] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 118784 csum 0x519e7044 expected csum 0x00000000 mirror 1
> [918062.584005] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 172032
> [918062.586782] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 221184
> [918062.587106] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 221184 csum 0xc9c50b2d expected csum 0x00000000 mirror 1
> [918062.590739] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 450560
> [918062.591255] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 450560 csum 0xd2e5e3eb expected csum 0x00000000 mirror 1
> [918062.601361] BTRFS warning (device dm-19): dm-19 checksum verify
> failed on 20043100258304 wanted 61da40d5 found 204cd10a level 0
> [918062.602875] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 503808 csum 0x183beedc expected csum 0x00000000 mirror 1
> [918085.820716] verify_parent_transid: 42 callbacks suppressed
> [918085.820730] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.837964] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.837992] __btrfs_lookup_bio_sums: 17 callbacks suppressed
> [918085.837994] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159711232
> [918085.838459] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.838987] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.838995] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159715328
> [918085.839349] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.840054] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.840065] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159719424
> [918085.840402] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.840943] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.840951] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159723520
> [918085.850438] btrfs_print_data_csum_error: 17 callbacks suppressed
> [918085.850442] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
> [918085.959941] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.960447] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.960452] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159711232
> [918085.960624] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
> [918085.961220] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159711232
> [918085.961341] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
> [918087.026984] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 193855488
> [918087.028152] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 193859584
> [918087.028598] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 193863680
> [918087.029005] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 193867776
> [918087.042810] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
> [918087.054812] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193904640 csum 0x279aea06 expected csum 0x00000000 mirror 1
> [918087.071004] BTRFS warning (device dm-19): dm-19 checksum verify
> failed on 20043549310976 wanted fcc4f73a found 3d3d8e83 level 0
> [918087.103303] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193937408 csum 0x1fedc264 expected csum 0x00000000 mirror 1
> [918087.118559] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
> [918087.131234] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
> [918087.132513] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
> [918087.135749] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
> [918119.990544] BTRFS warning (device dm-19): dm-19 checksum verify
> failed on 20043135205376 wanted 79d70254 found 93ac2b61 level 0
> [918120.010421] verify_parent_transid: 105 callbacks suppressed
> [918120.010428] BTRFS error (device dm-19): parent transid verify failed
> on 20043135205376 wanted 45126 found 45104
> [918120.010565] __btrfs_lookup_bio_sums: 48 callbacks suppressed
> [918120.010567] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 678543360
> [918120.028733] btrfs_print_data_csum_error: 5 callbacks suppressed
> [918120.028735] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
> [918120.040591] BTRFS error (device dm-19): parent transid verify failed
> on 20043135205376 wanted 45126 found 45104
> [918120.041097] BTRFS error (device dm-19): parent transid verify failed
> on 20043135205376 wanted 45126 found 45104
> [918120.041112] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 678543360
> [918120.041440] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1

Since you haven't rebooted, all these mismatch can be caused by aborted
transaction.

> 
> 
> Fortunately, after a reboot, everything appears healthy, no file
> corruption that I can observe, but I'm still running some checks.

That's great, write time tree checker prevented corruption.

But still, btrfs check, and scrub recommended.

And the most important thing, please provide the missing key log line.

Thanks,
Qu

> 
> I'm not sure what the write tree is or if this is related to the read
> tree issues mentioned on the mailing list, any advice/pointers would be
> most appreciated.  One thing I can note is that I've started to write
> heavily to this filesystem via nfs export (the nfs client happens to be
> a vm on the same machine, fwiw).
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: write time tree block corruption detected

[Kenneth Topp]

On 2020-01-17 03:33, Qu Wenruo wrote:
> On 2020/1/17 下午4:21, Kenneth Topp wrote:
>> I've been running 5.4.8 for 10 days and this popped up:
>> 
>> [917767.192190] BTRFS error (device dm-19): block=21374679056384 write
>> time tree block corruption detected
>> [917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
>> errno=-5 IO failure
>> [917767.193438] BTRFS info (device dm-19): forced readonly
> 
> You missed the most important line. One line before "write time tree
> block corruption detected" is the most important line.
> 

sorry, here is the missing line.

[917767.192187] BTRFS critical (device dm-19): corrupt leaf: 
root=18446744073709551610 block=21374679056384 slot=174 ino=219743 
file_offset=1385357312, file extent end range (1460178944) g
oes beyond start offset (1385385984) of the next file extent
[917767.192190] BTRFS error (device dm-19): block=21374679056384 write 
time tree block corruption detected
[917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104: 
errno=-5 IO failure
[917767.193438] BTRFS info (device dm-19): forced readonly

thanks for responding so quickly!

ken

>> 
>> 
>> This was a btrfs filesystem created on 5.4.8 with two 4kn drives, each
>> with a single whole disk dm-crypt partition.  The fs was created with
>> mkfs -O no-holes -d single -m raid1 and mounted with
>> clear_cache,space_cache=v2.
>> 
>> After that issue, I tried to update my backups, but it didn't go very 
>> well:
>> 
>> [918052.104923] BTRFS warning (device dm-19): dm-19 checksum verify
>> failed on 21374567694336 wanted 6c2341bd found 640f30e8 level 0
>> [918052.121801] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.121818] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397271552
>> [918052.138406] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.139080] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.139094] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397275648
>> [918052.139768] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.140158] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.140171] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397279744
>> [918052.140532] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.140868] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.140880] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397283840
>> [918052.159794] BTRFS warning (device dm-19): csum failed root 5 ino
>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 
>> 1
>> [918052.213959] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.216411] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.216418] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397271552
>> [918052.216570] BTRFS warning (device dm-19): csum failed root 5 ino
>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 
>> 1
>> [918052.216790] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.217037] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397271552
>> [918052.217165] BTRFS warning (device dm-19): csum failed root 5 ino
>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 
>> 1
>> [918062.496366] BTRFS warning (device dm-19): dm-19 checksum verify
>> failed on 20043100192768 wanted 8bf4c1c9 found 23b8dea8 level 0
>> [918062.504832] verify_parent_transid: 1 callbacks suppressed
>> [918062.504836] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.504848] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 28672
>> [918062.506174] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043100241920 wanted 45126 found 45104
>> [918062.509823] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 28672 csum 0xa0f9eaf7 expected csum 0x00000000 mirror 1
>> [918062.513826] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043100241920 wanted 45126 found 45104
>> [918062.513838] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 49152
>> [918062.519723] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 49152 csum 0x76421bb0 expected csum 0x00000000 mirror 1
>> [918062.530287] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.530866] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.530874] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 65536
>> [918062.539238] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.539560] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.539566] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 69632
>> [918062.540020] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.540261] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.540269] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 77824
>> [918062.540965] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.541199] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 114688
>> [918062.541753] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 118784
>> [918062.544496] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 65536 csum 0xdcb45a27 expected csum 0x00000000 mirror 1
>> [918062.547421] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 114688 csum 0x1d5a0f30 expected csum 0x00000000 mirror 1
>> [918062.549968] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 69632 csum 0x24dd186c expected csum 0x00000000 mirror 1
>> [918062.572502] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 77824 csum 0xba341c47 expected csum 0x00000000 mirror 1
>> [918062.575300] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 118784 csum 0x519e7044 expected csum 0x00000000 mirror 1
>> [918062.584005] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 172032
>> [918062.586782] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 221184
>> [918062.587106] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 221184 csum 0xc9c50b2d expected csum 0x00000000 mirror 1
>> [918062.590739] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 450560
>> [918062.591255] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 450560 csum 0xd2e5e3eb expected csum 0x00000000 mirror 1
>> [918062.601361] BTRFS warning (device dm-19): dm-19 checksum verify
>> failed on 20043100258304 wanted 61da40d5 found 204cd10a level 0
>> [918062.602875] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 503808 csum 0x183beedc expected csum 0x00000000 mirror 1
>> [918085.820716] verify_parent_transid: 42 callbacks suppressed
>> [918085.820730] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.837964] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.837992] __btrfs_lookup_bio_sums: 17 callbacks suppressed
>> [918085.837994] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159711232
>> [918085.838459] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.838987] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.838995] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159715328
>> [918085.839349] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.840054] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.840065] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159719424
>> [918085.840402] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.840943] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.840951] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159723520
>> [918085.850438] btrfs_print_data_csum_error: 17 callbacks suppressed
>> [918085.850442] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 
>> 1
>> [918085.959941] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.960447] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.960452] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159711232
>> [918085.960624] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 
>> 1
>> [918085.961220] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159711232
>> [918085.961341] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 
>> 1
>> [918087.026984] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 193855488
>> [918087.028152] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 193859584
>> [918087.028598] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 193863680
>> [918087.029005] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 193867776
>> [918087.042810] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 
>> 1
>> [918087.054812] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193904640 csum 0x279aea06 expected csum 0x00000000 mirror 
>> 1
>> [918087.071004] BTRFS warning (device dm-19): dm-19 checksum verify
>> failed on 20043549310976 wanted fcc4f73a found 3d3d8e83 level 0
>> [918087.103303] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193937408 csum 0x1fedc264 expected csum 0x00000000 mirror 
>> 1
>> [918087.118559] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 
>> 1
>> [918087.131234] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 
>> 1
>> [918087.132513] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 
>> 1
>> [918087.135749] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 
>> 1
>> [918119.990544] BTRFS warning (device dm-19): dm-19 checksum verify
>> failed on 20043135205376 wanted 79d70254 found 93ac2b61 level 0
>> [918120.010421] verify_parent_transid: 105 callbacks suppressed
>> [918120.010428] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043135205376 wanted 45126 found 45104
>> [918120.010565] __btrfs_lookup_bio_sums: 48 callbacks suppressed
>> [918120.010567] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 678543360
>> [918120.028733] btrfs_print_data_csum_error: 5 callbacks suppressed
>> [918120.028735] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 
>> 1
>> [918120.040591] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043135205376 wanted 45126 found 45104
>> [918120.041097] BTRFS error (device dm-19): parent transid verify 
>> failed
>> on 20043135205376 wanted 45126 found 45104
>> [918120.041112] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 678543360
>> [918120.041440] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 
>> 1
> 
> Since you haven't rebooted, all these mismatch can be caused by aborted
> transaction.
> 
>> 
>> 
>> Fortunately, after a reboot, everything appears healthy, no file
>> corruption that I can observe, but I'm still running some checks.
> 
> That's great, write time tree checker prevented corruption.
> 
> But still, btrfs check, and scrub recommended.
> 
> And the most important thing, please provide the missing key log line.
> 
> Thanks,
> Qu
> 
>> 
>> I'm not sure what the write tree is or if this is related to the read
>> tree issues mentioned on the mailing list, any advice/pointers would 
>> be
>> most appreciated.  One thing I can note is that I've started to write
>> heavily to this filesystem via nfs export (the nfs client happens to 
>> be
>> a vm on the same machine, fwiw).
>> 

Re: write time tree block corruption detected

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 14489 bytes --]



On 2020/1/17 下午4:48, Kenneth Topp wrote:
> On 2020-01-17 03:33, Qu Wenruo wrote:
>> On 2020/1/17 下午4:21, Kenneth Topp wrote:
>>> I've been running 5.4.8 for 10 days and this popped up:
>>>
>>> [917767.192190] BTRFS error (device dm-19): block=21374679056384 write
>>> time tree block corruption detected
>>> [917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
>>> errno=-5 IO failure
>>> [917767.193438] BTRFS info (device dm-19): forced readonly
>>
>> You missed the most important line. One line before "write time tree
>> block corruption detected" is the most important line.
>>
> 
> sorry, here is the missing line.
> 
> [917767.192187] BTRFS critical (device dm-19): corrupt leaf:
> root=18446744073709551610 block=21374679056384 slot=174 ino=219743
> file_offset=1385357312, file extent end range (1460178944) g
> oes beyond start offset (1385385984) of the next file extent
> [917767.192190] BTRFS error (device dm-19): block=21374679056384 write
> time tree block corruption detected
> [917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
> errno=-5 IO failure
> [917767.193438] BTRFS info (device dm-19): forced readonly

Hi Filipe,

This looks like one of your fixed problem for log tree, but I'm not very
confident.

Is this a fixed one or a new one?

Thanks,
Qu

> 
> thanks for responding so quickly!
> 
> ken
> 
>>>
>>>
>>> This was a btrfs filesystem created on 5.4.8 with two 4kn drives, each
>>> with a single whole disk dm-crypt partition.  The fs was created with
>>> mkfs -O no-holes -d single -m raid1 and mounted with
>>> clear_cache,space_cache=v2.
>>>
>>> After that issue, I tried to update my backups, but it didn't go very
>>> well:
>>>
>>> [918052.104923] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 21374567694336 wanted 6c2341bd found 640f30e8 level 0
>>> [918052.121801] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.121818] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397271552
>>> [918052.138406] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.139080] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.139094] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397275648
>>> [918052.139768] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140158] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140171] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397279744
>>> [918052.140532] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140868] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140880] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397283840
>>> [918052.159794] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
>>> [918052.213959] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.216411] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.216418] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397271552
>>> [918052.216570] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
>>> [918052.216790] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.217037] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397271552
>>> [918052.217165] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
>>> [918062.496366] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043100192768 wanted 8bf4c1c9 found 23b8dea8 level 0
>>> [918062.504832] verify_parent_transid: 1 callbacks suppressed
>>> [918062.504836] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.504848] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 28672
>>> [918062.506174] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100241920 wanted 45126 found 45104
>>> [918062.509823] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 28672 csum 0xa0f9eaf7 expected csum 0x00000000 mirror 1
>>> [918062.513826] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100241920 wanted 45126 found 45104
>>> [918062.513838] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 49152
>>> [918062.519723] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 49152 csum 0x76421bb0 expected csum 0x00000000 mirror 1
>>> [918062.530287] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.530866] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.530874] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 65536
>>> [918062.539238] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.539560] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.539566] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 69632
>>> [918062.540020] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.540261] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.540269] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 77824
>>> [918062.540965] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.541199] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 114688
>>> [918062.541753] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 118784
>>> [918062.544496] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 65536 csum 0xdcb45a27 expected csum 0x00000000 mirror 1
>>> [918062.547421] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 114688 csum 0x1d5a0f30 expected csum 0x00000000 mirror 1
>>> [918062.549968] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 69632 csum 0x24dd186c expected csum 0x00000000 mirror 1
>>> [918062.572502] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 77824 csum 0xba341c47 expected csum 0x00000000 mirror 1
>>> [918062.575300] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 118784 csum 0x519e7044 expected csum 0x00000000 mirror 1
>>> [918062.584005] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 172032
>>> [918062.586782] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 221184
>>> [918062.587106] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 221184 csum 0xc9c50b2d expected csum 0x00000000 mirror 1
>>> [918062.590739] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 450560
>>> [918062.591255] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 450560 csum 0xd2e5e3eb expected csum 0x00000000 mirror 1
>>> [918062.601361] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043100258304 wanted 61da40d5 found 204cd10a level 0
>>> [918062.602875] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 503808 csum 0x183beedc expected csum 0x00000000 mirror 1
>>> [918085.820716] verify_parent_transid: 42 callbacks suppressed
>>> [918085.820730] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.837964] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.837992] __btrfs_lookup_bio_sums: 17 callbacks suppressed
>>> [918085.837994] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159711232
>>> [918085.838459] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.838987] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.838995] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159715328
>>> [918085.839349] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840054] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840065] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159719424
>>> [918085.840402] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840943] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840951] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159723520
>>> [918085.850438] btrfs_print_data_csum_error: 17 callbacks suppressed
>>> [918085.850442] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
>>> [918085.959941] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.960447] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.960452] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159711232
>>> [918085.960624] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
>>> [918085.961220] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159711232
>>> [918085.961341] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
>>> [918087.026984] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193855488
>>> [918087.028152] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193859584
>>> [918087.028598] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193863680
>>> [918087.029005] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193867776
>>> [918087.042810] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
>>> [918087.054812] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193904640 csum 0x279aea06 expected csum 0x00000000 mirror 1
>>> [918087.071004] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043549310976 wanted fcc4f73a found 3d3d8e83 level 0
>>> [918087.103303] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193937408 csum 0x1fedc264 expected csum 0x00000000 mirror 1
>>> [918087.118559] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
>>> [918087.131234] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
>>> [918087.132513] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
>>> [918087.135749] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
>>> [918119.990544] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043135205376 wanted 79d70254 found 93ac2b61 level 0
>>> [918120.010421] verify_parent_transid: 105 callbacks suppressed
>>> [918120.010428] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043135205376 wanted 45126 found 45104
>>> [918120.010565] __btrfs_lookup_bio_sums: 48 callbacks suppressed
>>> [918120.010567] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 678543360
>>> [918120.028733] btrfs_print_data_csum_error: 5 callbacks suppressed
>>> [918120.028735] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
>>> [918120.040591] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043135205376 wanted 45126 found 45104
>>> [918120.041097] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043135205376 wanted 45126 found 45104
>>> [918120.041112] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 678543360
>>> [918120.041440] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
>>
>> Since you haven't rebooted, all these mismatch can be caused by aborted
>> transaction.
>>
>>>
>>>
>>> Fortunately, after a reboot, everything appears healthy, no file
>>> corruption that I can observe, but I'm still running some checks.
>>
>> That's great, write time tree checker prevented corruption.
>>
>> But still, btrfs check, and scrub recommended.
>>
>> And the most important thing, please provide the missing key log line.
>>
>> Thanks,
>> Qu
>>
>>>
>>> I'm not sure what the write tree is or if this is related to the read
>>> tree issues mentioned on the mailing list, any advice/pointers would be
>>> most appreciated.  One thing I can note is that I've started to write
>>> heavily to this filesystem via nfs export (the nfs client happens to be
>>> a vm on the same machine, fwiw).
>>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]