* Write time tree block corruption detected
@ 2020-08-12 13:49 Spencer Collyer
2020-08-12 14:15 ` Qu Wenruo
0 siblings, 1 reply; 16+ messages in thread
From: Spencer Collyer @ 2020-08-12 13:49 UTC (permalink / raw)
To: linux-btrfs
I have just received a 'write time tree block corruption detected' message on a BTRFS fs I use. As per
https://btrfs.wiki.kernel.org/index.php/Tree-checker it mentions sending a mail to this mailing list for this case.
The dmesg output from the error onwards is as follows:
[17434.620469] BTRFS error (device dm-1): block=13642806624256 write time tree block corruption detected
[17435.048842] BTRFS: error (device dm-1) in btrfs_commit_transaction:2323: errno=-5 IO failure (Error while writing out transaction)
[17435.048848] BTRFS info (device dm-1): forced readonly
[17435.048851] BTRFS warning (device dm-1): Skipping commit of aborted transaction.
[17435.048853] BTRFS: error (device dm-1) in cleanup_transaction:1894: errno=-5 IO failure
As can be seen, the fs was forced into readonly mode. I restarted the box that the fs was mounted on and am able to write to the fs again (tried 'touch'ing a non-existent file and that worked), but I'm reluctant to do any more with it until I know it is safe to do so.
Is there anything I should do to ensure the fs is OK? The page I quoted earlier said not to run 'btrfs check --repair' unless told to by a developer.
Thanks for your attention.
Spencer
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Write time tree block corruption detected
2020-08-12 13:49 Write time tree block corruption detected Spencer Collyer
@ 2020-08-12 14:15 ` Qu Wenruo
2020-08-13 5:53 ` Spencer Collyer
0 siblings, 1 reply; 16+ messages in thread
From: Qu Wenruo @ 2020-08-12 14:15 UTC (permalink / raw)
To: Spencer Collyer, linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 1443 bytes --]
On 2020/8/12 下午9:49, Spencer Collyer wrote:
> I have just received a 'write time tree block corruption detected' message on a BTRFS fs I use. As per
> https://btrfs.wiki.kernel.org/index.php/Tree-checker it mentions sending a mail to this mailing list for this case.
>
> The dmesg output from the error onwards is as follows:
>
> [17434.620469] BTRFS error (device dm-1): block=13642806624256 write time tree block corruption detected
What's the line before this line?
That's the most important line, and it's unfortunately not there...
Thanks,
Qu
> [17435.048842] BTRFS: error (device dm-1) in btrfs_commit_transaction:2323: errno=-5 IO failure (Error while writing out transaction)
> [17435.048848] BTRFS info (device dm-1): forced readonly
> [17435.048851] BTRFS warning (device dm-1): Skipping commit of aborted transaction.
> [17435.048853] BTRFS: error (device dm-1) in cleanup_transaction:1894: errno=-5 IO failure
>
> As can be seen, the fs was forced into readonly mode. I restarted the box that the fs was mounted on and am able to write to the fs again (tried 'touch'ing a non-existent file and that worked), but I'm reluctant to do any more with it until I know it is safe to do so.
>
> Is there anything I should do to ensure the fs is OK? The page I quoted earlier said not to run 'btrfs check --repair' unless told to by a developer.
>
> Thanks for your attention.
>
> Spencer
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Write time tree block corruption detected
2020-08-12 14:15 ` Qu Wenruo
@ 2020-08-13 5:53 ` Spencer Collyer
2020-08-13 5:58 ` Qu Wenruo
0 siblings, 1 reply; 16+ messages in thread
From: Spencer Collyer @ 2020-08-13 5:53 UTC (permalink / raw)
To: linux-btrfs
On Wed, 12 Aug 2020 22:15:23 +0800, Qu Wenruo wrote:
> On 2020/8/12 下午9:49, Spencer Collyer wrote:
> > I have just received a 'write time tree block corruption detected'
> > message on a BTRFS fs I use. As per
> > https://btrfs.wiki.kernel.org/index.php/Tree-checker it mentions
> > sending a mail to this mailing list for this case.
> >
> > The dmesg output from the error onwards is as follows:
> >
> > [17434.620469] BTRFS error (device dm-1): block=13642806624256
> > write time tree block corruption detected
>
> What's the line before this line?
>
> That's the most important line, and it's unfortunately not there...
>
> Thanks,
> Qu
Oh, didn't realise that. Unfortunately it has gone from the dmesg ring buffer, so I can't go back to it. If it happens again I'll take a copy of the full dmesg output so I can go back to it.
The page I mentioned previously (https://btrfs.wiki.kernel.org/index.php/Tree-checker) just mentions to report the error to this mailing list. Might be an idea to expand that line to explain what needs reporting.
Rereading the linked page, I notice that it says that if it is a write error that is prevented then the fs should still be OK, and you can run 'btrfs check --readonly' to check that. It is 'btrfs check --repair' that it says to not run unless told to by a developer. So I'm going to run the '--readonly' check and if that looks good I'll try re-running the command that caused the failure - does that seem reasonable?
Thanks for your attention,
Spencer
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Write time tree block corruption detected
2020-08-13 5:53 ` Spencer Collyer
@ 2020-08-13 5:58 ` Qu Wenruo
0 siblings, 0 replies; 16+ messages in thread
From: Qu Wenruo @ 2020-08-13 5:58 UTC (permalink / raw)
To: Spencer Collyer, linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 2186 bytes --]
On 2020/8/13 下午1:53, Spencer Collyer wrote:
> On Wed, 12 Aug 2020 22:15:23 +0800, Qu Wenruo wrote:
>> On 2020/8/12 下午9:49, Spencer Collyer wrote:
>>> I have just received a 'write time tree block corruption detected'
>>> message on a BTRFS fs I use. As per
>>> https://btrfs.wiki.kernel.org/index.php/Tree-checker it mentions
>>> sending a mail to this mailing list for this case.
>>>
>>> The dmesg output from the error onwards is as follows:
>>>
>>> [17434.620469] BTRFS error (device dm-1): block=13642806624256
>>> write time tree block corruption detected
>>
>> What's the line before this line?
>>
>> That's the most important line, and it's unfortunately not there...
>>
>> Thanks,
>> Qu
>
> Oh, didn't realise that. Unfortunately it has gone from the dmesg ring buffer, so I can't go back to it. If it happens again I'll take a copy of the full dmesg output so I can go back to it.
>
> The page I mentioned previously (https://btrfs.wiki.kernel.org/index.php/Tree-checker) just mentions to report the error to this mailing list. Might be an idea to expand that line to explain what needs reporting.
>
> Rereading the linked page, I notice that it says that if it is a write error that is prevented then the fs should still be OK, and you can run 'btrfs check --readonly' to check that. It is 'btrfs check --repair' that it says to not run unless told to by a developer. So I'm going to run the '--readonly' check and if that looks good I'll try re-running the command that caused the failure - does that seem reasonable?
Running btrfs check is always a good idea to ensure nothing is wrong.
Write time tree checker mostly detects the following types of errors:
- Memory bit flip
Then you'd better to at least run a memtest to ensure your memory is
fine.
- Btrfs internal bug
One example is the csum item overlap, which is a bug and soon get
fixed.
Although you can definitely try the same command, I doubt if it can
reproduce the problem.
But anyway, if you can reproduce the bug, it would definitely be helpful
for us to look into.
Thanks,
Qu
>
> Thanks for your attention,
>
> Spencer
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Write time tree block corruption detected
2021-06-09 8:32 ` Qu Wenruo
@ 2021-06-11 19:44 ` auxsvr
0 siblings, 0 replies; 16+ messages in thread
From: auxsvr @ 2021-06-11 19:44 UTC (permalink / raw)
To: linux-btrfs, Qu Wenruo
On Wednesday, 9 June 2021 11:32:57 EEST Qu Wenruo wrote:
> Would you please try to compile the following branch of btrfs-progs?
>
> https://github.com/adam900710/btrfs-progs/tree/device_size_fix
>
> And run "./btrfs check --repair --force /dev/sda2" for the newly
> compiled btrfs-progs to see if it solves your problem.
Ran the repair and everything seems to be fine so far.
Thank you,
Petros
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Write time tree block corruption detected
2021-06-09 8:04 ` auxsvr
@ 2021-06-09 8:32 ` Qu Wenruo
2021-06-11 19:44 ` auxsvr
0 siblings, 1 reply; 16+ messages in thread
From: Qu Wenruo @ 2021-06-09 8:32 UTC (permalink / raw)
To: auxsvr, linux-btrfs
Confirmed that the result in device tree is indeed smaller than the
superblock bytes_used.
Would you please try to compile the following branch of btrfs-progs?
https://github.com/adam900710/btrfs-progs/tree/device_size_fix
And run "./btrfs check --repair --force /dev/sda2" for the newly
compiled btrfs-progs to see if it solves your problem.
Thanks,
Qu
On 2021/6/9 下午4:04, auxsvr wrote:
> On Wednesday, 9 June 2021 04:05:18 EEST Qu Wenruo wrote:
>> # btrfs ins dump-tree -t dev /dev/sda2
> # btrfs ins dump-tree -t dev /dev/sda2
> btrfs-progs v4.19.1
> device tree key (DEV_TREE ROOT_ITEM 0)
> leaf 181304705024 items 59 free space 11984 generation 1049986 owner DEV_TREE
> leaf 181304705024 flags 0x1(WRITTEN) backref revision 1
> fs uuid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
> chunk uuid e09a0d7a-15ec-48ef-89fa-adaa5ec52990
> item 0 key (0 PERSISTENT_ITEM 1) itemoff 16243 itemsize 40
> persistent item objectid 0 offset 1
> device stats
> write_errs 0 read_errs 0 flush_errs 0 corruption_errs 0 generation 0
> item 1 key (1 DEV_EXTENT 22020096) itemoff 16195 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 229569986560 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 2 key (1 DEV_EXTENT 1095761920) itemoff 16147 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 537007226880 length 33554432
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 3 key (1 DEV_EXTENT 2169503744) itemoff 16099 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 227422502912 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 4 key (1 DEV_EXTENT 3243245568) itemoff 16051 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 226348761088 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 5 key (1 DEV_EXTENT 4316987392) itemoff 16003 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 225275019264 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 6 key (1 DEV_EXTENT 5390729216) itemoff 15955 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 224201277440 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 7 key (1 DEV_EXTENT 6464471040) itemoff 15907 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 223127535616 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 8 key (1 DEV_EXTENT 7538212864) itemoff 15859 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 222053793792 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 9 key (1 DEV_EXTENT 8611954688) itemoff 15811 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 220980051968 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 10 key (1 DEV_EXTENT 9685696512) itemoff 15763 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 315011104768 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 11 key (1 DEV_EXTENT 10759438336) itemoff 15715 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 295851524096 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 12 key (1 DEV_EXTENT 11833180160) itemoff 15667 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 437149237248 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 13 key (1 DEV_EXTENT 12906921984) itemoff 15619 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 187694055424 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 14 key (1 DEV_EXTENT 14014218240) itemoff 15571 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 181251604480 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 15 key (1 DEV_EXTENT 15087960064) itemoff 15523 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 328164442112 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 16 key (1 DEV_EXTENT 16161701888) itemoff 15475 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 379972485120 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 17 key (1 DEV_EXTENT 17235443712) itemoff 15427 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 531168755712 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 18 key (1 DEV_EXTENT 18309185536) itemoff 15379 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 457684549632 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 19 key (1 DEV_EXTENT 19382927360) itemoff 15331 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 347860893696 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 20 key (1 DEV_EXTENT 20456669184) itemoff 15283 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 345142984704 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 21 key (1 DEV_EXTENT 21530411008) itemoff 15235 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 532242497536 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 22 key (1 DEV_EXTENT 23644340224) itemoff 15187 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 183399088128 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 23 key (1 DEV_EXTENT 24718082048) itemoff 15139 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 537040781312 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 24 key (1 DEV_EXTENT 25791823872) itemoff 15091 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 208095150080 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 25 key (1 DEV_EXTENT 26865565696) itemoff 15043 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 207021408256 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 26 key (1 DEV_EXTENT 27939307520) itemoff 14995 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 205947666432 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 27 key (1 DEV_EXTENT 29013049344) itemoff 14947 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 204873924608 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 28 key (1 DEV_EXTENT 30086791168) itemoff 14899 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 203800182784 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 29 key (1 DEV_EXTENT 31160532992) itemoff 14851 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 202726440960 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 30 key (1 DEV_EXTENT 32234274816) itemoff 14803 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 201652699136 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 31 key (1 DEV_EXTENT 33308016640) itemoff 14755 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 200578957312 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 32 key (1 DEV_EXTENT 34381758464) itemoff 14707 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 199505215488 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 33 key (1 DEV_EXTENT 35455500288) itemoff 14659 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 198431473664 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 34 key (1 DEV_EXTENT 36529242112) itemoff 14611 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 460905775104 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 35 key (1 DEV_EXTENT 37602983936) itemoff 14563 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 196283990016 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 36 key (1 DEV_EXTENT 38676725760) itemoff 14515 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 259097886720 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 37 key (1 DEV_EXTENT 39750467584) itemoff 14467 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 184472829952 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 38 key (1 DEV_EXTENT 40824209408) itemoff 14419 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 195210248192 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 39 key (1 DEV_EXTENT 41897951232) itemoff 14371 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 182325346304 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 40 key (1 DEV_EXTENT 42971693056) itemoff 14323 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 194136506368 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 41 key (1 DEV_EXTENT 44045434880) itemoff 14275 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 186620313600 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 42 key (1 DEV_EXTENT 45119176704) itemoff 14227 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 188767797248 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 43 key (1 DEV_EXTENT 46192918528) itemoff 14179 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 189841539072 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 44 key (1 DEV_EXTENT 47266660352) itemoff 14131 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 193062764544 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 45 key (1 DEV_EXTENT 48340402176) itemoff 14083 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 191989022720 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 46 key (1 DEV_EXTENT 49414144000) itemoff 14035 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 190915280896 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 47 key (1 DEV_EXTENT 50487885824) itemoff 13987 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 280606277632 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 48 key (1 DEV_EXTENT 51561627648) itemoff 13939 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 471576084480 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 49 key (1 DEV_EXTENT 52635369472) itemoff 13891 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 282753761280 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 50 key (1 DEV_EXTENT 53709111296) itemoff 13843 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 523384127488 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 51 key (1 DEV_EXTENT 54782853120) itemoff 13795 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 524625641472 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 52 key (1 DEV_EXTENT 55856594944) itemoff 13747 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 526202699776 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 53 key (1 DEV_EXTENT 56930336768) itemoff 13699 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 533316239360 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 54 key (1 DEV_EXTENT 58004078592) itemoff 13651 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 264802140160 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 55 key (1 DEV_EXTENT 59077820416) itemoff 13603 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 433558913024 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 56 key (1 DEV_EXTENT 60151562240) itemoff 13555 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 534389981184 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 57 key (1 DEV_EXTENT 61225304064) itemoff 13507 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 290303508480 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> item 58 key (1 DEV_EXTENT 62299045888) itemoff 13459 itemsize 48
> dev extent chunk_tree 3
> chunk_objectid 256 chunk_offset 535463723008 length 1073741824
> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
> total bytes 64424509440
> bytes used 46488170496
> uuid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
>
>> # btrfs ins dump-super /dev/sda2
> # btrfs ins dump-super /dev/sda2
> superblock: bytenr=65536, device=/dev/sda2
> ---------------------------------------------------------
> csum_type 0 (crc32c)
> csum_size 4
> csum 0x5c8acbc2 [match]
> bytenr 65536
> flags 0x1
> ( WRITTEN )
> magic _BHRfS_M [match]
> fsid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
> metadata_uuid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
> label
> generation 1050015
> root 181625438208
> sys_array_size 97
> chunk_root_generation 1047576
> root_level 1
> chunk_root 537007226880
> chunk_root_level 0
> log_root 0
> log_root_transid 0
> log_root_level 0
> total_bytes 64424509440
> bytes_used 46488170496
> sectorsize 4096
> nodesize 16384
> leafsize (deprecated) 16384
> stripesize 4096
> root_dir 6
> num_devices 1
> compat_flags 0x0
> compat_ro_flags 0x0
> incompat_flags 0x161
> ( MIXED_BACKREF |
> BIG_METADATA |
> EXTENDED_IREF |
> SKINNY_METADATA )
> cache_generation 1050015
> uuid_tree_generation 1047664
> dev_item.uuid abc9d19a-d9d1-40a4-9bd5-f28654800249
> dev_item.fsid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4 [match]
> dev_item.type 0
> dev_item.total_bytes 64424509440
> dev_item.bytes_used 63451430912
> dev_item.io_align 0
> dev_item.io_width 0
> dev_item.sector_size 0
> dev_item.devid 1
> dev_item.dev_group 0
> dev_item.seek_speed 0
> dev_item.bandwidth 0
> dev_item.generation 0
>
> Thanks,
> Petros
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Write time tree block corruption detected
2021-06-09 1:05 ` Qu Wenruo
@ 2021-06-09 8:04 ` auxsvr
2021-06-09 8:32 ` Qu Wenruo
0 siblings, 1 reply; 16+ messages in thread
From: auxsvr @ 2021-06-09 8:04 UTC (permalink / raw)
To: linux-btrfs, Qu Wenruo
On Wednesday, 9 June 2021 04:05:18 EEST Qu Wenruo wrote:
> # btrfs ins dump-tree -t dev /dev/sda2
# btrfs ins dump-tree -t dev /dev/sda2
btrfs-progs v4.19.1
device tree key (DEV_TREE ROOT_ITEM 0)
leaf 181304705024 items 59 free space 11984 generation 1049986 owner DEV_TREE
leaf 181304705024 flags 0x1(WRITTEN) backref revision 1
fs uuid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
chunk uuid e09a0d7a-15ec-48ef-89fa-adaa5ec52990
item 0 key (0 PERSISTENT_ITEM 1) itemoff 16243 itemsize 40
persistent item objectid 0 offset 1
device stats
write_errs 0 read_errs 0 flush_errs 0 corruption_errs 0 generation 0
item 1 key (1 DEV_EXTENT 22020096) itemoff 16195 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 229569986560 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 2 key (1 DEV_EXTENT 1095761920) itemoff 16147 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 537007226880 length 33554432
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 3 key (1 DEV_EXTENT 2169503744) itemoff 16099 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 227422502912 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 4 key (1 DEV_EXTENT 3243245568) itemoff 16051 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 226348761088 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 5 key (1 DEV_EXTENT 4316987392) itemoff 16003 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 225275019264 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 6 key (1 DEV_EXTENT 5390729216) itemoff 15955 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 224201277440 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 7 key (1 DEV_EXTENT 6464471040) itemoff 15907 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 223127535616 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 8 key (1 DEV_EXTENT 7538212864) itemoff 15859 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 222053793792 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 9 key (1 DEV_EXTENT 8611954688) itemoff 15811 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 220980051968 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 10 key (1 DEV_EXTENT 9685696512) itemoff 15763 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 315011104768 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 11 key (1 DEV_EXTENT 10759438336) itemoff 15715 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 295851524096 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 12 key (1 DEV_EXTENT 11833180160) itemoff 15667 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 437149237248 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 13 key (1 DEV_EXTENT 12906921984) itemoff 15619 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 187694055424 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 14 key (1 DEV_EXTENT 14014218240) itemoff 15571 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 181251604480 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 15 key (1 DEV_EXTENT 15087960064) itemoff 15523 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 328164442112 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 16 key (1 DEV_EXTENT 16161701888) itemoff 15475 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 379972485120 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 17 key (1 DEV_EXTENT 17235443712) itemoff 15427 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 531168755712 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 18 key (1 DEV_EXTENT 18309185536) itemoff 15379 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 457684549632 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 19 key (1 DEV_EXTENT 19382927360) itemoff 15331 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 347860893696 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 20 key (1 DEV_EXTENT 20456669184) itemoff 15283 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 345142984704 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 21 key (1 DEV_EXTENT 21530411008) itemoff 15235 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 532242497536 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 22 key (1 DEV_EXTENT 23644340224) itemoff 15187 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 183399088128 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 23 key (1 DEV_EXTENT 24718082048) itemoff 15139 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 537040781312 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 24 key (1 DEV_EXTENT 25791823872) itemoff 15091 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 208095150080 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 25 key (1 DEV_EXTENT 26865565696) itemoff 15043 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 207021408256 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 26 key (1 DEV_EXTENT 27939307520) itemoff 14995 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 205947666432 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 27 key (1 DEV_EXTENT 29013049344) itemoff 14947 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 204873924608 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 28 key (1 DEV_EXTENT 30086791168) itemoff 14899 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 203800182784 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 29 key (1 DEV_EXTENT 31160532992) itemoff 14851 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 202726440960 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 30 key (1 DEV_EXTENT 32234274816) itemoff 14803 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 201652699136 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 31 key (1 DEV_EXTENT 33308016640) itemoff 14755 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 200578957312 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 32 key (1 DEV_EXTENT 34381758464) itemoff 14707 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 199505215488 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 33 key (1 DEV_EXTENT 35455500288) itemoff 14659 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 198431473664 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 34 key (1 DEV_EXTENT 36529242112) itemoff 14611 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 460905775104 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 35 key (1 DEV_EXTENT 37602983936) itemoff 14563 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 196283990016 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 36 key (1 DEV_EXTENT 38676725760) itemoff 14515 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 259097886720 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 37 key (1 DEV_EXTENT 39750467584) itemoff 14467 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 184472829952 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 38 key (1 DEV_EXTENT 40824209408) itemoff 14419 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 195210248192 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 39 key (1 DEV_EXTENT 41897951232) itemoff 14371 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 182325346304 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 40 key (1 DEV_EXTENT 42971693056) itemoff 14323 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 194136506368 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 41 key (1 DEV_EXTENT 44045434880) itemoff 14275 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 186620313600 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 42 key (1 DEV_EXTENT 45119176704) itemoff 14227 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 188767797248 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 43 key (1 DEV_EXTENT 46192918528) itemoff 14179 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 189841539072 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 44 key (1 DEV_EXTENT 47266660352) itemoff 14131 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 193062764544 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 45 key (1 DEV_EXTENT 48340402176) itemoff 14083 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 191989022720 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 46 key (1 DEV_EXTENT 49414144000) itemoff 14035 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 190915280896 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 47 key (1 DEV_EXTENT 50487885824) itemoff 13987 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 280606277632 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 48 key (1 DEV_EXTENT 51561627648) itemoff 13939 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 471576084480 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 49 key (1 DEV_EXTENT 52635369472) itemoff 13891 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 282753761280 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 50 key (1 DEV_EXTENT 53709111296) itemoff 13843 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 523384127488 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 51 key (1 DEV_EXTENT 54782853120) itemoff 13795 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 524625641472 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 52 key (1 DEV_EXTENT 55856594944) itemoff 13747 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 526202699776 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 53 key (1 DEV_EXTENT 56930336768) itemoff 13699 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 533316239360 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 54 key (1 DEV_EXTENT 58004078592) itemoff 13651 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 264802140160 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 55 key (1 DEV_EXTENT 59077820416) itemoff 13603 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 433558913024 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 56 key (1 DEV_EXTENT 60151562240) itemoff 13555 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 534389981184 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 57 key (1 DEV_EXTENT 61225304064) itemoff 13507 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 290303508480 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
item 58 key (1 DEV_EXTENT 62299045888) itemoff 13459 itemsize 48
dev extent chunk_tree 3
chunk_objectid 256 chunk_offset 535463723008 length 1073741824
chunk_tree_uuid 00000000-0000-0000-0000-000000000000
total bytes 64424509440
bytes used 46488170496
uuid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
> # btrfs ins dump-super /dev/sda2
# btrfs ins dump-super /dev/sda2
superblock: bytenr=65536, device=/dev/sda2
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x5c8acbc2 [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
metadata_uuid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
label
generation 1050015
root 181625438208
sys_array_size 97
chunk_root_generation 1047576
root_level 1
chunk_root 537007226880
chunk_root_level 0
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 64424509440
bytes_used 46488170496
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 1050015
uuid_tree_generation 1047664
dev_item.uuid abc9d19a-d9d1-40a4-9bd5-f28654800249
dev_item.fsid 44c67fa4-e2c4-4da5-9d07-98959ff77bc4 [match]
dev_item.type 0
dev_item.total_bytes 64424509440
dev_item.bytes_used 63451430912
dev_item.io_align 0
dev_item.io_width 0
dev_item.sector_size 0
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
Thanks,
Petros
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Write time tree block corruption detected
2021-06-08 22:39 ` Qu Wenruo
@ 2021-06-09 1:05 ` Qu Wenruo
2021-06-09 8:04 ` auxsvr
0 siblings, 1 reply; 16+ messages in thread
From: Qu Wenruo @ 2021-06-09 1:05 UTC (permalink / raw)
To: auxsvr, linux-btrfs
On 2021/6/9 上午6:39, Qu Wenruo wrote:
>
>
> On 2021/6/8 下午10:45, auxsvr wrote:
>> On Monday, 7 June 2021 14:55:10 EEST you wrote:
>>> It seems that every time free space is less than 1 GiB and I use
>>> zypper, I get a similar corruption message.
>>
>> Correction: I just got:
>>
>> [26355.330817] BTRFS critical (device sda2): corrupt leaf: root=3
>> block=537007243264 slot=0 devid=1 invalid bytes used: have 64503152640
>> expect [0, 64424509440]
>
> The line itself should explain itself, we're over allocating the device.
>
>> [26355.330819] BTRFS error (device sda2): block=537007243264 write
>> time tree block corruption detected
>
> Mind to dump the device tree?
>
> You can dump it with the following command, and the output contains
> nothing confidential, but only device extent layout:
>
> # btrfs ins dump-tree -t dev /dev/sda2
Sorry, you may also need to dump the super block:
# btrfs ins dump-super /dev/sda2
One of my concern is, the device used bytes may differ between super
block and device tree.
Thanks,
Qu
>
>>
>> and:
>>
>> # btrfs fi df /
>> Data, single: total=55.98GiB, used=43.04GiB
>> System, single: total=32.00MiB, used=16.00KiB
>> Metadata, single: total=2.00GiB, used=1.17GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> # btrfs fi show /
>> Label: none uuid: 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
>> Total devices 1 FS bytes used 44.22GiB
>> devid 1 size 60.00GiB used 60.07GiB path /dev/sda2
>
> My current guess is, there is something wrong in the past, leaving the
> accounting wrong.
>
> It leaves some ghost used bytes, thus causing the problem.
>
> Your previous btrfs-check is able to detect that, but unfortunately we
> don't have the ability to repair it in btrfs-check right now.
>
> I'll work on repair ability soon, thus if you can keep the fs, there is
> a high chance to repair it.
>
> Thanks,
> Qu
>
>>
>> 60.07GiB used again!
>>
>> Regards,
>> Petros
>>
>>
>>
>>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Write time tree block corruption detected
2021-06-08 14:45 ` auxsvr
@ 2021-06-08 22:39 ` Qu Wenruo
2021-06-09 1:05 ` Qu Wenruo
0 siblings, 1 reply; 16+ messages in thread
From: Qu Wenruo @ 2021-06-08 22:39 UTC (permalink / raw)
To: auxsvr, linux-btrfs
On 2021/6/8 下午10:45, auxsvr wrote:
> On Monday, 7 June 2021 14:55:10 EEST you wrote:
>> It seems that every time free space is less than 1 GiB and I use zypper, I get a similar corruption message.
>
> Correction: I just got:
>
> [26355.330817] BTRFS critical (device sda2): corrupt leaf: root=3 block=537007243264 slot=0 devid=1 invalid bytes used: have 64503152640 expect [0, 64424509440]
The line itself should explain itself, we're over allocating the device.
> [26355.330819] BTRFS error (device sda2): block=537007243264 write time tree block corruption detected
Mind to dump the device tree?
You can dump it with the following command, and the output contains
nothing confidential, but only device extent layout:
# btrfs ins dump-tree -t dev /dev/sda2
>
> and:
>
> # btrfs fi df /
> Data, single: total=55.98GiB, used=43.04GiB
> System, single: total=32.00MiB, used=16.00KiB
> Metadata, single: total=2.00GiB, used=1.17GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> # btrfs fi show /
> Label: none uuid: 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
> Total devices 1 FS bytes used 44.22GiB
> devid 1 size 60.00GiB used 60.07GiB path /dev/sda2
My current guess is, there is something wrong in the past, leaving the
accounting wrong.
It leaves some ghost used bytes, thus causing the problem.
Your previous btrfs-check is able to detect that, but unfortunately we
don't have the ability to repair it in btrfs-check right now.
I'll work on repair ability soon, thus if you can keep the fs, there is
a high chance to repair it.
Thanks,
Qu
>
> 60.07GiB used again!
>
> Regards,
> Petros
>
>
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Write time tree block corruption detected
2021-06-07 11:55 ` auxsvr
@ 2021-06-08 14:45 ` auxsvr
2021-06-08 22:39 ` Qu Wenruo
0 siblings, 1 reply; 16+ messages in thread
From: auxsvr @ 2021-06-08 14:45 UTC (permalink / raw)
To: linux-btrfs
On Monday, 7 June 2021 14:55:10 EEST you wrote:
> It seems that every time free space is less than 1 GiB and I use zypper, I get a similar corruption message.
Correction: I just got:
[26355.330817] BTRFS critical (device sda2): corrupt leaf: root=3 block=537007243264 slot=0 devid=1 invalid bytes used: have 64503152640 expect [0, 64424509440]
[26355.330819] BTRFS error (device sda2): block=537007243264 write time tree block corruption detected
and:
# btrfs fi df /
Data, single: total=55.98GiB, used=43.04GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=2.00GiB, used=1.17GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /
Label: none uuid: 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
Total devices 1 FS bytes used 44.22GiB
devid 1 size 60.00GiB used 60.07GiB path /dev/sda2
60.07GiB used again!
Regards,
Petros
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Write time tree block corruption detected
2021-05-30 21:30 auxsvr
@ 2021-06-07 11:55 ` auxsvr
2021-06-08 14:45 ` auxsvr
0 siblings, 1 reply; 16+ messages in thread
From: auxsvr @ 2021-06-07 11:55 UTC (permalink / raw)
To: linux-btrfs
On Monday, 31 May 2021 00:30:00 EEST auxsvr@gmail.com wrote:
> Hello,
>
> On linux 5.3.18-lp152.75-default (openSUSE 15.2), I got the following error message:
>
> [33573.070335] BTRFS critical (device sda2): corrupt leaf: root=3 block=531135201280 slot=0 devid=1 invalid bytes used: have 64503152640 expect [0, 64424509440]
> [33573.070338] BTRFS error (device sda2): block=531135201280 write time tree block corruption detected
> [33573.071909] BTRFS: error (device sda2) in btrfs_commit_transaction:2264: errno=-5 IO failure (Error while writing out transaction)
> [33573.071911] BTRFS info (device sda2): forced readonly
> [33573.071913] BTRFS warning (device sda2): Skipping commit of aborted transaction.
> [33573.071915] BTRFS: error (device sda2) in cleanup_transaction:1823: errno=-5 IO failure
> [33573.071917] BTRFS info (device sda2): delayed_refs has NO entry
> [33577.283856] BTRFS info (device sda2): delayed_refs has NO entry
After using btrfs check --repair /dev/sda2:
ref mismatch on [184344514560 4096] extent item 1024, found 0
owner ref check failed [184344514560 4096]
repair deleting extent record: key 184344514560 168 4096
Repaired extent references for 184344514560
Dev extent's total-byte(61236838400) is not equal to byte-used(63451430912) in dev[1, 216, 1]
without resolving the issue, as the filesystem would still become read-only, I deleted all snapshots and there hasn't been an issue for a few hours of heavy use. However, btrfs check still returns the error regarding size mismatch and this worries me a bit. Is there an easy way to resolve this or should I recreate the filesystem?
It seems that every time free space is less than 1 GiB and I use zypper, I get a similar corruption message. Thankfully, there hasn't been any data loss so far. There's also a backup of the filesystem right after the first incident, in case it is useful for further investigation.
Regards,
Petros
^ permalink raw reply [flat|nested] 16+ messages in thread
* Write time tree block corruption detected
@ 2021-05-30 21:30 auxsvr
2021-06-07 11:55 ` auxsvr
0 siblings, 1 reply; 16+ messages in thread
From: auxsvr @ 2021-05-30 21:30 UTC (permalink / raw)
To: linux-btrfs
Hello,
On linux 5.3.18-lp152.75-default (openSUSE 15.2), I got the following error message:
[33573.070335] BTRFS critical (device sda2): corrupt leaf: root=3 block=531135201280 slot=0 devid=1 invalid bytes used: have 64503152640 expect [0, 64424509440]
[33573.070338] BTRFS error (device sda2): block=531135201280 write time tree block corruption detected
[33573.071909] BTRFS: error (device sda2) in btrfs_commit_transaction:2264: errno=-5 IO failure (Error while writing out transaction)
[33573.071911] BTRFS info (device sda2): forced readonly
[33573.071913] BTRFS warning (device sda2): Skipping commit of aborted transaction.
[33573.071915] BTRFS: error (device sda2) in cleanup_transaction:1823: errno=-5 IO failure
[33573.071917] BTRFS info (device sda2): delayed_refs has NO entry
[33577.283856] BTRFS info (device sda2): delayed_refs has NO entry
Probably related is that the filesystem had less than 1 GiB free when this occurred. Rebooting and deleting some files seemed to have resolved it for a while, until:
[ 5201.020901] BTRFS critical (device sda2): corrupt leaf: root=3 block=536671682560 slot=0 devid=1 invalid bytes used: have 64503152640 expect [0, 64424509440]
[ 5201.020904] BTRFS error (device sda2): block=536671682560 write time tree block corruption detected
[ 5201.022983] BTRFS: error (device sda2) in btrfs_commit_transaction:2264: errno=-5 IO failure (Error while writing out transaction)
[ 5201.022985] BTRFS info (device sda2): forced readonly
[ 5201.022987] BTRFS warning (device sda2): Skipping commit of aborted transaction.
[ 5201.022988] BTRFS: error (device sda2) in cleanup_transaction:1823: errno=-5 IO failure
[ 5201.022991] BTRFS info (device sda2): delayed_refs has NO entry
[ 5201.023071] BTRFS info (device sda2): delayed_refs has NO entry
after a successful scrub and balance. This time df reports 8.8GiB free, but:
# btrfs fi df /
Data, single: total=55.98GiB, used=47.18GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=2.00GiB, used=1.01GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /
Label: none uuid: 44c67fa4-e2c4-4da5-9d07-98959ff77bc4
Total devices 1 FS bytes used 48.19GiB
devid 1 size 60.00GiB used 60.07GiB path /dev/sda2
60.07GiB used out of 60.00GiB! I recall that this is what I saw the first time too, so it might be related to the cause.
Is there anything I can do to ensure that the filesystem is not further damaged and that the issue is resolved?
Regards,
Petros
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: write time tree block corruption detected
2020-01-17 8:48 ` Kenneth Topp
@ 2020-01-17 10:09 ` Qu Wenruo
0 siblings, 0 replies; 16+ messages in thread
From: Qu Wenruo @ 2020-01-17 10:09 UTC (permalink / raw)
To: Kenneth Topp, Filipe Manana; +Cc: linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 14489 bytes --]
On 2020/1/17 下午4:48, Kenneth Topp wrote:
> On 2020-01-17 03:33, Qu Wenruo wrote:
>> On 2020/1/17 下午4:21, Kenneth Topp wrote:
>>> I've been running 5.4.8 for 10 days and this popped up:
>>>
>>> [917767.192190] BTRFS error (device dm-19): block=21374679056384 write
>>> time tree block corruption detected
>>> [917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
>>> errno=-5 IO failure
>>> [917767.193438] BTRFS info (device dm-19): forced readonly
>>
>> You missed the most important line. One line before "write time tree
>> block corruption detected" is the most important line.
>>
>
> sorry, here is the missing line.
>
> [917767.192187] BTRFS critical (device dm-19): corrupt leaf:
> root=18446744073709551610 block=21374679056384 slot=174 ino=219743
> file_offset=1385357312, file extent end range (1460178944) g
> oes beyond start offset (1385385984) of the next file extent
> [917767.192190] BTRFS error (device dm-19): block=21374679056384 write
> time tree block corruption detected
> [917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
> errno=-5 IO failure
> [917767.193438] BTRFS info (device dm-19): forced readonly
Hi Filipe,
This looks like one of your fixed problem for log tree, but I'm not very
confident.
Is this a fixed one or a new one?
Thanks,
Qu
>
> thanks for responding so quickly!
>
> ken
>
>>>
>>>
>>> This was a btrfs filesystem created on 5.4.8 with two 4kn drives, each
>>> with a single whole disk dm-crypt partition. The fs was created with
>>> mkfs -O no-holes -d single -m raid1 and mounted with
>>> clear_cache,space_cache=v2.
>>>
>>> After that issue, I tried to update my backups, but it didn't go very
>>> well:
>>>
>>> [918052.104923] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 21374567694336 wanted 6c2341bd found 640f30e8 level 0
>>> [918052.121801] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.121818] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397271552
>>> [918052.138406] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.139080] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.139094] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397275648
>>> [918052.139768] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140158] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140171] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397279744
>>> [918052.140532] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140868] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.140880] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397283840
>>> [918052.159794] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
>>> [918052.213959] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.216411] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.216418] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397271552
>>> [918052.216570] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
>>> [918052.216790] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918052.217037] BTRFS info (device dm-19): no csum found for inode
>>> 219677 start 3397271552
>>> [918052.217165] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
>>> [918062.496366] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043100192768 wanted 8bf4c1c9 found 23b8dea8 level 0
>>> [918062.504832] verify_parent_transid: 1 callbacks suppressed
>>> [918062.504836] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.504848] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 28672
>>> [918062.506174] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100241920 wanted 45126 found 45104
>>> [918062.509823] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 28672 csum 0xa0f9eaf7 expected csum 0x00000000 mirror 1
>>> [918062.513826] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100241920 wanted 45126 found 45104
>>> [918062.513838] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 49152
>>> [918062.519723] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 49152 csum 0x76421bb0 expected csum 0x00000000 mirror 1
>>> [918062.530287] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.530866] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.530874] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 65536
>>> [918062.539238] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.539560] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.539566] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 69632
>>> [918062.540020] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.540261] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.540269] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 77824
>>> [918062.540965] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043100192768 wanted 45126 found 45104
>>> [918062.541199] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 114688
>>> [918062.541753] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 118784
>>> [918062.544496] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 65536 csum 0xdcb45a27 expected csum 0x00000000 mirror 1
>>> [918062.547421] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 114688 csum 0x1d5a0f30 expected csum 0x00000000 mirror 1
>>> [918062.549968] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 69632 csum 0x24dd186c expected csum 0x00000000 mirror 1
>>> [918062.572502] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 77824 csum 0xba341c47 expected csum 0x00000000 mirror 1
>>> [918062.575300] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 118784 csum 0x519e7044 expected csum 0x00000000 mirror 1
>>> [918062.584005] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 172032
>>> [918062.586782] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 221184
>>> [918062.587106] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 221184 csum 0xc9c50b2d expected csum 0x00000000 mirror 1
>>> [918062.590739] BTRFS info (device dm-19): no csum found for inode
>>> 5464098 start 450560
>>> [918062.591255] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 450560 csum 0xd2e5e3eb expected csum 0x00000000 mirror 1
>>> [918062.601361] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043100258304 wanted 61da40d5 found 204cd10a level 0
>>> [918062.602875] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5464098 off 503808 csum 0x183beedc expected csum 0x00000000 mirror 1
>>> [918085.820716] verify_parent_transid: 42 callbacks suppressed
>>> [918085.820730] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.837964] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.837992] __btrfs_lookup_bio_sums: 17 callbacks suppressed
>>> [918085.837994] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159711232
>>> [918085.838459] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.838987] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.838995] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159715328
>>> [918085.839349] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840054] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840065] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159719424
>>> [918085.840402] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840943] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.840951] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159723520
>>> [918085.850438] btrfs_print_data_csum_error: 17 callbacks suppressed
>>> [918085.850442] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
>>> [918085.959941] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.960447] BTRFS error (device dm-19): parent transid verify failed
>>> on 21374567694336 wanted 45126 found 45104
>>> [918085.960452] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159711232
>>> [918085.960624] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
>>> [918085.961220] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 159711232
>>> [918085.961341] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
>>> [918087.026984] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193855488
>>> [918087.028152] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193859584
>>> [918087.028598] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193863680
>>> [918087.029005] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 193867776
>>> [918087.042810] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
>>> [918087.054812] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193904640 csum 0x279aea06 expected csum 0x00000000 mirror 1
>>> [918087.071004] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043549310976 wanted fcc4f73a found 3d3d8e83 level 0
>>> [918087.103303] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193937408 csum 0x1fedc264 expected csum 0x00000000 mirror 1
>>> [918087.118559] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
>>> [918087.131234] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
>>> [918087.132513] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
>>> [918087.135749] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
>>> [918119.990544] BTRFS warning (device dm-19): dm-19 checksum verify
>>> failed on 20043135205376 wanted 79d70254 found 93ac2b61 level 0
>>> [918120.010421] verify_parent_transid: 105 callbacks suppressed
>>> [918120.010428] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043135205376 wanted 45126 found 45104
>>> [918120.010565] __btrfs_lookup_bio_sums: 48 callbacks suppressed
>>> [918120.010567] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 678543360
>>> [918120.028733] btrfs_print_data_csum_error: 5 callbacks suppressed
>>> [918120.028735] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
>>> [918120.040591] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043135205376 wanted 45126 found 45104
>>> [918120.041097] BTRFS error (device dm-19): parent transid verify failed
>>> on 20043135205376 wanted 45126 found 45104
>>> [918120.041112] BTRFS info (device dm-19): no csum found for inode
>>> 5468073 start 678543360
>>> [918120.041440] BTRFS warning (device dm-19): csum failed root 5 ino
>>> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
>>
>> Since you haven't rebooted, all these mismatch can be caused by aborted
>> transaction.
>>
>>>
>>>
>>> Fortunately, after a reboot, everything appears healthy, no file
>>> corruption that I can observe, but I'm still running some checks.
>>
>> That's great, write time tree checker prevented corruption.
>>
>> But still, btrfs check, and scrub recommended.
>>
>> And the most important thing, please provide the missing key log line.
>>
>> Thanks,
>> Qu
>>
>>>
>>> I'm not sure what the write tree is or if this is related to the read
>>> tree issues mentioned on the mailing list, any advice/pointers would be
>>> most appreciated. One thing I can note is that I've started to write
>>> heavily to this filesystem via nfs export (the nfs client happens to be
>>> a vm on the same machine, fwiw).
>>>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: write time tree block corruption detected
2020-01-17 8:33 ` Qu Wenruo
@ 2020-01-17 8:48 ` Kenneth Topp
2020-01-17 10:09 ` Qu Wenruo
0 siblings, 1 reply; 16+ messages in thread
From: Kenneth Topp @ 2020-01-17 8:48 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs
On 2020-01-17 03:33, Qu Wenruo wrote:
> On 2020/1/17 下午4:21, Kenneth Topp wrote:
>> I've been running 5.4.8 for 10 days and this popped up:
>>
>> [917767.192190] BTRFS error (device dm-19): block=21374679056384 write
>> time tree block corruption detected
>> [917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
>> errno=-5 IO failure
>> [917767.193438] BTRFS info (device dm-19): forced readonly
>
> You missed the most important line. One line before "write time tree
> block corruption detected" is the most important line.
>
sorry, here is the missing line.
[917767.192187] BTRFS critical (device dm-19): corrupt leaf:
root=18446744073709551610 block=21374679056384 slot=174 ino=219743
file_offset=1385357312, file extent end range (1460178944) g
oes beyond start offset (1385385984) of the next file extent
[917767.192190] BTRFS error (device dm-19): block=21374679056384 write
time tree block corruption detected
[917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
errno=-5 IO failure
[917767.193438] BTRFS info (device dm-19): forced readonly
thanks for responding so quickly!
ken
>>
>>
>> This was a btrfs filesystem created on 5.4.8 with two 4kn drives, each
>> with a single whole disk dm-crypt partition. The fs was created with
>> mkfs -O no-holes -d single -m raid1 and mounted with
>> clear_cache,space_cache=v2.
>>
>> After that issue, I tried to update my backups, but it didn't go very
>> well:
>>
>> [918052.104923] BTRFS warning (device dm-19): dm-19 checksum verify
>> failed on 21374567694336 wanted 6c2341bd found 640f30e8 level 0
>> [918052.121801] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.121818] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397271552
>> [918052.138406] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.139080] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.139094] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397275648
>> [918052.139768] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.140158] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.140171] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397279744
>> [918052.140532] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.140868] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.140880] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397283840
>> [918052.159794] BTRFS warning (device dm-19): csum failed root 5 ino
>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror
>> 1
>> [918052.213959] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.216411] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.216418] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397271552
>> [918052.216570] BTRFS warning (device dm-19): csum failed root 5 ino
>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror
>> 1
>> [918052.216790] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918052.217037] BTRFS info (device dm-19): no csum found for inode
>> 219677 start 3397271552
>> [918052.217165] BTRFS warning (device dm-19): csum failed root 5 ino
>> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror
>> 1
>> [918062.496366] BTRFS warning (device dm-19): dm-19 checksum verify
>> failed on 20043100192768 wanted 8bf4c1c9 found 23b8dea8 level 0
>> [918062.504832] verify_parent_transid: 1 callbacks suppressed
>> [918062.504836] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.504848] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 28672
>> [918062.506174] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043100241920 wanted 45126 found 45104
>> [918062.509823] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 28672 csum 0xa0f9eaf7 expected csum 0x00000000 mirror 1
>> [918062.513826] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043100241920 wanted 45126 found 45104
>> [918062.513838] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 49152
>> [918062.519723] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 49152 csum 0x76421bb0 expected csum 0x00000000 mirror 1
>> [918062.530287] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.530866] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.530874] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 65536
>> [918062.539238] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.539560] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.539566] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 69632
>> [918062.540020] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.540261] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.540269] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 77824
>> [918062.540965] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043100192768 wanted 45126 found 45104
>> [918062.541199] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 114688
>> [918062.541753] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 118784
>> [918062.544496] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 65536 csum 0xdcb45a27 expected csum 0x00000000 mirror 1
>> [918062.547421] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 114688 csum 0x1d5a0f30 expected csum 0x00000000 mirror 1
>> [918062.549968] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 69632 csum 0x24dd186c expected csum 0x00000000 mirror 1
>> [918062.572502] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 77824 csum 0xba341c47 expected csum 0x00000000 mirror 1
>> [918062.575300] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 118784 csum 0x519e7044 expected csum 0x00000000 mirror 1
>> [918062.584005] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 172032
>> [918062.586782] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 221184
>> [918062.587106] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 221184 csum 0xc9c50b2d expected csum 0x00000000 mirror 1
>> [918062.590739] BTRFS info (device dm-19): no csum found for inode
>> 5464098 start 450560
>> [918062.591255] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 450560 csum 0xd2e5e3eb expected csum 0x00000000 mirror 1
>> [918062.601361] BTRFS warning (device dm-19): dm-19 checksum verify
>> failed on 20043100258304 wanted 61da40d5 found 204cd10a level 0
>> [918062.602875] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5464098 off 503808 csum 0x183beedc expected csum 0x00000000 mirror 1
>> [918085.820716] verify_parent_transid: 42 callbacks suppressed
>> [918085.820730] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.837964] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.837992] __btrfs_lookup_bio_sums: 17 callbacks suppressed
>> [918085.837994] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159711232
>> [918085.838459] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.838987] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.838995] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159715328
>> [918085.839349] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.840054] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.840065] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159719424
>> [918085.840402] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.840943] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.840951] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159723520
>> [918085.850438] btrfs_print_data_csum_error: 17 callbacks suppressed
>> [918085.850442] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror
>> 1
>> [918085.959941] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.960447] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 21374567694336 wanted 45126 found 45104
>> [918085.960452] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159711232
>> [918085.960624] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror
>> 1
>> [918085.961220] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 159711232
>> [918085.961341] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror
>> 1
>> [918087.026984] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 193855488
>> [918087.028152] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 193859584
>> [918087.028598] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 193863680
>> [918087.029005] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 193867776
>> [918087.042810] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror
>> 1
>> [918087.054812] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193904640 csum 0x279aea06 expected csum 0x00000000 mirror
>> 1
>> [918087.071004] BTRFS warning (device dm-19): dm-19 checksum verify
>> failed on 20043549310976 wanted fcc4f73a found 3d3d8e83 level 0
>> [918087.103303] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193937408 csum 0x1fedc264 expected csum 0x00000000 mirror
>> 1
>> [918087.118559] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror
>> 1
>> [918087.131234] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror
>> 1
>> [918087.132513] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror
>> 1
>> [918087.135749] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror
>> 1
>> [918119.990544] BTRFS warning (device dm-19): dm-19 checksum verify
>> failed on 20043135205376 wanted 79d70254 found 93ac2b61 level 0
>> [918120.010421] verify_parent_transid: 105 callbacks suppressed
>> [918120.010428] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043135205376 wanted 45126 found 45104
>> [918120.010565] __btrfs_lookup_bio_sums: 48 callbacks suppressed
>> [918120.010567] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 678543360
>> [918120.028733] btrfs_print_data_csum_error: 5 callbacks suppressed
>> [918120.028735] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror
>> 1
>> [918120.040591] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043135205376 wanted 45126 found 45104
>> [918120.041097] BTRFS error (device dm-19): parent transid verify
>> failed
>> on 20043135205376 wanted 45126 found 45104
>> [918120.041112] BTRFS info (device dm-19): no csum found for inode
>> 5468073 start 678543360
>> [918120.041440] BTRFS warning (device dm-19): csum failed root 5 ino
>> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror
>> 1
>
> Since you haven't rebooted, all these mismatch can be caused by aborted
> transaction.
>
>>
>>
>> Fortunately, after a reboot, everything appears healthy, no file
>> corruption that I can observe, but I'm still running some checks.
>
> That's great, write time tree checker prevented corruption.
>
> But still, btrfs check, and scrub recommended.
>
> And the most important thing, please provide the missing key log line.
>
> Thanks,
> Qu
>
>>
>> I'm not sure what the write tree is or if this is related to the read
>> tree issues mentioned on the mailing list, any advice/pointers would
>> be
>> most appreciated. One thing I can note is that I've started to write
>> heavily to this filesystem via nfs export (the nfs client happens to
>> be
>> a vm on the same machine, fwiw).
>>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: write time tree block corruption detected
2020-01-17 8:21 write " Kenneth Topp
@ 2020-01-17 8:33 ` Qu Wenruo
2020-01-17 8:48 ` Kenneth Topp
0 siblings, 1 reply; 16+ messages in thread
From: Qu Wenruo @ 2020-01-17 8:33 UTC (permalink / raw)
To: Kenneth Topp, linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 13139 bytes --]
On 2020/1/17 下午4:21, Kenneth Topp wrote:
> I've been running 5.4.8 for 10 days and this popped up:
>
> [917767.192190] BTRFS error (device dm-19): block=21374679056384 write
> time tree block corruption detected
> [917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
> errno=-5 IO failure
> [917767.193438] BTRFS info (device dm-19): forced readonly
You missed the most important line. One line before "write time tree
block corruption detected" is the most important line.
>
>
> This was a btrfs filesystem created on 5.4.8 with two 4kn drives, each
> with a single whole disk dm-crypt partition. The fs was created with
> mkfs -O no-holes -d single -m raid1 and mounted with
> clear_cache,space_cache=v2.
>
> After that issue, I tried to update my backups, but it didn't go very well:
>
> [918052.104923] BTRFS warning (device dm-19): dm-19 checksum verify
> failed on 21374567694336 wanted 6c2341bd found 640f30e8 level 0
> [918052.121801] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.121818] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397271552
> [918052.138406] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.139080] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.139094] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397275648
> [918052.139768] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.140158] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.140171] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397279744
> [918052.140532] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.140868] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.140880] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397283840
> [918052.159794] BTRFS warning (device dm-19): csum failed root 5 ino
> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
> [918052.213959] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.216411] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.216418] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397271552
> [918052.216570] BTRFS warning (device dm-19): csum failed root 5 ino
> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
> [918052.216790] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918052.217037] BTRFS info (device dm-19): no csum found for inode
> 219677 start 3397271552
> [918052.217165] BTRFS warning (device dm-19): csum failed root 5 ino
> 219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
> [918062.496366] BTRFS warning (device dm-19): dm-19 checksum verify
> failed on 20043100192768 wanted 8bf4c1c9 found 23b8dea8 level 0
> [918062.504832] verify_parent_transid: 1 callbacks suppressed
> [918062.504836] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.504848] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 28672
> [918062.506174] BTRFS error (device dm-19): parent transid verify failed
> on 20043100241920 wanted 45126 found 45104
> [918062.509823] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 28672 csum 0xa0f9eaf7 expected csum 0x00000000 mirror 1
> [918062.513826] BTRFS error (device dm-19): parent transid verify failed
> on 20043100241920 wanted 45126 found 45104
> [918062.513838] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 49152
> [918062.519723] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 49152 csum 0x76421bb0 expected csum 0x00000000 mirror 1
> [918062.530287] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.530866] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.530874] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 65536
> [918062.539238] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.539560] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.539566] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 69632
> [918062.540020] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.540261] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.540269] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 77824
> [918062.540965] BTRFS error (device dm-19): parent transid verify failed
> on 20043100192768 wanted 45126 found 45104
> [918062.541199] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 114688
> [918062.541753] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 118784
> [918062.544496] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 65536 csum 0xdcb45a27 expected csum 0x00000000 mirror 1
> [918062.547421] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 114688 csum 0x1d5a0f30 expected csum 0x00000000 mirror 1
> [918062.549968] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 69632 csum 0x24dd186c expected csum 0x00000000 mirror 1
> [918062.572502] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 77824 csum 0xba341c47 expected csum 0x00000000 mirror 1
> [918062.575300] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 118784 csum 0x519e7044 expected csum 0x00000000 mirror 1
> [918062.584005] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 172032
> [918062.586782] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 221184
> [918062.587106] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 221184 csum 0xc9c50b2d expected csum 0x00000000 mirror 1
> [918062.590739] BTRFS info (device dm-19): no csum found for inode
> 5464098 start 450560
> [918062.591255] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 450560 csum 0xd2e5e3eb expected csum 0x00000000 mirror 1
> [918062.601361] BTRFS warning (device dm-19): dm-19 checksum verify
> failed on 20043100258304 wanted 61da40d5 found 204cd10a level 0
> [918062.602875] BTRFS warning (device dm-19): csum failed root 5 ino
> 5464098 off 503808 csum 0x183beedc expected csum 0x00000000 mirror 1
> [918085.820716] verify_parent_transid: 42 callbacks suppressed
> [918085.820730] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.837964] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.837992] __btrfs_lookup_bio_sums: 17 callbacks suppressed
> [918085.837994] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159711232
> [918085.838459] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.838987] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.838995] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159715328
> [918085.839349] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.840054] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.840065] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159719424
> [918085.840402] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.840943] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.840951] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159723520
> [918085.850438] btrfs_print_data_csum_error: 17 callbacks suppressed
> [918085.850442] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
> [918085.959941] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.960447] BTRFS error (device dm-19): parent transid verify failed
> on 21374567694336 wanted 45126 found 45104
> [918085.960452] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159711232
> [918085.960624] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
> [918085.961220] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 159711232
> [918085.961341] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
> [918087.026984] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 193855488
> [918087.028152] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 193859584
> [918087.028598] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 193863680
> [918087.029005] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 193867776
> [918087.042810] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
> [918087.054812] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193904640 csum 0x279aea06 expected csum 0x00000000 mirror 1
> [918087.071004] BTRFS warning (device dm-19): dm-19 checksum verify
> failed on 20043549310976 wanted fcc4f73a found 3d3d8e83 level 0
> [918087.103303] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193937408 csum 0x1fedc264 expected csum 0x00000000 mirror 1
> [918087.118559] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
> [918087.131234] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
> [918087.132513] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
> [918087.135749] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
> [918119.990544] BTRFS warning (device dm-19): dm-19 checksum verify
> failed on 20043135205376 wanted 79d70254 found 93ac2b61 level 0
> [918120.010421] verify_parent_transid: 105 callbacks suppressed
> [918120.010428] BTRFS error (device dm-19): parent transid verify failed
> on 20043135205376 wanted 45126 found 45104
> [918120.010565] __btrfs_lookup_bio_sums: 48 callbacks suppressed
> [918120.010567] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 678543360
> [918120.028733] btrfs_print_data_csum_error: 5 callbacks suppressed
> [918120.028735] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
> [918120.040591] BTRFS error (device dm-19): parent transid verify failed
> on 20043135205376 wanted 45126 found 45104
> [918120.041097] BTRFS error (device dm-19): parent transid verify failed
> on 20043135205376 wanted 45126 found 45104
> [918120.041112] BTRFS info (device dm-19): no csum found for inode
> 5468073 start 678543360
> [918120.041440] BTRFS warning (device dm-19): csum failed root 5 ino
> 5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
Since you haven't rebooted, all these mismatch can be caused by aborted
transaction.
>
>
> Fortunately, after a reboot, everything appears healthy, no file
> corruption that I can observe, but I'm still running some checks.
That's great, write time tree checker prevented corruption.
But still, btrfs check, and scrub recommended.
And the most important thing, please provide the missing key log line.
Thanks,
Qu
>
> I'm not sure what the write tree is or if this is related to the read
> tree issues mentioned on the mailing list, any advice/pointers would be
> most appreciated. One thing I can note is that I've started to write
> heavily to this filesystem via nfs export (the nfs client happens to be
> a vm on the same machine, fwiw).
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* write time tree block corruption detected
@ 2020-01-17 8:21 Kenneth Topp
2020-01-17 8:33 ` Qu Wenruo
0 siblings, 1 reply; 16+ messages in thread
From: Kenneth Topp @ 2020-01-17 8:21 UTC (permalink / raw)
To: linux-btrfs
I've been running 5.4.8 for 10 days and this popped up:
[917767.192190] BTRFS error (device dm-19): block=21374679056384 write
time tree block corruption detected
[917767.193436] BTRFS: error (device dm-19) in btrfs_sync_log:3104:
errno=-5 IO failure
[917767.193438] BTRFS info (device dm-19): forced readonly
This was a btrfs filesystem created on 5.4.8 with two 4kn drives, each
with a single whole disk dm-crypt partition. The fs was created with
mkfs -O no-holes -d single -m raid1 and mounted with
clear_cache,space_cache=v2.
After that issue, I tried to update my backups, but it didn't go very
well:
[918052.104923] BTRFS warning (device dm-19): dm-19 checksum verify
failed on 21374567694336 wanted 6c2341bd found 640f30e8 level 0
[918052.121801] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918052.121818] BTRFS info (device dm-19): no csum found for inode
219677 start 3397271552
[918052.138406] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918052.139080] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918052.139094] BTRFS info (device dm-19): no csum found for inode
219677 start 3397275648
[918052.139768] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918052.140158] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918052.140171] BTRFS info (device dm-19): no csum found for inode
219677 start 3397279744
[918052.140532] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918052.140868] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918052.140880] BTRFS info (device dm-19): no csum found for inode
219677 start 3397283840
[918052.159794] BTRFS warning (device dm-19): csum failed root 5 ino
219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
[918052.213959] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918052.216411] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918052.216418] BTRFS info (device dm-19): no csum found for inode
219677 start 3397271552
[918052.216570] BTRFS warning (device dm-19): csum failed root 5 ino
219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
[918052.216790] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918052.217037] BTRFS info (device dm-19): no csum found for inode
219677 start 3397271552
[918052.217165] BTRFS warning (device dm-19): csum failed root 5 ino
219677 off 3397271552 csum 0x33c9a639 expected csum 0x00000000 mirror 1
[918062.496366] BTRFS warning (device dm-19): dm-19 checksum verify
failed on 20043100192768 wanted 8bf4c1c9 found 23b8dea8 level 0
[918062.504832] verify_parent_transid: 1 callbacks suppressed
[918062.504836] BTRFS error (device dm-19): parent transid verify failed
on 20043100192768 wanted 45126 found 45104
[918062.504848] BTRFS info (device dm-19): no csum found for inode
5464098 start 28672
[918062.506174] BTRFS error (device dm-19): parent transid verify failed
on 20043100241920 wanted 45126 found 45104
[918062.509823] BTRFS warning (device dm-19): csum failed root 5 ino
5464098 off 28672 csum 0xa0f9eaf7 expected csum 0x00000000 mirror 1
[918062.513826] BTRFS error (device dm-19): parent transid verify failed
on 20043100241920 wanted 45126 found 45104
[918062.513838] BTRFS info (device dm-19): no csum found for inode
5464098 start 49152
[918062.519723] BTRFS warning (device dm-19): csum failed root 5 ino
5464098 off 49152 csum 0x76421bb0 expected csum 0x00000000 mirror 1
[918062.530287] BTRFS error (device dm-19): parent transid verify failed
on 20043100192768 wanted 45126 found 45104
[918062.530866] BTRFS error (device dm-19): parent transid verify failed
on 20043100192768 wanted 45126 found 45104
[918062.530874] BTRFS info (device dm-19): no csum found for inode
5464098 start 65536
[918062.539238] BTRFS error (device dm-19): parent transid verify failed
on 20043100192768 wanted 45126 found 45104
[918062.539560] BTRFS error (device dm-19): parent transid verify failed
on 20043100192768 wanted 45126 found 45104
[918062.539566] BTRFS info (device dm-19): no csum found for inode
5464098 start 69632
[918062.540020] BTRFS error (device dm-19): parent transid verify failed
on 20043100192768 wanted 45126 found 45104
[918062.540261] BTRFS error (device dm-19): parent transid verify failed
on 20043100192768 wanted 45126 found 45104
[918062.540269] BTRFS info (device dm-19): no csum found for inode
5464098 start 77824
[918062.540965] BTRFS error (device dm-19): parent transid verify failed
on 20043100192768 wanted 45126 found 45104
[918062.541199] BTRFS info (device dm-19): no csum found for inode
5464098 start 114688
[918062.541753] BTRFS info (device dm-19): no csum found for inode
5464098 start 118784
[918062.544496] BTRFS warning (device dm-19): csum failed root 5 ino
5464098 off 65536 csum 0xdcb45a27 expected csum 0x00000000 mirror 1
[918062.547421] BTRFS warning (device dm-19): csum failed root 5 ino
5464098 off 114688 csum 0x1d5a0f30 expected csum 0x00000000 mirror 1
[918062.549968] BTRFS warning (device dm-19): csum failed root 5 ino
5464098 off 69632 csum 0x24dd186c expected csum 0x00000000 mirror 1
[918062.572502] BTRFS warning (device dm-19): csum failed root 5 ino
5464098 off 77824 csum 0xba341c47 expected csum 0x00000000 mirror 1
[918062.575300] BTRFS warning (device dm-19): csum failed root 5 ino
5464098 off 118784 csum 0x519e7044 expected csum 0x00000000 mirror 1
[918062.584005] BTRFS info (device dm-19): no csum found for inode
5464098 start 172032
[918062.586782] BTRFS info (device dm-19): no csum found for inode
5464098 start 221184
[918062.587106] BTRFS warning (device dm-19): csum failed root 5 ino
5464098 off 221184 csum 0xc9c50b2d expected csum 0x00000000 mirror 1
[918062.590739] BTRFS info (device dm-19): no csum found for inode
5464098 start 450560
[918062.591255] BTRFS warning (device dm-19): csum failed root 5 ino
5464098 off 450560 csum 0xd2e5e3eb expected csum 0x00000000 mirror 1
[918062.601361] BTRFS warning (device dm-19): dm-19 checksum verify
failed on 20043100258304 wanted 61da40d5 found 204cd10a level 0
[918062.602875] BTRFS warning (device dm-19): csum failed root 5 ino
5464098 off 503808 csum 0x183beedc expected csum 0x00000000 mirror 1
[918085.820716] verify_parent_transid: 42 callbacks suppressed
[918085.820730] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918085.837964] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918085.837992] __btrfs_lookup_bio_sums: 17 callbacks suppressed
[918085.837994] BTRFS info (device dm-19): no csum found for inode
5468073 start 159711232
[918085.838459] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918085.838987] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918085.838995] BTRFS info (device dm-19): no csum found for inode
5468073 start 159715328
[918085.839349] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918085.840054] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918085.840065] BTRFS info (device dm-19): no csum found for inode
5468073 start 159719424
[918085.840402] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918085.840943] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918085.840951] BTRFS info (device dm-19): no csum found for inode
5468073 start 159723520
[918085.850438] btrfs_print_data_csum_error: 17 callbacks suppressed
[918085.850442] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
[918085.959941] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918085.960447] BTRFS error (device dm-19): parent transid verify failed
on 21374567694336 wanted 45126 found 45104
[918085.960452] BTRFS info (device dm-19): no csum found for inode
5468073 start 159711232
[918085.960624] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
[918085.961220] BTRFS info (device dm-19): no csum found for inode
5468073 start 159711232
[918085.961341] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 159711232 csum 0x1e34dbf6 expected csum 0x00000000 mirror 1
[918087.026984] BTRFS info (device dm-19): no csum found for inode
5468073 start 193855488
[918087.028152] BTRFS info (device dm-19): no csum found for inode
5468073 start 193859584
[918087.028598] BTRFS info (device dm-19): no csum found for inode
5468073 start 193863680
[918087.029005] BTRFS info (device dm-19): no csum found for inode
5468073 start 193867776
[918087.042810] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
[918087.054812] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 193904640 csum 0x279aea06 expected csum 0x00000000 mirror 1
[918087.071004] BTRFS warning (device dm-19): dm-19 checksum verify
failed on 20043549310976 wanted fcc4f73a found 3d3d8e83 level 0
[918087.103303] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 193937408 csum 0x1fedc264 expected csum 0x00000000 mirror 1
[918087.118559] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
[918087.131234] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
[918087.132513] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 193855488 csum 0x86af03b1 expected csum 0x00000000 mirror 1
[918087.135749] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 193986560 csum 0xc47dad65 expected csum 0x00000000 mirror 1
[918119.990544] BTRFS warning (device dm-19): dm-19 checksum verify
failed on 20043135205376 wanted 79d70254 found 93ac2b61 level 0
[918120.010421] verify_parent_transid: 105 callbacks suppressed
[918120.010428] BTRFS error (device dm-19): parent transid verify failed
on 20043135205376 wanted 45126 found 45104
[918120.010565] __btrfs_lookup_bio_sums: 48 callbacks suppressed
[918120.010567] BTRFS info (device dm-19): no csum found for inode
5468073 start 678543360
[918120.028733] btrfs_print_data_csum_error: 5 callbacks suppressed
[918120.028735] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
[918120.040591] BTRFS error (device dm-19): parent transid verify failed
on 20043135205376 wanted 45126 found 45104
[918120.041097] BTRFS error (device dm-19): parent transid verify failed
on 20043135205376 wanted 45126 found 45104
[918120.041112] BTRFS info (device dm-19): no csum found for inode
5468073 start 678543360
[918120.041440] BTRFS warning (device dm-19): csum failed root 5 ino
5468073 off 678543360 csum 0xcd3c1a5d expected csum 0x00000000 mirror 1
Fortunately, after a reboot, everything appears healthy, no file
corruption that I can observe, but I'm still running some checks.
I'm not sure what the write tree is or if this is related to the read
tree issues mentioned on the mailing list, any advice/pointers would be
most appreciated. One thing I can note is that I've started to write
heavily to this filesystem via nfs export (the nfs client happens to be
a vm on the same machine, fwiw).
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2021-06-11 19:45 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-12 13:49 Write time tree block corruption detected Spencer Collyer
2020-08-12 14:15 ` Qu Wenruo
2020-08-13 5:53 ` Spencer Collyer
2020-08-13 5:58 ` Qu Wenruo
-- strict thread matches above, loose matches on Subject: below --
2021-05-30 21:30 auxsvr
2021-06-07 11:55 ` auxsvr
2021-06-08 14:45 ` auxsvr
2021-06-08 22:39 ` Qu Wenruo
2021-06-09 1:05 ` Qu Wenruo
2021-06-09 8:04 ` auxsvr
2021-06-09 8:32 ` Qu Wenruo
2021-06-11 19:44 ` auxsvr
2020-01-17 8:21 write " Kenneth Topp
2020-01-17 8:33 ` Qu Wenruo
2020-01-17 8:48 ` Kenneth Topp
2020-01-17 10:09 ` Qu Wenruo
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.