* corrupt leaf; unaligned key offset for csum item
@ 2018-06-09 12:22 Simon Kaiser
2018-06-09 22:53 ` Chris Murphy
2018-06-10 1:56 ` Qu Wenruo
0 siblings, 2 replies; 11+ messages in thread
From: Simon Kaiser @ 2018-06-09 12:22 UTC (permalink / raw)
To: linux-btrfs
Hi all,
I'm having trouble with my btrfs volume on a Samsung SSD 840 pro. On
startup I immediately get the error
BTRFS critical (device sda5): corrupt leaf: root=1 block=73263579136
slot=105, unaligned key offset for csum item, have 1271496708 should be
aligned to 4096
I used to make regular backups, but slacked off recently. Fixing the
filesystem isn't essential in any way, but would save me quite some time
on config tinkering and weeding out vacation photos. The second of
which I don't enjoy at all...
On similar issues faulty RAM seemed to be the most prominent cause. I
added a third memory module quite recently which could have caused this,
but I couldn't run memtest long enough yet to verify.
Here's the trimmed dmesg output (full output at
https://pastebin.com/7XF9i09K)
[ 4.062748] BTRFS critical (device sda5): corrupt leaf: root=1 block=73263579136 slot=105, unaligned key offset for csum item, have 1271496708 should be aligned to 4096
[ 4.063806] BTRFS info (device sda5): no csum found for inode 7810710 start 0
[ 4.064260] BTRFS warning (device sda5): csum failed root 257 ino 7810710 off 109201939968 csum 0x5307fa6f expected csum 0x00000000 mirror 1
[ 4.075834] BTRFS info (device sdb3): use lzo compression, level 0
[ 4.075836] BTRFS info (device sdb3): enabling auto defrag
[ 4.075837] BTRFS info (device sdb3): disk space caching is enabled
`brfs inspect-internal dump-tree -b 73263579136 /dev/sda5` (trimmed,
full output at https://pastebin.com/dqBJ3b6D)
btrfs-progs v4.16.1
leaf 73263579136 items 194 free space 4089 generation 530006 owner CSUM_TREE
leaf 73263579136 flags 0x1(WRITTEN) backref revision 1
fs uuid 95b4974b-a798-44b3-99aa-a4eef990aeeb
chunk uuid 9c8f46c3-ba51-4175-857c-8041543fa813
item 0 key (EXTENT_CSUM EXTENT_CSUM 1268850688) itemoff 16263 itemsize 20
range start 1268850688 end 1268871168 length 20480
item 1 key (EXTENT_CSUM EXTENT_CSUM 1268871168) itemoff 16259 itemsize 4
range start 1268871168 end 1268875264 length 4096
item 2 key (EXTENT_CSUM EXTENT_CSUM 1268875264) itemoff 16255 itemsize 4
range start 1268875264 end 1268879360 length 4096
item 3 key (EXTENT_CSUM EXTENT_CSUM 1268879360) itemoff 16239 itemsize 16
range start 1268879360 end 1268895744 length 16384
item 4 key (EXTENT_CSUM EXTENT_CSUM 1268895744) itemoff 16235 itemsize 4
range start 1268895744 end 1268899840 length 4096
item 5 key (EXTENT_CSUM EXTENT_CSUM 1268899840) itemoff 16223 itemsize 12
range start 1268899840 end 1268912128 length 12288
[...]
item 100 key (EXTENT_CSUM EXTENT_CSUM 1271410688) itemoff 13827 itemsize 8
range start 1271410688 end 1271418880 length 8192
item 101 key (EXTENT_CSUM EXTENT_CSUM 1271418880) itemoff 13811 itemsize 16
range start 1271418880 end 1271435264 length 16384
item 102 key (EXTENT_CSUM EXTENT_CSUM 1271435264) itemoff 13807 itemsize 4
range start 1271435264 end 1271439360 length 4096
item 103 key (EXTENT_CSUM EXTENT_CSUM 1271439360) itemoff 13771 itemsize 36
range start 1271439360 end 1271476224 length 36864
item 104 key (EXTENT_CSUM EXTENT_CSUM 1271488512) itemoff 13763 itemsize 8
range start 1271488512 end 1271496704 length 8192
item 105 key (EXTENT_CSUM EXTENT_CSUM 1271496708) itemoff 13403 itemsize 360
range start 1271496708 end 1271865348 length 368640
item 106 key (EXTENT_CSUM EXTENT_CSUM 1271873536) itemoff 13399 itemsize 4
range start 1271873536 end 1271877632 length 4096
item 107 key (EXTENT_CSUM EXTENT_CSUM 1271877632) itemoff 13379 itemsize 20
range start 1271877632 end 1271898112 length 20480
item 108 key (EXTENT_CSUM EXTENT_CSUM 1271898112) itemoff 13375 itemsize 4
range start 1271898112 end 1271902208 length 4096
item 109 key (EXTENT_CSUM EXTENT_CSUM 1271918592) itemoff 13363 itemsize 12
range start 1271918592 end 1271930880 length 12288
item 110 key (EXTENT_CSUM EXTENT_CSUM 1271930880) itemoff 13351 itemsize 12
range start 1271930880 end 1271943168 length 12288
[...]
`uname -a`
Linux hostname 4.16.12-1-ARCH #1 SMP PREEMPT Fri May 25 23:30:31 UTC 2018 x86_64 GNU/Linux
`btrfs fi show`
Label: 'arch' uuid: 95b4974b-a798-44b3-99aa-a4eef990aeeb
Total devices 1 FS bytes used 78.10GiB
devid 1 size 100.00GiB used 94.05GiB path /dev/sda5
`btrfs fi df`
Data, single: total=90.01GiB, used=76.32GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=4.01GiB, used=1.78GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
Is there any chance this can be fixed? Your help would be greatly
apperciated!
Yours,
Simon
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: corrupt leaf; unaligned key offset for csum item
2018-06-09 12:22 corrupt leaf; unaligned key offset for csum item Simon Kaiser
@ 2018-06-09 22:53 ` Chris Murphy
2018-06-10 1:56 ` Qu Wenruo
1 sibling, 0 replies; 11+ messages in thread
From: Chris Murphy @ 2018-06-09 22:53 UTC (permalink / raw)
To: Simon Kaiser; +Cc: Btrfs BTRFS
On Sat, Jun 9, 2018 at 6:22 AM, Simon Kaiser <simon.kaiser@kit.edu> wrote:
> Hi all,
>
> I'm having trouble with my btrfs volume on a Samsung SSD 840 pro. On
> startup I immediately get the error
>
> BTRFS critical (device sda5): corrupt leaf: root=1 block=73263579136
> slot=105, unaligned key offset for csum item, have 1271496708 should be
> aligned to 4096
>
> I used to make regular backups, but slacked off recently. Fixing the
> filesystem isn't essential in any way, but would save me quite some time
> on config tinkering and weeding out vacation photos. The second of
> which I don't enjoy at all...
>
> On similar issues faulty RAM seemed to be the most prominent cause. I
> added a third memory module quite recently which could have caused this,
> but I couldn't run memtest long enough yet to verify.
>
> Here's the trimmed dmesg output (full output at
> https://pastebin.com/7XF9i09K)
>
> [ 4.062748] BTRFS critical (device sda5): corrupt leaf: root=1 block=73263579136 slot=105, unaligned key offset for csum item, have 1271496708 should be aligned to 4096
> [ 4.063806] BTRFS info (device sda5): no csum found for inode 7810710 start 0
> [ 4.064260] BTRFS warning (device sda5): csum failed root 257 ino 7810710 off 109201939968 csum 0x5307fa6f expected csum 0x00000000 mirror 1
> [ 4.075834] BTRFS info (device sdb3): use lzo compression, level 0
> [ 4.075836] BTRFS info (device sdb3): enabling auto defrag
> [ 4.075837] BTRFS info (device sdb3): disk space caching is enabled
What are your mount options?
I suggest removing the suspected bad ram first, and then boot off
alternate media:
btrfs insp dump-s -f <dev>
Try to mount with -o ro, and if that doesn't work try -o
ro,norecovery. Read only mount is more tolerant of problems than rw
mount. If either work, then update your backups while you have a
chance. It's easier to backup from ro mount than it is to use 'btrfs
restore'
If neither -o ro nor -o ro,norecovery work try:
-o usebackuproot
There might be some recent changes data loss using that option. But if
you're using the discard mount option, it probably will fail.
If -o ro does not work, but -o ro,norecovery did work, it's a
candidate for 'btrfs rescue zero-log' again there might be some recent
change data loss if you zero the log.
btrfs check --repair is a last resort so I don't recommend it until
you've got a backup some other way.
--
Chris Murphy
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: corrupt leaf; unaligned key offset for csum item
2018-06-09 12:22 corrupt leaf; unaligned key offset for csum item Simon Kaiser
2018-06-09 22:53 ` Chris Murphy
@ 2018-06-10 1:56 ` Qu Wenruo
2018-06-10 3:53 ` Chris Murphy
2018-06-11 8:10 ` Qu Wenruo
1 sibling, 2 replies; 11+ messages in thread
From: Qu Wenruo @ 2018-06-10 1:56 UTC (permalink / raw)
To: Simon Kaiser, linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 5583 bytes --]
On 2018年06月09日 20:22, Simon Kaiser wrote:
> Hi all,
>
> I'm having trouble with my btrfs volume on a Samsung SSD 840 pro. On
> startup I immediately get the error
>
> BTRFS critical (device sda5): corrupt leaf: root=1 block=73263579136
> slot=105, unaligned key offset for csum item, have 1271496708 should be
> aligned to 4096
Tree-checker detects something.
And indeed the bytenr is not even aligned to 512.
So definitely a corruption.
>
> I used to make regular backups, but slacked off recently. Fixing the
> filesystem isn't essential in any way, but would save me quite some time
> on config tinkering and weeding out vacation photos. The second of
> which I don't enjoy at all...
>
> On similar issues faulty RAM seemed to be the most prominent cause. I
> added a third memory module quite recently which could have caused this,
> but I couldn't run memtest long enough yet to verify.
Indeed, the corrupted bytenr is 0x4bc98004.
Looks pretty like a bit flip in the 3rd lowest bit.
It can be fixed by manually patching the corrupted leaf to get rid of
the bitflip.
I could provide a special branch of btrfs-progs to fix it easily.
But before that, it's better to do a scrub to see if there is other
similar problems, so I could fix them all.
Thanks,
Qu
>
> Here's the trimmed dmesg output (full output at
> https://pastebin.com/7XF9i09K)
>
> [ 4.062748] BTRFS critical (device sda5): corrupt leaf: root=1 block=73263579136 slot=105, unaligned key offset for csum item, have 1271496708 should be aligned to 4096
> [ 4.063806] BTRFS info (device sda5): no csum found for inode 7810710 start 0
> [ 4.064260] BTRFS warning (device sda5): csum failed root 257 ino 7810710 off 109201939968 csum 0x5307fa6f expected csum 0x00000000 mirror 1
> [ 4.075834] BTRFS info (device sdb3): use lzo compression, level 0
> [ 4.075836] BTRFS info (device sdb3): enabling auto defrag
> [ 4.075837] BTRFS info (device sdb3): disk space caching is enabled
>
> `brfs inspect-internal dump-tree -b 73263579136 /dev/sda5` (trimmed,
> full output at https://pastebin.com/dqBJ3b6D)
>
> btrfs-progs v4.16.1
> leaf 73263579136 items 194 free space 4089 generation 530006 owner CSUM_TREE
> leaf 73263579136 flags 0x1(WRITTEN) backref revision 1
> fs uuid 95b4974b-a798-44b3-99aa-a4eef990aeeb
> chunk uuid 9c8f46c3-ba51-4175-857c-8041543fa813
> item 0 key (EXTENT_CSUM EXTENT_CSUM 1268850688) itemoff 16263 itemsize 20
> range start 1268850688 end 1268871168 length 20480
> item 1 key (EXTENT_CSUM EXTENT_CSUM 1268871168) itemoff 16259 itemsize 4
> range start 1268871168 end 1268875264 length 4096
> item 2 key (EXTENT_CSUM EXTENT_CSUM 1268875264) itemoff 16255 itemsize 4
> range start 1268875264 end 1268879360 length 4096
> item 3 key (EXTENT_CSUM EXTENT_CSUM 1268879360) itemoff 16239 itemsize 16
> range start 1268879360 end 1268895744 length 16384
> item 4 key (EXTENT_CSUM EXTENT_CSUM 1268895744) itemoff 16235 itemsize 4
> range start 1268895744 end 1268899840 length 4096
> item 5 key (EXTENT_CSUM EXTENT_CSUM 1268899840) itemoff 16223 itemsize 12
> range start 1268899840 end 1268912128 length 12288
> [...]
> item 100 key (EXTENT_CSUM EXTENT_CSUM 1271410688) itemoff 13827 itemsize 8
> range start 1271410688 end 1271418880 length 8192
> item 101 key (EXTENT_CSUM EXTENT_CSUM 1271418880) itemoff 13811 itemsize 16
> range start 1271418880 end 1271435264 length 16384
> item 102 key (EXTENT_CSUM EXTENT_CSUM 1271435264) itemoff 13807 itemsize 4
> range start 1271435264 end 1271439360 length 4096
> item 103 key (EXTENT_CSUM EXTENT_CSUM 1271439360) itemoff 13771 itemsize 36
> range start 1271439360 end 1271476224 length 36864
> item 104 key (EXTENT_CSUM EXTENT_CSUM 1271488512) itemoff 13763 itemsize 8
> range start 1271488512 end 1271496704 length 8192
> item 105 key (EXTENT_CSUM EXTENT_CSUM 1271496708) itemoff 13403 itemsize 360
> range start 1271496708 end 1271865348 length 368640
> item 106 key (EXTENT_CSUM EXTENT_CSUM 1271873536) itemoff 13399 itemsize 4
> range start 1271873536 end 1271877632 length 4096
> item 107 key (EXTENT_CSUM EXTENT_CSUM 1271877632) itemoff 13379 itemsize 20
> range start 1271877632 end 1271898112 length 20480
> item 108 key (EXTENT_CSUM EXTENT_CSUM 1271898112) itemoff 13375 itemsize 4
> range start 1271898112 end 1271902208 length 4096
> item 109 key (EXTENT_CSUM EXTENT_CSUM 1271918592) itemoff 13363 itemsize 12
> range start 1271918592 end 1271930880 length 12288
> item 110 key (EXTENT_CSUM EXTENT_CSUM 1271930880) itemoff 13351 itemsize 12
> range start 1271930880 end 1271943168 length 12288
> [...]
>
> `uname -a`
>
> Linux hostname 4.16.12-1-ARCH #1 SMP PREEMPT Fri May 25 23:30:31 UTC 2018 x86_64 GNU/Linux
>
> `btrfs fi show`
>
> Label: 'arch' uuid: 95b4974b-a798-44b3-99aa-a4eef990aeeb
> Total devices 1 FS bytes used 78.10GiB
> devid 1 size 100.00GiB used 94.05GiB path /dev/sda5
>
> `btrfs fi df`
>
> Data, single: total=90.01GiB, used=76.32GiB
> System, single: total=32.00MiB, used=16.00KiB
> Metadata, single: total=4.01GiB, used=1.78GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> Is there any chance this can be fixed? Your help would be greatly
> apperciated!
>
> Yours,
> Simon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: corrupt leaf; unaligned key offset for csum item
2018-06-10 1:56 ` Qu Wenruo
@ 2018-06-10 3:53 ` Chris Murphy
2018-06-10 5:12 ` Qu Wenruo
2018-06-11 8:10 ` Qu Wenruo
1 sibling, 1 reply; 11+ messages in thread
From: Chris Murphy @ 2018-06-10 3:53 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Simon Kaiser, Btrfs BTRFS
On Sat, Jun 9, 2018 at 7:56 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
> Indeed, the corrupted bytenr is 0x4bc98004.
> Looks pretty like a bit flip in the 3rd lowest bit.
>
> It can be fixed by manually patching the corrupted leaf to get rid of
> the bitflip.
> I could provide a special branch of btrfs-progs to fix it easily.
>
> But before that, it's better to do a scrub to see if there is other
> similar problems, so I could fix them all.
>
Do you think Simon should try -o ro and -o ro,norecovery and see if he
can update backups the easy way first? And then use the offline scrub
to check for additional problems?
Simon, the offline scrub is done unmounted with 'btrfs check
--check-data-csum <dev>' and it is a read-only check.
--
Chris Murphy
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: corrupt leaf; unaligned key offset for csum item
2018-06-10 3:53 ` Chris Murphy
@ 2018-06-10 5:12 ` Qu Wenruo
2018-06-10 13:38 ` Simon Kaiser
0 siblings, 1 reply; 11+ messages in thread
From: Qu Wenruo @ 2018-06-10 5:12 UTC (permalink / raw)
To: Chris Murphy; +Cc: Simon Kaiser, Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 1149 bytes --]
On 2018年06月10日 11:53, Chris Murphy wrote:
> On Sat, Jun 9, 2018 at 7:56 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>> Indeed, the corrupted bytenr is 0x4bc98004.
>> Looks pretty like a bit flip in the 3rd lowest bit.
>>
>> It can be fixed by manually patching the corrupted leaf to get rid of
>> the bitflip.
>> I could provide a special branch of btrfs-progs to fix it easily.
>>
>> But before that, it's better to do a scrub to see if there is other
>> similar problems, so I could fix them all.
>>
>
> Do you think Simon should try -o ro and -o ro,norecovery and see if he
> can update backups the easy way first?
If the bit flip is the only problem, I could fix it manually and nothing
is lost, the fs can be used as usual.
> And then use the offline scrub
> to check for additional problems?
I mean online scrub.
I didn't notice extra error, and I don't believe even for a faulty
memory, bit flip is that easy to happen, so on-line scrub should do the
work.
>
> Simon, the offline scrub is done unmounted with 'btrfs check
> --check-data-csum <dev>' and it is a read-only check.
>
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: corrupt leaf; unaligned key offset for csum item
2018-06-10 5:12 ` Qu Wenruo
@ 2018-06-10 13:38 ` Simon Kaiser
2018-06-10 21:30 ` Chris Murphy
0 siblings, 1 reply; 11+ messages in thread
From: Simon Kaiser @ 2018-06-10 13:38 UTC (permalink / raw)
To: Qu Wenruo, Chris Murphy; +Cc: Btrfs BTRFS
Qu Wenruo <quwenruo.btrfs@gmx.com> writes:
> On 2018年06月10日 11:53, Chris Murphy wrote:
>> On Sat, Jun 9, 2018 at 7:56 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>> Indeed, the corrupted bytenr is 0x4bc98004.
>>> Looks pretty like a bit flip in the 3rd lowest bit.
>>>
>>> It can be fixed by manually patching the corrupted leaf to get rid of
>>> the bitflip.
>>> I could provide a special branch of btrfs-progs to fix it easily.
>>>
>>> But before that, it's better to do a scrub to see if there is other
>>> similar problems, so I could fix them all.
>>>
Online scrub `btrfs scrub start -B -r /mnt/` (`mount -o ro,norecovery`)
ERROR: scrubbing /mnt/ failed for device id 1: ret=-1, errno=5 (Input/output error)
scrub canceled for 95b4974b-a798-44b3-99aa-a4eef990aeeb
scrub started at Sun Jun 10 13:25:37 2018 and was aborted after 00:00:03
total bytes scrubbed: 1.09GiB with 0 errors
Offline scrub `btrfs check --check-data-csum /dev/sda5`
Checking filesystem on /dev/sda5
UUID: 95b4974b-a798-44b3-99aa-a4eef990aeeb
checking extents
checking free space cache
checking fs roots
root 257 inode 7984709 errors 1000, some csum missing
root 257 inode 8016535 errors 800, odd csum item
root 407 inode 7984709 errors 1000, some csum missing
ERROR: errors found in fs roots
found 83859496960 bytes used, error(s) found
total csum bytes: 79588008
total tree bytes: 1907539968
total fs tree bytes: 1707540480
total extent tree bytes: 93782016
btree space waste bytes: 338263641
file data blocks allocated: 97127669760
referenced 108545990656
btrfs check --check-data-csum /dev/sda5 10.70s user 1.51s system 82% cpu 14.736 total
>>
>> Do you think Simon should try -o ro and -o ro,norecovery and see if he
>> can update backups the easy way first?
On copying from this drive, btrfs throws the error I posted initally on
lots of files, but the rest look ok. I also dd'ed the partition on a
backup drive, so I should be good to go on.
>
> If the bit flip is the only problem, I could fix it manually and nothing
> is lost, the fs can be used as usual.
>
>> And then use the offline scrub
>> to check for additional problems?
>
> I mean online scrub.
>
> I didn't notice extra error, and I don't believe even for a faulty
> memory, bit flip is that easy to happen, so on-line scrub should do the
> work.
>
>>
>> Simon, the offline scrub is done unmounted with 'btrfs check
>> --check-data-csum <dev>' and it is a read-only check.
>>
>>
Thank you both for looking into this!
Yours,
Simon
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: corrupt leaf; unaligned key offset for csum item
2018-06-10 13:38 ` Simon Kaiser
@ 2018-06-10 21:30 ` Chris Murphy
2018-06-10 22:32 ` Simon Kaiser
0 siblings, 1 reply; 11+ messages in thread
From: Chris Murphy @ 2018-06-10 21:30 UTC (permalink / raw)
To: Simon Kaiser; +Cc: Qu Wenruo, Chris Murphy, Btrfs BTRFS
On Sun, Jun 10, 2018 at 7:38 AM, Simon Kaiser <simon.kaiser@kit.edu> wrote:
> Qu Wenruo <quwenruo.btrfs@gmx.com> writes:
>
>> On 2018年06月10日 11:53, Chris Murphy wrote:
>>> On Sat, Jun 9, 2018 at 7:56 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>> Indeed, the corrupted bytenr is 0x4bc98004.
>>>> Looks pretty like a bit flip in the 3rd lowest bit.
>>>>
>>>> It can be fixed by manually patching the corrupted leaf to get rid of
>>>> the bitflip.
>>>> I could provide a special branch of btrfs-progs to fix it easily.
>>>>
>>>> But before that, it's better to do a scrub to see if there is other
>>>> similar problems, so I could fix them all.
>>>>
>
> Online scrub `btrfs scrub start -B -r /mnt/` (`mount -o ro,norecovery`)
>
> ERROR: scrubbing /mnt/ failed for device id 1: ret=-1, errno=5 (Input/output error)
> scrub canceled for 95b4974b-a798-44b3-99aa-a4eef990aeeb
> scrub started at Sun Jun 10 13:25:37 2018 and was aborted after 00:00:03
> total bytes scrubbed: 1.09GiB with 0 errors
Is there any kernel message at the time of this i/o error? Either
Btrfs or device error?
--
Chris Murphy
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: corrupt leaf; unaligned key offset for csum item
2018-06-10 21:30 ` Chris Murphy
@ 2018-06-10 22:32 ` Simon Kaiser
0 siblings, 0 replies; 11+ messages in thread
From: Simon Kaiser @ 2018-06-10 22:32 UTC (permalink / raw)
To: Chris Murphy; +Cc: Qu Wenruo, Btrfs BTRFS
Chris Murphy <lists@colorremedies.com> writes:
>> Online scrub `btrfs scrub start -B -r /mnt/` (`mount -o ro,norecovery`)
>>
>> ERROR: scrubbing /mnt/ failed for device id 1: ret=-1, errno=5 (Input/output error)
>> scrub canceled for 95b4974b-a798-44b3-99aa-a4eef990aeeb
>> scrub started at Sun Jun 10 13:25:37 2018 and was aborted after 00:00:03
>> total bytes scrubbed: 1.09GiB with 0 errors
>
> Is there any kernel message at the time of this i/o error? Either
> Btrfs or device error?
Around the time of the error, dmesg shows only the btrfs error I
initially posted:
[ 73.453640] BTRFS critical (device sda5): corrupt leaf: root=1 block=73263579136 slot=105, unaligned key offset for csum item, have 1271496708 should be aligned to 4096
[ 73.791479] BTRFS critical (device sda5): corrupt leaf: root=1 block=73263579136 slot=105, unaligned key offset for csum item, have 1271496708 should be aligned to 4096
Yours,
Simon
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: corrupt leaf; unaligned key offset for csum item
2018-06-10 1:56 ` Qu Wenruo
2018-06-10 3:53 ` Chris Murphy
@ 2018-06-11 8:10 ` Qu Wenruo
2018-06-11 16:42 ` Simon Kaiser
1 sibling, 1 reply; 11+ messages in thread
From: Qu Wenruo @ 2018-06-11 8:10 UTC (permalink / raw)
To: Simon Kaiser, linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 6146 bytes --]
On 2018年06月10日 09:56, Qu Wenruo wrote:
>
>
> On 2018年06月09日 20:22, Simon Kaiser wrote:
>> Hi all,
>>
>> I'm having trouble with my btrfs volume on a Samsung SSD 840 pro. On
>> startup I immediately get the error
>>
>> BTRFS critical (device sda5): corrupt leaf: root=1 block=73263579136
>> slot=105, unaligned key offset for csum item, have 1271496708 should be
>> aligned to 4096
>
> Tree-checker detects something.
>
> And indeed the bytenr is not even aligned to 512.
> So definitely a corruption.
>
>>
>> I used to make regular backups, but slacked off recently. Fixing the
>> filesystem isn't essential in any way, but would save me quite some time
>> on config tinkering and weeding out vacation photos. The second of
>> which I don't enjoy at all...
>>
>> On similar issues faulty RAM seemed to be the most prominent cause. I
>> added a third memory module quite recently which could have caused this,
>> but I couldn't run memtest long enough yet to verify.
>
> Indeed, the corrupted bytenr is 0x4bc98004.
> Looks pretty like a bit flip in the 3rd lowest bit.
>
> It can be fixed by manually patching the corrupted leaf to get rid of
> the bitflip.
> I could provide a special branch of btrfs-progs to fix it easily.
>
> But before that, it's better to do a scrub to see if there is other
> similar problems, so I could fix them all.
Since there is no other csum error exposed by btrfs check/scrub, the fix
for this particular csum item key bit flip can be fetched here:
https://github.com/adam900710/btrfs-progs/tree/dirty_fix
The usage will be (after the normal compile procedure)
$ ./btrfs-corrupt-block -X <device>
It needs to be executed offline (the <device> must be unmounted).
Thanks,
Qu
>
> Thanks,
> Qu
>
>>
>> Here's the trimmed dmesg output (full output at
>> https://pastebin.com/7XF9i09K)
>>
>> [ 4.062748] BTRFS critical (device sda5): corrupt leaf: root=1 block=73263579136 slot=105, unaligned key offset for csum item, have 1271496708 should be aligned to 4096
>> [ 4.063806] BTRFS info (device sda5): no csum found for inode 7810710 start 0
>> [ 4.064260] BTRFS warning (device sda5): csum failed root 257 ino 7810710 off 109201939968 csum 0x5307fa6f expected csum 0x00000000 mirror 1
>> [ 4.075834] BTRFS info (device sdb3): use lzo compression, level 0
>> [ 4.075836] BTRFS info (device sdb3): enabling auto defrag
>> [ 4.075837] BTRFS info (device sdb3): disk space caching is enabled
>>
>> `brfs inspect-internal dump-tree -b 73263579136 /dev/sda5` (trimmed,
>> full output at https://pastebin.com/dqBJ3b6D)
>>
>> btrfs-progs v4.16.1
>> leaf 73263579136 items 194 free space 4089 generation 530006 owner CSUM_TREE
>> leaf 73263579136 flags 0x1(WRITTEN) backref revision 1
>> fs uuid 95b4974b-a798-44b3-99aa-a4eef990aeeb
>> chunk uuid 9c8f46c3-ba51-4175-857c-8041543fa813
>> item 0 key (EXTENT_CSUM EXTENT_CSUM 1268850688) itemoff 16263 itemsize 20
>> range start 1268850688 end 1268871168 length 20480
>> item 1 key (EXTENT_CSUM EXTENT_CSUM 1268871168) itemoff 16259 itemsize 4
>> range start 1268871168 end 1268875264 length 4096
>> item 2 key (EXTENT_CSUM EXTENT_CSUM 1268875264) itemoff 16255 itemsize 4
>> range start 1268875264 end 1268879360 length 4096
>> item 3 key (EXTENT_CSUM EXTENT_CSUM 1268879360) itemoff 16239 itemsize 16
>> range start 1268879360 end 1268895744 length 16384
>> item 4 key (EXTENT_CSUM EXTENT_CSUM 1268895744) itemoff 16235 itemsize 4
>> range start 1268895744 end 1268899840 length 4096
>> item 5 key (EXTENT_CSUM EXTENT_CSUM 1268899840) itemoff 16223 itemsize 12
>> range start 1268899840 end 1268912128 length 12288
>> [...]
>> item 100 key (EXTENT_CSUM EXTENT_CSUM 1271410688) itemoff 13827 itemsize 8
>> range start 1271410688 end 1271418880 length 8192
>> item 101 key (EXTENT_CSUM EXTENT_CSUM 1271418880) itemoff 13811 itemsize 16
>> range start 1271418880 end 1271435264 length 16384
>> item 102 key (EXTENT_CSUM EXTENT_CSUM 1271435264) itemoff 13807 itemsize 4
>> range start 1271435264 end 1271439360 length 4096
>> item 103 key (EXTENT_CSUM EXTENT_CSUM 1271439360) itemoff 13771 itemsize 36
>> range start 1271439360 end 1271476224 length 36864
>> item 104 key (EXTENT_CSUM EXTENT_CSUM 1271488512) itemoff 13763 itemsize 8
>> range start 1271488512 end 1271496704 length 8192
>> item 105 key (EXTENT_CSUM EXTENT_CSUM 1271496708) itemoff 13403 itemsize 360
>> range start 1271496708 end 1271865348 length 368640
>> item 106 key (EXTENT_CSUM EXTENT_CSUM 1271873536) itemoff 13399 itemsize 4
>> range start 1271873536 end 1271877632 length 4096
>> item 107 key (EXTENT_CSUM EXTENT_CSUM 1271877632) itemoff 13379 itemsize 20
>> range start 1271877632 end 1271898112 length 20480
>> item 108 key (EXTENT_CSUM EXTENT_CSUM 1271898112) itemoff 13375 itemsize 4
>> range start 1271898112 end 1271902208 length 4096
>> item 109 key (EXTENT_CSUM EXTENT_CSUM 1271918592) itemoff 13363 itemsize 12
>> range start 1271918592 end 1271930880 length 12288
>> item 110 key (EXTENT_CSUM EXTENT_CSUM 1271930880) itemoff 13351 itemsize 12
>> range start 1271930880 end 1271943168 length 12288
>> [...]
>>
>> `uname -a`
>>
>> Linux hostname 4.16.12-1-ARCH #1 SMP PREEMPT Fri May 25 23:30:31 UTC 2018 x86_64 GNU/Linux
>>
>> `btrfs fi show`
>>
>> Label: 'arch' uuid: 95b4974b-a798-44b3-99aa-a4eef990aeeb
>> Total devices 1 FS bytes used 78.10GiB
>> devid 1 size 100.00GiB used 94.05GiB path /dev/sda5
>>
>> `btrfs fi df`
>>
>> Data, single: total=90.01GiB, used=76.32GiB
>> System, single: total=32.00MiB, used=16.00KiB
>> Metadata, single: total=4.01GiB, used=1.78GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> Is there any chance this can be fixed? Your help would be greatly
>> apperciated!
>>
>> Yours,
>> Simon
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: corrupt leaf; unaligned key offset for csum item
2018-06-11 8:10 ` Qu Wenruo
@ 2018-06-11 16:42 ` Simon Kaiser
2018-06-12 1:18 ` Qu Wenruo
0 siblings, 1 reply; 11+ messages in thread
From: Simon Kaiser @ 2018-06-11 16:42 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs
> On 2018年06月10日 09:56, Qu Wenruo wrote:
>
> Since there is no other csum error exposed by btrfs check/scrub, the fix
> for this particular csum item key bit flip can be fetched here:
> https://github.com/adam900710/btrfs-progs/tree/dirty_fix
>
> The usage will be (after the normal compile procedure)
> $ ./btrfs-corrupt-block -X <device>
>
> It needs to be executed offline (the <device> must be unmounted).
>
> Thanks,
> Qu
>
Thank you very much, that fixed the issue :)
btrfs scrub now shows 5 checksum errors on a single file
[ 1869.456426] BTRFS warning (device sda5): checksum error at logical 1271386112 on dev /dev/sda5, physical 1271386112, root 257, inode 8016535, offset 655360, length 4096, links 1 (path: var/log/journal/31d9cc011
2694cdab3887013fb7bbcf9/system@d8ae7586248c492d8d51c44efd7316da-0000000000000001-00056e3429e252bb.journal)
[ 1869.456429] BTRFS error (device sda5): bdev /dev/sda5 errs: wr 1, rd 370, flush 0, corrupt 1, gen 0
[ 1869.457975] BTRFS warning (device sda5): checksum error at logical 1271390208 on dev /dev/sda5, physical 1271390208, root 257, inode 8016535, offset 655360, length 4096, links 1 (path: var/log/journal/31d9cc011
2694cdab3887013fb7bbcf9/system@d8ae7586248c492d8d51c44efd7316da-0000000000000001-00056e3429e252bb.journal)
[ 1869.457978] BTRFS error (device sda5): bdev /dev/sda5 errs: wr 1, rd 370, flush 0, corrupt 2, gen 0
[ 1869.459381] BTRFS warning (device sda5): checksum error at logical 1271394304 on dev /dev/sda5, physical 1271394304, root 257, inode 8016535, offset 655360, length 4096, links 1 (path: var/log/journal/31d9cc011
2694cdab3887013fb7bbcf9/system@d8ae7586248c492d8d51c44efd7316da-0000000000000001-00056e3429e252bb.journal)
[ 1869.459383] BTRFS error (device sda5): bdev /dev/sda5 errs: wr 1, rd 370, flush 0, corrupt 3, gen 0
[ 1869.460827] BTRFS warning (device sda5): checksum error at logical 1271398400 on dev /dev/sda5, physical 1271398400, root 257, inode 8016535, offset 655360, length 4096, links 1 (path: var/log/journal/31d9cc011
2694cdab3887013fb7bbcf9/system@d8ae7586248c492d8d51c44efd7316da-0000000000000001-00056e3429e252bb.journal)
[ 1869.460829] BTRFS error (device sda5): bdev /dev/sda5 errs: wr 1, rd 370, flush 0, corrupt 4, gen 0
[ 1869.462118] BTRFS warning (device sda5): checksum error at logical 1271402496 on dev /dev/sda5, physical 1271402496, root 257, inode 8016535, offset 655360, length 4096, links 1 (path: var/log/journal/31d9cc011
2694cdab3887013fb7bbcf9/system@d8ae7586248c492d8d51c44efd7316da-0000000000000001-00056e3429e252bb.journal)
[ 1869.462120] BTRFS error (device sda5): bdev /dev/sda5 errs: wr 1, rd 370, flush 0, corrupt 5, gen 0
I don't really need that file, can I simply delete it and forget about
it or is there something else I should do?
Yours,
Simon
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: corrupt leaf; unaligned key offset for csum item
2018-06-11 16:42 ` Simon Kaiser
@ 2018-06-12 1:18 ` Qu Wenruo
0 siblings, 0 replies; 11+ messages in thread
From: Qu Wenruo @ 2018-06-12 1:18 UTC (permalink / raw)
To: Simon Kaiser, linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 3854 bytes --]
On 2018年06月12日 00:42, Simon Kaiser wrote:
>> On 2018年06月10日 09:56, Qu Wenruo wrote:
>>
>> Since there is no other csum error exposed by btrfs check/scrub, the fix
>> for this particular csum item key bit flip can be fetched here:
>> https://github.com/adam900710/btrfs-progs/tree/dirty_fix
>>
>> The usage will be (after the normal compile procedure)
>> $ ./btrfs-corrupt-block -X <device>
>>
>> It needs to be executed offline (the <device> must be unmounted).
>>
>> Thanks,
>> Qu
>>
>
> Thank you very much, that fixed the issue :)
>
> btrfs scrub now shows 5 checksum errors on a single file
>
> [ 1869.456426] BTRFS warning (device sda5): checksum error at logical 1271386112 on dev /dev/sda5, physical 1271386112, root 257, inode 8016535, offset 655360, length 4096, links 1 (path: var/log/journal/31d9cc011
> 2694cdab3887013fb7bbcf9/system@d8ae7586248c492d8d51c44efd7316da-0000000000000001-00056e3429e252bb.journal)
> [ 1869.456429] BTRFS error (device sda5): bdev /dev/sda5 errs: wr 1, rd 370, flush 0, corrupt 1, gen 0
> [ 1869.457975] BTRFS warning (device sda5): checksum error at logical 1271390208 on dev /dev/sda5, physical 1271390208, root 257, inode 8016535, offset 655360, length 4096, links 1 (path: var/log/journal/31d9cc011
> 2694cdab3887013fb7bbcf9/system@d8ae7586248c492d8d51c44efd7316da-0000000000000001-00056e3429e252bb.journal)
> [ 1869.457978] BTRFS error (device sda5): bdev /dev/sda5 errs: wr 1, rd 370, flush 0, corrupt 2, gen 0
> [ 1869.459381] BTRFS warning (device sda5): checksum error at logical 1271394304 on dev /dev/sda5, physical 1271394304, root 257, inode 8016535, offset 655360, length 4096, links 1 (path: var/log/journal/31d9cc011
> 2694cdab3887013fb7bbcf9/system@d8ae7586248c492d8d51c44efd7316da-0000000000000001-00056e3429e252bb.journal)
> [ 1869.459383] BTRFS error (device sda5): bdev /dev/sda5 errs: wr 1, rd 370, flush 0, corrupt 3, gen 0
> [ 1869.460827] BTRFS warning (device sda5): checksum error at logical 1271398400 on dev /dev/sda5, physical 1271398400, root 257, inode 8016535, offset 655360, length 4096, links 1 (path: var/log/journal/31d9cc011
> 2694cdab3887013fb7bbcf9/system@d8ae7586248c492d8d51c44efd7316da-0000000000000001-00056e3429e252bb.journal)
> [ 1869.460829] BTRFS error (device sda5): bdev /dev/sda5 errs: wr 1, rd 370, flush 0, corrupt 4, gen 0
> [ 1869.462118] BTRFS warning (device sda5): checksum error at logical 1271402496 on dev /dev/sda5, physical 1271402496, root 257, inode 8016535, offset 655360, length 4096, links 1 (path: var/log/journal/31d9cc011
> 2694cdab3887013fb7bbcf9/system@d8ae7586248c492d8d51c44efd7316da-0000000000000001-00056e3429e252bb.journal)
> [ 1869.462120] BTRFS error (device sda5): bdev /dev/sda5 errs: wr 1, rd 370, flush 0, corrupt 5, gen 0
>
> I don't really need that file, can I simply delete it and forget about
> it or is there something else I should do?
Of course you can delete it, and if nothing else went wrong, it should
fix the problem.
And of course, remove that offending memory stick.
Thanks,
Qu
>
> Yours,
> Simon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2018-06-12 1:18 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-06-09 12:22 corrupt leaf; unaligned key offset for csum item Simon Kaiser
2018-06-09 22:53 ` Chris Murphy
2018-06-10 1:56 ` Qu Wenruo
2018-06-10 3:53 ` Chris Murphy
2018-06-10 5:12 ` Qu Wenruo
2018-06-10 13:38 ` Simon Kaiser
2018-06-10 21:30 ` Chris Murphy
2018-06-10 22:32 ` Simon Kaiser
2018-06-11 8:10 ` Qu Wenruo
2018-06-11 16:42 ` Simon Kaiser
2018-06-12 1:18 ` Qu Wenruo
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.