BTRFS critical: corrupt leaf, slot offset bad; then read-only

BTRFS critical: corrupt leaf, slot offset bad; then read-only

[Lukas Tribus]

[-- Attachment #1: Type: text/plain, Size: 6153 bytes --]

Hi list!


I have btrfs pool consisting of 5x 2,72 TiB LUKS (dm-crypt) partitions 
in RAID1, mounted on Linux 4.4 with btrfs-progs 4.4. I never had any 
crashes or power loss here, but recently about every 60 - 120 minutes 
(while in use) btrfs detects corruptions, aborts the transaction and 
drops to read-only mode.
btrfs still mounts normally without any special options (it does take 
about 60 seconds, which I guess is normal for this kind of size). All 
LUKS partitions have at least 400GiB of free space.

I don't see any HW problems here; I doubt there is a corruption coming 
from the LUKS partition. I did test the RAM but it seems fine in 
multiple memtest86+ amd memtest86 runs.


Are there any known bugs in 4.4? Any suggestions would be greatly 
appreciated!


I have to admit I did not regularly scrub.


Thanks,
Lukas


---
~# uname -a
Linux srv1-dom0 4.4.0-63-generic #84-Ubuntu SMP Wed Feb 1 17:20:32 UTC 
2017 x86_64 x86_64 x86_64 GNU/Linux
~# btrfs --version
btrfs-progs v4.4
~# btrfs fi show
Label: 'dom0-os'  uuid: e475636c-21e0-4563-87d6-91f03c519a62
         Total devices 5 FS bytes used 3.52GiB
         devid    1 size 10.00GiB used 3.53GiB path /dev/sda2
         devid    2 size 10.00GiB used 4.25GiB path /dev/sdb2
         devid    3 size 10.00GiB used 3.28GiB path /dev/sdc2
         devid    4 size 10.00GiB used 4.00GiB path /dev/sdd2
         devid    5 size 10.00GiB used 4.00GiB path /dev/sde2

Label: 'storage_pool'  uuid: f50f980e-7640-49c7-bf8d-20d55cfe6005
         Total devices 5 FS bytes used 5.77TiB
         devid    1 size 2.72TiB used 2.31TiB path /dev/mapper/sda3_crypt
         devid    2 size 2.72TiB used 2.31TiB path /dev/mapper/sdb3_crypt
         devid    3 size 2.72TiB used 2.31TiB path /dev/mapper/sdc3_crypt
         devid    4 size 2.72TiB used 2.31TiB path /dev/mapper/sdd3_crypt
         devid    5 size 2.72TiB used 2.31TiB path /dev/mapper/sde3_crypt
~# btrfs fi df /storage/users/
Data, RAID1: total=5.77TiB, used=5.76TiB
System, RAID1: total=32.00MiB, used=832.00KiB
Metadata, RAID1: total=8.00GiB, used=6.96GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
~#

~#

partial dmesg:
[ 1509.033492] BTRFS: device label storage_pool devid 1 transid 238135 
/dev/dm-5
[ 1510.498804] BTRFS: device label storage_pool devid 2 transid 238135 
/dev/dm-6
[ 1511.980968] BTRFS: device label storage_pool devid 3 transid 238135 
/dev/dm-7
[ 1513.461799] BTRFS: device label storage_pool devid 4 transid 238135 
/dev/dm-8
[ 1514.838757] BTRFS: device label storage_pool devid 5 transid 238135 
/dev/dm-9
[ 1517.726471] BTRFS info (device dm-9): btrfs: use no compression
[ 1517.726477] BTRFS info (device dm-9): disk space caching is enabled
[ 1517.726479] BTRFS: has skinny extents
[ 1569.598633] BTRFS: checking UUID tree
[ 3540.825747] BTRFS critical (device dm-9): corrupt leaf, slot offset 
bad: block=5242107641856,root=1, slot=39
[ 3540.836168] BTRFS critical (device dm-9): corrupt leaf, slot offset 
bad: block=5242107641856,root=1, slot=39
[ 3540.846413] ------------[ cut here ]------------
[ 3540.846432] WARNING: CPU: 2 PID: 2757 at 
/build/linux-mPTI9s/linux-4.4.0/fs/btrfs/extent-tree.c:2930 
btrfs_run_delayed_refs+0x26b/0x2a0 [btrfs]()
[ 3540.846433] BTRFS: Transaction aborted (error -5)
[ 3540.846434] Modules linked in: algif_skcipher af_alg xen_gntdev 
xen_evtchn xenfs xen_privcmd drbg ansi_cprng dm_crypt nls_iso8859_1 
bridge stp llc intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp 
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel serio_raw joydev 
input_leds nuvoton_cir 8250_fintek ie31200_edac mac_hid rc_core lpc_ich 
edac_core shpchp mei_me mei ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad 
ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 
autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq 
async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear 
hid_generic usbhid hid mxm_wmi i915 i2c_algo_bit drm_kms_helper 
aesni_intel aes_x86_64 glue_helper syscopyarea sysfillrect firewire_ohci 
sysimgblt firewire_core fb_sys_fops lrw psmouse
[ 3540.846466]  tg3 gf128mul ablk_helper cryptd crc_itu_t ptp ahci drm 
pps_core libahci fjes wmi video
[ 3540.846473] CPU: 2 PID: 2757 Comm: btrfs-transacti Not tainted 
4.4.0-63-generic #84-Ubuntu
[ 3540.846475] Hardware name: To Be Filled By O.E.M. To Be Filled By 
O.E.M./Z77 Extreme6, BIOS P2.80 07/01/2013
[ 3540.846476]  0000000000000200 0000000002709bc3 ffff88007615fc90 
ffffffff813f8083
[ 3540.846478]  ffff88007615fcd8 ffffffffc048d498 ffff88007615fcc8 
ffffffff810812d2
[ 3540.846479]  ffff8802adf562f8 ffff8802a9c71800 ffff8800056caef0 
ffffffffffffffff
[ 3540.846481] Call Trace:
[ 3540.846486]  [<ffffffff813f8083>] dump_stack+0x63/0x90
[ 3540.846489]  [<ffffffff810812d2>] warn_slowpath_common+0x82/0xc0
[ 3540.846491]  [<ffffffff8108136c>] warn_slowpath_fmt+0x5c/0x80
[ 3540.846500]  [<ffffffffc03f16cd>] ? 
__btrfs_run_delayed_refs+0xcdd/0x1220 [btrfs]
[ 3540.846509]  [<ffffffffc03f4cab>] btrfs_run_delayed_refs+0x26b/0x2a0 
[btrfs]
[ 3540.846520]  [<ffffffffc04837b2>] commit_cowonly_roots+0x22b/0x2c2 
[btrfs]
[ 3540.846530]  [<ffffffffc040a1b6>] 
btrfs_commit_transaction+0x576/0xa90 [btrfs]
[ 3540.846533]  [<ffffffff810c41e0>] ? wake_atomic_t_function+0x60/0x60
[ 3540.846542]  [<ffffffffc04052e9>] transaction_kthread+0x229/0x240 [btrfs]
[ 3540.846558]  [<ffffffffc04050c0>] ? 
btrfs_cleanup_transaction+0x570/0x570 [btrfs]
[ 3540.846560]  [<ffffffff810a0ba8>] kthread+0xd8/0xf0
[ 3540.846562]  [<ffffffff810a0ad0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 3540.846564]  [<ffffffff8183c98f>] ret_from_fork+0x3f/0x70
[ 3540.846566]  [<ffffffff810a0ad0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 3540.846567] ---[ end trace 70830ce6f0e320dd ]---
[ 3540.846587] BTRFS: error (device dm-9) in 
btrfs_run_delayed_refs:2930: errno=-5 IO failure
[ 3540.855086] BTRFS info (device dm-9): forced readonly
[ 3540.855088] BTRFS warning (device dm-9): Skipping commit of aborted 
transaction.
[ 3540.855090] BTRFS: error (device dm-9) in cleanup_transaction:1746: 
errno=-5 IO failure



full dmesg attached and online at:
http://pastebin.com/raw/K8FNNEnS



[-- Attachment #2: dmesg-4.4.0-59-generic-btrfs-trace2.gz --]
[-- Type: application/gzip, Size: 21771 bytes --]

[-- Attachment #3: dmesg-4.4.0-47-generic-btrfs-trace3.gz --]
[-- Type: application/gzip, Size: 24291 bytes --]

[-- Attachment #4: dmesg-4.4.0-63-generic-btrfs-trace4.gz --]
[-- Type: application/gzip, Size: 20436 bytes --]

Re: BTRFS critical: corrupt leaf, slot offset bad; then read-only

[Lukas Tribus]

Upgrading to 4.8, the FS no longer causes a kernel calltrace and does 
not go read-only. It only shows the "corrupt leaf, slot offset bad" message.

A scrub completed without errors on 3 devices, while it was aborted on 2 
devices. Not sure why it was aborted, since there is no error message in 
dmesg?


Any suggestions why the scrub was aborted?



# uname -a
Linux srv1-dom0 4.8.0-36-generic #36~16.04.1-Ubuntu SMP Sun Feb 5 
09:39:57 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# btrfs scrub status /storage/users/
scrub status for f50f980e-7640-49c7-bf8d-20d55cfe6005
         scrub started at Wed Feb 22 00:07:33 2017 and was aborted after 
06:35:42
         total bytes scrubbed: 10.60TiB with 0 errors
/# btrfs scrub status /storage/users/ -d
scrub status for f50f980e-7640-49c7-bf8d-20d55cfe6005
scrub device /dev/dm-5 (id 1) history
         scrub started at Wed Feb 22 00:07:33 2017 and finished after 
06:35:36
         total bytes scrubbed: 2.30TiB with 0 errors
scrub device /dev/dm-6 (id 2) history
         scrub started at Wed Feb 22 00:07:33 2017 and finished after 
06:35:30
         total bytes scrubbed: 2.30TiB with 0 errors
scrub device /dev/dm-7 (id 3) history
         scrub started at Wed Feb 22 00:07:33 2017 and finished after 
06:35:42
         total bytes scrubbed: 2.30TiB with 0 errors
scrub device /dev/dm-8 (id 4) history
         scrub started at Wed Feb 22 00:07:33 2017 and was aborted after 
05:01:37
         total bytes scrubbed: 1.85TiB with 0 errors
scrub device /dev/mapper/sde3_crypt (id 5) history
         scrub started at Wed Feb 22 00:07:33 2017 and was aborted after 
05:01:37
         total bytes scrubbed: 1.85TiB with 0 errors
#dmesg | grep BTRFS
[  929.737119] BTRFS critical (device dm-9): corrupt leaf, slot offset 
bad: block=5242107641856,root=1, slot=39
[19772.594129] BTRFS critical (device dm-9): corrupt leaf, slot offset 
bad: block=5242107641856,root=1, slot=39
[19777.127704] BTRFS critical (device dm-9): corrupt leaf, slot offset 
bad: block=5242107641856,root=1, slot=39
[19777.552191] BTRFS critical (device dm-9): corrupt leaf, slot offset 
bad: block=5242107641856,root=1, slot=39
#

Re: BTRFS critical: corrupt leaf, slot offset bad; then read-only

[Lukas Tribus]

I did a "btrfs check" (--readonly):

Summary:
589x filetype 1 errors 4, no inode ref (--> Files)
597x filetype 2 errors 4, no inode ref (--> Directories)
1183x root xxx inode YYYYYY errors 2001, no inode item, link count wrong

I looked at a handful of reported files which are verifiable via public 
MD5/SHA1 checksums and they are not corrupted, the checksum is correct.

Any hints or suggestions would be much appreciated, please see below for 
the btrfs check output (repeating lines omitted and some filenames 
redacted):

Checking filesystem on /dev/dm-9
UUID: f50f980e-7640-49c7-bf8d-20d55cfe6005
checking extents [.]
[...]
incorrect offsets 14927 14415
bad block 5242107641856

Errors found in extent allocation tree or chunk allocation
checking free space cache [.]
[...]
checking fs roots [.]
[...]
incorrect offsets 14927 14415
incorrect offsets 14927 14415
root 261 inode 127094 errors 500, file extent discount, nbytes wrong
Found file extent holes:
     start: 0, len: 499712
     unresolved ref dir 127093 index 2 namelen 24 name ABC DE Fghij 
Klmnopr.tuv filetype 1 errors 4, no inode ref
root 261 inode 127095 errors 2001, no inode item, link count wrong
     unresolved ref dir 127080 index 13 namelen 17 name 
Whateverdir123456 filetype 2 errors 4, no inode ref
root 261 inode 127097 errors 2001, no inode item, link count wrong
     unresolved ref dir 127080 index 14 namelen 12 name 
WhateverDirectory2 filetype 2 errors 4, no inode ref
root 261 inode 127099 errors 2001, no inode item, link count wrong
     unresolved ref dir 127080 index 15 namelen 11 name AnyDir filetype 
2 errors 4, no inode ref
root 261 inode 127105 errors 2001, no inode item, link count wrong
     unresolved ref dir 127080 index 16 namelen 10 name AnotherDir 
filetype 2 errors 4, no inode ref
root 261 inode 127107 errors 2001, no inode item, link count wrong
     unresolved ref dir 127080 index 17 namelen 11 name Folder11 
filetype 2 errors 4, no inode ref
root 261 inode 127112 errors 2001, no inode item, link count wrong
     unresolved ref dir 126959 index 51 namelen 11 name Folder120 
filetype 2 errors 4, no inode ref
root 261 inode 127114 errors 2001, no inode item, link count wrong
     unresolved ref dir 126146 index 40 namelen 13 name GVC-dir filetype 
2 errors 4, no inode ref
root 261 inode 127396 errors 2001, no inode item, link count wrong
     unresolved ref dir 126146 index 41 namelen 4 name G3-dir filetype 2 
errors 4, no inode ref
root 261 inode 127527 errors 2001, no inode item, link count wrong
     unresolved ref dir 126146 index 42 namelen 11 name Hello Dir 2 
filetype 2 errors 4, no inode ref
root 261 inode 127535 errors 2001, no inode item, link count wrong
     unresolved ref dir 126146 index 43 namelen 4 name Hellodir filetype 
2 errors 4, no inode ref
root 261 inode 127573 errors 2001, no inode item, link count wrong
     unresolved ref dir 126146 index 44 namelen 6 name Hello 2 filetype 
2 errors 4, no inode ref
root 261 inode 127620 errors 2001, no inode item, link count wrong
[...]
root 261 inode 177273 errors 2001, no inode item, link count wrong
     unresolved ref dir 23439 index 23 namelen 24 name Firefox Setup 
51.0.1.exe filetype 1 errors 4, no inode ref
root 261 inode 177275 errors 2001, no inode item, link count wrong
     unresolved ref dir 23439 index 26 namelen 27 name Firefox Setup 
45.7.0esr.exe filetype 1 errors 4, no inode ref
root 261 inode 180457 errors 2001, no inode item, link count wrong
[...]
checking fs roots [o]
incorrect offsets 14927 14415
checking fs roots [.]
[...]
checking fs roots [o]
The following tree block(s) is corrupted in tree 263:
     tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)
checking fs roots [o]
incorrect offsets 14927 14415
checking fs roots [O]
The following tree block(s) is corrupted in tree 6685:
     tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)
checking fs roots [o]
checking fs roots [.]
incorrect offsets 14927 14415
The following tree block(s) is corrupted in tree 6879:
     tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)
checking fs roots [o]
incorrect offsets 14927 14415
incorrect offsets 14927 14415
root 6893 inode 127094 errors 500, file extent discount, nbytes wrong
Found file extent holes:
     start: 0, len: 499712
     unresolved ref dir 127093 index 2 namelen 24 name ABC DE Fghij 
Klmnopr.tuv filetype 1 errors 4, no inode ref
root 6893 inode 127095 errors 2001, no inode item, link count wrong
     unresolved ref dir 127080 index 13 namelen 17 name 
Whateverdir123456 filetype 2 errors 4, no inode ref
root 6893 inode 127097 errors 2001, no inode item, link count wrong
[...]
root 6893 inode 177273 errors 2001, no inode item, link count wrong
     unresolved ref dir 23439 index 23 namelen 24 name Firefox Setup 
51.0.1.exe filetype 1 errors 4, no inode ref
root 6893 inode 177275 errors 2001, no inode item, link count wrong
     unresolved ref dir 23439 index 26 namelen 27 name Firefox Setup 
45.7.0esr.exe filetype 1 errors 4, no inode ref
incorrect offsets 14927 14415
incorrect offsets 14927 14415
root 6896 inode 127094 errors 500, file extent discount, nbytes wrong
Found file extent holes:
     start: 0, len: 499712
     unresolved ref dir 127093 index 2 namelen 24 name ABC DE Fghij 
Klmnopr.tuv filetype 1 errors 4, no inode ref
root 6896 inode 127095 errors 2001, no inode item, link count wrong
     unresolved ref dir 127080 index 13 namelen 17 name 
Whateverdir123456 filetype 2 errors 4, no inode ref
root 6896 inode 127097 errors 2001, no inode item, link count wrong
[...]
root 6896 inode 177273 errors 2001, no inode item, link count wrong
     unresolved ref dir 23439 index 23 namelen 24 name Firefox Setup 
51.0.1.exe filetype 1 errors 4, no inode ref
root 6896 inode 177275 errors 2001, no inode item, link count wrong
     unresolved ref dir 23439 index 26 namelen 27 name Firefox Setup 
45.7.0esr.exe filetype 1 errors 4, no inode ref
The following tree block(s) is corrupted in tree 6893:
     tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)
The following tree block(s) is corrupted in tree 6896:
     tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)

found 4042079987465 bytes used err is 1
total csum bytes: 0
total tree bytes: 179142656
total fs tree bytes: 0
total extent tree bytes: 176128000
btree space waste bytes: 48203815
file data blocks allocated: 1545338880
  referenced 1545338880

Re: BTRFS critical: corrupt leaf, slot offset bad; then read-only

[Hans van Kranenburg]

On 02/22/2017 08:44 AM, Lukas Tribus wrote:
> Upgrading to 4.8, the FS no longer causes a kernel calltrace and does
> not go read-only. It only shows the "corrupt leaf, slot offset bad"
> message.
> 
> A scrub completed without errors on 3 devices, while it was aborted on 2
> devices. Not sure why it was aborted, since there is no error message in
> dmesg?
> 
> Any suggestions why the scrub was aborted?

Maybe because of the "corrupt leaf" error.

> # uname -a
> Linux srv1-dom0 4.8.0-36-generic #36~16.04.1-Ubuntu SMP Sun Feb 5
> 09:39:57 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
> # btrfs scrub status /storage/users/
> scrub status for f50f980e-7640-49c7-bf8d-20d55cfe6005
>         scrub started at Wed Feb 22 00:07:33 2017 and was aborted after
> 06:35:42
>         total bytes scrubbed: 10.60TiB with 0 errors
> /# btrfs scrub status /storage/users/ -d
> scrub status for f50f980e-7640-49c7-bf8d-20d55cfe6005
> scrub device /dev/dm-5 (id 1) history
>         scrub started at Wed Feb 22 00:07:33 2017 and finished after
> 06:35:36
>         total bytes scrubbed: 2.30TiB with 0 errors
> scrub device /dev/dm-6 (id 2) history
>         scrub started at Wed Feb 22 00:07:33 2017 and finished after
> 06:35:30
>         total bytes scrubbed: 2.30TiB with 0 errors
> scrub device /dev/dm-7 (id 3) history
>         scrub started at Wed Feb 22 00:07:33 2017 and finished after
> 06:35:42
>         total bytes scrubbed: 2.30TiB with 0 errors
> scrub device /dev/dm-8 (id 4) history
>         scrub started at Wed Feb 22 00:07:33 2017 and was aborted after
> 05:01:37
>         total bytes scrubbed: 1.85TiB with 0 errors
> scrub device /dev/mapper/sde3_crypt (id 5) history
>         scrub started at Wed Feb 22 00:07:33 2017 and was aborted after
> 05:01:37
>         total bytes scrubbed: 1.85TiB with 0 errors
> #dmesg | grep BTRFS
> [  929.737119] BTRFS critical (device dm-9): corrupt leaf, slot offset
> bad: block=5242107641856,root=1, slot=39
> [19772.594129] BTRFS critical (device dm-9): corrupt leaf, slot offset
> bad: block=5242107641856,root=1, slot=39
> [19777.127704] BTRFS critical (device dm-9): corrupt leaf, slot offset
> bad: block=5242107641856,root=1, slot=39
> [19777.552191] BTRFS critical (device dm-9): corrupt leaf, slot offset
> bad: block=5242107641856,root=1, slot=39

Ok, this is not a csum failure, so probably not the disk giving other
data back than what was sent to it when doing the writes, or a disk
controller which corrupted the data while writing.

And, it's a metadata page, in which part of the entries do not make
sense any more to btrfs. Specifically, it's in root 1, which is the tree
which contains information about all other subtrees containing metadata,
so it's quite an important one.

So, the corruption which is now present in there likely happened in
memory before writing it out. This is also a scenario in which DUP or
RAIDx on disk doesn't help you, because in memory it's stored just once.

If this is a bitflip like thing in memory, it would probably be possible
to spot it and manually correct it (using a patched btrfschk with
bitflip patch, or manually by hexediting++).

Another option is memory corruption or a bug somewhere else in the
kernel, which lead to a memory address of a pointer being changed,
leading to a write to memory end up in the middle of some btrfs metadata
waiting to be checksummed and written to disk.

Question here is... is it easier for you to nuke the filesystem and
restore the files from somewhere else, or do you want to figure out
manually if it's recoverable, and spend some time with dd, hexedit,
reading struct definitions in btrfs kernel C code etc...

If the regular --repair can't fix it (and it can't do magic if you shoot
a hole in it with a shotgun), then there's no automated other tool that
can do it now.

Since it's block 5242107641856 all the time, it might be worthwhile to
have a look at it. Either it's that block, or there's a bigger mess
hidden behind it.

-- 
Hans van Kranenburg

Re: BTRFS critical: corrupt leaf, slot offset bad; then read-only

[Lukas Tribus]

Hello Hans,


Am 22.02.2017 um 20:40 schrieb Hans van Kranenburg:
>
> Question here is... is it easier for you to nuke the filesystem and
> restore the files from somewhere else, or do you want to figure out
> manually if it's recoverable, and spend some time with dd, hexedit,
> reading struct definitions in btrfs kernel C code etc...
>
> If the regular --repair can't fix it (and it can't do magic if you shoot
> a hole in it with a shotgun), then there's no automated other tool that
> can do it now.
>
> Since it's block 5242107641856 all the time, it might be worthwhile to
> have a look at it. Either it's that block, or there's a bigger mess
> hidden behind it.
>

Thanks for all the inputs here and on IRC. I now have a good 
understanding of what can
and what cannot be done realistically.

The files are still fully readable and I'm going to backup as much data 
as I can over the
next few days.

Once that is done, I would like to go over the "btrfs recovery" thread 
and see if it can
be applied for my case as well. I will certainly need your help when 
that time comes...


Thanks for all your help,

Lukas

Re: BTRFS critical: corrupt leaf, slot offset bad; then read-only

[Hans van Kranenburg]

On 02/24/2017 12:47 AM, Lukas Tribus wrote:
> Hello Hans,
> 
> 
> Am 22.02.2017 um 20:40 schrieb Hans van Kranenburg:
>>
>> Question here is... is it easier for you to nuke the filesystem and
>> restore the files from somewhere else, or do you want to figure out
>> manually if it's recoverable, and spend some time with dd, hexedit,
>> reading struct definitions in btrfs kernel C code etc...
>>
>> If the regular --repair can't fix it (and it can't do magic if you shoot
>> a hole in it with a shotgun), then there's no automated other tool that
>> can do it now.
>>
>> Since it's block 5242107641856 all the time, it might be worthwhile to
>> have a look at it. Either it's that block, or there's a bigger mess
>> hidden behind it.
>>
> 
> Thanks for all the inputs here and on IRC. I now have a good
> understanding of what can
> and what cannot be done realistically.
> 
> The files are still fully readable and I'm going to backup as much data
> as I can over the
> next few days.
> 
> Once that is done, I would like to go over the "btrfs recovery" thread
> and see if it can
> be applied for my case as well. I will certainly need your help when
> that time comes...

We can take a stab at it.

-- 
Hans van Kranenburg

Re: BTRFS critical: corrupt leaf, slot offset bad; then read-only

[Lukas Tribus]

Hello Hans,


Am 24.02.2017 um 01:26 schrieb Hans van Kranenburg:
>
>> Once that is done, I would like to go over the "btrfs recovery" thread
>> and see if it can
>> be applied for my case as well. I will certainly need your help when
>> that time comes...
> We can take a stab at it.
>

I upgraded btrfs-tools to 4.8.1 as 4.4 didn't have btrfs 
inspect-internal dump-tree.
But I cannot find anything about 5242107641856 in the dump-tree output.

What does that mean?




Thanks,
Lukas

Re: BTRFS critical: corrupt leaf, slot offset bad; then read-only

[Hans van Kranenburg]

On 03/05/2017 11:50 PM, Lukas Tribus wrote:
> 
> Am 24.02.2017 um 01:26 schrieb Hans van Kranenburg:
>>
>>> Once that is done, I would like to go over the "btrfs recovery" thread
>>> and see if it can
>>> be applied for my case as well. I will certainly need your help when
>>> that time comes...
>> We can take a stab at it.
> 
> I upgraded btrfs-tools to 4.8.1 as 4.4 didn't have btrfs
> inspect-internal dump-tree.
> But I cannot find anything about 5242107641856 in the dump-tree output.
> 
> What does that mean?

I have no idea. It probably means it's gone. Did you use the filesystem
read/write? Are the symptoms also gone?

-- 
Hans van Kranenburg

Re: BTRFS critical: corrupt leaf, slot offset bad; then read-only

[Lukas Tribus]

Am 07.03.2017 um 15:12 schrieb Hans van Kranenburg:
> On 03/05/2017 11:50 PM, Lukas Tribus wrote:
>>
>> I upgraded btrfs-tools to 4.8.1 as 4.4 didn't have btrfs
>> inspect-internal dump-tree.
>> But I cannot find anything about 5242107641856 in the dump-tree output.
>>
>> What does that mean?
> I have no idea. It probably means it's gone. Did you use the filesystem
> read/write? Are the symptoms also gone?
>

Well I read basically everything and copied it to other drivers. Nothing 
appears corrupted
from what I can tell. I didn't write to the pool consciously, although I 
did not mount it
readonly either not that I'm thinking about it ...


btrfs check --readonly reports block corruption (and a number of "no 
inode ref" in files/folders):

Checking filesystem on /dev/mapper/sda3_crypt
UUID: f50f980e-7640-49c7-bf8d-20d55cfe6005
The following tree block(s) is corrupted in tree 261:
         tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)
The following tree block(s) is corrupted in tree 263:
         tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)
The following tree block(s) is corrupted in tree 6685:
         tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)
The following tree block(s) is corrupted in tree 6879:
         tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)
The following tree block(s) is corrupted in tree 6893:
         tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)
The following tree block(s) is corrupted in tree 6896:
         tree block bytenr: 5242107641856, level: 0, node key: 
(5241902333952, 169, 0)
found 4080263675904 bytes used err is 1
total csum bytes: 0
total tree bytes: 181780480
total fs tree bytes: 0
total extent tree bytes: 178765824
btree space waste bytes: 49102341
file data blocks allocated: 1545338880
  referenced 1545338880


Not sure how btrfs check finds a corrupted block that doesn't appear in 
the dump-tree output.


And I had an additional stack trace on the new btrfs pool I was copying 
the data to:

[873067.780479] BTRFS error (device sdf3): bdev /dev/sdf3 errs: wr 0, rd 
1, flush 0, corrupt 0, gen 0
[873067.790639] BTRFS error (device sdf3): bdev /dev/sdf3 errs: wr 0, rd 
2, flush 0, corrupt 0, gen 0
[873067.800708] ------------[ cut here ]------------
[873067.800727] WARNING: CPU: 3 PID: 12942 at 
/build/linux-hwe-6_oOe5/linux-hwe-4.8.0/fs/btrfs/extent-tree.c:6954 
__btrfs_free_extent.isra.71+0x2cb/0xcc0 [btrfs]
[873067.800730] BTRFS: Transaction aborted (error -5)
[873067.800731] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos 
jfs xfs algif_skcipher af_alg xen_gntdev xen_evtchn xenfs xen_privcmd 
dm_crypt intel_rapl x86_pkg_temp_thermal intel_powerclamp nls_iso8859_1 
coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel bridge stp 
llc intel_rapl_perf serio_raw lpc_ich joydev shpchp nuvoton_cir 
input_leds mei_me mei rc_core mac_hid ie31200_edac edac_core ib_iser 
rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi 
scsi_transport_iscsi autofs4 uas usb_storage btrfs raid10 raid456 
async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq 
libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid mxm_wmi 
aesni_intel aes_x86_64 i915 glue_helper lrw i2c_algo_bit ablk_helper tg3 
cryptd drm_kms_helper syscopyarea sysfillrect
[873067.800782]  firewire_ohci ptp sysimgblt psmouse firewire_core 
fb_sys_fops crc_itu_t pps_core ahci drm libahci wmi fjes video
[873067.800791] CPU: 3 PID: 12942 Comm: screen Tainted: G W       
4.8.0-39-generic #42~16.04.1-Ubuntu
[873067.800791] Hardware name: To Be Filled By O.E.M. To Be Filled By 
O.E.M./Z77 Extreme6, BIOS P2.80 07/01/2013
[873067.800793]  0000000000000200 00000000f56bf709 ffff880259f1f908 
ffffffff8142e043
[873067.800795]  ffff880259f1f958 0000000000000000 ffff880259f1f948 
ffffffff8108313b
[873067.800797]  00001b2a59f1faa0 00000000fffffffb 000001cda76bc000 
ffff8802a9fe0d20
[873067.800798] Call Trace:
[873067.800803]  [<ffffffff8142e043>] dump_stack+0x63/0x90
[873067.800805]  [<ffffffff8108313b>] __warn+0xcb/0xf0
[873067.800807]  [<ffffffff810831bf>] warn_slowpath_fmt+0x5f/0x80
[873067.800821]  [<ffffffffc03e22ab>] 
__btrfs_free_extent.isra.71+0x2cb/0xcc0 [btrfs]
[873067.800836]  [<ffffffffc04542af>] ? 
btrfs_merge_delayed_refs+0x8f/0x6a0 [btrfs]
[873067.800846]  [<ffffffffc03e7070>] 
__btrfs_run_delayed_refs+0xb10/0x12c0 [btrfs]
[873067.800857]  [<ffffffff811ad938>] ? set_page_dirty+0x58/0xb0
[873067.800869]  [<ffffffffc0428198>] ? 
set_extent_buffer_dirty+0x78/0xd0 [btrfs]
[873067.800879]  [<ffffffffc03ea8de>] btrfs_run_delayed_refs+0x8e/0x2b0 
[btrfs]
[873067.800890]  [<ffffffffc03fefee>] commit_cowonly_roots+0xae/0x300 
[btrfs]
[873067.800901]  [<ffffffffc0470fa4>] ? 
btrfs_qgroup_account_extents+0x84/0x180 [btrfs]
[873067.800911]  [<ffffffffc0401c33>] 
btrfs_commit_transaction+0x573/0xb00 [btrfs]
[873067.800920]  [<ffffffffc040225e>] ? start_transaction+0x9e/0x4c0 [btrfs]
[873067.800930]  [<ffffffffc03fa38f>] btrfs_commit_super+0x8f/0xa0 [btrfs]
[873067.800939]  [<ffffffffc03fc577>] close_ctree+0x2b7/0x360 [btrfs]
[873067.800947]  [<ffffffffc03cbf29>] btrfs_put_super+0x19/0x20 [btrfs]
[873067.800949]  [<ffffffff8123553f>] generic_shutdown_super+0x6f/0x100
[873067.800950]  [<ffffffff81235852>] kill_anon_super+0x12/0x20
[873067.800966]  [<ffffffffc03ccde8>] btrfs_kill_super+0x18/0x110 [btrfs]
[873067.800968]  [<ffffffff81235a23>] deactivate_locked_super+0x43/0x70
[873067.800969]  [<ffffffff81235efc>] deactivate_super+0x5c/0x60
[873067.800971]  [<ffffffff8125544f>] cleanup_mnt+0x3f/0x90
[873067.800972]  [<ffffffff812554e2>] __cleanup_mnt+0x12/0x20
[873067.800974]  [<ffffffff810a22fe>] task_work_run+0x7e/0xa0
[873067.800975]  [<ffffffff81087001>] do_exit+0x2d1/0xb50
[873067.800977]  [<ffffffff8106b4f5>] ? __do_page_fault+0x265/0x4e0
[873067.800978]  [<ffffffff81087903>] do_group_exit+0x43/0xb0
[873067.800979]  [<ffffffff81087984>] SyS_exit_group+0x14/0x20
[873067.800980]  [<ffffffff8189b7f6>] entry_SYSCALL_64_fastpath+0x1e/0xa8
[873067.800981] ---[ end trace b0a630aaaf9a5946 ]---
[873067.800983] BTRFS: error (device sdf3) in __btrfs_free_extent:6954: 
errno=-5 IO failure
[873067.810114] BTRFS info (device sdf3): forced readonly
[873067.810116] BTRFS: error (device sdf3) in 
btrfs_run_delayed_refs:2960: errno=-5 IO failure
[873067.819481] BTRFS warning (device sdf3): Skipping commit of aborted 
transaction.
[873067.819482] BTRFS: error (device sdf3) in cleanup_transaction:1854: 
errno=-5 IO failure
[873067.828668] BTRFS error (device sdf3): commit super ret -5
[873067.835748] BTRFS error (device sdf3): cleaner transaction attach 
returned -30



I guess its time to rebuild this FS from scratch, unless you have a 
better idea?




Thanks, much appreciated,
Lukas