4.11 relocate crash, null pointer

4.11 relocate crash, null pointer

[Marc MERLIN]

I have a filesystem that sadly got corrupted by a SAS card I just installed yesterday.

I don't think in a case like this, there is there a way to roll back all
writes across all subvolumes in the last 24H, correct?

Is the best thing to go in each subvolume, delete the recent snapshots and
rename the one from 24H as the current one?

BTRFS warning (device dm-5): failed to load free space cache for block group 6746013696000, rebuilding it now
BTRFS warning (device dm-5): block group 6754603630592 has wrong amount of free space
BTRFS warning (device dm-5): failed to load free space cache for block group 6754603630592, rebuilding it now
BTRFS warning (device dm-5): block group 7125178777600 has wrong amount of free space
BTRFS warning (device dm-5): failed to load free space cache for block group 7125178777600, rebuilding it now
BTRFS error (device dm-5): bad tree block start 3981076597540270796 2899180224512
BTRFS error (device dm-5): bad tree block start 942082474969670243 2899180224512
BTRFS: error (device dm-5) in __btrfs_free_extent:6944: errno=-5 IO failure
BTRFS info (device dm-5): forced readonly
BTRFS: error (device dm-5) in btrfs_run_delayed_refs:2961: errno=-5 IO failure
BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: __del_reloc_root+0x3f/0xa6
PGD 189a0e067
PUD 189a0f067
PMD 0

Oops: 0000 [#1] PREEMPT SMP
Modules linked in: veth ip6table_filter ip6_tables ebtable_nat ebtables ppdev lp xt_addrtype br_netfilter bridge stp llc tun autofs4 softdog binfmt_misc ftdi_sio nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ipt_REJECT nf_reject_ipv4 xt_conntrack xt_mark xt_nat xt_tcpudp nf_log_ipv4 nf_log_common xt_LOG iptable_mangle iptable_filter lm85 hwmon_vid pl2303 dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_nat nf_conntrack x_tables sg st snd_pcm_oss snd_mixer_oss bcache kvm_intel kvm irqbypass snd_hda_codec_realtek snd_cmipci snd_hda_codec_generic snd_hda_intel snd_mpu401_uart snd_hda_codec snd_opl3_lib snd_rawmidi snd_hda_core snd_seq_device snd_hwdep eeepc_wmi snd_pcm asus_wmi rc_ati_x10
 asix snd_timer ati_remote sparse_keymap usbnet rfkill snd hwmon soundcore rc_core evdev libphy tpm_infineon pcspkr i915 parport_pc i2c_i801 input_leds mei_me lpc_ich parport tpm_tis battery usbserial tpm_tis_core tpm wmi e1000e ptp pps_core fuse raid456 multipath mmc_block mmc_core lrw ablk_helper dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx crc32c_intel blowfish_x86_64 blowfish_common pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd xhci_pci ehci_pci sata_sil24 xhci_hcd mvsas ehci_hcd r8169 usbcore mii libsas scsi_transport_sas thermal fan [last unloaded: ftdi_sio]
CPU: 0 PID: 9056 Comm: btrfs Tainted: G     U          4.11.0-amd64-preempt-sysrq-20170406 #2
Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
task: ffff88374d2a60c0 task.stack: ffffa6f226424000
RIP: 0010:__del_reloc_root+0x3f/0xa6
RSP: 0018:ffffa6f226427a40 EFLAGS: 00210246
RAX: 0000000000000000 RBX: ffff8838ee256000 RCX: 00000000ffffffe2
RDX: 0000000000000001 RSI: ffffffff9f83b410 RDI: ffff8837992da568
RBP: ffffa6f226427a68 R08: 0000000000000000 R09: ffffffff9fd69480
R10: 0000000000000000 R11: 0000000000000000 R12: ffffa6f226427ab0
R13: ffff883768938000 R14: ffff8837992da568 R15: ffff8837992da570
FS:  00007facd18d28c0(0000) GS:ffff883a5e200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000189a10000 CR4: 00000000001406f0
Call Trace:
 free_reloc_roots+0x4f/0x5d
 merge_reloc_roots+0x159/0x1ba
 relocate_block_group+0x410/0x492
 btrfs_relocate_block_group+0x12d/0x253
 btrfs_relocate_chunk+0x3e/0xb1
 btrfs_balance+0xd16/0xf36
 btrfs_ioctl_balance+0x24f/0x2cd
 ? __alloc_pages_nodemask+0x134/0x1e0
 btrfs_ioctl+0x1447/0x1e22
 ? mem_cgroup_charge_statistics+0x1e/0x88
 ? get_page+0x9/0x26
 ? __lru_cache_add+0x2a/0x6c
 ? set_pte_at+0x9/0xd
 ? __handle_mm_fault+0x61d/0xa6f
 vfs_ioctl+0x21/0x38
 ? vfs_ioctl+0x21/0x38
 do_vfs_ioctl+0x4ef/0x537
 ? current_kernel_time64+0x10/0x36
 ? __audit_syscall_entry+0xc2/0xe6
 ? syscall_trace_enter+0x1ac/0x20e
 SyS_ioctl+0x57/0x7b
 do_syscall_64+0x6b/0x7d
 entry_SYSCALL64_slow_path+0x25/0x25
RIP: 0033:0x7facd097ecc7
RSP: 002b:00007ffefd3c3128 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007facd097ecc7
RDX: 00007ffefd3c31b8 RSI: 00000000c4009420 RDI: 0000000000000003
RBP: 00007ffefd3c31b8 R08: 0000000000000003 R09: 0000000000008040
R10: 0000000000000541 R11: 0000000000000206 R12: 0000000000000003
R13: 00007ffefd3c4cc9 R14: 0000000000000001 R15: 0000000000000001
Code: af f0 01 00 00 48 89 fb 4d 8b b5 10 0b 00 00 4d 8d be 70 05 00 00 49 81 c6 68 05 00 00 4c 89 ff e8 0f 44 43 00 48 8b 03 4c 89 f7 <48> 8b 30 e8 0e fc ff ff 48 85 c0 49 89 c4 74 0b 4c 89 f6 48 89
RIP: __del_reloc_root+0x3f/0xa6 RSP: ffffa6f226427a40
CR2: 0000000000000000
---[ end trace 64c3fa4dc953d295 ]---
Kernel panic - not syncing: Fatal exception
Kernel Offset: 0x1e000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Rebooting in 20 seconds..
ACPI MEMORY or I/O RESET_REG.

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Marc MERLIN]

So, I forgot to mention that it's my main media and backup server that got
corrupted. Yes, I do actually have a backup of a backup server, but it's
going to take days to recover due to the amount of data to copy back, not
counting lots of manual typing due to the number of subvolumes, btrfs
send/receive relationships and so forth.

Really, I should be able to roll back all writes from the last 24H, run a
check --repair/scrub on top just to be sure, and be back on track.

In the meantime, the good news is that the filesystem doesn't crash the
kernel (the poasted crash below) now that I was able to cancel the btrfs balance, 
but it goes read only at the drop of a hat, even when I'm trying to delete
recent snapshots and all data that was potentially written in the last 24H

On Mon, May 01, 2017 at 10:06:41AM -0700, Marc MERLIN wrote:
> I have a filesystem that sadly got corrupted by a SAS card I just installed yesterday.
> 
> I don't think in a case like this, there is there a way to roll back all
> writes across all subvolumes in the last 24H, correct?
> 
> Is the best thing to go in each subvolume, delete the recent snapshots and
> rename the one from 24H as the current one?
 
Well, just like I expected, it's a pain in the rear and this can't even help
fix the top level mountpoint which doesn't have snapshots, so I can't roll
it back.
btrfs should really have an easy way to roll back X hours, or days to
recover from garbage written after a good known point, given that it is COW
afterall.

Is there a way do this with check --repair maybe?

In the meantime, I got stuck while trying to delete snapshots:

Let's say I have this:
ID 428 gen 294021 top level 5 path backup
ID 2023 gen 294021 top level 5 path Soft
ID 3021 gen 294051 top level 428 path backup/debian32
ID 4400 gen 294018 top level 428 path backup/debian64
ID 4930 gen 294019 top level 428 path backup/ubuntu

I can easily
Delete subvolume (no-commit): '/mnt/btrfs_pool2/Soft'
and then:
gargamel:/mnt/btrfs_pool2# mv Soft_rw.20170430_01:50:22 Soft

But I can't delete backup, which actually is mostly only a directory
containing other things (in hindsight I shouldn't have made that a
subvolume)
Delete subvolume (no-commit): '/mnt/btrfs_pool2/backup'
ERROR: cannot delete '/mnt/btrfs_pool2/backup': Directory not empty

This is because backup has a lot of subvolumes due to btrfs send/receive
relationships.

Is it possible to recover there? Can you reparent subvolumes to a different
subvolume without doing a full copy via btrfs send/receive?

Thanks,
Marc

> BTRFS warning (device dm-5): failed to load free space cache for block group 6746013696000, rebuilding it now
> BTRFS warning (device dm-5): block group 6754603630592 has wrong amount of free space
> BTRFS warning (device dm-5): failed to load free space cache for block group 6754603630592, rebuilding it now
> BTRFS warning (device dm-5): block group 7125178777600 has wrong amount of free space
> BTRFS warning (device dm-5): failed to load free space cache for block group 7125178777600, rebuilding it now
> BTRFS error (device dm-5): bad tree block start 3981076597540270796 2899180224512
> BTRFS error (device dm-5): bad tree block start 942082474969670243 2899180224512
> BTRFS: error (device dm-5) in __btrfs_free_extent:6944: errno=-5 IO failure
> BTRFS info (device dm-5): forced readonly
> BTRFS: error (device dm-5) in btrfs_run_delayed_refs:2961: errno=-5 IO failure
> BUG: unable to handle kernel NULL pointer dereference at           (null)
> IP: __del_reloc_root+0x3f/0xa6
> PGD 189a0e067
> PUD 189a0f067
> PMD 0
> 
> Oops: 0000 [#1] PREEMPT SMP
> Modules linked in: veth ip6table_filter ip6_tables ebtable_nat ebtables ppdev lp xt_addrtype br_netfilter bridge stp llc tun autofs4 softdog binfmt_misc ftdi_sio nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ipt_REJECT nf_reject_ipv4 xt_conntrack xt_mark xt_nat xt_tcpudp nf_log_ipv4 nf_log_common xt_LOG iptable_mangle iptable_filter lm85 hwmon_vid pl2303 dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_nat nf_conntrack x_tables sg st snd_pcm_oss snd_mixer_oss bcache kvm_intel kvm irqbypass snd_hda_codec_realtek snd_cmipci snd_hda_codec_generic snd_hda_intel snd_mpu401_uart snd_hda_codec snd_opl3_lib snd_rawmidi snd_hda_core snd_seq_device snd_hwdep eeepc_wmi snd_pcm asus_wmi rc_ati_x10
>  asix snd_timer ati_remote sparse_keymap usbnet rfkill snd hwmon soundcore rc_core evdev libphy tpm_infineon pcspkr i915 parport_pc i2c_i801 input_leds mei_me lpc_ich parport tpm_tis battery usbserial tpm_tis_core tpm wmi e1000e ptp pps_core fuse raid456 multipath mmc_block mmc_core lrw ablk_helper dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx crc32c_intel blowfish_x86_64 blowfish_common pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd xhci_pci ehci_pci sata_sil24 xhci_hcd mvsas ehci_hcd r8169 usbcore mii libsas scsi_transport_sas thermal fan [last unloaded: ftdi_sio]
> CPU: 0 PID: 9056 Comm: btrfs Tainted: G     U          4.11.0-amd64-preempt-sysrq-20170406 #2
> Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> task: ffff88374d2a60c0 task.stack: ffffa6f226424000
> RIP: 0010:__del_reloc_root+0x3f/0xa6
> RSP: 0018:ffffa6f226427a40 EFLAGS: 00210246
> RAX: 0000000000000000 RBX: ffff8838ee256000 RCX: 00000000ffffffe2
> RDX: 0000000000000001 RSI: ffffffff9f83b410 RDI: ffff8837992da568
> RBP: ffffa6f226427a68 R08: 0000000000000000 R09: ffffffff9fd69480
> R10: 0000000000000000 R11: 0000000000000000 R12: ffffa6f226427ab0
> R13: ffff883768938000 R14: ffff8837992da568 R15: ffff8837992da570
> FS:  00007facd18d28c0(0000) GS:ffff883a5e200000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000000 CR3: 0000000189a10000 CR4: 00000000001406f0
> Call Trace:
>  free_reloc_roots+0x4f/0x5d
>  merge_reloc_roots+0x159/0x1ba
>  relocate_block_group+0x410/0x492
>  btrfs_relocate_block_group+0x12d/0x253
>  btrfs_relocate_chunk+0x3e/0xb1
>  btrfs_balance+0xd16/0xf36
>  btrfs_ioctl_balance+0x24f/0x2cd
>  ? __alloc_pages_nodemask+0x134/0x1e0
>  btrfs_ioctl+0x1447/0x1e22
>  ? mem_cgroup_charge_statistics+0x1e/0x88
>  ? get_page+0x9/0x26
>  ? __lru_cache_add+0x2a/0x6c
>  ? set_pte_at+0x9/0xd
>  ? __handle_mm_fault+0x61d/0xa6f
>  vfs_ioctl+0x21/0x38
>  ? vfs_ioctl+0x21/0x38
>  do_vfs_ioctl+0x4ef/0x537
>  ? current_kernel_time64+0x10/0x36
>  ? __audit_syscall_entry+0xc2/0xe6
>  ? syscall_trace_enter+0x1ac/0x20e
>  SyS_ioctl+0x57/0x7b
>  do_syscall_64+0x6b/0x7d
>  entry_SYSCALL64_slow_path+0x25/0x25
> RIP: 0033:0x7facd097ecc7
> RSP: 002b:00007ffefd3c3128 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
> RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007facd097ecc7
> RDX: 00007ffefd3c31b8 RSI: 00000000c4009420 RDI: 0000000000000003
> RBP: 00007ffefd3c31b8 R08: 0000000000000003 R09: 0000000000008040
> R10: 0000000000000541 R11: 0000000000000206 R12: 0000000000000003
> R13: 00007ffefd3c4cc9 R14: 0000000000000001 R15: 0000000000000001
> Code: af f0 01 00 00 48 89 fb 4d 8b b5 10 0b 00 00 4d 8d be 70 05 00 00 49 81 c6 68 05 00 00 4c 89 ff e8 0f 44 43 00 48 8b 03 4c 89 f7 <48> 8b 30 e8 0e fc ff ff 48 85 c0 49 89 c4 74 0b 4c 89 f6 48 89
> RIP: __del_reloc_root+0x3f/0xa6 RSP: ffffa6f226427a40
> CR2: 0000000000000000
> ---[ end trace 64c3fa4dc953d295 ]---
> Kernel panic - not syncing: Fatal exception
> Kernel Offset: 0x1e000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> Rebooting in 20 seconds..
> ACPI MEMORY or I/O RESET_REG.
> 
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/  

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Chris Murphy]

What about btfs check (no repair), without and then also with --mode=lowmem?

In theory I like the idea of a 24 hour rollback; but in normal usage
Btrfs will eventually free up space containing stale and no longer
necessary metadata. Like the chunk tree, it's always changing, so you
get to a point, even with snapshots, that the old state of that tree
is just - gone. A snapshot of an fs tree does not make the chunk tree
frozen in time.

To do what you want, maybe isn't a ton of work if it could be based on
a variation of the existing btrfs seed device code. Call it a "super
snapshot".

I like the idea of triage, where bad parts of the file system can just
be cut off, like triage. Compared to other filesystems, they'll say
this is hardware sabotage and nothing can be done. Btrfs is a bit
deceptive in that it sorta invites the idea we can use hardware that
isn't proven, and the fs can survive.

In any case, it's a big problem in my mind if no existing tools can
fix a file system of this size. So before making anymore changes, make
sure you have a btrfs-image somewhere, even if it's huge. The offline
checker needs to be able to repair it, right now it's all we have for
such a case.


Chris Murphy

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Marc MERLIN]

Hi Chris,

Thanks for the reply, much appreciated.

On Mon, May 01, 2017 at 07:50:22PM -0600, Chris Murphy wrote:
> What about btfs check (no repair), without and then also with --mode=lowmem?
> 
> In theory I like the idea of a 24 hour rollback; but in normal usage
> Btrfs will eventually free up space containing stale and no longer
> necessary metadata. Like the chunk tree, it's always changing, so you
> get to a point, even with snapshots, that the old state of that tree
> is just - gone. A snapshot of an fs tree does not make the chunk tree
> frozen in time.
 
Right, of course, I was being way over optimistic here. I kind of forgot
that metadata wasn't COW, my bad.

> In any case, it's a big problem in my mind if no existing tools can
> fix a file system of this size. So before making anymore changes, make
> sure you have a btrfs-image somewhere, even if it's huge. The offline
> checker needs to be able to repair it, right now it's all we have for
> such a case.

The image will be huge, and take maybe 24H to make (last time it took
some silly amount of time like that), and honestly I'm not sure how
useful it'll be.
Outside of the kernel crashing if I do a btrfs balance, and hopefully
the crash report I gave is good enough, the state I'm in is not btrfs'
fault.

If I can't roll back to a reasonably working state, with data loss of a
known quantity that I can recover from backup, I'll have to destroy and
filesystem and recover from scratch, which will take multiple days.
Since I can't wait too long before getting back to a working state, I
think I'm going to try btrfs check --repair after a scrub to get a list
of all the pathanmes/inodes that are known to be damaged, and work from
there.
Sounds reasonable?

Also, how is --mode=lowmem being useful?

And for re-parenting a sub-subvolume, is that possible?
(I want to delete /sub1/ but I can't because I have /sub1/sub2 that's also a subvolume
and I'm not sure how to re-parent sub2 to somewhere else so that I can subvolume delete
sub1)

In the meantime, a simple check without repair looks like this. It will
likely take many hours to complete:
gargamel:/var/local/space# btrfs check /dev/mapper/dshelf2
Checking filesystem on /dev/mapper/dshelf2
UUID: 03e9a50c-1ae6-4782-ab9c-5f310a98e653
checking extents
checksum verify failed on 3096461459456 found 0E6B7980 wanted FBE5477A
checksum verify failed on 3096461459456 found 0E6B7980 wanted FBE5477A
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
bytenr mismatch, want=2899180224512, have=3981076597540270796
checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
parent transid verify failed on 1671538819072 wanted 293964 found 293902
parent transid verify failed on 1671538819072 wanted 293964 found 293902
checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
checksum verify failed on 1759425052672 found 843B59F1 wanted F0FF7D00
checksum verify failed on 1759425052672 found 843B59F1 wanted F0FF7D00
checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
checksum verify failed on 2898779357184 found 96395131 wanted 433D6E09
checksum verify failed on 2898779357184 found 96395131 wanted 433D6E09
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
bytenr mismatch, want=2899180224512, have=3981076597540270796
checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
(...)

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Chris Murphy]

On Mon, May 1, 2017 at 9:23 PM, Marc MERLIN <marc@merlins.org> wrote:
> Hi Chris,
>
> Thanks for the reply, much appreciated.
>
> On Mon, May 01, 2017 at 07:50:22PM -0600, Chris Murphy wrote:
>> What about btfs check (no repair), without and then also with --mode=lowmem?
>>
>> In theory I like the idea of a 24 hour rollback; but in normal usage
>> Btrfs will eventually free up space containing stale and no longer
>> necessary metadata. Like the chunk tree, it's always changing, so you
>> get to a point, even with snapshots, that the old state of that tree
>> is just - gone. A snapshot of an fs tree does not make the chunk tree
>> frozen in time.
>
> Right, of course, I was being way over optimistic here. I kind of forgot
> that metadata wasn't COW, my bad.

Well it is COW. But there's more to the file system than fs trees, and
just because an fs tree gets snapshot doesn't mean all data is
snapshot. So whether snapshot or not, there's metadata that becomes
obsolete as the file system is updated and those areas get freed up
and eventually overwritten.


>
>> In any case, it's a big problem in my mind if no existing tools can
>> fix a file system of this size. So before making anymore changes, make
>> sure you have a btrfs-image somewhere, even if it's huge. The offline
>> checker needs to be able to repair it, right now it's all we have for
>> such a case.
>
> The image will be huge, and take maybe 24H to make (last time it took
> some silly amount of time like that), and honestly I'm not sure how
> useful it'll be.
> Outside of the kernel crashing if I do a btrfs balance, and hopefully
> the crash report I gave is good enough, the state I'm in is not btrfs'
> fault.
>
> If I can't roll back to a reasonably working state, with data loss of a
> known quantity that I can recover from backup, I'll have to destroy and
> filesystem and recover from scratch, which will take multiple days.
> Since I can't wait too long before getting back to a working state, I
> think I'm going to try btrfs check --repair after a scrub to get a list
> of all the pathanmes/inodes that are known to be damaged, and work from
> there.
> Sounds reasonable?

Yes.


>
> Also, how is --mode=lowmem being useful?

Testing. lowmem is a different implementation, so it might find
different things from the regular check.


>
> And for re-parenting a sub-subvolume, is that possible?
> (I want to delete /sub1/ but I can't because I have /sub1/sub2 that's also a subvolume
> and I'm not sure how to re-parent sub2 to somewhere else so that I can subvolume delete
> sub1)

Well you can move sub2 out of sub1 just like a directory and then
delete sub1. If it's read-only it can't be moved, but you can use
btrfs property get/set ro true/false to temporarily make it not
read-only, move it, then make it read-only again, and it's still fine
to use with btrfs send receive.





>
> In the meantime, a simple check without repair looks like this. It will
> likely take many hours to complete:
> gargamel:/var/local/space# btrfs check /dev/mapper/dshelf2
> Checking filesystem on /dev/mapper/dshelf2
> UUID: 03e9a50c-1ae6-4782-ab9c-5f310a98e653
> checking extents
> checksum verify failed on 3096461459456 found 0E6B7980 wanted FBE5477A
> checksum verify failed on 3096461459456 found 0E6B7980 wanted FBE5477A
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
> checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
> checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> parent transid verify failed on 1671538819072 wanted 293964 found 293902
> parent transid verify failed on 1671538819072 wanted 293964 found 293902
> checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
> checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
> checksum verify failed on 1759425052672 found 843B59F1 wanted F0FF7D00
> checksum verify failed on 1759425052672 found 843B59F1 wanted F0FF7D00
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2898779357184 found 96395131 wanted 433D6E09
> checksum verify failed on 2898779357184 found 96395131 wanted 433D6E09
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071

Not understanding the problem, it's by definition naive for me to
suggest it should go read-only sooner before hosing itself. But I'd
like to think it's possible for Btrfs to look backward every once in a
while for sanity checking, to limit damage should it be occurring even
if the hardware isn't reporting any problems.


-- 
Chris Murphy

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Duncan]

Marc MERLIN posted on Mon, 01 May 2017 20:23:46 -0700 as excerpted:

> Also, how is --mode=lowmem being useful?

FWIW, I just watched your talk that's linked from the wiki, and wondered 
what you were doing these days as I hadn't seen any posts from you here 
for awhile.

Well, that you're asking that question confirms you've not been following 
the list too closely...  Of course that's understandable as people have 
other stuff to do, but just sayin'.

The answer is... btrfs check in lowmem mode isn't simply lowmem, it's 
also effectively a very nearly entirely rewritten second implementation, 
which has already demonstrated its worth as it has already allowed 
finding and fixing a number of bugs in normal mode check.  Of course 
normal mode check has returned the favor a few times as well, so it is 
now reasonably standard list troubleshooting practice to ask for the 
output from both modes to see what and where they differ, especially if 
it's not something known to be directly fixable by normal mode, which of 
course remains the more mature default.

So even if neither one can actually fix the problem ATM, any differences 
in output both lend important clues to the real problem, and potentially 
help developers to find and fix bugs in one or the other implementation.

Tho it's worth noting that lowmem mode can be expected to take longer, as 
it favors lower memory usage over speed, just as the mode title suggests 
it will.  On a filesystem as big as yours... it may unfortunately not be 
entirely practical, especially if as you hint there's at least some time 
pressure here, tho it's not extreme.

Of course on-list I'm somewhat known for my arguments propounding the 
notion that any filesystem that's too big to be practically maintained 
(including time necessary to restore from backups, should that be 
necessary for whatever reason) is... too big... and should ideally be 
broken along logical and functional boundaries into a number of 
individual smaller filesystems until such point as each one is found to 
be practically maintainable within a reasonably practical time frame.  
Don't put all the eggs in one basket, and when the bottom of one of those 
baskets inevitably falls out, most of your eggs will be safe in other 
baskets. =:^)

But as someone else (pg, IIRC) on-list is fond of saying, lots of other 
people "know better" (TM).  Whatever.  It's your data, your systems and 
your time, not mine.  I just know what I've found (sometimes finding it 
the hard way!) to work best for me, and TBs on TBs of data on a single 
filesystem, even if it's a backup and is itself backed up, isn't 
something I'd be putting my own faith in, as the time even for a simple 
restore from backups is simply too high for me to consider it at all 
practical. =:^)

> And for re-parenting a sub-subvolume, is that possible?
> (I want to delete /sub1/ but I can't because I have /sub1/sub2 that's
> also a subvolume and I'm not sure how to re-parent sub2 to somewhere
> else so that I can subvolume delete sub1)

As I believe you know my own use-case doesn't deal with subvolumes and 
snapshots, so this may be of limited practicality, but FWIW, the 
sysadmin's guide discussion of snapshot management and special cases 
seems apropos as a first stop, before going further:

https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Managing_Snapshots

Note that toward the bottom of "management" it discusses moving 
subvolumes (which will obviously reparent them), but then below that in 
special cases it says that read-only subvolumes (and thus snapshots) 
cannot be moved, explaining why.


*BUT*, and here's the "go further" part, keep in mind that subvolume-read-
only is a property, gettable and settable by btrfs property.

So you should be able to unset the read-only property of a subvolume or 
snapshot, move it, then if desired, set it again.

Of course I wouldn't expect send -p to work with such a snapshot, but 
send -c /might/ still work, I'm not actually sure but I'd consider it 
worth trying.  (I'd try -p as well, but expect it to fail...)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Marc MERLIN]

On Mon, May 01, 2017 at 10:56:06PM -0600, Chris Murphy wrote:
> > Right, of course, I was being way over optimistic here. I kind of forgot
> > that metadata wasn't COW, my bad.
> 
> Well it is COW. But there's more to the file system than fs trees, and
> just because an fs tree gets snapshot doesn't mean all data is
> snapshot. So whether snapshot or not, there's metadata that becomes
> obsolete as the file system is updated and those areas get freed up
> and eventually overwritten.

Got it, thanks for explaining.

> > Also, how is --mode=lowmem being useful?
> 
> Testing. lowmem is a different implementation, so it might find
> different things from the regular check.
 
I see.
I've fired off some scrub -r and then check to run overnight, I'll see
if it finishes overnight assuming the kernel doesn't crash again (yeah,
just to make things simpler, I'm hitting another issue when I/O piles up
on btrfs on top of dmcrypt on top of bcache
http://lkml.iu.edu/hypermail/linux/kernel/1705.0/00626.html
https://pastebin.com/YqE4riw0
but that's not a bcache bug, just something else getting in the way.

> > And for re-parenting a sub-subvolume, is that possible?
> > (I want to delete /sub1/ but I can't because I have /sub1/sub2 that's also a subvolume
> > and I'm not sure how to re-parent sub2 to somewhere else so that I can subvolume delete
> > sub1)
> 
> Well you can move sub2 out of sub1 just like a directory and then
> delete sub1. If it's read-only it can't be moved, but you can use
> btrfs property get/set ro true/false to temporarily make it not
> read-only, move it, then make it read-only again, and it's still fine
> to use with btrfs send receive.

Ah, I didn't think mv would work from inside a subvolume to outside of a
subvolume without copying data (it doesn't for files) but I guess it
would for for subvolumes, good point.
I'll try that, thanks.

> Not understanding the problem, it's by definition naive for me to
> suggest it should go read-only sooner before hosing itself. But I'd
> like to think it's possible for Btrfs to look backward every once in a
> while for sanity checking, to limit damage should it be occurring even
> if the hardware isn't reporting any problems.

Fair point. To be honest, maybe btrfs could indeed have detected
problems earlier, but ultimately it's not really its fault if bad things
happen when I'm having repeated storage errors underneath. For all I
know, some data got written after getting corrupted and btrfs would not
notice that right away.
Now, I kind of naively thought I could simply unroll all writes done
after a certain point. You pointed right (rightfully so) that it's not
nearly as simple as I was hoping.

So at this point, I think it's just a matter of me providing
check/repair logs if they are useful, and someone looking into this
balance causing a kernel crash, which is IMO the only real thing that
btrfs should reasonably fix.

I'll update the thread when I have more logs and have moved further on
the recovery.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

btrfs check --repair: failed to repair damaged filesystem, aborting

[Marc MERLIN]

(cc trimmed)

The one in debian/unstable crashed:
gargamel:~# btrfs --version
btrfs-progs v4.7.3
gargamel:~# btrfs check --repair /dev/mapper/dshelf2
bytenr mismatch, want=2899180224512, have=3981076597540270796
extent-tree.c:2721: alloc_reserved_tree_block: Assertion `ret` failed.
btrfs[0x43e418]
btrfs[0x43e43f]
btrfs[0x43f276]
btrfs[0x43f46f]
btrfs[0x4407ef]
btrfs[0x440963]
btrfs(btrfs_inc_extent_ref+0x513)[0x44107a]
btrfs[0x420053]
btrfs[0x4265eb]
btrfs(cmd_check+0x1111)[0x427d6d]
btrfs(main+0x12f)[0x40a341]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f6b632e82b1]
btrfs(_start+0x2a)[0x40a37a]

Ok, it's old, let's take git from today:
gargamel:~# btrfs --version
btrfs-progs v4.10.2
As a note, 
gargamel:~# btrfs check --mode=lowmem --repair /dev/mapper/dshelf2
enabling repair mode
ERROR: low memory mode doesn't support repair yet

As a note, a 32bit binary on a 64bit kernel:
gargamel:~# btrfs check --repair /dev/mapper/dshelf2
enabling repair mode
Checking filesystem on /dev/mapper/dshelf2
UUID: 03e9a50c-1ae6-4782-ab9c-5f310a98e653
checking extents
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
bytenr mismatch, want=2899180224512, have=3981076597540270796
checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
parent transid verify failed on 1671538819072 wanted 293964 found 293902
parent transid verify failed on 1671538819072 wanted 293964 found 293902
checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
cmds-check.c:6291: add_data_backref: BUG_ON `!back` triggered, value 1
Aborted

let's try again with a 64bit binary built from git:
(...)
Repaired extent references for 4227617038336
ref mismatch on [4227872751616 4096] extent item 1, found 0
Incorrect local backref count on 4227872751616 parent 3493071667200 owner 0
offset 0 found 0 wanted 1 back 0x56470b18e7f0  
Backref disk bytenr does not match extent record, bytenr=4227872751616, ref
bytenr=0
backpointer mismatch on [4227872751616 4096]
owner ref check failed [4227872751616 4096]
repair deleting extent record: key 4227872751616 168 4096
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
bytenr mismatch, want=2899180224512, have=3981076597540270796
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E  
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
bytenr mismatch, want=2899180224512, have=3981076597540270796
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
bytenr mismatch, want=2899180224512, have=3981076597540270796
Repaired extent references for 4227872751616
ref mismatch on [6674127745024 32768] extent item 0, found 1
Backref 6674127745024 parent 7566652473344 owner 0 offset 0 num_refs 0 not
found in extent tree
Incorrect local backref count on 6674127745024 parent 7566652473344 owner 0
offset 0 found 1 wanted 0 back 0x5648afda0f20  
backpointer mismatch on [6674127745024 32768]
checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
checksum verify failed on 6983266418688 found BCBF9E15 wanted 785FF67E
checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
bytenr mismatch, want=6983266418688, have=13671317608077697645
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
bytenr mismatch, want=2899180224512, have=3981076597540270796
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
bytenr mismatch, want=2899180224512, have=3981076597540270796
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
bytenr mismatch, want=2899180224512, have=3981076597540270796
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
bytenr mismatch, want=2899180224512, have=3981076597540270796
failed to repair damaged filesystem, aborting


So, I'm out of luck now, full wipe and 3-5 day rebuild?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Kai Krakow]

Am Tue, 2 May 2017 05:01:02 +0000 (UTC)
schrieb Duncan <1i5t5.duncan@cox.net>:

> Of course on-list I'm somewhat known for my arguments propounding the 
> notion that any filesystem that's too big to be practically
> maintained (including time necessary to restore from backups, should
> that be necessary for whatever reason) is... too big... and should
> ideally be broken along logical and functional boundaries into a
> number of individual smaller filesystems until such point as each one
> is found to be practically maintainable within a reasonably practical
> time frame. Don't put all the eggs in one basket, and when the bottom
> of one of those baskets inevitably falls out, most of your eggs will
> be safe in other baskets. =:^)

Hehe... Yes, you're a fan of small filesystems. I'm more from the
opposite camp, preferring one big filesystem to not mess around with
size constraints of small filesystems fighting for the same volume
space. It also gives such filesystems better chances for data locality
of data staying in totally different parts across your fs mounts and
can reduce head movement. Of course, much of this is not true if you
use different devices per filesystem, or use SSDs, or SAN where you
have no real control over the physical placement of image stripes
anyway. But well...

In an ideal world, subvolumes of btrfs would be totally independent of
each other, just only share the same volume and dynamically allocating
chunks of space from it. If one is broken, it is simply not usable and
it should be destroyable. A garbage collector would grab the leftover
chunks from the subvolume and free them, and you could recreate this
subvolume from backup. In reality, shared extents will cross subvolume
borders so it is probably not how things could work anytime in the near
of far future.

This idea is more like having thinly provisioned LVM volumes which
allocate space as the filesystems on top need them, much like doing
thinly provisioned images with a VM host system. The problem here is,
unlike subvolumes, those chunks of space could never be given back to
the host as it doesn't know if it is still in use. Of course, there's
implementations available which allow thinning the images by passing
through TRIM from the guest to the host (or by other means of
communication channels between host and guest), but that is usually not
giving good performance, if even supported.

I tried once to exploit this in VirtualBox and hoped it would translate
guest discards into hole punching requests on the host, and it's even
documented to work that way... But (a) it was horrible slow, and (b) it
was incredibly unstable to the point of being useless. OTOH, it's not
announced as a stable feature and has to be enabled by manually editing
the XML config files.

But I still like the idea: Is it possible to make btrfs still work if
one subvolume gets corrupted? Of course it should have ways of telling
the user which other subvolumes are interconnected through shared
extents so those would be also discarded upon corruption cleanup - at
least if those extents couldn't be made any sense of any longer. Since
corruption is an issue mostly of subvolumes being written to, snapshots
should be mostly safe.

Such a feature would also only make sense if btrfs had an online repair
tool. BTW, are there plans for having an online repair tool in the
future? Maybe one that only scans and fixes part of the filesystems
(for obvious performance reasons, wrt Duncans idea of handling
filesystems), i.e. those parts that the kernel discovered having
corruptions? If I could then just delete and restore affected files,
this would be even better than having independent subvolumes like above.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Kai Krakow]

Am Mon, 1 May 2017 22:56:06 -0600
schrieb Chris Murphy <lists@colorremedies.com>:

> On Mon, May 1, 2017 at 9:23 PM, Marc MERLIN <marc@merlins.org> wrote:
> > Hi Chris,
> >
> > Thanks for the reply, much appreciated.
> >
> > On Mon, May 01, 2017 at 07:50:22PM -0600, Chris Murphy wrote:  
> >> What about btfs check (no repair), without and then also with
> >> --mode=lowmem?
> >>
> >> In theory I like the idea of a 24 hour rollback; but in normal
> >> usage Btrfs will eventually free up space containing stale and no
> >> longer necessary metadata. Like the chunk tree, it's always
> >> changing, so you get to a point, even with snapshots, that the old
> >> state of that tree is just - gone. A snapshot of an fs tree does
> >> not make the chunk tree frozen in time.  
> >
> > Right, of course, I was being way over optimistic here. I kind of
> > forgot that metadata wasn't COW, my bad.  
> 
> Well it is COW. But there's more to the file system than fs trees, and
> just because an fs tree gets snapshot doesn't mean all data is
> snapshot. So whether snapshot or not, there's metadata that becomes
> obsolete as the file system is updated and those areas get freed up
> and eventually overwritten.
> 
> 
> >  
> >> In any case, it's a big problem in my mind if no existing tools can
> >> fix a file system of this size. So before making anymore changes,
> >> make sure you have a btrfs-image somewhere, even if it's huge. The
> >> offline checker needs to be able to repair it, right now it's all
> >> we have for such a case.  
> >
> > The image will be huge, and take maybe 24H to make (last time it
> > took some silly amount of time like that), and honestly I'm not
> > sure how useful it'll be.
> > Outside of the kernel crashing if I do a btrfs balance, and
> > hopefully the crash report I gave is good enough, the state I'm in
> > is not btrfs' fault.
> >
> > If I can't roll back to a reasonably working state, with data loss
> > of a known quantity that I can recover from backup, I'll have to
> > destroy and filesystem and recover from scratch, which will take
> > multiple days. Since I can't wait too long before getting back to a
> > working state, I think I'm going to try btrfs check --repair after
> > a scrub to get a list of all the pathanmes/inodes that are known to
> > be damaged, and work from there.
> > Sounds reasonable?  
> 
> Yes.
> 
> 
> >
> > Also, how is --mode=lowmem being useful?  
> 
> Testing. lowmem is a different implementation, so it might find
> different things from the regular check.
> 
> 
> >
> > And for re-parenting a sub-subvolume, is that possible?
> > (I want to delete /sub1/ but I can't because I have /sub1/sub2
> > that's also a subvolume and I'm not sure how to re-parent sub2 to
> > somewhere else so that I can subvolume delete sub1)  
> 
> Well you can move sub2 out of sub1 just like a directory and then
> delete sub1. If it's read-only it can't be moved, but you can use
> btrfs property get/set ro true/false to temporarily make it not
> read-only, move it, then make it read-only again, and it's still fine
> to use with btrfs send receive.
> 
> 
> 
> 
> 
> >
> > In the meantime, a simple check without repair looks like this. It
> > will likely take many hours to complete:
> > gargamel:/var/local/space# btrfs check /dev/mapper/dshelf2
> > Checking filesystem on /dev/mapper/dshelf2
> > UUID: 03e9a50c-1ae6-4782-ab9c-5f310a98e653
> > checking extents
> > checksum verify failed on 3096461459456 found 0E6B7980 wanted
> > FBE5477A checksum verify failed on 3096461459456 found 0E6B7980
> > wanted FBE5477A checksum verify failed on 2899180224512 found
> > 7A6D427F wanted 7E899EE5 checksum verify failed on 2899180224512
> > found 7A6D427F wanted 7E899EE5 checksum verify failed on
> > 2899180224512 found ABBE39B0 wanted E0735D0E checksum verify failed
> > on 2899180224512 found 7A6D427F wanted 7E899EE5 bytenr mismatch,
> > want=2899180224512, have=3981076597540270796 checksum verify failed
> > on 1449488023552 found CECC36AF wanted 199FE6C5 checksum verify
> > failed on 1449488023552 found CECC36AF wanted 199FE6C5 checksum
> > verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> > checksum verify failed on 1449544613888 found 895D691B wanted
> > A0C64D2B parent transid verify failed on 1671538819072 wanted
> > 293964 found 293902 parent transid verify failed on 1671538819072
> > wanted 293964 found 293902 checksum verify failed on 1671603781632
> > found 18BC28D6 wanted 372655A0 checksum verify failed on
> > 1671603781632 found 18BC28D6 wanted 372655A0 checksum verify failed
> > on 1759425052672 found 843B59F1 wanted F0FF7D00 checksum verify
> > failed on 1759425052672 found 843B59F1 wanted F0FF7D00 checksum
> > verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> > checksum verify failed on 2182657212416 found CD8EFC0C wanted
> > 70847071 checksum verify failed on 2898779357184 found 96395131
> > wanted 433D6E09 checksum verify failed on 2898779357184 found
> > 96395131 wanted 433D6E09 checksum verify failed on 2899180224512
> > found 7A6D427F wanted 7E899EE5 checksum verify failed on
> > 2899180224512 found 7A6D427F wanted 7E899EE5 checksum verify failed
> > on 2899180224512 found ABBE39B0 wanted E0735D0E checksum verify
> > failed on 2899180224512 found 7A6D427F wanted 7E899EE5 bytenr
> > mismatch, want=2899180224512, have=3981076597540270796 checksum
> > verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> > checksum verify failed on 2182657212416 found CD8EFC0C wanted
> > 70847071 checksum verify failed on 2182657212416 found CD8EFC0C
> > wanted 70847071 checksum verify failed on 2182657212416 found
> > CD8EFC0C wanted 70847071 checksum verify failed on 2182657212416
> > found CD8EFC0C wanted 70847071  
> 
> Not understanding the problem, it's by definition naive for me to
> suggest it should go read-only sooner before hosing itself. But I'd
> like to think it's possible for Btrfs to look backward every once in a
> while for sanity checking, to limit damage should it be occurring even
> if the hardware isn't reporting any problems.

Would it be possible to make btrfs avoid using parts of the filesystem
it detected corruptions in? Then a still-in-theory online repair tool
could check these parts, maybe repair them (or destroy them upon
request), and make those parts of the fs available again... Such a
repair tool (scanning only known corrupted parts) would probably also
need less memory and time to run.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: btrfs check --repair: failed to repair damaged filesystem, aborting

[Marc MERLIN]

David,

I think you maintain btrfs-progs, but I'm not sure if you're in charge 
of check --repair.
Could you comment on the bottom of the mail, namely:
> failed to repair damaged filesystem, aborting
> So, I'm out of luck now, full wipe and 3-5 day rebuild?
 
Thanks,
Marc

Rest:
On Tue, May 02, 2017 at 11:47:22AM -0700, Marc MERLIN wrote:
> (cc trimmed)
> 
> The one in debian/unstable crashed:
> gargamel:~# btrfs --version
> btrfs-progs v4.7.3
> gargamel:~# btrfs check --repair /dev/mapper/dshelf2
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> extent-tree.c:2721: alloc_reserved_tree_block: Assertion `ret` failed.
> btrfs[0x43e418]
> btrfs[0x43e43f]
> btrfs[0x43f276]
> btrfs[0x43f46f]
> btrfs[0x4407ef]
> btrfs[0x440963]
> btrfs(btrfs_inc_extent_ref+0x513)[0x44107a]
> btrfs[0x420053]
> btrfs[0x4265eb]
> btrfs(cmd_check+0x1111)[0x427d6d]
> btrfs(main+0x12f)[0x40a341]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f6b632e82b1]
> btrfs(_start+0x2a)[0x40a37a]
> 
> Ok, it's old, let's take git from today:
> gargamel:~# btrfs --version
> btrfs-progs v4.10.2
> As a note, 
> gargamel:~# btrfs check --mode=lowmem --repair /dev/mapper/dshelf2
> enabling repair mode
> ERROR: low memory mode doesn't support repair yet
> 
> As a note, a 32bit binary on a 64bit kernel:
> gargamel:~# btrfs check --repair /dev/mapper/dshelf2
> enabling repair mode
> Checking filesystem on /dev/mapper/dshelf2
> UUID: 03e9a50c-1ae6-4782-ab9c-5f310a98e653
> checking extents
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
> checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
> checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> parent transid verify failed on 1671538819072 wanted 293964 found 293902
> parent transid verify failed on 1671538819072 wanted 293964 found 293902
> checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
> checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
> cmds-check.c:6291: add_data_backref: BUG_ON `!back` triggered, value 1
> Aborted
> 
> let's try again with a 64bit binary built from git:
> (...)
> Repaired extent references for 4227617038336
> ref mismatch on [4227872751616 4096] extent item 1, found 0
> Incorrect local backref count on 4227872751616 parent 3493071667200 owner 0
> offset 0 found 0 wanted 1 back 0x56470b18e7f0  
> Backref disk bytenr does not match extent record, bytenr=4227872751616, ref
> bytenr=0
> backpointer mismatch on [4227872751616 4096]
> owner ref check failed [4227872751616 4096]
> repair deleting extent record: key 4227872751616 168 4096
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E  
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> Repaired extent references for 4227872751616
> ref mismatch on [6674127745024 32768] extent item 0, found 1
> Backref 6674127745024 parent 7566652473344 owner 0 offset 0 num_refs 0 not
> found in extent tree
> Incorrect local backref count on 6674127745024 parent 7566652473344 owner 0
> offset 0 found 1 wanted 0 back 0x5648afda0f20  
> backpointer mismatch on [6674127745024 32768]
> checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
> checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
> checksum verify failed on 6983266418688 found BCBF9E15 wanted 785FF67E
> checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
> bytenr mismatch, want=6983266418688, have=13671317608077697645
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> failed to repair damaged filesystem, aborting
> 
> 
> So, I'm out of luck now, full wipe and 3-5 day rebuild?
> 
> Thanks,
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/  
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfs check --repair: failed to repair damaged filesystem, aborting

[Marc MERLIN]

On Tue, May 02, 2017 at 11:00:08PM -0700, Marc MERLIN wrote:
> David,
> 
> I think you maintain btrfs-progs, but I'm not sure if you're in charge 
> of check --repair.
> Could you comment on the bottom of the mail, namely:
> > failed to repair damaged filesystem, aborting
> > So, I'm out of luck now, full wipe and 3-5 day rebuild?
  
Actually, another thought:
Is there or should there be a way to repair around the bit that cannot
be repaired?
Separately, or not, can I locate which bits are causing the repair to
fail and maybe get a pointer to the path/inode so that I can hopefully
just delete those bad data structures (assuming deleting them is even
possible and that the FS won't just go read only as I try to do that)

Here is the full run if that helps:
https://pastebin.com/STMFHty4

> Thanks,
> Marc
> 
> Rest:
> On Tue, May 02, 2017 at 11:47:22AM -0700, Marc MERLIN wrote:
> > (cc trimmed)
> > 
> > The one in debian/unstable crashed:
> > gargamel:~# btrfs --version
> > btrfs-progs v4.7.3
> > gargamel:~# btrfs check --repair /dev/mapper/dshelf2
> > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > extent-tree.c:2721: alloc_reserved_tree_block: Assertion `ret` failed.
> > btrfs[0x43e418]
> > btrfs[0x43e43f]
> > btrfs[0x43f276]
> > btrfs[0x43f46f]
> > btrfs[0x4407ef]
> > btrfs[0x440963]
> > btrfs(btrfs_inc_extent_ref+0x513)[0x44107a]
> > btrfs[0x420053]
> > btrfs[0x4265eb]
> > btrfs(cmd_check+0x1111)[0x427d6d]
> > btrfs(main+0x12f)[0x40a341]
> > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f6b632e82b1]
> > btrfs(_start+0x2a)[0x40a37a]
> > 
> > Ok, it's old, let's take git from today:
> > gargamel:~# btrfs --version
> > btrfs-progs v4.10.2
> > As a note, 
> > gargamel:~# btrfs check --mode=lowmem --repair /dev/mapper/dshelf2
> > enabling repair mode
> > ERROR: low memory mode doesn't support repair yet
> > 
> > As a note, a 32bit binary on a 64bit kernel:
> > gargamel:~# btrfs check --repair /dev/mapper/dshelf2
> > enabling repair mode
> > Checking filesystem on /dev/mapper/dshelf2
> > UUID: 03e9a50c-1ae6-4782-ab9c-5f310a98e653
> > checking extents
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
> > checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
> > checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> > checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> > parent transid verify failed on 1671538819072 wanted 293964 found 293902
> > parent transid verify failed on 1671538819072 wanted 293964 found 293902
> > checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
> > checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
> > cmds-check.c:6291: add_data_backref: BUG_ON `!back` triggered, value 1
> > Aborted
> > 
> > let's try again with a 64bit binary built from git:
> > (...)
> > Repaired extent references for 4227617038336
> > ref mismatch on [4227872751616 4096] extent item 1, found 0
> > Incorrect local backref count on 4227872751616 parent 3493071667200 owner 0
> > offset 0 found 0 wanted 1 back 0x56470b18e7f0  
> > Backref disk bytenr does not match extent record, bytenr=4227872751616, ref
> > bytenr=0
> > backpointer mismatch on [4227872751616 4096]
> > owner ref check failed [4227872751616 4096]
> > repair deleting extent record: key 4227872751616 168 4096
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E  
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > Repaired extent references for 4227872751616
> > ref mismatch on [6674127745024 32768] extent item 0, found 1
> > Backref 6674127745024 parent 7566652473344 owner 0 offset 0 num_refs 0 not
> > found in extent tree
> > Incorrect local backref count on 6674127745024 parent 7566652473344 owner 0
> > offset 0 found 1 wanted 0 back 0x5648afda0f20  
> > backpointer mismatch on [6674127745024 32768]
> > checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
> > checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
> > checksum verify failed on 6983266418688 found BCBF9E15 wanted 785FF67E
> > checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
> > bytenr mismatch, want=6983266418688, have=13671317608077697645
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > failed to repair damaged filesystem, aborting
> > 
> > 
> > So, I'm out of luck now, full wipe and 3-5 day rebuild?
> > 
> > Thanks,
> > Marc
> > -- 
> > "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> > Microsoft is to operating systems ....
> >                                       .... what McDonalds is to gourmet cooking
> > Home page: http://marc.merlins.org/  
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfs check --repair: failed to repair damaged filesystem, aborting

[Roman Mamedov]

On Tue, 2 May 2017 23:17:11 -0700
Marc MERLIN <marc@merlins.org> wrote:

> On Tue, May 02, 2017 at 11:00:08PM -0700, Marc MERLIN wrote:
> > David,
> > 
> > I think you maintain btrfs-progs, but I'm not sure if you're in charge 
> > of check --repair.
> > Could you comment on the bottom of the mail, namely:
> > > failed to repair damaged filesystem, aborting
> > > So, I'm out of luck now, full wipe and 3-5 day rebuild?
>   
> Actually, another thought:
> Is there or should there be a way to repair around the bit that cannot
> be repaired?
> Separately, or not, can I locate which bits are causing the repair to
> fail and maybe get a pointer to the path/inode so that I can hopefully
> just delete those bad data structures (assuming deleting them is even
> possible and that the FS won't just go read only as I try to do that)

There is the "btrfs-corrupt-block" tool which helped me to kick Btrfsck
further along its course in a similar "unrepairable" situation.
https://www.spinics.net/lists/linux-btrfs/msg53061.html

In your case it appears like the block 2899180224512 is giving it the most
trouble, so you could start with killing that one. From what I can tell this
tool zeroes out the entire block, so Btrfsck can simply delete the reference
and forget it, rather than repeatedly trying to figure out solutions and
bailing out with "failed to repair damaged filesystem, aborting".

Depending on what was stored in it, you may have either no visible effect, or
a complete filesystem failure, or anything in between. Hence if you want to
experiment with this, find a way to work on writable overlay snapshots (also
described in the linked message).

> Here is the full run if that helps:
> https://pastebin.com/STMFHty4
> 
> > Thanks,
> > Marc
> > 
> > Rest:
> > On Tue, May 02, 2017 at 11:47:22AM -0700, Marc MERLIN wrote:
> > > (cc trimmed)
> > > 
> > > The one in debian/unstable crashed:
> > > gargamel:~# btrfs --version
> > > btrfs-progs v4.7.3
> > > gargamel:~# btrfs check --repair /dev/mapper/dshelf2
> > > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > > extent-tree.c:2721: alloc_reserved_tree_block: Assertion `ret` failed.
> > > btrfs[0x43e418]
> > > btrfs[0x43e43f]
> > > btrfs[0x43f276]
> > > btrfs[0x43f46f]
> > > btrfs[0x4407ef]
> > > btrfs[0x440963]
> > > btrfs(btrfs_inc_extent_ref+0x513)[0x44107a]
> > > btrfs[0x420053]
> > > btrfs[0x4265eb]
> > > btrfs(cmd_check+0x1111)[0x427d6d]
> > > btrfs(main+0x12f)[0x40a341]
> > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f6b632e82b1]
> > > btrfs(_start+0x2a)[0x40a37a]
> > > 
> > > Ok, it's old, let's take git from today:
> > > gargamel:~# btrfs --version
> > > btrfs-progs v4.10.2
> > > As a note, 
> > > gargamel:~# btrfs check --mode=lowmem --repair /dev/mapper/dshelf2
> > > enabling repair mode
> > > ERROR: low memory mode doesn't support repair yet
> > > 
> > > As a note, a 32bit binary on a 64bit kernel:
> > > gargamel:~# btrfs check --repair /dev/mapper/dshelf2
> > > enabling repair mode
> > > Checking filesystem on /dev/mapper/dshelf2
> > > UUID: 03e9a50c-1ae6-4782-ab9c-5f310a98e653
> > > checking extents
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > > checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
> > > checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
> > > checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> > > checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> > > parent transid verify failed on 1671538819072 wanted 293964 found 293902
> > > parent transid verify failed on 1671538819072 wanted 293964 found 293902
> > > checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
> > > checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
> > > cmds-check.c:6291: add_data_backref: BUG_ON `!back` triggered, value 1
> > > Aborted
> > > 
> > > let's try again with a 64bit binary built from git:
> > > (...)
> > > Repaired extent references for 4227617038336
> > > ref mismatch on [4227872751616 4096] extent item 1, found 0
> > > Incorrect local backref count on 4227872751616 parent 3493071667200 owner 0
> > > offset 0 found 0 wanted 1 back 0x56470b18e7f0  
> > > Backref disk bytenr does not match extent record, bytenr=4227872751616, ref
> > > bytenr=0
> > > backpointer mismatch on [4227872751616 4096]
> > > owner ref check failed [4227872751616 4096]
> > > repair deleting extent record: key 4227872751616 168 4096
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E  
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > > Repaired extent references for 4227872751616
> > > ref mismatch on [6674127745024 32768] extent item 0, found 1
> > > Backref 6674127745024 parent 7566652473344 owner 0 offset 0 num_refs 0 not
> > > found in extent tree
> > > Incorrect local backref count on 6674127745024 parent 7566652473344 owner 0
> > > offset 0 found 1 wanted 0 back 0x5648afda0f20  
> > > backpointer mismatch on [6674127745024 32768]
> > > checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
> > > checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
> > > checksum verify failed on 6983266418688 found BCBF9E15 wanted 785FF67E
> > > checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
> > > bytenr mismatch, want=6983266418688, have=13671317608077697645
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> > > checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> > > bytenr mismatch, want=2899180224512, have=3981076597540270796
> > > failed to repair damaged filesystem, aborting
> > > 
> > > 
> > > So, I'm out of luck now, full wipe and 3-5 day rebuild?
> > > 
> > > Thanks,
> > > Marc
> > > -- 
> > > "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> > > Microsoft is to operating systems ....
> > >                                       .... what McDonalds is to gourmet cooking
> > > Home page: http://marc.merlins.org/  
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > 
> > -- 
> > "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> > Microsoft is to operating systems ....
> >                                       .... what McDonalds is to gourmet cooking
> > Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 


-- 
With respect,
Roman

Re: btrfs check --repair: failed to repair damaged filesystem, aborting

[Marc MERLIN]

On Wed, May 03, 2017 at 11:32:26AM +0500, Roman Mamedov wrote:
> > Actually, another thought:
> > Is there or should there be a way to repair around the bit that cannot
> > be repaired?
> > Separately, or not, can I locate which bits are causing the repair to
> > fail and maybe get a pointer to the path/inode so that I can hopefully
> > just delete those bad data structures (assuming deleting them is even
> > possible and that the FS won't just go read only as I try to do that)
> 
> There is the "btrfs-corrupt-block" tool which helped me to kick Btrfsck
> further along its course in a similar "unrepairable" situation.
> https://www.spinics.net/lists/linux-btrfs/msg53061.html
> 
> In your case it appears like the block 2899180224512 is giving it the most
> trouble, so you could start with killing that one. From what I can tell this
> tool zeroes out the entire block, so Btrfsck can simply delete the reference
> and forget it, rather than repeatedly trying to figure out solutions and
> bailing out with "failed to repair damaged filesystem, aborting".
> 
> Depending on what was stored in it, you may have either no visible effect, or
> a complete filesystem failure, or anything in between. Hence if you want to
> experiment with this, find a way to work on writable overlay snapshots (also
> described in the linked message).

Thanks for the tip. This does not seem to have worked at all, though.
Did I do something wrong?

gargamel:/var/local/src/btrfs-progs# ./btrfs-corrupt-block -l  2899180224512 /dev/mapper/dshelf2
mirror 1 logical 2899180224512 physical 2814363009024 device /dev/mapper/dshelf2
corrupting 2899180224512 copy 1
mirror 2 logical 2899180224512 physical 2814899879936 device /dev/mapper/dshelf2
corrupting 2899180224512 copy 2


gargamel:/mnt/btrfs_pool1# btrfs check --repair /dev/mapper/dshelf2
(...)
checksum verify failed on 2899180224512 found F25BEE55 wanted 00000000
checksum verify failed on 2899180224512 found F25BEE55 wanted 00000000
checksum verify failed on 2899180224512 found E5245DBD wanted 00000000
checksum verify failed on 2899180224512 found F25BEE55 wanted 00000000
bytenr mismatch, want=2899180224512, have=0
Repaired extent references for 3566695825408
ref mismatch on [6674127745024 32768] extent item 0, found 1
Backref 6674127745024 parent 7566652473344 owner 0 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 6674127745024 parent 7566652473344 owner 0 offset 0 found 1 wanted 0 back 0x555cb4e9ced0
backpointer mismatch on [6674127745024 32768]
checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
checksum verify failed on 6983266418688 found BCBF9E15 wanted 785FF67E
checksum verify failed on 6983266418688 found 393B112A wanted 2B19CD5C
bytenr mismatch, want=6983266418688, have=13671317608077697645
checksum verify failed on 2899180224512 found F25BEE55 wanted 00000000
checksum verify failed on 2899180224512 found F25BEE55 wanted 00000000
checksum verify failed on 2899180224512 found E5245DBD wanted 00000000
checksum verify failed on 2899180224512 found F25BEE55 wanted 00000000
bytenr mismatch, want=2899180224512, have=0
failed to repair damaged filesystem, aborting

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Qu Wenruo]

At 05/02/2017 02:08 AM, Marc MERLIN wrote:
> So, I forgot to mention that it's my main media and backup server that got
> corrupted. Yes, I do actually have a backup of a backup server, but it's
> going to take days to recover due to the amount of data to copy back, not
> counting lots of manual typing due to the number of subvolumes, btrfs
> send/receive relationships and so forth.
> 
> Really, I should be able to roll back all writes from the last 24H, run a
> check --repair/scrub on top just to be sure, and be back on track.
> 
> In the meantime, the good news is that the filesystem doesn't crash the
> kernel (the poasted crash below) now that I was able to cancel the btrfs balance,
> but it goes read only at the drop of a hat, even when I'm trying to delete
> recent snapshots and all data that was potentially written in the last 24H
> 
> On Mon, May 01, 2017 at 10:06:41AM -0700, Marc MERLIN wrote:
>> I have a filesystem that sadly got corrupted by a SAS card I just installed yesterday.
>>
>> I don't think in a case like this, there is there a way to roll back all
>> writes across all subvolumes in the last 24H, correct?

Sorry for the late reply.
I thought the case is already finished as I see little chance to recover. :(

No, no way to roll back unless you're completely sure there is only 1 
transaction commit happened in last 24H.
(Well, not really possible in real world)

Btrfs is only capable to rollback to *previous* commit.
That's ensure by forced metadata CoW.

But beyond previous commit, only god knows.

If all metadata CoW write is done in some place never used by any 
previous metadata, then there is the chance to recover.

But mostly the possibility is very low, some mount option like ssd will 
change the extent allocator behavior to improve the possibility, but 
still need a lot of luck.

More detailed comment will be replied to btrfs check mail.

Thanks,
Qu

>>
>> Is the best thing to go in each subvolume, delete the recent snapshots and
>> rename the one from 24H as the current one?
>   
> Well, just like I expected, it's a pain in the rear and this can't even help
> fix the top level mountpoint which doesn't have snapshots, so I can't roll
> it back.
> btrfs should really have an easy way to roll back X hours, or days to
> recover from garbage written after a good known point, given that it is COW
> afterall.
> 
> Is there a way do this with check --repair maybe?
> 
> In the meantime, I got stuck while trying to delete snapshots:
> 
> Let's say I have this:
> ID 428 gen 294021 top level 5 path backup
> ID 2023 gen 294021 top level 5 path Soft
> ID 3021 gen 294051 top level 428 path backup/debian32
> ID 4400 gen 294018 top level 428 path backup/debian64
> ID 4930 gen 294019 top level 428 path backup/ubuntu
> 
> I can easily
> Delete subvolume (no-commit): '/mnt/btrfs_pool2/Soft'
> and then:
> gargamel:/mnt/btrfs_pool2# mv Soft_rw.20170430_01:50:22 Soft
> 
> But I can't delete backup, which actually is mostly only a directory
> containing other things (in hindsight I shouldn't have made that a
> subvolume)
> Delete subvolume (no-commit): '/mnt/btrfs_pool2/backup'
> ERROR: cannot delete '/mnt/btrfs_pool2/backup': Directory not empty
> 
> This is because backup has a lot of subvolumes due to btrfs send/receive
> relationships.
> 
> Is it possible to recover there? Can you reparent subvolumes to a different
> subvolume without doing a full copy via btrfs send/receive?
> 
> Thanks,
> Marc
> 
>> BTRFS warning (device dm-5): failed to load free space cache for block group 6746013696000, rebuilding it now
>> BTRFS warning (device dm-5): block group 6754603630592 has wrong amount of free space
>> BTRFS warning (device dm-5): failed to load free space cache for block group 6754603630592, rebuilding it now
>> BTRFS warning (device dm-5): block group 7125178777600 has wrong amount of free space
>> BTRFS warning (device dm-5): failed to load free space cache for block group 7125178777600, rebuilding it now
>> BTRFS error (device dm-5): bad tree block start 3981076597540270796 2899180224512
>> BTRFS error (device dm-5): bad tree block start 942082474969670243 2899180224512
>> BTRFS: error (device dm-5) in __btrfs_free_extent:6944: errno=-5 IO failure
>> BTRFS info (device dm-5): forced readonly
>> BTRFS: error (device dm-5) in btrfs_run_delayed_refs:2961: errno=-5 IO failure
>> BUG: unable to handle kernel NULL pointer dereference at           (null)
>> IP: __del_reloc_root+0x3f/0xa6
>> PGD 189a0e067
>> PUD 189a0f067
>> PMD 0
>>
>> Oops: 0000 [#1] PREEMPT SMP
>> Modules linked in: veth ip6table_filter ip6_tables ebtable_nat ebtables ppdev lp xt_addrtype br_netfilter bridge stp llc tun autofs4 softdog binfmt_misc ftdi_sio nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ipt_REJECT nf_reject_ipv4 xt_conntrack xt_mark xt_nat xt_tcpudp nf_log_ipv4 nf_log_common xt_LOG iptable_mangle iptable_filter lm85 hwmon_vid pl2303 dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_nat nf_conntrack x_tables sg st snd_pcm_oss snd_mixer_oss bcache kvm_intel kvm irqbypass snd_hda_codec_realtek snd_cmipci snd_hda_codec_generic snd_hda_intel snd_mpu401_uart snd_hda_codec snd_opl3_lib snd_rawmidi snd_hda_core snd_seq_device snd_hwdep eeepc_wmi snd_pcm asus_wmi rc_ati_x10
>>   asix snd_timer ati_remote sparse_keymap usbnet rfkill snd hwmon soundcore rc_core evdev libphy tpm_infineon pcspkr i915 parport_pc i2c_i801 input_leds mei_me lpc_ich parport tpm_tis battery usbserial tpm_tis_core tpm wmi e1000e ptp pps_core fuse raid456 multipath mmc_block mmc_core lrw ablk_helper dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx crc32c_intel blowfish_x86_64 blowfish_common pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd xhci_pci ehci_pci sata_sil24 xhci_hcd mvsas ehci_hcd r8169 usbcore mii libsas scsi_transport_sas thermal fan [last unloaded: ftdi_sio]
>> CPU: 0 PID: 9056 Comm: btrfs Tainted: G     U          4.11.0-amd64-preempt-sysrq-20170406 #2
>> Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
>> task: ffff88374d2a60c0 task.stack: ffffa6f226424000
>> RIP: 0010:__del_reloc_root+0x3f/0xa6
>> RSP: 0018:ffffa6f226427a40 EFLAGS: 00210246
>> RAX: 0000000000000000 RBX: ffff8838ee256000 RCX: 00000000ffffffe2
>> RDX: 0000000000000001 RSI: ffffffff9f83b410 RDI: ffff8837992da568
>> RBP: ffffa6f226427a68 R08: 0000000000000000 R09: ffffffff9fd69480
>> R10: 0000000000000000 R11: 0000000000000000 R12: ffffa6f226427ab0
>> R13: ffff883768938000 R14: ffff8837992da568 R15: ffff8837992da570
>> FS:  00007facd18d28c0(0000) GS:ffff883a5e200000(0000) knlGS:0000000000000000
>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 0000000000000000 CR3: 0000000189a10000 CR4: 00000000001406f0
>> Call Trace:
>>   free_reloc_roots+0x4f/0x5d
>>   merge_reloc_roots+0x159/0x1ba
>>   relocate_block_group+0x410/0x492
>>   btrfs_relocate_block_group+0x12d/0x253
>>   btrfs_relocate_chunk+0x3e/0xb1
>>   btrfs_balance+0xd16/0xf36
>>   btrfs_ioctl_balance+0x24f/0x2cd
>>   ? __alloc_pages_nodemask+0x134/0x1e0
>>   btrfs_ioctl+0x1447/0x1e22
>>   ? mem_cgroup_charge_statistics+0x1e/0x88
>>   ? get_page+0x9/0x26
>>   ? __lru_cache_add+0x2a/0x6c
>>   ? set_pte_at+0x9/0xd
>>   ? __handle_mm_fault+0x61d/0xa6f
>>   vfs_ioctl+0x21/0x38
>>   ? vfs_ioctl+0x21/0x38
>>   do_vfs_ioctl+0x4ef/0x537
>>   ? current_kernel_time64+0x10/0x36
>>   ? __audit_syscall_entry+0xc2/0xe6
>>   ? syscall_trace_enter+0x1ac/0x20e
>>   SyS_ioctl+0x57/0x7b
>>   do_syscall_64+0x6b/0x7d
>>   entry_SYSCALL64_slow_path+0x25/0x25
>> RIP: 0033:0x7facd097ecc7
>> RSP: 002b:00007ffefd3c3128 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
>> RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007facd097ecc7
>> RDX: 00007ffefd3c31b8 RSI: 00000000c4009420 RDI: 0000000000000003
>> RBP: 00007ffefd3c31b8 R08: 0000000000000003 R09: 0000000000008040
>> R10: 0000000000000541 R11: 0000000000000206 R12: 0000000000000003
>> R13: 00007ffefd3c4cc9 R14: 0000000000000001 R15: 0000000000000001
>> Code: af f0 01 00 00 48 89 fb 4d 8b b5 10 0b 00 00 4d 8d be 70 05 00 00 49 81 c6 68 05 00 00 4c 89 ff e8 0f 44 43 00 48 8b 03 4c 89 f7 <48> 8b 30 e8 0e fc ff ff 48 85 c0 49 89 c4 74 0b 4c 89 f6 48 89
>> RIP: __del_reloc_root+0x3f/0xa6 RSP: ffffa6f226427a40
>> CR2: 0000000000000000
>> ---[ end trace 64c3fa4dc953d295 ]---
>> Kernel panic - not syncing: Fatal exception
>> Kernel Offset: 0x1e000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>> Rebooting in 20 seconds..
>> ACPI MEMORY or I/O RESET_REG.
>>
>> -- 
>> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>> Microsoft is to operating systems ....
>>                                        .... what McDonalds is to gourmet cooking
>> Home page: http://marc.merlins.org/
> 

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Qu Wenruo]

At 05/02/2017 11:23 AM, Marc MERLIN wrote:
> Hi Chris,
> 
> Thanks for the reply, much appreciated.
> 
> On Mon, May 01, 2017 at 07:50:22PM -0600, Chris Murphy wrote:
>> What about btfs check (no repair), without and then also with --mode=lowmem?
>>
>> In theory I like the idea of a 24 hour rollback; but in normal usage
>> Btrfs will eventually free up space containing stale and no longer
>> necessary metadata. Like the chunk tree, it's always changing, so you
>> get to a point, even with snapshots, that the old state of that tree
>> is just - gone. A snapshot of an fs tree does not make the chunk tree
>> frozen in time.
>   
> Right, of course, I was being way over optimistic here. I kind of forgot
> that metadata wasn't COW, my bad.
> 
>> In any case, it's a big problem in my mind if no existing tools can
>> fix a file system of this size. So before making anymore changes, make
>> sure you have a btrfs-image somewhere, even if it's huge. The offline
>> checker needs to be able to repair it, right now it's all we have for
>> such a case.
> 
> The image will be huge, and take maybe 24H to make (last time it took
> some silly amount of time like that), and honestly I'm not sure how
> useful it'll be.
> Outside of the kernel crashing if I do a btrfs balance, and hopefully
> the crash report I gave is good enough, the state I'm in is not btrfs'
> fault.
> 
> If I can't roll back to a reasonably working state, with data loss of a
> known quantity that I can recover from backup, I'll have to destroy and
> filesystem and recover from scratch, which will take multiple days.
> Since I can't wait too long before getting back to a working state, I
> think I'm going to try btrfs check --repair after a scrub to get a list
> of all the pathanmes/inodes that are known to be damaged, and work from
> there.
> Sounds reasonable?
> 
> Also, how is --mode=lowmem being useful?
> 
> And for re-parenting a sub-subvolume, is that possible?
> (I want to delete /sub1/ but I can't because I have /sub1/sub2 that's also a subvolume
> and I'm not sure how to re-parent sub2 to somewhere else so that I can subvolume delete
> sub1)
> 
> In the meantime, a simple check without repair looks like this. It will
> likely take many hours to complete:
> gargamel:/var/local/space# btrfs check /dev/mapper/dshelf2
> Checking filesystem on /dev/mapper/dshelf2
> UUID: 03e9a50c-1ae6-4782-ab9c-5f310a98e653
> checking extents
> checksum verify failed on 3096461459456 found 0E6B7980 wanted FBE5477A
> checksum verify failed on 3096461459456 found 0E6B7980 wanted FBE5477A
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
> checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
> checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
> parent transid verify failed on 1671538819072 wanted 293964 found 293902
> parent transid verify failed on 1671538819072 wanted 293964 found 293902
> checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
> checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
> checksum verify failed on 1759425052672 found 843B59F1 wanted F0FF7D00
> checksum verify failed on 1759425052672 found 843B59F1 wanted F0FF7D00
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2898779357184 found 96395131 wanted 433D6E09
> checksum verify failed on 2898779357184 found 96395131 wanted 433D6E09
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
> bytenr mismatch, want=2899180224512, have=3981076597540270796
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
> (...)

Full output please.

I know it will be long, but the point here is, full output could help us 
to at least locate where the most corruption are.

If most corruption are only in extent tree, the chance to recover will 
increase hugely.

As extent tree is just a backref for all allocated extents, it's not 
really important if recovery (read) is the primary goal.

But if other tree (fs or subvolume tree important for you) also get 
corrupted, I'm afraid your last chance will be "btrfs restore" then.

Thanks,
Qu

> 
> Thanks,
> Marc
> 

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Qu Wenruo]

At 05/05/2017 09:19 AM, Qu Wenruo wrote:
> 
> 
> At 05/02/2017 11:23 AM, Marc MERLIN wrote:
>> Hi Chris,
>>
>> Thanks for the reply, much appreciated.
>>
>> On Mon, May 01, 2017 at 07:50:22PM -0600, Chris Murphy wrote:
>>> What about btfs check (no repair), without and then also with 
>>> --mode=lowmem?
>>>
>>> In theory I like the idea of a 24 hour rollback; but in normal usage
>>> Btrfs will eventually free up space containing stale and no longer
>>> necessary metadata. Like the chunk tree, it's always changing, so you
>>> get to a point, even with snapshots, that the old state of that tree
>>> is just - gone. A snapshot of an fs tree does not make the chunk tree
>>> frozen in time.
>> Right, of course, I was being way over optimistic here. I kind of forgot
>> that metadata wasn't COW, my bad.
>>
>>> In any case, it's a big problem in my mind if no existing tools can
>>> fix a file system of this size. So before making anymore changes, make
>>> sure you have a btrfs-image somewhere, even if it's huge. The offline
>>> checker needs to be able to repair it, right now it's all we have for
>>> such a case.
>>
>> The image will be huge, and take maybe 24H to make (last time it took
>> some silly amount of time like that), and honestly I'm not sure how
>> useful it'll be.
>> Outside of the kernel crashing if I do a btrfs balance, and hopefully
>> the crash report I gave is good enough, the state I'm in is not btrfs'
>> fault.
>>
>> If I can't roll back to a reasonably working state, with data loss of a
>> known quantity that I can recover from backup, I'll have to destroy and
>> filesystem and recover from scratch, which will take multiple days.
>> Since I can't wait too long before getting back to a working state, I
>> think I'm going to try btrfs check --repair after a scrub to get a list
>> of all the pathanmes/inodes that are known to be damaged, and work from
>> there.
>> Sounds reasonable?
>>
>> Also, how is --mode=lowmem being useful?
>>
>> And for re-parenting a sub-subvolume, is that possible?
>> (I want to delete /sub1/ but I can't because I have /sub1/sub2 that's 
>> also a subvolume
>> and I'm not sure how to re-parent sub2 to somewhere else so that I can 
>> subvolume delete
>> sub1)
>>
>> In the meantime, a simple check without repair looks like this. It will
>> likely take many hours to complete:
>> gargamel:/var/local/space# btrfs check /dev/mapper/dshelf2
>> Checking filesystem on /dev/mapper/dshelf2
>> UUID: 03e9a50c-1ae6-4782-ab9c-5f310a98e653
>> checking extents
>> checksum verify failed on 3096461459456 found 0E6B7980 wanted FBE5477A
>> checksum verify failed on 3096461459456 found 0E6B7980 wanted FBE5477A
>> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
>> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
>> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
>> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
>> bytenr mismatch, want=2899180224512, have=3981076597540270796
>> checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
>> checksum verify failed on 1449488023552 found CECC36AF wanted 199FE6C5
>> checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
>> checksum verify failed on 1449544613888 found 895D691B wanted A0C64D2B
>> parent transid verify failed on 1671538819072 wanted 293964 found 293902
>> parent transid verify failed on 1671538819072 wanted 293964 found 293902
>> checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
>> checksum verify failed on 1671603781632 found 18BC28D6 wanted 372655A0
>> checksum verify failed on 1759425052672 found 843B59F1 wanted F0FF7D00
>> checksum verify failed on 1759425052672 found 843B59F1 wanted F0FF7D00
>> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
>> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
>> checksum verify failed on 2898779357184 found 96395131 wanted 433D6E09
>> checksum verify failed on 2898779357184 found 96395131 wanted 433D6E09
>> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
>> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
>> checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
>> checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
>> bytenr mismatch, want=2899180224512, have=3981076597540270796
>> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
>> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
>> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
>> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
>> checksum verify failed on 2182657212416 found CD8EFC0C wanted 70847071
>> (...)
> 
> Full output please.

Sorry for not noticing the link.

[Conclusion]
After checking the full result, some of fs/subvolume trees are corrupted.

[Details]
Some example here:

---
ref mismatch on [6674127745024 32768] extent item 0, found 1
Backref 6674127745024 parent 7566652473344 owner 0 offset 0 num_refs 0 
not found in extent tree
Incorrect local backref count on 6674127745024 parent 7566652473344 
owner 0 offset 0 found 1 wanted 0 back 0x5648afda0f20
backpointer mismatch on [6674127745024 32768]
---

The extent at 6674127745024 seems to be an *DATA* extent.
While current default nodesize is 16K and ancient default node is 4K.

Unless you specified -n 32K at mkfs time, it's a DATA extent.

Further more, it's a shared data backref, it's using its parent tree 
block to do backref walk.

And its parent tree block is 7566652473344.
While such bytenr can't be found anywhere (including csum error output), 
that's to say either we can't find that tree block nor can't reach the 
tree root for it.

Considering it's data extent, its owner is either root or fs/subvolume tree.


Such cases are everywhere, as I found other extent sized from 4K to 44K, 
so I'm pretty sure there must be some fs/subvolume tree corrupted.
(Data extent in root tree is seldom 4K sized)

So unfortunately, your fs/subvolume trees are also corrupted.
And almost no chance to do a graceful recovery.

[Alternatives]
I would recommend to use "btrfs restore -f <subvolid>" to restore 
specified subvolume.

What we can do is to try to dump the tree of a subvolume, and manually 
gather what's still here and put them somewhere else.
And that's what btrfs-restore is doing.

The only good new is, your chunk tree seems to be good, so btrfs-restore 
shouldn't encounter too many problems.

Good luck.

Thanks,
Qu

> 
> I know it will be long, but the point here is, full output could help us 
> to at least locate where the most corruption are.
> 
> If most corruption are only in extent tree, the chance to recover will 
> increase hugely.
> 
> As extent tree is just a backref for all allocated extents, it's not 
> really important if recovery (read) is the primary goal.
> 
> But if other tree (fs or subvolume tree important for you) also get 
> corrupted, I'm afraid your last chance will be "btrfs restore" then.
> 
> Thanks,
> Qu
> 
>>
>> Thanks,
>> Marc
>>

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Marc MERLIN]

On Fri, May 05, 2017 at 09:19:29AM +0800, Qu Wenruo wrote:
> Sorry for not noticing the link.
 
no problem, it was only one line amongst many :)
Thanks much for having had a look.

> [Conclusion]
> After checking the full result, some of fs/subvolume trees are corrupted.
> 
> [Details]
> Some example here:
> 
> ---
> ref mismatch on [6674127745024 32768] extent item 0, found 1
> Backref 6674127745024 parent 7566652473344 owner 0 offset 0 num_refs 0 not
> found in extent tree
> Incorrect local backref count on 6674127745024 parent 7566652473344 owner 0
> offset 0 found 1 wanted 0 back 0x5648afda0f20
> backpointer mismatch on [6674127745024 32768]
> ---
> 
> The extent at 6674127745024 seems to be an *DATA* extent.
> While current default nodesize is 16K and ancient default node is 4K.
> 
> Unless you specified -n 32K at mkfs time, it's a DATA extent.

I did not, so you must be right about DATA, which should be good, right,
I don't mind losing data as long as the underlying metadata is correct.

I should have given more data on the FS:

gargamel:/var/local/src/btrfs-progs# btrfs fi df /mnt/btrfs_pool2/
Data, single: total=6.28TiB, used=6.12TiB
System, DUP: total=32.00MiB, used=720.00KiB
Metadata, DUP: total=97.00GiB, used=94.39GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

gargamel:/var/local/src/btrfs-progs# btrfs fi usage /mnt/btrfs_pool2
Overall:
    Device size:                   7.28TiB
    Device allocated:              6.47TiB
    Device unallocated:          824.48GiB
    Device missing:                  0.00B
    Used:                          6.30TiB
    Free (estimated):            994.45GiB      (min: 582.21GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:6.28TiB, Used:6.12TiB
   /dev/mapper/dshelf2     6.28TiB

Metadata,DUP: Size:97.00GiB, Used:94.39GiB
   /dev/mapper/dshelf2   194.00GiB

System,DUP: Size:32.00MiB, Used:720.00KiB
   /dev/mapper/dshelf2    64.00MiB

Unallocated:
   /dev/mapper/dshelf2   824.48GiB


> Further more, it's a shared data backref, it's using its parent tree block
> to do backref walk.
> 
> And its parent tree block is 7566652473344.
> While such bytenr can't be found anywhere (including csum error output),
> that's to say either we can't find that tree block nor can't reach the tree
> root for it.
> 
> Considering it's data extent, its owner is either root or fs/subvolume tree.
> 
> 
> Such cases are everywhere, as I found other extent sized from 4K to 44K, so
> I'm pretty sure there must be some fs/subvolume tree corrupted.
> (Data extent in root tree is seldom 4K sized)
> 
> So unfortunately, your fs/subvolume trees are also corrupted.
> And almost no chance to do a graceful recovery.
 
So I'm confused here. You're saying my metadata is not corrupted (and in
my case, I have DUP, so I should have 2 copies), but with data blocks
(which are not duped) corrupted, it's also possible to lose the
filesystem in a way that it can't be taken back to a clean state, even
by deleting some corrupted data?

> [Alternatives]
> I would recommend to use "btrfs restore -f <subvolid>" to restore specified
> subvolume.

I don't need to restore data, the data is a backup. It will just take
many days to recreate (plus many hours of typing from me because the
backup updates are automated, but recreating everything, is not
automated)

So if I understand correctly, my metadata is fine (and I guess I have 2
copies, so it would have been unlucky to get both copies corrupted), but
enough data blocks got corrupted that btrfs cannot recover, even by
deleting the corrupted data blocks. Correct?

And is it not possible to clear the corrupted blocks like this?
./btrfs-corrupt-block -l  2899180224512 /dev/mapper/dshelf2
and just accept the lost data but get btrfs check repair to deal with
the deleted blocks and bring the rest back to a clean state?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Qu Wenruo]

At 05/05/2017 10:40 AM, Marc MERLIN wrote:
> On Fri, May 05, 2017 at 09:19:29AM +0800, Qu Wenruo wrote:
>> Sorry for not noticing the link.
>   
> no problem, it was only one line amongst many :)
> Thanks much for having had a look.
> 
>> [Conclusion]
>> After checking the full result, some of fs/subvolume trees are corrupted.
>>
>> [Details]
>> Some example here:
>>
>> ---
>> ref mismatch on [6674127745024 32768] extent item 0, found 1
>> Backref 6674127745024 parent 7566652473344 owner 0 offset 0 num_refs 0 not
>> found in extent tree
>> Incorrect local backref count on 6674127745024 parent 7566652473344 owner 0
>> offset 0 found 1 wanted 0 back 0x5648afda0f20
>> backpointer mismatch on [6674127745024 32768]
>> ---
>>
>> The extent at 6674127745024 seems to be an *DATA* extent.
>> While current default nodesize is 16K and ancient default node is 4K.
>>
>> Unless you specified -n 32K at mkfs time, it's a DATA extent.
> 
> I did not, so you must be right about DATA, which should be good, right,
> I don't mind losing data as long as the underlying metadata is correct.
> 
> I should have given more data on the FS:
> 
> gargamel:/var/local/src/btrfs-progs# btrfs fi df /mnt/btrfs_pool2/
> Data, single: total=6.28TiB, used=6.12TiB
> System, DUP: total=32.00MiB, used=720.00KiB
> Metadata, DUP: total=97.00GiB, used=94.39GiB

Tons of metadata since the fs is so large.

> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> gargamel:/var/local/src/btrfs-progs# btrfs fi usage /mnt/btrfs_pool2
> Overall:
>      Device size:                   7.28TiB
>      Device allocated:              6.47TiB
>      Device unallocated:          824.48GiB
>      Device missing:                  0.00B
>      Used:                          6.30TiB
>      Free (estimated):            994.45GiB      (min: 582.21GiB)
>      Data ratio:                       1.00
>      Metadata ratio:                   2.00
>      Global reserve:              512.00MiB      (used: 0.00B)
> 
> Data,single: Size:6.28TiB, Used:6.12TiB
>     /dev/mapper/dshelf2     6.28TiB
> 
> Metadata,DUP: Size:97.00GiB, Used:94.39GiB
>     /dev/mapper/dshelf2   194.00GiB
> 
> System,DUP: Size:32.00MiB, Used:720.00KiB
>     /dev/mapper/dshelf2    64.00MiB
> 
> Unallocated:
>     /dev/mapper/dshelf2   824.48GiB
> 
> 
>> Further more, it's a shared data backref, it's using its parent tree block
>> to do backref walk.
>>
>> And its parent tree block is 7566652473344.
>> While such bytenr can't be found anywhere (including csum error output),
>> that's to say either we can't find that tree block nor can't reach the tree
>> root for it.
>>
>> Considering it's data extent, its owner is either root or fs/subvolume tree.
>>
>>
>> Such cases are everywhere, as I found other extent sized from 4K to 44K, so
>> I'm pretty sure there must be some fs/subvolume tree corrupted.
>> (Data extent in root tree is seldom 4K sized)
>>
>> So unfortunately, your fs/subvolume trees are also corrupted.
>> And almost no chance to do a graceful recovery.
>   
> So I'm confused here. You're saying my metadata is not corrupted (and in
> my case, I have DUP, so I should have 2 copies),

Nope, here I'm all talking about metadata (tree blocks).
Difference is the owner, either extent tree or fs/subvolume tree.

The fsck doesn't check data blocks.

> but with data blocks
> (which are not duped) corrupted, it's also possible to lose the
> filesystem in a way that it can't be taken back to a clean state, even
> by deleting some corrupted data?

No, it can't be repaired by deleting data.

The problem is, tree blocks (metadata) that refers these data blocks are 
corrupted.

And they are corrupted in such a way that both extent tree (tree 
contains extent allocation info) and fs tree (tree contains real fs 
info, like inode and data location) are corrupted.

So graceful recovery is not possible now.

> 
>> [Alternatives]
>> I would recommend to use "btrfs restore -f <subvolid>" to restore specified
>> subvolume.
> 
> I don't need to restore data, the data is a backup. It will just take
> many days to recreate (plus many hours of typing from me because the
> backup updates are automated, but recreating everything, is not
> automated)
> 
> So if I understand correctly, my metadata is fine (and I guess I have 2
> copies, so it would have been unlucky to get both copies corrupted), but
> enough data blocks got corrupted that btrfs cannot recover, even by
> deleting the corrupted data blocks. Correct?

Unfortunately, no, even you have 2 copies, a lot of tree blocks are 
corrupted that neither copy matches checksum.

Just like the following tree block, both copy have wrong checksum.
---
checksum verify failed on 2899180224512 found ABBE39B0 wanted E0735D0E
checksum verify failed on 2899180224512 found 7A6D427F wanted 7E899EE5
---

> 
> And is it not possible to clear the corrupted blocks like this?
> ./btrfs-corrupt-block -l  2899180224512 /dev/mapper/dshelf2
> and just accept the lost data but get btrfs check repair to deal with
> the deleted blocks and bring the rest back to a clean state?No, that won't help.

Corrupted blocks are corrupted, that command is just trying to corrupt 
it again.
It won't do the black magic to adjust tree blocks to avoid them.

That's done in btrfs check. (and --repair)
Btrfs check will just skip corrupted tree blocks and continue, while 
btrfs check --repair will try to rebuild the tree and avoid corrupted 
blocks.

But as you can see, btrfs check can't handle it, due to the complicated 
corruption combination.

So I'm afraid no good method to recover.

Thanks,
Qu
> 
> Thanks,
> Marc
> 

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Marc MERLIN]

Thanks again for your answer. Obviously even if my filesystem is toast,
it's useful to learn from what happened.

On Fri, May 05, 2017 at 01:03:02PM +0800, Qu Wenruo wrote:
> > > So unfortunately, your fs/subvolume trees are also corrupted.
> > > And almost no chance to do a graceful recovery.
> > So I'm confused here. You're saying my metadata is not corrupted (and in
> > my case, I have DUP, so I should have 2 copies),
> 
> Nope, here I'm all talking about metadata (tree blocks).
> Difference is the owner, either extent tree or fs/subvolume tree.
 
I see. I didn't realize that my filesystem managed to corrupt both
copies of its metadata.

> The fsck doesn't check data blocks.

Right, that's what scrub does, fair enough.

> The problem is, tree blocks (metadata) that refers these data blocks are
> corrupted.
> 
> And they are corrupted in such a way that both extent tree (tree contains
> extent allocation info) and fs tree (tree contains real fs info, like inode
> and data location) are corrupted.
> 
> So graceful recovery is not possible now.

I see, thanks for explaining.

> Unfortunately, no, even you have 2 copies, a lot of tree blocks are
> corrupted that neither copy matches checksum.
 
Thanks for confirming. I guess if I'm having corruption due to a bad
card, it makes sense that both get updated after one another and both
got corrupted for the same reason.

> Corrupted blocks are corrupted, that command is just trying to corrupt it
> again.
> It won't do the black magic to adjust tree blocks to avoid them.
 
I see. you may hve seen the earlier message from Kai Krakow who was
able to to recover his FS by trying this trick, but I understand it
can't work in all cases.

Thanks again for your answers.
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Kai Krakow]

Am Fri, 5 May 2017 08:43:23 -0700
schrieb Marc MERLIN <marc@merlins.org>:

[missing quote of the command]
> > Corrupted blocks are corrupted, that command is just trying to
> > corrupt it again.
> > It won't do the black magic to adjust tree blocks to avoid them.  
>  
> I see. you may hve seen the earlier message from Kai Krakow who was
> able to to recover his FS by trying this trick, but I understand it
> can't work in all cases.

Huh, what trick? I don't take credit for it... ;-)

The corrupt-block trick must've been someone else...


-- 
Regards,
Kai

Replies to list-only preferred.

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Marc MERLIN]

On Tue, May 02, 2017 at 05:01:02AM +0000, Duncan wrote:
> Marc MERLIN posted on Mon, 01 May 2017 20:23:46 -0700 as excerpted:
> 
> > Also, how is --mode=lowmem being useful?
> 
> FWIW, I just watched your talk that's linked from the wiki, and wondered 
> what you were doing these days as I hadn't seen any posts from you here 
> for awhile.
 
First, sorry for the late reply. Because you didn't Cc me in the answer,
it went to a different folder where I only saw your replies now.
Off topic, but basically I'm not dead or anything, I have btrfs working
well enough to not mess with it further because I have many other
hobbies :)  that is unless I put a new SAS card in my server, hit some
corruption bugs, and now I'm back spending days fixing the system.

> Well, that you're asking that question confirms you've not been following 
> the list too closely...  Of course that's understandable as people have 
> other stuff to do, but just sayin'.

That's exactly right. I'm subscribed to way too many lists on way too
many topics to be up to date with all, sadly :(

> Of course on-list I'm somewhat known for my arguments propounding the 
> notion that any filesystem that's too big to be practically maintained 
> (including time necessary to restore from backups, should that be 
> necessary for whatever reason) is... too big... and should ideally be 
> broken along logical and functional boundaries into a number of 
> individual smaller filesystems until such point as each one is found to 
> be practically maintainable within a reasonably practical time frame.  
> Don't put all the eggs in one basket, and when the bottom of one of those 
> baskets inevitably falls out, most of your eggs will be safe in other 
> baskets. =:^)
 
That's a valid point, and in my case, I can back it up/restore, it just
takes a bit of time, but most of the time is manually babysitting all
those subvolumes that I need to recreate by hand with btrfs send/restore
relationships, which all get lost during backup/restore.
This is the most painful part.
What's too big? I've only ever used a filesystem that fits on on a raid
of 4 data drives. That value has increased over time, but I don't have a
a crazy array of 20+ drives as a single filesystem, or anything.
Since drives have gotten bigger, but not that much faster, I use bcache
to make things more acceptable in speed.

> *BUT*, and here's the "go further" part, keep in mind that subvolume-read-
> only is a property, gettable and settable by btrfs property.
> 
> So you should be able to unset the read-only property of a subvolume or 
> snapshot, move it, then if desired, set it again.
> 
> Of course I wouldn't expect send -p to work with such a snapshot, but 
> send -c /might/ still work, I'm not actually sure but I'd consider it 
> worth trying.  (I'd try -p as well, but expect it to fail...)

That's an interesting point, thanks for making it.
In that case, I did have to destroy and recreate the filesystem since
btrfs check --repair was unable to fix it, but knowing how to reparent
read only subvolumes may be handy in the future, thanks.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.11 relocate crash, null pointer + rolling back a filesystem by X hours?

[Duncan]

Marc MERLIN posted on Tue, 23 May 2017 09:58:47 -0700 as excerpted:

> That's a valid point, and in my case, I can back it up/restore, it just
> takes a bit of time, but most of the time is manually babysitting all
> those subvolumes that I need to recreate by hand with btrfs send/restore
> relationships, which all get lost during backup/restore.
> This is the most painful part.
> What's too big? I've only ever used a filesystem that fits on on a raid
> of 4 data drives. That value has increased over time, but I don't have a
> a crazy array of 20+ drives as a single filesystem, or anything.
> Since drives have gotten bigger, but not that much faster, I use bcache
> to make things more acceptable in speed.

What's too big?  That depends on your tolerance for pain, but given the 
subvolumes manually recreated by hand with send/receive scenario, I'd 
probably try to break it down so while there's the same number of 
snapshots to restore, the number of subvolumes the snapshots are taken 
against are limited.

My own rule of thumb is if it's taking so long that it's a barrier to 
doing it, I really need to either break things down further, or upgrade 
to faster storage.  The latter is why I'm actually looking at upgrading 
my media and second backup set, on spinning rust, to ssd.  Because while 
I used to do backups spinning rust to spinning rust of that size all the 
time, ssds have spoiled me, and now I dread doing the spinning rust 
backups... or restores.   Tho in my case the spinning rust is only a half-
TB, so a pair of half-TB to 1 TB ssds for an upgrade is still cost 
effective.  It's not like I'm going multi-TB, which would still be cost 
prohibitive on SSD, particularly since I want raid1, so doubling the 
number of SSDs.

Meanwhile, what I'd do with that raid of four drives (and /did/ do with 
my 4-drive raid back a few storage generations ago, when 300 GB spinning-
rust disks were still quite big, and what I do with my paired SSDs with 
btrfs now) is partition them up and do raids of partitions on each drive.

One thing that's nice about that is that you can actually do a set of 
backups on a second set of partitions on the same physical devices, 
because the physical device redundancy of the raids covers loss of a 
device, and the separate partitions and raids (btrfs raid1 now) cover the 
fat-finger or simple loss of filesystem risk.  A second set of backups to 
separate devices can then be made just in case, and depending on the 
need, swapped out to off-premises or uploaded to the cloud or whatever, 
but you always have the primary backup at hand to boot to or mount if the 
working copy fails, by simply pointing to the backup partitions and 
filesystem instead of the normal working copy.  For root, I even have a 
grub menu item that switches to the backup copy, and for fstab, I have a 
set of stubs that are assembled via script into three copies of fstab 
that swap working and backup copies as necessary, with /etc/fstab itself 
being a symlink to the working copy one, that I simply switch to point to 
the one that loads the backup copies as working, on the backup.  Or I can 
mount the root filesystem for maintenance from the initramfs, and switch 
the fstab symlink from there, before exiting maintenance and booting the 
main system.

I learned this "split it up" method the hard way back before mdraid had 
write-intent bitmaps, and I had only two much larger raids, working and 
backup, where if one device dropped out and I brought it back in, I had 
to wait way too long for the huge working raid to resync.  When I split 
things up by function into multiple raids, most of the time only some of 
them were active and only one or two of the active ones would actually 
have been being written at the time so were out of sync, and syncing them 
was fast as they were much smaller than the larger full system raids I 
had been using previously.

>> *BUT*, and here's the "go further" part, keep in mind that
>> subvolume-read-
>> only is a property, gettable and settable by btrfs property.
>> 
>> So you should be able to unset the read-only property of a subvolume or
>> snapshot, move it, then if desired, set it again.
>> 
>> Of course I wouldn't expect send -p to work with such a snapshot, but
>> send -c /might/ still work, I'm not actually sure but I'd consider it
>> worth trying.  (I'd try -p as well, but expect it to fail...)
> 
> That's an interesting point, thanks for making it.
> In that case, I did have to destroy and recreate the filesystem since
> btrfs check --repair was unable to fix it, but knowing how to reparent
> read only subvolumes may be handy in the future, thanks.

Hopefully you won't end up testing it any time soon, but if you do, 
please confirm whether my suspicions that send -p won't work after 
toggling and reparenting, but send -c still will, are correct.

(For those who read this out of thread context where I believe I already 
stated it, my own use-case involves neither snapshots nor send-receive.  
But it'd be useful information to confirm, both for others, and in case I 
suddenly find myself with a different use-case for some reason or other.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Marc MERLIN]

My filesystem got remounted read only, and yet after a lengthy
btrfs check --repair, it ran clean.

Any idea what went wrong?
[846332.992285] WARNING: CPU: 4 PID: 4095 at fs/btrfs/free-space-cache.c:1476 tree_insert_offset+0x78/0xb1
[846333.744721] BTRFS critical (device dm-1): unable to add free space :-17
[847312.529660] BTRFS: Transaction aborted (error -17)
[847313.218391] BTRFS: error (device dm-1) in btrfs_run_delayed_refs:2961: errno=-17 Object already exists
[847313.247668] BTRFS info (device dm-1): forced readonly

gargamel:~# btrfs check --repair /dev/mapper/dshelf2
enabling repair mode
Checking filesystem on /dev/mapper/dshelf2
UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 5544539336704 bytes used, no error found
total csum bytes: 5344305964
total tree bytes: 70455754752
total fs tree bytes: 58427670528
total extent tree bytes: 5372461056
btree space waste bytes: 10620592981
file data blocks allocated: 7735818444800
 referenced 6155805896704


this is how it went read only:
[846332.977964] ------------[ cut here ]------------
[846332.992285] WARNING: CPU: 4 PID: 4095 at fs/btrfs/free-space-cache.c:1476 tree_insert_offset+0x78/0xb1
[846333.402648] CPU: 4 PID: 4095 Comm: btrfs-transacti Tainted: G     U          4.11.3-amd64-preempt-sysrq-20170406 #5
[846333.434917] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
[846333.463597] Call Trace:
[846333.469942] usb 2-1-port4: device 2-1.4 not suspended yet
[846333.489639]  dump_stack+0x61/0x7d
[846333.500480]  __warn+0xc2/0xdd
[846333.510956]  warn_slowpath_null+0x1d/0x1f
[846333.524103]  tree_insert_offset+0x78/0xb1
[846333.537337]  link_free_space+0x2c/0x41
[846333.549991]  __btrfs_add_free_space+0x89/0x3aa
[846333.564236]  ? kmem_cache_free+0x3d/0x92
[846333.577702]  btrfs_add_free_space+0x1d/0x1f
[846333.591179]  unpin_extent_range+0xf3/0x2b0
[846333.605220]  btrfs_finish_extent_commit+0xda/0x1d4
[846333.621324]  btrfs_commit_transaction+0x629/0x79a
[846333.637205]  ? add_wait_queue+0x44/0x44
[846333.649680]  transaction_kthread+0xe2/0x178
[846333.663201]  ? btrfs_cleanup_transaction+0x3e8/0x3e8
[846333.679033]  kthread+0xfb/0x100
[846333.690261]  ? init_completion+0x24/0x24
[846333.703239]  ? do_fast_syscall_32+0xb7/0xfe
[846333.717649]  ret_from_fork+0x2c/0x40
[846333.729656] ---[ end trace 27aa532d1886e536 ]---
[846333.744721] BTRFS critical (device dm-1): unable to add free space :-17

[847312.529660] BTRFS: Transaction aborted (error -17)
[847312.912784] CPU: 6 PID: 4094 Comm: btrfs-cleaner Tainted: G     U  W       4.11.3-amd64-preempt-sysrq-20170406 #5
[847312.913132] usb 2-1-port4: device 2-1.4 not suspended yet
[847312.962394] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
[847312.990936] Call Trace:
[847312.999347]  dump_stack+0x61/0x7d
[847313.010383]  __warn+0xc2/0xdd
[847313.020351]  warn_slowpath_fmt+0x5a/0x76
[847313.033274]  btrfs_run_delayed_refs+0xb1/0x1cc
[847313.047655]  btrfs_should_end_transaction+0x50/0x57
[847313.063910]  btrfs_drop_snapshot+0x38a/0x6c4
[847313.078619]  ? btrfs_kill_all_delayed_nodes+0x5f/0xd7
[847313.094916]  ? _raw_spin_lock+0x15/0x17
[847313.108325]  btrfs_clean_one_deleted_snapshot+0xce/0xdc
[847313.125493]  cleaner_kthread+0x91/0x14b
[847313.138228]  ? btrfs_destroy_pinned_extent+0xd2/0xd2
[847313.154308]  kthread+0xfb/0x100
[847313.164900]  ? init_completion+0x24/0x24
[847313.177781]  ? do_fast_syscall_32+0xb7/0xfe
[847313.191490]  ret_from_fork+0x2c/0x40
[847313.203432] ---[ end trace 27aa532d1886e537 ]---
[847313.218391] BTRFS: error (device dm-1) in btrfs_run_delayed_refs:2961: errno=-17 Object already exists
[847313.247668] BTRFS info (device dm-1): forced readonly

[849789.173126] BTRFS error (device dm-1): parent transid verify failed on 1935589703680 wanted 37959 found 3229
[849789.218675] BTRFS error (device dm-1): parent transid verify failed on 1935589703680 wanted 37959 found 3229

[863279.783590] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
[863279.827526] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
[863279.857797] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
[863279.888096] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
[863279.918393] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
[863279.948740] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
[863279.979033] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
[863280.009362] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
[863280.040438] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
[863280.070966] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 5818 bytes --]

On Tue, Jun 20, 2017 at 07:39:16AM -0700, Marc MERLIN wrote:
> My filesystem got remounted read only, and yet after a lengthy
> btrfs check --repair, it ran clean.
> 
> Any idea what went wrong?
> [846332.992285] WARNING: CPU: 4 PID: 4095 at fs/btrfs/free-space-cache.c:1476 tree_insert_offset+0x78/0xb1
> [846333.744721] BTRFS critical (device dm-1): unable to add free space :-17
> [847312.529660] BTRFS: Transaction aborted (error -17)
> [847313.218391] BTRFS: error (device dm-1) in btrfs_run_delayed_refs:2961: errno=-17 Object already exists

   Error 17 is EEXIST, so I'd guess (and it is a guess) that it's
trying to add a free space cache record for some space that already
has such a record. This might also match with:

[...]
> gargamel:~# btrfs check --repair /dev/mapper/dshelf2
[...]
> cache and super generation don't match, space cache will be invalidated
[...]

   I'd try clearing the cache (mount with -o clear_cache, once), and
then letting it rebuild.

   Hugo.

> checking fs roots
> checking csums
> checking root refs
> found 5544539336704 bytes used, no error found
> total csum bytes: 5344305964
> total tree bytes: 70455754752
> total fs tree bytes: 58427670528
> total extent tree bytes: 5372461056
> btree space waste bytes: 10620592981
> file data blocks allocated: 7735818444800
>  referenced 6155805896704
> 
> 
> this is how it went read only:
> [846332.977964] ------------[ cut here ]------------
> [846332.992285] WARNING: CPU: 4 PID: 4095 at fs/btrfs/free-space-cache.c:1476 tree_insert_offset+0x78/0xb1
> [846333.402648] CPU: 4 PID: 4095 Comm: btrfs-transacti Tainted: G     U          4.11.3-amd64-preempt-sysrq-20170406 #5
> [846333.434917] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> [846333.463597] Call Trace:
> [846333.469942] usb 2-1-port4: device 2-1.4 not suspended yet
> [846333.489639]  dump_stack+0x61/0x7d
> [846333.500480]  __warn+0xc2/0xdd
> [846333.510956]  warn_slowpath_null+0x1d/0x1f
> [846333.524103]  tree_insert_offset+0x78/0xb1
> [846333.537337]  link_free_space+0x2c/0x41
> [846333.549991]  __btrfs_add_free_space+0x89/0x3aa
> [846333.564236]  ? kmem_cache_free+0x3d/0x92
> [846333.577702]  btrfs_add_free_space+0x1d/0x1f
> [846333.591179]  unpin_extent_range+0xf3/0x2b0
> [846333.605220]  btrfs_finish_extent_commit+0xda/0x1d4
> [846333.621324]  btrfs_commit_transaction+0x629/0x79a
> [846333.637205]  ? add_wait_queue+0x44/0x44
> [846333.649680]  transaction_kthread+0xe2/0x178
> [846333.663201]  ? btrfs_cleanup_transaction+0x3e8/0x3e8
> [846333.679033]  kthread+0xfb/0x100
> [846333.690261]  ? init_completion+0x24/0x24
> [846333.703239]  ? do_fast_syscall_32+0xb7/0xfe
> [846333.717649]  ret_from_fork+0x2c/0x40
> [846333.729656] ---[ end trace 27aa532d1886e536 ]---
> [846333.744721] BTRFS critical (device dm-1): unable to add free space :-17
> 
> [847312.529660] BTRFS: Transaction aborted (error -17)
> [847312.912784] CPU: 6 PID: 4094 Comm: btrfs-cleaner Tainted: G     U  W       4.11.3-amd64-preempt-sysrq-20170406 #5
> [847312.913132] usb 2-1-port4: device 2-1.4 not suspended yet
> [847312.962394] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> [847312.990936] Call Trace:
> [847312.999347]  dump_stack+0x61/0x7d
> [847313.010383]  __warn+0xc2/0xdd
> [847313.020351]  warn_slowpath_fmt+0x5a/0x76
> [847313.033274]  btrfs_run_delayed_refs+0xb1/0x1cc
> [847313.047655]  btrfs_should_end_transaction+0x50/0x57
> [847313.063910]  btrfs_drop_snapshot+0x38a/0x6c4
> [847313.078619]  ? btrfs_kill_all_delayed_nodes+0x5f/0xd7
> [847313.094916]  ? _raw_spin_lock+0x15/0x17
> [847313.108325]  btrfs_clean_one_deleted_snapshot+0xce/0xdc
> [847313.125493]  cleaner_kthread+0x91/0x14b
> [847313.138228]  ? btrfs_destroy_pinned_extent+0xd2/0xd2
> [847313.154308]  kthread+0xfb/0x100
> [847313.164900]  ? init_completion+0x24/0x24
> [847313.177781]  ? do_fast_syscall_32+0xb7/0xfe
> [847313.191490]  ret_from_fork+0x2c/0x40
> [847313.203432] ---[ end trace 27aa532d1886e537 ]---
> [847313.218391] BTRFS: error (device dm-1) in btrfs_run_delayed_refs:2961: errno=-17 Object already exists
> [847313.247668] BTRFS info (device dm-1): forced readonly
> 
> [849789.173126] BTRFS error (device dm-1): parent transid verify failed on 1935589703680 wanted 37959 found 3229
> [849789.218675] BTRFS error (device dm-1): parent transid verify failed on 1935589703680 wanted 37959 found 3229
> 
> [863279.783590] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
> [863279.827526] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
> [863279.857797] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
> [863279.888096] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
> [863279.918393] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
> [863279.948740] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
> [863279.979033] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
> [863280.009362] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
> [863280.040438] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
> [863280.070966] BTRFS error (device dm-1): parent transid verify failed on 1932065177600 wanted 37959 found 3634
> 

-- 
Hugo Mills             | I believe that it's closely correlated with the
hugo@... carfax.org.uk | aeroswine coefficient
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                       Adrian Bridgett

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Marc MERLIN]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

On Tue, Jun 20, 2017 at 03:23:54PM +0000, Hugo Mills wrote:
> On Tue, Jun 20, 2017 at 07:39:16AM -0700, Marc MERLIN wrote:
> > My filesystem got remounted read only, and yet after a lengthy
> > btrfs check --repair, it ran clean.
> > 
> > Any idea what went wrong?
> > [846332.992285] WARNING: CPU: 4 PID: 4095 at fs/btrfs/free-space-cache.c:1476 tree_insert_offset+0x78/0xb1
> > [846333.744721] BTRFS critical (device dm-1): unable to add free space :-17
> > [847312.529660] BTRFS: Transaction aborted (error -17)
> > [847313.218391] BTRFS: error (device dm-1) in btrfs_run_delayed_refs:2961: errno=-17 Object already exists
> 
>    Error 17 is EEXIST, so I'd guess (and it is a guess) that it's
> trying to add a free space cache record for some space that already
> has such a record. This might also match with:
 
Thanks for having a look. Is it a bug, or is it a problem with my storage
subsystem?

> [...]
> > gargamel:~# btrfs check --repair /dev/mapper/dshelf2
> [...]
> > cache and super generation don't match, space cache will be invalidated
> [...]
> 
>    I'd try clearing the cache (mount with -o clear_cache, once), and
> then letting it rebuild.

"space cache will be invalidated " => doesn't that mean that my cache was
already cleared by check --repair, or are you saying I need to clear it
again?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1999 bytes --]

On Tue, Jun 20, 2017 at 08:26:48AM -0700, Marc MERLIN wrote:
> On Tue, Jun 20, 2017 at 03:23:54PM +0000, Hugo Mills wrote:
> > On Tue, Jun 20, 2017 at 07:39:16AM -0700, Marc MERLIN wrote:
> > > My filesystem got remounted read only, and yet after a lengthy
> > > btrfs check --repair, it ran clean.
> > > 
> > > Any idea what went wrong?
> > > [846332.992285] WARNING: CPU: 4 PID: 4095 at fs/btrfs/free-space-cache.c:1476 tree_insert_offset+0x78/0xb1
> > > [846333.744721] BTRFS critical (device dm-1): unable to add free space :-17
> > > [847312.529660] BTRFS: Transaction aborted (error -17)
> > > [847313.218391] BTRFS: error (device dm-1) in btrfs_run_delayed_refs:2961: errno=-17 Object already exists
> > 
> >    Error 17 is EEXIST, so I'd guess (and it is a guess) that it's
> > trying to add a free space cache record for some space that already
> > has such a record. This might also match with:
>  
> Thanks for having a look. Is it a bug, or is it a problem with my storage
> subsystem?

   Well, I'd say it's probably a problem with some inconsistent data
on the disk. How that data got there is another matter -- it may be
due to a bug which wrote the inconsistent data some time ago, and has
only now been found out.

> > [...]
> > > gargamel:~# btrfs check --repair /dev/mapper/dshelf2
> > [...]
> > > cache and super generation don't match, space cache will be invalidated
> > [...]
> > 
> >    I'd try clearing the cache (mount with -o clear_cache, once), and
> > then letting it rebuild.
> 
> "space cache will be invalidated " => doesn't that mean that my cache was
> already cleared by check --repair, or are you saying I need to clear it
> again?

   I'm never quite sure about that one. :)

   It can't hurt to clear it manually as well.

   Hugo.

-- 
Hugo Mills             | I believe that it's closely correlated with the
hugo@... carfax.org.uk | aeroswine coefficient
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                       Adrian Bridgett

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Marc MERLIN]

[-- Attachment #1: Type: text/plain, Size: 1797 bytes --]

On Tue, Jun 20, 2017 at 03:36:01PM +0000, Hugo Mills wrote:
> > Thanks for having a look. Is it a bug, or is it a problem with my storage
> > subsystem?
> 
>    Well, I'd say it's probably a problem with some inconsistent data
> on the disk. How that data got there is another matter -- it may be
> due to a bug which wrote the inconsistent data some time ago, and has
> only now been found out.
 
Understood.

> > "space cache will be invalidated " => doesn't that mean that my cache was
> > already cleared by check --repair, or are you saying I need to clear it
> > again?
> 
>    I'm never quite sure about that one. :)
> 
>    It can't hurt to clear it manually as well.

Sounds good, done.

In the meantime, I ran into this again:
https://bugzilla.kernel.org/show_bug.cgi?id=195863
btrfs check of a big filesystem kills the kernel due to OOM (but btrfs userspace is not OOM killed)

Is it achievable at all for btrfs check to realize that it's taking all the
available RAM in kernel space, is about to crash the system, and cancel the
check before the system crashes?
I've already confirmed that it doesn't use swap. I've just had to order new
RAM to upgrade my machine from 24GB to 32GB, but 32GB is max for that
hardware, so hopefully the lowmem repair stuff will work before I hit the
32GB limit next time.

In the meantime, though, it really shouldn't crash your system (potentially
causing more damage in the process because you end up with an unclean
shutdown).
Can anyone look at this?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Marc MERLIN]

[-- Attachment #1: Type: text/plain, Size: 3513 bytes --]

On Tue, Jun 20, 2017 at 08:44:29AM -0700, Marc MERLIN wrote:
> On Tue, Jun 20, 2017 at 03:36:01PM +0000, Hugo Mills wrote:
> > > Thanks for having a look. Is it a bug, or is it a problem with my storage
> > > subsystem?
> > 
> >    Well, I'd say it's probably a problem with some inconsistent data
> > on the disk. How that data got there is another matter -- it may be
> > due to a bug which wrote the inconsistent data some time ago, and has
> > only now been found out.
>  
> Understood.
> 
> > > "space cache will be invalidated " => doesn't that mean that my cache was
> > > already cleared by check --repair, or are you saying I need to clear it
> > > again?
> > 
> >    I'm never quite sure about that one. :)
> > 
> >    It can't hurt to clear it manually as well.
> 
> Sounds good, done.
 
Except it didn't help :(
It worked for a while, and failed again.

It looks like I'm hitting a persistent bug :(

[   86.383988] BTRFS: device label dshelf2 devid 1 transid 37975 /dev/mapper/dshelf2
[   98.232529] BTRFS info (device dm-1): use lzo compression
[   98.251982] BTRFS info (device dm-1): disk space caching is enabled
[   98.274847] BTRFS info (device dm-1): has skinny extents
[  104.171597] BTRFS info (device dm-1): detected SSD devices, enabling SSD mode
[  165.429894] BTRFS error (device dm-1): Duplicate entries in free space cache, dumping
[  165.455673] BTRFS warning (device dm-1): failed to load free space cache for block group 2039601954816, rebuilding it now
[  234.221435] BTRFS warning (device dm-1): block group 2837392130048 has wrong amount of free space
[  234.249264] BTRFS warning (device dm-1): failed to load free space cache for block group 2837392130048, rebuilding it now
[  234.636396] BTRFS warning (device dm-1): block group 2885173641216 has wrong amount of free space
[  234.664015] BTRFS warning (device dm-1): failed to load free space cache for block group 2885173641216, rebuilding it now
[  242.042940] BTRFS warning (device dm-1): block group 3116565004288 has wrong amount of free space
[  242.071207] BTRFS warning (device dm-1): failed to load free space cache for block group 3116565004288, rebuilding it now
[  273.910918] BTRFS warning (device dm-1): block group 3209980542976 has wrong amount of free space
[  273.937625] BTRFS warning (device dm-1): failed to load free space cache for block group 3209980542976, rebuilding it now
[  298.578615] BTRFS warning (device dm-1): block group 2305889927168 has wrong amount of free space
[  298.605250] BTRFS warning (device dm-1): failed to load free space cache for block group 2305889927168, rebuilding it now
[  873.265687] BTRFS: Transaction aborted (error -17)
[  873.948245] BTRFS: error (device dm-1) in btrfs_run_delayed_refs:2961: errno=-17 Object already exists
[  873.978884] BTRFS info (device dm-1): forced readonly

Given that check --repair ran clean when I ran it yesterday after this first happened,
and I then ran  mount -o clear_cache , the cache got rebuilt, and I got the problem again, 
this is not looking good, seems like a persistent bug :-/

I'm now going to remount this with nospace_cache to see if your guess about
space_cache was correct.
Other suggestions also welcome :)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Marc MERLIN]

[-- Attachment #1: Type: text/plain, Size: 4740 bytes --]

On Tue, Jun 20, 2017 at 04:12:03PM -0700, Marc MERLIN wrote:
> Given that check --repair ran clean when I ran it yesterday after this first happened,
> and I then ran  mount -o clear_cache , the cache got rebuilt, and I got the problem again, 
> this is not looking good, seems like a persistent bug :-/
> 
> I'm now going to remount this with nospace_cache to see if your guess about
> space_cache was correct.

Now, it seems that disabling the cache is causing some serious hangs:
[ 2055.473113] INFO: task kworker/u16:17:7579 blocked for more than 120 seconds.
[ 2055.496148]       Tainted: G     U          4.11.6-amd64-preempt-sysrq-20170406 #6
[ 2055.520611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2055.545675] kworker/u16:17  D    0  7579      2 0x00000080
[ 2055.563626] Workqueue: writeback wb_workfn (flush-btrfs-4)
[ 2055.581458] Call Trace:
[ 2055.590154]  __schedule+0x4ef/0x627
[ 2055.602830]  schedule+0x89/0x9a
[ 2055.613618]  io_schedule+0x16/0x38
[ 2055.625324]  wait_on_page_bit_common+0xd8/0x151
[ 2055.640413]  ? inode_to_bdi+0x35/0x35
[ 2055.653701]  __lock_page+0x40/0x42
[ 2055.665431]  lock_page+0x19/0x1c
[ 2055.676315]  extent_write_cache_pages.constprop.31+0x173/0x368
[ 2055.695049]  ? update_load_avg+0x227/0x3c6
[ 2055.708592]  ? update_load_avg+0x3b1/0x3c6
[ 2055.722340]  ? list_add+0x1a/0x34
[ 2055.733520]  ? cfs_rq_throttled.isra.24+0xd/0x1d
[ 2055.748503]  ? update_cfs_shares+0x2e/0xcf
[ 2055.761891]  extent_writepages+0x5b/0x80
[ 2055.774854]  ? __percpu_counter_compare+0x29/0x72
[ 2055.790054]  ? insert_reserved_file_extent.constprop.41+0x28e/0x28e
[ 2055.809869]  btrfs_writepages+0x28/0x2a
[ 2055.822516]  do_writepages+0x20/0x29
[ 2055.834251]  __writeback_single_inode+0x8a/0x328
[ 2055.849159]  ? inode_cgwb_enabled+0xd/0x3b
[ 2055.862521]  writeback_sb_inodes+0x22e/0x400
[ 2055.876310]  __writeback_inodes_wb+0x6e/0xb0
[ 2055.890057]  wb_writeback+0x163/0x2ca
[ 2055.902436]  wb_workfn+0x1f7/0x2bf
[ 2055.913520]  ? wb_workfn+0x1f7/0x2bf
[ 2055.925090]  ? __switch_to+0x2c8/0x45f
[ 2055.937184]  process_one_work+0x193/0x2b0
[ 2055.950034]  ? rescuer_thread+0x2b1/0x2b1
[ 2055.962833]  worker_thread+0x1e9/0x2c1
[ 2055.974826]  ? rescuer_thread+0x2b1/0x2b1
[ 2055.988016]  kthread+0xfb/0x100
[ 2055.998183]  ? init_completion+0x24/0x24
[ 2056.010902]  ? do_syscall_64+0x77/0x7d
[ 2056.022802]  ret_from_fork+0x2c/0x40
[ 2056.034224] INFO: task rsync:27554 blocked for more than 120 seconds.
[ 2056.054213]       Tainted: G     U          4.11.6-amd64-preempt-sysrq-20170406 #6
[ 2056.077611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2056.101705] rsync           D    0 27554  27526 0x20020080
[ 2056.119102] Call Trace:
[ 2056.127019]  __schedule+0x4ef/0x627
[ 2056.138385]  schedule+0x89/0x9a
[ 2056.148616]  io_schedule+0x16/0x38
[ 2056.159682]  wait_on_page_bit_common+0xd8/0x151
[ 2056.173787]  ? inode_to_bdi+0x35/0x35
[ 2056.185336]  __lock_page+0x40/0x42
[ 2056.196176]  lock_page+0x19/0x1c
[ 2056.206420]  extent_write_cache_pages.constprop.31+0x173/0x368
[ 2056.224786]  ? _raw_read_unlock+0xe/0x1e
[ 2056.237221]  ? btrfs_set_lock_blocking_rw+0x9a/0x9d
[ 2056.252388]  extent_writepages+0x5b/0x80
[ 2056.264687]  ? insert_reserved_file_extent.constprop.41+0x28e/0x28e
[ 2056.284051]  btrfs_writepages+0x28/0x2a
[ 2056.296117]  do_writepages+0x20/0x29
[ 2056.307426]  __filemap_fdatawrite_range+0x97/0xc3
[ 2056.322374]  filemap_flush+0x1c/0x1e
[ 2056.333627]  btrfs_rename2+0x894/0xf6f
[ 2056.345376]  ? capable_wrt_inode_uidgid+0x3f/0x4e
[ 2056.359977]  ? generic_permission+0x11e/0x175
[ 2056.373719]  vfs_rename+0x234/0x391
[ 2056.384805]  ? vfs_rename+0x234/0x391
[ 2056.396341]  SYSC_renameat2+0x327/0x448
[ 2056.408349]  SyS_rename+0x1e/0x20
[ 2056.418806]  do_fast_syscall_32+0xb7/0xfe
[ 2056.431325]  entry_SYSENTER_compat+0x4c/0x5b
[ 2056.444642] RIP: 0023:0xf76feb39
[ 2056.454861] RSP: 002b:00000000ffe177bc EFLAGS: 00000292 ORIG_RAX: 0000000000000026
[ 2056.478081] RAX: ffffffffffffffda RBX: 00000000ffe18890 RCX: 00000000ffe1a890
[ 2056.500019] RDX: 0000000000000001 RSI: 00000000ffe1a890 RDI: 0000000000000003
[ 2056.521948] RBP: 00000000ffe177f8 R08: 0000000000000000 R09: 0000000000000000
[ 2056.543858] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 2056.565809] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000


Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Chris Murphy]

On Tue, Jun 20, 2017 at 9:44 AM, Marc MERLIN <marc@merlins.org> wrote:

> In the meantime, I ran into this again:
> https://bugzilla.kernel.org/show_bug.cgi?id=195863
> btrfs check of a big filesystem kills the kernel due to OOM (but btrfs userspace is not OOM killed)
>
> Is it achievable at all for btrfs check to realize that it's taking all the
> available RAM in kernel space, is about to crash the system, and cancel the
> check before the system crashes?
> I've already confirmed that it doesn't use swap. I've just had to order new
> RAM to upgrade my machine from 24GB to 32GB, but 32GB is max for that
> hardware, so hopefully the lowmem repair stuff will work before I hit the
> 32GB limit next time.

Right now Btrfs isn't scalable if you have to repair it because large
volumes run into this problem; one of the reasons for the lowmem mode.

It's a separate bug that it OOMs even with swap, I don't know why it
won't use that, it should be up to kernel memory management to deal
with this; I know this works with xfs_repair. I don't know if the idea
is that normal mode will go away, in favor of lowmem mode, or if there
are fixes still planned for normal mode. If it's going to stick
around, it needs to be able to use swap, same for lowmem mode. Just
running into a total inability to --repair isn't OK.


-- 
Chris Murphy

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Chris Murphy]

On Tue, Jun 20, 2017 at 5:12 PM, Marc MERLIN <marc@merlins.org> wrote:

> I'm now going to remount this with nospace_cache to see if your guess about
> space_cache was correct.
> Other suggestions also welcome :)

What results do you get with lowmem mode? It won't repair without
additional patches, but might give a dev a clue what's going on. I
regularly see normal mode check finds no problems, and lowmem mode
finds problems. Lowmem mode is a total rewrite so it's a different
implementation and can find things normal mode won't.


-- 
Chris Murphy

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Marc MERLIN]

On Tue, Jun 20, 2017 at 09:31:42PM -0600, Chris Murphy wrote:
> On Tue, Jun 20, 2017 at 5:12 PM, Marc MERLIN <marc@merlins.org> wrote:
> 
> > I'm now going to remount this with nospace_cache to see if your guess about
> > space_cache was correct.
> > Other suggestions also welcome :)
> 
> What results do you get with lowmem mode? It won't repair without
> additional patches, but might give a dev a clue what's going on. I
> regularly see normal mode check finds no problems, and lowmem mode
> finds problems. Lowmem mode is a total rewrite so it's a different
> implementation and can find things normal mode won't.

Oh, I kind of forgot that lowmem mode looked for more things than regular
mode.
I will run this tonight and see what it says.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Marc MERLIN]

On Tue, Jun 20, 2017 at 09:26:27PM -0600, Chris Murphy wrote:
> Right now Btrfs isn't scalable if you have to repair it because large
> volumes run into this problem; one of the reasons for the lowmem mode.
> 
> It's a separate bug that it OOMs even with swap, I don't know why it
> won't use that, it should be up to kernel memory management to deal

The thing is that it doesn't even get OOM'ed.
I didn't look at the code, but I'm assuming it must be using kernel RAM
instead of user space RAM, which is why it can't be OOM'ed and why it gets
the kernel to deadlock.

If that is the case, then the user space code should monitor kernel space
usage and cancel the check if it's about to run out of usable RAM (better
than deadlocking the system).

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: 4.11.3: BTRFS critical (device dm-1): unable to add free space :-17 => btrfs check --repair runs clean

[Duncan]

Marc MERLIN posted on Tue, 20 Jun 2017 16:12:03 -0700 as excerpted:

> On Tue, Jun 20, 2017 at 08:44:29AM -0700, Marc MERLIN wrote:
>> On Tue, Jun 20, 2017 at 03:36:01PM +0000, Hugo Mills wrote:
>> 
>>>> "space cache will be invalidated " => doesn't that mean that my
>>>> cache was already cleared by check --repair, or are you saying I
>>>> need to clear it again?
>>> 
>>>    I'm never quite sure about that one. :)
>>> 
>>>    It can't hurt to clear it manually as well.
>> 
>> Sounds good, done.
>  
> Except it didn't help :(
> It worked for a while, and failed again.
> 
> It looks like I'm hitting a persistent bug :(

[Omitted free space cache dmesg errors]

> Given that check --repair ran clean when I ran it yesterday after this
> first happened, and I then ran  mount -o clear_cache , the cache got
> rebuilt, and I got the problem again, this is not looking good, seems
> like a persistent bug :-/

Keep in mind this quote from a recent (I'm quoting -progs 4.11) btrfs-
check manpage (reformatted for posting):

>>>>> 

--clear-space-cache v1|v2

completely wipe all free space cache of given type

For free space cache v1, the clear_cache kernel mount option only 
rebuilds the free space cache for block groups that are modified while the
filesystem is mounted with that option. Thus, using this option with v1 
makes it possible to actually clear the entire free space cache.

For free space cache v2, the clear_cache kernel mount option does destroy 
the entire free space cache. This option with v2 provides an alternative
method of clearing the free space cache that doesn’t require mounting the 
filesystem.

<<<<<

Given the dmesg, seems you're still running the space cache, not the v2/
tree (which is fine, I'm conservative enough not to have switched yet 
either).  So try the check option instead of the mount option.  The mount 
option might simply have not caught all the badness while it was active.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Marc MERLIN]

On Tue, Jun 20, 2017 at 08:43:52PM -0700, Marc MERLIN wrote:
> On Tue, Jun 20, 2017 at 09:31:42PM -0600, Chris Murphy wrote:
> > On Tue, Jun 20, 2017 at 5:12 PM, Marc MERLIN <marc@merlins.org> wrote:
> > 
> > > I'm now going to remount this with nospace_cache to see if your guess about
> > > space_cache was correct.
> > > Other suggestions also welcome :)
> > 
> > What results do you get with lowmem mode? It won't repair without
> > additional patches, but might give a dev a clue what's going on. I
> > regularly see normal mode check finds no problems, and lowmem mode
> > finds problems. Lowmem mode is a total rewrite so it's a different
> > implementation and can find things normal mode won't.
> 
> Oh, I kind of forgot that lowmem mode looked for more things than regular
> mode.
> I will run this tonight and see what it says.
 
It's probably still a ways from being finished given how slow lowmem is in
comparison, but sadly it found a bunch of problems which regular mode didn't
find.

I'm pretty bummed. I just spent way too long recreating this filesystem and
the multiple btrfs send/receive relationships from other machines. Too a bit
over a week :(

It looks like the errors are not major (especially if the regular mode
doesn't even see them), but without lowmem --repair, I'm kind of screwed.

I'm wondering if I could/should leave those errors unfixed until lowmem --repair
finally happens, or whether I'm looking at spending another week rebuilding
this filesystem :-/


gargamel:~# btrfs check -p --mode lowmem  /dev/mapper/dshelf2
Checking filesystem on /dev/mapper/dshelf2
UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
ERROR: extent[3886187384832, 81920] referencer count mismatch (root: 11930, owner: 375444, offset: 1851654144) wanted: 1, have: 4
ERROR: extent[3886189391872, 122880] referencer count mismatch (root: 11712, owner: 863395, offset: 79659008) wanted: 1, have: 2
ERROR: extent[3933249708032, 69632] referencer count mismatch (root: 12424, owner: 6945, offset: 2083389440) wanted: 1, have: 2
ERROR: extent[3933249708032, 69632] referencer count mismatch (root: 12172, owner: 6945, offset: 2083389440) wanted: 1, have: 2
ERROR: extent[4571729862656, 876544] referencer count mismatch (root: 11058, owner: 375442, offset: 907706368) wanted: 6, have: 21
ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11712, owner: 375444, offset: 1848672256) wanted: 3, have: 5
ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11276, owner: 375444, offset: 1848672256) wanted: 3, have: 5
ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11058, owner: 375444, offset: 1848672256) wanted: 3, have: 5
ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11494, owner: 375444, offset: 1848672256) wanted: 3, have: 5
ERROR: extent[4658555617280, 122880] referencer count mismatch (root: 11712, owner: 375444, offset: 1848705024) wanted: 1, have: 3
ERROR: extent[4677858123776, 417792] referencer count mismatch (root: 11712, owner: 863395, offset: 79523840) wanted: 1, have: 3
ERROR: extent[4677858123776, 417792] referencer count mismatch (root: 11930, owner: 863395, offset: 79523840) wanted: 1, have: 3
ERROR: extent[4698380947456, 409600] referencer count mismatch (root: 11930, owner: 375444, offset: 1851596800) wanted: 3, have: 4
ERROR: extent[4720470421504, 667648] referencer count mismatch (root: 11058, owner: 3463478, offset: 2334720) wanted: 2, have: 10
ERROR: extent[4783941246976, 65536] referencer count mismatch (root: 9365, owner: 24493, offset: 4562944) wanted: 2, have: 3
ERROR: extent[5077564477440, 106496] referencer count mismatch (root: 9370, owner: 1602694, offset: 734756864) wanted: 1, have: 2
ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11712, owner: 375441, offset: 910999552) wanted: 16, have: 1864
ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11276, owner: 375441, offset: 910999552) wanted: 867, have: 1865
ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11058, owner: 375441, offset: 910999552) wanted: 126, have: 1872
ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11494, owner: 375441, offset: 910999552) wanted: 866, have: 1864
ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11930, owner: 375441, offset: 910999552) wanted: 861, have: 1859
ERROR: extent[5136649891840, 66781184] referencer count mismatch (root: 11058, owner: 375442, offset: 192659456) wanted: 5, have: 19
ERROR: extent[5136879157248, 134217728] referencer count mismatch (root: 11930, owner: 375442, offset: 394543104) wanted: 10, have: 33
ERROR: extent[5137380671488, 80945152] referencer count mismatch (root: 11058, owner: 375442, offset: 875233280) wanted: 1, have: 21
ERROR: extent[5137380671488, 80945152] referencer count mismatch (root: 11930, owner: 375442, offset: 875233280) wanted: 11, have: 21
ERROR: extent[5138641395712, 524288] referencer count mismatch (root: 11494, owner: 375445, offset: 39845888) wanted: 1, have: 3
ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11712, owner: 863395, offset: 51118080) wanted: 1, have: 4
ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11276, owner: 863395, offset: 51118080) wanted: 1, have: 4
ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11494, owner: 863395, offset: 51118080) wanted: 1, have: 4
ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11276, owner: 863395, offset: 74952704) wanted: 3, have: 5
ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11058, owner: 863395, offset: 74952704) wanted: 3, have: 5
ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11494, owner: 863395, offset: 74952704) wanted: 3, have: 5
ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11930, owner: 863395, offset: 74952704) wanted: 3, have: 5
ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11712, owner: 863395, offset: 77705216) wanted: 1, have: 6
ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11276, owner: 863395, offset: 77705216) wanted: 5, have: 6
ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11058, owner: 863395, offset: 77705216) wanted: 5, have: 6
ERROR: extent[5427326599168, 61440] referencer count mismatch (root: 9370, owner: 1225712, offset: 29753344) wanted: 1, have: 5
ERROR: extent[5456623030272, 24576] referencer count mismatch (root: 11058, owner: 2278892, offset: 786432) wanted: 2, have: 3
ERROR: extent[5851251269632, 134217728] referencer count mismatch (root: 9370, owner: 1602695, offset: 534061056) wanted: 3, have: 4
ERROR: errors found in extent allocation tree or chunk allocation
cache and super generation don't match, space cache will be invalidated
ERROR: root 3857 EXTENT_DATA[108864 4096] interrupt
ERROR: root 3857 EXTENT_DATA[133050 4096] interrupt
ERROR: root 3857 EXTENT_DATA[388570 4096] interrupt
ERROR: root 3857 EXTENT_DATA[729583 4096] interrupt
ERROR: root 3857 EXTENT_DATA[984778 4096] interrupt
ERROR: root 3857 EXTENT_DATA[997394 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1002954 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1007491 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1111463 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1111506 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1111536 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1111536 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1134500 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1136498 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1175965 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1185977 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1190919 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1201340 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1230370 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1230530 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1230530 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1235960 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1248784 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1271827 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1295242 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1406074 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1410780 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1412938 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1413532 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1413532 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1421245 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1423365 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1425985 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1429229 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1430615 4096] interrupt
ERROR: root 3857 EXTENT_DATA[1443769 4096] interrupt
ERROR: root 3860 EXTENT_DATA[599089 4096] interrupt

(not finished, still going on)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Chris Murphy]

On Wed, Jun 21, 2017 at 9:13 AM, Marc MERLIN <marc@merlins.org> wrote:
> On Tue, Jun 20, 2017 at 08:43:52PM -0700, Marc MERLIN wrote:
>> On Tue, Jun 20, 2017 at 09:31:42PM -0600, Chris Murphy wrote:
>> > On Tue, Jun 20, 2017 at 5:12 PM, Marc MERLIN <marc@merlins.org> wrote:
>> >
>> > > I'm now going to remount this with nospace_cache to see if your guess about
>> > > space_cache was correct.
>> > > Other suggestions also welcome :)
>> >
>> > What results do you get with lowmem mode? It won't repair without
>> > additional patches, but might give a dev a clue what's going on. I
>> > regularly see normal mode check finds no problems, and lowmem mode
>> > finds problems. Lowmem mode is a total rewrite so it's a different
>> > implementation and can find things normal mode won't.
>>
>> Oh, I kind of forgot that lowmem mode looked for more things than regular
>> mode.
>> I will run this tonight and see what it says.
>
> It's probably still a ways from being finished given how slow lowmem is in
> comparison, but sadly it found a bunch of problems which regular mode didn't
> find.
>
> I'm pretty bummed. I just spent way too long recreating this filesystem and
> the multiple btrfs send/receive relationships from other machines. Too a bit
> over a week :(
>
> It looks like the errors are not major (especially if the regular mode
> doesn't even see them), but without lowmem --repair, I'm kind of screwed.
>
> I'm wondering if I could/should leave those errors unfixed until lowmem --repair
> finally happens, or whether I'm looking at spending another week rebuilding
> this filesystem :-/
>
>
> gargamel:~# btrfs check -p --mode lowmem  /dev/mapper/dshelf2
> Checking filesystem on /dev/mapper/dshelf2
> UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
> ERROR: extent[3886187384832, 81920] referencer count mismatch (root: 11930, owner: 375444, offset: 1851654144) wanted: 1, have: 4
> ERROR: extent[3886189391872, 122880] referencer count mismatch (root: 11712, owner: 863395, offset: 79659008) wanted: 1, have: 2
> ERROR: extent[3933249708032, 69632] referencer count mismatch (root: 12424, owner: 6945, offset: 2083389440) wanted: 1, have: 2
> ERROR: extent[3933249708032, 69632] referencer count mismatch (root: 12172, owner: 6945, offset: 2083389440) wanted: 1, have: 2
> ERROR: extent[4571729862656, 876544] referencer count mismatch (root: 11058, owner: 375442, offset: 907706368) wanted: 6, have: 21
> ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11712, owner: 375444, offset: 1848672256) wanted: 3, have: 5
> ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11276, owner: 375444, offset: 1848672256) wanted: 3, have: 5
> ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11058, owner: 375444, offset: 1848672256) wanted: 3, have: 5
> ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11494, owner: 375444, offset: 1848672256) wanted: 3, have: 5
> ERROR: extent[4658555617280, 122880] referencer count mismatch (root: 11712, owner: 375444, offset: 1848705024) wanted: 1, have: 3
> ERROR: extent[4677858123776, 417792] referencer count mismatch (root: 11712, owner: 863395, offset: 79523840) wanted: 1, have: 3
> ERROR: extent[4677858123776, 417792] referencer count mismatch (root: 11930, owner: 863395, offset: 79523840) wanted: 1, have: 3
> ERROR: extent[4698380947456, 409600] referencer count mismatch (root: 11930, owner: 375444, offset: 1851596800) wanted: 3, have: 4
> ERROR: extent[4720470421504, 667648] referencer count mismatch (root: 11058, owner: 3463478, offset: 2334720) wanted: 2, have: 10
> ERROR: extent[4783941246976, 65536] referencer count mismatch (root: 9365, owner: 24493, offset: 4562944) wanted: 2, have: 3
> ERROR: extent[5077564477440, 106496] referencer count mismatch (root: 9370, owner: 1602694, offset: 734756864) wanted: 1, have: 2
> ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11712, owner: 375441, offset: 910999552) wanted: 16, have: 1864
> ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11276, owner: 375441, offset: 910999552) wanted: 867, have: 1865
> ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11058, owner: 375441, offset: 910999552) wanted: 126, have: 1872
> ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11494, owner: 375441, offset: 910999552) wanted: 866, have: 1864
> ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11930, owner: 375441, offset: 910999552) wanted: 861, have: 1859
> ERROR: extent[5136649891840, 66781184] referencer count mismatch (root: 11058, owner: 375442, offset: 192659456) wanted: 5, have: 19
> ERROR: extent[5136879157248, 134217728] referencer count mismatch (root: 11930, owner: 375442, offset: 394543104) wanted: 10, have: 33
> ERROR: extent[5137380671488, 80945152] referencer count mismatch (root: 11058, owner: 375442, offset: 875233280) wanted: 1, have: 21
> ERROR: extent[5137380671488, 80945152] referencer count mismatch (root: 11930, owner: 375442, offset: 875233280) wanted: 11, have: 21
> ERROR: extent[5138641395712, 524288] referencer count mismatch (root: 11494, owner: 375445, offset: 39845888) wanted: 1, have: 3
> ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11712, owner: 863395, offset: 51118080) wanted: 1, have: 4
> ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11276, owner: 863395, offset: 51118080) wanted: 1, have: 4
> ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11494, owner: 863395, offset: 51118080) wanted: 1, have: 4
> ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11276, owner: 863395, offset: 74952704) wanted: 3, have: 5
> ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11058, owner: 863395, offset: 74952704) wanted: 3, have: 5
> ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11494, owner: 863395, offset: 74952704) wanted: 3, have: 5
> ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11930, owner: 863395, offset: 74952704) wanted: 3, have: 5
> ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11712, owner: 863395, offset: 77705216) wanted: 1, have: 6
> ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11276, owner: 863395, offset: 77705216) wanted: 5, have: 6
> ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11058, owner: 863395, offset: 77705216) wanted: 5, have: 6
> ERROR: extent[5427326599168, 61440] referencer count mismatch (root: 9370, owner: 1225712, offset: 29753344) wanted: 1, have: 5
> ERROR: extent[5456623030272, 24576] referencer count mismatch (root: 11058, owner: 2278892, offset: 786432) wanted: 2, have: 3
> ERROR: extent[5851251269632, 134217728] referencer count mismatch (root: 9370, owner: 1602695, offset: 534061056) wanted: 3, have: 4
> ERROR: errors found in extent allocation tree or chunk allocation
> cache and super generation don't match, space cache will be invalidated
> ERROR: root 3857 EXTENT_DATA[108864 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[133050 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[388570 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[729583 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[984778 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[997394 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1002954 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1007491 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1111463 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1111506 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1111536 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1111536 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1134500 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1136498 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1175965 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1185977 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1190919 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1201340 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1230370 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1230530 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1230530 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1235960 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1248784 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1271827 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1295242 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1406074 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1410780 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1412938 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1413532 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1413532 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1421245 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1423365 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1425985 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1429229 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1430615 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1443769 4096] interrupt
> ERROR: root 3860 EXTENT_DATA[599089 4096] interrupt
>
> (not finished, still going on)


I don't know what it means. Maybe Qu has some idea. He might want a
btrfs-image  of this file system to see if it's a bug. There are still
some bugs found with lowmem mode, so these could be bogus messages.
But the file system clearly has problems, the question is why does
such a new file system have these kinds of problems that can't be
fixed by normal repair because they aren't even being detected; or
maybe there is no problem on disk per se, the problem might be a bug.

In which case, off chance going back to a substantially older kernel
might help. Maybe the latest 4.9 series kernel?

-- 
Chris Murphy

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Marc MERLIN]

On Wed, Jun 21, 2017 at 05:22:15PM -0600, Chris Murphy wrote:
> I don't know what it means. Maybe Qu has some idea. He might want a
> btrfs-image  of this file system to see if it's a bug. There are still
> some bugs found with lowmem mode, so these could be bogus messages.
> But the file system clearly has problems, the question is why does
> such a new file system have these kinds of problems that can't be
> fixed by normal repair because they aren't even being detected; or
> maybe there is no problem on disk per se, the problem might be a bug.
 
Yes, that's indeed the question I was asking myself too :)
Now, I did have a couple of drives that got kicked out of a (mdadm, not
btrfs) raid array, causing the array to go away while btrfs was trying to
write to it, but my understanding of btrfs write journalling is that the new
data that was being written should have been discarded and I should have
ended up at the previous good state.
AFAIK, I'm pretty sure I didn't get any block layer corruption this time, I
just got a drive effectively pulled from a running array (well 2, one went
to degraded, and the 2nd one killed the array. I re-added them carefully and
correctly in the right order and mdadm rebuilt what it needed using the
extent tree)

For what it's worth, I've had no end of trouble with Sata SAS cards and their 4
sata cables in one:
https://www.amazon.com/gp/product/B0050SLTPC/ref=oh_aui_search_detailpage?ie=UTF8&psc=1
https://www.amazon.com/gp/product/B013G4EMH8/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

I have it stable now, but those cables are super sensitive and have caused
drives to get kicked out if they weren't air canned first, and plugged in
just right :-/

> In which case, off chance going back to a substantially older kernel
> might help. Maybe the latest 4.9 series kernel?

If there is reasonable evidence that it will help, I can give it a shot.

Qu, or anyone, given that btrfs-image is going to take a long time (maybe a
day or more), given that I have to use at least -s before I can share the
image, and if I need -ss, then it's even slower from what I remember.

Basically please suggest the fastest image algorithm I can use. It's a quad
core HT machine, so should I use
btrfs-image -c0 -t8 -s /dev/ image
(I'm assuing -c9 will not be faster and that -ss will be even slower)

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Qu Wenruo]

At 06/21/2017 11:13 PM, Marc MERLIN wrote:
> On Tue, Jun 20, 2017 at 08:43:52PM -0700, Marc MERLIN wrote:
>> On Tue, Jun 20, 2017 at 09:31:42PM -0600, Chris Murphy wrote:
>>> On Tue, Jun 20, 2017 at 5:12 PM, Marc MERLIN <marc@merlins.org> wrote:
>>>
>>>> I'm now going to remount this with nospace_cache to see if your guess about
>>>> space_cache was correct.
>>>> Other suggestions also welcome :)
>>>
>>> What results do you get with lowmem mode? It won't repair without
>>> additional patches, but might give a dev a clue what's going on. I
>>> regularly see normal mode check finds no problems, and lowmem mode
>>> finds problems. Lowmem mode is a total rewrite so it's a different
>>> implementation and can find things normal mode won't.
>>
>> Oh, I kind of forgot that lowmem mode looked for more things than regular
>> mode.
>> I will run this tonight and see what it says.
>   
> It's probably still a ways from being finished given how slow lowmem is in
> comparison, but sadly it found a bunch of problems which regular mode didn't
> find.
> 
> I'm pretty bummed. I just spent way too long recreating this filesystem and
> the multiple btrfs send/receive relationships from other machines. Too a bit
> over a week :(
> 
> It looks like the errors are not major (especially if the regular mode
> doesn't even see them), but without lowmem --repair, I'm kind of screwed.
> 
> I'm wondering if I could/should leave those errors unfixed until lowmem --repair
> finally happens, or whether I'm looking at spending another week rebuilding
> this filesystem :-/
> 
> 
> gargamel:~# btrfs check -p --mode lowmem  /dev/mapper/dshelf2
> Checking filesystem on /dev/mapper/dshelf2
> UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
> ERROR: extent[3886187384832, 81920] referencer count mismatch (root: 11930, owner: 375444, offset: 1851654144) wanted: 1, have: 4

This means that in extent tree, btrfs says there is only one referring 
to this extent, but lowmem mode find 4.

It would provide great help if you could dump extent tree for it.
# btrfs-debug-tree <dev> | grep -C 10 3886187384832


> ERROR: extent[3886189391872, 122880] referencer count mismatch (root: 11712, owner: 863395, offset: 79659008) wanted: 1, have: 2
> ERROR: extent[3933249708032, 69632] referencer count mismatch (root: 12424, owner: 6945, offset: 2083389440) wanted: 1, have: 2
> ERROR: extent[3933249708032, 69632] referencer count mismatch (root: 12172, owner: 6945, offset: 2083389440) wanted: 1, have: 2
> ERROR: extent[4571729862656, 876544] referencer count mismatch (root: 11058, owner: 375442, offset: 907706368) wanted: 6, have: 21
> ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11712, owner: 375444, offset: 1848672256) wanted: 3, have: 5
> ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11276, owner: 375444, offset: 1848672256) wanted: 3, have: 5
> ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11058, owner: 375444, offset: 1848672256) wanted: 3, have: 5
> ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11494, owner: 375444, offset: 1848672256) wanted: 3, have: 5
> ERROR: extent[4658555617280, 122880] referencer count mismatch (root: 11712, owner: 375444, offset: 1848705024) wanted: 1, have: 3
> ERROR: extent[4677858123776, 417792] referencer count mismatch (root: 11712, owner: 863395, offset: 79523840) wanted: 1, have: 3
> ERROR: extent[4677858123776, 417792] referencer count mismatch (root: 11930, owner: 863395, offset: 79523840) wanted: 1, have: 3
> ERROR: extent[4698380947456, 409600] referencer count mismatch (root: 11930, owner: 375444, offset: 1851596800) wanted: 3, have: 4
> ERROR: extent[4720470421504, 667648] referencer count mismatch (root: 11058, owner: 3463478, offset: 2334720) wanted: 2, have: 10
> ERROR: extent[4783941246976, 65536] referencer count mismatch (root: 9365, owner: 24493, offset: 4562944) wanted: 2, have: 3
> ERROR: extent[5077564477440, 106496] referencer count mismatch (root: 9370, owner: 1602694, offset: 734756864) wanted: 1, have: 2
> ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11712, owner: 375441, offset: 910999552) wanted: 16, have: 1864
> ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11276, owner: 375441, offset: 910999552) wanted: 867, have: 1865
> ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11058, owner: 375441, offset: 910999552) wanted: 126, have: 1872
> ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11494, owner: 375441, offset: 910999552) wanted: 866, have: 1864
> ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11930, owner: 375441, offset: 910999552) wanted: 861, have: 1859
> ERROR: extent[5136649891840, 66781184] referencer count mismatch (root: 11058, owner: 375442, offset: 192659456) wanted: 5, have: 19
> ERROR: extent[5136879157248, 134217728] referencer count mismatch (root: 11930, owner: 375442, offset: 394543104) wanted: 10, have: 33
> ERROR: extent[5137380671488, 80945152] referencer count mismatch (root: 11058, owner: 375442, offset: 875233280) wanted: 1, have: 21
> ERROR: extent[5137380671488, 80945152] referencer count mismatch (root: 11930, owner: 375442, offset: 875233280) wanted: 11, have: 21
> ERROR: extent[5138641395712, 524288] referencer count mismatch (root: 11494, owner: 375445, offset: 39845888) wanted: 1, have: 3
> ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11712, owner: 863395, offset: 51118080) wanted: 1, have: 4
> ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11276, owner: 863395, offset: 51118080) wanted: 1, have: 4
> ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11494, owner: 863395, offset: 51118080) wanted: 1, have: 4
> ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11276, owner: 863395, offset: 74952704) wanted: 3, have: 5
> ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11058, owner: 863395, offset: 74952704) wanted: 3, have: 5
> ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11494, owner: 863395, offset: 74952704) wanted: 3, have: 5
> ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11930, owner: 863395, offset: 74952704) wanted: 3, have: 5
> ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11712, owner: 863395, offset: 77705216) wanted: 1, have: 6
> ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11276, owner: 863395, offset: 77705216) wanted: 5, have: 6
> ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11058, owner: 863395, offset: 77705216) wanted: 5, have: 6
> ERROR: extent[5427326599168, 61440] referencer count mismatch (root: 9370, owner: 1225712, offset: 29753344) wanted: 1, have: 5
> ERROR: extent[5456623030272, 24576] referencer count mismatch (root: 11058, owner: 2278892, offset: 786432) wanted: 2, have: 3
> ERROR: extent[5851251269632, 134217728] referencer count mismatch (root: 9370, owner: 1602695, offset: 534061056) wanted: 3, have: 4
> ERROR: errors found in extent allocation tree or chunk allocation
> cache and super generation don't match, space cache will be invalidated
> ERROR: root 3857 EXTENT_DATA[108864 4096] interrupt

This means that, for root 3857, inode 108864, file offset 4096, there is 
a gap before that extent.
In NO_HOLES mode it's allowed, but if NO_HOLES incompat flag is not set, 
this should be a problem.

I wonder if this is a problem caused by inlined compressed file extent.

This can also be dumped by the following command.
# btrfs-debug-tree -t 3857 <dev> | grep -C 10 108864

Thanks,
Qu

> ERROR: root 3857 EXTENT_DATA[133050 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[388570 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[729583 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[984778 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[997394 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1002954 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1007491 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1111463 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1111506 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1111536 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1111536 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1134500 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1136498 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1175965 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1185977 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1190919 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1201340 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1230370 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1230530 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1230530 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1235960 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1248784 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1271827 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1295242 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1406074 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1410780 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1412938 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1413532 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1413532 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1421245 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1423365 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1425985 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1429229 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1430615 4096] interrupt
> ERROR: root 3857 EXTENT_DATA[1443769 4096] interrupt
> ERROR: root 3860 EXTENT_DATA[599089 4096] interrupt
> 
> (not finished, still going on)
> 
> Marc
> 

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Marc MERLIN]

[-- Attachment #1: Type: text/plain, Size: 3830 bytes --]

Ok, first it finished (almost 24H)

(...)
ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt
ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt
ERROR: root 3864 EXTENT_DATA[109336 4096] interrupt
ERROR: errors found in fs roots
found 5544779108352 bytes used, error(s) found
total csum bytes: 5344523140
total tree bytes: 71323041792
total fs tree bytes: 59288403968
total extent tree bytes: 5378260992
btree space waste bytes: 10912166856
file data blocks allocated: 7830914256896
 referenced 6244104495104

Thanks for your reply Qu

On Thu, Jun 22, 2017 at 10:22:57AM +0800, Qu Wenruo wrote:
> >gargamel:~# btrfs check -p --mode lowmem  /dev/mapper/dshelf2
> >Checking filesystem on /dev/mapper/dshelf2
> >UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
> >ERROR: extent[3886187384832, 81920] referencer count mismatch (root: 
> >11930, owner: 375444, offset: 1851654144) wanted: 1, have: 4
> 
> This means that in extent tree, btrfs says there is only one referring 
> to this extent, but lowmem mode find 4.
> 
> It would provide great help if you could dump extent tree for it.
> # btrfs-debug-tree <dev> | grep -C 10 3886187384832
 
                extent data backref root 11712 objectid 375444 offset 1851572224 count 1
                extent data backref root 11276 objectid 375444 offset 1851572224 count 1
                extent data backref root 11058 objectid 375444 offset 1851572224 count 1
                extent data backref root 11494 objectid 375444 offset 1851572224 count 1
        item 37 key (3886187352064 EXTENT_ITEM 32768) itemoff 11381 itemsize 140
                extent refs 4 gen 32382 flags DATA
                extent data backref root 11712 objectid 375444 offset 1851596800 count 1
                extent data backref root 11276 objectid 375444 offset 1851596800 count 1
                extent data backref root 11058 objectid 375444 offset 1851596800 count 1
                extent data backref root 11494 objectid 375444 offset 1851596800 count 1
        item 38 key (3886187384832 EXTENT_ITEM 81920) itemoff 11212 itemsize 169
                extent refs 16 gen 32382 flags DATA
                extent data backref root 11712 objectid 375444 offset 1851654144 count 4
                extent data backref root 11276 objectid 375444 offset 1851654144 count 4
                extent data backref root 11058 objectid 375444 offset 1851654144 count 3
                extent data backref root 11494 objectid 375444 offset 1851654144 count 4
                extent data backref root 11930 objectid 375444 offset 1851654144 count 1
        item 39 key (3886187466752 EXTENT_ITEM 16384) itemoff 11043 itemsize 169
                extent refs 5 gen 32382 flags DATA
                extent data backref root 11712 objectid 375444 offset 1851744256 count 1
                extent data backref root 11276 objectid 375444 offset 1851744256 count 1

 
> >ERROR: errors found in extent allocation tree or chunk allocation
> >cache and super generation don't match, space cache will be invalidated
> >ERROR: root 3857 EXTENT_DATA[108864 4096] interrupt
> 
> This means that, for root 3857, inode 108864, file offset 4096, there is 
> a gap before that extent.
> In NO_HOLES mode it's allowed, but if NO_HOLES incompat flag is not set, 
> this should be a problem.
> 
> I wonder if this is a problem caused by inlined compressed file extent.
> 
> This can also be dumped by the following command.
> # btrfs-debug-tree -t 3857 <dev> | grep -C 10 108864

This one is much bigger (192KB), I've bzipped and attached it.

Thanks for having a look, I appreciate it.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

[-- Attachment #2: out.bz2 --]
[-- Type: application/octet-stream, Size: 23826 bytes --]

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Qu Wenruo]

At 06/22/2017 10:53 AM, Marc MERLIN wrote:
> Ok, first it finished (almost 24H)
> 
> (...)
> ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt
> ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt
> ERROR: root 3864 EXTENT_DATA[109336 4096] interrupt
> ERROR: errors found in fs roots
> found 5544779108352 bytes used, error(s) found
> total csum bytes: 5344523140
> total tree bytes: 71323041792
> total fs tree bytes: 59288403968
> total extent tree bytes: 5378260992
> btree space waste bytes: 10912166856
> file data blocks allocated: 7830914256896
>   referenced 6244104495104
> 
> Thanks for your reply Qu
> 
> On Thu, Jun 22, 2017 at 10:22:57AM +0800, Qu Wenruo wrote:
>>> gargamel:~# btrfs check -p --mode lowmem  /dev/mapper/dshelf2
>>> Checking filesystem on /dev/mapper/dshelf2
>>> UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
>>> ERROR: extent[3886187384832, 81920] referencer count mismatch (root:
>>> 11930, owner: 375444, offset: 1851654144) wanted: 1, have: 4
>>
>> This means that in extent tree, btrfs says there is only one referring
>> to this extent, but lowmem mode find 4.
>>
>> It would provide great help if you could dump extent tree for it.
>> # btrfs-debug-tree <dev> | grep -C 10 3886187384832
>   
>                  extent data backref root 11712 objectid 375444 offset 1851572224 count 1
>                  extent data backref root 11276 objectid 375444 offset 1851572224 count 1
>                  extent data backref root 11058 objectid 375444 offset 1851572224 count 1
>                  extent data backref root 11494 objectid 375444 offset 1851572224 count 1
>          item 37 key (3886187352064 EXTENT_ITEM 32768) itemoff 11381 itemsize 140
>                  extent refs 4 gen 32382 flags DATA
>                  extent data backref root 11712 objectid 375444 offset 1851596800 count 1
>                  extent data backref root 11276 objectid 375444 offset 1851596800 count 1
>                  extent data backref root 11058 objectid 375444 offset 1851596800 count 1
>                  extent data backref root 11494 objectid 375444 offset 1851596800 count 1
>          item 38 key (3886187384832 EXTENT_ITEM 81920) itemoff 11212 itemsize 169
>                  extent refs 16 gen 32382 flags DATA
>                  extent data backref root 11712 objectid 375444 offset 1851654144 count 4
>                  extent data backref root 11276 objectid 375444 offset 1851654144 count 4
>                  extent data backref root 11058 objectid 375444 offset 1851654144 count 3
>                  extent data backref root 11494 objectid 375444 offset 1851654144 count 4
>                  extent data backref root 11930 objectid 375444 offset 1851654144 count 1
>          item 39 key (3886187466752 EXTENT_ITEM 16384) itemoff 11043 itemsize 169
>                  extent refs 5 gen 32382 flags DATA
>                  extent data backref root 11712 objectid 375444 offset 1851744256 count 1
>                  extent data backref root 11276 objectid 375444 offset 1851744256 count 1

Well, there is only the output from extent tree.

I was also expecting output from subvolue (11930) tree.

It could be done by
# btrfs-debug-tree -t 11930 | grep -C 10 3886187384832

But please pay attention that, this dump may contain filenames, feel 
free to mask the filenames.

Thanks,
Qu

> 
>   
>>> ERROR: errors found in extent allocation tree or chunk allocation
>>> cache and super generation don't match, space cache will be invalidated
>>> ERROR: root 3857 EXTENT_DATA[108864 4096] interrupt
>>
>> This means that, for root 3857, inode 108864, file offset 4096, there is
>> a gap before that extent.
>> In NO_HOLES mode it's allowed, but if NO_HOLES incompat flag is not set,
>> this should be a problem.
>>
>> I wonder if this is a problem caused by inlined compressed file extent.
>>
>> This can also be dumped by the following command.
>> # btrfs-debug-tree -t 3857 <dev> | grep -C 10 108864
> 
> This one is much bigger (192KB), I've bzipped and attached it.

Thanks for this one.
And it is caused by inlined compressed extent.

Lu Fengqi will send patch fixing it.

Thanks,
Qu

> 
> Thanks for having a look, I appreciate it.
> 
> Marc
> 

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Qu Wenruo]

At 06/22/2017 10:53 AM, Marc MERLIN wrote:
> Ok, first it finished (almost 24H)
> 
> (...)
> ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt
> ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt
> ERROR: root 3864 EXTENT_DATA[109336 4096] interrupt
> ERROR: errors found in fs roots
> found 5544779108352 bytes used, error(s) found
> total csum bytes: 5344523140
> total tree bytes: 71323041792
> total fs tree bytes: 59288403968
> total extent tree bytes: 5378260992
> btree space waste bytes: 10912166856
> file data blocks allocated: 7830914256896
>   referenced 6244104495104
> 
> Thanks for your reply Qu
> 
> On Thu, Jun 22, 2017 at 10:22:57AM +0800, Qu Wenruo wrote:
>>> gargamel:~# btrfs check -p --mode lowmem  /dev/mapper/dshelf2
>>> Checking filesystem on /dev/mapper/dshelf2
>>> UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
>>> ERROR: extent[3886187384832, 81920] referencer count mismatch (root:
>>> 11930, owner: 375444, offset: 1851654144) wanted: 1, have: 4
>>
>> This means that in extent tree, btrfs says there is only one referring
>> to this extent, but lowmem mode find 4.
>>
>> It would provide great help if you could dump extent tree for it.
>> # btrfs-debug-tree <dev> | grep -C 10 3886187384832
>   
>                  extent data backref root 11712 objectid 375444 offset 1851572224 count 1
>                  extent data backref root 11276 objectid 375444 offset 1851572224 count 1
>                  extent data backref root 11058 objectid 375444 offset 1851572224 count 1
>                  extent data backref root 11494 objectid 375444 offset 1851572224 count 1
>          item 37 key (3886187352064 EXTENT_ITEM 32768) itemoff 11381 itemsize 140
>                  extent refs 4 gen 32382 flags DATA
>                  extent data backref root 11712 objectid 375444 offset 1851596800 count 1
>                  extent data backref root 11276 objectid 375444 offset 1851596800 count 1
>                  extent data backref root 11058 objectid 375444 offset 1851596800 count 1
>                  extent data backref root 11494 objectid 375444 offset 1851596800 count 1
>          item 38 key (3886187384832 EXTENT_ITEM 81920) itemoff 11212 itemsize 169
>                  extent refs 16 gen 32382 flags DATA
>                  extent data backref root 11712 objectid 375444 offset 1851654144 count 4
>                  extent data backref root 11276 objectid 375444 offset 1851654144 count 4
>                  extent data backref root 11058 objectid 375444 offset 1851654144 count 3
>                  extent data backref root 11494 objectid 375444 offset 1851654144 count 4
>                  extent data backref root 11930 objectid 375444 offset 1851654144 count 1
>          item 39 key (3886187466752 EXTENT_ITEM 16384) itemoff 11043 itemsize 169
>                  extent refs 5 gen 32382 flags DATA
>                  extent data backref root 11712 objectid 375444 offset 1851744256 count 1
>                  extent data backref root 11276 objectid 375444 offset 1851744256 count 1

Well, there is only the output from extent tree.

I was also expecting output from subvolue (11930) tree.

It could be done by
# btrfs-debug-tree -t 11930 | grep -C 10 3886187384832

But please pay attention that, this dump may contain filenames, feel 
free to mask the filenames.

> 
>   
>>> ERROR: errors found in extent allocation tree or chunk allocation
>>> cache and super generation don't match, space cache will be invalidated
>>> ERROR: root 3857 EXTENT_DATA[108864 4096] interrupt
>>
>> This means that, for root 3857, inode 108864, file offset 4096, there is
>> a gap before that extent.
>> In NO_HOLES mode it's allowed, but if NO_HOLES incompat flag is not set,
>> this should be a problem.
>>
>> I wonder if this is a problem caused by inlined compressed file extent.
>>
>> This can also be dumped by the following command.
>> # btrfs-debug-tree -t 3857 <dev> | grep -C 10 108864
> 
> This one is much bigger (192KB), I've bzipped and attached it.

Thanks for this one.
And it is caused by inlined compressed extent.

Lu Fengqi will send patch fixing it.

Thanks,
Qu

> 
> Thanks for having a look, I appreciate it.
> 
> Marc
> 

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Marc MERLIN]

On Thu, Jun 22, 2017 at 12:08:44PM +0800, Qu Wenruo wrote:
> > On Thu, Jun 22, 2017 at 10:22:57AM +0800, Qu Wenruo wrote:
> > > > gargamel:~# btrfs check -p --mode lowmem  /dev/mapper/dshelf2
> > > > Checking filesystem on /dev/mapper/dshelf2
> > > > UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
> > > > ERROR: extent[3886187384832, 81920] referencer count mismatch (root:
> > > > 11930, owner: 375444, offset: 1851654144) wanted: 1, have: 4
> > > 
> > > This means that in extent tree, btrfs says there is only one referring
> > > to this extent, but lowmem mode find 4.
> > > 
> > > It would provide great help if you could dump extent tree for it.
> > > # btrfs-debug-tree <dev> | grep -C 10 3886187384832
> >                  extent data backref root 11712 objectid 375444 offset 1851572224 count 1
> >                  extent data backref root 11276 objectid 375444 offset 1851572224 count 1
> >                  extent data backref root 11058 objectid 375444 offset 1851572224 count 1
> >                  extent data backref root 11494 objectid 375444 offset 1851572224 count 1
> >          item 37 key (3886187352064 EXTENT_ITEM 32768) itemoff 11381 itemsize 140
> >                  extent refs 4 gen 32382 flags DATA
> >                  extent data backref root 11712 objectid 375444 offset 1851596800 count 1
> >                  extent data backref root 11276 objectid 375444 offset 1851596800 count 1
> >                  extent data backref root 11058 objectid 375444 offset 1851596800 count 1
> >                  extent data backref root 11494 objectid 375444 offset 1851596800 count 1
> >          item 38 key (3886187384832 EXTENT_ITEM 81920) itemoff 11212 itemsize 169
> >                  extent refs 16 gen 32382 flags DATA
> >                  extent data backref root 11712 objectid 375444 offset 1851654144 count 4
> >                  extent data backref root 11276 objectid 375444 offset 1851654144 count 4
> >                  extent data backref root 11058 objectid 375444 offset 1851654144 count 3
> >                  extent data backref root 11494 objectid 375444 offset 1851654144 count 4
> >                  extent data backref root 11930 objectid 375444 offset 1851654144 count 1
> >          item 39 key (3886187466752 EXTENT_ITEM 16384) itemoff 11043 itemsize 169
> >                  extent refs 5 gen 32382 flags DATA
> >                  extent data backref root 11712 objectid 375444 offset 1851744256 count 1
> >                  extent data backref root 11276 objectid 375444 offset 1851744256 count 1
> 
> Well, there is only the output from extent tree.
> 
> I was also expecting output from subvolue (11930) tree.
> 
> It could be done by
> # btrfs-debug-tree -t 11930 | grep -C 10 3886187384832
> 
> But please pay attention that, this dump may contain filenames, feel free to
> mask the filenames.
 
There you go:
gargamel:~# btrfs-debug-tree /dev/mapper/dsh | grep -C 10 3886187384832  
dshelf1@ dshelf2@  
                extent compression 0 (none)
        item 201 key (375444 EXTENT_DATA 1851654144) itemoff 5577 itemsize 53
                extent data disk byte 5613689888768 nr 8192
                extent data offset 0 nr 8192 ram 8192
                extent compression 0 (none)
        item 3 key (375444 EXTENT_DATA 1851744256) itemoff 16071 itemsize 53
                generation 32382 type 1 (regular)
                extent data disk byte 5613689888768 nr 8192
                extent data offset 0 nr 8192 ram 8192
                extent compression 0 (none)
        item 201 key (375444 EXTENT_DATA 1851654144) itemoff 5577 itemsize 53
                generation 32961 type 1 (regular)
                extent data disk byte 4686293291008 nr 16384
                extent data offset 0 nr 16384 ram 16384
                extent compression 0 (none)
        item 202 key (375444 EXTENT_DATA 1851670528) itemoff 5524 itemsize 53
                generation 32382 type 1 (regular)
                extent data disk byte 3886187384832 nr 81920
                extent data offset 16384 nr 8192 ram 81920
                extent compression 0 (none)
        item 203 key (375444 EXTENT_DATA 1851678720) itemoff 5471 itemsize 53
                generation 33534 type 1 (regular)
                extent data disk byte 5540480962560 nr 8192
                extent data offset 0 nr 8192 ram 8192
                extent compression 0 (none)
        item 204 key (375444 EXTENT_DATA 1851686912) itemoff 5418 itemsize 53
                generation 32961 type 1 (regular)
                extent data disk byte 4686293307392 nr 16384
                extent data offset 8192 nr 8192 ram 16384
                extent compression 0 (none)
        item 205 key (375444 EXTENT_DATA 1851695104) itemoff 5365 itemsize 53
                generation 32382 type 1 (regular)
                extent data disk byte 3886187384832 nr 81920
                extent data offset 40960 nr 8192 ram 81920
                extent compression 0 (none)
        item 206 key (375444 EXTENT_DATA 1851703296) itemoff 5312 itemsize 53
                generation 32961 type 1 (regular)
                extent data disk byte 4686293323776 nr 8192
                extent data offset 0 nr 8192 ram 8192
                extent compression 0 (none)
        item 207 key (375444 EXTENT_DATA 1851711488) itemoff 5259 itemsize 53
                generation 32382 type 1 (regular)
                extent data disk byte 3886187384832 nr 81920
                extent data offset 57344 nr 8192 ram 81920
                extent compression 0 (none)
leaf 5715801047040 items 105 free space 8093 generation 36595 owner 11930
fs uuid 85441c59-ad11-4b25-b1fe-974f9e4acede
chunk uuid ed705b7b-2fa6-43f6-a4a1-941c8463ee68
        item 0 key (375444 EXTENT_DATA 1851719680) itemoff 16230 itemsize 53
                generation 34868 type 1 (regular)
                extent data disk byte 5591266127872 nr 8192
                extent data offset 0 nr 8192 ram 8192
                extent compression 0 (none)
        item 1 key (375444 EXTENT_DATA 1851727872) itemoff 16177 itemsize 53
                generation 32382 type 1 (regular)
                extent data disk byte 3886187384832 nr 81920
                extent data offset 73728 nr 8192 ram 81920
                extent compression 0 (none)
        item 2 key (375444 EXTENT_DATA 1851736064) itemoff 16124 itemsize 53
                generation 31782 type 1 (regular)
                extent data disk byte 5922189430784 nr 106496
                extent data offset 81920 nr 8192 ram 106496
                extent compression 0 (none)
        item 3 key (375444 EXTENT_DATA 1851744256) itemoff 16071 itemsize 53
                generation 32382 type 1 (regular)
                extent data disk byte 3886187466752 nr 16384


> Thanks for this one.
> And it is caused by inlined compressed extent.
> 
> Lu Fengqi will send patch fixing it.

I got the patch and will test it, thank you.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Lu Fengqi]

On 2017年06月23日 12:06, Marc MERLIN wrote:
>> Well, there is only the output from extent tree.
>>
>> I was also expecting output from subvolue (11930) tree.
>>
>> It could be done by
>> # btrfs-debug-tree -t 11930 | grep -C 10 3886187384832
>>
I apologize if this was not made clear.

>> But please pay attention that, this dump may contain filenames, feel free to
>> mask the filenames.
>   
> There you go:
> gargamel:~# btrfs-debug-tree /dev/mapper/dsh | grep -C 10 3886187384832

Could you dump file tree (11930) by the following command.
# btrfs-debug-tree -t 11930 /dev/mapper/dsh | grep -C 10 3886187384832

I wonder if this extent was referenced by this file tree four times.

Hoping that this will not cause you too much trouble.


-- 
Thanks,
Lu

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Marc MERLIN]

On Fri, Jun 23, 2017 at 04:54:01PM +0800, Lu Fengqi wrote:
> On 2017年06月23日 12:06, Marc MERLIN wrote:
> > > Well, there is only the output from extent tree.
> > > 
> > > I was also expecting output from subvolue (11930) tree.
> > > 
> > > It could be done by
> > > # btrfs-debug-tree -t 11930 | grep -C 10 3886187384832
> > > 
> I apologize if this was not made clear.
> 
> > > But please pay attention that, this dump may contain filenames, feel free to
> > > mask the filenames.
> > There you go:
> > gargamel:~# btrfs-debug-tree /dev/mapper/dsh | grep -C 10 3886187384832
> 
> Could you dump file tree (11930) by the following command.
> # btrfs-debug-tree -t 11930 /dev/mapper/dsh | grep -C 10 3886187384832

Sure thing, there you go:
		extent data disk byte 5613689888768 nr 8192
		extent data offset 0 nr 8192 ram 8192
		extent compression 0 (none)
	item 201 key (375444 EXTENT_DATA 1851654144) itemoff 5577 itemsize 53
		generation 32961 type 1 (regular)
		extent data disk byte 4686293291008 nr 16384
		extent data offset 0 nr 16384 ram 16384
		extent compression 0 (none)
	item 202 key (375444 EXTENT_DATA 1851670528) itemoff 5524 itemsize 53
		generation 32382 type 1 (regular)
		extent data disk byte 3886187384832 nr 81920
		extent data offset 16384 nr 8192 ram 81920
		extent compression 0 (none)
	item 203 key (375444 EXTENT_DATA 1851678720) itemoff 5471 itemsize 53
		generation 33534 type 1 (regular)
		extent data disk byte 5540480962560 nr 8192
		extent data offset 0 nr 8192 ram 8192
		extent compression 0 (none)
	item 204 key (375444 EXTENT_DATA 1851686912) itemoff 5418 itemsize 53
		generation 32961 type 1 (regular)
		extent data disk byte 4686293307392 nr 16384
		extent data offset 8192 nr 8192 ram 16384
		extent compression 0 (none)
	item 205 key (375444 EXTENT_DATA 1851695104) itemoff 5365 itemsize 53
		generation 32382 type 1 (regular)
		extent data disk byte 3886187384832 nr 81920
		extent data offset 40960 nr 8192 ram 81920
		extent compression 0 (none)
	item 206 key (375444 EXTENT_DATA 1851703296) itemoff 5312 itemsize 53
		generation 32961 type 1 (regular)
		extent data disk byte 4686293323776 nr 8192
		extent data offset 0 nr 8192 ram 8192
		extent compression 0 (none)
	item 207 key (375444 EXTENT_DATA 1851711488) itemoff 5259 itemsize 53
		generation 32382 type 1 (regular)
		extent data disk byte 3886187384832 nr 81920
		extent data offset 57344 nr 8192 ram 81920
		extent compression 0 (none)
leaf 5715801047040 items 105 free space 8093 generation 36595 owner 11930
leaf 5715801047040 flags 0x1(WRITTEN) backref revision 1
fs uuid 85441c59-ad11-4b25-b1fe-974f9e4acede
chunk uuid ed705b7b-2fa6-43f6-a4a1-941c8463ee68
	item 0 key (375444 EXTENT_DATA 1851719680) itemoff 16230 itemsize 53
		generation 34868 type 1 (regular)
		extent data disk byte 5591266127872 nr 8192
		extent data offset 0 nr 8192 ram 8192
		extent compression 0 (none)
	item 1 key (375444 EXTENT_DATA 1851727872) itemoff 16177 itemsize 53
		generation 32382 type 1 (regular)
		extent data disk byte 3886187384832 nr 81920
		extent data offset 73728 nr 8192 ram 81920
		extent compression 0 (none)
	item 2 key (375444 EXTENT_DATA 1851736064) itemoff 16124 itemsize 53
		generation 31782 type 1 (regular)
		extent data disk byte 5922189430784 nr 106496
		extent data offset 81920 nr 8192 ram 106496
		extent compression 0 (none)
	item 3 key (375444 EXTENT_DATA 1851744256) itemoff 16071 itemsize 53
		generation 32382 type 1 (regular)
		extent data disk byte 3886187466752 nr 16384


Thanks for looking at this.
I have applied your patch and I'm still re-running check in lowmem. It takes about 24H so I'll
post the full results when it's done.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Marc MERLIN]

On Fri, Jun 23, 2017 at 09:17:50AM -0700, Marc MERLIN wrote:
> Thanks for looking at this.
> I have applied your patch and I'm still re-running check in lowmem. It takes about 24H so I'll
> post the full results when it's done.

Ok, here is the output of the check with btrfs-progs freshly synced from
git, including Lu's just added patch.

Obviously while I'm happy to give further debug info on why my filesystem is in that state and
while check --repair sees nothing to repair, suggestions on how to clean those warnings up, unless they are not going to affect filesystem operation, would be greatly appreciated :)

Thanks,
Marc


Checking filesystem on /dev/mapper/dshelf2
UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
ERROR: extent[3886187384832, 81920] referencer count mismatch (root: 11930, owner: 375444, offset: 1851654144) wanted: 1, have: 4
ERROR: extent[3886189391872, 122880] referencer count mismatch (root: 11712, owner: 863395, offset: 79659008) wanted: 1, have: 2
ERROR: extent[3886189391872, 122880] referencer count mismatch (root: 11494, owner: 863395, offset: 79659008) wanted: 1, have: 2
ERROR: extent[3886189391872, 122880] referencer count mismatch (root: 11930, owner: 863395, offset: 79659008) wanted: 1, have: 2
ERROR: extent[3933249708032, 69632] referencer count mismatch (root: 12424, owner: 6945, offset: 2083389440) wanted: 1, have: 2
ERROR: extent[3933249708032, 69632] referencer count mismatch (root: 12172, owner: 6945, offset: 2083389440) wanted: 1, have: 2
ERROR: extent[4571729862656, 876544] referencer count mismatch (root: 11058, owner: 375442, offset: 907706368) wanted: 6, have: 21
ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11712, owner: 375444, offset: 1848672256) wanted: 3, have: 5
ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11276, owner: 375444, offset: 1848672256) wanted: 3, have: 5
ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11058, owner: 375444, offset: 1848672256) wanted: 3, have: 5
ERROR: extent[4641490833408, 270336] referencer count mismatch (root: 11494, owner: 375444, offset: 1848672256) wanted: 3, have: 5
ERROR: extent[4658555617280, 122880] referencer count mismatch (root: 11712, owner: 375444, offset: 1848705024) wanted: 1, have: 3
ERROR: extent[4658555617280, 122880] referencer count mismatch (root: 11276, owner: 375444, offset: 1848705024) wanted: 1, have: 3
ERROR: extent[4658555617280, 122880] referencer count mismatch (root: 11058, owner: 375444, offset: 1848705024) wanted: 1, have: 3
ERROR: extent[4658555617280, 122880] referencer count mismatch (root: 11494, owner: 375444, offset: 1848705024) wanted: 1, have: 3
ERROR: extent[4677858123776, 417792] referencer count mismatch (root: 11712, owner: 863395, offset: 79523840) wanted: 1, have: 3
ERROR: extent[4677858123776, 417792] referencer count mismatch (root: 11494, owner: 863395, offset: 79523840) wanted: 1, have: 3
ERROR: extent[4677858123776, 417792] referencer count mismatch (root: 11930, owner: 863395, offset: 79523840) wanted: 1, have: 3
ERROR: extent[4698380947456, 409600] referencer count mismatch (root: 11930, owner: 375444, offset: 1851596800) wanted: 3, have: 4
ERROR: extent[4720470421504, 667648] referencer count mismatch (root: 11058, owner: 3463478, offset: 2334720) wanted: 2, have: 10
ERROR: extent[4783941246976, 65536] referencer count mismatch (root: 9365, owner: 24493, offset: 4562944) wanted: 2, have: 3
ERROR: extent[5077564477440, 106496] referencer count mismatch (root: 9370, owner: 1602694, offset: 734756864) wanted: 1, have: 2
ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11712, owner: 375441, offset: 910999552) wanted: 16, have: 1864
ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11276, owner: 375441, offset: 910999552) wanted: 867, have: 1865
ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11058, owner: 375441, offset: 910999552) wanted: 126, have: 1872
ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11494, owner: 375441, offset: 910999552) wanted: 866, have: 1864
ERROR: extent[5136306929664, 131489792] referencer count mismatch (root: 11930, owner: 375441, offset: 910999552) wanted: 861, have: 1859
ERROR: extent[5136649891840, 66781184] referencer count mismatch (root: 11058, owner: 375442, offset: 192659456) wanted: 5, have: 19
ERROR: extent[5136879157248, 134217728] referencer count mismatch (root: 11930, owner: 375442, offset: 394543104) wanted: 10, have: 33
ERROR: extent[5137380671488, 80945152] referencer count mismatch (root: 11058, owner: 375442, offset: 875233280) wanted: 1, have: 21
ERROR: extent[5137380671488, 80945152] referencer count mismatch (root: 11930, owner: 375442, offset: 875233280) wanted: 11, have: 21
ERROR: extent[5138641395712, 524288] referencer count mismatch (root: 11494, owner: 375445, offset: 39845888) wanted: 1, have: 3
ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11712, owner: 863395, offset: 51118080) wanted: 1, have: 4
ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11276, owner: 863395, offset: 51118080) wanted: 1, have: 4
ERROR: extent[5190245990400, 53248] referencer count mismatch (root: 11494, owner: 863395, offset: 51118080) wanted: 1, have: 4
ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11276, owner: 863395, offset: 74952704) wanted: 3, have: 5
ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11058, owner: 863395, offset: 74952704) wanted: 3, have: 5
ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11494, owner: 863395, offset: 74952704) wanted: 3, have: 5
ERROR: extent[5190274174976, 77824] referencer count mismatch (root: 11930, owner: 863395, offset: 74952704) wanted: 3, have: 5
ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11712, owner: 863395, offset: 77705216) wanted: 1, have: 6
ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11276, owner: 863395, offset: 77705216) wanted: 5, have: 6
ERROR: extent[5190275895296, 77824] referencer count mismatch (root: 11058, owner: 863395, offset: 77705216) wanted: 5, have: 6
ERROR: extent[5427326599168, 61440] referencer count mismatch (root: 9370, owner: 1225712, offset: 29753344) wanted: 1, have: 5
ERROR: extent[5456623030272, 24576] referencer count mismatch (root: 11058, owner: 2278892, offset: 786432) wanted: 2, have: 3
ERROR: extent[5851251269632, 134217728] referencer count mismatch (root: 9370, owner: 1602695, offset: 534061056) wanted: 3, have: 4
ERROR: errors found in extent allocation tree or chunk allocation
cache and super generation don't match, space cache will be invalidated
ERROR: root 3862 EXTENT_DATA[18170706 4096] interrupt
ERROR: root 3862 EXTENT_DATA[18170706 16384] interrupt
ERROR: root 3862 EXTENT_DATA[18170706 20480] interrupt
ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt
ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt
ERROR: errors found in fs roots
found 5544779108352 bytes used, error(s) found
total csum bytes: 5344523140
total tree bytes: 71323041792
total fs tree bytes: 59288403968
total extent tree bytes: 5378260992
btree space waste bytes: 10912166856
file data blocks allocated: 7830914256896
 referenced 6244104495104

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Lu Fengqi]

On 2017年06月24日 10:34, Marc MERLIN wrote:
> On Fri, Jun 23, 2017 at 09:17:50AM -0700, Marc MERLIN wrote:
>> Thanks for looking at this.
>> I have applied your patch and I'm still re-running check in lowmem. It takes about 24H so I'll
>> post the full results when it's done.
> 
> Ok, here is the output of the check with btrfs-progs freshly synced from
> git, including Lu's just added patch.
> 
> Obviously while I'm happy to give further debug info on why my filesystem is in that state and
> while check --repair sees nothing to repair, suggestions on how to clean those warnings up, unless they are not going to affect filesystem operation, would be greatly appreciated :)
> 
> Thanks,
> Marc

Thanks for the updated information. I'm sorry that the false alert make 
you feel nervous.

> 
> ERROR: root 3862 EXTENT_DATA[18170706 4096] interrupt
> ERROR: root 3862 EXTENT_DATA[18170706 16384] interrupt
> ERROR: root 3862 EXTENT_DATA[18170706 20480] interrupt
> ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt
> ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt
> ERROR: errors found in fs roots

However, this looks like another problem. Could you dump this file tree 
by the following command?
# btrfs-debug-tree -t 3862 <dev> | grep -C 10 18170706

-- 
Thanks,
Lu

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Marc MERLIN]

On Mon, Jun 26, 2017 at 06:46:16PM +0800, Lu Fengqi wrote:
> Thanks for the updated information. I'm sorry that the false alert make 
> you feel nervous.

If you can help me find out whether those are real errors that I need to fix
(and can't yet since there is no --repair), or whether they are not real
problems, I can ignore them as long as the other check --mode normal runs
clean (it does), we'll be good :)

> >ERROR: root 3862 EXTENT_DATA[18170706 4096] interrupt
> >ERROR: root 3862 EXTENT_DATA[18170706 16384] interrupt
> >ERROR: root 3862 EXTENT_DATA[18170706 20480] interrupt
> >ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt
> >ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt
> >ERROR: errors found in fs roots
> 
> However, this looks like another problem. Could you dump this file tree 
> by the following command?
> # btrfs-debug-tree -t 3862 <dev> | grep -C 10 18170706

argamel:~# btrfs-debug-tree -t 3862 /dev/mapper/dshelf2 | grep -C 10 18170706  
                transid 522 data_len 0 name_len 45
                name: 007b01c69a8d_9582a070_425a45d7@mindspring.com
        item 89 key (835232 DIR_ITEM 1181375325) itemoff 10304 itemsize 64
                location key (877605 INODE_ITEM 0) type FILE
                transid 530 data_len 0 name_len 34
                name: PO.5078346.7051988@codeweavers.com
        item 90 key (835232 DIR_ITEM 1181489720) itemoff 10226 itemsize 78
                location key (873230 INODE_ITEM 0) type FILE
                transid 529 data_len 0 name_len 48
                name: 8-8356087-5CruUNCbuG3Kg9zuO@mail1.fireflypro.com
        item 91 key (835232 DIR_ITEM 1181707066) itemoff 10148 itemsize 78
                location key (869906 INODE_ITEM 0) type FILE
                transid 528 data_len 0 name_len 48
                name: 7-7026088-1uLzZJuFzYD6h4rzV@max.firmalliance.biz
        item 92 key (835232 DIR_ITEM 1181727135) itemoff 10084 itemsize 64
                location key (877380 INODE_ITEM 0) type FILE
                transid 530 data_len 0 name_len 34
                name: NJ.5943286.7059518@codeweavers.com
        item 93 key (835232 DIR_ITEM 1181873033) itemoff 10038 itemsize 46
                location key (859092 INODE_ITEM 0) type FILE
                transid 526 data_len 0 name_len 16
                name: mdadm_detail.0
        item 83 key (2640780 DIR_ITEM 3316050734) itemoff 12739 itemsize 39
                location key (15689752 INODE_ITEM 0) type FILE
                transid 8178 data_len 0 name_len 9
                name: sda3.dd.0
        item 84 key (2640780 DIR_ITEM 3349213389) itemoff 12697 itemsize 42
                location key (2667656 INODE_ITEM 0) type FILE
                transid 885 data_len 0 name_len 12
                name: sdb2.dd.1.gz
        item 85 key (2640780 DIR_ITEM 3351742419) itemoff 12663 itemsize 34
                location key (18170706 INODE_ITEM 0) type FILE
                transid 37866 data_len 0 name_len 4
                name: dm-0
        item 86 key (2640780 DIR_ITEM 3354578455) itemoff 12624 itemsize 39
                location key (13847590 INODE_ITEM 0) type FILE
                transid 2387 data_len 0 name_len 9
                name: sda7.3.gz
        item 87 key (2640780 DIR_ITEM 3361267344) itemoff 12586 itemsize 38
                location key (2667594 INODE_ITEM 0) type FILE
                transid 885 data_len 0 name_len 8
                name: .profile
--
                name: sdc1.dd.4.gz
        item 70 key (2640780 DIR_INDEX 1685) itemoff 13162 itemsize 42
                location key (17548883 INODE_ITEM 0) type FILE
                transid 34469 data_len 0 name_len 12
                name: sdc1.dd.5.gz
        item 71 key (2640780 DIR_INDEX 1687) itemoff 13120 itemsize 42
                location key (17548884 INODE_ITEM 0) type FILE
                transid 34469 data_len 0 name_len 12
                name: sdc1.dd.6.gz
        item 72 key (2640780 DIR_INDEX 2039) itemoff 13086 itemsize 34
                location key (18170706 INODE_ITEM 0) type FILE
                transid 37866 data_len 0 name_len 4
                name: dm-0
        item 73 key (2640780 DIR_INDEX 2041) itemoff 13051 itemsize 35
                location key (18170707 INODE_ITEM 0) type FILE
                transid 37866 data_len 0 name_len 5
                name: fdisk
        item 74 key (2640780 DIR_INDEX 2043) itemoff 13007 itemsize 44
                location key (18170708 INODE_ITEM 0) type FILE
                transid 37866 data_len 0 name_len 14
                name: mdadm_detail.0

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Lu Fengqi]

On Tue, Jun 27, 2017 at 04:11:46PM -0700, Marc MERLIN wrote:
>On Mon, Jun 26, 2017 at 06:46:16PM +0800, Lu Fengqi wrote:
>> Thanks for the updated information. I'm sorry that the false alert make 
>> you feel nervous.
>
>If you can help me find out whether those are real errors that I need to fix
>(and can't yet since there is no --repair), or whether they are not real
>problems, I can ignore them as long as the other check --mode normal runs
>clean (it does), we'll be good :)

:)

>
>> >ERROR: root 3862 EXTENT_DATA[18170706 4096] interrupt
>> >ERROR: root 3862 EXTENT_DATA[18170706 16384] interrupt
>> >ERROR: root 3862 EXTENT_DATA[18170706 20480] interrupt
>> >ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt
>> >ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt
>> >ERROR: errors found in fs roots
>> 
>> However, this looks like another problem. Could you dump this file tree 
>> by the following command?
>> # btrfs-debug-tree -t 3862 <dev> | grep -C 10 18170706
>

Because the output is abnormal, except for the relevant DIR_ITEM and
DIR_INDEX, I can't find the above mentiond INODE_ITEM and EXTENT_DATA.
I wonder if the file system is online when this command is executed? If
so, please re-execute it offline again; if not, could you apply my
patches re-check it again?

>argamel:~# btrfs-debug-tree -t 3862 /dev/mapper/dshelf2 | grep -C 10 18170706  
>                transid 522 data_len 0 name_len 45
>                name: 007b01c69a8d_9582a070_425a45d7@mindspring.com
>        item 89 key (835232 DIR_ITEM 1181375325) itemoff 10304 itemsize 64
>                location key (877605 INODE_ITEM 0) type FILE
>                transid 530 data_len 0 name_len 34
>                name: PO.5078346.7051988@codeweavers.com
>        item 90 key (835232 DIR_ITEM 1181489720) itemoff 10226 itemsize 78
>                location key (873230 INODE_ITEM 0) type FILE
>                transid 529 data_len 0 name_len 48
>                name: 8-8356087-5CruUNCbuG3Kg9zuO@mail1.fireflypro.com
>        item 91 key (835232 DIR_ITEM 1181707066) itemoff 10148 itemsize 78
>                location key (869906 INODE_ITEM 0) type FILE
>                transid 528 data_len 0 name_len 48
>                name: 7-7026088-1uLzZJuFzYD6h4rzV@max.firmalliance.biz
>        item 92 key (835232 DIR_ITEM 1181727135) itemoff 10084 itemsize 64
>                location key (877380 INODE_ITEM 0) type FILE
>                transid 530 data_len 0 name_len 34
>                name: NJ.5943286.7059518@codeweavers.com
>        item 93 key (835232 DIR_ITEM 1181873033) itemoff 10038 itemsize 46
>                location key (859092 INODE_ITEM 0) type FILE
>                transid 526 data_len 0 name_len 16
>                name: mdadm_detail.0
>        item 83 key (2640780 DIR_ITEM 3316050734) itemoff 12739 itemsize 39
>                location key (15689752 INODE_ITEM 0) type FILE
>                transid 8178 data_len 0 name_len 9
>                name: sda3.dd.0
>        item 84 key (2640780 DIR_ITEM 3349213389) itemoff 12697 itemsize 42
>                location key (2667656 INODE_ITEM 0) type FILE
>                transid 885 data_len 0 name_len 12
>                name: sdb2.dd.1.gz
>        item 85 key (2640780 DIR_ITEM 3351742419) itemoff 12663 itemsize 34
>                location key (18170706 INODE_ITEM 0) type FILE
>                transid 37866 data_len 0 name_len 4
>                name: dm-0
>        item 86 key (2640780 DIR_ITEM 3354578455) itemoff 12624 itemsize 39
>                location key (13847590 INODE_ITEM 0) type FILE
>                transid 2387 data_len 0 name_len 9
>                name: sda7.3.gz
>        item 87 key (2640780 DIR_ITEM 3361267344) itemoff 12586 itemsize 38
>                location key (2667594 INODE_ITEM 0) type FILE
>                transid 885 data_len 0 name_len 8
>                name: .profile
>--
>                name: sdc1.dd.4.gz
>        item 70 key (2640780 DIR_INDEX 1685) itemoff 13162 itemsize 42
>                location key (17548883 INODE_ITEM 0) type FILE
>                transid 34469 data_len 0 name_len 12
>                name: sdc1.dd.5.gz
>        item 71 key (2640780 DIR_INDEX 1687) itemoff 13120 itemsize 42
>                location key (17548884 INODE_ITEM 0) type FILE
>                transid 34469 data_len 0 name_len 12
>                name: sdc1.dd.6.gz
>        item 72 key (2640780 DIR_INDEX 2039) itemoff 13086 itemsize 34
>                location key (18170706 INODE_ITEM 0) type FILE
>                transid 37866 data_len 0 name_len 4
>                name: dm-0
>        item 73 key (2640780 DIR_INDEX 2041) itemoff 13051 itemsize 35
>                location key (18170707 INODE_ITEM 0) type FILE
>                transid 37866 data_len 0 name_len 5
>                name: fdisk
>        item 74 key (2640780 DIR_INDEX 2043) itemoff 13007 itemsize 44
>                location key (18170708 INODE_ITEM 0) type FILE
>                transid 37866 data_len 0 name_len 14
>                name: mdadm_detail.0
>
>Marc
>-- 
>"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>Microsoft is to operating systems ....
>                                      .... what McDonalds is to gourmet cooking
>Home page: http://marc.merlins.org/  
>
>

-- 
Thanks,
Lu

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Marc MERLIN]

[-- Attachment #1: Type: text/plain, Size: 740 bytes --]

[cc trimmed]

On Wed, Jun 28, 2017 at 03:10:27PM +0800, Lu Fengqi wrote:
> Because the output is abnormal, except for the relevant DIR_ITEM and
> DIR_INDEX, I can't find the above mentiond INODE_ITEM and EXTENT_DATA.
> I wonder if the file system is online when this command is executed? If
> so, please re-execute it offline again; if not, could you apply my
> patches re-check it again?

The filesystem was offline and I had those 2 patches applied.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

[-- Attachment #2: p1.patch --]
[-- Type: text/x-diff, Size: 4038 bytes --]

>From lufq.fnst@cn.fujitsu.com Mon Jun 26 03:37:46 2017
Received: from [59.151.112.132] (port=50126 helo=heian.cn.fujitsu.com)
	by mail1.merlins.org with esmtp (Exim 4.87 #1)
	id 1dPROn-0001kT-Ud
	for <marc@merlins.org>; Mon, 26 Jun 2017 03:37:46 -0700
X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; 
   d="scan'208";a="20491849"
Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5])
  by heian.cn.fujitsu.com with ESMTP; 26 Jun 2017 18:37:30 +0800
Received: from G08CNEXCHPEKD02.g08.fujitsu.local (unknown [10.167.33.83])
	by cn.fujitsu.com (Postfix) with ESMTP id 2694647E64CC;
	Mon, 26 Jun 2017 18:37:30 +0800 (CST)
Received: from lufq.5F.lufq.5F (10.167.225.63) by
 G08CNEXCHPEKD02.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server
 (TLS) id 14.3.319.2; Mon, 26 Jun 2017 18:37:31 +0800
From: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
To: <linux-btrfs@vger.kernel.org>
CC: <marc@merlins.org>
Date: Mon, 26 Jun 2017 18:37:24 +0800
Message-ID: <20170626103727.8945-1-lufq.fnst@cn.fujitsu.com>
X-Mailer: git-send-email 2.13.1
MIME-Version: 1.0
Content-Type: text/plain
X-Originating-IP: [10.167.225.63]
X-yoursite-MailScanner-ID: 2694647E64CC.AB674
X-yoursite-MailScanner: Found to be clean
X-yoursite-MailScanner-From: lufq.fnst@cn.fujitsu.com
X-Broken-Reverse-DNS: no host name for IP address 59.151.112.132
X-SA-Exim-Connect-IP: 59.151.112.132
X-SA-Exim-Rcpt-To: marc@merlins.org
X-SA-Exim-Mail-From: lufq.fnst@cn.fujitsu.com
X-Spam-Checker-Version: SpamAssassin 3.4.1-mmrules_20121111 (2015-04-28) on
	magic.merlins.org
X-Spam-Level: 
X-Spam-Status: No, score=-2.6 required=7.0 tests=BAYES_00,GREYLIST_ISWHITE,
	RDNS_NONE autolearn=ham autolearn_force=no version=3.4.1-mmrules_20121111
X-Spam-Report: 
	* -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
	*      [score: 0.0000]
	*  0.8 RDNS_NONE Delivered to internal network by a host with no rDNS
	* -1.5 GREYLIST_ISWHITE The incoming server has been whitelisted for this
	*      receipient and sender
Subject: [PATCH v3 1/4] btrfs-progs: lowmem check: Fix false alert about file extent interrupt
X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000)
X-SA-Exim-Scanned: Yes (on mail1.merlins.org)
Status: RO
Content-Length: 1811
Lines: 52

As Qu mentioned in this thread
(https://www.spinics.net/lists/linux-btrfs/msg64469.html), compression
can cause regular extent to co-exist with inlined extent. This coexistence
makes things confusing. Since it was permitted currently, so fix
btrfsck to prevent a bunch of error logs that will make user feel
panic.

When check file extent, record the extent_end of regular extent to check
if there is a gap between the regular extents. Normally there is only one
inlined extent, so the extent_end of inlined extent is useless. However,
if regular extent can co-exist with inlined extent, the extent_end of
inlined extent also need to record.

Reported-by: Marc MERLIN <marc@merlins.org>
Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
---

Changlog:
v2: Just fix reported-by
v3: Output verbose information when file extent interrupt

 cmds-check.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index c052f66e..70d2b7f2 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -4782,6 +4782,7 @@ static int check_file_extent(struct btrfs_root *root, struct btrfs_key *fkey,
 				extent_num_bytes, item_inline_len);
 			err |= FILE_EXTENT_ERROR;
 		}
+		*end += extent_num_bytes;
 		*size += extent_num_bytes;
 		return err;
 	}
@@ -4847,8 +4848,8 @@ static int check_file_extent(struct btrfs_root *root, struct btrfs_key *fkey,
 		      root->objectid, fkey->objectid, fkey->offset);
 	} else if (!no_holes && *end != fkey->offset) {
 		err |= FILE_EXTENT_ERROR;
-		error("root %llu EXTENT_DATA[%llu %llu] interrupt",
-		      root->objectid, fkey->objectid, fkey->offset);
+		error("root %llu EXTENT_DATA[%llu %llu] interrupt, should start at %llu",
+		      root->objectid, fkey->objectid, fkey->offset, *end);
 	}
 
 	*end += extent_num_bytes;
-- 
2.13.1






[-- Attachment #3: p2.patch --]
[-- Type: text/x-diff, Size: 3267 bytes --]

>From lufq.fnst@cn.fujitsu.com Mon Jun 26 03:37:41 2017
Received: from [59.151.112.132] (port=50126 helo=heian.cn.fujitsu.com)
	by mail1.merlins.org with esmtp (Exim 4.87 #1)
	id 1dPROj-0001kT-Tq
	for <marc@merlins.org>; Mon, 26 Jun 2017 03:37:41 -0700
X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; 
   d="scan'208";a="20491848"
Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5])
  by heian.cn.fujitsu.com with ESMTP; 26 Jun 2017 18:37:30 +0800
Received: from G08CNEXCHPEKD02.g08.fujitsu.local (unknown [10.167.33.83])
	by cn.fujitsu.com (Postfix) with ESMTP id B3C5047E64D5;
	Mon, 26 Jun 2017 18:37:30 +0800 (CST)
Received: from lufq.5F.lufq.5F (10.167.225.63) by
 G08CNEXCHPEKD02.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server
 (TLS) id 14.3.319.2; Mon, 26 Jun 2017 18:37:32 +0800
From: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
To: <linux-btrfs@vger.kernel.org>
CC: <marc@merlins.org>
Date: Mon, 26 Jun 2017 18:37:25 +0800
Message-ID: <20170626103727.8945-2-lufq.fnst@cn.fujitsu.com>
X-Mailer: git-send-email 2.13.1
In-Reply-To: <20170626103727.8945-1-lufq.fnst@cn.fujitsu.com>
References: <20170626103727.8945-1-lufq.fnst@cn.fujitsu.com>
MIME-Version: 1.0
Content-Type: text/plain
X-Originating-IP: [10.167.225.63]
X-yoursite-MailScanner-ID: B3C5047E64D5.AC56F
X-yoursite-MailScanner: Found to be clean
X-yoursite-MailScanner-From: lufq.fnst@cn.fujitsu.com
X-Broken-Reverse-DNS: no host name for IP address 59.151.112.132
X-SA-Exim-Connect-IP: 59.151.112.132
X-SA-Exim-Rcpt-To: marc@merlins.org
X-SA-Exim-Mail-From: lufq.fnst@cn.fujitsu.com
X-Spam-Checker-Version: SpamAssassin 3.4.1-mmrules_20121111 (2015-04-28) on
	magic.merlins.org
X-Spam-Level: 
X-Spam-Status: No, score=-2.6 required=7.0 tests=BAYES_00,GREYLIST_ISWHITE,
	RDNS_NONE autolearn=ham autolearn_force=no version=3.4.1-mmrules_20121111
X-Spam-Report: 
	* -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
	*      [score: 0.0000]
	*  0.8 RDNS_NONE Delivered to internal network by a host with no rDNS
	* -1.5 GREYLIST_ISWHITE The incoming server has been whitelisted for this
	*      receipient and sender
Subject: [PATCH v3 2/4] btrfs-progs: lowmem check: Fix false alert about referencer count mismatch
X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000)
X-SA-Exim-Scanned: Yes (on mail1.merlins.org)
Status: O
Content-Length: 915
Lines: 29

The normal back reference counting doesn't care about the extent referred
by the extent data in the shared leaf. The check_extent_data_backref
function need to skip the leaf that owner mismatch with the root_id.

Reported-by: Marc MERLIN <marc@merlins.org>
Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
---
 cmds-check.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cmds-check.c b/cmds-check.c
index 70d2b7f2..f42968cd 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -10692,7 +10692,8 @@ static int check_extent_data_backref(struct btrfs_fs_info *fs_info,
 		leaf = path.nodes[0];
 		slot = path.slots[0];
 
-		if (slot >= btrfs_header_nritems(leaf))
+		if (slot >= btrfs_header_nritems(leaf) ||
+		    btrfs_header_owner(leaf) != root_id)
 			goto next;
 		btrfs_item_key_to_cpu(leaf, &key, slot);
 		if (key.objectid != objectid || key.type != BTRFS_EXTENT_DATA_KEY)
-- 
2.13.1

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Lu Fengqi]

On Wed, Jun 28, 2017 at 07:43:48AM -0700, Marc MERLIN wrote:
>[cc trimmed]
>
>On Wed, Jun 28, 2017 at 03:10:27PM +0800, Lu Fengqi wrote:
>> Because the output is abnormal, except for the relevant DIR_ITEM and
>> DIR_INDEX, I can't find the above mentiond INODE_ITEM and EXTENT_DATA.
>> I wonder if the file system is online when this command is executed? If
>> so, please re-execute it offline again; if not, could you apply my
>> patches re-check it again?
>
>The filesystem was offline and I had those 2 patches applied.

I am afraid I don't know why the inode item disappers. Besides, if
btrfs-debug-tree can't find the inode item, btrfs check shouldn't report
this inode item's extent data interrupt. Could you check the disk
again? The error output may have changed.

>
>Marc
>-- 
>"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>Microsoft is to operating systems ....
>                                      .... what McDonalds is to gourmet cooking
>Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901


-- 
Thanks,
Lu

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Marc MERLIN]

On Thu, Jun 29, 2017 at 09:36:15PM +0800, Lu Fengqi wrote:
> On Wed, Jun 28, 2017 at 07:43:48AM -0700, Marc MERLIN wrote:
> >[cc trimmed]
> >
> >On Wed, Jun 28, 2017 at 03:10:27PM +0800, Lu Fengqi wrote:
> >> Because the output is abnormal, except for the relevant DIR_ITEM and
> >> DIR_INDEX, I can't find the above mentiond INODE_ITEM and EXTENT_DATA.
> >> I wonder if the file system is online when this command is executed? If
> >> so, please re-execute it offline again; if not, could you apply my
> >> patches re-check it again?
> >
> >The filesystem was offline and I had those 2 patches applied.
> 
> I am afraid I don't know why the inode item disappers. Besides, if
> btrfs-debug-tree can't find the inode item, btrfs check shouldn't report
> this inode item's extent data interrupt. Could you check the disk
> again? The error output may have changed.

I just did but it takes 24H. I just have the results now: 
gargamel:~# btrfs check --mode lowmem  /dev/mapper/dshelf2
Checking filesystem on /dev/mapper/dshelf2
UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
ERROR: root 3862 EXTENT_DATA[18170706 4096] interrupt
ERROR: root 3862 EXTENT_DATA[18170706 16384] interrupt
ERROR: root 3862 EXTENT_DATA[18170706 20480] interrupt
ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt
ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt
ERROR: errors found in fs roots
found 5544779124736 bytes used, error(s) found
total csum bytes: 5344523140
total tree bytes: 71323058176
total fs tree bytes: 59288403968
total extent tree bytes: 5378277376
btree space waste bytes: 10912183048
file data blocks allocated: 7830914256896
 referenced 6244104495104


This is looking better, but not 0.
Can I ignore these or should we look into them still?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: How to fix errors that check --mode lomem finds, but --mode normal doesn't?

[Lu Fengqi]

On Thu, Jun 29, 2017 at 08:30:35AM -0700, Marc MERLIN wrote:
>On Thu, Jun 29, 2017 at 09:36:15PM +0800, Lu Fengqi wrote:
>> On Wed, Jun 28, 2017 at 07:43:48AM -0700, Marc MERLIN wrote:
>> >[cc trimmed]
>> >
>> >On Wed, Jun 28, 2017 at 03:10:27PM +0800, Lu Fengqi wrote:
>> >> Because the output is abnormal, except for the relevant DIR_ITEM and
>> >> DIR_INDEX, I can't find the above mentiond INODE_ITEM and EXTENT_DATA.
>> >> I wonder if the file system is online when this command is executed? If
>> >> so, please re-execute it offline again; if not, could you apply my
>> >> patches re-check it again?
>> >
>> >The filesystem was offline and I had those 2 patches applied.
>> 
>> I am afraid I don't know why the inode item disappers. Besides, if
>> btrfs-debug-tree can't find the inode item, btrfs check shouldn't report
>> this inode item's extent data interrupt. Could you check the disk
>> again? The error output may have changed.
>
>I just did but it takes 24H. I just have the results now: 
>gargamel:~# btrfs check --mode lowmem  /dev/mapper/dshelf2
>Checking filesystem on /dev/mapper/dshelf2
>UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
>checking extents
>checking free space cache
>cache and super generation don't match, space cache will be invalidated
>checking fs roots
>ERROR: root 3862 EXTENT_DATA[18170706 4096] interrupt
>ERROR: root 3862 EXTENT_DATA[18170706 16384] interrupt
>ERROR: root 3862 EXTENT_DATA[18170706 20480] interrupt
>ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt
>ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt
>ERROR: errors found in fs roots
>found 5544779124736 bytes used, error(s) found
>total csum bytes: 5344523140
>total tree bytes: 71323058176
>total fs tree bytes: 59288403968
>total extent tree bytes: 5378277376
>btree space waste bytes: 10912183048
>file data blocks allocated: 7830914256896
> referenced 6244104495104
>
>
>This is looking better, but not 0.
>Can I ignore these or should we look into them still?
>
>Marc
>-- 
>"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>Microsoft is to operating systems ....
>                                      .... what McDonalds is to gourmet cooking
>Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
>
>

Personally, I think since the normal mode didn't report any error
related this inode, then these error maybe caused by the bug of lowmem
mode and btrfs-debug-tree.

At your convenience, would you please give me all items about this
inode? I think it can provide some clues regarding the disappearance
of inode and the extent interrupt. It can be dumped by this following
command:

# btrfs-debug-tree /dev/mapper/dshelf2 | grep -C 10 18170706

Please pay attention that, this dump may contain filenames, feel free
to mask the filenames.

Thank you for your assistance.

-- 
Thanks,
Lu

ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5

[Marc MERLIN]

I'm still trying to fix my filesystem.
It seems to work well enough since the damage is apparently localized, but
I'd really want check --repair to actually bring it back to a working
state, but now it's crashing

This is btrfs tools from git from a few days ago

Failed to find [4068943577088, 168, 16384]
btrfs unable to find ref byte nr 4068943577088 parent 0 root 4  owner 1 offset 0
Failed to find [5905106075648, 168, 16384]
btrfs unable to find ref byte nr 5906282119168 parent 0 root 4  owner 0 offset 1
Failed to find [21037056, 168, 16384]
btrfs unable to find ref byte nr 21037056 parent 0 root 3  owner 1 offset 0
Failed to find [21053440, 168, 16384]
btrfs unable to find ref byte nr 21053440 parent 0 root 3  owner 0 offset 1
Failed to find [21299200, 168, 16384]
btrfs unable to find ref byte nr 21299200 parent 0 root 3  owner 0 offset 1
Failed to find [5523931971584, 168, 16384]
btrfs unable to find ref byte nr 5524037566464 parent 0 root 3861  owner 3 offset 0
ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5
btrfs(+0x113cf)[0x5651e60443cf]
btrfs(__btrfs_cow_block+0x576)[0x5651e6045848]
btrfs(btrfs_cow_block+0xea)[0x5651e6045dc6]
btrfs(btrfs_search_slot+0x11df)[0x5651e604969d]
btrfs(+0x59184)[0x5651e608c184]
btrfs(cmd_check+0x2bd4)[0x5651e60987b3]
btrfs(main+0x85)[0x5651e60442c3]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f34f523d2b1]
btrfs(_start+0x2a)[0x5651e6043e3a]


Full log:
enabling repair mode
Checking filesystem on /dev/mapper/dshelf2
UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checksum verify failed on 3037243965440 found 179689AF wanted 82B97043
checksum verify failed on 3037243965440 found 179689AF wanted 82B97043
checksum verify failed on 3037243998208 found 60EA5C5B wanted 0CF5948F
checksum verify failed on 3037243998208 found 60EA5C5B wanted 0CF5948F
checksum verify failed on 3037244293120 found 38382803 wanted 39E4F85E
checksum verify failed on 3037244293120 found 38382803 wanted 39E4F85E
checksum verify failed on 3037244342272 found E84F1D8F wanted 472DA98C
checksum verify failed on 3037244342272 found E84F1D8F wanted 472DA98C
checksum verify failed on 3037244669952 found 2F6E4C0E wanted E00BBF09
checksum verify failed on 3037244669952 found 2F6E4C0E wanted E00BBF09
checksum verify failed on 3037248913408 found CE2E4AEE wanted EF22F9CA
checksum verify failed on 3037248913408 found CE2E4AEE wanted EF22F9CA
checksum verify failed on 3037248929792 found C989CB0E wanted E27527BC
checksum verify failed on 3037248929792 found C989CB0E wanted E27527BC
checksum verify failed on 3037247569920 found 05848C79 wanted EF3D5598
checksum verify failed on 3037247569920 found 05848C79 wanted EF3D5598
checksum verify failed on 3037247586304 found 9D1E4E39 wanted F1EC8135
checksum verify failed on 3037247586304 found 9D1E4E39 wanted F1EC8135
checksum verify failed on 3037247619072 found BFE40520 wanted 627DB20D
checksum verify failed on 3037247619072 found BFE40520 wanted 627DB20D
checksum verify failed on 3037249208320 found A6B5775F wanted B1E6C0FC
checksum verify failed on 3037249208320 found A6B5775F wanted B1E6C0FC
checksum verify failed on 3037252534272 found 207AD7DF wanted DE72BDF7
checksum verify failed on 3037252534272 found 207AD7DF wanted DE72BDF7
checksum verify failed on 3111569391616 found 3C623707 wanted D955D668
checksum verify failed on 3111569391616 found 3C623707 wanted D955D668
checksum verify failed on 3111569768448 found 0C129F3C wanted C509003A
checksum verify failed on 3111569768448 found 0C129F3C wanted C509003A
checksum verify failed on 3111569735680 found E94C9D41 wanted 55836DD2
checksum verify failed on 3111569735680 found E94C9D41 wanted 55836DD2
checksum verify failed on 3037253435392 found 8E124EB5 wanted A3291C35
checksum verify failed on 3037253435392 found 8E124EB5 wanted A3291C35
checksum verify failed on 3037253746688 found 2B6A4DCD wanted 4323B339
checksum verify failed on 3037253746688 found 2B6A4DCD wanted 4323B339
checksum verify failed on 3111569702912 found 1048610C wanted 9856BB43
checksum verify failed on 3111569702912 found 1048610C wanted 9856BB43
checksum verify failed on 3111569801216 found CD7AAF82 wanted C1DA44DF
checksum verify failed on 3111569801216 found CD7AAF82 wanted C1DA44DF
checksum verify failed on 3037251878912 found 86FB02F3 wanted 728772CE
checksum verify failed on 3037251878912 found 86FB02F3 wanted 728772CE
checksum verify failed on 3037252861952 found CFD54426 wanted E91774C0
checksum verify failed on 3037252861952 found CFD54426 wanted E91774C0
checksum verify failed on 3037255974912 found E3655B7C wanted 8163FDDE
checksum verify failed on 3037255974912 found E3655B7C wanted 8163FDDE
checksum verify failed on 3037252927488 found E7AD88A3 wanted F6BA5B10
checksum verify failed on 3037252927488 found E7AD88A3 wanted F6BA5B10
checksum verify failed on 3037253500928 found 514A55B2 wanted 3611CD81
checksum verify failed on 3037253500928 found 514A55B2 wanted 3611CD81
checksum verify failed on 3037256105984 found 41ADA274 wanted 8F7F0A0B
checksum verify failed on 3037256105984 found 41ADA274 wanted 8F7F0A0B
Csum didn't match
The following tree block(s) is corrupted in tree 3861:
	tree block bytenr: 1710573748224, level: 1, node key: (1073956, 12, 959325)
Try to repair the btree for root 3861
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Csum didn't match
Failed to find [4068943577088, 168, 16384]
btrfs unable to find ref byte nr 4068943577088 parent 0 root 4  owner 1 offset 0
Failed to find [5905106075648, 168, 16384]
btrfs unable to find ref byte nr 5906282119168 parent 0 root 4  owner 0 offset 1
Failed to find [21037056, 168, 16384]
btrfs unable to find ref byte nr 21037056 parent 0 root 3  owner 1 offset 0
Failed to find [21053440, 168, 16384]
btrfs unable to find ref byte nr 21053440 parent 0 root 3  owner 0 offset 1
Failed to find [21299200, 168, 16384]
btrfs unable to find ref byte nr 21299200 parent 0 root 3  owner 0 offset 1
Failed to find [5523931971584, 168, 16384]
btrfs unable to find ref byte nr 5524037566464 parent 0 root 3861  owner 3 offset 0
ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5
btrfs(+0x113cf)[0x5651e60443cf]
btrfs(__btrfs_cow_block+0x576)[0x5651e6045848]
btrfs(btrfs_cow_block+0xea)[0x5651e6045dc6]
btrfs(btrfs_search_slot+0x11df)[0x5651e604969d]
btrfs(+0x59184)[0x5651e608c184]
btrfs(cmd_check+0x2bd4)[0x5651e60987b3]
btrfs(main+0x85)[0x5651e60442c3]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f34f523d2b1]
btrfs(_start+0x2a)[0x5651e6043e3a]
Aborted
gargamel:~# 
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5

[Marc MERLIN]

On Thu, Jul 06, 2017 at 10:37:18PM -0700, Marc MERLIN wrote:
> I'm still trying to fix my filesystem.
> It seems to work well enough since the damage is apparently localized, but
> I'd really want check --repair to actually bring it back to a working
> state, but now it's crashing
> 
> This is btrfs tools from git from a few days ago
> 
> Failed to find [4068943577088, 168, 16384]
> btrfs unable to find ref byte nr 4068943577088 parent 0 root 4  owner 1 offset 0
> Failed to find [5905106075648, 168, 16384]
> btrfs unable to find ref byte nr 5906282119168 parent 0 root 4  owner 0 offset 1
> Failed to find [21037056, 168, 16384]
> btrfs unable to find ref byte nr 21037056 parent 0 root 3  owner 1 offset 0
> Failed to find [21053440, 168, 16384]
> btrfs unable to find ref byte nr 21053440 parent 0 root 3  owner 0 offset 1
> Failed to find [21299200, 168, 16384]
> btrfs unable to find ref byte nr 21299200 parent 0 root 3  owner 0 offset 1
> Failed to find [5523931971584, 168, 16384]
> btrfs unable to find ref byte nr 5524037566464 parent 0 root 3861  owner 3 offset 0
> ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5
> btrfs(+0x113cf)[0x5651e60443cf]
> btrfs(__btrfs_cow_block+0x576)[0x5651e6045848]
> btrfs(btrfs_cow_block+0xea)[0x5651e6045dc6]
> btrfs(btrfs_search_slot+0x11df)[0x5651e604969d]
> btrfs(+0x59184)[0x5651e608c184]
> btrfs(cmd_check+0x2bd4)[0x5651e60987b3]
> btrfs(main+0x85)[0x5651e60442c3]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f34f523d2b1]
> btrfs(_start+0x2a)[0x5651e6043e3a]

Mmmh, never mind, it seems that the software raid suffered yet another
double disk failure due to some undermined flakiness in the underlying block
device cabling :-/
That would likely explain the failures here.

 
> Full log:
> enabling repair mode
> Checking filesystem on /dev/mapper/dshelf2
> UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede
> checking extents
> Fixed 0 roots.
> checking free space cache
> cache and super generation don't match, space cache will be invalidated
> checking fs roots
> checksum verify failed on 3037243965440 found 179689AF wanted 82B97043
> checksum verify failed on 3037243965440 found 179689AF wanted 82B97043
> checksum verify failed on 3037243998208 found 60EA5C5B wanted 0CF5948F
> checksum verify failed on 3037243998208 found 60EA5C5B wanted 0CF5948F
> checksum verify failed on 3037244293120 found 38382803 wanted 39E4F85E
> checksum verify failed on 3037244293120 found 38382803 wanted 39E4F85E
> checksum verify failed on 3037244342272 found E84F1D8F wanted 472DA98C
> checksum verify failed on 3037244342272 found E84F1D8F wanted 472DA98C
> checksum verify failed on 3037244669952 found 2F6E4C0E wanted E00BBF09
> checksum verify failed on 3037244669952 found 2F6E4C0E wanted E00BBF09
> checksum verify failed on 3037248913408 found CE2E4AEE wanted EF22F9CA
> checksum verify failed on 3037248913408 found CE2E4AEE wanted EF22F9CA
> checksum verify failed on 3037248929792 found C989CB0E wanted E27527BC
> checksum verify failed on 3037248929792 found C989CB0E wanted E27527BC
> checksum verify failed on 3037247569920 found 05848C79 wanted EF3D5598
> checksum verify failed on 3037247569920 found 05848C79 wanted EF3D5598
> checksum verify failed on 3037247586304 found 9D1E4E39 wanted F1EC8135
> checksum verify failed on 3037247586304 found 9D1E4E39 wanted F1EC8135
> checksum verify failed on 3037247619072 found BFE40520 wanted 627DB20D
> checksum verify failed on 3037247619072 found BFE40520 wanted 627DB20D
> checksum verify failed on 3037249208320 found A6B5775F wanted B1E6C0FC
> checksum verify failed on 3037249208320 found A6B5775F wanted B1E6C0FC
> checksum verify failed on 3037252534272 found 207AD7DF wanted DE72BDF7
> checksum verify failed on 3037252534272 found 207AD7DF wanted DE72BDF7
> checksum verify failed on 3111569391616 found 3C623707 wanted D955D668
> checksum verify failed on 3111569391616 found 3C623707 wanted D955D668
> checksum verify failed on 3111569768448 found 0C129F3C wanted C509003A
> checksum verify failed on 3111569768448 found 0C129F3C wanted C509003A
> checksum verify failed on 3111569735680 found E94C9D41 wanted 55836DD2
> checksum verify failed on 3111569735680 found E94C9D41 wanted 55836DD2
> checksum verify failed on 3037253435392 found 8E124EB5 wanted A3291C35
> checksum verify failed on 3037253435392 found 8E124EB5 wanted A3291C35
> checksum verify failed on 3037253746688 found 2B6A4DCD wanted 4323B339
> checksum verify failed on 3037253746688 found 2B6A4DCD wanted 4323B339
> checksum verify failed on 3111569702912 found 1048610C wanted 9856BB43
> checksum verify failed on 3111569702912 found 1048610C wanted 9856BB43
> checksum verify failed on 3111569801216 found CD7AAF82 wanted C1DA44DF
> checksum verify failed on 3111569801216 found CD7AAF82 wanted C1DA44DF
> checksum verify failed on 3037251878912 found 86FB02F3 wanted 728772CE
> checksum verify failed on 3037251878912 found 86FB02F3 wanted 728772CE
> checksum verify failed on 3037252861952 found CFD54426 wanted E91774C0
> checksum verify failed on 3037252861952 found CFD54426 wanted E91774C0
> checksum verify failed on 3037255974912 found E3655B7C wanted 8163FDDE
> checksum verify failed on 3037255974912 found E3655B7C wanted 8163FDDE
> checksum verify failed on 3037252927488 found E7AD88A3 wanted F6BA5B10
> checksum verify failed on 3037252927488 found E7AD88A3 wanted F6BA5B10
> checksum verify failed on 3037253500928 found 514A55B2 wanted 3611CD81
> checksum verify failed on 3037253500928 found 514A55B2 wanted 3611CD81
> checksum verify failed on 3037256105984 found 41ADA274 wanted 8F7F0A0B
> checksum verify failed on 3037256105984 found 41ADA274 wanted 8F7F0A0B
> Csum didn't match
> The following tree block(s) is corrupted in tree 3861:
> 	tree block bytenr: 1710573748224, level: 1, node key: (1073956, 12, 959325)
> Try to repair the btree for root 3861
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Csum didn't match
> Failed to find [4068943577088, 168, 16384]
> btrfs unable to find ref byte nr 4068943577088 parent 0 root 4  owner 1 offset 0
> Failed to find [5905106075648, 168, 16384]
> btrfs unable to find ref byte nr 5906282119168 parent 0 root 4  owner 0 offset 1
> Failed to find [21037056, 168, 16384]
> btrfs unable to find ref byte nr 21037056 parent 0 root 3  owner 1 offset 0
> Failed to find [21053440, 168, 16384]
> btrfs unable to find ref byte nr 21053440 parent 0 root 3  owner 0 offset 1
> Failed to find [21299200, 168, 16384]
> btrfs unable to find ref byte nr 21299200 parent 0 root 3  owner 0 offset 1
> Failed to find [5523931971584, 168, 16384]
> btrfs unable to find ref byte nr 5524037566464 parent 0 root 3861  owner 3 offset 0
> ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5
> btrfs(+0x113cf)[0x5651e60443cf]
> btrfs(__btrfs_cow_block+0x576)[0x5651e6045848]
> btrfs(btrfs_cow_block+0xea)[0x5651e6045dc6]
> btrfs(btrfs_search_slot+0x11df)[0x5651e604969d]
> btrfs(+0x59184)[0x5651e608c184]
> btrfs(cmd_check+0x2bd4)[0x5651e60987b3]
> btrfs(main+0x85)[0x5651e60442c3]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f34f523d2b1]
> btrfs(_start+0x2a)[0x5651e6043e3a]
> Aborted
> gargamel:~# 
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/  

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5

[Lu Fengqi]

On Thu, Jul 06, 2017 at 10:39:53PM -0700, Marc MERLIN wrote:
>On Thu, Jul 06, 2017 at 10:37:18PM -0700, Marc MERLIN wrote:
>> I'm still trying to fix my filesystem.
>> It seems to work well enough since the damage is apparently localized, but
>> I'd really want check --repair to actually bring it back to a working
>> state, but now it's crashing

I apologise for my late reply. As a colleague left, I have to take over his
work recently.

>> 
>> This is btrfs tools from git from a few days ago
>> 
>> Failed to find [4068943577088, 168, 16384]
>> btrfs unable to find ref byte nr 4068943577088 parent 0 root 4  owner 1 offset 0
>> Failed to find [5905106075648, 168, 16384]
>> btrfs unable to find ref byte nr 5906282119168 parent 0 root 4  owner 0 offset 1
>> Failed to find [21037056, 168, 16384]
>> btrfs unable to find ref byte nr 21037056 parent 0 root 3  owner 1 offset 0
>> Failed to find [21053440, 168, 16384]
>> btrfs unable to find ref byte nr 21053440 parent 0 root 3  owner 0 offset 1
>> Failed to find [21299200, 168, 16384]
>> btrfs unable to find ref byte nr 21299200 parent 0 root 3  owner 0 offset 1
>> Failed to find [5523931971584, 168, 16384]
>> btrfs unable to find ref byte nr 5524037566464 parent 0 root 3861  owner 3 offset 0
>> ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5
>> btrfs(+0x113cf)[0x5651e60443cf]
>> btrfs(__btrfs_cow_block+0x576)[0x5651e6045848]
>> btrfs(btrfs_cow_block+0xea)[0x5651e6045dc6]
>> btrfs(btrfs_search_slot+0x11df)[0x5651e604969d]
>> btrfs(+0x59184)[0x5651e608c184]
>> btrfs(cmd_check+0x2bd4)[0x5651e60987b3]
>> btrfs(main+0x85)[0x5651e60442c3]
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f34f523d2b1]
>> btrfs(_start+0x2a)[0x5651e6043e3a]
>
>Mmmh, never mind, it seems that the software raid suffered yet another
>double disk failure due to some undermined flakiness in the underlying block
>device cabling :-/
>That would likely explain the failures here.

I'm sorry for hear this. Which raid level are you using? So could you recover
from this double disk failure?

-- 
Thanks,
Lu

Re: ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5

[Marc MERLIN]

On Fri, Jul 07, 2017 at 05:33:20PM +0800, Lu Fengqi wrote:
> I apologise for my late reply. As a colleague left, I have to take over his
> work recently.
 
no worries.

> >Mmmh, never mind, it seems that the software raid suffered yet another
> >double disk failure due to some undermined flakiness in the underlying block
> >device cabling :-/
> >That would likely explain the failures here.
> 
> I'm sorry for hear this. Which raid level are you using? So could you recover
> from this double disk failure?

The disks aren't failed, and the array wasn't being written to.
It's just a matter of putting the disks back in the md raid5 array in the
right order.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Marc MERLIN]

Sigh,

This is now the 3rd filesystem I have (on 3 different machines) that is
getting corruption of some kind (on 4.11.6).
This is starting to look suspicious :-/

Can I fix this filesystem in some other way?
gargamel:/var/local/scr/host# btrfs check --repair /dev/mapper/crypt_bcache2 
enabling repair mode
Checking filesystem on /dev/mapper/crypt_bcache2
UUID: c4e6f9ca-e9a2-43d7-befa-763fc2cd5a57
checking extents
ref mismatch on [14655689654272 16384] extent item 0, found 1
Backref 14655689654272 parent 15455 root 15455 not found in extent tree
backpointer mismatch on [14655689654272 16384]
owner ref check failed [14655689654272 16384]
repair deleting extent record: key 14655689654272 169 1
adding new tree backref on start 14655689654272 len 16384 parent 0 root 15455
Repaired extent references for 14655689654272
root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)
ERROR: failed to repair root items: Invalid argument

Recreating the filesystem is going to take me a week of work, a lot of if
manual, and I'm not feeling very good with doing this since the backup
server this is a backup of, is also seeing some hopefully minor) problems
too.

I really hope there isn't a new corruption problem in 4.11, because when
I'm getting corruption on my laptop, my backup server, and the backup of my
backup server, I'm starting to run out of redundant backups :(
(and I'm not mentioning all the time this is costing me)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

We really need a better/working btrfs check --repair

[Marc MERLIN]

+Chris

On Sat, Jul 08, 2017 at 09:34:17PM -0700, Marc MERLIN wrote:
> gargamel:/var/local/scr/host# btrfs check --repair /dev/mapper/crypt_bcache2 
> enabling repair mode
> Checking filesystem on /dev/mapper/crypt_bcache2
> UUID: c4e6f9ca-e9a2-43d7-befa-763fc2cd5a57
> checking extents
> ref mismatch on [14655689654272 16384] extent item 0, found 1
> Backref 14655689654272 parent 15455 root 15455 not found in extent tree
> backpointer mismatch on [14655689654272 16384]
> owner ref check failed [14655689654272 16384]
> repair deleting extent record: key 14655689654272 169 1
> adding new tree backref on start 14655689654272 len 16384 parent 0 root 15455
> Repaired extent references for 14655689654272
> root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)
> ERROR: failed to repair root items: Invalid argument

On this note, getting hit 3 times on 3 different filesystems, that are not
badly damaged, but in none of those caess can btrfs check --repair put them
in a working state, is really bringing home the problem with lack of proper
fsck.

I understand that some errors are hard to fix without unknown data loss, but
btrfs check --repair should just do what it takes to put the filesystem back
into a consistent state, never mind what data is lost.
Restoring 10 to 20TB of data is getting old and is not really an acceptable
answer as the only way out.
I should not have to recreate a filesystem as the only way to bring it back
to a working state. 

Before Duncan tells me my filesystem is too big, and I should keep to very
small filesystems so that it's less work for each time btrfs gets corrupted
again, and fails again to bring back the filesystem to a usable state after
discarding some data, that's just not an acceptable answer long term, and by
long term honestly I mean now.
I just have data that doesn't segment well and the more small filesystems I
make the more time I'm going to waste managing them all and dealing with
which one gets full first :(

So, whether 4.11 has a corruption problem, or not, please put some resources
behind btrfs check --repair, be it the lowmem mode, or not.

Thank you
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Marc MERLIN]

On Sat, Jul 08, 2017 at 09:34:17PM -0700, Marc MERLIN wrote:
> Sigh,
> 
> This is now the 3rd filesystem I have (on 3 different machines) that is
> getting corruption of some kind (on 4.11.6).
> This is starting to look suspicious :-/
> 
> Can I fix this filesystem in some other way?
> gargamel:/var/local/scr/host# btrfs check --repair /dev/mapper/crypt_bcache2 
> enabling repair mode
> Checking filesystem on /dev/mapper/crypt_bcache2
> UUID: c4e6f9ca-e9a2-43d7-befa-763fc2cd5a57
> checking extents
> ref mismatch on [14655689654272 16384] extent item 0, found 1
> Backref 14655689654272 parent 15455 root 15455 not found in extent tree
> backpointer mismatch on [14655689654272 16384]
> owner ref check failed [14655689654272 16384]
> repair deleting extent record: key 14655689654272 169 1
> adding new tree backref on start 14655689654272 len 16384 parent 0 root 15455
> Repaired extent references for 14655689654272
> root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)
> ERROR: failed to repair root items: Invalid argument

Mmmh, actually to be fair, this was the 2nd run, I didn't scroll back
enough and missed the first run (doing too many recoveries at once,
I'm getting mixed up).
This first run looks like a lot more things happened:
http://marc.merlins.org/tmp/btrfs_check_ds5.txt

The amount of things that went wrong here are very worrisome, given that
there were no issues with those drives and that array has been working
for over a year without problems, until I recently upgraded to 4.11 :(

Now mind you, despite the 21MB of things that got fixed, I still kind of
have the expectation that btrfs check --repairs continues and fixes
everything until the filesystem is clean again, just like e2fsck -f
would, but I understand that this filesystem somehow got corrupted to a
point that it's maybe not that simple to do so.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Martin Steigerwald]

Hello Marc.

Marc MERLIN - 08.07.17, 21:34:
> Sigh,
> 
> This is now the 3rd filesystem I have (on 3 different machines) that is
> getting corruption of some kind (on 4.11.6).

Anyone else getting corruptions with 4.11?

I happily switch back to 4.10.17 or even 4.9 if that is the case. I may even 
do so just from your reports. Well, yes, I will do exactly that. I just switch 
back for 4.10 for now. Better be safe, than sorry.

I know how you feel, Marc. I posted about a corruption on one of my backup 
harddisks here some time ago that btrfs check --repair wasn´t able to handle. 
I redid that disk from scratch and it took a long, long time.

I agree with you that this has to stop. Before that I will never *ever* 
recommend this to a customer. Ideally no corruptions in stable kernels, 
especially when its a .6 at the end of the version number. But if so… then 
fixable. Other filesystems like Ext4 and XFS can do it… so this should be 
possible with BTRFS as well.

Thanks,
-- 
Martin

RE: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Paul Jones]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1175 bytes --]

> -----Original Message-----
> From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
> owner@vger.kernel.org] On Behalf Of Martin Steigerwald
> Sent: Sunday, 9 July 2017 5:58 PM
> To: Marc MERLIN <marc@merlins.org>
> Cc: Lu Fengqi <lufq.fnst@cn.fujitsu.com>; Btrfs BTRFS <linux-
> btrfs@vger.kernel.org>; David Sterba <dsterba@suse.cz>
> Subject: Re: 4.11.6 / more corruption / root 15455 has a root item with a more
> recent gen (33682) compared to the found root node (0)
> 
> Hello Marc.
> 
> Marc MERLIN - 08.07.17, 21:34:
> > Sigh,
> >
> > This is now the 3rd filesystem I have (on 3 different machines) that
> > is getting corruption of some kind (on 4.11.6).
> 
> Anyone else getting corruptions with 4.11?
> 
> I happily switch back to 4.10.17 or even 4.9 if that is the case. I may even do
> so just from your reports. Well, yes, I will do exactly that. I just switch back
> for 4.10 for now. Better be safe, than sorry.

No corruption for me - I've been on 4.11 since about .2 and everything seems fine. Currently on 4.11.8

Paul.
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Duncan]

Paul Jones posted on Sun, 09 Jul 2017 09:16:36 +0000 as excerpted:

>> Marc MERLIN - 08.07.17, 21:34:
>> >
>> > This is now the 3rd filesystem I have (on 3 different machines) that
>> > is getting corruption of some kind (on 4.11.6).
>> 
>> Anyone else getting corruptions with 4.11?
>> 
>> I happily switch back to 4.10.17 or even 4.9 if that is the case. I may
>> even do so just from your reports. Well, yes, I will do exactly that. I
>> just switch back for 4.10 for now. Better be safe, than sorry.
> 
> No corruption for me - I've been on 4.11 since about .2 and everything
> seems fine. Currently on 4.11.8

No corruptions here either. 4.12.0 now, previously 4.12-rc5(ish, git), 
before that 4.11.0.

I have however just upgraded to new ssds then wiped and setup the old 
ones as another backup set, so everything is on brand new filesystems on 
fast ssds, no possibility of old undetected corruption suddenly 
triggering problems.

Also, all my btrfs are raid1 or dup for checksummed redundancy, and 
relatively small, the largest now 80 GiB per device, after the upgrade.  
And my use-case doesn't involve snapshots or subvolumes.  

So any bug that is most likely on older filesystems, say those without 
the no-holes feature, for instance, or that doesn't tend to hit raid1 or 
dup mode, or that is less likely on small filesystems on fast ssds, or 
that triggers most often with reflinks and thus on filesystems with 
snapshots, is unlikely to hit me.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Martin Steigerwald]

Hello Duncan.

Duncan - 09.07.17, 11:17:
> Paul Jones posted on Sun, 09 Jul 2017 09:16:36 +0000 as excerpted:
> >> Marc MERLIN - 08.07.17, 21:34:
> >> > This is now the 3rd filesystem I have (on 3 different machines) that
> >> > is getting corruption of some kind (on 4.11.6).
> >> 
> >> Anyone else getting corruptions with 4.11?
> >> 
> >> I happily switch back to 4.10.17 or even 4.9 if that is the case. I may
> >> even do so just from your reports. Well, yes, I will do exactly that. I
> >> just switch back for 4.10 for now. Better be safe, than sorry.
> > 
> > No corruption for me - I've been on 4.11 since about .2 and everything
> > seems fine. Currently on 4.11.8
> 
> No corruptions here either. 4.12.0 now, previously 4.12-rc5(ish, git),
> before that 4.11.0.
> 
> I have however just upgraded to new ssds then wiped and setup the old
[…]
> Also, all my btrfs are raid1 or dup for checksummed redundancy, and
> relatively small, the largest now 80 GiB per device, after the upgrade.
> And my use-case doesn't involve snapshots or subvolumes.
> 
> So any bug that is most likely on older filesystems, say those without
> the no-holes feature, for instance, or that doesn't tend to hit raid1 or
> dup mode, or that is less likely on small filesystems on fast ssds, or
> that triggers most often with reflinks and thus on filesystems with
> snapshots, is unlikely to hit me.

Hmmm, the BTRFS filesystems on my laptop 3 to 5 or even more years old. I stick 
with 4.10 for now, I think.

The older ones are RAID 1 across two SSDs, the newer one is single device, on 
one SSD.

These filesystems didn´t fail me in years and since 4.5 or 4.6 even the "I 
search for free space" kernel hang (hung tasks and all that) is gone as well.

Thanks,
-- 
Martin

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Imran Geriskovan]

On 7/9/17, Duncan <1i5t5.duncan@cox.net> wrote:
> I have however just upgraded to new ssds then wiped and setup the old
> ones as another backup set, so everything is on brand new filesystems on
> fast ssds, no possibility of old undetected corruption suddenly
> triggering problems.
>
> Also, all my btrfs are raid1 or dup for checksummed redundancy

Do you have any experience/advice/comment regarding
dup data on ssds?

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Duncan]

Imran Geriskovan posted on Sat, 29 Jul 2017 21:29:46 +0200 as excerpted:

> On 7/9/17, Duncan <1i5t5.duncan@cox.net> wrote:
>> I have however just upgraded to new ssds then wiped and setup the old
>> ones as another backup set, so everything is on brand new filesystems 
on
>> fast ssds, no possibility of old undetected corruption suddenly
>> triggering problems.
>>
>> Also, all my btrfs are raid1 or dup for checksummed redundancy
> 
> Do you have any experience/advice/comment regarding
> dup data on ssds?

Very good question. =:^)

Limited.  Most of my btrfs are raid1, with dup only used on the device-
respective /boot btrfs (of which there are four, one on each of the two 
ssds that otherwise form the btrfs raid1 pairs, for each of the working 
and backup copy pairs -- I can use BIOS to select any of the four to 
boot), and those are all sub-GiB mixed-bg mode.

So all my dup experience is sub-GiB mixed-blockgroup mode.

Within that limitation, my only btrfs problem has been that at my 
initially chosen size of 256 MiB, mkfs.btrfs at least used to create an 
initial data/metadata chunk of 64 MiB.  Remember, this is dup mode, so 
there's two of them = 128 MiB.  Because there's also a system chunk, that 
means the initial chunk cannot be balanced even with an entirely empty 
filesystem, because there's not enough space to write a second 64 MiB 
chunk duped to 128 MiB.

Between that and the 256 MiB in dup mode size meaning under 128 MiB 
usable, and the fact that I routinely run and sometimes need to bisect 
pre-release kernels, I was routinely running out of space, then cleaning 
up, but not being able to do a full cleanup without a blow-away and new 
mkfs.btrfs, because I couldn't balance.

When I recently purchased the second pair of (now larger) ssds in ordered 
to put everything, including the media and backups that were previously 
still on spinning rust, on ssd, I redid the layout and made the /boots 
512 MiB, still mixed-bg dup mode.  That seems to have solved the problem, 
and I can now rebalance the first mkfs.btrfs-created mixed-bg chunk, as 
it's now small enough that it's less than half the filesystem even when 
duped.

Because it's now 512 MiB, however, I can't say for sure whether the 
previous problem with mkfs.btrfs creating an initial mixed-bg chunk of a 
quarter the 256 MiB filesystem size, so in dup mode it can't be balanced 
because it's half the total filesystem size and with the system chunk as 
well, the other half is partially used so there's no space to write the 
balance destination chunks, is fixed, or not.  What I can say is that the 
problem doesn't affect the new 512 MiB size, at least with btrfs-progs 
4.11.x, which is what I used to mkfs.btrfs the new layout.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Imran Geriskovan]

On 7/30/17, Duncan <1i5t5.duncan@cox.net> wrote:
>>> Also, all my btrfs are raid1 or dup for checksummed redundancy

>> Do you have any experience/advice/comment regarding
>> dup data on ssds?

> Very good question. =:^)

> Limited.  Most of my btrfs are raid1, with dup only used on the device-
> respective /boot btrfs (of which there are four, one on each of the two
> ssds that otherwise form the btrfs raid1 pairs, for each of the working
> and backup copy pairs -- I can use BIOS to select any of the four to
> boot), and those are all sub-GiB mixed-bg mode.

Is this a military or deep space device? ;)

> So all my dup experience is sub-GiB mixed-blockgroup mode.
>
> Within that limitation, my only btrfs problem has been that at my
> initially chosen size of 256 MiB, mkfs.btrfs at least used to create an
> initial data/metadata chunk of 64 MiB.  Remember, this is dup mode, so
> there's two of them = 128 MiB.  Because there's also a system chunk, that
> means the initial chunk cannot be balanced even with an entirely empty
> filesystem, because there's not enough space to write a second 64 MiB
> chunk duped to 128 MiB.

For /boot, I've also tried dup data.

But because of combinations of constraints you've mentioned,
I totally give-up trying to have a bullet proof /boot
as my poor laptop is not mission critical as your device and
as I do always have bootable backups and always carry
some bootable sdcards.

Perhaps that has something to do with me kicking
out all systemd, inits, initramfs, mkinitcpio, dracut, etc, etc.

Now the init on /boot is a "19 lines" shell script, including lines
for keymap, hdparm, crytpsetup. And let's not forget this is
possible by a custom kernel, its reliable buddy syslinux.

Interestingly my seach for reliability started with
"dup data" and ended up here. :)

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Duncan]

Imran Geriskovan posted on Sun, 30 Jul 2017 16:54:25 +0200 as excerpted:

> On 7/30/17, Duncan <1i5t5.duncan@cox.net> wrote:
>>>> Also, all my btrfs are raid1 or dup for checksummed redundancy
> 
>>> Do you have any experience/advice/comment regarding dup data on ssds?
> 
>> Very good question. =:^)
> 
>> Limited.  Most of my btrfs are raid1, with dup only used on the device-
>> respective /boot btrfs (of which there are four, one on each of the two
>> ssds that otherwise form the btrfs raid1 pairs, for each of the working
>> and backup copy pairs -- I can use BIOS to select any of the four to
>> boot), and those are all sub-GiB mixed-bg mode.
> 
> Is this a military or deep space device? ;)

Just happens to have four physical ssds, two pairs, with everything but 
/boot being paired btrfs raid1.  Because I wanted similar partition 
layout for ease of management, that's a /boot on each one, and because 
bios can only point to one at a time, that's four separate grub installs
[1], each of which is configured to load its own /boot.

While four is a bit much, three can certainly be very useful, because it 
allows a bad grub upgrade to be core-installed to one BIOS-boot 
partition, while allowing me to fat-finger point it to the wrong /boot on 
a second device destroying my ability to boot to it as well, and still 
have a third untouched to boot from.  The forth is simply bonus insurance 
on that, more by accident due to having two pair than because I really 
needed it.

A minimum of three /boots is also quite convenient for my kernel update 
routine, given I routinely test and sometimes bisect pre-release 
kernels.  The default/working /boot gets the prereleases with a release 
and stable fallback, the first backup the releases and a stable fallback, 
and the secondary backups get updated less frequently, generally when I'm 
doing a / backup cycle as well and there has been either a kernel config 
or system change substantial enough that I'm no longer confident the 
older kernels will work correctly with the updated system.

Of course the same general testing/release/stable /boot system works well 
for other related updates, say to the grub menu (I use grub2's bash-like 
scripting language directly, not the high level stuff which I find too 
difficult to tweak to my liking) or the initrd, which I attach to the 
individual kernels at build-time, so a tested kernel selection is a 
tested initramfs selection as well.

> For /boot, I've also tried dup data.
> 
> But because of combinations of constraints you've mentioned,
> I totally give-up trying to have a bullet proof /boot as my poor laptop
> is not mission critical as your device and as I do always have bootable
> backups and always carry some bootable sdcards.

When I complained about the 64-MiB default mixed-bg mode chunk size on a 
256 MiB filesystem being too big to allow balance in dup mode, a dev 
answered that in theory chunk sizes are supposed to be limited to 1/8 
filesystem size (down to something like a 16 MiB minimum chunk size I 
think, but might be 8 or 32), but something about my setup, likely the 
mixed-bg mode as it's less tested, was short-circuiting that, thus the 
quarter-fs-size 64 MiB chunk sizes, which he agreed didn't make much 
sense on a 256 MiB filesystem in dup mode.

He was able to duplicate the problem, and there seemed no disagreement is 
was a bug, but I'm not sure if mkfs.btrfs was ever patched to fix it, and 
of course now with the bigger half-gig filesystem the same 64-MiB initial 
chunk size is fine.

And my other quarter-gig btrfs, log, is raid1, quarter-gig per device, so 
I'd not see the problem there, mixed-mode or not.  (As mentioned in the 
footnote below, at least in this go-round it's not... more by accident 
than intent.)

Meanwhile, such bugs come with the territory when you're running what 
might be roughly compared at the commercial software level to late beta 
or rc level software, or even initial release, pre-service-release-1, 
level, which I'd argue is a more accurate btrfs comparison at this 
point.  As long as you stay within the known stable areas the danger of 
it eating your data is relatively small now, but the full feature set 
isn't there yet, and some of the features that are there are 
significantly less mature and stable than others.

> Perhaps that has something to do with me kicking out all systemd, inits,
> initramfs, mkinitcpio, dracut, etc, etc.
> 
> Now the init on /boot is a "19 lines" shell script, including lines for
> keymap, hdparm, crytpsetup. And let's not forget this is possible by a
> custom kernel, its reliable buddy syslinux.

FWIW...

I really like grub2, especially it's quite flexible bash-like scripting 
language (the higher level stuff intended for normal users just isn't 
flexible enough for me, so I need the scripting language anyway, and once 
I knew that, the higher level stuff only got in the way) and command line 
that allow all sorts of stuff like browsing for kernel commandline 
documentation at the boot prompt that I never imagined possible in a boot 
manager.

And after holding off for awhile, I'm now a cautious adopter and 
supporter of systemd in general, tho I don't use its solutions for 
/everything/ and don't like its extremely aggressive feature expansion.

And after resisting an initr* for years as unnecessary, I've been a 
reluctant adopter since a btrfs raid1 root effectively requires it 
(rootflags=device= doesn't seem to work, for whatever reason, or at least 
didn't when I initially converted to btrfs, so at least a limited initr* 
seems the only viable solution for a btrfs raid1 root).

And I'm using dracut for that, tho quite cut down from its default, with 
a monolithic kernel and only installing necessary dracut modules.

But particularly after the last dracut update pulled in kmod as a 
mandatory dep as it now links against its libs, despite my monolithic 
kernel built without module support, I've been considering similar initr* 
alternatives, including hand-rolling my own initr* build scripts.

Because I'm still not happy having to run an initr* at all, especially 
since there's more "magic" there than I'm particularly comfortable with 
since I like to grok the boot and thus potential recovery process better 
than I do this, and dracut was just the most convenient option at the 
time.

But kmod isn't a /huge/ dep, particularly with the executables and docs 
install-masked so it's only the library, headers and *.pc config file 
installed, and the current dracut solution works /reasonably/ well, so 
finding/creating an alternative isn't particularly high on my priority 
list, and I'll probably never do it unless dracut suddenly decides some 
of its other modules are going to need mandatory deps, or something else 
radically changes the current fragile balance and I really do need that 
currently lacking initr* grok.

> Interestingly my seach for reliability started with "dup data" and ended
> up here. :)

=:^)

---
[1] Grub and partition layout:  I install grub-core (i386-pc) to a raw 
GPT legacy BIOS boot partition.  While this only requires a partition 
size of about a third of a MiB, I use gdisk's default 1 MiB alignment and 
the first MiB is the GTP and the alignment gap, so this first BIOS boot 
partition starts at 1 MiB and must be a whole MiB unit in size.  Because 
I wanted plenty of room, however, and wanted additional partitions a 
minimum of 4 MiB aligned, I configured a 3 MiB BIOS boot partition for 
grub to use, thus accomplishing that 4 MiB alignment for further 
partitions.

The second partition is a currently unused GPT EFI partition for forward 
compatibility, 252 MiB in size so further partitions are quarter-GiB 
aligned.

The third partition is the /boot partition we've been discussing, a half 
GiB in size, thus ending at 3/4 GiB.  It's my only btrfs mixed-mode dup 
in the layout, so a half gig in size but a quarter gig usable.  As 
mentioned, with four physical ssds that's a total of four /boots, each 
pointed at by the grub-core installation in the first partition on the 
corresponding ssd.

Partition 4 is the log partition, a quarter GiB in size as log rotation 
keeps typical usage under 50 MiB, but the quarter gig size means it ends 
on the 1 GiB boundary and further partitions are GiB aligned.  In the 
last layout generation this was a half gig and /boot a quarter gig, but I 
decided /boot could use the extra quarter gig more than log so I traded 
sizes.  This, like all further partitions, is btrfs raid1.  I intended to 
make it mixed-bg mode, as it was in the previous generation layout, but 
forgot the mkfs.btrfs switch for that and it no longer defaults to mixed 
at under a gig, so I got standard mode.  Never-the-less, with raid1 
instead of dup, and low normal usage, the chunk size is small enough that 
balance shouldn't be an issue, and if it is I can always blow it away and 
recreate in mixed mode.

All further partitions are gig-aligned btrfs raid1 pair-device, three 
copies, working/0 and backups 1 and 2, on two separate pairs of ssds.  
The older pair is 256GB/238GiB with the backup/1 copy, the newer pair is 
1TB/931GiB with working/0 and backup/2.  The partition size and layout is 
identical on all four thru the sub-GiB and first copy, with the second 
copy on the larger pair being a same-sequence same-size repeat of the 
first, beyond the non-duplicated sub-GiB, of course.  So as long as the 
GPT on one of the four remains intact and bootable, I can easily recreate 
the other three.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Imran Geriskovan]

>>>> Do you have any experience/advice/comment regarding dup data on ssds?

>>> Very good question. =:^)

>> Now the init on /boot is a "19 lines" shell script, including lines for
>> keymap, hdparm, crytpsetup. And let's not forget this is possible by a
>> custom kernel and its reliable buddy syslinux.
>
> FWIW...
> And I'm using dracut for that, tho quite cut down from its default, with
> a monolithic kernel and only installing necessary dracut modules.

Just create minimal bootable /boot for running below init.
(Your initramfs/rd is a bloated and packaged version of
this anyway.) Kick the rest. Since you a have your own
kernel you are not far away from it.


#!/bin/sh
# This is actually busybox ash or hush. Cant remember now.
# You may compile/customize your busybox as well. Easy.

mount proc /proc -t proc
mount sys  /sys  -t sysfs
mount run  /run  -t tmpfs
mkdir /dev/pts /dev/shm /run/lock
mount devpts /dev/pts -t devpts &
mount shm    /dev/shm -t tmpfs &
mount -o remount,rw,noatime / &

# '&' is for backgrounding/parallel_execution.
# Use responsibly double checking its side effects
# depending on your setup.

hdparm -B 254 /dev/sda &
loadkmap < /boot/trq.bkmap

cryptsetup -T 10 luksOpen /dev/sdXX sdXX
mount /dev/mapper/sdXX /mnt/new_root -t btrfs -o noatime,compress=lzo

cd /mnt/new_root
mount --move /dev  ./dev
mount --move /proc ./proc
mount --move /sys  ./sys
mount --move /run  ./run
pivot_root . boot

exec chroot . busybox init
# Jump to your real roots init. Whatever it may be.


> But particularly after the last dracut update pulled in kmod as a
> mandatory dep as it now links against its libs, despite my monolithic
> kernel built without module support, I've been considering similar initr*
> alternatives, including hand-rolling my own initr* build scripts.
>
> Because I'm still not happy having to run an initr* at all, especially
> since there's more "magic" there than I'm particularly comfortable with
> since I like to grok the boot and thus potential recovery process better
> than I do this, and dracut was just the most convenient option at the
> time.

>> Interestingly my seach for reliability started with "dup data" and ended
>> up here. :)
> =:^)

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Ivan Sizov]

2017-07-09 10:57 GMT+03:00 Martin Steigerwald <martin@lichtvoll.de>:
> Hello Marc.
>
> Marc MERLIN - 08.07.17, 21:34:
>> Sigh,
>>
>> This is now the 3rd filesystem I have (on 3 different machines) that is
>> getting corruption of some kind (on 4.11.6).
>
> Anyone else getting corruptions with 4.11?
Yes, a lot. There are at least 3 cases, probably I've missed something.
https://www.spinics.net/lists/linux-btrfs/msg67177.html
https://www.spinics.net/lists/linux-btrfs/msg67681.html
https://unix.stackexchange.com/questions/369133/dealing-with-btrfs-ref-backpointer-mismatches-backref-missing/369275

If an additional debug info is needed, I'm ready to provide it.

-- 
Ivan Sizov

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Marc MERLIN]

On Tue, Aug 01, 2017 at 12:07:14AM +0300, Ivan Sizov wrote:
> 2017-07-09 10:57 GMT+03:00 Martin Steigerwald <martin@lichtvoll.de>:
> > Hello Marc.
> >
> > Marc MERLIN - 08.07.17, 21:34:
> >> Sigh,
> >>
> >> This is now the 3rd filesystem I have (on 3 different machines) that is
> >> getting corruption of some kind (on 4.11.6).
> >
> > Anyone else getting corruptions with 4.11?
> Yes, a lot. There are at least 3 cases, probably I've missed something.
> https://www.spinics.net/lists/linux-btrfs/msg67177.html
> https://www.spinics.net/lists/linux-btrfs/msg67681.html
> https://unix.stackexchange.com/questions/369133/dealing-with-btrfs-ref-backpointer-mismatches-backref-missing/369275

Indeed. My main server is happy back on 4.9.36 and while my laptop is
stuck on 4.11 due to other kernel issues that prevent me from going back
to 4.9, it only corrupted a single filesystem so far, and no other ones
that I've noticed yet.
Hopefully that will hold :-/

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Ivan Sizov]

2017-08-01 0:17 GMT+03:00 Marc MERLIN <marc@merlins.org>:
> On Tue, Aug 01, 2017 at 12:07:14AM +0300, Ivan Sizov wrote:
>> 2017-07-09 10:57 GMT+03:00 Martin Steigerwald <martin@lichtvoll.de>:
>> > Hello Marc.
>> >
>> > Marc MERLIN - 08.07.17, 21:34:
>> >> Sigh,
>> >>
>> >> This is now the 3rd filesystem I have (on 3 different machines) that is
>> >> getting corruption of some kind (on 4.11.6).
>> >
>> > Anyone else getting corruptions with 4.11?
>> Yes, a lot. There are at least 3 cases, probably I've missed something.
>> https://www.spinics.net/lists/linux-btrfs/msg67177.html
>> https://www.spinics.net/lists/linux-btrfs/msg67681.html
>> https://unix.stackexchange.com/questions/369133/dealing-with-btrfs-ref-backpointer-mismatches-backref-missing/369275
>
> Indeed. My main server is happy back on 4.9.36 and while my laptop is
> stuck on 4.11 due to other kernel issues that prevent me from going back
> to 4.9, it only corrupted a single filesystem so far, and no other ones
> that I've noticed yet.
> Hopefully that will hold :-/
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

I want to try mounting and checking FS under Live images with
different kernels tomorrow. Today's Fedora Rawhide image seems to be
built incorrectly. Can you advice me where to get a fresh live image
with 4.12 kernel (it's not important which distro that will be)?

-- 
Ivan Sizov

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Justin Maggard]

On Mon, Jul 31, 2017 at 2:17 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Tue, Aug 01, 2017 at 12:07:14AM +0300, Ivan Sizov wrote:
>> 2017-07-09 10:57 GMT+03:00 Martin Steigerwald <martin@lichtvoll.de>:
>> > Hello Marc.
>> >
>> > Marc MERLIN - 08.07.17, 21:34:
>> >> Sigh,
>> >>
>> >> This is now the 3rd filesystem I have (on 3 different machines) that is
>> >> getting corruption of some kind (on 4.11.6).
>> >
>> > Anyone else getting corruptions with 4.11?
>> Yes, a lot. There are at least 3 cases, probably I've missed something.
>> https://www.spinics.net/lists/linux-btrfs/msg67177.html
>> https://www.spinics.net/lists/linux-btrfs/msg67681.html
>> https://unix.stackexchange.com/questions/369133/dealing-with-btrfs-ref-backpointer-mismatches-backref-missing/369275
>
> Indeed. My main server is happy back on 4.9.36 and while my laptop is
> stuck on 4.11 due to other kernel issues that prevent me from going back
> to 4.9, it only corrupted a single filesystem so far, and no other ones
> that I've noticed yet.
> Hopefully that will hold :-/
>

Marc, do you have quotas enabled?  IIRC, you're a send/receive user.
The combination of quotas and btrfs receive can corrupt your
filesystem, as shown by the xfstest I sent to the list a little while
ago.

-Justin

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Duncan]

Imran Geriskovan posted on Mon, 31 Jul 2017 22:32:39 +0200 as excerpted:

>>> Now the init on /boot is a "19 lines" shell script, including lines
>>> for keymap, hdparm, crytpsetup. And let's not forget this is possible
>>> by a custom kernel and its reliable buddy syslinux.
>>
>> FWIW...
>> And I'm using dracut for that, tho quite cut down from its default,
>> with a monolithic kernel and only installing necessary dracut modules.
> 
> Just create minimal bootable /boot for running below init.
> (Your initramfs/rd is a bloated and packaged version of this anyway.)
> Kick the rest. Since you a have your own kernel you are not far away
> from it.

Thanks.  You just solved my primary problem of needing to take the time 
to actually research all the steps and in what order I needed to do them, 
for a hand-rolled script. =:^)

Unfortunately, while I've been laid-up the last ~5 days due to a twisted 
knee and have been spending more time on the lists, etc, and would have 
loved to spend a day or so testing and setting this up, I'm back to work 
tomorrow, so I've no idea when I'll actually get to play with this.

But meanwhile, I'm saving your message for reference when the time 
comes.  It should be /very/ useful!  =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Marc MERLIN]

On Mon, Jul 31, 2017 at 03:00:53PM -0700, Justin Maggard wrote:
> Marc, do you have quotas enabled?  IIRC, you're a send/receive user.
> The combination of quotas and btrfs receive can corrupt your
> filesystem, as shown by the xfstest I sent to the list a little while
> ago.

Thanks for checking. I do not use quota given the problems I had with
them early on over 2y ago.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Imran Geriskovan]

On 8/1/17, Duncan <1i5t5.duncan@cox.net> wrote:
> Imran Geriskovan posted on Mon, 31 Jul 2017 22:32:39 +0200 as excerpted:
>>>> Now the init on /boot is a "19 lines" shell script, including lines
>>>> for keymap, hdparm, crytpsetup. And let's not forget this is possible
>>>> by a custom kernel and its reliable buddy syslinux.

>>> And I'm using dracut for that, tho quite cut down from its default,
>>> with a monolithic kernel and only installing necessary dracut modules.

>> Just create minimal bootable /boot for running below init.
>> (Your initramfs/rd is a bloated and packaged version of this anyway.)
>> Kick the rest. Since you a have your own kernel you are not far away
>> from it.

> Thanks.  You just solved my primary problem of needing to take the time
> to actually research all the steps and in what order I needed to do them,
> for a hand-rolled script. =:^)

It's just a minimal one. But it is a good start. For possible extensions
extract your initramfs and explore it. Dracut is bloated. Try mkinitcpio.

Once your have your self hosting bootmng, kernel, modules, /boot, init, etc
chain, you'll be shocked to realize you have been spending so much time for
that bullshit while trying to keep them up..

Get to this point in the shortest possible time. Save your precious
time. And reclaim your systems reliability.

For X, you'll still need udev or eudev.

Re: 4.11.6 / more corruption / root 15455 has a root item with a more recent gen (33682) compared to the found root node (0)

[Ivan Sizov]

2017-08-01 0:39 GMT+03:00 Ivan Sizov <sivan606@gmail.com>:
> 2017-08-01 0:17 GMT+03:00 Marc MERLIN <marc@merlins.org>:
>> On Tue, Aug 01, 2017 at 12:07:14AM +0300, Ivan Sizov wrote:
>>> 2017-07-09 10:57 GMT+03:00 Martin Steigerwald <martin@lichtvoll.de>:
>>> > Hello Marc.
>>> >
>>> > Marc MERLIN - 08.07.17, 21:34:
>>> >> Sigh,
>>> >>
>>> >> This is now the 3rd filesystem I have (on 3 different machines) that is
>>> >> getting corruption of some kind (on 4.11.6).
>>> >
>>> > Anyone else getting corruptions with 4.11?
>>> Yes, a lot. There are at least 3 cases, probably I've missed something.
>>> https://www.spinics.net/lists/linux-btrfs/msg67177.html
>>> https://www.spinics.net/lists/linux-btrfs/msg67681.html
>>> https://unix.stackexchange.com/questions/369133/dealing-with-btrfs-ref-backpointer-mismatches-backref-missing/369275
>>
>> Indeed. My main server is happy back on 4.9.36 and while my laptop is
>> stuck on 4.11 due to other kernel issues that prevent me from going back
>> to 4.9, it only corrupted a single filesystem so far, and no other ones
>> that I've noticed yet.
>> Hopefully that will hold :-/
>>
>> Marc
>> --
>> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>> Microsoft is to operating systems ....
>>                                       .... what McDonalds is to gourmet cooking
>> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
>
> I want to try mounting and checking FS under Live images with
> different kernels tomorrow. Today's Fedora Rawhide image seems to be
> built incorrectly. Can you advice me where to get a fresh live image
> with 4.12 kernel (it's not important which distro that will be)?
>
> --
> Ivan Sizov
Mounting problem persists:
on 4.13.0 with btrfs-progs v4.11.1 (latest Fedora Rawhide Live)
on 4.10.0 with btrfs-progs v4.9.1 (Ubuntu 17.04 Live)
on 4.9.0 with btrfs-progs v 4.7.3 (Debian 9 Stretch Live)
"btrfs check --readonly" also gives the same output on 4.11, 4.10 and 4.9.

Marc, how did you roll back and fix those errors?

-- 
Ivan Sizov