* Bad superblock when mounting rw, ro mount works @ 2018-06-07 17:07 Daniel Underwood 2018-06-07 20:38 ` Chris Murphy 2018-06-08 3:39 ` Qu Wenruo 0 siblings, 2 replies; 8+ messages in thread From: Daniel Underwood @ 2018-06-07 17:07 UTC (permalink / raw) To: linux-btrfs [-- Attachment #1: Type: text/plain, Size: 1363 bytes --] Hi, I have a raid10-like setup that is failing to mount in rw mode with the error mount: /mnt/media: wrong fs type, bad option, bad superblock on /dev/mapper/em1, missing codepage or helper program, or other error read-only mounts seem to work and the files seem to be there. I started having issues after a system crash during the process of deleting a number of large files. After this (Ubuntu 16.04/Kernel 4.4), any attempt to mount the array in rw mode would cause a similar crash. I did an upgrade to Ubuntu 18.04/Kernel 4.15 and now get the error above. I have looked through a variety of posts on the mailing list, but couldn't find anything with the same issue. I have done a scrub on the array that resulted in 6 verify errors with dmesg showing something about extent trees. It didn't list them as uncorrectable errors, but couldn't correct them either as I can't mount in rw. I also tried `btrfs rescue zero-log /dev/mapper/em1`, which changes the above to (say) em5, but then zero-log on em5 causes it to go back to em1. Any direction would be appreciated. From what I could tell, my next steps would be a check with --repair or --init-extent-tree, though I'm reluctant to try those without being explicitly told to do so. I have attached a dmesg.log file and hope I haven't greped out anything important. Thanks, Daniel -- Daniel Underwood [-- Attachment #2: dmesg.log --] [-- Type: application/octet-stream, Size: 53851 bytes --] [95639.431822] BTRFS error (device dm-11): unable to find ref byte nr 73691507081216 parent 0 root 7 owner 0 offset 0 [95639.432428] ------------[ cut here ]------------ [95639.432429] BTRFS: Transaction aborted (error -2) [95639.432466] WARNING: CPU: 2 PID: 3503 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6980 __btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [95639.432466] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [95639.432511] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [95639.432566] CPU: 2 PID: 3503 Comm: mount Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [95639.432567] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [95639.432585] RIP: 0010:__btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [95639.432586] RSP: 0018:ffffaaf089f8b610 EFLAGS: 00010286 [95639.432588] RAX: 0000000000000000 RBX: 00004305a41a4000 RCX: 0000000000000006 [95639.432590] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff9ebc1f296490 [95639.432591] RBP: ffffaaf089f8b6b8 R08: 0000000000000001 R09: 0000000000002731 [95639.432593] R10: 0000000000000002 R11: 0000000000000000 R12: ffff9eb907020000 [95639.432594] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9ebb3f36d700 [95639.432597] FS: 00007f010a095080(0000) GS:ffff9ebc1f280000(0000) knlGS:0000000000000000 [95639.432598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [95639.432600] CR2: 00007f5373821d30 CR3: 000000015e6a6002 CR4: 00000000000606e0 [95639.432601] Call Trace: [95639.432627] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [95639.432647] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [95639.432667] ? btree_set_page_dirty+0xe/0x10 [btrfs] [95639.432671] ? set_page_dirty+0x5e/0xb0 [95639.432690] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [95639.432730] ? reserve_metadata_bytes+0xb66/0xc00 [btrfs] [95639.432751] btrfs_evict_inode+0x4bd/0x5c0 [btrfs] [95639.432755] evict+0xca/0x1a0 [95639.432757] iput+0x156/0x220 [95639.432776] btrfs_orphan_cleanup+0x209/0x490 [btrfs] [95639.432795] ? btrfs_insert_fs_root+0x52/0x80 [btrfs] [95639.432816] open_ctree+0x211b/0x2440 [btrfs] [95639.432833] btrfs_mount+0xe47/0xf60 [btrfs] [95639.432849] ? btrfs_mount+0xe47/0xf60 [btrfs] [95639.432852] ? find_next_bit+0xb/0x10 [95639.432854] ? pcpu_next_unpop+0x3c/0x50 [95639.432858] mount_fs+0x37/0x150 [95639.432861] ? alloc_vfsmnt+0x1b3/0x230 [95639.432865] vfs_kern_mount.part.23+0x5d/0x110 [95639.432869] vfs_kern_mount+0x13/0x20 [95639.432884] btrfs_mount+0x199/0xf60 [btrfs] [95639.432887] ? find_next_bit+0xb/0x10 [95639.432889] ? pcpu_next_unpop+0x3c/0x50 [95639.432893] mount_fs+0x37/0x150 [95639.432896] ? alloc_vfsmnt+0x1b3/0x230 [95639.432899] vfs_kern_mount.part.23+0x5d/0x110 [95639.432902] do_mount+0x5ed/0xce0 [95639.432905] ? memdup_user+0x4f/0x80 [95639.432908] SyS_mount+0x98/0xe0 [95639.432911] do_syscall_64+0x73/0x130 [95639.432914] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [95639.432916] RIP: 0033:0x7f01099623ca [95639.432917] RSP: 002b:00007fffa9ce2158 EFLAGS: 00000202 ORIG_RAX: 00000000000000a5 [95639.432920] RAX: ffffffffffffffda RBX: 000055fa63522a40 RCX: 00007f01099623ca [95639.432921] RDX: 000055fa63529010 RSI: 000055fa63522c80 RDI: 000055fa63522d40 [95639.432923] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [95639.432924] R10: 00000000c0ed0000 R11: 0000000000000202 R12: 000055fa63522d40 [95639.432926] R13: 000055fa63529010 R14: 0000000000000000 R15: 00007f0109e838a4 [95639.432928] Code: fe ff 48 8b 45 90 48 8b 40 60 f0 0f ba a8 d8 cd 00 00 02 0f 92 c0 84 c0 41 59 75 11 44 89 ee 48 c7 c7 38 26 94 c0 e8 6e 7e 1f f5 <0f> 0b 48 8b 7d 90 b9 fe ff ff ff ba 44 1b 00 00 48 c7 c6 80 b8 [95639.432973] ---[ end trace a565f676a06845cf ]--- [95639.432976] BTRFS: error (device dm-11) in __btrfs_free_extent:6980: errno=-2 No such entry [95639.433638] BTRFS: error (device dm-11) in btrfs_run_delayed_refs:3076: errno=-2 No such entry [95639.490889] BTRFS error (device dm-11): Error removing orphan entry, stopping orphan cleanup [95639.491223] BTRFS error (device dm-11): could not do orphan cleanup -22 [95639.491566] BTRFS error (device dm-11): commit super ret -30 [95639.492347] BTRFS error (device dm-11): pending csums is 44810240 [95639.492791] BTRFS error (device dm-11): cleaner transaction attach returned -30 [95639.609171] BTRFS error (device dm-11): open_ctree failed [96614.907963] BTRFS info (device dm-11): disk space caching is enabled [96614.907965] BTRFS info (device dm-11): has skinny extents [96615.207692] BTRFS info (device dm-11): bdev /dev/mapper/em3 errs: wr 0, rd 0, flush 0, corrupt 0, gen 3 [96615.207702] BTRFS info (device dm-11): bdev /dev/mapper/em5 errs: wr 164286, rd 3444262, flush 2110, corrupt 3, gen 181 [96778.188543] WARNING: CPU: 7 PID: 3725 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6974 __btrfs_free_extent.isra.63+0x1fc/0xdd0 [btrfs] [96778.188545] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [96778.188595] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [96778.188660] CPU: 7 PID: 3725 Comm: kworker/u16:1 Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [96778.188661] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [96778.188690] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [96778.188710] RIP: 0010:__btrfs_free_extent.isra.63+0x1fc/0xdd0 [btrfs] [96778.188711] RSP: 0018:ffffaaf081e9fbe8 EFLAGS: 00010246 [96778.188714] RAX: 00000000fffffffe RBX: 00004305a41a4000 RCX: 0000000000000000 [96778.188715] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9eb9020ab4c8 [96778.188717] RBP: ffffaaf081e9fc90 R08: ffff9eb9fd0c7000 R09: 0000000000000046 [96778.188718] R10: 0000000000000000 R11: ffffaaf081e9fb54 R12: ffff9eb907020000 [96778.188720] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9eb9fd0c7000 [96778.188722] FS: 0000000000000000(0000) GS:ffff9ebc1f3c0000(0000) knlGS:0000000000000000 [96778.188724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [96778.188725] CR2: 00007f17ef0b85d0 CR3: 00000003fe40a004 CR4: 00000000000606e0 [96778.188727] Call Trace: [96778.188756] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [96778.188776] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [96778.188797] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [96778.188814] delayed_ref_async_start+0x98/0xb0 [btrfs] [96778.188838] normal_work_helper+0x7f/0x300 [btrfs] [96778.188859] btrfs_extent_refs_helper+0x12/0x20 [btrfs] [96778.188864] process_one_work+0x1de/0x410 [96778.188867] worker_thread+0x32/0x410 [96778.188870] kthread+0x121/0x140 [96778.188873] ? process_one_work+0x410/0x410 [96778.188876] ? kthread_create_worker_on_cpu+0x70/0x70 [96778.188880] ret_from_fork+0x35/0x40 [96778.188882] Code: ff 75 10 48 89 d9 48 c7 45 a0 00 00 00 00 e8 5c c0 ff ff 41 89 c5 58 5a 45 85 ed 0f 84 81 00 00 00 41 83 fd fe 0f 85 04 ff ff ff <0f> 0b 49 8b 3f e8 ea da 00 00 4c 8b 4d 10 4c 8b 85 78 ff ff ff [96778.188928] ---[ end trace a565f676a06845d0 ]--- [96778.188933] BTRFS info (device dm-11): leaf 49664442974208 total ptrs 204 free space 4460 [96778.189846] BTRFS error (device dm-11): unable to find ref byte nr 73691507081216 parent 0 root 7 owner 0 offset 0 [96778.190749] ------------[ cut here ]------------ [96778.190750] BTRFS: Transaction aborted (error -2) [96778.190787] WARNING: CPU: 7 PID: 3725 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6980 __btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [96778.190787] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [96778.190832] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [96778.190887] CPU: 7 PID: 3725 Comm: kworker/u16:1 Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [96778.190888] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [96778.190910] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [96778.190928] RIP: 0010:__btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [96778.190930] RSP: 0018:ffffaaf081e9fbe8 EFLAGS: 00010286 [96778.190932] RAX: 0000000000000000 RBX: 00004305a41a4000 RCX: 0000000000000006 [96778.190934] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9ebc1f3d6490 [96778.190935] RBP: ffffaaf081e9fc90 R08: 0000000000000001 R09: 00000000000029fc [96778.190937] R10: 0000000000000002 R11: 0000000000000000 R12: ffff9eb907020000 [96778.190938] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9eb9fd0c7000 [96778.190940] FS: 0000000000000000(0000) GS:ffff9ebc1f3c0000(0000) knlGS:0000000000000000 [96778.190942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [96778.190943] CR2: 00007f17ef0b85d0 CR3: 00000003fe40a004 CR4: 00000000000606e0 [96778.190944] Call Trace: [96778.190969] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [96778.190988] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [96778.191008] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [96778.191026] delayed_ref_async_start+0x98/0xb0 [btrfs] [96778.191047] normal_work_helper+0x7f/0x300 [btrfs] [96778.191067] btrfs_extent_refs_helper+0x12/0x20 [btrfs] [96778.191070] process_one_work+0x1de/0x410 [96778.191073] worker_thread+0x32/0x410 [96778.191076] kthread+0x121/0x140 [96778.191079] ? process_one_work+0x410/0x410 [96778.191082] ? kthread_create_worker_on_cpu+0x70/0x70 [96778.191085] ret_from_fork+0x35/0x40 [96778.191087] Code: fe ff 48 8b 45 90 48 8b 40 60 f0 0f ba a8 d8 cd 00 00 02 0f 92 c0 84 c0 41 59 75 11 44 89 ee 48 c7 c7 38 26 94 c0 e8 6e 7e 1f f5 <0f> 0b 48 8b 7d 90 b9 fe ff ff ff ba 44 1b 00 00 48 c7 c6 80 b8 [96778.191132] ---[ end trace a565f676a06845d1 ]--- [96778.191136] BTRFS: error (device dm-11) in __btrfs_free_extent:6980: errno=-2 No such entry [96778.192090] BTRFS: error (device dm-11) in btrfs_run_delayed_refs:3076: errno=-2 No such entry [96778.245582] BTRFS error (device dm-11): Error removing orphan entry, stopping orphan cleanup [96778.245590] BTRFS info (device dm-11): delayed_refs has NO entry [96778.246035] BTRFS error (device dm-11): could not do orphan cleanup -22 [96778.246495] BTRFS error (device dm-11): commit super ret -30 [96778.247027] BTRFS error (device dm-11): cleaner transaction attach returned -30 [96778.583103] BTRFS error (device dm-11): open_ctree failed [103826.659536] mlx4_en: enp2s0: Link Down [103898.564227] mlx4_en: enp2s0: Link Up [105750.300738] mlx4_en: enp2s0: Link Down [106134.258941] mlx4_en: enp2s0: Link Up [106134.308952] mlx4_en: enp2s0: Link Down [106134.358902] mlx4_en: enp2s0: Link Up [111045.066139] mlx4_en: enp2s0: Link Down [147344.448018] mlx4_en: enp2s0: Link Up [147521.378303] BTRFS info (device dm-11): disk space caching is enabled [147521.378306] BTRFS info (device dm-11): has skinny extents [147521.688090] BTRFS info (device dm-11): bdev /dev/mapper/em3 errs: wr 0, rd 0, flush 0, corrupt 0, gen 3 [147521.688100] BTRFS info (device dm-11): bdev /dev/mapper/em5 errs: wr 164286, rd 3444262, flush 2110, corrupt 3, gen 181 [147681.165701] WARNING: CPU: 0 PID: 11627 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6974 __btrfs_free_extent.isra.63+0x1fc/0xdd0 [btrfs] [147681.165703] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [147681.165753] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [147681.165817] CPU: 0 PID: 11627 Comm: kworker/u16:3 Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [147681.165818] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [147681.165846] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [147681.165866] RIP: 0010:__btrfs_free_extent.isra.63+0x1fc/0xdd0 [btrfs] [147681.165868] RSP: 0018:ffffaaf08247bbe8 EFLAGS: 00010246 [147681.165870] RAX: 00000000fffffffe RBX: 00004305a41a4000 RCX: 0000000000000000 [147681.165872] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ebb8bf2c690 [147681.165874] RBP: ffffaaf08247bc90 R08: ffff9eb9fd0c7bd0 R09: 0000000000000046 [147681.165875] R10: 0000000000000000 R11: ffffaaf08247bb54 R12: ffff9eb9847f0000 [147681.165877] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9eb9fd0c7bd0 [147681.165879] FS: 0000000000000000(0000) GS:ffff9ebc1f200000(0000) knlGS:0000000000000000 [147681.165881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [147681.165882] CR2: 0000565064000920 CR3: 00000003fe40a005 CR4: 00000000000606f0 [147681.165884] Call Trace: [147681.165913] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [147681.165933] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [147681.165940] ? mempool_free_slab+0x17/0x20 [147681.165943] ? mempool_free+0x2f/0x90 [147681.165948] ? update_curr+0xf5/0x1d0 [147681.165967] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [147681.165985] delayed_ref_async_start+0x98/0xb0 [btrfs] [147681.166009] normal_work_helper+0x7f/0x300 [btrfs] [147681.166031] btrfs_extent_refs_helper+0x12/0x20 [btrfs] [147681.166034] process_one_work+0x1de/0x410 [147681.166037] worker_thread+0x32/0x410 [147681.166041] kthread+0x121/0x140 [147681.166043] ? process_one_work+0x410/0x410 [147681.166047] ? kthread_create_worker_on_cpu+0x70/0x70 [147681.166051] ret_from_fork+0x35/0x40 [147681.166052] Code: ff 75 10 48 89 d9 48 c7 45 a0 00 00 00 00 e8 5c c0 ff ff 41 89 c5 58 5a 45 85 ed 0f 84 81 00 00 00 41 83 fd fe 0f 85 04 ff ff ff <0f> 0b 49 8b 3f e8 ea da 00 00 4c 8b 4d 10 4c 8b 85 78 ff ff ff [147681.166098] ---[ end trace a565f676a06845d2 ]--- [147681.166103] BTRFS info (device dm-11): leaf 49664027328512 total ptrs 204 free space 4460 [147681.167018] BTRFS error (device dm-11): unable to find ref byte nr 73691507081216 parent 0 root 7 owner 0 offset 0 [147681.169264] ------------[ cut here ]------------ [147681.169266] BTRFS: Transaction aborted (error -2) [147681.169303] WARNING: CPU: 0 PID: 11627 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6980 __btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [147681.169308] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [147681.169359] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [147681.169415] CPU: 0 PID: 11627 Comm: kworker/u16:3 Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [147681.169416] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [147681.169440] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [147681.169458] RIP: 0010:__btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [147681.169460] RSP: 0018:ffffaaf08247bbe8 EFLAGS: 00010286 [147681.169462] RAX: 0000000000000000 RBX: 00004305a41a4000 RCX: 0000000000000006 [147681.169464] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9ebc1f216490 [147681.169465] RBP: ffffaaf08247bc90 R08: 0000000000000001 R09: 0000000000002cd2 [147681.169467] R10: 0000000000000002 R11: 0000000000000000 R12: ffff9eb9847f0000 [147681.169468] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9eb9fd0c7bd0 [147681.169470] FS: 0000000000000000(0000) GS:ffff9ebc1f200000(0000) knlGS:0000000000000000 [147681.169472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [147681.169473] CR2: 0000565064000920 CR3: 00000003fe40a005 CR4: 00000000000606f0 [147681.169474] Call Trace: [147681.169500] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [147681.169519] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [147681.169527] ? mempool_free_slab+0x17/0x20 [147681.169530] ? mempool_free+0x2f/0x90 [147681.169534] ? update_curr+0xf5/0x1d0 [147681.169553] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [147681.169571] delayed_ref_async_start+0x98/0xb0 [btrfs] [147681.169593] normal_work_helper+0x7f/0x300 [btrfs] [147681.169613] btrfs_extent_refs_helper+0x12/0x20 [btrfs] [147681.169616] process_one_work+0x1de/0x410 [147681.169619] worker_thread+0x32/0x410 [147681.169622] kthread+0x121/0x140 [147681.169625] ? process_one_work+0x410/0x410 [147681.169628] ? kthread_create_worker_on_cpu+0x70/0x70 [147681.169631] ret_from_fork+0x35/0x40 [147681.169633] Code: fe ff 48 8b 45 90 48 8b 40 60 f0 0f ba a8 d8 cd 00 00 02 0f 92 c0 84 c0 41 59 75 11 44 89 ee 48 c7 c7 38 26 94 c0 e8 6e 7e 1f f5 <0f> 0b 48 8b 7d 90 b9 fe ff ff ff ba 44 1b 00 00 48 c7 c6 80 b8 [147681.169678] ---[ end trace a565f676a06845d3 ]--- [147681.169682] BTRFS: error (device dm-11) in __btrfs_free_extent:6980: errno=-2 No such entry [147681.171788] BTRFS: error (device dm-11) in btrfs_run_delayed_refs:3076: errno=-2 No such entry [147681.515206] BTRFS error (device dm-11): Error removing orphan entry, stopping orphan cleanup [147681.517379] BTRFS error (device dm-11): could not do orphan cleanup -22 [147681.519556] BTRFS error (device dm-11): commit super ret -30 [147681.570580] BTRFS error (device dm-11): cleaner transaction attach returned -30 [147681.890468] BTRFS error (device dm-11): open_ctree failed [148054.941961] BTRFS info (device dm-11): disabling disk space caching [148054.941967] BTRFS info (device dm-11): force clearing of disk cache [148054.941971] BTRFS info (device dm-11): has skinny extents [148055.346429] BTRFS info (device dm-11): bdev /dev/mapper/em3 errs: wr 0, rd 0, flush 0, corrupt 0, gen 3 [148055.346439] BTRFS info (device dm-11): bdev /dev/mapper/em5 errs: wr 164286, rd 3444262, flush 2110, corrupt 3, gen 181 [148219.310882] WARNING: CPU: 5 PID: 11243 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6974 __btrfs_free_extent.isra.63+0x1fc/0xdd0 [btrfs] [148219.310884] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [148219.310933] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [148219.310997] CPU: 5 PID: 11243 Comm: kworker/u16:1 Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [148219.310999] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [148219.311027] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [148219.311048] RIP: 0010:__btrfs_free_extent.isra.63+0x1fc/0xdd0 [btrfs] [148219.311049] RSP: 0018:ffffaaf081b7fbe8 EFLAGS: 00010246 [148219.311052] RAX: 00000000fffffffe RBX: 00004305a41a4000 RCX: 0000000000000000 [148219.311053] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9eb93a33f180 [148219.311055] RBP: ffffaaf081b7fc90 R08: ffff9eb95b403b60 R09: 0000000000000046 [148219.311056] R10: 0000000000000000 R11: ffffaaf081b7fb54 R12: ffff9eba1c5b0000 [148219.311058] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9eb95b403b60 [148219.311060] FS: 0000000000000000(0000) GS:ffff9ebc1f340000(0000) knlGS:0000000000000000 [148219.311062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [148219.311063] CR2: 000055767f714458 CR3: 00000003fe40a005 CR4: 00000000000606e0 [148219.311065] Call Trace: [148219.311094] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [148219.311114] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [148219.311119] ? intel_pstate_update_util+0x1fd/0x390 [148219.311123] ? __update_load_avg_se.isra.38+0x1c0/0x1d0 [148219.311126] ? __update_load_avg_se.isra.38+0x1c0/0x1d0 [148219.311130] ? update_curr+0xf5/0x1d0 [148219.311150] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [148219.311168] delayed_ref_async_start+0x98/0xb0 [btrfs] [148219.311191] normal_work_helper+0x7f/0x300 [btrfs] [148219.311213] btrfs_extent_refs_helper+0x12/0x20 [btrfs] [148219.311217] process_one_work+0x1de/0x410 [148219.311219] worker_thread+0x32/0x410 [148219.311223] kthread+0x121/0x140 [148219.311225] ? process_one_work+0x410/0x410 [148219.311229] ? kthread_create_worker_on_cpu+0x70/0x70 [148219.311232] ? do_syscall_64+0x73/0x130 [148219.311235] ? SyS_exit_group+0x14/0x20 [148219.311240] ret_from_fork+0x35/0x40 [148219.311242] Code: ff 75 10 48 89 d9 48 c7 45 a0 00 00 00 00 e8 5c c0 ff ff 41 89 c5 58 5a 45 85 ed 0f 84 81 00 00 00 41 83 fd fe 0f 85 04 ff ff ff <0f> 0b 49 8b 3f e8 ea da 00 00 4c 8b 4d 10 4c 8b 85 78 ff ff ff [148219.311287] ---[ end trace a565f676a06845d4 ]--- [148219.311292] BTRFS info (device dm-11): leaf 49664426655744 total ptrs 204 free space 4460 [148219.312209] BTRFS error (device dm-11): unable to find ref byte nr 73691507081216 parent 0 root 7 owner 0 offset 0 [148219.314260] ------------[ cut here ]------------ [148219.314265] BTRFS: Transaction aborted (error -2) [148219.314320] WARNING: CPU: 5 PID: 11243 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6980 __btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [148219.314330] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [148219.314529] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [148219.314611] CPU: 5 PID: 11243 Comm: kworker/u16:1 Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [148219.314613] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [148219.314650] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [148219.314680] RIP: 0010:__btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [148219.314682] RSP: 0018:ffffaaf081b7fbe8 EFLAGS: 00010286 [148219.314686] RAX: 0000000000000000 RBX: 00004305a41a4000 RCX: 0000000000000006 [148219.314688] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9ebc1f356490 [148219.314690] RBP: ffffaaf081b7fc90 R08: 0000000000000001 R09: 0000000000002f8c [148219.314693] R10: 0000000000000002 R11: 0000000000000000 R12: ffff9eba1c5b0000 [148219.314695] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9eb95b403b60 [148219.314698] FS: 0000000000000000(0000) GS:ffff9ebc1f340000(0000) knlGS:0000000000000000 [148219.314701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [148219.314703] CR2: 000055767f714458 CR3: 00000003fe40a005 CR4: 00000000000606e0 [148219.314705] Call Trace: [148219.314744] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [148219.314774] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [148219.314782] ? intel_pstate_update_util+0x1fd/0x390 [148219.314792] ? __update_load_avg_se.isra.38+0x1c0/0x1d0 [148219.314799] ? __update_load_avg_se.isra.38+0x1c0/0x1d0 [148219.314808] ? update_curr+0xf5/0x1d0 [148219.314837] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [148219.314866] delayed_ref_async_start+0x98/0xb0 [btrfs] [148219.314906] normal_work_helper+0x7f/0x300 [btrfs] [148219.314939] btrfs_extent_refs_helper+0x12/0x20 [btrfs] [148219.314948] process_one_work+0x1de/0x410 [148219.314958] worker_thread+0x32/0x410 [148219.314965] kthread+0x121/0x140 [148219.314971] ? process_one_work+0x410/0x410 [148219.314979] ? kthread_create_worker_on_cpu+0x70/0x70 [148219.314986] ? do_syscall_64+0x73/0x130 [148219.314994] ? SyS_exit_group+0x14/0x20 [148219.315004] ret_from_fork+0x35/0x40 [148219.315009] Code: fe ff 48 8b 45 90 48 8b 40 60 f0 0f ba a8 d8 cd 00 00 02 0f 92 c0 84 c0 41 59 75 11 44 89 ee 48 c7 c7 38 26 94 c0 e8 6e 7e 1f f5 <0f> 0b 48 8b 7d 90 b9 fe ff ff ff ba 44 1b 00 00 48 c7 c6 80 b8 [148219.315236] ---[ end trace a565f676a06845d5 ]--- [148219.315241] BTRFS: error (device dm-11) in __btrfs_free_extent:6980: errno=-2 No such entry [148219.317700] BTRFS: error (device dm-11) in btrfs_run_delayed_refs:3076: errno=-2 No such entry [148219.462857] BTRFS error (device dm-11): Error removing orphan entry, stopping orphan cleanup [148219.464634] BTRFS error (device dm-11): could not do orphan cleanup -22 [148219.466438] BTRFS error (device dm-11): commit super ret -30 [148219.468327] BTRFS error (device dm-11): cleaner transaction attach returned -30 [148219.580656] BTRFS error (device dm-11): open_ctree failed [148707.539851] BTRFS info (device dm-11): disabling disk space caching [148707.539855] BTRFS info (device dm-11): force clearing of disk cache [148707.539859] BTRFS info (device dm-11): trying to use backup root at mount time [148707.539862] BTRFS info (device dm-11): has skinny extents [148708.034249] BTRFS info (device dm-11): bdev /dev/mapper/em3 errs: wr 0, rd 0, flush 0, corrupt 0, gen 3 [148708.034259] BTRFS info (device dm-11): bdev /dev/mapper/em5 errs: wr 164286, rd 3444262, flush 2110, corrupt 3, gen 181 [148871.614524] WARNING: CPU: 1 PID: 11716 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6974 __btrfs_free_extent.isra.63+0x1fc/0xdd0 [btrfs] [148871.614526] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [148871.614575] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [148871.614640] CPU: 1 PID: 11716 Comm: mount Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [148871.614641] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [148871.614661] RIP: 0010:__btrfs_free_extent.isra.63+0x1fc/0xdd0 [btrfs] [148871.614662] RSP: 0018:ffffaaf08216f610 EFLAGS: 00010246 [148871.614665] RAX: 00000000fffffffe RBX: 00004305a41a4000 RCX: 0000000000000000 [148871.614666] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9eba116474c8 [148871.614668] RBP: ffffaaf08216f6b8 R08: ffff9eba050337e0 R09: 0000000000000046 [148871.614669] R10: 0000000000000000 R11: ffffaaf08216f57c R12: ffff9eba1c5b0000 [148871.614671] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9eba050337e0 [148871.614673] FS: 00007fd94bef1080(0000) GS:ffff9ebc1f240000(0000) knlGS:0000000000000000 [148871.614675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [148871.614677] CR2: 00007f1806090010 CR3: 0000000322914001 CR4: 00000000000606e0 [148871.614679] Call Trace: [148871.614709] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [148871.614730] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [148871.614751] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [148871.614793] ? reserve_metadata_bytes+0xb66/0xc00 [btrfs] [148871.614816] btrfs_evict_inode+0x4bd/0x5c0 [btrfs] [148871.614821] evict+0xca/0x1a0 [148871.614824] iput+0x156/0x220 [148871.614842] btrfs_orphan_cleanup+0x209/0x490 [btrfs] [148871.614862] ? btrfs_insert_fs_root+0x52/0x80 [btrfs] [148871.614882] open_ctree+0x211b/0x2440 [btrfs] [148871.614900] btrfs_mount+0xe47/0xf60 [btrfs] [148871.614916] ? btrfs_mount+0xe47/0xf60 [btrfs] [148871.614920] ? find_next_bit+0xb/0x10 [148871.614923] ? pcpu_next_unpop+0x3c/0x50 [148871.614927] mount_fs+0x37/0x150 [148871.614931] ? alloc_vfsmnt+0x1b3/0x230 [148871.614934] vfs_kern_mount.part.23+0x5d/0x110 [148871.614938] vfs_kern_mount+0x13/0x20 [148871.614954] btrfs_mount+0x199/0xf60 [btrfs] [148871.614956] ? find_next_bit+0xb/0x10 [148871.614958] ? pcpu_next_unpop+0x3c/0x50 [148871.614962] mount_fs+0x37/0x150 [148871.614965] ? alloc_vfsmnt+0x1b3/0x230 [148871.614969] vfs_kern_mount.part.23+0x5d/0x110 [148871.614971] do_mount+0x5ed/0xce0 [148871.614974] ? copy_mount_options+0x2c/0x220 [148871.614977] SyS_mount+0x98/0xe0 [148871.614980] do_syscall_64+0x73/0x130 [148871.614985] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [148871.614987] RIP: 0033:0x7fd94b7bf3ca [148871.614988] RSP: 002b:00007ffe908b2098 EFLAGS: 00000202 ORIG_RAX: 00000000000000a5 [148871.614991] RAX: ffffffffffffffda RBX: 0000559877038a40 RCX: 00007fd94b7bf3ca [148871.614992] RDX: 000055987703f090 RSI: 0000559877038d40 RDI: 0000559877038dc0 [148871.614994] RBP: 0000000000000000 R08: 0000559877038c90 R09: 0000000000000000 [148871.614995] R10: 00000000c0ed0000 R11: 0000000000000202 R12: 0000559877038dc0 [148871.614997] R13: 000055987703f090 R14: 0000000000000000 R15: 00007fd94bce08a4 [148871.614999] Code: ff 75 10 48 89 d9 48 c7 45 a0 00 00 00 00 e8 5c c0 ff ff 41 89 c5 58 5a 45 85 ed 0f 84 81 00 00 00 41 83 fd fe 0f 85 04 ff ff ff <0f> 0b 49 8b 3f e8 ea da 00 00 4c 8b 4d 10 4c 8b 85 78 ff ff ff [148871.615044] ---[ end trace a565f676a06845d6 ]--- [148871.615049] BTRFS info (device dm-11): leaf 49664442646528 total ptrs 204 free space 4460 [148871.615999] BTRFS error (device dm-11): unable to find ref byte nr 73691507081216 parent 0 root 7 owner 0 offset 0 [148871.617913] ------------[ cut here ]------------ [148871.617915] BTRFS: Transaction aborted (error -2) [148871.617952] WARNING: CPU: 1 PID: 11716 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6980 __btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [148871.617953] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [148871.617999] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [148871.618054] CPU: 1 PID: 11716 Comm: mount Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [148871.618055] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [148871.618073] RIP: 0010:__btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [148871.618074] RSP: 0018:ffffaaf08216f610 EFLAGS: 00010286 [148871.618076] RAX: 0000000000000000 RBX: 00004305a41a4000 RCX: 0000000000000006 [148871.618078] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff9ebc1f256490 [148871.618079] RBP: ffffaaf08216f6b8 R08: 0000000000000001 R09: 000000000000325c [148871.618081] R10: 0000000000000002 R11: 0000000000000000 R12: ffff9eba1c5b0000 [148871.618082] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9eba050337e0 [148871.618085] FS: 00007fd94bef1080(0000) GS:ffff9ebc1f240000(0000) knlGS:0000000000000000 [148871.618086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [148871.618088] CR2: 00007f1806090010 CR3: 0000000322914001 CR4: 00000000000606e0 [148871.618089] Call Trace: [148871.618116] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [148871.618136] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [148871.618156] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [148871.618197] ? reserve_metadata_bytes+0xb66/0xc00 [btrfs] [148871.618219] btrfs_evict_inode+0x4bd/0x5c0 [btrfs] [148871.618223] evict+0xca/0x1a0 [148871.618225] iput+0x156/0x220 [148871.618244] btrfs_orphan_cleanup+0x209/0x490 [btrfs] [148871.618264] ? btrfs_insert_fs_root+0x52/0x80 [btrfs] [148871.618284] open_ctree+0x211b/0x2440 [btrfs] [148871.618301] btrfs_mount+0xe47/0xf60 [btrfs] [148871.618317] ? btrfs_mount+0xe47/0xf60 [btrfs] [148871.618320] ? find_next_bit+0xb/0x10 [148871.618322] ? pcpu_next_unpop+0x3c/0x50 [148871.618326] mount_fs+0x37/0x150 [148871.618330] ? alloc_vfsmnt+0x1b3/0x230 [148871.618333] vfs_kern_mount.part.23+0x5d/0x110 [148871.618337] vfs_kern_mount+0x13/0x20 [148871.618352] btrfs_mount+0x199/0xf60 [btrfs] [148871.618355] ? find_next_bit+0xb/0x10 [148871.618357] ? pcpu_next_unpop+0x3c/0x50 [148871.618361] mount_fs+0x37/0x150 [148871.618364] ? alloc_vfsmnt+0x1b3/0x230 [148871.618367] vfs_kern_mount.part.23+0x5d/0x110 [148871.618370] do_mount+0x5ed/0xce0 [148871.618372] ? copy_mount_options+0x2c/0x220 [148871.618375] SyS_mount+0x98/0xe0 [148871.618378] do_syscall_64+0x73/0x130 [148871.618381] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [148871.618383] RIP: 0033:0x7fd94b7bf3ca [148871.618385] RSP: 002b:00007ffe908b2098 EFLAGS: 00000202 ORIG_RAX: 00000000000000a5 [148871.618387] RAX: ffffffffffffffda RBX: 0000559877038a40 RCX: 00007fd94b7bf3ca [148871.618389] RDX: 000055987703f090 RSI: 0000559877038d40 RDI: 0000559877038dc0 [148871.618390] RBP: 0000000000000000 R08: 0000559877038c90 R09: 0000000000000000 [148871.618392] R10: 00000000c0ed0000 R11: 0000000000000202 R12: 0000559877038dc0 [148871.618393] R13: 000055987703f090 R14: 0000000000000000 R15: 00007fd94bce08a4 [148871.618395] Code: fe ff 48 8b 45 90 48 8b 40 60 f0 0f ba a8 d8 cd 00 00 02 0f 92 c0 84 c0 41 59 75 11 44 89 ee 48 c7 c7 38 26 94 c0 e8 6e 7e 1f f5 <0f> 0b 48 8b 7d 90 b9 fe ff ff ff ba 44 1b 00 00 48 c7 c6 80 b8 [148871.618440] ---[ end trace a565f676a06845d7 ]--- [148871.618474] BTRFS: error (device dm-11) in __btrfs_free_extent:6980: errno=-2 No such entry [148871.620391] BTRFS: error (device dm-11) in btrfs_run_delayed_refs:3076: errno=-2 No such entry [148871.671775] BTRFS error (device dm-11): Error removing orphan entry, stopping orphan cleanup [148871.672641] BTRFS error (device dm-11): could not do orphan cleanup -22 [148871.672874] BTRFS error (device dm-11): pending csums is 490401792 [148871.674904] BTRFS error (device dm-11): commit super ret -30 [148871.676079] BTRFS error (device dm-11): cleaner transaction attach returned -30 [148872.055414] BTRFS error (device dm-11): open_ctree failed [149310.279091] BTRFS info (device dm-11): disabling disk space caching [149310.279096] BTRFS info (device dm-11): force clearing of disk cache [149310.279100] BTRFS info (device dm-11): trying to use backup root at mount time [149310.279103] BTRFS info (device dm-11): has skinny extents [149310.745966] BTRFS info (device dm-11): bdev /dev/mapper/em3 errs: wr 0, rd 0, flush 0, corrupt 0, gen 3 [149310.745976] BTRFS info (device dm-11): bdev /dev/mapper/em5 errs: wr 164286, rd 3444262, flush 2110, corrupt 3, gen 181 [149471.569512] WARNING: CPU: 7 PID: 11807 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6974 __btrfs_free_extent.isra.63+0x1fc/0xdd0 [btrfs] [149471.569514] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [149471.569563] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [149471.569627] CPU: 7 PID: 11807 Comm: kworker/u16:2 Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [149471.569629] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [149471.569657] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [149471.569677] RIP: 0010:__btrfs_free_extent.isra.63+0x1fc/0xdd0 [btrfs] [149471.569678] RSP: 0018:ffffaaf08262fbe8 EFLAGS: 00010246 [149471.569681] RAX: 00000000fffffffe RBX: 00004305a41a4000 RCX: 0000000000000000 [149471.569682] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ebafca0a578 [149471.569684] RBP: ffffaaf08262fc90 R08: ffff9eb9fd0c7070 R09: 0000000000000046 [149471.569685] R10: 0000000000000000 R11: ffffaaf08262fb54 R12: ffff9ebaeb7c0000 [149471.569687] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9eb9fd0c7070 [149471.569689] FS: 0000000000000000(0000) GS:ffff9ebc1f3c0000(0000) knlGS:0000000000000000 [149471.569691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [149471.569692] CR2: 000055e9f2c9bcd0 CR3: 00000003fe40a001 CR4: 00000000000606e0 [149471.569694] Call Trace: [149471.569723] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [149471.569743] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [149471.569749] ? mempool_free_slab+0x17/0x20 [149471.569752] ? mempool_free+0x2f/0x90 [149471.569758] ? update_curr+0xf5/0x1d0 [149471.569777] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [149471.569795] delayed_ref_async_start+0x98/0xb0 [btrfs] [149471.569818] normal_work_helper+0x7f/0x300 [btrfs] [149471.569840] btrfs_extent_refs_helper+0x12/0x20 [btrfs] [149471.569843] process_one_work+0x1de/0x410 [149471.569846] worker_thread+0x32/0x410 [149471.569850] kthread+0x121/0x140 [149471.569852] ? process_one_work+0x410/0x410 [149471.569855] ? kthread_create_worker_on_cpu+0x70/0x70 [149471.569859] ret_from_fork+0x35/0x40 [149471.569861] Code: ff 75 10 48 89 d9 48 c7 45 a0 00 00 00 00 e8 5c c0 ff ff 41 89 c5 58 5a 45 85 ed 0f 84 81 00 00 00 41 83 fd fe 0f 85 04 ff ff ff <0f> 0b 49 8b 3f e8 ea da 00 00 4c 8b 4d 10 4c 8b 85 78 ff ff ff [149471.569907] ---[ end trace a565f676a06845d8 ]--- [149471.569912] BTRFS info (device dm-11): leaf 49663964168192 total ptrs 204 free space 4460 [149471.571196] BTRFS error (device dm-11): unable to find ref byte nr 73691507081216 parent 0 root 7 owner 0 offset 0 [149471.572976] ------------[ cut here ]------------ [149471.572977] BTRFS: Transaction aborted (error -2) [149471.573014] WARNING: CPU: 7 PID: 11807 at /build/linux-lZKWha/linux-4.15.0/fs/btrfs/extent-tree.c:6980 __btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [149471.573017] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_crypt algif_skcipher af_alg nls_iso8859_1 intel_rapl input_leds x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass snd_hda_codec_hdmi intel_cstate intel_rapl_perf serio_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer [149471.573064] snd mei_me soundcore mei lpc_ich shpchp mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core hid_generic usbhid hid i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit ghash_clmulni_intel pcbc mxm_wmi drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd firewire_ohci glue_helper fb_sys_fops cryptd psmouse mpt3sas ahci drm mlx4_core r8169 firewire_core libahci mii raid_class crc_itu_t devlink scsi_transport_sas wmi video [149471.573121] CPU: 7 PID: 11807 Comm: kworker/u16:2 Tainted: G W OE 4.15.0-22-generic #24-Ubuntu [149471.573123] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68X-UD3H-B3, BIOS U1n 07/11/2016 [149471.573145] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [149471.573163] RIP: 0010:__btrfs_free_extent.isra.63+0x252/0xdd0 [btrfs] [149471.573167] RSP: 0018:ffffaaf08262fbe8 EFLAGS: 00010286 [149471.573169] RAX: 0000000000000000 RBX: 00004305a41a4000 RCX: 0000000000000006 [149471.573171] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9ebc1f3d6490 [149471.573172] RBP: ffffaaf08262fc90 R08: 0000000000000001 R09: 000000000000352a [149471.573174] R10: 0000000000000002 R11: 0000000000000000 R12: ffff9ebaeb7c0000 [149471.573175] R13: 00000000fffffffe R14: 0000000000000000 R15: ffff9eb9fd0c7070 [149471.573177] FS: 0000000000000000(0000) GS:ffff9ebc1f3c0000(0000) knlGS:0000000000000000 [149471.573179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [149471.573180] CR2: 000055e9f2c9bcd0 CR3: 00000003fe40a001 CR4: 00000000000606e0 [149471.573182] Call Trace: [149471.573206] ? btrfs_merge_delayed_refs+0x345/0x3a0 [btrfs] [149471.573226] __btrfs_run_delayed_refs+0x4ed/0x1230 [btrfs] [149471.573232] ? mempool_free_slab+0x17/0x20 [149471.573235] ? mempool_free+0x2f/0x90 [149471.573239] ? update_curr+0xf5/0x1d0 [149471.573258] btrfs_run_delayed_refs+0xff/0x1c0 [btrfs] [149471.573276] delayed_ref_async_start+0x98/0xb0 [btrfs] [149471.573300] normal_work_helper+0x7f/0x300 [btrfs] [149471.573320] btrfs_extent_refs_helper+0x12/0x20 [btrfs] [149471.573323] process_one_work+0x1de/0x410 [149471.573326] worker_thread+0x32/0x410 [149471.573332] kthread+0x121/0x140 [149471.573334] ? process_one_work+0x410/0x410 [149471.573337] ? kthread_create_worker_on_cpu+0x70/0x70 [149471.573340] ret_from_fork+0x35/0x40 [149471.573342] Code: fe ff 48 8b 45 90 48 8b 40 60 f0 0f ba a8 d8 cd 00 00 02 0f 92 c0 84 c0 41 59 75 11 44 89 ee 48 c7 c7 38 26 94 c0 e8 6e 7e 1f f5 <0f> 0b 48 8b 7d 90 b9 fe ff ff ff ba 44 1b 00 00 48 c7 c6 80 b8 [149471.573390] ---[ end trace a565f676a06845d9 ]--- [149471.573393] BTRFS: error (device dm-11) in __btrfs_free_extent:6980: errno=-2 No such entry [149471.574742] BTRFS: error (device dm-11) in btrfs_run_delayed_refs:3076: errno=-2 No such entry [149471.683878] BTRFS info (device dm-11): delayed_refs has NO entry [149471.683882] BTRFS error (device dm-11): Error removing orphan entry, stopping orphan cleanup [149471.683887] BTRFS error (device dm-11): could not do orphan cleanup -22 [149471.683894] BTRFS error (device dm-11): commit super ret -30 [149471.687817] BTRFS error (device dm-11): cleaner transaction attach returned -30 [149472.029885] BTRFS error (device dm-11): open_ctree failed ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Bad superblock when mounting rw, ro mount works 2018-06-07 17:07 Bad superblock when mounting rw, ro mount works Daniel Underwood @ 2018-06-07 20:38 ` Chris Murphy 2018-06-07 20:50 ` Chris Murphy 2018-06-08 3:39 ` Qu Wenruo 1 sibling, 1 reply; 8+ messages in thread From: Chris Murphy @ 2018-06-07 20:38 UTC (permalink / raw) To: Daniel Underwood; +Cc: Btrfs BTRFS So you zero'd the log? Why? You need to slow down and stop making changes to your file system unless you hate your data and want to lose it. The log is there to help protect your data, you don't zero the log except under very specific situations, none of which appear to apply to your situation. 'btrfs check' is safe, --repair may or may not be safe, you didn't say what version of btrfs-progs you're using, I would consider 4.14.1 at the oldest, it should be straightforward with any distro to get something relatively new, maybe from a testing repo. Current is 4.16. And good god don't use --init-extent-tree, that's a hammer. The absolute safest way to troubleshoot Btrfs problems is using newer kernel versions, even though 4.15 is not old, there are still tons of bug fixes in every release. Suspicious items from your dmesg: [95639.492347] BTRFS error (device dm-11): pending csums is 44810240 [95639.492791] BTRFS error (device dm-11): cleaner transaction attach returned -30 [95639.609171] BTRFS error (device dm-11): open_ctree failed [96614.907963] BTRFS info (device dm-11): disk space caching is enabled [96614.907965] BTRFS info (device dm-11): has skinny extents [96615.207692] BTRFS info (device dm-11): bdev /dev/mapper/em3 errs: wr 0, rd 0, flush 0, corrupt 0, gen 3 [96615.207702] BTRFS info (device dm-11): bdev /dev/mapper/em5 errs: wr 164286, rd 3444262, flush 2110, corrupt 3, gen 181 OK so it's a multiple device Btrfs volume, and you have one device missing a metric done of reads and writes and flushes, with a few corruptions and generation mismatches. What's wrong with /dev/mapper/em5? Have you done any troubleshooting to figure out what that problem is? Because Btrfs can't fix it, and it can't make up for it. That looks like the device was missing for a long time, and writes were only going to em3. So what's the backstory? Your very first task right now is to mount ro, and update your backups. Don't do anything else until you've done that. It's a testimony to Btrfs that this file system mounts at all, even ro, so take advantage of this fact before you can't mount it anymore. -- Chris Murphy ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Bad superblock when mounting rw, ro mount works 2018-06-07 20:38 ` Chris Murphy @ 2018-06-07 20:50 ` Chris Murphy 2018-06-14 14:23 ` Daniel Underwood 0 siblings, 1 reply; 8+ messages in thread From: Chris Murphy @ 2018-06-07 20:50 UTC (permalink / raw) To: Chris Murphy; +Cc: Daniel Underwood, Btrfs BTRFS On Thu, Jun 7, 2018 at 2:38 PM, Chris Murphy <lists@colorremedies.com> wrote: > Your very first task right now is to mount ro, and update your > backups. Don't do anything else until you've done that. It's a > testimony to Btrfs that this file system mounts at all, even ro, so > take advantage of this fact before you can't mount it anymore. After you've done the backup, you need to find out why one of these devices is being so unreliable. That has to be fixed first. You can recreate a new Btrfs or some other file system, and you'll just run into the exact same problem down the road. Next, it might be useful to see the output from btrfs-progs 4.16.1 'btrfs check' and 'btrfs check --mode=lowmem' both of which are slow, the second one is really slow but is a different implementation so it's helpful to see both outputs. That's safe as long as you do not use --repair. Also we need to see the output from 'btrfs fi us <mountpoint>' with the volume mounted (ro). Off hand I think the most likely outcome is that you get a backup from the ro mounted file system, and you'll have to recreate it from scratch and restore from backups. In other words, no matter what you need a current backup. -- Chris Murphy ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Bad superblock when mounting rw, ro mount works 2018-06-07 20:50 ` Chris Murphy @ 2018-06-14 14:23 ` Daniel Underwood 2018-06-14 14:43 ` Qu Wenruo 0 siblings, 1 reply; 8+ messages in thread From: Daniel Underwood @ 2018-06-14 14:23 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 2830 bytes --] >> Your very first task right now is to mount ro, and update your >> backups. Don't do anything else until you've done that. It's a >> testimony to Btrfs that this file system mounts at all, even ro, so >> take advantage of this fact before you can't mount it anymore. Backups are in place for the important parts, though I'd prefer not to use them if possible. For btrfs-progs, I am using 4.16.1 installed from https://github.com/kdave/btrfs-progs. Regarding em5, it had errors when I initially added it to the array that were due to a faulty SAS card. The card was replaced and I haven't seen errors since until this popped up. Regarding what you said about my dmesg, the numbers are actually still the same (bdev /dev/mapper/em5 errs: wr 164286, rd 3444262, flush 2110, corrupt 3, gen 181) after doing a backup and reboot. I would think they would have changed unless btrfs is just ignoring that disk now. Just to make sure, I've checked to make sure there weren't any loose connections. The logs for check (with and without lowmem mode), `btrfs fi us`, and `smartctl -x` are attached. The checks are only for em5, but I can do other disks if necessary (it's a 6-disk raid10-style setup). Note that there are a lot of errors on the SMART info, but they seem to be back from when I was having hardware issues. --- On Thu, Jun 7, 2018 at 4:50 PM, Chris Murphy <lists@colorremedies.com> wrote: > On Thu, Jun 7, 2018 at 2:38 PM, Chris Murphy <lists@colorremedies.com> wrote: > > >> Your very first task right now is to mount ro, and update your >> backups. Don't do anything else until you've done that. It's a >> testimony to Btrfs that this file system mounts at all, even ro, so >> take advantage of this fact before you can't mount it anymore. > > After you've done the backup, you need to find out why one of these > devices is being so unreliable. That has to be fixed first. You can > recreate a new Btrfs or some other file system, and you'll just run > into the exact same problem down the road. Next, it might be useful to > see the output from btrfs-progs 4.16.1 'btrfs check' and 'btrfs check > --mode=lowmem' both of which are slow, the second one is really slow > but is a different implementation so it's helpful to see both outputs. > That's safe as long as you do not use --repair. > > Also we need to see the output from 'btrfs fi us <mountpoint>' with > the volume mounted (ro). Off hand I think the most likely outcome is > that you get a backup from the ro mounted file system, and you'll have > to recreate it from scratch and restore from backups. In other words, > no matter what you need a current backup. > > -- > Chris Murphy -- Daniel Underwood NCSU Physics 2016 Undergraduate Researcher, Triangle Universities Nuclear Laboratory (704) 244-0244 daniel.underwood13@gmail.com djunder2@ncsu.edu [-- Attachment #2: media-usage.log --] [-- Type: application/octet-stream, Size: 1094 bytes --] Overall: Device size: 43.66TiB Device allocated: 43.09TiB Device unallocated: 583.94GiB Device missing: 0.00B Used: 41.61TiB Free (estimated): 1.02TiB (min: 1.02TiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Data,RAID1: Size:21.52TiB, Used:20.78TiB /dev/mapper/em1 7.17TiB /dev/mapper/em2 7.17TiB /dev/mapper/em3 7.18TiB /dev/mapper/em4 7.18TiB /dev/mapper/em5 7.18TiB /dev/mapper/em6 7.15TiB Metadata,RAID1: Size:30.00GiB, Used:25.65GiB /dev/mapper/em1 10.00GiB /dev/mapper/em2 8.00GiB /dev/mapper/em3 12.00GiB /dev/mapper/em4 10.00GiB /dev/mapper/em5 8.00GiB /dev/mapper/em6 12.00GiB System,RAID1: Size:64.00MiB, Used:3.17MiB /dev/mapper/em2 32.00MiB /dev/mapper/em4 32.00MiB /dev/mapper/em6 64.00MiB Unallocated: /dev/mapper/em1 96.00GiB /dev/mapper/em2 96.97GiB /dev/mapper/em3 90.00GiB /dev/mapper/em4 89.00GiB /dev/mapper/em5 89.00GiB /dev/mapper/em6 122.97GiB [-- Attachment #3: sdh-smart.log --] [-- Type: application/octet-stream, Size: 15524 bytes --] smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-23-generic] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD80EFZX-68UW8N0 Serial Number: VK1E13UY LU WWN Device Id: 5 000cca 3b7d4048b Firmware Version: 83.H0A83 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Thu Jun 14 10:20:31 2018 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM level is: 254 (maximum performance) Rd look-ahead is: Enabled Write cache is: Enabled ATA Security is: Disabled, NOT FROZEN [SEC1] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 101) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: (1175) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0 2 Throughput_Performance P-S--- 131 131 054 - 116 3 Spin_Up_Time POS--- 153 153 024 - 432 (Average 428) 4 Start_Stop_Count -O--C- 100 100 000 - 17 5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0 7 Seek_Error_Rate PO-R-- 100 100 067 - 0 8 Seek_Time_Performance P-S--- 128 128 020 - 18 9 Power_On_Hours -O--C- 100 100 000 - 5597 10 Spin_Retry_Count PO--C- 100 100 060 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 17 22 Helium_Level PO---K 100 100 025 - 100 192 Power-Off_Retract_Count -O--CK 100 100 000 - 234 193 Load_Cycle_Count -O--C- 100 100 000 - 234 194 Temperature_Celsius -O---- 166 166 000 - 36 (Min/Max 25/40) 196 Reallocated_Event_Count -O--CK 100 100 000 - 0 197 Current_Pending_Sector -O---K 100 100 000 - 0 198 Offline_Uncorrectable ---R-- 100 100 000 - 0 199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 3695257 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 1 Comprehensive SMART error log 0x03 GPL R/O 1 Ext. Comprehensive SMART error log 0x04 GPL,SL R/O 8 Device Statistics log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 1 Extended self-test log 0x08 GPL R/O 2 Power Conditions log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 SATA NCQ Queued Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x12 GPL R/O 1 SATA NCQ NON-DATA log 0x15 GPL,SL R/W 1 SATA Rebuild Assist log 0x21 GPL R/O 1 Write stream error log 0x22 GPL R/O 1 Read stream error log 0x24 GPL R/O 256 Current Device Internal Status Data log 0x25 GPL R/O 256 Saved Device Internal Status Data log 0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (1 sectors) Device Error Count: 65535 (device log contains only the most recent 4 errors) CR = Command Register FEATR = Features Register COUNT = Count (was: Sector Count) Register LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8 LH = LBA High (was: Cylinder High) Register ] LBA LM = LBA Mid (was: Cylinder Low) Register ] Register LL = LBA Low (was: Sector Number) Register ] DV = Device (was: Device/Head) Register DC = Device Control Register ER = Error register ST = Status register Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 65535 [2] occurred at disk power-on lifetime: 539 hours (22 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 84 -- 41 00 00 00 00 00 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 04 00 00 18 00 01 54 2b c8 80 40 00 02:20:56.374 READ FPDMA QUEUED 60 00 60 00 38 00 01 54 2b d8 80 40 00 02:20:56.345 READ FPDMA QUEUED 60 04 00 00 30 00 01 54 2b d4 80 40 00 02:20:56.345 READ FPDMA QUEUED 60 04 00 00 28 00 01 54 2b d0 80 40 00 02:20:56.345 READ FPDMA QUEUED 60 04 00 00 20 00 01 54 2b cc 80 40 00 02:20:56.345 READ FPDMA QUEUED Error 65534 [1] occurred at disk power-on lifetime: 539 hours (22 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 84 -- 41 00 00 00 00 00 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 20 00 00 00 00 fd 93 49 00 40 00 02:20:55.851 READ FPDMA QUEUED 2f 00 00 00 01 00 00 00 00 00 10 00 00 02:20:55.826 READ LOG EXT 2f 00 00 00 01 00 00 00 00 00 10 00 00 02:20:55.826 READ LOG EXT 60 00 20 00 00 00 00 fd 93 49 00 40 00 02:20:55.815 READ FPDMA QUEUED 60 00 20 00 00 00 00 96 54 b8 c0 40 00 02:20:55.810 READ FPDMA QUEUED Error 65533 [0] occurred at disk power-on lifetime: 539 hours (22 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 84 -- 41 00 00 00 00 00 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 20 00 00 00 00 fd 93 49 00 40 00 02:20:55.826 READ FPDMA QUEUED 60 00 20 00 00 00 00 96 54 b8 c0 40 00 02:20:55.810 READ FPDMA QUEUED 2f 00 00 00 01 00 00 00 00 00 10 00 00 02:20:55.797 READ LOG EXT 2f 00 00 00 01 00 00 00 00 00 10 00 00 02:20:55.797 READ LOG EXT 60 00 20 00 00 00 00 96 54 b8 c0 40 00 02:20:55.794 READ FPDMA QUEUED Error 65532 [3] occurred at disk power-on lifetime: 539 hours (22 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 84 -- 41 00 00 00 00 00 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 20 00 00 00 00 96 54 b8 c0 40 00 02:20:55.797 READ FPDMA QUEUED 2f 00 00 00 01 00 00 00 00 00 10 00 00 02:20:55.781 READ LOG EXT 2f 00 00 00 01 00 00 00 00 00 10 00 00 02:20:55.781 READ LOG EXT 60 00 20 00 00 00 00 96 54 b8 c0 40 00 02:20:55.774 READ FPDMA QUEUED 60 00 20 00 00 00 00 96 5f 13 20 40 00 02:20:55.736 READ FPDMA QUEUED SMART Extended Self-test Log Version: 1 (1 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 5592 - # 2 Extended offline Completed without error 00% 539 - # 3 Extended offline Interrupted (host reset) 90% 537 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 3 SCT Version (vendor specific): 256 (0x0100) SCT Support Level: 1 Device State: Active (0) Current Temperature: 36 Celsius Power Cycle Min/Max Temperature: 33/39 Celsius Lifetime Min/Max Temperature: 25/40 Celsius Under/Over Temperature Limit Count: 0/0 SCT Temperature History Version: 2 Temperature Sampling Period: 1 minute Temperature Logging Interval: 1 minute Min/Max recommended Temperature: 0/60 Celsius Min/Max Temperature Limit: -40/70 Celsius Temperature History Size (Index): 128 (27) Index Estimated Time Temperature Celsius 28 2018-06-14 08:13 36 ***************** 29 2018-06-14 08:14 36 ***************** 30 2018-06-14 08:15 35 **************** 31 2018-06-14 08:16 36 ***************** ... ..(123 skipped). .. ***************** 27 2018-06-14 10:20 36 ***************** SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) Device Statistics (GP Log 0x04) Page Offset Size Value Flags Description 0x01 ===== = = === == General Statistics (rev 2) == 0x01 0x008 4 17 --- Lifetime Power-On Resets 0x01 0x018 6 200168319922 --- Logical Sectors Written 0x01 0x020 6 375111281 --- Number of Write Commands 0x01 0x028 6 225788674009 --- Logical Sectors Read 0x01 0x030 6 485549497 --- Number of Read Commands 0x01 0x038 6 20151956150 --- Date and Time TimeStamp 0x03 ===== = = === == Rotating Media Statistics (rev 1) == 0x03 0x008 4 5554 --- Spindle Motor Power-on Hours 0x03 0x010 4 5554 --- Head Flying Hours 0x03 0x018 4 234 --- Head Load Events 0x03 0x020 4 0 --- Number of Reallocated Logical Sectors 0x03 0x028 4 597501 --- Read Recovery Attempts 0x03 0x030 4 0 --- Number of Mechanical Start Failures 0x04 ===== = = === == General Errors Statistics (rev 1) == 0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors 0x04 0x010 4 4 --- Resets Between Cmd Acceptance and Completion 0x05 ===== = = === == Temperature Statistics (rev 1) == 0x05 0x008 1 36 --- Current Temperature 0x05 0x010 1 36 N-- Average Short Term Temperature 0x05 0x018 1 36 N-- Average Long Term Temperature 0x05 0x020 1 40 --- Highest Temperature 0x05 0x028 1 25 --- Lowest Temperature 0x05 0x030 1 38 N-- Highest Average Short Term Temperature 0x05 0x038 1 25 N-- Lowest Average Short Term Temperature 0x05 0x040 1 36 N-- Highest Average Long Term Temperature 0x05 0x048 1 25 N-- Lowest Average Long Term Temperature 0x05 0x050 4 0 --- Time in Over-Temperature 0x05 0x058 1 60 --- Specified Maximum Operating Temperature 0x05 0x060 4 0 --- Time in Under-Temperature 0x05 0x068 1 0 --- Specified Minimum Operating Temperature 0x06 ===== = = === == Transport Statistics (rev 1) == 0x06 0x008 4 1 --- Number of Hardware Resets 0x06 0x010 4 54 --- Number of ASR Events 0x06 0x018 4 3695257 --- Number of Interface CRC Errors |||_ C monitored condition met ||__ D supports DSN |___ N normalized value SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 2 0 Command failed due to ICRC error 0x0002 2 0 R_ERR response for data FIS 0x0003 2 0 R_ERR response for device-to-host data FIS 0x0004 2 0 R_ERR response for host-to-device data FIS 0x0005 2 0 R_ERR response for non-data FIS 0x0006 2 0 R_ERR response for device-to-host non-data FIS 0x0007 2 0 R_ERR response for host-to-device non-data FIS 0x0008 2 0 Device-to-host non-data FIS retries 0x0009 2 1 Transition from drive PhyRdy to drive PhyNRdy 0x000a 2 2 Device-to-host register FISes sent due to a COMRESET 0x000b 2 0 CRC errors within host-to-device FIS 0x000d 2 0 Non-CRC errors within host-to-device FIS [-- Attachment #4: em5-check.log --] [-- Type: application/octet-stream, Size: 2013 bytes --] checking extents corrupt extent record: key [73691506982912,169,16384] tree backref 73691506262016 parent 7 root 7 not found in extent tree backref 73691506262016 root 39 not referenced back 0x562c8f7b7100 incorrect global backref count on 73691506262016 found 2 wanted 1 backpointer mismatch on [73691506262016 16384] tree backref 73691506343936 parent 7 root 7 not found in extent tree backref 73691506343936 root 528280977415 not referenced back 0x562c8f7b7600 incorrect global backref count on 73691506343936 found 2 wanted 1 backpointer mismatch on [73691506343936 16384] ref mismatch on [73691506393088 16384] extent item 1, found 0 owner ref check failed [73691506393088 16384] ref mismatch on [73691506491392 16384] extent item 300647710721, found 1 tree backref 73691506982912 parent 7 root 7 not found in extent tree backpointer mismatch on [73691506982912 16384] tree backref 73691507081216 parent 7 root 7 not found in extent tree backref 73691507081216 root 2499805191 not referenced back 0x562c8f7b98a0 incorrect global backref count on 73691507081216 found 2 wanted 1 backpointer mismatch on [73691507081216 16384] tree backref 73691507277824 parent 5 root 5 not found in extent tree backref 73691507277824 root 144115188075855877 not referenced back 0x562c8f7ba0a0 incorrect global backref count on 73691507277824 found 2 wanted 1 backpointer mismatch on [73691507277824 16384] ref mismatch on [73691507425280 16384] extent item 1392508929, found 1 ref mismatch on [73691507539968 16384] extent item 2738188573441261569, found 1 ERROR: errors found in extent allocation tree or chunk allocation checking free space cache checking fs roots check/main.c:1174: leave_shared_node: BUG_ON `node->refs <= 1` triggered, value 1 btrfs(+0x570a1)[0x562bdc33a0a1] btrfs(+0x57147)[0x562bdc33a147] btrfs(+0x5c3d4)[0x562bdc33f3d4] btrfs(cmd_check+0x292c)[0x562bdc34888f] btrfs(main+0x88)[0x562bdc2f62b0] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7fc69a72fb97] btrfs(_start+0x2a)[0x562bdc2f5e4a] [-- Attachment #5: em5-check-lowmem.log --] [-- Type: application/octet-stream, Size: 16888 bytes --] checking extents ERROR: extent[51820655280128, 134217728] referencer count mismatch (root: 5, owner: 122543, offset: 19339149312) wanted: 1, have: 2 ERROR: extent[51835956101120, 134217728] referencer count mismatch (root: 5, owner: 122543, offset: 2701656064) wanted: 1, have: 2 ERROR: extent[65834385539072, 524288] referencer count mismatch (root: 5, owner: 44623, offset: 524288) wanted: 1, have: 4 ERROR: extent[65861179932672, 524288] referencer count mismatch (root: 5, owner: 44624, offset: 7340032) wanted: 1, have: 4 ERROR: extent[65990596698112, 90968064] referencer count mismatch (root: 5, owner: 44623, offset: 805306368) wanted: 133, have: 695 ERROR: extent[65992872951808, 53018624] referencer count mismatch (root: 5, owner: 44624, offset: 930611200) wanted: 130, have: 405 ERROR: extent[66004413411328, 134217728] referencer count mismatch (root: 5, owner: 44626, offset: 671088640) wanted: 3, have: 1024 ERROR: extent [73691506262016 16384] lost referencer (owner: 39, level: 0) ERROR: extent [73691506343936 16384] lost referencer (owner: 528280977415, level: 0) ERROR: extent[73691506982912 169 0] has unknown ref type: 118 ERROR: extent [73691507081216 16384] lost referencer (owner: 2499805191, level: 0) ERROR: extent [73691507277824 16384] lost referencer (owner: 144115188075855877, level: 0) ERROR: extent[73691507605504 16384] backref generation mismatch, wanted: 564673, have: 889757121 ERROR: extent[73691506982912 169 0] has unknown ref type: 118 ERROR: extent[73691506589696 16384] backref generation mismatch, wanted: 713105, have: 712960 ERROR: extent[73691507425280 16384] is referred by other roots than 7 ERROR: extent[73691507703808 16384] backref generation mismatch, wanted: 687611, have: 936748722493750779 ERROR: extent[73691506491392 16384] is referred by other roots than 7 ERROR: errors found in extent allocation tree or chunk allocation checking free space cache checking fs roots ERROR: root 5 EXTENT_DATA[1173015 623292416] gap exists, expected: EXTENT_DATA[1173015 621219840] ERROR: root 5 EXTENT_DATA[1173015 623980544] gap exists, expected: EXTENT_DATA[1173015 621907968] ERROR: root 5 EXTENT_DATA[1173015 624664576] gap exists, expected: EXTENT_DATA[1173015 622592000] ERROR: root 5 EXTENT_DATA[1173015 625348608] gap exists, expected: EXTENT_DATA[1173015 623276032] ERROR: root 5 EXTENT_DATA[1173015 625688576] gap exists, expected: EXTENT_DATA[1173015 623616000] ERROR: root 5 EXTENT_DATA[1173015 626024448] gap exists, expected: EXTENT_DATA[1173015 623951872] ERROR: root 5 EXTENT_DATA[1173015 626360320] gap exists, expected: EXTENT_DATA[1173015 624287744] ERROR: root 5 EXTENT_DATA[1173015 626692096] gap exists, expected: EXTENT_DATA[1173015 624619520] ERROR: root 5 EXTENT_DATA[1173015 627023872] gap exists, expected: EXTENT_DATA[1173015 624951296] ERROR: root 5 EXTENT_DATA[1173015 629280768] gap exists, expected: EXTENT_DATA[1173015 625278976] ERROR: root 5 EXTENT_DATA[1173015 629596160] gap exists, expected: EXTENT_DATA[1173015 625594368] ERROR: root 5 EXTENT_DATA[1173015 629907456] gap exists, expected: EXTENT_DATA[1173015 625905664] ERROR: root 5 EXTENT_DATA[1173015 630530048] gap exists, expected: EXTENT_DATA[1173015 626528256] ERROR: root 5 EXTENT_DATA[1173015 630841344] gap exists, expected: EXTENT_DATA[1173015 626839552] ERROR: root 5 EXTENT_DATA[1173015 631902208] gap exists, expected: EXTENT_DATA[1173015 627146752] ERROR: root 5 EXTENT_DATA[1173015 632193024] gap exists, expected: EXTENT_DATA[1173015 627437568] ERROR: root 5 EXTENT_DATA[1173015 632336384] gap exists, expected: EXTENT_DATA[1173015 627580928] ERROR: root 5 EXTENT_DATA[1173015 632479744] gap exists, expected: EXTENT_DATA[1173015 627724288] ERROR: root 5 EXTENT_DATA[1173015 632762368] gap exists, expected: EXTENT_DATA[1173015 628006912] ERROR: root 5 EXTENT_DATA[1173015 633176064] gap exists, expected: EXTENT_DATA[1173015 628146176] ERROR: root 5 EXTENT_DATA[1173015 633446400] gap exists, expected: EXTENT_DATA[1173015 628416512] ERROR: root 5 EXTENT_DATA[1173015 633716736] gap exists, expected: EXTENT_DATA[1173015 628686848] ERROR: root 5 EXTENT_DATA[1173015 633987072] gap exists, expected: EXTENT_DATA[1173015 628957184] ERROR: root 5 EXTENT_DATA[1173015 634257408] gap exists, expected: EXTENT_DATA[1173015 629227520] ERROR: root 5 EXTENT_DATA[1173015 636690432] gap exists, expected: EXTENT_DATA[1173015 629497856] ERROR: root 5 EXTENT_DATA[1173015 637501440] gap exists, expected: EXTENT_DATA[1173015 629768192] ERROR: root 5 EXTENT_DATA[1173015 637771776] gap exists, expected: EXTENT_DATA[1173015 630038528] ERROR: root 5 EXTENT_DATA[1173015 638042112] gap exists, expected: EXTENT_DATA[1173015 630308864] ERROR: root 5 EXTENT_DATA[1173015 638312448] gap exists, expected: EXTENT_DATA[1173015 630579200] ERROR: root 5 EXTENT_DATA[1173015 638447616] gap exists, expected: EXTENT_DATA[1173015 630714368] ERROR: root 5 EXTENT_DATA[1173015 640311296] gap exists, expected: EXTENT_DATA[1173015 630980608] ERROR: root 5 EXTENT_DATA[1173015 641642496] gap exists, expected: EXTENT_DATA[1173015 631246848] ERROR: root 5 EXTENT_DATA[1173015 643239936] gap exists, expected: EXTENT_DATA[1173015 631513088] ERROR: root 5 EXTENT_DATA[1173015 644038656] gap exists, expected: EXTENT_DATA[1173015 631779328] ERROR: root 5 EXTENT_DATA[1173015 644169728] gap exists, expected: EXTENT_DATA[1173015 631910400] ERROR: root 5 EXTENT_DATA[1173015 644431872] gap exists, expected: EXTENT_DATA[1173015 632172544] ERROR: root 5 EXTENT_DATA[1173015 644694016] gap exists, expected: EXTENT_DATA[1173015 632434688] ERROR: root 5 EXTENT_DATA[1173015 644956160] gap exists, expected: EXTENT_DATA[1173015 632696832] ERROR: root 5 EXTENT_DATA[1173015 645218304] gap exists, expected: EXTENT_DATA[1173015 632958976] ERROR: root 5 EXTENT_DATA[1173015 645480448] gap exists, expected: EXTENT_DATA[1173015 633221120] ERROR: root 5 EXTENT_DATA[1173015 645742592] gap exists, expected: EXTENT_DATA[1173015 633483264] ERROR: root 5 EXTENT_DATA[1173015 646004736] gap exists, expected: EXTENT_DATA[1173015 633745408] ERROR: root 5 EXTENT_DATA[1173015 646266880] gap exists, expected: EXTENT_DATA[1173015 634007552] ERROR: root 5 EXTENT_DATA[1173015 646529024] gap exists, expected: EXTENT_DATA[1173015 634269696] ERROR: root 5 EXTENT_DATA[1173015 646791168] gap exists, expected: EXTENT_DATA[1173015 634531840] ERROR: root 5 EXTENT_DATA[1173015 647053312] gap exists, expected: EXTENT_DATA[1173015 634793984] ERROR: root 5 EXTENT_DATA[1173015 647315456] gap exists, expected: EXTENT_DATA[1173015 635056128] ERROR: root 5 EXTENT_DATA[1173015 647839744] gap exists, expected: EXTENT_DATA[1173015 635318272] ERROR: root 5 EXTENT_DATA[1173015 648101888] gap exists, expected: EXTENT_DATA[1173015 635580416] ERROR: root 5 EXTENT_DATA[1173015 648364032] gap exists, expected: EXTENT_DATA[1173015 635842560] ERROR: root 5 EXTENT_DATA[1173015 648626176] gap exists, expected: EXTENT_DATA[1173015 636104704] ERROR: root 5 EXTENT_DATA[1173015 648888320] gap exists, expected: EXTENT_DATA[1173015 636366848] ERROR: root 5 EXTENT_DATA[1173015 649150464] gap exists, expected: EXTENT_DATA[1173015 636628992] ERROR: root 5 EXTENT_DATA[1173015 649412608] gap exists, expected: EXTENT_DATA[1173015 636891136] ERROR: root 5 EXTENT_DATA[1173015 649674752] gap exists, expected: EXTENT_DATA[1173015 637153280] ERROR: root 5 EXTENT_DATA[1173015 649936896] gap exists, expected: EXTENT_DATA[1173015 637415424] ERROR: root 5 EXTENT_DATA[1173015 650199040] gap exists, expected: EXTENT_DATA[1173015 637677568] ERROR: root 5 EXTENT_DATA[1173015 650461184] gap exists, expected: EXTENT_DATA[1173015 637939712] ERROR: root 5 EXTENT_DATA[1173015 650723328] gap exists, expected: EXTENT_DATA[1173015 638201856] ERROR: root 5 EXTENT_DATA[1173015 650985472] gap exists, expected: EXTENT_DATA[1173015 638464000] ERROR: root 5 EXTENT_DATA[1173015 651247616] gap exists, expected: EXTENT_DATA[1173015 638726144] ERROR: root 5 EXTENT_DATA[1173015 652296192] gap exists, expected: EXTENT_DATA[1173015 638988288] ERROR: root 5 EXTENT_DATA[1173015 652558336] gap exists, expected: EXTENT_DATA[1173015 639250432] ERROR: root 5 EXTENT_DATA[1173015 652820480] gap exists, expected: EXTENT_DATA[1173015 639512576] ERROR: root 5 EXTENT_DATA[1173015 653082624] gap exists, expected: EXTENT_DATA[1173015 639774720] ERROR: root 5 EXTENT_DATA[1173015 653344768] gap exists, expected: EXTENT_DATA[1173015 640036864] ERROR: root 5 EXTENT_DATA[1173015 653869056] gap exists, expected: EXTENT_DATA[1173015 640299008] ERROR: root 5 EXTENT_DATA[1173015 654131200] gap exists, expected: EXTENT_DATA[1173015 640561152] ERROR: root 5 EXTENT_DATA[1173015 654393344] gap exists, expected: EXTENT_DATA[1173015 640823296] ERROR: root 5 EXTENT_DATA[1173015 654655488] gap exists, expected: EXTENT_DATA[1173015 641085440] ERROR: root 5 EXTENT_DATA[1173015 654917632] gap exists, expected: EXTENT_DATA[1173015 641347584] ERROR: root 5 EXTENT_DATA[1173015 655179776] gap exists, expected: EXTENT_DATA[1173015 641609728] ERROR: root 5 EXTENT_DATA[1173015 655441920] gap exists, expected: EXTENT_DATA[1173015 641871872] ERROR: root 5 EXTENT_DATA[1173015 655704064] gap exists, expected: EXTENT_DATA[1173015 642134016] ERROR: root 5 EXTENT_DATA[1173015 655966208] gap exists, expected: EXTENT_DATA[1173015 642396160] ERROR: root 5 EXTENT_DATA[1173015 656228352] gap exists, expected: EXTENT_DATA[1173015 642658304] ERROR: root 5 EXTENT_DATA[1173015 656752640] gap exists, expected: EXTENT_DATA[1173015 642920448] ERROR: root 5 EXTENT_DATA[1173015 657014784] gap exists, expected: EXTENT_DATA[1173015 643182592] ERROR: root 5 EXTENT_DATA[1173015 657276928] gap exists, expected: EXTENT_DATA[1173015 643444736] ERROR: root 5 EXTENT_DATA[1173015 657539072] gap exists, expected: EXTENT_DATA[1173015 643706880] ERROR: root 5 EXTENT_DATA[1173015 657801216] gap exists, expected: EXTENT_DATA[1173015 643969024] ERROR: root 5 EXTENT_DATA[1173015 658063360] gap exists, expected: EXTENT_DATA[1173015 644231168] ERROR: root 5 EXTENT_DATA[1173015 658325504] gap exists, expected: EXTENT_DATA[1173015 644493312] ERROR: root 5 EXTENT_DATA[1173015 658587648] gap exists, expected: EXTENT_DATA[1173015 644755456] ERROR: root 5 EXTENT_DATA[1173015 658849792] gap exists, expected: EXTENT_DATA[1173015 645017600] ERROR: root 5 EXTENT_DATA[1173015 659111936] gap exists, expected: EXTENT_DATA[1173015 645279744] ERROR: root 5 EXTENT_DATA[1173015 659374080] gap exists, expected: EXTENT_DATA[1173015 645541888] ERROR: root 5 EXTENT_DATA[1173015 659636224] gap exists, expected: EXTENT_DATA[1173015 645804032] ERROR: root 5 EXTENT_DATA[1173015 659898368] gap exists, expected: EXTENT_DATA[1173015 646066176] ERROR: root 5 EXTENT_DATA[1173015 660160512] gap exists, expected: EXTENT_DATA[1173015 646328320] ERROR: root 5 EXTENT_DATA[1173015 660422656] gap exists, expected: EXTENT_DATA[1173015 646590464] ERROR: root 5 EXTENT_DATA[1173015 660684800] gap exists, expected: EXTENT_DATA[1173015 646852608] ERROR: root 5 EXTENT_DATA[1173015 660946944] gap exists, expected: EXTENT_DATA[1173015 647114752] ERROR: root 5 EXTENT_DATA[1173015 661209088] gap exists, expected: EXTENT_DATA[1173015 647376896] ERROR: root 5 EXTENT_DATA[1173015 661471232] gap exists, expected: EXTENT_DATA[1173015 647639040] ERROR: root 5 EXTENT_DATA[1173015 661733376] gap exists, expected: EXTENT_DATA[1173015 647901184] ERROR: root 5 EXTENT_DATA[1173015 661995520] gap exists, expected: EXTENT_DATA[1173015 648163328] ERROR: root 5 EXTENT_DATA[1173015 662257664] gap exists, expected: EXTENT_DATA[1173015 648425472] ERROR: root 5 EXTENT_DATA[1173015 662519808] gap exists, expected: EXTENT_DATA[1173015 648687616] ERROR: root 5 EXTENT_DATA[1173015 662781952] gap exists, expected: EXTENT_DATA[1173015 648949760] ERROR: root 5 EXTENT_DATA[1173015 663044096] gap exists, expected: EXTENT_DATA[1173015 649211904] ERROR: root 5 EXTENT_DATA[1173015 663306240] gap exists, expected: EXTENT_DATA[1173015 649474048] ERROR: root 5 EXTENT_DATA[1173015 663568384] gap exists, expected: EXTENT_DATA[1173015 649736192] ERROR: root 5 EXTENT_DATA[1173015 663830528] gap exists, expected: EXTENT_DATA[1173015 649998336] ERROR: root 5 EXTENT_DATA[1173015 664092672] gap exists, expected: EXTENT_DATA[1173015 650260480] ERROR: root 5 EXTENT_DATA[1173015 664354816] gap exists, expected: EXTENT_DATA[1173015 650522624] ERROR: root 5 EXTENT_DATA[1173015 664616960] gap exists, expected: EXTENT_DATA[1173015 650784768] ERROR: root 5 EXTENT_DATA[1173015 664879104] gap exists, expected: EXTENT_DATA[1173015 651046912] ERROR: root 5 EXTENT_DATA[1173015 665927680] gap exists, expected: EXTENT_DATA[1173015 651309056] ERROR: root 5 EXTENT_DATA[1173015 666189824] gap exists, expected: EXTENT_DATA[1173015 651571200] ERROR: root 5 EXTENT_DATA[1173015 666451968] gap exists, expected: EXTENT_DATA[1173015 651833344] ERROR: root 5 EXTENT_DATA[1173015 666714112] gap exists, expected: EXTENT_DATA[1173015 652095488] ERROR: root 5 EXTENT_DATA[1173015 666976256] gap exists, expected: EXTENT_DATA[1173015 652357632] ERROR: root 5 EXTENT_DATA[1173015 667238400] gap exists, expected: EXTENT_DATA[1173015 652619776] ERROR: root 5 EXTENT_DATA[1173015 667500544] gap exists, expected: EXTENT_DATA[1173015 652881920] ERROR: root 5 EXTENT_DATA[1173015 667762688] gap exists, expected: EXTENT_DATA[1173015 653144064] ERROR: root 5 EXTENT_DATA[1173015 668024832] gap exists, expected: EXTENT_DATA[1173015 653406208] ERROR: root 5 EXTENT_DATA[1173015 668286976] gap exists, expected: EXTENT_DATA[1173015 653668352] ERROR: root 5 EXTENT_DATA[1173015 668549120] gap exists, expected: EXTENT_DATA[1173015 653930496] ERROR: root 5 EXTENT_DATA[1173015 668811264] gap exists, expected: EXTENT_DATA[1173015 654192640] ERROR: root 5 EXTENT_DATA[1173015 669073408] gap exists, expected: EXTENT_DATA[1173015 654454784] ERROR: root 5 EXTENT_DATA[1173015 669335552] gap exists, expected: EXTENT_DATA[1173015 654716928] ERROR: root 5 EXTENT_DATA[1173015 669597696] gap exists, expected: EXTENT_DATA[1173015 654979072] ERROR: root 5 EXTENT_DATA[1173015 669859840] gap exists, expected: EXTENT_DATA[1173015 655241216] ERROR: root 5 EXTENT_DATA[1173015 670121984] gap exists, expected: EXTENT_DATA[1173015 655503360] ERROR: root 5 EXTENT_DATA[1173015 670384128] gap exists, expected: EXTENT_DATA[1173015 655765504] ERROR: root 5 EXTENT_DATA[1173015 670646272] gap exists, expected: EXTENT_DATA[1173015 656027648] ERROR: root 5 EXTENT_DATA[1173015 670908416] gap exists, expected: EXTENT_DATA[1173015 656289792] ERROR: root 5 EXTENT_DATA[1173015 671170560] gap exists, expected: EXTENT_DATA[1173015 656551936] ERROR: root 5 EXTENT_DATA[1173015 671432704] gap exists, expected: EXTENT_DATA[1173015 656814080] ERROR: root 5 EXTENT_DATA[1173015 671956992] gap exists, expected: EXTENT_DATA[1173015 657076224] ERROR: root 5 EXTENT_DATA[1173015 672219136] gap exists, expected: EXTENT_DATA[1173015 657338368] ERROR: root 5 EXTENT_DATA[1173015 672481280] gap exists, expected: EXTENT_DATA[1173015 657600512] ERROR: root 5 EXTENT_DATA[1173015 677724160] gap exists, expected: EXTENT_DATA[1173015 657862656] ERROR: root 5 EXTENT_DATA[1173015 677986304] gap exists, expected: EXTENT_DATA[1173015 658124800] ERROR: root 5 EXTENT_DATA[1173015 678510592] gap exists, expected: EXTENT_DATA[1173015 658386944] ERROR: root 5 EXTENT_DATA[1173015 678772736] gap exists, expected: EXTENT_DATA[1173015 658649088] ERROR: root 5 EXTENT_DATA[1173015 679034880] gap exists, expected: EXTENT_DATA[1173015 658911232] ERROR: root 5 EXTENT_DATA[1173015 692666368] gap exists, expected: EXTENT_DATA[1173015 659173376] ERROR: root 5 EXTENT_DATA[1173015 692928512] gap exists, expected: EXTENT_DATA[1173015 659435520] ERROR: root 5 EXTENT_DATA[1173015 693190656] gap exists, expected: EXTENT_DATA[1173015 659697664] ERROR: root 5 EXTENT_DATA[1173015 693452800] gap exists, expected: EXTENT_DATA[1173015 659959808] ERROR: root 5 EXTENT_DATA[1173015 693714944] gap exists, expected: EXTENT_DATA[1173015 660221952] ERROR: root 5 EXTENT_DATA[1173015 693977088] gap exists, expected: EXTENT_DATA[1173015 660484096] ERROR: root 5 EXTENT_DATA[1173015 694239232] gap exists, expected: EXTENT_DATA[1173015 660746240] ERROR: root 5 EXTENT_DATA[1173015 694501376] gap exists, expected: EXTENT_DATA[1173015 661008384] ERROR: root 5 EXTENT_DATA[1173015 694763520] gap exists, expected: EXTENT_DATA[1173015 661270528] ERROR: root 5 INODE[1570999] nbytes 25251581952 not equal to extent_size 18313383936 ERROR: errors found in fs roots Checking filesystem on /dev/mapper/em5 UUID: 632108aa-e869-4ca0-9bc4-c8e667b2da38 cache and super generation don't match, space cache will be invalidated found 22876746559488 bytes used, error(s) found total csum bytes: 22308429932 total tree bytes: 27549319168 total fs tree bytes: 1652686848 total extent tree bytes: 1216888832 btree space waste bytes: 2454986066 file data blocks allocated: 748724628103168 referenced 22986126295040 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Bad superblock when mounting rw, ro mount works 2018-06-14 14:23 ` Daniel Underwood @ 2018-06-14 14:43 ` Qu Wenruo 2018-06-22 21:20 ` Daniel Underwood 0 siblings, 1 reply; 8+ messages in thread From: Qu Wenruo @ 2018-06-14 14:43 UTC (permalink / raw) To: Daniel Underwood, Chris Murphy; +Cc: Btrfs BTRFS [-- Attachment #1.1: Type: text/plain, Size: 3873 bytes --] On 2018年06月14日 22:23, Daniel Underwood wrote: >>> Your very first task right now is to mount ro, and update your >>> backups. Don't do anything else until you've done that. It's a >>> testimony to Btrfs that this file system mounts at all, even ro, so >>> take advantage of this fact before you can't mount it anymore. > > Backups are in place for the important parts, though I'd prefer not to > use them if possible. > > For btrfs-progs, I am using 4.16.1 installed from > https://github.com/kdave/btrfs-progs. > > Regarding em5, it had errors when I initially added it to the array > that were due to a faulty SAS card. The card was replaced and I > haven't seen errors since until this popped up. Regarding what you > said about my dmesg, the numbers are actually still the same (bdev > /dev/mapper/em5 errs: wr 164286, rd 3444262, flush 2110, corrupt 3, > gen 181) after doing a backup and reboot. I would think they would > have changed unless btrfs is just ignoring that disk now. Just to make > sure, I've checked to make sure there weren't any loose connections. > The logs for check (with and without lowmem mode), `btrfs fi us`, and > `smartctl -x` are attached. The checks are only for em5, but I can do > other disks if necessary (it's a 6-disk raid10-style setup). Note that > there are a lot of errors on the SMART info, but they seem to be back > from when I was having hardware issues. From the output, specially the lowmem mode output since original mode handles extent tree corruption poorly and aborted, it's your extent tree corrupted and causing the bug. Thus, you should be able to mount the fs RO and copy all the data back without much hassle. Just need to pay attention for csum error. And considering how many extent tree corruption, I don't think it's a good idea to manually fix the fs. The last chance is, to try --repair --init-extent-tree as your last chance, if you still want to salvage the filesystem. The lowmem mode shows no extra bug, thus it's possible for --init-extent-tree to re-init extent tree and save the day. But personally speaking I'm not fully confident of the operation, thus it may fail and you may need to use the backup. BTW, even --init-extent-tree succeeded, you may still need to run btrfs check again to check if all the bugs are fixed. But at least from the lowmem output, the remaining errors are all fixable. Thanks, Qu > > --- > > On Thu, Jun 7, 2018 at 4:50 PM, Chris Murphy <lists@colorremedies.com> wrote: >> On Thu, Jun 7, 2018 at 2:38 PM, Chris Murphy <lists@colorremedies.com> wrote: >> >> >>> Your very first task right now is to mount ro, and update your >>> backups. Don't do anything else until you've done that. It's a >>> testimony to Btrfs that this file system mounts at all, even ro, so >>> take advantage of this fact before you can't mount it anymore. >> >> After you've done the backup, you need to find out why one of these >> devices is being so unreliable. That has to be fixed first. You can >> recreate a new Btrfs or some other file system, and you'll just run >> into the exact same problem down the road. Next, it might be useful to >> see the output from btrfs-progs 4.16.1 'btrfs check' and 'btrfs check >> --mode=lowmem' both of which are slow, the second one is really slow >> but is a different implementation so it's helpful to see both outputs. >> That's safe as long as you do not use --repair. >> >> Also we need to see the output from 'btrfs fi us <mountpoint>' with >> the volume mounted (ro). Off hand I think the most likely outcome is >> that you get a backup from the ro mounted file system, and you'll have >> to recreate it from scratch and restore from backups. In other words, >> no matter what you need a current backup. >> >> -- >> Chris Murphy > > > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Bad superblock when mounting rw, ro mount works 2018-06-14 14:43 ` Qu Wenruo @ 2018-06-22 21:20 ` Daniel Underwood 2018-06-23 0:02 ` Qu Wenruo 0 siblings, 1 reply; 8+ messages in thread From: Daniel Underwood @ 2018-06-22 21:20 UTC (permalink / raw) To: Qu Wenruo; +Cc: Chris Murphy, Btrfs BTRFS Thanks for the help, I started `check --repair --init-extent-tree` right around a week ago as a last effort before restoring from backup. Unfortunately, that command is still running. It does seem to be using about half of the system's RAM (8 of 16GB) and 100% load on a single core. Is this type of run time expected for an 8TB drive? The byte numbers it's referencing seem to be a bit odd to me as they're larger than the number of bytes on the drive. Here's the head and tail of the current run (separated by -----) if that's indicative of progress: btrfs unable to find ref byte nr 49673574858752 parent 0 root 1 owner 2 offset 0 btrfs unable to find ref byte nr 49673719529472 parent 0 root 1 owner 1 offset 1 btrfs unable to find ref byte nr 62243448012800 parent 0 root 1 owner 0 offset 1 btrfs unable to find ref byte nr 49673575120896 parent 0 root 1 owner 1 offset 1 btrfs unable to find ref byte nr 49673575251968 parent 0 root 1 owner 0 offset 1 checking extents ref mismatch on [49218307751936 67108864] extent item 0, found 1 data backref 49218307751936 root 5 owner 1359193 offset 536870912 num_refs 0 not found in extent tree incorrect local backref count on 49218307751936 root 5 owner 1359193 offset 536870912 found 1 wanted 0 back 0x5583559bd790 backpointer mismatch on [49218307751936 67108864] ----------------- data backref 49230998138880 root 5 owner 1409678 offset 7408779264 num_refs 0 not found in extent tree incorrect local backref count on 49230998138880 root 5 owner 1409678 offset 7408779264 found 1 wanted 0 back 0x5583b95f0e20 backpointer mismatch on [49230998138880 16384] adding new data backref on 49230998138880 root 5 owner 1409678 offset 7408779264 found 1 Repaired extent references for 49230998138880 ref mismatch on [49230998155264 16384] extent item 0, found 1 data backref 49230998155264 root 5 owner 669291 offset 3905650688 num_refs 0 not found in extent tree incorrect local backref count on 49230998155264 root 5 owner 669291 offset 3905650688 found 1 wanted 0 back 0x5582efb12930 backpointer mismatch on [49230998155264 16384] adding new data backref on 49230998155264 root 5 owner 669291 offset 3905650688 found 1 Thanks, Daniel On Thu, Jun 14, 2018 at 10:43 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > From the output, specially the lowmem mode output since original mode > handles extent tree corruption poorly and aborted, it's your extent tree > corrupted and causing the bug. > > Thus, you should be able to mount the fs RO and copy all the data back > without much hassle. > Just need to pay attention for csum error. > > And considering how many extent tree corruption, I don't think it's a > good idea to manually fix the fs. > > The last chance is, to try --repair --init-extent-tree as your last > chance, if you still want to salvage the filesystem. > The lowmem mode shows no extra bug, thus it's possible for > --init-extent-tree to re-init extent tree and save the day. > > But personally speaking I'm not fully confident of the operation, thus > it may fail and you may need to use the backup. > > BTW, even --init-extent-tree succeeded, you may still need to run btrfs > check again to check if all the bugs are fixed. > But at least from the lowmem output, the remaining errors are all fixable. > > Thanks, > Qu ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Bad superblock when mounting rw, ro mount works 2018-06-22 21:20 ` Daniel Underwood @ 2018-06-23 0:02 ` Qu Wenruo 0 siblings, 0 replies; 8+ messages in thread From: Qu Wenruo @ 2018-06-23 0:02 UTC (permalink / raw) To: Daniel Underwood; +Cc: Chris Murphy, Btrfs BTRFS [-- Attachment #1.1: Type: text/plain, Size: 4341 bytes --] On 2018年06月23日 05:20, Daniel Underwood wrote: > Thanks for the help, I started `check --repair --init-extent-tree` > right around a week ago as a last effort before restoring from backup. > Unfortunately, that command is still running. It does seem to be using > about half of the system's RAM (8 of 16GB) and 100% load on a single > core. Is this type of run time expected for an 8TB drive? It depends. Since it will iterate the whole disk serval times, it takes a long time, especially if you have a lot of used space of that 8T drive. To determine if it's doing dead loop, we need to verify the output is not duplicating itself. > The byte > numbers it's referencing seem to be a bit odd to me as they're larger > than the number of bytes on the drive. Completely valid. As btrfs is using logical address, it's completely valid that logical bytenr is larger than your device if you have balanced the device several times before. > Here's the head and tail of the > current run (separated by -----) if that's indicative of progress: > > btrfs unable to find ref byte nr 49673574858752 parent 0 root 1 owner > 2 offset 0 > btrfs unable to find ref byte nr 49673719529472 parent 0 root 1 owner > 1 offset 1 > btrfs unable to find ref byte nr 62243448012800 parent 0 root 1 owner > 0 offset 1 > btrfs unable to find ref byte nr 49673575120896 parent 0 root 1 owner > 1 offset 1 > btrfs unable to find ref byte nr 49673575251968 parent 0 root 1 owner > 0 offset 1 > checking extents > ref mismatch on [49218307751936 67108864] extent item 0, found 1 > data backref 49218307751936 root 5 owner 1359193 offset 536870912 > num_refs 0 not found in extent tree > incorrect local backref count on 49218307751936 root 5 owner 1359193 > offset 536870912 found 1 wanted 0 back 0x5583559bd790 > backpointer mismatch on [49218307751936 67108864] > ----------------- > data backref 49230998138880 root 5 owner 1409678 offset 7408779264 > num_refs 0 not found in extent tree > incorrect local backref count on 49230998138880 root 5 owner 1409678 > offset 7408779264 found 1 wanted 0 back 0x5583b95f0e20 > backpointer mismatch on [49230998138880 16384] > adding new data backref on 49230998138880 root 5 owner 1409678 offset > 7408779264 found 1 > Repaired extent references for 49230998138880 > ref mismatch on [49230998155264 16384] extent item 0, found 1 > data backref 49230998155264 root 5 owner 669291 offset 3905650688 > num_refs 0 not found in extent tree > incorrect local backref count on 49230998155264 root 5 owner 669291 > offset 3905650688 found 1 wanted 0 back 0x5582efb12930 > backpointer mismatch on [49230998155264 16384] > adding new data backref on 49230998155264 root 5 owner 669291 offset > 3905650688 found 1 Looks it's processing. But it's better to keep all the output and verify the same section of bytenr doesn't re-appear too many times. > > Thanks, > Daniel > > > On Thu, Jun 14, 2018 at 10:43 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: >> From the output, specially the lowmem mode output since original mode >> handles extent tree corruption poorly and aborted, it's your extent tree >> corrupted and causing the bug. >> >> Thus, you should be able to mount the fs RO and copy all the data back >> without much hassle. >> Just need to pay attention for csum error. >> >> And considering how many extent tree corruption, I don't think it's a >> good idea to manually fix the fs. >> >> The last chance is, to try --repair --init-extent-tree as your last >> chance, if you still want to salvage the filesystem. >> The lowmem mode shows no extra bug, thus it's possible for >> --init-extent-tree to re-init extent tree and save the day. >> >> But personally speaking I'm not fully confident of the operation, thus >> it may fail and you may need to use the backup. >> >> BTW, even --init-extent-tree succeeded, you may still need to run btrfs >> check again to check if all the bugs are fixed. >> But at least from the lowmem output, the remaining errors are all fixable. >> >> Thanks, >> Qu > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Bad superblock when mounting rw, ro mount works 2018-06-07 17:07 Bad superblock when mounting rw, ro mount works Daniel Underwood 2018-06-07 20:38 ` Chris Murphy @ 2018-06-08 3:39 ` Qu Wenruo 1 sibling, 0 replies; 8+ messages in thread From: Qu Wenruo @ 2018-06-08 3:39 UTC (permalink / raw) To: Daniel Underwood, linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 2175 bytes --] On 2018年06月08日 01:07, Daniel Underwood wrote: > Hi, > > I have a raid10-like setup that is failing to mount in rw mode with the error > > > mount: /mnt/media: wrong fs type, bad option, bad superblock on > /dev/mapper/em1, missing codepage or helper program, or other error > > read-only mounts seem to work and the files seem to be there. > > I started having issues after a system crash during the process of > deleting a number of large files. After this (Ubuntu 16.04/Kernel > 4.4), any attempt to mount the array in rw mode would cause a similar > crash. I did an upgrade to Ubuntu 18.04/Kernel 4.15 and now get the > error above. > > I have looked through a variety of posts on the mailing list, but > couldn't find anything with the same issue. I have done a scrub on the > array that resulted in 6 verify errors with dmesg showing something > about extent trees. It didn't list them as uncorrectable errors, but > couldn't correct them either as I can't mount in rw. I also tried > `btrfs rescue zero-log /dev/mapper/em1`, which changes the above to > (say) em5, but then zero-log on em5 causes it to go back to em1. The problem is, your extent tree is corrupted. And normally, it indicates your whole filesystem has more or less extra errors. In this case, btrfs check could help you to know how badly your fs is corrupted. (don't run --repair). Optionally, btrfs check --mode=lowmem is pretty a good try, as it would provide better error message. (And help us enhance btrfs-progs) > > Any direction would be appreciated. From what I could tell, my next > steps would be a check with --repair or --init-extent-tree, though I'm > reluctant to try those without being explicitly told to do so. Never run this two, unless you're 100% sure about: 1) What's the problem 2) What --repair can fix Personally speaking, and also stated in man page, unless you're a btrfs-progs developer or has enough experience, don't do any write operation with btrfs check. Thanks, Qu > > I have attached a dmesg.log file and hope I haven't greped out > anything important. > > Thanks, > Daniel > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-06-23 0:02 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-06-07 17:07 Bad superblock when mounting rw, ro mount works Daniel Underwood 2018-06-07 20:38 ` Chris Murphy 2018-06-07 20:50 ` Chris Murphy 2018-06-14 14:23 ` Daniel Underwood 2018-06-14 14:43 ` Qu Wenruo 2018-06-22 21:20 ` Daniel Underwood 2018-06-23 0:02 ` Qu Wenruo 2018-06-08 3:39 ` Qu Wenruo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.