* Unmountable degraded BTRFS RAID6 filesystem @ 2019-09-03 22:20 Edmund Urbani 2019-09-03 23:30 ` Chris Murphy 0 siblings, 1 reply; 12+ messages in thread From: Edmund Urbani @ 2019-09-03 22:20 UTC (permalink / raw) To: linux-btrfs Hi all, two days ago my btrfs filesystem became quite slow and the logs showed a lot of I/O errors on one of the HDDs. I ordered a replacement drive and tried to remove the failing drive from the filesystem (btrfs device remove). That removal command did not finish but just sat there without any output. Today the new drive arrived. Device removal still had not finished, but the filesystem had entered read-only mode last night. I shut down the system to replace the defective drive. However, after the reboot I am no longer able to mount the filesystem at all or recover any data from it.:( ***** uname -a Linux phoenix 4.14.78-gentoo #1 SMP Mon Dec 3 09:25:24 CET 2018 x86_64 AMD Opteron(tm) Processor 6174 AuthenticAMD GNU/Linux ***** btrfs --version btrfs-progs v4.19 ***** btrfs fi show warning, device 8 is missing warning, device 8 is missing checksum verify failed on 71133554540544 found B52922D9 wanted C8FB97CF checksum verify failed on 71133554540544 found 9820D207 wanted 189B50C0 checksum verify failed on 71133554540544 found 9820D207 wanted 189B50C0 bad tree block 71133554540544, bytenr mismatch, want=71133554540544, have=7227596181724576485 ERROR: cannot read chunk root Label: none uuid: 108df6ea-2846-4a88-8a50-61aedeef92b4 Total devices 10 FS bytes used 14.71TiB devid 1 size 2.73TiB used 2.04TiB path /dev/sdg1 devid 2 size 2.73TiB used 2.04TiB path /dev/sdh1 devid 3 size 2.73TiB used 2.04TiB path /dev/sdj1 devid 4 size 2.73TiB used 2.04TiB path /dev/sdi1 devid 5 size 2.73TiB used 2.04TiB path /dev/sde1 devid 6 size 2.73TiB used 2.04TiB path /dev/sdf1 devid 7 size 2.73TiB used 2.04TiB path /dev/sda1 devid 9 size 2.73TiB used 2.04TiB path /dev/sdc1 devid 10 size 2.73TiB used 2.04TiB path /dev/sdd1 *** Some devices missing ***** dmesg (after attempting mount with -o degraded) ... [ 8904.358084] BTRFS info (device sda1): turning on discard [ 8904.358088] BTRFS info (device sda1): allowing degraded mounts [ 8904.358089] BTRFS info (device sda1): disk space caching is enabled [ 8904.358091] BTRFS info (device sda1): has skinny extents [ 8904.361743] BTRFS warning (device sda1): devid 8 uuid 0e8b4aff-6d64-4d31-a135-705421928f94 is missing [ 8905.705036] BTRFS info (device sda1): bdev (null) errs: wr 0, rd 14809, flush 0, corrupt 4, gen 0 [ 8905.705041] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd 4, flush 0, corrupt 0, gen 0 [ 8905.705052] BTRFS info (device sda1): bdev /dev/sdf1 errs: wr 0, rd 10543, flush 0, corrupt 0, gen 0 [ 8905.705062] BTRFS info (device sda1): bdev /dev/sdc1 errs: wr 0, rd 8, flush 0, corrupt 0, gen 0 [ 8909.565118] BTRFS error (device sda1): bad tree block start 12170572967447269873 34958581399552 [ 8909.565978] BTRFS error (device sda1): bad tree block start 12170572967447269873 34958581399552 [ 8909.567462] BTRFS error (device sda1): bad tree block start 12170572967447269873 34958581399552 [ 8909.568439] BTRFS error (device sda1): bad tree block start 12170572967447269873 34958581399552 [ 8909.569861] BTRFS error (device sda1): bad tree block start 12170572967447269873 34958581399552 [ 8909.570695] BTRFS error (device sda1): bad tree block start 12170572967447269873 34958581399552 [ 8909.572146] BTRFS error (device sda1): bad tree block start 12170572967447269873 34958581399552 [ 8909.572969] BTRFS error (device sda1): bad tree block start 12170572967447269873 34958581399552 [ 8909.574175] BTRFS error (device sda1): bad tree block start 12170572967447269873 34958581399552 [ 8909.574189] BTRFS error (device sda1): failed to read block groups: -5 [ 8909.635991] BTRFS error (device sda1): open_ctree failed ***** btrfs check /dev/sda1 Opening filesystem to check... warning, device 8 is missing warning, device 8 is missing checksum verify failed on 71133554540544 found B52922D9 wanted C8FB97CF checksum verify failed on 71133554540544 found 9820D207 wanted 189B50C0 checksum verify failed on 71133554540544 found 9820D207 wanted 189B50C0 bad tree block 71133554540544, bytenr mismatch, want=71133554540544, have=7227596181724576485 ERROR: cannot read chunk root ERROR: cannot open file system ***** I have tried all the mount / restore options listed here: https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/?tab=comments#comment-543490 ... and all I keep getting is "bad tree block" errors. Superblocks seem fine (btrfs rescue super-reecover found no problem). I am considering trying "btrfs rescue chunk-recover" at this point. Could this help in my situation? What do you think? Kind regards Edmund -- *Liland IT GmbH* Ferlach ● Wien ● München Tel: +43 463 220111 Tel: +49 89 458 15 940 office@Liland.com https://Liland.com <https://Liland.com> Copyright © 2019 Liland IT GmbH Diese Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese Email irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorised copying, disclosure or distribution of the material in this email is strictly forbidden. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-03 22:20 Unmountable degraded BTRFS RAID6 filesystem Edmund Urbani @ 2019-09-03 23:30 ` Chris Murphy 2019-09-04 4:39 ` Edmund Urbani 0 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2019-09-03 23:30 UTC (permalink / raw) To: Edmund Urbani, Qu Wenruo; +Cc: Btrfs BTRFS On Tue, Sep 3, 2019 at 2:20 PM Edmund Urbani <edmund.urbani@liland.com> wrote: > > > Hi all, > > two days ago my btrfs filesystem became quite slow and the logs showed a > lot of I/O errors on one of the HDDs. I ordered a replacement drive and > tried to remove the failing drive from the filesystem (btrfs device > remove). That removal command did not finish but just sat there without > any output. What exact commands? 'btrfs device del missing' I expect causes reconstruction from parity as well as a balance to create the new 9 device stripe width (well, 7 data + 2 parity). This is not an inherently bad thing to do, it should work and should be COW. And there's one extra copy available in case of an unrecoverable read error, it can still do additional reconstruction. Because it's a balance though, it might be really, really slow and I don't think there is no way to cancel device removal. I don't think it's possible to cancel it with btrfs balance stop. How many subvolumes and snapshots? Are quotas enabled? > Today the new drive arrived. Device removal still had not finished, but > the filesystem had entered read-only mode last night. Likely pre-existing problem is discovered during the balance, or bug triggered, or both, and the file system goes read only to avoid further corruption. Do you have kernel messages for the entire time starting at 'device delete' until the file system goes read only? > Linux phoenix 4.14.78-gentoo #1 SMP Mon Dec 3 09:25:24 CET 2018 x86_64 kernel 4.14.141 is the current version LTS for that series, and there are hundreds of bug fix insertions/removals between just those two versions https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/?id=v4.14.141&id2=v4.14.78&dt=2 between kernel 4.14.141 and 5.2.11, there are thousands of changes just in Btrfs... thousands https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/?id=v5.2.11&id2=v4.14.141&dt=2 And quite a few in raid56.c which isn't that big to begin with, but there are a lot of simplifications and improvements from what I can tell https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/fs/btrfs/raid56.c?id=v5.2.11&id2=v4.14.141 Anyway, it's worth a try to try and mount with 5.2.11 using '-o ro,degraded' and at least see if it will mount. But it gives you some idea why there's a strong bias toward using newer kernels. It's too hard to remember all the changes, even for developers. > AMD Opteron(tm) Processor 6174 AuthenticAMD GNU/Linux > > ***** > btrfs --version > > btrfs-progs v4.19 This is OK, but the change log will show lots of bug fixes here too. I wouldn't make changes (no repair attempts at all, including chunk recover or --repair) until you get some dev advice about the next step. > [ 8904.358084] BTRFS info (device sda1): turning on discard Unexpected. > [ 8904.358088] BTRFS info (device sda1): allowing degraded mounts > [ 8904.358089] BTRFS info (device sda1): disk space caching is enabled > [ 8904.358091] BTRFS info (device sda1): has skinny extents > [ 8904.361743] BTRFS warning (device sda1): devid 8 uuid > 0e8b4aff-6d64-4d31-a135-705421928f94 is missing > [ 8905.705036] BTRFS info (device sda1): bdev (null) errs: wr 0, rd > 14809, flush 0, corrupt 4, gen 0 > [ 8905.705041] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd > 4, flush 0, corrupt 0, gen 0 > [ 8905.705052] BTRFS info (device sda1): bdev /dev/sdf1 errs: wr 0, rd > 10543, flush 0, corrupt 0, gen 0 > [ 8905.705062] BTRFS info (device sda1): bdev /dev/sdc1 errs: wr 0, rd > 8, flush 0, corrupt 0, gen 0 four devices with read errors When was the last time the volume was scrubbed? Do you know for sure these errors have not gone up at all since the last successful scrub? And were any errors reported for that last scrub? > I have tried all the mount / restore options listed here: > https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/?tab=comments#comment-543490 Good. Stick with ro attempts for now. Including if you want to try a newer kernel. If it succeeds to mount ro, my advice is to update backups so at least critical information isn't lost. Back up while you can. Any repair attempt makes changes that will risk the data being permanently lost. So it's important to be really deliberate about any changes. > ... and all I keep getting is "bad tree block" errors. Superblocks seem > fine (btrfs rescue super-reecover found no problem). I am considering > trying "btrfs rescue chunk-recover" at this point. > > Could this help in my situation? What do you think? I'm not sure if chunk recover can work on degraded volumes. Your best bet is to not make any further changes to the volume itself. Preserve all logs. -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-03 23:30 ` Chris Murphy @ 2019-09-04 4:39 ` Edmund Urbani 2019-09-04 5:36 ` Chris Murphy 0 siblings, 1 reply; 12+ messages in thread From: Edmund Urbani @ 2019-09-04 4:39 UTC (permalink / raw) To: linux-btrfs On 09/04/2019 01:30 AM, Chris Murphy wrote: > On Tue, Sep 3, 2019 at 2:20 PM Edmund Urbani <edmund.urbani@liland.com> wrote: >> >> Hi all, >> >> two days ago my btrfs filesystem became quite slow and the logs showed a >> lot of I/O errors on one of the HDDs. I ordered a replacement drive and >> tried to remove the failing drive from the filesystem (btrfs device >> remove). That removal command did not finish but just sat there without >> any output. > What exact commands? btrfs device delete /dev/sdb1 /mnt/shared > > 'btrfs device del missing' I expect causes reconstruction from parity > as well as a balance to create the new 9 device stripe width (well, 7 > data + 2 parity). This is not an inherently bad thing to do, it should > work and should be COW. And there's one extra copy available in case > of an unrecoverable read error, it can still do additional > reconstruction. > > Because it's a balance though, it might be really, really slow and I > don't think there is no way to cancel device removal. I don't think > it's possible to cancel it with btrfs balance stop. > > How many subvolumes and snapshots? Are quotas enabled? One subvolume, no snapshot (I used to have some, but I think I removed them all), no quotes. > > >> Today the new drive arrived. Device removal still had not finished, but >> the filesystem had entered read-only mode last night. > Likely pre-existing problem is discovered during the balance, or bug > triggered, or both, and the file system goes read only to avoid > further corruption. Do you have kernel messages for the entire time > starting at 'device delete' until the file system goes read only? Well, I found this in the logs and that might be around the time I started device removal: Sep 1 16:19:30 phoenix kernel: ------------[ cut here ]------------ Sep 1 16:19:30 phoenix kernel: WARNING: CPU: 9 PID: 6401 at fs/btrfs/ctree.h:1564 btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: Modules linked in: nfnetlink_queue nfnetlink_log nvidia_uvm(O) nvidia(PO) tun nfsd auth_rpcgss oid_registry nfs_acl ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter bridge stp llc nf_conntrack_irc xt_CT xt_tcpudp xt_helper nf_conntrack_ftp nf_log_ipv4 nf_log_common ip6table_raw ip6table_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_TCPMSS xt_LOG ipt_REJECT nf_reject_ipv4 iptable_raw iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack ip6table_filter ip6_tables iptable_filter ip_tables x_tables binfm t_misc snd_hda_codec_hdmi ata_generic kvm_amd kvm snd_hda_intel irqbypass ftdi_sio snd_hda_codec usbserial snd_hda_core pcspkr ipmi_si snd_pcm i2c_piix4 k10temp snd_timer pata_acpi ohci_pci Sep 1 16:19:30 phoenix kernel: snd e1000e nvidiafb vgastate shpchp evdev xts crypto_simd cryptd glue_helper aes_x86_64 ixgb ixgbe tulip cxgb3 cxgb mdio bonding vxlan ip6_udp_tunnel udp_tunnel macvlan tg3 libphy sky2 r8169 pcnet32 mii igb ptp pps_core dca e1000 bnx2 msdos fat fscrypto configfs overlay fuse nfs lockd grace sunrpc fscache btrfs zstd_decompress zstd_compress xxhash zlib_deflate dm_thin_pool dm_persistent_data dm_bio_prison hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_logitech_dj hid_gyration sl811_hcd usbhid xhci_pci xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd qla2xxx megaraid_sas megaraid aa craid sx8 DAC960 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc mptspi mptscsih mptbase atp870u dc395x qla1280 imm parport dmx3191d sym53c8xx gdth BusLogic aic7xxx aic79xx Sep 1 16:19:30 phoenix kernel: scsi_transport_spi sr_mod cdrom sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_cypress pata_oldpiix pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix [last unloaded: nvid ia] Sep 1 16:19:30 phoenix kernel: CPU: 9 PID: 6401 Comm: btrfs Tainted: P W O 4.14.78-gentoo #1 Sep 1 16:19:30 phoenix kernel: Hardware name: System manufacturer System Product Name/KGP(M)E-D16, BIOS 2202 03/29/2012 Sep 1 16:19:30 phoenix kernel: task: ffff880090b7c700 task.stack: ffffc9000b394000 Sep 1 16:19:30 phoenix kernel: RIP: 0010:btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: RSP: 0018:ffffc9000b397b98 EFLAGS: 00010206 Sep 1 16:19:30 phoenix kernel: RAX: 0000000000000fff RBX: ffff88035d1b4690 RCX: 000002baa1371e00 Sep 1 16:19:30 phoenix kernel: RDX: 0000000000001000 RSI: ffff880000000000 RDI: ffff88031455fd20 Sep 1 16:19:30 phoenix kernel: RBP: ffff88022c53e400 R08: ffffc9000b397b50 R09: ffffc9000b397b58 Sep 1 16:19:30 phoenix kernel: R10: 0000000000000003 R11: 0000000000003000 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000003cf0 R14: ffff88031455fd20 R15: ffff880826250348 Sep 1 16:19:30 phoenix kernel: FS: 00007f16f8e4e8c0(0000) GS:ffff88042f980000(0000) knlGS:0000000000000000 Sep 1 16:19:30 phoenix kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 1 16:19:30 phoenix kernel: CR2: 00007f0c0564b180 CR3: 0000000438a32000 CR4: 00000000000006e0 Sep 1 16:19:30 phoenix kernel: Call Trace: Sep 1 16:19:30 phoenix kernel: btrfs_remove_chunk+0x2f9/0x700 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_relocate_chunk+0x9c/0xd0 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_shrink_device+0x1c0/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? btrfs_find_device_missing_or_by_path+0x30/0x120 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_rm_device+0x19b/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_from_user+0x3f/0x80 Sep 1 16:19:30 phoenix kernel: btrfs_ioctl+0x2129/0x2380 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_to_user+0x22/0x30 Sep 1 16:19:30 phoenix kernel: ? cp_new_stat+0x138/0x150 Sep 1 16:19:30 phoenix kernel: do_vfs_ioctl+0x9b/0x5e0 Sep 1 16:19:30 phoenix kernel: ? SyS_newstat+0x35/0x40 Sep 1 16:19:30 phoenix kernel: SyS_ioctl+0x47/0x90 Sep 1 16:19:30 phoenix kernel: do_syscall_64+0x55/0x100 Sep 1 16:19:30 phoenix kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2 Sep 1 16:19:30 phoenix kernel: RIP: 0033:0x7f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RSP: 002b:00007ffd701d26e8 EFLAGS: 00000206 ORIG_RAX: 0000000000000010 Sep 1 16:19:30 phoenix kernel: RAX: ffffffffffffffda RBX: 00007ffd701d4880 RCX: 00007f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RDX: 00007ffd701d3720 RSI: 000000005000943a RDI: 0000000000000003 Sep 1 16:19:30 phoenix kernel: RBP: 00007ffd701d3720 R08: 0000000000000000 R09: 000000000000000c Sep 1 16:19:30 phoenix kernel: R10: 0000000000000572 R11: 0000000000000206 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000000000 R14: 0000000000000003 R15: 00007ffd701d4888 Sep 1 16:19:30 phoenix kernel: Code: 4c 89 f7 45 31 c0 ba 10 00 00 00 4c 89 ee e8 9a 40 ff ff 4c 89 f7 e8 42 29 fd ff e9 de fe ff ff 41 bc f4 ff ff ff e9 db fe ff ff <0f> 0b eb b7 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 53 31 d2 Sep 1 16:19:30 phoenix kernel: ---[ end trace edd626af3a502d93 ]--- Sep 1 16:19:30 phoenix kernel: ------------[ cut here ]------------ Sep 1 16:19:30 phoenix kernel: WARNING: CPU: 10 PID: 6401 at fs/btrfs/ctree.h:1564 btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: Modules linked in: nfnetlink_queue nfnetlink_log nvidia_uvm(O) nvidia(PO) tun nfsd auth_rpcgss oid_registry nfs_acl ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter bridge stp llc nf_conntrack_irc xt_CT xt_tcpudp xt_helper nf_conntrack_ftp nf_log_ipv4 nf_log_common ip6table_raw ip6table_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_TCPMSS xt_LOG ipt_REJECT nf_reject_ipv4 iptable_raw iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack ip6table_filter ip6_tables iptable_filter ip_tables x_tables binfm t_misc snd_hda_codec_hdmi ata_generic kvm_amd kvm snd_hda_intel irqbypass ftdi_sio snd_hda_codec usbserial snd_hda_core pcspkr ipmi_si snd_pcm i2c_piix4 k10temp snd_timer pata_acpi ohci_pci Sep 1 16:19:30 phoenix kernel: snd e1000e nvidiafb vgastate shpchp evdev xts crypto_simd cryptd glue_helper aes_x86_64 ixgb ixgbe tulip cxgb3 cxgb mdio bonding vxlan ip6_udp_tunnel udp_tunnel macvlan tg3 libphy sky2 r8169 pcnet32 mii igb ptp pps_core dca e1000 bnx2 msdos fat fscrypto configfs overlay fuse nfs lockd grace sunrpc fscache btrfs zstd_decompress zstd_compress xxhash zlib_deflate dm_thin_pool dm_persistent_data dm_bio_prison hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_logitech_dj hid_gyration sl811_hcd usbhid xhci_pci xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd qla2xxx megaraid_sas megaraid aa craid sx8 DAC960 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc mptspi mptscsih mptbase atp870u dc395x qla1280 imm parport dmx3191d sym53c8xx gdth BusLogic aic7xxx aic79xx Sep 1 16:19:30 phoenix kernel: scsi_transport_spi sr_mod cdrom sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_cypress pata_oldpiix pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix [last unloaded: nvid ia] Sep 1 16:19:30 phoenix kernel: CPU: 10 PID: 6401 Comm: btrfs Tainted: P W O 4.14.78-gentoo #1 Sep 1 16:19:30 phoenix kernel: Hardware name: System manufacturer System Product Name/KGP(M)E-D16, BIOS 2202 03/29/2012 Sep 1 16:19:30 phoenix kernel: task: ffff880090b7c700 task.stack: ffffc9000b394000 Sep 1 16:19:30 phoenix kernel: RIP: 0010:btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: RSP: 0018:ffffc9000b397b98 EFLAGS: 00010206 Sep 1 16:19:30 phoenix kernel: RAX: 0000000000000fff RBX: ffff88035d1b4690 RCX: 000002baa1371e00 Sep 1 16:19:30 phoenix kernel: RDX: 0000000000001000 RSI: ffff880000000000 RDI: ffff88031455fd20 Sep 1 16:19:30 phoenix kernel: RBP: ffff8808261adc00 R08: ffffc9000b397b50 R09: ffffc9000b397b58 Sep 1 16:19:30 phoenix kernel: R10: 0000000000000003 R11: 0000000000003000 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000003e16 R14: ffff88031455fd20 R15: ffff880826250348 Sep 1 16:19:30 phoenix kernel: FS: 00007f16f8e4e8c0(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 Sep 1 16:19:30 phoenix kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 1 16:19:30 phoenix kernel: CR2: 00007f0c0564b180 CR3: 0000000438a32000 CR4: 00000000000006e0 Sep 1 16:19:30 phoenix kernel: Call Trace: Sep 1 16:19:30 phoenix kernel: btrfs_remove_chunk+0x2f9/0x700 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_relocate_chunk+0x9c/0xd0 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_shrink_device+0x1c0/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? btrfs_find_device_missing_or_by_path+0x30/0x120 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_rm_device+0x19b/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_from_user+0x3f/0x80 Sep 1 16:19:30 phoenix kernel: btrfs_ioctl+0x2129/0x2380 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_to_user+0x22/0x30 Sep 1 16:19:30 phoenix kernel: ? cp_new_stat+0x138/0x150 Sep 1 16:19:30 phoenix kernel: do_vfs_ioctl+0x9b/0x5e0 Sep 1 16:19:30 phoenix kernel: ? SyS_newstat+0x35/0x40 Sep 1 16:19:30 phoenix kernel: SyS_ioctl+0x47/0x90 Sep 1 16:19:30 phoenix kernel: do_syscall_64+0x55/0x100 Sep 1 16:19:30 phoenix kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2 Sep 1 16:19:30 phoenix kernel: RIP: 0033:0x7f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RSP: 002b:00007ffd701d26e8 EFLAGS: 00000206 ORIG_RAX: 0000000000000010 Sep 1 16:19:30 phoenix kernel: RAX: ffffffffffffffda RBX: 00007ffd701d4880 RCX: 00007f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RDX: 00007ffd701d3720 RSI: 000000005000943a RDI: 0000000000000003 Sep 1 16:19:30 phoenix kernel: RBP: 00007ffd701d3720 R08: 0000000000000000 R09: 000000000000000c Sep 1 16:19:30 phoenix kernel: R10: 0000000000000572 R11: 0000000000000206 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000000000 R14: 0000000000000003 R15: 00007ffd701d4888 Sep 1 16:19:30 phoenix kernel: Code: 4c 89 f7 45 31 c0 ba 10 00 00 00 4c 89 ee e8 9a 40 ff ff 4c 89 f7 e8 42 29 fd ff e9 de fe ff ff 41 bc f4 ff ff ff e9 db fe ff ff <0f> 0b eb b7 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 53 31 d2 Sep 1 16:19:30 phoenix kernel: ---[ end trace edd626af3a502d94 ]--- Sep 1 16:19:30 phoenix kernel: ------------[ cut here ]------------ Sep 1 16:19:30 phoenix kernel: WARNING: CPU: 11 PID: 6401 at fs/btrfs/ctree.h:1564 btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: Modules linked in: nfnetlink_queue nfnetlink_log nvidia_uvm(O) nvidia(PO) tun nfsd auth_rpcgss oid_registry nfs_acl ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter bridge stp llc nf_conntrack_irc xt_CT xt_tcpudp xt_helper nf_conntrack_ftp nf_log_ipv4 nf_log_common ip6table_raw ip6table_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_TCPMSS xt_LOG ipt_REJECT nf_reject_ipv4 iptable_raw iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack ip6table_filter ip6_tables iptable_filter ip_tables x_tables binfmt_misc snd_hda_codec_hdmi ata_generic kvm_amd kvm snd_hda_intel irqbypass ftdi_sio snd_hda_codec usbserial snd_hda_core pcspkr ipmi_si snd_pcm i2c_piix4 k10temp snd_timer pata_acpi ohci_pci Sep 1 16:19:30 phoenix kernel: snd e1000e nvidiafb vgastate shpchp evdev xts crypto_simd cryptd glue_helper aes_x86_64 ixgb ixgbe tulip cxgb3 cxgb mdio bonding vxlan ip6_udp_tunnel udp_tunnel macvlan tg3 libphy sky2 r8169 pcnet32 mii igb ptp pps_core dca e1000 bnx2 msdos fat fscrypto configfs overlay fuse nfs lockd grace sunrpc fscache btrfs zstd_decompress zstd_compress xxhash zlib_deflate dm_thin_pool dm_persistent_data dm_bio_prison hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_logitech_dj hid_gyration sl811_hcd usbhid xhci_pci xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd qla2xxx megaraid_sas megaraid aacraid sx8 DAC960 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc mptspi mptscsih mptbase atp870u dc395x qla1280 imm parport dmx3191d sym53c8xx gdth BusLogic aic7xxx aic79xx Sep 1 16:19:30 phoenix kernel: scsi_transport_spi sr_mod cdrom sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_cypress pata_oldpiix pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix [last unloaded: nvidia] Sep 1 16:19:30 phoenix kernel: CPU: 11 PID: 6401 Comm: btrfs Tainted: P W O 4.14.78-gentoo #1 Sep 1 16:19:30 phoenix kernel: Hardware name: System manufacturer System Product Name/KGP(M)E-D16, BIOS 2202 03/29/2012 Sep 1 16:19:30 phoenix kernel: task: ffff880090b7c700 task.stack: ffffc9000b394000 Sep 1 16:19:30 phoenix kernel: RIP: 0010:btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: RSP: 0018:ffffc9000b397b98 EFLAGS: 00010206 Sep 1 16:19:30 phoenix kernel: RAX: 0000000000000fff RBX: ffff88035d1b4690 RCX: 000002baa1371e00 Sep 1 16:19:30 phoenix kernel: RDX: 0000000000001000 RSI: ffff880000000000 RDI: ffff88031455fd20 Sep 1 16:19:30 phoenix kernel: RBP: ffff880825590400 R08: ffffc9000b397b50 R09: ffffc9000b397b58 Sep 1 16:19:30 phoenix kernel: R10: 0000000000000003 R11: 0000000000003000 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000003db4 R14: ffff88031455fd20 R15: ffff880826250348 Sep 1 16:19:30 phoenix kernel: FS: 00007f16f8e4e8c0(0000) GS:ffff88042fa80000(0000) knlGS:0000000000000000 Sep 1 16:19:30 phoenix kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 1 16:19:30 phoenix kernel: CR2: 000056068cac8ec8 CR3: 0000000438a32000 CR4: 00000000000006e0 Sep 1 16:19:30 phoenix kernel: Call Trace: Sep 1 16:19:30 phoenix kernel: btrfs_remove_chunk+0x2f9/0x700 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_relocate_chunk+0x9c/0xd0 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_shrink_device+0x1c0/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? btrfs_find_device_missing_or_by_path+0x30/0x120 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_rm_device+0x19b/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_from_user+0x3f/0x80 Sep 1 16:19:30 phoenix kernel: btrfs_ioctl+0x2129/0x2380 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_to_user+0x22/0x30 Sep 1 16:19:30 phoenix kernel: ? cp_new_stat+0x138/0x150 Sep 1 16:19:30 phoenix kernel: do_vfs_ioctl+0x9b/0x5e0 Sep 1 16:19:30 phoenix kernel: ? SyS_newstat+0x35/0x40 Sep 1 16:19:30 phoenix kernel: SyS_ioctl+0x47/0x90 Sep 1 16:19:30 phoenix kernel: do_syscall_64+0x55/0x100 Sep 1 16:19:30 phoenix kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2 Sep 1 16:19:30 phoenix kernel: RIP: 0033:0x7f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RSP: 002b:00007ffd701d26e8 EFLAGS: 00000206 ORIG_RAX: 0000000000000010 Sep 1 16:19:30 phoenix kernel: RAX: ffffffffffffffda RBX: 00007ffd701d4880 RCX: 00007f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RDX: 00007ffd701d3720 RSI: 000000005000943a RDI: 0000000000000003 Sep 1 16:19:30 phoenix kernel: RBP: 00007ffd701d3720 R08: 0000000000000000 R09: 000000000000000c Sep 1 16:19:30 phoenix kernel: R13: 0000000000000000 R14: 0000000000000003 R15: 00007ffd701d4888 Sep 1 16:19:30 phoenix kernel: Code: 4c 89 f7 45 31 c0 ba 10 00 00 00 4c 89 ee e8 9a 40 ff ff 4c 89 f7 e8 42 29 fd ff e9 de fe ff ff 41 bc f4 ff ff ff e9 db fe ff ff <0f> 0b eb b7 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 53 31 d2 Sep 1 16:19:30 phoenix kernel: ---[ end trace edd626af3a502d95 ]--- Sep 1 16:19:30 phoenix kernel: ------------[ cut here ]------------ Sep 1 16:19:30 phoenix kernel: WARNING: CPU: 6 PID: 6401 at fs/btrfs/ctree.h:1564 btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: Modules linked in: nfnetlink_queue nfnetlink_log nvidia_uvm(O) nvidia(PO) tun nfsd auth_rpcgss oid_registry nfs_acl ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter bridge stp llc nf_conntrack_irc xt_CT xt_tcpudp xt_helper nf_conntrack_ftp nf_log_ipv4 nf_log_common ip6table_raw ip6table_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_TCPMSS xt_LOG ipt_REJECT nf_reject_ipv4 iptable_raw iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack ip6table_filter ip6_tables iptable_filter ip_tables x_tables binfmt_misc snd_hda_codec_hdmi ata_generic kvm_amd kvm snd_hda_intel irqbypass ftdi_sio snd_hda_codec usbserial snd_hda_core pcspkr ipmi_si snd_pcm i2c_piix4 k10temp snd_timer pata_acpi ohci_pci Sep 1 16:19:30 phoenix kernel: snd e1000e nvidiafb vgastate shpchp evdev xts crypto_simd cryptd glue_helper aes_x86_64 ixgb ixgbe tulip cxgb3 cxgb mdio bonding vxlan ip6_udp_tunnel udp_tunnel macvlan tg3 libphy sky2 r8169 pcnet32 mii igb ptp pps_core dca e1000 bnx2 msdos fat fscrypto configfs overlay fuse nfs lockd grace sunrpc fscache btrfs zstd_decompress zstd_compress xxhash zlib_deflate dm_thin_pool dm_persistent_data dm_bio_prison hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_logitech_dj hid_gyration sl811_hcd usbhid xhci_pci xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd qla2xxx megaraid_sas megaraid aacraid sx8 DAC960 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc mptspi mptscsih mptbase atp870u dc395x qla1280 imm parport dmx3191d sym53c8xx gdth BusLogic aic7xxx aic79xx Sep 1 16:19:30 phoenix kernel: scsi_transport_spi sr_mod cdrom sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_cypress pata_oldpiix pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix [last unloaded: nvidia] Sep 1 16:19:30 phoenix kernel: CPU: 6 PID: 6401 Comm: btrfs Tainted: P W O 4.14.78-gentoo #1 Sep 1 16:19:30 phoenix kernel: Hardware name: System manufacturer System Product Name/KGP(M)E-D16, BIOS 2202 03/29/2012 Sep 1 16:19:30 phoenix kernel: task: ffff880090b7c700 task.stack: ffffc9000b394000 Sep 1 16:19:30 phoenix kernel: RIP: 0010:btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: RSP: 0018:ffffc9000b397b98 EFLAGS: 00010206 Sep 1 16:19:30 phoenix kernel: RAX: 0000000000000fff RBX: ffff88035d1b4690 RCX: 000002baa1371e00 Sep 1 16:19:30 phoenix kernel: RDX: 0000000000001000 RSI: ffff880000000000 RDI: ffff880250a8a388 Sep 1 16:19:30 phoenix kernel: RBP: ffff880825886400 R08: ffffc9000b397b50 R09: ffffc9000b397b58 Sep 1 16:19:30 phoenix kernel: R10: 0000000000000003 R11: 0000000000003000 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000003f9e R14: ffff880250a8a388 R15: ffff880826250348 Sep 1 16:19:30 phoenix kernel: FS: 00007f16f8e4e8c0(0000) GS:ffff88042f800000(0000) knlGS:0000000000000000 Sep 1 16:19:30 phoenix kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 1 16:19:30 phoenix kernel: CR2: 00007f0c0564b180 CR3: 0000000438a32000 CR4: 00000000000006e0 Sep 1 16:19:30 phoenix kernel: Call Trace: Sep 1 16:19:30 phoenix kernel: btrfs_remove_chunk+0x2f9/0x700 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_relocate_chunk+0x9c/0xd0 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_shrink_device+0x1c0/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? btrfs_find_device_missing_or_by_path+0x30/0x120 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_rm_device+0x19b/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_from_user+0x3f/0x80 Sep 1 16:19:30 phoenix kernel: btrfs_ioctl+0x2129/0x2380 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_to_user+0x22/0x30 Sep 1 16:19:30 phoenix kernel: ? cp_new_stat+0x138/0x150 Sep 1 16:19:30 phoenix kernel: do_vfs_ioctl+0x9b/0x5e0 Sep 1 16:19:30 phoenix kernel: ? SyS_newstat+0x35/0x40 Sep 1 16:19:30 phoenix kernel: SyS_ioctl+0x47/0x90 Sep 1 16:19:30 phoenix kernel: do_syscall_64+0x55/0x100 Sep 1 16:19:30 phoenix kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2 Sep 1 16:19:30 phoenix kernel: RIP: 0033:0x7f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RSP: 002b:00007ffd701d26e8 EFLAGS: 00000206 ORIG_RAX: 0000000000000010 Sep 1 16:19:30 phoenix kernel: RAX: ffffffffffffffda RBX: 00007ffd701d4880 RCX: 00007f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RDX: 00007ffd701d3720 RSI: 000000005000943a RDI: 0000000000000003 Sep 1 16:19:30 phoenix kernel: RBP: 00007ffd701d3720 R08: 0000000000000000 R09: 000000000000000c Sep 1 16:19:30 phoenix kernel: R10: 0000000000000572 R11: 0000000000000206 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000000000 R14: 0000000000000003 R15: 00007ffd701d4888 Sep 1 16:19:30 phoenix kernel: Code: 4c 89 f7 45 31 c0 ba 10 00 00 00 4c 89 ee e8 9a 40 ff ff 4c 89 f7 e8 42 29 fd ff e9 de fe ff ff 41 bc f4 ff ff ff e9 db fe ff ff <0f> 0b eb b7 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 53 31 d2 Sep 1 16:19:30 phoenix kernel: ---[ end trace edd626af3a502d96 ]--- Sep 1 16:19:30 phoenix kernel: ------------[ cut here ]------------ Sep 1 16:19:30 phoenix kernel: WARNING: CPU: 7 PID: 6401 at fs/btrfs/ctree.h:1564 btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: Modules linked in: nfnetlink_queue nfnetlink_log nvidia_uvm(O) nvidia(PO) tun nfsd auth_rpcgss oid_registry nfs_acl ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter bridge stp llc nf_conntrack_irc xt_CT xt_tcpudp xt_helper nf_conntrack_ftp nf_log_ipv4 nf_log_common ip6table_raw ip6table_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_TCPMSS xt_LOG ipt_REJECT nf_reject_ipv4 iptable_raw iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack ip6table_filter ip6_tables iptable_filter ip_tables x_tables binfmt_misc snd_hda_codec_hdmi ata_generic kvm_amd kvm snd_hda_intel irqbypass ftdi_sio snd_hda_codec usbserial snd_hda_core pcspkr ipmi_si snd_pcm i2c_piix4 k10temp snd_timer pata_acpi ohci_pci Sep 1 16:19:30 phoenix kernel: snd e1000e nvidiafb vgastate shpchp evdev xts crypto_simd cryptd glue_helper aes_x86_64 ixgb ixgbe tulip cxgb3 cxgb mdio bonding vxlan ip6_udp_tunnel udp_tunnel macvlan tg3 libphy sky2 r8169 pcnet32 mii igb ptp pps_core dca e1000 bnx2 msdos fat fscrypto configfs overlay fuse nfs lockd grace sunrpc fscache btrfs zstd_decompress zstd_compress xxhash zlib_deflate dm_thin_pool dm_persistent_data dm_bio_prison hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_logitech_dj hid_gyration sl811_hcd usbhid xhci_pci xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd qla2xxx megaraid_sas megaraid aacraid sx8 DAC960 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc mptspi mptscsih mptbase atp870u dc395x qla1280 imm parport dmx3191d sym53c8xx gdth BusLogic aic7xxx aic79xx Sep 1 16:19:30 phoenix kernel: scsi_transport_spi sr_mod cdrom sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_cypress pata_oldpiix pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix [last unloaded: nvidia] Sep 1 16:19:30 phoenix kernel: CPU: 7 PID: 6401 Comm: btrfs Tainted: P W O 4.14.78-gentoo #1 Sep 1 16:19:30 phoenix kernel: Hardware name: System manufacturer System Product Name/KGP(M)E-D16, BIOS 2202 03/29/2012 Sep 1 16:19:30 phoenix kernel: task: ffff880090b7c700 task.stack: ffffc9000b394000 Sep 1 16:19:30 phoenix kernel: RIP: 0010:btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: RSP: 0018:ffffc9000b397b98 EFLAGS: 00010206 Sep 1 16:19:30 phoenix kernel: RAX: 0000000000000fff RBX: ffff88035d1b4690 RCX: 000002baa1371e00 Sep 1 16:19:30 phoenix kernel: RDX: 0000000000001000 RSI: ffff880000000000 RDI: ffff88031455fd20 Sep 1 16:19:30 phoenix kernel: RBP: ffff880825881400 R08: ffffc9000b397b50 R09: ffffc9000b397b58 Sep 1 16:19:30 phoenix kernel: R10: 0000000000000003 R11: 0000000000003000 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000003c8e R14: ffff88031455fd20 R15: ffff880826250348 Sep 1 16:19:30 phoenix kernel: FS: 00007f16f8e4e8c0(0000) GS:ffff88042f880000(0000) knlGS:0000000000000000 Sep 1 16:19:30 phoenix kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 1 16:19:30 phoenix kernel: CR2: 00007f0bfd599394 CR3: 0000000438a32000 CR4: 00000000000006e0 Sep 1 16:19:30 phoenix kernel: Call Trace: Sep 1 16:19:30 phoenix kernel: btrfs_remove_chunk+0x2f9/0x700 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_relocate_chunk+0x9c/0xd0 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_shrink_device+0x1c0/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? btrfs_find_device_missing_or_by_path+0x30/0x120 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_rm_device+0x19b/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_from_user+0x3f/0x80 Sep 1 16:19:30 phoenix kernel: btrfs_ioctl+0x2129/0x2380 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_to_user+0x22/0x30 Sep 1 16:19:30 phoenix kernel: ? cp_new_stat+0x138/0x150 Sep 1 16:19:30 phoenix kernel: do_vfs_ioctl+0x9b/0x5e0 Sep 1 16:19:30 phoenix kernel: ? SyS_newstat+0x35/0x40 Sep 1 16:19:30 phoenix kernel: SyS_ioctl+0x47/0x90 Sep 1 16:19:30 phoenix kernel: do_syscall_64+0x55/0x100 Sep 1 16:19:30 phoenix kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2 Sep 1 16:19:30 phoenix kernel: RIP: 0033:0x7f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RSP: 002b:00007ffd701d26e8 EFLAGS: 00000206 ORIG_RAX: 0000000000000010 Sep 1 16:19:30 phoenix kernel: RAX: ffffffffffffffda RBX: 00007ffd701d4880 RCX: 00007f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RDX: 00007ffd701d3720 RSI: 000000005000943a RDI: 0000000000000003 Sep 1 16:19:30 phoenix kernel: RBP: 00007ffd701d3720 R08: 0000000000000000 R09: 000000000000000c Sep 1 16:19:30 phoenix kernel: R10: 0000000000000572 R11: 0000000000000206 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000000000 R14: 0000000000000003 R15: 00007ffd701d4888 Sep 1 16:19:30 phoenix kernel: Code: 4c 89 f7 45 31 c0 ba 10 00 00 00 4c 89 ee e8 9a 40 ff ff 4c 89 f7 e8 42 29 fd ff e9 de fe ff ff 41 bc f4 ff ff ff e9 db fe ff ff <0f> 0b eb b7 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 53 31 d2 Sep 1 16:19:30 phoenix kernel: ---[ end trace edd626af3a502d97 ]--- Sep 1 16:19:30 phoenix kernel: ------------[ cut here ]------------ Sep 1 16:19:30 phoenix kernel: WARNING: CPU: 8 PID: 6401 at fs/btrfs/ctree.h:1564 btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: Modules linked in: nfnetlink_queue nfnetlink_log nvidia_uvm(O) nvidia(PO) tun nfsd auth_rpcgss oid_registry nfs_acl ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter bridge stp llc nf_conntrack_irc xt_CT xt_tcpudp xt_helper nf_conntrack_ftp nf_log_ipv4 nf_log_common ip6table_raw ip6table_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_TCPMSS xt_LOG ipt_REJECT nf_reject_ipv4 iptable_raw iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack ip6table_filter ip6_tables iptable_filter ip_tables x_tables binfmt_misc snd_hda_codec_hdmi ata_generic kvm_amd kvm snd_hda_intel irqbypass ftdi_sio snd_hda_codec usbserial snd_hda_core pcspkr ipmi_si snd_pcm i2c_piix4 k10temp snd_timer pata_acpi ohci_pci Sep 1 16:19:30 phoenix kernel: snd e1000e nvidiafb vgastate shpchp evdev xts crypto_simd cryptd glue_helper aes_x86_64 ixgb ixgbe tulip cxgb3 cxgb mdio bonding vxlan ip6_udp_tunnel udp_tunnel macvlan tg3 libphy sky2 r8169 pcnet32 mii igb ptp pps_core dca e1000 bnx2 msdos fat fscrypto configfs overlay fuse nfs lockd grace sunrpc fscache btrfs zstd_decompress zstd_compress xxhash zlib_deflate dm_thin_pool dm_persistent_data dm_bio_prison hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_logitech_dj hid_gyration sl811_hcd usbhid xhci_pci xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd qla2xxx megaraid_sas megaraid aacraid sx8 DAC960 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc mptspi mptscsih mptbase atp870u dc395x qla1280 imm parport dmx3191d sym53c8xx gdth BusLogic aic7xxx aic79xx Sep 1 16:19:30 phoenix kernel: scsi_transport_spi sr_mod cdrom sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_cypress pata_oldpiix pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix [last unloaded: nvidia] Sep 1 16:19:30 phoenix kernel: CPU: 8 PID: 6401 Comm: btrfs Tainted: P W O 4.14.78-gentoo #1 Sep 1 16:19:30 phoenix kernel: Hardware name: System manufacturer System Product Name/KGP(M)E-D16, BIOS 2202 03/29/2012 Sep 1 16:19:30 phoenix kernel: task: ffff880090b7c700 task.stack: ffffc9000b394000 Sep 1 16:19:30 phoenix kernel: RIP: 0010:btrfs_update_device+0x1ae/0x1c0 [btrfs] Sep 1 16:19:30 phoenix kernel: RSP: 0018:ffffc9000b397b98 EFLAGS: 00010206 Sep 1 16:19:30 phoenix kernel: RAX: 0000000000000fff RBX: ffff88035d1b4690 RCX: 000002baa1371e00 Sep 1 16:19:30 phoenix kernel: RDX: 0000000000001000 RSI: ffff880000000000 RDI: ffff88031455fd20 Sep 1 16:19:30 phoenix kernel: RBP: ffff88022bb44c00 R08: ffffc9000b397b50 R09: ffffc9000b397b58 Sep 1 16:19:30 phoenix kernel: R10: 0000000000000003 R11: 0000000000003000 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000003d52 R14: ffff88031455fd20 R15: ffff880826250348 Sep 1 16:19:30 phoenix kernel: FS: 00007f16f8e4e8c0(0000) GS:ffff88042f900000(0000) knlGS:0000000000000000 Sep 1 16:19:30 phoenix kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 1 16:19:30 phoenix kernel: CR2: 00007f5854176078 CR3: 0000000438a32000 CR4: 00000000000006e0 Sep 1 16:19:30 phoenix kernel: Call Trace: Sep 1 16:19:30 phoenix kernel: btrfs_remove_chunk+0x2f9/0x700 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_relocate_chunk+0x9c/0xd0 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_shrink_device+0x1c0/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? btrfs_find_device_missing_or_by_path+0x30/0x120 [btrfs] Sep 1 16:19:30 phoenix kernel: btrfs_rm_device+0x19b/0x4f0 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_from_user+0x3f/0x80 Sep 1 16:19:30 phoenix kernel: btrfs_ioctl+0x2129/0x2380 [btrfs] Sep 1 16:19:30 phoenix kernel: ? _copy_to_user+0x22/0x30 Sep 1 16:19:30 phoenix kernel: ? cp_new_stat+0x138/0x150 Sep 1 16:19:30 phoenix kernel: do_vfs_ioctl+0x9b/0x5e0 Sep 1 16:19:30 phoenix kernel: ? SyS_newstat+0x35/0x40 Sep 1 16:19:30 phoenix kernel: SyS_ioctl+0x47/0x90 Sep 1 16:19:30 phoenix kernel: do_syscall_64+0x55/0x100 Sep 1 16:19:30 phoenix kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2 Sep 1 16:19:30 phoenix kernel: RIP: 0033:0x7f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RSP: 002b:00007ffd701d26e8 EFLAGS: 00000206 ORIG_RAX: 0000000000000010 Sep 1 16:19:30 phoenix kernel: RAX: ffffffffffffffda RBX: 00007ffd701d4880 RCX: 00007f16f7be80f7 Sep 1 16:19:30 phoenix kernel: RDX: 00007ffd701d3720 RSI: 000000005000943a RDI: 0000000000000003 Sep 1 16:19:30 phoenix kernel: RBP: 00007ffd701d3720 R08: 0000000000000000 R09: 000000000000000c Sep 1 16:19:30 phoenix kernel: R10: 0000000000000572 R11: 0000000000000206 R12: 0000000000000000 Sep 1 16:19:30 phoenix kernel: R13: 0000000000000000 R14: 0000000000000003 R15: 00007ffd701d4888 Sep 1 16:19:30 phoenix kernel: Code: 4c 89 f7 45 31 c0 ba 10 00 00 00 4c 89 ee e8 9a 40 ff ff 4c 89 f7 e8 42 29 fd ff e9 de fe ff ff 41 bc f4 ff ff ff e9 db fe ff ff <0f> 0b eb b7 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 53 31 d2 Sep 1 16:19:30 phoenix kernel: ---[ end trace edd626af3a502d98 ]--- Sep 1 16:19:30 phoenix kernel: BTRFS info (device sdb1): relocating block group 71262403559424 flags data|raid6 There are more of this sort with other blocks groups to be found in the log. Also there are a few of these: Sep 1 21:10:17 phoenix kernel: ata6.00: exception Emask 0x0 SAct 0x10000020 SErr 0x0 action 0x0 Sep 1 21:10:17 phoenix kernel: ata6.00: irq_stat 0x40000008 Sep 1 21:10:17 phoenix kernel: ata6.00: failed command: READ FPDMA QUEUED Sep 1 21:10:17 phoenix kernel: ata6.00: cmd 60/20:28:80:66:09/00:00:50:01:00/40 tag 5 ncq dma 16384 in\x0a res 41/40:00:88:66:09/00:00:50:01:00/40 Emask 0x409 (media error) <F> Sep 1 21:10:17 phoenix kernel: ata6.00: status: { DRDY ERR } Sep 1 21:10:17 phoenix kernel: ata6.00: error: { UNC } Sep 1 21:10:17 phoenix kernel: ata6.00: configured for UDMA/133 Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 Sense Key : 0x3 [current] Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 ASC=0x11 ASCQ=0x4 Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 CDB: opcode=0x88 88 00 00 00 00 01 50 09 66 80 00 00 00 20 00 00 Sep 1 21:10:17 phoenix kernel: print_req_error: I/O error, dev sdf, sector 5637760640 Sep 1 21:10:17 phoenix kernel: BTRFS error (device sdb1): bdev /dev/sdf1 errs: wr 0, rd 289, flush 0, corrupt 0, gen 0 Sep 1 21:10:17 phoenix kernel: ata6: EH complete Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 level 0 Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 level 0 Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 level 0 Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 level 0 Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 level 0 Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 level 0 Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 level 0 Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 level 0 Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 level 0 So, yes /dev/sdf is also causing problems. I was going to replace that one next. I still have the old /dev/sdb lying around BTW. It has not completely failed yet, it's just not installed at the moment. I am still looking for log entries related to the filesystem going read-only. Not sure when exactly that happened and the logs are spammed with plenty of the above... > >> Linux phoenix 4.14.78-gentoo #1 SMP Mon Dec 3 09:25:24 CET 2018 x86_64 > kernel 4.14.141 is the current version LTS for that series, and there > are hundreds of bug fix insertions/removals between just those two > versions > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/?id=v4.14.141&id2=v4.14.78&dt=2 > > between kernel 4.14.141 and 5.2.11, there are thousands of changes > just in Btrfs... thousands > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/?id=v5.2.11&id2=v4.14.141&dt=2 > > And quite a few in raid56.c which isn't that big to begin with, but > there are a lot of simplifications and improvements from what I can > tell > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/fs/btrfs/raid56.c?id=v5.2.11&id2=v4.14.141 > > Anyway, it's worth a try to try and mount with 5.2.11 using '-o > ro,degraded' and at least see if it will mount. But it gives you some > idea why there's a strong bias toward using newer kernels. It's too > hard to remember all the changes, even for developers. The latest kernel version in the gentoo tree is 5.2.9. I am compiling that now... > > >> AMD Opteron(tm) Processor 6174 AuthenticAMD GNU/Linux >> >> ***** >> btrfs --version >> >> btrfs-progs v4.19 > This is OK, but the change log will show lots of bug fixes here too. I > wouldn't make changes (no repair attempts at all, including chunk > recover or --repair) until you get some dev advice about the next > step. I already tried --repair as well, but it would not do anything anyway in the filesystem's current state. > > > > >> [ 8904.358084] BTRFS info (device sda1): turning on discard > Unexpected. I had still that in fstab for some reason. Not sure why/when I added that. These are not SSDs. > >> [ 8904.358088] BTRFS info (device sda1): allowing degraded mounts >> [ 8904.358089] BTRFS info (device sda1): disk space caching is enabled >> [ 8904.358091] BTRFS info (device sda1): has skinny extents >> [ 8904.361743] BTRFS warning (device sda1): devid 8 uuid >> 0e8b4aff-6d64-4d31-a135-705421928f94 is missing >> [ 8905.705036] BTRFS info (device sda1): bdev (null) errs: wr 0, rd >> 14809, flush 0, corrupt 4, gen 0 >> [ 8905.705041] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd >> 4, flush 0, corrupt 0, gen 0 >> [ 8905.705052] BTRFS info (device sda1): bdev /dev/sdf1 errs: wr 0, rd >> 10543, flush 0, corrupt 0, gen 0 >> [ 8905.705062] BTRFS info (device sda1): bdev /dev/sdc1 errs: wr 0, rd >> 8, flush 0, corrupt 0, gen 0 > four devices with read errors > > When was the last time the volume was scrubbed? Do you know for sure > these errors have not gone up at all since the last successful scrub? > And were any errors reported for that last scrub? Oh, that must have been quite a while ago. Sometime in 2018? Maybe? All these drives have been up and running for several years now. sda and sdc should still be fine, the replaced drive is sdb and sdf is next in line. > > >> I have tried all the mount / restore options listed here: >> https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/?tab=comments#comment-543490 > Good. Stick with ro attempts for now. Including if you want to try a > newer kernel. If it succeeds to mount ro, my advice is to update > backups so at least critical information isn't lost. Back up while you > can. Any repair attempt makes changes that will risk the data being > permanently lost. So it's important to be really deliberate about any > changes. I'll let you know, when I have the new kernel up and running. > > >> ... and all I keep getting is "bad tree block" errors. Superblocks seem >> fine (btrfs rescue super-reecover found no problem). I am considering >> trying "btrfs rescue chunk-recover" at this point. >> >> Could this help in my situation? What do you think? > I'm not sure if chunk recover can work on degraded volumes. Your best > bet is to not make any further changes to the volume itself. > > Preserve all logs. > ok -- *Liland IT GmbH* Ferlach ● Wien ● München Tel: +43 463 220111 Tel: +49 89 458 15 940 office@Liland.com https://Liland.com <https://Liland.com> Copyright © 2019 Liland IT GmbH Diese Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese Email irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorised copying, disclosure or distribution of the material in this email is strictly forbidden. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-04 4:39 ` Edmund Urbani @ 2019-09-04 5:36 ` Chris Murphy 2019-09-04 6:18 ` Edmund Urbani 2019-09-05 19:17 ` Edmund Urbani 0 siblings, 2 replies; 12+ messages in thread From: Chris Murphy @ 2019-09-04 5:36 UTC (permalink / raw) To: Edmund Urbani; +Cc: Btrfs BTRFS On Tue, Sep 3, 2019 at 10:41 PM Edmund Urbani <edmund.urbani@liland.com> wrote: > > Also there are a few of these: > Sep 1 21:10:17 phoenix kernel: ata6.00: exception Emask 0x0 SAct > 0x10000020 SErr 0x0 action 0x0 > Sep 1 21:10:17 phoenix kernel: ata6.00: irq_stat 0x40000008 > Sep 1 21:10:17 phoenix kernel: ata6.00: failed command: READ FPDMA QUEUED > Sep 1 21:10:17 phoenix kernel: ata6.00: cmd > 60/20:28:80:66:09/00:00:50:01:00/40 tag 5 ncq dma 16384 in\x0a > res 41/40:00:88:66:09/00:00:50:01:00/40 Emask 0x409 (media error) <F> > Sep 1 21:10:17 phoenix kernel: ata6.00: status: { DRDY ERR } > Sep 1 21:10:17 phoenix kernel: ata6.00: error: { UNC } > Sep 1 21:10:17 phoenix kernel: ata6.00: configured for UDMA/133 > Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 UNKNOWN(0x2003) > Result: hostbyte=0x00 driverbyte=0x08 > Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 Sense Key : 0x3 > [current] > Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 ASC=0x11 ASCQ=0x4 > Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 CDB: opcode=0x88 > 88 00 00 00 00 01 50 09 66 80 00 00 00 20 00 00 > Sep 1 21:10:17 phoenix kernel: print_req_error: I/O error, dev sdf, > sector 5637760640 > Sep 1 21:10:17 phoenix kernel: BTRFS error (device sdb1): bdev > /dev/sdf1 errs: wr 0, rd 289, flush 0, corrupt 0, gen 0 > Sep 1 21:10:17 phoenix kernel: ata6: EH complete > Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 > checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 > level 0 > Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 > checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 > level 0 > Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 > checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 > level 0 > Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 > checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 > level 0 > Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 > checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 > level 0 > Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 > checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 > level 0 > Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 > checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 > level 0 > Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 > checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 > level 0 > Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 > checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 > level 0 OK so the file system is not degraded, but sdb1 is giving you problems, so you've deleted it and its in the process of being removed (fs shrink, move chunks, and restripe). Here /dev/sdf has issued an uncorrectable read error. Classic case of bad sector. And btrfs is trying to get data off sdb1 to try and fix it, but this fails with checksum errors multiple times. So basically it is a two device failure for the stripe currently being read. It should still be possible to recover the stripe unless there is one more error from another drive - but the included dmesg doesn't go on far enough to tell us how this event turned out. > I am still looking for log entries related to the filesystem going > read-only. Not sure when exactly that happened and the logs are spammed > with plenty of the above... They're relevant because if there's a third failure at the same time, and if it affects metadata, reconstruction isn't possible, the metadata is missing. So then it's, what's missing and can it be manually reconstructed. It's super tedious. > >> [ 8904.358088] BTRFS info (device sda1): allowing degraded mounts > >> [ 8904.358089] BTRFS info (device sda1): disk space caching is enabled > >> [ 8904.358091] BTRFS info (device sda1): has skinny extents > >> [ 8904.361743] BTRFS warning (device sda1): devid 8 uuid > >> 0e8b4aff-6d64-4d31-a135-705421928f94 is missing > >> [ 8905.705036] BTRFS info (device sda1): bdev (null) errs: wr 0, rd > >> 14809, flush 0, corrupt 4, gen 0 > >> [ 8905.705041] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd > >> 4, flush 0, corrupt 0, gen 0 > >> [ 8905.705052] BTRFS info (device sda1): bdev /dev/sdf1 errs: wr 0, rd > >> 10543, flush 0, corrupt 0, gen 0 > >> [ 8905.705062] BTRFS info (device sda1): bdev /dev/sdc1 errs: wr 0, rd > >> 8, flush 0, corrupt 0, gen 0 > > four devices with read errors > > > > When was the last time the volume was scrubbed? Do you know for sure > > these errors have not gone up at all since the last successful scrub? > > And were any errors reported for that last scrub? > Oh, that must have been quite a while ago. Sometime in 2018? Maybe? All > these drives have been up and running for several years now. sda and sdc > should still be fine, the replaced drive is sdb and sdf is next in line. There's evidence of four drives with problems at some point in time. And there's evidence in the kernel messages above of at least two problems at the same time with the same stripe. So, all it takes is one more problem with that stripe, and then that stripe can't be recovered - and if it's a metadata stripe? That's 512KiB of metadata lost, which is quite a lot, it probably kills the file system, depending on where it happens. If it's data - no big deal. Btrfs won't even care, it will just report EIO and the path to the bad file, and continue on. The whole point of regular scrubs is to prevent single sector corruptions and failures. If you don't do that, they can accumulate over time and then it's a huge problem when just one drive dies. So when did you last do a scrub? Are they all the same make model drive? Do they all have the same SCT ERC value? And is that value, for all drives, less than the value found at /sys/block/sdN/device/timeout ? > > > > > >> I have tried all the mount / restore options listed here: > >> https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/?tab=comments#comment-543490 > > Good. Stick with ro attempts for now. Including if you want to try a > > newer kernel. If it succeeds to mount ro, my advice is to update > > backups so at least critical information isn't lost. Back up while you > > can. Any repair attempt makes changes that will risk the data being > > permanently lost. So it's important to be really deliberate about any > > changes. > I'll let you know, when I have the new kernel up and running. I think you should have all the original drives installed, and try to mount -o ro first. And if that doesn't work, try -o ro,degraded, and then we'll just have to see which drive it doesn't like. -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-04 5:36 ` Chris Murphy @ 2019-09-04 6:18 ` Edmund Urbani 2019-09-04 12:35 ` Piotr Szymaniak 2019-09-05 19:17 ` Edmund Urbani 1 sibling, 1 reply; 12+ messages in thread From: Edmund Urbani @ 2019-09-04 6:18 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS On 09/04/2019 07:36 AM, Chris Murphy wrote: > On Tue, Sep 3, 2019 at 10:41 PM Edmund Urbani <edmund.urbani@liland.com> wrote: >> Also there are a few of these: >> Sep 1 21:10:17 phoenix kernel: ata6.00: exception Emask 0x0 SAct >> 0x10000020 SErr 0x0 action 0x0 >> Sep 1 21:10:17 phoenix kernel: ata6.00: irq_stat 0x40000008 >> Sep 1 21:10:17 phoenix kernel: ata6.00: failed command: READ FPDMA QUEUED >> Sep 1 21:10:17 phoenix kernel: ata6.00: cmd >> 60/20:28:80:66:09/00:00:50:01:00/40 tag 5 ncq dma 16384 in\x0a >> res 41/40:00:88:66:09/00:00:50:01:00/40 Emask 0x409 (media error) <F> >> Sep 1 21:10:17 phoenix kernel: ata6.00: status: { DRDY ERR } >> Sep 1 21:10:17 phoenix kernel: ata6.00: error: { UNC } >> Sep 1 21:10:17 phoenix kernel: ata6.00: configured for UDMA/133 >> Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 UNKNOWN(0x2003) >> Result: hostbyte=0x00 driverbyte=0x08 >> Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 Sense Key : 0x3 >> [current] >> Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 ASC=0x11 ASCQ=0x4 >> Sep 1 21:10:17 phoenix kernel: sd 5:0:0:0: [sdf] tag#5 CDB: opcode=0x88 >> 88 00 00 00 00 01 50 09 66 80 00 00 00 20 00 00 >> Sep 1 21:10:17 phoenix kernel: print_req_error: I/O error, dev sdf, >> sector 5637760640 >> Sep 1 21:10:17 phoenix kernel: BTRFS error (device sdb1): bdev >> /dev/sdf1 errs: wr 0, rd 289, flush 0, corrupt 0, gen 0 >> Sep 1 21:10:17 phoenix kernel: ata6: EH complete >> Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 >> checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 >> level 0 >> Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 >> checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 >> level 0 >> Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 >> checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 >> level 0 >> Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 >> checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 >> level 0 >> Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 >> checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 >> level 0 >> Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 >> checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 >> level 0 >> Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 >> checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 >> level 0 >> Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 >> checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 >> level 0 >> Sep 1 21:10:17 phoenix kernel: BTRFS warning (device sdb1): sdb1 >> checksum verify failed on 70943861833728 wanted 49137758 found 776101D6 >> level 0 > OK so the file system is not degraded, but sdb1 is giving you > problems, so you've deleted it and its in the process of being removed > (fs shrink, move chunks, and restripe). I am about to install the old sdb in another system to try ddrescue on it and see what I can salvage. > > Here /dev/sdf has issued an uncorrectable read error. Classic case of > bad sector. And btrfs is trying to get data off sdb1 to try and fix > it, but this fails with checksum errors multiple times. So basically > it is a two device failure for the stripe currently being read. It > should still be possible to recover the stripe unless there is one > more error from another drive - but the included dmesg doesn't go on > far enough to tell us how this event turned out. I suspect that whatever is in sector 5637760640 on sdf is metadata. I grepped the log for it and that bad sector pops up 10000+ times. > > >> I am still looking for log entries related to the filesystem going >> read-only. Not sure when exactly that happened and the logs are spammed >> with plenty of the above... > They're relevant because if there's a third failure at the same time, > and if it affects metadata, reconstruction isn't possible, the > metadata is missing. So then it's, what's missing and can it be > manually reconstructed. It's super tedious. > apparently, this is where it went read-only: Sep 3 00:31:31 phoenix kernel: ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Sep 3 00:31:31 phoenix kernel: ata1.01: BMDMA stat 0x64 Sep 3 00:31:31 phoenix kernel: ata1.01: failed command: READ DMA EXT Sep 3 00:31:31 phoenix kernel: ata1.01: cmd 25/00:20:80:72:5a/00:00:a9:00:00/f0 tag 0 dma 16384 in\x0a res 51/40:20:80:72:5a/40:00:a9:00:00/f0 Emask 0x9 (media error) Sep 3 00:31:31 phoenix kernel: ata1.01: status: { DRDY ERR } Sep 3 00:31:31 phoenix kernel: ata1.01: error: { UNC } Sep 3 00:31:31 phoenix kernel: ata1.00: configured for UDMA/100 Sep 3 00:31:31 phoenix kernel: ata1.01: configured for UDMA/100 Sep 3 00:31:31 phoenix kernel: sd 0:0:1:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Sep 3 00:31:31 phoenix kernel: sd 0:0:1:0: [sdb] tag#0 Sense Key : 0x3 [current] Sep 3 00:31:31 phoenix kernel: sd 0:0:1:0: [sdb] tag#0 ASC=0x11 ASCQ=0x4 Sep 3 00:31:31 phoenix kernel: sd 0:0:1:0: [sdb] tag#0 CDB: opcode=0x88 88 00 00 00 00 00 a9 5a 72 80 00 00 00 20 00 00 Sep 3 00:31:31 phoenix kernel: print_req_error: I/O error, dev sdb, sector 2841277056 Sep 3 00:31:31 phoenix kernel: BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 0, rd 14786, flush 0, corrupt 0, gen 0 Sep 3 00:31:31 phoenix kernel: ata1: EH complete Sep 3 00:31:31 phoenix kernel: BTRFS info (device sdb1): read error corrected: ino 0 off 34958490861568 (dev /dev/sdb1 sector 2841275008) Sep 3 00:31:31 phoenix kernel: BTRFS info (device sdb1): read error corrected: ino 0 off 34958490865664 (dev /dev/sdb1 sector 2841275016) Sep 3 00:31:31 phoenix kernel: BTRFS info (device sdb1): read error corrected: ino 0 off 34958490869760 (dev /dev/sdb1 sector 2841275024) Sep 3 00:31:31 phoenix kernel: BTRFS info (device sdb1): read error corrected: ino 0 off 34958490873856 (dev /dev/sdb1 sector 2841275032) Sep 3 00:31:39 phoenix kernel: ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Sep 3 00:31:39 phoenix kernel: ata1.01: BMDMA stat 0x64 Sep 3 00:31:39 phoenix kernel: ata1.01: failed command: READ DMA EXT Sep 3 00:31:39 phoenix kernel: ata1.01: cmd 25/00:20:60:74:5b/00:00:a9:00:00/f0 tag 0 dma 16384 in\x0a res 51/40:20:60:74:5b/40:00:a9:00:00/f0 Emask 0x9 (media error) Sep 3 00:31:39 phoenix kernel: ata1.01: status: { DRDY ERR } Sep 3 00:31:39 phoenix kernel: ata1.01: error: { UNC } Sep 3 00:31:39 phoenix kernel: ata1.00: configured for UDMA/100 Sep 3 00:31:39 phoenix kernel: ata1.01: configured for UDMA/100 Sep 3 00:31:39 phoenix kernel: sd 0:0:1:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Sep 3 00:31:39 phoenix kernel: sd 0:0:1:0: [sdb] tag#0 Sense Key : 0x3 [current] Sep 3 00:31:39 phoenix kernel: sd 0:0:1:0: [sdb] tag#0 ASC=0x11 ASCQ=0x4 Sep 3 00:31:39 phoenix kernel: sd 0:0:1:0: [sdb] tag#0 CDB: opcode=0x88 88 00 00 00 00 00 a9 5b 74 60 00 00 00 20 00 00 Sep 3 00:31:39 phoenix kernel: print_req_error: I/O error, dev sdb, sector 2841343072 Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 0, rd 14787, flush 0, corrupt 0, gen 0 Sep 3 00:31:39 phoenix kernel: ata1: EH complete Sep 3 00:31:39 phoenix kernel: btree_readpage_end_io_hook: 8 callbacks suppressed Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): bad tree block start 3417494780583899951 34958760591360 Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): bad tree block start 3417494780583899951 34958760591360 Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): bad tree block start 3417494780583899951 34958760591360 Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): bad tree block start 3417494780583899951 34958760591360 Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): bad tree block start 3417494780583899951 34958760591360 Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): bad tree block start 3417494780583899951 34958760591360 Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): bad tree block start 3417494780583899951 34958760591360 Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): bad tree block start 3417494780583899951 34958760591360 Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): bad tree block start 3417494780583899951 34958760591360 Sep 3 00:31:39 phoenix kernel: BTRFS: error (device sdb1) in __btrfs_free_extent:7084: errno=-5 IO failure Sep 3 00:31:39 phoenix kernel: BTRFS info (device sdb1): forced readonly Sep 3 00:31:39 phoenix kernel: BTRFS: error (device sdb1) in btrfs_run_delayed_refs:3089: errno=-5 IO failure Sep 3 00:31:39 phoenix kernel: BTRFS error (device sdb1): pending csums is 24776704 > > > >>>> [ 8904.358088] BTRFS info (device sda1): allowing degraded mounts >>>> [ 8904.358089] BTRFS info (device sda1): disk space caching is enabled >>>> [ 8904.358091] BTRFS info (device sda1): has skinny extents >>>> [ 8904.361743] BTRFS warning (device sda1): devid 8 uuid >>>> 0e8b4aff-6d64-4d31-a135-705421928f94 is missing >>>> [ 8905.705036] BTRFS info (device sda1): bdev (null) errs: wr 0, rd >>>> 14809, flush 0, corrupt 4, gen 0 >>>> [ 8905.705041] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd >>>> 4, flush 0, corrupt 0, gen 0 >>>> [ 8905.705052] BTRFS info (device sda1): bdev /dev/sdf1 errs: wr 0, rd >>>> 10543, flush 0, corrupt 0, gen 0 >>>> [ 8905.705062] BTRFS info (device sda1): bdev /dev/sdc1 errs: wr 0, rd >>>> 8, flush 0, corrupt 0, gen 0 >>> four devices with read errors >>> >>> When was the last time the volume was scrubbed? Do you know for sure >>> these errors have not gone up at all since the last successful scrub? >>> And were any errors reported for that last scrub? >> Oh, that must have been quite a while ago. Sometime in 2018? Maybe? All >> these drives have been up and running for several years now. sda and sdc >> should still be fine, the replaced drive is sdb and sdf is next in line. > There's evidence of four drives with problems at some point in time. > And there's evidence in the kernel messages above of at least two > problems at the same time with the same stripe. So, all it takes is > one more problem with that stripe, and then that stripe can't be > recovered - and if it's a metadata stripe? That's 512KiB of metadata > lost, which is quite a lot, it probably kills the file system, > depending on where it happens. If it's data - no big deal. Btrfs won't > even care, it will just report EIO and the path to the bad file, and > continue on. > > The whole point of regular scrubs is to prevent single sector > corruptions and failures. If you don't do that, they can accumulate > over time and then it's a huge problem when just one drive dies. So > when did you last do a scrub? Are they all the same make model drive? > Do they all have the same SCT ERC value? And is that value, for all > drives, less than the value found at /sys/block/sdN/device/timeout ? I'll make sure to setup a cron job for monthly scrubbing once this is over. I really can't remember exactly when I last did a manual scrub. They are all WD Red 3TB, though not all the same generation. smartctl reports these models: Device Model: WDC WD30EFRX-68AX9N0 Device Model: WDC WD30EFRX-68EUZN0 The timeout under /sys is 30 for all of them. I am not sure what you mean by SCT ERC value and where to look for that. Here's all the info smartctl gives me about sda: smartctl 6.6 2017-11-05 r4594 [x86_64-linux-5.2.9-gentoo] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD30EFRX-68AX9N0 Serial Number: WD-WMC1T1311633 LU WWN Device Id: 5 0014ee 602ee2554 Firmware Version: 80.00A80 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Wed Sep 4 06:04:20 2019 -00 SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (40320) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 404) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 178 177 021 Pre-fail Always - 6058 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 61 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 020 020 000 Old_age Always - 58487 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 61 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 43 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 17 194 Temperature_Celsius 0x0022 110 098 000 Old_age Always - 40 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 13 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. > > > >>> >>>> I have tried all the mount / restore options listed here: >>>> https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/?tab=comments#comment-543490 >>> Good. Stick with ro attempts for now. Including if you want to try a >>> newer kernel. If it succeeds to mount ro, my advice is to update >>> backups so at least critical information isn't lost. Back up while you >>> can. Any repair attempt makes changes that will risk the data being >>> permanently lost. So it's important to be really deliberate about any >>> changes. >> I'll let you know, when I have the new kernel up and running. > I think you should have all the original drives installed, and try to > mount -o ro first. And if that doesn't work, try -o ro,degraded, and > then we'll just have to see which drive it doesn't like. > > > > -- > Chris Murphy Well, I already tried mounting with 5.2.9 in the meantime without success (without the original sdb drive). It still does not mount (mount -o ro,degraded /dev/sda1 /mnt/shared/): [ 209.459309] BTRFS info (device sdg1): allowing degraded mounts [ 209.459313] BTRFS info (device sdg1): disk space caching is enabled [ 209.459314] BTRFS info (device sdg1): has skinny extents [ 209.461246] BTRFS warning (device sdg1): devid 8 uuid 0e8b4aff-6d64-4d31-a135-705421928f94 is missing [ 209.544603] BTRFS warning (device sdg1): devid 8 uuid 0e8b4aff-6d64-4d31-a135-705421928f94 is missing [ 211.401375] BTRFS info (device sdg1): bdev (efault) errs: wr 0, rd 14809, flush 0, corrupt 4, gen 0 [ 211.401388] BTRFS info (device sdg1): bdev /dev/sdf1 errs: wr 0, rd 10543, flush 0, corrupt 0, gen 0 [ 211.401391] BTRFS info (device sdg1): bdev /dev/sda1 errs: wr 0, rd 4, flush 0, corrupt 0, gen 0 [ 211.401394] BTRFS info (device sdg1): bdev /dev/sdc1 errs: wr 0, rd 8, flush 0, corrupt 0, gen 0 [ 215.381805] BTRFS error (device sdg1): bad tree block start, want 34958581399552 have 12170572967447269873 [ 215.382603] BTRFS error (device sdg1): bad tree block start, want 34958581399552 have 12170572967447269873 [ 215.386155] BTRFS error (device sdg1): bad tree block start, want 34958581399552 have 12170572967447269873 [ 215.389539] BTRFS error (device sdg1): bad tree block start, want 34958581399552 have 12170572967447269873 [ 215.393053] BTRFS error (device sdg1): bad tree block start, want 34958581399552 have 12170572967447269873 [ 215.395223] BTRFS error (device sdg1): bad tree block start, want 34958581399552 have 12170572967447269873 [ 215.397188] BTRFS error (device sdg1): bad tree block start, want 34958581399552 have 12170572967447269873 [ 215.399307] BTRFS error (device sdg1): bad tree block start, want 34958581399552 have 12170572967447269873 [ 215.402410] BTRFS error (device sdg1): bad tree block start, want 34958581399552 have 12170572967447269873 [ 215.402447] BTRFS error (device sdg1): failed to read block groups: -5 [ 215.844527] BTRFS error (device sdg1): open_ctree failed I'm planning to install the copy of sdb once ddrescue is done. Regards, Edmund -- *Liland IT GmbH* Ferlach ● Wien ● München Tel: +43 463 220111 Tel: +49 89 458 15 940 office@Liland.com https://Liland.com <https://Liland.com> Copyright © 2019 Liland IT GmbH Diese Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese Email irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorised copying, disclosure or distribution of the material in this email is strictly forbidden. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-04 6:18 ` Edmund Urbani @ 2019-09-04 12:35 ` Piotr Szymaniak 2019-09-04 14:04 ` Edmund Urbani 0 siblings, 1 reply; 12+ messages in thread From: Piotr Szymaniak @ 2019-09-04 12:35 UTC (permalink / raw) To: Edmund Urbani; +Cc: Chris Murphy, Btrfs BTRFS On Wed, Sep 04, 2019 at 08:18:09AM +0200, Edmund Urbani wrote: > *snip* > The timeout under /sys is 30 for all of them. > > I am not sure what you mean by SCT ERC value and where to look for that. > Here's all the info smartctl gives me about sda: Try: smartctl -l scterc /dev/ice ie. one of my drives (also WD Red) outputs: $ smartctl -l scterc /dev/sdb smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.62] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) SCT ERC value should be lower then the value in /sys. Best regards, Piotr Szymaniak. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-04 12:35 ` Piotr Szymaniak @ 2019-09-04 14:04 ` Edmund Urbani 0 siblings, 0 replies; 12+ messages in thread From: Edmund Urbani @ 2019-09-04 14:04 UTC (permalink / raw) To: Piotr Szymaniak; +Cc: Chris Murphy, Btrfs BTRFS On 9/4/19 2:35 PM, Piotr Szymaniak wrote: > On Wed, Sep 04, 2019 at 08:18:09AM +0200, Edmund Urbani wrote: >> *snip* >> The timeout under /sys is 30 for all of them. >> >> I am not sure what you mean by SCT ERC value and where to look for that. >> Here's all the info smartctl gives me about sda: > Try: > smartctl -l scterc /dev/ice > > ie. one of my drives (also WD Red) outputs: > $ smartctl -l scterc /dev/sdb > smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.62] (local build) > Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org > > SCT Error Recovery Control: > Read: 70 (7.0 seconds) > Write: 70 (7.0 seconds) > > SCT ERC value should be lower then the value in /sys. > > > Best regards, > Piotr Szymaniak. Ok, thanks. Mine also all have 7 seconds set. Kind regards, Edmund -- *Liland IT GmbH* Ferlach ● Wien ● München Tel: +43 463 220111 Tel: +49 89 458 15 940 office@Liland.com https://Liland.com <https://Liland.com> Copyright © 2019 Liland IT GmbH Diese Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese Email irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorised copying, disclosure or distribution of the material in this email is strictly forbidden. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-04 5:36 ` Chris Murphy 2019-09-04 6:18 ` Edmund Urbani @ 2019-09-05 19:17 ` Edmund Urbani 2019-09-05 19:57 ` Chris Murphy 1 sibling, 1 reply; 12+ messages in thread From: Edmund Urbani @ 2019-09-05 19:17 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS On 04.09.2019 07:36, Chris Murphy wrote: > >>> >>>> I have tried all the mount / restore options listed here: >>>> https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/?tab=comments#comment-543490 >>> Good. Stick with ro attempts for now. Including if you want to try a >>> newer kernel. If it succeeds to mount ro, my advice is to update >>> backups so at least critical information isn't lost. Back up while you >>> can. Any repair attempt makes changes that will risk the data being >>> permanently lost. So it's important to be really deliberate about any >>> changes. >> I'll let you know, when I have the new kernel up and running. > I think you should have all the original drives installed, and try to > mount -o ro first. And if that doesn't work, try -o ro,degraded, and > then we'll just have to see which drive it doesn't like. Things are finally looking up. I have replaced both sdb and sdf with ddrescue'd copies. sdb had some 10MB bad sectors and sdf 8KB which could not be recovered. I am now able to mount the volume again. :) btrfsck /dev/sda1 Opening filesystem to check... Checking filesystem on /dev/sda1 UUID: 108df6ea-2846-4a88-8a50-61aedeef92b4 [1/7] checking root items checksum verify failed on 34958760591360 found E4E3BDB6 wanted 00000000 checksum verify failed on 34958760591360 found E4E3BDB6 wanted 00000000 parent transid verify failed on 34958760591360 wanted 3331734 found 1544337 checksum verify failed on 34958760591360 found 04DEBA71 wanted B9FBE54D checksum verify failed on 34958760591360 found 04DEBA71 wanted B9FBE54D bad tree block 34958760591360, bytenr mismatch, want=34958760591360, have=27967614209536 ERROR: failed to repair root items: Input/output error Anyway, I am about to mount it read-only again to try and backup a few things. And once I am done with that, should I run btrfs scrub? Kind regards, Edmund -- *Liland IT GmbH* Ferlach ● Wien ● München Tel: +43 463 220111 Tel: +49 89 458 15 940 office@Liland.com https://Liland.com <https://Liland.com> Copyright © 2019 Liland IT GmbH Diese Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese Email irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorised copying, disclosure or distribution of the material in this email is strictly forbidden. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-05 19:17 ` Edmund Urbani @ 2019-09-05 19:57 ` Chris Murphy 2019-09-05 20:44 ` Edmund Urbani 0 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2019-09-05 19:57 UTC (permalink / raw) To: Btrfs BTRFS; +Cc: Edmund Urbani, Qu Wenruo On Thu, Sep 5, 2019 at 1:18 PM Edmund Urbani <edmund.urbani@liland.com> wrote: > > > On 04.09.2019 07:36, Chris Murphy wrote: > > > >>> > >>>> I have tried all the mount / restore options listed here: > >>>> https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/?tab=comments#comment-543490 > >>> Good. Stick with ro attempts for now. Including if you want to try a > >>> newer kernel. If it succeeds to mount ro, my advice is to update > >>> backups so at least critical information isn't lost. Back up while you > >>> can. Any repair attempt makes changes that will risk the data being > >>> permanently lost. So it's important to be really deliberate about any > >>> changes. > >> I'll let you know, when I have the new kernel up and running. > > I think you should have all the original drives installed, and try to > > mount -o ro first. And if that doesn't work, try -o ro,degraded, and > > then we'll just have to see which drive it doesn't like. > > Things are finally looking up. I have replaced both sdb and sdf with > ddrescue'd copies. sdb had some 10MB bad sectors and sdf 8KB which could > not be recovered. > > I am now able to mount the volume again. :) > > btrfsck /dev/sda1 > > Opening filesystem to check... > Checking filesystem on /dev/sda1 > UUID: 108df6ea-2846-4a88-8a50-61aedeef92b4 > [1/7] checking root items > checksum verify failed on 34958760591360 found E4E3BDB6 wanted 00000000 > checksum verify failed on 34958760591360 found E4E3BDB6 wanted 00000000 > parent transid verify failed on 34958760591360 wanted 3331734 found 1544337 > checksum verify failed on 34958760591360 found 04DEBA71 wanted B9FBE54D > checksum verify failed on 34958760591360 found 04DEBA71 wanted B9FBE54D > bad tree block 34958760591360, bytenr mismatch, want=34958760591360, > have=27967614209536 > ERROR: failed to repair root items: Input/output error > > Anyway, I am about to mount it read-only again to try and backup a few > things. And once I am done with that, should I run btrfs scrub? Did it mount with ro alone, or did you need ro,degraded? I'm a little confused by the i/o error, which I'd expect will also produce a message at the same time in dmesg that will hint what the nature of the i/o error is. That suggests some kind of hardware issue still exists, even if it is an uncorrectable sector read error. For sure rw mounted scrubs can fix those thing, if enough redundancy exists, and those copies aren't also corrupt. But I'm off hand not sure whether 'btrfs check --repair' can fixup bad sectors like scrub can. Anyway, I suggest 'btfs check --repair' is a last resort, no matter the version of btrfs-progs. 'btrfs check' alone is safe. So in order: * you've done these *dmesg *btrfs check --readonly ##safe, makes no changes, maybe gives a hint of the problem *mount -o ro *mount -o ro,degraded mount -o rw ## all devices available mount -o rw,degraded I'm not sure a read only scrub helps much. It might be interesting? What you really want is to be able to mount rw with all devices, and then scrub. But even rw,degraded is better, because you must be rw mounted to make scrub repairs, and also to do device replacements. I personally would not do a degraded scrub, because that scrub requires reading the whole volume. If you're going to read the whole volume anyway, you might as well rebuild the bad/missing device, so that you can more quickly get back to undegraded/normal RAID6 operation. If you can only mount 'rw,degraded' we need to see 'btrfs fi show' and the kernel messages for the failed mount and the successful degraded mount, so we can figure out what devices are affected, maybe why, and then what the next step is. Anyone know if latest kernel and progs now reliably supports 'btrfs replace' for RAID6? For a bit it was recommended to do it the old way, with 'btrfs device add' followed by 'btrfs device delete'. Main difference for the user is that 'replace' requires that the replacement drive is at least as big (in bytes) as the one being replaced and also that 'replace' will not resize the volume after replacement is finished, that has to be done manually. Otherwise I think it's preferred? -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-05 19:57 ` Chris Murphy @ 2019-09-05 20:44 ` Edmund Urbani 2019-09-05 22:33 ` Chris Murphy 0 siblings, 1 reply; 12+ messages in thread From: Edmund Urbani @ 2019-09-05 20:44 UTC (permalink / raw) To: Chris Murphy, Btrfs BTRFS; +Cc: Qu Wenruo On 05.09.2019 21:57, Chris Murphy wrote: > On Thu, Sep 5, 2019 at 1:18 PM Edmund Urbani <edmund.urbani@liland.com> wrote: >> >> On 04.09.2019 07:36, Chris Murphy wrote: >>>>>> I have tried all the mount / restore options listed here: >>>>>> https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/?tab=comments#comment-543490 >>>>> Good. Stick with ro attempts for now. Including if you want to try a >>>>> newer kernel. If it succeeds to mount ro, my advice is to update >>>>> backups so at least critical information isn't lost. Back up while you >>>>> can. Any repair attempt makes changes that will risk the data being >>>>> permanently lost. So it's important to be really deliberate about any >>>>> changes. >>>> I'll let you know, when I have the new kernel up and running. >>> I think you should have all the original drives installed, and try to >>> mount -o ro first. And if that doesn't work, try -o ro,degraded, and >>> then we'll just have to see which drive it doesn't like. >> Things are finally looking up. I have replaced both sdb and sdf with >> ddrescue'd copies. sdb had some 10MB bad sectors and sdf 8KB which could >> not be recovered. >> >> I am now able to mount the volume again. :) >> >> btrfsck /dev/sda1 >> >> Opening filesystem to check... >> Checking filesystem on /dev/sda1 >> UUID: 108df6ea-2846-4a88-8a50-61aedeef92b4 >> [1/7] checking root items >> checksum verify failed on 34958760591360 found E4E3BDB6 wanted 00000000 >> checksum verify failed on 34958760591360 found E4E3BDB6 wanted 00000000 >> parent transid verify failed on 34958760591360 wanted 3331734 found 1544337 >> checksum verify failed on 34958760591360 found 04DEBA71 wanted B9FBE54D >> checksum verify failed on 34958760591360 found 04DEBA71 wanted B9FBE54D >> bad tree block 34958760591360, bytenr mismatch, want=34958760591360, >> have=27967614209536 >> ERROR: failed to repair root items: Input/output error >> >> Anyway, I am about to mount it read-only again to try and backup a few >> things. And once I am done with that, should I run btrfs scrub? > Did it mount with ro alone, or did you need ro,degraded? > > I'm a little confused by the i/o error, which I'd expect will also > produce a message at the same time in dmesg that will hint what the > nature of the i/o error is. That suggests some kind of hardware issue > still exists, even if it is an uncorrectable sector read error. For > sure rw mounted scrubs can fix those thing, if enough redundancy > exists, and those copies aren't also corrupt. But I'm off hand not > sure whether 'btrfs check --repair' can fixup bad sectors like scrub > can. > > Anyway, I suggest 'btfs check --repair' is a last resort, no matter > the version of btrfs-progs. 'btrfs check' alone is safe. So in order: > > * you've done these > > *dmesg > *btrfs check --readonly ##safe, makes no changes, maybe gives a hint > of the problem > *mount -o ro > *mount -o ro,degraded > mount -o rw ## all devices available > mount -o rw,degraded > > I'm not sure a read only scrub helps much. It might be interesting? > What you really want is to be able to mount rw with all devices, and > then scrub. > > But even rw,degraded is better, because you must be rw mounted to make > scrub repairs, and also to do device replacements. I personally would > not do a degraded scrub, because that scrub requires reading the whole > volume. If you're going to read the whole volume anyway, you might as > well rebuild the bad/missing device, so that you can more quickly get > back to undegraded/normal RAID6 operation. > > If you can only mount 'rw,degraded' we need to see 'btrfs fi show' and > the kernel messages for the failed mount and the successful degraded > mount, so we can figure out what devices are affected, maybe why, and > then what the next step is. > > Anyone know if latest kernel and progs now reliably supports 'btrfs > replace' for RAID6? For a bit it was recommended to do it the old way, > with 'btrfs device add' followed by 'btrfs device delete'. Main > difference for the user is that 'replace' requires that the > replacement drive is at least as big (in bytes) as the one being > replaced and also that 'replace' will not resize the volume after > replacement is finished, that has to be done manually. Otherwise I > think it's preferred? > I did not need the degraded option. And so far I see no HW I/O errors in dmesg. I have encountered a few errors while copying files and found these in the log: [ 3560.273634] btrfs_print_data_csum_error: 50 callbacks suppressed [ 3560.273639] BTRFS warning (device sdg1): csum failed root 262 ino 1838364 off 14467072 csum 0x98f94189 expected csum 0xcb3af09a mirror 1 [ 3560.825942] BTRFS warning (device sdg1): csum failed root 262 ino 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 2 [ 3560.826588] BTRFS warning (device sdg1): csum failed root 262 ino 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 3 [ 3560.827813] BTRFS warning (device sdg1): csum failed root 262 ino 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 4 [ 3560.829063] BTRFS warning (device sdg1): csum failed root 262 ino 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 5 [ 3560.830366] BTRFS warning (device sdg1): csum failed root 262 ino 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 6 [ 3560.831559] BTRFS warning (device sdg1): csum failed root 262 ino 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 7 [ 3560.832998] BTRFS warning (device sdg1): csum failed root 262 ino 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 8 [ 3560.834649] BTRFS warning (device sdg1): csum failed root 262 ino 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 9 [ 3560.836188] BTRFS warning (device sdg1): csum failed root 262 ino 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 10 and also: [ 3889.813300] btree_readpage_end_io_hook: 1860 callbacks suppressed [ 3889.813304] BTRFS error (device sdg1): bad tree block start, want 34958548107264 have 0 [ 3889.825732] BTRFS error (device sdg1): bad tree block start, want 34958548107264 have 12157064991241308972 [ 3889.826375] BTRFS error (device sdg1): bad tree block start, want 34958548107264 have 12157064991241308972 [ 3889.828149] BTRFS error (device sdg1): bad tree block start, want 34958548107264 have 12157064991241308972 [ 3889.829649] BTRFS error (device sdg1): bad tree block start, want 34958548107264 have 12157064991241308972 [ 3889.831592] BTRFS error (device sdg1): bad tree block start, want 34958548107264 have 12157064991241308972 [ 3889.833436] BTRFS error (device sdg1): bad tree block start, want 34958548107264 have 12157064991241308972 [ 3889.835458] BTRFS error (device sdg1): bad tree block start, want 34958548107264 have 12157064991241308972 [ 3889.836968] BTRFS error (device sdg1): bad tree block start, want 34958548107264 have 12157064991241308972 [ 3889.848545] BTRFS error (device sdg1): bad tree block start, want 34958548107264 have 12157064991241308972 I think that Input/output error btrfsck is showing is actually a filesystem checksum error and not triggered by faulty hardware (not anymore, I hope). If there actually are any more failing drives here, I will most likely do the ddrescue thing again. Currently there are no free SATA ports in that system to connect an additional drive, so I cannot simply add one (at least not without also installing an additional SATA controller). Anyway, I have some peace of mind now that most of my data is accessible again. Time to get some sleep... Thank you, Chris! Kind regards, Edmund -- *Liland IT GmbH* Ferlach ● Wien ● München Tel: +43 463 220111 Tel: +49 89 458 15 940 office@Liland.com https://Liland.com <https://Liland.com> Copyright © 2019 Liland IT GmbH Diese Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese Email irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorised copying, disclosure or distribution of the material in this email is strictly forbidden. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-05 20:44 ` Edmund Urbani @ 2019-09-05 22:33 ` Chris Murphy 2019-09-16 14:22 ` Urbani, Edmund 0 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2019-09-05 22:33 UTC (permalink / raw) To: Btrfs BTRFS; +Cc: Qu Wenruo, Edmund Urbani On Thu, Sep 5, 2019 at 2:44 PM Edmund Urbani <edmund.urbani@liland.com> wrote: > > I did not need the degraded option. And so far I see no HW I/O errors in > dmesg. I have encountered a few errors while copying files and found > these in the log: > > [ 3560.273634] btrfs_print_data_csum_error: 50 callbacks suppressed > [ 3560.273639] BTRFS warning (device sdg1): csum failed root 262 ino > 1838364 off 14467072 csum 0x98f94189 expected csum 0xcb3af09a mirror 1 Not a bit flip 0x98f94189 10011000111110010100000110001001 0xcb3af09a 11001011001110101111000010011010 > [ 3560.825942] BTRFS warning (device sdg1): csum failed root 262 ino > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 2 > [ 3560.826588] BTRFS warning (device sdg1): csum failed root 262 ino > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 3 > [ 3560.827813] BTRFS warning (device sdg1): csum failed root 262 ino > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 4 > [ 3560.829063] BTRFS warning (device sdg1): csum failed root 262 ino > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 5 > [ 3560.830366] BTRFS warning (device sdg1): csum failed root 262 ino > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 6 > [ 3560.831559] BTRFS warning (device sdg1): csum failed root 262 ino > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 7 > [ 3560.832998] BTRFS warning (device sdg1): csum failed root 262 ino > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 8 > [ 3560.834649] BTRFS warning (device sdg1): csum failed root 262 ino > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 9 > [ 3560.836188] BTRFS warning (device sdg1): csum failed root 262 ino > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 10 Also not a bit flip. 0xc0248289 11000000001001001000001010001001 0xcb3af09a 11001011001110101111000010011010 I'm not sure what it means or suggests has happened, that all the copies are wrong. Plausible with raid5 metadata. But seems unlikely with raid6 metadata, and also with all devices accounted for. The file itself is probably fine - these look like metadata complaints. If you find the file this inode belongs to, either duplicating it or deleting it is fine, should cause this bad leaf to just go away. Make sure you delete the correct file, each subvolume has its own list of inodes, this one is in subvol id 262. > > and also: > > [ 3889.813300] btree_readpage_end_io_hook: 1860 callbacks suppressed > [ 3889.813304] BTRFS error (device sdg1): bad tree block start, want > 34958548107264 have 0 > [ 3889.825732] BTRFS error (device sdg1): bad tree block start, want > 34958548107264 have 12157064991241308972 > [ 3889.826375] BTRFS error (device sdg1): bad tree block start, want > 34958548107264 have 12157064991241308972 > [ 3889.828149] BTRFS error (device sdg1): bad tree block start, want > 34958548107264 have 12157064991241308972 > [ 3889.829649] BTRFS error (device sdg1): bad tree block start, want > 34958548107264 have 12157064991241308972 > [ 3889.831592] BTRFS error (device sdg1): bad tree block start, want > 34958548107264 have 12157064991241308972 > [ 3889.833436] BTRFS error (device sdg1): bad tree block start, want > 34958548107264 have 12157064991241308972 > [ 3889.835458] BTRFS error (device sdg1): bad tree block start, want > 34958548107264 have 12157064991241308972 > [ 3889.836968] BTRFS error (device sdg1): bad tree block start, want > 34958548107264 have 12157064991241308972 > [ 3889.848545] BTRFS error (device sdg1): bad tree block start, want > 34958548107264 have 12157064991241308972 I'm skeptical that a scrub will fix these things, because Btrfs is passively scrubbing on reads, so any checksum mismatches should get fixed up, if they can be fixed, from reconstruction, on the fly as well as scrub. This is a different problem, I'm not sure how serious it is. I would still do the full scrub. And then unmount it and run 'btrfs check --mode=lowmem'. On a file system of this size it will take a long time. So maybe do it over a weekend > > I think that Input/output error btrfsck is showing is actually a > filesystem checksum error and not triggered by faulty hardware (not > anymore, I hope). If there actually are any more failing drives here, I > will most likely do the ddrescue thing again. Currently there are no > free SATA ports in that system to connect an additional drive, so I > cannot simply add one (at least not without also installing an > additional SATA controller). I suggest start planning how to migrate the data to a new Btrfs volume. If the problems can't be repaired, this becomes inevitable. A reasonable strategy is to take read-only snapshots of each subvolume you want to preserve. And either 'btrfs send/receive' or 'rsync' to new storage. That way you can keep using the volume rw in the meantime. Once that completes, do another read only snapshot of each subvolume, and do an incremental 'send -p' or rsync to migrate the much smaller changes. -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unmountable degraded BTRFS RAID6 filesystem 2019-09-05 22:33 ` Chris Murphy @ 2019-09-16 14:22 ` Urbani, Edmund 0 siblings, 0 replies; 12+ messages in thread From: Urbani, Edmund @ 2019-09-16 14:22 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS, Qu Wenruo Am Fr., 6. Sept. 2019 um 00:33 Uhr schrieb Chris Murphy <lists@colorremedies.com>: > > On Thu, Sep 5, 2019 at 2:44 PM Edmund Urbani <edmund.urbani@liland.com> wrote: > > > > I did not need the degraded option. And so far I see no HW I/O errors in > > dmesg. I have encountered a few errors while copying files and found > > these in the log: > > > > [ 3560.273634] btrfs_print_data_csum_error: 50 callbacks suppressed > > [ 3560.273639] BTRFS warning (device sdg1): csum failed root 262 ino > > 1838364 off 14467072 csum 0x98f94189 expected csum 0xcb3af09a mirror 1 > > Not a bit flip > 0x98f94189 > 10011000111110010100000110001001 > 0xcb3af09a > 11001011001110101111000010011010 > > > > [ 3560.825942] BTRFS warning (device sdg1): csum failed root 262 ino > > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 2 > > [ 3560.826588] BTRFS warning (device sdg1): csum failed root 262 ino > > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 3 > > [ 3560.827813] BTRFS warning (device sdg1): csum failed root 262 ino > > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 4 > > [ 3560.829063] BTRFS warning (device sdg1): csum failed root 262 ino > > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 5 > > [ 3560.830366] BTRFS warning (device sdg1): csum failed root 262 ino > > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 6 > > [ 3560.831559] BTRFS warning (device sdg1): csum failed root 262 ino > > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 7 > > [ 3560.832998] BTRFS warning (device sdg1): csum failed root 262 ino > > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 8 > > [ 3560.834649] BTRFS warning (device sdg1): csum failed root 262 ino > > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 9 > > [ 3560.836188] BTRFS warning (device sdg1): csum failed root 262 ino > > 1838364 off 14467072 csum 0xc0248289 expected csum 0xcb3af09a mirror 10 > > Also not a bit flip. > 0xc0248289 > 11000000001001001000001010001001 > 0xcb3af09a > 11001011001110101111000010011010 > > I'm not sure what it means or suggests has happened, that all the > copies are wrong. Plausible with raid5 metadata. But seems unlikely > with raid6 metadata, and also with all devices accounted for. > > The file itself is probably fine - these look like metadata > complaints. If you find the file this inode belongs to, either > duplicating it or deleting it is fine, should cause this bad leaf to > just go away. Make sure you delete the correct file, each subvolume > has its own list of inodes, this one is in subvol id 262. > > > > > and also: > > > > [ 3889.813300] btree_readpage_end_io_hook: 1860 callbacks suppressed > > [ 3889.813304] BTRFS error (device sdg1): bad tree block start, want > > 34958548107264 have 0 > > [ 3889.825732] BTRFS error (device sdg1): bad tree block start, want > > 34958548107264 have 12157064991241308972 > > [ 3889.826375] BTRFS error (device sdg1): bad tree block start, want > > 34958548107264 have 12157064991241308972 > > [ 3889.828149] BTRFS error (device sdg1): bad tree block start, want > > 34958548107264 have 12157064991241308972 > > [ 3889.829649] BTRFS error (device sdg1): bad tree block start, want > > 34958548107264 have 12157064991241308972 > > [ 3889.831592] BTRFS error (device sdg1): bad tree block start, want > > 34958548107264 have 12157064991241308972 > > [ 3889.833436] BTRFS error (device sdg1): bad tree block start, want > > 34958548107264 have 12157064991241308972 > > [ 3889.835458] BTRFS error (device sdg1): bad tree block start, want > > 34958548107264 have 12157064991241308972 > > [ 3889.836968] BTRFS error (device sdg1): bad tree block start, want > > 34958548107264 have 12157064991241308972 > > [ 3889.848545] BTRFS error (device sdg1): bad tree block start, want > > 34958548107264 have 12157064991241308972 > > I'm skeptical that a scrub will fix these things, because Btrfs is > passively scrubbing on reads, so any checksum mismatches should get > fixed up, if they can be fixed, from reconstruction, on the fly as > well as scrub. This is a different problem, I'm not sure how serious > it is. > > I would still do the full scrub. And then unmount it and run 'btrfs > check --mode=lowmem'. On a file system of this size it will take a > long time. So maybe do it over a weekend > > > > > I think that Input/output error btrfsck is showing is actually a > > filesystem checksum error and not triggered by faulty hardware (not > > anymore, I hope). If there actually are any more failing drives here, I > > will most likely do the ddrescue thing again. Currently there are no > > free SATA ports in that system to connect an additional drive, so I > > cannot simply add one (at least not without also installing an > > additional SATA controller). > > I suggest start planning how to migrate the data to a new Btrfs > volume. If the problems can't be repaired, this becomes inevitable. A > reasonable strategy is to take read-only snapshots of each subvolume > you want to preserve. And either 'btrfs send/receive' or 'rsync' to > new storage. That way you can keep using the volume rw in the > meantime. Once that completes, do another read only snapshot of each > subvolume, and do an incremental 'send -p' or rsync to migrate the > much smaller changes. > > > -- > Chris Murphy Here's a little status update. I am still in the process of salvaging files (remounting rw did not work for long and btrfs soon reverted to read-only state and I left it that way for now). After completing my first rsync pass I was still missing several large directory trees and found corresponding errors in the logs: Sep 15 20:34:39 phoenix kernel: BTRFS error (device sdg1): parent transid verify failed on 34960626352128 wanted 3332854 found 3332691 I remounted with ro,recover,nospace_cache,clear_cache. Now I am able to access more of the filesystem, but some errors still remain. I am seeing plenty of csum errors in the logs: Sep 16 12:08:53 phoenix kernel: BTRFS info (device sdg1): no csum found for inode 6126287 start 1673527296 then there's these (for all 10 mirrors): Sep 16 12:09:13 phoenix kernel: BTRFS warning (device sdg1): csum failed root 261 ino 6126287 off 1734606848 csum 0x7430ddcb expected csum 0x00000000 mirror 10 curiously at least the recent log entries all refer to inode 6126287 (start, offset etc. vary). And then there's also still occasionally this: Sep 16 12:09:19 phoenix kernel: BTRFS error (device sdg1): parent transid verify failed on 34960627597312 wanted 3332854 found 3332691 I'll investigate the logs further when the second rsync pass is done. Kind regards, Edmund -- *Liland IT GmbH* Ferlach ● Wien ● München Tel: +43 463 220111 Tel: +49 89 458 15 940 office@Liland.com https://Liland.com <https://Liland.com> Copyright © 2019 Liland IT GmbH Diese Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese Email irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorised copying, disclosure or distribution of the material in this email is strictly forbidden. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2019-09-16 14:22 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-09-03 22:20 Unmountable degraded BTRFS RAID6 filesystem Edmund Urbani 2019-09-03 23:30 ` Chris Murphy 2019-09-04 4:39 ` Edmund Urbani 2019-09-04 5:36 ` Chris Murphy 2019-09-04 6:18 ` Edmund Urbani 2019-09-04 12:35 ` Piotr Szymaniak 2019-09-04 14:04 ` Edmund Urbani 2019-09-05 19:17 ` Edmund Urbani 2019-09-05 19:57 ` Chris Murphy 2019-09-05 20:44 ` Edmund Urbani 2019-09-05 22:33 ` Chris Murphy 2019-09-16 14:22 ` Urbani, Edmund
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).