mount troubles after crash

mount troubles after crash

[Patrick Schmid]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

Hi togehter,

during a balance run, the server crash. after the crash i got the 
attached error messages....

Kernel: 4.9.11
Btrfs-Tools: v4.9.1

btrfs check --repair failed with:

couldn't open RDWR because of unsupported option features (3).
Open ctree failed


is there a way to fix this 140TB filesystem?

Regards Patrick
-- 
Patrick Schmid  <schmid@phys.ethz.ch>     support: +41 44 633 2668
IT Services Group, HPT H 8                voice:   +41 44 633 3997
Departement Physik, ETH Zurich
CH-8093 Zurich, Switzerland

[-- Attachment #2: after_crash.txt --]
[-- Type: text/plain, Size: 6137 bytes --]

Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.622208] BUG: unable to handle kernel paging request at fffffffffffffe10 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.630033] IP: [<ffffffffa093e1db>] qgroup_fix_relocated_data_extents+0x2b/0x290 [btrfs] 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.639224] PGD 1c0a067  
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.641864] PUD 1c0c067  
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.644708] PMD 0  
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.645306]  
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.646983] Oops: 0000 [#1] SMP 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.650495] Modules linked in: nfsv3(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_tcpudp(E) xt_multiport(E) iptable_filter(E) ip_tables(E) x_tables(E) autofs4(E) ib_iser(E) rdma_cm(E) iw_cm(E) ib_cm(E) ib_core(E) configfs(E) iscsi_tcp(E) libiscsi_tcp(E) libiscsi(E) scsi_transport_iscsi(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) sunrpc(E) fscache(E) btrfs(E) xor(E) raid6_pq(E) sb_edac(E) edac_core(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) ablk_helper(E) cryptd(E) bonding(E) ipmi_ssif(E) mousedev(E) lpc_ich(E) wmi(E) mei_me(E) mei(E) ioatdma(E) shpchp(E) tpm_tis(E) tpm_tis_core(E) ipmi_si(E) ipmi_poweroff(E) tcp_nv(E) ipmi_devintf(E) ipmi_msghandler(E) 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.730264]  sch_fq(E) hid_generic(E) igb(E) isci(E) ixgbe(E) i2c_algo_bit(E) libsas(E) dca(E) ahci(E) usbhid(E) ptp(E) scsi_transport_sas(E) arcmsr(OE) libahci(E) hid(E) mdio(E) pps_core(E) 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.747985] CPU: 6 PID: 86374 Comm: mount Tainted: G           OE   4.9.11-stable-blkmq #41 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.757389] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.03.0003.041920141333 04/19/2014 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.768937] task: ffff880103230000 task.stack: ffffc90026238000 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.775601] RIP: 0010:[<ffffffffa093e1db>]  [<ffffffffa093e1db>] qgroup_fix_relocated_data_extents+0x2b/0x290 [btrfs] 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.787571] RSP: 0018:ffffc9002623ba00  EFLAGS: 00010246 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.793542] RAX: 0000000000000000 RBX: ffff880b6b60c000 RCX: 0000000000000000 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.801559] RDX: ffff880103230000 RSI: ffff881f18e2d000 RDI: ffff881f48535f40 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.809572] RBP: ffffc9002623ba78 R08: 000060e0008071d0 R09: ffff881f48535f40 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.817588] R10: 000000009cd50001 R11: ffff880d9cd508c0 R12: ffff881ff59b5800 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.825604] R13: ffffc9002623baa0 R14: 0000000000000000 R15: 0000000000000000 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.833619] FS:  00007fb6570ee880(0000) GS:ffff881ffe400000(0000) knlGS:0000000000000000 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.842731] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.849189] CR2: fffffffffffffe10 CR3: 0000000c93c9d000 CR4: 00000000000406e0 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.857211] Stack: 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.859498]  ffff881f48535f40 0000000000000801 ffffc9002623ba68 ffffffffa08ea66b 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.867884]  ffff881f48535f40 0000000000000000 ffff880103230000 0000000000000000 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.876269]  ffff880dc489a1c0 ffff881ff59b5800 ffff880dc489a1c0 ffff881ff59b5800 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.884655] Call Trace: 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.887444]  [<ffffffffa08ea66b>] ? start_transaction+0x9b/0x4b0 [btrfs] 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.894986]  [<ffffffffa0942968>] btrfs_recover_relocation+0x378/0x420 [btrfs] 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.903136]  [<ffffffffa08e6cb7>] open_ctree+0x22e7/0x27f0 [btrfs] 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.910089]  [<ffffffffa08ba804>] btrfs_mount+0xca4/0xec0 [btrfs] 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.916942]  [<ffffffff813c8c6e>] ? find_next_zero_bit+0x1e/0x20 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.923703]  [<ffffffff811a9cee>] ? pcpu_next_unpop+0x3e/0x50 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.930164]  [<ffffffff813c8c49>] ? find_next_bit+0x19/0x20 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.936432]  [<ffffffff81213999>] mount_fs+0x39/0x160 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.942114]  [<ffffffff811aaf45>] ? __alloc_percpu+0x15/0x20 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.948476]  [<ffffffff8122fa47>] vfs_kern_mount+0x67/0x110 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.954741]  [<ffffffffa08b9ceb>] btrfs_mount+0x18b/0xec0 [btrfs] 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.961590]  [<ffffffff813c8c6e>] ? find_next_zero_bit+0x1e/0x20 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.968342]  [<ffffffff81213999>] mount_fs+0x39/0x160 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.974025]  [<ffffffff811aaf45>] ? __alloc_percpu+0x15/0x20 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.980386]  [<ffffffff8122fa47>] vfs_kern_mount+0x67/0x110 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.986644]  [<ffffffff8123208b>] do_mount+0x1bb/0xc80 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.992423]  [<ffffffff81232e53>] SyS_mount+0x83/0xd0 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6011.998107]  [<ffffffff817c417b>] entry_SYSCALL_64_fastpath+0x1e/0xad 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6012.005343] Code: 66 66 66 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 50 48 8b 46 08 4c 8b 76 10 48 89 7d a8 48 8b 98 f0 01 00 00 31 c0 <4d> 8b ae 10 fe ff ff 48 8b 53 20 83 e2 40 74 09 80 be b0 05 00  
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6012.027375] RIP  [<ffffffffa093e1db>] qgroup_fix_relocated_data_extents+0x2b/0x290 [btrfs] 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6012.036702]  RSP <ffffc9002623ba00> 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6012.040631] CR2: fffffffffffffe10 
Feb 20 14:42:01 phd-bkp-gw kernel: [ 6012.044758] ---[] end trace a297bd1ef95a0092 ]--- 

Re: mount troubles after crash

[Qu Wenruo]

At 02/20/2017 10:13 PM, Patrick Schmid wrote:
> Hi togehter,
>
> during a balance run, the server crash. after the crash i got the
> attached error messages....
>
> Kernel: 4.9.11

A known bug when qgroup is enabled.
Fixed in v4.10-rcs.

Please mount using v4.10 kernels and it will mount without problem.

Thanks,
Qu

> Btrfs-Tools: v4.9.1
>
> btrfs check --repair failed with:
>
> couldn't open RDWR because of unsupported option features (3).
> Open ctree failed
>
>
> is there a way to fix this 140TB filesystem?
>
> Regards Patrick

Re: mount troubles after crash

[Patrick Schmid]

Hi Qu

> A known bug when qgroup is enabled.
> Fixed in v4.10-rcs.
>
> Please mount using v4.10 kernels and it will mount without problem.

I have rebootet with the v4.10 kernel and everything works again!
Thank you very much, you save me a lot of work!

Regards Patrick
--
Patrick Schmid  <schmid@phys.ethz.ch>     support: +41 44 633 2668
IT Services Group, HPT H 8                voice:   +41 44 633 3997
Departement Physik, ETH Zurich
CH-8093 Zurich, Switzerland