* linux 4.7.0 rbd client kernel panic when OSD process was killed by OOM
@ 2016-08-08 19:57 Victor Payno
2016-08-08 20:14 ` Ilya Dryomov
0 siblings, 1 reply; 6+ messages in thread
From: Victor Payno @ 2016-08-08 19:57 UTC (permalink / raw)
To: Ilya Dryomov, Patrick McLean, Roderick Colenbrander,
Cliff Pajaro, Ceph Development
We have another problem where an RBD client was killed when an OSD was
killed by the OOM on a server. The servers have 4.4.16 kernels.
ams2 login: [789881.620147] ------------[ cut here ]------------
[789881.625094] kernel BUG at drivers/block/rbd.c:4638!
[789881.630311] invalid opcode: 0000 [#1] SMP
[789881.634650] Modules linked in: rbd libceph sg rpcsec_gss_krb5
xt_nat xt_UDPLB(O) xt_multiport xt_addrtype iptable_mangle iptable_raw
iptable_nat nf_nat_ipv4 nf_nat ext4 jbd2 mbcache x86_pkg_temp_thermal
gkuart(O) usbserial ie31200_edac edac_core tpm_tis raid1 crc32c_intel
[789881.661718] CPU: 4 PID: 4111 Comm: kworker/u16:0 Tainted: G
O 4.7.0-vanilla-ams-3 #1
[789881.671091] Hardware name: Quanta T6BC-S1N/T6BC, BIOS T6BC2A01 03/26/2014
[789881.678212] Workqueue: ceph-watch-notify do_watch_notify [libceph]
[789881.684814] task: ffff88032069ea00 ti: ffff8803f0c90000 task.ti:
ffff8803f0c90000
[789881.692802] RIP: 0010:[<ffffffffa016d1c9>] [<ffffffffa016d1c9>]
rbd_dev_header_info+0x5a9/0x940 [rbd]
[789881.702702] RSP: 0018:ffff8803f0c93d30 EFLAGS: 00010286
[789881.708344] RAX: 0000000000000077 RBX: ffff8802a6a63800 RCX:
0000000000000000
[789881.715985] RDX: 0000000000000077 RSI: ffff88041fd0dd08 RDI:
ffff88041fd0dd08
[789881.723625] RBP: ffff8803f0c93d98 R08: 0000000000000030 R09:
0000000000000000
[789881.731261] R10: 0000000000000000 R11: 0000000000004479 R12:
ffff8800d6eaf000
[789881.738899] R13: ffff8802a6a639b0 R14: 0000000000000000 R15:
ffff880327e6e780
[789881.746533] FS: 0000000000000000(0000) GS:ffff88041fd00000(0000)
knlGS:0000000000000000
[789881.755120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[789881.761197] CR2: 00007fbb18242838 CR3: 0000000001e07000 CR4:
00000000001406e0
[789881.768846] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[789881.776482] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[789881.784118] Stack:
[789881.786457] ffffffff8113a91a ffff88032069ea00 ffff88041fd17ef0
ffff88041fd17ef0
[789881.794713] ffff88041fd17ef0 0000000000030289 ffff8803f0c93dd8
ffffffff8113d968
[789881.802965] ffff8802a6a63800 ffff8800d6eaf000 ffff8802a6a639b0
0000000000000000
[789881.811207] Call Trace:
[789881.813988] [<ffffffff8113a91a>] ? update_curr+0x8a/0x110
[789881.819810] [<ffffffff8113d968>] ? dequeue_task_fair+0x618/0x1150
[789881.826321] [<ffffffffa016d591>] rbd_dev_refresh+0x31/0xf0 [rbd]
[789881.832760] [<ffffffffa016d719>] rbd_watch_cb+0x29/0xa0 [rbd]
[789881.838930] [<ffffffffa0138fdc>] do_watch_notify+0x4c/0x80 [libceph]
[789881.845706] [<ffffffff811258e9>] process_one_work+0x149/0x3c0
[789881.856639] [<ffffffff81125bae>] worker_thread+0x4e/0x490
[789881.862453] [<ffffffff81125b60>] ? process_one_work+0x3c0/0x3c0
[789881.868823] [<ffffffff8112b1e9>] kthread+0xc9/0xe0
[789881.874033] [<ffffffff8185e4ff>] ret_from_fork+0x1f/0x40
[789881.879764] [<ffffffff8112b120>] ? kthread_create_on_node+0x170/0x170
[789881.886618] Code: 0b 44 8b 6d b8 e9 1d ff ff ff 48 c7 c1 f0 00 17
a0 ba 1e 12 00 00 48 c7 c6 90 0e 17 a0 48 c7 c7 20 f8 16 a0 31 c0 e8
8a 5d 08 e1 <0f> 0b 75 14 49 8b 7f 68 41 bd 92 ff ff ff e8 d4 e0 fc ff
e9 dc
[789881.911744] RIP [<ffffffffa016d1c9>] rbd_dev_header_info+0x5a9/0x940 [rbd]
[789881.919116] RSP <ffff8803f0c93d30>
[789881.922989] ---[ end trace 12b8d1c2ed74d6c1 ]---
[789881.927971] BUG: unable to handle kernel paging request at ffffffffffffffd8
[789881.935435] IP: [<ffffffff8112b821>] kthread_data+0x11/0x20
[789881.941427] PGD 1e0a067 PUD 1e0c067 PMD 0
[789881.946117] Oops: 0000 [#2] SMP
[789881.949591] Modules linked in: rbd libceph sg rpcsec_gss_krb5
xt_nat xt_UDPLB(O) xt_multiport xt_addrtype iptable_mangle iptable_raw
iptable_nat nf_nat_ipv4 nf_nat ext4 jbd2 mbcache x86_pkg_temp_thermal
gkuart(O) usbserial ie31200_edac edac_core tpm_tis raid1 crc32c_intel
[789881.976900] CPU: 4 PID: 4111 Comm: kworker/u16:0 Tainted: G D
O 4.7.0-vanilla-ams-3 #1
[789881.986280] Hardware name: Quanta T6BC-S1N/T6BC, BIOS T6BC2A01 03/26/2014
[789881.993410] task: ffff88032069ea00 ti: ffff8803f0c90000 task.ti:
ffff8803f0c90000
[789882.001411] RIP: 0010:[<ffffffff8112b821>] [<ffffffff8112b821>]
kthread_data+0x11/0x20
[789882.010024] RSP: 0018:ffff8803f0c93a28 EFLAGS: 00010002
[789882.015682] RAX: 0000000000000000 RBX: ffff88041fd17e80 RCX:
0000000000000004
[789882.023342] RDX: ffff88040f005000 RSI: ffff88032069ea00 RDI:
ffff88032069ea00
[789882.030996] RBP: ffff8803f0c93a30 R08: 0000000000000000 R09:
0000000000079800
[789882.038645] R10: 0000000000000001 R11: 0000000000000001 R12:
0000000000017e80
[789882.046288] R13: 0000000000000000 R14: ffff88032069eec0 R15:
ffff88032069ea00
[789882.053926] FS: 0000000000000000(0000) GS:ffff88041fd00000(0000)
knlGS:0000000000000000
[789882.062524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[789882.068603] CR2: 0000000000000028 CR3: 0000000001e07000 CR4:
00000000001406e0
[789882.076261] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[789882.083920] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[789882.091577] Stack:
[789882.093935] ffffffff8112645e ffff8803f0c93a78 ffffffff8185ab3e
ffff88032069ea00
[789882.102194] ffff8803f0c93a78 ffff8803f0c94000 ffff8803f0c93ad0
ffff8803f0c936e8
[789882.110438] ffff88040d5c8000 0000000000000000 ffff8803f0c93a90
ffffffff8185aef5
[789882.118689] Call Trace:
[789882.121484] [<ffffffff8112645e>] ? wq_worker_sleeping+0xe/0x90
[789882.127752] [<ffffffff8185ab3e>] __schedule+0x36e/0x6f0
[789882.133411] [<ffffffff8185aef5>] schedule+0x35/0x80
[789882.138712] [<ffffffff81110ff9>] do_exit+0x739/0xb50
[789882.144098] [<ffffffff8108833c>] oops_end+0x9c/0xd0
[789882.149400] [<ffffffff810887ab>] die+0x4b/0x70
[789882.154276] [<ffffffff81085b26>] do_trap+0xb6/0x150
[789882.159583] [<ffffffff81085d87>] do_error_trap+0x77/0xe0
[789882.165322] [<ffffffffa016d1c9>] ? rbd_dev_header_info+0x5a9/0x940 [rbd]
[789882.172446] [<ffffffff811d7a3d>] ? irq_work_queue+0x6d/0x80
[789882.178441] [<ffffffff811575d4>] ? wake_up_klogd+0x34/0x40
[789882.184363] [<ffffffff81157aa6>] ? console_unlock+0x4c6/0x510
[789882.190532] [<ffffffff810863c0>] do_invalid_op+0x20/0x30
[789882.196265] [<ffffffff8185fb6e>] invalid_op+0x1e/0x30
[789882.201740] [<ffffffffa016d1c9>] ? rbd_dev_header_info+0x5a9/0x940 [rbd]
[789882.208866] [<ffffffff8113a91a>] ? update_curr+0x8a/0x110
[789882.214694] [<ffffffff8113d968>] ? dequeue_task_fair+0x618/0x1150
[789882.221225] [<ffffffffa016d591>] rbd_dev_refresh+0x31/0xf0 [rbd]
[789882.227662] [<ffffffffa016d719>] rbd_watch_cb+0x29/0xa0 [rbd]
[789882.233855] [<ffffffffa0138fdc>] do_watch_notify+0x4c/0x80 [libceph]
[789882.240647] [<ffffffff811258e9>] process_one_work+0x149/0x3c0
[789882.246811] [<ffffffff81125bae>] worker_thread+0x4e/0x490
[789882.252629] [<ffffffff81125b60>] ? process_one_work+0x3c0/0x3c0
[789882.258969] [<ffffffff8112b1e9>] kthread+0xc9/0xe0
[789882.264182] [<ffffffff8185e4ff>] ret_from_fork+0x1f/0x40
[789882.269917] [<ffffffff8112b120>] ? kthread_create_on_node+0x170/0x170
[789882.276784] Code: 02 00 00 00 e8 a1 fd ff ff 5d c3 0f 1f 44 00 00
66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 60 04 00 00 55
48 89 e5 5d <48> 8b 40 d8 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00
00 55
[789882.301941] RIP [<ffffffff8112b821>] kthread_data+0x11/0x20
[789882.308016] RSP <ffff8803f0c93a28>
[789882.311847] CR2: ffffffffffffffd8
[789882.315505] ---[ end trace 12b8d1c2ed74d6c2 ]---
[789882.320462] Fixing recursive fault but reboot is needed!
--
Victor Payno
ビクター·ペイン
Sr. Release Engineer
シニアリリースエンジニア
Gaikai, a Sony Computer Entertainment Company ∆○×□
ガイカイ、ソニー・コンピュータエンタテインメント傘下会社
65 Enterprise
Aliso Viejo, CA 92656 USA
Web: www.gaikai.com
Email: vpayno@gaikai.com
Phone: (949) 330-6850
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: linux 4.7.0 rbd client kernel panic when OSD process was killed by OOM
2016-08-08 19:57 linux 4.7.0 rbd client kernel panic when OSD process was killed by OOM Victor Payno
@ 2016-08-08 20:14 ` Ilya Dryomov
2016-08-08 20:16 ` Victor Payno
0 siblings, 1 reply; 6+ messages in thread
From: Ilya Dryomov @ 2016-08-08 20:14 UTC (permalink / raw)
To: Victor Payno
Cc: Patrick McLean, Roderick Colenbrander, Cliff Pajaro, Ceph Development
On Mon, Aug 8, 2016 at 9:57 PM, Victor Payno <vpayno@gaikai.com> wrote:
> We have another problem where an RBD client was killed when an OSD was
> killed by the OOM on a server. The servers have 4.4.16 kernels.
>
> ams2 login: [789881.620147] ------------[ cut here ]------------
> [789881.625094] kernel BUG at drivers/block/rbd.c:4638!
> [789881.630311] invalid opcode: 0000 [#1] SMP
> [789881.634650] Modules linked in: rbd libceph sg rpcsec_gss_krb5
> xt_nat xt_UDPLB(O) xt_multiport xt_addrtype iptable_mangle iptable_raw
> iptable_nat nf_nat_ipv4 nf_nat ext4 jbd2 mbcache x86_pkg_temp_thermal
> gkuart(O) usbserial ie31200_edac edac_core tpm_tis raid1 crc32c_intel
> [789881.661718] CPU: 4 PID: 4111 Comm: kworker/u16:0 Tainted: G
> O 4.7.0-vanilla-ams-3 #1
> [789881.671091] Hardware name: Quanta T6BC-S1N/T6BC, BIOS T6BC2A01 03/26/2014
> [789881.678212] Workqueue: ceph-watch-notify do_watch_notify [libceph]
> [789881.684814] task: ffff88032069ea00 ti: ffff8803f0c90000 task.ti:
> ffff8803f0c90000
> [789881.692802] RIP: 0010:[<ffffffffa016d1c9>] [<ffffffffa016d1c9>]
> rbd_dev_header_info+0x5a9/0x940 [rbd]
> [789881.702702] RSP: 0018:ffff8803f0c93d30 EFLAGS: 00010286
> [789881.708344] RAX: 0000000000000077 RBX: ffff8802a6a63800 RCX:
> 0000000000000000
> [789881.715985] RDX: 0000000000000077 RSI: ffff88041fd0dd08 RDI:
> ffff88041fd0dd08
> [789881.723625] RBP: ffff8803f0c93d98 R08: 0000000000000030 R09:
> 0000000000000000
> [789881.731261] R10: 0000000000000000 R11: 0000000000004479 R12:
> ffff8800d6eaf000
> [789881.738899] R13: ffff8802a6a639b0 R14: 0000000000000000 R15:
> ffff880327e6e780
> [789881.746533] FS: 0000000000000000(0000) GS:ffff88041fd00000(0000)
> knlGS:0000000000000000
> [789881.755120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [789881.761197] CR2: 00007fbb18242838 CR3: 0000000001e07000 CR4:
> 00000000001406e0
> [789881.768846] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [789881.776482] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [789881.784118] Stack:
> [789881.786457] ffffffff8113a91a ffff88032069ea00 ffff88041fd17ef0
> ffff88041fd17ef0
> [789881.794713] ffff88041fd17ef0 0000000000030289 ffff8803f0c93dd8
> ffffffff8113d968
> [789881.802965] ffff8802a6a63800 ffff8800d6eaf000 ffff8802a6a639b0
> 0000000000000000
> [789881.811207] Call Trace:
> [789881.813988] [<ffffffff8113a91a>] ? update_curr+0x8a/0x110
> [789881.819810] [<ffffffff8113d968>] ? dequeue_task_fair+0x618/0x1150
> [789881.826321] [<ffffffffa016d591>] rbd_dev_refresh+0x31/0xf0 [rbd]
> [789881.832760] [<ffffffffa016d719>] rbd_watch_cb+0x29/0xa0 [rbd]
> [789881.838930] [<ffffffffa0138fdc>] do_watch_notify+0x4c/0x80 [libceph]
> [789881.845706] [<ffffffff811258e9>] process_one_work+0x149/0x3c0
> [789881.856639] [<ffffffff81125bae>] worker_thread+0x4e/0x490
> [789881.862453] [<ffffffff81125b60>] ? process_one_work+0x3c0/0x3c0
> [789881.868823] [<ffffffff8112b1e9>] kthread+0xc9/0xe0
> [789881.874033] [<ffffffff8185e4ff>] ret_from_fork+0x1f/0x40
> [789881.879764] [<ffffffff8112b120>] ? kthread_create_on_node+0x170/0x170
> [789881.886618] Code: 0b 44 8b 6d b8 e9 1d ff ff ff 48 c7 c1 f0 00 17
> a0 ba 1e 12 00 00 48 c7 c6 90 0e 17 a0 48 c7 c7 20 f8 16 a0 31 c0 e8
> 8a 5d 08 e1 <0f> 0b 75 14 49 8b 7f 68 41 bd 92 ff ff ff e8 d4 e0 fc ff
> e9 dc
> [789881.911744] RIP [<ffffffffa016d1c9>] rbd_dev_header_info+0x5a9/0x940 [rbd]
> [789881.919116] RSP <ffff8803f0c93d30>
> [789881.922989] ---[ end trace 12b8d1c2ed74d6c1 ]---
> [789881.927971] BUG: unable to handle kernel paging request at ffffffffffffffd8
> [789881.935435] IP: [<ffffffff8112b821>] kthread_data+0x11/0x20
> [789881.941427] PGD 1e0a067 PUD 1e0c067 PMD 0
> [789881.946117] Oops: 0000 [#2] SMP
> [789881.949591] Modules linked in: rbd libceph sg rpcsec_gss_krb5
> xt_nat xt_UDPLB(O) xt_multiport xt_addrtype iptable_mangle iptable_raw
> iptable_nat nf_nat_ipv4 nf_nat ext4 jbd2 mbcache x86_pkg_temp_thermal
> gkuart(O) usbserial ie31200_edac edac_core tpm_tis raid1 crc32c_intel
> [789881.976900] CPU: 4 PID: 4111 Comm: kworker/u16:0 Tainted: G D
> O 4.7.0-vanilla-ams-3 #1
> [789881.986280] Hardware name: Quanta T6BC-S1N/T6BC, BIOS T6BC2A01 03/26/2014
> [789881.993410] task: ffff88032069ea00 ti: ffff8803f0c90000 task.ti:
> ffff8803f0c90000
> [789882.001411] RIP: 0010:[<ffffffff8112b821>] [<ffffffff8112b821>]
> kthread_data+0x11/0x20
> [789882.010024] RSP: 0018:ffff8803f0c93a28 EFLAGS: 00010002
> [789882.015682] RAX: 0000000000000000 RBX: ffff88041fd17e80 RCX:
> 0000000000000004
> [789882.023342] RDX: ffff88040f005000 RSI: ffff88032069ea00 RDI:
> ffff88032069ea00
> [789882.030996] RBP: ffff8803f0c93a30 R08: 0000000000000000 R09:
> 0000000000079800
> [789882.038645] R10: 0000000000000001 R11: 0000000000000001 R12:
> 0000000000017e80
> [789882.046288] R13: 0000000000000000 R14: ffff88032069eec0 R15:
> ffff88032069ea00
> [789882.053926] FS: 0000000000000000(0000) GS:ffff88041fd00000(0000)
> knlGS:0000000000000000
> [789882.062524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [789882.068603] CR2: 0000000000000028 CR3: 0000000001e07000 CR4:
> 00000000001406e0
> [789882.076261] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [789882.083920] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [789882.091577] Stack:
> [789882.093935] ffffffff8112645e ffff8803f0c93a78 ffffffff8185ab3e
> ffff88032069ea00
> [789882.102194] ffff8803f0c93a78 ffff8803f0c94000 ffff8803f0c93ad0
> ffff8803f0c936e8
> [789882.110438] ffff88040d5c8000 0000000000000000 ffff8803f0c93a90
> ffffffff8185aef5
> [789882.118689] Call Trace:
> [789882.121484] [<ffffffff8112645e>] ? wq_worker_sleeping+0xe/0x90
> [789882.127752] [<ffffffff8185ab3e>] __schedule+0x36e/0x6f0
> [789882.133411] [<ffffffff8185aef5>] schedule+0x35/0x80
> [789882.138712] [<ffffffff81110ff9>] do_exit+0x739/0xb50
> [789882.144098] [<ffffffff8108833c>] oops_end+0x9c/0xd0
> [789882.149400] [<ffffffff810887ab>] die+0x4b/0x70
> [789882.154276] [<ffffffff81085b26>] do_trap+0xb6/0x150
> [789882.159583] [<ffffffff81085d87>] do_error_trap+0x77/0xe0
> [789882.165322] [<ffffffffa016d1c9>] ? rbd_dev_header_info+0x5a9/0x940 [rbd]
> [789882.172446] [<ffffffff811d7a3d>] ? irq_work_queue+0x6d/0x80
> [789882.178441] [<ffffffff811575d4>] ? wake_up_klogd+0x34/0x40
> [789882.184363] [<ffffffff81157aa6>] ? console_unlock+0x4c6/0x510
> [789882.190532] [<ffffffff810863c0>] do_invalid_op+0x20/0x30
> [789882.196265] [<ffffffff8185fb6e>] invalid_op+0x1e/0x30
> [789882.201740] [<ffffffffa016d1c9>] ? rbd_dev_header_info+0x5a9/0x940 [rbd]
> [789882.208866] [<ffffffff8113a91a>] ? update_curr+0x8a/0x110
> [789882.214694] [<ffffffff8113d968>] ? dequeue_task_fair+0x618/0x1150
> [789882.221225] [<ffffffffa016d591>] rbd_dev_refresh+0x31/0xf0 [rbd]
> [789882.227662] [<ffffffffa016d719>] rbd_watch_cb+0x29/0xa0 [rbd]
> [789882.233855] [<ffffffffa0138fdc>] do_watch_notify+0x4c/0x80 [libceph]
> [789882.240647] [<ffffffff811258e9>] process_one_work+0x149/0x3c0
> [789882.246811] [<ffffffff81125bae>] worker_thread+0x4e/0x490
> [789882.252629] [<ffffffff81125b60>] ? process_one_work+0x3c0/0x3c0
> [789882.258969] [<ffffffff8112b1e9>] kthread+0xc9/0xe0
> [789882.264182] [<ffffffff8185e4ff>] ret_from_fork+0x1f/0x40
> [789882.269917] [<ffffffff8112b120>] ? kthread_create_on_node+0x170/0x170
> [789882.276784] Code: 02 00 00 00 e8 a1 fd ff ff 5d c3 0f 1f 44 00 00
> 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 60 04 00 00 55
> 48 89 e5 5d <48> 8b 40 d8 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00
> 00 55
> [789882.301941] RIP [<ffffffff8112b821>] kthread_data+0x11/0x20
> [789882.308016] RSP <ffff8803f0c93a28>
> [789882.311847] CR2: ffffffffffffffd8
> [789882.315505] ---[ end trace 12b8d1c2ed74d6c2 ]---
> [789882.320462] Fixing recursive fault but reboot is needed!
That's the same one you've reported in the "ceph osd kernel divide
error", right? I've filed http://tracker.ceph.com/issues/16963 and
should get to it later this week.
What did you mean by "no networking stack" in that thread?
Thanks,
Ilya
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: linux 4.7.0 rbd client kernel panic when OSD process was killed by OOM
2016-08-08 20:14 ` Ilya Dryomov
@ 2016-08-08 20:16 ` Victor Payno
2016-08-08 20:35 ` Victor Payno
0 siblings, 1 reply; 6+ messages in thread
From: Victor Payno @ 2016-08-08 20:16 UTC (permalink / raw)
To: Ilya Dryomov
Cc: Patrick McLean, Roderick Colenbrander, Cliff Pajaro, Ceph Development
ping doesn't respond, usually when I see a non-kernel panic even if
SSH is unresponsive, the kernel still responds to pings and
applications ports are still open but not usually working. That might
just be on older kernels now.
On Mon, Aug 8, 2016 at 1:14 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
> On Mon, Aug 8, 2016 at 9:57 PM, Victor Payno <vpayno@gaikai.com> wrote:
>> We have another problem where an RBD client was killed when an OSD was
>> killed by the OOM on a server. The servers have 4.4.16 kernels.
>>
>> ams2 login: [789881.620147] ------------[ cut here ]------------
>> [789881.625094] kernel BUG at drivers/block/rbd.c:4638!
>> [789881.630311] invalid opcode: 0000 [#1] SMP
>> [789881.634650] Modules linked in: rbd libceph sg rpcsec_gss_krb5
>> xt_nat xt_UDPLB(O) xt_multiport xt_addrtype iptable_mangle iptable_raw
>> iptable_nat nf_nat_ipv4 nf_nat ext4 jbd2 mbcache x86_pkg_temp_thermal
>> gkuart(O) usbserial ie31200_edac edac_core tpm_tis raid1 crc32c_intel
>> [789881.661718] CPU: 4 PID: 4111 Comm: kworker/u16:0 Tainted: G
>> O 4.7.0-vanilla-ams-3 #1
>> [789881.671091] Hardware name: Quanta T6BC-S1N/T6BC, BIOS T6BC2A01 03/26/2014
>> [789881.678212] Workqueue: ceph-watch-notify do_watch_notify [libceph]
>> [789881.684814] task: ffff88032069ea00 ti: ffff8803f0c90000 task.ti:
>> ffff8803f0c90000
>> [789881.692802] RIP: 0010:[<ffffffffa016d1c9>] [<ffffffffa016d1c9>]
>> rbd_dev_header_info+0x5a9/0x940 [rbd]
>> [789881.702702] RSP: 0018:ffff8803f0c93d30 EFLAGS: 00010286
>> [789881.708344] RAX: 0000000000000077 RBX: ffff8802a6a63800 RCX:
>> 0000000000000000
>> [789881.715985] RDX: 0000000000000077 RSI: ffff88041fd0dd08 RDI:
>> ffff88041fd0dd08
>> [789881.723625] RBP: ffff8803f0c93d98 R08: 0000000000000030 R09:
>> 0000000000000000
>> [789881.731261] R10: 0000000000000000 R11: 0000000000004479 R12:
>> ffff8800d6eaf000
>> [789881.738899] R13: ffff8802a6a639b0 R14: 0000000000000000 R15:
>> ffff880327e6e780
>> [789881.746533] FS: 0000000000000000(0000) GS:ffff88041fd00000(0000)
>> knlGS:0000000000000000
>> [789881.755120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [789881.761197] CR2: 00007fbb18242838 CR3: 0000000001e07000 CR4:
>> 00000000001406e0
>> [789881.768846] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [789881.776482] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
>> 0000000000000400
>> [789881.784118] Stack:
>> [789881.786457] ffffffff8113a91a ffff88032069ea00 ffff88041fd17ef0
>> ffff88041fd17ef0
>> [789881.794713] ffff88041fd17ef0 0000000000030289 ffff8803f0c93dd8
>> ffffffff8113d968
>> [789881.802965] ffff8802a6a63800 ffff8800d6eaf000 ffff8802a6a639b0
>> 0000000000000000
>> [789881.811207] Call Trace:
>> [789881.813988] [<ffffffff8113a91a>] ? update_curr+0x8a/0x110
>> [789881.819810] [<ffffffff8113d968>] ? dequeue_task_fair+0x618/0x1150
>> [789881.826321] [<ffffffffa016d591>] rbd_dev_refresh+0x31/0xf0 [rbd]
>> [789881.832760] [<ffffffffa016d719>] rbd_watch_cb+0x29/0xa0 [rbd]
>> [789881.838930] [<ffffffffa0138fdc>] do_watch_notify+0x4c/0x80 [libceph]
>> [789881.845706] [<ffffffff811258e9>] process_one_work+0x149/0x3c0
>> [789881.856639] [<ffffffff81125bae>] worker_thread+0x4e/0x490
>> [789881.862453] [<ffffffff81125b60>] ? process_one_work+0x3c0/0x3c0
>> [789881.868823] [<ffffffff8112b1e9>] kthread+0xc9/0xe0
>> [789881.874033] [<ffffffff8185e4ff>] ret_from_fork+0x1f/0x40
>> [789881.879764] [<ffffffff8112b120>] ? kthread_create_on_node+0x170/0x170
>> [789881.886618] Code: 0b 44 8b 6d b8 e9 1d ff ff ff 48 c7 c1 f0 00 17
>> a0 ba 1e 12 00 00 48 c7 c6 90 0e 17 a0 48 c7 c7 20 f8 16 a0 31 c0 e8
>> 8a 5d 08 e1 <0f> 0b 75 14 49 8b 7f 68 41 bd 92 ff ff ff e8 d4 e0 fc ff
>> e9 dc
>> [789881.911744] RIP [<ffffffffa016d1c9>] rbd_dev_header_info+0x5a9/0x940 [rbd]
>> [789881.919116] RSP <ffff8803f0c93d30>
>> [789881.922989] ---[ end trace 12b8d1c2ed74d6c1 ]---
>> [789881.927971] BUG: unable to handle kernel paging request at ffffffffffffffd8
>> [789881.935435] IP: [<ffffffff8112b821>] kthread_data+0x11/0x20
>> [789881.941427] PGD 1e0a067 PUD 1e0c067 PMD 0
>> [789881.946117] Oops: 0000 [#2] SMP
>> [789881.949591] Modules linked in: rbd libceph sg rpcsec_gss_krb5
>> xt_nat xt_UDPLB(O) xt_multiport xt_addrtype iptable_mangle iptable_raw
>> iptable_nat nf_nat_ipv4 nf_nat ext4 jbd2 mbcache x86_pkg_temp_thermal
>> gkuart(O) usbserial ie31200_edac edac_core tpm_tis raid1 crc32c_intel
>> [789881.976900] CPU: 4 PID: 4111 Comm: kworker/u16:0 Tainted: G D
>> O 4.7.0-vanilla-ams-3 #1
>> [789881.986280] Hardware name: Quanta T6BC-S1N/T6BC, BIOS T6BC2A01 03/26/2014
>> [789881.993410] task: ffff88032069ea00 ti: ffff8803f0c90000 task.ti:
>> ffff8803f0c90000
>> [789882.001411] RIP: 0010:[<ffffffff8112b821>] [<ffffffff8112b821>]
>> kthread_data+0x11/0x20
>> [789882.010024] RSP: 0018:ffff8803f0c93a28 EFLAGS: 00010002
>> [789882.015682] RAX: 0000000000000000 RBX: ffff88041fd17e80 RCX:
>> 0000000000000004
>> [789882.023342] RDX: ffff88040f005000 RSI: ffff88032069ea00 RDI:
>> ffff88032069ea00
>> [789882.030996] RBP: ffff8803f0c93a30 R08: 0000000000000000 R09:
>> 0000000000079800
>> [789882.038645] R10: 0000000000000001 R11: 0000000000000001 R12:
>> 0000000000017e80
>> [789882.046288] R13: 0000000000000000 R14: ffff88032069eec0 R15:
>> ffff88032069ea00
>> [789882.053926] FS: 0000000000000000(0000) GS:ffff88041fd00000(0000)
>> knlGS:0000000000000000
>> [789882.062524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [789882.068603] CR2: 0000000000000028 CR3: 0000000001e07000 CR4:
>> 00000000001406e0
>> [789882.076261] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [789882.083920] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
>> 0000000000000400
>> [789882.091577] Stack:
>> [789882.093935] ffffffff8112645e ffff8803f0c93a78 ffffffff8185ab3e
>> ffff88032069ea00
>> [789882.102194] ffff8803f0c93a78 ffff8803f0c94000 ffff8803f0c93ad0
>> ffff8803f0c936e8
>> [789882.110438] ffff88040d5c8000 0000000000000000 ffff8803f0c93a90
>> ffffffff8185aef5
>> [789882.118689] Call Trace:
>> [789882.121484] [<ffffffff8112645e>] ? wq_worker_sleeping+0xe/0x90
>> [789882.127752] [<ffffffff8185ab3e>] __schedule+0x36e/0x6f0
>> [789882.133411] [<ffffffff8185aef5>] schedule+0x35/0x80
>> [789882.138712] [<ffffffff81110ff9>] do_exit+0x739/0xb50
>> [789882.144098] [<ffffffff8108833c>] oops_end+0x9c/0xd0
>> [789882.149400] [<ffffffff810887ab>] die+0x4b/0x70
>> [789882.154276] [<ffffffff81085b26>] do_trap+0xb6/0x150
>> [789882.159583] [<ffffffff81085d87>] do_error_trap+0x77/0xe0
>> [789882.165322] [<ffffffffa016d1c9>] ? rbd_dev_header_info+0x5a9/0x940 [rbd]
>> [789882.172446] [<ffffffff811d7a3d>] ? irq_work_queue+0x6d/0x80
>> [789882.178441] [<ffffffff811575d4>] ? wake_up_klogd+0x34/0x40
>> [789882.184363] [<ffffffff81157aa6>] ? console_unlock+0x4c6/0x510
>> [789882.190532] [<ffffffff810863c0>] do_invalid_op+0x20/0x30
>> [789882.196265] [<ffffffff8185fb6e>] invalid_op+0x1e/0x30
>> [789882.201740] [<ffffffffa016d1c9>] ? rbd_dev_header_info+0x5a9/0x940 [rbd]
>> [789882.208866] [<ffffffff8113a91a>] ? update_curr+0x8a/0x110
>> [789882.214694] [<ffffffff8113d968>] ? dequeue_task_fair+0x618/0x1150
>> [789882.221225] [<ffffffffa016d591>] rbd_dev_refresh+0x31/0xf0 [rbd]
>> [789882.227662] [<ffffffffa016d719>] rbd_watch_cb+0x29/0xa0 [rbd]
>> [789882.233855] [<ffffffffa0138fdc>] do_watch_notify+0x4c/0x80 [libceph]
>> [789882.240647] [<ffffffff811258e9>] process_one_work+0x149/0x3c0
>> [789882.246811] [<ffffffff81125bae>] worker_thread+0x4e/0x490
>> [789882.252629] [<ffffffff81125b60>] ? process_one_work+0x3c0/0x3c0
>> [789882.258969] [<ffffffff8112b1e9>] kthread+0xc9/0xe0
>> [789882.264182] [<ffffffff8185e4ff>] ret_from_fork+0x1f/0x40
>> [789882.269917] [<ffffffff8112b120>] ? kthread_create_on_node+0x170/0x170
>> [789882.276784] Code: 02 00 00 00 e8 a1 fd ff ff 5d c3 0f 1f 44 00 00
>> 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 60 04 00 00 55
>> 48 89 e5 5d <48> 8b 40 d8 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00
>> 00 55
>> [789882.301941] RIP [<ffffffff8112b821>] kthread_data+0x11/0x20
>> [789882.308016] RSP <ffff8803f0c93a28>
>> [789882.311847] CR2: ffffffffffffffd8
>> [789882.315505] ---[ end trace 12b8d1c2ed74d6c2 ]---
>> [789882.320462] Fixing recursive fault but reboot is needed!
>
> That's the same one you've reported in the "ceph osd kernel divide
> error", right? I've filed http://tracker.ceph.com/issues/16963 and
> should get to it later this week.
>
> What did you mean by "no networking stack" in that thread?
>
> Thanks,
>
> Ilya
--
Victor Payno
ビクター·ペイン
Sr. Release Engineer
シニアリリースエンジニア
Gaikai, a Sony Computer Entertainment Company ∆○×□
ガイカイ、ソニー・コンピュータエンタテインメント傘下会社
65 Enterprise
Aliso Viejo, CA 92656 USA
Web: www.gaikai.com
Email: vpayno@gaikai.com
Phone: (949) 330-6850
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: linux 4.7.0 rbd client kernel panic when OSD process was killed by OOM
2016-08-08 20:16 ` Victor Payno
@ 2016-08-08 20:35 ` Victor Payno
2016-08-09 9:02 ` Ilya Dryomov
0 siblings, 1 reply; 6+ messages in thread
From: Victor Payno @ 2016-08-08 20:35 UTC (permalink / raw)
To: Ilya Dryomov
Cc: Patrick McLean, Roderick Colenbrander, Cliff Pajaro, Ceph Development
Ilya:
I think it is the same error, I was confusing it with the 4.4 kernel error.
Do you have documentation on settings we can use to limit the memory
growth of OSD processes?
So far all I have is changing these from
osd_min_pg_log_entries = 3000
osd_max_pg_log_entries = 10000
to
osd_min_pg_log_entries = 300
osd_max_pg_log_entries = 1000
and now I'm trying these settings
osd_min_pg_log_entries = 150
osd_max_pg_log_entries = 500
The hosts have 12 OSDs (8TB HDDs with SSDs for journals) and 32 GB of
RAM. We're having a hard time with getting more memory because Ceph
documentation says 2GB per OSD (24 GB).
The bigger problem I have is that once an OSD OOMs, I can't recover
it, I have to destroy it and create it again. Unfortunately that
starts a domino effect and other nodes start loosing 1 OSD to OOM.
Eventually I end up destroying the cluster and starting over again.
This cluster had 2 pools, the second pool had a single 100TB RBD with
3.6 TB of data (was currently mapped and mounted but idle).
Are there memory recommendations per pool? Eventually we'll probably
have a minimum of 5 pools per cluster (pool == application).
On Mon, Aug 8, 2016 at 1:16 PM, Victor Payno <vpayno@gaikai.com> wrote:
> ping doesn't respond, usually when I see a non-kernel panic even if
> SSH is unresponsive, the kernel still responds to pings and
> applications ports are still open but not usually working. That might
> just be on older kernels now.
>
> On Mon, Aug 8, 2016 at 1:14 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
>> On Mon, Aug 8, 2016 at 9:57 PM, Victor Payno <vpayno@gaikai.com> wrote:
>>> We have another problem where an RBD client was killed when an OSD was
>>> killed by the OOM on a server. The servers have 4.4.16 kernels.
>>>
>>> ams2 login: [789881.620147] ------------[ cut here ]------------
>>> [789881.625094] kernel BUG at drivers/block/rbd.c:4638!
>>> [789881.630311] invalid opcode: 0000 [#1] SMP
>>> [789881.634650] Modules linked in: rbd libceph sg rpcsec_gss_krb5
>>> xt_nat xt_UDPLB(O) xt_multiport xt_addrtype iptable_mangle iptable_raw
>>> iptable_nat nf_nat_ipv4 nf_nat ext4 jbd2 mbcache x86_pkg_temp_thermal
>>> gkuart(O) usbserial ie31200_edac edac_core tpm_tis raid1 crc32c_intel
>>> [789881.661718] CPU: 4 PID: 4111 Comm: kworker/u16:0 Tainted: G
>>> O 4.7.0-vanilla-ams-3 #1
>>> [789881.671091] Hardware name: Quanta T6BC-S1N/T6BC, BIOS T6BC2A01 03/26/2014
>>> [789881.678212] Workqueue: ceph-watch-notify do_watch_notify [libceph]
>>> [789881.684814] task: ffff88032069ea00 ti: ffff8803f0c90000 task.ti:
>>> ffff8803f0c90000
>>> [789881.692802] RIP: 0010:[<ffffffffa016d1c9>] [<ffffffffa016d1c9>]
>>> rbd_dev_header_info+0x5a9/0x940 [rbd]
>>> [789881.702702] RSP: 0018:ffff8803f0c93d30 EFLAGS: 00010286
>>> [789881.708344] RAX: 0000000000000077 RBX: ffff8802a6a63800 RCX:
>>> 0000000000000000
>>> [789881.715985] RDX: 0000000000000077 RSI: ffff88041fd0dd08 RDI:
>>> ffff88041fd0dd08
>>> [789881.723625] RBP: ffff8803f0c93d98 R08: 0000000000000030 R09:
>>> 0000000000000000
>>> [789881.731261] R10: 0000000000000000 R11: 0000000000004479 R12:
>>> ffff8800d6eaf000
>>> [789881.738899] R13: ffff8802a6a639b0 R14: 0000000000000000 R15:
>>> ffff880327e6e780
>>> [789881.746533] FS: 0000000000000000(0000) GS:ffff88041fd00000(0000)
>>> knlGS:0000000000000000
>>> [789881.755120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [789881.761197] CR2: 00007fbb18242838 CR3: 0000000001e07000 CR4:
>>> 00000000001406e0
>>> [789881.768846] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>>> 0000000000000000
>>> [789881.776482] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
>>> 0000000000000400
>>> [789881.784118] Stack:
>>> [789881.786457] ffffffff8113a91a ffff88032069ea00 ffff88041fd17ef0
>>> ffff88041fd17ef0
>>> [789881.794713] ffff88041fd17ef0 0000000000030289 ffff8803f0c93dd8
>>> ffffffff8113d968
>>> [789881.802965] ffff8802a6a63800 ffff8800d6eaf000 ffff8802a6a639b0
>>> 0000000000000000
>>> [789881.811207] Call Trace:
>>> [789881.813988] [<ffffffff8113a91a>] ? update_curr+0x8a/0x110
>>> [789881.819810] [<ffffffff8113d968>] ? dequeue_task_fair+0x618/0x1150
>>> [789881.826321] [<ffffffffa016d591>] rbd_dev_refresh+0x31/0xf0 [rbd]
>>> [789881.832760] [<ffffffffa016d719>] rbd_watch_cb+0x29/0xa0 [rbd]
>>> [789881.838930] [<ffffffffa0138fdc>] do_watch_notify+0x4c/0x80 [libceph]
>>> [789881.845706] [<ffffffff811258e9>] process_one_work+0x149/0x3c0
>>> [789881.856639] [<ffffffff81125bae>] worker_thread+0x4e/0x490
>>> [789881.862453] [<ffffffff81125b60>] ? process_one_work+0x3c0/0x3c0
>>> [789881.868823] [<ffffffff8112b1e9>] kthread+0xc9/0xe0
>>> [789881.874033] [<ffffffff8185e4ff>] ret_from_fork+0x1f/0x40
>>> [789881.879764] [<ffffffff8112b120>] ? kthread_create_on_node+0x170/0x170
>>> [789881.886618] Code: 0b 44 8b 6d b8 e9 1d ff ff ff 48 c7 c1 f0 00 17
>>> a0 ba 1e 12 00 00 48 c7 c6 90 0e 17 a0 48 c7 c7 20 f8 16 a0 31 c0 e8
>>> 8a 5d 08 e1 <0f> 0b 75 14 49 8b 7f 68 41 bd 92 ff ff ff e8 d4 e0 fc ff
>>> e9 dc
>>> [789881.911744] RIP [<ffffffffa016d1c9>] rbd_dev_header_info+0x5a9/0x940 [rbd]
>>> [789881.919116] RSP <ffff8803f0c93d30>
>>> [789881.922989] ---[ end trace 12b8d1c2ed74d6c1 ]---
>>> [789881.927971] BUG: unable to handle kernel paging request at ffffffffffffffd8
>>> [789881.935435] IP: [<ffffffff8112b821>] kthread_data+0x11/0x20
>>> [789881.941427] PGD 1e0a067 PUD 1e0c067 PMD 0
>>> [789881.946117] Oops: 0000 [#2] SMP
>>> [789881.949591] Modules linked in: rbd libceph sg rpcsec_gss_krb5
>>> xt_nat xt_UDPLB(O) xt_multiport xt_addrtype iptable_mangle iptable_raw
>>> iptable_nat nf_nat_ipv4 nf_nat ext4 jbd2 mbcache x86_pkg_temp_thermal
>>> gkuart(O) usbserial ie31200_edac edac_core tpm_tis raid1 crc32c_intel
>>> [789881.976900] CPU: 4 PID: 4111 Comm: kworker/u16:0 Tainted: G D
>>> O 4.7.0-vanilla-ams-3 #1
>>> [789881.986280] Hardware name: Quanta T6BC-S1N/T6BC, BIOS T6BC2A01 03/26/2014
>>> [789881.993410] task: ffff88032069ea00 ti: ffff8803f0c90000 task.ti:
>>> ffff8803f0c90000
>>> [789882.001411] RIP: 0010:[<ffffffff8112b821>] [<ffffffff8112b821>]
>>> kthread_data+0x11/0x20
>>> [789882.010024] RSP: 0018:ffff8803f0c93a28 EFLAGS: 00010002
>>> [789882.015682] RAX: 0000000000000000 RBX: ffff88041fd17e80 RCX:
>>> 0000000000000004
>>> [789882.023342] RDX: ffff88040f005000 RSI: ffff88032069ea00 RDI:
>>> ffff88032069ea00
>>> [789882.030996] RBP: ffff8803f0c93a30 R08: 0000000000000000 R09:
>>> 0000000000079800
>>> [789882.038645] R10: 0000000000000001 R11: 0000000000000001 R12:
>>> 0000000000017e80
>>> [789882.046288] R13: 0000000000000000 R14: ffff88032069eec0 R15:
>>> ffff88032069ea00
>>> [789882.053926] FS: 0000000000000000(0000) GS:ffff88041fd00000(0000)
>>> knlGS:0000000000000000
>>> [789882.062524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [789882.068603] CR2: 0000000000000028 CR3: 0000000001e07000 CR4:
>>> 00000000001406e0
>>> [789882.076261] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>>> 0000000000000000
>>> [789882.083920] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
>>> 0000000000000400
>>> [789882.091577] Stack:
>>> [789882.093935] ffffffff8112645e ffff8803f0c93a78 ffffffff8185ab3e
>>> ffff88032069ea00
>>> [789882.102194] ffff8803f0c93a78 ffff8803f0c94000 ffff8803f0c93ad0
>>> ffff8803f0c936e8
>>> [789882.110438] ffff88040d5c8000 0000000000000000 ffff8803f0c93a90
>>> ffffffff8185aef5
>>> [789882.118689] Call Trace:
>>> [789882.121484] [<ffffffff8112645e>] ? wq_worker_sleeping+0xe/0x90
>>> [789882.127752] [<ffffffff8185ab3e>] __schedule+0x36e/0x6f0
>>> [789882.133411] [<ffffffff8185aef5>] schedule+0x35/0x80
>>> [789882.138712] [<ffffffff81110ff9>] do_exit+0x739/0xb50
>>> [789882.144098] [<ffffffff8108833c>] oops_end+0x9c/0xd0
>>> [789882.149400] [<ffffffff810887ab>] die+0x4b/0x70
>>> [789882.154276] [<ffffffff81085b26>] do_trap+0xb6/0x150
>>> [789882.159583] [<ffffffff81085d87>] do_error_trap+0x77/0xe0
>>> [789882.165322] [<ffffffffa016d1c9>] ? rbd_dev_header_info+0x5a9/0x940 [rbd]
>>> [789882.172446] [<ffffffff811d7a3d>] ? irq_work_queue+0x6d/0x80
>>> [789882.178441] [<ffffffff811575d4>] ? wake_up_klogd+0x34/0x40
>>> [789882.184363] [<ffffffff81157aa6>] ? console_unlock+0x4c6/0x510
>>> [789882.190532] [<ffffffff810863c0>] do_invalid_op+0x20/0x30
>>> [789882.196265] [<ffffffff8185fb6e>] invalid_op+0x1e/0x30
>>> [789882.201740] [<ffffffffa016d1c9>] ? rbd_dev_header_info+0x5a9/0x940 [rbd]
>>> [789882.208866] [<ffffffff8113a91a>] ? update_curr+0x8a/0x110
>>> [789882.214694] [<ffffffff8113d968>] ? dequeue_task_fair+0x618/0x1150
>>> [789882.221225] [<ffffffffa016d591>] rbd_dev_refresh+0x31/0xf0 [rbd]
>>> [789882.227662] [<ffffffffa016d719>] rbd_watch_cb+0x29/0xa0 [rbd]
>>> [789882.233855] [<ffffffffa0138fdc>] do_watch_notify+0x4c/0x80 [libceph]
>>> [789882.240647] [<ffffffff811258e9>] process_one_work+0x149/0x3c0
>>> [789882.246811] [<ffffffff81125bae>] worker_thread+0x4e/0x490
>>> [789882.252629] [<ffffffff81125b60>] ? process_one_work+0x3c0/0x3c0
>>> [789882.258969] [<ffffffff8112b1e9>] kthread+0xc9/0xe0
>>> [789882.264182] [<ffffffff8185e4ff>] ret_from_fork+0x1f/0x40
>>> [789882.269917] [<ffffffff8112b120>] ? kthread_create_on_node+0x170/0x170
>>> [789882.276784] Code: 02 00 00 00 e8 a1 fd ff ff 5d c3 0f 1f 44 00 00
>>> 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 60 04 00 00 55
>>> 48 89 e5 5d <48> 8b 40 d8 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00
>>> 00 55
>>> [789882.301941] RIP [<ffffffff8112b821>] kthread_data+0x11/0x20
>>> [789882.308016] RSP <ffff8803f0c93a28>
>>> [789882.311847] CR2: ffffffffffffffd8
>>> [789882.315505] ---[ end trace 12b8d1c2ed74d6c2 ]---
>>> [789882.320462] Fixing recursive fault but reboot is needed!
>>
>> That's the same one you've reported in the "ceph osd kernel divide
>> error", right? I've filed http://tracker.ceph.com/issues/16963 and
>> should get to it later this week.
>>
>> What did you mean by "no networking stack" in that thread?
>>
>> Thanks,
>>
>> Ilya
>
>
>
> --
> Victor Payno
> ビクター·ペイン
>
> Sr. Release Engineer
> シニアリリースエンジニア
>
>
>
> Gaikai, a Sony Computer Entertainment Company ∆○×□
> ガイカイ、ソニー・コンピュータエンタテインメント傘下会社
> 65 Enterprise
> Aliso Viejo, CA 92656 USA
>
> Web: www.gaikai.com
> Email: vpayno@gaikai.com
> Phone: (949) 330-6850
--
Victor Payno
ビクター·ペイン
Sr. Release Engineer
シニアリリースエンジニア
Gaikai, a Sony Computer Entertainment Company ∆○×□
ガイカイ、ソニー・コンピュータエンタテインメント傘下会社
65 Enterprise
Aliso Viejo, CA 92656 USA
Web: www.gaikai.com
Email: vpayno@gaikai.com
Phone: (949) 330-6850
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: linux 4.7.0 rbd client kernel panic when OSD process was killed by OOM
2016-08-08 20:35 ` Victor Payno
@ 2016-08-09 9:02 ` Ilya Dryomov
2016-08-09 20:29 ` Victor Payno
0 siblings, 1 reply; 6+ messages in thread
From: Ilya Dryomov @ 2016-08-09 9:02 UTC (permalink / raw)
To: Victor Payno
Cc: Patrick McLean, Roderick Colenbrander, Cliff Pajaro, Ceph Development
On Mon, Aug 8, 2016 at 10:35 PM, Victor Payno <vpayno@gaikai.com> wrote:
> Ilya:
>
> I think it is the same error, I was confusing it with the 4.4 kernel error.
The kernel client shouldn't crash no matter what - I'll look into it
later this or next week.
>
> Do you have documentation on settings we can use to limit the memory
> growth of OSD processes?
>
> So far all I have is changing these from
>
> osd_min_pg_log_entries = 3000
> osd_max_pg_log_entries = 10000
>
> to
>
> osd_min_pg_log_entries = 300
> osd_max_pg_log_entries = 1000
>
> and now I'm trying these settings
>
> osd_min_pg_log_entries = 150
> osd_max_pg_log_entries = 500
>
>
>
> The hosts have 12 OSDs (8TB HDDs with SSDs for journals) and 32 GB of
> RAM. We're having a hard time with getting more memory because Ceph
> documentation says 2GB per OSD (24 GB).
>
> The bigger problem I have is that once an OSD OOMs, I can't recover
> it, I have to destroy it and create it again. Unfortunately that
> starts a domino effect and other nodes start loosing 1 OSD to OOM.
> Eventually I end up destroying the cluster and starting over again.
>
>
> This cluster had 2 pools, the second pool had a single 100TB RBD with
> 3.6 TB of data (was currently mapped and mounted but idle).
How many PGs in each pool?
Was there a recovery underway? I know from experience that during
recovery it can blow past the 2G mark, especially if more than a few
OSDs on the host are being rebalanced at the same time.
Do you have an idea on what leads to the OOM kill? A test workload you
are running, general state of the cluster, etc.
To get out of it, I'd add a big swapfile, set the flags to pause any
background operations (backfill, scrub, etc) and prevent OSD flapping
and just babysit it, monitoring memory usage. You might want to search
ceph-users archive for more tricks.
Thanks,
Ilya
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: linux 4.7.0 rbd client kernel panic when OSD process was killed by OOM
2016-08-09 9:02 ` Ilya Dryomov
@ 2016-08-09 20:29 ` Victor Payno
0 siblings, 0 replies; 6+ messages in thread
From: Victor Payno @ 2016-08-09 20:29 UTC (permalink / raw)
To: Ilya Dryomov
Cc: Patrick McLean, Roderick Colenbrander, Cliff Pajaro, Ceph Development
The problem with the swap file/partition is that I've seen OSD grow
past 64GB in size.
With previous OOMs, I've seen one to four (out of eight to twelve
OSDs) OOM on a single host.
And I've had times when one OOM cascades into additional OOMs on other hosts.
For this last OOM event, we think that the client rbds (168+) hit the
100 snapshot limit and they all started pruning at the same time.
It's possible that we had both pruning and snapshoting happening at
the same time (not on the same rbd).
Also, when the rbds reach 100 snapshots, a prune follows every
snapshot creation (oops, we're fixing that).
On Tue, Aug 9, 2016 at 2:02 AM, Ilya Dryomov <idryomov@gmail.com> wrote:
> On Mon, Aug 8, 2016 at 10:35 PM, Victor Payno <vpayno@gaikai.com> wrote:
>> Ilya:
>>
>> I think it is the same error, I was confusing it with the 4.4 kernel error.
>
> The kernel client shouldn't crash no matter what - I'll look into it
> later this or next week.
>
>>
>> Do you have documentation on settings we can use to limit the memory
>> growth of OSD processes?
>>
>> So far all I have is changing these from
>>
>> osd_min_pg_log_entries = 3000
>> osd_max_pg_log_entries = 10000
>>
>> to
>>
>> osd_min_pg_log_entries = 300
>> osd_max_pg_log_entries = 1000
>>
>> and now I'm trying these settings
>>
>> osd_min_pg_log_entries = 150
>> osd_max_pg_log_entries = 500
>>
>>
>>
>> The hosts have 12 OSDs (8TB HDDs with SSDs for journals) and 32 GB of
>> RAM. We're having a hard time with getting more memory because Ceph
>> documentation says 2GB per OSD (24 GB).
>
>>
>> The bigger problem I have is that once an OSD OOMs, I can't recover
>> it, I have to destroy it and create it again. Unfortunately that
>> starts a domino effect and other nodes start loosing 1 OSD to OOM.
>> Eventually I end up destroying the cluster and starting over again.
>>
>>
>> This cluster had 2 pools, the second pool had a single 100TB RBD with
>> 3.6 TB of data (was currently mapped and mounted but idle).
>
> How many PGs in each pool?
>
> Was there a recovery underway? I know from experience that during
> recovery it can blow past the 2G mark, especially if more than a few
> OSDs on the host are being rebalanced at the same time.
>
> Do you have an idea on what leads to the OOM kill? A test workload you
> are running, general state of the cluster, etc.
>
> To get out of it, I'd add a big swapfile, set the flags to pause any
> background operations (backfill, scrub, etc) and prevent OSD flapping
> and just babysit it, monitoring memory usage. You might want to search
> ceph-users archive for more tricks.
>
> Thanks,
>
> Ilya
--
Victor Payno
ビクター·ペイン
Sr. Release Engineer
シニアリリースエンジニア
Gaikai, a Sony Computer Entertainment Company ∆○×□
ガイカイ、ソニー・コンピュータエンタテインメント傘下会社
65 Enterprise
Aliso Viejo, CA 92656 USA
Web: www.gaikai.com
Email: vpayno@gaikai.com
Phone: (949) 330-6850
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-08-09 20:30 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-08-08 19:57 linux 4.7.0 rbd client kernel panic when OSD process was killed by OOM Victor Payno
2016-08-08 20:14 ` Ilya Dryomov
2016-08-08 20:16 ` Victor Payno
2016-08-08 20:35 ` Victor Payno
2016-08-09 9:02 ` Ilya Dryomov
2016-08-09 20:29 ` Victor Payno
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.