Kernel crashes with RBD

Kernel crashes with RBD

[Danny Kukawka]

[-- Attachment #1: Type: text/plain, Size: 2005 bytes --]

Hi,

we are currently testing CEPH with RBD on a cluster with 1GBit and
10Gbit interfaces. While we see no kernel crashes with RBD if the
cluster runs on the 1GBit interfaces, we see very frequent kernel
crashes with the 10Gbit network while running tests with e.g. fio
against the RBDs.

I've tested it with kernel v3.0 and also 3.3.0 (with the patches from
the 'for-linus' branch from ceph-client.git at git.kernel.org).

With more client machines running tests the crashes occur even much
faster. The issue is fully reproducible here.

Has anyone seen similar problems? See the backtrace below.

Regards

Danny

PID: 10902  TASK: ffff88032a9a2080  CPU: 0   COMMAND: "kworker/0:0"
 #0 [ffff8803235fd950] machine_kexec at ffffffff810265ee
 #1 [ffff8803235fd9a0] crash_kexec at ffffffff810a3bda
 #2 [ffff8803235fda70] oops_end at ffffffff81444688
 #3 [ffff8803235fda90] __bad_area_nosemaphore at ffffffff81032a35
 #4 [ffff8803235fdb50] do_page_fault at ffffffff81446d3e
 #5 [ffff8803235fdc50] page_fault at ffffffff81443865
    [exception RIP: read_partial_message+816]
    RIP: ffffffffa041e500  RSP: ffff8803235fdd00  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: 00000000000009d7  RCX: 0000000000008000
    RDX: 0000000000000000  RSI: 00000000000009d7  RDI: ffffffff813c8d78
    RBP: ffff880328827030   R8: 00000000000009d7   R9: 0000000000004000
    R10: 0000000000000000  R11: ffffffff81205800  R12: 0000000000000000
    R13: 0000000000000069  R14: ffff88032a9bc780  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #6 [ffff8803235fdd38] thread_return at ffffffff81440e82
 #7 [ffff8803235fdd78] try_read at ffffffffa041ed58 [libceph]
 #8 [ffff8803235fddf8] con_work at ffffffffa041fb2e [libceph]
 #9 [ffff8803235fde28] process_one_work at ffffffff8107487c
#10 [ffff8803235fde78] worker_thread at ffffffff8107740a
#11 [ffff8803235fdee8] kthread at ffffffff8107b736
#12 [ffff8803235fdf48] kernel_thread_helper at ffffffff8144c144


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: Kernel crashes with RBD

[Josh Durgin]

On 04/11/2012 03:30 PM, Danny Kukawka wrote:
> Hi,
>
> we are currently testing CEPH with RBD on a cluster with 1GBit and
> 10Gbit interfaces. While we see no kernel crashes with RBD if the
> cluster runs on the 1GBit interfaces, we see very frequent kernel
> crashes with the 10Gbit network while running tests with e.g. fio
> against the RBDs.
>
> I've tested it with kernel v3.0 and also 3.3.0 (with the patches from
> the 'for-linus' branch from ceph-client.git at git.kernel.org).
>
> With more client machines running tests the crashes occur even much
> faster. The issue is fully reproducible here.
>
> Has anyone seen similar problems? See the backtrace below.
>
> Regards
>
> Danny
>
> PID: 10902  TASK: ffff88032a9a2080  CPU: 0   COMMAND: "kworker/0:0"
>   #0 [ffff8803235fd950] machine_kexec at ffffffff810265ee
>   #1 [ffff8803235fd9a0] crash_kexec at ffffffff810a3bda
>   #2 [ffff8803235fda70] oops_end at ffffffff81444688
>   #3 [ffff8803235fda90] __bad_area_nosemaphore at ffffffff81032a35
>   #4 [ffff8803235fdb50] do_page_fault at ffffffff81446d3e
>   #5 [ffff8803235fdc50] page_fault at ffffffff81443865
>      [exception RIP: read_partial_message+816]
>      RIP: ffffffffa041e500  RSP: ffff8803235fdd00  RFLAGS: 00010246
>      RAX: 0000000000000000  RBX: 00000000000009d7  RCX: 0000000000008000
>      RDX: 0000000000000000  RSI: 00000000000009d7  RDI: ffffffff813c8d78
>      RBP: ffff880328827030   R8: 00000000000009d7   R9: 0000000000004000
>      R10: 0000000000000000  R11: ffffffff81205800  R12: 0000000000000000
>      R13: 0000000000000069  R14: ffff88032a9bc780  R15: 0000000000000000
>      ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
>   #6 [ffff8803235fdd38] thread_return at ffffffff81440e82
>   #7 [ffff8803235fdd78] try_read at ffffffffa041ed58 [libceph]
>   #8 [ffff8803235fddf8] con_work at ffffffffa041fb2e [libceph]
>   #9 [ffff8803235fde28] process_one_work at ffffffff8107487c
> #10 [ffff8803235fde78] worker_thread at ffffffff8107740a
> #11 [ffff8803235fdee8] kthread at ffffffff8107b736
> #12 [ffff8803235fdf48] kernel_thread_helper at ffffffff8144c144
>

This looks similar to http://tracker.newdream.net/issues/2261. What do 
you think Alex?

Josh

Re: Kernel crashes with RBD

[Danny Kukawka]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

Hi

Am 13.04.2012 19:48, schrieb Josh Durgin:
> On 04/11/2012 03:30 PM, Danny Kukawka wrote:
[...]
> 
> This looks similar to http://tracker.newdream.net/issues/2261. What do
> you think Alex?

Not sure about that, since this crashes only the clients and not the
OSDs. We see no crashes in the cluster.

I analyzed it a bit more and found that the last working version was
0.43. Any later released version leads to this crash sooner or later,
but as I already said only on a 10Gbit (FC) network. I didn't see any
crash on the 1Gbit net on the same machines.

What kind of network do you use at dreamhost for testing?

If you need more info, let me known.

Danny


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: Kernel crashes with RBD

[Josh Durgin]

On 04/13/2012 11:18 AM, Danny Kukawka wrote:
> Hi
>
> Am 13.04.2012 19:48, schrieb Josh Durgin:
>> On 04/11/2012 03:30 PM, Danny Kukawka wrote:
> [...]
>>
>> This looks similar to http://tracker.newdream.net/issues/2261. What do
>> you think Alex?
>
> Not sure about that, since this crashes only the clients and not the
> OSDs. We see no crashes in the cluster.

These are both rbd kernel client crashes, but looking again they seem
like different underlying issues, so I opened
http://tracker.newdream.net/issues/2287 to track this problem.

>
> I analyzed it a bit more and found that the last working version was
> 0.43. Any later released version leads to this crash sooner or later,
> but as I already said only on a 10Gbit (FC) network. I didn't see any
> crash on the 1Gbit net on the same machines.
>
> What kind of network do you use at dreamhost for testing?

Mostly 1Gbit, some 10Gbit, but I don't think we've tested the kernel
client on 10Gbit yet.

> If you need more info, let me known.

Do the crashes always have the same stack trace? When you say 10Gbit
for the cluster, does that include the client using rbd, or just the
osds?

Josh

Re: Kernel crashes with RBD

[Danny Kukawka]

[-- Attachment #1: Type: text/plain, Size: 2178 bytes --]

Am 13.04.2012 22:56, schrieb Josh Durgin:
> On 04/13/2012 11:18 AM, Danny Kukawka wrote:
>> Hi
>>
>> Am 13.04.2012 19:48, schrieb Josh Durgin:
>>> On 04/11/2012 03:30 PM, Danny Kukawka wrote:
>> [...]
>>>
>>> This looks similar to http://tracker.newdream.net/issues/2261. What do
>>> you think Alex?
>>
>> Not sure about that, since this crashes only the clients and not the
>> OSDs. We see no crashes in the cluster.
> 
> These are both rbd kernel client crashes, but looking again they seem
> like different underlying issues, so I opened
> http://tracker.newdream.net/issues/2287 to track this problem.
> 
>>
>> I analyzed it a bit more and found that the last working version was
>> 0.43. Any later released version leads to this crash sooner or later,
>> but as I already said only on a 10Gbit (FC) network. I didn't see any
>> crash on the 1Gbit net on the same machines.
>>
>> What kind of network do you use at dreamhost for testing?
> 
> Mostly 1Gbit, some 10Gbit, but I don't think we've tested the kernel
> client on 10Gbit yet.

That's what I assumed ;-)

>> If you need more info, let me known.
> 
> Do the crashes always have the same stack trace? When you say 10Gbit
> for the cluster, does that include the client using rbd, or just the
> osds?

It's always the same stack trace (sometimes a address is different, but
everything else looks identical).

We tested basically the following setups and the crash happend with all
of them:
1) OSD, MON and Clients in the same 10Gbit network
2) OSD, MON and Clients in different public/cluster 10Gbit networks
3) OSD and Clients in the same 10Gbit network, MON in 1Gbit network
3) OSD and Clients in the same 10Gbit network, MON in 1Gbit network
   different public/cluster networks

The number of OSDs (tested 4 nodes with 10 OSDs per node, each one
physical harddisk) didn't matter in this case. If I use 2 clients
running fio tests against one 50GByte RBD per client, I hit the problem
faster than with one client. If you need information about the used fio
tests, let me know.

As already I said: we didn't hit this problem with 1Gbit networks yet.

Danny


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: Kernel crashes with RBD

[Danny Kukawka]

[-- Attachment #1: Type: text/plain, Size: 5488 bytes --]

Am 14.04.2012 01:03, schrieb Danny Kukawka:
> Am 13.04.2012 22:56, schrieb Josh Durgin:
>> On 04/13/2012 11:18 AM, Danny Kukawka wrote:
>>> Hi
>>>
>>> Am 13.04.2012 19:48, schrieb Josh Durgin:
>>>> On 04/11/2012 03:30 PM, Danny Kukawka wrote:
>>> [...]
>>>>
>>>> This looks similar to http://tracker.newdream.net/issues/2261. What do
>>>> you think Alex?
>>>
>>> Not sure about that, since this crashes only the clients and not the
>>> OSDs. We see no crashes in the cluster.
>>
>> These are both rbd kernel client crashes, but looking again they seem
>> like different underlying issues, so I opened
>> http://tracker.newdream.net/issues/2287 to track this problem.
>>
>>>
>>> I analyzed it a bit more and found that the last working version was
>>> 0.43. Any later released version leads to this crash sooner or later,
>>> but as I already said only on a 10Gbit (FC) network. I didn't see any
>>> crash on the 1Gbit net on the same machines.
>>>
>>> What kind of network do you use at dreamhost for testing?
>>
>> Mostly 1Gbit, some 10Gbit, but I don't think we've tested the kernel
>> client on 10Gbit yet.
> 
> That's what I assumed ;-)
> 
>>> If you need more info, let me known.
>>
>> Do the crashes always have the same stack trace? When you say 10Gbit
>> for the cluster, does that include the client using rbd, or just the
>> osds?
> 
> It's always the same stack trace (sometimes a address is different, but
> everything else looks identical).
> 
> We tested basically the following setups and the crash happend with all
> of them:
> 1) OSD, MON and Clients in the same 10Gbit network
> 2) OSD, MON and Clients in different public/cluster 10Gbit networks
> 3) OSD and Clients in the same 10Gbit network, MON in 1Gbit network
> 3) OSD and Clients in the same 10Gbit network, MON in 1Gbit network
>    different public/cluster networks
> 
> The number of OSDs (tested 4 nodes with 10 OSDs per node, each one
> physical harddisk) didn't matter in this case. If I use 2 clients
> running fio tests against one 50GByte RBD per client, I hit the problem
> faster than with one client. If you need information about the used fio
> tests, let me know.
> 
> As already I said: we didn't hit this problem with 1Gbit networks yet.

Now I see this kind of crash also on 1Gbit and running tests against RBD
with the following setup:
- 30 OSD nodes, 3 OSDs per node
- 11 clients
- 3 MON
- 2 MDS (1 active, 1 standby)
- different public/cluster 1Gbit networks
- ceph 0.45

ceph -s:
------------
2012-04-14 15:29:53.744829    pg v5609: 18018 pgs: 18018 active+clean;
549 GB data, 1107 GB used, 22382 GB / 23490 GB avail
2012-04-14 15:29:53.781966   mds e6: 1/1/1 up {0=alpha=up:active}, 1
up:standby-replay
2012-04-14 15:29:53.782002   osd e26: 90 osds: 90 up, 90 in
2012-04-14 15:29:53.782233   log 2012-04-14 15:11:51.857127 osd.68
192.168.111.43:6801/23591 1372 : [WRN] old request
osd_sub_op(client.4180.1:931208 2.dfc eca8dfc/rb.0.3.000000000e76/head
[] v 26'1534 snapset=0=[]:[] snapc=0=[]) v6 received at 2012-04-14
15:11:20.882595 currently startedold request
osd_sub_op(client.4250.1:885302 2.898 6bf1d898/rb.0.6.000000000b61/head
[] v 26'1662 snapset=0=[]:[] snapc=0=[]) v6 received at 2012-04-14
15:11:21.663493 currently startedold request osd_op(client.4262.1:888106
rb.0.9.000000000672 [write 1286144~4096] 2.45aa94a) received at
2012-04-14 15:11:19.879144 currently waiting for sub opsold request
osd_op(client.4250.1:885252 rb.0.6.0000000018a2 [write 1073152~4096]
2.fb3ab288) received at 2012-04-14 15:11:19.927412 currently waiting for
sub opsold request osd_op(client.4241.1:930301 rb.0.1.000000002fdc
[write 8192~4096] 2.9110ba90) received at 2012-04-14 15:11:20.645040
currently waiting for sub opsold request osd_op(client.4250.1:885278
rb.0.6.0000000013cf [write 1892352~4096] 2.cfe911f0) received at
2012-04-14 15:11:20.616330 currently waiting for sub ops
2012-04-14 15:29:53.782370   mon e1: 3 mons at
{alpha=192.168.111.33:6789/0,beta=192.168.111.34:6789/0,gamma=192.168.111.35:6789/0}
------------

crash backtrace:
------------
PID: 86     TASK: ffff880432326040  CPU: 13  COMMAND: "kworker/13:1"
 #0 [ffff880432329970] machine_kexec at ffffffff810265ee
 #1 [ffff8804323299c0] crash_kexec at ffffffff810a3bda
 #2 [ffff880432329a90] oops_end at ffffffff81444688
 #3 [ffff880432329ab0] __bad_area_nosemaphore at ffffffff81032a35
 #4 [ffff880432329b70] do_page_fault at ffffffff81446d3e
 #5 [ffff880432329c70] page_fault at ffffffff81443865
    [exception RIP: write_partial_msg_pages+1181]
    RIP: ffffffffa0352e8d  RSP: ffff880432329d20  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff88043268a030  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000000001000
    RBP: 0000000000000000   R8: 0000000000000000   R9: 00000000479aae8f
    R10: 00000000000005a8  R11: 0000000000000000  R12: 0000000000001000
    R13: ffffea000e917608  R14: 0000160000000000  R15: ffff88043217fbc0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #6 [ffff880432329d78] try_write at ffffffffa0355345 [libceph]
 #7 [ffff880432329df8] con_work at ffffffffa0355bdd [libceph]
 #8 [ffff880432329e28] process_one_work at ffffffff8107487c
 #9 [ffff880432329e78] worker_thread at ffffffff8107740a
#10 [ffff880432329ee8] kthread at ffffffff8107b736
#11 [ffff880432329f48] kernel_thread_helper at ffffffff8144c144
------------

Danny


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: Kernel crashes with RBD

[Yan, Zheng]

I think I tracked this bug down, the Oops is due to 'msg->bio_iter == NULL'.

---
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index f0993af..ac16f13 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -549,6 +549,10 @@ static void prepare_write_message(struct
ceph_connection *con)
 	}

 	m = list_first_entry(&con->out_queue, struct ceph_msg, list_head);
+#ifdef CONFIG_BLOCK
+	if (m->bio && m->bio_iter)
+		m->bio_iter = NULL;
+#endif
 	con->out_msg = m;

 	/* put message on sent list */


On Thu, Apr 12, 2012 at 6:30 AM, Danny Kukawka <danny.kukawka@bisect.de> wrote:
> Hi,
>
> we are currently testing CEPH with RBD on a cluster with 1GBit and
> 10Gbit interfaces. While we see no kernel crashes with RBD if the
> cluster runs on the 1GBit interfaces, we see very frequent kernel
> crashes with the 10Gbit network while running tests with e.g. fio
> against the RBDs.
>
> I've tested it with kernel v3.0 and also 3.3.0 (with the patches from
> the 'for-linus' branch from ceph-client.git at git.kernel.org).
>
> With more client machines running tests the crashes occur even much
> faster. The issue is fully reproducible here.
>
> Has anyone seen similar problems? See the backtrace below.
>
> Regards
>
> Danny
>
> PID: 10902  TASK: ffff88032a9a2080  CPU: 0   COMMAND: "kworker/0:0"
>  #0 [ffff8803235fd950] machine_kexec at ffffffff810265ee
>  #1 [ffff8803235fd9a0] crash_kexec at ffffffff810a3bda
>  #2 [ffff8803235fda70] oops_end at ffffffff81444688
>  #3 [ffff8803235fda90] __bad_area_nosemaphore at ffffffff81032a35
>  #4 [ffff8803235fdb50] do_page_fault at ffffffff81446d3e
>  #5 [ffff8803235fdc50] page_fault at ffffffff81443865
>    [exception RIP: read_partial_message+816]
>    RIP: ffffffffa041e500  RSP: ffff8803235fdd00  RFLAGS: 00010246
>    RAX: 0000000000000000  RBX: 00000000000009d7  RCX: 0000000000008000
>    RDX: 0000000000000000  RSI: 00000000000009d7  RDI: ffffffff813c8d78
>    RBP: ffff880328827030   R8: 00000000000009d7   R9: 0000000000004000
>    R10: 0000000000000000  R11: ffffffff81205800  R12: 0000000000000000
>    R13: 0000000000000069  R14: ffff88032a9bc780  R15: 0000000000000000
>    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
>  #6 [ffff8803235fdd38] thread_return at ffffffff81440e82
>  #7 [ffff8803235fdd78] try_read at ffffffffa041ed58 [libceph]
>  #8 [ffff8803235fddf8] con_work at ffffffffa041fb2e [libceph]
>  #9 [ffff8803235fde28] process_one_work at ffffffff8107487c
> #10 [ffff8803235fde78] worker_thread at ffffffff8107740a
> #11 [ffff8803235fdee8] kthread at ffffffff8107b736
> #12 [ffff8803235fdf48] kernel_thread_helper at ffffffff8144c144
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html