Re: Kernel crashes with RBD

[Danny Kukawka <danny.kukawka@bisect.de>]

[-- Attachment #1: Type: text/plain, Size: 2178 bytes --]

Am 13.04.2012 22:56, schrieb Josh Durgin:
> On 04/13/2012 11:18 AM, Danny Kukawka wrote:
>> Hi
>>
>> Am 13.04.2012 19:48, schrieb Josh Durgin:
>>> On 04/11/2012 03:30 PM, Danny Kukawka wrote:
>> [...]
>>>
>>> This looks similar to http://tracker.newdream.net/issues/2261. What do
>>> you think Alex?
>>
>> Not sure about that, since this crashes only the clients and not the
>> OSDs. We see no crashes in the cluster.
> 
> These are both rbd kernel client crashes, but looking again they seem
> like different underlying issues, so I opened
> http://tracker.newdream.net/issues/2287 to track this problem.
> 
>>
>> I analyzed it a bit more and found that the last working version was
>> 0.43. Any later released version leads to this crash sooner or later,
>> but as I already said only on a 10Gbit (FC) network. I didn't see any
>> crash on the 1Gbit net on the same machines.
>>
>> What kind of network do you use at dreamhost for testing?
> 
> Mostly 1Gbit, some 10Gbit, but I don't think we've tested the kernel
> client on 10Gbit yet.

That's what I assumed ;-)

>> If you need more info, let me known.
> 
> Do the crashes always have the same stack trace? When you say 10Gbit
> for the cluster, does that include the client using rbd, or just the
> osds?

It's always the same stack trace (sometimes a address is different, but
everything else looks identical).

We tested basically the following setups and the crash happend with all
of them:
1) OSD, MON and Clients in the same 10Gbit network
2) OSD, MON and Clients in different public/cluster 10Gbit networks
3) OSD and Clients in the same 10Gbit network, MON in 1Gbit network
3) OSD and Clients in the same 10Gbit network, MON in 1Gbit network
   different public/cluster networks

The number of OSDs (tested 4 nodes with 10 OSDs per node, each one
physical harddisk) didn't matter in this case. If I use 2 clients
running fio tests against one 50GByte RBD per client, I hit the problem
faster than with one client. If you need information about the used fio
tests, let me know.

As already I said: we didn't hit this problem with 1Gbit networks yet.

Danny


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]