Re: domino-style OSD crash

[Yann Dupont <Yann.Dupont@univ-nantes.fr>]

Le 05/07/2012 23:32, Gregory Farnum a écrit :

[...]
>> ok, so as all nodes were identical, I probably have hit a btrfs bug (like a
>> erroneous out of space ) in more or less the same time. And when 1 osd was
>> out,

OH , I didn't finish the sentence... When 1 osd was out, missing data 
was copied on another nodes, probably speeding btrfs problem on those 
nodes (I suspect erroneous out of space conditions)

I've reformatted OSD with xfs. Performance is slightly worse for the 
moment (well, depend on the workload, and maybe lack of syncfs is to 
blame), but at least I hope to have the storage layer rock-solid. BTW, 
I've managed to keep the faulty btrfs volumes .

[...]

>>> I wonder if maybe there's a confounding factor here — are all your nodes
>>> similar to each other,
>> Yes. I designed the cluster that way. All nodes are identical hardware
>> (powerEdge M610, 10G intel ethernet + emulex fibre channel attached to
>> storage (1 Array for 2 OSD nodes, 1 controller dedicated for each OSD)
> Oh, interesting. Are the broken nodes all on the same set of arrays?

No. There are 4 completely independant raid arrays, in 4 different 
locations. They are similar (same brand & model, but slighltly different 
disks, and 1 different firmware), all arrays are multipathed. I don't 
think the raid array is the problem. We use those particular models 
since 2/3 years, and in the logs I don't see any problem that can be 
caused by the storage itself (like scsi or multipath errors)

Cheers,

-- 
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html