From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Atchley, Scott" Subject: Re: flashcache Date: Thu, 17 Jan 2013 10:00:55 -0500 Message-ID: References: <50F72150.7080002@inktank.com> <50F7FE9B.6050908@inktank.com> <44CC6A16-2ADD-4DBB-8B07-0E16032AEDA2@ornl.gov> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7BIT Return-path: Received: from mta01.ornl.gov ([128.219.177.14]:52109 "EHLO mta01.ornl.gov" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756027Ab3AQPA6 convert rfc822-to-8bit (ORCPT ); Thu, 17 Jan 2013 10:00:58 -0500 In-Reply-To: Content-Language: en-US Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Gandalf Corvotempesta Cc: Mark Nelson , Joseph Glanville , Sage Weil , "ceph-devel@vger.kernel.org" On Jan 17, 2013, at 9:48 AM, Gandalf Corvotempesta wrote: > 2013/1/17 Atchley, Scott : >> IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE. > > Which kind of tuning? Do you have a paper about this? No, I followed the Mellanox tuning guide and modified their interrupt affinity scripts. > But, actually, is possible to use ceph with IPoIB in a stable way or > is this experimental ? IPoIB appears as a traditional Ethernet device to Linux and can be used as such. Ceph has no idea that it is not Ethernet. > I don't know if i support for rsocket that is experimental/untested > and IPoIB is a stable workaroud or what else. IPoIB is much more used and pretty stable, while rsockets is new with limited testing. That said, more people using it will help Sean improve it. Ideally, we would like support for zero-copy and reduced CPU usage (via OS-bypass) and with more interconnects than just InfiniBand. :-) > And is a dual controller needed on each OSD node? Ceph is able to > handle OSD network failures? This is really important to know. It > change the whole network topology. I will let others answer this. Scott