All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Atchley, Scott" <atchleyes@ornl.gov>
To: Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com>
Cc: Mark Nelson <mark.nelson@inktank.com>,
	Joseph Glanville <joseph.glanville@orionvm.com.au>,
	Sage Weil <sage@inktank.com>,
	"ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: flashcache
Date: Thu, 17 Jan 2013 10:00:55 -0500	[thread overview]
Message-ID: <F926561F-9D73-4B74-AFC7-F5B962810733@ornl.gov> (raw)
In-Reply-To: <CAJH6TXhMXfsgx607aOxpLmpq15vg44AW8NHbm_Yw8yC45CHyKA@mail.gmail.com>

On Jan 17, 2013, at 9:48 AM, Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:

> 2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
>> IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE.
> 
> Which kind of tuning? Do you have a paper about this?

No, I followed the Mellanox tuning guide and modified their interrupt affinity scripts.

> But, actually, is possible to use ceph with IPoIB in a stable way or
> is this experimental ?

IPoIB appears as a traditional Ethernet device to Linux and can be used as such. Ceph has no idea that it is not Ethernet.

> I don't know if i support for rsocket that is experimental/untested
> and IPoIB is a stable workaroud or what else.

IPoIB is much more used and pretty stable, while rsockets is new with limited testing. That said, more people using it will help Sean improve it.

Ideally, we would like support for zero-copy and reduced CPU usage (via OS-bypass) and with more interconnects than just InfiniBand. :-)

> And is a dual controller needed on each OSD node? Ceph is able to
> handle OSD network failures? This is really important to know. It
> change the whole network topology.

I will let others answer this.

Scott

  reply	other threads:[~2013-01-17 15:00 UTC|newest]

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-01-16 21:22 flashcache Gandalf Corvotempesta
2013-01-16 21:29 ` flashcache Sage Weil
2013-01-16 21:42   ` flashcache Gandalf Corvotempesta
2013-01-16 21:46     ` flashcache Sage Weil
2013-01-16 21:55       ` flashcache Mark Nelson
2013-01-16 21:59         ` flashcache Gandalf Corvotempesta
2013-01-16 21:57       ` flashcache Gandalf Corvotempesta
2013-01-16 21:53   ` flashcache Mark Nelson
2013-01-16 22:04     ` flashcache Gandalf Corvotempesta
2013-01-17  5:47     ` flashcache Stefan Priebe - Profihost AG
2013-01-17 13:34       ` flashcache Mark Nelson
2013-01-17  9:46     ` flashcache Gandalf Corvotempesta
2013-01-17 13:32       ` flashcache Joseph Glanville
2013-01-17 13:37         ` flashcache Mark Nelson
2013-01-17 13:44           ` flashcache Gandalf Corvotempesta
2013-01-17 14:30           ` flashcache Atchley, Scott
2013-01-17 14:48             ` flashcache Gandalf Corvotempesta
2013-01-17 15:00               ` Atchley, Scott [this message]
2013-01-17 15:07                 ` flashcache Andrey Korolyov
2013-01-17 15:47                   ` flashcache Atchley, Scott
2013-01-17 16:39                     ` flashcache Andrey Korolyov
2013-01-20  2:56                       ` flashcache Joseph Glanville
2013-01-21 23:57                         ` flashcache John Nielsen
2013-01-30 20:10                           ` flashcache Joseph Glanville
2013-01-17 15:14                 ` flashcache Gandalf Corvotempesta
2013-01-17 15:50                   ` flashcache Atchley, Scott
2013-01-17 16:01                     ` flashcache Gandalf Corvotempesta
2013-01-17 16:12                       ` flashcache Atchley, Scott
2013-01-17 16:19                         ` flashcache Gandalf Corvotempesta
2013-01-22 21:06                           ` flashcache Atchley, Scott
2013-01-22 21:08                             ` flashcache Atchley, Scott
2013-01-17 16:20                         ` flashcache Stefan Priebe
2013-01-17 16:21                           ` flashcache Gandalf Corvotempesta
2013-01-17 16:24                             ` flashcache Stefan Priebe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=F926561F-9D73-4B74-AFC7-F5B962810733@ornl.gov \
    --to=atchleyes@ornl.gov \
    --cc=ceph-devel@vger.kernel.org \
    --cc=gandalf.corvotempesta@gmail.com \
    --cc=joseph.glanville@orionvm.com.au \
    --cc=mark.nelson@inktank.com \
    --cc=sage@inktank.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.