flashcache

flashcache

[Gandalf Corvotempesta]

In a ceph cluster, flashcache with writeback is considered safe?
In case of SSD failure, the flashcache contents should be already been
replicated (by ceph) in other servers, right?

I'm planning to use this configuration: Supermicro with 12 spinning
disks e 2 SSD.
6 spinning disks will have ceph journal on SSD1, the other 6 disks
will have ceph journal on disks2.

One OSD for each spinning disk (a single XFS filesystem for the whole disk).
XFS metadata to a parition of SSD1
XFS flashcache to another partition of SSD1

So, 3 partitions for each OSD on the SSD.
How big should be these partitions? Any advice?

No raid at all, except for 1 RAID-1 volume made with a 10GB partitions
on each SSD, for the OS. Log files will be replicated to a remote
server, so writes on OS partitions are very very low.

Any hint? Adivice? Critics?

Re: flashcache

[Sage Weil]

On Wed, 16 Jan 2013, Gandalf Corvotempesta wrote:
> In a ceph cluster, flashcache with writeback is considered safe?
> In case of SSD failure, the flashcache contents should be already been
> replicated (by ceph) in other servers, right?

This sort of configuration effectively bundles the disk and SSD into a 
single unit, where the failure of either results in the loss of both.  
From Ceph's perspective, it doesn't matter if the thing it is sitting on 
is a single disk, an SSD+disk flashcache thing, or a big RAID array.  All 
that changes is the probability of failure.

The thing to watch out for *knowing* that the whole is lost when one part 
fails (vs plowing ahead with a corrupt fs).

> I'm planning to use this configuration: Supermicro with 12 spinning
> disks e 2 SSD.
> 6 spinning disks will have ceph journal on SSD1, the other 6 disks
> will have ceph journal on disks2.
> 
> One OSD for each spinning disk (a single XFS filesystem for the whole disk).
> XFS metadata to a parition of SSD1
> XFS flashcache to another partition of SSD1
> 
> So, 3 partitions for each OSD on the SSD.
> How big should be these partitions? Any advice?
> 
> No raid at all, except for 1 RAID-1 volume made with a 10GB partitions
> on each SSD, for the OS. Log files will be replicated to a remote
> server, so writes on OS partitions are very very low.
> 
> Any hint? Adivice? Critics?

I would worry that there is a lot of stuff piling onto the SSD and it may 
become your bottleneck.  My guess is that another 1-2 SSDs will be a 
better 'balance', but only experiementation will really tell us that.

Otherwise, those seem to all be good things to put on teh SSD!

sage

Re: flashcache

[Gandalf Corvotempesta]

2013/1/16 Sage Weil <sage@inktank.com>:
> This sort of configuration effectively bundles the disk and SSD into a
> single unit, where the failure of either results in the loss of both.
> From Ceph's perspective, it doesn't matter if the thing it is sitting on
> is a single disk, an SSD+disk flashcache thing, or a big RAID array.  All
> that changes is the probability of failure.

Ok, it will fail, but this should not be an issue, in a cluster like
ceph, right?
With or without flashcache or SSD, ceph should be able to handle
disks/nodes/osds failures on its own by replicating in real time to
multiple server.

Should I worry about loosing data in case of failure? It should
rebalance automatically in case of failure with no data loss.

> I would worry that there is a lot of stuff piling onto the SSD and it may
> become your bottleneck.  My guess is that another 1-2 SSDs will be a
> better 'balance', but only experiementation will really tell us that.
>
> Otherwise, those seem to all be good things to put on teh SSD!

I can't add more than 2 SSD, I don't have enough space.
I can move OS to the first 2 spinning disks in raid1 software, if this
will improve performance of SSD

What about swap? I'm thinking to no use swap at all and start with 16/32GB RAM

Re: flashcache

[Sage Weil]

On Wed, 16 Jan 2013, Gandalf Corvotempesta wrote:
> 2013/1/16 Sage Weil <sage@inktank.com>:
> > This sort of configuration effectively bundles the disk and SSD into a
> > single unit, where the failure of either results in the loss of both.
> > From Ceph's perspective, it doesn't matter if the thing it is sitting on
> > is a single disk, an SSD+disk flashcache thing, or a big RAID array.  All
> > that changes is the probability of failure.
> 
> Ok, it will fail, but this should not be an issue, in a cluster like
> ceph, right?
> With or without flashcache or SSD, ceph should be able to handle
> disks/nodes/osds failures on its own by replicating in real time to
> multiple server.

Exactly.

> Should I worry about loosing data in case of failure? It should
> rebalance automatically in case of failure with no data loss.

You should not worry, except to the extent that 2 might fail 
simultaneously, and failures in general are not good things.

> > I would worry that there is a lot of stuff piling onto the SSD and it may
> > become your bottleneck.  My guess is that another 1-2 SSDs will be a
> > better 'balance', but only experiementation will really tell us that.
> >
> > Otherwise, those seem to all be good things to put on teh SSD!
> 
> I can't add more than 2 SSD, I don't have enough space.
> I can move OS to the first 2 spinning disks in raid1 software, if this
> will improve performance of SSD
> 
> What about swap? I'm thinking to no use swap at all and start with 
> 16/32GB RAM

You could use the first (single) disk for os and logs.  You might not even 
bother with raid1, since you will presumably be replicating across hosts.  
When the OSD disk dies, you can re-run your chef/juju/puppet rule or 
whatever provisioning tool is at work to reinstall/configure the OS disk.  
The data on the SSDs and data disks will all be intact.

sage

Re: flashcache

[Mark Nelson]

On 01/16/2013 03:29 PM, Sage Weil wrote:
> On Wed, 16 Jan 2013, Gandalf Corvotempesta wrote:
>> In a ceph cluster, flashcache with writeback is considered safe?
>> In case of SSD failure, the flashcache contents should be already been
>> replicated (by ceph) in other servers, right?
>
> This sort of configuration effectively bundles the disk and SSD into a
> single unit, where the failure of either results in the loss of both.
>  From Ceph's perspective, it doesn't matter if the thing it is sitting on
> is a single disk, an SSD+disk flashcache thing, or a big RAID array.  All
> that changes is the probability of failure.
>
> The thing to watch out for *knowing* that the whole is lost when one part
> fails (vs plowing ahead with a corrupt fs).
>
>> I'm planning to use this configuration: Supermicro with 12 spinning
>> disks e 2 SSD.
>> 6 spinning disks will have ceph journal on SSD1, the other 6 disks
>> will have ceph journal on disks2.
>>
>> One OSD for each spinning disk (a single XFS filesystem for the whole disk).
>> XFS metadata to a parition of SSD1
>> XFS flashcache to another partition of SSD1
>>
>> So, 3 partitions for each OSD on the SSD.
>> How big should be these partitions? Any advice?
>>
>> No raid at all, except for 1 RAID-1 volume made with a 10GB partitions
>> on each SSD, for the OS. Log files will be replicated to a remote
>> server, so writes on OS partitions are very very low.
>>
>> Any hint? Adivice? Critics?

Looks like a fun configuration to test!  Having said that, I have no 
idea how stable flashcache is.  It's certainly not something we've used 
in production before!  Keep that in mind.

With only 2 SSDs for 12 spinning disks, you'll need to make sure the 
SSDs are really fast.  I use Intel 520s for testing which are great, but 
I wouldn't use them in  production.  The S3700 might be a good bet at 
larger sizes, but it looks like the 100GB version is a lot slower than 
the 200GB version, and that's still a bit slower than the 400GB version. 
  Assuming you have 10GbE, you'll probably be capped by the SSDs for 
large block sequential workloads.  Having said that, I still think this 
has potential to be a nice setup.  Just be aware that we usually don't 
stick that much stuff on the SSDs!

>
> I would worry that there is a lot of stuff piling onto the SSD and it may
> become your bottleneck.  My guess is that another 1-2 SSDs will be a
> better 'balance', but only experiementation will really tell us that.
>

It'd be amazing if supermicro could cram another 2 SSD slots in the 
back.  Maybe by that time we'll all be using PCIE flash storage though. :)

> Otherwise, those seem to all be good things to put on teh SSD!
>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: flashcache

[Mark Nelson]

On 01/16/2013 03:46 PM, Sage Weil wrote:
> On Wed, 16 Jan 2013, Gandalf Corvotempesta wrote:
>> 2013/1/16 Sage Weil <sage@inktank.com>:
>>> This sort of configuration effectively bundles the disk and SSD into a
>>> single unit, where the failure of either results in the loss of both.
>>>  From Ceph's perspective, it doesn't matter if the thing it is sitting on
>>> is a single disk, an SSD+disk flashcache thing, or a big RAID array.  All
>>> that changes is the probability of failure.
>>
>> Ok, it will fail, but this should not be an issue, in a cluster like
>> ceph, right?
>> With or without flashcache or SSD, ceph should be able to handle
>> disks/nodes/osds failures on its own by replicating in real time to
>> multiple server.
>
> Exactly.
>
>> Should I worry about loosing data in case of failure? It should
>> rebalance automatically in case of failure with no data loss.
>
> You should not worry, except to the extent that 2 might fail
> simultaneously, and failures in general are not good things.
>
>>> I would worry that there is a lot of stuff piling onto the SSD and it may
>>> become your bottleneck.  My guess is that another 1-2 SSDs will be a
>>> better 'balance', but only experiementation will really tell us that.
>>>
>>> Otherwise, those seem to all be good things to put on teh SSD!
>>
>> I can't add more than 2 SSD, I don't have enough space.
>> I can move OS to the first 2 spinning disks in raid1 software, if this
>> will improve performance of SSD
>>
>> What about swap? I'm thinking to no use swap at all and start with
>> 16/32GB RAM
>
> You could use the first (single) disk for os and logs.  You might not even
> bother with raid1, since you will presumably be replicating across hosts.
> When the OSD disk dies, you can re-run your chef/juju/puppet rule or
> whatever provisioning tool is at work to reinstall/configure the OS disk.
> The data on the SSDs and data disks will all be intact.

Other options might be network boot or even usb stick boot.

>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: flashcache

[Gandalf Corvotempesta]

2013/1/16 Sage Weil <sage@inktank.com>:
> You should not worry, except to the extent that 2 might fail
> simultaneously, and failures in general are not good things.

Are you talking about 2 OSDs failure (with the same replicated data)?
Ok, but it's very very hard.

> You could use the first (single) disk for os and logs.  You might not even
> bother with raid1, since you will presumably be replicating across hosts.
> When the OSD disk dies, you can re-run your chef/juju/puppet rule or
> whatever provisioning tool is at work to reinstall/configure the OS disk.
> The data on the SSDs and data disks will all be intact.

OS failure will result in the whole node down. This will cause a
rebalance of many TB of data that I prefer to avoid, if possible.
I also prefer to avoid a dedicated disks for the OS. I'll loose to much space.
With OS in a RAID-1 partition of the first and second spinning disks
(the same disks will also act as OSD with another, bigger, partition),
OS failure will not bring down anything and I can hot-replace the
failed disk.

RAID1 will handle OS reconstruction, ceph will handle the cluster rebalance

Re: flashcache

[Gandalf Corvotempesta]

2013/1/16 Mark Nelson <mark.nelson@inktank.com>:
> Other options might be network boot or even usb stick boot.

GREAT IDEA!
I'll go with a network boot.

Re: flashcache

[Gandalf Corvotempesta]

2013/1/16 Mark Nelson <mark.nelson@inktank.com>:
> Looks like a fun configuration to test!  Having said that, I have no idea
> how stable flashcache is.  It's certainly not something we've used in
> production before!  Keep that in mind.

As wrote before, this should not be an issue, ceph should handle failures.
But is ceph able to detect a badly-wrote blocks?

> With only 2 SSDs for 12 spinning disks, you'll need to make sure the SSDs
> are really fast.  I use Intel 520s for testing which are great, but I
> wouldn't use them in  production.  The S3700 might be a good bet at larger
> sizes, but it looks like the 100GB version is a lot slower than the 200GB
> version, and that's still a bit slower than the 400GB version.  Assuming you
> have 10GbE, you'll probably be capped by the SSDs for large block sequential
> workloads.  Having said that, I still think this has potential to be a nice
> setup.  Just be aware that we usually don't stick that much stuff on the
> SSDs!

12 spinning disks will be the worse scenario.
When in production i'll start with 5 server, with 6 disks each (3+3
for each SSD)
Then i'll try to add hosts instead of adding disks.
I don't have 10GbE actually but only 2GbE bonded. I can evaluate 4GbE.

> It'd be amazing if supermicro could cram another 2 SSD slots in the back.
> Maybe by that time we'll all be using PCIE flash storage though. :)

How much a PCIE storage?

Re: flashcache

[Stefan Priebe - Profihost AG]

Hi Mark,

Am 16.01.2013 um 22:53 schrieb Mark 
> With only 2 SSDs for 12 spinning disks, you'll need to make sure the SSDs are really fast.  I use Intel 520s for testing which are great, but I wouldn't use them in  production.

Why not? I use them for a ssd only ceph cluster.

Stefan

Re: flashcache

[Gandalf Corvotempesta]

2013/1/16 Mark Nelson <mark.nelson@inktank.com>:
> Assuming you have 10GbE, you'll probably be capped by the SSDs for
> large block sequential workloads.  Having said that, I still think this has
> potential to be a nice setup.  Just be aware that we usually don't stick
> that much stuff on the SSDs!

10GbE for which network, OSDs or client network? I think you are
talking about OSD network.

I'm evaluating a small Infiniband 10GB or 20GB network , but I'm still
not sure if should I
make the OSD network in the node by using 2 IB card. What happens when
the OSD network
fails on a single node but that node is still reachable from the client network?

Ceph will remove it from the cluster?

I don't know if I have to use a single two port IB card (switch
redundancy and no card redundancy) or
I have to use two single port cards. (or a single one port IB?)

Re: flashcache

[Joseph Glanville]

On 17 January 2013 20:46, Gandalf Corvotempesta
<gandalf.corvotempesta@gmail.com> wrote:
> 2013/1/16 Mark Nelson <mark.nelson@inktank.com>:

> I don't know if I have to use a single two port IB card (switch
> redundancy and no card redundancy) or
> I have to use two single port cards. (or a single one port IB?)

On the topic of IB..

But slightly off-topic all the same..  I would love to attempt getting
Ceph running on rsockets if I could find the time (alas we don't run
Ceph).
rsockets is a fully userland implementation of BSD sockets over RDMA,
supporting fork and all the usual goodies, in theory unless you are
using the kernel RBD module (of the kernel FS module etc) you should
be able to run it on rsockets and enjoy a considerable performance
increase.

rsockets is available in the librdmacm git up on Open Fabrics and dev
+ support happens on the linux-rdma list.

-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: flashcache

[Mark Nelson]

On 01/16/2013 11:47 PM, Stefan Priebe - Profihost AG wrote:
> Hi Mark,
>
> Am 16.01.2013 um 22:53 schrieb Mark
>> With only 2 SSDs for 12 spinning disks, you'll need to make sure the SSDs are really fast.  I use Intel 520s for testing which are great, but I wouldn't use them in  production.
>
> Why not? I use them for a ssd only ceph cluster.
>
> Stefan

It's pretty tough to get an apples-to-apples comparison of endurance 
when looking at the Intel 520 vs something like the DC S3700. If I were 
actually building out a production deployment I'd probably stick with 
the DC S3700 (especially if sticking journals, flashcache, and xfs 
journals for 6 osds on 1 drive!).  There's probably a reasonable 
endurance per cost argument for a severely under-subscribed 520 (or 
other similar drive) as well.  It'd be an interesting study to look at 
how long it takes a small enterpise drives to die vs a larger 
under-subscribed consumer drive.

-- 
Mark Nelson
Performance Engineer
Inktank

Re: flashcache

[Mark Nelson]

On 01/17/2013 07:32 AM, Joseph Glanville wrote:
> On 17 January 2013 20:46, Gandalf Corvotempesta
> <gandalf.corvotempesta@gmail.com>  wrote:
>> 2013/1/16 Mark Nelson<mark.nelson@inktank.com>:
>
>> I don't know if I have to use a single two port IB card (switch
>> redundancy and no card redundancy) or
>> I have to use two single port cards. (or a single one port IB?)
>
> On the topic of IB..
>
> But slightly off-topic all the same..  I would love to attempt getting
> Ceph running on rsockets if I could find the time (alas we don't run
> Ceph).
> rsockets is a fully userland implementation of BSD sockets over RDMA,
> supporting fork and all the usual goodies, in theory unless you are
> using the kernel RBD module (of the kernel FS module etc) you should
> be able to run it on rsockets and enjoy a considerable performance
> increase.
>
> rsockets is available in the librdmacm git up on Open Fabrics and dev
> + support happens on the linux-rdma list.
>

There's been some talk about rsockets on the list before.  I think there 
are a couple of different folks that have tried (succeeded?) in getting 
it working.  barring that, it sounds like if you tune interrupt affinity 
settings and various other bits you can get IPoIB up into the 2GB/s+ 
range which while not RDMA speed, is at least better than 10GbE.

-- 
Mark Nelson
Performance Engineer
Inktank

Re: flashcache

[Gandalf Corvotempesta]

2013/1/17 Mark Nelson <mark.nelson@inktank.com>:
> There's been some talk about rsockets on the list before.  I think there are
> a couple of different folks that have tried (succeeded?) in getting it
> working.  barring that, it sounds like if you tune interrupt affinity
> settings and various other bits you can get IPoIB up into the 2GB/s+ range
> which while not RDMA speed, is at least better than 10GbE.

More over, an SDR/DDR IB switch is much more cheaper than a full 10GbE switch.

Re: flashcache

[Atchley, Scott]

On Jan 17, 2013, at 8:37 AM, Mark Nelson <mark.nelson@inktank.com> wrote:

> On 01/17/2013 07:32 AM, Joseph Glanville wrote:
>> On 17 January 2013 20:46, Gandalf Corvotempesta
>> <gandalf.corvotempesta@gmail.com>  wrote:
>>> 2013/1/16 Mark Nelson<mark.nelson@inktank.com>:
>> 
>>> I don't know if I have to use a single two port IB card (switch
>>> redundancy and no card redundancy) or
>>> I have to use two single port cards. (or a single one port IB?)
>> 
>> On the topic of IB..
>> 
>> But slightly off-topic all the same..  I would love to attempt getting
>> Ceph running on rsockets if I could find the time (alas we don't run
>> Ceph).
>> rsockets is a fully userland implementation of BSD sockets over RDMA,
>> supporting fork and all the usual goodies, in theory unless you are
>> using the kernel RBD module (of the kernel FS module etc) you should
>> be able to run it on rsockets and enjoy a considerable performance
>> increase.
>> 
>> rsockets is available in the librdmacm git up on Open Fabrics and dev
>> + support happens on the linux-rdma list.
>> 
> 
> There's been some talk about rsockets on the list before.  I think there 
> are a couple of different folks that have tried (succeeded?) in getting 
> it working.  barring that, it sounds like if you tune interrupt affinity 
> settings and various other bits you can get IPoIB up into the 2GB/s+ 
> range which while not RDMA speed, is at least better than 10GbE.

IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE.

I would be interested in seeing if rsockets helps ceph. I have questions though.

By default, rsockets still has to copy data in and out. It has extensions for zero-copy, but they do not work with non-blocking sockets. Does ceph use non-blocking sockets?

How many simultaneous connections does it support before falling over into the non-rsockets path?

If rsockets has a ceiling on connections and falls over to the non-socket path, can the application determine if it is safe to use the zero-copy extensions for a specific connection or do they fail gracefully?

Scott

Re: flashcache

[Gandalf Corvotempesta]

2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
> IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE.

Which kind of tuning? Do you have a paper about this?

But, actually, is possible to use ceph with IPoIB in a stable way or
is this experimental ?
I don't know if i support for rsocket that is experimental/untested
and IPoIB is a stable workaroud or what else.

And is a dual controller needed on each OSD node? Ceph is able to
handle OSD network failures? This is really important to know. It
change the whole network topology.

Re: flashcache

[Atchley, Scott]

On Jan 17, 2013, at 9:48 AM, Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:

> 2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
>> IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE.
> 
> Which kind of tuning? Do you have a paper about this?

No, I followed the Mellanox tuning guide and modified their interrupt affinity scripts.

> But, actually, is possible to use ceph with IPoIB in a stable way or
> is this experimental ?

IPoIB appears as a traditional Ethernet device to Linux and can be used as such. Ceph has no idea that it is not Ethernet.

> I don't know if i support for rsocket that is experimental/untested
> and IPoIB is a stable workaroud or what else.

IPoIB is much more used and pretty stable, while rsockets is new with limited testing. That said, more people using it will help Sean improve it.

Ideally, we would like support for zero-copy and reduced CPU usage (via OS-bypass) and with more interconnects than just InfiniBand. :-)

> And is a dual controller needed on each OSD node? Ceph is able to
> handle OSD network failures? This is really important to know. It
> change the whole network topology.

I will let others answer this.

Scott

Re: flashcache

[Andrey Korolyov]

On Thu, Jan 17, 2013 at 7:00 PM, Atchley, Scott <atchleyes@ornl.gov> wrote:
> On Jan 17, 2013, at 9:48 AM, Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:
>
>> 2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
>>> IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE.
>>
>> Which kind of tuning? Do you have a paper about this?
>
> No, I followed the Mellanox tuning guide and modified their interrupt affinity scripts.

Did you tried to bind interrupts only to core to which QPI link
belongs in reality and measure difference with spread-over-all-cores
binding?

>
>> But, actually, is possible to use ceph with IPoIB in a stable way or
>> is this experimental ?
>
> IPoIB appears as a traditional Ethernet device to Linux and can be used as such.

Not exactly, this summer kernel added additional driver for fully
featured L2(ib ethernet driver), before that it was quite painful to
do any possible failover using ipoib.

>
>> I don't know if i support for rsocket that is experimental/untested
>> and IPoIB is a stable workaroud or what else.
>
> IPoIB is much more used and pretty stable, while rsockets is new with limited testing. That said, more people using it will help Sean improve it.
>
> Ideally, we would like support for zero-copy and reduced CPU usage (via OS-bypass) and with more interconnects than just InfiniBand. :-)
>
>> And is a dual controller needed on each OSD node? Ceph is able to
>> handle OSD network failures? This is really important to know. It
>> change the whole network topology.
>
> I will let others answer this.
>
> Scott--
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: flashcache

[Gandalf Corvotempesta]

2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
> IPoIB appears as a traditional Ethernet device to Linux and can be used as such. Ceph has no idea that it is not Ethernet.

Ok. Now it's clear.
AFAIK, a standard SDR IB card should give use more speed than GbE
(less overhead?) and lower latency, I think.

Re: flashcache

[Atchley, Scott]

On Jan 17, 2013, at 10:07 AM, Andrey Korolyov <andrey@xdel.ru> wrote:

> On Thu, Jan 17, 2013 at 7:00 PM, Atchley, Scott <atchleyes@ornl.gov> wrote:
>> On Jan 17, 2013, at 9:48 AM, Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:
>> 
>>> 2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
>>>> IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE.
>>> 
>>> Which kind of tuning? Do you have a paper about this?
>> 
>> No, I followed the Mellanox tuning guide and modified their interrupt affinity scripts.
> 
> Did you tried to bind interrupts only to core to which QPI link
> belongs in reality and measure difference with spread-over-all-cores
> binding?

This is the modified part. I bound the mlx4-async handler to core 0 and the mlx4-ib-1-0 handle to core 1 for our machines.

>>> But, actually, is possible to use ceph with IPoIB in a stable way or
>>> is this experimental ?
>> 
>> IPoIB appears as a traditional Ethernet device to Linux and can be used as such.
> 
> Not exactly, this summer kernel added additional driver for fully
> featured L2(ib ethernet driver), before that it was quite painful to
> do any possible failover using ipoib.

I assume it is now an EoIB driver. Does it replace the IPoIB driver?

>>> I don't know if i support for rsocket that is experimental/untested
>>> and IPoIB is a stable workaroud or what else.
>> 
>> IPoIB is much more used and pretty stable, while rsockets is new with limited testing. That said, more people using it will help Sean improve it.
>> 
>> Ideally, we would like support for zero-copy and reduced CPU usage (via OS-bypass) and with more interconnects than just InfiniBand. :-)
>> 
>>> And is a dual controller needed on each OSD node? Ceph is able to
>>> handle OSD network failures? This is really important to know. It
>>> change the whole network topology.
>> 
>> I will let others answer this.
>> 
>> Scott--
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: flashcache

[Atchley, Scott]

On Jan 17, 2013, at 10:14 AM, Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:

> 2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
>> IPoIB appears as a traditional Ethernet device to Linux and can be used as such. Ceph has no idea that it is not Ethernet.
> 
> Ok. Now it's clear.
> AFAIK, a standard SDR IB card should give use more speed than GbE
> (less overhead?) and lower latency, I think.

Yes. It should get close to 1 GB/s where 1GbE is limited to about 125 MB/s. Lower latency? Probably since most Ethernet drivers set interrupt coalescing by default. Intel e1000 driver, for example, have a cluster mode that reduces (or turns off) interrupt coalescing. I don't know if ceph is latency sensitive or not.

Scott

Re: flashcache

[Gandalf Corvotempesta]

2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
> Yes. It should get close to 1 GB/s where 1GbE is limited to about 125 MB/s. Lower latency? Probably since most Ethernet drivers set interrupt coalescing by default. Intel e1000 driver, for example, have a cluster mode that reduces (or turns off) interrupt coalescing. I don't know if ceph is latency sensitive or not.

Sorry, I meant 10GbE.

Re: flashcache

[Atchley, Scott]

On Jan 17, 2013, at 11:01 AM, Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:

> 2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
>> Yes. It should get close to 1 GB/s where 1GbE is limited to about 125 MB/s. Lower latency? Probably since most Ethernet drivers set interrupt coalescing by default. Intel e1000 driver, for example, have a cluster mode that reduces (or turns off) interrupt coalescing. I don't know if ceph is latency sensitive or not.
> 
> Sorry, I meant 10GbE.

10GbE should get close to 1.2 GB/s compared to 1 GB/s for IB SDR. Latency again depends on the Ethernet driver.

Scott

Re: flashcache

[Gandalf Corvotempesta]

2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
> 10GbE should get close to 1.2 GB/s compared to 1 GB/s for IB SDR. Latency again depends on the Ethernet driver.

10GbE faster than IB SDR? Really ?

Re: flashcache

[Stefan Priebe]

Hi,

Am 17.01.2013 17:12, schrieb Atchley, Scott:
> On Jan 17, 2013, at 11:01 AM, Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:
>
>> 2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
>>> Yes. It should get close to 1 GB/s where 1GbE is limited to about 125 MB/s. Lower latency? Probably since most Ethernet drivers set interrupt coalescing by default. Intel e1000 driver, for example, have a cluster mode that reduces (or turns off) interrupt coalescing. I don't know if ceph is latency sensitive or not.
>>
>> Sorry, I meant 10GbE.
>
> 10GbE should get close to 1.2 GB/s compared to 1 GB/s for IB SDR. Latency again depends on the Ethernet driver.

We're using bonded active/active 2x10GbE with Intel ixgbe and i'm able 
to get 2.3GB/s.

Not sure how to measure latency effectively.

Stefan

Re: flashcache

[Gandalf Corvotempesta]

2013/1/17 Stefan Priebe <s.priebe@profihost.ag>:
> We're using bonded active/active 2x10GbE with Intel ixgbe and i'm able to
> get 2.3GB/s.

Which kind of switch do you use?

Re: flashcache

[Stefan Priebe]

Hi,

Am 17.01.2013 17:21, schrieb Gandalf Corvotempesta:
> 2013/1/17 Stefan Priebe <s.priebe@profihost.ag>:
>> We're using bonded active/active 2x10GbE with Intel ixgbe and i'm able to
>> get 2.3GB/s.
>
> Which kind of switch do you use?

HP 5920

Stefan

Re: flashcache

[Andrey Korolyov]

On Thu, Jan 17, 2013 at 7:47 PM, Atchley, Scott <atchleyes@ornl.gov> wrote:
> On Jan 17, 2013, at 10:07 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>
>> On Thu, Jan 17, 2013 at 7:00 PM, Atchley, Scott <atchleyes@ornl.gov> wrote:
>>> On Jan 17, 2013, at 9:48 AM, Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:
>>>
>>>> 2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
>>>>> IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE.
>>>>
>>>> Which kind of tuning? Do you have a paper about this?
>>>
>>> No, I followed the Mellanox tuning guide and modified their interrupt affinity scripts.
>>
>> Did you tried to bind interrupts only to core to which QPI link
>> belongs in reality and measure difference with spread-over-all-cores
>> binding?
>
> This is the modified part. I bound the mlx4-async handler to core 0 and the mlx4-ib-1-0 handle to core 1 for our machines.
>
>>>> But, actually, is possible to use ceph with IPoIB in a stable way or
>>>> is this experimental ?
>>>
>>> IPoIB appears as a traditional Ethernet device to Linux and can be used as such.
>>
>> Not exactly, this summer kernel added additional driver for fully
>> featured L2(ib ethernet driver), before that it was quite painful to
>> do any possible failover using ipoib.
>
> I assume it is now an EoIB driver. Does it replace the IPoIB driver?
>
Nope, it is upper-layer thing: https://lwn.net/Articles/509448/

>>>> I don't know if i support for rsocket that is experimental/untested
>>>> and IPoIB is a stable workaroud or what else.
>>>
>>> IPoIB is much more used and pretty stable, while rsockets is new with limited testing. That said, more people using it will help Sean improve it.
>>>
>>> Ideally, we would like support for zero-copy and reduced CPU usage (via OS-bypass) and with more interconnects than just InfiniBand. :-)
>>>
>>>> And is a dual controller needed on each OSD node? Ceph is able to
>>>> handle OSD network failures? This is really important to know. It
>>>> change the whole network topology.
>>>
>>> I will let others answer this.
>>>
>>> Scott--
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: flashcache

[Joseph Glanville]

>> I assume it is now an EoIB driver. Does it replace the IPoIB driver?
>>
> Nope, it is upper-layer thing: https://lwn.net/Articles/509448/

Aye, its effectively a NAT translation layer that strips Ethernet
headers and grafts on IPoIB headers, thus using the same wire protocol
and allowing communication from EoIB to IPoIB.

However this approach is a little dirty and has been nacked by the
netdev community so we aren't likely to see it in the mainline
kernel.. basically ever.

Joseph.


-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: flashcache

[John Nielsen]

On Jan 19, 2013, at 7:56 PM, Joseph Glanville <joseph.glanville@orionvm.com.au> wrote:

>>> I assume it is now an EoIB driver. Does it replace the IPoIB driver?
>>> 
>> Nope, it is upper-layer thing: https://lwn.net/Articles/509448/
> 
> Aye, its effectively a NAT translation layer that strips Ethernet
> headers and grafts on IPoIB headers, thus using the same wire protocol
> and allowing communication from EoIB to IPoIB.
> 
> However this approach is a little dirty and has been nacked by the
> netdev community so we aren't likely to see it in the mainline
> kernel.. basically ever.

Just to clarify:

EoIB has been around for a while (at least in the Mellanox software, not sure about mainline). It uses the mlx4_vnic module and is a true Ethernet encapsulation over InfiniBand. Unfortunately the newer Mellanox switches won't support it any more and the ones that to have entered "Limited Support." (Not to be confused with mlx4_en, which just turns a ConnectX card into a 10G Ethernet NIC.)

IPoIB is IP over InfiniBand without Ethernet (the data link layer is straight InfiniBand).

eIPoIB is (or will be, maybe) Ethernet over IP over InfiniBand. It is intended to work with both Linux bridging and regular IB switches that support IPoIB. (Allowing e.g. unmodified KVM guests on hypervisors connected to an IPoIB fabric.) Both Joseph's comments and the LWN link above are referring to eIPoIB. Last I heard (from a pretty direct source in the last couple of weeks) Mellanox is still working on this but doesn't have anything generally available yet. Here's hoping.. but the feedback on netdev was quite negative, to put it mildly.

JN

Re: flashcache

[Atchley, Scott]

On Jan 17, 2013, at 11:19 AM, Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:

> 2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
>> 10GbE should get close to 1.2 GB/s compared to 1 GB/s for IB SDR. Latency again depends on the Ethernet driver.
> 
> 10GbE faster than IB SDR? Really ?

Define faster. Throughput or latency?

Throughput, yes. You can easily measure 1.2 GB/s using many 10GbE NICs.

Latency, (mostly) no. IB SDR is about 4 us while the best TCP performance I have measured over 10GbE is about 16 us (interrupt coalescing turned off, NAGLE off, etc). Some vendors (e.g. Myricom and SolarFlare) have userspace, OS-bypass socket libraries for their NICs that get this down to 3-6 us.

Scott

Re: flashcache

[Atchley, Scott]

On Jan 22, 2013, at 4:06 PM, "Atchley, Scott" <atchleyes@ornl.gov> wrote:

> On Jan 17, 2013, at 11:19 AM, Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:
> 
>> 2013/1/17 Atchley, Scott <atchleyes@ornl.gov>:
>>> 10GbE should get close to 1.2 GB/s compared to 1 GB/s for IB SDR. Latency again depends on the Ethernet driver.
>> 
>> 10GbE faster than IB SDR? Really ?
> 
> Define faster. Throughput or latency?
> 
> Throughput, yes. You can easily measure 1.2 GB/s using many 10GbE NICs.
> 
> Latency, (mostly) no. IB SDR is about 4 us while the best TCP performance I have measured over 10GbE is about 16 us (interrupt coalescing turned off, NAGLE off, etc). Some vendors (e.g. Myricom and SolarFlare) have userspace, OS-bypass socket libraries for their NICs that get this down to 3-6 us.

Argh, this is native IB for SDR latency. Latency of sockets on top of IB SDR will be closer to 16-20 us.

Scott

Re: flashcache

[Joseph Glanville]

On 22 January 2013 10:57, John Nielsen <lists@jnielsen.net> wrote:
> On Jan 19, 2013, at 7:56 PM, Joseph Glanville <joseph.glanville@orionvm.com.au> wrote:
>
>>>> I assume it is now an EoIB driver. Does it replace the IPoIB driver?
>>>>
>>> Nope, it is upper-layer thing: https://lwn.net/Articles/509448/
>>
>> Aye, its effectively a NAT translation layer that strips Ethernet
>> headers and grafts on IPoIB headers, thus using the same wire protocol
>> and allowing communication from EoIB to IPoIB.
>>
>> However this approach is a little dirty and has been nacked by the
>> netdev community so we aren't likely to see it in the mainline
>> kernel.. basically ever.
>
> Just to clarify:
>
> EoIB has been around for a while (at least in the Mellanox software, not sure about mainline). It uses the mlx4_vnic module and is a true Ethernet encapsulation over InfiniBand. Unfortunately the newer Mellanox switches won't support it any more and the ones that to have entered "Limited Support." (Not to be confused with mlx4_en, which just turns a ConnectX card into a 10G Ethernet NIC.)
>
> IPoIB is IP over InfiniBand without Ethernet (the data link layer is straight InfiniBand).
>
> eIPoIB is (or will be, maybe) Ethernet over IP over InfiniBand. It is intended to work with both Linux bridging and regular IB switches that support IPoIB. (Allowing e.g. unmodified KVM guests on hypervisors connected to an IPoIB fabric.) Both Joseph's comments and the LWN link above are referring to eIPoIB. Last I heard (from a pretty direct source in the last couple of weeks) Mellanox is still working on this but doesn't have anything generally available yet. Here's hoping.. but the feedback on netdev was quite negative, to put it mildly.
>
> JN
>

Apolgies, I misread EoIB as eIPoIB as others were discussing IPoIB
appearing as an Ethernet device.

Personally I would like to see a pure software implementation of EoIB
ala IPoIB, using the subnet manager to manage addressing etc rather
than trying to implement a NAT style solution.
The usefulness of communicating to IPoIB devices via the same
interface as IPoIB doesn't sufficiently offset the dirtiness of NAT
IMO.

-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846