questions on networks and hardware

questions on networks and hardware

[John Nielsen]

I'm planning a Ceph deployment which will include:
	10Gbit/s public/client network
	10Gbit/s cluster network
	dedicated mon hosts (3 to start)
	dedicated storage hosts (multiple disks, one XFS and OSD per disk, 3-5 to start)
	dedicated RADOS gateway host (1 to start)

I've done some initial testing and read through most of the docs but I still have a few questions. Please respond even if you just have a suggestion or response for one of them.

If I have "cluster network" and "public network" entries under [global] or [osd], do I still need to specify "public addr" and "cluster addr" for each OSD individually?

Which network(s) should the monitor hosts be on? If both, is it valid to have more than one "mon addr" entry per mon host or is there a different way to do it?

Is it worthwhile to have 10G NIC's on the monitor hosts? (The storage hosts will each have 2x 10Gbit/s NIC's.)

I'd like to have 2x 10Gbit/s NIC's on the gateway host and maximize throughput. Any suggestions on how to best do that? I'm assuming it will talk to the OSD's on the Ceph public/client network, so does that imply a third even-more-public network for the gateway's clients?

I think this has come up before, but has anyone written up something with more details on setting up gateways? Hardware recommendations, strategies to improve caching and performance, multiple gateway setups with and without a load balancer, etc.

Thanks!

JN

RE: questions on networks and hardware

[Holcombe, Christopher]

Hi John,

I have a public/cluster network option setup in my config file.  You do not need to also specify an addr for each osd individually.  Here's an example of my working config:
[global]
        auth cluster required = none
        auth service required = none
        auth client required = none
        public network = 172.20.41.0/25
        cluster network = 172.20.41.128/25
        osd mkfs type = xfs
[osd]
        osd journal size = 1000
        filestore max sync interval = 30

[mon.a]
        host = plcephd01
        mon addr = 172.20.41.4:6789
[mon.b]
        host = plcephd03
        mon addr = 172.20.41.6:6789
[mon.c]
        host = plcephd05
        mon addr = 172.20.41.8:6789

[osd.0]
        host = plcephd01
        devs = /dev/sda3
[osd.X]
.... and so on...


-Chris
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of John Nielsen
Sent: Friday, January 18, 2013 6:35 PM
To: ceph-devel@vger.kernel.org
Subject: questions on networks and hardware

I'm planning a Ceph deployment which will include:
        10Gbit/s public/client network
        10Gbit/s cluster network
        dedicated mon hosts (3 to start)
        dedicated storage hosts (multiple disks, one XFS and OSD per disk, 3-5 to start)
        dedicated RADOS gateway host (1 to start)

I've done some initial testing and read through most of the docs but I still have a few questions. Please respond even if you just have a suggestion or response for one of them.

If I have "cluster network" and "public network" entries under [global] or [osd], do I still need to specify "public addr" and "cluster addr" for each OSD individually?

Which network(s) should the monitor hosts be on? If both, is it valid to have more than one "mon addr" entry per mon host or is there a different way to do it?

Is it worthwhile to have 10G NIC's on the monitor hosts? (The storage hosts will each have 2x 10Gbit/s NIC's.)

I'd like to have 2x 10Gbit/s NIC's on the gateway host and maximize throughput. Any suggestions on how to best do that? I'm assuming it will talk to the OSD's on the Ceph public/client network, so does that imply a third even-more-public network for the gateway's clients?

I think this has come up before, but has anyone written up something with more details on setting up gateways? Hardware recommendations, strategies to improve caching and performance, multiple gateway setups with and without a load balancer, etc.

Thanks!

JN

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

________________________________

NOTICE: This e-mail and any attachments is intended only for use by the addressee(s) named herein and may contain legally privileged, proprietary or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify me via reply email or at (800) 927-9800 and permanently delete the original copy and any copy of any e-mail, and any printout.

Re: questions on networks and hardware

[Wido den Hollander]

Hi,

On 01/19/2013 12:34 AM, John Nielsen wrote:
> I'm planning a Ceph deployment which will include:
> 	10Gbit/s public/client network
> 	10Gbit/s cluster network
> 	dedicated mon hosts (3 to start)
> 	dedicated storage hosts (multiple disks, one XFS and OSD per disk, 3-5 to start)
> 	dedicated RADOS gateway host (1 to start)
>
> I've done some initial testing and read through most of the docs but I still have a few questions. Please respond even if you just have a suggestion or response for one of them.
>
> If I have "cluster network" and "public network" entries under [global] or [osd], do I still need to specify "public addr" and "cluster addr" for each OSD individually?
>

Already answered, but no. You don't need to. The OSDs will bind to the 
available IP in that network.

> Which network(s) should the monitor hosts be on? If both, is it valid to have more than one "mon addr" entry per mon host or is there a different way to do it?
>

They should be on the "public" network since the clients also need to be 
able to reach the monitors.

> Is it worthwhile to have 10G NIC's on the monitor hosts? (The storage hosts will each have 2x 10Gbit/s NIC's.)
>

No, not really. 1Gbit should be more then enough for your monitors. 3 
monitors should also be good. No need to go for 5 or 7.

> I'd like to have 2x 10Gbit/s NIC's on the gateway host and maximize throughput. Any suggestions on how to best do that? I'm assuming it will talk to the OSD's on the Ceph public/client network, so does that imply a third even-more-public network for the gateway's clients?
>

No, there is no third network. You could trunk the 2 NICs with LACP or 
something. Since the client will open TCP connections to all the OSDs 
you will get a pretty good balancing over the available bandwith.

> I think this has come up before, but has anyone written up something with more details on setting up gateways? Hardware recommendations, strategies to improve caching and performance, multiple gateway setups with and without a load balancer, etc.

Not that I know. I'm still trying to play with RGW and Varnish in front 
of it, but haven't really taken the time yet. The goal is to offload a 
lot of the caching to Varnish and have Varnish 'ban' objects when they 
change.

You could also use Varnish as a loadbalancer. But in this case you could 
also use RR-DNS or LVS with Direct Routing, that way only incoming 
traffic goes through the loadbalancer and return traffic goes directly 
to your network's gateway.

Wido

>
> Thanks!
>
> JN
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: questions on networks and hardware

[Jeff Mitchell]

Wido den Hollander wrote:
> No, not really. 1Gbit should be more then enough for your monitors. 3
> monitors should also be good. No need to go for 5 or 7.

I have 5 monitors, across 16 different OSD-hosting machines...is that 
going to *harm* anything?

(I have had issues in my cluster when doing upgrades where my monitor 
count fell to 3, so it felt like having the extra was nice, at that point.)

Thanks,
Jeff

Re: questions on networks and hardware

[Stefan Priebe]

Hi,

Am 20.01.2013 18:16, schrieb Wido den Hollander:
>> If I have "cluster network" and "public network" entries under
>> [global] or [osd], do I still need to specify "public addr" and
>> "cluster addr" for each OSD individually?
>>
> Already answered, but no. You don't need to. The OSDs will bind to the
> available IP in that network.

What happens if there are more than one IP available in that network?

Stefan

Re: questions on networks and hardware

[Sage Weil]

On Sun, 20 Jan 2013, Stefan Priebe wrote:
> Hi,
> 
> Am 20.01.2013 18:16, schrieb Wido den Hollander:
> > > If I have "cluster network" and "public network" entries under
> > > [global] or [osd], do I still need to specify "public addr" and
> > > "cluster addr" for each OSD individually?
> > > 
> > Already answered, but no. You don't need to. The OSDs will bind to the
> > available IP in that network.
> 
> What happens if there are more than one IP available in that network?

It'll take either the first or last one returned by the kernel.. you'd 
have to check the code to know which.  If you need finer control than 
that, 'public addr' and 'cluster addr' are there :)

sage

Re: questions on networks and hardware

[Sage Weil]

On Sun, 20 Jan 2013, Jeff Mitchell wrote:
> Wido den Hollander wrote:
> > No, not really. 1Gbit should be more then enough for your monitors. 3
> > monitors should also be good. No need to go for 5 or 7.
> 
> I have 5 monitors, across 16 different OSD-hosting machines...is that going to
> *harm* anything?

The updates will be marginally more expensive, but I don't think it will 
cause any problems--not until you have 10 or more mons, I'd say.

The only thing to watch out for is the ceph-mon interfering with the 
ceph-osds.  If you have an old kernel without syncfs(2) (<2.6.39 or 
thereabouts), or if you are sharing the mon disk with an osd, there can be 
a negative interaction.

sage

> (I have had issues in my cluster when doing upgrades where my monitor count
> fell to 3, so it felt like having the extra was nice, at that point.)

5 will give you protection from 2 ceph-mon failues, which is nice.

sage

Re: questions on networks and hardware

[Gandalf Corvotempesta]

2013/1/19 John Nielsen <lists@jnielsen.net>:
> I'm planning a Ceph deployment which will include:
>         10Gbit/s public/client network
>         10Gbit/s cluster network

I'm still trying to know if a redundant cluster networks is needed or not.
Is ceph able to manage a cluster network failure from on OSD?

What I would like to know if ceph will monitor OSD from the public
side (that should be redundant, because client will use it to connect
to the cluster) or from the cluster side.

Re: questions on networks and hardware

[Gregory Farnum]

On Sunday, January 20, 2013 at 12:30 PM, Gandalf Corvotempesta wrote:
> 2013/1/19 John Nielsen <lists@jnielsen.net (mailto:lists@jnielsen.net)>:
> > I'm planning a Ceph deployment which will include:
> > 10Gbit/s public/client network
> > 10Gbit/s cluster network
> 
> 
> 
> I'm still trying to know if a redundant cluster networks is needed or not.
> Is ceph able to manage a cluster network failure from on OSD?
> 
> What I would like to know if ceph will monitor OSD from the public
> side (that should be redundant, because client will use it to connect
> to the cluster) or from the cluster side.
> 
This is a bit embarrassing, but if you're actually using two networks and the cluster network fails but the client network stays up, things behave pretty badly (the OSDs will keep insisting it's failed, while the monitor will insist it's still working). I believe there's a branch working on this problem, but I haven't been involved with it.
It's not necessary to have split networks though, no.

Does that answer your question?
-Greg

Re: questions on networks and hardware

[Gandalf Corvotempesta]

2013/1/20 Gregory Farnum <greg@inktank.com>:
> This is a bit embarrassing, but if you're actually using two networks and the cluster network fails but the client network stays up, things behave pretty badly (the OSDs will keep insisting it's failed, while the monitor will insist it's still working). I believe there's a branch working on this problem, but I haven't been involved with it.
> It's not necessary to have split networks though, no.

Ok.

> Does that answer your question?

Absolutely, but usually cluster network is faster than client network
and being forced to use two cluster network is very very expensive.

Re: questions on networks and hardware

[Gregory Farnum]

On Sunday, January 20, 2013 at 2:43 PM, Gandalf Corvotempesta wrote:
> 2013/1/20 Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>:
> > This is a bit embarrassing, but if you're actually using two networks and the cluster network fails but the client network stays up, things behave pretty badly (the OSDs will keep insisting it's failed, while the monitor will insist it's still working). I believe there's a branch working on this problem, but I haven't been involved with it.
> > It's not necessary to have split networks though, no.
>  
>  
>  
> Ok.
>  
> > Does that answer your question?
>  
> Absolutely, but usually cluster network is faster than client network
> and being forced to use two cluster network is very very expensive.

I'm not quite sure what you mean…the use of the "cluster network" and "public network" are really just intended as conveniences for people with multiple NICs on their box. There's nothing preventing you from running everything on the same network…(and more specifically, from different speed grades to different boxes, but keeping them all on the same network).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: questions on networks and hardware

[Gandalf Corvotempesta]

2013/1/21 Gregory Farnum <greg@inktank.com>:
> I'm not quite sure what you mean…the use of the "cluster network" and "public network" are really just intended as conveniences for people with multiple NICs on their box. There's nothing preventing you from running everything on the same network…(and more specifically, from different speed grades to different boxes, but keeping them all on the same network).

I mean using two cluster network and two pubic networks. 4 NICs in total.
Our cluster network will be 10GBe or Infiniband DDR/QDR, making a
fully redundany cluster network (two nics on each server, two
switches) will double our costs.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: questions on networks and hardware

[John Nielsen]

Thanks all for your responses! Some comments inline.

On Jan 20, 2013, at 10:16 AM, Wido den Hollander <wido@widodh.nl> wrote:

> On 01/19/2013 12:34 AM, John Nielsen wrote:
>> I'm planning a Ceph deployment which will include:
>> 	10Gbit/s public/client network
>> 	10Gbit/s cluster network
>> 	dedicated mon hosts (3 to start)
>> 	dedicated storage hosts (multiple disks, one XFS and OSD per disk, 3-5 to start)
>> 	dedicated RADOS gateway host (1 to start)
>> 
>> I've done some initial testing and read through most of the docs but I still have a few questions. Please respond even if you just have a suggestion or response for one of them.
>> 
>> If I have "cluster network" and "public network" entries under [global] or [osd], do I still need to specify "public addr" and "cluster addr" for each OSD individually?
> 
> Already answered, but no. You don't need to. The OSDs will bind to the available IP in that network.

Nice. That's how I hoped/assumed it would work but I have seen some configurations on the web that include both so I wanted to make sure.

>> Which network(s) should the monitor hosts be on? If both, is it valid to have more than one "mon addr" entry per mon host or is there a different way to do it?
> 
> They should be on the "public" network since the clients also need to be able to reach the monitors.

It sounds (from other followups) like there is work on adding some awareness of the cluster network to the monitor but it's not there yet. I'll stay tuned. It would be nice if the monitors and OSD's together could form a reachability map for the cluster and give the option of using the public network for the affected subset of OSD traffic in the event of a problem on the cluster network. Having the monitor(s) connected to the cluster network might be useful for that...

Then again, a human should be involved if there is a network failure anyway; manually changing the network settings in the Ceph config as a temporary workaround is already an option I suppose.

>> I'd like to have 2x 10Gbit/s NIC's on the gateway host and maximize throughput. Any suggestions on how to best do that? I'm assuming it will talk to the OSD's on the Ceph public/client network, so does that imply a third even-more-public network for the gateway's clients?
> 
> No, there is no third network. You could trunk the 2 NICs with LACP or something. Since the client will open TCP connections to all the OSDs you will get a pretty good balancing over the available bandwith.

Good suggestion, thanks. I actually just started using LACP to pair up 1Gb NIC's on a small test cluster and it's proven beneficial, even with less-than-ideal hashing from the switch.

To put my question here a different way: suppose I really really want to segregate the Ceph traffic from the HTTP traffic and I set up the IP's and routing necessary to do so (3rd network idea). Is there any reason NOT to do that?

>> Is it worthwhile to have 10G NIC's on the monitor hosts? (The storage hosts will each have 2x 10Gbit/s NIC's.)
> 
> No, not really. 1Gbit should be more then enough for your monitors. 3 monitors should also be good. No need to go for 5 or 7.

Maybe I'll have the monitors just be on the public network but use LACP with dual 1Gbit NIC's for fault tolerance (since I'll have the NIC's onboard anyway).

>> I think this has come up before, but has anyone written up something with more details on setting up gateways? Hardware recommendations, strategies to improve caching and performance, multiple gateway setups with and without a load balancer, etc.
> 
> Not that I know. I'm still trying to play with RGW and Varnish in front of it, but haven't really taken the time yet. The goal is to offload a lot of the caching to Varnish and have Varnish 'ban' objects when they change.
> 
> You could also use Varnish as a loadbalancer. But in this case you could also use RR-DNS or LVS with Direct Routing, that way only incoming traffic goes through the loadbalancer and return traffic goes directly to your network's gateway.

I definitely plan to look in to Varnish at some point, I'll be sure to follow up here if I learn anything interesting.

Thanks again,

JN

Re: questions on networks and hardware

[Wido den Hollander]

On 01/22/2013 12:15 AM, John Nielsen wrote:
> Thanks all for your responses! Some comments inline.
>
> On Jan 20, 2013, at 10:16 AM, Wido den Hollander <wido@widodh.nl> wrote:
>
>> On 01/19/2013 12:34 AM, John Nielsen wrote:
>>> I'm planning a Ceph deployment which will include:
>>> 	10Gbit/s public/client network
>>> 	10Gbit/s cluster network
>>> 	dedicated mon hosts (3 to start)
>>> 	dedicated storage hosts (multiple disks, one XFS and OSD per disk, 3-5 to start)
>>> 	dedicated RADOS gateway host (1 to start)
>>>
>>> I've done some initial testing and read through most of the docs but I still have a few questions. Please respond even if you just have a suggestion or response for one of them.
>>>
>>> If I have "cluster network" and "public network" entries under [global] or [osd], do I still need to specify "public addr" and "cluster addr" for each OSD individually?
>>
>> Already answered, but no. You don't need to. The OSDs will bind to the available IP in that network.
>
> Nice. That's how I hoped/assumed it would work but I have seen some configurations on the web that include both so I wanted to make sure.
>
>>> Which network(s) should the monitor hosts be on? If both, is it valid to have more than one "mon addr" entry per mon host or is there a different way to do it?
>>
>> They should be on the "public" network since the clients also need to be able to reach the monitors.
>
> It sounds (from other followups) like there is work on adding some awareness of the cluster network to the monitor but it's not there yet. I'll stay tuned. It would be nice if the monitors and OSD's together could form a reachability map for the cluster and give the option of using the public network for the affected subset of OSD traffic in the event of a problem on the cluster network. Having the monitor(s) connected to the cluster network might be useful for that...
>
> Then again, a human should be involved if there is a network failure anyway; manually changing the network settings in the Ceph config as a temporary workaround is already an option I suppose.
>
>>> I'd like to have 2x 10Gbit/s NIC's on the gateway host and maximize throughput. Any suggestions on how to best do that? I'm assuming it will talk to the OSD's on the Ceph public/client network, so does that imply a third even-more-public network for the gateway's clients?
>>
>> No, there is no third network. You could trunk the 2 NICs with LACP or something. Since the client will open TCP connections to all the OSDs you will get a pretty good balancing over the available bandwith.
>
> Good suggestion, thanks. I actually just started using LACP to pair up 1Gb NIC's on a small test cluster and it's proven beneficial, even with less-than-ideal hashing from the switch.
>
> To put my question here a different way: suppose I really really want to segregate the Ceph traffic from the HTTP traffic and I set up the IP's and routing necessary to do so (3rd network idea). Is there any reason NOT to do that?
>

No, no problem! The RGW gateways simply have 10G to the internet and 10G 
to the Ceph cluster.

The OSDs on their part have a "cluster network" which they use for 
internal communication.

Ceph doesn't need to know about your 3rd network.

>>> Is it worthwhile to have 10G NIC's on the monitor hosts? (The storage hosts will each have 2x 10Gbit/s NIC's.)
>>
>> No, not really. 1Gbit should be more then enough for your monitors. 3 monitors should also be good. No need to go for 5 or 7.
>
> Maybe I'll have the monitors just be on the public network but use LACP with dual 1Gbit NIC's for fault tolerance (since I'll have the NIC's onboard anyway).
>
>>> I think this has come up before, but has anyone written up something with more details on setting up gateways? Hardware recommendations, strategies to improve caching and performance, multiple gateway setups with and without a load balancer, etc.
>>
>> Not that I know. I'm still trying to play with RGW and Varnish in front of it, but haven't really taken the time yet. The goal is to offload a lot of the caching to Varnish and have Varnish 'ban' objects when they change.
>>
>> You could also use Varnish as a loadbalancer. But in this case you could also use RR-DNS or LVS with Direct Routing, that way only incoming traffic goes through the loadbalancer and return traffic goes directly to your network's gateway.
>
> I definitely plan to look in to Varnish at some point, I'll be sure to follow up here if I learn anything interesting.
>

One thing is still having multiple Varnish caches and object banning. I 
proposed something for this some time ago, some "hook" in RGW you could 
use to inform a upstream cache to "purge" something from it's cache.

That isn't there yet though.

Wido

> Thanks again,
>
> JN
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: questions on networks and hardware

[Jeff Mitchell]

Wido den Hollander wrote:
> One thing is still having multiple Varnish caches and object banning. I
> proposed something for this some time ago, some "hook" in RGW you could
> use to inform a upstream cache to "purge" something from it's cache.

Hopefully not Varnish-specific; something like the Last-Modified header 
would be good.

Also there are tricks you can do with queries; see for instance 
http://forum.nginx.org/read.php?2,1047,1052

--Jeff

Re: questions on networks and hardware

[Dan Mick]

On 01/21/2013 12:19 AM, Gandalf Corvotempesta wrote:
> 2013/1/21 Gregory Farnum <greg@inktank.com>:
>> I'm not quite sure what you mean…the use of the "cluster network" and "public network" are really just intended as conveniences for people with multiple NICs on their box. There's nothing preventing you from running everything on the same network…(and more specifically, from different speed grades to different boxes, but keeping them all on the same network).
>
> I mean using two cluster network and two pubic networks. 4 NICs in total.
> Our cluster network will be 10GBe or Infiniband DDR/QDR, making a
> fully redundany cluster network (two nics on each server, two
> switches) will double our costs.

Well, again, you needn't use two different networks for Ceph; that 
capability is there so that you *can* only use the faster gear for the 
cluster traffic if you want to limit costs but still segregate traffic. 
  If you're just after a fully-redundant network (by which I assume you 
also mean fully-redundant fabric, power, etc.), there's no reason it 
can't all be fast gear.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html