Statically binding ports for ceph-osd

Statically binding ports for ceph-osd

[Nick Bartos]

I'm trying to get ceph-osd's listening ports to be set statically for
firewall reasons.  I am able to get 2 of the 3 ports set statically,
however the 3rd one is still getting set dynamically.

I am using:

[osd.48]
    host = 172.16.0.13
    cluster addr = 172.16.0.13:6944
    public addr = 172.16.0.13:6945

The daemon will successfully bind to 6944 and 6945, but also binds to
6800.  What additional option do I need?  I started looking at the
code and thought "hb addr = 172.16.0.13:6946" would do it, but
specifying that option seems to have no effect (or at least does not
achieve the desired result).

Re: [PATCH] Statically binding ports for ceph-osd

[Nick Bartos]

[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]

After doing some more looking at the code, it appears that this option
is not supported.  I created a small patch (attached) which adds the
functionality.  Is there any way we could get this, or something like
this, applied upstream?  I think this is important functionality for
firewalled environments, and seems like a simple fix since all the
other services (including ones for ceph-mon and ceph-mds) already
allow you to specify a static port.


On Mon, Apr 9, 2012 at 5:27 PM, Nick Bartos <nick@pistoncloud.com> wrote:
> I'm trying to get ceph-osd's listening ports to be set statically for
> firewall reasons.  I am able to get 2 of the 3 ports set statically,
> however the 3rd one is still getting set dynamically.
>
> I am using:
>
> [osd.48]
>    host = 172.16.0.13
>    cluster addr = 172.16.0.13:6944
>    public addr = 172.16.0.13:6945
>
> The daemon will successfully bind to 6944 and 6945, but also binds to
> 6800.  What additional option do I need?  I started looking at the
> code and thought "hb addr = 172.16.0.13:6946" would do it, but
> specifying that option seems to have no effect (or at least does not
> achieve the desired result).

[-- Attachment #2: ceph-0.41-osd_hb_port.patch --]
[-- Type: application/octet-stream, Size: 1367 bytes --]

diff -U3 -r ceph-0.41.orig/src/ceph_osd.cc ceph-0.41/src/ceph_osd.cc
--- ceph-0.41.orig/src/ceph_osd.cc	2012-01-27 12:35:39.000000000 -0800
+++ ceph-0.41/src/ceph_osd.cc	2012-04-10 09:57:08.600817954 -0700
@@ -307,7 +307,11 @@
   // hb should bind to same ip as cluster_addr (if specified)
   entity_addr_t hb_addr = g_conf->cluster_addr;
   if (!hb_addr.is_blank_ip())
-    hb_addr.set_port(0);
+    // Only use osd_hb_port if it's a valid port.
+    if (g_conf->osd_hb_port > 0 && g_conf->osd_hb_port < 65536)
+        hb_addr.set_port(g_conf->osd_hb_port);
+    else
+        hb_addr.set_port(0);
   r = messenger_hbout->bind(hb_addr, getpid());
   if (r < 0)
     exit(1);
diff -U3 -r ceph-0.41.orig/src/common/config_opts.h ceph-0.41/src/common/config_opts.h
--- ceph-0.41.orig/src/common/config_opts.h	2012-01-23 10:24:35.000000000 -0800
+++ ceph-0.41/src/common/config_opts.h	2012-04-10 09:46:42.409428811 -0700
@@ -295,6 +295,7 @@
 OPTION(osd_max_notify_timeout, OPT_U32, 30) // max notify timeout in seconds
 OPTION(osd_kill_backfill_at, OPT_INT, 0)
 OPTION(osd_min_pg_log_entries, OPT_U32, 1000) // number of entries to keep in the pg log when trimming it
+OPTION(osd_hb_port, OPT_INT, 0)
 OPTION(filestore, OPT_BOOL, false)
 OPTION(filestore_max_sync_interval, OPT_DOUBLE, 5)    // seconds
 OPTION(filestore_min_sync_interval, OPT_DOUBLE, .01)  // seconds

Re: [PATCH] Statically binding ports for ceph-osd

[Greg Farnum]

Yep, you're absolutely correct. Might as well let users specify the whole address rather than just the port, though — since your patch won't apply to current upstream due to some heartbeating changes I whipped up another one which adds the "osd heartbeat addr" option. It's pushed it to master in commit 6fbac10dc68e67d1c700421f311cf5e26991d39c, but you'll want to backport (easy) or carry your change until you upgrade (and remember to change the config!). :)
Thanks for the report!
-Greg


On Tuesday, April 10, 2012 at 12:56 PM, Nick Bartos wrote:

> After doing some more looking at the code, it appears that this option
> is not supported. I created a small patch (attached) which adds the
> functionality. Is there any way we could get this, or something like
> this, applied upstream? I think this is important functionality for
> firewalled environments, and seems like a simple fix since all the
> other services (including ones for ceph-mon and ceph-mds) already
> allow you to specify a static port.
>  
>  
> On Mon, Apr 9, 2012 at 5:27 PM, Nick Bartos <nick@pistoncloud.com (mailto:nick@pistoncloud.com)> wrote:
> > I'm trying to get ceph-osd's listening ports to be set statically for
> > firewall reasons. I am able to get 2 of the 3 ports set statically,
> > however the 3rd one is still getting set dynamically.
> >  
> > I am using:
> >  
> > [osd.48]
> > host = 172.16.0.13
> > cluster addr = 172.16.0.13:6944
> > public addr = 172.16.0.13:6945
> >  
> > The daemon will successfully bind to 6944 and 6945, but also binds to
> > 6800. What additional option do I need? I started looking at the
> > code and thought "hb addr = 172.16.0.13:6946" would do it, but
> > specifying that option seems to have no effect (or at least does not
> > achieve the desired result).
>  
>  
>  
> Attachments:  
> - ceph-0.41-osd_hb_port.patch
>  



--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Statically binding ports for ceph-osd

[Nick Bartos]

Awesome, thanks so much!  Can I assume this will make it into the next
ceph stable release?  I'll probably just backport it now before we
actually start using it, so I don't have to change the config later.

On Tue, Apr 10, 2012 at 4:16 PM, Greg Farnum
<gregory.farnum@dreamhost.com> wrote:
> Yep, you're absolutely correct. Might as well let users specify the whole address rather than just the port, though — since your patch won't apply to current upstream due to some heartbeating changes I whipped up another one which adds the "osd heartbeat addr" option. It's pushed it to master in commit 6fbac10dc68e67d1c700421f311cf5e26991d39c, but you'll want to backport (easy) or carry your change until you upgrade (and remember to change the config!). :)
> Thanks for the report!
> -Greg
>
>
> On Tuesday, April 10, 2012 at 12:56 PM, Nick Bartos wrote:
>
>> After doing some more looking at the code, it appears that this option
>> is not supported. I created a small patch (attached) which adds the
>> functionality. Is there any way we could get this, or something like
>> this, applied upstream? I think this is important functionality for
>> firewalled environments, and seems like a simple fix since all the
>> other services (including ones for ceph-mon and ceph-mds) already
>> allow you to specify a static port.
>>
>>
>> On Mon, Apr 9, 2012 at 5:27 PM, Nick Bartos <nick@pistoncloud.com (mailto:nick@pistoncloud.com)> wrote:
>> > I'm trying to get ceph-osd's listening ports to be set statically for
>> > firewall reasons. I am able to get 2 of the 3 ports set statically,
>> > however the 3rd one is still getting set dynamically.
>> >
>> > I am using:
>> >
>> > [osd.48]
>> > host = 172.16.0.13
>> > cluster addr = 172.16.0.13:6944
>> > public addr = 172.16.0.13:6945
>> >
>> > The daemon will successfully bind to 6944 and 6945, but also binds to
>> > 6800. What additional option do I need? I started looking at the
>> > code and thought "hb addr = 172.16.0.13:6946" would do it, but
>> > specifying that option seems to have no effect (or at least does not
>> > achieve the desired result).
>>
>>
>>
>> Attachments:
>> - ceph-0.41-osd_hb_port.patch
>>
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Statically binding ports for ceph-osd

[Greg Farnum]

I think we've already branched off 0.45, so it'll have to wait until 0.46 unless we decide to pull it over. Sage could probably be talked into it if you ask nicely?  
-Greg


On Tuesday, April 10, 2012 at 4:45 PM, Nick Bartos wrote:

> Awesome, thanks so much! Can I assume this will make it into the next
> ceph stable release? I'll probably just backport it now before we
> actually start using it, so I don't have to change the config later.
>  
> On Tue, Apr 10, 2012 at 4:16 PM, Greg Farnum
> <gregory.farnum@dreamhost.com (mailto:gregory.farnum@dreamhost.com)> wrote:
> > Yep, you're absolutely correct. Might as well let users specify the whole address rather than just the port, though — since your patch won't apply to current upstream due to some heartbeating changes I whipped up another one which adds the "osd heartbeat addr" option. It's pushed it to master in commit 6fbac10dc68e67d1c700421f311cf5e26991d39c, but you'll want to backport (easy) or carry your change until you upgrade (and remember to change the config!). :)
> > Thanks for the report!
> > -Greg
> >  
> >  
> > On Tuesday, April 10, 2012 at 12:56 PM, Nick Bartos wrote:
> >  
> > > After doing some more looking at the code, it appears that this option
> > > is not supported. I created a small patch (attached) which adds the
> > > functionality. Is there any way we could get this, or something like
> > > this, applied upstream? I think this is important functionality for
> > > firewalled environments, and seems like a simple fix since all the
> > > other services (including ones for ceph-mon and ceph-mds) already
> > > allow you to specify a static port.
> > >  
> > >  
> > > On Mon, Apr 9, 2012 at 5:27 PM, Nick Bartos <nick@pistoncloud.com (mailto:nick@pistoncloud.com)> wrote:
> > > > I'm trying to get ceph-osd's listening ports to be set statically for
> > > > firewall reasons. I am able to get 2 of the 3 ports set statically,
> > > > however the 3rd one is still getting set dynamically.
> > > >  
> > > > I am using:
> > > >  
> > > > [osd.48]
> > > > host = 172.16.0.13
> > > > cluster addr = 172.16.0.13:6944
> > > > public addr = 172.16.0.13:6945
> > > >  
> > > > The daemon will successfully bind to 6944 and 6945, but also binds to
> > > > 6800. What additional option do I need? I started looking at the
> > > > code and thought "hb addr = 172.16.0.13:6946" would do it, but
> > > > specifying that option seems to have no effect (or at least does not
> > > > achieve the desired result).
> > >  
> > >  
> > >  
> > >  
> > >  
> > > Attachments:
> > > - ceph-0.41-osd_hb_port.patch
> >  
>  



--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Statically binding ports for ceph-osd

[Sage Weil]

On Tue, 10 Apr 2012, Nick Bartos wrote:
> Awesome, thanks so much!  Can I assume this will make it into the next
> ceph stable release?  I'll probably just backport it now before we
> actually start using it, so I don't have to change the config later.

v0.45 is out today/tomorrow, but it'll be in v0.46.

sage


> 
> On Tue, Apr 10, 2012 at 4:16 PM, Greg Farnum
> <gregory.farnum@dreamhost.com> wrote:
> > Yep, you're absolutely correct. Might as well let users specify the whole address rather than just the port, though ? since your patch won't apply to current upstream due to some heartbeating changes I whipped up another one which adds the "osd heartbeat addr" option. It's pushed it to master in commit 6fbac10dc68e67d1c700421f311cf5e26991d39c, but you'll want to backport (easy) or carry your change until you upgrade (and remember to change the config!). :)
> > Thanks for the report!
> > -Greg
> >
> >
> > On Tuesday, April 10, 2012 at 12:56 PM, Nick Bartos wrote:
> >
> >> After doing some more looking at the code, it appears that this option
> >> is not supported. I created a small patch (attached) which adds the
> >> functionality. Is there any way we could get this, or something like
> >> this, applied upstream? I think this is important functionality for
> >> firewalled environments, and seems like a simple fix since all the
> >> other services (including ones for ceph-mon and ceph-mds) already
> >> allow you to specify a static port.
> >>
> >>
> >> On Mon, Apr 9, 2012 at 5:27 PM, Nick Bartos <nick@pistoncloud.com (mailto:nick@pistoncloud.com)> wrote:
> >> > I'm trying to get ceph-osd's listening ports to be set statically for
> >> > firewall reasons. I am able to get 2 of the 3 ports set statically,
> >> > however the 3rd one is still getting set dynamically.
> >> >
> >> > I am using:
> >> >
> >> > [osd.48]
> >> > host = 172.16.0.13
> >> > cluster addr = 172.16.0.13:6944
> >> > public addr = 172.16.0.13:6945
> >> >
> >> > The daemon will successfully bind to 6944 and 6945, but also binds to
> >> > 6800. What additional option do I need? I started looking at the
> >> > code and thought "hb addr = 172.16.0.13:6946" would do it, but
> >> > specifying that option seems to have no effect (or at least does not
> >> > achieve the desired result).
> >>
> >>
> >>
> >> Attachments:
> >> - ceph-0.41-osd_hb_port.patch
> >>
> >
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: [PATCH] Statically binding ports for ceph-osd

[Nick Bartos]

Good enough for me, I'll just patch it for the short term.

Thanks!

On Tue, Apr 10, 2012 at 4:51 PM, Sage Weil <sage@newdream.net> wrote:
> On Tue, 10 Apr 2012, Nick Bartos wrote:
>> Awesome, thanks so much!  Can I assume this will make it into the next
>> ceph stable release?  I'll probably just backport it now before we
>> actually start using it, so I don't have to change the config later.
>
> v0.45 is out today/tomorrow, but it'll be in v0.46.
>
> sage
>
>
>>
>> On Tue, Apr 10, 2012 at 4:16 PM, Greg Farnum
>> <gregory.farnum@dreamhost.com> wrote:
>> > Yep, you're absolutely correct. Might as well let users specify the whole address rather than just the port, though ? since your patch won't apply to current upstream due to some heartbeating changes I whipped up another one which adds the "osd heartbeat addr" option. It's pushed it to master in commit 6fbac10dc68e67d1c700421f311cf5e26991d39c, but you'll want to backport (easy) or carry your change until you upgrade (and remember to change the config!). :)
>> > Thanks for the report!
>> > -Greg
>> >
>> >
>> > On Tuesday, April 10, 2012 at 12:56 PM, Nick Bartos wrote:
>> >
>> >> After doing some more looking at the code, it appears that this option
>> >> is not supported. I created a small patch (attached) which adds the
>> >> functionality. Is there any way we could get this, or something like
>> >> this, applied upstream? I think this is important functionality for
>> >> firewalled environments, and seems like a simple fix since all the
>> >> other services (including ones for ceph-mon and ceph-mds) already
>> >> allow you to specify a static port.
>> >>
>> >>
>> >> On Mon, Apr 9, 2012 at 5:27 PM, Nick Bartos <nick@pistoncloud.com (mailto:nick@pistoncloud.com)> wrote:
>> >> > I'm trying to get ceph-osd's listening ports to be set statically for
>> >> > firewall reasons. I am able to get 2 of the 3 ports set statically,
>> >> > however the 3rd one is still getting set dynamically.
>> >> >
>> >> > I am using:
>> >> >
>> >> > [osd.48]
>> >> > host = 172.16.0.13
>> >> > cluster addr = 172.16.0.13:6944
>> >> > public addr = 172.16.0.13:6945
>> >> >
>> >> > The daemon will successfully bind to 6944 and 6945, but also binds to
>> >> > 6800. What additional option do I need? I started looking at the
>> >> > code and thought "hb addr = 172.16.0.13:6946" would do it, but
>> >> > specifying that option seems to have no effect (or at least does not
>> >> > achieve the desired result).
>> >>
>> >>
>> >>
>> >> Attachments:
>> >> - ceph-0.41-osd_hb_port.patch
>> >>
>> >
>> >
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Statically binding ports for ceph-osd

[Greg Farnum]

You're unlikely to hit it since you're setting all addresses, but we somehow managed to introduce an error even in that small patch -- you may want to pull in commit cd4a760e9b22047fa5a45d0211ec4130809d725e as well.
-Greg

On Tuesday, April 10, 2012 at 5:13 PM, Nick Bartos wrote:
> Good enough for me, I'll just patch it for the short term.
> 
> Thanks!
> 
> On Tue, Apr 10, 2012 at 4:51 PM, Sage Weil <sage@newdream.net (mailto:sage@newdream.net)> wrote:
> > On Tue, 10 Apr 2012, Nick Bartos wrote:
> > > Awesome, thanks so much!  Can I assume this will make it into the next
> > > ceph stable release?  I'll probably just backport it now before we
> > > actually start using it, so I don't have to change the config later.
> > > 
> > 
> > 
> > v0.45 is out today/tomorrow, but it'll be in v0.46.
> > 
> > sage
> > 
> > 
> > > 
> > > On Tue, Apr 10, 2012 at 4:16 PM, Greg Farnum
> > > <gregory.farnum@dreamhost.com (mailto:gregory.farnum@dreamhost.com)> wrote:
> > > > Yep, you're absolutely correct. Might as well let users specify the whole address rather than just the port, though ? since your patch won't apply to current upstream due to some heartbeating changes I whipped up another one which adds the "osd heartbeat addr" option. It's pushed it to master in commit 6fbac10dc68e67d1c700421f311cf5e26991d39c, but you'll want to backport (easy) or carry your change until you upgrade (and remember to change the config!). :)
> > > > Thanks for the report!
> > > > -Greg
> > > > 
> > > > 
> > > > On Tuesday, April 10, 2012 at 12:56 PM, Nick Bartos wrote:
> > > > 
> > > > > After doing some more looking at the code, it appears that this option
> > > > > is not supported. I created a small patch (attached) which adds the
> > > > > functionality. Is there any way we could get this, or something like
> > > > > this, applied upstream? I think this is important functionality for
> > > > > firewalled environments, and seems like a simple fix since all the
> > > > > other services (including ones for ceph-mon and ceph-mds) already
> > > > > allow you to specify a static port.
> > > > > 
> > > > > 
> > > > > On Mon, Apr 9, 2012 at 5:27 PM, Nick Bartos <nick@pistoncloud.com (mailto:nick@pistoncloud.com)> wrote:
> > > > > > I'm trying to get ceph-osd's listening ports to be set statically for
> > > > > > firewall reasons. I am able to get 2 of the 3 ports set statically,
> > > > > > however the 3rd one is still getting set dynamically.
> > > > > > 
> > > > > > I am using:
> > > > > > 
> > > > > > [osd.48]
> > > > > > host = 172.16.0.13
> > > > > > cluster addr = 172.16.0.13:6944
> > > > > > public addr = 172.16.0.13:6945
> > > > > > 
> > > > > > The daemon will successfully bind to 6944 and 6945, but also binds to
> > > > > > 6800. What additional option do I need? I started looking at the
> > > > > > code and thought "hb addr = 172.16.0.13:6946" would do it, but
> > > > > > specifying that option seems to have no effect (or at least does not
> > > > > > achieve the desired result).
> > > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > Attachments:
> > > > > - ceph-0.41-osd_hb_port.patch
> > > > > 
> > > > 
> > > > 
> > > 
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > > the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > 
> > 
> 
> 
> 
> 

Re: [PATCH] Statically binding ports for ceph-osd

[Tommi Virtanen]

On Tue, Apr 10, 2012 at 16:16, Greg Farnum <gregory.farnum@dreamhost.com> wrote:
> Yep, you're absolutely correct. Might as well let users specify the whole address rather than just the port, though — since your patch won't apply to current upstream due to some heartbeating changes I whipped up another one which adds the "osd heartbeat addr" option. It's pushed it to master in commit 6fbac10dc68e67d1c700421f311cf5e26991d39c, but you'll want to backport (easy) or carry your change until you upgrade (and remember to change the config!). :)

We're trying to push the "public addr" and "cluster addr" to have a
"public network" and "cluster network" alternatives that don't require
per-node IP address changes.. Adding more "addr" variants without
"network" ones is probably not the most awesome idea.

Now, because this is about port allocation, it runs into problems
trying to run multiple OSDs per machine without per-osd [osd.42]
sections, so I guess that's not as relevant here. I don't have a good
suggestion as to what to do for port allocation, apart from ensuring
that the port range being used is configurable (or even easily
discoverable).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html