autoconfigured haproxy service?

autoconfigured haproxy service?

[Sage Weil]

Hi all,

Luminous features a new 'service map' that lets rgw's (and rgw nfs 
gateways and iscsi gateways and rbd mirror daemons and ...) advertise 
themselves to the cluster along with some metadata (like the addresses 
they are binding to and the services the provide).

It should be pretty straightforward to build a service that 
auto-configures haproxy based on this information so that you can deploy 
an rgw front-end that dynamically reconfigures itself when additional 
rgw's are deployed or removed.  haproxy has a facility to adjust its 
backend configuration at runtime[1].

Anybody interested in tackling this?  Setting up the load balancer in 
front of rgw is one of the more annoying pieces of getting ceph up and 
running in production and until now has been mostly treated as out of 
scope.  It would be awesome if there was an autoconfigured service that 
did it out of the box (and had all the right haproxy options set).

sage

Re: autoconfigured haproxy service?

[Sage Weil]

On Tue, 11 Jul 2017, Sage Weil wrote:
> Hi all,
> 
> Luminous features a new 'service map' that lets rgw's (and rgw nfs 
> gateways and iscsi gateways and rbd mirror daemons and ...) advertise 
> themselves to the cluster along with some metadata (like the addresses 
> they are binding to and the services the provide).
> 
> It should be pretty straightforward to build a service that 
> auto-configures haproxy based on this information so that you can deploy 
> an rgw front-end that dynamically reconfigures itself when additional 
> rgw's are deployed or removed.  haproxy has a facility to adjust its 
> backend configuration at runtime[1].
> 
> Anybody interested in tackling this?  Setting up the load balancer in 
> front of rgw is one of the more annoying pieces of getting ceph up and 
> running in production and until now has been mostly treated as out of 
> scope.  It would be awesome if there was an autoconfigured service that 
> did it out of the box (and had all the right haproxy options set).

[1] https://stackoverflow.com/questions/42678269/haproxy-dynamic-configuration

Re: autoconfigured haproxy service?

[Haomai Wang]

On Tue, Jul 11, 2017 at 11:11 PM, Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> On Tue, 11 Jul 2017, Sage Weil wrote:
>> Hi all,
>>
>> Luminous features a new 'service map' that lets rgw's (and rgw nfs
>> gateways and iscsi gateways and rbd mirror daemons and ...) advertise
>> themselves to the cluster along with some metadata (like the addresses
>> they are binding to and the services the provide).
>>
>> It should be pretty straightforward to build a service that
>> auto-configures haproxy based on this information so that you can deploy
>> an rgw front-end that dynamically reconfigures itself when additional
>> rgw's are deployed or removed.  haproxy has a facility to adjust its
>> backend configuration at runtime[1].
>>
>> Anybody interested in tackling this?  Setting up the load balancer in
>> front of rgw is one of the more annoying pieces of getting ceph up and
>> running in production and until now has been mostly treated as out of
>> scope.  It would be awesome if there was an autoconfigured service that
>> did it out of the box (and had all the right haproxy options set).
>
> [1] https://stackoverflow.com/questions/42678269/haproxy-dynamic-configuration

it looks we do more compared to before. do we need to care the
lifecycle of haproxy?  we need to manage haproxy in ceph command?

> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: autoconfigured haproxy service?

[Sage Weil]

On Tue, 11 Jul 2017, Haomai Wang wrote:
> On Tue, Jul 11, 2017 at 11:11 PM, Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > On Tue, 11 Jul 2017, Sage Weil wrote:
> >> Hi all,
> >>
> >> Luminous features a new 'service map' that lets rgw's (and rgw nfs
> >> gateways and iscsi gateways and rbd mirror daemons and ...) advertise
> >> themselves to the cluster along with some metadata (like the addresses
> >> they are binding to and the services the provide).
> >>
> >> It should be pretty straightforward to build a service that
> >> auto-configures haproxy based on this information so that you can deploy
> >> an rgw front-end that dynamically reconfigures itself when additional
> >> rgw's are deployed or removed.  haproxy has a facility to adjust its
> >> backend configuration at runtime[1].
> >>
> >> Anybody interested in tackling this?  Setting up the load balancer in
> >> front of rgw is one of the more annoying pieces of getting ceph up and
> >> running in production and until now has been mostly treated as out of
> >> scope.  It would be awesome if there was an autoconfigured service that
> >> did it out of the box (and had all the right haproxy options set).
> >
> > [1] https://stackoverflow.com/questions/42678269/haproxy-dynamic-configuration
> 
> it looks we do more compared to before. do we need to care the
> lifecycle of haproxy?  we need to manage haproxy in ceph command?

I don't think so, although not having done this much I'm not the 
expert.

My suggestion would be a new package like radosgw-haproxy-agent that 
depends on haproxy and includes a script and some systemd units etc so 
that with minimal configuration (i.e., set up ceph.conf auth key or 
something) it will wake up periodically and refresh the running haproxy's 
config.

We could add a 'ceph-deploy haproxy create ...' command to deploy it, 
along with something similar in ceph-ansible...

sage

Re: [ceph-users] autoconfigured haproxy service?

[Dan van der Ster]

On Tue, Jul 11, 2017 at 5:40 PM, Sage Weil <sage@newdream.net> wrote:
> On Tue, 11 Jul 2017, Haomai Wang wrote:
>> On Tue, Jul 11, 2017 at 11:11 PM, Sage Weil <sage@newdream.net> wrote:
>> > On Tue, 11 Jul 2017, Sage Weil wrote:
>> >> Hi all,
>> >>
>> >> Luminous features a new 'service map' that lets rgw's (and rgw nfs
>> >> gateways and iscsi gateways and rbd mirror daemons and ...) advertise
>> >> themselves to the cluster along with some metadata (like the addresses
>> >> they are binding to and the services the provide).
>> >>
>> >> It should be pretty straightforward to build a service that
>> >> auto-configures haproxy based on this information so that you can deploy
>> >> an rgw front-end that dynamically reconfigures itself when additional
>> >> rgw's are deployed or removed.  haproxy has a facility to adjust its
>> >> backend configuration at runtime[1].
>> >>
>> >> Anybody interested in tackling this?  Setting up the load balancer in
>> >> front of rgw is one of the more annoying pieces of getting ceph up and
>> >> running in production and until now has been mostly treated as out of
>> >> scope.  It would be awesome if there was an autoconfigured service that
>> >> did it out of the box (and had all the right haproxy options set).
>> >
>> > [1] https://stackoverflow.com/questions/42678269/haproxy-dynamic-configuration
>>
>> it looks we do more compared to before. do we need to care the
>> lifecycle of haproxy?  we need to manage haproxy in ceph command?
>
> I don't think so, although not having done this much I'm not the
> expert.
>
> My suggestion would be a new package like radosgw-haproxy-agent that
> depends on haproxy and includes a script and some systemd units etc so
> that with minimal configuration (i.e., set up ceph.conf auth key or
> something) it will wake up periodically and refresh the running haproxy's
> config.

So IIUC you want to periodically discover the set of radosgw backends
to fill haproxy.cfg, then reload the haproxy daemons. That would be
useful to (a) keep the set of radosgw hosts up to date and (b) to
provide a high quality haproxy configuration OOTB.

The stackoverflow link you sent is about another interesting use-case
of haproxy -- mapping different urls to different backends. Indeed we
used this in the past to migrate between ceph clusters, bucket by
bucket. And we still use it today to redirect a few very busy buckets
to an isolated set of radosgw's. I can share our config if that helps
explain how this works [2]. And maybe that config can already start a
debate about which are the best settings for an haproxy frontend (I
won't claim ours is generally correct -- happy to hear about how it
could be improved).

I don't know if the bucket mapping concept is generally applicable.
Maybe this haproxy-agent should focus on configuring a single backend
populated with the radosgw's, and leave more complex configurations up
to their admins?

(BTW, we generate this haproxy.cfg dynamically via puppet, which fills
a template by discovering the radosgw hosts in our PuppetDB).

Cheers, Dan

[2] https://gist.github.com/dvanders/857ffcf7249849cffc8d784c55b1a4d5

> We could add a 'ceph-deploy haproxy create ...' command to deploy it,
> along with something similar in ceph-ansible...
>
> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: autoconfigured haproxy service?

[Sage Weil]

On Tue, 11 Jul 2017, Dan van der Ster wrote:
> On Tue, Jul 11, 2017 at 5:40 PM, Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > On Tue, 11 Jul 2017, Haomai Wang wrote:
> >> On Tue, Jul 11, 2017 at 11:11 PM, Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> >> > On Tue, 11 Jul 2017, Sage Weil wrote:
> >> >> Hi all,
> >> >>
> >> >> Luminous features a new 'service map' that lets rgw's (and rgw nfs
> >> >> gateways and iscsi gateways and rbd mirror daemons and ...) advertise
> >> >> themselves to the cluster along with some metadata (like the addresses
> >> >> they are binding to and the services the provide).
> >> >>
> >> >> It should be pretty straightforward to build a service that
> >> >> auto-configures haproxy based on this information so that you can deploy
> >> >> an rgw front-end that dynamically reconfigures itself when additional
> >> >> rgw's are deployed or removed.  haproxy has a facility to adjust its
> >> >> backend configuration at runtime[1].
> >> >>
> >> >> Anybody interested in tackling this?  Setting up the load balancer in
> >> >> front of rgw is one of the more annoying pieces of getting ceph up and
> >> >> running in production and until now has been mostly treated as out of
> >> >> scope.  It would be awesome if there was an autoconfigured service that
> >> >> did it out of the box (and had all the right haproxy options set).
> >> >
> >> > [1] https://stackoverflow.com/questions/42678269/haproxy-dynamic-configuration
> >>
> >> it looks we do more compared to before. do we need to care the
> >> lifecycle of haproxy?  we need to manage haproxy in ceph command?
> >
> > I don't think so, although not having done this much I'm not the
> > expert.
> >
> > My suggestion would be a new package like radosgw-haproxy-agent that
> > depends on haproxy and includes a script and some systemd units etc so
> > that with minimal configuration (i.e., set up ceph.conf auth key or
> > something) it will wake up periodically and refresh the running haproxy's
> > config.
> 
> So IIUC you want to periodically discover the set of radosgw backends
> to fill haproxy.cfg, then reload the haproxy daemons. That would be
> useful to (a) keep the set of radosgw hosts up to date and (b) to
> provide a high quality haproxy configuration OOTB.

Right.

> The stackoverflow link you sent is about another interesting use-case
> of haproxy -- mapping different urls to different backends. Indeed we
> used this in the past to migrate between ceph clusters, bucket by
> bucket. And we still use it today to redirect a few very busy buckets
> to an isolated set of radosgw's. I can share our config if that helps
> explain how this works [2]. And maybe that config can already start a
> debate about which are the best settings for an haproxy frontend (I
> won't claim ours is generally correct -- happy to hear about how it
> could be improved).

Oops, yeah, I didn't look at the link carefully.  I was just verifying 
that haproxy can be reconfigured on the fly without a restart.

> I don't know if the bucket mapping concept is generally applicable.
> Maybe this haproxy-agent should focus on configuring a single backend
> populated with the radosgw's, and leave more complex configurations up
> to their admins?

Yeah.  (The dynamic remapping is interesting, though!  That could 
potentially be controlled by rgw as well to automatically isolate busy 
buckets or objects.)
 
> (BTW, we generate this haproxy.cfg dynamically via puppet, which fills
> a template by discovering the radosgw hosts in our PuppetDB).

Right.  The idea here is to remove the puppet dependency by discovering 
the rgw's directly from the cluster.

sage


> Cheers, Dan
> 
> [2] https://gist.github.com/dvanders/857ffcf7249849cffc8d784c55b1a4d5
> 
> > We could add a 'ceph-deploy haproxy create ...' command to deploy it,
> > along with something similar in ceph-ansible...
> >
> > sage
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 

Re: autoconfigured haproxy service?

[Robin H. Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 2254 bytes --]

(Trim lots of good related content).

The upcoming HAProxy 1.8 has landed further patches for improving hot
restarts/reloads of HAProxy, which previously lead to a brief gap period
when new connections were not serviced. Lots of other approaches had
been seen, including delaying TCP SYN momentarily [1]. It also notably
fixes how you needed a separate wrapper for using haproxy w/ systemd.

In the DreamHost RGW usage, I've been building additional functionality
into HAProxy via Lua, but I'm not convinced it's the best place for some
of it:
- rate-limiting by access key (with explicitly whitelisted & blacklisted
  keys), over multiple frontends [2]
- Planned: different rates for different operations/customer classes.
- Planned: issue redirects for buckets being moved (because getting
  federated metadata between existing clusters is painful) [3].
  This differs from the CERN backend-selection-by-bucket, because the
  bucket is going to be moving regions!

RGW doesn't seem like the right place for some of this functionality
either, because storing rate data in multiple places means lots of
cross-talk.

[1] https://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html
[2] http://blog.armbruster-it.de/2015/08/neo4j-and-haproxy-some-best-practices-and-tricks/
[3] http://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.html

> > I don't know if the bucket mapping concept is generally applicable.
> > Maybe this haproxy-agent should focus on configuring a single backend
> > populated with the radosgw's, and leave more complex configurations up
> > to their admins?
> Yeah.  (The dynamic remapping is interesting, though!  That could 
> potentially be controlled by rgw as well to automatically isolate busy 
> buckets or objects.)
Yes, this would fit well to populate a single HAProxy backend w/ the
available RGWs, and then that can be wrapped with whatever frontend
configuration (esp. SSL) that the admin wants.


-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2-aBrp7R+bbdUdnm+yROfE0A@public.gmane.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: autoconfigured haproxy service?

[Wido den Hollander]

> Op 11 juli 2017 om 17:03 schreef Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>:
> 
> 
> Hi all,
> 
> Luminous features a new 'service map' that lets rgw's (and rgw nfs 
> gateways and iscsi gateways and rbd mirror daemons and ...) advertise 
> themselves to the cluster along with some metadata (like the addresses 
> they are binding to and the services the provide).
> 
> It should be pretty straightforward to build a service that 
> auto-configures haproxy based on this information so that you can deploy 
> an rgw front-end that dynamically reconfigures itself when additional 
> rgw's are deployed or removed.  haproxy has a facility to adjust its 
> backend configuration at runtime[1].
> 
> Anybody interested in tackling this?  Setting up the load balancer in 
> front of rgw is one of the more annoying pieces of getting ceph up and 
> running in production and until now has been mostly treated as out of 
> scope.  It would be awesome if there was an autoconfigured service that 
> did it out of the box (and had all the right haproxy options set).
> 

Are there easy Python bindings for this? I mean querying the service map.

I'm personally a fan of running Varnish (with Hitch for SSL) in front of RGW. Some people might also prefer Traefik [0] since that also supports dynamic configs.

Wido

[0]: https://traefik.io/

> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] autoconfigured haproxy service?

[John Spray]

On Tue, Jul 11, 2017 at 8:36 PM, Wido den Hollander <wido@42on.com> wrote:
>
>> Op 11 juli 2017 om 17:03 schreef Sage Weil <sweil@redhat.com>:
>>
>>
>> Hi all,
>>
>> Luminous features a new 'service map' that lets rgw's (and rgw nfs
>> gateways and iscsi gateways and rbd mirror daemons and ...) advertise
>> themselves to the cluster along with some metadata (like the addresses
>> they are binding to and the services the provide).
>>
>> It should be pretty straightforward to build a service that
>> auto-configures haproxy based on this information so that you can deploy
>> an rgw front-end that dynamically reconfigures itself when additional
>> rgw's are deployed or removed.  haproxy has a facility to adjust its
>> backend configuration at runtime[1].
>>
>> Anybody interested in tackling this?  Setting up the load balancer in
>> front of rgw is one of the more annoying pieces of getting ceph up and
>> running in production and until now has been mostly treated as out of
>> scope.  It would be awesome if there was an autoconfigured service that
>> did it out of the box (and had all the right haproxy options set).
>>
>
> Are there easy Python bindings for this? I mean querying the service map.

This could be a good use-case for extending the new `restful` module,
to expose a read-only endpoint for reading the servicemap entries for
a particular type of service.

John

>
> I'm personally a fan of running Varnish (with Hitch for SSL) in front of RGW. Some people might also prefer Traefik [0] since that also supports dynamic configs.
>
> Wido
>
> [0]: https://traefik.io/
>
>> sage
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] autoconfigured haproxy service?

[Sage Weil]

On Tue, 11 Jul 2017, Wido den Hollander wrote:
> > Op 11 juli 2017 om 17:03 schreef Sage Weil <sweil@redhat.com>:
> > 
> > 
> > Hi all,
> > 
> > Luminous features a new 'service map' that lets rgw's (and rgw nfs 
> > gateways and iscsi gateways and rbd mirror daemons and ...) advertise 
> > themselves to the cluster along with some metadata (like the addresses 
> > they are binding to and the services the provide).
> > 
> > It should be pretty straightforward to build a service that 
> > auto-configures haproxy based on this information so that you can deploy 
> > an rgw front-end that dynamically reconfigures itself when additional 
> > rgw's are deployed or removed.  haproxy has a facility to adjust its 
> > backend configuration at runtime[1].
> > 
> > Anybody interested in tackling this?  Setting up the load balancer in 
> > front of rgw is one of the more annoying pieces of getting ceph up and 
> > running in production and until now has been mostly treated as out of 
> > scope.  It would be awesome if there was an autoconfigured service that 
> > did it out of the box (and had all the right haproxy options set).
> > 
> 
> Are there easy Python bindings for this? I mean querying the service map.

Yes and no.  There are no special librados hooks (or python wrappers) to 
get the map, but you can issue a mon_command for 'service dump' and get it 
in JSON, which works just as well for python users.

> I'm personally a fan of running Varnish (with Hitch for SSL) in front of 
> RGW. Some people might also prefer Traefik [0] since that also supports 
> dynamic configs.

How would you go about autoconfiguring varnish via the rgw service map in 
this case?

sage

Re: autoconfigured haproxy service?

[Kyle Bader]

With civetweb, radosgw needs a thread per connection, so recently I've
been using haproxy on each node running radosgw, with the backend
being only the local radosgw process. Combined with the haproxy
configuration setting "option http-server-close", haproxy will hold
open the connection to the client, but close the connection between it
and civetweb/rgw after the request finishes. Putting a pile of radosgw
nodes behind haproxy is fine and all, but doesn't scale cluster
ingress/egress beyond the pipe size of the haproxy node.

It might be a bigger win to look into doing something with IPVS / LVS,
which can do direct server return (scaling egress). Perhaps some sort
of service that pokes the service map and uses it to run commands with
ipvsadm and ensures a heartbeat service is setup via
keepalived/corosync.

To scale ingress *and* egress, we'd need to wire the service map
feature into a bgp/osfp speaker like bird/quagga in order to
inject/remove equal cost multipath routes to $virtual_ip/32 and/or
$virtual_ip6/128 with the upstream router(s).

On Tue, Jul 11, 2017 at 8:03 AM, Sage Weil <sweil@redhat.com> wrote:
> Hi all,
>
> Luminous features a new 'service map' that lets rgw's (and rgw nfs
> gateways and iscsi gateways and rbd mirror daemons and ...) advertise
> themselves to the cluster along with some metadata (like the addresses
> they are binding to and the services the provide).
>
> It should be pretty straightforward to build a service that
> auto-configures haproxy based on this information so that you can deploy
> an rgw front-end that dynamically reconfigures itself when additional
> rgw's are deployed or removed.  haproxy has a facility to adjust its
> backend configuration at runtime[1].
>
> Anybody interested in tackling this?  Setting up the load balancer in
> front of rgw is one of the more annoying pieces of getting ceph up and
> running in production and until now has been mostly treated as out of
> scope.  It would be awesome if there was an autoconfigured service that
> did it out of the box (and had all the right haproxy options set).
>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 

Kyle Bader

Re: [ceph-users] autoconfigured haproxy service?

[Matt Benjamin]

Hi Robin,

On Tue, Jul 11, 2017 at 2:53 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> (Trim lots of good related content).
>
> The upcoming HAProxy 1.8 has landed further patches for improving hot
> restarts/reloads of HAProxy, which previously lead to a brief gap period
> when new connections were not serviced. Lots of other approaches had
> been seen, including delaying TCP SYN momentarily [1]. It also notably
> fixes how you needed a separate wrapper for using haproxy w/ systemd.
>
> In the DreamHost RGW usage, I've been building additional functionality
> into HAProxy via Lua, but I'm not convinced it's the best place for some
> of it:
> - rate-limiting by access key (with explicitly whitelisted & blacklisted
>   keys), over multiple frontends [2]
> - Planned: different rates for different operations/customer classes.
> - Planned: issue redirects for buckets being moved (because getting
>   federated metadata between existing clusters is painful) [3].
>   This differs from the CERN backend-selection-by-bucket, because the
>   bucket is going to be moving regions!

This is really cool.

>
> RGW doesn't seem like the right place for some of this functionality
> either, because storing rate data in multiple places means lots of
> cross-talk.

I'm pretty skeptical about this, though.  RGW isn't becoming a load
balancer, but also needs to make scheduling decisions.  Probably
should know what the load balancer knows when it is present?

>
> [1] https://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html
> [2] http://blog.armbruster-it.de/2015/08/neo4j-and-haproxy-some-best-practices-and-tricks/
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.html
>
>> > I don't know if the bucket mapping concept is generally applicable.
>> > Maybe this haproxy-agent should focus on configuring a single backend
>> > populated with the radosgw's, and leave more complex configurations up
>> > to their admins?
>> Yeah.  (The dynamic remapping is interesting, though!  That could
>> potentially be controlled by rgw as well to automatically isolate busy
>> buckets or objects.)
> Yes, this would fit well to populate a single HAProxy backend w/ the
> available RGWs, and then that can be wrapped with whatever frontend
> configuration (esp. SSL) that the admin wants.
>
>
> --
> Robin Hugh Johnson
> Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
> E-Mail   : robbat2@gentoo.org
> GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
> GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
>


Matt

Re: autoconfigured haproxy service?

[David Turner]

[-- Attachment #1.1: Type: text/plain, Size: 2232 bytes --]

Would this be able to be modular for other front ends as well? We really
like using nginx for load balancing and it is capable of reloading the
config after modifications as well.

On Tue, Jul 11, 2017, 4:36 PM Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On Tue, 11 Jul 2017, Wido den Hollander wrote:
> > > Op 11 juli 2017 om 17:03 schreef Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>:
> > >
> > >
> > > Hi all,
> > >
> > > Luminous features a new 'service map' that lets rgw's (and rgw nfs
> > > gateways and iscsi gateways and rbd mirror daemons and ...) advertise
> > > themselves to the cluster along with some metadata (like the addresses
> > > they are binding to and the services the provide).
> > >
> > > It should be pretty straightforward to build a service that
> > > auto-configures haproxy based on this information so that you can
> deploy
> > > an rgw front-end that dynamically reconfigures itself when additional
> > > rgw's are deployed or removed.  haproxy has a facility to adjust its
> > > backend configuration at runtime[1].
> > >
> > > Anybody interested in tackling this?  Setting up the load balancer in
> > > front of rgw is one of the more annoying pieces of getting ceph up and
> > > running in production and until now has been mostly treated as out of
> > > scope.  It would be awesome if there was an autoconfigured service that
> > > did it out of the box (and had all the right haproxy options set).
> > >
> >
> > Are there easy Python bindings for this? I mean querying the service map.
>
> Yes and no.  There are no special librados hooks (or python wrappers) to
> get the map, but you can issue a mon_command for 'service dump' and get it
> in JSON, which works just as well for python users.
>
> > I'm personally a fan of running Varnish (with Hitch for SSL) in front of
> > RGW. Some people might also prefer Traefik [0] since that also supports
> > dynamic configs.
>
> How would you go about autoconfiguring varnish via the rgw service map in
> this case?
>
> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

[-- Attachment #1.2: Type: text/html, Size: 3012 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: autoconfigured haproxy service?

[Chris Jones]

[-- Attachment #1.1: Type: text/plain, Size: 1706 bytes --]

Hi Sage,

The automated tool Cepheus https://github.com/cepheus-io/cepheus does this
with ceph-chef. It's based on json data for a given environment. It uses
Chef and Ansible. If someone wanted to break out the haproxy (ADC) portion
into a package then it has a good model for HAProxy they could look at.
Originally created due to the need for our own software LB solution over a
hardware LB. It also supports keep-alived and bird (BGP).

Thanks

On Tue, Jul 11, 2017 at 11:03 AM, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> Hi all,
>
> Luminous features a new 'service map' that lets rgw's (and rgw nfs
> gateways and iscsi gateways and rbd mirror daemons and ...) advertise
> themselves to the cluster along with some metadata (like the addresses
> they are binding to and the services the provide).
>
> It should be pretty straightforward to build a service that
> auto-configures haproxy based on this information so that you can deploy
> an rgw front-end that dynamically reconfigures itself when additional
> rgw's are deployed or removed.  haproxy has a facility to adjust its
> backend configuration at runtime[1].
>
> Anybody interested in tackling this?  Setting up the load balancer in
> front of rgw is one of the more annoying pieces of getting ceph up and
> running in production and until now has been mostly treated as out of
> scope.  It would be awesome if there was an autoconfigured service that
> did it out of the box (and had all the right haproxy options set).
>
> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

[-- Attachment #1.2: Type: text/html, Size: 2413 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: autoconfigured haproxy service?

[Wido den Hollander]

> Op 11 juli 2017 om 22:35 schreef Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>:
> 
> 
> On Tue, 11 Jul 2017, Wido den Hollander wrote:
> > > Op 11 juli 2017 om 17:03 schreef Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>:
> > > 
> > > 
> > > Hi all,
> > > 
> > > Luminous features a new 'service map' that lets rgw's (and rgw nfs 
> > > gateways and iscsi gateways and rbd mirror daemons and ...) advertise 
> > > themselves to the cluster along with some metadata (like the addresses 
> > > they are binding to and the services the provide).
> > > 
> > > It should be pretty straightforward to build a service that 
> > > auto-configures haproxy based on this information so that you can deploy 
> > > an rgw front-end that dynamically reconfigures itself when additional 
> > > rgw's are deployed or removed.  haproxy has a facility to adjust its 
> > > backend configuration at runtime[1].
> > > 
> > > Anybody interested in tackling this?  Setting up the load balancer in 
> > > front of rgw is one of the more annoying pieces of getting ceph up and 
> > > running in production and until now has been mostly treated as out of 
> > > scope.  It would be awesome if there was an autoconfigured service that 
> > > did it out of the box (and had all the right haproxy options set).
> > > 
> > 
> > Are there easy Python bindings for this? I mean querying the service map.
> 
> Yes and no.  There are no special librados hooks (or python wrappers) to 
> get the map, but you can issue a mon_command for 'service dump' and get it 
> in JSON, which works just as well for python users.
> 
> > I'm personally a fan of running Varnish (with Hitch for SSL) in front of 
> > RGW. Some people might also prefer Traefik [0] since that also supports 
> > dynamic configs.
> 
> How would you go about autoconfiguring varnish via the rgw service map in 
> this case?
> 

Something like this works with RGW: https://gist.github.com/wido/d93f18810f40ecf405a5be0272821999

You see two backends configured there, but you can have more.

You can also replace that by:

include "backends.vcl"

Where the backends.vcl would then contain:

backend rgw1 {
    .host = "rgw1";
    .port = "7480";
    .connect_timeout = 5s;
    .first_byte_timeout = 15s;
    .between_bytes_timeout = 5s;
    .probe = {
        .timeout   = 30s;
        .interval  = 3s;
        .window    = 10;
        .threshold = 3;
        .request =
            "GET / HTTP/1.1"
            "Host: localhost"
            "User-Agent: Varnish-health-check"
            "Connection: close";
    }
}

backend rgw2 {
    .host = "rgw2";
    .port = "7480";
    .connect_timeout = 5s;
    .first_byte_timeout = 15s;
    .between_bytes_timeout = 5s;
    .probe = {
        .timeout   = 30s;
        .interval  = 3s;
        .window    = 10;
        .threshold = 3;
        .request =
            "GET / HTTP/1.1"
            "Host: localhost"
            "User-Agent: Varnish-health-check"
            "Connection: close";
    }
}

A very simple piece of code would generate these backends based on the servicemap in Ceph.

Wido

> sage