Ideas for new ceph-mgr service

Ideas for new ceph-mgr service

[John Spray]

Hi all,

We currently have an unfulfilled need for a high level
management/monitoring service that can take some of the non-essential
tasks away from the mon (like handling the volume of advisory pg
stats), and provide a place to implement new features (like
cluster-wide command and control of the new scrub stuff in cephfs).

We've had a couple of attempts in this area historically:
 * ceph-rest-api, which is a stateless HTTP wrapper around the
MonCommand interface.  All calls hit the mons directly, it's really
just a protocol converter, and it's really more RPC than REST.
 * Calamari, which has a very extensible architecture, but suffers
from being rather heavyweight, with lots of dependencies like its own
database, and requires its own separate agents running on all the Ceph
servers.

So, the idea is to create a new lightweight service (ceph-mgr) that
runs alongside the mon, and uses the existing Ceph network channels to
talk to remote hosts.  The address of this service would be published
in the OSDMap, and OSDs and other daemons would send their
non-essential stats to the mgr instead of the mon.  For HA we would
probably run a mgr alongside each mon, and use whichever mgr instance
lived with the current leader mon.

Internally, the mgr itself then has three main components:
 * The server (a Messenger), which receives telemetry from daemons
elsewhere in the system, and receives cluster map updates from the mon
 * A simple in memory store of all the structures that we receive from
the cluster (the maps, the daemon metadata, the pg stats)
 * An embedded python interpreter that hosts high level functionality
like a REST API.

The mgr embodies the interface between "C++ Ceph land" (cephx auth,
Messenger, and ::encode/::decode serialization) and "admin land"
(JSON-like structures, REST APIs, Python modules).  The reason for
doing this in one process, rather than putting the Python parts in a
separate service (like calamari) is twofold:
 * Code simplicity: avoid inventing a C++->Python network API that
re-implements things like cluster map subscription and incremental
OSDmaps.
 * Efficiency: transmit data in its native encoding, hold it in memory
in native structs, and only expose what's needed up into Python-land
at runtime.

That last part involves a bit of a trick: because Python (specifically
the CPython interpreter) is so flexible, we can do neat things like
implementing functions in C++ that have access to our native Ceph data
structures, but are callable from high level Python code.  We can also
cast our C++ structures into Python dicts directly, without an
intermediate JSON step, using a magic Formatter subclass that
generates python objects instead of serializing.  In general the
PyFormatter is still not quite as efficient as writing full blown
wrappers for C++ structures, but it's way more efficient that
serializing stuff to JSON and sending it over the network.

Most of the business logic would then be written in python.  This
would include the obvious status/health REST APIs, but potentially
also things like pool management (similar to how the Calamari API
handles these).  As well as being accessible via a REST API, the stats
that live in the mgr could also be streamed on to a full featured time
series database like influxdb, for users that want to deploy that kind
of thing.  Our service would store some very recent history, so that
folks without a full featured TSDB can still load things like the last
60s of bandwidth into a graph in their GUI, if they have a GUI that
uses our API.

I've written a small proof-of-concept service that just subscribes to
cluster maps, loads a python module that acts as an HTTP server, and
exposes the maps to the module.  It's here:
https://github.com/jcsp/ceph/tree/wip-pyfoo/src/pyfoo

I appreciate that this might not all be completely clear in text form,
probably some more detailed design and pictures will be needed in due
course, but I wanted to put this out there to get feedback.

Cheers,
John

Re: Ideas for new ceph-mgr service

[Matt Benjamin]

Hi,


----- Original Message -----
> From: "John Spray" <jspray@redhat.com>
> To: "Ceph Development" <ceph-devel@vger.kernel.org>
> Sent: Wednesday, January 13, 2016 8:13:27 AM
> Subject: Ideas for new ceph-mgr service
> 
> Hi all,
> 
> We currently have an unfulfilled need for a high level
> management/monitoring service that can take some of the non-essential
> tasks away from the mon (like handling the volume of advisory pg
> stats), and provide a place to implement new features (like
> cluster-wide command and control of the new scrub stuff in cephfs).

I (and our group as a whole) think this will be a HUGE win, it's something we've talked about conceptually for years.  Thank you sincerely for proposing and prototyping this!

> 
> We've had a couple of attempts in this area historically:
>  * ceph-rest-api, which is a stateless HTTP wrapper around the
> MonCommand interface.  All calls hit the mons directly, it's really
> just a protocol converter, and it's really more RPC than REST.
>  * Calamari, which has a very extensible architecture, but suffers
> from being rather heavyweight, with lots of dependencies like its own
> database, and requires its own separate agents running on all the Ceph
> servers.
> 
> So, the idea is to create a new lightweight service (ceph-mgr) that
> runs alongside the mon, and uses the existing Ceph network channels to
> talk to remote hosts.  The address of this service would be published
> in the OSDMap, and OSDs and other daemons would send their
> non-essential stats to the mgr instead of the mon.  For HA we would
> probably run a mgr alongside each mon, and use whichever mgr instance
> lived with the current leader mon.
> 
> Internally, the mgr itself then has three main components:
>  * The server (a Messenger), which receives telemetry from daemons
> elsewhere in the system, and receives cluster map updates from the mon
>  * A simple in memory store of all the structures that we receive from
> the cluster (the maps, the daemon metadata, the pg stats)
>  * An embedded python interpreter that hosts high level functionality
> like a REST API.
> 
> The mgr embodies the interface between "C++ Ceph land" (cephx auth,
> Messenger, and ::encode/::decode serialization) and "admin land"
> (JSON-like structures, REST APIs, Python modules).  The reason for
> doing this in one process, rather than putting the Python parts in a
> separate service (like calamari) is twofold:
>  * Code simplicity: avoid inventing a C++->Python network API that
> re-implements things like cluster map subscription and incremental
> OSDmaps.
>  * Efficiency: transmit data in its native encoding, hold it in memory
> in native structs, and only expose what's needed up into Python-land
> at runtime.

I defer to your intuition on keeping this localized in the ceph-mon process (there are obviously a ton of reaosns to do this).

I would -strongly- request that we not use a hybrid C++ & Python server as a production version of this capability.  If the proof of concept is as successful as I intuit, I think it would be highly desirable to design a scalable, native-code framework for the core management service runtime.

Any apparent advantage from flexibility of Cython interfacing is, honestly, I think, strongly outweighed by the drawbacks of supporting the hybrid interfaces, not to mention the pervasive serialization and latency properties of a Python-driven runtime model.  (That's not to say I think that Python shouldn't be used to implement routines called from a core management runtime, if you strongly prefer not to run such code out-of-process [as systems like Salt, iirc, do].)

Matt

> 
> That last part involves a bit of a trick: because Python (specifically
> the CPython interpreter) is so flexible, we can do neat things like
> implementing functions in C++ that have access to our native Ceph data
> structures, but are callable from high level Python code.  We can also
> cast our C++ structures into Python dicts directly, without an
> intermediate JSON step, using a magic Formatter subclass that
> generates python objects instead of serializing.  In general the
> PyFormatter is still not quite as efficient as writing full blown
> wrappers for C++ structures, but it's way more efficient that
> serializing stuff to JSON and sending it over the network.
> 
> Most of the business logic would then be written in python.  This
> would include the obvious status/health REST APIs, but potentially
> also things like pool management (similar to how the Calamari API
> handles these).  As well as being accessible via a REST API, the stats
> that live in the mgr could also be streamed on to a full featured time
> series database like influxdb, for users that want to deploy that kind
> of thing.  Our service would store some very recent history, so that
> folks without a full featured TSDB can still load things like the last
> 60s of bandwidth into a graph in their GUI, if they have a GUI that
> uses our API.
> 
> I've written a small proof-of-concept service that just subscribes to
> cluster maps, loads a python module that acts as an HTTP server, and
> exposes the maps to the module.  It's here:
> https://github.com/jcsp/ceph/tree/wip-pyfoo/src/pyfoo
> 
> I appreciate that this might not all be completely clear in text form,
> probably some more detailed design and pictures will be needed in due
> course, but I wanted to put this out there to get feedback.
> 
> Cheers,
> John
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

Re: Ideas for new ceph-mgr service

[Matt Benjamin]

Sorry, ignore the part about ceph-mon, I misread that sentence in the original mail.

Matt

----- Original Message -----
> From: "Matt Benjamin" <mbenjamin@redhat.com>
> To: "John Spray" <jspray@redhat.com>
> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
> Sent: Wednesday, January 13, 2016 10:53:50 AM
> Subject: Re: Ideas for new ceph-mgr service
> 
> Hi,
> 
> 
> ----- Original Message -----
> > From: "John Spray" <jspray@redhat.com>
> > To: "Ceph Development" <ceph-devel@vger.kernel.org>
> > Sent: Wednesday, January 13, 2016 8:13:27 AM
> > Subject: Ideas for new ceph-mgr service
> > 
> > Hi all,
> > 
> > We currently have an unfulfilled need for a high level
> > management/monitoring service that can take some of the non-essential
> > tasks away from the mon (like handling the volume of advisory pg
> > stats), and provide a place to implement new features (like
> > cluster-wide command and control of the new scrub stuff in cephfs).
> 
> I (and our group as a whole) think this will be a HUGE win, it's something
> we've talked about conceptually for years.  Thank you sincerely for
> proposing and prototyping this!
> 
> > 
> > We've had a couple of attempts in this area historically:
> >  * ceph-rest-api, which is a stateless HTTP wrapper around the
> > MonCommand interface.  All calls hit the mons directly, it's really
> > just a protocol converter, and it's really more RPC than REST.
> >  * Calamari, which has a very extensible architecture, but suffers
> > from being rather heavyweight, with lots of dependencies like its own
> > database, and requires its own separate agents running on all the Ceph
> > servers.
> > 
> > So, the idea is to create a new lightweight service (ceph-mgr) that
> > runs alongside the mon, and uses the existing Ceph network channels to
> > talk to remote hosts.  The address of this service would be published
> > in the OSDMap, and OSDs and other daemons would send their
> > non-essential stats to the mgr instead of the mon.  For HA we would
> > probably run a mgr alongside each mon, and use whichever mgr instance
> > lived with the current leader mon.
> > 
> > Internally, the mgr itself then has three main components:
> >  * The server (a Messenger), which receives telemetry from daemons
> > elsewhere in the system, and receives cluster map updates from the mon
> >  * A simple in memory store of all the structures that we receive from
> > the cluster (the maps, the daemon metadata, the pg stats)
> >  * An embedded python interpreter that hosts high level functionality
> > like a REST API.
> > 
> > The mgr embodies the interface between "C++ Ceph land" (cephx auth,
> > Messenger, and ::encode/::decode serialization) and "admin land"
> > (JSON-like structures, REST APIs, Python modules).  The reason for
> > doing this in one process, rather than putting the Python parts in a
> > separate service (like calamari) is twofold:
> >  * Code simplicity: avoid inventing a C++->Python network API that
> > re-implements things like cluster map subscription and incremental
> > OSDmaps.
> >  * Efficiency: transmit data in its native encoding, hold it in memory
> > in native structs, and only expose what's needed up into Python-land
> > at runtime.

> 
> I would -strongly- request that we not use a hybrid C++ & Python server as a
> production version of this capability.  If the proof of concept is as
> successful as I intuit, I think it would be highly desirable to design a
> scalable, native-code framework for the core management service runtime.
> 
> Any apparent advantage from flexibility of Cython interfacing is, honestly, I
> think, strongly outweighed by the drawbacks of supporting the hybrid
> interfaces, not to mention the pervasive serialization and latency
> properties of a Python-driven runtime model.  (That's not to say I think
> that Python shouldn't be used to implement routines called from a core
> management runtime, if you strongly prefer not to run such code
> out-of-process [as systems like Salt, iirc, do].)
> 
> Matt
> 
> > 
> > That last part involves a bit of a trick: because Python (specifically
> > the CPython interpreter) is so flexible, we can do neat things like
> > implementing functions in C++ that have access to our native Ceph data
> > structures, but are callable from high level Python code.  We can also
> > cast our C++ structures into Python dicts directly, without an
> > intermediate JSON step, using a magic Formatter subclass that
> > generates python objects instead of serializing.  In general the
> > PyFormatter is still not quite as efficient as writing full blown
> > wrappers for C++ structures, but it's way more efficient that
> > serializing stuff to JSON and sending it over the network.
> > 
> > Most of the business logic would then be written in python.  This
> > would include the obvious status/health REST APIs, but potentially
> > also things like pool management (similar to how the Calamari API
> > handles these).  As well as being accessible via a REST API, the stats
> > that live in the mgr could also be streamed on to a full featured time
> > series database like influxdb, for users that want to deploy that kind
> > of thing.  Our service would store some very recent history, so that
> > folks without a full featured TSDB can still load things like the last
> > 60s of bandwidth into a graph in their GUI, if they have a GUI that
> > uses our API.
> > 
> > I've written a small proof-of-concept service that just subscribes to
> > cluster maps, loads a python module that acts as an HTTP server, and
> > exposes the maps to the module.  It's here:
> > https://github.com/jcsp/ceph/tree/wip-pyfoo/src/pyfoo
> > 
> > I appreciate that this might not all be completely clear in text form,
> > probably some more detailed design and pictures will be needed in due
> > course, but I wanted to put this out there to get feedback.
> > 
> > Cheers,
> > John
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> --
> --
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
> 
> http://www.redhat.com/en/technologies/storage
> 
> tel.  734-707-0660
> fax.  734-769-8938
> cel.  734-216-5309
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

Re: Ideas for new ceph-mgr service

[John Spray]

On Wed, Jan 13, 2016 at 3:53 PM, Matt Benjamin <mbenjamin@redhat.com> wrote:
> Hi,
>
>
> ----- Original Message -----
>> From: "John Spray" <jspray@redhat.com>
>> To: "Ceph Development" <ceph-devel@vger.kernel.org>
>> Sent: Wednesday, January 13, 2016 8:13:27 AM
>> Subject: Ideas for new ceph-mgr service
>>
>> Hi all,
>>
>> We currently have an unfulfilled need for a high level
>> management/monitoring service that can take some of the non-essential
>> tasks away from the mon (like handling the volume of advisory pg
>> stats), and provide a place to implement new features (like
>> cluster-wide command and control of the new scrub stuff in cephfs).
>
> I (and our group as a whole) think this will be a HUGE win, it's something we've talked about conceptually for years.  Thank you sincerely for proposing and prototyping this!

Good to hear!

>>
>> We've had a couple of attempts in this area historically:
>>  * ceph-rest-api, which is a stateless HTTP wrapper around the
>> MonCommand interface.  All calls hit the mons directly, it's really
>> just a protocol converter, and it's really more RPC than REST.
>>  * Calamari, which has a very extensible architecture, but suffers
>> from being rather heavyweight, with lots of dependencies like its own
>> database, and requires its own separate agents running on all the Ceph
>> servers.
>>
>> So, the idea is to create a new lightweight service (ceph-mgr) that
>> runs alongside the mon, and uses the existing Ceph network channels to
>> talk to remote hosts.  The address of this service would be published
>> in the OSDMap, and OSDs and other daemons would send their
>> non-essential stats to the mgr instead of the mon.  For HA we would
>> probably run a mgr alongside each mon, and use whichever mgr instance
>> lived with the current leader mon.
>>
>> Internally, the mgr itself then has three main components:
>>  * The server (a Messenger), which receives telemetry from daemons
>> elsewhere in the system, and receives cluster map updates from the mon
>>  * A simple in memory store of all the structures that we receive from
>> the cluster (the maps, the daemon metadata, the pg stats)
>>  * An embedded python interpreter that hosts high level functionality
>> like a REST API.
>>
>> The mgr embodies the interface between "C++ Ceph land" (cephx auth,
>> Messenger, and ::encode/::decode serialization) and "admin land"
>> (JSON-like structures, REST APIs, Python modules).  The reason for
>> doing this in one process, rather than putting the Python parts in a
>> separate service (like calamari) is twofold:
>>  * Code simplicity: avoid inventing a C++->Python network API that
>> re-implements things like cluster map subscription and incremental
>> OSDmaps.
>>  * Efficiency: transmit data in its native encoding, hold it in memory
>> in native structs, and only expose what's needed up into Python-land
>> at runtime.
>
> I defer to your intuition on keeping this localized in the ceph-mon process (there are obviously a ton of reaosns to do this).
>
> I would -strongly- request that we not use a hybrid C++ & Python server as a production version of this capability.  If the proof of concept is as successful as I intuit, I think it would be highly desirable to design a scalable, native-code framework for the core management service runtime.
>
> Any apparent advantage from flexibility of Cython interfacing is, honestly, I think, strongly outweighed by the drawbacks of supporting the hybrid interfaces, not to mention the pervasive serialization and latency properties of a Python-driven runtime model.  (That's not to say I think that Python shouldn't be used to implement routines called from a core management runtime, if you strongly prefer not to run such code out-of-process [as systems like Salt, iirc, do].)

The idea is that the core of the service is in C++, handling talking
to the Ceph cluster, and updating all the state.  The python stuff
would be loaded and plug into a very narrow interface: actually pretty
similar to what the python code in the current CLI can do (send
commands, read status json objects), but without the serialization
overhead, and with efficient notifications for subscribing to cluster
map updates.  What I definitely *don't* want to do is write thousands
of lines of wrapper/interface code.

Latency: the python layer is stuff that will be called remotely over
HTTP from GUIs or other tools: once you're going down the whole
HTTP/JSON route, the efficiency of the language runtime tends to
become less significant, and the chances are that the code at the
other end of the collection is not super-efficient either.  The key
efficiency part IMHO is how we handle the firehose of data coming from
the cluster (in C++) rather than the individual request handling (in
Python).  I tend to think of the C++ side as the "downwards" interface
to Ceph, and the Python as the "upwards" interface to the wider world,
and this process is really straddling the line between the part where
raw execution performance is very important, and the part where ease
of integration with other code is most important.

Aside from performance, I recognise that there would be an argument to
use C++ throughout in the interests of uniformity.  For me, that's
outweighed by the how radically quicker it is to build web services in
Python, and an outward-facing REST API would be one of the biggest
parts of this service LOC-wise.

All that said... my big get-out here is that I don't think it's an
either-or choice.  We could readily define the interface to these
modules in C++ terms, and then implement the python-wrapping as a
special C++ module that happens to just load up and run python code.
That way, when there were pieces of functionality that made sense in
C++ (for example if we wanted to efficiently scan over lots of PG data
to generate some other statistics or state) we could use it, and in
other cases (for example a third party wants to plug our stats output
into their system) they can write lightweight python modules.

John

P.S. I think this was just a typo in your mail but to clarify, I'm
talking about CPython (the default python interpreter) rather than
Cython (the static compiler for python)

>
> Matt
>
>>
>> That last part involves a bit of a trick: because Python (specifically
>> the CPython interpreter) is so flexible, we can do neat things like
>> implementing functions in C++ that have access to our native Ceph data
>> structures, but are callable from high level Python code.  We can also
>> cast our C++ structures into Python dicts directly, without an
>> intermediate JSON step, using a magic Formatter subclass that
>> generates python objects instead of serializing.  In general the
>> PyFormatter is still not quite as efficient as writing full blown
>> wrappers for C++ structures, but it's way more efficient that
>> serializing stuff to JSON and sending it over the network.
>>
>> Most of the business logic would then be written in python.  This
>> would include the obvious status/health REST APIs, but potentially
>> also things like pool management (similar to how the Calamari API
>> handles these).  As well as being accessible via a REST API, the stats
>> that live in the mgr could also be streamed on to a full featured time
>> series database like influxdb, for users that want to deploy that kind
>> of thing.  Our service would store some very recent history, so that
>> folks without a full featured TSDB can still load things like the last
>> 60s of bandwidth into a graph in their GUI, if they have a GUI that
>> uses our API.
>>
>> I've written a small proof-of-concept service that just subscribes to
>> cluster maps, loads a python module that acts as an HTTP server, and
>> exposes the maps to the module.  It's here:
>> https://github.com/jcsp/ceph/tree/wip-pyfoo/src/pyfoo
>>
>> I appreciate that this might not all be completely clear in text form,
>> probably some more detailed design and pictures will be needed in due
>> course, but I wanted to put this out there to get feedback.
>>
>> Cheers,
>> John
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> --
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://www.redhat.com/en/technologies/storage
>
> tel.  734-707-0660
> fax.  734-769-8938
> cel.  734-216-5309

Re: Ideas for new ceph-mgr service

[Mark Nelson]

On 01/13/2016 11:27 AM, John Spray wrote:
> On Wed, Jan 13, 2016 at 3:53 PM, Matt Benjamin <mbenjamin@redhat.com> wrote:
>> Hi,
>>
>>
>> ----- Original Message -----
>>> From: "John Spray" <jspray@redhat.com>
>>> To: "Ceph Development" <ceph-devel@vger.kernel.org>
>>> Sent: Wednesday, January 13, 2016 8:13:27 AM
>>> Subject: Ideas for new ceph-mgr service
>>>
>>> Hi all,
>>>
>>> We currently have an unfulfilled need for a high level
>>> management/monitoring service that can take some of the non-essential
>>> tasks away from the mon (like handling the volume of advisory pg
>>> stats), and provide a place to implement new features (like
>>> cluster-wide command and control of the new scrub stuff in cephfs).
>>
>> I (and our group as a whole) think this will be a HUGE win, it's something we've talked about conceptually for years.  Thank you sincerely for proposing and prototyping this!
>
> Good to hear!
>
>>>
>>> We've had a couple of attempts in this area historically:
>>>   * ceph-rest-api, which is a stateless HTTP wrapper around the
>>> MonCommand interface.  All calls hit the mons directly, it's really
>>> just a protocol converter, and it's really more RPC than REST.
>>>   * Calamari, which has a very extensible architecture, but suffers
>>> from being rather heavyweight, with lots of dependencies like its own
>>> database, and requires its own separate agents running on all the Ceph
>>> servers.
>>>
>>> So, the idea is to create a new lightweight service (ceph-mgr) that
>>> runs alongside the mon, and uses the existing Ceph network channels to
>>> talk to remote hosts.  The address of this service would be published
>>> in the OSDMap, and OSDs and other daemons would send their
>>> non-essential stats to the mgr instead of the mon.  For HA we would
>>> probably run a mgr alongside each mon, and use whichever mgr instance
>>> lived with the current leader mon.
>>>
>>> Internally, the mgr itself then has three main components:
>>>   * The server (a Messenger), which receives telemetry from daemons
>>> elsewhere in the system, and receives cluster map updates from the mon
>>>   * A simple in memory store of all the structures that we receive from
>>> the cluster (the maps, the daemon metadata, the pg stats)
>>>   * An embedded python interpreter that hosts high level functionality
>>> like a REST API.
>>>
>>> The mgr embodies the interface between "C++ Ceph land" (cephx auth,
>>> Messenger, and ::encode/::decode serialization) and "admin land"
>>> (JSON-like structures, REST APIs, Python modules).  The reason for
>>> doing this in one process, rather than putting the Python parts in a
>>> separate service (like calamari) is twofold:
>>>   * Code simplicity: avoid inventing a C++->Python network API that
>>> re-implements things like cluster map subscription and incremental
>>> OSDmaps.
>>>   * Efficiency: transmit data in its native encoding, hold it in memory
>>> in native structs, and only expose what's needed up into Python-land
>>> at runtime.
>>
>> I defer to your intuition on keeping this localized in the ceph-mon process (there are obviously a ton of reaosns to do this).
>>
>> I would -strongly- request that we not use a hybrid C++ & Python server as a production version of this capability.  If the proof of concept is as successful as I intuit, I think it would be highly desirable to design a scalable, native-code framework for the core management service runtime.
>>
>> Any apparent advantage from flexibility of Cython interfacing is, honestly, I think, strongly outweighed by the drawbacks of supporting the hybrid interfaces, not to mention the pervasive serialization and latency properties of a Python-driven runtime model.  (That's not to say I think that Python shouldn't be used to implement routines called from a core management runtime, if you strongly prefer not to run such code out-of-process [as systems like Salt, iirc, do].)
>
> The idea is that the core of the service is in C++, handling talking
> to the Ceph cluster, and updating all the state.  The python stuff
> would be loaded and plug into a very narrow interface: actually pretty
> similar to what the python code in the current CLI can do (send
> commands, read status json objects), but without the serialization
> overhead, and with efficient notifications for subscribing to cluster
> map updates.  What I definitely *don't* want to do is write thousands
> of lines of wrapper/interface code.
>
> Latency: the python layer is stuff that will be called remotely over
> HTTP from GUIs or other tools: once you're going down the whole
> HTTP/JSON route, the efficiency of the language runtime tends to
> become less significant, and the chances are that the code at the
> other end of the collection is not super-efficient either.  The key
> efficiency part IMHO is how we handle the firehose of data coming from
> the cluster (in C++) rather than the individual request handling (in
> Python).  I tend to think of the C++ side as the "downwards" interface
> to Ceph, and the Python as the "upwards" interface to the wider world,
> and this process is really straddling the line between the part where
> raw execution performance is very important, and the part where ease
> of integration with other code is most important.
>
> Aside from performance, I recognise that there would be an argument to
> use C++ throughout in the interests of uniformity.  For me, that's
> outweighed by the how radically quicker it is to build web services in
> Python, and an outward-facing REST API would be one of the biggest
> parts of this service LOC-wise.

My gut instinct is to agree with Matt on this one, but I know the pain 
of trying to develop web services in C++ so I can't get too ornery about 
it.  If there are ways to keep it C++ throughout without too much pain 
I'd advocate that route.

>
> All that said... my big get-out here is that I don't think it's an
> either-or choice.  We could readily define the interface to these
> modules in C++ terms, and then implement the python-wrapping as a
> special C++ module that happens to just load up and run python code.
> That way, when there were pieces of functionality that made sense in
> C++ (for example if we wanted to efficiently scan over lots of PG data
> to generate some other statistics or state) we could use it, and in
> other cases (for example a third party wants to plug our stats output
> into their system) they can write lightweight python modules.

If nothing else it's a good sales pitch anyway! :)  Seriously, it does 
sound nice.  Just the hairs on the back of my neck go up a bit.

>
> John
>
> P.S. I think this was just a typo in your mail but to clarify, I'm
> talking about CPython (the default python interpreter) rather than
> Cython (the static compiler for python)
>
>>
>> Matt
>>
>>>
>>> That last part involves a bit of a trick: because Python (specifically
>>> the CPython interpreter) is so flexible, we can do neat things like
>>> implementing functions in C++ that have access to our native Ceph data
>>> structures, but are callable from high level Python code.  We can also
>>> cast our C++ structures into Python dicts directly, without an
>>> intermediate JSON step, using a magic Formatter subclass that
>>> generates python objects instead of serializing.  In general the
>>> PyFormatter is still not quite as efficient as writing full blown
>>> wrappers for C++ structures, but it's way more efficient that
>>> serializing stuff to JSON and sending it over the network.
>>>
>>> Most of the business logic would then be written in python.  This
>>> would include the obvious status/health REST APIs, but potentially
>>> also things like pool management (similar to how the Calamari API
>>> handles these).  As well as being accessible via a REST API, the stats
>>> that live in the mgr could also be streamed on to a full featured time
>>> series database like influxdb, for users that want to deploy that kind
>>> of thing.  Our service would store some very recent history, so that
>>> folks without a full featured TSDB can still load things like the last
>>> 60s of bandwidth into a graph in their GUI, if they have a GUI that
>>> uses our API.
>>>
>>> I've written a small proof-of-concept service that just subscribes to
>>> cluster maps, loads a python module that acts as an HTTP server, and
>>> exposes the maps to the module.  It's here:
>>> https://github.com/jcsp/ceph/tree/wip-pyfoo/src/pyfoo
>>>
>>> I appreciate that this might not all be completely clear in text form,
>>> probably some more detailed design and pictures will be needed in due
>>> course, but I wanted to put this out there to get feedback.
>>>
>>> Cheers,
>>> John
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> --
>> --
>> Matt Benjamin
>> Red Hat, Inc.
>> 315 West Huron Street, Suite 140A
>> Ann Arbor, Michigan 48103
>>
>> http://www.redhat.com/en/technologies/storage
>>
>> tel.  734-707-0660
>> fax.  734-769-8938
>> cel.  734-216-5309
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Ideas for new ceph-mgr service

[Adam C. Emerson]

On 13/01/2016, Mark Nelson wrote:
[snip]
> My gut instinct is to agree with Matt on this one, but I know the pain of
> trying to develop web services in C++ so I can't get too ornery about it.
> If there are ways to keep it C++ throughout without too much pain I'd
> advocate that route.
[snip]

If the main concern is the annoyance of writing web services, would it make
sense to use some widely supported binary protocol (Protocol Buffers, Flat
Buffers, Cap'n Proto, etc.) that can be implemented easily in C++ and is well
supported in Python and other languages?

If we /need/ HTTP and JSON specifically (presumably for someone hacking up
management software in a web browser's JavaScript runtime) could we have a
Python translator that just shovels messages from the binary protocol to the
HTTP JSON protocol and back?

-- 
Senior Software Engineer           Red Hat Storage, Ann Arbor, MI, US
IRC: Aemerson@{RedHat, OFTC, Freenode}
0x80F7544B90EDBFB9 E707 86BA 0C1B 62CC 152C  7C12 80F7 544B 90ED BFB9

Re: Ideas for new ceph-mgr service

[John Spray]

On Wed, Jan 13, 2016 at 9:15 PM, Adam C. Emerson <aemerson@redhat.com> wrote:
> On 13/01/2016, Mark Nelson wrote:
> [snip]
>> My gut instinct is to agree with Matt on this one, but I know the pain of
>> trying to develop web services in C++ so I can't get too ornery about it.
>> If there are ways to keep it C++ throughout without too much pain I'd
>> advocate that route.
> [snip]
>
> If the main concern is the annoyance of writing web services, would it make
> sense to use some widely supported binary protocol (Protocol Buffers, Flat
> Buffers, Cap'n Proto, etc.) that can be implemented easily in C++ and is well
> supported in Python and other languages?

When writing our upward-facing interfaces, the #1 goal is low friction
interoperability.  REST is essentially the closest thing we have to a
language-agnostic lingua franca.

To pick up on the examples you mention, Protobuf and Flat buffers are
both just serialization formats, not protocols as such.  Cap'n Proto
is an RPC framework, and describes itself as beta, and its RPC as
"particularly experimental".  I've spent a fair bit of time in this
space, and there really isn't a more popular or widely supported
protocol for services like this than vanilla REST+JSON.  Avoiding the
need for client libraries is key to the usefulness of it, as is the
familiarity and comfort level of third parties we might like to
interact with our API.

Clearly binary encodings and RPC libraries have a reason to exist:
they are especially useful within systems where both ends are
controlled by the same developers, and performance is important.  I'm
a big fan of protobuf, I just don't think it's a good fit here.

> If we /need/ HTTP and JSON specifically (presumably for someone hacking up
> management software in a web browser's JavaScript runtime) could we have a
> Python translator that just shovels messages from the binary protocol to the
> HTTP JSON protocol and back?

You could, but the resulting HTTP service would not be RESTful.  You
can have a "REST API" with an endpoint per RPC call, that only accepts
POSTs (that's what ceph-rest-api is), but it's not really a REST API
(try GET'ing a list of OSDs with ceph-rest-api: it's just not the
model it implements).

More broadly than the questions of REST vs RPC, or JSON vs protobuf, I
can imagine all kinds of modules someone might drop into this new
service:
 * Emit SNMP
 * Emit graphite, or influxdb
 * Run a simple GUI locally in process
 * Fuse the Ceph state with something from another API, like a
hardware status, and serve the result.
 * Talk to vendor X's custom protocol for reporting status

Most of those things would be natural fits for a language like python.
Remember that we're also going to be part of an ecosystem of tools,
including installers, things akin to ceph-deploy, openstack services,
etc etc -- a suitably high level language will make the outward-facing
parts of the code that much more accessible to our non-C++ colleagues.
Using a language that's so widely understood by people doing
integration work is a big win, pragmatically.

In a way I'm seeking to sidestep the question of the actual protocol:
rather than trying to come up with an all-singing-all-dancing high
performance procotol that is also very easy to consume (there's no
such thing!) I want to create a simple interface from C++ to Python,
so that anyone can readily write a Python module to generate whatever
output or interface they needed between Ceph and their wider
environment.

I see implementing a REST API in Python as the cheap, low risk option.
I'm not arguing that it's so superior that we should invest lots of
work in it, I'm arguing that it's appealing because of its simplicity
and convenience[1], and if we need to come back with something native
later, we could do that without upsetting the design of the service as
a whole.

John

Re: Ideas for new ceph-mgr service

[Brad Hubbard]

----- Original Message -----
> From: "John Spray" <jspray@redhat.com>
> To: "Mark Nelson" <mnelson@redhat.com>, "John Spray" <jspray@redhat.com>, "Matt Benjamin" <mbenjamin@redhat.com>,
> "Ceph Development" <ceph-devel@vger.kernel.org>
> Sent: Thursday, 14 January, 2016 9:02:57 AM
> Subject: Re: Ideas for new ceph-mgr service
> 
> On Wed, Jan 13, 2016 at 9:15 PM, Adam C. Emerson <aemerson@redhat.com> wrote:
> > On 13/01/2016, Mark Nelson wrote:
> > [snip]
> >> My gut instinct is to agree with Matt on this one, but I know the pain of
> >> trying to develop web services in C++ so I can't get too ornery about it.
> >> If there are ways to keep it C++ throughout without too much pain I'd
> >> advocate that route.
> > [snip]
> >
> > If the main concern is the annoyance of writing web services, would it make
> > sense to use some widely supported binary protocol (Protocol Buffers, Flat
> > Buffers, Cap'n Proto, etc.) that can be implemented easily in C++ and is
> > well
> > supported in Python and other languages?
> 
> When writing our upward-facing interfaces, the #1 goal is low friction
> interoperability.  REST is essentially the closest thing we have to a
> language-agnostic lingua franca.
> 
> To pick up on the examples you mention, Protobuf and Flat buffers are
> both just serialization formats, not protocols as such.  Cap'n Proto
> is an RPC framework, and describes itself as beta, and its RPC as
> "particularly experimental".  I've spent a fair bit of time in this
> space, and there really isn't a more popular or widely supported
> protocol for services like this than vanilla REST+JSON.  Avoiding the
> need for client libraries is key to the usefulness of it, as is the
> familiarity and comfort level of third parties we might like to
> interact with our API.

Something like this then maybe?

https://github.com/corvusoft/restbed

> 
> Clearly binary encodings and RPC libraries have a reason to exist:
> they are especially useful within systems where both ends are
> controlled by the same developers, and performance is important.  I'm
> a big fan of protobuf, I just don't think it's a good fit here.
> 
> > If we /need/ HTTP and JSON specifically (presumably for someone hacking up
> > management software in a web browser's JavaScript runtime) could we have a
> > Python translator that just shovels messages from the binary protocol to
> > the
> > HTTP JSON protocol and back?
> 
> You could, but the resulting HTTP service would not be RESTful.  You
> can have a "REST API" with an endpoint per RPC call, that only accepts
> POSTs (that's what ceph-rest-api is), but it's not really a REST API
> (try GET'ing a list of OSDs with ceph-rest-api: it's just not the
> model it implements).
> 
> More broadly than the questions of REST vs RPC, or JSON vs protobuf, I
> can imagine all kinds of modules someone might drop into this new
> service:
>  * Emit SNMP
>  * Emit graphite, or influxdb
>  * Run a simple GUI locally in process
>  * Fuse the Ceph state with something from another API, like a
> hardware status, and serve the result.
>  * Talk to vendor X's custom protocol for reporting status
> 
> Most of those things would be natural fits for a language like python.
> Remember that we're also going to be part of an ecosystem of tools,
> including installers, things akin to ceph-deploy, openstack services,
> etc etc -- a suitably high level language will make the outward-facing
> parts of the code that much more accessible to our non-C++ colleagues.
> Using a language that's so widely understood by people doing
> integration work is a big win, pragmatically.
> 
> In a way I'm seeking to sidestep the question of the actual protocol:
> rather than trying to come up with an all-singing-all-dancing high
> performance procotol that is also very easy to consume (there's no
> such thing!) I want to create a simple interface from C++ to Python,
> so that anyone can readily write a Python module to generate whatever
> output or interface they needed between Ceph and their wider
> environment.
> 
> I see implementing a REST API in Python as the cheap, low risk option.
> I'm not arguing that it's so superior that we should invest lots of
> work in it, I'm arguing that it's appealing because of its simplicity
> and convenience[1], and if we need to come back with something native
> later, we could do that without upsetting the design of the service as
> a whole.
> 
> John
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Ideas for new ceph-mgr service

[Marcus Watts]

On Wed, Jan 13, 2016 at 12:02:12PM -0600, Mark Nelson wrote:
Various wrote:
...
> >>>We currently have an unfulfilled need for a high level
> >>>management/monitoring service that can take some of the non-essential
...
> >>>So, the idea is to create a new lightweight service (ceph-mgr) that
> >>>runs alongside the mon, and uses the existing Ceph network channels to
> >>>talk to remote hosts.  The address of this service would be published
...

I think I'm going to take this in a slightly different direction.
[ ie, "blue sky" warning. ]

What was described up there (ceph-mgr) is pretty much an "admin" server.
Which is a nice idea.  But, -- well, I think what you've described is
pretty much a web server running python, except maybe you've got a weird
transport instead of http?  (and some local cached data...)

I think there are a variety of things ceph-mon does that it shouldn't,
and some things it doesn't do that maybe it shouldn't, but something should.

So some things ceph-mon doesn't do: start/stop ceph services.
Why this is useful: better integration, control of when services
start and stop, ability to migrate services between machines depending
on load or other factors.

Things ceph-mon does that it *should* do: location service/broker.
Where are things running?  This is the "fixed point" that you have
to advertise in ceph.conf, because you have to start somewhere.
Once you can locate everything elsewhere, you no longer have to run
it all in the mon.

ceph-mon provides consensus data services, and a variety of databases
on top of that.  This doesn't necessarily need to be the same set
of machines or service that does location brokering, and it might
be useful to move different databases to different sets of machines.
It may also be useful to separate one database into subsets that
get managed by different machines.  Also it should be separate from
the next bit.

I think this is the part that's actually already been discussed,
but:
ceph-mon provides an "exec" environment.  They all get "argv",and
they produce "stdout" and "stderr" output.  And there are a bunch
of canned apps.  It would be nice to de-couple argument parsing and
stdout/stderr output from the actual operation logic.  
So the seperation I would try to have is;
	1/ lowlevel "fixed" logic, as C++, providing a variety of
	simpler operations or where operation speed or consistency
	matters.
	2/ mid-level "programmarable" logic, perhaps as python
	where it can be threaded/operation speed doesn't matter.
	Eats and produces binary structures, perhaps json, or not.
	Runs with "system privileges".
	Also provide ability (*new*) - to add additional scripts,
	and to provide for periodic scheduled internal operations
	("cron".)
	Operations at this layer may not correspond exactly to user commands.
		If you want C++ -- how about a plugin shared-object
		architecture for this bit?
	3/ high-level "outer" logic.  runs on end-user user machines
	with user privileges (which might not be trustable.)  Might parse
	arg lines, might be a menu, cgi applet, might produce text output,
	xml, json, or html.  Might in some cases execute
	multiple operations in response to one user request.

The consensus and "exec" mechanisms in ceph today are kind of
conmingled, and each consensus database has its own specific list
of data.  Separating it could be icky, but I think there's a win.

One goal in this should to avoid a "firehouse" concentration
of too much stuff in any one place.  Ie, should allow for as
much parallelism as possible, and to require the least degree of
serialization or exclusion.

Anyways, those are some of my thoughts about where this could go.

					-Marcus Watts

Re: Ideas for new ceph-mgr service

[John Spray]

On Thu, Jan 14, 2016 at 4:49 AM, Marcus Watts <mwatts@redhat.com> wrote:
> On Wed, Jan 13, 2016 at 12:02:12PM -0600, Mark Nelson wrote:
> Various wrote:
> ...
>> >>>We currently have an unfulfilled need for a high level
>> >>>management/monitoring service that can take some of the non-essential
> ...
>> >>>So, the idea is to create a new lightweight service (ceph-mgr) that
>> >>>runs alongside the mon, and uses the existing Ceph network channels to
>> >>>talk to remote hosts.  The address of this service would be published
> ...
>
> I think I'm going to take this in a slightly different direction.
> [ ie, "blue sky" warning. ]

OK, duly warned!

> What was described up there (ceph-mgr) is pretty much an "admin" server.
> Which is a nice idea.  But, -- well, I think what you've described is
> pretty much a web server running python, except maybe you've got a weird
> transport instead of http?  (and some local cached data...)

It's primarily a service for holding onto live stats and metadata
about the cluster, and providing an execution environment for code
that wants to use all that data, either for monitoring or for doing
interesting cluster-wide management operations.

Then it loads a module that just happens to be a python web server.

> I think there are a variety of things ceph-mon does that it shouldn't,
> and some things it doesn't do that maybe it shouldn't, but something should.
>
> So some things ceph-mon doesn't do: start/stop ceph services.
> Why this is useful: better integration, control of when services
> start and stop, ability to migrate services between machines depending
> on load or other factors.

That's a useful thing to bring up.  Starting/stopping services is
actually a line that I actively don't want to cross: anything that
requires SSHing out into the host environment should IMHO be separate
to the things that we can do within the Ceph cluster.  Service
management is something for general purpose management tools.

> Things ceph-mon does that it *should* do: location service/broker.
> Where are things running?  This is the "fixed point" that you have
> to advertise in ceph.conf, because you have to start somewhere.
> Once you can locate everything elsewhere, you no longer have to run
> it all in the mon.
>
> ceph-mon provides consensus data services, and a variety of databases
> on top of that.  This doesn't necessarily need to be the same set
> of machines or service that does location brokering, and it might
> be useful to move different databases to different sets of machines.
> It may also be useful to separate one database into subsets that
> get managed by different machines.  Also it should be separate from
> the next bit.

These are fair points about designing a more scale-out-able mon
cluster, but I think this is a pretty long way apart from the concept
of adding a layer on top of the mon cluster for management/monitoring.

> I think this is the part that's actually already been discussed,
> but:
> ceph-mon provides an "exec" environment.  They all get "argv",and
> they produce "stdout" and "stderr" output.  And there are a bunch
> of canned apps.  It would be nice to de-couple argument parsing and
> stdout/stderr output from the actual operation logic.

Strictly speaking they actually get a dict of arguments that
ceph_argparse.py composes by interpreting an argv and mapping it to a
command from MonCommands.h.  That mapping of a command line into a
named command and set of named arguments is already happening in the
python cli.

What I envision is using the existing mon commands as the low level
primitives for friendlier interfaces layered on top.  The classic
example of this is increasing the number of PGs in a pool.  We have
the mon commands for setting pg_num, setting pgp_num, and querying
whether the pg creation is complete.  On top of that, one needs a
friendlier command that knows how to iterate through that process in
chunks of N pgs at a time.  Calamari's management stuff never got very
far, but that was one of the neat things that it was capable of.

Supporting long-running operations generally is important to me.
Spitting commands at a mon cluster is already a fully solved problem,
which could be cleaner but basically works.  The interesting (new)
thing is making higher level operations, like "scrub this OSD and
complete when we've scrubbed all those PGs" or "mark this OSD out and
complete when all the data has migrated away".  The kind of thing
where someone writing a UI would naturally put a progress bar on it.

> So the seperation I would try to have is;
>         1/ lowlevel "fixed" logic, as C++, providing a variety of
>         simpler operations or where operation speed or consistency
>         matters.
>         2/ mid-level "programmarable" logic, perhaps as python
>         where it can be threaded/operation speed doesn't matter.
>         Eats and produces binary structures, perhaps json, or not.
>         Runs with "system privileges".
>         Also provide ability (*new*) - to add additional scripts,
>         and to provide for periodic scheduled internal operations
>         ("cron".)
>         Operations at this layer may not correspond exactly to user commands.
>                 If you want C++ -- how about a plugin shared-object
>                 architecture for this bit?
>         3/ high-level "outer" logic.  runs on end-user user machines
>         with user privileges (which might not be trustable.)  Might parse
>         arg lines, might be a menu, cgi applet, might produce text output,
>         xml, json, or html.  Might in some cases execute
>         multiple operations in response to one user request.

So my vision is that in the above list:
 (1) is the existing ceph-mon
 (2) is ceph-mgr
 (3) is outside consumers of our APIs

For the moment, I'd still have the Ceph CLI pointing straight at (1)
for all the commands that it already has.  It could also learn to talk
to (2) for newly added functionality, but I wouldn't want to convert
it all over and pass commands through ceph-mgr just for the heck of
it.

I'm not about re-implementing the entire existing command set.  I'm
interested in creating a place where we can add new, higher level
functionality.  That might create a situation where a CLI would talk
to one interface for some operations, and another interface for
others, but I think that is the price of incremental improvement
rather than trying to do a big bang redesign of the existing stuff.

> The consensus and "exec" mechanisms in ceph today are kind of
> conmingled, and each consensus database has its own specific list
> of data.  Separating it could be icky, but I think there's a win.

I'm not sure they really are that commingled.  The mon is pretty
clearly structured into the generic paxos parts, and the specific
subsystem message handling.  The code could be cleaner (it would be
nice to avoid the massive if() blocks) but there is already a clear
separation between message handling and the consensus mechanisms.

> One goal in this should to avoid a "firehouse" concentration
> of too much stuff in any one place.  Ie, should allow for as
> much parallelism as possible, and to require the least degree of
> serialization or exclusion.

To some extent there will always be a (logical) firehose, even if we
shard the (physical) firehose across multiple nodes.  For example, if
we shard the nonessential PG stats updates from OSDs across multiple
ceph-mgr instances, we'll still at some point need a way to merge them
back together to do health reporting.  The first, simplest victory
would be to move them out of the mon: parallelising that their
handling more would be a further improvement.

John

Re: Ideas for new ceph-mgr service

[John Spray]

On Thu, Jan 14, 2016 at 2:33 AM, Brad Hubbard <bhubbard@redhat.com> wrote:
> ----- Original Message -----
>> From: "John Spray" <jspray@redhat.com>
>> To: "Mark Nelson" <mnelson@redhat.com>, "John Spray" <jspray@redhat.com>, "Matt Benjamin" <mbenjamin@redhat.com>,
>> "Ceph Development" <ceph-devel@vger.kernel.org>
>> Sent: Thursday, 14 January, 2016 9:02:57 AM
>> Subject: Re: Ideas for new ceph-mgr service
>>
>> On Wed, Jan 13, 2016 at 9:15 PM, Adam C. Emerson <aemerson@redhat.com> wrote:
>> > On 13/01/2016, Mark Nelson wrote:
>> > [snip]
>> >> My gut instinct is to agree with Matt on this one, but I know the pain of
>> >> trying to develop web services in C++ so I can't get too ornery about it.
>> >> If there are ways to keep it C++ throughout without too much pain I'd
>> >> advocate that route.
>> > [snip]
>> >
>> > If the main concern is the annoyance of writing web services, would it make
>> > sense to use some widely supported binary protocol (Protocol Buffers, Flat
>> > Buffers, Cap'n Proto, etc.) that can be implemented easily in C++ and is
>> > well
>> > supported in Python and other languages?
>>
>> When writing our upward-facing interfaces, the #1 goal is low friction
>> interoperability.  REST is essentially the closest thing we have to a
>> language-agnostic lingua franca.
>>
>> To pick up on the examples you mention, Protobuf and Flat buffers are
>> both just serialization formats, not protocols as such.  Cap'n Proto
>> is an RPC framework, and describes itself as beta, and its RPC as
>> "particularly experimental".  I've spent a fair bit of time in this
>> space, and there really isn't a more popular or widely supported
>> protocol for services like this than vanilla REST+JSON.  Avoiding the
>> need for client libraries is key to the usefulness of it, as is the
>> familiarity and comfort level of third parties we might like to
>> interact with our API.
>
> Something like this then maybe?
>
> https://github.com/corvusoft/restbed

Or this: https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rest.cc

It's certainly possible to write anything in C++.  But just because we
can, doesn't mean we should.  We already have a mixture of C++ and
Python in our codebase, and I'm firmly of the opinion that when
writing a web frontend we should be using Python unless there's a
super-good reason to take the hit of writing it in C++.

While I'm not deaf to the feedback that a single language solution
would be desirable, I'm also conscious that I've emailed a list of
mostly C++ developers suggesting a piece of Python code.  I could
probably email a list of Python developers with a suggestion to write
a web service in C++ and see what feedback I get there :-)

I think we'll need to remain flexible on this part.  It could be that
there's a neat solution where the serving part is C++ (a la civetweb)
with the option to write request handlers in either C++ or Python.
The relative benefits of languages might be clearer once we have some
examples, and in practice whoever writes the code will have a pretty
strong ability to influence the choice.

John

P.S. I also noticed that restbed is AGPL, it's not clear to me that
that's compatible with linking into our LGPL codebase.

>>
>> Clearly binary encodings and RPC libraries have a reason to exist:
>> they are especially useful within systems where both ends are
>> controlled by the same developers, and performance is important.  I'm
>> a big fan of protobuf, I just don't think it's a good fit here.
>>
>> > If we /need/ HTTP and JSON specifically (presumably for someone hacking up
>> > management software in a web browser's JavaScript runtime) could we have a
>> > Python translator that just shovels messages from the binary protocol to
>> > the
>> > HTTP JSON protocol and back?
>>
>> You could, but the resulting HTTP service would not be RESTful.  You
>> can have a "REST API" with an endpoint per RPC call, that only accepts
>> POSTs (that's what ceph-rest-api is), but it's not really a REST API
>> (try GET'ing a list of OSDs with ceph-rest-api: it's just not the
>> model it implements).
>>
>> More broadly than the questions of REST vs RPC, or JSON vs protobuf, I
>> can imagine all kinds of modules someone might drop into this new
>> service:
>>  * Emit SNMP
>>  * Emit graphite, or influxdb
>>  * Run a simple GUI locally in process
>>  * Fuse the Ceph state with something from another API, like a
>> hardware status, and serve the result.
>>  * Talk to vendor X's custom protocol for reporting status
>>
>> Most of those things would be natural fits for a language like python.
>> Remember that we're also going to be part of an ecosystem of tools,
>> including installers, things akin to ceph-deploy, openstack services,
>> etc etc -- a suitably high level language will make the outward-facing
>> parts of the code that much more accessible to our non-C++ colleagues.
>> Using a language that's so widely understood by people doing
>> integration work is a big win, pragmatically.
>>
>> In a way I'm seeking to sidestep the question of the actual protocol:
>> rather than trying to come up with an all-singing-all-dancing high
>> performance procotol that is also very easy to consume (there's no
>> such thing!) I want to create a simple interface from C++ to Python,
>> so that anyone can readily write a Python module to generate whatever
>> output or interface they needed between Ceph and their wider
>> environment.
>>
>> I see implementing a REST API in Python as the cheap, low risk option.
>> I'm not arguing that it's so superior that we should invest lots of
>> work in it, I'm arguing that it's appealing because of its simplicity
>> and convenience[1], and if we need to come back with something native
>> later, we could do that without upsetting the design of the service as
>> a whole.
>>
>> John
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: Ideas for new ceph-mgr service

[Matt Benjamin]

Hi John,

The language preference of the individual programmer has a lot of red-herring in it.  The question I ask is, what codebase do I want to own and maintain?  I am as biased as many, but I think the long-term value of a Python codebase in a server is limited--whether that matters depends on what happens to our product in the market.

Python provides a readable surface and it's cheaper to write than C++, but there are real issues with long-running servers written in it.  When I think of starting a new coding project of potentially substantial size, I really think it matters what kind of performance, scalability, future-facing serviceability it's going to have.

I think it's hard to argue with coding the devops/edge pieces in Python--but if a server-hosted Ceph management service inherits Python's GIL in any part of its core workflow, I think that would be a fail.

Matt

----- Original Message -----
> From: "John Spray" <jspray@redhat.com>
> To: "Brad Hubbard" <bhubbard@redhat.com>
> Cc: "Mark Nelson" <mnelson@redhat.com>, "Matt Benjamin" <mbenjamin@redhat.com>, "Ceph Development"
> <ceph-devel@vger.kernel.org>
> Sent: Thursday, January 14, 2016 6:31:18 AM
> Subject: Re: Ideas for new ceph-mgr service
> 
> On Thu, Jan 14, 2016 at 2:33 AM, Brad Hubbard <bhubbard@redhat.com> wrote:
> > ----- Original Message -----
> >> From: "John Spray" <jspray@redhat.com>
> >> To: "Mark Nelson" <mnelson@redhat.com>, "John Spray" <jspray@redhat.com>,
> >> "Matt Benjamin" <mbenjamin@redhat.com>,
> >> "Ceph Development" <ceph-devel@vger.kernel.org>
> >> Sent: Thursday, 14 January, 2016 9:02:57 AM
> >> Subject: Re: Ideas for new ceph-mgr service
> >>
> >> On Wed, Jan 13, 2016 at 9:15 PM, Adam C. Emerson <aemerson@redhat.com>
> >> wrote:
> >> > On 13/01/2016, Mark Nelson wrote:
> >> > [snip]
> >> >> My gut instinct is to agree with Matt on this one, but I know the pain
> >> >> of
> >> >> trying to develop web services in C++ so I can't get too ornery about
> >> >> it.
> >> >> If there are ways to keep it C++ throughout without too much pain I'd
> >> >> advocate that route.
> >> > [snip]
> >> >
> >> > If the main concern is the annoyance of writing web services, would it
> >> > make
> >> > sense to use some widely supported binary protocol (Protocol Buffers,
> >> > Flat
> >> > Buffers, Cap'n Proto, etc.) that can be implemented easily in C++ and is
> >> > well
> >> > supported in Python and other languages?
> >>
> >> When writing our upward-facing interfaces, the #1 goal is low friction
> >> interoperability.  REST is essentially the closest thing we have to a
> >> language-agnostic lingua franca.
> >>
> >> To pick up on the examples you mention, Protobuf and Flat buffers are
> >> both just serialization formats, not protocols as such.  Cap'n Proto
> >> is an RPC framework, and describes itself as beta, and its RPC as
> >> "particularly experimental".  I've spent a fair bit of time in this
> >> space, and there really isn't a more popular or widely supported
> >> protocol for services like this than vanilla REST+JSON.  Avoiding the
> >> need for client libraries is key to the usefulness of it, as is the
> >> familiarity and comfort level of third parties we might like to
> >> interact with our API.
> >
> > Something like this then maybe?
> >
> > https://github.com/corvusoft/restbed
> 
> Or this: https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rest.cc
> 
> It's certainly possible to write anything in C++.  But just because we
> can, doesn't mean we should.  We already have a mixture of C++ and
> Python in our codebase, and I'm firmly of the opinion that when
> writing a web frontend we should be using Python unless there's a
> super-good reason to take the hit of writing it in C++.
> 
> While I'm not deaf to the feedback that a single language solution
> would be desirable, I'm also conscious that I've emailed a list of
> mostly C++ developers suggesting a piece of Python code.  I could
> probably email a list of Python developers with a suggestion to write
> a web service in C++ and see what feedback I get there :-)
> 
> I think we'll need to remain flexible on this part.  It could be that
> there's a neat solution where the serving part is C++ (a la civetweb)
> with the option to write request handlers in either C++ or Python.
> The relative benefits of languages might be clearer once we have some
> examples, and in practice whoever writes the code will have a pretty
> strong ability to influence the choice.
> 
> John
> 
> P.S. I also noticed that restbed is AGPL, it's not clear to me that
> that's compatible with linking into our LGPL codebase.
> 
> >>
> >> Clearly binary encodings and RPC libraries have a reason to exist:
> >> they are especially useful within systems where both ends are
> >> controlled by the same developers, and performance is important.  I'm
> >> a big fan of protobuf, I just don't think it's a good fit here.
> >>
> >> > If we /need/ HTTP and JSON specifically (presumably for someone hacking
> >> > up
> >> > management software in a web browser's JavaScript runtime) could we have
> >> > a
> >> > Python translator that just shovels messages from the binary protocol to
> >> > the
> >> > HTTP JSON protocol and back?
> >>
> >> You could, but the resulting HTTP service would not be RESTful.  You
> >> can have a "REST API" with an endpoint per RPC call, that only accepts
> >> POSTs (that's what ceph-rest-api is), but it's not really a REST API
> >> (try GET'ing a list of OSDs with ceph-rest-api: it's just not the
> >> model it implements).
> >>
> >> More broadly than the questions of REST vs RPC, or JSON vs protobuf, I
> >> can imagine all kinds of modules someone might drop into this new
> >> service:
> >>  * Emit SNMP
> >>  * Emit graphite, or influxdb
> >>  * Run a simple GUI locally in process
> >>  * Fuse the Ceph state with something from another API, like a
> >> hardware status, and serve the result.
> >>  * Talk to vendor X's custom protocol for reporting status
> >>
> >> Most of those things would be natural fits for a language like python.
> >> Remember that we're also going to be part of an ecosystem of tools,
> >> including installers, things akin to ceph-deploy, openstack services,
> >> etc etc -- a suitably high level language will make the outward-facing
> >> parts of the code that much more accessible to our non-C++ colleagues.
> >> Using a language that's so widely understood by people doing
> >> integration work is a big win, pragmatically.
> >>
> >> In a way I'm seeking to sidestep the question of the actual protocol:
> >> rather than trying to come up with an all-singing-all-dancing high
> >> performance procotol that is also very easy to consume (there's no
> >> such thing!) I want to create a simple interface from C++ to Python,
> >> so that anyone can readily write a Python module to generate whatever
> >> output or interface they needed between Ceph and their wider
> >> environment.
> >>
> >> I see implementing a REST API in Python as the cheap, low risk option.
> >> I'm not arguing that it's so superior that we should invest lots of
> >> work in it, I'm arguing that it's appealing because of its simplicity
> >> and convenience[1], and if we need to come back with something native
> >> later, we could do that without upsetting the design of the service as
> >> a whole.
> >>
> >> John
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> 

-- 
-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

Re: Ideas for new ceph-mgr service

[Brad Hubbard]

----- Original Message -----
> From: "John Spray" <jspray@redhat.com>
> To: "Brad Hubbard" <bhubbard@redhat.com>
> Cc: "Mark Nelson" <mnelson@redhat.com>, "Matt Benjamin" <mbenjamin@redhat.com>, "Ceph Development"
> <ceph-devel@vger.kernel.org>
> Sent: Thursday, 14 January, 2016 9:31:18 PM
> Subject: Re: Ideas for new ceph-mgr service
> 
> On Thu, Jan 14, 2016 at 2:33 AM, Brad Hubbard <bhubbard@redhat.com> wrote:
> > ----- Original Message -----
> >> From: "John Spray" <jspray@redhat.com>
> >> To: "Mark Nelson" <mnelson@redhat.com>, "John Spray" <jspray@redhat.com>,
> >> "Matt Benjamin" <mbenjamin@redhat.com>,
> >> "Ceph Development" <ceph-devel@vger.kernel.org>
> >> Sent: Thursday, 14 January, 2016 9:02:57 AM
> >> Subject: Re: Ideas for new ceph-mgr service
> >>
> >> On Wed, Jan 13, 2016 at 9:15 PM, Adam C. Emerson <aemerson@redhat.com>
> >> wrote:
> >> > On 13/01/2016, Mark Nelson wrote:
> >> > [snip]
> >> >> My gut instinct is to agree with Matt on this one, but I know the pain
> >> >> of
> >> >> trying to develop web services in C++ so I can't get too ornery about
> >> >> it.
> >> >> If there are ways to keep it C++ throughout without too much pain I'd
> >> >> advocate that route.
> >> > [snip]
> >> >
> >> > If the main concern is the annoyance of writing web services, would it
> >> > make
> >> > sense to use some widely supported binary protocol (Protocol Buffers,
> >> > Flat
> >> > Buffers, Cap'n Proto, etc.) that can be implemented easily in C++ and is
> >> > well
> >> > supported in Python and other languages?
> >>
> >> When writing our upward-facing interfaces, the #1 goal is low friction
> >> interoperability.  REST is essentially the closest thing we have to a
> >> language-agnostic lingua franca.
> >>
> >> To pick up on the examples you mention, Protobuf and Flat buffers are
> >> both just serialization formats, not protocols as such.  Cap'n Proto
> >> is an RPC framework, and describes itself as beta, and its RPC as
> >> "particularly experimental".  I've spent a fair bit of time in this
> >> space, and there really isn't a more popular or widely supported
> >> protocol for services like this than vanilla REST+JSON.  Avoiding the
> >> need for client libraries is key to the usefulness of it, as is the
> >> familiarity and comfort level of third parties we might like to
> >> interact with our API.
> >
> > Something like this then maybe?
> >
> > https://github.com/corvusoft/restbed
> 
> Or this: https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rest.cc
> 
> It's certainly possible to write anything in C++.  But just because we
> can, doesn't mean we should.  We already have a mixture of C++ and
> Python in our codebase, and I'm firmly of the opinion that when
> writing a web frontend we should be using Python unless there's a
> super-good reason to take the hit of writing it in C++.
> 
> While I'm not deaf to the feedback that a single language solution
> would be desirable, I'm also conscious that I've emailed a list of
> mostly C++ developers suggesting a piece of Python code.  I could
> probably email a list of Python developers with a suggestion to write
> a web service in C++ and see what feedback I get there :-)

Fair point.

> 
> I think we'll need to remain flexible on this part.  It could be that
> there's a neat solution where the serving part is C++ (a la civetweb)
> with the option to write request handlers in either C++ or Python.
> The relative benefits of languages might be clearer once we have some
> examples, and in practice whoever writes the code will have a pretty
> strong ability to influence the choice.

Sure.

> 
> John
> 
> P.S. I also noticed that restbed is AGPL, it's not clear to me that
> that's compatible with linking into our LGPL codebase.

Yes. I looked at that, but I was really just offering it as an example that I
was aware of.

Cheers,
Brad

> 
> >>
> >> Clearly binary encodings and RPC libraries have a reason to exist:
> >> they are especially useful within systems where both ends are
> >> controlled by the same developers, and performance is important.  I'm
> >> a big fan of protobuf, I just don't think it's a good fit here.
> >>
> >> > If we /need/ HTTP and JSON specifically (presumably for someone hacking
> >> > up
> >> > management software in a web browser's JavaScript runtime) could we have
> >> > a
> >> > Python translator that just shovels messages from the binary protocol to
> >> > the
> >> > HTTP JSON protocol and back?
> >>
> >> You could, but the resulting HTTP service would not be RESTful.  You
> >> can have a "REST API" with an endpoint per RPC call, that only accepts
> >> POSTs (that's what ceph-rest-api is), but it's not really a REST API
> >> (try GET'ing a list of OSDs with ceph-rest-api: it's just not the
> >> model it implements).
> >>
> >> More broadly than the questions of REST vs RPC, or JSON vs protobuf, I
> >> can imagine all kinds of modules someone might drop into this new
> >> service:
> >>  * Emit SNMP
> >>  * Emit graphite, or influxdb
> >>  * Run a simple GUI locally in process
> >>  * Fuse the Ceph state with something from another API, like a
> >> hardware status, and serve the result.
> >>  * Talk to vendor X's custom protocol for reporting status
> >>
> >> Most of those things would be natural fits for a language like python.
> >> Remember that we're also going to be part of an ecosystem of tools,
> >> including installers, things akin to ceph-deploy, openstack services,
> >> etc etc -- a suitably high level language will make the outward-facing
> >> parts of the code that much more accessible to our non-C++ colleagues.
> >> Using a language that's so widely understood by people doing
> >> integration work is a big win, pragmatically.
> >>
> >> In a way I'm seeking to sidestep the question of the actual protocol:
> >> rather than trying to come up with an all-singing-all-dancing high
> >> performance procotol that is also very easy to consume (there's no
> >> such thing!) I want to create a simple interface from C++ to Python,
> >> so that anyone can readily write a Python module to generate whatever
> >> output or interface they needed between Ceph and their wider
> >> environment.
> >>
> >> I see implementing a REST API in Python as the cheap, low risk option.
> >> I'm not arguing that it's so superior that we should invest lots of
> >> work in it, I'm arguing that it's appealing because of its simplicity
> >> and convenience[1], and if we need to come back with something native
> >> later, we could do that without upsetting the design of the service as
> >> a whole.
> >>
> >> John
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
>