multiple cherrypys in ceph-mgr modules stomp on each other

multiple cherrypys in ceph-mgr modules stomp on each other

[Tim Serong]

Hi All,

We have a problem using cherrypy in mgr modules.  Specifically, multiple
modules that use cherrypy stomp on each other.  I discovered this while
playing with John's guilolz experiment
(https://github.com/ceph/ceph/pull/13725).

Both rest and guilolz modules follow this pattern:

  cherrypy.config.update({'server.socket_port': ....})
  cherrypy.tree.graft(...)  # or cherrypy.tree.mount(...)
  cherrypy.engine.start()
  cherrypy.engine.block()

So:

- They're both first setting the port to listen on globally.
- They both eat the whole web namespace (graft/mount).
- They both call cherrypy.engine.start() to start the server.

In each of the above cases, the first module loaded will win and the
other won't (or, it might be racey and you'll kinda get a mix?).  For
example, the second call to cherrypy.engine.start() fails complaining it
can't be called more than once from the same thread.

I'm not sure how best to deal with this.  Assuming we *do* want it to be
possible for multiple modules that provide a web service to be loaded
simultaneously, we need to:

- not set the port globally.
- not stomp on each others namespaces.
- only call cherrypy.engine.start() once.

You can have multiple cherrypy servers listening on different ports, so
that gets us part of the way there, e.g.:

  from cherrypy._cpserver import Server
  server = Server()
  server.socket_port = .... # whatever
  server.subscribe()

But, this doesn't fix the namespace eating problem.  That might be
doable with cherrypy virtualhosts
(http://tools.cherrypy.org/wiki/VirtualHosts), but I haven't
experimented further with that yet.

Also, those subscribe()s have to happen before the
cherrypy.engine.start() call or they have no effect.  One way to do that
would be to have some magic cherrypy module which is always loaded last
to start the engine, but that might be ugly.

Another option might be to have a separate python interpreter per module
(can't stomp on each other then, right?) but that might be too resource
intensive.

Thoughts?

Thanks,

Tim
-- 
Tim Serong
Senior Clustering Engineer
SUSE
tserong@suse.com

Re: multiple cherrypys in ceph-mgr modules stomp on each other

[Ricardo Dias]

[-- Attachment #1: Type: text/plain, Size: 2893 bytes --]

Hi,

What if ceph-mgr modules could actually depend on each other. That way,
we could have a single cherrypy module and other modules could depend
on the cherrypy module.
The cherrypy module would redirect requests to the corresponding module.
Would that be an option?

I know nothing about the ceph-mgr architecture, so ignore my suggestion if
does not make sense at all.

Ricardo Dias

> On 11 Apr 2017, at 11:41, Tim Serong <tserong@suse.com> wrote:
> 
> Hi All,
> 
> We have a problem using cherrypy in mgr modules.  Specifically, multiple
> modules that use cherrypy stomp on each other.  I discovered this while
> playing with John's guilolz experiment
> (https://github.com/ceph/ceph/pull/13725).
> 
> Both rest and guilolz modules follow this pattern:
> 
>  cherrypy.config.update({'server.socket_port': ....})
>  cherrypy.tree.graft(...)  # or cherrypy.tree.mount(...)
>  cherrypy.engine.start()
>  cherrypy.engine.block()
> 
> So:
> 
> - They're both first setting the port to listen on globally.
> - They both eat the whole web namespace (graft/mount).
> - They both call cherrypy.engine.start() to start the server.
> 
> In each of the above cases, the first module loaded will win and the
> other won't (or, it might be racey and you'll kinda get a mix?).  For
> example, the second call to cherrypy.engine.start() fails complaining it
> can't be called more than once from the same thread.
> 
> I'm not sure how best to deal with this.  Assuming we *do* want it to be
> possible for multiple modules that provide a web service to be loaded
> simultaneously, we need to:
> 
> - not set the port globally.
> - not stomp on each others namespaces.
> - only call cherrypy.engine.start() once.
> 
> You can have multiple cherrypy servers listening on different ports, so
> that gets us part of the way there, e.g.:
> 
>  from cherrypy._cpserver import Server
>  server = Server()
>  server.socket_port = .... # whatever
>  server.subscribe()
> 
> But, this doesn't fix the namespace eating problem.  That might be
> doable with cherrypy virtualhosts
> (http://tools.cherrypy.org/wiki/VirtualHosts), but I haven't
> experimented further with that yet.
> 
> Also, those subscribe()s have to happen before the
> cherrypy.engine.start() call or they have no effect.  One way to do that
> would be to have some magic cherrypy module which is always loaded last
> to start the engine, but that might be ugly.
> 
> Another option might be to have a separate python interpreter per module
> (can't stomp on each other then, right?) but that might be too resource
> intensive.
> 
> Thoughts?
> 
> Thanks,
> 
> Tim
> --
> Tim Serong
> Senior Clustering Engineer
> SUSE
> tserong@suse.com
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: multiple cherrypys in ceph-mgr modules stomp on each other

[John Spray]

On Tue, Apr 11, 2017 at 11:41 AM, Tim Serong <tserong@suse.com> wrote:
> Hi All,
>
> We have a problem using cherrypy in mgr modules.  Specifically, multiple
> modules that use cherrypy stomp on each other.  I discovered this while
> playing with John's guilolz experiment
> (https://github.com/ceph/ceph/pull/13725).
>
> Both rest and guilolz modules follow this pattern:
>
>   cherrypy.config.update({'server.socket_port': ....})
>   cherrypy.tree.graft(...)  # or cherrypy.tree.mount(...)
>   cherrypy.engine.start()
>   cherrypy.engine.block()
>
> So:
>
> - They're both first setting the port to listen on globally.
> - They both eat the whole web namespace (graft/mount).
> - They both call cherrypy.engine.start() to start the server.
>
> In each of the above cases, the first module loaded will win and the
> other won't (or, it might be racey and you'll kinda get a mix?).  For
> example, the second call to cherrypy.engine.start() fails complaining it
> can't be called more than once from the same thread.
>
> I'm not sure how best to deal with this.  Assuming we *do* want it to be
> possible for multiple modules that provide a web service to be loaded
> simultaneously, we need to:
>
> - not set the port globally.
> - not stomp on each others namespaces.
> - only call cherrypy.engine.start() once.
>
> You can have multiple cherrypy servers listening on different ports, so
> that gets us part of the way there, e.g.:
>
>   from cherrypy._cpserver import Server
>   server = Server()
>   server.socket_port = .... # whatever
>   server.subscribe()
>
> But, this doesn't fix the namespace eating problem.  That might be
> doable with cherrypy virtualhosts
> (http://tools.cherrypy.org/wiki/VirtualHosts), but I haven't
> experimented further with that yet.
>
> Also, those subscribe()s have to happen before the
> cherrypy.engine.start() call or they have no effect.  One way to do that
> would be to have some magic cherrypy module which is always loaded last
> to start the engine, but that might be ugly.
>
> Another option might be to have a separate python interpreter per module
> (can't stomp on each other then, right?) but that might be too resource
> intensive.

My assumption has been that the global cherrypy stuff was a
convenience that could be un-wrapped into something simpler as and
when, but that may not be the case...

We could have a small shared python module next to mgr_module.py that
modules could use to cooperate nicely on the global cherrypy bits.  It
wouldn't be a total hack, as it would make some of "make me a
webserver" code less repetitive between modules that want to do that.

Doing separate sub-interpreters would also be an option that would
give us more robustness generally in the face of python modules that
do global things.  I don't think there was any fundamental reason I
didn't use sub-interpreters when writing this stuff originally,
they're just a comparatively sparesely documented part of CPython.

I think either one is probably fine?

John

>
>
> Thoughts?
>
> Thanks,
>
> Tim
> --
> Tim Serong
> Senior Clustering Engineer
> SUSE
> tserong@suse.com
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multiple cherrypys in ceph-mgr modules stomp on each other

[John Spray]

On Tue, Apr 11, 2017 at 12:17 PM, Ricardo Dias <rdias@suse.com> wrote:
> Hi,
>
> What if ceph-mgr modules could actually depend on each other. That way,
> we could have a single cherrypy module and other modules could depend
> on the cherrypy module.
> The cherrypy module would redirect requests to the corresponding module.
> Would that be an option?

That's already possible, it's just that the cherrypy module wouldn't
be a mgr module per se, it would just be a python module that the mgr
modules used.

John

> I know nothing about the ceph-mgr architecture, so ignore my suggestion if
> does not make sense at all.
>
> Ricardo Dias
>
>> On 11 Apr 2017, at 11:41, Tim Serong <tserong@suse.com> wrote:
>>
>> Hi All,
>>
>> We have a problem using cherrypy in mgr modules.  Specifically, multiple
>> modules that use cherrypy stomp on each other.  I discovered this while
>> playing with John's guilolz experiment
>> (https://github.com/ceph/ceph/pull/13725).
>>
>> Both rest and guilolz modules follow this pattern:
>>
>>  cherrypy.config.update({'server.socket_port': ....})
>>  cherrypy.tree.graft(...)  # or cherrypy.tree.mount(...)
>>  cherrypy.engine.start()
>>  cherrypy.engine.block()
>>
>> So:
>>
>> - They're both first setting the port to listen on globally.
>> - They both eat the whole web namespace (graft/mount).
>> - They both call cherrypy.engine.start() to start the server.
>>
>> In each of the above cases, the first module loaded will win and the
>> other won't (or, it might be racey and you'll kinda get a mix?).  For
>> example, the second call to cherrypy.engine.start() fails complaining it
>> can't be called more than once from the same thread.
>>
>> I'm not sure how best to deal with this.  Assuming we *do* want it to be
>> possible for multiple modules that provide a web service to be loaded
>> simultaneously, we need to:
>>
>> - not set the port globally.
>> - not stomp on each others namespaces.
>> - only call cherrypy.engine.start() once.
>>
>> You can have multiple cherrypy servers listening on different ports, so
>> that gets us part of the way there, e.g.:
>>
>>  from cherrypy._cpserver import Server
>>  server = Server()
>>  server.socket_port = .... # whatever
>>  server.subscribe()
>>
>> But, this doesn't fix the namespace eating problem.  That might be
>> doable with cherrypy virtualhosts
>> (http://tools.cherrypy.org/wiki/VirtualHosts), but I haven't
>> experimented further with that yet.
>>
>> Also, those subscribe()s have to happen before the
>> cherrypy.engine.start() call or they have no effect.  One way to do that
>> would be to have some magic cherrypy module which is always loaded last
>> to start the engine, but that might be ugly.
>>
>> Another option might be to have a separate python interpreter per module
>> (can't stomp on each other then, right?) but that might be too resource
>> intensive.
>>
>> Thoughts?
>>
>> Thanks,
>>
>> Tim
>> --
>> Tim Serong
>> Senior Clustering Engineer
>> SUSE
>> tserong@suse.com
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: multiple cherrypys in ceph-mgr modules stomp on each other

[Tim Serong]

On 04/11/2017 09:17 PM, John Spray wrote:
> On Tue, Apr 11, 2017 at 11:41 AM, Tim Serong <tserong@suse.com> wrote:
>> [...]
>> cherrypy.engine.start() call or they have no effect.  One way to do that
>> would be to have some magic cherrypy module which is always loaded last
>> to start the engine, but that might be ugly.
>>
>> Another option might be to have a separate python interpreter per module
>> (can't stomp on each other then, right?) but that might be too resource
>> intensive.
> 
> My assumption has been that the global cherrypy stuff was a
> convenience that could be un-wrapped into something simpler as and
> when, but that may not be the case...
> 
> We could have a small shared python module next to mgr_module.py that
> modules could use to cooperate nicely on the global cherrypy bits.  It
> wouldn't be a total hack, as it would make some of "make me a
> webserver" code less repetitive between modules that want to do that.
> 
> Doing separate sub-interpreters would also be an option that would
> give us more robustness generally in the face of python modules that
> do global things.  I don't think there was any fundamental reason I
> didn't use sub-interpreters when writing this stuff originally,
> they're just a comparatively sparesely documented part of CPython.

I think I'm liking the second option for general robustness.  My best
guess right now is that's probably adding a Py_NewInterpreter() call to
MgrPyModule's constructor, and PyThreadState_Swap()s in its methods so
the right thread state is active at the right time.  I reckon this'll
require some other rejigging in various places though -- note how
https://docs.python.org/2/c-api/init.html#bugs-and-caveats says that
combining sub-interpreters with the PyGILState_*() APIs "is delicate",
and further up it flat out states "The following functions use
thread-local storage, and are not compatible with sub-interpreters:
PyGILState_Ensure() ... PyGILState_Release()..." etc.

Given the sparseness of the docs it's probably best to just try it and
see what catches on fire.  I'll keep you posted.

Regards,

Tim
-- 
Tim Serong
Senior Clustering Engineer
SUSE
tserong@suse.com

Re: multiple cherrypys in ceph-mgr modules stomp on each other

[Tim Serong]

On 04/13/2017 11:21 PM, Tim Serong wrote:
> On 04/11/2017 09:17 PM, John Spray wrote:
>> Doing separate sub-interpreters would also be an option that would
>> give us more robustness generally in the face of python modules that
>> do global things.  I don't think there was any fundamental reason I
>> didn't use sub-interpreters when writing this stuff originally,
>> they're just a comparatively sparesely documented part of CPython.
> 
> I think I'm liking the second option for general robustness.  My best
> guess right now is that's probably adding a Py_NewInterpreter() call to
> MgrPyModule's constructor, and PyThreadState_Swap()s in its methods so
> the right thread state is active at the right time.  I reckon this'll
> require some other rejigging in various places though -- note how
> https://docs.python.org/2/c-api/init.html#bugs-and-caveats says that
> combining sub-interpreters with the PyGILState_*() APIs "is delicate",
> and further up it flat out states "The following functions use
> thread-local storage, and are not compatible with sub-interpreters:
> PyGILState_Ensure() ... PyGILState_Release()..." etc.
> 
> Given the sparseness of the docs it's probably best to just try it and
> see what catches on fire.  I'll keep you posted.

Coming back to this again, just for the record I've got something that
seems to work:

  https://github.com/ceph/ceph/pull/14971

Note that this should be considered experimental until further notice
:-) but is hopefully promising.

Cheers,

Tim
-- 
Tim Serong
Senior Clustering Engineer
SUSE
tserong@suse.com