debugging mgr python modules

debugging mgr python modules

[Tim Serong]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Hi All,

Right now, if you want to debug an mgr python module, you're pretty much
restricted to dumping interesting stuff to log files, right?

What about if you wanted to run an interactive python debugger?

I had two half-formed ideas for this.  One is that apparently there's
something called prdb, which could theoretically be loaded in an mgr
module to enable remote debugging over a socket (although
https://pypi.python.org/pypi/rpdb/ says it reroutes stdout and stdin,
which I assume might not work given mgr is already eating those -- not
sure, haven't had a chance to try it yet).

The second idea was to create a python wrapper which provides a stub
implementation of the pieces that ceph-mgr exposes from C++ land
(ceph_state, etc.).  This would expose some fake cluster state, much as
the minion sim thing did in calamari.  Then you could (somehow) load/run
a mgr python module completely outside mgr to experimentally poke at it.
 Having typed all that out  now, I suspect this might be a PITA to
maintain and be ultimately less useful for live debugging that the first
half-formed idea, although it might be interesting for unit testing.

Any thoughts on whether any of the above is worth pursuing?

Thanks,

Tim
-- 
Tim Serong
Senior Clustering Engineer
SUSE
tserong@suse.com

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: tserong.vcf --]
[-- Type: text/x-vcard; name="tserong.vcf", Size: 4 bytes --]

null

Re: debugging mgr python modules

[Ricardo Dias]

On 23-01-2018 00:38, Tim Serong wrote:
> Hi All,
> 
> Right now, if you want to debug an mgr python module, you're pretty much
> restricted to dumping interesting stuff to log files, right?
> 
> What about if you wanted to run an interactive python debugger?
> 
> I had two half-formed ideas for this.  One is that apparently there's
> something called prdb, which could theoretically be loaded in an mgr
> module to enable remote debugging over a socket (although
> https://pypi.python.org/pypi/rpdb/ says it reroutes stdout and stdin,
> which I assume might not work given mgr is already eating those -- not
> sure, haven't had a chance to try it yet).

Even without having the stdout/err it's very useful to be able to stop 
execution at breakpoints and do a step-by-step execution. I think it 
should be easy to make prdb to work with ceph-mgr plugins. Thanks for 
the research.

> 
> The second idea was to create a python wrapper which provides a stub
> implementation of the pieces that ceph-mgr exposes from C++ land
> (ceph_state, etc.).  This would expose some fake cluster state, much as
> the minion sim thing did in calamari.  Then you could (somehow) load/run
> a mgr python module completely outside mgr to experimentally poke at it.
>   Having typed all that out  now, I suspect this might be a PITA to
> maintain and be ultimately less useful for live debugging that the first
> half-formed idea, although it might be interesting for unit testing.

I think this idea is also very useful but for a different use-case like 
unit/functional testing without having to start a Ceph cluster.
This way we could add unit tests to the ceph-mgr plugins and run them 
along with the already existing tests.

> 
> Any thoughts on whether any of the above is worth pursuing?
> 
> Thanks,
> 
> Tim
> 

-- 
Ricardo Dias
Senior Software Engineer - Storage Team
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284
(AG Nürnberg)

Re: debugging mgr python modules

[Lars Marowsky-Bree]

On 2018-01-23T07:30:43, Ricardo Dias <rdias@suse.com> wrote:

> I think this idea is also very useful but for a different use-case like
> unit/functional testing without having to start a Ceph cluster.
> This way we could add unit tests to the ceph-mgr plugins and run them along
> with the already existing tests.

As far as crazy ideas go, it'd be useful if we could get some sort of
API recorder going at the ceph-mgr / plugin interfaces?




-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

Re: debugging mgr python modules

[John Spray]

On Tue, Jan 23, 2018 at 12:38 AM, Tim Serong <tserong@suse.com> wrote:
>
> Hi All,
>
> Right now, if you want to debug an mgr python module, you're pretty much
> restricted to dumping interesting stuff to log files, right?
>
> What about if you wanted to run an interactive python debugger?
>
> I had two half-formed ideas for this.  One is that apparently there's
> something called prdb, which could theoretically be loaded in an mgr
> module to enable remote debugging over a socket (although
> https://pypi.python.org/pypi/rpdb/ says it reroutes stdout and stdin,
> which I assume might not work given mgr is already eating those -- not
> sure, haven't had a chance to try it yet).


This kind of remote interface would be the way to go -- another option
would be https://pypi.python.org/pypi/manhole


>
>
> The second idea was to create a python wrapper which provides a stub
> implementation of the pieces that ceph-mgr exposes from C++ land
> (ceph_state, etc.).  This would expose some fake cluster state, much as
> the minion sim thing did in calamari.  Then you could (somehow) load/run
> a mgr python module completely outside mgr to experimentally poke at it.
>  Having typed all that out  now, I suspect this might be a PITA to
> maintain and be ultimately less useful for live debugging that the first
> half-formed idea, although it might be interesting for unit testing.


PITA to maintain is correct, but even more than that, it's important
that the mgr tests are using the real data structures.  The primary
motivation for the self test functions in modules today is to quickly
detect any changes in underlying structures that violate module
expectations -- if we started using mocks then well-meaning developers
would end up writing unit tests that gave a false sense of security.

However, if there are classes in python modules that don't directly
talk to Ceph, and those classes need out-of-cluster unit tests, then
obviously that's much easier.

Tangentially related: I'm gradually trying to raise awareness of
qa/tasks/vstart_runner.py -- writing tests against running clusters
(like those in qa/tasks/mgr) can still be quick and convenient.

Cheers,
John

>
>
> Any thoughts on whether any of the above is worth pursuing?
>
> Thanks,
>
> Tim
> --
> Tim Serong
> Senior Clustering Engineer
> SUSE
> tserong@suse.com