RFC: bugfix for 3575

RFC: bugfix for 3575

[Marinescu, Bogdan A]

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

Hi,

Please review my fix of bug 3575 (Ensure windows hob client communication
with bitbake backend via xmlrpc). I'd like to get feedback before uploading
the patches. I've pushed my fix on poky-contrib:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=bogdanm/b3575

Currently, the fix allows bitbake and hob to run on different machines and
"talk" to each other via XMLRPC (no windows version yet, but it can be
tested in Linux). Hopefully the commit messages will tell the rest of the
story.

Thanks,
Bogdan

[-- Attachment #2: Type: text/html, Size: 763 bytes --]

Re: RFC: bugfix for 3575

[Richard Purdie]

On Wed, 2013-01-23 at 12:08 +0200, Marinescu, Bogdan A wrote:
> Please review my fix of bug 3575 (Ensure windows hob client
> communication with bitbake backend via xmlrpc). I'd like to get
> feedback before uploading the patches. I've pushed my fix on
> poky-contrib:
>http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=bogdanm/b3575
>
> Currently, the fix allows bitbake and hob to run on different machines
> and "talk" to each other via XMLRPC (no windows version yet, but it
> can be tested in Linux). Hopefully the commit messages will tell the
> rest of the story.

First two commits are no problem.

Third Commit:

I don't like the merging of all the extra cache data. I think when a
given UI registers, it needs to call into cooker and inform bitbake it
requires the extra caches (if it does so). With your current code, we
might as well throw away the extra cache code as it effectively works as
the greatest set. I see this only happens in server mode but I'm still
not 100% convinced its the right thing to do.

I also don't see why knotty can't work in server mode?

Fourth Commit:

I have a strong opinion that we want to support multiple clients
connected to one server. I appreciate this complicates things and
currently would be chaotic. I think the easiest way to deal with this is
to have some notion of a single "controller" which would perhaps be
represented by the token. We'd then have some kind of handoff process so
that a given UI could claim the token and then control the server. We
might be able to lose the need for the manual reclaim option too if we
can tie this to the registered handlers somehow?

Any thoughts on that approach? 

Cheers,

Richard

Re: RFC: bugfix for 3575

[Marinescu, Bogdan A]

[-- Attachment #1: Type: text/plain, Size: 9397 bytes --]

On Wed, Jan 23, 2013 at 5:14 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2013-01-23 at 12:08 +0200, Marinescu, Bogdan A wrote:
> > Please review my fix of bug 3575 (Ensure windows hob client
> > communication with bitbake backend via xmlrpc). I'd like to get
> > feedback before uploading the patches. I've pushed my fix on
> > poky-contrib:
> >
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=bogdanm/b3575
> >
> > Currently, the fix allows bitbake and hob to run on different machines
> > and "talk" to each other via XMLRPC (no windows version yet, but it
> > can be tested in Linux). Hopefully the commit messages will tell the
> > rest of the story.
>
> First two commits are no problem.
>
> Third Commit:
>
> I don't like the merging of all the extra cache data. I think when a
> given UI registers, it needs to call into cooker and inform bitbake it
> requires the extra caches (if it does so). With your current code, we
> might as well throw away the extra cache code as it effectively works as
> the greatest set. I see this only happens in server mode but I'm still
> not 100% convinced its the right thing to do.
>

I considered that. It would have been the most generic and elegant
solution, but I decided against it mainly because currently only Hob uses
an extra cache (and judging from its size, it doesn't appear to incur a
large speed and/or memory penalty), which didn't seem to compensate for the
amount of work needed to implement the extra-caches-load-on-demand feature.
Plus, if we go for the load-on-demand, we'll probably want to implement
unload-on-exit (when the client exits and the caches aren't needed anymore)
for completion, which adds even more complexity. That said, I can start
working on this if you think it is the right way to do.


>
> I also don't see why knotty can't work in server mode?
>

I added my thoughts on that in the bug description page, I'll copy/paste
them here:

=========================================================================================================

> There is no good reason knotty shouldn't work with a remote server over
> xmlrpc?

There are a few that I've found. The main problem comes from the fact
that the server is initialized like this:

    server = server.BitBakeServer()
    if configuration.bind:
        server.initServer((configuration.bind, 0))
    else:
        server.initServer()

    idle = server.getServerIdleCB()

    cooker = bb.cooker.BBCooker(configuration, idle, initialenv)
    cooker.parseCommandLine()

    server.addcooker(cooker)
    server.saveConnectionDetails()
    server.detach()

As you can see, parseCommandLine is called when the server is
initialized. If we switch to client/server model, the command line on
the client side won't be taken into account, since it's not ever
parsed. Fortunately, this is fairly easy to fix (by moving the command
line parsing on the client side).
The real problem, which is related to the one above, comes from
dealing with conf/local.conf. This is parsed early in the cooker
(loadConfigurationData) and again, the cooker is "linked" with the
server and initialized at server creation time. Even if we add a
"reparseConfigurationData" method to the cooker, if the client
'bitbake' is started from a different build directory, we need access
to that build's directory conf/ files. Hob works differently in this
respect, since the user sets the required configuration data in the
interface and then Hob calls setVariable on the server, thus setting
the proper configuration. Of course, this can be done with knotty or
any other user interface too, but it definitely needs some work and
should probably be the subject of another bug.

=========================================================================================================


>
> Fourth Commit:
>
> I have a strong opinion that we want to support multiple clients
> connected to one server. I appreciate this complicates things and
> currently would be chaotic. I think the easiest way to deal with this is
> to have some notion of a single "controller" which would perhaps be
> represented by the token. We'd then have some kind of handoff process so
> that a given UI could claim the token and then control the server. We
> might be able to lose the need for the manual reclaim option too if we
> can tie this to the registered handlers somehow?
>
> Any thoughts on that approach?
>

I've considered this too, but I dismissed it quickly. I admit that my
knowledge about bitbake/hob is still pretty basic and that must have scared
me away, but still, there are some hard questions to be answered here. What
happens if there is one server connected to two clients simultaneously,
each client with its own set of configuration variables? When we switch
from one client to the other, we'd need to "reset" the server to use the
configuration variables of this last client. In fact, we probably need to
restore the full previous _state_ of this specific server/client
combination, which might be more than just the configuration variables.
Implementing this would probably take considerable effort, but even if we
do it, will it be worth from a performance standpoint? It seems like a lot
of time and memory would be spend in these save/restore state operations.
Of course, the problem will get progressively worse as the number of
clients increases.
I had a different idea for this: leave bitbake alone and implement a
"meta-server". In short, this meta-server would be a service listening on a
specific port on the build machine. The meta-server keeps a "pool" of
bitbake servers running on the build machine. The bitbake servers are
separate processes, bound to separate build directories, but configured to
share as much data as possible (shared state, downloads and everything else
that can be shared). When the meta-server receives a request from a client,
it would:

1. check if a bitbake server from its server pool is available (idle).
2. if not, create a new bitbake server (properly set to share data as
described above). Add this sever to the server pool.
3. the result of steps 1 or 2 above will be a free bitbake server, ready to
be connected to the client, so the meta-server will pass the host/port data
of the bitbake server to the client, so they can talk directly.
4. when the client finishes, it sends a "release server" request to the
meta-server, which will mark the server as "idle" (and maybe restart it to
reset its internal state? not sure if this is needed).

To add knotty into the picture, step 3 above might be augmented with a file
transfer step that would send the whole "conf/local.conf" directory on the
client side to the (meta-) server. The server would then receive the
client's configuration, interpret it, ignore what needs to be ignored
(mostly the variables used to share various data with the other bitbake
servers on the build machine) and start the build.
This is still quite a bit of work, but for someone with my basic level of
understanding of the bitbake internals it sounds much easier to implement.
For me, this is conceptually like chosing between modifying Python to
remove its GIL restrictions versus simply using multiprocessing for fixing
a problem that can be fixed with both approaches.

Thanks,
Bogdan


On Wed, Jan 23, 2013 at 5:14 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2013-01-23 at 12:08 +0200, Marinescu, Bogdan A wrote:
> > Please review my fix of bug 3575 (Ensure windows hob client
> > communication with bitbake backend via xmlrpc). I'd like to get
> > feedback before uploading the patches. I've pushed my fix on
> > poky-contrib:
> >
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=bogdanm/b3575
> >
> > Currently, the fix allows bitbake and hob to run on different machines
> > and "talk" to each other via XMLRPC (no windows version yet, but it
> > can be tested in Linux). Hopefully the commit messages will tell the
> > rest of the story.
>
> First two commits are no problem.
>
> Third Commit:
>
> I don't like the merging of all the extra cache data. I think when a
> given UI registers, it needs to call into cooker and inform bitbake it
> requires the extra caches (if it does so). With your current code, we
> might as well throw away the extra cache code as it effectively works as
> the greatest set. I see this only happens in server mode but I'm still
> not 100% convinced its the right thing to do.
>
> I also don't see why knotty can't work in server mode?
>
> Fourth Commit:
>
> I have a strong opinion that we want to support multiple clients
> connected to one server. I appreciate this complicates things and
> currently would be chaotic. I think the easiest way to deal with this is
> to have some notion of a single "controller" which would perhaps be
> represented by the token. We'd then have some kind of handoff process so
> that a given UI could claim the token and then control the server. We
> might be able to lose the need for the manual reclaim option too if we
> can tie this to the registered handlers somehow?
>
> Any thoughts on that approach?
>
> Cheers,
>
> Richard
>
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 11379 bytes --]