[RFC PATCH 0/3] memres: fix for --observe-only

[RFC PATCH 0/3] memres: fix for --observe-only

[Robert Yang]

Hello,

These are just RFC patches, bitbake memeres still has other problems as
listed here:
https://wiki.yoctoproject.org/wiki/Tinfoil2

I sent these patches earlier to make sure that I'm on the correct way.

Things I will work on recently:
1) Nongthing in bitbake-cookerdaemon.log, this is incorrect
2) Sometimes, Ctrl-C can't stop the build correctly
3) Sometimes, "bitbake -m" can't shutdown the server

// Robert

The following changes since commit 854c8c2e4c24ea3ddfec6e5b5f6477f0620510e4:

  oeqa/tinfoil: Improve test_wait_event for race issues (2017-07-08 13:34:46 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib rbt/memres
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/memres

Robert Yang (3):
  bitbake: knotty.py: add MonitorDiskEvent and RecipeTaskPreProcess
  bitbake: lib: fix --observe-only
  bitbake: uihelper.py: check event.pid is present

 bitbake/lib/bb/main.py          |  2 +-
 bitbake/lib/bb/server/xmlrpc.py | 13 ++++++++-----
 bitbake/lib/bb/ui/knotty.py     |  2 ++
 bitbake/lib/bb/ui/uievent.py    |  5 +++--
 bitbake/lib/bb/ui/uihelper.py   | 41 ++++++++++++++++++++++-------------------
 5 files changed, 36 insertions(+), 27 deletions(-)

-- 
2.11.0

[RFC PATCH 1/3] bitbake: knotty.py: add MonitorDiskEvent and RecipeTaskPreProcess

[Robert Yang]

Sometimes, we may see the errors:
$ bitbake --observe-only
ERROR: Unknown event: <bb.event.MonitorDiskEvent object at 0x7fbd2e0a8438>
ERROR: Unknown event: <bb.event.RecipeTaskPreProcess object at 0x7fdc6b7e7b00>

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/ui/knotty.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index 11afb3e7441..71ec168fa6c 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -667,11 +667,13 @@ def main(server, eventHandler, params, tf = TerminalFilter):
                                   bb.event.MultiConfigParsed,
                                   bb.event.RecipeParsed,
                                   bb.event.RecipePreFinalise,
+                                  bb.event.RecipeTaskPreProcess,
                                   bb.runqueue.runQueueEvent,
                                   bb.event.OperationStarted,
                                   bb.event.OperationCompleted,
                                   bb.event.OperationProgress,
                                   bb.event.DiskFull,
+                                  bb.event.MonitorDiskEvent,
                                   bb.event.HeartbeatEvent,
                                   bb.build.TaskProgress)):
                 continue
-- 
2.11.0

[RFC PATCH 2/3] bitbake: lib: fix --observe-only

[Robert Yang]

Fixed:
$ . ../poky/oe-init-build-env-memres .
$ bitbake world
In another terminal
$ . ../poky/oe-init-build-env-memres .
$ bitbake --observe-only
Traceback (most recent call last):
  File "/buildarea/lyang1/poky/bitbake/bin/bitbake", line 48, in <module>
    cookerdata.CookerConfiguration()))
  File "/buildarea/lyang1/poky/bitbake/lib/bb/main.py", line 456, in bitbake_main
    server, server_connection, ui_module = setup_bitbake(configParams, configuration)
  File "/buildarea/lyang1/poky/bitbake/lib/bb/main.py", line 544, in setup_bitbake
    server_connection.setupEventQueue()
  File "/buildarea/lyang1/poky/bitbake/lib/bb/server/xmlrpc.py", line 363, in setupEventQueue
    self.events = uievent.BBUIEventQueue(self.connection, self.clientinfo)
  File "/buildarea/lyang1/poky/bitbake/lib/bb/ui/uievent.py", line 72, in __init__
    raise Exception(errmsg)
Exception: Could not register UI event handler. Error: Cooker is busy: running, host 127.0.0.1, port 36350
WARNING: Could not register UI event handler. Error: Cooker is busy: running, host 127.0.0.1, port 36350, retry
[snip]

This was becuase xmlrpc doesn't allow connections when cooker is running, but
--observe-only doesn't do any builds, so allow it to connect is safe.

[YOCTO #10731]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/main.py          |  2 +-
 bitbake/lib/bb/server/xmlrpc.py | 13 ++++++++-----
 bitbake/lib/bb/ui/uievent.py    |  5 +++--
 3 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
index 29e391162e6..f998e9dfef4 100755
--- a/bitbake/lib/bb/main.py
+++ b/bitbake/lib/bb/main.py
@@ -541,7 +541,7 @@ def setup_bitbake(configParams, configuration, extrafeatures=None, setup_logging
             bb.event.ui_queue = []
             return None, None, None
 
-        server_connection.setupEventQueue()
+        server_connection.setupEventQueue(configParams.observe_only)
 
         # Restore the environment in case the UI needs it
         for k in cleanedvars:
diff --git a/bitbake/lib/bb/server/xmlrpc.py b/bitbake/lib/bb/server/xmlrpc.py
index 1a475e04ba5..bbc37d72f9b 100644
--- a/bitbake/lib/bb/server/xmlrpc.py
+++ b/bitbake/lib/bb/server/xmlrpc.py
@@ -108,14 +108,14 @@ class BitBakeServerCommands():
         self.server = server
         self.has_client = False
 
-    def registerEventHandler(self, host, port):
+    def registerEventHandler(self, host, port, observe_only=False):
         """
         Register a remote UI Event Handler
         """
         s, t = _create_server(host, port)
 
-        # we don't allow connections if the cooker is running
-        if (self.cooker.state in [bb.cooker.state.parsing, bb.cooker.state.running]):
+        # Only allow observe_only connections if the cooker is running
+        if (self.cooker.state in [bb.cooker.state.parsing, bb.cooker.state.running]) and not observe_only:
             return None, "Cooker is busy: %s" % bb.cooker.state.get_name(self.cooker.state)
 
         self.event_handle = bb.event.register_UIHhandler(s, True)
@@ -359,11 +359,14 @@ class BitBakeXMLRPCServerConnection(BitBakeBaseServerConnection):
         self.transport.set_connection_token(token)
         return self
 
-    def setupEventQueue(self):
-        self.events = uievent.BBUIEventQueue(self.connection, self.clientinfo)
+    def setupEventQueue(self, observe_only=False):
+        self.events = uievent.BBUIEventQueue(self.connection, self.clientinfo, observe_only)
         for event in bb.event.ui_queue:
             self.events.queue_event(event)
 
+        if observe_only:
+            return
+
         _, error = self.connection.runCommand(["setFeatures", self.featureset])
         if error:
             # disconnect the client, we can't make the setFeature work
diff --git a/bitbake/lib/bb/ui/uievent.py b/bitbake/lib/bb/ui/uievent.py
index 9542b911ca0..7dfe02c6c97 100644
--- a/bitbake/lib/bb/ui/uievent.py
+++ b/bitbake/lib/bb/ui/uievent.py
@@ -28,7 +28,7 @@ import socket, threading, pickle, collections
 from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
 
 class BBUIEventQueue:
-    def __init__(self, BBServer, clientinfo=("localhost, 0")):
+    def __init__(self, BBServer, clientinfo=("localhost, 0"), observe_only=False):
 
         self.eventQueue = []
         self.eventQueueLock = threading.Lock()
@@ -36,6 +36,7 @@ class BBUIEventQueue:
 
         self.BBServer = BBServer
         self.clientinfo = clientinfo
+        self.observe_only = observe_only
 
         server = UIXMLRPCServer(self.clientinfo)
         self.host, self.port = server.socket.getsockname()
@@ -51,7 +52,7 @@ class BBUIEventQueue:
         # giving up
 
         for count_tries in range(5):
-            ret = self.BBServer.registerEventHandler(self.host, self.port)
+            ret = self.BBServer.registerEventHandler(self.host, self.port, self.observe_only)
 
             if isinstance(ret, collections.Iterable):
                 self.EventHandle, error = ret
-- 
2.11.0

[RFC PATCH 3/3] bitbake: uihelper.py: check event.pid is present

[Robert Yang]

Fixed:
$ . ../poky/oe-init-build-env-memres .
$ bitbake world
In another terminal:
$ . ../poky/oe-init-build-env-memres .
Run the following command and Ctrl-C several times:
$ bitbake --observe-only
Traceback (most recent call last):
  File "/buildarea/lyang1/poky/bitbake/lib/bb/ui/knotty.py", line 441, in main
    helper.eventHandler(event)
  File "/buildarea/lyang1/poky/bitbake/lib/bb/ui/uihelper.py", line 42, in eventHandler
    del self.running_tasks[event.pid]
KeyError: 43646

This was becase the ui maybe connected at any time, so the event.pid may not be
present.

[YOCTO #11781]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/ui/uihelper.py | 41 ++++++++++++++++++++++-------------------
 1 file changed, 22 insertions(+), 19 deletions(-)

diff --git a/bitbake/lib/bb/ui/uihelper.py b/bitbake/lib/bb/ui/uihelper.py
index 113fcedeaf1..307c87e149c 100644
--- a/bitbake/lib/bb/ui/uihelper.py
+++ b/bitbake/lib/bb/ui/uihelper.py
@@ -38,29 +38,32 @@ class BBUIHelper:
                 self.running_tasks[event.pid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time() }
             self.running_pids.append(event.pid)
             self.needUpdate = True
-        elif isinstance(event, bb.build.TaskSucceeded):
-            del self.running_tasks[event.pid]
-            self.running_pids.remove(event.pid)
-            self.needUpdate = True
-        elif isinstance(event, bb.build.TaskFailedSilent):
-            del self.running_tasks[event.pid]
-            self.running_pids.remove(event.pid)
-            # Don't add to the failed tasks list since this is e.g. a setscene task failure
-            self.needUpdate = True
-        elif isinstance(event, bb.build.TaskFailed):
-            del self.running_tasks[event.pid]
-            self.running_pids.remove(event.pid)
-            self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)})
-            self.needUpdate = True
+        # The ui maybe connected at any time, so make sure that the
+        # event.pid is already in self.running_tasks.
+        elif getattr(event, 'pid', 'none_pid') in self.running_tasks:
+            if isinstance(event, bb.build.TaskSucceeded):
+                del self.running_tasks[event.pid]
+                self.running_pids.remove(event.pid)
+                self.needUpdate = True
+            elif isinstance(event, bb.build.TaskFailedSilent):
+                del self.running_tasks[event.pid]
+                self.running_pids.remove(event.pid)
+                # Don't add to the failed tasks list since this is e.g. a setscene task failure
+                self.needUpdate = True
+            elif isinstance(event, bb.build.TaskFailed):
+                del self.running_tasks[event.pid]
+                self.running_pids.remove(event.pid)
+                self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)})
+                self.needUpdate = True
+            elif isinstance(event, bb.build.TaskProgress):
+                if event.pid > 0:
+                    self.running_tasks[event.pid]['progress'] = event.progress
+                    self.running_tasks[event.pid]['rate'] = event.rate
+                    self.needUpdate = True
         elif isinstance(event, bb.runqueue.runQueueTaskStarted) or isinstance(event, bb.runqueue.sceneQueueTaskStarted):
             self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
             self.tasknumber_total = event.stats.total
             self.needUpdate = True
-        elif isinstance(event, bb.build.TaskProgress):
-            if event.pid > 0:
-                self.running_tasks[event.pid]['progress'] = event.progress
-                self.running_tasks[event.pid]['rate'] = event.rate
-                self.needUpdate = True
 
     def getTasks(self):
         self.needUpdate = False
-- 
2.11.0

Re: [RFC PATCH 1/3] bitbake: knotty.py: add MonitorDiskEvent and RecipeTaskPreProcess

[Richard Purdie]

On Tue, 2017-07-11 at 03:27 -0700, Robert Yang wrote:
> Sometimes, we may see the errors:
> $ bitbake --observe-only
> ERROR: Unknown event: <bb.event.MonitorDiskEvent object at
> 0x7fbd2e0a8438>
> ERROR: Unknown event: <bb.event.RecipeTaskPreProcess object at
> 0x7fdc6b7e7b00>
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/ui/knotty.py | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/bitbake/lib/bb/ui/knotty.py
> b/bitbake/lib/bb/ui/knotty.py
> index 11afb3e7441..71ec168fa6c 100644
> --- a/bitbake/lib/bb/ui/knotty.py
> +++ b/bitbake/lib/bb/ui/knotty.py
> @@ -667,11 +667,13 @@ def main(server, eventHandler, params, tf =
> TerminalFilter):
>                                    bb.event.MultiConfigParsed,
>                                    bb.event.RecipeParsed,
>                                    bb.event.RecipePreFinalise,
> +                                  bb.event.RecipeTaskPreProcess,
>                                    bb.runqueue.runQueueEvent,
>                                    bb.event.OperationStarted,
>                                    bb.event.OperationCompleted,
>                                    bb.event.OperationProgress,
>                                    bb.event.DiskFull,
> +                                  bb.event.MonitorDiskEvent,
>                                    bb.event.HeartbeatEvent,
>                                    bb.build.TaskProgress)):
>                  continue

Do you know why we don't either always see these or always don't see
them? I'm a bit worried there may be a deeper issue lurking here. Are
those events part of the event mask being set?

For reference, I've been looking at the server abstraction in bitbake
and am close to rewriting a large part of bb.server.* and bb.main with
a view to simplifying the code structure and making things easier to
understand.

I've noticed I see some new events with my change, equally I think its
an event mask issue with my new code...

I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-cont
rib/commit/?h=rpurdie/wip-
rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
however I am well aware things don't work properly yet and its full of
debug. When finished I should be able to delete server/__init__.py and
server/xmlrpc.py.

Cheers,

Richard

Re: [RFC PATCH 1/3] bitbake: knotty.py: add MonitorDiskEvent and RecipeTaskPreProcess

[Robert Yang]

Hi RP,

On 07/12/2017 06:56 AM, Richard Purdie wrote:
> On Tue, 2017-07-11 at 03:27 -0700, Robert Yang wrote:
>> Sometimes, we may see the errors:
>> $ bitbake --observe-only
>> ERROR: Unknown event: <bb.event.MonitorDiskEvent object at
>> 0x7fbd2e0a8438>
>> ERROR: Unknown event: <bb.event.RecipeTaskPreProcess object at
>> 0x7fdc6b7e7b00>
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>  bitbake/lib/bb/ui/knotty.py | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/bitbake/lib/bb/ui/knotty.py
>> b/bitbake/lib/bb/ui/knotty.py
>> index 11afb3e7441..71ec168fa6c 100644
>> --- a/bitbake/lib/bb/ui/knotty.py
>> +++ b/bitbake/lib/bb/ui/knotty.py
>> @@ -667,11 +667,13 @@ def main(server, eventHandler, params, tf =
>> TerminalFilter):
>>                                    bb.event.MultiConfigParsed,
>>                                    bb.event.RecipeParsed,
>>                                    bb.event.RecipePreFinalise,
>> +                                  bb.event.RecipeTaskPreProcess,
>>                                    bb.runqueue.runQueueEvent,
>>                                    bb.event.OperationStarted,
>>                                    bb.event.OperationCompleted,
>>                                    bb.event.OperationProgress,
>>                                    bb.event.DiskFull,
>> +                                  bb.event.MonitorDiskEvent,
>>                                    bb.event.HeartbeatEvent,
>>                                    bb.build.TaskProgress)):
>>                  continue
>
> Do you know why we don't either always see these or always don't see
> them? I'm a bit worried there may be a deeper issue lurking here. Are
> those events part of the event mask being set?

I digged this today, and find the reason, finally. It is harder than I
had thought. Take MonitorDiskEvent as an example, the reproducer is:

In terminal 1:
$ bitbake -m ; . ../poky/oe-init-build-env-memres . && bitbake world -k

In Terminal 2:
$ bitbake --observe-only

Note, the time and order is very important, otherwise we can't reproduce
the problem, we must run "bitbake target" in terminal 1 firstly, and then
run "bitbake --observe-only" in terminal 2 after "cache loaded"
before runqueue is ready, in another words, run "bitbake --observe-only"
when "Initialising tasks" is displaying in terminal 1, no early or late,
then we will see the error in terminal 2 (And only in terminal 2):

ERROR: Unknown event: <bb.event.MonitorDiskEvent object at 0x7f5c3776f940>

The "bitbake world" (first) does everything well, but "bitbake --observe-only"
(second) doesn't becuase of the following code in lib/bb/main.py:

         try:
             return ui_module.main(server_connection.connection, 
server_connection.events,
                                   configParams)

The second run of "ui_module.main()" sends server_connection.events 
(bb.event.MonitorDiskEvent) which is prepared by first run, and knotty.py
doesn't know how to handle it, and we will get the error. So we can't
get the error if we run second bitbake earlier (the event is not ready)
or later (the event had been handled by first bitbake).

I think that bb.event.RecipeTaskPreProcess is similar, add them
to ignore list should be fine, and I will add these explanations to
commit message.

>
> For reference, I've been looking at the server abstraction in bitbake
> and am close to rewriting a large part of bb.server.* and bb.main with
> a view to simplifying the code structure and making things easier to
> understand.

That's great, something var name is not easy for me to understand, for example,
observe_only vs observer_only. And I think that we need a simple way to
restart the server, something like bitbake --restart.

>
> I've noticed I see some new events with my change, equally I think its
> an event mask issue with my new code...
>
> I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-cont
> rib/commit/?h=rpurdie/wip-
> rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
> however I am well aware things don't work properly yet and its full of
> debug. When finished I should be able to delete server/__init__.py and
> server/xmlrpc.py.

I will hold on my patches until you've finished the rewriting, and please
let me know if anything I can do.

// Robert

>
> Cheers,
>
> Richard
>
>
>

Re: [RFC PATCH 1/3] bitbake: knotty.py: add MonitorDiskEvent and RecipeTaskPreProcess

[Richard Purdie]

On Wed, 2017-07-12 at 17:45 +0800, Robert Yang wrote:
> On 07/12/2017 06:56 AM, Richard Purdie wrote:
> > 
> > Do you know why we don't either always see these or always don't
> > see
> > them? I'm a bit worried there may be a deeper issue lurking here.
> > Are
> > those events part of the event mask being set?
> I digged this today, and find the reason, finally. It is harder than
> I had thought. Take MonitorDiskEvent as an example, the reproducer
> is:
> 
> In terminal 1:
> $ bitbake -m ; . ../poky/oe-init-build-env-memres . && bitbake world
> -k
> 
> In Terminal 2:
> $ bitbake --observe-only
> 
> Note, the time and order is very important, otherwise we can't
> reproduce
> the problem, we must run "bitbake target" in terminal 1 firstly, and
> then
> run "bitbake --observe-only" in terminal 2 after "cache loaded"
> before runqueue is ready, in another words, run "bitbake --observe-
> only"
> when "Initialising tasks" is displaying in terminal 1, no early or
> late,
> then we will see the error in terminal 2 (And only in terminal 2):
> 
> ERROR: Unknown event: <bb.event.MonitorDiskEvent object at
> 0x7f5c3776f940>
> 
> The "bitbake world" (first) does everything well, but "bitbake --
> observe-only"
> (second) doesn't becuase of the following code in lib/bb/main.py:
> 
>          try:
>              return ui_module.main(server_connection.connection, 
> server_connection.events,
>                                    configParams)
> 
> The second run of "ui_module.main()" sends server_connection.events 
> (bb.event.MonitorDiskEvent) which is prepared by first run, and
> knotty.py
> doesn't know how to handle it, and we will get the error. So we can't
> get the error if we run second bitbake earlier (the event is not
> ready)
> or later (the event had been handled by first bitbake).
> 
> I think that bb.event.RecipeTaskPreProcess is similar, add them
> to ignore list should be fine, and I will add these explanations to
> commit message.

I'm not sure I agree here, I think you're right that there is a race
issue however I'd rather solve any race issue than workaround it.
Whilst you need to add these events today, there are likely other
events which might appear tomorrow.

> > For reference, I've been looking at the server abstraction in
> > bitbake and am close to rewriting a large part of bb.server.* and
> > bb.main with a view to simplifying the code structure and making
> > things easier to understand.
> That's great, something var name is not easy for me to understand,
> for example, observe_only vs observer_only. And I think that we need
> a simple way to restart the server, something like bitbake --restart.
> 
> > 
> > 
> > I've noticed I see some new events with my change, equally I think
> > its
> > an event mask issue with my new code...
> > 
> > I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-
> > cont
> > rib/commit/?h=rpurdie/wip-
> > rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
> > however I am well aware things don't work properly yet and its full
> > of
> > debug. When finished I should be able to delete server/__init__.py
> > and
> > server/xmlrpc.py.
> I will hold on my patches until you've finished the rewriting, and
> please let me know if anything I can do.

The changes are now in master-next. I plan to split the patch up and
have been revising it a bit as I find issues. It would be great if you
could apply this locally and see if the issues you've found are still
present after this change and help with stressing the changes a bit,
see what issues we have?

One thing I know is probably broken at the moment is observe-only
unfortunately. I am tempted to merge the code changes I have as this
gives us a platform to build on which is much more understandable than
the existing codebase even if there are issues like that.

Cheers,

Richard

Re: [RFC PATCH 1/3] bitbake: knotty.py: add MonitorDiskEvent and RecipeTaskPreProcess

[Robert Yang]

On 07/14/2017 03:34 AM, Richard Purdie wrote:
> On Wed, 2017-07-12 at 17:45 +0800, Robert Yang wrote:
>> On 07/12/2017 06:56 AM, Richard Purdie wrote:
>>>
>>> Do you know why we don't either always see these or always don't
>>> see
>>> them? I'm a bit worried there may be a deeper issue lurking here.
>>> Are
>>> those events part of the event mask being set?
>> I digged this today, and find the reason, finally. It is harder than
>> I had thought. Take MonitorDiskEvent as an example, the reproducer
>> is:
>>
>> In terminal 1:
>> $ bitbake -m ; . ../poky/oe-init-build-env-memres . && bitbake world
>> -k
>>
>> In Terminal 2:
>> $ bitbake --observe-only
>>
>> Note, the time and order is very important, otherwise we can't
>> reproduce
>> the problem, we must run "bitbake target" in terminal 1 firstly, and
>> then
>> run "bitbake --observe-only" in terminal 2 after "cache loaded"
>> before runqueue is ready, in another words, run "bitbake --observe-
>> only"
>> when "Initialising tasks" is displaying in terminal 1, no early or
>> late,
>> then we will see the error in terminal 2 (And only in terminal 2):
>>
>> ERROR: Unknown event: <bb.event.MonitorDiskEvent object at
>> 0x7f5c3776f940>
>>
>> The "bitbake world" (first) does everything well, but "bitbake --
>> observe-only"
>> (second) doesn't becuase of the following code in lib/bb/main.py:
>>
>>          try:
>>              return ui_module.main(server_connection.connection,
>> server_connection.events,
>>                                    configParams)
>>
>> The second run of "ui_module.main()" sends server_connection.events
>> (bb.event.MonitorDiskEvent) which is prepared by first run, and
>> knotty.py
>> doesn't know how to handle it, and we will get the error. So we can't
>> get the error if we run second bitbake earlier (the event is not
>> ready)
>> or later (the event had been handled by first bitbake).
>>
>> I think that bb.event.RecipeTaskPreProcess is similar, add them
>> to ignore list should be fine, and I will add these explanations to
>> commit message.
>
> I'm not sure I agree here, I think you're right that there is a race
> issue however I'd rather solve any race issue than workaround it.
> Whilst you need to add these events today, there are likely other
> events which might appear tomorrow.

A rough thoughts on fixing this might be:

The original code:
try:
     return ui_module.main(server_connection.connection, 
server_connection.events, configParams)

The problem is server_connection.events contains more events than
ui_module.main() can handle, we can filter them out: (fake code)

filtered_events = []

for event in server_connection.events:
	if event in ui_module._evt_list:
		filtered_events.append(event)

And use filtered_events to call ui_module.main():
return ui_module.main(server_connection.connection, filtered_events, configParams)

Then we can totally remove the ignore list in ui like knotty.py:

             # ignore
             if isinstance(event, (bb.event.BuildBase,
                                   bb.event.MetadataEvent,
                                   bb.event.StampUpdate,
                                   bb.event.ConfigParsed,
                                   bb.event.MultiConfigParsed,
                                   bb.event.RecipeParsed,
                                   bb.event.RecipePreFinalise,
                                   bb.runqueue.runQueueEvent,
                                   bb.event.OperationStarted,
                                   bb.event.OperationCompleted,
                                   bb.event.OperationProgress,
                                   bb.event.DiskFull,
                                   bb.event.HeartbeatEvent,
                                   bb.build.TaskProgress)):
                 continue

>
>>> For reference, I've been looking at the server abstraction in
>>> bitbake and am close to rewriting a large part of bb.server.* and
>>> bb.main with a view to simplifying the code structure and making
>>> things easier to understand.
>> That's great, something var name is not easy for me to understand,
>> for example, observe_only vs observer_only. And I think that we need
>> a simple way to restart the server, something like bitbake --restart.
>>
>>>
>>>
>>> I've noticed I see some new events with my change, equally I think
>>> its
>>> an event mask issue with my new code...
>>>
>>> I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-
>>> cont
>>> rib/commit/?h=rpurdie/wip-
>>> rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
>>> however I am well aware things don't work properly yet and its full
>>> of
>>> debug. When finished I should be able to delete server/__init__.py
>>> and
>>> server/xmlrpc.py.
>> I will hold on my patches until you've finished the rewriting, and
>> please let me know if anything I can do.
>
> The changes are now in master-next. I plan to split the patch up and
> have been revising it a bit as I find issues. It would be great if you
> could apply this locally and see if the issues you've found are still
> present after this change and help with stressing the changes a bit,
> see what issues we have?
>
> One thing I know is probably broken at the moment is observe-only
> unfortunately. I am tempted to merge the code changes I have as this
> gives us a platform to build on which is much more understandable than
> the existing codebase even if there are issues like that.

Sure, I'd like to test them today.

// Robert

>
> Cheers,
>
> Richard
>
>
>

Re: [RFC PATCH 1/3] bitbake: knotty.py: add MonitorDiskEvent and RecipeTaskPreProcess

[Robert Yang]

Hi RP,

I tested master-next + memres today, unfortunately, I couldn't make memres
work by now, I fixes part of the problems, but there are still other issues.
Here are 3 fixes:

   git://git.pokylinux.org/poky-contrib rbt/master-next-memres
   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/master-next-memres

Robert Yang (3):
   oe-init-build-env-memres: remove "-t xmlrpc" when run bitbake
   server/process.py: fix self.bitbake_lock.write()
   bb/main.py: fix infinite loop for --server-only


The main remaing issue is "bitbake --status-only" doesn't work:
* Reproducer:
$ . ../poky/oe-init-build-env-memres .
$ bitbake --status-only
[snip]
OverflowError: getsockaddrarg: port must be 0-65535.

I think that it is because it can't handle BBSERVER or "port == -1"
correctly, we may need read port from bitbake.lock or BBSERVER,
the similar to "bitbake -m":

* Reproducer:
$ . ../poky/oe-init-build-env-memres .
$ bitbake -m
OverflowError: getsockaddrarg: port must be 0-65535.
WARNING: Could not create socket for localhost:-1 (getsockaddrarg: port must be 
0-65535.)

Other problems:
1) $ . ../poky/oe-init-build-env-memres .
Starting bitbake server...
DEBUG: Removed the following variables from the environment: TERMCAP, ..
It prints DEBUG messages, maybe because logger is not enabled or set correctly
when --server-only

2) Errors when run oe-init-build-env-memres again: (when bitbake.lock exists)
   $ . ../poky/oe-init-build-env-memres .
   $ . ../poky/oe-init-build-env-memres .
   File "/buildarea/lyang1/poky/bitbake/lib/bb/server/xmlrpcclient.py", line 
140, in connectXMLRPC
     s.connect((host, port))
OSError: [Errno 22] Invalid argument


3) See the code of 2), there is a bb.warn() in xmlrpcclient.py:
    bb.warn("Could not create socket for %s:%s (%s)" % (host, port, str(e)))

    But it doesn't print to screen, maybe the similar reason to step 1),
    and print() works.

// Robert

On 07/14/2017 09:57 AM, Robert Yang wrote:
>
>
> On 07/14/2017 03:34 AM, Richard Purdie wrote:
>> On Wed, 2017-07-12 at 17:45 +0800, Robert Yang wrote:
>>> On 07/12/2017 06:56 AM, Richard Purdie wrote:
>>>>
>>>> Do you know why we don't either always see these or always don't
>>>> see
>>>> them? I'm a bit worried there may be a deeper issue lurking here.
>>>> Are
>>>> those events part of the event mask being set?
>>> I digged this today, and find the reason, finally. It is harder than
>>> I had thought. Take MonitorDiskEvent as an example, the reproducer
>>> is:
>>>
>>> In terminal 1:
>>> $ bitbake -m ; . ../poky/oe-init-build-env-memres . && bitbake world
>>> -k
>>>
>>> In Terminal 2:
>>> $ bitbake --observe-only
>>>
>>> Note, the time and order is very important, otherwise we can't
>>> reproduce
>>> the problem, we must run "bitbake target" in terminal 1 firstly, and
>>> then
>>> run "bitbake --observe-only" in terminal 2 after "cache loaded"
>>> before runqueue is ready, in another words, run "bitbake --observe-
>>> only"
>>> when "Initialising tasks" is displaying in terminal 1, no early or
>>> late,
>>> then we will see the error in terminal 2 (And only in terminal 2):
>>>
>>> ERROR: Unknown event: <bb.event.MonitorDiskEvent object at
>>> 0x7f5c3776f940>
>>>
>>> The "bitbake world" (first) does everything well, but "bitbake --
>>> observe-only"
>>> (second) doesn't becuase of the following code in lib/bb/main.py:
>>>
>>>          try:
>>>              return ui_module.main(server_connection.connection,
>>> server_connection.events,
>>>                                    configParams)
>>>
>>> The second run of "ui_module.main()" sends server_connection.events
>>> (bb.event.MonitorDiskEvent) which is prepared by first run, and
>>> knotty.py
>>> doesn't know how to handle it, and we will get the error. So we can't
>>> get the error if we run second bitbake earlier (the event is not
>>> ready)
>>> or later (the event had been handled by first bitbake).
>>>
>>> I think that bb.event.RecipeTaskPreProcess is similar, add them
>>> to ignore list should be fine, and I will add these explanations to
>>> commit message.
>>
>> I'm not sure I agree here, I think you're right that there is a race
>> issue however I'd rather solve any race issue than workaround it.
>> Whilst you need to add these events today, there are likely other
>> events which might appear tomorrow.
>
> A rough thoughts on fixing this might be:
>
> The original code:
> try:
>     return ui_module.main(server_connection.connection,
> server_connection.events, configParams)
>
> The problem is server_connection.events contains more events than
> ui_module.main() can handle, we can filter them out: (fake code)
>
> filtered_events = []
>
> for event in server_connection.events:
>     if event in ui_module._evt_list:
>         filtered_events.append(event)
>
> And use filtered_events to call ui_module.main():
> return ui_module.main(server_connection.connection, filtered_events, configParams)
>
> Then we can totally remove the ignore list in ui like knotty.py:
>
>             # ignore
>             if isinstance(event, (bb.event.BuildBase,
>                                   bb.event.MetadataEvent,
>                                   bb.event.StampUpdate,
>                                   bb.event.ConfigParsed,
>                                   bb.event.MultiConfigParsed,
>                                   bb.event.RecipeParsed,
>                                   bb.event.RecipePreFinalise,
>                                   bb.runqueue.runQueueEvent,
>                                   bb.event.OperationStarted,
>                                   bb.event.OperationCompleted,
>                                   bb.event.OperationProgress,
>                                   bb.event.DiskFull,
>                                   bb.event.HeartbeatEvent,
>                                   bb.build.TaskProgress)):
>                 continue
>
>>
>>>> For reference, I've been looking at the server abstraction in
>>>> bitbake and am close to rewriting a large part of bb.server.* and
>>>> bb.main with a view to simplifying the code structure and making
>>>> things easier to understand.
>>> That's great, something var name is not easy for me to understand,
>>> for example, observe_only vs observer_only. And I think that we need
>>> a simple way to restart the server, something like bitbake --restart.
>>>
>>>>
>>>>
>>>> I've noticed I see some new events with my change, equally I think
>>>> its
>>>> an event mask issue with my new code...
>>>>
>>>> I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-
>>>> cont
>>>> rib/commit/?h=rpurdie/wip-
>>>> rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
>>>> however I am well aware things don't work properly yet and its full
>>>> of
>>>> debug. When finished I should be able to delete server/__init__.py
>>>> and
>>>> server/xmlrpc.py.
>>> I will hold on my patches until you've finished the rewriting, and
>>> please let me know if anything I can do.
>>
>> The changes are now in master-next. I plan to split the patch up and
>> have been revising it a bit as I find issues. It would be great if you
>> could apply this locally and see if the issues you've found are still
>> present after this change and help with stressing the changes a bit,
>> see what issues we have?
>>
>> One thing I know is probably broken at the moment is observe-only
>> unfortunately. I am tempted to merge the code changes I have as this
>> gives us a platform to build on which is much more understandable than
>> the existing codebase even if there are issues like that.
>
> Sure, I'd like to test them today.
>
> // Robert
>
>>
>> Cheers,
>>
>> Richard
>>
>>
>>

Re: [RFC PATCH 1/3] bitbake: knotty.py: add MonitorDiskEvent and RecipeTaskPreProcess

[Robert Yang]

Hi RP,

I can make master-next + memres partially work now, all the patches
are here:

   git://git.pokylinux.org/poky-contrib rbt/master-next-memres
   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/master-next-memres

Robert Yang (8):
   oe-init-build-env-memres: remove "-t xmlrpc" when run bitbake
   server/process.py: fix self.bitbake_lock.write()
   bb/main.py: fix infinite loop for --server-only
   bb/main.py: read port and host from bitbake.lock
   oe-init-build-env-memres: don't export BBSERVER
   bb/main.py: avoid starting server when not needed
   server/process.py: don't quit when server_only
   bb/main.py: fix logic for --observe-only


I'm not sure about:
oe-init-build-env-memres: don't export BBSERVER

The BBSERVER did cause problems, but don't export it makes bitbake
know nothing about whether we need memeres or not, for example:
$ . ../poky/oe-init-build-env-memres .
$ bitbake -m #shutdown it
$ bitbake core-image-minimal
Then an one time bitbake will start (not memres), compared to old
*master* branch:
$ bitbake core-image-minimal
WARNING: Could not connect to server at 127.0.0.1:44006 ([Errno 111] Connection 
refused)
ERROR: Could not connect to server 127.0.0.1:44006
: [Errno 111] Connection refused

I think that once the user uses "oe-init-build-env-memres" to intialize
the build, then all bitbake commands should be memeres by default.
(No matter connect to an existed server or shutdown and restart).
So maybe we still need export BBSERVER and add more checking in bitbake
to make it work as expected.

BTW., we still have the warnings on master-next:
ERROR: Unknown event: <bb.event.MonitorDiskEvent object at 0x7f8777de5c50>

And:
File "/buildarea/lyang1/poky/bitbake/lib/bb/ui/uihelper.py", line 42, in 
eventHandler
     del self.running_tasks[event.pid]
KeyError: 37402


I will rebase the 3 patches and resubmit after the new code is stable.

// Robert

On 07/14/2017 06:27 PM, Robert Yang wrote:
> Hi RP,
>
> I tested master-next + memres today, unfortunately, I couldn't make memres
> work by now, I fixes part of the problems, but there are still other issues.
> Here are 3 fixes:
>
>   git://git.pokylinux.org/poky-contrib rbt/master-next-memres
>   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/master-next-memres
>
> Robert Yang (3):
>   oe-init-build-env-memres: remove "-t xmlrpc" when run bitbake
>   server/process.py: fix self.bitbake_lock.write()
>   bb/main.py: fix infinite loop for --server-only
>
>
> The main remaing issue is "bitbake --status-only" doesn't work:
> * Reproducer:
> $ . ../poky/oe-init-build-env-memres .
> $ bitbake --status-only
> [snip]
> OverflowError: getsockaddrarg: port must be 0-65535.
>
> I think that it is because it can't handle BBSERVER or "port == -1"
> correctly, we may need read port from bitbake.lock or BBSERVER,
> the similar to "bitbake -m":
>
> * Reproducer:
> $ . ../poky/oe-init-build-env-memres .
> $ bitbake -m
> OverflowError: getsockaddrarg: port must be 0-65535.
> WARNING: Could not create socket for localhost:-1 (getsockaddrarg: port must be
> 0-65535.)
>
> Other problems:
> 1) $ . ../poky/oe-init-build-env-memres .
> Starting bitbake server...
> DEBUG: Removed the following variables from the environment: TERMCAP, ..
> It prints DEBUG messages, maybe because logger is not enabled or set correctly
> when --server-only
>
> 2) Errors when run oe-init-build-env-memres again: (when bitbake.lock exists)
>   $ . ../poky/oe-init-build-env-memres .
>   $ . ../poky/oe-init-build-env-memres .
>   File "/buildarea/lyang1/poky/bitbake/lib/bb/server/xmlrpcclient.py", line 140,
> in connectXMLRPC
>     s.connect((host, port))
> OSError: [Errno 22] Invalid argument
>
>
> 3) See the code of 2), there is a bb.warn() in xmlrpcclient.py:
>    bb.warn("Could not create socket for %s:%s (%s)" % (host, port, str(e)))
>
>    But it doesn't print to screen, maybe the similar reason to step 1),
>    and print() works.
>
> // Robert
>
> On 07/14/2017 09:57 AM, Robert Yang wrote:
>>
>>
>> On 07/14/2017 03:34 AM, Richard Purdie wrote:
>>> On Wed, 2017-07-12 at 17:45 +0800, Robert Yang wrote:
>>>> On 07/12/2017 06:56 AM, Richard Purdie wrote:
>>>>>
>>>>> Do you know why we don't either always see these or always don't
>>>>> see
>>>>> them? I'm a bit worried there may be a deeper issue lurking here.
>>>>> Are
>>>>> those events part of the event mask being set?
>>>> I digged this today, and find the reason, finally. It is harder than
>>>> I had thought. Take MonitorDiskEvent as an example, the reproducer
>>>> is:
>>>>
>>>> In terminal 1:
>>>> $ bitbake -m ; . ../poky/oe-init-build-env-memres . && bitbake world
>>>> -k
>>>>
>>>> In Terminal 2:
>>>> $ bitbake --observe-only
>>>>
>>>> Note, the time and order is very important, otherwise we can't
>>>> reproduce
>>>> the problem, we must run "bitbake target" in terminal 1 firstly, and
>>>> then
>>>> run "bitbake --observe-only" in terminal 2 after "cache loaded"
>>>> before runqueue is ready, in another words, run "bitbake --observe-
>>>> only"
>>>> when "Initialising tasks" is displaying in terminal 1, no early or
>>>> late,
>>>> then we will see the error in terminal 2 (And only in terminal 2):
>>>>
>>>> ERROR: Unknown event: <bb.event.MonitorDiskEvent object at
>>>> 0x7f5c3776f940>
>>>>
>>>> The "bitbake world" (first) does everything well, but "bitbake --
>>>> observe-only"
>>>> (second) doesn't becuase of the following code in lib/bb/main.py:
>>>>
>>>>          try:
>>>>              return ui_module.main(server_connection.connection,
>>>> server_connection.events,
>>>>                                    configParams)
>>>>
>>>> The second run of "ui_module.main()" sends server_connection.events
>>>> (bb.event.MonitorDiskEvent) which is prepared by first run, and
>>>> knotty.py
>>>> doesn't know how to handle it, and we will get the error. So we can't
>>>> get the error if we run second bitbake earlier (the event is not
>>>> ready)
>>>> or later (the event had been handled by first bitbake).
>>>>
>>>> I think that bb.event.RecipeTaskPreProcess is similar, add them
>>>> to ignore list should be fine, and I will add these explanations to
>>>> commit message.
>>>
>>> I'm not sure I agree here, I think you're right that there is a race
>>> issue however I'd rather solve any race issue than workaround it.
>>> Whilst you need to add these events today, there are likely other
>>> events which might appear tomorrow.
>>
>> A rough thoughts on fixing this might be:
>>
>> The original code:
>> try:
>>     return ui_module.main(server_connection.connection,
>> server_connection.events, configParams)
>>
>> The problem is server_connection.events contains more events than
>> ui_module.main() can handle, we can filter them out: (fake code)
>>
>> filtered_events = []
>>
>> for event in server_connection.events:
>>     if event in ui_module._evt_list:
>>         filtered_events.append(event)
>>
>> And use filtered_events to call ui_module.main():
>> return ui_module.main(server_connection.connection, filtered_events,
>> configParams)
>>
>> Then we can totally remove the ignore list in ui like knotty.py:
>>
>>             # ignore
>>             if isinstance(event, (bb.event.BuildBase,
>>                                   bb.event.MetadataEvent,
>>                                   bb.event.StampUpdate,
>>                                   bb.event.ConfigParsed,
>>                                   bb.event.MultiConfigParsed,
>>                                   bb.event.RecipeParsed,
>>                                   bb.event.RecipePreFinalise,
>>                                   bb.runqueue.runQueueEvent,
>>                                   bb.event.OperationStarted,
>>                                   bb.event.OperationCompleted,
>>                                   bb.event.OperationProgress,
>>                                   bb.event.DiskFull,
>>                                   bb.event.HeartbeatEvent,
>>                                   bb.build.TaskProgress)):
>>                 continue
>>
>>>
>>>>> For reference, I've been looking at the server abstraction in
>>>>> bitbake and am close to rewriting a large part of bb.server.* and
>>>>> bb.main with a view to simplifying the code structure and making
>>>>> things easier to understand.
>>>> That's great, something var name is not easy for me to understand,
>>>> for example, observe_only vs observer_only. And I think that we need
>>>> a simple way to restart the server, something like bitbake --restart.
>>>>
>>>>>
>>>>>
>>>>> I've noticed I see some new events with my change, equally I think
>>>>> its
>>>>> an event mask issue with my new code...
>>>>>
>>>>> I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-
>>>>> cont
>>>>> rib/commit/?h=rpurdie/wip-
>>>>> rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
>>>>> however I am well aware things don't work properly yet and its full
>>>>> of
>>>>> debug. When finished I should be able to delete server/__init__.py
>>>>> and
>>>>> server/xmlrpc.py.
>>>> I will hold on my patches until you've finished the rewriting, and
>>>> please let me know if anything I can do.
>>>
>>> The changes are now in master-next. I plan to split the patch up and
>>> have been revising it a bit as I find issues. It would be great if you
>>> could apply this locally and see if the issues you've found are still
>>> present after this change and help with stressing the changes a bit,
>>> see what issues we have?
>>>
>>> One thing I know is probably broken at the moment is observe-only
>>> unfortunately. I am tempted to merge the code changes I have as this
>>> gives us a platform to build on which is much more understandable than
>>> the existing codebase even if there are issues like that.
>>
>> Sure, I'd like to test them today.
>>
>> // Robert
>>
>>>
>>> Cheers,
>>>
>>> Richard
>>>
>>>
>>>

Re: [RFC PATCH 1/3] bitbake: knotty.py: add MonitorDiskEvent and RecipeTaskPreProcess

[Richard Purdie]

Hi Robert,

Thanks for these. I've queued some of this, I fixed some things
differently and I have a request for some tweaks please! :)

For: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rbt/ma
ster-next-memres&id=1d7df2f917cbcfa5f21c437ba2f0cc656e256eaa could you
please leave pid in the lockfile. We are going to likely need an easier
way to find the server in future so I'd like to have that there.

With oe-init-build-env-memres, I'm simply planning to delete this. If
someone wants to start a standalone server over xmlrpc, they still can
but going forward most people will just set BB_SERVER_TIMEOUT=X. We can
therefore drop that script (patch queued).

For: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rbt/ma
ster-next-memres&id=78333885f4117e4586336c7aff6b5a9e3a6378b8
could you rename the "server_only" parameter in ProcessServer to
stay_resident please?

I've queued everything in bitbake master-next, I can't update poky
master-next at the moment as a build is in progress.

I'm hopeful the patches are becoming more stable now, not sure how many
of your other issues remain if we remove the memres init script and use
BB_SERVER_TIMEOUT as the preferred way to enable memres?

Cheers,

Richard