[PATCH v3 0/6] Fixes for toastergui

[PATCH v3 0/6] Fixes for toastergui

[Ed Bartosh]

Hi,

This patchset fixes creation of Target objects for command line builds
and couple of usability fixes for toastergui module.

Changes in v2: used better name for the API: get_or_create_targets.
Changes in v3: made sure 'build' task is not set in Target objects.
               rebased pathset on top of poky-contrib:master-next

The following changes since commit f8bd18184a84e6de1976ecb4d0d929c55bc19a66:

  toaster: orm remove the complicated querying on the ORM (2015-09-28 20:07:38 -0700)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/toaster/target-task
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/target-task

Ed Bartosh (6):
  event: add new property to BuildBase class
  event: add new parameter to Build* event APIs
  toaster: add task to the target_information
  toaster: add get_or_create_targets API
  toaster: ignore ReachableStamps event
  toaster: use meaningful logging levels

 bitbake/lib/bb/cooker.py             |  8 ++++----
 bitbake/lib/bb/event.py              | 18 +++++++++++++-----
 bitbake/lib/bb/ui/buildinfohelper.py | 35 +++++++++++++++++++++--------------
 bitbake/lib/bb/ui/toasterui.py       | 11 ++++++-----
 4 files changed, 44 insertions(+), 28 deletions(-)

-- 
2.1.4

[PATCH 1/6] event: add new property to BuildBase class

[Ed Bartosh]

Added 'task' property to the base class of BuildStarted and
BuildCompleted classes to store bitbake task if it's specified.

This is done as without task Build* events can't fully represent
bitbake build. Task information is needed by UI classes to properly
show or process Build* events.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/event.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
index 366bc41..68496e5 100644
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -331,6 +331,7 @@ class BuildBase(Event):
     def __init__(self, n, p, failures = 0):
         self._name = n
         self._pkgs = p
+        self._task = None
         Event.__init__(self)
         self._failures = failures
 
@@ -352,6 +353,12 @@ class BuildBase(Event):
     def setCfg(self, cfg):
         self.data = cfg
 
+    def getTask(self):
+        return self._task
+
+    def setTask(self, task):
+        self._task = task
+
     def getFailures(self):
         """
         Return the number of failed packages
@@ -361,6 +368,7 @@ class BuildBase(Event):
     pkgs = property(getPkgs, setPkgs, None, "pkgs property")
     name = property(getName, setName, None, "name property")
     cfg = property(getCfg, setCfg, None, "cfg property")
+    task = property(getTask, setTask, None, "task property")
 
 
 
-- 
2.1.4

[PATCH 2/6] event: add new parameter to Build* event APIs

[Ed Bartosh]

Added 't'(task) parameter to BuildStarted and BuildCompleted
event APIs. Modified existing calls of above mentioned APIs:
passed bitbake task name as a parameter.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/cooker.py |  8 ++++----
 bitbake/lib/bb/event.py  | 12 ++++++------
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index a0d7d59..ef81c3a 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1313,7 +1313,7 @@ class BBCooker:
         taskdata.add_provider(self.data, self.recipecache, item)
 
         buildname = self.data.getVar("BUILDNAME", True)
-        bb.event.fire(bb.event.BuildStarted(buildname, [item]), self.expanded_data)
+        bb.event.fire(bb.event.BuildStarted(buildname, [item], task), self.expanded_data)
 
         # Execute the runqueue
         if not task.startswith("do_"):
@@ -1345,7 +1345,7 @@ class BBCooker:
                 return False
 
             if not retval:
-                bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, item, failures, interrupted), self.expanded_data)
+                bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, item, task, failures, interrupted), self.expanded_data)
                 self.command.finishAsyncCommand(msg)
                 return False
             if retval is True:
@@ -1381,7 +1381,7 @@ class BBCooker:
                 return False
 
             if not retval:
-                bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, targets, failures, interrupted), self.data)
+                bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, targets, task, failures, interrupted), self.data)
                 self.command.finishAsyncCommand(msg)
                 return False
             if retval is True:
@@ -1394,7 +1394,7 @@ class BBCooker:
         taskdata, runlist, fulltargetlist = self.buildTaskData(targets, task, self.configuration.abort)
 
         buildname = self.data.getVar("BUILDNAME", False)
-        bb.event.fire(bb.event.BuildStarted(buildname, fulltargetlist), self.data)
+        bb.event.fire(bb.event.BuildStarted(buildname, fulltargetlist, task), self.data)
 
         rq = bb.runqueue.RunQueue(self, self.data, self.recipecache, taskdata, runlist)
         if 'universe' in targets:
diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
index 68496e5..bf67ee9 100644
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -328,10 +328,10 @@ class StampUpdate(Event):
 class BuildBase(Event):
     """Base class for bbmake run events"""
 
-    def __init__(self, n, p, failures = 0):
+    def __init__(self, n, p, t, failures = 0):
         self._name = n
         self._pkgs = p
-        self._task = None
+        self._task = t
         Event.__init__(self)
         self._failures = failures
 
@@ -376,19 +376,19 @@ class BuildBase(Event):
 
 class BuildStarted(BuildBase, OperationStarted):
     """bbmake build run started"""
-    def __init__(self, n, p, failures = 0):
+    def __init__(self, n, p, t, failures = 0):
         OperationStarted.__init__(self, "Building Started")
-        BuildBase.__init__(self, n, p, failures)
+        BuildBase.__init__(self, n, p, t, failures)
 
 class BuildCompleted(BuildBase, OperationCompleted):
     """bbmake build run completed"""
-    def __init__(self, total, n, p, failures=0, interrupted=0):
+    def __init__(self, total, n, p, t, failures=0, interrupted=0):
         if not failures:
             OperationCompleted.__init__(self, total, "Building Succeeded")
         else:
             OperationCompleted.__init__(self, total, "Building Failed")
         self._interrupted = interrupted
-        BuildBase.__init__(self, n, p, failures)
+        BuildBase.__init__(self, n, p, t, failures)
 
 class DiskFull(Event):
     """Disk full case build aborted"""
-- 
2.1.4

[PATCH 3/6] toaster: add task to the target_information

[Ed Bartosh]

Added task information from BuildStarted event to the
target_information structure. Made sure 'build' task
is not set in Target objects as it shouldn't be shown
in Toaster UI.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 9fdde29..34a8694 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -925,6 +925,12 @@ class BuildInfoHelper(object):
         target_information = {}
         target_information['targets'] = event._pkgs
         target_information['build'] = build_obj
+        task = getattr(event, 'task', '')
+        if task.startswith('do_'):
+            task = task[3:]
+        if task == 'build':
+            task = ''
+        target_information['task'] = task
 
         self.internal_state['targets'] = self.orm_wrapper.create_target_objects(target_information)
 
-- 
2.1.4

[PATCH 4/6] toaster: add get_or_create_targets API

[Ed Bartosh]

Target objects are created before the build if build is
started from UI in build mode. However, in analysis mode Target
objects don't exist and need to be created using information
from bitbake events.

Added new API call get_or_create_targets to retrive existing
target objects or create them if they don't exist yet.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 34a8694..404f114 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -184,18 +184,19 @@ class ORMWrapper(object):
 
         return build
 
-    def create_target_objects(self, target_info):
-        assert 'build' in target_info
-        assert 'targets' in target_info
-
-        targets = []
-        for tgt_name in target_info['targets']:
-            tgt_object = Target.objects.create( build = target_info['build'],
-                                    target = tgt_name,
-                                    is_image = False,
-                                    )
-            targets.append(tgt_object)
-        return targets
+    @staticmethod
+    def get_or_create_targets(target_info):
+        result = []
+        for target in target_info['targets']:
+            obj, created = Target.objects.get_or_create(build=target_info['build'],
+                                                        target=target)
+            if created:
+                obj.is_image = False
+                if target_info['task']:
+                    obj.task = target_info['task']
+                obj.save()
+            result.append(obj)
+        return result
 
     def update_build_object(self, build, errors, warnings, taskfailures):
         assert isinstance(build,Build)
@@ -932,7 +933,7 @@ class BuildInfoHelper(object):
             task = ''
         target_information['task'] = task
 
-        self.internal_state['targets'] = self.orm_wrapper.create_target_objects(target_information)
+        self.internal_state['targets'] = self.orm_wrapper.get_or_create_targets(target_information)
 
         # Save build configuration
         data = self.server.runCommand(["getAllKeysWithFlags", ["doc", "func"]])[0]
-- 
2.1.4

[PATCH 5/6] toaster: ignore ReachableStamps event

[Ed Bartosh]

Added ReachableStamps event to the list of known events to
ignore. This should stop toaster throwing error message:
ERROR: Unknown event: <bb.event.ReachableStamps>

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/toasterui.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index e0c278b..3891e63 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -308,7 +308,8 @@ def main(server, eventHandler, params ):
                                   bb.event.OperationProgress,
                                   bb.command.CommandFailed,
                                   bb.command.CommandExit,
-                                  bb.command.CommandCompleted)):
+                                  bb.command.CommandCompleted,
+                                  bb.event.ReachableStamps)):
                 continue
 
             if isinstance(event, bb.event.DepTreeGenerated):
-- 
2.1.4

[PATCH 6/6] toaster: use meaningful logging levels

[Ed Bartosh]

Changed logging levels to more appropriate ones.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/toasterui.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index 3891e63..dbe0d09 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -130,14 +130,14 @@ def main(server, eventHandler, params ):
 
             if isinstance(event, (bb.build.TaskStarted, bb.build.TaskSucceeded, bb.build.TaskFailedSilent)):
                 buildinfohelper.update_and_store_task(event)
-                logger.warn("Logfile for task %s", event.logfile)
+                logger.info("Logfile for task %s", event.logfile)
                 continue
 
             if isinstance(event, bb.build.TaskBase):
                 logger.info(event._message)
 
             if isinstance(event, bb.event.LogExecTTY):
-                logger.warn(event.msg)
+                logger.info(event.msg)
                 continue
 
             if isinstance(event, logging.LogRecord):
@@ -316,7 +316,7 @@ def main(server, eventHandler, params ):
                 buildinfohelper.store_dependency_information(event)
                 continue
 
-            logger.error("Unknown event: %s", event)
+            logger.warn("Unknown event: %s", event)
             return_value += 1
 
         except EnvironmentError as ioerror:
@@ -336,7 +336,7 @@ def main(server, eventHandler, params ):
             if tb is not None:
                 curr = tb
                 while curr is not None:
-                    logger.warn("Error data dump %s\n%s\n" , traceback.format_tb(curr,1), pformat(curr.tb_frame.f_locals))
+                    logger.error("Error data dump %s\n%s\n" , traceback.format_tb(curr,1), pformat(curr.tb_frame.f_locals))
                     curr = curr.tb_next
 
             # save them to database, if possible; if it fails, we already logged to console.
-- 
2.1.4

Re: [PATCH v3 0/6] Fixes for toastergui

[Barros Pena, Belen]

On 29/09/2015 08:55, "toaster-bounces@yoctoproject.org on behalf of Ed
Bartosh" <toaster-bounces@yoctoproject.org on behalf of
ed.bartosh@linux.intel.com> wrote:

>Hi,
>
>This patchset fixes creation of Target objects for command line builds
>and couple of usability fixes for toastergui module.
>
>Changes in v2: used better name for the API: get_or_create_targets.
>Changes in v3: made sure 'build' task is not set in Target objects.
>               rebased pathset on top of poky-contrib:master-next
>
>The following changes since commit
>f8bd18184a84e6de1976ecb4d0d929c55bc19a66:
>
>  toaster: orm remove the complicated querying on the ORM (2015-09-28
>20:07:38 -0700)
>
>are available in the git repository at:
>
>  git://git.yoctoproject.org/poky-contrib ed/toaster/target-task
>  
>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/target
>-task

I am a bit confused about what the UI is showing with these changes. This
is what I am seeing:

In analysis (interactive) mode, everything looks good. If I run

bitbake -c clean core-image-minimal

The Toaster UI shows me core-image-minimal:clean while the build is in
progress, and shows the same thing when the build completes

In build (managed) mode, if I run

core-image-minimal:clean

The Toaster UI shows me core-image-minimal:clean while the build is in
progress, but just 'core-image-minimal' once the build completes.

The behaviour in analysis mode is the correct one. I should see
target:task while the build is in progress and when the build completes.

Cheers

Belén


>
>Ed Bartosh (6):
>  event: add new property to BuildBase class
>  event: add new parameter to Build* event APIs
>  toaster: add task to the target_information
>  toaster: add get_or_create_targets API
>  toaster: ignore ReachableStamps event
>  toaster: use meaningful logging levels
>
> bitbake/lib/bb/cooker.py             |  8 ++++----
> bitbake/lib/bb/event.py              | 18 +++++++++++++-----
> bitbake/lib/bb/ui/buildinfohelper.py | 35
>+++++++++++++++++++++--------------
> bitbake/lib/bb/ui/toasterui.py       | 11 ++++++-----
> 4 files changed, 44 insertions(+), 28 deletions(-)
>
>-- 
>2.1.4
>
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster

Re: [PATCH v3 0/6] Fixes for toastergui

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

On 29 September 2015 at 08:55, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> Hi,
>
> This patchset fixes creation of Target objects for command line builds
> and couple of usability fixes for toastergui module.
>
> Changes in v2: used better name for the API: get_or_create_targets.
> Changes in v3: made sure 'build' task is not set in Target objects.
>                rebased pathset on top of poky-contrib:master-next
>

This now works for me *during* the build: in managed mode, if I build
"zlib" (with the default ":build" target), I don't see the ":build" in the
builds pages. Similarly, if I do "zlib:clean", I see the ":clean" target
during the build.

However, when the "zlib:clean" build completes, I no longer see the
":clean" target next to the build name (NB in managed mode).

In analysis mode, if I do "bitbake -c clean zlib", the target ":clean" is
next to the build name both during the build and when it has completed.

Which is correct?

Elliot


>
> The following changes since commit
> f8bd18184a84e6de1976ecb4d0d929c55bc19a66:
>
>   toaster: orm remove the complicated querying on the ORM (2015-09-28
> 20:07:38 -0700)
>
> are available in the git repository at:
>
>   git://git.yoctoproject.org/poky-contrib ed/toaster/target-task
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/target-task
>
> Ed Bartosh (6):
>   event: add new property to BuildBase class
>   event: add new parameter to Build* event APIs
>   toaster: add task to the target_information
>   toaster: add get_or_create_targets API
>   toaster: ignore ReachableStamps event
>   toaster: use meaningful logging levels
>
>  bitbake/lib/bb/cooker.py             |  8 ++++----
>  bitbake/lib/bb/event.py              | 18 +++++++++++++-----
>  bitbake/lib/bb/ui/buildinfohelper.py | 35
> +++++++++++++++++++++--------------
>  bitbake/lib/bb/ui/toasterui.py       | 11 ++++++-----
>  4 files changed, 44 insertions(+), 28 deletions(-)
>
> --
> 2.1.4
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 3622 bytes --]

Re: [PATCH v3 0/6] Fixes for toastergui

[Barros Pena, Belen]

On 30/09/2015 10:57, "toaster-bounces@yoctoproject.org on behalf of Smith,
Elliot" <toaster-bounces@yoctoproject.org on behalf of
elliot.smith@intel.com> wrote:

>On 29 September 2015 at 08:55, Ed Bartosh
><ed.bartosh@linux.intel.com> wrote:
>
>Hi,
>
>This patchset fixes creation of Target objects for command line builds
>and couple of usability fixes for toastergui module.
>
>Changes in v2: used better name for the API: get_or_create_targets.
>Changes in v3: made sure 'build' task is not set in Target objects.
>               rebased pathset on top of poky-contrib:master-next
>
>
>
>
>This now works for me *during* the build: in managed mode, if I build
>"zlib" (with the default ":build" target), I don't see the ":build" in
>the builds pages. Similarly, if I do "zlib:clean", I see the ":clean"
>target during the build.
>
>
>However, when the "zlib:clean" build completes, I no longer see the
>":clean" target next to the build name (NB in managed mode).
>
>
>In analysis mode, if I do "bitbake -c clean zlib", the target ":clean" is
>next to the build name both during the build and when it has completed.
>
>
>Which is correct?

You seem to be seen exactly what I am seeing. The correct behaviour is the
one you are seeing in analysis mode. If a task is specified, the task
should appear next to the target during the build and once the build
completes.

Cheers

Belén

>
>
>Elliot
> 
>
>
>The following changes since commit
>f8bd18184a84e6de1976ecb4d0d929c55bc19a66:
>
>  toaster: orm remove the complicated querying on the ORM (2015-09-28
>20:07:38 -0700)
>
>are available in the git repository at:
>
>  git://git.yoctoproject.org/poky-contrib
><http://git.yoctoproject.org/poky-contrib> ed/toaster/target-task
>  
>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/target
>-task 
><http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/targe
>t-task>
>
>Ed Bartosh (6):
>  event: add new property to BuildBase class
>  event: add new parameter to Build* event APIs
>  toaster: add task to the target_information
>  toaster: add get_or_create_targets API
>  toaster: ignore ReachableStamps event
>  toaster: use meaningful logging levels
>
> bitbake/lib/bb/cooker.py             |  8 ++++----
> bitbake/lib/bb/event.py              | 18 +++++++++++++-----
> bitbake/lib/bb/ui/buildinfohelper.py | 35
>+++++++++++++++++++++--------------
> bitbake/lib/bb/ui/toasterui.py       | 11 ++++++-----
> 4 files changed, 44 insertions(+), 28 deletions(-)
>
>--
>2.1.4
>
>--
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster
>
>
>
>
>
>
>
>-- 
>Elliot Smith
>Software Engineer
>Intel Open Source Technology Centre
>
>
>
>

Re: [PATCH v3 0/6] Fixes for toastergui

[Ed Bartosh]

On Tue, Sep 29, 2015 at 12:50:54PM +0000, Barros Pena, Belen wrote:
> 
> I am a bit confused about what the UI is showing with these changes. This
> is what I am seeing:
> 
> In analysis (interactive) mode, everything looks good. If I run
> 
> bitbake -c clean core-image-minimal
> 
> The Toaster UI shows me core-image-minimal:clean while the build is in
> progress, and shows the same thing when the build completes
> 
> In build (managed) mode, if I run
> 
> core-image-minimal:clean
> 
> The Toaster UI shows me core-image-minimal:clean while the build is in
> progress, but just 'core-image-minimal' once the build completes.
> 
> The behaviour in analysis mode is the correct one. I should see
> target:task while the build is in progress and when the build completes.
> 

It's a bug caused by rebase on top of toaster-next. Sorry, missed it.
I'll send next version ASAP.

--
Regards,
Ed