bitbake-devel.lists.openembedded.org archive mirror
 help / color / mirror / Atom feed
* [hardknott][PATCH 0/8] Review request
@ 2021-09-21  8:35 Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 1/8] cookerdata: Show error for no BBLAYERS in bblayers.conf Anuj Mittal
                   ` (8 more replies)
  0 siblings, 9 replies; 11+ messages in thread
From: Anuj Mittal @ 2021-09-21  8:35 UTC (permalink / raw)
  To: bitbake-devel

Please review these changes for hardknott/1.50. Tested with oe-core and
no related problems seen.

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2625

Thanks,

Anuj

The following changes since commit 4d83d9103d9225a2ac42c0f3dfada2f5cf140b68:

  build: Catch and error upon circular task references (2021-09-16 22:17:07 +0100)

are available in the Git repository at:

  git://push.openembedded.org/bitbake-contrib stable/1.50-next

Martin Jansa (1):
  cooker/process: Fix typos in exiting message

Richard Purdie (7):
  cookerdata: Show error for no BBLAYERS in bblayers.conf
  cookerdata: Improve missing core layer error message
  data_smart: Improve error display for handled exceptions
  runqueue: Clean up task stats handling
  bitbake-worker: Improve error handling
  cookerdata: Show a readable error for invalid multiconfig name
  runqueue/knotty: Improve UI handling of setscene task counting

 bin/bitbake-worker       | 10 ++++++----
 lib/bb/cooker.py         |  2 +-
 lib/bb/cookerdata.py     |  7 +++++++
 lib/bb/data_smart.py     |  2 ++
 lib/bb/runqueue.py       | 43 +++++++++++++++++++++++-----------------
 lib/bb/server/process.py |  2 +-
 lib/bb/ui/knotty.py      |  2 +-
 lib/bb/ui/uihelper.py    |  4 ++--
 8 files changed, 45 insertions(+), 27 deletions(-)

-- 
2.31.1


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [hardknott][PATCH 1/8] cookerdata: Show error for no BBLAYERS in bblayers.conf
  2021-09-21  8:35 [hardknott][PATCH 0/8] Review request Anuj Mittal
@ 2021-09-21  8:35 ` Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 2/8] cookerdata: Improve missing core layer error message Anuj Mittal
                   ` (7 subsequent siblings)
  8 siblings, 0 replies; 11+ messages in thread
From: Anuj Mittal @ 2021-09-21  8:35 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If there is no BBLAYERS set in bblayers.conf show a more helpful
error and exit.

[YOCTO #14340]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 97183e10faf9862b5d9489d6e2c27ac77c3b697d)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cookerdata.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index 1c1e008c6..24002b879 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -342,6 +342,9 @@ class CookerDataBuilder(object):
             layers = (data.getVar('BBLAYERS') or "").split()
             broken_layers = []
 
+            if not layers:
+                bb.fatal("The bblayers.conf file doesn't contain any BBLAYERS definition")
+
             data = bb.data.createCopy(data)
             approved = bb.utils.approved_variables()
 
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [hardknott][PATCH 2/8] cookerdata: Improve missing core layer error message
  2021-09-21  8:35 [hardknott][PATCH 0/8] Review request Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 1/8] cookerdata: Show error for no BBLAYERS in bblayers.conf Anuj Mittal
@ 2021-09-21  8:35 ` Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 3/8] data_smart: Improve error display for handled exceptions Anuj Mittal
                   ` (6 subsequent siblings)
  8 siblings, 0 replies; 11+ messages in thread
From: Anuj Mittal @ 2021-09-21  8:35 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If the core layer is missing from bblayers.conf, the message the user sees is
hard to understand. Improve it.

[YOCTO #14340]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5815a7258ebb8a989e0c6f5798853559d9413f02)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cookerdata.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index 24002b879..d6206b731 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -399,6 +399,8 @@ class CookerDataBuilder(object):
                 if c in collections_tmp:
                     bb.fatal("Found duplicated BBFILE_COLLECTIONS '%s', check bblayers.conf or layer.conf to fix it." % c)
                 compat = set((data.getVar("LAYERSERIES_COMPAT_%s" % c) or "").split())
+                if compat and not layerseries:
+                    bb.fatal("No core layer found to work with layer '%s'. Missing entry in bblayers.conf?" % c)
                 if compat and not (compat & layerseries):
                     bb.fatal("Layer %s is not compatible with the core layer which only supports these series: %s (layer is compatible with %s)"
                               % (c, " ".join(layerseries), " ".join(compat)))
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [hardknott][PATCH 3/8] data_smart: Improve error display for handled exceptions
  2021-09-21  8:35 [hardknott][PATCH 0/8] Review request Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 1/8] cookerdata: Show error for no BBLAYERS in bblayers.conf Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 2/8] cookerdata: Improve missing core layer error message Anuj Mittal
@ 2021-09-21  8:35 ` Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 4/8] runqueue: Clean up task stats handling Anuj Mittal
                   ` (5 subsequent siblings)
  8 siblings, 0 replies; 11+ messages in thread
From: Anuj Mittal @ 2021-09-21  8:35 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We don't need tracebacks for BBHandledException. Reduces confusing output like:

ERROR: /meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio
ERROR: ExpansionError during parsing /meta/recipes-core/images/core-image-tiny-initramfs.bb
Traceback (most recent call last):
  File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_qa', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']):
                     if task in deps:
    >                    follow_chain(othertask, endtask, chain)
             chain.pop()
  File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']):
                     if task in deps:
    >                    follow_chain(othertask, endtask, chain)
             chain.pop()
  File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_cpio', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']):
                     if task in deps:
    >                    follow_chain(othertask, endtask, chain)
             chain.pop()
  File "/bitbake/lib/bb/build.py", line 1038, in follow_chain(task='do_image_complete', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']):
             if task in chain:
    >            bb.fatal("Circular task dependencies as %s depends itself via the chain %s?!" % (task, " -> ".join(chain)))
             chain.append(task)
  File "/bitbake/lib/bb/__init__.py", line 165, in fatal:
         mainlogger.critical(''.join(args), extra=kwargs)
    >    raise BBHandledException()

to the real error:

ERROR: /media/build1/poky/meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 551d4c0576a0a0c3406000029df9238b312f2263)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/data_smart.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index b4ed62a4e..8291ca65e 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -411,6 +411,8 @@ class DataSmart(MutableMapping):
                 raise
             except bb.parse.SkipRecipe:
                 raise
+            except bb.BBHandledException:
+                raise
             except Exception as exc:
                 tb = sys.exc_info()[2]
                 raise ExpansionError(varname, s, exc).with_traceback(tb) from exc
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [hardknott][PATCH 4/8] runqueue: Clean up task stats handling
  2021-09-21  8:35 [hardknott][PATCH 0/8] Review request Anuj Mittal
                   ` (2 preceding siblings ...)
  2021-09-21  8:35 ` [hardknott][PATCH 3/8] data_smart: Improve error display for handled exceptions Anuj Mittal
@ 2021-09-21  8:35 ` Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 5/8] cooker/process: Fix typos in exiting message Anuj Mittal
                   ` (4 subsequent siblings)
  8 siblings, 0 replies; 11+ messages in thread
From: Anuj Mittal @ 2021-09-21  8:35 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

When we parallelised normal and setscene tasks, the task stats
handling was left separate pending further thought. We had to remove
handling of the setscene tasks from the UI in order to maintain
consistent task numbering.

Currently, "0 of 0" tasks can be shown as setscene tasks execute
until the first normal task runs.

The only use left for sq_stats is in the active task numbers which
we can use the length of sq_ive for instead. We can therefore
drop it and return stats in all cases. This removes the "0 of 0" task
problem since the stats in all normal and setscene tasks matches.

[YOCTO #14479]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit eae6e947e37e18cded053814bd2a268b44fb25cd)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/runqueue.py    | 22 +++++++---------------
 lib/bb/ui/uihelper.py |  2 +-
 2 files changed, 8 insertions(+), 16 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 5ccf755f1..6ce0ce80f 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1736,7 +1736,6 @@ class RunQueueExecute:
         self.sqdone = False
 
         self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
-        self.sq_stats = RunQueueStats(len(self.rqdata.runq_setscene_tids))
 
         for mc in rq.worker:
             rq.worker[mc].pipe.setrunqueueexec(self)
@@ -1820,7 +1819,7 @@ class RunQueueExecute:
     def finish(self):
         self.rq.state = runQueueCleanUp
 
-        active = self.stats.active + self.sq_stats.active
+        active = self.stats.active + len(self.sq_live)
         if active > 0:
             bb.event.fire(runQueueExitWait(active), self.cfgData)
             self.rq.read_workers()
@@ -1853,7 +1852,7 @@ class RunQueueExecute:
         return valid
 
     def can_start_task(self):
-        active = self.stats.active + self.sq_stats.active
+        active = self.stats.active + len(self.sq_live)
         can_start = active < self.number_tasks
         return can_start
 
@@ -1956,7 +1955,7 @@ class RunQueueExecute:
         err = False
         if not self.sqdone:
             logger.debug('We could skip tasks %s', "\n".join(sorted(self.scenequeue_covered)))
-            completeevent = sceneQueueComplete(self.sq_stats, self.rq)
+            completeevent = sceneQueueComplete(self.stats, self.rq)
             bb.event.fire(completeevent, self.cfgData)
         if self.sq_deferred:
             logger.error("Scenequeue had deferred entries: %s" % pprint.pformat(self.sq_deferred))
@@ -2067,7 +2066,7 @@ class RunQueueExecute:
                 self.sq_task_failoutright(task)
                 return True
 
-            startevent = sceneQueueTaskStarted(task, self.sq_stats, self.rq)
+            startevent = sceneQueueTaskStarted(task, self.stats, self.rq)
             bb.event.fire(startevent, self.cfgData)
 
             taskdepdata = self.sq_build_taskdepdata(task)
@@ -2088,7 +2087,6 @@ class RunQueueExecute:
             self.build_stamps2.append(self.build_stamps[task])
             self.sq_running.add(task)
             self.sq_live.add(task)
-            self.sq_stats.taskActive()
             if self.can_start_task():
                 return True
 
@@ -2178,7 +2176,7 @@ class RunQueueExecute:
             if self.can_start_task():
                 return True
 
-        if self.stats.active > 0 or self.sq_stats.active > 0:
+        if self.stats.active > 0 or len(self.sq_live) > 0:
             self.rq.read_workers()
             return self.rq.active_fds()
 
@@ -2520,13 +2518,11 @@ class RunQueueExecute:
                 self.rq.state = runQueueCleanUp
 
     def sq_task_complete(self, task):
-        self.sq_stats.taskCompleted()
-        bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats, self.rq), self.cfgData)
+        bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), self.cfgData)
         self.sq_task_completeoutright(task)
 
     def sq_task_fail(self, task, result):
-        self.sq_stats.taskFailed()
-        bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats, result, self), self.cfgData)
+        bb.event.fire(sceneQueueTaskFailed(task, self.stats, result, self), self.cfgData)
         self.scenequeue_notcovered.add(task)
         self.scenequeue_updatecounters(task, True)
         self.sq_check_taskfail(task)
@@ -2534,8 +2530,6 @@ class RunQueueExecute:
     def sq_task_failoutright(self, task):
         self.sq_running.add(task)
         self.sq_buildable.add(task)
-        self.sq_stats.taskSkipped()
-        self.sq_stats.taskCompleted()
         self.scenequeue_notcovered.add(task)
         self.scenequeue_updatecounters(task, True)
 
@@ -2543,8 +2537,6 @@ class RunQueueExecute:
         self.sq_running.add(task)
         self.sq_buildable.add(task)
         self.sq_task_completeoutright(task)
-        self.sq_stats.taskSkipped()
-        self.sq_stats.taskCompleted()
 
     def sq_build_taskdepdata(self, task):
         def getsetscenedeps(tid):
diff --git a/lib/bb/ui/uihelper.py b/lib/bb/ui/uihelper.py
index 48d808ae2..40c7c5a5e 100644
--- a/lib/bb/ui/uihelper.py
+++ b/lib/bb/ui/uihelper.py
@@ -49,7 +49,7 @@ class BBUIHelper:
             tid = event._fn + ":" + event._task
             removetid(event.pid, tid)
             self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)})
-        elif isinstance(event, bb.runqueue.runQueueTaskStarted):
+        elif isinstance(event, bb.runqueue.runQueueTaskStarted) or isinstance(event, bb.runqueue.sceneQueueTaskStarted):
             self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
             self.tasknumber_total = event.stats.total
             self.needUpdate = True
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [hardknott][PATCH 5/8] cooker/process: Fix typos in exiting message
  2021-09-21  8:35 [hardknott][PATCH 0/8] Review request Anuj Mittal
                   ` (3 preceding siblings ...)
  2021-09-21  8:35 ` [hardknott][PATCH 4/8] runqueue: Clean up task stats handling Anuj Mittal
@ 2021-09-21  8:35 ` Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 6/8] bitbake-worker: Improve error handling Anuj Mittal
                   ` (3 subsequent siblings)
  8 siblings, 0 replies; 11+ messages in thread
From: Anuj Mittal @ 2021-09-21  8:35 UTC (permalink / raw)
  To: bitbake-devel

From: Martin Jansa <Martin.Jansa@gmail.com>

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1ff1ea3880d293b14ce0fc65e3bc4c938d587a2f)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 bin/bitbake-worker       | 2 +-
 lib/bb/cooker.py         | 2 +-
 lib/bb/server/process.py | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/bin/bitbake-worker b/bin/bitbake-worker
index 7765b9368..7d982f90b 100755
--- a/bin/bitbake-worker
+++ b/bin/bitbake-worker
@@ -517,5 +517,5 @@ except BaseException as e:
 worker_thread_exit = True
 worker_thread.join()
 
-workerlog_write("exitting")
+workerlog_write("exiting")
 sys.exit(0)
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index f2beea2d3..89f1fad08 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -382,7 +382,7 @@ class BBCooker:
         try:
             self.prhost = prserv.serv.auto_start(self.data)
         except prserv.serv.PRServiceConfigError as e:
-            bb.fatal("Unable to start PR Server, exitting")
+            bb.fatal("Unable to start PR Server, exiting")
 
         if self.data.getVar("BB_HASHSERVE") == "auto":
             # Create a new hash server bound to a unix domain socket
diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
index a0955722e..07bb785a1 100644
--- a/lib/bb/server/process.py
+++ b/lib/bb/server/process.py
@@ -471,7 +471,7 @@ class BitBakeServer(object):
             try:
                 r = ready.get()
             except EOFError:
-                # Trap the child exitting/closing the pipe and error out
+                # Trap the child exiting/closing the pipe and error out
                 r = None
         if not r or r[0] != "r":
             ready.close()
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [hardknott][PATCH 6/8] bitbake-worker: Improve error handling
  2021-09-21  8:35 [hardknott][PATCH 0/8] Review request Anuj Mittal
                   ` (4 preceding siblings ...)
  2021-09-21  8:35 ` [hardknott][PATCH 5/8] cooker/process: Fix typos in exiting message Anuj Mittal
@ 2021-09-21  8:35 ` Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 7/8] cookerdata: Show a readable error for invalid multiconfig name Anuj Mittal
                   ` (2 subsequent siblings)
  8 siblings, 0 replies; 11+ messages in thread
From: Anuj Mittal @ 2021-09-21  8:35 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If bitbake-worker fails, return an error code showing that. Also
make the thread cleanup code explict in a finally clause as it would
otherwise hang.

[YOCTO #14393]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7e0af70fb53fb13f824ca954b8cc1dffee730233)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 bin/bitbake-worker | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/bin/bitbake-worker b/bin/bitbake-worker
index 7d982f90b..3ca8c1853 100755
--- a/bin/bitbake-worker
+++ b/bin/bitbake-worker
@@ -513,9 +513,11 @@ except BaseException as e:
         import traceback
         sys.stderr.write(traceback.format_exc())
         sys.stderr.write(str(e))
-
-worker_thread_exit = True
-worker_thread.join()
+finally:
+    worker_thread_exit = True
+    worker_thread.join()
 
 workerlog_write("exiting")
+if not normalexit:
+    sys.exit(1)
 sys.exit(0)
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [hardknott][PATCH 7/8] cookerdata: Show a readable error for invalid multiconfig name
  2021-09-21  8:35 [hardknott][PATCH 0/8] Review request Anuj Mittal
                   ` (5 preceding siblings ...)
  2021-09-21  8:35 ` [hardknott][PATCH 6/8] bitbake-worker: Improve error handling Anuj Mittal
@ 2021-09-21  8:35 ` Anuj Mittal
  2021-09-21  8:35 ` [hardknott][PATCH 8/8] runqueue/knotty: Improve UI handling of setscene task counting Anuj Mittal
  2021-09-21 17:22 ` [bitbake-devel] [hardknott][PATCH 0/8] Review request Martin Jansa
  8 siblings, 0 replies; 11+ messages in thread
From: Anuj Mittal @ 2021-09-21  8:35 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If a multiconfig starts with a digit, users would see pages of
errors as we use the multiconfig as a python function name prefix
and python functions cannot start with a digit. We could avoid doing
that but it is easier just to ask users to name multiconfigs not
starting with digits.

This tweak ensures the user sees an easier to understand error.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f9cddaeef35b2ea0dadf717101ed896f6b857abd)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cookerdata.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index d6206b731..ba657c03b 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -291,6 +291,8 @@ class CookerDataBuilder(object):
 
             multiconfig = (self.data.getVar("BBMULTICONFIG") or "").split()
             for config in multiconfig:
+                if config[0].isdigit():
+                    bb.fatal("Multiconfig name '%s' is invalid as multiconfigs cannot start with a digit" % config)
                 mcdata = self.parseConfigurationFiles(self.prefiles, self.postfiles, config)
                 bb.event.fire(bb.event.ConfigParsed(), mcdata)
                 self.mcdata[config] = mcdata
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [hardknott][PATCH 8/8] runqueue/knotty: Improve UI handling of setscene task counting
  2021-09-21  8:35 [hardknott][PATCH 0/8] Review request Anuj Mittal
                   ` (6 preceding siblings ...)
  2021-09-21  8:35 ` [hardknott][PATCH 7/8] cookerdata: Show a readable error for invalid multiconfig name Anuj Mittal
@ 2021-09-21  8:35 ` Anuj Mittal
  2021-09-21 17:22 ` [bitbake-devel] [hardknott][PATCH 0/8] Review request Martin Jansa
  8 siblings, 0 replies; 11+ messages in thread
From: Anuj Mittal @ 2021-09-21  8:35 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The recent fixes to merge setscene and normal task accounting in runqueue
fixed some display issues but broke the task numbering of setscene tasks.

Add new accounting methods to the stats structure specifically designed
for setscene. This accounts for the fact that setscene tasks can rerun
multiple times in the build.

Then use the new data in the UI to correctly display the numbers the
user wants to see to understand progress.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ed7e2da88bf4b7bfc7ebfc12b9bd6c0fb7d8c1aa)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/runqueue.py    | 21 ++++++++++++++++++---
 lib/bb/ui/knotty.py   |  2 +-
 lib/bb/ui/uihelper.py |  2 +-
 3 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 6ce0ce80f..10511a09d 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -85,15 +85,19 @@ class RunQueueStats:
     """
     Holds statistics on the tasks handled by the associated runQueue
     """
-    def __init__(self, total):
+    def __init__(self, total, setscene_total):
         self.completed = 0
         self.skipped = 0
         self.failed = 0
         self.active = 0
+        self.setscene_active = 0
+        self.setscene_covered = 0
+        self.setscene_notcovered = 0
+        self.setscene_total = setscene_total
         self.total = total
 
     def copy(self):
-        obj = self.__class__(self.total)
+        obj = self.__class__(self.total, self.setscene_total)
         obj.__dict__.update(self.__dict__)
         return obj
 
@@ -112,6 +116,13 @@ class RunQueueStats:
     def taskActive(self):
         self.active = self.active + 1
 
+    def updateCovered(self, covered, notcovered):
+        self.setscene_covered = covered
+        self.setscene_notcovered = notcovered
+
+    def updateActiveSetscene(self, active):
+        self.setscene_active = active
+
 # These values indicate the next step due to be run in the
 # runQueue state machine
 runQueuePrepare = 2
@@ -1735,7 +1746,7 @@ class RunQueueExecute:
         self.holdoff_need_update = True
         self.sqdone = False
 
-        self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
+        self.stats = RunQueueStats(len(self.rqdata.runtaskentries), len(self.rqdata.runq_setscene_tids))
 
         for mc in rq.worker:
             rq.worker[mc].pipe.setrunqueueexec(self)
@@ -1786,6 +1797,7 @@ class RunQueueExecute:
             else:
                 self.sq_task_complete(task)
             self.sq_live.remove(task)
+            self.stats.updateActiveSetscene(len(self.sq_live))
         else:
             if status != 0:
                 self.task_fail(task, status, fakerootlog=fakerootlog)
@@ -2087,6 +2099,7 @@ class RunQueueExecute:
             self.build_stamps2.append(self.build_stamps[task])
             self.sq_running.add(task)
             self.sq_live.add(task)
+            self.stats.updateActiveSetscene(len(self.sq_live))
             if self.can_start_task():
                 return True
 
@@ -2462,6 +2475,7 @@ class RunQueueExecute:
                 self.sq_task_failoutright(tid)
 
         if changed:
+            self.stats.updateCovered(len(self.scenequeue_covered), len(self.scenequeue_notcovered))
             self.holdoff_need_update = True
 
     def scenequeue_updatecounters(self, task, fail=False):
@@ -2495,6 +2509,7 @@ class RunQueueExecute:
                         new.add(dep)
             next = new
 
+        self.stats.updateCovered(len(self.scenequeue_covered), len(self.scenequeue_notcovered))
         self.holdoff_need_update = True
 
     def sq_task_completeoutright(self, task):
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 0efa614df..04285eac6 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -745,7 +745,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
                 continue
 
             if isinstance(event, bb.runqueue.sceneQueueTaskStarted):
-                logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring))
+                logger.info("Running setscene task %d of %d (%s)" % (event.stats.setscene_covered + event.stats.setscene_active + event.stats.setscene_notcovered + 1, event.stats.setscene_total, event.taskstring))
                 continue
 
             if isinstance(event, bb.runqueue.runQueueTaskStarted):
diff --git a/lib/bb/ui/uihelper.py b/lib/bb/ui/uihelper.py
index 40c7c5a5e..52fdae3fe 100644
--- a/lib/bb/ui/uihelper.py
+++ b/lib/bb/ui/uihelper.py
@@ -50,7 +50,7 @@ class BBUIHelper:
             removetid(event.pid, tid)
             self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)})
         elif isinstance(event, bb.runqueue.runQueueTaskStarted) or isinstance(event, bb.runqueue.sceneQueueTaskStarted):
-            self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
+            self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + event.stats.setscene_active + 1
             self.tasknumber_total = event.stats.total
             self.needUpdate = True
         elif isinstance(event, bb.build.TaskProgress):
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* Re: [bitbake-devel] [hardknott][PATCH 0/8] Review request
  2021-09-21  8:35 [hardknott][PATCH 0/8] Review request Anuj Mittal
                   ` (7 preceding siblings ...)
  2021-09-21  8:35 ` [hardknott][PATCH 8/8] runqueue/knotty: Improve UI handling of setscene task counting Anuj Mittal
@ 2021-09-21 17:22 ` Martin Jansa
  2021-09-21 17:30   ` Steve Sakoman
  8 siblings, 1 reply; 11+ messages in thread
From: Martin Jansa @ 2021-09-21 17:22 UTC (permalink / raw)
  To: Anuj Mittal, Steve Sakoman; +Cc: bitbake-devel

[-- Attachment #1: Type: text/plain, Size: 1943 bytes --]

LGTM

For next round of bitbake patches, can you please include 3 bitbake changes
from:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14542
?

All 3 (fc58ad84a9deb2620ad90611684dad65dafedb11
e2c1afda4cb8023ed4ffeb5dc5bee4f0055659a8
9eee9fd4f2f96789ad2b037e74d561bdc1426856) apply cleanly on bitbake 1.46,
1.48 and 1.50.

+Steve Sakoman <steve@sakoman.com> to consider for dunfell as well.

Thanks

On Tue, Sep 21, 2021 at 10:35 AM Anuj Mittal <anuj.mittal@intel.com> wrote:

> Please review these changes for hardknott/1.50. Tested with oe-core and
> no related problems seen.
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2625
>
> Thanks,
>
> Anuj
>
> The following changes since commit
> 4d83d9103d9225a2ac42c0f3dfada2f5cf140b68:
>
>   build: Catch and error upon circular task references (2021-09-16
> 22:17:07 +0100)
>
> are available in the Git repository at:
>
>   git://push.openembedded.org/bitbake-contrib stable/1.50-next
>
> Martin Jansa (1):
>   cooker/process: Fix typos in exiting message
>
> Richard Purdie (7):
>   cookerdata: Show error for no BBLAYERS in bblayers.conf
>   cookerdata: Improve missing core layer error message
>   data_smart: Improve error display for handled exceptions
>   runqueue: Clean up task stats handling
>   bitbake-worker: Improve error handling
>   cookerdata: Show a readable error for invalid multiconfig name
>   runqueue/knotty: Improve UI handling of setscene task counting
>
>  bin/bitbake-worker       | 10 ++++++----
>  lib/bb/cooker.py         |  2 +-
>  lib/bb/cookerdata.py     |  7 +++++++
>  lib/bb/data_smart.py     |  2 ++
>  lib/bb/runqueue.py       | 43 +++++++++++++++++++++++-----------------
>  lib/bb/server/process.py |  2 +-
>  lib/bb/ui/knotty.py      |  2 +-
>  lib/bb/ui/uihelper.py    |  4 ++--
>  8 files changed, 45 insertions(+), 27 deletions(-)
>
> --
> 2.31.1
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 2866 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [bitbake-devel] [hardknott][PATCH 0/8] Review request
  2021-09-21 17:22 ` [bitbake-devel] [hardknott][PATCH 0/8] Review request Martin Jansa
@ 2021-09-21 17:30   ` Steve Sakoman
  0 siblings, 0 replies; 11+ messages in thread
From: Steve Sakoman @ 2021-09-21 17:30 UTC (permalink / raw)
  To: Martin Jansa; +Cc: Anuj Mittal, bitbake-devel

On Tue, Sep 21, 2021 at 7:23 AM Martin Jansa <martin.jansa@gmail.com> wrote:
>
> LGTM
>
> For next round of bitbake patches, can you please include 3 bitbake changes from:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14542
> ?
>
> All 3 (fc58ad84a9deb2620ad90611684dad65dafedb11 e2c1afda4cb8023ed4ffeb5dc5bee4f0055659a8 9eee9fd4f2f96789ad2b037e74d561bdc1426856) apply cleanly on bitbake 1.46, 1.48 and 1.50.
>
> +Steve Sakoman to consider for dunfell as well.

I've been testing those (and a few others), so they will be in my next
set of bitbake patches for review.

Steve

>
> Thanks
>
> On Tue, Sep 21, 2021 at 10:35 AM Anuj Mittal <anuj.mittal@intel.com> wrote:
>>
>> Please review these changes for hardknott/1.50. Tested with oe-core and
>> no related problems seen.
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2625
>>
>> Thanks,
>>
>> Anuj
>>
>> The following changes since commit 4d83d9103d9225a2ac42c0f3dfada2f5cf140b68:
>>
>>   build: Catch and error upon circular task references (2021-09-16 22:17:07 +0100)
>>
>> are available in the Git repository at:
>>
>>   git://push.openembedded.org/bitbake-contrib stable/1.50-next
>>
>> Martin Jansa (1):
>>   cooker/process: Fix typos in exiting message
>>
>> Richard Purdie (7):
>>   cookerdata: Show error for no BBLAYERS in bblayers.conf
>>   cookerdata: Improve missing core layer error message
>>   data_smart: Improve error display for handled exceptions
>>   runqueue: Clean up task stats handling
>>   bitbake-worker: Improve error handling
>>   cookerdata: Show a readable error for invalid multiconfig name
>>   runqueue/knotty: Improve UI handling of setscene task counting
>>
>>  bin/bitbake-worker       | 10 ++++++----
>>  lib/bb/cooker.py         |  2 +-
>>  lib/bb/cookerdata.py     |  7 +++++++
>>  lib/bb/data_smart.py     |  2 ++
>>  lib/bb/runqueue.py       | 43 +++++++++++++++++++++++-----------------
>>  lib/bb/server/process.py |  2 +-
>>  lib/bb/ui/knotty.py      |  2 +-
>>  lib/bb/ui/uihelper.py    |  4 ++--
>>  8 files changed, 45 insertions(+), 27 deletions(-)
>>
>> --
>> 2.31.1
>>
>>
>> 
>>

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2021-09-21 17:30 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-21  8:35 [hardknott][PATCH 0/8] Review request Anuj Mittal
2021-09-21  8:35 ` [hardknott][PATCH 1/8] cookerdata: Show error for no BBLAYERS in bblayers.conf Anuj Mittal
2021-09-21  8:35 ` [hardknott][PATCH 2/8] cookerdata: Improve missing core layer error message Anuj Mittal
2021-09-21  8:35 ` [hardknott][PATCH 3/8] data_smart: Improve error display for handled exceptions Anuj Mittal
2021-09-21  8:35 ` [hardknott][PATCH 4/8] runqueue: Clean up task stats handling Anuj Mittal
2021-09-21  8:35 ` [hardknott][PATCH 5/8] cooker/process: Fix typos in exiting message Anuj Mittal
2021-09-21  8:35 ` [hardknott][PATCH 6/8] bitbake-worker: Improve error handling Anuj Mittal
2021-09-21  8:35 ` [hardknott][PATCH 7/8] cookerdata: Show a readable error for invalid multiconfig name Anuj Mittal
2021-09-21  8:35 ` [hardknott][PATCH 8/8] runqueue/knotty: Improve UI handling of setscene task counting Anuj Mittal
2021-09-21 17:22 ` [bitbake-devel] [hardknott][PATCH 0/8] Review request Martin Jansa
2021-09-21 17:30   ` Steve Sakoman

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).