All of lore.kernel.org
 help / color / mirror / Atom feed
* [1.50][PATCH 0/5] Review request
@ 2021-09-15 23:19 Anuj Mittal
  2021-09-15 23:19 ` [1.50][PATCH 1/5] runqueue: Avoid deadlock avoidance task graph corruption Anuj Mittal
                   ` (4 more replies)
  0 siblings, 5 replies; 7+ messages in thread
From: Anuj Mittal @ 2021-09-15 23:19 UTC (permalink / raw)
  To: bitbake-devel

Please review these changes for 1.50. Tested with oe-core and no issues
seen.

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2592

Thanks,

Anuj

The following changes since commit c7b506d11df78cfc4610db6578745eaa6220b13a:

  server: Fix early parsing errors preventing zombie bitbake (2021-09-09 09:41:51 +0100)

are available in the Git repository at:

  git://push.openembedded.org/bitbake-contrib stable/1.50-next

Joshua Watt (1):
  cooker: Allow upstream for local hash equivalence server

Richard Purdie (4):
  runqueue: Avoid deadlock avoidance task graph corruption
  runqueue: Fix issues with multiconfig deferred task deadlock messages
  runqueue: Improve multiconfig deferred task issues
  build: Catch and error upon circular task references

 lib/bb/build.py    |  2 ++
 lib/bb/cooker.py   |  7 ++++-
 lib/bb/runqueue.py | 67 +++++++++++++++++++++++++++-------------------
 3 files changed, 48 insertions(+), 28 deletions(-)

-- 
2.31.1


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [1.50][PATCH 1/5] runqueue: Avoid deadlock avoidance task graph corruption
  2021-09-15 23:19 [1.50][PATCH 0/5] Review request Anuj Mittal
@ 2021-09-15 23:19 ` Anuj Mittal
  2021-09-15 23:19 ` [1.50][PATCH 2/5] runqueue: Fix issues with multiconfig deferred task deadlock messages Anuj Mittal
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 7+ messages in thread
From: Anuj Mittal @ 2021-09-15 23:19 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If the deferred task deadlock avoidance code triggers, it could mark an executed
task as failed which leads to "covered and not covered" error messages. Improve
the logic so if the deadlock code is triggered, it doesn't cause the errors.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 51bdd6cb3bd9e2c02e261fb578bb945b86b82c75)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/runqueue.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 6c41fe6d4..014ee37bf 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2180,7 +2180,8 @@ class RunQueueExecute:
         if self.sq_deferred:
             tid = self.sq_deferred.pop(list(self.sq_deferred.keys())[0])
             logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s" % tid)
-            self.sq_task_failoutright(tid)
+            if tid not in self.runq_complete:
+                self.sq_task_failoutright(tid)
             return True
 
         if len(self.failed_tids) != 0:
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [1.50][PATCH 2/5] runqueue: Fix issues with multiconfig deferred task deadlock messages
  2021-09-15 23:19 [1.50][PATCH 0/5] Review request Anuj Mittal
  2021-09-15 23:19 ` [1.50][PATCH 1/5] runqueue: Avoid deadlock avoidance task graph corruption Anuj Mittal
@ 2021-09-15 23:19 ` Anuj Mittal
  2021-09-15 23:19 ` [1.50][PATCH 3/5] cooker: Allow upstream for local hash equivalence server Anuj Mittal
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 7+ messages in thread
From: Anuj Mittal @ 2021-09-15 23:19 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

In multiconfig builds with large numbers of identical tasks, builds were
deadlocking after recent runqueue changes upon rebuilds where there was
heavy sstate usage (i.e. on second builds after a first completed).

The issue was that deferred tasks were being left indefinitely on
the deferred list. The deadlock handler was then "breaking" things
by failing tasks that had already succeeded, leading to the task
being on both covered and not covered lists, giving a further error.

The fix is to clean up the deferred task list when each setscene task
completes. I'd previously been hoping to avoid iterating that list
but it appears unavoidable.

[YOCTO #14342]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ae24a0f2d2d8b4b5ec10efabd0e9362e560832ea)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/runqueue.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 014ee37bf..f67981309 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1904,6 +1904,12 @@ class RunQueueExecute:
                 self.setbuildable(revdep)
                 logger.debug("Marking task %s as buildable", revdep)
 
+        for t in self.sq_deferred.copy():
+            if self.sq_deferred[t] == task:
+                logger.debug2("Deferred task %s now buildable" % t)
+                del self.sq_deferred[t]
+                update_scenequeue_data([t], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False)
+
     def task_complete(self, task):
         self.stats.taskCompleted()
         bb.event.fire(runQueueTaskCompleted(task, self.stats, self.rq), self.cfgData)
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [1.50][PATCH 3/5] cooker: Allow upstream for local hash equivalence server
  2021-09-15 23:19 [1.50][PATCH 0/5] Review request Anuj Mittal
  2021-09-15 23:19 ` [1.50][PATCH 1/5] runqueue: Avoid deadlock avoidance task graph corruption Anuj Mittal
  2021-09-15 23:19 ` [1.50][PATCH 2/5] runqueue: Fix issues with multiconfig deferred task deadlock messages Anuj Mittal
@ 2021-09-15 23:19 ` Anuj Mittal
  2021-09-15 23:19 ` [1.50][PATCH 4/5] runqueue: Improve multiconfig deferred task issues Anuj Mittal
  2021-09-15 23:19 ` [1.50][PATCH 5/5] build: Catch and error upon circular task references Anuj Mittal
  4 siblings, 0 replies; 7+ messages in thread
From: Anuj Mittal @ 2021-09-15 23:19 UTC (permalink / raw)
  To: bitbake-devel

From: Joshua Watt <jpewhacker@gmail.com>

The hash equivalence server has had the option to support a read-only
upstream server for some time now when launched as a standalone program,
but there was no way to set the upstream when using a locally started
server. Add a new variable called BB_HASHSERVE_UPSTREAM that can be used
to specify an upstream server when a local hash equivalence server is
used (e.g. BB_HASHSERVE is "auto")

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 250fa17f1391ff1ee01ab9b51d2a4f9aa35c1d1e)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cooker.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 39e10e613..f2beea2d3 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -389,7 +389,12 @@ class BBCooker:
             if not self.hashserv:
                 dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db"
                 self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR")
-                self.hashserv = hashserv.create_server(self.hashservaddr, dbfile, sync=False)
+                self.hashserv = hashserv.create_server(
+                    self.hashservaddr,
+                    dbfile,
+                    sync=False,
+                    upstream=self.data.getVar("BB_HASHSERVE_UPSTREAM") or None,
+                )
                 self.hashserv.process = multiprocessing.Process(target=self.hashserv.serve_forever)
                 self.hashserv.process.start()
             self.data.setVar("BB_HASHSERVE", self.hashservaddr)
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [1.50][PATCH 4/5] runqueue: Improve multiconfig deferred task issues
  2021-09-15 23:19 [1.50][PATCH 0/5] Review request Anuj Mittal
                   ` (2 preceding siblings ...)
  2021-09-15 23:19 ` [1.50][PATCH 3/5] cooker: Allow upstream for local hash equivalence server Anuj Mittal
@ 2021-09-15 23:19 ` Anuj Mittal
  2021-09-15 23:19 ` [1.50][PATCH 5/5] build: Catch and error upon circular task references Anuj Mittal
  4 siblings, 0 replies; 7+ messages in thread
From: Anuj Mittal @ 2021-09-15 23:19 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The previous patches have exposed new issues with this code path,
the issues being around what should happen when the hash of a task
changes and the task is or is not on the deferred task list.

Rather than rebuilding the deferred task list during each rehash
event, build it once at the start of a build. This avoids the problem
of tasks being added back after they have run and also avoids problems
of always ensuring the same task is deferred. It also allows the
'outrightfail' codepath to be handled separately as the conditions
are subtly differnt.

One significant win for the new approch is the build is not continually
printing out lists of deferred tasks, that list remains fairly static
from the start of the build. Logic is added in to ensure a rehashed
task with a hash matching other deferred tasks is deferred along with
them as a small optimization.

An interesting test case for this code was reported by Mark Hatle
with four multiconfigs, each the same apart from TMPDIR and running a
build of:

bitbake buildtools-tarball mc:{one,two,three,four}:core-image-minimal

which is interesting in that the build of buildtools partially overlaps
core-image-minimal and the build has a rehash event for qemuwrapper-cross
even without any external hash equivalence server or preexisting data.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit bb424e0a6d274d398f434f7df63951da9ce305b3)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/runqueue.py | 58 +++++++++++++++++++++++++---------------------
 1 file changed, 32 insertions(+), 26 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index f67981309..5ccf755f1 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2450,6 +2450,11 @@ class RunQueueExecute:
 
         if update_tasks:
             self.sqdone = False
+            for tid in [t[0] for t in update_tasks]:
+                h = pending_hash_index(tid, self.rqdata)
+                if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]:
+                    self.sq_deferred[tid] = self.sqdata.hashes[h]
+                    bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h]))
             update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False)
 
         for (tid, harddepfail, origvalid) in update_tasks:
@@ -2793,6 +2798,19 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
     sqdata.stamppresent = set()
     sqdata.valid = set()
 
+    sqdata.hashes = {}
+    sqrq.sq_deferred = {}
+    for mc in sorted(sqdata.multiconfigs):
+        for tid in sorted(sqdata.sq_revdeps):
+            if mc_from_tid(tid) != mc:
+                continue
+            h = pending_hash_index(tid, rqdata)
+            if h not in sqdata.hashes:
+                sqdata.hashes[h] = tid
+            else:
+                sqrq.sq_deferred[tid] = sqdata.hashes[h]
+                bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h]))
+
     update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq, cooker, stampcache, sqrq, summary=True)
 
     # Compute a list of 'stale' sstate tasks where the current hash does not match the one
@@ -2857,32 +2875,20 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s
 
     sqdata.valid |= rq.validate_hashes(tocheck, cooker.data, len(sqdata.stamppresent), False, summary=summary)
 
-    sqdata.hashes = {}
-    sqrq.sq_deferred = {}
-    for mc in sorted(sqdata.multiconfigs):
-        for tid in sorted(sqdata.sq_revdeps):
-            if mc_from_tid(tid) != mc:
-                continue
-            if tid in sqdata.stamppresent:
-                continue
-            if tid in sqdata.valid:
-                continue
-            if tid in sqdata.noexec:
-                continue
-            if tid in sqrq.scenequeue_notcovered:
-                continue
-            if tid in sqrq.scenequeue_covered:
-                continue
-
-            h = pending_hash_index(tid, rqdata)
-            if h not in sqdata.hashes:
-                if tid in tids:
-                    sqdata.outrightfail.add(tid)
-                sqdata.hashes[h] = tid
-            else:
-                sqrq.sq_deferred[tid] = sqdata.hashes[h]
-                bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h]))
-
+    for tid in tids:
+        if tid in sqdata.stamppresent:
+            continue
+        if tid in sqdata.valid:
+            continue
+        if tid in sqdata.noexec:
+            continue
+        if tid in sqrq.scenequeue_covered:
+            continue
+        if tid in sqrq.scenequeue_notcovered:
+            continue
+        if tid in sqrq.sq_deferred:
+            continue
+        sqdata.outrightfail.add(tid)
 
 class TaskFailure(Exception):
     """
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [1.50][PATCH 5/5] build: Catch and error upon circular task references
  2021-09-15 23:19 [1.50][PATCH 0/5] Review request Anuj Mittal
                   ` (3 preceding siblings ...)
  2021-09-15 23:19 ` [1.50][PATCH 4/5] runqueue: Improve multiconfig deferred task issues Anuj Mittal
@ 2021-09-15 23:19 ` Anuj Mittal
  4 siblings, 0 replies; 7+ messages in thread
From: Anuj Mittal @ 2021-09-15 23:19 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If there are circular task references, error on them rather than show
a recursion error. A simple reproducer is:

"""
do_packageswu () {
       :
}

addtask do_packageswu after do_image_complete before do_image_qa
"""

into image_types.bbclass. There is code in runqueue to detect these but
we never get that far with the current codebase.

[YOCTO #13140]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 339d4d6be515a71311b81fb9e99742af0d8a5130)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/build.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index b2715fc53..24ce327c5 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -1025,6 +1025,8 @@ def tasksbetween(task_start, task_end, d):
     def follow_chain(task, endtask, chain=None):
         if not chain:
             chain = []
+        if task in chain:
+            bb.fatal("Circular task dependencies as %s depends on itself via the chain %s" % (task, " -> ".join(chain)))
         chain.append(task)
         for othertask in tasks:
             if othertask == task:
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [1.50][PATCH 0/5] review request
@ 2021-04-26  3:27 Anuj Mittal
  0 siblings, 0 replies; 7+ messages in thread
From: Anuj Mittal @ 2021-04-26  3:27 UTC (permalink / raw)
  To: bitbake-devel

Please review these changes for hardknott/1.50.

Thanks,

Anuj

The following changes since commit b32e6c8d4ea2f83fe77021207e9db883fec82d97:

  bitbake-user-manual: add REQUIRED_VERSION and adjust PREFERRED_VERSION entry (2021-04-15 15:01:47 +0100)

are available in the Git repository at:

  git://push.openembedded.org/bitbake-contrib stable/1.50-next

Mikko Rapeli (2):
  bitbake: tests/fetch: fix test execution without .gitconfig
  bitbake: tests/fetch: remove write protected files too

Niels Avonds (1):
  fetch/gitsm: Fix crash when using git LFS and submodules

Richard Purdie (1):
  runqueue: Fix deferred task issues

Ross Burton (1):
  bitbake-server: ensure server timeout is a float

 bin/bitbake-server       |  2 +-
 lib/bb/fetch2/git.py     |  2 +-
 lib/bb/runqueue.py       |  4 ++++
 lib/bb/server/process.py |  2 +-
 lib/bb/tests/fetch.py    | 20 ++++++++++++++++++--
 5 files changed, 25 insertions(+), 5 deletions(-)

-- 
2.30.2


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-09-15 23:19 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-15 23:19 [1.50][PATCH 0/5] Review request Anuj Mittal
2021-09-15 23:19 ` [1.50][PATCH 1/5] runqueue: Avoid deadlock avoidance task graph corruption Anuj Mittal
2021-09-15 23:19 ` [1.50][PATCH 2/5] runqueue: Fix issues with multiconfig deferred task deadlock messages Anuj Mittal
2021-09-15 23:19 ` [1.50][PATCH 3/5] cooker: Allow upstream for local hash equivalence server Anuj Mittal
2021-09-15 23:19 ` [1.50][PATCH 4/5] runqueue: Improve multiconfig deferred task issues Anuj Mittal
2021-09-15 23:19 ` [1.50][PATCH 5/5] build: Catch and error upon circular task references Anuj Mittal
  -- strict thread matches above, loose matches on Subject: below --
2021-04-26  3:27 [1.50][PATCH 0/5] review request Anuj Mittal

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.