[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this next set of patches for dunfell/1.46 and have comments back
by end of day Tuesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1962

The following changes since commit e0dd20a006ce6fdc656e6be6ac581e93e2ed4a95:

  __init__.py: Fix bitbake debug log handling (2021-03-09 00:03:26 +0000)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Richard Purdie (2):
  runqueue: Fix task execution corruption issue
  runqueue: Add setscene task overlap sanity check

 lib/bb/runqueue.py | 7 +++++++
 1 file changed, 7 insertions(+)

-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 1/2] runqueue: Fix task execution corruption issue

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We've seen occasional issues where linux-yocto:do_compile_kernelmodules would
run without do_shared_workdir running before it. do_shared_workdir is an
setscene task but never has an sstate object generated so it will always
rerun. This should not happen since compile_kernemodules should only
execute if a setscene that depends on it didn't run and that should trigger
do_shared_workdir not to be marked as covered.

The issue is that build-appliance-image:do_package is one of the tasks which
covers linux-yocto:do_compile_kernelmodules but it is also a noexec task
and has a dependecy on pseudo-native:do_populate_sysroot.

In the problem case, pseudo-native:do_populate_sysroot is unavailable but
marked as covered since it is noexec. The "harddeps" code then also marks it
as notcovered. No task should ever be both covered and notcovered and this
is where the problems come from.

The solution is for the harddeps code only to to fail tasks if they've not
already been handled in some way. The code is assuming code couldn't have
handled revdeps at this point but we now have clear evidence they can.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f66556bbb38449789ceea2fd105e9f68df7fb660)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/runqueue.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 30cab537..6186a83f 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2421,6 +2421,9 @@ class RunQueueExecute:
 
         for dep in sorted(self.sqdata.sq_deps[task]):
             if fail and task in self.sqdata.sq_harddeps and dep in self.sqdata.sq_harddeps[task]:
+                if dep in self.scenequeue_covered or dep in self.scenequeue_notcovered:
+                    # dependency could be already processed, e.g. noexec setscene task
+                    continue
                 logger.debug(2, "%s was unavailable and is a hard dependency of %s so skipping" % (task, dep))
                 self.sq_task_failoutright(dep)
                 continue
-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 2/2] runqueue: Add setscene task overlap sanity check

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We've seen hard to debug issues where a task ends up in both the
covered and notcovered list. Add a sanity check to ensure if this
happens in future, we see it in the logs.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6e001410854792f9bb66a0409a2ac176171b0507)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/runqueue.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 6186a83f..2bb97b6e 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1934,6 +1934,10 @@ class RunQueueExecute:
             logger.error("Scenequeue had holdoff tasks: %s" % pprint.pformat(self.holdoff_tasks))
             err = True
 
+        for tid in self.scenequeue_covered.intersection(self.scenequeue_notcovered):
+            # No task should end up in both covered and uncovered, that is a bug.
+            logger.error("Setscene task %s in both covered and notcovered." % tid)
+
         for tid in self.rqdata.runq_setscene_tids:
             if tid not in self.scenequeue_covered and tid not in self.scenequeue_notcovered:
                 err = True
-- 
2.25.1

Re: [bitbake-devel] [bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1930 bytes --]

On Sat, Mar 13, 2021 at 09:04:50AM -1000, Steve Sakoman wrote:
> Please review this next set of patches for dunfell/1.46 and have comments back
> by end of day Tuesday.
> 
> Passed a-full on autobuilder:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1962
> 
> The following changes since commit e0dd20a006ce6fdc656e6be6ac581e93e2ed4a95:
> 
>   __init__.py: Fix bitbake debug log handling (2021-03-09 00:03:26 +0000)
> 
> are available in the Git repository at:
> 
>   git://git.openembedded.org/bitbake-contrib stable/1.46-nut
>   http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
> 
> Richard Purdie (2):
>   runqueue: Fix task execution corruption issue
>   runqueue: Add setscene task overlap sanity check
> 
>  lib/bb/runqueue.py | 7 +++++++
>  1 file changed, 7 insertions(+)

Following fix from master might be useful for 1.46 and 1.48 as well.

The bebef58b21bdff7a3ee1fa2449b7df19144f26fd mentioned in commit message was introduced in 1.40 and I've included this
change to all my builds since it was sent to ML and haven't seen any issues caused by that (but cannot confirm that
it _completely_ resolve the hangs I was seeing after parsing issues).

commit 915330e1dbae1ee8fd9a0358decf2c294f771961
Author: Jan Brzezanski <jan.brzezanski@gmail.com>
Date:   Thu Feb 18 17:53:32 2021 +0100

    Force parser shutdown after catching an exception

    Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing
    parser shutdown as default in case of build abort.

    In this case bitbake sometimes hangs after facing error during parsing,
    waiting for child processes to finish. Killing it then will spawn zombie
    processes.

    Thus we force the shutdown after catching an exception.

    Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this set of patches for dunfell/1.46 and have comments back by
end of day Tuesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4512

The following changes since commit 8ea8e443005ad92f4ad264d9abd9e90e33fb5c17:

  tests/fetch: Allow handling of a file:// url within a submodule (2022-10-27 16:38:59 +0100)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Richard Purdie (2):
  utils: Handle lockfile filenames that are too long for filesystems
  utils: Fix lockfile path length issues

 lib/bb/utils.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this set of patches for dunfell/1.46 and have comments back by end of
day Tuesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3433

The following changes since commit d22cc1e587c650fd5f90cda32f5720f8a3105aac:

  tests/fetch: Handle upstream master -> main branch change (2022-02-23 18:11:51 +0000)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Richard Purdie (2):
  server/process: Note when commands complete in logs
  tinfoil: Allow run_command not to wait on events

 lib/bb/server/process.py | 1 +
 lib/bb/tinfoil.py        | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this set of patches for dunfell/1.46 and have comments back by
end of day Monday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3112

The following changes since commit c5969eedd035648e3258bade386cc67ce3bb0e03:

  cooker/command: Add a dummy event for tinfoil testing (2022-01-07 23:43:41 +0000)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Richard Purdie (2):
  utils: Update to use exec_module() instead of load_module()
  tests/fetch: Drop gnu urls from wget connectivity test

 lib/bb/tests/fetch.py | 3 ---
 lib/bb/utils.py       | 7 +++++--
 2 files changed, 5 insertions(+), 5 deletions(-)

-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this set of patches for dunfell/1.46 and have comments back by
end of day Tuesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2988

with the exception of a known autobuilder intermittent issue (on qemuppc test)
which passed on subsequent retest:

https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/4402

The following changes since commit 7fdd43c5cbde38daa013076de2fdedcf3c3d3107:

  process/knotty: Improve early exception handling (2021-11-22 04:15:15 -1000)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Richard Purdie (2):
  cooker: Ensure reparsing is handled correctly
  bblayers/action: When adding layers, catch BBHandledException

 lib/bb/cooker.py       | 2 ++
 lib/bblayers/action.py | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this set of patches for 1.46/dunfell and have comments back by
end of day Monday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2876

The following changes since commit c222eddcebe892ae209aea7776cfc1147ac1df6e:

  fetch/git: Handle github dropping git:// support (2021-11-03 11:30:57 +0000)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Richard Purdie (2):
  tests/fetch2: Fix quoting warning
  tests/fetch: Update github urls

 lib/bb/tests/fetch.py | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)

-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this set of patches for dunfell/1.46 and have comments back by
end of day Monday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2787

The following changes since commit 35f01508d687eb44c2dc488cd74a9b35124859d9:

  hashserv: let asyncio discover the running loop (2021-10-21 05:57:38 -1000)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Richard Purdie (2):
  fetch2/git: Avoid races over mirror tarball creation
  fetch2/git: Use os.rename instead of mv

 lib/bb/fetch2/git.py | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this next set of patches for 1.46 (dunfell) and have comments
back by end of day Wednesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1466

The following changes since commit 18e1957337fd9f06bc673d28dd4f8277321d07bc:

  tests/fetch: Move away from problematic freedesktop.org urls (2020-09-17 09:13:32 -1000)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Charlie Davies (2):
  bitbake: fetch/git: add support for SRC_URI containing spaces in url
  bitbake: tests/fetch: add unit tests for SRC_URI with spaces in url

 lib/bb/fetch2/git.py  | 12 +++++------
 lib/bb/tests/fetch.py | 47 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+), 6 deletions(-)

-- 
2.17.1

[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this set of patches for dunfell and have comments back
by end of day Wednesday.

Tested on dunfell autobuilder builds over the past few weeks.

The following changes since commit 838a89141fd6a75da9f982ad2b035e823b692d2a:

  server/process: Account for xmlrpc connections (2020-07-29 00:21:29 +0100)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Jean-Francois Dagenais (1):
  siggen: clean_basepath: remove recipe full path when virtual:xyz
    present

Richard Purdie (1):
  fetch2: Drop cups.org from wget status checks

 lib/bb/siggen.py      | 2 +-
 lib/bb/tests/fetch.py | 4 +---
 2 files changed, 2 insertions(+), 4 deletions(-)

-- 
2.17.1

[bitbake][dunfell][1.46][PATCH 0/2] Patch review

[Steve Sakoman]

These patches cherry-picked from master should fix the toaster issue.

The following changes since commit cc11dfa4eb3616547a8a3909f89da0cc4f35dc57:

  cooker: Handle multiconfig name mappings correctly (2020-07-23 16:50:51 +0100)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Richard Purdie (2):
  server/process: Fix UI first connection tracking
  server/process: Account for xmlrpc connections

 lib/bb/server/process.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

-- 
2.17.1