[OE-core][dunfell 00/21] Patch review

[OE-core][dunfell 00/21] Patch review

[Steve Sakoman]

Please review this next set of patches for dunfell and have comments back by end
of day Wednesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1807

with the exception of an autobuilder intermittent issue on qemuppc-alt, which
passed on subsequent retest:

https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/1320

The following changes since commit 1988d07b65ad38bdf8fac8615f11fb6536a75806:

  wic: Optimise fstab modification for ext2/3/4 and msdos partitions (2021-01-28 04:41:47 -1000)

are available in the Git repository at:

  git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut

Alexander Kanavin (1):
  oeqa/ptest: print a warning if ptests failed

Dorinda (2):
  sanity: Verify that user isn't building in PSEUDO_IGNORE_PATHS
  sanity.bbclass: sanity check for if bitbake is present in PATH

Oleksiy Obitotskyy (2):
  flex: Fix --noline option behavior
  dtc: improve reproducibility

Oleksiy Obitotskyy yIEf0zt.mo (1):
  toolchain-shar-relocate.sh: Fix handling files with colons

Ovidiu Panait (1):
  timezone: upgrade to 2021a

Richard Purdie (3):
  ncurses: Don't put terminfo into the sysroot
  python3: Avoid installing test data into recipe-sysroot
  staging: Clean up files installed into the sysroot

Ross Burton (3):
  flex: fix build with autoconf 2.70
  core-image-sato-sdk-ptest: these images need ptest
  ovmf-shell-image: image is only buildable on x86-64

Tomasz Dziendzielski (5):
  devtool: Fix source extraction for gcc shared source
  externalsrc: Fix parsing error with devtool non-git sources
  devtool: Fix file:// fetcher symlink directory structure
  selftest/devtool: Add modify_localfiles_only test checking symlink
    path
  externalsrc: Detect code changes in submodules

Yi Fan Yu (2):
  oeqa/selftest/cases/tinfoil.py: increase timeout 10->60s
    test_wait_event
  strace: increase ptest timeout duration 120->240s

zangrc (1):
  flex: Refresh patch

 meta/classes/externalsrc.bbclass              | 16 ++++-
 meta/classes/sanity.bbclass                   | 12 ++++
 meta/classes/staging.bbclass                  |  4 ++
 meta/files/toolchain-shar-relocate.sh         |  2 +-
 meta/lib/oeqa/runtime/cases/ptest.py          |  1 +
 meta/lib/oeqa/selftest/cases/devtool.py       | 37 ++++++++++
 meta/lib/oeqa/selftest/cases/tinfoil.py       |  6 +-
 meta/recipes-core/ncurses/ncurses.inc         |  5 ++
 meta/recipes-core/ovmf/ovmf-shell-image.bb    |  1 +
 ...directives-if-gen_line_dirs-is-false.patch | 32 +++++++++
 .../flex/flex/check-funcs.patch               | 67 +++++++++++++++++++
 meta/recipes-devtools/flex/flex_2.6.4.bb      |  2 +
 meta/recipes-devtools/python/python3_3.8.2.bb |  6 ++
 meta/recipes-devtools/strace/strace/run-ptest |  2 +-
 meta/recipes-extended/timezone/timezone.inc   |  6 +-
 meta/recipes-kernel/dtc/dtc.inc               |  2 +
 ...-Makefile-to-add-CFLAGS-not-override.patch | 36 ++++++++++
 .../images/core-image-sato-ptest-fast.bb      |  3 +
 .../images/core-image-sato-sdk-ptest.bb       |  3 +
 scripts/lib/devtool/standard.py               | 36 ++++++++--
 20 files changed, 266 insertions(+), 13 deletions(-)
 create mode 100644 meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch
 create mode 100644 meta/recipes-devtools/flex/flex/check-funcs.patch
 create mode 100644 meta/recipes-kernel/dtc/dtc/0001-dtc-Fix-Makefile-to-add-CFLAGS-not-override.patch

-- 
2.25.1

[OE-core][dunfell 01/21] sanity: Verify that user isn't building in PSEUDO_IGNORE_PATHS

[Steve Sakoman]

From: Dorinda <dorindabassey@gmail.com>

If a user builds in a path in PSEUDO_IGNORE_PATHS, random failures
are generated. Hence this patch adds a sanity check in sanity.bbclass
to ensure that a user isn't building in PSEUDO_IGNORE_PATHS.

[YOCTO #14179]

Signed-off-by: Dorinda Bassey <dorindabassey@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7a681525e904914e938de25df5cc64209097d15d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/classes/sanity.bbclass | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index cdef8e0c7d..4dc21bf0a9 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -703,6 +703,13 @@ def check_sanity_version_change(status, d):
     if (tmpdirmode & stat.S_ISUID):
         status.addresult("TMPDIR is setuid, please don't build in a setuid directory")
 
+    # Check that a user isn't building in a path in PSEUDO_IGNORE_PATHS
+    pseudoignorepaths = d.getVar('PSEUDO_IGNORE_PATHS', expand=True).split(",")
+    workdir = d.getVar('WORKDIR', expand=True)
+    for i in pseudoignorepaths:
+        if i and workdir.startswith(i):
+            status.addresult("You are building in a path included in PSEUDO_IGNORE_PATHS " + str(i) + " please locate the build outside this path.\n")
+
     # Some third-party software apparently relies on chmod etc. being suid root (!!)
     import stat
     suid_check_bins = "chown chmod mknod".split()
-- 
2.25.1

[OE-core][dunfell 02/21] sanity.bbclass: sanity check for if bitbake is present in PATH

[Steve Sakoman]

From: Dorinda <dorindabassey@gmail.com>

If a user executes the environment script instead of sourcing it,
there's an error about an empty element in PATH. This is because
bitbake isn't present in environment variable PATH. Hence, this
patch adds a sanity check to verify if bitbake is present in
PATH and if bitbake isn't present issue a warning message.

[YOCTO #13822]

Signed-off-by: Dorinda Bassey <dorindabassey@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e08799913a7f207bc63e085eb98196fd61ed57bc)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/classes/sanity.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 4dc21bf0a9..b678284554 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -794,6 +794,11 @@ def check_sanity_everybuild(status, d):
     if "." in paths or "./" in paths or "" in paths:
         status.addresult("PATH contains '.', './' or '' (empty element), which will break the build, please remove this.\nParsed PATH is " + str(paths) + "\n")
 
+    #Check if bitbake is present in PATH environment variable
+    bb_check = bb.utils.which(d.getVar('PATH'), 'bitbake')
+    if not bb_check:
+        bb.warn("bitbake binary is not found in PATH, did you source the script?")
+
     # Check whether 'inherit' directive is found (used for a class to inherit)
     # in conf file it's supposed to be uppercase INHERIT
     inherit = d.getVar('inherit')
-- 
2.25.1

[OE-core][dunfell 03/21] oeqa/ptest: print a warning if ptests failed

[Steve Sakoman]

From: Alexander Kanavin <alex.kanavin@gmail.com>

This allows spotting ptest regressions without having hard ptest failures
(for that full ptest stability should be achieved).

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 79b54d902df2788b0822b6c9cc14705ab00e6a5b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
(cherry picked from commit 5232c0b480eba4537c8db71c95189950011fb64d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/lib/oeqa/runtime/cases/ptest.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/lib/oeqa/runtime/cases/ptest.py b/meta/lib/oeqa/runtime/cases/ptest.py
index ef0470da7e..2066d009c3 100644
--- a/meta/lib/oeqa/runtime/cases/ptest.py
+++ b/meta/lib/oeqa/runtime/cases/ptest.py
@@ -104,4 +104,5 @@ class PtestRunnerTest(OERuntimeTestCase):
             failmsg = failmsg + "Failed ptests:\n%s" % pprint.pformat(failed_tests)
 
         if failmsg:
+            self.logger.warning("There were failing ptests.")
             self.fail(failmsg)
-- 
2.25.1

[OE-core][dunfell 04/21] flex: fix build with autoconf 2.70

[Steve Sakoman]

From: Ross Burton <ross.burton@intel.com>

autoconf 2.70 is coming soon which has some small behavioural fixes, so
backport a patch from upstream to fix the build with that release.

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 946610f2cc8cd42265ee3fab6c73ad4831aaeb4e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 .../flex/flex/check-funcs.patch               | 79 +++++++++++++++++++
 meta/recipes-devtools/flex/flex_2.6.4.bb      |  1 +
 2 files changed, 80 insertions(+)
 create mode 100644 meta/recipes-devtools/flex/flex/check-funcs.patch

diff --git a/meta/recipes-devtools/flex/flex/check-funcs.patch b/meta/recipes-devtools/flex/flex/check-funcs.patch
new file mode 100644
index 0000000000..037ca81fdc
--- /dev/null
+++ b/meta/recipes-devtools/flex/flex/check-funcs.patch
@@ -0,0 +1,79 @@
+Upstream-Status: Backport
+Signed-off-by: Ross Burton <ross.burton@arm.com>
+
+From c42de062bbdc7c31d7181c10a74202d493280ada Mon Sep 17 00:00:00 2001
+From: Explorer09 <explorer09@gmail.com>
+Date: Tue, 27 Feb 2018 09:10:12 +0800
+Subject: [PATCH] build: Move dnl comments out of AC_CHECK_FUNCS
+
+Due to a bug, autoheader (2.69) will treat M4 dnl comments in a quoted
+argument of AC_CHECK_FUNCS as function tokens and generate a lot of
+redundant and useless HAVE_* macros in config.h.in.
+(Examples: HAVE_DNL, HAVE_AVAILABLE_, HAVE_BY)
+
+It seems to be this commit dbb4e94dc7bacbcfd4acef4f085ef752fe1aa03f of
+mine that revealed this autoheader bug, and the affected config.h.in
+had been shipped within flex-2.6.4 release tarball.
+
+I have reported the autoheader bug here:
+<https://lists.gnu.org/archive/html/bug-autoconf/2018-02/msg00005.html>
+
+As a workaround, let's move comments out of AC_CHECK_FUNCS.
+
+Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
+---
+ configure.ac | 29 +++++++++++++----------------
+ 1 file changed, 13 insertions(+), 16 deletions(-)
+
+diff --git a/configure.ac b/configure.ac
+index 55e774b0..d0f3b7da 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -166,6 +166,7 @@ strtol dnl
+ AC_CHECK_FUNCS([dnl
+ pow dnl           Used only by "examples/manual/expr"
+ setlocale dnl     Needed only if NLS is enabled
++reallocarr dnl    NetBSD function. Use reallocarray if not available.
+ reallocarray dnl  OpenBSD function. We have replacement if not available.
+ ])
+ 
+diff --git a/configure.ac b/configure.ac
+index 3c977a4e..9c53590f 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -158,22 +158,19 @@ AC_FUNC_REALLOC
+ AS_IF([test "$cross_compiling" = yes],
+    AC_MSG_WARN([result $ac_cv_func_realloc_0_nonnull guessed because of cross compilation]))
+ 
+-AC_CHECK_FUNCS([dup2 dnl
+-memset dnl
+-regcomp dnl
+-strcasecmp dnl
+-strchr dnl
+-strdup dnl
+-strtol dnl
+-], [], [AC_MSG_ERROR(required library function not found on your system)])
+-
+-# Optional library functions
+-AC_CHECK_FUNCS([dnl
+-pow dnl           Used only by "examples/manual/expr"
+-setlocale dnl     Needed only if NLS is enabled
+-reallocarr dnl    NetBSD function. Use reallocarray if not available.
+-reallocarray dnl  OpenBSD function. We have replacement if not available.
+-])
++dnl Autoheader (<= 2.69) bug: "dnl" comments in a quoted argument of
++dnl AC_CHECK_FUNCS will expand wierdly in config.h.in.
++dnl (https://lists.gnu.org/archive/html/bug-autoconf/2018-02/msg00005.html)
++
++AC_CHECK_FUNCS([dup2 memset regcomp strcasecmp strchr strdup strtol], [],
++  [AC_MSG_ERROR(required library function not found on your system)])
++
++# Optional library functions:
++# pow - Used only by "examples/manual/expr".
++# setlocale - Needed only if NLS is enabled.
++# reallocarr - NetBSD function. Use reallocarray if not available.
++# reallocarray - OpenBSD function. We have replacement if not available.
++AC_CHECK_FUNCS([pow setlocale reallocarr reallocarray])
+ 
+ AC_CONFIG_FILES(
+ Makefile
diff --git a/meta/recipes-devtools/flex/flex_2.6.4.bb b/meta/recipes-devtools/flex/flex_2.6.4.bb
index 43b2547fc6..3d57572865 100644
--- a/meta/recipes-devtools/flex/flex_2.6.4.bb
+++ b/meta/recipes-devtools/flex/flex_2.6.4.bb
@@ -15,6 +15,7 @@ SRC_URI = "https://github.com/westes/flex/releases/download/v${PV}/flex-${PV}.ta
            file://0001-tests-add-a-target-for-building-tests-without-runnin.patch \
            ${@bb.utils.contains('PTEST_ENABLED', '1', '', 'file://disable-tests.patch', d)} \
            file://0001-build-AC_USE_SYSTEM_EXTENSIONS-in-configure.ac.patch \
+           file://check-funcs.patch \
            "
 
 SRC_URI[md5sum] = "2882e3179748cc9f9c23ec593d6adc8d"
-- 
2.25.1

[OE-core][dunfell 05/21] flex: Refresh patch

[Steve Sakoman]

From: zangrc <zangrc.fnst@cn.fujitsu.com>

The patch changes the same file twice which causes issues with some patch
tools (not used in YP by default). Refresh the patch to clean this up
anyway.

RP: Tweak commit message
Signed-off-by: Zang Ruochen <zangrc.fnst@cn.fujitsu.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cb6e65f8196fd6587c75c3ab2cf060c00a074ec6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 .../flex/flex/check-funcs.patch               | 36 +++++++------------
 1 file changed, 12 insertions(+), 24 deletions(-)

diff --git a/meta/recipes-devtools/flex/flex/check-funcs.patch b/meta/recipes-devtools/flex/flex/check-funcs.patch
index 037ca81fdc..762275e7f8 100644
--- a/meta/recipes-devtools/flex/flex/check-funcs.patch
+++ b/meta/recipes-devtools/flex/flex/check-funcs.patch
@@ -1,10 +1,4 @@
-Upstream-Status: Backport
-Signed-off-by: Ross Burton <ross.burton@arm.com>
-
-From c42de062bbdc7c31d7181c10a74202d493280ada Mon Sep 17 00:00:00 2001
-From: Explorer09 <explorer09@gmail.com>
-Date: Tue, 27 Feb 2018 09:10:12 +0800
-Subject: [PATCH] build: Move dnl comments out of AC_CHECK_FUNCS
+Subject: build: Move dnl comments out of AC_CHECK_FUNCS
 
 Due to a bug, autoheader (2.69) will treat M4 dnl comments in a quoted
 argument of AC_CHECK_FUNCS as function tokens and generate a lot of
@@ -20,28 +14,20 @@ I have reported the autoheader bug here:
 
 As a workaround, let's move comments out of AC_CHECK_FUNCS.
 
+Upstream-Status: Backport
+Signed-off-by: Ross Burton <ross.burton@arm.com>
+
 Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
+Signed-off-by: Zang Ruochen <zangrc.fnst@cn.fujitsu.com>
 ---
- configure.ac | 29 +++++++++++++----------------
- 1 file changed, 13 insertions(+), 16 deletions(-)
+ configure.ac | 28 +++++++++++++---------------
+ 1 file changed, 13 insertions(+), 15 deletions(-)
 
 diff --git a/configure.ac b/configure.ac
-index 55e774b0..d0f3b7da 100644
+index 55e774b..5ea3a93 100644
 --- a/configure.ac
 +++ b/configure.ac
-@@ -166,6 +166,7 @@ strtol dnl
- AC_CHECK_FUNCS([dnl
- pow dnl           Used only by "examples/manual/expr"
- setlocale dnl     Needed only if NLS is enabled
-+reallocarr dnl    NetBSD function. Use reallocarray if not available.
- reallocarray dnl  OpenBSD function. We have replacement if not available.
- ])
- 
-diff --git a/configure.ac b/configure.ac
-index 3c977a4e..9c53590f 100644
---- a/configure.ac
-+++ b/configure.ac
-@@ -158,22 +158,19 @@ AC_FUNC_REALLOC
+@@ -153,21 +153,19 @@ AC_FUNC_REALLOC
  AS_IF([test "$cross_compiling" = yes],
     AC_MSG_WARN([result $ac_cv_func_realloc_0_nonnull guessed because of cross compilation]))
  
@@ -58,7 +44,6 @@ index 3c977a4e..9c53590f 100644
 -AC_CHECK_FUNCS([dnl
 -pow dnl           Used only by "examples/manual/expr"
 -setlocale dnl     Needed only if NLS is enabled
--reallocarr dnl    NetBSD function. Use reallocarray if not available.
 -reallocarray dnl  OpenBSD function. We have replacement if not available.
 -])
 +dnl Autoheader (<= 2.69) bug: "dnl" comments in a quoted argument of
@@ -77,3 +62,6 @@ index 3c977a4e..9c53590f 100644
  
  AC_CONFIG_FILES(
  Makefile
+-- 
+2.25.1
+
-- 
2.25.1

[OE-core][dunfell 06/21] flex: Fix --noline option behavior

[Steve Sakoman]

From: Oleksiy Obitotskyy <oobitots@cisco.com>

Option --noline or -L does not handled properly.
So generated code contains #line directives with
file absolute path and prevents to create
reproducible builds.

Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0bf2cb7dc2123f220accf1542c2ae4c4b4b8275a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 ...directives-if-gen_line_dirs-is-false.patch | 32 +++++++++++++++++++
 meta/recipes-devtools/flex/flex_2.6.4.bb      |  1 +
 2 files changed, 33 insertions(+)
 create mode 100644 meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch

diff --git a/meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch b/meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch
new file mode 100644
index 0000000000..c8202b6bd5
--- /dev/null
+++ b/meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch
@@ -0,0 +1,32 @@
+From 440f3f55739468cd26e22f31871eca8cbbd53294 Mon Sep 17 00:00:00 2001
+From: Oleksiy Obitotskyy <oobitots@cisco.com>
+Date: Wed, 6 Jan 2021 06:12:14 -0800
+Subject: [PATCH] Emit no #line directives if gen_line_dirs is false
+
+If we set --noline we should not print line directives.
+But setting --noline means gen_line_dirs is false.
+
+Upstream-Status: Submitted
+Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
+---
+ src/buf.c | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/src/buf.c b/src/buf.c
+index 185083c..4439e28 100644
+--- a/src/buf.c
++++ b/src/buf.c
+@@ -95,8 +95,8 @@ struct Buf *buf_linedir (struct Buf *buf, const char* filename, int lineno)
+     const char *src;
+     size_t tsz;
+ 
+-    if (gen_line_dirs)
+-	return buf;
++    if (!gen_line_dirs)
++        return buf;
+ 
+     tsz = strlen("#line \"\"\n")                +   /* constant parts */
+                2 * strlen (filename)            +   /* filename with possibly all backslashes escaped */
+-- 
+2.26.2.Cisco
+
diff --git a/meta/recipes-devtools/flex/flex_2.6.4.bb b/meta/recipes-devtools/flex/flex_2.6.4.bb
index 3d57572865..1d43d2228a 100644
--- a/meta/recipes-devtools/flex/flex_2.6.4.bb
+++ b/meta/recipes-devtools/flex/flex_2.6.4.bb
@@ -16,6 +16,7 @@ SRC_URI = "https://github.com/westes/flex/releases/download/v${PV}/flex-${PV}.ta
            ${@bb.utils.contains('PTEST_ENABLED', '1', '', 'file://disable-tests.patch', d)} \
            file://0001-build-AC_USE_SYSTEM_EXTENSIONS-in-configure.ac.patch \
            file://check-funcs.patch \
+           file://0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch \
            "
 
 SRC_URI[md5sum] = "2882e3179748cc9f9c23ec593d6adc8d"
-- 
2.25.1

[OE-core][dunfell 07/21] toolchain-shar-relocate.sh: Fix handling files with colons

[Steve Sakoman]

From: "Oleksiy Obitotskyy yIEf0zt.mo" <oobitots@cisco.com>

Files could contain colons in name and we should not use
colons (':') as field separator. E.g. perl/python man
pages packages.

Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 50d2ed0689f1aed6f33b4992d37e2e991c99eb07)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/files/toolchain-shar-relocate.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/files/toolchain-shar-relocate.sh b/meta/files/toolchain-shar-relocate.sh
index 94d288ce05..5433741296 100644
--- a/meta/files/toolchain-shar-relocate.sh
+++ b/meta/files/toolchain-shar-relocate.sh
@@ -55,7 +55,7 @@ fi
 for replace in "$target_sdk_dir -maxdepth 1" "$native_sysroot"; do
 	$SUDO_EXEC find $replace -type f
 done | xargs -n100 file | grep ":.*\(ASCII\|script\|source\).*text" | \
-    awk -F':' '{printf "\"%s\"\n", $1}' | \
+    awk -F': ' '{printf "\"%s\"\n", $1}' | \
     grep -Fv -e "$target_sdk_dir/environment-setup-" \
              -e "$target_sdk_dir/relocate_sdk" \
              -e "$target_sdk_dir/post-relocate-setup" \
-- 
2.25.1

[OE-core][dunfell 08/21] devtool: Fix source extraction for gcc shared source

[Steve Sakoman]

From: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>

If do_patch task is disabled then prepare do_configure dependencies to
fetch external sources and create symlink to ${S} in devtool workspace.

[YOCTO #13036]

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9e47319bfe62d289b90f7545a64dbdc1cbde7f1d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/lib/devtool/standard.py | 30 ++++++++++++++++++++++++++----
 1 file changed, 26 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 261d642d4a..f7d8a82117 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -589,6 +589,16 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             else:
                 task = 'do_patch'
 
+                if 'noexec' in (d.getVarFlags(task, False) or []) or 'task' not in (d.getVarFlags(task, False) or []):
+                    logger.info('The %s recipe has %s disabled. Running only '
+                                       'do_configure task dependencies' % (pn, task))
+
+                    if 'depends' in d.getVarFlags('do_configure', False):
+                        pn = d.getVarFlags('do_configure', False)['depends']
+                        pn = pn.replace('${PV}', d.getVar('PV'))
+                        pn = pn.replace('${COMPILERDEP}', d.getVar('COMPILERDEP'))
+                        task = None
+
             # Run the fetch + unpack tasks
             res = tinfoil.build_targets(pn,
                                         task,
@@ -600,6 +610,17 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
         if not res:
             raise DevtoolError('Extracting source for %s failed' % pn)
 
+        if not is_kernel_yocto and ('noexec' in (d.getVarFlags('do_patch', False) or []) or 'task' not in (d.getVarFlags('do_patch', False) or [])):
+            workshareddir = d.getVar('S')
+            if os.path.islink(srctree):
+                os.unlink(srctree)
+
+            os.symlink(workshareddir, srctree)
+
+            # The initial_rev file is created in devtool_post_unpack function that will not be executed if
+            # do_unpack/do_patch tasks are disabled so we have to directly say that source extraction was successful
+            return True, True
+
         try:
             with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
                 initial_rev = f.read()
@@ -847,10 +868,11 @@ def modify(args, config, basepath, workspace):
             if not initial_rev:
                 return 1
             logger.info('Source tree extracted to %s' % srctree)
-            # Get list of commits since this revision
-            (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
-            commits = stdout.split()
-            check_commits = True
+            if os.path.exists(os.path.join(srctree, '.git')):
+                # Get list of commits since this revision
+                (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
+                commits = stdout.split()
+                check_commits = True
         else:
             if os.path.exists(os.path.join(srctree, '.git')):
                 # Check if it's a tree previously extracted by us. This is done
-- 
2.25.1

[OE-core][dunfell 09/21] externalsrc: Fix parsing error with devtool non-git sources

[Steve Sakoman]

From: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>

If srcdir is under poky directory (e.g. devtool poky/build/workspace/sources)
and is not a git repository then ${@srctree_hash_files(d)} will run "git
rev-parse --git-dir" and detect poky directory as git-dir and run "'git', 'add',
'-A', '.'], cwd=s_dir" trying to add srcdir but build dir is in .gitignore and
latest git will fail with "The following paths are ignored by one of your
.gitignore files: build" which will end with "ExpansionError during parsing".

In this commit I added a check if git_dir is the same as git-dir from
TOPDIR (which will detect poky directory) and if yes, then treat srcdir
as non-git sources.

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 95fbac8dcad6c93f4c9737e9fe13e92ab6befa09)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/classes/externalsrc.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index d200129987..259ccbc79f 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -197,6 +197,10 @@ def srctree_hash_files(d, srcdir=None):
     try:
         git_dir = os.path.join(s_dir,
             subprocess.check_output(['git', '-C', s_dir, 'rev-parse', '--git-dir'], stderr=subprocess.DEVNULL).decode("utf-8").rstrip())
+        top_git_dir = os.path.join(s_dir, subprocess.check_output(['git', '-C', d.getVar("TOPDIR"), 'rev-parse', '--git-dir'],
+            stderr=subprocess.DEVNULL).decode("utf-8").rstrip())
+        if git_dir == top_git_dir:
+            git_dir = None
     except subprocess.CalledProcessError:
         pass
 
-- 
2.25.1

[OE-core][dunfell 10/21] oeqa/selftest/cases/tinfoil.py: increase timeout 10->60s test_wait_event

[Steve Sakoman]

From: Yi Fan Yu <yifan.yu@windriver.com>

The test would timeout on autobuilders. This patch increases the
timeout to 60s

The test will now also exit as soon as we receive the 2 expected events
Expected runtime is around 1s if successful

Bug 14158

Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit be02aa9283f805de718badd5ea12c4968da8774f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/lib/oeqa/selftest/cases/tinfoil.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/tinfoil.py b/meta/lib/oeqa/selftest/cases/tinfoil.py
index 206168ed00..a51c6048d3 100644
--- a/meta/lib/oeqa/selftest/cases/tinfoil.py
+++ b/meta/lib/oeqa/selftest/cases/tinfoil.py
@@ -100,9 +100,11 @@ class TinfoilTests(OESelftestTestCase):
             eventreceived = False
             commandcomplete = False
             start = time.time()
-            # Wait for 10s in total so we'd detect spurious heartbeat events for example
+            # Wait for maximum 60s in total so we'd detect spurious heartbeat events for example
             # The test is IO load sensitive too
-            while time.time() - start < 10:
+            while (not (eventreceived == True and commandcomplete == True) 
+                    and (time.time() - start < 60)):
+                # if we received both events (on let's say a good day), we are done  
                 event = tinfoil.wait_event(1)
                 if event:
                     if isinstance(event, bb.command.CommandCompleted):
-- 
2.25.1

[OE-core][dunfell 11/21] devtool: Fix file:// fetcher symlink directory structure

[Steve Sakoman]

From: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>

Add relative path if file is under directory.

[YOCTO #13738]

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 19ddacc1b38f9ebb86a9359963ccc3c707f7125e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/lib/devtool/standard.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index f7d8a82117..7b62b7e7b8 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -474,7 +474,11 @@ def symlink_oelocal_files_srctree(rd,srctree):
                 destpth = os.path.join(srctree, relpth, fn)
                 if os.path.exists(destpth):
                     os.unlink(destpth)
-                os.symlink('oe-local-files/%s' % fn, destpth)
+                if relpth != '.':
+                    back_relpth = os.path.relpath(local_files_dir, root)
+                    os.symlink('%s/oe-local-files/%s/%s' % (back_relpth, relpth, fn), destpth)
+                else:
+                    os.symlink('oe-local-files/%s' % fn, destpth)
                 addfiles.append(os.path.join(relpth, fn))
         if addfiles:
             bb.process.run('git add %s' % ' '.join(addfiles), cwd=srctree)
-- 
2.25.1

[OE-core][dunfell 12/21] dtc: improve reproducibility

[Steve Sakoman]

From: Oleksiy Obitotskyy <oobitots@cisco.com>

Makefile override CFLAGS, so -fmacro/debug-prefix-map
optiions was omitted and binaries contains absolute
patch to sources.

Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b58e808a087bbc0a5abd78fd34bb6f1c0c93ba25)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/recipes-kernel/dtc/dtc.inc               |  2 ++
 ...-Makefile-to-add-CFLAGS-not-override.patch | 36 +++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100644 meta/recipes-kernel/dtc/dtc/0001-dtc-Fix-Makefile-to-add-CFLAGS-not-override.patch

diff --git a/meta/recipes-kernel/dtc/dtc.inc b/meta/recipes-kernel/dtc/dtc.inc
index 0650e3c82e..5da6c24fbf 100644
--- a/meta/recipes-kernel/dtc/dtc.inc
+++ b/meta/recipes-kernel/dtc/dtc.inc
@@ -7,7 +7,9 @@ DEPENDS = "flex-native bison-native"
 
 SRC_URI = "git://git.kernel.org/pub/scm/utils/dtc/dtc.git \
            file://make_install.patch \
+           file://0001-dtc-Fix-Makefile-to-add-CFLAGS-not-override.patch \
            "
+
 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>\d+(\.\d+)+)"
 
 EXTRA_OEMAKE='NO_PYTHON=1 PREFIX="${prefix}" LIBDIR="${libdir}" DESTDIR="${D}"'
diff --git a/meta/recipes-kernel/dtc/dtc/0001-dtc-Fix-Makefile-to-add-CFLAGS-not-override.patch b/meta/recipes-kernel/dtc/dtc/0001-dtc-Fix-Makefile-to-add-CFLAGS-not-override.patch
new file mode 100644
index 0000000000..a2deb12d4b
--- /dev/null
+++ b/meta/recipes-kernel/dtc/dtc/0001-dtc-Fix-Makefile-to-add-CFLAGS-not-override.patch
@@ -0,0 +1,36 @@
+From f0119060ef1b9bd80e2cae487df1e4aedffb0e9b Mon Sep 17 00:00:00 2001
+From: Oleksiy Obitotskyy <oobitots@cisco.com>
+Date: Fri, 22 Jan 2021 09:12:48 +0200
+Subject: [PATCH] dtc: Fix Makefile to add CFLAGS not override
+
+Makefile override CFLAGS not extend them, so some of them
+missing. Sources builds out of kernel tree and probably not all
+options could be used (?). We need at least -fmacro-prefix-map/
+debug-prefix-map to eliminate absolute path in binaries.
+
+Upstream-Status: Pending
+Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
+---
+ Makefile | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/Makefile b/Makefile
+index 35d936f..b5b13cf 100644
+--- a/Makefile
++++ b/Makefile
+@@ -20,10 +20,10 @@ CONFIG_LOCALVERSION =
+ # See libfdt_internal.h for details
+ ASSUME_MASK ?= 0
+ 
+-CPPFLAGS = -I libfdt -I . -DFDT_ASSUME_MASK=$(ASSUME_MASK)
++CPPFLAGS += -I libfdt -I . -DFDT_ASSUME_MASK=$(ASSUME_MASK)
+ WARNINGS = -Wall -Wpointer-arith -Wcast-qual -Wnested-externs \
+ 	-Wstrict-prototypes -Wmissing-prototypes -Wredundant-decls -Wshadow
+-CFLAGS = -g -Os $(SHAREDLIB_CFLAGS) -Werror $(WARNINGS) $(EXTRA_CFLAGS)
++CFLAGS += -g -Os $(SHAREDLIB_CFLAGS) -Werror $(WARNINGS) $(EXTRA_CFLAGS)
+ 
+ BISON = bison
+ LEX = flex
+-- 
+2.25.1
+
-- 
2.25.1

[OE-core][dunfell 13/21] core-image-sato-sdk-ptest: these images need ptest

[Steve Sakoman]

From: Ross Burton <ross@burtonini.com>

As this image isn't buildable without ptest (the packages won't exist),
depend on the ptest DISTRO_FEATURE to ensure we don't try and build it.

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0fe856d726c6d9c35533e32e70fbe05ef2b88b17)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/recipes-sato/images/core-image-sato-ptest-fast.bb | 3 +++
 meta/recipes-sato/images/core-image-sato-sdk-ptest.bb  | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/meta/recipes-sato/images/core-image-sato-ptest-fast.bb b/meta/recipes-sato/images/core-image-sato-ptest-fast.bb
index 3641217306..4f08d6eb64 100644
--- a/meta/recipes-sato/images/core-image-sato-ptest-fast.bb
+++ b/meta/recipes-sato/images/core-image-sato-ptest-fast.bb
@@ -1,3 +1,6 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES = "ptest"
+
 require core-image-sato-sdk.bb
 require conf/distro/include/ptest-packagelists.inc
 
diff --git a/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb b/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb
index bf749acd79..4d59c9536b 100644
--- a/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb
+++ b/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb
@@ -1,3 +1,6 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES = "ptest"
+
 require core-image-sato-sdk.bb
 require conf/distro/include/ptest-packagelists.inc
 
-- 
2.25.1

[OE-core][dunfell 14/21] ovmf-shell-image: image is only buildable on x86-64

[Steve Sakoman]

From: Ross Burton <ross@burtonini.com>

This image is only buildable for x86-64, so add a COMPATIBLE assignment
to ensure it isn't attempted on others.

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit bdd8208675c8a0c0232c678804a8b62cd74f1d48)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/recipes-core/ovmf/ovmf-shell-image.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/ovmf/ovmf-shell-image.bb b/meta/recipes-core/ovmf/ovmf-shell-image.bb
index 0d2b8bf52f..fd4fb5b732 100644
--- a/meta/recipes-core/ovmf/ovmf-shell-image.bb
+++ b/meta/recipes-core/ovmf/ovmf-shell-image.bb
@@ -1,4 +1,5 @@
 DESCRIPTION = "boot image with UEFI shell and tools"
+COMPATIBLE_HOST_class-target='(i.86|x86_64).*'
 
 # For this image recipe, only the wic format with a
 # single vfat partition makes sense. Because we have no
-- 
2.25.1

[OE-core][dunfell 15/21] selftest/devtool: Add modify_localfiles_only test checking symlink path

[Steve Sakoman]

From: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>

If recipe uses only file:// fetcher devtool unpacks sources under
oe-local-files/ and adds symlink in source root directory. We need to
verify if the symlink in subdirectory has correct path.

See [YOCTO #13738] for details.

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 044c90d152bfd1b2a23272df39327ba4cd862dff)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/lib/oeqa/selftest/cases/devtool.py | 37 +++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index ddf46547de..0985434238 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -693,7 +693,44 @@ class DevtoolModifyTests(DevtoolBase):
 
         self.assertTrue(bbclassextended, 'None of these recipes are BBCLASSEXTENDed to native - need to adjust testrecipes list: %s' % ', '.join(testrecipes))
         self.assertTrue(inheritnative, 'None of these recipes do "inherit native" - need to adjust testrecipes list: %s' % ', '.join(testrecipes))
+    def test_devtool_modify_localfiles_only(self):
+        # Check preconditions
+        testrecipe = 'base-files'
+        src_uri = (get_bb_var('SRC_URI', testrecipe) or '').split()
+        foundlocalonly = False
+        correct_symlink = False
+        for item in src_uri:
+            if item.startswith('file://'):
+                if '.patch' not in item:
+                    foundlocalonly = True
+            else:
+                foundlocalonly = False
+                break
+        self.assertTrue(foundlocalonly, 'This test expects the %s recipe to fetch local files only and it seems that it no longer does' % testrecipe)
+        # Clean up anything in the workdir/sysroot/sstate cache
+        bitbake('%s -c cleansstate' % testrecipe)
+        # Try modifying a recipe
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake -c clean %s' % testrecipe)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
+        srcfile = os.path.join(tempdir, 'oe-local-files/share/dot.bashrc')
+        srclink = os.path.join(tempdir, 'share/dot.bashrc')
+        self.assertExists(srcfile, 'Extracted source could not be found')
+        if os.path.islink(srclink) and os.path.exists(srclink) and os.path.samefile(srcfile, srclink):
+            correct_symlink = True
+        self.assertTrue(correct_symlink, 'Source symlink to oe-local-files is broken')
 
+        matches = glob.glob(os.path.join(self.workspacedir, 'appends', '%s_*.bbappend' % testrecipe))
+        self.assertTrue(matches, 'bbappend not created')
+        # Test devtool status
+        result = runCmd('devtool status')
+        self.assertIn(testrecipe, result.output)
+        self.assertIn(tempdir, result.output)
+        # Try building
+        bitbake(testrecipe)
 
     def test_devtool_modify_git(self):
         # Check preconditions
-- 
2.25.1

[OE-core][dunfell 16/21] timezone: upgrade to 2021a

[Steve Sakoman]

From: Ovidiu Panait <ovidiu.panait@windriver.com>

Release 2021a - 2021-01-24 10:54:57 -0800

  Changes to future timestamps

    South Sudan changes from +03 to +02 on 2021-02-01 at 00:00.
    (Thanks to Steffen Thorsen.)

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ed9114df2086c4ad0544cf99c9c1ff8fb7b830b9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/recipes-extended/timezone/timezone.inc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-extended/timezone/timezone.inc b/meta/recipes-extended/timezone/timezone.inc
index 9a19093e24..a89560b424 100644
--- a/meta/recipes-extended/timezone/timezone.inc
+++ b/meta/recipes-extended/timezone/timezone.inc
@@ -6,7 +6,7 @@ SECTION = "base"
 LICENSE = "PD & BSD & BSD-3-Clause"
 LIC_FILES_CHKSUM = "file://LICENSE;md5=c679c9d6b02bc2757b3eaf8f53c43fba"
 
-PV = "2020f"
+PV = "2021a"
 
 SRC_URI =" http://www.iana.org/time-zones/repository/releases/tzcode${PV}.tar.gz;name=tzcode \
            http://www.iana.org/time-zones/repository/releases/tzdata${PV}.tar.gz;name=tzdata \
@@ -14,5 +14,5 @@ SRC_URI =" http://www.iana.org/time-zones/repository/releases/tzcode${PV}.tar.gz
 
 UPSTREAM_CHECK_URI = "http://www.iana.org/time-zones"
 
-SRC_URI[tzcode.sha256sum] = "cfeeea2a7745164f64bd9f6d76e47916f4ac820c4434493674adbbd4324329c5"
-SRC_URI[tzdata.sha256sum] = "121131918c3ae6dc5d40f0eb87563a2be920b71a76e2392c09519a5e4a666881"
+SRC_URI[tzcode.sha256sum] = "eb46bfa124b5b6bd13d61a609bfde8351bd192894708d33aa06e5c1e255802d0"
+SRC_URI[tzdata.sha256sum] = "39e7d2ba08c68cbaefc8de3227aab0dec2521be8042cf56855f7dc3a9fb14e08"
-- 
2.25.1

[OE-core][dunfell 17/21] strace: increase ptest timeout duration 120->240s

[Steve Sakoman]

From: Yi Fan Yu <yifan.yu@windriver.com>

solve qual_fault-syscall.test and qual_fault.test
failing due to timeout.

Bug 14165

Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5af9f32d9b12654793289f44366251f978f6378a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/recipes-devtools/strace/strace/run-ptest | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/strace/strace/run-ptest b/meta/recipes-devtools/strace/strace/run-ptest
index 4660207220..3a51fb0be9 100755
--- a/meta/recipes-devtools/strace/strace/run-ptest
+++ b/meta/recipes-devtools/strace/strace/run-ptest
@@ -1,5 +1,5 @@
 #!/bin/sh
-export TIMEOUT_DURATION=120
+export TIMEOUT_DURATION=240
 chown nobody tests
 chown nobody tests/*
 chown nobody ../ptest
-- 
2.25.1

[OE-core][dunfell 18/21] externalsrc: Detect code changes in submodules

[Steve Sakoman]

From: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>

The srctree_hash was calculated only from main source directory ignoring
changes in submodules.

[YOCTO #13748]

Use submodule--helper to determine all submodules, and calculate hash
from all git tree objects names combined.

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 50ff9afb3990bcf60b4fa1f937506cb84028c32d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/classes/externalsrc.bbclass | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 259ccbc79f..1d7300d65b 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -190,6 +190,7 @@ def srctree_hash_files(d, srcdir=None):
     import shutil
     import subprocess
     import tempfile
+    import hashlib
 
     s_dir = srcdir or d.getVar('EXTERNALSRC')
     git_dir = None
@@ -214,7 +215,16 @@ def srctree_hash_files(d, srcdir=None):
             env = os.environ.copy()
             env['GIT_INDEX_FILE'] = tmp_index.name
             subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
-            sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
+            git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
+            submodule_helper = subprocess.check_output(['git', 'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8")
+            for line in submodule_helper.splitlines():
+                module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
+                proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+                proc.communicate()
+                proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+                stdout, _ = proc.communicate()
+                git_sha1 += stdout.decode("utf-8")
+            sha1 = hashlib.sha1(git_sha1.encode("utf-8")).hexdigest()
         with open(oe_hash_file, 'w') as fobj:
             fobj.write(sha1)
         ret = oe_hash_file + ':True'
-- 
2.25.1

[OE-core][dunfell 19/21] ncurses: Don't put terminfo into the sysroot

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

This recudes the file count from ~2850 to ~100 which is a huge win
for reducing build directory clutter, its unlikely anything uses the
terminfo data or man pages in the sysroot. This is especially helpful
as we usually end up with two copies of these sets of files.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 443633dfc20177ef88a388d96745675817510c99)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/recipes-core/ncurses/ncurses.inc | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/recipes-core/ncurses/ncurses.inc b/meta/recipes-core/ncurses/ncurses.inc
index 630c5f26d6..7f1834f0dc 100644
--- a/meta/recipes-core/ncurses/ncurses.inc
+++ b/meta/recipes-core/ncurses/ncurses.inc
@@ -319,3 +319,8 @@ FILES_${PN}-terminfo-base = "\
 
 RSUGGESTS_${PN}-libtinfo = "${PN}-terminfo"
 RRECOMMENDS_${PN}-libtinfo = "${PN}-terminfo-base"
+
+# Putting terminfo into the sysroot adds around 2800 files to
+# each recipe specific sysroot. We can live without this, particularly
+# as many recipes may have native and target copies.
+SYSROOT_DIRS_remove = "${datadir}"
-- 
2.25.1

[OE-core][dunfell 20/21] python3: Avoid installing test data into recipe-sysroot

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

There are several thousand files in the test directory which we don't need.
Adding these for the native and target sysroots is a crazy amount of files
to be throwing around needlessly. Delete the files from the sysroot side
of things to tidy up the sysroots and improve performance.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f6bced03011ad1663d68b0322a2f8aeb4d836646)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/recipes-devtools/python/python3_3.8.2.bb | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/recipes-devtools/python/python3_3.8.2.bb b/meta/recipes-devtools/python/python3_3.8.2.bb
index 9eddad4ad4..a86542e547 100644
--- a/meta/recipes-devtools/python/python3_3.8.2.bb
+++ b/meta/recipes-devtools/python/python3_3.8.2.bb
@@ -360,3 +360,9 @@ RDEPENDS_${PN}-dev = ""
 
 RDEPENDS_${PN}-tests_append_class-target = " bash"
 RDEPENDS_${PN}-tests_append_class-nativesdk = " bash"
+
+# Python's tests contain large numbers of files we don't need in the recipe sysroots
+SYSROOT_PREPROCESS_FUNCS += " py3_sysroot_cleanup"
+py3_sysroot_cleanup () {
+	rm -rf ${SYSROOT_DESTDIR}${libdir}/python${PYTHON_MAJMIN}/test
+}
-- 
2.25.1

[OE-core][dunfell 21/21] staging: Clean up files installed into the sysroot

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

There are a variety of files being installed into $datadir which we
don't need. Pick the top "offenders" which amount of thousands of files
and simply don't install them. These include things like test data,
terminfo data, locale data for native tools and so on. This saves
copying these files into native and target sysroots and should improve
performance (smaller sstate, fewer files to copy around).

With this and the python recipe change, alsa-tools went from:

recipe-sysroot: 18357
recipe-sysroot-native: 14129

to

recipe-sysroot: 10809
recipe-sysroot-native: 8079

which is a decent improvement.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 366c72941fe1c24d0b1d96df46e13cb9eb4e79d6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/classes/staging.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index 5b04f88b2d..506ce0665e 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -27,11 +27,15 @@ SYSROOT_DIRS_BLACKLIST = " \
     ${mandir} \
     ${docdir} \
     ${infodir} \
+    ${datadir}/X11/locale \
     ${datadir}/applications \
+    ${datadir}/bash-completion \
     ${datadir}/fonts \
     ${datadir}/gtk-doc/html \
+    ${datadir}/installed-tests \
     ${datadir}/locale \
     ${datadir}/pixmaps \
+    ${datadir}/terminfo \
     ${libdir}/${BPN}/ptest \
 "
 
-- 
2.25.1