* [gatesgarth][PATCH 01/20] zip: whitelist CVE-2018-13410 and CVE-2018-13684
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 02/20] scripts: oe-run-native, fix *-native directories Anuj Mittal
` (18 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Mikko Rapeli <mikko.rapeli@bmw.de>
https://nvd.nist.gov/vuln/detail/CVE-2018-13410 is disputed and
also Debian considers it not a vulnerability:
https://security-tracker.debian.org/tracker/CVE-2018-13410
http://seclists.org/fulldisclosure/2018/Jul/24
"Negligible security impact, would involve that a untrusted party controls the -TT value."
https://nvd.nist.gov/vuln/detail/CVE-2018-13684 is not for zip, also Debian concludes this:
https://security-tracker.debian.org/tracker/CVE-2018-13684
"NOT-FOR-US: smart contract implementation for ZIP"
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
meta/recipes-extended/zip/zip_3.0.bb | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/meta/recipes-extended/zip/zip_3.0.bb b/meta/recipes-extended/zip/zip_3.0.bb
index c00a932763..97e5e57533 100644
--- a/meta/recipes-extended/zip/zip_3.0.bb
+++ b/meta/recipes-extended/zip/zip_3.0.bb
@@ -19,6 +19,12 @@ UPSTREAM_VERSION_UNKNOWN = "1"
SRC_URI[md5sum] = "7b74551e63f8ee6aab6fbc86676c0d37"
SRC_URI[sha256sum] = "f0e8bb1f9b7eb0b01285495a2699df3a4b766784c1765a8f1aeedf63c0806369"
+# Disputed and also Debian doesn't consider a vulnerability
+CVE_CHECK_WHITELIST += "CVE-2018-13410"
+
+# Not for zip but for smart contract implementation for it
+CVE_CHECK_WHITELIST += "CVE-2018-13684"
+
# zip.inc sets CFLAGS, but what Makefile actually uses is
# CFLAGS_NOOPT. It will also force -O3 optimization, overriding
# whatever we set.
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 02/20] scripts: oe-run-native, fix *-native directories
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 01/20] zip: whitelist CVE-2018-13410 and CVE-2018-13684 Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 03/20] toolchain-shar-extract.sh: Handle special characters in script path Anuj Mittal
` (17 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Adrian Herrera <adrian.herrera@arm.com>
This fixes a crash with "find" when running a native tool and *-native
directories do not exist under the binary directory in the sysroot.
This happened because the directory wildcard was passed as part of the
root directory.
The directory wildcard is now passed by "-name", which returns an empty
result if no matching directory.
Signed-off-by: Adrian Herrera <adrian.herrera@arm.com>
Change-Id: Iba7acd8bbd7e0beb4d25c984f6af7a4fd21486e6
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f6c90ed0ad24b7d4f892e22e088b1578824eb1d3)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
scripts/oe-run-native | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/oe-run-native b/scripts/oe-run-native
index 4e63e69cc4..22958d97e7 100755
--- a/scripts/oe-run-native
+++ b/scripts/oe-run-native
@@ -43,7 +43,7 @@ fi
OLD_PATH=$PATH
# look for a tool only in native sysroot
-PATH=$OECORE_NATIVE_SYSROOT/usr/bin:$OECORE_NATIVE_SYSROOT/bin:$OECORE_NATIVE_SYSROOT/usr/sbin:$OECORE_NATIVE_SYSROOT/sbin$(find $OECORE_NATIVE_SYSROOT/usr/bin/*-native -maxdepth 1 -type d -printf ":%p")
+PATH=$OECORE_NATIVE_SYSROOT/usr/bin:$OECORE_NATIVE_SYSROOT/bin:$OECORE_NATIVE_SYSROOT/usr/sbin:$OECORE_NATIVE_SYSROOT/sbin$(find $OECORE_NATIVE_SYSROOT/usr/bin -maxdepth 1 -name "*-native" -type d -printf ":%p")
tool_find=`/usr/bin/which $tool 2>/dev/null`
if [ -n "$tool_find" ] ; then
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 03/20] toolchain-shar-extract.sh: Handle special characters in script path
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 01/20] zip: whitelist CVE-2018-13410 and CVE-2018-13684 Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 02/20] scripts: oe-run-native, fix *-native directories Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 04/20] systemd.bbclass: improve error message when a service unit specified in SYSTEMD_SERVICE is not found Anuj Mittal
` (16 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Andrey Mozzhuhin <amozzhuhin@yandex.ru>
Extracting SDK archive may fail if the script is run using a path with
special characters such as space or asterisk. This is because the shell
interprets such characters after expanding the $0 variable.
Added quotes to all uses of the shell variable $0 to fix this.
Signed-off-by: Andrey Mozzhuhin <amozzhuhin@yandex.ru>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0453acbbd45604537090ec7a3295b34309e6eecb)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
| 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
--git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index bea6d4189a..dd9342758b 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -95,7 +95,7 @@ while getopts ":yd:npDRSl" OPT; do
listcontents=1
;;
*)
- echo "Usage: $(basename $0) [-y] [-d <dir>]"
+ echo "Usage: $(basename "$0") [-y] [-d <dir>]"
echo " -y Automatic yes to all prompts"
echo " -d <dir> Install the SDK to <dir>"
echo "======== Extensible SDK only options ============"
@@ -111,17 +111,17 @@ while getopts ":yd:npDRSl" OPT; do
esac
done
-payload_offset=$(($(grep -na -m1 "^MARKER:$" $0|cut -d':' -f1) + 1))
+payload_offset=$(($(grep -na -m1 "^MARKER:$" "$0"|cut -d':' -f1) + 1))
if [ "$listcontents" = "1" ] ; then
if [ @SDK_ARCHIVE_TYPE@ = "zip" ]; then
- tail -n +$payload_offset $0 > sdk.zip
+ tail -n +$payload_offset "$0" > sdk.zip
if unzip -l sdk.zip;then
rm sdk.zip
else
rm sdk.zip && exit 1
fi
else
- tail -n +$payload_offset $0| tar tvJ || exit 1
+ tail -n +$payload_offset "$0"| tar tvJ || exit 1
fi
exit
fi
@@ -242,14 +242,14 @@ fi
printf "Extracting SDK..."
if [ @SDK_ARCHIVE_TYPE@ = "zip" ]; then
- tail -n +$payload_offset $0 > sdk.zip
+ tail -n +$payload_offset "$0" > sdk.zip
if $SUDO_EXEC unzip $EXTRA_TAR_OPTIONS sdk.zip -d $target_sdk_dir;then
rm sdk.zip
else
rm sdk.zip && exit 1
fi
else
- tail -n +$payload_offset $0| $SUDO_EXEC tar mxJ -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1
+ tail -n +$payload_offset "$0"| $SUDO_EXEC tar mxJ -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1
fi
echo "done"
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 04/20] systemd.bbclass: improve error message when a service unit specified in SYSTEMD_SERVICE is not found
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (2 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 03/20] toolchain-shar-extract.sh: Handle special characters in script path Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 05/20] boost: drop arm-intrinsics.patch Anuj Mittal
` (15 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Chris Laplante <mostthingsweb@gmail.com>
The previous message was fairly useless without diving into the bbclass.
Signed-off-by: Chris Laplante <mostthingsweb@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ef6117b148be65536e89409a83cbfd22049c652e)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
meta/classes/systemd.bbclass | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/classes/systemd.bbclass b/meta/classes/systemd.bbclass
index 9e8a82c9f1..a4bff732b9 100644
--- a/meta/classes/systemd.bbclass
+++ b/meta/classes/systemd.bbclass
@@ -174,7 +174,8 @@ python systemd_populate_packages() {
if path_found != '':
systemd_add_files_and_parse(pkg_systemd, path_found, service, keys)
else:
- bb.fatal("SYSTEMD_SERVICE_%s value %s does not exist" % (pkg_systemd, service))
+ bb.fatal("Didn't find service unit '{0}', specified in SYSTEMD_SERVICE_{1}. {2}".format(
+ service, pkg_systemd, "Also looked for service unit '{0}'.".format(base) if base is not None else ""))
def systemd_create_presets(pkg, action):
presetf = oe.path.join(d.getVar("PKGD"), d.getVar("systemd_unitdir"), "system-preset/98-%s.preset" % pkg)
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 05/20] boost: drop arm-intrinsics.patch
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (3 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 04/20] systemd.bbclass: improve error message when a service unit specified in SYSTEMD_SERVICE is not found Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 06/20] binutils: Fix CVE-2020-35448 Anuj Mittal
` (14 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Mans Rullgard <mans@mansr.com>
This patch makes gcc produce broken code. It is unclear why it is there
in the first place. Drop it.
Signed-off-by: Mans Rullgard <mans@mansr.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5f3cace37496fe1dc4fd045f688f7d441505c437)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
.../boost/boost/arm-intrinsics.patch | 55 -------------------
meta/recipes-support/boost/boost_1.74.0.bb | 2 +-
2 files changed, 1 insertion(+), 56 deletions(-)
delete mode 100644 meta/recipes-support/boost/boost/arm-intrinsics.patch
diff --git a/meta/recipes-support/boost/boost/arm-intrinsics.patch b/meta/recipes-support/boost/boost/arm-intrinsics.patch
deleted file mode 100644
index fe85c69a82..0000000000
--- a/meta/recipes-support/boost/boost/arm-intrinsics.patch
+++ /dev/null
@@ -1,55 +0,0 @@
-Upstream-Status: Backport
-
-8/17/2010 - rebased to 1.44 by Qing He <qing.he@intel.com>
-
-diff --git a/boost/smart_ptr/detail/atomic_count_sync.hpp b/boost/smart_ptr/detail/atomic_count_sync.hpp
-index b6359b5..78b1cc2 100644
---- a/boost/smart_ptr/detail/atomic_count_sync.hpp
-+++ b/boost/smart_ptr/detail/atomic_count_sync.hpp
-@@ -33,17 +33,46 @@ public:
-
- long operator++()
- {
-+#ifdef __ARM_ARCH_7A__
-+ int v1, tmp;
-+ asm volatile ("1: \n\t"
-+ "ldrex %0, %1 \n\t"
-+ "add %0 ,%0, #1 \n\t"
-+ "strex %2, %0, %1 \n\t"
-+ "cmp %2, #0 \n\t"
-+ "bne 1b \n\t"
-+ : "=&r" (v1), "+Q"(value_), "=&r"(tmp)
-+ );
-+#else
- return __sync_add_and_fetch( &value_, 1 );
-+#endif
- }
-
- long operator--()
- {
-+#ifdef __ARM_ARCH_7A__
-+ int v1, tmp;
-+ asm volatile ("1: \n\t"
-+ "ldrex %0, %1 \n\t"
-+ "sub %0 ,%0, #1 \n\t"
-+ "strex %2, %0, %1 \n\t"
-+ "cmp %2, #0 \n\t"
-+ "bne 1b \n\t"
-+ : "=&r" (v1), "+Q"(value_), "=&r"(tmp)
-+ );
-+ return value_;
-+#else
- return __sync_add_and_fetch( &value_, -1 );
-+#endif
- }
-
- operator long() const
- {
-+#if __ARM_ARCH_7A__
-+ return value_;
-+#else
- return __sync_fetch_and_add( &value_, 0 );
-+#endif
- }
-
- private:
diff --git a/meta/recipes-support/boost/boost_1.74.0.bb b/meta/recipes-support/boost/boost_1.74.0.bb
index b01b390a59..9188303195 100644
--- a/meta/recipes-support/boost/boost_1.74.0.bb
+++ b/meta/recipes-support/boost/boost_1.74.0.bb
@@ -1,7 +1,7 @@
require boost-${PV}.inc
require boost.inc
-SRC_URI += "file://arm-intrinsics.patch \
+SRC_URI += " \
file://boost-CVE-2012-2677.patch \
file://boost-math-disable-pch-for-gcc.patch \
file://0001-Apply-boost-1.62.0-no-forced-flags.patch.patch \
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 06/20] binutils: Fix CVE-2020-35448
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (4 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 05/20] boost: drop arm-intrinsics.patch Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 07/20] meta: toolchain-shar-relocate.sh: Do not use $target_sdk_dir as regex Anuj Mittal
` (13 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Yi Fan Yu <yifan.yu@windriver.com>
Fix related to a buffer overflow in bfd library
CVE Details https://nvd.nist.gov/vuln/detail/CVE-2020-35448
Upstream Tracking https://sourceware.org/bugzilla/show_bug.cgi?id=26574
Patch from Upstream
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;
h=8642dafaef21aa6747cec01df1977e9c52eb4679
Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cb83312131f6c4f69d89d639085e07ea1f53167e)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
.../binutils/binutils-2.35.1.inc | 1 +
.../binutils/binutils/CVE-2020-35448.patch | 85 +++++++++++++++++++
2 files changed, 86 insertions(+)
create mode 100644 meta/recipes-devtools/binutils/binutils/CVE-2020-35448.patch
diff --git a/meta/recipes-devtools/binutils/binutils-2.35.1.inc b/meta/recipes-devtools/binutils/binutils-2.35.1.inc
index c92cb75543..775af2b8f2 100644
--- a/meta/recipes-devtools/binutils/binutils-2.35.1.inc
+++ b/meta/recipes-devtools/binutils/binutils-2.35.1.inc
@@ -43,5 +43,6 @@ SRC_URI = "\
file://0016-Check-for-clang-before-checking-gcc-version.patch \
file://0017-gas-improve-reproducibility-for-stabs-debugging-data.patch \
file://0001-aarch64-Return-an-error-on-conditional-branch-to-an-.patch \
+ file://CVE-2020-35448.patch \
"
S = "${WORKDIR}/git"
diff --git a/meta/recipes-devtools/binutils/binutils/CVE-2020-35448.patch b/meta/recipes-devtools/binutils/binutils/CVE-2020-35448.patch
new file mode 100644
index 0000000000..3bc64776e5
--- /dev/null
+++ b/meta/recipes-devtools/binutils/binutils/CVE-2020-35448.patch
@@ -0,0 +1,85 @@
+From 6caa41daeb7aa17c400b7300fb78d207cf064d70 Mon Sep 17 00:00:00 2001
+From: Alan Modra <amodra@gmail.com>
+Date: Fri, 4 Sep 2020 19:19:18 +0930
+Subject: [PATCH] PR26574, heap buffer overflow in
+ _bfd_elf_slurp_secondary_reloc_section
+
+A horribly fuzzed object with section headers inside the ELF header.
+Disallow that, and crazy reloc sizes.
+
+ PR 26574
+ * elfcode.h (elf_object_p): Sanity check section header offset.
+ * elf.c (_bfd_elf_slurp_secondary_reloc_section): Sanity check
+ sh_entsize.
+
+Upstream-Status: Backport
+CVE: CVE-2020-35448
+
+Reference to upstream patch:
+https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;
+ h=8642dafaef21aa6747cec01df1977e9c52eb4679
+
+Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com>
+---
+ bfd/elf.c | 4 +++-
+ bfd/elfcode.h | 8 ++++----
+ 2 files changed, 7 insertions(+), 5 deletions(-)
+
+diff --git a/bfd/elf.c b/bfd/elf.c
+index fe375e7346..9f29166399 100644
+--- a/bfd/elf.c
++++ b/bfd/elf.c
+@@ -12527,7 +12527,9 @@ _bfd_elf_slurp_secondary_reloc_section (bfd * abfd,
+ Elf_Internal_Shdr * hdr = & elf_section_data (relsec)->this_hdr;
+
+ if (hdr->sh_type == SHT_SECONDARY_RELOC
+- && hdr->sh_info == (unsigned) elf_section_data (sec)->this_idx)
++ && hdr->sh_info == (unsigned) elf_section_data (sec)->this_idx
++ && (hdr->sh_entsize == ebd->s->sizeof_rel
++ || hdr->sh_entsize == ebd->s->sizeof_rela))
+ {
+ bfd_byte * native_relocs;
+ bfd_byte * native_reloc;
+diff --git a/bfd/elfcode.h b/bfd/elfcode.h
+index f4a7829f27..54ef890637 100644
+--- a/bfd/elfcode.h
++++ b/bfd/elfcode.h
+@@ -568,7 +568,7 @@ elf_object_p (bfd *abfd)
+
+ /* If this is a relocatable file and there is no section header
+ table, then we're hosed. */
+- if (i_ehdrp->e_shoff == 0 && i_ehdrp->e_type == ET_REL)
++ if (i_ehdrp->e_shoff < sizeof (x_ehdr) && i_ehdrp->e_type == ET_REL)
+ goto got_wrong_format_error;
+
+ /* As a simple sanity check, verify that what BFD thinks is the
+@@ -578,7 +578,7 @@ elf_object_p (bfd *abfd)
+ goto got_wrong_format_error;
+
+ /* Further sanity check. */
+- if (i_ehdrp->e_shoff == 0 && i_ehdrp->e_shnum != 0)
++ if (i_ehdrp->e_shoff < sizeof (x_ehdr) && i_ehdrp->e_shnum != 0)
+ goto got_wrong_format_error;
+
+ ebd = get_elf_backend_data (abfd);
+@@ -615,7 +615,7 @@ elf_object_p (bfd *abfd)
+ && ebd->elf_osabi != ELFOSABI_NONE)
+ goto got_wrong_format_error;
+
+- if (i_ehdrp->e_shoff != 0)
++ if (i_ehdrp->e_shoff >= sizeof (x_ehdr))
+ {
+ file_ptr where = (file_ptr) i_ehdrp->e_shoff;
+
+@@ -807,7 +807,7 @@ elf_object_p (bfd *abfd)
+ }
+ }
+
+- if (i_ehdrp->e_shstrndx != 0 && i_ehdrp->e_shoff != 0)
++ if (i_ehdrp->e_shstrndx != 0 && i_ehdrp->e_shoff >= sizeof (x_ehdr))
+ {
+ unsigned int num_sec;
+
+--
+2.29.2
+
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 07/20] meta: toolchain-shar-relocate.sh: Do not use $target_sdk_dir as regex
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (5 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 06/20] binutils: Fix CVE-2020-35448 Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 08/20] meta: toolchain-shar-relocate.sh: Filter out post-relocate-setup script Anuj Mittal
` (12 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Marek Vasut <marex@denx.de>
The $target_sdk_dir path might contain special characters, for example if
the path is /opt/poky/3.2+snapshot . Prevent grep from interpreting those
as part of the regex by using the -F parameter and multiple -e parameters
to specify which strings to filter out. Also note that the previous regex
was using asterisk as wildcard (e.g. environment-setup-*), but that should
have been regex (e.g. environment-setup-.*, with dot) to match correctly,
this is also fixed by this change.
Fixes: 9721378688 ("toolchain-shar-template.sh: Make relocation optional.")
Signed-off-by: Marek Vasut <marex@denx.de>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Krzysztof Zawadzki <krzysztof.zawadzki@nokia.com>
Cc: Randy Witt <randy.e.witt@linux.intel.com>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
Cc: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 19d9fa7ab6c851000bc5d24281739e1b2bb8f057)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
meta/files/toolchain-shar-relocate.sh | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/meta/files/toolchain-shar-relocate.sh b/meta/files/toolchain-shar-relocate.sh
index e3c10018ef..9c358a53e2 100644
--- a/meta/files/toolchain-shar-relocate.sh
+++ b/meta/files/toolchain-shar-relocate.sh
@@ -56,7 +56,9 @@ for replace in "$target_sdk_dir -maxdepth 1" "$native_sysroot"; do
$SUDO_EXEC find $replace -type f
done | xargs -n100 file | grep ":.*\(ASCII\|script\|source\).*text" | \
awk -F':' '{printf "\"%s\"\n", $1}' | \
- grep -Ev "$target_sdk_dir/(environment-setup-*|relocate_sdk*|${0##*/})" | \
+ grep -Fv -e "$target_sdk_dir/environment-setup-" \
+ -e "$target_sdk_dir/relocate_sdk" \
+ -e "$target_sdk_dir/${0##*/}" | \
xargs -n100 $SUDO_EXEC sed -i \
-e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:g" \
-e "s:^#! */usr/bin/perl.*:#! /usr/bin/env perl:g" \
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 08/20] meta: toolchain-shar-relocate.sh: Filter out post-relocate-setup script
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (6 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 07/20] meta: toolchain-shar-relocate.sh: Do not use $target_sdk_dir as regex Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 09/20] lib/oe/utils: Return empty string in parallel_make Anuj Mittal
` (11 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Marek Vasut <marex@denx.de>
The toolchain-shar-extract.sh script updates the SDK relocation paths in
post-relocate-setup.sh, so avoid doing this twice. This is generally not
a problem, unless the SDK path is a subset of the SDK relocation path, in
which case the resulting path is substituted twice. To trigger the issue,
$ ./tmp/deploy/sdk/poky-glibc-x86_64-core-image-base-core2-64-qemux86-64-toolchain-3.2+snapshot.sh -y -d /home/oe/.local/opt/poky/3.2+snapshot
which generates relocation path
/home/oe/.local/home/oe/.local/opt/poky/3.2+snapshot
instead of
/home/oe/.local/opt/poky/3.2+snapshot
Fixes: 93ec145f42 ("toolchain-shar-extract: Add post-relocate scripts")
Signed-off-by: Marek Vasut <marex@denx.de>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Krzysztof Zawadzki <krzysztof.zawadzki@nokia.com>
Cc: Randy Witt <randy.e.witt@linux.intel.com>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
Cc: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5000aabe6ac336e7b424dafa1bf76271dee6a6f1)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
meta/files/toolchain-shar-relocate.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/files/toolchain-shar-relocate.sh b/meta/files/toolchain-shar-relocate.sh
index 9c358a53e2..94d288ce05 100644
--- a/meta/files/toolchain-shar-relocate.sh
+++ b/meta/files/toolchain-shar-relocate.sh
@@ -58,6 +58,7 @@ done | xargs -n100 file | grep ":.*\(ASCII\|script\|source\).*text" | \
awk -F':' '{printf "\"%s\"\n", $1}' | \
grep -Fv -e "$target_sdk_dir/environment-setup-" \
-e "$target_sdk_dir/relocate_sdk" \
+ -e "$target_sdk_dir/post-relocate-setup" \
-e "$target_sdk_dir/${0##*/}" | \
xargs -n100 $SUDO_EXEC sed -i \
-e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:g" \
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 09/20] lib/oe/utils: Return empty string in parallel_make
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (7 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 08/20] meta: toolchain-shar-relocate.sh: Filter out post-relocate-setup script Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 10/20] gcc: Backport patch to resolve i*86 tune configuration overrides Anuj Mittal
` (10 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
In cmake.bbclass we set CMAKE_BUILD_PARALLEL_LEVEL using parallel_make
function and if PARALLEL_MAKE is set to empty string then this variable
is exported as "None" causing cmake to fail with:
"'CMAKE_BUILD_PARALLEL_LEVEL' environment variable
invalid number 'None' given."
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2f790ded554a52ac18d1c28002142f9c62abec8b)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
meta/lib/oe/utils.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index 468c76f30f..9a2187e36f 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -193,7 +193,7 @@ def parallel_make(d, makeinst=False):
return int(v)
- return None
+ return ''
def parallel_make_argument(d, fmt, limit=None, makeinst=False):
"""
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 10/20] gcc: Backport patch to resolve i*86 tune configuration overrides
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (8 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 09/20] lib/oe/utils: Return empty string in parallel_make Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 11/20] wic: Add workdir argument Anuj Mittal
` (9 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Nathan Rossi <nathan@nathanrossi.com>
Upstream has merged a change to handle detection of target overrides so
that they are only applied as required. Backport this patch from
upstream and remove the OE specific patch.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d430f425e1a5725e93833a652e5c6988576f7347)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
meta/recipes-devtools/gcc/gcc-10.2.inc | 2 +-
...0001-lib-Remove-i-86-march-overrides.patch | 113 ----------
...libgomp-libitc-Fix-bootstrap-PR70454.patch | 208 ++++++++++++++++++
3 files changed, 209 insertions(+), 114 deletions(-)
delete mode 100644 meta/recipes-devtools/gcc/gcc/0001-lib-Remove-i-86-march-overrides.patch
create mode 100644 meta/recipes-devtools/gcc/gcc/0001-libatomic-libgomp-libitc-Fix-bootstrap-PR70454.patch
diff --git a/meta/recipes-devtools/gcc/gcc-10.2.inc b/meta/recipes-devtools/gcc/gcc-10.2.inc
index 87dc988d70..82f180db77 100644
--- a/meta/recipes-devtools/gcc/gcc-10.2.inc
+++ b/meta/recipes-devtools/gcc/gcc-10.2.inc
@@ -69,7 +69,7 @@ SRC_URI = "\
file://0002-aarch64-Introduce-SLS-mitigation-for-RET-and-BR-inst.patch \
file://0003-aarch64-Mitigate-SLS-for-BLR-instruction.patch \
file://0001-aarch64-Fix-up-__aarch64_cas16_acq_rel-fallback.patch \
- file://0001-lib-Remove-i-86-march-overrides.patch \
+ file://0001-libatomic-libgomp-libitc-Fix-bootstrap-PR70454.patch \
"
SRC_URI[sha256sum] = "b8dd4368bb9c7f0b98188317ee0254dd8cc99d1e3a18d0ff146c855fe16c1d8c"
diff --git a/meta/recipes-devtools/gcc/gcc/0001-lib-Remove-i-86-march-overrides.patch b/meta/recipes-devtools/gcc/gcc/0001-lib-Remove-i-86-march-overrides.patch
deleted file mode 100644
index 3f04e121ba..0000000000
--- a/meta/recipes-devtools/gcc/gcc/0001-lib-Remove-i-86-march-overrides.patch
+++ /dev/null
@@ -1,113 +0,0 @@
-From 3776789cde95916f95deef48acf9532b0ebf91ae Mon Sep 17 00:00:00 2001
-From: Nathan Rossi <nathan@nathanrossi.com>
-Date: Tue, 15 Dec 2020 11:43:36 +1000
-Subject: [PATCH] lib*: Remove i*86 march overrides
-
-OE does not pass the '--with-arch' option to gccs configure, as such
-some gcc-runtime libraries try to override the value of '-march' and
-'-mtune' which OE already provides. This causes conflicts with other
-i*86 instruction/architecture flags (e.g. -msse* and -mfpmath=*).
-Additionally this caused the following libraries to be built with less
-optimized tune configurations.
-
-Upstream suggests that this should be detecting or otherwise checking if
-the target supports the desired functionality before overriding.
-
- https://gcc.gnu.org/legacy-ml/gcc-patches/2016-04/msg01344.html
-
-libatomic also overrides the '-march' specifically for IFUNC objects.
-OE already supplies the correct march flag, so remove setting.
-
-Upstream-Status: Inappropriate [OE Specific]
-Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
----
- libatomic/Makefile.am | 1 -
- libatomic/Makefile.in | 1 -
- libatomic/configure.tgt | 9 ---------
- libgomp/configure.tgt | 8 --------
- libitm/configure.tgt | 9 ---------
- 5 files changed, 28 deletions(-)
-
-diff --git a/libatomic/Makefile.am b/libatomic/Makefile.am
-index ac1ca64587..5aa16e0699 100644
---- a/libatomic/Makefile.am
-+++ b/libatomic/Makefile.am
-@@ -133,7 +133,6 @@ libatomic_la_LIBADD += $(foreach s,$(SIZES),$(addsuffix _$(s)_1_.lo,$(SIZEOBJS))
- libatomic_la_LIBADD += $(addsuffix _8_2_.lo,$(SIZEOBJS))
- endif
- if ARCH_I386
--IFUNC_OPTIONS = -march=i586
- libatomic_la_LIBADD += $(addsuffix _8_1_.lo,$(SIZEOBJS))
- endif
- if ARCH_X86_64
-diff --git a/libatomic/Makefile.in b/libatomic/Makefile.in
-index 97df2d7ff0..82c511d420 100644
---- a/libatomic/Makefile.in
-+++ b/libatomic/Makefile.in
-@@ -432,7 +432,6 @@ libatomic_la_LIBADD = $(foreach s,$(SIZES),$(addsuffix \
- _$(s)_.lo,$(SIZEOBJS))) $(am__append_1) $(am__append_2) \
- $(am__append_3) $(am__append_4)
- @ARCH_ARM_LINUX_TRUE@@HAVE_IFUNC_TRUE@IFUNC_OPTIONS = -march=armv7-a+fp -DHAVE_KERNEL64
--@ARCH_I386_TRUE@@HAVE_IFUNC_TRUE@IFUNC_OPTIONS = -march=i586
- @ARCH_X86_64_TRUE@@HAVE_IFUNC_TRUE@IFUNC_OPTIONS = -mcx16
- libatomic_convenience_la_SOURCES = $(libatomic_la_SOURCES)
- libatomic_convenience_la_LIBADD = $(libatomic_la_LIBADD)
-diff --git a/libatomic/configure.tgt b/libatomic/configure.tgt
-index 5dd0926d20..3cc41773c3 100644
---- a/libatomic/configure.tgt
-+++ b/libatomic/configure.tgt
-@@ -82,15 +82,6 @@ case "${target_cpu}" in
- ;;
-
- i[3456]86)
-- case " ${CC} ${CFLAGS} " in
-- *" -m64 "*|*" -mx32 "*)
-- ;;
-- *)
-- if test -z "$with_arch"; then
-- XCFLAGS="${XCFLAGS} -march=i486 -mtune=${target_cpu}"
-- XCFLAGS="${XCFLAGS} -fomit-frame-pointer"
-- fi
-- esac
- ARCH=x86
- # ??? Detect when -march=i686 is already enabled.
- try_ifunc=yes
-diff --git a/libgomp/configure.tgt b/libgomp/configure.tgt
-index 4790a31e39..46c4c958e6 100644
---- a/libgomp/configure.tgt
-+++ b/libgomp/configure.tgt
-@@ -72,14 +72,6 @@ if test x$enable_linux_futex = xyes; then
- # Note that bare i386 is not included here. We need cmpxchg.
- i[456]86-*-linux*)
- config_path="linux/x86 linux posix"
-- case " ${CC} ${CFLAGS} " in
-- *" -m64 "*|*" -mx32 "*)
-- ;;
-- *)
-- if test -z "$with_arch"; then
-- XCFLAGS="${XCFLAGS} -march=i486 -mtune=${target_cpu}"
-- fi
-- esac
- ;;
-
- # Similar jiggery-pokery for x86_64 multilibs, except here we
-diff --git a/libitm/configure.tgt b/libitm/configure.tgt
-index 04109160e9..3d78ea609d 100644
---- a/libitm/configure.tgt
-+++ b/libitm/configure.tgt
-@@ -59,15 +59,6 @@ case "${target_cpu}" in
- arm*) ARCH=arm ;;
-
- i[3456]86)
-- case " ${CC} ${CFLAGS} " in
-- *" -m64 "*|*" -mx32 "*)
-- ;;
-- *)
-- if test -z "$with_arch"; then
-- XCFLAGS="${XCFLAGS} -march=i486 -mtune=${target_cpu}"
-- XCFLAGS="${XCFLAGS} -fomit-frame-pointer"
-- fi
-- esac
- XCFLAGS="${XCFLAGS} -mrtm"
- ARCH=x86
- ;;
diff --git a/meta/recipes-devtools/gcc/gcc/0001-libatomic-libgomp-libitc-Fix-bootstrap-PR70454.patch b/meta/recipes-devtools/gcc/gcc/0001-libatomic-libgomp-libitc-Fix-bootstrap-PR70454.patch
new file mode 100644
index 0000000000..addecb4bd8
--- /dev/null
+++ b/meta/recipes-devtools/gcc/gcc/0001-libatomic-libgomp-libitc-Fix-bootstrap-PR70454.patch
@@ -0,0 +1,208 @@
+From 2824d2418605e092899117e77bc8ebf332321807 Mon Sep 17 00:00:00 2001
+From: Jakub Jelinek <jakub@redhat.com>
+Date: Fri, 15 Jan 2021 13:12:59 +0100
+Subject: [PATCH] libatomic, libgomp, libitc: Fix bootstrap [PR70454]
+
+The recent changes to error on mixing -march=i386 and -fcf-protection broke
+bootstrap. This patch changes lib{atomic,gomp,itm} configury, so that it
+only adds -march=i486 to flags if really needed (i.e. when 486 or later isn't
+on by default already). Similarly, it will not use ifuncs if -mcx16
+(or -march=i686 for 32-bit) is on by default.
+
+2021-01-15 Jakub Jelinek <jakub@redhat.com>
+
+ PR target/70454
+libatomic/
+ * configure.tgt: For i?86 and x86_64 determine if -march=i486 needs to
+ be added through preprocessor check on
+ __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4. Determine if try_ifunc is needed
+ based on preprocessor check on __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
+ or __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8.
+libgomp/
+ * configure.tgt: For i?86 and x86_64 determine if -march=i486 needs to
+ be added through preprocessor check on
+ __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4.
+libitm/
+ * configure.tgt: For i?86 and x86_64 determine if -march=i486 needs to
+ be added through preprocessor check on
+ __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4.
+
+Upstream-Status: Backport [master post 10.x release]
+---
+ libatomic/configure.tgt | 56 +++++++++++++++++++++++------------------
+ libgomp/configure.tgt | 35 +++++++++++---------------
+ libitm/configure.tgt | 37 +++++++++++++--------------
+ 3 files changed, 64 insertions(+), 64 deletions(-)
+
+diff --git a/libatomic/configure.tgt b/libatomic/configure.tgt
+index 5dd0926d20..6ea082a29b 100644
+--- a/libatomic/configure.tgt
++++ b/libatomic/configure.tgt
+@@ -81,32 +81,40 @@ case "${target_cpu}" in
+ ARCH=sparc
+ ;;
+
+- i[3456]86)
+- case " ${CC} ${CFLAGS} " in
+- *" -m64 "*|*" -mx32 "*)
+- ;;
+- *)
+- if test -z "$with_arch"; then
+- XCFLAGS="${XCFLAGS} -march=i486 -mtune=${target_cpu}"
+- XCFLAGS="${XCFLAGS} -fomit-frame-pointer"
+- fi
+- esac
+- ARCH=x86
+- # ??? Detect when -march=i686 is already enabled.
+- try_ifunc=yes
+- ;;
+- x86_64)
+- case " ${CC} ${CFLAGS} " in
+- *" -m32 "*)
++ i[3456]86 | x86_64)
++ cat > conftestx.c <<EOF
++#ifndef __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
++#error need -march=i486
++#endif
++EOF
++ if ${CC} ${CFLAGS} -E conftestx.c > /dev/null 2>&1; then
++ :
++ else
++ if test "${target_cpu}" = x86_64; then
+ XCFLAGS="${XCFLAGS} -march=i486 -mtune=generic"
+- XCFLAGS="${XCFLAGS} -fomit-frame-pointer"
+- ;;
+- *)
+- ;;
+- esac
++ else
++ XCFLAGS="${XCFLAGS} -march=i486 -mtune=${target_cpu}"
++ fi
++ XCFLAGS="${XCFLAGS} -fomit-frame-pointer"
++ fi
++ cat > conftestx.c <<EOF
++#ifdef __x86_64__
++#ifndef __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
++#error need -mcx16
++#endif
++#else
++#ifndef __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
++#error need -march=i686
++#endif
++#endif
++EOF
++ if ${CC} ${CFLAGS} -E conftestx.c > /dev/null 2>&1; then
++ try_ifunc=no
++ else
++ try_ifunc=yes
++ fi
++ rm -f conftestx.c
+ ARCH=x86
+- # ??? Detect when -mcx16 is already enabled.
+- try_ifunc=yes
+ ;;
+
+ *) ARCH="${target_cpu}" ;;
+diff --git a/libgomp/configure.tgt b/libgomp/configure.tgt
+index 4790a31e39..761ef2a7db 100644
+--- a/libgomp/configure.tgt
++++ b/libgomp/configure.tgt
+@@ -70,28 +70,23 @@ if test x$enable_linux_futex = xyes; then
+ ;;
+
+ # Note that bare i386 is not included here. We need cmpxchg.
+- i[456]86-*-linux*)
++ i[456]86-*-linux* | x86_64-*-linux*)
+ config_path="linux/x86 linux posix"
+- case " ${CC} ${CFLAGS} " in
+- *" -m64 "*|*" -mx32 "*)
+- ;;
+- *)
+- if test -z "$with_arch"; then
+- XCFLAGS="${XCFLAGS} -march=i486 -mtune=${target_cpu}"
+- fi
+- esac
+- ;;
+-
+- # Similar jiggery-pokery for x86_64 multilibs, except here we
+- # can't rely on the --with-arch configure option, since that
+- # applies to the 64-bit side.
+- x86_64-*-linux*)
+- config_path="linux/x86 linux posix"
+- case " ${CC} ${CFLAGS} " in
+- *" -m32 "*)
++ cat > conftestx.c <<EOF
++#ifndef __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
++#error need -march=i486
++#endif
++EOF
++ if ${CC} ${CFLAGS} -E conftestx.c > /dev/null 2>&1; then
++ :
++ else
++ if test "${target_cpu}" = x86_64; then
+ XCFLAGS="${XCFLAGS} -march=i486 -mtune=generic"
+- ;;
+- esac
++ else
++ XCFLAGS="${XCFLAGS} -march=i486 -mtune=${target_cpu}"
++ fi
++ fi
++ rm -f conftestx.c
+ ;;
+
+ # Note that sparcv7 and sparcv8 is not included here. We need cas.
+diff --git a/libitm/configure.tgt b/libitm/configure.tgt
+index 04109160e9..ca62bac627 100644
+--- a/libitm/configure.tgt
++++ b/libitm/configure.tgt
+@@ -58,16 +58,23 @@ case "${target_cpu}" in
+
+ arm*) ARCH=arm ;;
+
+- i[3456]86)
+- case " ${CC} ${CFLAGS} " in
+- *" -m64 "*|*" -mx32 "*)
+- ;;
+- *)
+- if test -z "$with_arch"; then
+- XCFLAGS="${XCFLAGS} -march=i486 -mtune=${target_cpu}"
+- XCFLAGS="${XCFLAGS} -fomit-frame-pointer"
+- fi
+- esac
++ i[3456]86 | x86_64)
++ cat > conftestx.c <<EOF
++#ifndef __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
++#error need -march=i486
++#endif
++EOF
++ if ${CC} ${CFLAGS} -E conftestx.c > /dev/null 2>&1; then
++ :
++ else
++ if test "${target_cpu}" = x86_64; then
++ XCFLAGS="${XCFLAGS} -march=i486 -mtune=generic"
++ else
++ XCFLAGS="${XCFLAGS} -march=i486 -mtune=${target_cpu}"
++ fi
++ XCFLAGS="${XCFLAGS} -fomit-frame-pointer"
++ fi
++ rm -f conftestx.c
+ XCFLAGS="${XCFLAGS} -mrtm"
+ ARCH=x86
+ ;;
+@@ -102,16 +109,6 @@ case "${target_cpu}" in
+ ARCH=sparc
+ ;;
+
+- x86_64)
+- case " ${CC} ${CFLAGS} " in
+- *" -m32 "*)
+- XCFLAGS="${XCFLAGS} -march=i486 -mtune=generic"
+- XCFLAGS="${XCFLAGS} -fomit-frame-pointer"
+- ;;
+- esac
+- XCFLAGS="${XCFLAGS} -mrtm"
+- ARCH=x86
+- ;;
+ s390|s390x)
+ XCFLAGS="${XCFLAGS} -mzarch -mhtm"
+ ARCH=s390
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 11/20] wic: Add workdir argument
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (9 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 10/20] gcc: Backport patch to resolve i*86 tune configuration overrides Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 12/20] wic: Ensure internal workdir is not reused Anuj Mittal
` (8 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
This allows the path for the temporary workdir used by wic to be set
when running wic from bitbake or directly from the command line.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2e7314ac3a3cab89de93c932e8efc75de0feb0a7)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
scripts/lib/wic/plugins/imager/direct.py | 2 +-
scripts/wic | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index 0ca67b77d5..b329568c7a 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -62,7 +62,7 @@ class DirectPlugin(ImagerPlugin):
self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0],
strftime("%Y%m%d%H%M"))
- self.workdir = tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+ self.workdir = options.workdir or tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
self._image = None
self.ptable_format = self.ks.bootloader.ptable
self.parts = self.ks.partitions
diff --git a/scripts/wic b/scripts/wic
index 24700f380f..a741aed364 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -312,6 +312,8 @@ def wic_init_parser_create(subparser):
subparser.add_argument("-o", "--outdir", dest="outdir", default='.',
help="name of directory to create image in")
+ subparser.add_argument("-w", "--workdir",
+ help="temporary workdir to use for intermediate files")
subparser.add_argument("-e", "--image-name", dest="image_name",
help="name of the image to use the artifacts from "
"e.g. core-image-sato")
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 12/20] wic: Ensure internal workdir is not reused
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (10 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 11/20] wic: Add workdir argument Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 13/20] wic: Allow exec_native_cmd to run HOSTTOOLS Anuj Mittal
` (7 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
If a path is specified for the internal wic working directory using
the -w/--workdir argument then it must not already exist. Re-using a
previous workdir could easily result in rootfs and intermediate files
from a previous build being added to the current image.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2e40c8d4109024ff704c5ce40d98050ca7f34dd5)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
scripts/lib/wic/plugins/imager/direct.py | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index b329568c7a..f107e60089 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -62,7 +62,7 @@ class DirectPlugin(ImagerPlugin):
self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0],
strftime("%Y%m%d%H%M"))
- self.workdir = options.workdir or tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+ self.workdir = self.setup_workdir(options.workdir)
self._image = None
self.ptable_format = self.ks.bootloader.ptable
self.parts = self.ks.partitions
@@ -78,6 +78,16 @@ class DirectPlugin(ImagerPlugin):
self._image = PartitionedImage(image_path, self.ptable_format,
self.parts, self.native_sysroot)
+ def setup_workdir(self, workdir):
+ if workdir:
+ if os.path.exists(workdir):
+ raise WicError("Internal workdir '%s' specified in wic arguments already exists!" % (workdir))
+
+ os.makedirs(workdir)
+ return workdir
+ else:
+ return tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+
def do_create(self):
"""
Plugin entry point.
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 13/20] wic: Allow exec_native_cmd to run HOSTTOOLS
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (11 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 12/20] wic: Ensure internal workdir is not reused Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:21 ` [gatesgarth][PATCH 14/20] image_types_wic: Move wic working directory Anuj Mittal
` (6 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
This allows programs from HOSTTOOLS (e.g. 'install', 'rm', 'mv', etc) to
be more easily executed by wic. Without this change only programs from
an actual *-native recipe built by bitbake can be executed by wic.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8eb186acdecfbb3151c9a0ab148358e3fe5cce39)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
meta/classes/image_types_wic.bbclass | 2 +-
scripts/lib/wic/misc.py | 7 +++++--
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 286e0f5d54..000ee4249f 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -3,7 +3,7 @@
WICVARS ?= "\
BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE FAKEROOTCMD IMAGE_BASENAME IMAGE_EFI_BOOT_FILES IMAGE_BOOT_FILES \
IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD INITRD_LIVE ISODIR RECIPE_SYSROOT_NATIVE \
- ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS \
+ ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS HOSTTOOLS_DIR \
KERNEL_IMAGETYPE MACHINE INITRAMFS_IMAGE INITRAMFS_IMAGE_BUNDLE INITRAMFS_LINK_NAME APPEND \
ASSUME_PROVIDED PSEUDO_IGNORE_PATHS"
diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 4b08d649c6..75b219cd3f 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -138,9 +138,12 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
if pseudo:
cmd_and_args = pseudo + cmd_and_args
- native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \
+ hosttools_dir = get_bitbake_var("HOSTTOOLS_DIR")
+
+ native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin:%s" % \
(native_sysroot, native_sysroot,
- native_sysroot, native_sysroot)
+ native_sysroot, native_sysroot,
+ hosttools_dir)
native_cmd_and_args = "export PATH=%s:$PATH;%s" % \
(native_paths, cmd_and_args)
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 14/20] image_types_wic: Move wic working directory
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (12 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 13/20] wic: Allow exec_native_cmd to run HOSTTOOLS Anuj Mittal
@ 2021-01-21 16:21 ` Anuj Mittal
2021-01-21 16:22 ` [gatesgarth][PATCH 15/20] wic: Update pseudo db when excluding content from rootfs Anuj Mittal
` (5 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:21 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
By default the wic working directory is placed under the output
directory. When invoking wic under bitbake, the wic output directory is
added to PSEUDO_PATHS_IGNORE to avoid issues with files being removed
from outside a pseudo environment (see oe-core commit ad8f5532ff).
However, wic will copy the rootfs directory into its working directory
if it needs to add or remove content before creating a filesystem image.
This copy of the rootfs directory must be tracked by pseudo in order to
keep the permissions correct in the resulting image. So we can't have
the wic working directory under a path in PSEUDO_PATHS_IGNORE unless
we like broken permissions.
To fix this the new '-w' argument to wic is used to move the working
directory away from the output directory.
Note that wic deletes the temporary working directory automatically
when it finishes creating an image so users won't normally see the
'tmp-wic' directory under WORKDIR.
Fixes [Yocto #14129]
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4d07169499c47fa9dc759e6f81843416a6dc25c5)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
meta/classes/image_types_wic.bbclass | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 000ee4249f..49be1da77a 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -29,11 +29,17 @@ WIC_CREATE_EXTRA_ARGS ?= ""
IMAGE_CMD_wic () {
out="${IMGDEPLOYDIR}/${IMAGE_NAME}"
build_wic="${WORKDIR}/build-wic"
+ tmp_wic="${WORKDIR}/tmp-wic"
wks="${WKS_FULL_PATH}"
+ if [ -e "$tmp_wic" ]; then
+ # Ensure we don't have any junk leftover from a previously interrupted
+ # do_image_wic execution
+ rm -rf "$tmp_wic"
+ fi
if [ -z "$wks" ]; then
bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
fi
- BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS}
+ BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
mv "$build_wic/$(basename "${wks%.wks}")"*.direct "$out${IMAGE_NAME_SUFFIX}.wic"
}
IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 15/20] wic: Update pseudo db when excluding content from rootfs
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (13 preceding siblings ...)
2021-01-21 16:21 ` [gatesgarth][PATCH 14/20] image_types_wic: Move wic working directory Anuj Mittal
@ 2021-01-21 16:22 ` Anuj Mittal
2021-01-21 16:22 ` [gatesgarth][PATCH 16/20] wic: Copy rootfs dir if fstab needs updating Anuj Mittal
` (4 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:22 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
To exclude content from the rootfs, wic makes a copy (using hardlinks if
possible) of the rootfs directory and associated pseudo db, then removes
files & directories as needed. However if these files and directories
are removed using the python functions os.remove and shutil.rmtree, the
copied pseudo db will not be updated correctly. For files copied from
the original rootfs, if hardlinks were used successfully when copying
the rootfs this should mean that the relevant inodes can't be reused and
so the risk of pseudo aborts should be avoided. However, this logic
doesn't apply for directories (as they can't be hardlinked) or for files
added via the '--include-path' argument (as they weren't present in the
original rootfs) and so there remains some risk of inodes being reused
and the pseudo db becoming corrupted.
To fix this, use the 'rm' command under pseudo when removing files &
directories from the copied rootfs to ensure that the copied pseudo db
is updated.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d5db7e268947f0392c2126137571a44acd29ccd6)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
scripts/lib/wic/plugins/source/rootfs.py | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index f1db83f8a1..afb1eb9202 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -199,17 +199,20 @@ class RootfsPlugin(SourcePlugin):
if not os.path.lexists(full_path):
continue
+ if new_pseudo:
+ pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
+ else:
+ pseudo = None
if path.endswith(os.sep):
# Delete content only.
for entry in os.listdir(full_path):
full_entry = os.path.join(full_path, entry)
- if os.path.isdir(full_entry) and not os.path.islink(full_entry):
- shutil.rmtree(full_entry)
- else:
- os.remove(full_entry)
+ rm_cmd = "rm -rf %s" % (full_entry)
+ exec_native_cmd(rm_cmd, native_sysroot, pseudo)
else:
# Delete whole directory.
- shutil.rmtree(full_path)
+ rm_cmd = "rm -rf %s" % (full_path)
+ exec_native_cmd(rm_cmd, native_sysroot, pseudo)
part.prepare_rootfs(cr_workdir, oe_builddir,
new_rootfs or part.rootfs_dir, native_sysroot,
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 16/20] wic: Copy rootfs dir if fstab needs updating
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (14 preceding siblings ...)
2021-01-21 16:22 ` [gatesgarth][PATCH 15/20] wic: Update pseudo db when excluding content from rootfs Anuj Mittal
@ 2021-01-21 16:22 ` Anuj Mittal
2021-01-21 16:22 ` [gatesgarth][PATCH 17/20] wic: Optimise fstab modification for ext2/3/4 and msdos partitions Anuj Mittal
` (3 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:22 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
By default, wic updates the /etc/fstab in the rootfs to include details
of additional partitions described in the selected wks file. If this
modification is performed in place, other tasks which create an image
file from the rootfs directory (e.g. do_image_tar and do_image_ext4)
will pick up the modified fstab file which would not be appropriate for
those images as they do not include the additional partitions described
in the wks file. wic does undo modifications to the fstab file once it
has finished creating the filesystem image, however this leaves open a
race condition if one of the other tasks reads the contents of the fstab
file from the rootfs directory between the point where wic modifies the
fstab file and the point where wic restores the files original content.
This could be solved by adding a lockfile for tasks which use the rootfs
directory to ensure that no other such task is reading the rootfs
directory while do_image_wic is running. This would serialize several
do_image_* tasks and result in slower builds, especially for large
images. Another drawback of this solution is that it is hard to
selectively optimise - adding lockfiles to do_image_* tasks would result
in these tasks always being serialized even if no fstab modification
will take place.
An alternative solution is to copy the rootfs directory when fstab needs
to be modified. The code to do this in wic already exists as it is
needed when including or excluding content in the rootfs. This still
results in an impact on build times but the copy uses hardlinks if
possible (so little data is actually copied) and we can make selective
optimisations to improve things. The rootfs copy will only take place if
fstab modification is required (or if it was already needed to include
or exclude rootfs content). We can also follow up with further
optimisations after this commit. So this second solution is chosen.
Fixes [Yocto #13994]
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ce682a73b7447652f898ce1d1d0416a456df5416)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
scripts/lib/wic/partition.py | 5 +++-
scripts/lib/wic/plugins/imager/direct.py | 36 ++++++++----------------
scripts/lib/wic/plugins/source/rootfs.py | 17 +++++++++--
3 files changed, 29 insertions(+), 29 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 286c7867cb..f59eceb23d 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -54,6 +54,7 @@ class Partition():
self.uuid = args.uuid
self.fsuuid = args.fsuuid
self.type = args.type
+ self.updated_fstab_path = None
self.lineno = lineno
self.source_file = ""
@@ -118,11 +119,13 @@ class Partition():
return self.fixed_size if self.fixed_size else self.size
def prepare(self, creator, cr_workdir, oe_builddir, rootfs_dir,
- bootimg_dir, kernel_dir, native_sysroot):
+ bootimg_dir, kernel_dir, native_sysroot, updated_fstab_path):
"""
Prepare content for individual partitions, depending on
partition command parameters.
"""
+ self.updated_fstab_path = updated_fstab_path
+
if not self.source:
if not self.size and not self.fixed_size:
raise WicError("The %s partition has a size of zero. Please "
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index f107e60089..7e1c1c03ab 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -58,7 +58,7 @@ class DirectPlugin(ImagerPlugin):
self.compressor = options.compressor
self.bmap = options.bmap
self.no_fstab_update = options.no_fstab_update
- self.original_fstab = None
+ self.updated_fstab_path = None
self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0],
strftime("%Y%m%d%H%M"))
@@ -100,11 +100,8 @@ class DirectPlugin(ImagerPlugin):
finally:
self.cleanup()
- def _write_fstab(self, image_rootfs):
- """overriden to generate fstab (temporarily) in rootfs. This is called
- from _create, make sure it doesn't get called from
- BaseImage.create()
- """
+ def update_fstab(self, image_rootfs):
+ """Assume partition order same as in wks"""
if not image_rootfs:
return
@@ -114,18 +111,9 @@ class DirectPlugin(ImagerPlugin):
with open(fstab_path) as fstab:
fstab_lines = fstab.readlines()
- self.original_fstab = fstab_lines.copy()
-
- if self._update_fstab(fstab_lines, self.parts):
- with open(fstab_path, "w") as fstab:
- fstab.writelines(fstab_lines)
- else:
- self.original_fstab = None
- def _update_fstab(self, fstab_lines, parts):
- """Assume partition order same as in wks"""
updated = False
- for part in parts:
+ for part in self.parts:
if not part.realnum or not part.mountpoint \
or part.mountpoint == "/":
continue
@@ -154,7 +142,10 @@ class DirectPlugin(ImagerPlugin):
fstab_lines.append(line)
updated = True
- return updated
+ if updated:
+ self.updated_fstab_path = os.path.join(self.workdir, "fstab")
+ with open(self.updated_fstab_path, "w") as f:
+ f.writelines(fstab_lines)
def _full_path(self, path, name, extention):
""" Construct full file path to a file we generate. """
@@ -170,7 +161,7 @@ class DirectPlugin(ImagerPlugin):
a partitioned image.
"""
if not self.no_fstab_update:
- self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
+ self.update_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
for part in self.parts:
# get rootfs size from bitbake variable if it's not set in .ks file
@@ -283,12 +274,6 @@ class DirectPlugin(ImagerPlugin):
if os.path.isfile(path):
shutil.move(path, os.path.join(self.outdir, fname))
- #Restore original fstab
- if self.original_fstab:
- fstab_path = self.rootfs_dir.get("ROOTFS_DIR") + "/etc/fstab"
- with open(fstab_path, "w") as fstab:
- fstab.writelines(self.original_fstab)
-
# remove work directory
shutil.rmtree(self.workdir, ignore_errors=True)
@@ -368,7 +353,8 @@ class PartitionedImage():
# sizes before we can add them and do the layout.
part.prepare(imager, imager.workdir, imager.oe_builddir,
imager.rootfs_dir, imager.bootimg_dir,
- imager.kernel_dir, imager.native_sysroot)
+ imager.kernel_dir, imager.native_sysroot,
+ imager.updated_fstab_path)
# Converting kB to sectors for parted
part.size_sec = part.disk_size * 1024 // self.sector_size
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index afb1eb9202..6fd415af5b 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -103,9 +103,9 @@ class RootfsPlugin(SourcePlugin):
new_rootfs = None
new_pseudo = None
# Handle excluded paths.
- if part.exclude_path or part.include_path or part.change_directory:
- # We need a new rootfs directory we can delete files from. Copy to
- # workdir.
+ if part.exclude_path or part.include_path or part.change_directory or part.updated_fstab_path:
+ # We need a new rootfs directory we can safely modify without
+ # interfering with other tasks. Copy to workdir.
new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
if os.path.lexists(new_rootfs):
@@ -214,6 +214,17 @@ class RootfsPlugin(SourcePlugin):
rm_cmd = "rm -rf %s" % (full_path)
exec_native_cmd(rm_cmd, native_sysroot, pseudo)
+ has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab"))
+ if part.updated_fstab_path and has_fstab:
+ fstab_path = os.path.join(new_rootfs, "etc/fstab")
+ # Assume that fstab should always be owned by root with fixed permissions
+ install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
+ if new_pseudo:
+ pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
+ else:
+ pseudo = None
+ exec_native_cmd(install_cmd, native_sysroot, pseudo)
+
part.prepare_rootfs(cr_workdir, oe_builddir,
new_rootfs or part.rootfs_dir, native_sysroot,
pseudo_dir = new_pseudo or pseudo_dir)
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 17/20] wic: Optimise fstab modification for ext2/3/4 and msdos partitions
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (15 preceding siblings ...)
2021-01-21 16:22 ` [gatesgarth][PATCH 16/20] wic: Copy rootfs dir if fstab needs updating Anuj Mittal
@ 2021-01-21 16:22 ` Anuj Mittal
2021-01-21 16:22 ` [gatesgarth][PATCH 18/20] toolchain-shar-relocate.sh: Fix handling files with colons Anuj Mittal
` (2 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:22 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
The fix for [Yocto #13994] required the rootfs directory to be copied
(using hardlinks if possible) when modifying the fstab file under wic.
We can optimise this copy away for filesystems where we have the tools
to modify the contents of the partition image after it is created. For
ext2/3/4 filesystems we have the debugfs tool and for msdos/vfat
filesystems we have the mcopy tool. So for any of these filesystems we
skip the modification of the fstab file in the rootfs directory (and
skip the associated copy unless it is otherwise necessary) and update
the contents of fstab directly in the partition image.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5fb8ae0e9159597d7eaa9307a3a8543800bf9405)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
scripts/lib/wic/partition.py | 27 +++++++++++++++++++-----
scripts/lib/wic/plugins/source/rootfs.py | 9 +++++---
2 files changed, 28 insertions(+), 8 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index f59eceb23d..e574f40c47 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -55,6 +55,8 @@ class Partition():
self.fsuuid = args.fsuuid
self.type = args.type
self.updated_fstab_path = None
+ self.has_fstab = False
+ self.update_fstab_in_rootfs = False
self.lineno = lineno
self.source_file = ""
@@ -125,6 +127,8 @@ class Partition():
partition command parameters.
"""
self.updated_fstab_path = updated_fstab_path
+ if self.updated_fstab_path and not (self.fstype.startswith("ext") or self.fstype == "msdos"):
+ self.update_fstab_in_rootfs = True
if not self.source:
if not self.size and not self.fixed_size:
@@ -250,7 +254,7 @@ class Partition():
prefix = "ext" if self.fstype.startswith("ext") else self.fstype
method = getattr(self, "prepare_rootfs_" + prefix)
- method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
+ method(rootfs, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo)
self.source_file = rootfs
# get the rootfs size in the right units for kickstart (kB)
@@ -258,7 +262,7 @@ class Partition():
out = exec_cmd(du_cmd)
self.size = int(out.split()[0])
- def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
+ def prepare_rootfs_ext(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
"""
Prepare content for an ext2/3/4 rootfs partition.
@@ -282,10 +286,19 @@ class Partition():
(self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
+ if self.updated_fstab_path and self.has_fstab:
+ debugfs_script_path = os.path.join(cr_workdir, "debugfs_script")
+ with open(debugfs_script_path, "w") as f:
+ f.write("cd etc\n")
+ f.write("rm fstab\n")
+ f.write("write %s fstab\n" % (self.updated_fstab_path))
+ debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs)
+ exec_native_cmd(debugfs_cmd, native_sysroot)
+
mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
- def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
+ def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
"""
Prepare content for a btrfs rootfs partition.
@@ -308,7 +321,7 @@ class Partition():
self.mkfs_extraopts, self.fsuuid, rootfs)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
- def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
+ def prepare_rootfs_msdos(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
"""
Prepare content for a msdos/vfat rootfs partition.
@@ -337,12 +350,16 @@ class Partition():
mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
exec_native_cmd(mcopy_cmd, native_sysroot)
+ if self.updated_fstab_path and self.has_fstab:
+ mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path)
+ exec_native_cmd(mcopy_cmd, native_sysroot)
+
chmod_cmd = "chmod 644 %s" % rootfs
exec_cmd(chmod_cmd)
prepare_rootfs_vfat = prepare_rootfs_msdos
- def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir,
+ def prepare_rootfs_squashfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
"""
Prepare content for a squashfs rootfs partition.
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 6fd415af5b..96d940a91d 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -94,6 +94,7 @@ class RootfsPlugin(SourcePlugin):
"it is not a valid path, exiting" % part.rootfs_dir)
part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+ part.has_fstab = os.path.exists(os.path.join(part.rootfs_dir, "etc/fstab"))
pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
if not os.path.lexists(pseudo_dir):
logger.warn("%s folder does not exist. "
@@ -103,7 +104,7 @@ class RootfsPlugin(SourcePlugin):
new_rootfs = None
new_pseudo = None
# Handle excluded paths.
- if part.exclude_path or part.include_path or part.change_directory or part.updated_fstab_path:
+ if part.exclude_path or part.include_path or part.change_directory or part.update_fstab_in_rootfs:
# We need a new rootfs directory we can safely modify without
# interfering with other tasks. Copy to workdir.
new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
@@ -214,8 +215,10 @@ class RootfsPlugin(SourcePlugin):
rm_cmd = "rm -rf %s" % (full_path)
exec_native_cmd(rm_cmd, native_sysroot, pseudo)
- has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab"))
- if part.updated_fstab_path and has_fstab:
+ # Update part.has_fstab here as fstab may have been added or
+ # removed by the above modifications.
+ part.has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab"))
+ if part.update_fstab_in_rootfs and part.has_fstab:
fstab_path = os.path.join(new_rootfs, "etc/fstab")
# Assume that fstab should always be owned by root with fixed permissions
install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 18/20] toolchain-shar-relocate.sh: Fix handling files with colons
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (16 preceding siblings ...)
2021-01-21 16:22 ` [gatesgarth][PATCH 17/20] wic: Optimise fstab modification for ext2/3/4 and msdos partitions Anuj Mittal
@ 2021-01-21 16:22 ` Anuj Mittal
2021-01-21 16:22 ` [gatesgarth][PATCH 19/20] devtool: Fix source extraction for gcc shared source Anuj Mittal
2021-01-21 16:22 ` [gatesgarth][PATCH 20/20] flex: Fix --noline option behavior Anuj Mittal
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:22 UTC (permalink / raw)
To: openembedded-core
From: "Oleksiy Obitotskyy yIEf0zt.mo" <oobitots@cisco.com>
Files could contain colons in name and we should not use
colons (':') as field separator. E.g. perl/python man
pages packages.
Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 50d2ed0689f1aed6f33b4992d37e2e991c99eb07)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
meta/files/toolchain-shar-relocate.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/files/toolchain-shar-relocate.sh b/meta/files/toolchain-shar-relocate.sh
index 94d288ce05..5433741296 100644
--- a/meta/files/toolchain-shar-relocate.sh
+++ b/meta/files/toolchain-shar-relocate.sh
@@ -55,7 +55,7 @@ fi
for replace in "$target_sdk_dir -maxdepth 1" "$native_sysroot"; do
$SUDO_EXEC find $replace -type f
done | xargs -n100 file | grep ":.*\(ASCII\|script\|source\).*text" | \
- awk -F':' '{printf "\"%s\"\n", $1}' | \
+ awk -F': ' '{printf "\"%s\"\n", $1}' | \
grep -Fv -e "$target_sdk_dir/environment-setup-" \
-e "$target_sdk_dir/relocate_sdk" \
-e "$target_sdk_dir/post-relocate-setup" \
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 19/20] devtool: Fix source extraction for gcc shared source
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (17 preceding siblings ...)
2021-01-21 16:22 ` [gatesgarth][PATCH 18/20] toolchain-shar-relocate.sh: Fix handling files with colons Anuj Mittal
@ 2021-01-21 16:22 ` Anuj Mittal
2021-01-21 16:22 ` [gatesgarth][PATCH 20/20] flex: Fix --noline option behavior Anuj Mittal
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:22 UTC (permalink / raw)
To: openembedded-core
From: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
If do_patch task is disabled then prepare do_configure dependencies to
fetch external sources and create symlink to ${S} in devtool workspace.
[YOCTO #13036]
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9e47319bfe62d289b90f7545a64dbdc1cbde7f1d)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
scripts/lib/devtool/standard.py | 30 ++++++++++++++++++++++++++----
1 file changed, 26 insertions(+), 4 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 261d642d4a..f7d8a82117 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -589,6 +589,16 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
else:
task = 'do_patch'
+ if 'noexec' in (d.getVarFlags(task, False) or []) or 'task' not in (d.getVarFlags(task, False) or []):
+ logger.info('The %s recipe has %s disabled. Running only '
+ 'do_configure task dependencies' % (pn, task))
+
+ if 'depends' in d.getVarFlags('do_configure', False):
+ pn = d.getVarFlags('do_configure', False)['depends']
+ pn = pn.replace('${PV}', d.getVar('PV'))
+ pn = pn.replace('${COMPILERDEP}', d.getVar('COMPILERDEP'))
+ task = None
+
# Run the fetch + unpack tasks
res = tinfoil.build_targets(pn,
task,
@@ -600,6 +610,17 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
if not res:
raise DevtoolError('Extracting source for %s failed' % pn)
+ if not is_kernel_yocto and ('noexec' in (d.getVarFlags('do_patch', False) or []) or 'task' not in (d.getVarFlags('do_patch', False) or [])):
+ workshareddir = d.getVar('S')
+ if os.path.islink(srctree):
+ os.unlink(srctree)
+
+ os.symlink(workshareddir, srctree)
+
+ # The initial_rev file is created in devtool_post_unpack function that will not be executed if
+ # do_unpack/do_patch tasks are disabled so we have to directly say that source extraction was successful
+ return True, True
+
try:
with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
initial_rev = f.read()
@@ -847,10 +868,11 @@ def modify(args, config, basepath, workspace):
if not initial_rev:
return 1
logger.info('Source tree extracted to %s' % srctree)
- # Get list of commits since this revision
- (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
- commits = stdout.split()
- check_commits = True
+ if os.path.exists(os.path.join(srctree, '.git')):
+ # Get list of commits since this revision
+ (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
+ commits = stdout.split()
+ check_commits = True
else:
if os.path.exists(os.path.join(srctree, '.git')):
# Check if it's a tree previously extracted by us. This is done
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [gatesgarth][PATCH 20/20] flex: Fix --noline option behavior
2021-01-21 16:21 [gatesgarth][PATCH 00/20] patch review request Anuj Mittal
` (18 preceding siblings ...)
2021-01-21 16:22 ` [gatesgarth][PATCH 19/20] devtool: Fix source extraction for gcc shared source Anuj Mittal
@ 2021-01-21 16:22 ` Anuj Mittal
19 siblings, 0 replies; 21+ messages in thread
From: Anuj Mittal @ 2021-01-21 16:22 UTC (permalink / raw)
To: openembedded-core
From: Oleksiy Obitotskyy <oobitots@cisco.com>
Option --noline or -L does not handled properly.
So generated code contains #line directives with
file absolute path and prevents to create
reproducible builds.
Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0bf2cb7dc2123f220accf1542c2ae4c4b4b8275a)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
...directives-if-gen_line_dirs-is-false.patch | 32 +++++++++++++++++++
meta/recipes-devtools/flex/flex_2.6.4.bb | 1 +
2 files changed, 33 insertions(+)
create mode 100644 meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch
diff --git a/meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch b/meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch
new file mode 100644
index 0000000000..c8202b6bd5
--- /dev/null
+++ b/meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch
@@ -0,0 +1,32 @@
+From 440f3f55739468cd26e22f31871eca8cbbd53294 Mon Sep 17 00:00:00 2001
+From: Oleksiy Obitotskyy <oobitots@cisco.com>
+Date: Wed, 6 Jan 2021 06:12:14 -0800
+Subject: [PATCH] Emit no #line directives if gen_line_dirs is false
+
+If we set --noline we should not print line directives.
+But setting --noline means gen_line_dirs is false.
+
+Upstream-Status: Submitted
+Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
+---
+ src/buf.c | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/src/buf.c b/src/buf.c
+index 185083c..4439e28 100644
+--- a/src/buf.c
++++ b/src/buf.c
+@@ -95,8 +95,8 @@ struct Buf *buf_linedir (struct Buf *buf, const char* filename, int lineno)
+ const char *src;
+ size_t tsz;
+
+- if (gen_line_dirs)
+- return buf;
++ if (!gen_line_dirs)
++ return buf;
+
+ tsz = strlen("#line \"\"\n") + /* constant parts */
+ 2 * strlen (filename) + /* filename with possibly all backslashes escaped */
+--
+2.26.2.Cisco
+
diff --git a/meta/recipes-devtools/flex/flex_2.6.4.bb b/meta/recipes-devtools/flex/flex_2.6.4.bb
index 3d57572865..1d43d2228a 100644
--- a/meta/recipes-devtools/flex/flex_2.6.4.bb
+++ b/meta/recipes-devtools/flex/flex_2.6.4.bb
@@ -16,6 +16,7 @@ SRC_URI = "https://github.com/westes/flex/releases/download/v${PV}/flex-${PV}.ta
${@bb.utils.contains('PTEST_ENABLED', '1', '', 'file://disable-tests.patch', d)} \
file://0001-build-AC_USE_SYSTEM_EXTENSIONS-in-configure.ac.patch \
file://check-funcs.patch \
+ file://0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch \
"
SRC_URI[md5sum] = "2882e3179748cc9f9c23ec593d6adc8d"
--
2.29.2
^ permalink raw reply related [flat|nested] 21+ messages in thread