[PATCH 1/8] python: Run pybench once for optimisation task in pgo

[PATCH 1/8] python: Run pybench once for optimisation task in pgo

[Richard Purdie]

From: Ross Burton <ross.burton@intel.com>

There is no good reason to loop 10 times on the test since the profile
optimisation won't change between runs, we don't need/want a statistical
average. This is just burning cpu cycles, just make 1 run.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-devtools/python/python3_3.5.5.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/python/python3_3.5.5.bb b/meta/recipes-devtools/python/python3_3.5.5.bb
index bfe91fb1c71..8afe15eee6a 100644
--- a/meta/recipes-devtools/python/python3_3.5.5.bb
+++ b/meta/recipes-devtools/python/python3_3.5.5.bb
@@ -74,7 +74,7 @@ TARGET_CC_ARCH += "-DNDEBUG -fno-inline"
 SDK_CC_ARCH += "-DNDEBUG -fno-inline"
 EXTRA_OEMAKE += "CROSS_COMPILE=yes"
 EXTRA_OECONF += "CROSSPYTHONPATH=${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dynload/ --without-ensurepip --enable-optimizations"
-PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 10"
+PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 1"
 
 export CROSS_COMPILE = "${TARGET_PREFIX}"
 export _PYTHON_PROJECT_BASE = "${B}"
-- 
2.17.1

[PATCH 2/8] os-release: fix to install in the expected location

[Richard Purdie]

From: Joshua Lock <joshua.g.lock@intel.com>

os-release (5) recommends that the os-release file be installed in
/usr/lib/os-release and that /etc/os-release be a relative symlink to it.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-core/os-release/os-release.bb | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/os-release/os-release.bb b/meta/recipes-core/os-release/os-release.bb
index f9887047561..dae5748ba7a 100644
--- a/meta/recipes-core/os-release/os-release.bb
+++ b/meta/recipes-core/os-release/os-release.bb
@@ -1,7 +1,7 @@
 inherit allarch
 
 SUMMARY = "Operating system identification"
-DESCRIPTION = "The /etc/os-release file contains operating system identification data."
+DESCRIPTION = "The /usr/lib/os-release file contains operating system identification data."
 LICENSE = "MIT"
 INHIBIT_DEFAULT_DEPS = "1"
 
@@ -42,6 +42,9 @@ python do_compile () {
 do_compile[vardeps] += "${OS_RELEASE_FIELDS}"
 
 do_install () {
-    install -d ${D}${sysconfdir}
-    install -m 0644 os-release ${D}${sysconfdir}/
+    install -d ${D}${libdir} ${D}${sysconfdir}
+    install -m 0644 os-release ${D}${libdir}/
+    lnr ${D}${libdir}/os-release ${D}${sysconfdir}/os-relea
 }
+
+FILES_${PN} += "${libdir}/os-release"
-- 
2.17.1

[PATCH 3/8] gcc: Merge two related patches together

[Richard Purdie]

These patches may as well be merged together as they affect the same code
and things are clearer this way.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-devtools/gcc/gcc-8.2.inc         |  1 -
 .../0003-gcc-poison-system-directories.patch  |  6 ++-
 .../gcc-8.2/0004-gcc-poison-dir-extend.patch  | 39 -------------------
 3 files changed, 4 insertions(+), 42 deletions(-)
 delete mode 100644 meta/recipes-devtools/gcc/gcc-8.2/0004-gcc-poison-dir-extend.patch

diff --git a/meta/recipes-devtools/gcc/gcc-8.2.inc b/meta/recipes-devtools/gcc/gcc-8.2.inc
index 8503a66b074..889187e9c42 100644
--- a/meta/recipes-devtools/gcc/gcc-8.2.inc
+++ b/meta/recipes-devtools/gcc/gcc-8.2.inc
@@ -33,7 +33,6 @@ SRC_URI = "\
            ${BASEURI} \
            file://0001-gcc-4.3.1-ARCH_FLAGS_FOR_TARGET.patch \
            file://0003-gcc-poison-system-directories.patch \
-           file://0004-gcc-poison-dir-extend.patch \
            file://0005-gcc-4.3.3-SYSROOT_CFLAGS_FOR_TARGET.patch \
            file://0006-64-bit-multilib-hack.patch \
            file://0007-optional-libstdc.patch \
diff --git a/meta/recipes-devtools/gcc/gcc-8.2/0003-gcc-poison-system-directories.patch b/meta/recipes-devtools/gcc/gcc-8.2/0003-gcc-poison-system-directories.patch
index 1839d4c7b8e..05138976c3b 100644
--- a/meta/recipes-devtools/gcc/gcc-8.2/0003-gcc-poison-system-directories.patch
+++ b/meta/recipes-devtools/gcc/gcc-8.2/0003-gcc-poison-system-directories.patch
@@ -162,7 +162,7 @@ index b11c6a57939..e3b7a21966f 100644
  
  /* Microsoft Windows does not natively support inodes.
     VMS has non-numeric inodes.  */
-@@ -393,6 +394,24 @@ merge_include_chains (const char *sysroot, cpp_reader *pfile, int verbose)
+@@ -393,6 +394,26 @@ merge_include_chains (const char *sysroot, cpp_reader *pfile, int verbose)
  	}
        fprintf (stderr, _("End of search list.\n"));
      }
@@ -176,7 +176,9 @@ index b11c6a57939..e3b7a21966f 100644
 +         {
 +          if ((!strncmp (p->name, "/usr/include", 12))
 +              || (!strncmp (p->name, "/usr/local/include", 18))
-+              || (!strncmp (p->name, "/usr/X11R6/include", 18)))
++              || (!strncmp (p->name, "/usr/X11R6/include", 18))
++              || (!strncmp (p->name, "/sw/include", 11))
++              || (!strncmp (p->name, "/opt/include", 12)))
 +            warning (OPT_Wpoison_system_directories,
 +                     "include location \"%s\" is unsafe for "
 +                     "cross-compilation",
diff --git a/meta/recipes-devtools/gcc/gcc-8.2/0004-gcc-poison-dir-extend.patch b/meta/recipes-devtools/gcc/gcc-8.2/0004-gcc-poison-dir-extend.patch
deleted file mode 100644
index 1072d5b18fb..00000000000
--- a/meta/recipes-devtools/gcc/gcc-8.2/0004-gcc-poison-dir-extend.patch
+++ /dev/null
@@ -1,39 +0,0 @@
-From 97370eadcd10da5da529dfb734df56993ce24644 Mon Sep 17 00:00:00 2001
-From: Khem Raj <raj.khem@gmail.com>
-Date: Fri, 29 Mar 2013 09:00:34 +0400
-Subject: [PATCH 04/39] gcc-poison-dir-extend
-
-Add /sw/include and /opt/include based on the original
-zecke-no-host-includes.patch patch.  The original patch checked for
-/usr/include, /sw/include and /opt/include and then triggered a failure and
-aborted.
-
-Instead, we add the two missing items to the current scan.  If the user
-wants this to be a failure, they can add "-Werror=poison-system-directories".
-
-Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
-Signed-off-by: Khem Raj <raj.khem@gmail.com>
-
-Upstream-Status: Pending
----
- gcc/incpath.c | 4 +++-
- 1 file changed, 3 insertions(+), 1 deletion(-)
-
-diff --git a/gcc/incpath.c b/gcc/incpath.c
-index e3b7a21966f..9a457e02dd3 100644
---- a/gcc/incpath.c
-+++ b/gcc/incpath.c
-@@ -404,7 +404,9 @@ merge_include_chains (const char *sysroot, cpp_reader *pfile, int verbose)
-          {
-           if ((!strncmp (p->name, "/usr/include", 12))
-               || (!strncmp (p->name, "/usr/local/include", 18))
--              || (!strncmp (p->name, "/usr/X11R6/include", 18)))
-+              || (!strncmp (p->name, "/usr/X11R6/include", 18))
-+              || (!strncmp (p->name, "/sw/include", 11))
-+              || (!strncmp (p->name, "/opt/include", 12)))
-             warning (OPT_Wpoison_system_directories,
-                      "include location \"%s\" is unsafe for "
-                      "cross-compilation",
--- 
-2.17.0
-
-- 
2.17.1

[PATCH 4/8] binutils: Fix variable conflict

[Richard Purdie]

A recent binutils patch added the LDGOLD variable but its already used
for controlling EXTRA_OECONF options for gold. Separate the two variables
to different names to avoid build warnings and confusion.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-devtools/binutils/binutils.inc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/binutils/binutils.inc b/meta/recipes-devtools/binutils/binutils.inc
index b56fcd4cd01..81ecbb50cc1 100644
--- a/meta/recipes-devtools/binutils/binutils.inc
+++ b/meta/recipes-devtools/binutils/binutils.inc
@@ -38,8 +38,8 @@ FILES_${PN}-dev = " \
 # Rather than duplicating multiple entries for these, make one
 # list and reuse it.
 
-LDGOLD ?= "ld.gold dwp"
-LDGOLD_riscv64 = ""
+LDGOLD_ALTS ?= "ld.gold dwp"
+LDGOLD_ALTS_riscv64 = ""
 
 USE_ALTERNATIVES_FOR = " \
 	addr2line \
@@ -50,7 +50,7 @@ USE_ALTERNATIVES_FOR = " \
 	gprof \
 	ld \
 	ld.bfd \
-	${LDGOLD} \
+	${LDGOLD_ALTS} \
 	nm \
 	objcopy \
 	objdump \
-- 
2.17.1

[PATCH 5/8] populate_base_sdk: Stop running gcc --version all the time

[Richard Purdie]

Running 'gcc --version' for every image recipe is slow and increases parsing
time/resource usage for no good reason. Only compute the value in when we're
really running the task/function.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/populate_sdk_base.bbclass | 2 +-
 meta/lib/oe/utils.py                   | 5 ++++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index c456c52866a..bbf1ff1cd29 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -259,7 +259,7 @@ EOF
 		-e 's#@SDK_VERSION@#${SDK_VERSION}#g' \
 		-e '/@SDK_PRE_INSTALL_COMMAND@/d' \
 		-e '/@SDK_POST_INSTALL_COMMAND@/d' \
-		-e 's#@SDK_GCC_VER@#${@oe.utils.host_gcc_version(d)}#g' \
+		-e 's#@SDK_GCC_VER@#${@oe.utils.host_gcc_version(d, taskcontextonly=True)}#g' \
 		${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	# add execution permission
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index f22a6ab596d..fbda7035f1e 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -342,9 +342,12 @@ def format_pkg_list(pkg_dict, ret_format=None):
 
     return '\n'.join(output)
 
-def host_gcc_version(d):
+def host_gcc_version(d, taskcontextonly=False):
     import re, subprocess
 
+    if taskcontextonly and d.getVar('BB_WORKERCONTEXT') != '1':
+        return
+
     compiler = d.getVar("BUILD_CC")
     try:
         env = os.environ.copy()
-- 
2.17.1

[PATCH 6/8] maintainers: Add entry for fribidi

[Richard Purdie]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/conf/distro/include/maintainers.inc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index de4f9af47f6..4ec3c20b3bc 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -164,6 +164,7 @@ RECIPE_MAINTAINER_pn-fontconfig = "Maxin B. John <maxin.john@intel.com>"
 RECIPE_MAINTAINER_pn-foomatic-filters = "Chen Qi <Qi.Chen@windriver.com>"
 RECIPE_MAINTAINER_pn-formfactor = "Maxin B. John <maxin.john@intel.com>"
 RECIPE_MAINTAINER_pn-freetype = "Maxin B. John <maxin.john@intel.com>"
+RECIPE_MAINTAINER_pn-fribidi = "Ross Burton <ross.burton@intel.com>"
 RECIPE_MAINTAINER_pn-fts = "Khem Raj <raj.khem@gmail.com>"
 RECIPE_MAINTAINER_pn-gawk = "Chen Qi <Qi.Chen@windriver.com>"
 RECIPE_MAINTAINER_pn-gcc = "Khem Raj <raj.khem@gmail.com>"
-- 
2.17.1

[PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

[Richard Purdie]

Currently these are in ${PN} and ${PN}-syslog may get replaced by
other packages but update-alternatives would error in the postinst
if other files were installed first. Avoid the problems by putting
the links in the correct package.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-core/busybox/busybox.inc | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc
index 586d5342e60..c26ef56bf78 100644
--- a/meta/recipes-core/busybox/busybox.inc
+++ b/meta/recipes-core/busybox/busybox.inc
@@ -367,7 +367,10 @@ python do_package_prepend () {
             # Match coreutils
             if alt_name == '[':
                 alt_name = 'lbracket'
-            d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name)
+            if alt_name == 'klogd' or alt_name == 'syslogd':
+                d.appendVar('ALTERNATIVE_%s-syslog' % (pn), ' ' + alt_name)
+            else:
+                d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name)
             d.setVarFlag('ALTERNATIVE_LINK_NAME', alt_name, alt_link_name)
             if os.path.exists('%s%s' % (dvar, target)):
                 d.setVarFlag('ALTERNATIVE_TARGET', alt_name, target)
-- 
2.17.1

[PATCH 8/8] busybox: Sync arch and link locations with coreutils

[Richard Purdie]

coreutils installs these into bindir, folllow its lead to avoid postinst
failures when installing coreutils and busybox together.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-core/busybox/busybox.inc | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc
index c26ef56bf78..c02e0623e37 100644
--- a/meta/recipes-core/busybox/busybox.inc
+++ b/meta/recipes-core/busybox/busybox.inc
@@ -208,6 +208,10 @@ do_install () {
 	sed -i "s:^/usr/bin/:BINDIR/:" busybox.links*
 	sed -i "s:^/usr/sbin/:SBINDIR/:" busybox.links*
 
+	# Move arch/link to BINDIR to match coreutils
+	sed -i "s:^BASE_BINDIR/arch:BINDIR/arch:" busybox.links*
+	sed -i "s:^BASE_BINDIR/link:BINDIR/link:" busybox.links*
+
 	sed -i "s:^BASE_BINDIR/:${base_bindir}/:" busybox.links*
 	sed -i "s:^BASE_SBINDIR/:${base_sbindir}/:" busybox.links*
 	sed -i "s:^BINDIR/:${bindir}/:" busybox.links*
-- 
2.17.1

Re: [PATCH 2/8] os-release: fix to install in the expected location

[Andre McCurdy]

On Wed, Sep 5, 2018 at 9:52 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> From: Joshua Lock <joshua.g.lock@intel.com>
>
> os-release (5) recommends that the os-release file be installed in
> /usr/lib/os-release and that /etc/os-release be a relative symlink to it.
>
> Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
> Signed-off-by: Ross Burton <ross.burton@intel.com>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/recipes-core/os-release/os-release.bb | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/meta/recipes-core/os-release/os-release.bb b/meta/recipes-core/os-release/os-release.bb
> index f9887047561..dae5748ba7a 100644
> --- a/meta/recipes-core/os-release/os-release.bb
> +++ b/meta/recipes-core/os-release/os-release.bb
> @@ -1,7 +1,7 @@
>  inherit allarch
>
>  SUMMARY = "Operating system identification"
> -DESCRIPTION = "The /etc/os-release file contains operating system identification data."
> +DESCRIPTION = "The /usr/lib/os-release file contains operating system identification data."
>  LICENSE = "MIT"
>  INHIBIT_DEFAULT_DEPS = "1"
>
> @@ -42,6 +42,9 @@ python do_compile () {
>  do_compile[vardeps] += "${OS_RELEASE_FIELDS}"
>
>  do_install () {
> -    install -d ${D}${sysconfdir}
> -    install -m 0644 os-release ${D}${sysconfdir}/
> +    install -d ${D}${libdir} ${D}${sysconfdir}
> +    install -m 0644 os-release ${D}${libdir}/
> +    lnr ${D}${libdir}/os-release ${D}${sysconfdir}/os-relea

Typo.

Please add the rm too when you fix that.

>  }
> +
> +FILES_${PN} += "${libdir}/os-release"
> --
> 2.17.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 2/8] os-release: fix to install in the expected location

[richard.purdie]

On Wed, 2018-09-05 at 10:16 -0700, Andre McCurdy wrote:
> On Wed, Sep 5, 2018 at 9:52 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > From: Joshua Lock <joshua.g.lock@intel.com>
> > 
> > os-release (5) recommends that the os-release file be installed in
> > /usr/lib/os-release and that /etc/os-release be a relative symlink
> > to it.
> > 
> > Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
> > Signed-off-by: Ross Burton <ross.burton@intel.com>
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  meta/recipes-core/os-release/os-release.bb | 9 ++++++---
> >  1 file changed, 6 insertions(+), 3 deletions(-)
> > 
> > diff --git a/meta/recipes-core/os-release/os-release.bb
> > b/meta/recipes-core/os-release/os-release.bb
> > index f9887047561..dae5748ba7a 100644
> > --- a/meta/recipes-core/os-release/os-release.bb
> > +++ b/meta/recipes-core/os-release/os-release.bb
> > @@ -1,7 +1,7 @@
> >  inherit allarch
> > 
> >  SUMMARY = "Operating system identification"
> > -DESCRIPTION = "The /etc/os-release file contains operating system
> > identification data."
> > +DESCRIPTION = "The /usr/lib/os-release file contains operating
> > system identification data."
> >  LICENSE = "MIT"
> >  INHIBIT_DEFAULT_DEPS = "1"
> > 
> > @@ -42,6 +42,9 @@ python do_compile () {
> >  do_compile[vardeps] += "${OS_RELEASE_FIELDS}"
> > 
> >  do_install () {
> > -    install -d ${D}${sysconfdir}
> > -    install -m 0644 os-release ${D}${sysconfdir}/
> > +    install -d ${D}${libdir} ${D}${sysconfdir}
> > +    install -m 0644 os-release ${D}${libdir}/
> > +    lnr ${D}${libdir}/os-release ${D}${sysconfdir}/os-relea
> 
> Typo.
> 
> Please add the rm too when you fix that.

Too late, was already spotted and fixed. I'm not adding the rm since
we're not going through all the do_installs adding rm statements in for
things, I am totally against doing that.

Cheers,

Richard

Re: [PATCH 4/8] binutils: Fix variable conflict

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 1230 bytes --]



On 9/5/18 9:52 AM, Richard Purdie wrote:
> A recent binutils patch added the LDGOLD variable but its already used
> for controlling EXTRA_OECONF options for gold. Separate the two variables
> to different names to avoid build warnings and confusion.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/recipes-devtools/binutils/binutils.inc | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/recipes-devtools/binutils/binutils.inc b/meta/recipes-devtools/binutils/binutils.inc
> index b56fcd4cd01..81ecbb50cc1 100644
> --- a/meta/recipes-devtools/binutils/binutils.inc
> +++ b/meta/recipes-devtools/binutils/binutils.inc
> @@ -38,8 +38,8 @@ FILES_${PN}-dev = " \
>  # Rather than duplicating multiple entries for these, make one
>  # list and reuse it.
>  
> -LDGOLD ?= "ld.gold dwp"
> -LDGOLD_riscv64 = ""
> +LDGOLD_ALTS ?= "ld.gold dwp"
> +LDGOLD_ALTS_riscv64 = ""
>  
>  USE_ALTERNATIVES_FOR = " \
>  	addr2line \
> @@ -50,7 +50,7 @@ USE_ALTERNATIVES_FOR = " \
>  	gprof \
>  	ld \
>  	ld.bfd \
> -	${LDGOLD} \
> +	${LDGOLD_ALTS} \

aw my bad. Thanks for taking care of it.

>  	nm \
>  	objcopy \
>  	objdump \
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

[ChenQi]

Hi Richard,

I think my previous patch has covered this case.

   git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives
   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busybox-alternatives

Chen Qi (1):
   busybox: fix the alternatives logic


Is there some problem with the patch?

Best Regards,
Chen Qi

On 09/06/2018 12:52 AM, Richard Purdie wrote:
> Currently these are in ${PN} and ${PN}-syslog may get replaced by
> other packages but update-alternatives would error in the postinst
> if other files were installed first. Avoid the problems by putting
> the links in the correct package.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>   meta/recipes-core/busybox/busybox.inc | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc
> index 586d5342e60..c26ef56bf78 100644
> --- a/meta/recipes-core/busybox/busybox.inc
> +++ b/meta/recipes-core/busybox/busybox.inc
> @@ -367,7 +367,10 @@ python do_package_prepend () {
>               # Match coreutils
>               if alt_name == '[':
>                   alt_name = 'lbracket'
> -            d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name)
> +            if alt_name == 'klogd' or alt_name == 'syslogd':
> +                d.appendVar('ALTERNATIVE_%s-syslog' % (pn), ' ' + alt_name)
> +            else:
> +                d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name)
>               d.setVarFlag('ALTERNATIVE_LINK_NAME', alt_name, alt_link_name)
>               if os.path.exists('%s%s' % (dvar, target)):
>                   d.setVarFlag('ALTERNATIVE_TARGET', alt_name, target)

Re: [PATCH 1/8] python: Run pybench once for optimisation task in pgo

[Robert Yang]

Hi RP,

On 09/06/2018 12:52 AM, Richard Purdie wrote:
> From: Ross Burton <ross.burton@intel.com>
> 
> There is no good reason to loop 10 times on the test since the profile
> optimisation won't change between runs, we don't need/want a statistical
> average. This is just burning cpu cycles, just make 1 run.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>   meta/recipes-devtools/python/python3_3.5.5.bb | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-devtools/python/python3_3.5.5.bb b/meta/recipes-devtools/python/python3_3.5.5.bb
> index bfe91fb1c71..8afe15eee6a 100644
> --- a/meta/recipes-devtools/python/python3_3.5.5.bb
> +++ b/meta/recipes-devtools/python/python3_3.5.5.bb
> @@ -74,7 +74,7 @@ TARGET_CC_ARCH += "-DNDEBUG -fno-inline"
>   SDK_CC_ARCH += "-DNDEBUG -fno-inline"
>   EXTRA_OEMAKE += "CROSS_COMPILE=yes"
>   EXTRA_OECONF += "CROSSPYTHONPATH=${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dynload/ --without-ensurepip --enable-optimizations"
> -PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 10"
> +PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 1"

What's the fault if I don't run it, please ? This is very slow, here are some
rough thoughts to speed it up:

1) Make it optional to run, and default to no if there isn't any faults without 
running it.

2) Use qemu-kvm to run the tests if qemu kvm has been setup (I tried it, it can
    reduce form 12m -> 4m, saved 8 mins:
    * Without kvm
      $ bitbake python3 -ccleansstate && time bitbake python3
      real    12m28.070s
      user    0m1.316s
      sys     0m0.184s

    * With kvm:
    real    3m53.781s
    user    0m1.080s
    sys     0m0.220s

   We may need a oe-core/scripts/qemu-kvm-check to check strictly whether kvm
   can be used or not.

Please feel free to give your comments.

// Robert

>   
>   export CROSS_COMPILE = "${TARGET_PREFIX}"
>   export _PYTHON_PROJECT_BASE = "${B}"
> 

Re: [PATCH 1/8] python: Run pybench once for optimisation task in pgo

[richard.purdie]

On Thu, 2018-09-06 at 16:15 +0800, Robert Yang wrote:
> What's the fault if I don't run it, please ? This is very slow, here
> are some rough thoughts to speed it up:
> 
> 1) Make it optional to run, and default to no if there isn't any
> faults without running it.

Without running it, python is not profile optimised and runs slower.

> 2) Use qemu-kvm to run the tests if qemu kvm has been setup (I tried
> it, it can
>     reduce form 12m -> 4m, saved 8 mins:
>     * Without kvm
>       $ bitbake python3 -ccleansstate && time bitbake python3
>       real    12m28.070s
>       user    0m1.316s
>       sys     0m0.184s
> 
>     * With kvm:
>     real    3m53.781s
>     user    0m1.080s
>     sys     0m0.220s
> 
>    We may need a oe-core/scripts/qemu-kvm-check to check strictly
> whether kvm
>    can be used or not.
> 
> Please feel free to give your comments.

Ross has some patches coming in this area which should improve things a
lot so I'd suggest we wait for those. I merged the n10 -> n1 change
simply because it was an easy fix to same some time whilst the other
patches a debugged.

Cheers,

Richard

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

[richard.purdie]

On Thu, 2018-09-06 at 10:11 +0800, ChenQi wrote:
> Hi Richard,
> 
> I think my previous patch has covered this case.
> 
>    git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busyb
> ox-alternatives
> 
> Chen Qi (1):
>    busybox: fix the alternatives logic
> 
> 
> Is there some problem with the patch?

Sorry, I thought I'd replied about this. There were build failures from
this patch and I had to drop it from the test branch. I'm afraid with
all the build failures we're had recently I'm struggling to find which
failures those were...

I merged my simpler but less complete fix simply to get patches moving
as I'd assumed this one was being debugged. I guess we need the
failures for that though...

Cheers,

Richard

Re: [PATCH 1/8] python: Run pybench once for optimisation task in pgo

[Robert Yang]

On 09/06/2018 04:17 PM, richard.purdie@linuxfoundation.org wrote:
> On Thu, 2018-09-06 at 16:15 +0800, Robert Yang wrote:
>> What's the fault if I don't run it, please ? This is very slow, here
>> are some rough thoughts to speed it up:
>>
>> 1) Make it optional to run, and default to no if there isn't any
>> faults without running it.
> 
> Without running it, python is not profile optimised and runs slower.
> 
>> 2) Use qemu-kvm to run the tests if qemu kvm has been setup (I tried
>> it, it can
>>      reduce form 12m -> 4m, saved 8 mins:
>>      * Without kvm
>>        $ bitbake python3 -ccleansstate && time bitbake python3
>>        real    12m28.070s
>>        user    0m1.316s
>>        sys     0m0.184s
>>
>>      * With kvm:
>>      real    3m53.781s
>>      user    0m1.080s
>>      sys     0m0.220s
>>
>>     We may need a oe-core/scripts/qemu-kvm-check to check strictly
>> whether kvm
>>     can be used or not.
>>
>> Please feel free to give your comments.
> 
> Ross has some patches coming in this area which should improve things a
> lot so I'd suggest we wait for those. I merged the n10 -> n1 change
> simply because it was an easy fix to same some time whilst the other
> patches a debugged.

Sounds great, thanks.

// Robert

> 
> Cheers,
> 
> Richard
> 

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

[ChenQi]

On 09/06/2018 04:19 PM, richard.purdie@linuxfoundation.org wrote:
> On Thu, 2018-09-06 at 10:11 +0800, ChenQi wrote:
>> Hi Richard,
>>
>> I think my previous patch has covered this case.
>>
>>     git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives
>>     http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busyb
>> ox-alternatives
>>
>> Chen Qi (1):
>>     busybox: fix the alternatives logic
>>
>>
>> Is there some problem with the patch?
> Sorry, I thought I'd replied about this. There were build failures from
> this patch and I had to drop it from the test branch. I'm afraid with
> all the build failures we're had recently I'm struggling to find which
> failures those were...
>
> I merged my simpler but less complete fix simply to get patches moving
> as I'd assumed this one was being debugged. I guess we need the
> failures for that though...
>
> Cheers,
>
> Richard
>
Got it.
I just looked at some build failures, the ones that might be related are 
those of testimage failures complaining 'No syslog daemon'.
I'll check things locally first.

Best Regards,
Chen Qi

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

[richard.purdie]

On Thu, 2018-09-06 at 17:04 +0800, ChenQi wrote:
> On 09/06/2018 04:19 PM, richard.purdie@linuxfoundation.org wrote:
> > On Thu, 2018-09-06 at 10:11 +0800, ChenQi wrote:
> > > Hi Richard,
> > > 
> > > I think my previous patch has covered this case.
> > > 
> > >     git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives
> > >     http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busyb
> > > ox-alternatives
> > > 
> > > Chen Qi (1):
> > >     busybox: fix the alternatives logic
> > > 
> > > 
> > > Is there some problem with the patch?
> > 
> > Sorry, I thought I'd replied about this. There were build failures from
> > this patch and I had to drop it from the test branch. I'm afraid with
> > all the build failures we're had recently I'm struggling to find which
> > failures those were...
> > 
> > I merged my simpler but less complete fix simply to get patches moving
> > as I'd assumed this one was being debugged. I guess we need the
> > failures for that though...
> > 
> > Cheers,
> > 
> > Richard
> > 
> 
> Got it.
> I just looked at some build failures, the ones that might be related are 
> those of testimage failures complaining 'No syslog daemon'.
> I'll check things locally first.

It seems that we have problems in master even with my version of the
patch. Its not a deterministic problem, it depends upon rpm/dnf package
installation order.

The dnf.DnfRepoTest.test_dnf_installroot image test does:

dnf --repofrompath=oe-testimage-repo-i586,http://192.168.7.7:36895/i586 --repofrompath=oe-testimage-repo-qemux86,http://192.168.7.7:36895/qemux86 --repofrompath=oe-testimage-repo-noarch,http://192.168.7.7:36895/noarch --nogpgcheck install --installroot=/home/root/chroot/test -v -y --rpmverbosity=debug busybox run-postinst

which figures out:

================================================================================
|  Package                   Arch    Version      Repository                 Size
| ================================================================================
| Installing:
|  busybox                   i586    1.29.2-r0.0  oe-testimage-repo-i586    352 k
|  run-postinsts             noarch  1.0-r10.0    oe-testimage-repo-noarch  8.7 k
| Installing dependencies:
|  libc6                     i586    2.28-r0.0    oe-testimage-repo-i586    1.3 M
|  update-alternatives-opkg  i586    0.3.6-r0.0   oe-testimage-repo-i586    8.5 k
| Installing weak dependencies:
|  busybox-syslog            i586    1.29.2-r0.0  oe-testimage-repo-i586     10 k
|  busybox-udhcpc            i586    1.29.2-r0.0  oe-testimage-repo-i586    8.1 k
| 

so far so good. It then tries to install busybox-syslog *before*
busybox. During the postinstall of busybox-syslog we see:

update-alternatives --install /sbin/klogd klogd /bin/busybox.nosuid 50
| /usr/bin/update-alternatives: line 145: sed: command not found
| /usr/bin/update-alternatives: line 145: echo: write error: Broken pipe

which makes sense as sed comes from busybox which hasn't installed yet.

I added:

RDEPENDS_${PN}-syslog = "busybox"

to the busybox recipe but this doesn't seem to help. I see in the
busybox.spec file for the -syslog package:

Requires: /bin/sh
Requires: busybox
Requires: update-alternatives-opkg
Requires(post): /bin/sh
Requires(post): busybox
Requires(post): update-alternatives-opkg

but it still tries to install busybox-syslog *before* busybox.
(busybox only RRECOMMENDS busybox-syslog).

The autobuilder with the above RDEPENDS added also shows this.

Interestingly this test doesn't fail, its the parselogs test later
which fails due to the postinstall errors being detected in the dnf log
files.

This would seem to be an rpm/dnf bug :(. 

Sometimes we don't see the error as if it chooses to install busybox
first, things work ok. THis is currently breaking builds quite badly on
master :(.

Cheers,

Richard

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

[Richard Purdie]

Just to follow up, I'm trying separating out busybox-syslog's
recommends from busybox into the higher level packagegroups. The patch
is in -next and will run tests overnight.

Doing this is probably good anyway but it does feel like we're working
around an rpm/dnf issue. I'm open to better ideas...

Cheers,

Richard

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

[Mark Hatle]

On 9/6/18 5:51 PM, Richard Purdie wrote:
> Just to follow up, I'm trying separating out busybox-syslog's
> recommends from busybox into the higher level packagegroups. The patch
> is in -next and will run tests overnight.
> 
> Doing this is probably good anyway but it does feel like we're working
> around an rpm/dnf issue. I'm open to better ideas...

Circular dependencies can result in non-deterministic behavior.  RPM does not
have any difference in dependency resolution (ordering) for depends vs
recommends.  (The only difference is a recommend is not an error.)  Due to this,
a circular dependency will result in behavior that may change the order of
circular dependent items based on external factors such as file list order, date
codes or ???

RPM does (or at least used to) permit you to specify a way to break circular
dependencies.  But this was a static list of known circular dependencies which
you would simply specify in the macros file which had to be installed first.

The 'correct' way to break this is only specify ACTUAL dependencies, and
realistically there should never be a circular dependency based on a pre-post
install action.  Otherwise, expect non-deterministic breakage.

There is also a way to explicitly set dependencies for the install scriptlets
themselves, but I don't know how this influences the ordering of the
installation.   But this would likely require some special syntax for busybox
(or other early load environments) which likely would be worse then a package group.

(I don't know how or if DNF handles this differently BTW.)

--Mark