[PATCH 0/3] Fixes for build failures for multilibs + usrmerge, NLS and GCONV

[PATCH 0/3] Fixes for build failures for multilibs + usrmerge, NLS and GCONV

[Jason Wessel]

Here are a few more fixes for usrmerge, multilibs, NLS, and GCONV.

Thanks,
Jason.

[PATCH 1/3] psmisc: Fix dependency for USE_NLS=no

[Jason Wessel]

When using USE_NLS="no" in the local.conf psmisc will fail to
compile as follows:

| autoreconf: Entering directory `.'
| autoreconf: running: autopoint --force
| autoreconf: failed to run autopoint: No such file or directory
| autoreconf: autopoint is needed because this package uses Gettext
| ERROR: autoreconf execution failed.

This is because the gettext.bbclass returns gettext-minimal-native for
the host dependency which does not include autopoint.  The autopoint
utility is required to build psmisc, so it needs to list
gettext-native as a dependency.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/recipes-extended/psmisc/psmisc.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-extended/psmisc/psmisc.inc b/meta/recipes-extended/psmisc/psmisc.inc
index 82ef947709..594a10cf22 100644
--- a/meta/recipes-extended/psmisc/psmisc.inc
+++ b/meta/recipes-extended/psmisc/psmisc.inc
@@ -7,7 +7,7 @@ command sends a specified signal (SIGTERM if nothing is specified) to \
 processes identified by name.  The fuser command identifies the PIDs \
 of processes that are using specified files or filesystems."
 SECTION = "base"
-DEPENDS = "ncurses virtual/libintl"
+DEPENDS = "ncurses virtual/libintl gettext-native"
 LICENSE = "GPLv2"
 
 SRC_URI = "${SOURCEFORGE_MIRROR}/psmisc/psmisc-${PV}.tar.gz"
-- 
2.21.0

[PATCH 2/3] glibc-locale: Fix build error with PACKAGE_NO_GCONV = "1"

[Jason Wessel]

When the PACKAGE_NO_GCONV is set to 1 an empty directory is left behind from the do_install rule:

=====
ERROR: glibc-locale-2.29-r0 do_package: QA Issue: glibc-locale: Files/directories were installed but not shipped in any package:
  /usr/lib
  /usr/lib/locale
Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.
glibc-locale: 2 installed and not shipped files. [installed-vs-shipped]
ERROR: glibc-locale-2.29-r0 do_package: Fatal QA errors found, failing task.
=====

The simple fix is to prune the empty directory.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/recipes-core/glibc/glibc-locale.inc | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
index f033635bd0..bf5eaee938 100644
--- a/meta/recipes-core/glibc/glibc-locale.inc
+++ b/meta/recipes-core/glibc/glibc-locale.inc
@@ -87,6 +87,9 @@ do_install() {
 	if [ ${PACKAGE_NO_GCONV} -eq 0 ]; then
 		copy_locale_files ${libdir}/gconv 0755
 		copy_locale_files ${datadir}/i18n 0644
+	else
+		# Remove the libdir if it is empty when gconv is not copied
+		find ${D}${libdir} -type d -empty -delete
 	fi
 	copy_locale_files ${datadir}/locale 0644
 	install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
-- 
2.21.0

[PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

[Jason Wessel]

The do_stash_locale was not working consistently across the 4 build
configurations and the multilib, usrmerge configuration would fail
entirely with the obscure message:

| DEBUG: Executing shell function do_prep_locale_tree
| tar: i18n: Cannot stat: No such file or directory
| tar: Exiting with failure status due to previous errors
| gzip: /poky/build/tmp/work/core2-64-poky-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz: No such file or directory
| WARNING: /poky/build/tmp/work/core2-64-poky-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.124690:1 exit 1 from 'gunzip $i'

Here is the 4 build configurations without the patch applied:

A) x86-64 no multilibs, no usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l
909
B) x86-64 no multilibs, usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l
909
C) x86-64 multilibs, no usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l
885
D) x86-64 multilibs, usrmerge
find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v nscd.service |wc -l
864

The issue here is that all the moves should be processed first, then a
copy should be made of the lib directories, but only in the case they
are different when using the usrmerge feature.  Even though the build
worked for the multilib configuration without usrmerge, the content
was not the same.

After applying the patch the same number of files are in all the
configurations.  The list of files was also diffed, after normalizing
the directory names to ensure all the correct files were copied.

Ultimately there are probably additional files that should be pruned
from what is copied to the stated_locale, but the purpose of this
patch is make it 100% consistent between the build types and fix the
builds.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/recipes-core/glibc/glibc-package.inc | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index a1d79b3075..ff17a193c3 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -162,21 +162,28 @@ bashscripts = "mtrace sotruss xtrace"
 do_stash_locale () {
 	dest=${LOCALESTASH}
 	install -d ${dest}${base_libdir} ${dest}${bindir} ${dest}${libdir} ${dest}${datadir}
-	if [ "${base_libdir}" != "${libdir}" ]; then
-		cp -fpPR ${D}${base_libdir}/* ${dest}${base_libdir}
-	fi
+	# Hide away the locale data from the deployment
 	if [ -e ${D}${bindir}/localedef ]; then
 		mv -f ${D}${bindir}/localedef ${dest}${bindir}
 	fi
 	if [ -e ${D}${libdir}/gconv ]; then
 		mv -f ${D}${libdir}/gconv ${dest}${libdir}
 	fi
-	if [ -e ${D}${exec_prefix}/lib ]; then
-		cp -fpPR ${D}${exec_prefix}/lib ${dest}${exec_prefix}
-	fi
 	if [ -e ${D}${datadir}/i18n ]; then
 		mv ${D}${datadir}/i18n ${dest}${datadir}
 	fi
+
+	# Make a copy of all the libraries into the locale stash
+	cp -fpPR ${D}${libdir}/* ${dest}${libdir}
+	if [ "${base_libdir}" != "${libdir}" ]; then
+		cp -fpPR ${D}${base_libdir}/* ${dest}${base_libdir}
+	fi
+	if [ -e ${D}${exec_prefix}/lib ]; then
+		if [ ${exec_prefix}/lib != ${base_libdir} ] && [ ${exec_prefix}/lib != ${libdir} ]; then
+			cp -fpPR ${D}${exec_prefix}/lib ${dest}${exec_prefix}
+		fi
+	fi
+
 	cp -fpPR ${D}${datadir}/* ${dest}${datadir}
 	rm -rf ${D}${datadir}/locale/
 	cp -fpPR ${WORKDIR}/SUPPORTED ${dest}
-- 
2.21.0

Re: [PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 5088 bytes --]

I don't see how this change (or the previous glibc-locale one) would cause
that, but since the oe-core upgrade yesterday I'm seeing following
glibc-locale.do_package failure:
http://errors.yoctoproject.org/Errors/Details/250557/

DEBUG: Executing shell function do_prep_locale_tree
tar: i18n: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
gzip:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
No such file or directory
WARNING:
TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.23303:1
exit 1 from 'gunzip $i'

anyone else seeing this?

On Tue, Jul 2, 2019 at 10:16 PM Jason Wessel <jason.wessel@windriver.com>
wrote:

> The do_stash_locale was not working consistently across the 4 build
> configurations and the multilib, usrmerge configuration would fail
> entirely with the obscure message:
>
> | DEBUG: Executing shell function do_prep_locale_tree
> | tar: i18n: Cannot stat: No such file or directory
> | tar: Exiting with failure status due to previous errors
> | gzip:
> /poky/build/tmp/work/core2-64-poky-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
> No such file or directory
> | WARNING:
> /poky/build/tmp/work/core2-64-poky-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.124690:1
> exit 1 from 'gunzip $i'
>
> Here is the 4 build configurations without the patch applied:
>
> A) x86-64 no multilibs, no usrmerge
> find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v
> nscd.service |wc -l
> 909
> B) x86-64 no multilibs, usrmerge
> find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v
> nscd.service |wc -l
> 909
> C) x86-64 multilibs, no usrmerge
> find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v
> nscd.service |wc -l
> 885
> D) x86-64 multilibs, usrmerge
> find ./tmp/work/*/glibc/2.29-r0/stashed-locale -type f |grep -v
> nscd.service |wc -l
> 864
>
> The issue here is that all the moves should be processed first, then a
> copy should be made of the lib directories, but only in the case they
> are different when using the usrmerge feature.  Even though the build
> worked for the multilib configuration without usrmerge, the content
> was not the same.
>
> After applying the patch the same number of files are in all the
> configurations.  The list of files was also diffed, after normalizing
> the directory names to ensure all the correct files were copied.
>
> Ultimately there are probably additional files that should be pruned
> from what is copied to the stated_locale, but the purpose of this
> patch is make it 100% consistent between the build types and fix the
> builds.
>
> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
> ---
>  meta/recipes-core/glibc/glibc-package.inc | 19 +++++++++++++------
>  1 file changed, 13 insertions(+), 6 deletions(-)
>
> diff --git a/meta/recipes-core/glibc/glibc-package.inc
> b/meta/recipes-core/glibc/glibc-package.inc
> index a1d79b3075..ff17a193c3 100644
> --- a/meta/recipes-core/glibc/glibc-package.inc
> +++ b/meta/recipes-core/glibc/glibc-package.inc
> @@ -162,21 +162,28 @@ bashscripts = "mtrace sotruss xtrace"
>  do_stash_locale () {
>         dest=${LOCALESTASH}
>         install -d ${dest}${base_libdir} ${dest}${bindir} ${dest}${libdir}
> ${dest}${datadir}
> -       if [ "${base_libdir}" != "${libdir}" ]; then
> -               cp -fpPR ${D}${base_libdir}/* ${dest}${base_libdir}
> -       fi
> +       # Hide away the locale data from the deployment
>         if [ -e ${D}${bindir}/localedef ]; then
>                 mv -f ${D}${bindir}/localedef ${dest}${bindir}
>         fi
>         if [ -e ${D}${libdir}/gconv ]; then
>                 mv -f ${D}${libdir}/gconv ${dest}${libdir}
>         fi
> -       if [ -e ${D}${exec_prefix}/lib ]; then
> -               cp -fpPR ${D}${exec_prefix}/lib ${dest}${exec_prefix}
> -       fi
>         if [ -e ${D}${datadir}/i18n ]; then
>                 mv ${D}${datadir}/i18n ${dest}${datadir}
>         fi
> +
> +       # Make a copy of all the libraries into the locale stash
> +       cp -fpPR ${D}${libdir}/* ${dest}${libdir}
> +       if [ "${base_libdir}" != "${libdir}" ]; then
> +               cp -fpPR ${D}${base_libdir}/* ${dest}${base_libdir}
> +       fi
> +       if [ -e ${D}${exec_prefix}/lib ]; then
> +               if [ ${exec_prefix}/lib != ${base_libdir} ] && [
> ${exec_prefix}/lib != ${libdir} ]; then
> +                       cp -fpPR ${D}${exec_prefix}/lib
> ${dest}${exec_prefix}
> +               fi
> +       fi
> +
>         cp -fpPR ${D}${datadir}/* ${dest}${datadir}
>         rm -rf ${D}${datadir}/locale/
>         cp -fpPR ${WORKDIR}/SUPPORTED ${dest}
> --
> 2.21.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 6167 bytes --]

Re: [PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

[Adrian Bunk]

On Thu, Jul 04, 2019 at 11:13:18AM +0200, Martin Jansa wrote:
> I don't see how this change (or the previous glibc-locale one) would cause
> that, but since the oe-core upgrade yesterday I'm seeing following
> glibc-locale.do_package failure:
> http://errors.yoctoproject.org/Errors/Details/250557/
> 
> DEBUG: Executing shell function do_prep_locale_tree
> tar: i18n: Cannot stat: No such file or directory
> tar: Exiting with failure status due to previous errors
> gzip:
> TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
> No such file or directory
> WARNING:
> TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.23303:1
> exit 1 from 'gunzip $i'
> 
> anyone else seeing this?

Yes.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

[Jason Wessel]

On 7/4/19 5:57 AM, Adrian Bunk wrote:
> On Thu, Jul 04, 2019 at 11:13:18AM +0200, Martin Jansa wrote:
>> I don't see how this change (or the previous glibc-locale one) would cause
>> that, but since the oe-core upgrade yesterday I'm seeing following
>> glibc-locale.do_package failure:
>> http://errors.yoctoproject.org/Errors/Details/250557/
>>
>> DEBUG: Executing shell function do_prep_locale_tree
>> tar: i18n: Cannot stat: No such file or directory
>> tar: Exiting with failure status due to previous errors
>> gzip:
>> TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
>> No such file or directory
>> WARNING:
>> TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.23303:1
>> exit 1 from 'gunzip $i'
>>
>> anyone else seeing this?
> 
> Yes.
> 


I don't know that it was specifically this change that causes the problem but rather any change to this function.   If that is not the case can either of you send me your local.conf file, and I'll try to reproduce it.  If the following doesn't work around the problem, there is clearly another corner case that is not well understood.

bitbake -c cleansstate glibc glibc-locale

I believe that any change to the .inc file which has locale stash function will cause the problem due to the fact it performs and "mv" operation to move the files.  When it runs a second time, there are no more files to move and you will not have them for use later on.  When I was experimenting earlier, I tried running the glibc temp/run.do_install script to put the files back, but then hit a problem with localdef being missing, because it comes from the temp/run.do_compile functions.

My conclusion thus far has been is that the ties between the glibc and glibc-locale are very subtle and break easily.  This is a separate problem than what I was attempting to fix, where the usrmerge + multilibs didn't even get the correct files in the first place.

Cheers,

Jason.

Re: [PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

[Adrian Bunk]

On Thu, Jul 04, 2019 at 06:57:50AM -0500, Jason Wessel wrote:
> On 7/4/19 5:57 AM, Adrian Bunk wrote:
> > On Thu, Jul 04, 2019 at 11:13:18AM +0200, Martin Jansa wrote:
> > > I don't see how this change (or the previous glibc-locale one) would cause
> > > that, but since the oe-core upgrade yesterday I'm seeing following
> > > glibc-locale.do_package failure:
> > > http://errors.yoctoproject.org/Errors/Details/250557/
> > > 
> > > DEBUG: Executing shell function do_prep_locale_tree
> > > tar: i18n: Cannot stat: No such file or directory
> > > tar: Exiting with failure status due to previous errors
> > > gzip:
> > > TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
> > > No such file or directory
> > > WARNING:
> > > TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.23303:1
> > > exit 1 from 'gunzip $i'
> > > 
> > > anyone else seeing this?
> > 
> > Yes.
> 
> 
> I don't know that it was specifically this change that causes the problem but rather any change to this function.   If that is not the case can either of you send me your local.conf file, and I'll try to reproduce it.  If the following doesn't work around the problem, there is clearly another corner case that is not well understood.
> 
> bitbake -c cleansstate glibc glibc-locale
>...

bitbake -c cleansstate glibc glibc-locale nativesdk-glibc nativesdk-glibc-locale

This seems to workaround the problem in the cases I have seen so far,
so what you said about the cause looks plausible to me.

> Cheers,
> 
> Jason.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH 3/3] glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs

[Jason Wessel]

On 7/6/19 2:31 PM, Adrian Bunk wrote:
> On Thu, Jul 04, 2019 at 06:57:50AM -0500, Jason Wessel wrote:
>> On 7/4/19 5:57 AM, Adrian Bunk wrote:
>>> On Thu, Jul 04, 2019 at 11:13:18AM +0200, Martin Jansa wrote:
>>>> I don't see how this change (or the previous glibc-locale one) would cause
>>>> that, but since the oe-core upgrade yesterday I'm seeing following
>>>> glibc-locale.do_package failure:
>>>> http://errors.yoctoproject.org/Errors/Details/250557/
>>>>
>>>> DEBUG: Executing shell function do_prep_locale_tree
>>>> tar: i18n: Cannot stat: No such file or directory
>>>> tar: Exiting with failure status due to previous errors
>>>> gzip:
>>>> TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/locale-tree//usr/share/i18n/charmaps/*gz.gz:
>>>> No such file or directory
>>>> WARNING:
>>>> TOPDIR/tmp-glibc/work/core2-64-oe-linux/glibc-locale/2.29-r0/temp/run.do_prep_locale_tree.23303:1
>>>> exit 1 from 'gunzip $i'
>>>>
>>>> anyone else seeing this?
>>>
>>> Yes.
>>
>>
>> I don't know that it was specifically this change that causes the problem but rather any change to this function.   If that is not the case can either of you send me your local.conf file, and I'll try to reproduce it.  If the following doesn't work around the problem, there is clearly another corner case that is not well understood.
>>
>> bitbake -c cleansstate glibc glibc-locale
>> ...
> 
> bitbake -c cleansstate glibc glibc-locale nativesdk-glibc nativesdk-glibc-locale
> 
> This seems to workaround the problem in the cases I have seen so far,
> so what you said about the cause looks plausible to me.
> 


I chatted with Richard to figure out how to fix the original implementation, so this will not continue to happen.  I am no fan of incremental or sstate based builds failing.  The pending fix is below.

https://patchwork.openembedded.org/patch/162886/

Cheers,
Jason.