[PATCH 0/2 v2] glibc: fixes for nscd and libnss-db

[PATCH 0/2 v2] glibc: fixes for nscd and libnss-db

[jackie.huang]

From: Jackie Huang <jackie.huang@windriver.com>

v2 comments:
Add a note for external toolchain in the commit log.

--
The following changes since commit 514a808f21c37b6ad704ce397bb2740ecc9a93bc:

  ref-manual: Updates to "Image Generation" section. (2017-10-30 15:55:08 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib.git jhuang0/d_nscd_171102_0
  http://git.pokylinux.org/cgit.cgi//log/?h=jhuang0/d_nscd_171102_0

Jackie Huang (2):
  glibc: re-package for libnss-db
  glibc/nscd: do not cache for netgroup by default

 meta/recipes-core/glibc/glibc-package.inc | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

-- 
2.11.0

[PATCH 1/2 v2] glibc: re-package for libnss-db

[jackie.huang]

From: Jackie Huang <jackie.huang@windriver.com>

On other distros like ubuntu/centos, libnss-db usually provides:
- The libraries
- The Makefile to create database
  (in /var/db for centos, /var/lib/misc/ for ubuntu)
- The makedb command (it's in glibc-common for centos7)

What we had is:
- The libraries are in glibc-extra-nss
- The Makefile is removed
- The makedb command is in glibc-utils (lack of dependency)

So when glibc-extra-nss is installed but glibc-utils is not,
we see error like:
nscd[165]: 165 checking for monitored file `/var/db/group.db': No such file or directory
nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No such file or directory

And there is not an easy way to create these databases.

To fix the issue:
- Re-package the libraries into libnss-db
- Don't remove the Makefile and add it in libnss-db
- Add RDEPENDS for libnss-db on glibc-utils and make

Notes:
For external toolchain, an extra package 'libnss-db' need to be provided
If replacing glibc from core.

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
---
 meta/recipes-core/glibc/glibc-package.inc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index df3db2cc45..1c3782dbb0 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -1,6 +1,6 @@
 INHIBIT_SYSROOT_STRIP = "1"
 
-PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
+PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libnss-db libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
 
 # The ld.so in this glibc supports the GNU_HASH
 RPROVIDES_${PN} = "eglibc rtld(GNU_HASH)"
@@ -23,6 +23,8 @@ FILES_ldd = "${bindir}/ldd"
 FILES_libsegfault = "${base_libdir}/libSegFault*"
 FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
 FILES_libmemusage = "${base_libdir}/libmemusage.so"
+FILES_libnss-db = "${base_libdir}/libnss_db.so.* ${base_libdir}/libnss_db-*.so ${localstatedir}/db/Makefile"
+RDEPENDS_libnss-db = "${PN}-utils make"
 FILES_glibc-extra-nss = "${base_libdir}/libnss_*-*.so ${base_libdir}/libnss_*.so.*"
 FILES_sln = "${base_sbindir}/sln"
 FILES_${PN}-pic = "${libdir}/*_pic.a ${libdir}/*_pic.map ${libdir}/libc_pic/*.o"
@@ -59,7 +61,6 @@ inherit libc-common multilib_header
 
 do_install_append () {
 	rm -f ${D}${sysconfdir}/localtime
-	rm -rf ${D}${localstatedir}
 
 	# remove empty glibc dir
 	if [ -d ${D}${libexecdir} ]; then
-- 
2.11.0

[PATCH 2/2 v2] glibc/nscd: do not cache for netgroup by default

[jackie.huang]

From: Jackie Huang <jackie.huang@windriver.com>

We don't have /etc/netgroup by default, so do not
cache for netgroup by default to avoid:
nscd[529]: 529 disabled inotify-based monitoring for file `/etc/netgroup': No such file or directory
nscd[529]: 529 stat failed for file `/etc/netgroup'; will try again later: No such file or directory

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
---
 meta/recipes-core/glibc/glibc-package.inc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index 1c3782dbb0..c4ca94c981 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -98,6 +98,7 @@ do_install_append () {
 	install -m 0755 ${S}/nscd/nscd.init ${D}${sysconfdir}/init.d/nscd
 	install -m 0755 ${S}/nscd/nscd.conf ${D}${sysconfdir}/nscd.conf
 	sed -i "s%daemon%start-stop-daemon --start --exec%g" ${D}${sysconfdir}/init.d/nscd
+	sed -i "s|\(enable-cache\t\+netgroup\t\+\)yes|\1no|" ${D}${sysconfdir}/nscd.conf
 
 	install -d ${D}${systemd_unitdir}/system
 	install -m 0644 ${S}/nscd/nscd.service ${D}${systemd_unitdir}/system/
-- 
2.11.0

Re: [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db

[Huang, Jie (Jackie)]

Ping, I didn't see any objection on this, but it's not merged yet, do I miss anything?

Thanks,
Jackie

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> jackie.huang@windriver.com
> Sent: Thursday, November 02, 2017 14:41
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db
> 
> From: Jackie Huang <jackie.huang@windriver.com>
> 
> v2 comments:
> Add a note for external toolchain in the commit log.
> 
> --
> The following changes since commit
> 514a808f21c37b6ad704ce397bb2740ecc9a93bc:
> 
>   ref-manual: Updates to "Image Generation" section. (2017-10-30 15:55:08
> +0000)
> 
> are available in the git repository at:
> 
>   git://git.pokylinux.org/poky-contrib.git jhuang0/d_nscd_171102_0
>   http://git.pokylinux.org/cgit.cgi//log/?h=jhuang0/d_nscd_171102_0
> 
> Jackie Huang (2):
>   glibc: re-package for libnss-db
>   glibc/nscd: do not cache for netgroup by default
> 
>  meta/recipes-core/glibc/glibc-package.inc | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> --
> 2.11.0
> 
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db

[Richard Purdie]

On Fri, 2017-12-22 at 02:08 +0000, Huang, Jie (Jackie) wrote:
> Ping, I didn't see any objection on this, but it's not merged yet, do
> I miss anything?

When we test it we see:

WARNING: glibc-2.26-r0 do_package: glibc-extra-nss-2.26 was registered as shlib provider for libnss_db.so.2, changing it to libnss-db-2.26 because it was built later

so it looks like you swapped one race for another?

Cheers,

Richard

Re: [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db

[Huang, Jie (Jackie)]

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: Wednesday, January 03, 2018 21:54
> To: Huang, Jie (Jackie); openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db
> 
> On Fri, 2017-12-22 at 02:08 +0000, Huang, Jie (Jackie) wrote:
> > Ping, I didn't see any objection on this, but it's not merged yet, do
> > I miss anything?
> 
> When we test it we see:
> 
> WARNING: glibc-2.26-r0 do_package: glibc-extra-nss-2.26 was registered as
> shlib provider for libnss_db.so.2, changing it to libnss-db-2.26 because it was
> built later

Sorry, but I haven't seen this warning in our builds after this patch merged in our 
local branch for two weeks. Maybe we missed some cases in our test builds, so
could you show me how to reproduce it? Thanks!

This patch is simply re-package libnss_db.so* from glibc-extra-nss into libnss-db,
I don't understand why libnss_db.so* is still provided by both of them, did I miss
anything when I do a re-packaging for a recipe?

Thanks,
Jackie

> 
> so it looks like you swapped one race for another?
> 
> Cheers,
> 
> Richard

Re: [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db

[Richard Purdie]

On Thu, 2018-01-04 at 01:31 +0000, Huang, Jie (Jackie) wrote:
> 
> > 
> > -----Original Message-----
> > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > Sent: Wednesday, January 03, 2018 21:54
> > To: Huang, Jie (Jackie); openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH 0/2 v2] glibc: fixes for nscd and
> > libnss-db
> > 
> > On Fri, 2017-12-22 at 02:08 +0000, Huang, Jie (Jackie) wrote:
> > > 
> > > Ping, I didn't see any objection on this, but it's not merged
> > > yet, do
> > > I miss anything?
> > When we test it we see:
> > 
> > WARNING: glibc-2.26-r0 do_package: glibc-extra-nss-2.26 was
> > registered as
> > shlib provider for libnss_db.so.2, changing it to libnss-db-2.26
> > because it was
> > built later
> Sorry, but I haven't seen this warning in our builds after this patch
> merged in our local branch for two weeks. Maybe we missed some cases
> in our test builds, so could you show me how to reproduce it? Thanks!
> 
> This patch is simply re-package libnss_db.so* from glibc-extra-nss
> into libnss-db, I don't understand why libnss_db.so* is still
> provided by both of them, did I miss anything when I do a re-
> packaging for a recipe?


I was able to reproduce this by starting with an existing build without
the patch, then applying the patch. The error shows once during the
rebuild and then not again. Its therefore probably less of an issue
(but does raise a question about why that is happening).

The new make dependency does cause an additional failure though:

$ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3

2018-01-12 08:31:25,196 - oe-selftest - INFO - test_non_gplv3 (bbtests.BitbakeTests)
2018-01-12 08:34:02,940 - oe-selftest - INFO -  ... FAIL
2018-01-12 08:34:02,941 - oe-selftest - INFO - ======================================================================
2018-01-12 08:34:02,941 - oe-selftest - INFO - FAIL: test_non_gplv3 (bbtests.BitbakeTests)
2018-01-12 08:34:02,941 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-01-12 08:34:02,941 - oe-selftest - INFO - Traceback (most recent call last):
  File "/media/build1/poky/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/media/build1/poky/meta/lib/oeqa/selftest/cases/bbtests.py", line 237, in test_non_gplv3
    self.assertEqual(result.status, 0, "Bitbake failed, exit code %s, output %s" % (result.status, result.output))
AssertionError: 1 != 0 : Bitbake failed, exit code 1, output Parsing recipes...done.
Parsing of 829 .bb files complete (0 cached, 829 parsed). 1287 targets, 99 skipped, 0 masked, 0 errors.
Removing 35 recipes from the core2-64 sysroot...done.
Removing 35 recipes from the qemux86-64 sysroot...done.
NOTE: Resolving any missing task queue dependencies
ERROR: Nothing RPROVIDES 'make' (but /media/build1/poky/meta/recipes-core/glibc/glibc_2.26.bb RDEPENDS on or otherwise requires it)
make was skipped: it has an incompatible license: GPLv3 & LGPLv2
NOTE: Runtime target 'make' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['make']
ERROR: Required build target 'selftest-ed' has no buildable providers.
Missing or unbuildable dependency chain was: ['selftest-ed', 'virtual/libc', 'make']


Cheers,

Richard

Re: [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db

[Huang, Jie (Jackie)]

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: Friday, January 12, 2018 16:43
> To: Huang, Jie (Jackie)
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db
> 
> On Thu, 2018-01-04 at 01:31 +0000, Huang, Jie (Jackie) wrote:
> >
> > >
> > > -----Original Message-----
> > > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > > Sent: Wednesday, January 03, 2018 21:54
> > > To: Huang, Jie (Jackie); openembedded-core@lists.openembedded.org
> > > Subject: Re: [OE-core] [PATCH 0/2 v2] glibc: fixes for nscd and
> > > libnss-db
> > >
> > > On Fri, 2017-12-22 at 02:08 +0000, Huang, Jie (Jackie) wrote:
> > > >
> > > > Ping, I didn't see any objection on this, but it's not merged
> > > > yet, do
> > > > I miss anything?
> > > When we test it we see:
> > >
> > > WARNING: glibc-2.26-r0 do_package: glibc-extra-nss-2.26 was
> > > registered as
> > > shlib provider for libnss_db.so.2, changing it to libnss-db-2.26
> > > because it was
> > > built later
> > Sorry, but I haven't seen this warning in our builds after this patch
> > merged in our local branch for two weeks. Maybe we missed some cases
> > in our test builds, so could you show me how to reproduce it? Thanks!
> >
> > This patch is simply re-package libnss_db.so* from glibc-extra-nss
> > into libnss-db, I don't understand why libnss_db.so* is still
> > provided by both of them, did I miss anything when I do a re-
> > packaging for a recipe?
> 
> 
> I was able to reproduce this by starting with an existing build without
> the patch, then applying the patch. The error shows once during the
> rebuild and then not again. Its therefore probably less of an issue
> (but does raise a question about why that is happening).
> 
> The new make dependency does cause an additional failure though:

Sorry, I didn't noticed that make is gplv3. I change the dependency to
RSUGGESTS and the test passed:

$ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3
2018-01-16 21:24:54,920 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2018-01-16 21:25:05,010 - oe-selftest - INFO - Adding layer libraries:
2018-01-16 21:25:05,011 - oe-selftest - INFO -  /buildarea2/jhuang0/git/yp/poky/meta/lib
2018-01-16 21:25:05,011 - oe-selftest - INFO -  /buildarea2/jhuang0/git/yp/poky/meta-yocto-bsp/lib
2018-01-16 21:25:05,011 - oe-selftest - INFO -  /buildarea2/jhuang0/git/yp/poky/meta-selftest/lib
2018-01-16 21:25:05,012 - oe-selftest - INFO - Running bitbake -p
2018-01-16 21:25:10,565 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea2/jhuang0/yp/y_x64_180112/conf/local.conf
2018-01-16 21:25:10,566 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2018-01-16 21:25:10,566 - oe-selftest - INFO - test_non_gplv3 (bbtests.BitbakeTests)
2018-01-16 21:46:22,538 - oe-selftest - INFO -  ... ok
2018-01-16 21:46:22,539 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-01-16 21:46:22,539 - oe-selftest - INFO - Ran 1 test in 1271.974s
2018-01-16 21:46:22,539 - oe-selftest - INFO - OK
2018-01-16 21:46:22,539 - oe-selftest - INFO - RESULTS:
2018-01-16 21:46:22,539 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_non_gplv3 - Testcase 1119: PASSED
2018-01-16 21:46:22,540 - oe-selftest - INFO - SUMMARY:
2018-01-16 21:46:22,540 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 1271.975s
2018-01-16 21:46:22,540 - oe-selftest - INFO - oe-selftest - OK - All required tests passed

Thanks,
Jackie

> 
> $ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3
> 
> 2018-01-12 08:31:25,196 - oe-selftest - INFO - test_non_gplv3
> (bbtests.BitbakeTests)
> 2018-01-12 08:34:02,940 - oe-selftest - INFO -  ... FAIL
> 2018-01-12 08:34:02,941 - oe-selftest - INFO -
> =================================================================
> =====
> 2018-01-12 08:34:02,941 - oe-selftest - INFO - FAIL: test_non_gplv3
> (bbtests.BitbakeTests)
> 2018-01-12 08:34:02,941 - oe-selftest - INFO - -----------------------------------------
> -----------------------------
> 2018-01-12 08:34:02,941 - oe-selftest - INFO - Traceback (most recent call last):
>   File "/media/build1/poky/meta/lib/oeqa/core/decorator/__init__.py", line 32,
> in wrapped_f
>     return func(*args, **kwargs)
>   File "/media/build1/poky/meta/lib/oeqa/selftest/cases/bbtests.py", line 237, in
> test_non_gplv3
>     self.assertEqual(result.status, 0, "Bitbake failed, exit code %s, output %s" %
> (result.status, result.output))
> AssertionError: 1 != 0 : Bitbake failed, exit code 1, output Parsing recipes...done.
> Parsing of 829 .bb files complete (0 cached, 829 parsed). 1287 targets, 99
> skipped, 0 masked, 0 errors.
> Removing 35 recipes from the core2-64 sysroot...done.
> Removing 35 recipes from the qemux86-64 sysroot...done.
> NOTE: Resolving any missing task queue dependencies
> ERROR: Nothing RPROVIDES 'make' (but /media/build1/poky/meta/recipes-
> core/glibc/glibc_2.26.bb RDEPENDS on or otherwise requires it)
> make was skipped: it has an incompatible license: GPLv3 & LGPLv2
> NOTE: Runtime target 'make' is unbuildable, removing...
> Missing or unbuildable dependency chain was: ['make']
> ERROR: Required build target 'selftest-ed' has no buildable providers.
> Missing or unbuildable dependency chain was: ['selftest-ed', 'virtual/libc', 'make']
> 
> 
> Cheers,
> 
> Richard