[PATCH 0/1 v3] glibc: re-package for libnss-db

[PATCH 0/1 v3] glibc: re-package for libnss-db

[jackie.huang]

From: Jackie Huang <jackie.huang@windriver.com>

v3 comments:
Change the dependency on make to RSUGGESTS.

--
The following changes since commit b9dcab772fd37ed8376a87d3194195e13ff52d10:

  qemu: CVE-2017-17381 (2018-01-14 09:11:58 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib.git jhuang0/d_nscd_v3_180116_0
  http://git.pokylinux.org/cgit.cgi//log/?h=jhuang0/d_nscd_v3_180116_0

Jackie Huang (1):
  glibc: re-package for libnss-db

 meta/recipes-core/glibc/glibc-package.inc | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

-- 
2.11.0

[PATCH 1/1] glibc: re-package for libnss-db

[jackie.huang]

From: Jackie Huang <jackie.huang@windriver.com>

On other distros like ubuntu/centos, libnss-db usually provides:
- The libraries
- The Makefile to create database
  (in /var/db for centos, /var/lib/misc/ for ubuntu)
- The makedb command (it's in glibc-common for centos7)

What we had is:
- The libraries are in glibc-extra-nss
- The Makefile is removed
- The makedb command is in glibc-utils (lack of dependency)

So when glibc-extra-nss is installed but glibc-utils is not,
we see error like:
nscd[165]: 165 checking for monitored file `/var/db/group.db': No such file or directory
nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No such file or directory

And there is not an easy way to create these databases.

To fix the issue:
- Re-package the libraries into libnss-db
- Don't remove the Makefile and add it in libnss-db
- Add RDEPENDS on glibc-utils for libnss-db, and add RSUGGESTS on make
  since it's optional to generate the db with make.

Notes:
For external toolchain, an extra package 'libnss-db' need to be provided
If replacing glibc from core.

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
---
 meta/recipes-core/glibc/glibc-package.inc | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index 817d850358..6ee33e61e6 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -1,6 +1,6 @@
 INHIBIT_SYSROOT_STRIP = "1"
 
-PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
+PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libnss-db libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
 
 # The ld.so in this glibc supports the GNU_HASH
 RPROVIDES_${PN} = "eglibc rtld(GNU_HASH)"
@@ -23,6 +23,9 @@ FILES_ldd = "${bindir}/ldd"
 FILES_libsegfault = "${base_libdir}/libSegFault*"
 FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
 FILES_libmemusage = "${base_libdir}/libmemusage.so"
+FILES_libnss-db = "${base_libdir}/libnss_db.so.* ${base_libdir}/libnss_db-*.so ${localstatedir}/db/Makefile"
+RDEPENDS_libnss-db = "${PN}-utils"
+RSUGGESTS_libnss-db = "make"
 FILES_glibc-extra-nss = "${base_libdir}/libnss_*-*.so ${base_libdir}/libnss_*.so.*"
 FILES_sln = "${base_sbindir}/sln"
 FILES_${PN}-pic = "${libdir}/*_pic.a ${libdir}/*_pic.map ${libdir}/libc_pic/*.o"
@@ -59,7 +62,6 @@ inherit libc-common multilib_header
 
 do_install_append () {
 	rm -f ${D}${sysconfdir}/localtime
-	rm -rf ${D}${localstatedir}
 
 	# remove empty glibc dir
 	if [ -d ${D}${libexecdir} ]; then
-- 
2.11.0

Re: [PATCH 1/1] glibc: re-package for libnss-db

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 10433 bytes --]



On 8/16/18 3:37 AM, Chen Qi wrote:
> On other distros like ubuntu/centos, libnss-db usually provides:
> - The libraries
> - The Makefile to create database
>   (in /var/db for centos, /var/lib/misc/ for ubuntu)
> - The makedb command (it's in glibc-common for centos7)
> 
> What we had is:
> - The libraries are in glibc-extra-nss
> - The Makefile is removed
> - The makedb command is in glibc-utils (lack of dependency)
> 
> So when glibc-extra-nss is installed but glibc-utils is not,
> we see error like:
> nscd[165]: 165 checking for monitored file `/var/db/group.db': No such file or directory
> nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No such file or directory
> 
> And there is not an easy way to create these databases.
> 
> To fix the issue:
> - Re-package the libraries into libnss-db
> - Don't remove the Makefile and add it in libnss-db
> - Add RDEPENDS for libnss-db on glibc-utils
> - Provide a shell script, makedbs.sh, to generate the db files.
>   This is to avoid dependency on 'make'.
> 
> Notes:
> 1. For external toolchain, an extra package 'libnss-db' need to be provided
>    If replacing glibc from core.
> 2. I've check the git history of nss/db-Makefile, the last two functionality
>    fix is as below.
>    - fix non-portable `echo -n` usage   --  Date:   Thu Aug 6 04:14:20 2015 -0400
>    - Fix db makefile rule for group.db  --  Date:   Fri Nov 11 14:43:36 2011 +0100
>    So I think this file is stable enough. And using makedbs.sh which is crafted according
>    to that file is not likely to cause maintanence problem.


using a shell script instead of Makefile is worthwhile change to submit
upstream. Please go ahead and submit it to glibc mailing lists.

Overall the changes look ok to me.
> 
> Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/recipes-core/glibc/glibc-package.inc |   6 +-
>  meta/recipes-core/glibc/glibc/makedbs.sh  | 177 ++++++++++++++++++++++++++++++
>  meta/recipes-core/glibc/glibc_2.28.bb     |   1 +
>  3 files changed, 182 insertions(+), 2 deletions(-)
>  create mode 100755 meta/recipes-core/glibc/glibc/makedbs.sh
> 
> diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
> index 07ce75d..9ea41b7 100644
> --- a/meta/recipes-core/glibc/glibc-package.inc
> +++ b/meta/recipes-core/glibc/glibc-package.inc
> @@ -1,6 +1,6 @@
>  INHIBIT_SYSROOT_STRIP = "1"
>  
> -PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
> +PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libnss-db libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
>  
>  # The ld.so in this glibc supports the GNU_HASH
>  RPROVIDES_${PN} = "eglibc rtld(GNU_HASH)"
> @@ -23,6 +23,8 @@ FILES_ldd = "${bindir}/ldd"
>  FILES_libsegfault = "${base_libdir}/libSegFault*"
>  FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
>  FILES_libmemusage = "${base_libdir}/libmemusage.so"
> +FILES_libnss-db = "${base_libdir}/libnss_db.so.* ${base_libdir}/libnss_db-*.so ${localstatedir}/db/Makefile ${localstatedir}/db/makedbs.sh"
> +RDEPENDS_libnss-db = "${PN}-utils"
>  FILES_glibc-extra-nss = "${base_libdir}/libnss_*-*.so ${base_libdir}/libnss_*.so.*"
>  FILES_sln = "${base_sbindir}/sln"
>  FILES_${PN}-pic = "${libdir}/*_pic.a ${libdir}/*_pic.map ${libdir}/libc_pic/*.o"
> @@ -59,7 +61,6 @@ inherit libc-common multilib_header
>  
>  do_install_append () {
>  	rm -f ${D}${sysconfdir}/localtime
> -	rm -rf ${D}${localstatedir}
>  
>  	# remove empty glibc dir
>  	if [ -d ${D}${libexecdir} ]; then
> @@ -95,6 +96,7 @@ do_install_append () {
>  	install -d ${D}${localstatedir}/db/nscd
>  	install -m 0755 ${S}/nscd/nscd.init ${D}${sysconfdir}/init.d/nscd
>  	install -m 0755 ${S}/nscd/nscd.conf ${D}${sysconfdir}/nscd.conf
> +	install -m 0755 ${WORKDIR}/makedbs.sh ${D}${localstatedir}/db
>  	sed -i "s%daemon%start-stop-daemon --start --exec%g" ${D}${sysconfdir}/init.d/nscd
>  	sed -i "s|\(enable-cache\t\+netgroup\t\+\)yes|\1no|" ${D}${sysconfdir}/nscd.conf
>  
> diff --git a/meta/recipes-core/glibc/glibc/makedbs.sh b/meta/recipes-core/glibc/glibc/makedbs.sh
> new file mode 100755
> index 0000000..7d51a67
> --- /dev/null
> +++ b/meta/recipes-core/glibc/glibc/makedbs.sh
> @@ -0,0 +1,177 @@
> +#!/bin/sh
> +
> +#
> +# Make passwd.db, group.db, etc.
> +#
> +
> +VAR_DB=/var/db
> +
> +# Use make if available
> +if [ -x /usr/bin/make -o -x /bin/make ]; then
> +	make -C $VAR_DB
> +	exit 0
> +fi
> +
> +# No make available, do it in hard way
> +
> +# passwd.db
> +if [ -e /etc/passwd ]; then
> +target=$VAR_DB/passwd.db
> +echo -n "passwd... "
> +awk 'BEGIN { FS=":"; OFS=":" } \
> + /^[ \t]*$$/ { next } \
> + /^[ \t]*#/ { next } \
> + /^[^#]/ { printf ".%s ", $$1; print; \
> +	   printf "=%s ", $$3; print }' /etc/passwd | \
> +makedb --quiet -o $target -
> +echo "done."
> +fi
> +
> +# group.db
> +if [ -e /etc/group ]; then
> +target=$VAR_DB/group.db
> +echo -n "group... "
> +awk 'BEGIN { FS=":"; OFS=":" } \
> + /^[ \t]*$$/ { next } \
> + /^[ \t]*#/ { next } \
> + /^[^#]/ { printf ".%s ", $$1; print; \
> +	   printf "=%s ", $$3; print; \
> +	   if ($$4 != "") { \
> +	     split($$4, grmems, ","); \
> +	     for (memidx in grmems) { \
> +	       mem=grmems[memidx]; \
> +	       if (members[mem] == "") \
> +		 members[mem]=$$3; \
> +	       else \
> +		 members[mem]=members[mem] "," $$3; \
> +	     } \
> +	     delete grmems; } } \
> + END { for (mem in members) \
> +	 printf ":%s %s %s\n", mem, mem, members[mem]; }' /etc/group | \
> +makedb --quiet -o $target -
> +echo "done."
> +fi
> +
> +# ethers.db
> +if [ -e /etc/ethers ]; then
> +target=$VAR_DB/ethers.db
> +echo -n "ethers... "
> +awk '/^[ \t]*$$/ { next } \
> + /^[ \t]*#/ { next } \
> + /^[^#]/ { printf ".%s ", $$1; print; \
> +	   printf "=%s ", $$2; print }' /etc/ethers | \
> +makedb --quiet -o $target -
> +echo "done."
> +fi
> +
> +# protocols.db
> +if [ -e /etc/protocols ]; then
> +target=$VAR_DB/protocols.db
> +echo -n "protocols... "
> +awk '/^[ \t]*$$/ { next } \
> + /^[ \t]*#/ { next } \
> + /^[^#]/ { printf ".%s ", $$1; print; \
> +	   printf "=%s ", $$2; print; \
> +	   for (i = 3; i <= NF && !($$i ~ /^#/); ++i) \
> +	     { printf ".%s ", $$i; print } }' /etc/protocols | \
> +makedb --quiet -o $target -
> +echo "done."
> +fi
> +
> +# rpc.db
> +if [ -e /etc/rpc ]; then
> +target=$VAR_DB/rpc.db
> +echo -n "rpc... "
> +awk '/^[ \t]*$$/ { next } \
> + /^[ \t]*#/ { next } \
> + /^[^#]/ { printf ".%s ", $$1; print; \
> +	   printf "=%s ", $$2; print; \
> +	   for (i = 3; i <= NF && !($$i ~ /^#/); ++i) \
> +	     { printf ".%s ", $$i; print } }' /etc/rpc | \
> +makedb --quiet -o $target -
> +echo "done."
> +fi
> +
> +# services.db
> +if [ -e /etc/services ]; then
> +target=$VAR_DB/services.db
> +echo -n "services... "
> +awk 'BEGIN { FS="[ \t/]+" } \
> + /^[ \t]*$$/ { next } \
> + /^[ \t]*#/ { next } \
> + /^[^#]/ { sub(/[ \t]*#.*$$/, "");\
> +	   printf ":%s/%s ", $$1, $$3; print; \
> +	   printf ":%s/ ", $$1; print; \
> +	   printf "=%s/%s ", $$2, $$3; print; \
> +	   printf "=%s/ ", $$2; print; \
> +	   for (i = 4; i <= NF && !($$i ~ /^#/); ++i) \
> +	     { printf ":%s/%s ", $$i, $$3; print; \
> +	       printf ":%s/ ", $$i; print } }' /etc/services | \
> +makedb --quiet -o $target -
> +echo "done."
> +fi
> +
> +# shadow.db
> +if [ -e /etc/shadow ]; then
> +target=$VAR_DB/shadow.db
> +echo -n "shadow... "
> +awk 'BEGIN { FS=":"; OFS=":" } \
> + /^[ \t]*$$/ { next } \
> + /^[ \t]*#/ { next } \
> + /^[^#]/ { printf ".%s ", $$1; print }' /etc/shadow | \
> +(umask 077 && makedb --quiet -o $target -)
> +echo "done."
> +if chgrp shadow $target 2>/dev/null; then
> +	chmod g+r $target
> +else
> +	chown 0 $target; chgrp 0 $target; chmod 600 $target;
> +	echo
> +	echo "Warning: The shadow password database $target"
> +	echo "has been set to be readable only by root.  You may want"
> +	echo "to make it readable by the \`shadow' group depending"
> +	echo "on your configuration."
> +	echo
> +fi
> +fi
> +
> +# gshadow.db
> +if [ -e /etc/gshadow ]; then
> +target=$VAR_DB/gshadow.db
> +echo -n "gshadow... "
> +awk 'BEGIN { FS=":"; OFS=":" } \
> + /^[ \t]*$$/ { next } \
> + /^[ \t]*#/ { next } \
> + /^[^#]/ { printf ".%s ", $$1; print }' /etc/gshadow | \
> +(umask 077 && makedb --quiet -o $target -)
> +echo "done."
> +if chgrp shadow $target 2>/dev/null; then
> +	chmod g+r $target
> +else
> +	chown 0 $target; chgrp 0 $target; chmod 600 $target
> +	echo
> +	echo "Warning: The shadow group database $target"
> +	echo "has been set to be readable only by root.  You may want"
> +	echo "to make it readable by the \`shadow' group depending"
> +	echo "on your configuration."
> +	echo
> +fi
> +fi
> +
> +# netgroup.db
> +if [ -e /etc/netgroup ]; then
> +target=$VAR_DB/netgroup.db
> +echo -n "netgroup... "
> +awk 'BEGIN { ini=1 } \
> + /^[ \t]*$$/ { next } \
> + /^[ \t]*#/ { next } \
> + /^[^#]/ { if (sub(/[ \t]*\\$$/, " ") == 0) end="\n"; \
> +	   else end=""; \
> +	   gsub(/[ \t]+/, " "); \
> +	   sub(/^[ \t]*/, ""); \
> +	   if (ini == 0) printf "%s%s", $$0, end; \
> +	   else printf ".%s %s%s", $$1, $$0, end; \
> +	   ini=end == "" ? 0 : 1; } \
> + END { if (ini==0) printf "\n" }' /etc/netgroup | \
> +makedb --quiet -o $target
> +echo "done."
> +fi
> diff --git a/meta/recipes-core/glibc/glibc_2.28.bb b/meta/recipes-core/glibc/glibc_2.28.bb
> index 95e333d..0ebbaf9 100644
> --- a/meta/recipes-core/glibc/glibc_2.28.bb
> +++ b/meta/recipes-core/glibc/glibc_2.28.bb
> @@ -17,6 +17,7 @@ UPSTREAM_CHECK_GITTAGREGEX = "(?P<pver>\d+\.\d+(\.(?!90)\d+)*)"
>  SRC_URI = "${GLIBC_GIT_URI};branch=${SRCBRANCH};name=glibc \
>             file://etc/ld.so.conf \
>             file://generate-supported.mk \
> +           file://makedbs.sh \
>             \
>             ${NATIVESDKFIXES} \
>             file://0006-fsl-e500-e5500-e6500-603e-fsqrt-implementation.patch \
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

[PATCH 1/1] glibc: re-package for libnss-db

[Chen Qi]

On other distros like ubuntu/centos, libnss-db usually provides:
- The libraries
- The Makefile to create database
  (in /var/db for centos, /var/lib/misc/ for ubuntu)
- The makedb command (it's in glibc-common for centos7)

What we had is:
- The libraries are in glibc-extra-nss
- The Makefile is removed
- The makedb command is in glibc-utils (lack of dependency)

So when glibc-extra-nss is installed but glibc-utils is not,
we see error like:
nscd[165]: 165 checking for monitored file `/var/db/group.db': No such file or directory
nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No such file or directory

And there is not an easy way to create these databases.

To fix the issue:
- Re-package the libraries into libnss-db
- Don't remove the Makefile and add it in libnss-db
- Add RDEPENDS for libnss-db on glibc-utils
- Provide a shell script, makedbs.sh, to generate the db files.
  This is to avoid dependency on 'make'.

Notes:
1. For external toolchain, an extra package 'libnss-db' need to be provided
   If replacing glibc from core.
2. I've check the git history of nss/db-Makefile, the last two functionality
   fix is as below.
   - fix non-portable `echo -n` usage   --  Date:   Thu Aug 6 04:14:20 2015 -0400
   - Fix db makefile rule for group.db  --  Date:   Fri Nov 11 14:43:36 2011 +0100
   So I think this file is stable enough. And using makedbs.sh which is crafted according
   to that file is not likely to cause maintanence problem.

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/glibc/glibc-package.inc |   6 +-
 meta/recipes-core/glibc/glibc/makedbs.sh  | 177 ++++++++++++++++++++++++++++++
 meta/recipes-core/glibc/glibc_2.28.bb     |   1 +
 3 files changed, 182 insertions(+), 2 deletions(-)
 create mode 100755 meta/recipes-core/glibc/glibc/makedbs.sh

diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index 07ce75d..9ea41b7 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -1,6 +1,6 @@
 INHIBIT_SYSROOT_STRIP = "1"
 
-PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
+PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libnss-db libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
 
 # The ld.so in this glibc supports the GNU_HASH
 RPROVIDES_${PN} = "eglibc rtld(GNU_HASH)"
@@ -23,6 +23,8 @@ FILES_ldd = "${bindir}/ldd"
 FILES_libsegfault = "${base_libdir}/libSegFault*"
 FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
 FILES_libmemusage = "${base_libdir}/libmemusage.so"
+FILES_libnss-db = "${base_libdir}/libnss_db.so.* ${base_libdir}/libnss_db-*.so ${localstatedir}/db/Makefile ${localstatedir}/db/makedbs.sh"
+RDEPENDS_libnss-db = "${PN}-utils"
 FILES_glibc-extra-nss = "${base_libdir}/libnss_*-*.so ${base_libdir}/libnss_*.so.*"
 FILES_sln = "${base_sbindir}/sln"
 FILES_${PN}-pic = "${libdir}/*_pic.a ${libdir}/*_pic.map ${libdir}/libc_pic/*.o"
@@ -59,7 +61,6 @@ inherit libc-common multilib_header
 
 do_install_append () {
 	rm -f ${D}${sysconfdir}/localtime
-	rm -rf ${D}${localstatedir}
 
 	# remove empty glibc dir
 	if [ -d ${D}${libexecdir} ]; then
@@ -95,6 +96,7 @@ do_install_append () {
 	install -d ${D}${localstatedir}/db/nscd
 	install -m 0755 ${S}/nscd/nscd.init ${D}${sysconfdir}/init.d/nscd
 	install -m 0755 ${S}/nscd/nscd.conf ${D}${sysconfdir}/nscd.conf
+	install -m 0755 ${WORKDIR}/makedbs.sh ${D}${localstatedir}/db
 	sed -i "s%daemon%start-stop-daemon --start --exec%g" ${D}${sysconfdir}/init.d/nscd
 	sed -i "s|\(enable-cache\t\+netgroup\t\+\)yes|\1no|" ${D}${sysconfdir}/nscd.conf
 
diff --git a/meta/recipes-core/glibc/glibc/makedbs.sh b/meta/recipes-core/glibc/glibc/makedbs.sh
new file mode 100755
index 0000000..7d51a67
--- /dev/null
+++ b/meta/recipes-core/glibc/glibc/makedbs.sh
@@ -0,0 +1,177 @@
+#!/bin/sh
+
+#
+# Make passwd.db, group.db, etc.
+#
+
+VAR_DB=/var/db
+
+# Use make if available
+if [ -x /usr/bin/make -o -x /bin/make ]; then
+	make -C $VAR_DB
+	exit 0
+fi
+
+# No make available, do it in hard way
+
+# passwd.db
+if [ -e /etc/passwd ]; then
+target=$VAR_DB/passwd.db
+echo -n "passwd... "
+awk 'BEGIN { FS=":"; OFS=":" } \
+ /^[ \t]*$$/ { next } \
+ /^[ \t]*#/ { next } \
+ /^[^#]/ { printf ".%s ", $$1; print; \
+	   printf "=%s ", $$3; print }' /etc/passwd | \
+makedb --quiet -o $target -
+echo "done."
+fi
+
+# group.db
+if [ -e /etc/group ]; then
+target=$VAR_DB/group.db
+echo -n "group... "
+awk 'BEGIN { FS=":"; OFS=":" } \
+ /^[ \t]*$$/ { next } \
+ /^[ \t]*#/ { next } \
+ /^[^#]/ { printf ".%s ", $$1; print; \
+	   printf "=%s ", $$3; print; \
+	   if ($$4 != "") { \
+	     split($$4, grmems, ","); \
+	     for (memidx in grmems) { \
+	       mem=grmems[memidx]; \
+	       if (members[mem] == "") \
+		 members[mem]=$$3; \
+	       else \
+		 members[mem]=members[mem] "," $$3; \
+	     } \
+	     delete grmems; } } \
+ END { for (mem in members) \
+	 printf ":%s %s %s\n", mem, mem, members[mem]; }' /etc/group | \
+makedb --quiet -o $target -
+echo "done."
+fi
+
+# ethers.db
+if [ -e /etc/ethers ]; then
+target=$VAR_DB/ethers.db
+echo -n "ethers... "
+awk '/^[ \t]*$$/ { next } \
+ /^[ \t]*#/ { next } \
+ /^[^#]/ { printf ".%s ", $$1; print; \
+	   printf "=%s ", $$2; print }' /etc/ethers | \
+makedb --quiet -o $target -
+echo "done."
+fi
+
+# protocols.db
+if [ -e /etc/protocols ]; then
+target=$VAR_DB/protocols.db
+echo -n "protocols... "
+awk '/^[ \t]*$$/ { next } \
+ /^[ \t]*#/ { next } \
+ /^[^#]/ { printf ".%s ", $$1; print; \
+	   printf "=%s ", $$2; print; \
+	   for (i = 3; i <= NF && !($$i ~ /^#/); ++i) \
+	     { printf ".%s ", $$i; print } }' /etc/protocols | \
+makedb --quiet -o $target -
+echo "done."
+fi
+
+# rpc.db
+if [ -e /etc/rpc ]; then
+target=$VAR_DB/rpc.db
+echo -n "rpc... "
+awk '/^[ \t]*$$/ { next } \
+ /^[ \t]*#/ { next } \
+ /^[^#]/ { printf ".%s ", $$1; print; \
+	   printf "=%s ", $$2; print; \
+	   for (i = 3; i <= NF && !($$i ~ /^#/); ++i) \
+	     { printf ".%s ", $$i; print } }' /etc/rpc | \
+makedb --quiet -o $target -
+echo "done."
+fi
+
+# services.db
+if [ -e /etc/services ]; then
+target=$VAR_DB/services.db
+echo -n "services... "
+awk 'BEGIN { FS="[ \t/]+" } \
+ /^[ \t]*$$/ { next } \
+ /^[ \t]*#/ { next } \
+ /^[^#]/ { sub(/[ \t]*#.*$$/, "");\
+	   printf ":%s/%s ", $$1, $$3; print; \
+	   printf ":%s/ ", $$1; print; \
+	   printf "=%s/%s ", $$2, $$3; print; \
+	   printf "=%s/ ", $$2; print; \
+	   for (i = 4; i <= NF && !($$i ~ /^#/); ++i) \
+	     { printf ":%s/%s ", $$i, $$3; print; \
+	       printf ":%s/ ", $$i; print } }' /etc/services | \
+makedb --quiet -o $target -
+echo "done."
+fi
+
+# shadow.db
+if [ -e /etc/shadow ]; then
+target=$VAR_DB/shadow.db
+echo -n "shadow... "
+awk 'BEGIN { FS=":"; OFS=":" } \
+ /^[ \t]*$$/ { next } \
+ /^[ \t]*#/ { next } \
+ /^[^#]/ { printf ".%s ", $$1; print }' /etc/shadow | \
+(umask 077 && makedb --quiet -o $target -)
+echo "done."
+if chgrp shadow $target 2>/dev/null; then
+	chmod g+r $target
+else
+	chown 0 $target; chgrp 0 $target; chmod 600 $target;
+	echo
+	echo "Warning: The shadow password database $target"
+	echo "has been set to be readable only by root.  You may want"
+	echo "to make it readable by the \`shadow' group depending"
+	echo "on your configuration."
+	echo
+fi
+fi
+
+# gshadow.db
+if [ -e /etc/gshadow ]; then
+target=$VAR_DB/gshadow.db
+echo -n "gshadow... "
+awk 'BEGIN { FS=":"; OFS=":" } \
+ /^[ \t]*$$/ { next } \
+ /^[ \t]*#/ { next } \
+ /^[^#]/ { printf ".%s ", $$1; print }' /etc/gshadow | \
+(umask 077 && makedb --quiet -o $target -)
+echo "done."
+if chgrp shadow $target 2>/dev/null; then
+	chmod g+r $target
+else
+	chown 0 $target; chgrp 0 $target; chmod 600 $target
+	echo
+	echo "Warning: The shadow group database $target"
+	echo "has been set to be readable only by root.  You may want"
+	echo "to make it readable by the \`shadow' group depending"
+	echo "on your configuration."
+	echo
+fi
+fi
+
+# netgroup.db
+if [ -e /etc/netgroup ]; then
+target=$VAR_DB/netgroup.db
+echo -n "netgroup... "
+awk 'BEGIN { ini=1 } \
+ /^[ \t]*$$/ { next } \
+ /^[ \t]*#/ { next } \
+ /^[^#]/ { if (sub(/[ \t]*\\$$/, " ") == 0) end="\n"; \
+	   else end=""; \
+	   gsub(/[ \t]+/, " "); \
+	   sub(/^[ \t]*/, ""); \
+	   if (ini == 0) printf "%s%s", $$0, end; \
+	   else printf ".%s %s%s", $$1, $$0, end; \
+	   ini=end == "" ? 0 : 1; } \
+ END { if (ini==0) printf "\n" }' /etc/netgroup | \
+makedb --quiet -o $target
+echo "done."
+fi
diff --git a/meta/recipes-core/glibc/glibc_2.28.bb b/meta/recipes-core/glibc/glibc_2.28.bb
index 95e333d..0ebbaf9 100644
--- a/meta/recipes-core/glibc/glibc_2.28.bb
+++ b/meta/recipes-core/glibc/glibc_2.28.bb
@@ -17,6 +17,7 @@ UPSTREAM_CHECK_GITTAGREGEX = "(?P<pver>\d+\.\d+(\.(?!90)\d+)*)"
 SRC_URI = "${GLIBC_GIT_URI};branch=${SRCBRANCH};name=glibc \
            file://etc/ld.so.conf \
            file://generate-supported.mk \
+           file://makedbs.sh \
            \
            ${NATIVESDKFIXES} \
            file://0006-fsl-e500-e5500-e6500-603e-fsqrt-implementation.patch \
-- 
1.9.1

Re: [PATCH 1/1] glibc: re-package for libnss-db

[ChenQi]

On 08/16/2018 04:41 AM, Randy MacLeod wrote:
> On 08/15/2018 10:02 AM, Richard Purdie wrote:
>> On Wed, 2018-08-15 at 15:00 +0800, Chen Qi wrote:
>>> From: Jackie Huang <jackie.huang@windriver.com>
>>>
>>> On other distros like ubuntu/centos, libnss-db usually provides:
>>> - The libraries
>>> - The Makefile to create database
>>>    (in /var/db for centos, /var/lib/misc/ for ubuntu)
>>> - The makedb command (it's in glibc-common for centos7)
>>>
>>> What we had is:
>>> - The libraries are in glibc-extra-nss
>>> - The Makefile is removed
>>> - The makedb command is in glibc-utils (lack of dependency)
>>>
>>> So when glibc-extra-nss is installed but glibc-utils is not,
>>> we see error like:
>>> nscd[165]: 165 checking for monitored file `/var/db/group.db': No
>>> such file or directory
>>> nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No
>>> such file or directory
>>>
>>> And there is not an easy way to create these databases.
>>>
>>> To fix the issue:
>>> - Re-package the libraries into libnss-db
>>> - Don't remove the Makefile and add it in libnss-db
>>> - Add RDEPENDS for libnss-db on glibc-utils and make
>>>
>>> Notes:
>>> For external toolchain, an extra package 'libnss-db' need to be
>>> provided
>>> If replacing glibc from core.
>>>
>>> Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
>>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>>> ---
>>>   meta/recipes-core/glibc/glibc-package.inc | 5 +++--
>>>   1 file changed, 3 insertions(+), 2 deletions(-)
>>
>> This breaks on the autobuilder in oe-selftest:
>>
>> 2018-08-15 13:09:15,551 - oe-selftest - INFO - 
>> ======================================================================
>
>
> ...
>
>> NOTE: Resolving any missing task queue dependencies
>> ERROR: Nothing RPROVIDES 'make' (but 
>> /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/recipes-core/glibc/glibc_2.28.bb 
>> RDEPENDS on or otherwise requires it)
>> make was skipped: it has an incompatible license: GPLv3 & LGPLv2
>> NOTE: Runtime target 'make' is unbuildable, removing...
>> Missing or unbuildable dependency chain was: ['make']
>> ERROR: Required build target 'selftest-ed' has no buildable providers.
>> Missing or unbuildable dependency chain was: ['selftest-ed', 
>> 'virtual/libc', 'make']
>>
>> Summary: There were 2 ERROR messages shown, returning a non-zero exit 
>> code.
>>
>> https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/1281/steps/Running%20oe-selftest/logs/stdio 
>>
>>
>> We really don't want a make dependency in glibc :(
>
> Switching from RDEPENDS to RSUGGESTS  seems like a
> reasonable solution. The GPLv2 build would work but
> libnss-db would fail unless meta-gplv2 were used to
> get an old version of make.
>
> One way to avoid needing make is to demand that the bdb
> files are generated during the build. That would be nice
> but it shouldn't be a requirement since it's not the
> typical workflow and since the db will usually be dynamic.
>
> History and more details below.
>
> ../Randy
>
> To support on target generation of the bdb files that
> libnss-db uses then we need to:
>  - separate the libnss-db files into a libnss-db pkg
>  - *suggest* a 'make' as described below:
>
> Back in January, Jackie had replied to:
>    [OE-core] [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db
>
> with:
>
>> Sorry, I didn't noticed that make is gplv3. I change the dependency to
>> RSUGGESTS and the test passed:
>>
>> $ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3
>> ...
>> 2018-01-16 21:46:22,540 - oe-selftest - INFO - oe-selftest - OK - All 
>> required tests passed
>
> but of course that papers over the failure by allowing the build to
> pass even though runtime needs to include make to be able to
> rebuild the db(s). And of course libnss-db isn't even being tested yet
> so once we add that to oeqa, the test will fail for GPLv2-only images
> where meta-gplv2 isn't added to the list of layers. I'm fine with
> such oeqa failing when meta-gplv2 isn't around.
>
>
> Qi, if no one comes up with a better idea,
> please test that as well as:
>   $ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3
> so that Richard doesn't burn another build cycle on it.
>
> This isn't urgent but it would be nice to resolve the
> issue properly for M3 without driving Richard around the bend!
>
> Thanks,
> ../Randy
>
>
>>
>> Cheers,
>>
>> Richard
>>
>
>
Sorry for breaking the autobuilder.

I checked the Makefile and found that it's a bunch of `awd [expression] 
/etc/xxx | makedb -o /var/db/xxx.db -'.
I also checked the glibc repo and this Makefile is quite stable. So I 
think we can use a shell script to get rid of the make dependency. And 
it should not create maintenance problem.

I'll send out V2.
I've tested it locally and it should pass the oe-selftest.

Best Regards,
Chen Qi

Re: [PATCH 1/1] glibc: re-package for libnss-db

[Randy MacLeod]

On 08/15/2018 10:02 AM, Richard Purdie wrote:
> On Wed, 2018-08-15 at 15:00 +0800, Chen Qi wrote:
>> From: Jackie Huang <jackie.huang@windriver.com>
>>
>> On other distros like ubuntu/centos, libnss-db usually provides:
>> - The libraries
>> - The Makefile to create database
>>    (in /var/db for centos, /var/lib/misc/ for ubuntu)
>> - The makedb command (it's in glibc-common for centos7)
>>
>> What we had is:
>> - The libraries are in glibc-extra-nss
>> - The Makefile is removed
>> - The makedb command is in glibc-utils (lack of dependency)
>>
>> So when glibc-extra-nss is installed but glibc-utils is not,
>> we see error like:
>> nscd[165]: 165 checking for monitored file `/var/db/group.db': No
>> such file or directory
>> nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No
>> such file or directory
>>
>> And there is not an easy way to create these databases.
>>
>> To fix the issue:
>> - Re-package the libraries into libnss-db
>> - Don't remove the Makefile and add it in libnss-db
>> - Add RDEPENDS for libnss-db on glibc-utils and make
>>
>> Notes:
>> For external toolchain, an extra package 'libnss-db' need to be
>> provided
>> If replacing glibc from core.
>>
>> Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>> ---
>>   meta/recipes-core/glibc/glibc-package.inc | 5 +++--
>>   1 file changed, 3 insertions(+), 2 deletions(-)
> 
> This breaks on the autobuilder in oe-selftest:
> 
> 2018-08-15 13:09:15,551 - oe-selftest - INFO - ======================================================================


...

> NOTE: Resolving any missing task queue dependencies
> ERROR: Nothing RPROVIDES 'make' (but /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/recipes-core/glibc/glibc_2.28.bb RDEPENDS on or otherwise requires it)
> make was skipped: it has an incompatible license: GPLv3 & LGPLv2
> NOTE: Runtime target 'make' is unbuildable, removing...
> Missing or unbuildable dependency chain was: ['make']
> ERROR: Required build target 'selftest-ed' has no buildable providers.
> Missing or unbuildable dependency chain was: ['selftest-ed', 'virtual/libc', 'make']
> 
> Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
> 
> https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/1281/steps/Running%20oe-selftest/logs/stdio
> 
> We really don't want a make dependency in glibc :(

Switching from RDEPENDS to RSUGGESTS  seems like a
reasonable solution. The GPLv2 build would work but
libnss-db would fail unless meta-gplv2 were used to
get an old version of make.

One way to avoid needing make is to demand that the bdb
files are generated during the build. That would be nice
but it shouldn't be a requirement since it's not the
typical workflow and since the db will usually be dynamic.

History and more details below.

../Randy

To support on target generation of the bdb files that
libnss-db uses then we need to:
  - separate the libnss-db files into a libnss-db pkg
  - *suggest* a 'make' as described below:

Back in January, Jackie had replied to:
    [OE-core] [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db

with:

> Sorry, I didn't noticed that make is gplv3. I change the dependency to
> RSUGGESTS and the test passed:
> 
> $ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3
> ...
> 2018-01-16 21:46:22,540 - oe-selftest - INFO - oe-selftest - OK - All required tests passed

but of course that papers over the failure by allowing the build to
pass even though runtime needs to include make to be able to
rebuild the db(s). And of course libnss-db isn't even being tested yet
so once we add that to oeqa, the test will fail for GPLv2-only images
where meta-gplv2 isn't added to the list of layers. I'm fine with
such oeqa failing when meta-gplv2 isn't around.


Qi, if no one comes up with a better idea,
please test that as well as:
   $ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3
so that Richard doesn't burn another build cycle on it.

This isn't urgent but it would be nice to resolve the
issue properly for M3 without driving Richard around the bend!

Thanks,
../Randy


> 
> Cheers,
> 
> Richard
> 


-- 
# Randy MacLeod
# Wind River Linux

Re: [PATCH 1/1] glibc: re-package for libnss-db

[Richard Purdie]

On Wed, 2018-08-15 at 15:00 +0800, Chen Qi wrote:
> From: Jackie Huang <jackie.huang@windriver.com>
> 
> On other distros like ubuntu/centos, libnss-db usually provides:
> - The libraries
> - The Makefile to create database
>   (in /var/db for centos, /var/lib/misc/ for ubuntu)
> - The makedb command (it's in glibc-common for centos7)
> 
> What we had is:
> - The libraries are in glibc-extra-nss
> - The Makefile is removed
> - The makedb command is in glibc-utils (lack of dependency)
> 
> So when glibc-extra-nss is installed but glibc-utils is not,
> we see error like:
> nscd[165]: 165 checking for monitored file `/var/db/group.db': No
> such file or directory
> nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No
> such file or directory
> 
> And there is not an easy way to create these databases.
> 
> To fix the issue:
> - Re-package the libraries into libnss-db
> - Don't remove the Makefile and add it in libnss-db
> - Add RDEPENDS for libnss-db on glibc-utils and make
> 
> Notes:
> For external toolchain, an extra package 'libnss-db' need to be
> provided
> If replacing glibc from core.
> 
> Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/recipes-core/glibc/glibc-package.inc | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)

This breaks on the autobuilder in oe-selftest:

2018-08-15 13:09:15,551 - oe-selftest - INFO - ======================================================================
2018-08-15 13:09:15,551 - oe-selftest - INFO - FAIL: bbtests.BitbakeTests.test_non_gplv3 (subunit.RemotedTestCase)
2018-08-15 13:09:15,552 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-08-15 13:09:15,552 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/bbtests.py", line 246, in test_non_gplv3
    self.assertEqual(result.status, 0, "Bitbake failed, exit code %s, output %s" % (result.status, result.output))
  File "/usr/lib64/python3.6/unittest/case.py", line 829, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib64/python3.6/unittest/case.py", line 822, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 1 != 0 : Bitbake failed, exit code 1, output Parsing recipes...done.
Parsing of 811 .bb files complete (0 cached, 811 parsed). 1277 targets, 99 skipped, 0 masked, 0 errors.
Removing 10 recipes from the core2-64 sysroot...done.
Removing 10 recipes from the qemux86_64 sysroot...done.
NOTE: Resolving any missing task queue dependencies
ERROR: Nothing RPROVIDES 'make' (but /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/recipes-core/glibc/glibc_2.28.bb RDEPENDS on or otherwise requires it)
make was skipped: it has an incompatible license: GPLv3 & LGPLv2
NOTE: Runtime target 'make' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['make']
ERROR: Required build target 'selftest-ed' has no buildable providers.
Missing or unbuildable dependency chain was: ['selftest-ed', 'virtual/libc', 'make']

Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/1281/steps/Running%20oe-selftest/logs/stdio

We really don't want a make dependency in glibc :(

Cheers,

Richard

[PATCH 1/1] glibc: re-package for libnss-db

[Chen Qi]

From: Jackie Huang <jackie.huang@windriver.com>

On other distros like ubuntu/centos, libnss-db usually provides:
- The libraries
- The Makefile to create database
  (in /var/db for centos, /var/lib/misc/ for ubuntu)
- The makedb command (it's in glibc-common for centos7)

What we had is:
- The libraries are in glibc-extra-nss
- The Makefile is removed
- The makedb command is in glibc-utils (lack of dependency)

So when glibc-extra-nss is installed but glibc-utils is not,
we see error like:
nscd[165]: 165 checking for monitored file `/var/db/group.db': No such file or directory
nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No such file or directory

And there is not an easy way to create these databases.

To fix the issue:
- Re-package the libraries into libnss-db
- Don't remove the Makefile and add it in libnss-db
- Add RDEPENDS for libnss-db on glibc-utils and make

Notes:
For external toolchain, an extra package 'libnss-db' need to be provided
If replacing glibc from core.

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/glibc/glibc-package.inc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index 8ac01ce..305094f 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -1,6 +1,6 @@
 INHIBIT_SYSROOT_STRIP = "1"
 
-PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
+PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libnss-db libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
 
 # The ld.so in this glibc supports the GNU_HASH
 RPROVIDES_${PN} = "eglibc rtld(GNU_HASH)"
@@ -23,6 +23,8 @@ FILES_ldd = "${bindir}/ldd"
 FILES_libsegfault = "${base_libdir}/libSegFault*"
 FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
 FILES_libmemusage = "${base_libdir}/libmemusage.so"
+FILES_libnss-db = "${base_libdir}/libnss_db.so.* ${base_libdir}/libnss_db-*.so ${localstatedir}/db/Makefile"
+RDEPENDS_libnss-db = "${PN}-utils make"
 FILES_glibc-extra-nss = "${base_libdir}/libnss_*-*.so ${base_libdir}/libnss_*.so.*"
 FILES_sln = "${base_sbindir}/sln"
 FILES_${PN}-pic = "${libdir}/*_pic.a ${libdir}/*_pic.map ${libdir}/libc_pic/*.o"
@@ -59,7 +61,6 @@ inherit libc-common multilib_header
 
 do_install_append () {
 	rm -f ${D}${sysconfdir}/localtime
-	rm -rf ${D}${localstatedir}
 
 	# remove empty glibc dir
 	if [ -d ${D}${libexecdir} ]; then
-- 
1.9.1