[PATCH 1/7] glibc: make nscd optional

[PATCH 1/7] glibc: make nscd optional

[Ross Burton]

NSCD itself is already packaged into its own package, but if it is never
going to be needed then it can be disabled and it won't be called from
glibc at all.  Add a PACKAGECONFIG for NSCD that is enabled by default.

As a side effect if NSCD is disable, glibc with and without systemd is
binary identical, which helps sstate reuse.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/recipes-core/glibc/glibc_2.32.bb | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/glibc/glibc_2.32.bb b/meta/recipes-core/glibc/glibc_2.32.bb
index 7049e61625..2a0e464385 100644
--- a/meta/recipes-core/glibc/glibc_2.32.bb
+++ b/meta/recipes-core/glibc/glibc_2.32.bb
@@ -69,13 +69,15 @@ EXTRA_OECONF = "--enable-kernel=${OLDEST_KERNEL} \
                 --enable-stackguard-randomization \
                 --disable-crypt \
                 --with-default-link \
-                --enable-nscd \
                 ${@bb.utils.contains_any('SELECTED_OPTIMIZATION', '-O0 -Og', '--disable-werror', '', d)} \
                 ${GLIBCPIE} \
                 ${GLIBC_EXTRA_OECONF}"
 
 EXTRA_OECONF += "${@get_libc_fpu_setting(bb, d)}"
 
+PACKAGECONFIG ??= "nscd"
+PACKAGECONFIG[nscd] = "--enable-nscd,--disable-nscd"
+
 do_patch_append() {
     bb.build.exec_func('do_fix_readlib_c', d)
 }
-- 
2.25.1

[PATCH 2/7] utils: respect scheduler affinity in cpu_count()

[Ross Burton]

cpu_count() returns multiprocessing.cpu_count() but that is simply returns
os.cpu_count() so we could use that directly.

However this returns the number of CPUs on the host, not the number of
usable CPUs on the host. If the user is using scheduler affinity then
the number of usable CPUs may be less, so when determining how many cores
we can use check the affinity instead.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/lib/oe/utils.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index 13f4271da0..468c76f30f 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -249,8 +249,7 @@ def trim_version(version, num_parts=2):
     return trimmed
 
 def cpu_count(at_least=1):
-    import multiprocessing
-    cpus = multiprocessing.cpu_count()
+    cpus = len(os.sched_getaffinity(0))
     return max(cpus, at_least)
 
 def execute_pre_post_process(d, cmds):
-- 
2.25.1

[PATCH 3/7] rpm: disable libarchive use

[Ross Burton]

libarchive is only needed for the rpm2archive tool, as this is of
limited use and libarchive is a non-trivial build dependency make this
optional and disabled by default.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/recipes-devtools/rpm/rpm_4.15.1.bb | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/rpm/rpm_4.15.1.bb b/meta/recipes-devtools/rpm/rpm_4.15.1.bb
index c9258632d2..71e2c033fd 100644
--- a/meta/recipes-devtools/rpm/rpm_4.15.1.bb
+++ b/meta/recipes-devtools/rpm/rpm_4.15.1.bb
@@ -49,7 +49,7 @@ SRCREV = "ab2179452c5be276a6b96c591afded485c7e58c3"
 
 S = "${WORKDIR}/git"
 
-DEPENDS = "openssl libarchive db file popt xz bzip2 dbus elfutils python3"
+DEPENDS = "openssl db file popt xz bzip2 dbus elfutils python3"
 DEPENDS_append_class-native = " file-replacement-native bzip2-replacement-native"
 
 inherit autotools gettext pkgconfig python3native
@@ -74,6 +74,7 @@ BBCLASSEXTEND = "native nativesdk"
 
 PACKAGECONFIG ??= ""
 PACKAGECONFIG[imaevm] = "--with-imaevm,,ima-evm-utils"
+PACKAGECONFIG[rpm2archive] = "--with-archive,--without-archive,libarchive"
 
 ASNEEDED = ""
 
@@ -95,7 +96,7 @@ WRAPPER_TOOLS = " \
 
 do_install_append_class-native() {
         for tool in ${WRAPPER_TOOLS}; do
-                create_wrapper ${D}$tool \
+                test -x ${D}$tool && create_wrapper ${D}$tool \
                         RPM_CONFIGDIR=${STAGING_LIBDIR_NATIVE}/rpm \
                         RPM_ETCCONFIGDIR=${STAGING_DIR_NATIVE} \
                         MAGIC=${STAGING_DIR_NATIVE}${datadir_native}/misc/magic.mgc \
@@ -105,7 +106,7 @@ do_install_append_class-native() {
 
 do_install_append_class-nativesdk() {
         for tool in ${WRAPPER_TOOLS}; do
-                create_wrapper ${D}$tool \
+                test -x ${D}$tool && create_wrapper ${D}$tool \
                         RPM_CONFIGDIR='`dirname $''realpath`'/${@os.path.relpath(d.getVar('libdir'), d.getVar('bindir'))}/rpm \
                         RPM_ETCCONFIGDIR='$'{RPM_ETCCONFIGDIR-'`dirname $''realpath`'/${@os.path.relpath(d.getVar('sysconfdir'), d.getVar('bindir'))}/..} \
                         MAGIC='`dirname $''realpath`'/${@os.path.relpath(d.getVar('datadir'), d.getVar('bindir'))}/misc/magic.mgc \
-- 
2.25.1

[PATCH 4/7] rpm: add PACKAGECONFIG for the systemd inhibit plugin

[Ross Burton]

RPM ships a systemd inhibit plugin, that will tell systemd to inhibit a
reboot or sleep during a package upgrades.

For native RPM this is entirely useless, and for target it's only useful
if you're using systemd+logind+rpm.

This plugin uses DBus which means it depends on expat -> cmake-native ->
libarchive-native curl-native, which is quite a dependency tree to need
in early build (required to build packages via rpm-native).

It was previously forcibly disabled for native packages but the build
dependency on DBus remained.  Remove that logic and add a PACKAGECONFIG
for the plugin that is only enabled for target builds with systemd and
explicitly disabled for native/nativesdk builds.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/recipes-devtools/rpm/rpm_4.15.1.bb | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/meta/recipes-devtools/rpm/rpm_4.15.1.bb b/meta/recipes-devtools/rpm/rpm_4.15.1.bb
index 71e2c033fd..a5b811087b 100644
--- a/meta/recipes-devtools/rpm/rpm_4.15.1.bb
+++ b/meta/recipes-devtools/rpm/rpm_4.15.1.bb
@@ -49,7 +49,7 @@ SRCREV = "ab2179452c5be276a6b96c591afded485c7e58c3"
 
 S = "${WORKDIR}/git"
 
-DEPENDS = "openssl db file popt xz bzip2 dbus elfutils python3"
+DEPENDS = "openssl db file popt xz bzip2 elfutils python3"
 DEPENDS_append_class-native = " file-replacement-native bzip2-replacement-native"
 
 inherit autotools gettext pkgconfig python3native
@@ -62,18 +62,19 @@ EXTRA_OECONF_append = " --without-lua --enable-python --with-crypto=openssl"
 EXTRA_OECONF_append_libc-musl = " --disable-nls --disable-openmp"
 
 # --sysconfdir prevents rpm from attempting to access machine-specific configuration in sysroot/etc; we need to have it in rootfs
-#
 # --localstatedir prevents rpm from writing its database to native sysroot when building images
-#
-# Disable dbus for native, so that rpm doesn't attempt to inhibit shutdown via session dbus even when plugins support is enabled.
-# Also disable plugins by default for native.
-EXTRA_OECONF_append_class-native = " --sysconfdir=/etc --localstatedir=/var --disable-plugins"
-EXTRA_OECONF_append_class-nativesdk = " --sysconfdir=/etc --disable-plugins"
+EXTRA_OECONF_append_class-native = " --sysconfdir=/etc --localstatedir=/var"
+EXTRA_OECONF_append_class-nativesdk = " --sysconfdir=/etc"
 
 BBCLASSEXTEND = "native nativesdk"
 
-PACKAGECONFIG ??= ""
+PACKAGECONFIG ??= "${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'inhibit', '', d)}"
+# The inhibit plugin serves no purpose outside of the target
+PACKAGECONFIG_remove_class-native = "inhibit"
+PACKAGECONFIG_remove_class-nativesdk = "inhibit"
+
 PACKAGECONFIG[imaevm] = "--with-imaevm,,ima-evm-utils"
+PACKAGECONFIG[inhibit] = "--enable-inhibit-plugin,--disable-inhibit-plugin,dbus"
 PACKAGECONFIG[rpm2archive] = "--with-archive,--without-archive,libarchive"
 
 ASNEEDED = ""
-- 
2.25.1

[PATCH 5/7] utils: add umask changing context manager

[Ross Burton]

Add a umask context manager which can be used to temporarily change the
umask in a 'with' block.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 bitbake/lib/bb/utils.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 0b79f92e25..f73d31fb73 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -944,6 +944,17 @@ def which(path, item, direction = 0, history = False, executable=False):
         return "", hist
     return ""
 
+@contextmanager
+def umask(new_mask):
+    """
+    Context manager to set the umask to a specific mask, and restore it afterwards.
+    """
+    current_mask = os.umask(new_mask)
+    try:
+        yield
+    finally:
+        os.umask(current_mask)
+
 def to_boolean(string, default=None):
     if not string:
         return default
-- 
2.25.1

[PATCH 6/7] siggen: use correct umask when writing siginfo

[Ross Burton]

We try to write sstate with group-write permissions so that sstate-cache
can be shared between multiple users.  However the siginfo files are
created in various tasks which may set their own umask (such as
do_populate_sysroot, 0022).  This results in no group write permission
on the intermediate directories, which is fatal when sharing a cache.

Fix this by wrapping the siginfo mkdir in a umask change to 0002.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 bitbake/lib/bb/siggen.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index 1456324a70..86e0e16f39 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -358,7 +358,8 @@ class SignatureGeneratorBasic(SignatureGenerator):
         else:
             sigfile = stampbase + "." + task + ".sigbasedata" + "." + self.basehash[tid]
 
-        bb.utils.mkdirhier(os.path.dirname(sigfile))
+        with bb.utils.umask(0o002):
+            bb.utils.mkdirhier(os.path.dirname(sigfile))
 
         data = {}
         data['task'] = task
-- 
2.25.1

[PATCH 7/7] sstate: set mode explicitly when creating directories in sstate-cache

[Ross Burton]

When creating directories in the sstate-cache, explicitly set the mode
passed to mkdir to 0775 so that the directories are group writable, as
otherwise they cannot be shared with other users.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 66a96a7603..a8ae75101d 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -787,7 +787,7 @@ sstate_create_package () {
 		return
 	fi
 
-	mkdir -p `dirname ${SSTATE_PKG}`
+	mkdir --mode=0775 -p `dirname ${SSTATE_PKG}`
 	TFILE=`mktemp ${SSTATE_PKG}.XXXXXXXX`
 
 	# Use pigz if available
-- 
2.25.1

Re: [OE-core] [PATCH 7/7] sstate: set mode explicitly when creating directories in sstate-cache

[Mark Hatle]

I worry that this could be problematic for other ways.  Security requirements
for an org or even users who want to share the sstate case 0777 etc.

Wouldn't it be better to warn the user that their umask won't allow them to
share this with others, and give them instructions and opening the umask?

--Mark

On 9/28/20 11:19 AM, Ross Burton wrote:
> When creating directories in the sstate-cache, explicitly set the mode
> passed to mkdir to 0775 so that the directories are group writable, as
> otherwise they cannot be shared with other users.
> 
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  meta/classes/sstate.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 66a96a7603..a8ae75101d 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -787,7 +787,7 @@ sstate_create_package () {
>  		return
>  	fi
>  
> -	mkdir -p `dirname ${SSTATE_PKG}`
> +	mkdir --mode=0775 -p `dirname ${SSTATE_PKG}`
>  	TFILE=`mktemp ${SSTATE_PKG}.XXXXXXXX`
>  
>  	# Use pigz if available
> 
> 
> 
> 
> 

Re: [OE-core] [PATCH 7/7] sstate: set mode explicitly when creating directories in sstate-cache

[Ross Burton]

On Mon, 28 Sep 2020 at 17:40, Mark Hatle <mark.hatle@kernel.crashing.org> wrote:
> I worry that this could be problematic for other ways.  Security requirements
> for an org or even users who want to share the sstate case 0777 etc.
>
> Wouldn't it be better to warn the user that their umask won't allow them to
> share this with others, and give them instructions and opening the umask?

FWIW, ten lines lower we chmod 664 the actual data, so this is just in
keeping with existing behaviour.

Ross

Re: [OE-core] [PATCH 1/7] glibc: make nscd optional

[Khem Raj]

On Mon, Sep 28, 2020 at 9:19 AM Ross Burton <ross@burtonini.com> wrote:
>
> NSCD itself is already packaged into its own package, but if it is never
> going to be needed then it can be disabled and it won't be called from
> glibc at all.  Add a PACKAGECONFIG for NSCD that is enabled by default.
>
> As a side effect if NSCD is disable, glibc with and without systemd is
> binary identical, which helps sstate reuse.
>
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  meta/recipes-core/glibc/glibc_2.32.bb | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/meta/recipes-core/glibc/glibc_2.32.bb b/meta/recipes-core/glibc/glibc_2.32.bb
> index 7049e61625..2a0e464385 100644
> --- a/meta/recipes-core/glibc/glibc_2.32.bb
> +++ b/meta/recipes-core/glibc/glibc_2.32.bb
> @@ -69,13 +69,15 @@ EXTRA_OECONF = "--enable-kernel=${OLDEST_KERNEL} \
>                  --enable-stackguard-randomization \
>                  --disable-crypt \
>                  --with-default-link \
> -                --enable-nscd \
>                  ${@bb.utils.contains_any('SELECTED_OPTIMIZATION', '-O0 -Og', '--disable-werror', '', d)} \
>                  ${GLIBCPIE} \
>                  ${GLIBC_EXTRA_OECONF}"
>
>  EXTRA_OECONF += "${@get_libc_fpu_setting(bb, d)}"
>
> +PACKAGECONFIG ??= "nscd"
> +PACKAGECONFIG[nscd] = "--enable-nscd,--disable-nscd"
> +
>  do_patch_append() {
>      bb.build.exec_func('do_fix_readlib_c', d)
>  }

The patch looks good to me but
I wonder if this goes beyond glibc and more like a distro features,
where you can also ignore nsswitch.conf and
packages being aware of it.

> --
> 2.25.1
>
>
> 
>