[RFC PATCH 0/2] Install rdependent of allarch package with same multilib variant of image

[RFC PATCH 0/2] Install rdependent of allarch package with same multilib variant of image

[kai.kang]

From: Kai Kang <kai.kang@windriver.com>

Hi Richard and Mark,

The series rfc patch is trying to install 64bits packages which is
required by allarch package to 32 bits image. Work for rpm and need
little more work and test for dpkg and opkg.

Take ca-certificates as example. It requires openssl. So it intalls
openssl for both core-image-sato and lib32-core-image-sato. But
lib32-openssl is expected to be installed to lib32-core-image-sato.

So extend rdpendent of allarch package with all multilib variants and
'noarch-' as well.

Extend
ca-certificates -> openssl

to

ca-certificates -> openssl, lib32-openssl, noarch-openssl

Rdepends on openssl and lib32-openssl make both of them exist in
oe-core-repo when do rootfs. And then make both openssl and
lib32-openssl provide noarch-openssl. And only keep noarch-openssl in
RDEPENDS when write package for ca-certificates. When do image rootfs
for lib32-core-image-sato, remove openssl if installed already and
install lib32-openssl to provide noarch-openssl for allarch pacakge
ca-certificates.

And extend rprovides of allarch package with prefix 'noarch-' too that
it may be required by other allarch packages. Put all dependent packages
of all allarch recipes in oe-core to a new ariable 'ALLARCH_RDEP_PKGS'.

Any comment for this method is welcomed. Thanks a lot.

--Kai

Kai Kang (2):
  make allarch packages don't dependent specific multilib package
  WIP: install rdependent packages of allarch packages according to
    image rootfs

 meta/classes/allarch.bbclass         |  3 --
 meta/classes/multilib.bbclass        |  3 +-
 meta/classes/multilib_global.bbclass | 31 ++++++++++++++++-
 meta/classes/nativesdk.bbclass       |  8 +++++
 meta/classes/package_deb.bbclass     |  7 +++-
 meta/classes/package_ipk.bbclass     |  7 +++-
 meta/classes/package_rpm.bbclass     |  8 ++++-
 meta/conf/multilib.conf              | 16 +++++++++
 meta/lib/oe/package_manager.py       | 51 ++++++++++++++++++++++++++--
 meta/lib/oe/rootfs.py                | 28 +++++++++++++++
 10 files changed, 152 insertions(+), 10 deletions(-)

-- 
2.17.1

[RFC PATCH 1/2] make allarch packages don't dependent specific multilib package

[kai.kang]

From: Kai Kang <kai.kang@windriver.com>

The dependencies of allarch packages doesn't not extend when enable
multilib. So take ca-certificates as an example, it always requires
openssl no matter whether multilib is enabled. For multilib image such
as lib32-core-image-sato, it installs 64 bits openssl rather than
lib32-openssl which is expected.

So extend rdependencies of allarch package with prefix 'noarch-' and all
of multilib variants. For

ca-certificates -> openssl

exends to

ca-certificates -> openssl, lib32-openssl, noarch-openssl

Rdepends on openssl and lib32-openssl make both of them exist in
oe-core-repo when do rootfs. And then make both openssl and
lib32-openssl provide noarch-openssl. And only keep noarch-openssl in
RDEPENDS when write package for ca-certificates. So when do image rootfs
for lib32-core-image-sato, remove openssl if installed already and
install lib32-openssl for allarch pacakge ca-certificates.

And extend rprovides of allarch package with prefix 'noarch-' too that
it may be required by other allarch packages. Put all dependent packages
of all allarch recipes in oe-core to a new ariable 'ALLARCH_RDEP_PKGS'.

Signed-off-by: Kai Kang <kai.kang@windriver.com>
---
 meta/classes/allarch.bbclass         |  3 ---
 meta/classes/multilib.bbclass        |  3 ++-
 meta/classes/multilib_global.bbclass | 31 +++++++++++++++++++++++++++-
 meta/classes/nativesdk.bbclass       |  8 +++++++
 meta/classes/package_deb.bbclass     |  7 ++++++-
 meta/classes/package_ipk.bbclass     |  7 ++++++-
 meta/classes/package_rpm.bbclass     |  8 ++++++-
 meta/conf/multilib.conf              | 16 ++++++++++++++
 8 files changed, 75 insertions(+), 8 deletions(-)

diff --git a/meta/classes/allarch.bbclass b/meta/classes/allarch.bbclass
index 1eebe0bf2e..52efba405a 100644
--- a/meta/classes/allarch.bbclass
+++ b/meta/classes/allarch.bbclass
@@ -42,9 +42,6 @@ python () {
         d.setVar("INHIBIT_PACKAGE_DEBUG_SPLIT", "1")
         d.setVar("INHIBIT_PACKAGE_STRIP", "1")
 
-        # These multilib values shouldn't change allarch packages so exclude them
-        d.appendVarFlag("emit_pkgdata", "vardepsexclude", " MULTILIB_VARIANTS")
-        d.appendVarFlag("write_specfile", "vardepsexclude", " MULTILIBS")
         d.appendVarFlag("do_package", "vardepsexclude", " package_do_shlibs")
     elif bb.data.inherits_class('packagegroup', d) and not bb.data.inherits_class('nativesdk', d):
         bb.error("Please ensure recipe %s sets PACKAGE_ARCH before inherit packagegroup" % d.getVar("FILE"))
diff --git a/meta/classes/multilib.bbclass b/meta/classes/multilib.bbclass
index f2ac8bdfef..f9417ce69e 100644
--- a/meta/classes/multilib.bbclass
+++ b/meta/classes/multilib.bbclass
@@ -134,7 +134,8 @@ python do_package_qa_multilib() {
                 i = i[len('virtual/'):]
             if (not i.startswith('kernel-module')) and (not i.startswith(mlprefix)) and \
                 (not 'cross-canadian' in i) and (not i.startswith("nativesdk-")) and \
-                (not i.startswith("rtld")) and (not i.startswith('kernel-vmlinux')):
+                (not i.startswith("rtld")) and (not i.startswith('kernel-vmlinux')) and \
+                (not i.startswith("noarch")):
                 candidates.append(i)
         if len(candidates) > 0:
             msg = "%s package %s - suspicious values '%s' in %s" \
diff --git a/meta/classes/multilib_global.bbclass b/meta/classes/multilib_global.bbclass
index d2ec1adfea..a5e3046498 100644
--- a/meta/classes/multilib_global.bbclass
+++ b/meta/classes/multilib_global.bbclass
@@ -161,6 +161,16 @@ multilib_virtclass_handler_vendor[eventmask] = "bb.event.ConfigParsed"
 
 python multilib_virtclass_handler_global () {
     variant = e.data.getVar("BBEXTENDVARIANT")
+
+    # make packages from ALLARCH_RDEP_PKGS provide noarch-pkg
+    if not (bb.data.inherits_class('nativesdk', e.data) or bb.data.inherits_class('crosssdk', e.data)):
+        for pkg in (e.data.getVar("PACKAGES") or "").split():
+            bpkg = pkg.lstrip(variant + "-" if variant else "")
+            if bpkg in (e.data.getVar("ALLARCH_RDEP_PKGS") or ""):
+                rprovs = d.getVar("RPROVIDES_%s" % pkg) or ""
+                rprovs = rprovs + " noarch-" + bpkg
+                d.setVar("RPROVIDES_%s" % pkg, rprovs.strip())
+
     if variant:
         return
 
@@ -194,9 +204,28 @@ python multilib_virtclass_handler_global () {
                 for clsextend in clsextends:
                     rprovs = rprovs + " " + clsextend.map_variable("RPROVIDES_%s" % pkg, setvar=False)
                     rprovs = rprovs + " " + clsextend.extname + "-" + pkg
+
+                if bb.data.inherits_class('allarch', e.data):
+                    # make allarch pkg rprovides noarch-itself
+                    if not pkg.endswith(("-dbg", "-dev", "-staticdev")):
+                        rprovs = rprovs + " noarch-" + pkg
+
                 e.data.setVar("RPROVIDES_%s" % pkg, rprovs)
+
+            # Process RDEPENDS for noarch recipe
+            if bb.data.inherits_class('allarch', e.data) and \
+                not (bb.data.inherits_class('nativesdk', e.data) or bb.data.inherits_class('crosssdk', e.data)):
+                noarch_clsext = oe.classextend.ClassExtender('noarch', e.data)
+                for pkg in (e.data.getVar("PACKAGES") or "").split():
+                    if pkg.endswith(("-dbg", "-dev", "-staticdev")):
+                        continue
+
+                    rdeps = e.data.getVar("RDEPENDS_%s" % pkg) or ""
+                    rdeps = rdeps + " " + noarch_clsext.map_variable("RDEPENDS_%s" % pkg, setvar=False)
+                    for clsextend in clsextends:
+                        rdeps = rdeps + " " + clsextend.map_variable("RDEPENDS_%s" % pkg, setvar=False)
+                    e.data.setVar("RDEPENDS_%s" % pkg, rdeps)
 }
 
 addhandler multilib_virtclass_handler_global
 multilib_virtclass_handler_global[eventmask] = "bb.event.RecipeParsed"
-
diff --git a/meta/classes/nativesdk.bbclass b/meta/classes/nativesdk.bbclass
index ab566e9803..8f8af8f188 100644
--- a/meta/classes/nativesdk.bbclass
+++ b/meta/classes/nativesdk.bbclass
@@ -95,6 +95,14 @@ python () {
     clsextend.map_packagevars()
     clsextend.map_variable("PROVIDES")
     clsextend.map_regexp_variable("PACKAGES_DYNAMIC")
+
+    # make variant-pkg from ALLARCH_RDEP_PKGS provides noarch-pkg
+    for pkg in (d.getVar("PACKAGES") or "").split():
+        bpkg = pkg.replace("nativesdk-", "")
+        if bpkg in (d.getVar("ALLARCH_RDEP_PKGS") or "").split():
+            rprovs = d.getVar("RPROVIDES_%s" % pkg) or ""
+            rprovs = rprovs + " noarch-" + pkg
+            d.setVar("RPROVIDES_%s" % pkg, rprovs)
 }
 
 addhandler nativesdk_virtclass_handler
diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 2e8d17d3c7..71f69b9ac3 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -245,7 +245,12 @@ def deb_write_pkg(pkg, d):
                     elif (v or "").startswith("> "):
                         var[dep][i] = var[dep][i].replace("> ", ">> ")
 
-        rdepends = bb.utils.explode_dep_versions2(localdata.getVar("RDEPENDS") or "")
+        rdepends = localdata.getVar("RDEPENDS") or ""
+        if bb.data.inherits_class('allarch', localdata) \
+            and not bb.data.inherits_class('packagegroup', localdata) \
+            and not pkg.endswith(("-dbg", "-dev", "-staticdev")):
+            rdepends =  " ".join(list(filter(lambda dep: dep.startswith('noarch-'), rdepends.split())))
+        rdepends = bb.utils.explode_dep_versions2(rdepends)
         debian_cmp_remap(rdepends)
         for dep in list(rdepends.keys()):
                 if dep == pkg:
diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index a0b34fa7a5..783a2302bf 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -225,7 +225,12 @@ def ipk_write_pkg(pkg, d):
                     elif (v or "").startswith("> "):
                         var[dep][i] = var[dep][i].replace("> ", ">> ")
 
-        rdepends = bb.utils.explode_dep_versions2(localdata.getVar("RDEPENDS") or "")
+        rdepends = localdata.getVar("RDEPENDS") or ""
+        if bb.data.inherits_class('allarch', localdata) \
+            and not bb.data.inherits_class('packagegroup', localdata) \
+            and not pkg.endswith(("-dbg", "-dev", "-staticdev")):
+            rdepends = " ".join(list(filter(lambda dep: dep.startswith('noarch-'), rdepends)))
+        rdepends = bb.utils.explode_dep_versions2(rdepends)
         debian_cmp_remap(rdepends)
         rrecommends = bb.utils.explode_dep_versions2(localdata.getVar("RRECOMMENDS") or "")
         debian_cmp_remap(rrecommends)
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 21ada348aa..dbd7cc3701 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -333,6 +333,13 @@ python write_specfile () {
 
         localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
 
+        if bb.data.inherits_class('allarch', localdata) \
+            and not bb.data.inherits_class('packagegroup', localdata) \
+            and not pkg.endswith(("-dbg", "-dev", "-staticdev")):
+            rdeps = (localdata.getVar('RDEPENDS') or "").split()
+            rdeps = list(filter(lambda dep: dep.startswith('noarch-'), rdeps))
+            localdata.setVar('RDEPENDS', ' '.join(rdeps))
+
         conffiles = get_conffiles(pkg, d)
         dirfiles = localdata.getVar('DIRFILES')
         if dirfiles is not None:
@@ -372,7 +379,6 @@ python write_specfile () {
         splitrprerm    = localdata.getVar('pkg_prerm')
         splitrpostrm   = localdata.getVar('pkg_postrm')
 
-
         if not perfiledeps:
             # Add in summary of per file dependencies
             splitrdepends = splitrdepends + " " + get_perfile('RDEPENDS', pkg, d)
diff --git a/meta/conf/multilib.conf b/meta/conf/multilib.conf
index e74dec81a8..2466befbca 100644
--- a/meta/conf/multilib.conf
+++ b/meta/conf/multilib.conf
@@ -26,3 +26,19 @@ OPKG_ARGS_append = " --force-maintainer --force-overwrite"
 PKG_CONFIG_PATH .= ":${WORKDIR}/recipe-sysroot/${datadir}/pkgconfig"
 PKG_CONFIG_PATH[vardepsexclude] = "datadir WORKDIR"
 PKG_CONFIG_PATH[vardepvalueexclude] = ":${WORKDIR}/recipe-sysroot/${datadir}/pkgconfig"
+
+# resolve runtime dependencies of allarch packages when enable multilib
+ALLARCH_RDEP_PKGS ?= "\
+    bash \
+    busybox-mdev \
+    font-util \
+    fontconfig-utils \
+    kbd \
+    openssl \
+    perl \
+    ppp \
+    udev \
+    udev-extraconf \
+    weston \
+    ${VIRTUAL-RUNTIME_base-utils} \
+"
-- 
2.17.1

[RFC PATCH 2/2] WIP: install rdependent packages of allarch packages according to image rootfs

[kai.kang]

From: Kai Kang <kai.kang@windriver.com>

Add interface is_leaf to package managers. If any multilib variant of
package from 'ALLARCH_RDEP_PKGS' is installed to image rootfs, then
install the packages with 'BBEXTENDVARIANT' of current image and remove
the already installed package if not required by others and not add to
image intentionally.

Signed-off-by: Kai Kang <kai.kang@windriver.com>
---
 meta/lib/oe/package_manager.py | 51 ++++++++++++++++++++++++++++++++--
 meta/lib/oe/rootfs.py          | 28 +++++++++++++++++++
 2 files changed, 77 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 9aa5847c8a..eeaf297432 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -249,7 +249,7 @@ class DpkgIndexer(Indexer):
                 release.write("Label: %s\n" % arch)
 
             cmd += "PSEUDO_UNLOAD=1 %s release . >> Release" % apt_ftparchive
-            
+
             index_cmds.append(cmd)
 
             deb_dirs_found = True
@@ -448,6 +448,13 @@ class PackageManager(object, metaclass=ABCMeta):
     def list_installed(self):
         pass
 
+    @abstractmethod
+    def is_leaf(self, pkg):
+        """
+        Whether pkg is required by other packages
+        """
+        pass
+
     @abstractmethod
     def extract(self, pkg):
         """
@@ -587,7 +594,7 @@ def create_packages_dir(d, rpm_repo_dir, deploydir, taskname, filterbydependenci
     pn = d.getVar("PN")
     seendirs = set()
     multilibs = {}
-   
+
     rpm_subrepo_dir = oe.path.join(rpm_repo_dir, "rpm")
 
     bb.utils.remove(rpm_subrepo_dir, recurse=True)
@@ -895,6 +902,20 @@ class RpmPM(PackageManager):
 
         return packages
 
+    def is_leaf(self, pkg):
+        if not pkg:
+            return False
+
+        cmd = bb.utils.which(os.getenv('PATH'), "rpm")
+        args = ["-q", "--whatrequires", "--root=%s" % self.target_rootfs]
+
+        bb.note("Running %s" % ' '.join([cmd] + args + [pkg]))
+        # return value 0 stands for other(s) require(s) pkg
+        if subprocess.call([cmd] + args + [pkg], shell=True):
+            return True
+        else:
+            return False
+
     def update(self):
         self._invoke_dnf(["makecache", "--refresh"])
 
@@ -1327,6 +1348,12 @@ class OpkgPM(OpkgDpkgPM):
     def list_installed(self):
         return OpkgPkgsList(self.d, self.target_rootfs, self.config_file).list_pkgs()
 
+    def is_leaf(self, pkg):
+        if not pkg:
+            return False
+
+
+
     def handle_bad_recommendations(self):
         bad_recommendations = self.d.getVar("BAD_RECOMMENDATIONS") or ""
         if bad_recommendations.strip() == "":
@@ -1753,6 +1780,26 @@ class DpkgPM(OpkgDpkgPM):
     def list_installed(self):
         return DpkgPkgsList(self.d, self.target_rootfs).list_pkgs()
 
+    def is_leaf(self, pkg):
+        bb.warn("=====================in dpkg is_leaf()\n")
+        if not pkg:
+            return False
+
+        cmd = "%s rdepends %s" % (self.apt_cache_cmd, pkg)
+
+        try:
+            output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).decode("utf-8")
+        except subprocess.CalledProcessError as e:
+            bb.fatal("Unable to list rdepends packages. Command '%s' "
+                     "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
+
+        bb.warn(output)
+        bb.warn(len(output.split()))
+        if len(output.split()) > 2:
+            return False
+        else:
+            return True
+
     def package_info(self, pkg):
         """
         Returns a dictionary with the package info.
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 28642abbd9..54bde6798b 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -170,6 +170,7 @@ class Rootfs(object, metaclass=ABCMeta):
 
     def create(self):
         bb.note("###### Generate rootfs #######")
+        bb.warn(" in create ###### Generate rootfs #######")
         pre_process_cmds = self.d.getVar("ROOTFS_PREPROCESS_COMMAND")
         post_process_cmds = self.d.getVar("ROOTFS_POSTPROCESS_COMMAND")
         rootfs_post_install_cmds = self.d.getVar('ROOTFS_POSTINSTALL_COMMAND')
@@ -213,6 +214,9 @@ class Rootfs(object, metaclass=ABCMeta):
 
         self._uninstall_unneeded()
 
+        bb.warn("========= in rootfs() before self._fix_allarch_ml_rdeps\n")
+        self._fix_allarch_ml_rdeps()
+
         if self.progress_reporter:
             self.progress_reporter.next_stage()
 
@@ -279,6 +283,30 @@ class Rootfs(object, metaclass=ABCMeta):
             # Remove the package manager data files
             self.pm.remove_packaging_data()
 
+    def _fix_allarch_ml_rdeps(self):
+        variant = self.d.getVar("BBEXTENDVARIANT") or ""
+        ml_variants = self.d.getVar("MULTILIB_VARIANTS") or ""
+
+        packages = self.pm.list_installed()
+
+        pkg_to_remove = list()
+        bb.warn("======= in _fix_allarch_ml_rdeps \n")
+        for var in ml_variants.split():
+            for pkg in (self.d.getVar("ALLARCH_RDEP_PKGS") or "").split():
+                ml_pkg = pkg if var == variant else var + "-" + pkg
+                if ml_pkg in packages:
+                    cur_pkg = variant + "-" + pkg if variant else pkg
+                    if not cur_pkg in packages:
+                        self.pm.install([cur_pkg])
+                    if self.pm.is_leaf(ml_pkg) \
+                        and ml_pkg not in (self.d.getVar("IMAGE_INSTALL") or "").split():
+                        pkg_to_remove.append(ml_pkg)
+
+        bb.note("Following duplicated packages required by all arch packages \
+                will be removed: %s" % " ".join(pkg_to_remove))
+        self.pm.update()
+        self.pm.remove(pkg_to_remove)
+
     def _run_ldconfig(self):
         if self.d.getVar('LDCONFIGDEPEND'):
             bb.note("Executing: ldconfig -r" + self.image_rootfs + "-c new -v")
-- 
2.17.1

Re: [RFC PATCH 0/2] Install rdependent of allarch package with same multilib variant of image

[richard.purdie]

Hi Kai,

On Thu, 2018-07-26 at 00:41 +0800, kai.kang@windriver.com wrote:
> The series rfc patch is trying to install 64bits packages which is
> required by allarch package to 32 bits image. Work for rpm and need
> little more work and test for dpkg and opkg.
> 
> Take ca-certificates as example. It requires openssl. So it intalls
> openssl for both core-image-sato and lib32-core-image-sato. But
> lib32-openssl is expected to be installed to lib32-core-image-sato.
> 
> So extend rdpendent of allarch package with all multilib variants and
> 'noarch-' as well.
> 
> Extend
> ca-certificates -> openssl
> 
> to
> 
> ca-certificates -> openssl, lib32-openssl, noarch-openssl
> 
> Rdepends on openssl and lib32-openssl make both of them exist in
> oe-core-repo when do rootfs. And then make both openssl and
> lib32-openssl provide noarch-openssl. And only keep noarch-openssl in
> RDEPENDS when write package for ca-certificates. When do image rootfs
> for lib32-core-image-sato, remove openssl if installed already and
> install lib32-openssl to provide noarch-openssl for allarch pacakge
> ca-certificates.
> 
> And extend rprovides of allarch package with prefix 'noarch-' too
> that
> it may be required by other allarch packages. Put all dependent
> packages
> of all allarch recipes in oe-core to a new ariable
> 'ALLARCH_RDEP_PKGS'.
> 
> Any comment for this method is welcomed. Thanks a lot.

I think this is kind of the right direction but its not ready in this
form. Taking a step back and thinking about this problem, its actually
similar to the allarch recipes signature problem.

Normally if A DEPENDS on B, the taskhash of A includes the taskhash of
B. In the allarch case this is problematic so you see ca-certificates-
>openssl listed in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS.

I think (but am not 100% sure) that everything you list in
ALLARCH_RDEP_PKGS is already covered somewhere in
SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS.

I'd therefore propose using SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS instead of
adding a ALLARCH_RDEP_PKGS which duplicates information. I can't see
any case where you'd want an entry in one but not the other.

Rather than changing the RPROVIDES/RDEPENDS in package_{deb|ipk|rpm},
you should do this once in package.bbclass.

There is a bigger problem though since the allarch recipe signature
filtering doesn't change what bitbake will actually build, only how its
represented in sstate and the circumstances under which it would
rebuild it. If ca-certificates DEPENDS on openssl, it will build
openssl whether you're targeting core-image-sato or lib32-core-image-
sato. This potentially means you can end up in a situation where it
wouldn't build lib32-openssl if nothing else in lib32-core-image-sato
depended upon it more directly.

There is no easy way to solve this problem since somehow, if you're
building lib32-<image> it needs to build lib32-openssl and if you're
building <image> it needs to build openssl. "pushing" that information
back up the dependency chain is very hard, we have no such mechanism.

I'm afraid I don't have a solution for this piece, its a very hard
problem to solve. Your current patches happen to work as there are
other more direct dependencies which tend to ensure the right thing
happens but we can't rely on that and it probably isn't the case for
all the dependencies.

Finally, the package manager shouldn't really need changes for this
approach to work. If things RPROVIDE and RDEPEND on the noarch
versions, the feed priorities should mean the right things happen. If
that doesn't happen automatically, the feed prorities are not working
correctly and that needs fixing, not the 2/2 patch you're proposing. I
suspect this issue needs fixing regardless of anything else.

Cheers,

Richard

Re: [RFC PATCH 0/2] Install rdependent of allarch package with same multilib variant of image

[Kang Kai]

On 2018年07月26日 06:47, richard.purdie@linuxfoundation.org wrote:
> Hi Kai,
>
> On Thu, 2018-07-26 at 00:41 +0800, kai.kang@windriver.com wrote:
>> The series rfc patch is trying to install 64bits packages which is
>> required by allarch package to 32 bits image. Work for rpm and need
>> little more work and test for dpkg and opkg.
>>
>> Take ca-certificates as example. It requires openssl. So it intalls
>> openssl for both core-image-sato and lib32-core-image-sato. But
>> lib32-openssl is expected to be installed to lib32-core-image-sato.
>>
>> So extend rdpendent of allarch package with all multilib variants and
>> 'noarch-' as well.
>>
>> Extend
>> ca-certificates -> openssl
>>
>> to
>>
>> ca-certificates -> openssl, lib32-openssl, noarch-openssl
>>
>> Rdepends on openssl and lib32-openssl make both of them exist in
>> oe-core-repo when do rootfs. And then make both openssl and
>> lib32-openssl provide noarch-openssl. And only keep noarch-openssl in
>> RDEPENDS when write package for ca-certificates. When do image rootfs
>> for lib32-core-image-sato, remove openssl if installed already and
>> install lib32-openssl to provide noarch-openssl for allarch pacakge
>> ca-certificates.
>>
>> And extend rprovides of allarch package with prefix 'noarch-' too
>> that
>> it may be required by other allarch packages. Put all dependent
>> packages
>> of all allarch recipes in oe-core to a new ariable
>> 'ALLARCH_RDEP_PKGS'.
>>
>> Any comment for this method is welcomed. Thanks a lot.

Hi Richard,

Thanks for detailed reply.

> I think this is kind of the right direction but its not ready in this
> form. Taking a step back and thinking about this problem, its actually
> similar to the allarch recipes signature problem.
>
> Normally if A DEPENDS on B, the taskhash of A includes the taskhash of
> B. In the allarch case this is problematic so you see ca-certificates-
>> openssl listed in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS.
> I think (but am not 100% sure) that everything you list in
> ALLARCH_RDEP_PKGS is already covered somewhere in
> SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS.
>
> I'd therefore propose using SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS instead of
> adding a ALLARCH_RDEP_PKGS which duplicates information. I can't see
> any case where you'd want an entry in one but not the other.
Yes, it seems ALLARCH_RDEP_PKGS and I'll try to use 
SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS instead.

>
> Rather than changing the RPROVIDES/RDEPENDS in package_{deb|ipk|rpm},
> you should do this once in package.bbclass.
ok.

>
> There is a bigger problem though since the allarch recipe signature
> filtering doesn't change what bitbake will actually build, only how its
> represented in sstate and the circumstances under which it would
> rebuild it. If ca-certificates DEPENDS on openssl, it will build
> openssl whether you're targeting core-image-sato or lib32-core-image-
> sato. This potentially means you can end up in a situation where it
> wouldn't build lib32-openssl if nothing else in lib32-core-image-sato
> depended upon it more directly.

That's why extend rdepends of allarch package with all multilib variant. 
In bitbake inner environment, make
ca-certificates depends on openssl, lib32-openssl and even 
libn32-openssl for qemumips64. It may cost a little
more build time(I didn't test yet), but the dependency relationship is 
fixed and make sure that all multilib version
of openssl exist in 'oe-core-repo' which is used for do_rootfs.

When build ca-certificates, both openssl and lib32-openssl will be 
built. And when write package of ca-certificates,
remove openssl and lib32-openssl from RDEPENDS and keep noarch-openssl. 
And make both openssl and
lib32-openssl provides noarch-openssl. Then some tweak according image 
multilib type could be done at then end
of do_rootfs.


>
> There is no easy way to solve this problem since somehow, if you're
> building lib32-<image> it needs to build lib32-openssl and if you're
> building <image> it needs to build openssl. "pushing" that information
> back up the dependency chain is very hard, we have no such mechanism.
I once want to set 'PREFERRED_PROVIDER_noarch-openssl' in image.bbclass, 
and finally find that the setting only exists
in d of <image>, and can't pass to outer context of 'defaultpkgname'

>
> I'm afraid I don't have a solution for this piece, its a very hard
> problem to solve. Your current patches happen to work as there are
> other more direct dependencies which tend to ensure the right thing
> happens but we can't rely on that and it probably isn't the case for
> all the dependencies.
>
> Finally, the package manager shouldn't really need changes for this
> approach to work. If things RPROVIDE and RDEPEND on the noarch
> versions, the feed priorities should mean the right things happen. If
> that doesn't happen automatically, the feed prorities are not working
> correctly and that needs fixing, not the 2/2 patch you're proposing. I
> suspect this issue needs fixing regardless of anything else.
Make feed repo with different priority once was an option. But I 
selected this way first. I'll work on it.

Regards,
Kai

>
> Cheers,
>
> Richard
>

-- 
Regards,
Neil | Kai Kang

Re: [RFC PATCH 0/2] Install rdependent of allarch package with same multilib variant of image

[richard.purdie]

On Thu, 2018-07-26 at 14:53 +0800, Kang Kai wrote:
> On 2018年07月26日 06:47, richard.purdie@linuxfoundation.org wrote:
> > There is a bigger problem though since the allarch recipe signature
> > filtering doesn't change what bitbake will actually build, only how
> > its
> > represented in sstate and the circumstances under which it would
> > rebuild it. If ca-certificates DEPENDS on openssl, it will build
> > openssl whether you're targeting core-image-sato or lib32-core-
> > image-
> > sato. This potentially means you can end up in a situation where it
> > wouldn't build lib32-openssl if nothing else in lib32-core-image-
> > sato
> > depended upon it more directly.
> 
> That's why extend rdepends of allarch package with all multilib
> variant. 
> In bitbake inner environment, make ca-certificates depends on
> openssl, lib32-openssl and even libn32-openssl for qemumips64. It may
> cost a little more build time(I didn't test yet), but the dependency
> relationship is fixed and make sure that all multilib version
> of openssl exist in 'oe-core-repo' which is used for do_rootfs.
> 
> When build ca-certificates, both openssl and lib32-openssl will be 
> built. And when write package of ca-certificates, remove openssl and
> lib32-openssl from RDEPENDS and keep noarch-openssl. 
> And make both openssl and lib32-openssl provides noarch-openssl. Then
> some tweak according image multilib type could be done at then end
> of do_rootfs.

Requiring that all variants of openssl build in order to build ca-
certificates will have huge build time impact as it will need to build
both multilib toolchains.

At that point we'd probably be better either building allarch recipes
twice (which I don't like), or, banning non-allarch dependencies from
allarch recipes. If a recipe needs a non-allarch dependency we'd make
it non-allarch. We may just have to accept the latter...

Cheers,

Richard

Re: [RFC PATCH 0/2] Install rdependent of allarch package with same multilib variant of image

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3162 bytes --]

FWIW: the later is what I did in morty based build to get multilib working
with allarch.

It was actually combination of both, in some cases the dependency was
already pulled by something else and it was relatively safe to drop from
allarch recipe (e.g. I did that for ca-certificates), in some cases I've
just dropped allarch.

I also don't like building things twice, we have allarch web apps which
have runtime dependency on web engine (which includes chromium build) and I
really don't want to build chromium twice just to satisfy allarch
dependencies, when we really want to have just one chromium in the rootfs
in the end. So it's not only the multilib toochain and small things like
openssl, but it can get huge. Building small allarch web app multiple times
(per TUNE_PKGARCH) on the other hand is relatively small price to pay.

On Thu, Jul 26, 2018 at 11:07 AM <richard.purdie@linuxfoundation.org> wrote:

> On Thu, 2018-07-26 at 14:53 +0800, Kang Kai wrote:
> > On 2018年07月26日 06:47, richard.purdie@linuxfoundation.org wrote:
> > > There is a bigger problem though since the allarch recipe signature
> > > filtering doesn't change what bitbake will actually build, only how
> > > its
> > > represented in sstate and the circumstances under which it would
> > > rebuild it. If ca-certificates DEPENDS on openssl, it will build
> > > openssl whether you're targeting core-image-sato or lib32-core-
> > > image-
> > > sato. This potentially means you can end up in a situation where it
> > > wouldn't build lib32-openssl if nothing else in lib32-core-image-
> > > sato
> > > depended upon it more directly.
> >
> > That's why extend rdepends of allarch package with all multilib
> > variant.
> > In bitbake inner environment, make ca-certificates depends on
> > openssl, lib32-openssl and even libn32-openssl for qemumips64. It may
> > cost a little more build time(I didn't test yet), but the dependency
> > relationship is fixed and make sure that all multilib version
> > of openssl exist in 'oe-core-repo' which is used for do_rootfs.
> >
> > When build ca-certificates, both openssl and lib32-openssl will be
> > built. And when write package of ca-certificates, remove openssl and
> > lib32-openssl from RDEPENDS and keep noarch-openssl.
> > And make both openssl and lib32-openssl provides noarch-openssl. Then
> > some tweak according image multilib type could be done at then end
> > of do_rootfs.
>
> Requiring that all variants of openssl build in order to build ca-
> certificates will have huge build time impact as it will need to build
> both multilib toolchains.
>
> At that point we'd probably be better either building allarch recipes
> twice (which I don't like), or, banning non-allarch dependencies from
> allarch recipes. If a recipe needs a non-allarch dependency we'd make
> it non-allarch. We may just have to accept the latter...
>
> Cheers,
>
> Richard
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3982 bytes --]

Re: [RFC PATCH 0/2] Install rdependent of allarch package with same multilib variant of image

[Alexander Kanavin]

2018-07-26 11:07 GMT+02:00  <richard.purdie@linuxfoundation.org>:
> Requiring that all variants of openssl build in order to build ca-
> certificates will have huge build time impact as it will need to build
> both multilib toolchains.
>
> At that point we'd probably be better either building allarch recipes
> twice (which I don't like), or, banning non-allarch dependencies from
> allarch recipes. If a recipe needs a non-allarch dependency we'd make
> it non-allarch. We may just have to accept the latter...

Another issue is that whatever solution ends up being merged it needs
to be understandable by a human. I remember I ripped out a large chunk
multilib-related renaming of packages (probably dealing with the same
issue) when transitioning from rpm5 to rpm4, and the reason was,
essentially, that I was not able to understand what the code is doing
and why.

Alex

Re: [RFC PATCH 0/2] Install rdependent of allarch package with same multilib variant of image

[Mark Hatle]

On 7/26/18 4:07 AM, richard.purdie@linuxfoundation.org wrote:
> On Thu, 2018-07-26 at 14:53 +0800, Kang Kai wrote:
>> On 2018年07月26日 06:47, richard.purdie@linuxfoundation.org wrote:
>>> There is a bigger problem though since the allarch recipe signature
>>> filtering doesn't change what bitbake will actually build, only how
>>> its
>>> represented in sstate and the circumstances under which it would
>>> rebuild it. If ca-certificates DEPENDS on openssl, it will build
>>> openssl whether you're targeting core-image-sato or lib32-core-
>>> image-
>>> sato. This potentially means you can end up in a situation where it
>>> wouldn't build lib32-openssl if nothing else in lib32-core-image-
>>> sato
>>> depended upon it more directly.
>>
>> That's why extend rdepends of allarch package with all multilib
>> variant. 
>> In bitbake inner environment, make ca-certificates depends on
>> openssl, lib32-openssl and even libn32-openssl for qemumips64. It may
>> cost a little more build time(I didn't test yet), but the dependency
>> relationship is fixed and make sure that all multilib version
>> of openssl exist in 'oe-core-repo' which is used for do_rootfs.
>>
>> When build ca-certificates, both openssl and lib32-openssl will be 
>> built. And when write package of ca-certificates, remove openssl and
>> lib32-openssl from RDEPENDS and keep noarch-openssl. 
>> And make both openssl and lib32-openssl provides noarch-openssl. Then
>> some tweak according image multilib type could be done at then end
>> of do_rootfs.
> 
> Requiring that all variants of openssl build in order to build ca-
> certificates will have huge build time impact as it will need to build
> both multilib toolchains.
> 
> At that point we'd probably be better either building allarch recipes
> twice (which I don't like), or, banning non-allarch dependencies from
> allarch recipes. If a recipe needs a non-allarch dependency we'd make
> it non-allarch. We may just have to accept the latter...

allarch requiring (runtime) dependencies on non-allarch might just have to be
banned.  I hadn't considered that -- but it would resolve most of the problems
that are being worked on.

If we can produce the allarch package -once- w/o rdepends on anything then they
should still be fine.. since the dependency would be:

arch1 |
      | -> allarch
arch2 |

Avoiding conflicts and preserving the approach we want.

--Mark

> Cheers,
> 
> Richard
>