* [PATCH v2 0/3] Debian multilib packaging fixes
@ 2015-12-18 14:34 Matt Madison
2015-12-18 14:34 ` [PATCH v2 1/3] package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping function Matt Madison
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Matt Madison @ 2015-12-18 14:34 UTC (permalink / raw)
To: openembedded-core
This adds a third patch that fixes the problem where APT couldn't
resolve dependencies between when two packages of different architectures
both depended on an allarch package.
Matt Madison (3):
package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping
function
package_manager.py: fixes for multilib deb packaging builds
package_deb.bbclass: add 'Multi-Arch: foreign' tag to allarch packages
meta/classes/cross-canadian.bbclass | 2 +-
meta/classes/package_deb.bbclass | 37 +++++++++++++++++++++++++++----------
meta/lib/oe/package_manager.py | 17 +++++++++++------
3 files changed, 39 insertions(+), 17 deletions(-)
--
2.5.0
^ permalink raw reply [flat|nested] 8+ messages in thread* [PATCH v2 1/3] package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping function 2015-12-18 14:34 [PATCH v2 0/3] Debian multilib packaging fixes Matt Madison @ 2015-12-18 14:34 ` Matt Madison 2015-12-18 14:34 ` [PATCH v2 2/3] package_manager.py: fixes for multilib deb packaging builds Matt Madison ` (2 subsequent siblings) 3 siblings, 0 replies; 8+ messages in thread From: Matt Madison @ 2015-12-18 14:34 UTC (permalink / raw) To: openembedded-core Have DPKG_ARCH set by directly invoking a mapping function, rather than using an anonymous Python function modify the variable under the hood, so we can have proper handling of overrides. Also bring in some additional mappings to Debian architecture names that weren't being handled. Signed-off-by: Matt Madison <matt@madison.systems> --- meta/classes/cross-canadian.bbclass | 2 +- meta/classes/package_deb.bbclass | 35 +++++++++++++++++++++++++---------- 2 files changed, 26 insertions(+), 11 deletions(-) diff --git a/meta/classes/cross-canadian.bbclass b/meta/classes/cross-canadian.bbclass index 7b26ed6..e07b1bd 100644 --- a/meta/classes/cross-canadian.bbclass +++ b/meta/classes/cross-canadian.bbclass @@ -103,7 +103,7 @@ HOST_LD_ARCH = "${SDK_LD_ARCH}" HOST_AS_ARCH = "${SDK_AS_ARCH}" #assign DPKG_ARCH -DPKG_ARCH = "${SDK_ARCH}" +DPKG_ARCH = "${@debian_arch_map(d.getVar('SDK_ARCH', True), '')}" CPPFLAGS = "${BUILDSDK_CPPFLAGS}" CFLAGS = "${BUILDSDK_CFLAGS}" diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass index 5feeeb0..8d27adf 100644 --- a/meta/classes/package_deb.bbclass +++ b/meta/classes/package_deb.bbclass @@ -6,7 +6,7 @@ inherit package IMAGE_PKGTYPE ?= "deb" -DPKG_ARCH ?= "${TARGET_ARCH}" +DPKG_ARCH ?= "${@debian_arch_map(d.getVar('TARGET_ARCH', True), d.getVar('TUNE_FEATURES', True))}" PKGWRITEDIRDEB = "${WORKDIR}/deploy-debs" @@ -14,6 +14,28 @@ APTCONF_TARGET = "${WORKDIR}" APT_ARGS = "${@['', '--no-install-recommends'][d.getVar("NO_RECOMMENDATIONS", True) == "1"]}" +def debian_arch_map(arch, tune): + tune_features = tune.split() + if arch in ["i586", "i686"]: + return "i386" + if arch == "x86_64": + if "mx32" in tune_features: + return "x32" + return "amd64" + if arch.startswith("mips"): + endian = ["el", ""]["bigendian" in tune_features] + if "n64" in tune_features: + return "mips64" + endian + if "n32" in tune_features: + return "mipsn32" + endian + return "mips" + endian + if arch == "powerpc": + return arch + ["", "spe"]["spe" in tune_features] + if arch == "aarch64": + return "arm64" + if arch == "arm": + return arch + ["el", "hf"]["callconvention-hard" in tune_features] + return arch # # install a bunch of packages using apt # the following shell variables needs to be set before calling this func: @@ -288,6 +310,8 @@ python do_package_deb () { cleanupcontrol(root) bb.utils.unlockfile(lf) } +# Indirect references to these vars +do_package_write_deb[vardeps] += "PKGV PKGR PKGV DESCRIPTION SECTION PRIORITY MAINTAINER DPKG_ARCH PN HOMEPAGE" # Otherwise allarch packages may change depending on override configuration do_package_deb[vardepsexclude] = "OVERRIDES" @@ -311,15 +335,6 @@ python () { deps = ' dpkg-native:do_populate_sysroot virtual/fakeroot-native:do_populate_sysroot' d.appendVarFlag('do_package_write_deb', 'depends', deps) d.setVarFlag('do_package_write_deb', 'fakeroot', "1") - - # Map TARGET_ARCH to Debian's ideas about architectures - darch = d.getVar('DPKG_ARCH', True) - if darch in ["x86", "i486", "i586", "i686", "pentium"]: - d.setVar('DPKG_ARCH', 'i386') - elif darch == "x86_64": - d.setVar('DPKG_ARCH', 'amd64') - elif darch == "arm": - d.setVar('DPKG_ARCH', 'armel') } python do_package_write_deb () { -- 2.5.0 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v2 2/3] package_manager.py: fixes for multilib deb packaging builds 2015-12-18 14:34 [PATCH v2 0/3] Debian multilib packaging fixes Matt Madison 2015-12-18 14:34 ` [PATCH v2 1/3] package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping function Matt Madison @ 2015-12-18 14:34 ` Matt Madison 2015-12-18 14:34 ` [PATCH v2 3/3] package_deb.bbclass: add 'Multi-Arch: foreign' tag to allarch packages Matt Madison 2016-01-04 20:56 ` [PATCH v2 0/3] Debian multilib packaging fixes Aníbal Limón 3 siblings, 0 replies; 8+ messages in thread From: Matt Madison @ 2015-12-18 14:34 UTC (permalink / raw) To: openembedded-core * tmp/deploy/deb subdirectories do not get hyphens replaced with underscores, so don't do that translation when building the sources list. * Fix MULTILIB_VARIANTS handling to be more general and work for all architectures * Also include a fix for a warning generated by apt due to missing apt/preferences.d directory. Signed-off-by: Matt Madison <matt@madison.systems> --- meta/lib/oe/package_manager.py | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py index fd9caa3..b78d291 100644 --- a/meta/lib/oe/package_manager.py +++ b/meta/lib/oe/package_manager.py @@ -247,7 +247,7 @@ class DpkgIndexer(Indexer): if a not in pkg_archs: arch_list.append(a) - all_mlb_pkg_arch_list = (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or "").replace('-', '_').split() + all_mlb_pkg_arch_list = (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or "").split() arch_list.extend(arch for arch in all_mlb_pkg_arch_list if arch not in arch_list) apt_ftparchive = bb.utils.which(os.getenv('PATH'), "apt-ftparchive") @@ -1750,7 +1750,7 @@ class DpkgPM(PackageManager): self.apt_args = d.getVar("APT_ARGS", True) self.all_arch_list = archs.split() - all_mlb_pkg_arch_list = (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or "").replace('-', '_').split() + all_mlb_pkg_arch_list = (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or "").split() self.all_arch_list.extend(arch for arch in all_mlb_pkg_arch_list if arch not in self.all_arch_list) self._create_configs(archs, base_archs) @@ -1941,6 +1941,7 @@ class DpkgPM(PackageManager): bb.utils.mkdirhier(self.apt_conf_dir) bb.utils.mkdirhier(self.apt_conf_dir + "/lists/partial/") bb.utils.mkdirhier(self.apt_conf_dir + "/apt.conf.d/") + bb.utils.mkdirhier(self.apt_conf_dir + "/preferences.d/") arch_list = [] for arch in self.all_arch_list: @@ -1975,10 +1976,14 @@ class DpkgPM(PackageManager): base_arch_list = base_archs.split() multilib_variants = self.d.getVar("MULTILIB_VARIANTS", True); for variant in multilib_variants.split(): - if variant == "lib32": - base_arch_list.append("i386") - elif variant == "lib64": - base_arch_list.append("amd64") + localdata = bb.data.createCopy(self.d) + variant_tune = localdata.getVar("DEFAULTTUNE_virtclass-multilib-" + variant, False) + orig_arch = localdata.getVar("DPKG_ARCH", True) + localdata.setVar("DEFAULTTUNE", variant_tune) + bb.data.update_data(localdata) + variant_arch = localdata.getVar("DPKG_ARCH", True) + if variant_arch not in base_arch_list: + base_arch_list.append(variant_arch) with open(self.apt_conf_file, "w+") as apt_conf: with open(self.d.expand("${STAGING_ETCDIR_NATIVE}/apt/apt.conf.sample")) as apt_conf_sample: -- 2.5.0 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v2 3/3] package_deb.bbclass: add 'Multi-Arch: foreign' tag to allarch packages 2015-12-18 14:34 [PATCH v2 0/3] Debian multilib packaging fixes Matt Madison 2015-12-18 14:34 ` [PATCH v2 1/3] package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping function Matt Madison 2015-12-18 14:34 ` [PATCH v2 2/3] package_manager.py: fixes for multilib deb packaging builds Matt Madison @ 2015-12-18 14:34 ` Matt Madison 2016-01-04 20:56 ` [PATCH v2 0/3] Debian multilib packaging fixes Aníbal Limón 3 siblings, 0 replies; 8+ messages in thread From: Matt Madison @ 2015-12-18 14:34 UTC (permalink / raw) To: openembedded-core This tells APT that it can use such packages to resolve dependencies from packages of any architecture in a multilib build. Signed-off-by: Matt Madison <matt@madison.systems> --- meta/classes/package_deb.bbclass | 2 ++ 1 file changed, 2 insertions(+) diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass index 8d27adf..25218d0 100644 --- a/meta/classes/package_deb.bbclass +++ b/meta/classes/package_deb.bbclass @@ -161,6 +161,8 @@ python do_package_deb () { return l2 ctrlfile.write("Package: %s\n" % pkgname) + if d.getVar('PACKAGE_ARCH', True) == "all": + ctrlfile.write("Multi-Arch: foreign\n") # check for required fields try: for (c, fs) in fields: -- 2.5.0 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH v2 0/3] Debian multilib packaging fixes 2015-12-18 14:34 [PATCH v2 0/3] Debian multilib packaging fixes Matt Madison ` (2 preceding siblings ...) 2015-12-18 14:34 ` [PATCH v2 3/3] package_deb.bbclass: add 'Multi-Arch: foreign' tag to allarch packages Matt Madison @ 2016-01-04 20:56 ` Aníbal Limón 2016-01-05 17:00 ` Matt Madison 3 siblings, 1 reply; 8+ messages in thread From: Aníbal Limón @ 2016-01-04 20:56 UTC (permalink / raw) To: Matt Madison, openembedded-core Hi Matt, Sorry for the delay in the response, i'll test again your patches with the new one and the same error appears when build core-image-full-cmdline or core-image-sato. MACHINE ??= "qemux86-64" IMAGE_INSTALL_append = " lib32-connman" require conf/multilib.conf MULTILIBS = "multilib:lib32" DEFAULTTUNE_virtclass-multilib-lib32 = "x86" I'm reviewing if this an error in the meta data (because core-image-minimal builds) or in the class. Best regards, alimon On 12/18/2015 08:34 AM, Matt Madison wrote: > This adds a third patch that fixes the problem where APT couldn't > resolve dependencies between when two packages of different architectures > both depended on an allarch package. > > Matt Madison (3): > package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping > function > package_manager.py: fixes for multilib deb packaging builds > package_deb.bbclass: add 'Multi-Arch: foreign' tag to allarch packages > > meta/classes/cross-canadian.bbclass | 2 +- > meta/classes/package_deb.bbclass | 37 +++++++++++++++++++++++++++---------- > meta/lib/oe/package_manager.py | 17 +++++++++++------ > 3 files changed, 39 insertions(+), 17 deletions(-) > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2 0/3] Debian multilib packaging fixes 2016-01-04 20:56 ` [PATCH v2 0/3] Debian multilib packaging fixes Aníbal Limón @ 2016-01-05 17:00 ` Matt Madison 2016-01-05 23:04 ` Aníbal Limón 0 siblings, 1 reply; 8+ messages in thread From: Matt Madison @ 2016-01-05 17:00 UTC (permalink / raw) To: Aníbal Limón; +Cc: openembedded-core On Mon, Jan 4, 2016 at 12:56 PM, Aníbal Limón <anibal.limon@linux.intel.com> wrote: > Hi Matt, > > Sorry for the delay in the response, i'll test again your patches with > the new one and the same error appears when build > core-image-full-cmdline or core-image-sato. No problem, I figured things would be slow around the holidays. I think the problem with this specific configuration has to do with trying to add lib32-connman to an image build that already had a dependency on connman (from packagegroup-core-x11-sato-base, for example). I modified the DpkgPM class in package-manager.py to run an 'apt-get -f install' to fix the dependencies that APT thinks are broken if the initial installation attempt fails. That appears to work around the issue adequately, and I see APT reporting ignore old unsatisfied important dependency on connman-conf:amd64 which is there because it cannot install both connman-conf and lib32-connman-conf, since PACKAGE_ARCH is ${MACHINE_ARCH} in the connman-conf recipe. For my particular application, I didn't have any conflicts between 32-bit and 64-bit packages, so the patches I submitted were enough. I'm not sure what the right answer for this should be; hacking in the additional 'apt-get -f install' works around the problem here, but I'm not sure it solves the general problem correctly. APT implements the Debian policy, and it's complicated - to do it "right" , we'd have to ensure that all of the packages get tagged with the right combination of Architecture and Multi-Arch headers, and I think we'd have to change how we write dependencies in the control files to use the '<pkg>:any' notation when that's appropriate. I'm not even sure if that could be computed automatically; it could require a significant amount of manual review. Thanks, -Matt > > MACHINE ??= "qemux86-64" > > IMAGE_INSTALL_append = " lib32-connman" > require conf/multilib.conf > MULTILIBS = "multilib:lib32" > DEFAULTTUNE_virtclass-multilib-lib32 = "x86" > > > I'm reviewing if this an error in the meta data (because > core-image-minimal builds) or in the class. > > Best regards, > alimon > > On 12/18/2015 08:34 AM, Matt Madison wrote: >> This adds a third patch that fixes the problem where APT couldn't >> resolve dependencies between when two packages of different architectures >> both depended on an allarch package. >> >> Matt Madison (3): >> package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping >> function >> package_manager.py: fixes for multilib deb packaging builds >> package_deb.bbclass: add 'Multi-Arch: foreign' tag to allarch packages >> >> meta/classes/cross-canadian.bbclass | 2 +- >> meta/classes/package_deb.bbclass | 37 +++++++++++++++++++++++++++---------- >> meta/lib/oe/package_manager.py | 17 +++++++++++------ >> 3 files changed, 39 insertions(+), 17 deletions(-) >> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2 0/3] Debian multilib packaging fixes 2016-01-05 17:00 ` Matt Madison @ 2016-01-05 23:04 ` Aníbal Limón 2016-01-06 12:17 ` Matt Madison 0 siblings, 1 reply; 8+ messages in thread From: Aníbal Limón @ 2016-01-05 23:04 UTC (permalink / raw) To: Matt Madison; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 3889 bytes --] Hi, I fixed the problem and add a 4 patch and sign-off your first tree patches also i tested your changes building images in machines: x86-64 (sato, minimal, multilib), x86(minimal), arm(minimal) and mips(minimal). Details below... Can you send the 4 patches again? alimon On 01/05/2016 11:00 AM, Matt Madison wrote: > On Mon, Jan 4, 2016 at 12:56 PM, Aníbal Limón > <anibal.limon@linux.intel.com> wrote: >> Hi Matt, >> >> Sorry for the delay in the response, i'll test again your patches with >> the new one and the same error appears when build >> core-image-full-cmdline or core-image-sato. > > No problem, I figured things would be slow around the holidays. > > I think the problem with this specific configuration has to do with > trying to add lib32-connman to an image build that already had a > dependency on connman (from packagegroup-core-x11-sato-base, for > example). I modified the DpkgPM class in package-manager.py to run an > 'apt-get -f install' to fix the dependencies that APT thinks are > broken if the initial installation attempt fails. That appears to > work around the issue adequately, and I see APT reporting I figured out the issue is related to addition of log check feature into rootfs image (that's good) but i didn't take into account DpkgPM because it assumes that can be missing dependencies and executes apt-get -f install, see the final patch attached. I'll make the changes to the AB multilib buildset for add package_deb to the CI cycle. > > ignore old unsatisfied important dependency on connman-conf:amd64 > > which is there because it cannot install both connman-conf and > lib32-connman-conf, since PACKAGE_ARCH is ${MACHINE_ARCH} in the > connman-conf recipe. > > For my particular application, I didn't have any conflicts between > 32-bit and 64-bit packages, so the patches I submitted were enough. > > I'm not sure what the right answer for this should be; hacking in the > additional 'apt-get -f install' works around the problem here, but I'm > not sure it solves the general problem correctly. APT implements the > Debian policy, and it's complicated - to do it "right" , we'd have to > ensure that all of the packages get tagged with the right combination > of Architecture and Multi-Arch headers, and I think we'd have to > change how we write dependencies in the control files to use the > '<pkg>:any' notation when that's appropriate. I'm not even sure if > that could be computed automatically; it could require a significant > amount of manual review. I agree, there are work here to do to make consistent with debian way but for now is a good start. > > Thanks, > -Matt > >> >> MACHINE ??= "qemux86-64" >> >> IMAGE_INSTALL_append = " lib32-connman" >> require conf/multilib.conf >> MULTILIBS = "multilib:lib32" >> DEFAULTTUNE_virtclass-multilib-lib32 = "x86" >> >> >> I'm reviewing if this an error in the meta data (because >> core-image-minimal builds) or in the class. >> >> Best regards, >> alimon >> >> On 12/18/2015 08:34 AM, Matt Madison wrote: >>> This adds a third patch that fixes the problem where APT couldn't >>> resolve dependencies between when two packages of different architectures >>> both depended on an allarch package. >>> >>> Matt Madison (3): >>> package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping >>> function >>> package_manager.py: fixes for multilib deb packaging builds >>> package_deb.bbclass: add 'Multi-Arch: foreign' tag to allarch packages >>> >>> meta/classes/cross-canadian.bbclass | 2 +- >>> meta/classes/package_deb.bbclass | 37 +++++++++++++++++++++++++++---------- >>> meta/lib/oe/package_manager.py | 17 +++++++++++------ >>> 3 files changed, 39 insertions(+), 17 deletions(-) >>> [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-package_deb.bbclass-cross-canadian.bbclass-DPKG_ARCH.patch --] [-- Type: text/x-diff; name="0001-package_deb.bbclass-cross-canadian.bbclass-DPKG_ARCH.patch", Size: 3985 bytes --] From 059def3b1ee80c47dec9e7614f9f5df0bccf0385 Mon Sep 17 00:00:00 2001 From: Matt Madison <matt@madison.systems> Date: Fri, 18 Dec 2015 06:34:29 -0800 Subject: [PATCH 1/4] package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping function MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Have DPKG_ARCH set by directly invoking a mapping function, rather than using an anonymous Python function modify the variable under the hood, so we can have proper handling of overrides. Also bring in some additional mappings to Debian architecture names that weren't being handled. Signed-off-by: Matt Madison <matt@madison.systems> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> --- meta/classes/cross-canadian.bbclass | 2 +- meta/classes/package_deb.bbclass | 35 +++++++++++++++++++++++++---------- 2 files changed, 26 insertions(+), 11 deletions(-) diff --git a/meta/classes/cross-canadian.bbclass b/meta/classes/cross-canadian.bbclass index 7b26ed6..e07b1bd 100644 --- a/meta/classes/cross-canadian.bbclass +++ b/meta/classes/cross-canadian.bbclass @@ -103,7 +103,7 @@ HOST_LD_ARCH = "${SDK_LD_ARCH}" HOST_AS_ARCH = "${SDK_AS_ARCH}" #assign DPKG_ARCH -DPKG_ARCH = "${SDK_ARCH}" +DPKG_ARCH = "${@debian_arch_map(d.getVar('SDK_ARCH', True), '')}" CPPFLAGS = "${BUILDSDK_CPPFLAGS}" CFLAGS = "${BUILDSDK_CFLAGS}" diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass index 5feeeb0..8d27adf 100644 --- a/meta/classes/package_deb.bbclass +++ b/meta/classes/package_deb.bbclass @@ -6,7 +6,7 @@ inherit package IMAGE_PKGTYPE ?= "deb" -DPKG_ARCH ?= "${TARGET_ARCH}" +DPKG_ARCH ?= "${@debian_arch_map(d.getVar('TARGET_ARCH', True), d.getVar('TUNE_FEATURES', True))}" PKGWRITEDIRDEB = "${WORKDIR}/deploy-debs" @@ -14,6 +14,28 @@ APTCONF_TARGET = "${WORKDIR}" APT_ARGS = "${@['', '--no-install-recommends'][d.getVar("NO_RECOMMENDATIONS", True) == "1"]}" +def debian_arch_map(arch, tune): + tune_features = tune.split() + if arch in ["i586", "i686"]: + return "i386" + if arch == "x86_64": + if "mx32" in tune_features: + return "x32" + return "amd64" + if arch.startswith("mips"): + endian = ["el", ""]["bigendian" in tune_features] + if "n64" in tune_features: + return "mips64" + endian + if "n32" in tune_features: + return "mipsn32" + endian + return "mips" + endian + if arch == "powerpc": + return arch + ["", "spe"]["spe" in tune_features] + if arch == "aarch64": + return "arm64" + if arch == "arm": + return arch + ["el", "hf"]["callconvention-hard" in tune_features] + return arch # # install a bunch of packages using apt # the following shell variables needs to be set before calling this func: @@ -288,6 +310,8 @@ python do_package_deb () { cleanupcontrol(root) bb.utils.unlockfile(lf) } +# Indirect references to these vars +do_package_write_deb[vardeps] += "PKGV PKGR PKGV DESCRIPTION SECTION PRIORITY MAINTAINER DPKG_ARCH PN HOMEPAGE" # Otherwise allarch packages may change depending on override configuration do_package_deb[vardepsexclude] = "OVERRIDES" @@ -311,15 +335,6 @@ python () { deps = ' dpkg-native:do_populate_sysroot virtual/fakeroot-native:do_populate_sysroot' d.appendVarFlag('do_package_write_deb', 'depends', deps) d.setVarFlag('do_package_write_deb', 'fakeroot', "1") - - # Map TARGET_ARCH to Debian's ideas about architectures - darch = d.getVar('DPKG_ARCH', True) - if darch in ["x86", "i486", "i586", "i686", "pentium"]: - d.setVar('DPKG_ARCH', 'i386') - elif darch == "x86_64": - d.setVar('DPKG_ARCH', 'amd64') - elif darch == "arm": - d.setVar('DPKG_ARCH', 'armel') } python do_package_write_deb () { -- 2.1.4 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: 0002-package_manager.py-fixes-for-multilib-deb-packaging-.patch --] [-- Type: text/x-diff; name="0002-package_manager.py-fixes-for-multilib-deb-packaging-.patch", Size: 3590 bytes --] From e526b852a185bcdde6484eb3a88ff8ea7c08e24d Mon Sep 17 00:00:00 2001 From: Matt Madison <matt@madison.systems> Date: Fri, 18 Dec 2015 06:34:30 -0800 Subject: [PATCH 2/4] package_manager.py: fixes for multilib deb packaging builds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * tmp/deploy/deb subdirectories do not get hyphens replaced with underscores, so don't do that translation when building the sources list. * Fix MULTILIB_VARIANTS handling to be more general and work for all architectures * Also include a fix for a warning generated by apt due to missing apt/preferences.d directory. Signed-off-by: Matt Madison <matt@madison.systems> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> --- meta/lib/oe/package_manager.py | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py index 32afeaf..bc39721 100644 --- a/meta/lib/oe/package_manager.py +++ b/meta/lib/oe/package_manager.py @@ -247,7 +247,7 @@ class DpkgIndexer(Indexer): if a not in pkg_archs: arch_list.append(a) - all_mlb_pkg_arch_list = (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or "").replace('-', '_').split() + all_mlb_pkg_arch_list = (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or "").split() arch_list.extend(arch for arch in all_mlb_pkg_arch_list if arch not in arch_list) apt_ftparchive = bb.utils.which(os.getenv('PATH'), "apt-ftparchive") @@ -1775,7 +1775,7 @@ class DpkgPM(PackageManager): self.apt_args = d.getVar("APT_ARGS", True) self.all_arch_list = archs.split() - all_mlb_pkg_arch_list = (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or "").replace('-', '_').split() + all_mlb_pkg_arch_list = (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or "").split() self.all_arch_list.extend(arch for arch in all_mlb_pkg_arch_list if arch not in self.all_arch_list) self._create_configs(archs, base_archs) @@ -1966,6 +1966,7 @@ class DpkgPM(PackageManager): bb.utils.mkdirhier(self.apt_conf_dir) bb.utils.mkdirhier(self.apt_conf_dir + "/lists/partial/") bb.utils.mkdirhier(self.apt_conf_dir + "/apt.conf.d/") + bb.utils.mkdirhier(self.apt_conf_dir + "/preferences.d/") arch_list = [] for arch in self.all_arch_list: @@ -2000,10 +2001,14 @@ class DpkgPM(PackageManager): base_arch_list = base_archs.split() multilib_variants = self.d.getVar("MULTILIB_VARIANTS", True); for variant in multilib_variants.split(): - if variant == "lib32": - base_arch_list.append("i386") - elif variant == "lib64": - base_arch_list.append("amd64") + localdata = bb.data.createCopy(self.d) + variant_tune = localdata.getVar("DEFAULTTUNE_virtclass-multilib-" + variant, False) + orig_arch = localdata.getVar("DPKG_ARCH", True) + localdata.setVar("DEFAULTTUNE", variant_tune) + bb.data.update_data(localdata) + variant_arch = localdata.getVar("DPKG_ARCH", True) + if variant_arch not in base_arch_list: + base_arch_list.append(variant_arch) with open(self.apt_conf_file, "w+") as apt_conf: with open(self.d.expand("${STAGING_ETCDIR_NATIVE}/apt/apt.conf.sample")) as apt_conf_sample: -- 2.1.4 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #4: 0003-package_deb.bbclass-add-Multi-Arch-foreign-tag-to-al.patch --] [-- Type: text/x-diff; name="0003-package_deb.bbclass-add-Multi-Arch-foreign-tag-to-al.patch", Size: 1201 bytes --] From 22a2a8425982403bed9cbde3e1d43db450b071da Mon Sep 17 00:00:00 2001 From: Matt Madison <matt@madison.systems> Date: Fri, 18 Dec 2015 06:34:31 -0800 Subject: [PATCH 3/4] package_deb.bbclass: add 'Multi-Arch: foreign' tag to allarch packages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This tells APT that it can use such packages to resolve dependencies from packages of any architecture in a multilib build. Signed-off-by: Matt Madison <matt@madison.systems> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> --- meta/classes/package_deb.bbclass | 2 ++ 1 file changed, 2 insertions(+) diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass index 8d27adf..25218d0 100644 --- a/meta/classes/package_deb.bbclass +++ b/meta/classes/package_deb.bbclass @@ -161,6 +161,8 @@ python do_package_deb () { return l2 ctrlfile.write("Package: %s\n" % pkgname) + if d.getVar('PACKAGE_ARCH', True) == "all": + ctrlfile.write("Multi-Arch: foreign\n") # check for required fields try: for (c, fs) in fields: -- 2.1.4 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #5: 0004-DpkgRootfs-Fix-logcheck_error-false-positive-when-us.patch --] [-- Type: text/x-diff; name="0004-DpkgRootfs-Fix-logcheck_error-false-positive-when-us.patch", Size: 2227 bytes --] From b467e9813fd5ce414e3f4a449f02a0208dc408e4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?An=C3=ADbal=20Lim=C3=B3n?= <anibal.limon@linux.intel.com> Date: Tue, 5 Jan 2016 11:38:42 -0600 Subject: [PATCH 4/4] DpkgRootfs: Fix logcheck_error false-positive when use multilib MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rootfs with dpkg was failing due to false-positive in logcheck_error because current logic of DpkgPM handles missing dependencies failure using apt-get -f install [1][2]. This support was broken due to addition of logcheck and don't take into account dpkgpm cases, in order to fix add an attr for specify expected errors regex'es by package manager. [1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oe/rootfs.py#n659 [2] http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oe/package_manager.py#n2038 Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> --- meta/lib/oe/rootfs.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py index a2af332..f677d03 100644 --- a/meta/lib/oe/rootfs.py +++ b/meta/lib/oe/rootfs.py @@ -63,6 +63,15 @@ class Rootfs(object): if 'log_check' in line: continue + if hasattr(self, 'log_check_expected_errors_regexes'): + m = None + for ee in self.log_check_expected_errors_regexes: + m = re.search(ee, line) + if m: + break + if m: + continue + m = r.search(line) if m: found_error = 1 @@ -623,6 +632,10 @@ class DpkgRootfs(DpkgOpkgRootfs): def __init__(self, d, manifest_dir): super(DpkgRootfs, self).__init__(d) self.log_check_regex = '^E:' + self.log_check_expected_errors_regexes = \ + [ + "^E: Unmet dependencies." + ] bb.utils.remove(self.image_rootfs, True) bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) -- 2.1.4 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH v2 0/3] Debian multilib packaging fixes 2016-01-05 23:04 ` Aníbal Limón @ 2016-01-06 12:17 ` Matt Madison 0 siblings, 0 replies; 8+ messages in thread From: Matt Madison @ 2016-01-06 12:17 UTC (permalink / raw) To: Aníbal Limón; +Cc: openembedded-core On Tue, Jan 5, 2016 at 3:04 PM, Aníbal Limón <anibal.limon@linux.intel.com> wrote: > Hi, > > I fixed the problem and add a 4 patch and sign-off your first tree > patches also i tested your changes building images in machines: x86-64 > (sato, minimal, multilib), x86(minimal), arm(minimal) and mips(minimal). > Details below... > > Can you send the 4 patches again? Thanks, will send out V3 shortly. -Matt > > alimon > > On 01/05/2016 11:00 AM, Matt Madison wrote: >> On Mon, Jan 4, 2016 at 12:56 PM, Aníbal Limón >> <anibal.limon@linux.intel.com> wrote: >>> Hi Matt, >>> >>> Sorry for the delay in the response, i'll test again your patches with >>> the new one and the same error appears when build >>> core-image-full-cmdline or core-image-sato. >> >> No problem, I figured things would be slow around the holidays. >> >> I think the problem with this specific configuration has to do with >> trying to add lib32-connman to an image build that already had a >> dependency on connman (from packagegroup-core-x11-sato-base, for >> example). I modified the DpkgPM class in package-manager.py to run an >> 'apt-get -f install' to fix the dependencies that APT thinks are >> broken if the initial installation attempt fails. That appears to >> work around the issue adequately, and I see APT reporting > > > I figured out the issue is related to addition of log check feature into > rootfs image (that's good) but i didn't take into account DpkgPM because > it assumes that can be missing dependencies and executes apt-get -f > install, see the final patch attached. > > I'll make the changes to the AB multilib buildset for add package_deb to > the CI cycle. Great, that will help. > > >> >> ignore old unsatisfied important dependency on connman-conf:amd64 >> >> which is there because it cannot install both connman-conf and >> lib32-connman-conf, since PACKAGE_ARCH is ${MACHINE_ARCH} in the >> connman-conf recipe. >> >> For my particular application, I didn't have any conflicts between >> 32-bit and 64-bit packages, so the patches I submitted were enough. >> >> I'm not sure what the right answer for this should be; hacking in the >> additional 'apt-get -f install' works around the problem here, but I'm >> not sure it solves the general problem correctly. APT implements the >> Debian policy, and it's complicated - to do it "right" , we'd have to >> ensure that all of the packages get tagged with the right combination >> of Architecture and Multi-Arch headers, and I think we'd have to >> change how we write dependencies in the control files to use the >> '<pkg>:any' notation when that's appropriate. I'm not even sure if >> that could be computed automatically; it could require a significant >> amount of manual review. > > I agree, there are work here to do to make consistent with debian way > but for now is a good start. > >> >> Thanks, >> -Matt >> >>> >>> MACHINE ??= "qemux86-64" >>> >>> IMAGE_INSTALL_append = " lib32-connman" >>> require conf/multilib.conf >>> MULTILIBS = "multilib:lib32" >>> DEFAULTTUNE_virtclass-multilib-lib32 = "x86" >>> >>> >>> I'm reviewing if this an error in the meta data (because >>> core-image-minimal builds) or in the class. >>> >>> Best regards, >>> alimon >>> >>> On 12/18/2015 08:34 AM, Matt Madison wrote: >>>> This adds a third patch that fixes the problem where APT couldn't >>>> resolve dependencies between when two packages of different architectures >>>> both depended on an allarch package. >>>> >>>> Matt Madison (3): >>>> package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping >>>> function >>>> package_manager.py: fixes for multilib deb packaging builds >>>> package_deb.bbclass: add 'Multi-Arch: foreign' tag to allarch packages >>>> >>>> meta/classes/cross-canadian.bbclass | 2 +- >>>> meta/classes/package_deb.bbclass | 37 +++++++++++++++++++++++++++---------- >>>> meta/lib/oe/package_manager.py | 17 +++++++++++------ >>>> 3 files changed, 39 insertions(+), 17 deletions(-) >>>> ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-01-06 12:17 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-12-18 14:34 [PATCH v2 0/3] Debian multilib packaging fixes Matt Madison 2015-12-18 14:34 ` [PATCH v2 1/3] package_deb.bbclass, cross-canadian.bbclass: DPKG_ARCH mapping function Matt Madison 2015-12-18 14:34 ` [PATCH v2 2/3] package_manager.py: fixes for multilib deb packaging builds Matt Madison 2015-12-18 14:34 ` [PATCH v2 3/3] package_deb.bbclass: add 'Multi-Arch: foreign' tag to allarch packages Matt Madison 2016-01-04 20:56 ` [PATCH v2 0/3] Debian multilib packaging fixes Aníbal Limón 2016-01-05 17:00 ` Matt Madison 2016-01-05 23:04 ` Aníbal Limón 2016-01-06 12:17 ` Matt Madison
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.