[PATCH 0/4] insane.bbclass: add QA check: package-missing

[PATCH 0/4] insane.bbclass: add QA check: package-missing

[Hongxu Jia]

We also tried to fix perl, linux-libc-headers and busybox according to
the result of QA checking.

Here is a test to explain what the QA check showing, we could figure out 
package missing issue at package generation time rather than do_rootfs
time.

Test Steps:

1. Edit xz and gzip recipe, to simulate a package missing situation (the manually added
rdepends is not real but):
..............
diff --git a/meta/recipes-extended/gzip/gzip.inc b/meta/recipes-extended/gzip/gzip.inc
index 671b1eb..7e643e5 100644
--- a/meta/recipes-extended/gzip/gzip.inc
+++ b/meta/recipes-extended/gzip/gzip.inc
@@ -32,3 +32,8 @@ ALTERNATIVE_LINK_NAME[gzip] = "${base_bindir}/gzip"
 ALTERNATIVE_LINK_NAME[zcat] = "${base_bindir}/zcat"
 
 export CONFIG_SHELL="/bin/sh"
+
+PACKAGES =+ "${PN}-missing1 ${PN}-missing2"
+FILES_${PN}-missing1 = ""
+FILES_${PN}-missing2 = ""
+RPROVIDES_${PN}-missing1 = "${PN}-missing"
diff --git a/meta/recipes-extended/xz/xz_5.1.3alpha.bb b/meta/recipes-extended/xz/xz_5.1.3alpha.bb
index 168e383..c5d8b57 100644
--- a/meta/recipes-extended/xz/xz_5.1.3alpha.bb
+++ b/meta/recipes-extended/xz/xz_5.1.3alpha.bb
@@ -37,6 +37,9 @@ FILES_liblzma-dev = "${includedir}/lzma* ${libdir}/liblzma*${SOLIBSDEV} ${libdir
 FILES_liblzma-staticdev = "${libdir}/liblzma.a"
 FILES_liblzma-dbg = "${libdir}/.debug/liblzma*"
 
+
+RDEPENDS_${PN}_class-target += "gzip-missing gzip-missing2"
+
 BBCLASSEXTEND = "native nativesdk"
 
 export CONFIG_SHELL="/bin/sh"
..............

2. Vim local.conf
...
IMAGE_INSTALL_append = " xz"
WARN_QA_append = " package-missing"
...

3. bitbake core-image-minimal
..............
|WARNING: QA Issue: package gzip-missing1 not generated, but listed in PACKAGES
|Compute rdepends chains, '-->' means 'runtime depends'
|("gzip-missing1" rprovides "gzip-missing")
|"xz" -> "gzip-missing"
| [package-missing]
|WARNING: QA Issue: package gzip-missing2 not generated, but listed in PACKAGES
|Compute rdepends chains, '-->' means 'runtime depends'
|"xz" -> "gzip-missing2"
| [package-missing]
|ERROR: Unable to install packages. Command '/home/jiahongxu/yocto/build-2014120
2-mips/tmp/sysroots/x86_64-linux/usr/bin/smart --quiet --data-dir=/home/jiahongx
u/yocto/build-20141202-mips/tmp/work/qemumips-poky-linux/core-image-minimal/1.0
-r0/rootfs/var/lib/smart install -y xz@mips32r2 run-postinsts@all packagegroup-
core-boot@qemumips' returned 1:
error: Can't install xz-5.1.3alpha-r0@mips32r2: no package provides gzip-missing


|ERROR: Function failed: do_rootfs
|ERROR: Logfile of failure stored in: /home/jiahongxu/yocto/build-20141202-mips/
tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.24338
|ERROR: Task 7 (/home/jiahongxu/yocto/poky/meta/recipes-core/images/core-image-
minimal.bb, do_rootfs) failed with exit code '1'
..............

//Hongxu

The following changes since commit ec6377bcf52d105cd23ac6bbbeddd38fee9337e4:

  bitbake: bitbake-user-manual-metadata.xml: Updated do_package_write example (2014-12-09 22:25:36 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/package-missing
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/package-missing

Hongxu Jia (4):
  insane.bbclass: add QA check: package-missing
  perl_5.20.0: fix QA package-missing warnings
  linux-libc-headers.inc: fix QA package-missing warnings
  busybox.inc: fix QA package-missing warnings

 meta/classes/insane.bbclass                        | 93 +++++++++++++++++++++-
 meta/recipes-core/busybox/busybox.inc              | 10 +++
 meta/recipes-devtools/perl/perl_5.20.0.bb          | 11 +--
 .../linux-libc-headers/linux-libc-headers.inc      |  2 +
 4 files changed, 110 insertions(+), 6 deletions(-)

-- 
1.9.1

[PATCH 1/4] insane.bbclass: add QA check: package-missing

[Hongxu Jia]

During packaging, check if there are any dependencies (RDEPENDS)
on packages that have ended up empty and not produced; and if so
produce an warn/error as soon as possible, instead of allowing
the build to proceed up to do_rootfs and then the package manager
reporting the package as missing.

At the moment, we use bb.persist_data as a global database to
collect all available packages rdepends and rprovides. So in the
do_package_qa task, while package not produced, we could compute
rdepends chain, and report warn/error immediately.

[YOCTO #5531]
[YOCTO #6420]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/insane.bbclass | 93 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 92 insertions(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 0b45374..a0eff37 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -29,7 +29,7 @@ QA_SANE = "True"
 WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
             textrel already-stripped incompatible-license files-invalid \
             installed-vs-shipped compile-host-path install-host-path \
-            pn-overrides infodir build-deps file-rdeps \
+            pn-overrides infodir build-deps file-rdeps package-missing \
             "
 ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
@@ -868,6 +868,15 @@ def package_qa_check_rdepends(pkg, pkgdest, skip, taskdeps, packages, d):
                             (pkg, ', '.join(str(e) for e in filerdepends))
                 sane = package_qa_handle_error("file-rdeps", error_msg, d)
 
+        if "package-missing" not in skip and \
+               not pkg.endswith("-dev") and not pkg.endswith("-staticdev") and \
+               not pkg.endswith("-locale") and not pkg.endswith("-dbg") and \
+               not pkg.endswith("-doc"):
+            if not oe.packagedata.packaged(pkg, d):
+                error_msg = "package %s not generated, but listed in PACKAGES\n" % pkg
+                error_msg += compute_rdepends_chain(pkg, d)
+                sane = package_qa_handle_error("package-missing", error_msg, d)
+
     return sane
 
 def package_qa_check_deps(pkg, pkgdest, skip, d):
@@ -1152,3 +1161,85 @@ python () {
     for i in issues:
         package_qa_handle_error("pkgvarcheck", "%s: Variable %s is set as not being package specific, please fix this." % (d.getVar("FILE", True), i), d)
 }
+
+# Add a handler to collect all available packages rdepends and rprovides
+# to a global database (bb.persist_data). So the recipe could find out
+# which others rdepends on it. We filter out "-dev, -staticdev, -locale,
+# -dbg, -doc" packages.
+# Such as:
+# RDEPENDS_pkg1 = "pkg2 pkg3"
+# RDEPENDS_pkg4 = "pkg3 pkg5"
+# equals:
+# rdeps_dict = {'pkg2': 'pkg1', 'pkg3': 'pkg1 pkg4', 'pkg5':'pkg4'}
+#
+# RPROVIDES_pkg6 = "pkg2 pkg7"
+# RPROVIDES_pkg7 = "pkg5"
+# equals:
+# rprovides_dict = {'pkg6': 'pkg2 pkg7', 'pkg7': 'pkg5'}
+python collect_package_rdepends_handler () {
+    d = e.data
+    pkgs = []
+    for pkg in (d.getVar('PACKAGES', True) or '').split():
+        if pkg.endswith("-dev") or pkg.endswith("-staticdev") or \
+           pkg.endswith("-locale") or pkg.endswith("-dbg") or \
+           pkg.endswith("-doc"):
+            continue
+        pkgs.append(pkg)
+
+    if not pkgs:
+        return
+
+    rdeps_dict = bb.persist_data.persist('BB_RDEPENDED_CHAIN', d)
+    for pkg in pkgs:
+        rdeps = (d.getVar('RDEPENDS_%s' % pkg, True) or "").split()
+        for rdep in rdeps:
+            # Filter out pkg which rdepends itself
+            if pkg == rdep:
+                continue
+
+            if rdep in rdeps_dict and pkg not in rdeps_dict[rdep].split():
+                rdeps_dict[rdep] += ' %s' % pkg
+            elif rdep not in rdeps_dict:
+                rdeps_dict[rdep] = '%s' % pkg
+
+    rprovides_dict = bb.persist_data.persist('BB_RPROVIDES_CHAIN', d)
+    for pkg in pkgs:
+        rprovides = (d.getVar('RPROVIDES_%s' % pkg, True) or "").split()
+        if rprovides:
+            rprovides_dict[pkg] = ' '.join(set(rprovides))
+}
+addhandler collect_package_rdepends_handler
+collect_package_rdepends_handler[eventmask] = "bb.event.RecipeParsed"
+
+def compute_rdepends_chain(pkg, d):
+    rdeps_dict = bb.persist_data.persist('BB_RDEPENDED_CHAIN', d)
+    rprovides_dict = bb.persist_data.persist('BB_RPROVIDES_CHAIN', d)
+
+    def get_parents(pkg):
+        parents = []
+        message = ""
+
+        if pkg not in rdeps_dict and pkg in rprovides_dict:
+            for rprovide in rprovides_dict[pkg].split():
+                # Use rprovide to instead of pkg
+                if rprovide in rdeps_dict:
+                    message = '("%s" rprovides "%s")\n' % (pkg, rprovide)
+                    pkg = rprovide
+                    break
+
+        if pkg in rdeps_dict:
+            parents = rdeps_dict[pkg].split()
+            for parent in parents:
+                message += '"%s" -> "%s"\n' % (parent, pkg)
+
+            for parent in parents:
+                message += get_parents(parent)
+
+        return message
+
+    message = get_parents(pkg)
+    if message:
+        message = "Compute rdepends chains, '-->' means 'runtime depends'\n" \
+                  + message
+    return message
+
-- 
1.9.1

[PATCH 2/4] perl_5.20.0: fix QA package-missing warnings

[Hongxu Jia]

While building perl with QA package-missing checking enabled, there was warning:
...
WARNING: QA Issue: package perl-module-cpanplus not generated, but listed in PACKAGES
WARNING: QA Issue: package perl-module-config not generated, but listed in PACKAGES
WARNING: QA Issue: package perl-module-config-heavy not generated, but listed in PACKAGES
WARNING: QA Issue: package perl-module-strict not generated, but listed in PACKAGES
WARNING: QA Issue: package perl-module-vars not generated, but listed in PACKAGES
WARNING: QA Issue: package perl-module-warnings not generated, but listed in PACKAGES
WARNING: QA Issue: package perl-module-warnings-register not generated, but listed in PACKAGES
...

Drop perl-module-cpanplus, from git://perl5.git.perl.org/perl.git:
...
commit fb598ba5e55920eb59105c932df653f4fea6966c
Author: Chris 'BinGOs' Williams <chris@bingosnet.co.uk>
Date:   Sat May 18 01:01:42 2013 +0100

    Remove cpan/CPANPLUS and associated utilities
...

Drop perl-module-config from PACKAGES_DYNAMIC:
Config.pm was listed in "FILES_${PN}", we should filter it out from
do_split_packages.

Drop perl-module-config-heavy from PACKAGES_DYNAMIC:
Config_heavy.pl was listed in "FILES_${PN}-lib", we should filter it out from
do_split_packages.

Drop perl-module-strict from PACKAGES_DYNAMIC:
strict.pm was listed in "FILES_${PN}", we should filter it out from
do_split_packages.

Drop perl-module-vars from PACKAGES_DYNAMIC:
vars.pm was listed in "FILES_${PN}", we should filter it out from
do_split_packages.

Drop perl-module-warnings from PACKAGES_DYNAMIC:
warnings.pm was listed in "FILES_${PN}", we should filter it out from
do_split_packages.

Drop perl-module-warnings-register from PACKAGES_DYNAMIC:
Dir warnings was listed in "FILES_${PN}", we should filter it out from
do_split_packages.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-devtools/perl/perl_5.20.0.bb | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/meta/recipes-devtools/perl/perl_5.20.0.bb b/meta/recipes-devtools/perl/perl_5.20.0.bb
index 3ca0f53..c7f21a9 100644
--- a/meta/recipes-devtools/perl/perl_5.20.0.bb
+++ b/meta/recipes-devtools/perl/perl_5.20.0.bb
@@ -272,7 +272,7 @@ perl_package_preprocess () {
 }
 
 PACKAGES = "perl-dbg perl perl-misc perl-dev perl-pod perl-doc perl-lib \
-            perl-module-cpan perl-module-cpanplus perl-module-unicore"
+            perl-module-cpan perl-module-unicore"
 FILES_${PN} = "${bindir}/perl ${bindir}/perl${PV} \
                ${libdir}/perl/${PV}/Config.pm \
                ${libdir}/perl/${PV}/strict.pm \
@@ -311,7 +311,6 @@ FILES_${PN}-doc = "${libdir}/perl/${PV}/*/*.txt \
                    ${libdir}/perl/${PV}/CGI/eg \
                    ${libdir}/perl/${PV}/CPAN/PAUSE2003.pub \
                    ${libdir}/perl/${PV}/CPAN/SIGNATURE \
-		   ${libdir}/perl/${PV}/CPANPLUS/Shell/Default/Plugins/HOWTO.pod \
                    ${libdir}/perl/${PV}/Encode/encode.h \
                    ${libdir}/perl/${PV}/ExtUtils/MANIFEST.SKIP \
                    ${libdir}/perl/${PV}/ExtUtils/NOTES \
@@ -326,8 +325,6 @@ FILES_${PN}-doc = "${libdir}/perl/${PV}/*/*.txt \
 
 FILES_perl-module-cpan += "${libdir}/perl/${PV}/CPAN \
                            ${libdir}/perl/${PV}/CPAN.pm"
-FILES_perl-module-cpanplus += "${libdir}/perl/${PV}/CPANPLUS \
-                               ${libdir}/perl/${PV}/CPANPLUS.pm"
 FILES_perl-module-unicore += "${libdir}/perl/${PV}/unicore"
 
 # Create a perl-modules package recommending all the other perl
@@ -340,7 +337,11 @@ python populate_packages_prepend () {
     do_split_packages(d, libdir, 'auto/([^.]*)/[^/]*\.(so|ld|ix|al)', 'perl-module-%s', 'perl module %s', recursive=True, match_path=True, prepend=False)
     do_split_packages(d, libdir, 'Module/([^\/]*)\.pm', 'perl-module-%s', 'perl module %s', recursive=True, allow_dirs=False, match_path=True, prepend=False)
     do_split_packages(d, libdir, 'Module/([^\/]*)/.*', 'perl-module-%s', 'perl module %s', recursive=True, allow_dirs=False, match_path=True, prepend=False)
-    do_split_packages(d, libdir, '(^(?!(CPAN\/|CPANPLUS\/|Module\/|unicore\/|auto\/)[^\/]).*)\.(pm|pl|e2x)', 'perl-module-%s', 'perl module %s', recursive=True, allow_dirs=False, match_path=True, prepend=False)
+
+    unmatch_dirs = '(CPAN\/|Module\/|unicore\/|auto\/|warnings\/)[^\/]'
+    unmatch_files = 'vars.pm$|Config.pm$|strict.pm$|Config_heavy.pl$|warnings.pm$'
+    file_regex = '(^(?!(%s|%s)).*)\.(pm|pl|e2x)' % (unmatch_dirs, unmatch_files)
+    do_split_packages(d, libdir, file_regex, 'perl-module-%s', 'perl module %s', recursive=True, allow_dirs=False, match_path=True, prepend=False)
 
     # perl-modules should recommend every perl module, and only the
     # modules. Don't attempt to use the result of do_split_packages() as some
-- 
1.9.1

[PATCH 3/4] linux-libc-headers.inc: fix QA package-missing warnings

[Hongxu Jia]

While building linux-libc-headers with QA package-missing checking enabled,
there was warning:
...
QA Issue: package linux-libc-headers not generated, but listed in PACKAGES
...

Only linux-libc-headers-dev was generated, the others are empty, we should
set PACKAGE manually.

We also set linux-libc-headers-dev as linux-libc-headers's provider, so
package management could install linux-libc-headers.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc b/meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc
index b18d09f..a69bbe6 100644
--- a/meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc
+++ b/meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc
@@ -71,3 +71,5 @@ RRECOMMENDS_${PN}-dbg = "${PN}-dev (= ${EXTENDPKGV})"
 
 INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS += "unifdef-native"
+PACKAGES="${PN}-dev"
+RPROVIDES_${PN}-dev = "${PN}"
-- 
1.9.1

[PATCH 4/4] busybox.inc: fix QA package-missing warnings

[Hongxu Jia]

While building busybox with QA package-missing checking enabled,
there was warning:
...
WARNING: QA Issue: package busybox-mdev not generated, but listed in PACKAGES
WARNING: QA Issue: package busybox-httpd not generated, but listed in PACKAGES
...

The config in ${B}/.config decides to generate busybox-mdev,
busybox-httpd or not. Files are copied if the packages generated.
So set PACKAGES in do_package according to the existance of files.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-core/busybox/busybox.inc | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc
index bd66e4f..23115ae 100644
--- a/meta/recipes-core/busybox/busybox.inc
+++ b/meta/recipes-core/busybox/busybox.inc
@@ -314,6 +314,16 @@ python () {
 }
 
 python do_package_prepend () {
+    if not os.path.exists(d.expand("${D}${sysconfdir}/init.d/busybox-httpd")):
+        packages = (d.getVar('PACKAGES', True) or '').split()
+        packages.remove(d.expand('${PN}-httpd'))
+        d.setVar('PACKAGES', ' '.join(packages))
+
+    if not os.path.exists(d.expand("${D}${sysconfdir}/init.d/mdev")):
+        packages = (d.getVar('PACKAGES', True) or '').split()
+        packages.remove(d.expand('${PN}-mdev'))
+        d.setVar('PACKAGES', ' '.join(packages))
+
     # We need to load the full set of busybox provides from the /etc/busybox.links
     # Use this to see the update-alternatives with the right information
 
-- 
1.9.1

Re: [PATCH 0/4] insane.bbclass: add QA check: package-missing

[Hongxu Jia]

Ping

//Hongxu

On 12/11/2014 06:27 PM, Hongxu Jia wrote:
> We also tried to fix perl, linux-libc-headers and busybox according to
> the result of QA checking.
>
> Here is a test to explain what the QA check showing, we could figure out
> package missing issue at package generation time rather than do_rootfs
> time.
>
> Test Steps:
>
> 1. Edit xz and gzip recipe, to simulate a package missing situation (the manually added
> rdepends is not real but):
> ..............
> diff --git a/meta/recipes-extended/gzip/gzip.inc b/meta/recipes-extended/gzip/gzip.inc
> index 671b1eb..7e643e5 100644
> --- a/meta/recipes-extended/gzip/gzip.inc
> +++ b/meta/recipes-extended/gzip/gzip.inc
> @@ -32,3 +32,8 @@ ALTERNATIVE_LINK_NAME[gzip] = "${base_bindir}/gzip"
>   ALTERNATIVE_LINK_NAME[zcat] = "${base_bindir}/zcat"
>   
>   export CONFIG_SHELL="/bin/sh"
> +
> +PACKAGES =+ "${PN}-missing1 ${PN}-missing2"
> +FILES_${PN}-missing1 = ""
> +FILES_${PN}-missing2 = ""
> +RPROVIDES_${PN}-missing1 = "${PN}-missing"
> diff --git a/meta/recipes-extended/xz/xz_5.1.3alpha.bb b/meta/recipes-extended/xz/xz_5.1.3alpha.bb
> index 168e383..c5d8b57 100644
> --- a/meta/recipes-extended/xz/xz_5.1.3alpha.bb
> +++ b/meta/recipes-extended/xz/xz_5.1.3alpha.bb
> @@ -37,6 +37,9 @@ FILES_liblzma-dev = "${includedir}/lzma* ${libdir}/liblzma*${SOLIBSDEV} ${libdir
>   FILES_liblzma-staticdev = "${libdir}/liblzma.a"
>   FILES_liblzma-dbg = "${libdir}/.debug/liblzma*"
>   
> +
> +RDEPENDS_${PN}_class-target += "gzip-missing gzip-missing2"
> +
>   BBCLASSEXTEND = "native nativesdk"
>   
>   export CONFIG_SHELL="/bin/sh"
> ..............
>
> 2. Vim local.conf
> ...
> IMAGE_INSTALL_append = " xz"
> WARN_QA_append = " package-missing"
> ...
>
> 3. bitbake core-image-minimal
> ..............
> |WARNING: QA Issue: package gzip-missing1 not generated, but listed in PACKAGES
> |Compute rdepends chains, '-->' means 'runtime depends'
> |("gzip-missing1" rprovides "gzip-missing")
> |"xz" -> "gzip-missing"
> | [package-missing]
> |WARNING: QA Issue: package gzip-missing2 not generated, but listed in PACKAGES
> |Compute rdepends chains, '-->' means 'runtime depends'
> |"xz" -> "gzip-missing2"
> | [package-missing]
> |ERROR: Unable to install packages. Command '/home/jiahongxu/yocto/build-2014120
> 2-mips/tmp/sysroots/x86_64-linux/usr/bin/smart --quiet --data-dir=/home/jiahongx
> u/yocto/build-20141202-mips/tmp/work/qemumips-poky-linux/core-image-minimal/1.0
> -r0/rootfs/var/lib/smart install -y xz@mips32r2 run-postinsts@all packagegroup-
> core-boot@qemumips' returned 1:
> error: Can't install xz-5.1.3alpha-r0@mips32r2: no package provides gzip-missing
>
>
> |ERROR: Function failed: do_rootfs
> |ERROR: Logfile of failure stored in: /home/jiahongxu/yocto/build-20141202-mips/
> tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.24338
> |ERROR: Task 7 (/home/jiahongxu/yocto/poky/meta/recipes-core/images/core-image-
> minimal.bb, do_rootfs) failed with exit code '1'
> ..............
>
> //Hongxu
>
> The following changes since commit ec6377bcf52d105cd23ac6bbbeddd38fee9337e4:
>
>    bitbake: bitbake-user-manual-metadata.xml: Updated do_package_write example (2014-12-09 22:25:36 +0000)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib hongxu/package-missing
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/package-missing
>
> Hongxu Jia (4):
>    insane.bbclass: add QA check: package-missing
>    perl_5.20.0: fix QA package-missing warnings
>    linux-libc-headers.inc: fix QA package-missing warnings
>    busybox.inc: fix QA package-missing warnings
>
>   meta/classes/insane.bbclass                        | 93 +++++++++++++++++++++-
>   meta/recipes-core/busybox/busybox.inc              | 10 +++
>   meta/recipes-devtools/perl/perl_5.20.0.bb          | 11 +--
>   .../linux-libc-headers/linux-libc-headers.inc      |  2 +
>   4 files changed, 110 insertions(+), 6 deletions(-)
>

Re: [PATCH 1/4] insane.bbclass: add QA check: package-missing

[Richard Purdie]

On Thu, 2014-12-11 at 18:27 +0800, Hongxu Jia wrote:
> During packaging, check if there are any dependencies (RDEPENDS)
> on packages that have ended up empty and not produced; and if so
> produce an warn/error as soon as possible, instead of allowing
> the build to proceed up to do_rootfs and then the package manager
> reporting the package as missing.
> 
> At the moment, we use bb.persist_data as a global database to
> collect all available packages rdepends and rprovides. So in the
> do_package_qa task, while package not produced, we could compute
> rdepends chain, and report warn/error immediately.

I liked this until I saw you were using bb.persist_data. That isn't
under sstate control and will not result in deterministic builds.

Have you looked at the data saved by do_packageinfo into the pkgdata
structures? Can we not use the information on RDEPENDS and RPROVIDES
there instead?

Cheers,

Richard

[PATCH 1/4] insane.bbclass: add QA check: package-missing

[Hongxu Jia]

During packaging, check if there are any dependencies (RDEPENDS)
on packages that have ended up empty and not produced; and if so
produce an warn/error as soon as possible, instead of allowing
the build to proceed up to do_rootfs and then the package manager
reporting the package as missing.

At the moment, we use packageinfo which saved in $PKGDATA_DIR to
collect all available packages rdepends and rprovides. So in the
do_package_qa task, while package not produced, we could compute
rdepends chain, and report warn/error immediately.

[YOCTO #5531]
[YOCTO #6420]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/insane.bbclass | 100 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 99 insertions(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 0b45374..59129b7 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -29,7 +29,7 @@ QA_SANE = "True"
 WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
             textrel already-stripped incompatible-license files-invalid \
             installed-vs-shipped compile-host-path install-host-path \
-            pn-overrides infodir build-deps file-rdeps \
+            pn-overrides infodir build-deps file-rdeps package-missing \
             "
 ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
@@ -868,6 +868,15 @@ def package_qa_check_rdepends(pkg, pkgdest, skip, taskdeps, packages, d):
                             (pkg, ', '.join(str(e) for e in filerdepends))
                 sane = package_qa_handle_error("file-rdeps", error_msg, d)
 
+        if "package-missing" not in skip and \
+               not pkg.endswith("-dev") and not pkg.endswith("-staticdev") and \
+               not pkg.endswith("-locale") and not pkg.endswith("-dbg") and \
+               not pkg.endswith("-doc"):
+            if not oe.packagedata.packaged(pkg, d):
+                error_msg = "package %s not generated, but listed in PACKAGES\n" % pkg
+                error_msg += compute_rdepends_chain(pkg, d)
+                sane = package_qa_handle_error("package-missing", error_msg, d)
+
     return sane
 
 def package_qa_check_deps(pkg, pkgdest, skip, d):
@@ -1152,3 +1161,92 @@ python () {
     for i in issues:
         package_qa_handle_error("pkgvarcheck", "%s: Variable %s is set as not being package specific, please fix this." % (d.getVar("FILE", True), i), d)
 }
+
+# It collects all available packages rdepends and rprovides,
+# except the "-dev, -staticdev, -locale, -dbg, -doc" packages.
+# Such as:
+# RDEPENDS_pkg1 = "pkg2 pkg3"
+# RDEPENDS_pkg4 = "pkg3 pkg5"
+# equals:
+# rdeps_dict = {'pkg2': 'pkg1', 'pkg3': 'pkg1 pkg4', 'pkg5':'pkg4'}
+#
+# RPROVIDES_pkg6 = "pkg2 pkg7"
+# RPROVIDES_pkg7 = "pkg5"
+# equals:
+# rprovides_dict = {'pkg6': 'pkg2 pkg7', 'pkg7': 'pkg5'}
+def collect_packageinfo(d):
+    import re
+    rdeps_dict = dict()
+    rprovides_dict = dict()
+    pkgdata_dir = d.getVar("PKGDATA_DIR", True) + '/runtime/'
+    if os.path.exists(pkgdata_dir):
+        for root, dirs, files in os.walk(pkgdata_dir):
+            for pkgname in files:
+                if pkgname.endswith('.packaged') or \
+                   pkgname.endswith("-dev") or \
+                   pkgname.endswith("-staticdev") or \
+                   pkgname.endswith("-locale") or \
+                   pkgname.endswith("-dbg") or pkgname.endswith("-doc"):
+                    continue
+
+                pkgdatafile = root + pkgname
+                try:
+                    sdata = oe.packagedata.read_pkgdatafile(pkgdatafile)
+                    rdepends = sdata.get('RDEPENDS_%s' % pkgname)
+                    if rdepends:
+                        # Filter out sub string "(***)" in RDEPENDS
+                        p = re.compile('\([^\)]*\)')
+                        for rdep in p.sub('', rdepends).split():
+                            # Filter out pkg which rdepends itself
+                            if pkgname == rdep:
+                                continue
+
+                            if rdep in rdeps_dict and \
+                               pkgname not in rdeps_dict[rdep].split():
+                                rdeps_dict[rdep] += ' %s' % pkgname
+                            elif rdep not in rdeps_dict:
+                                rdeps_dict[rdep] = '%s' % pkgname
+
+                    rprovides = sdata.get('RPROVIDES_%s' % pkgname)
+                    if rprovides:
+                        rprovides_dict[pkgname] = rprovides
+
+                except Exception as e:
+                    bb.warn("%s: Failed to read pkgdata file %s: %s: %s" % (pkgname, pkgdatafile, e.__class__, str(e)))
+
+    return rdeps_dict, rprovides_dict
+
+def compute_rdepends_chain(pkg, d):
+    rdeps_dict, rprovides_dict = collect_packageinfo(d)
+    def get_parents(pkg, messages):
+        parents = []
+
+        if pkg not in rdeps_dict and pkg in rprovides_dict:
+            for rprovide in rprovides_dict[pkg].split():
+                # Use rprovide to instead of pkg
+                if rprovide in rdeps_dict:
+                    messages.append('("%s" rprovides "%s")' % (pkg, rprovide))
+                    pkg = rprovide
+                    break
+
+        if pkg in rdeps_dict:
+            parents = rdeps_dict[pkg].split()
+            for parent in parents:
+                message = '"%s" -> "%s"' % (parent, pkg)
+                if message in messages:
+                    return
+                messages.append(message)
+
+            for parent in parents:
+                get_parents(parent, messages)
+
+        return
+
+    messages = []
+    get_parents(pkg, messages)
+    if messages:
+        message = "Compute rdepends chains, '-->' means 'runtime depends'\n" \
+                  + '\n'.join(messages)
+        return message
+    return ''
+
-- 
1.9.1