[PATCH 0/2] fix smart install with --attempt failed

[PATCH 0/2] fix smart install with --attempt failed

[Hongxu Jia]

Test steps:

1. vim local.conf
...
IMAGE_INSTALL_append = "shadow man-pages hdparm"
EXTRA_IMAGE_FEATURES += "doc-pkgs"
...

2. bitbake core-image-minimal
...
|WARNING: Unable attempt to install packages. Command '...'
returned 1:
|error: file /usr/share/man/man5/passwd.5 from install of
shadow-doc-4.2.1-r0.i586 conflicts with file from package
man-pages-3.71-r0.i586
|error: file /usr/share/man/man3/getspnam.3 from install
of shadow-doc-4.2.1-r0.i586 conflicts with file from package
man-pages-3.71-r0.i586


|WARNING: Attempt to install again:...
|NOTE: Tasks Summary: Attempted 2198 tasks of which 2172 didn't
need to be rerun and all succeeded.
...

The rest doc packages installed successfully.

//Hongxu

The following changes since commit 8ac8eca2e3bd8c78e2b31ea974930ed0243258a3:

  build-appliance-image: Update to dizzy head revision (2014-09-23 22:10:26 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/fix-attempt-install
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-attempt-install

Hongxu Jia (2):
  rootfs.py: catch inner warn message
  package_manager.py: ignore failed smart install with --attempt

 meta/lib/oe/package_manager.py | 38 +++++++++++++++++++++++++++++++++++++-
 meta/lib/oe/rootfs.py          | 20 +++++++++++++++++++-
 2 files changed, 56 insertions(+), 2 deletions(-)

-- 
1.9.1

[PATCH 1/2] rootfs.py: catch inner warn message

[Hongxu Jia]

Package managements (smart/apt-get/opkg-cl) generate some warn messages
to stdout, and we need to catch them and output by bb.warn.

Here is an example, while invoking smart to attempt install doc packages,
if install failed, it generates warn message to stdout.
...
|warning: Can't install util-linux-doc-2.24.2-r1@i586: Can't
install util-linux-doc-2.24.2-r1@i586: no package provides info
...

The fix catches it and outputs:
...
|WARNING: log_check: There is a warn message in the logfile
|WARNING: log_check: Matched keyword: [warn]
|WARNING: log_check: warning: Can't install util-linux-doc-2.24.2-r1@
i586: Can't install util-linux-doc-2.24.2-r1@i586: no package provides
info
...

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/lib/oe/rootfs.py | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 3d8ae81..c4735f2 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -347,7 +347,21 @@ class RpmRootfs(Rootfs):
         # already saved in /etc/rpm-postinsts
         pass
 
-    def _log_check(self):
+    def _log_check_warn(self):
+        r = re.compile('(warn|Warn)')
+        log_path = self.d.expand("${T}/log.do_rootfs")
+        with open(log_path, 'r') as log:
+            for line in log.read().split('\n'):
+                if 'log_check' in line:
+                    continue
+
+                m = r.search(line)
+                if m:
+                    bb.warn('log_check: There is a warn message in the logfile')
+                    bb.warn('log_check: Matched keyword: [%s]' % m.group())
+                    bb.warn('log_check: %s\n' % line)
+
+    def _log_check_error(self):
         r = re.compile('(unpacking of archive failed|Cannot find package|exit 1|ERR|Fail)')
         log_path = self.d.expand("${T}/log.do_rootfs")
         with open(log_path, 'r') as log:
@@ -370,6 +384,10 @@ class RpmRootfs(Rootfs):
                 if found_error == 6:
                     bb.fatal(message)
 
+    def _log_check(self):
+        self._log_check_warn()
+        self._log_check_error()
+
     def _handle_intercept_failure(self, registered_pkgs):
         rpm_postinsts_dir = self.image_rootfs + self.d.expand('${sysconfdir}/rpm-postinsts/')
         bb.utils.mkdirhier(rpm_postinsts_dir)
-- 
1.9.1

[PATCH 2/2] package_manager.py: ignore failed smart install with --attempt

[Hongxu Jia]

Paul's patch added checking in smart to ignore failed install with --attempt
...
commit e9c1191da4f84c9743b053e7030ab28b48f57c44
Author: Paul Eggleton <paul.eggleton@linux.intel.com>
Date:   Thu Feb 6 13:39:01 2014 +0000

    python-smartpm: really ignore conflicts during install with --attempt
...

But if the failure is generated by rpm which smart doesn't resolve, we check
the output of smart, filter the failed packages, and attempt to install the
rest.

Here is an example, assume there are two packages which have identical file
name but different content, that cause confliction. If we install one without
--attempt, after it installed and install another with --attempt, the rpm
will generate an error which smart does not resolve:
...
|error: file /usr/share/man/man5/passwd.5 from install of
shadow-doc-4.2.1-r0.i586 conflicts with file from package
man-pages-3.71-r0.i586
|error: file /usr/share/man/man3/getspnam.3 from install of
shadow-doc-4.2.1-r0.i586 conflicts with file from package
man-pages-3.71-r0.i586
...

In this situation, remove failed packages, and attempt to install the rest.

[YOCTO #6769]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/lib/oe/package_manager.py | 38 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index b3b3b2d..20997d3 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -955,7 +955,43 @@ class RpmPM(PackageManager):
             output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
             bb.note(output)
         except subprocess.CalledProcessError as e:
-            bb.fatal("Unable to install packages. Command '%s' "
+            if not attempt_only:
+                bb.fatal("Unable to install packages. Command '%s' "
+                         "returned %d:\n%s" % (cmd, e.returncode, e.output))
+            else:
+                bb.warn("Unable attempt to install packages. Command '%s' "
+                         "returned %d:\n%s" % (cmd, e.returncode, e.output))
+
+                # Remove failed packages, and attempt to install the rest
+                reinstall_pkgs = pkgs[:]
+                for line in e.output.split("\n"):
+                    if line.strip() == "":
+                        continue
+                    for pkg in pkgs[:]:
+                        name = pkg.split("@")[0]
+                        arch = pkg.split("@")[1]
+                        r = re.compile(r'%s\S+%s' % (name, arch))
+                        if r.search(line):
+                            pkgs.remove(pkg)
+
+                bb.warn("Attempt to install again: %s" % ' '.join(pkgs))
+                self._attempt_install(pkgs)
+
+    '''
+    Attempt to install spkgs, the spkgs is a pkg list with smart format
+    '''
+    def _attempt_install(self, spkgs):
+        if len(spkgs) == 0:
+            return
+
+        bb.note("Attempt to install the following packages: %s" % ' '.join(spkgs))
+        cmd = "%s %s install --attempt -y %s" % \
+              (self.smart_cmd, self.smart_opt, ' '.join(spkgs))
+        try:
+            output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
+            bb.note(output)
+        except subprocess.CalledProcessError as e:
+            bb.warn("Unable attempt to install packages. Command '%s' "
                      "returned %d:\n%s" % (cmd, e.returncode, e.output))
 
     '''
-- 
1.9.1

Re: [PATCH 2/2] package_manager.py: ignore failed smart install with --attempt

[Paul Eggleton]

Hi Hongxu,

On Friday 26 September 2014 19:36:02 Hongxu Jia wrote:
> Paul's patch added checking in smart to ignore failed install with --attempt
> ...
> commit e9c1191da4f84c9743b053e7030ab28b48f57c44
> Author: Paul Eggleton <paul.eggleton@linux.intel.com>
> Date:   Thu Feb 6 13:39:01 2014 +0000
> 
>     python-smartpm: really ignore conflicts during install with --attempt
> ...
> 
> But if the failure is generated by rpm which smart doesn't resolve, we check
> the output of smart, filter the failed packages, and attempt to install the
> rest.
> 
> Here is an example, assume there are two packages which have identical file
> name but different content, that cause confliction. If we install one
> without --attempt, after it installed and install another with --attempt,
> the rpm will generate an error which smart does not resolve:
> ...
> 
> |error: file /usr/share/man/man5/passwd.5 from install of
> 
> shadow-doc-4.2.1-r0.i586 conflicts with file from package
> man-pages-3.71-r0.i586
> 
> |error: file /usr/share/man/man3/getspnam.3 from install of
> 
> shadow-doc-4.2.1-r0.i586 conflicts with file from package
> man-pages-3.71-r0.i586
> ...
> 
> In this situation, remove failed packages, and attempt to install the rest.

Apologies for the delay in reviewing this.

I see what you're trying to do here. However I am concerned about the other 
kinds of failure cases we might end up obscuring with this kind of approach. 
My understanding of attemptonly is that it should ignore conflicts, but any 
other kind of error should trigger build failure - otherwise you can miss 
genuine errors that need to be fixed.

If possible I would prefer if we would detect and handle this kind of conflict 
within smart itself and allow it to continue rather than detecting the failure 
and retrying externally.

Mark (on CC), any opinions?

Cheers,
Paul



-- 

Paul Eggleton
Intel Open Source Technology Centre