[PATCH 00/12] [auh] make it easy to use by recipe maintainer

[PATCH 00/12] [auh] make it easy to use by recipe maintainer

[Robert Yang]

* Usages: (No settings is required by default)
  - Upgrade one recipe:
    $ upgradehelper.py less

  - Upgrade multiple recipes:
    $ upgradehelper.py less bash

    The commit will be kept in the repo when *succeed*.
    No commit when *failed* by default unless -f is used.

  - Upgrade recipes and apply failed patches after the upgrade is done:
    $ upgradehelper.py less bash -f

  - Upgrade all recipes and send emails:
    $ upgradehelper.py all -e

* Next:
  - Auto fix do_fetch error when thera are more than one checksums, e.g., git repo.
  - Add it into oe-core and suggest maintainer uses it ?

// Robert

The following changes since commit ccf93239c7def7b475fe20f8fcd91043bc59bd61:

  upgradehelper.py: Layer mode do a full cleanup of poky (2017-06-05 16:14:23 -0500)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib rbt/auh
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/auh

Robert Yang (12):
  upgradehelper.py: fix checking for do_checkpkg
  upgradehelper.py: support upgrade multiple recipes
  upgradehelper.py: use UniverseUpdater for all cases
  modules/steps.py: fix warn when skip compilation
  modules/steps.py: fix warn when skip compilation
  upgradehelper.py: only check email settings when -e is specified
  upgradehelper.py: always do upgrade when recipes are specified
  upgradehelper.py: clean repo only once when recipes are specified
  upgradehelper.py: use git user in commit when recipes are specified
  upgradehelper.py: add --apply-failed option
  upgradehelper.py: print info when recipe is skipped to upgrade
  upgradehelper.py: don't build gcc-runtime when --skip-compilation

 modules/steps.py |   4 --
 upgradehelper.py | 118 ++++++++++++++++++++++++++++++++++---------------------
 2 files changed, 73 insertions(+), 49 deletions(-)

-- 
2.7.4

[PATCH 01/12] upgradehelper.py: fix checking for do_checkpkg

[Robert Yang]

The error message in the log is:
Initialising tasks...ERROR: Task do_checkpkg does not exist for target strace [snip]

So line.find("ERROR: Task do_checkpkg does not exist") == 0 doesn't
work, use != -1 to fix the problem.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 upgradehelper.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 8e5466e..d439a4c 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -753,7 +753,7 @@ class UniverseUpdater(Updater):
             self.bb.checkpkg(recipe)
         except Error as e:
             for line in e.stdout.split('\n'):
-                if line.find("ERROR: Task do_checkpkg does not exist") == 0:
+                if line.find("ERROR: Task do_checkpkg does not exist") != -1:
                     C(" \"distrodata.bbclass\" not inherited. Consider adding "
                       "the following to your local.conf:\n\n"
                       "INHERIT =+ \"distrodata\"\n")
-- 
2.7.4

[PATCH 02/12] upgradehelper.py: support upgrade multiple recipes

[Robert Yang]

Now we can use:
$ upgradehelper.py recipe1 recipe2

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 upgradehelper.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index d439a4c..220e459 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -74,7 +74,7 @@ def parse_cmdline():
     parser = argparse.ArgumentParser(description='Package Upgrade Helper',
                                      formatter_class=argparse.RawTextHelpFormatter,
                                      epilog=help_text)
-    parser.add_argument("recipe", help="recipe to be upgraded")
+    parser.add_argument("recipe", nargs = '+', action='store', default='', help="recipe to be upgraded")
 
     parser.add_argument("-t", "--to_version",
                         help="version to upgrade the recipe to")
@@ -928,7 +928,7 @@ if __name__ == "__main__":
                     level=debug_levels[args.debug_level - 1])
     settings, maintainer_override = parse_config_file(args.config_file)
 
-    recipes = args.recipe.split()
+    recipes = args.recipe
 
     if len(recipes) == 1 and recipes[0] == "all":
         updater = UniverseUpdater()
@@ -942,9 +942,12 @@ if __name__ == "__main__":
             args.maintainer = "Upgrade Helper <%s>" % \
                 settings.get('from', 'uh@not.set')
 
-        pkg_list = [(args.recipe, args.to_version, args.maintainer)]
+        pkg_list = [(recipes[0], args.to_version, args.maintainer)]
         updater = Updater(args.auto_mode, args.send_emails, args.skip_compilation)
         updater.run(pkg_list)
+    elif len(recipes) > 1 and args.to_version:
+        E(" -t is only supported when upgrade one recipe\n")
+        exit(1)
     else:
         updater = UniverseUpdater(recipes)
         updater.run()
-- 
2.7.4

[PATCH 03/12] upgradehelper.py: use UniverseUpdater for all cases

[Robert Yang]

* Use UniverseUpdater() for the following 3 cases:
  + Upgrade all recipes
  + Upgrade 1 recipe
    - '--maintainer' is not a must when any more when use --send-emails, the
      maintainer be got from distrodata.
  + Upgrade multiple recipes

* Use "args" as the parameter of UniverseUpdater() and Updater(), this can make
  the parameters simple, and easy for extending.

* Make upgrade "all" recipes respect the args, it didn't care --send-emails or
  --maintainer which would suprise the user.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 upgradehelper.py | 68 +++++++++++++++++++++++++-------------------------------
 1 file changed, 30 insertions(+), 38 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 220e459..4a3f3f0 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -128,10 +128,11 @@ def parse_config_file(config_file):
     return (settings, maintainer_override)
 
 class Updater(object):
-    def __init__(self, auto_mode=False, send_email=False, skip_compilation=False):
+    def __init__(self, args):
         build_dir = get_build_dir()
 
         self.bb = Bitbake(build_dir)
+        self.args = args
 
         try:
             self.base_env = self.bb.env()
@@ -141,7 +142,7 @@ class Updater(object):
             E( " Bitbake output:\n%s" % (e.stdout))
             exit(1)
 
-        self._set_options(auto_mode, send_email, skip_compilation)
+        self._set_options()
 
         self._make_dirs(build_dir)
 
@@ -150,7 +151,7 @@ class Updater(object):
         self.email_handler = Email(settings)
         self.statistics = Statistics()
 
-    def _set_options(self, auto_mode, send_email, skip_compilation):
+    def _set_options(self):
         self.opts = {}
         self.opts['layer_mode'] = settings.get('layer_mode', '')
         if self.opts['layer_mode'] == 'yes':
@@ -174,11 +175,11 @@ class Updater(object):
             self.opts['machines'] = settings.get('machines',
                 'qemux86 qemux86-64 qemuarm qemumips qemuppc').split()
 
-        self.opts['interactive'] = not auto_mode
-        self.opts['send_email'] = send_email
+        self.opts['interactive'] = not self.args.auto_mode
+        self.opts['send_email'] = self.args.send_emails
         self.opts['author'] = "Upgrade Helper <%s>" % \
                 settings.get('from', 'uh@not.set')
-        self.opts['skip_compilation'] = skip_compilation
+        self.opts['skip_compilation'] = self.args.skip_compilation
         self.opts['buildhistory'] = self._buildhistory_is_enabled()
         self.opts['testimage'] = self._testimage_is_enabled()
 
@@ -659,17 +660,25 @@ class Updater(object):
                 self.send_status_mail(statistics_summary)
 
 class UniverseUpdater(Updater):
-    def __init__(self, recipes=None):
-        Updater.__init__(self, True, True)
+    def __init__(self, args):
+        Updater.__init__(self, args)
+
+        if len(args.recipe) == 1 and args.recipe[0] == "all":
+            self.recipes = []
+        else:
+            self.recipes = args.recipe
 
         # to filter recipes in upgrade
-        if not recipes and self.opts['layer_mode'] == 'yes':
+        if not self.recipes and self.opts['layer_mode'] == 'yes':
             # when layer mode is enabled and no recipes are specified
             # we need to figure out what recipes are provided by the
             # layer to try upgrade
             self.recipes = self._get_recipes_by_layer()
-        else:
-            self.recipes = recipes
+
+        if args.to_version:
+            if len(self.recipes) != 1:
+                E(" -t is only supported when upgrade one recipe\n")
+                exit(1)
 
         # read history file
         self.history_file = os.path.join(get_build_dir(), "upgrade-helper", "history.uh")
@@ -772,9 +781,15 @@ class UniverseUpdater(Updater):
 
                 pn = row[0]
                 cur_ver = row[1]
-                next_ver = row[2]
+                if self.args.to_version:
+                    next_ver = self.args.to_version
+                else:
+                    next_ver = row[2]
                 status = row[11]
-                maintainer = row[14]
+                if self.args.maintainer:
+                    maintainer = self.args.maintainer
+                else:
+                    maintainer = row[14]
                 no_upgrade_reason = row[15]
 
                 if status == 'UPDATE' and not no_upgrade_reason:
@@ -928,28 +943,5 @@ if __name__ == "__main__":
                     level=debug_levels[args.debug_level - 1])
     settings, maintainer_override = parse_config_file(args.config_file)
 
-    recipes = args.recipe
-
-    if len(recipes) == 1 and recipes[0] == "all":
-        updater = UniverseUpdater()
-        updater.run()
-    elif len(recipes) == 1 and args.to_version:
-        if not args.maintainer and args.send_emails:
-            E(" For upgrade one recipe and send email you must specify --maintainer\n")
-            exit(1)
-
-        if not args.maintainer:
-            args.maintainer = "Upgrade Helper <%s>" % \
-                settings.get('from', 'uh@not.set')
-
-        pkg_list = [(recipes[0], args.to_version, args.maintainer)]
-        updater = Updater(args.auto_mode, args.send_emails, args.skip_compilation)
-        updater.run(pkg_list)
-    elif len(recipes) > 1 and args.to_version:
-        E(" -t is only supported when upgrade one recipe\n")
-        exit(1)
-    else:
-        updater = UniverseUpdater(recipes)
-        updater.run()
-
-
+    updater = UniverseUpdater(args)
+    updater.run()
-- 
2.7.4

[PATCH 04/12] modules/steps.py: fix warn when skip compilation

[Robert Yang]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 modules/steps.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/modules/steps.py b/modules/steps.py
index 0c527a4..677d101 100644
--- a/modules/steps.py
+++ b/modules/steps.py
@@ -137,7 +137,7 @@ def unpack_original_workdir(bb, git, opts, pkg_ctx):
 
 def compile(bb, git, opts, pkg_ctx):
     if opts['skip_compilation']:
-        W(" %s: Compilation was skipped by user choice!")
+        W(" %s: Compilation was skipped by user choice!" % pkg_ctx['PN'])
         return
 
     for machine in opts['machines']:
-- 
2.7.4

[PATCH 05/12] modules/steps.py: fix warn when skip compilation

[Robert Yang]

* The previous code lacks a "% var" in the end:
  W(" %s: Compilation was skipped by user choice!")

* Move the skipping steps to upgradehelper.py rather than
  modules/steps.py, do not run compile is more straight-forward  than
  return early from it when skipping.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 modules/steps.py | 4 ----
 upgradehelper.py | 7 +++++--
 2 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/modules/steps.py b/modules/steps.py
index 677d101..c78cabc 100644
--- a/modules/steps.py
+++ b/modules/steps.py
@@ -136,10 +136,6 @@ def unpack_original_workdir(bb, git, opts, pkg_ctx):
         pass
 
 def compile(bb, git, opts, pkg_ctx):
-    if opts['skip_compilation']:
-        W(" %s: Compilation was skipped by user choice!" % pkg_ctx['PN'])
-        return
-
     for machine in opts['machines']:
         I(" %s: compiling for %s ..." % (pkg_ctx['PN'], machine))
         pkg_ctx['recipe'].compile(machine)
diff --git a/upgradehelper.py b/upgradehelper.py
index 4a3f3f0..605114b 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -56,6 +56,7 @@ from utils.emailhandler import Email
 
 from statistics import Statistics
 from steps import upgrade_steps
+from steps import compile
 from testimage import TestImage
 
 help_text = """Usage examples:
@@ -179,7 +180,6 @@ class Updater(object):
         self.opts['send_email'] = self.args.send_emails
         self.opts['author'] = "Upgrade Helper <%s>" % \
                 settings.get('from', 'uh@not.set')
-        self.opts['skip_compilation'] = self.args.skip_compilation
         self.opts['buildhistory'] = self._buildhistory_is_enabled()
         self.opts['testimage'] = self._testimage_is_enabled()
 
@@ -231,7 +231,7 @@ class Updater(object):
                       " but need BUILDHISTORY_COMMIT=1 please set.")
                     exit(1)
 
-                if self.opts['skip_compilation']:
+                if self.args.skip_compilation:
                     W(" Buildhistory disabled because user" \
                             " skip compilation!")
                 else:
@@ -577,6 +577,9 @@ class Updater(object):
             try:
                 I(" %s: Upgrading to %s" % (pkg_ctx['PN'], pkg_ctx['NPV']))
                 for step, msg in upgrade_steps:
+                    if step == compile and self.args.skip_compilation:
+                        W(" %s: Skipping compile by user choice" % pkg_ctx['PN'])
+                        continue
                     if msg is not None:
                         I(" %s: %s" % (pkg_ctx['PN'], msg))
                     step(self.bb, self.git, self.opts, pkg_ctx)
-- 
2.7.4

[PATCH 06/12] upgradehelper.py: only check email settings when -e is specified

[Robert Yang]

Fixed:
$ upgradehelper.py less
ERROR: smtp host not set! Sending emails disabled!
ERROR: 'From' address not set! Sending emails disabled!

Only check email settings when "-e" is specified can fix the problem.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 upgradehelper.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 605114b..70b9e8d 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -149,7 +149,8 @@ class Updater(object):
 
         self._add_file_logger()
 
-        self.email_handler = Email(settings)
+        if self.args.send_emails:
+            self.email_handler = Email(settings)
         self.statistics = Statistics()
 
     def _set_options(self):
-- 
2.7.4

[PATCH 07/12] upgradehelper.py: always do upgrade when recipes are specified

[Robert Yang]

For example, first run:
$ upgradehelper.py less -d 5
It did the upgrade

Second run:
$ upgradehelper.py less -d 5
DEBUG: Skipping upgrade of less: is in history and not 30 days passed

Let it always do the upgrade makes it easier to use when do upgrade locally.
It will still do the check when the recipe is all.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 upgradehelper.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 70b9e8d..e38a281 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -879,7 +879,7 @@ class UniverseUpdater(Updater):
                 if not os.path.exists(last_checkpkg_file):
                     last_checkpkg_file = None
 
-        if last_master_commit != cur_master_commit or last_date_checked != current_date or \
+        if self.recipes or last_master_commit != cur_master_commit or last_date_checked != current_date or \
                 last_checkpkg_file is None:
             self._check_upstream_versions()
             last_checkpkg_file = os.path.realpath(get_build_dir() + "/tmp/log/checkpkg.csv")
@@ -889,7 +889,10 @@ class UniverseUpdater(Updater):
 
         pkgs_list = []
         for pkg in self._parse_checkpkg_file(last_checkpkg_file):
-            if self._pkg_upgradable(pkg[0], pkg[1], pkg[2]):
+            # Always do the upgrade if recipes are specified
+            if self.recipes and pkg[0] in self.recipes:
+                pkgs_list.append(pkg)
+            elif self._pkg_upgradable(pkg[0], pkg[1], pkg[2]):
                 pkgs_list.append(pkg)
 
         # Update last_checkpkg_run only after the version check has been completed
-- 
2.7.4

[PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Robert Yang]

E.g.:
$ upgradehelper.py less strace bash git

The commit is removed when failed, and kept when succeed, but it would be
removed when next recipe runs, so only run clean_repo once can keep the commit,
which is helpful for the user.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 upgradehelper.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/upgradehelper.py b/upgradehelper.py
index e38a281..ff27b97 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -57,6 +57,7 @@ from utils.emailhandler import Email
 from statistics import Statistics
 from steps import upgrade_steps
 from steps import compile
+from steps import clean_repo
 from testimage import TestImage
 
 help_text = """Usage examples:
@@ -419,6 +420,10 @@ class Updater(object):
                 else:
                     I(" %s: Save patch in directory: %s." %
                         (pkg_ctx['PN'], pkg_ctx['workdir']))
+                    if pkg_ctx['error']:
+                        I(" %s: Remove it from repo since failed!" % pkg_ctx['PN'])
+                        self.git.reset_hard(1)
+
         except Error as e:
             msg = ''
 
@@ -569,6 +574,7 @@ class Updater(object):
         succeeded_pkgs_ctx = []
         failed_pkgs_ctx = []
         attempted_pkgs = 0
+        repo_cleaned = False
         for pn, _, _ in pkgs_to_upgrade:
             pkg_ctx = pkgs_ctx[pn]
             pkg_ctx['error'] = None
@@ -581,6 +587,12 @@ class Updater(object):
                     if step == compile and self.args.skip_compilation:
                         W(" %s: Skipping compile by user choice" % pkg_ctx['PN'])
                         continue
+                    if step == clean_repo and self.recipes:
+                        if repo_cleaned:
+                            I(" %s: Skipping clean_repo since it had been run by previous recipe" % pkg_ctx['PN'])
+                            continue
+                        else:
+                            repo_cleaned = True
                     if msg is not None:
                         I(" %s: %s" % (pkg_ctx['PN'], msg))
                     step(self.bb, self.git, self.opts, pkg_ctx)
-- 
2.7.4

[PATCH 09/12] upgradehelper.py: use git user in commit when recipes are specified

[Robert Yang]

E.g.,:
$ upgradehelper.py less

commit d9b551438037e105eebabbbcb2da6d8b8cae7504
Author: Robert Yang <liezhi.yang@windriver.com>
Date:   Wed Dec 6 16:38:55 2017 +0800

    less: upgrade to 529

    Signed-off-by: Robert Yang <liezhi.yang@windriver.com>

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 upgradehelper.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index ff27b97..0678c58 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -407,7 +407,11 @@ class Updater(object):
 
             if 'recipe' in pkg_ctx:
                 I(" %s: Auto commit changes ..." % pkg_ctx['PN'])
-                self.git.commit(pkg_ctx['recipe'].commit_msg, self.opts['author'])
+                commit_msg = pkg_ctx['recipe'].commit_msg
+                if self.recipes:
+                    self.git.commit(commit_msg)
+                else:
+                    self.git.commit(commit_msg, self.opts['author'])
 
                 stdout = self.git.create_patch(pkg_ctx['workdir'])
                 pkg_ctx['patch_file'] = stdout.strip()
-- 
2.7.4

[PATCH 10/12] upgradehelper.py: add --apply-failed option

[Robert Yang]

  -f, --apply-failed    Apply failed patch in the repo after upgrade is done

And add "FAILED:" in subject when failed. So that the user can go on
working based on the commit.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 upgradehelper.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 0678c58..3ad33ae 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -94,6 +94,8 @@ def parse_cmdline():
                         help="do not compile, just change the checksums, remove PR, and commit")
     parser.add_argument("-c", "--config-file", default=None,
                         help="Path to the configuration file. Default is $BUILDDIR/upgrade-helper/upgrade-helper.conf")
+    parser.add_argument("-f", "--apply-failed", action="store_true", default=False,
+                        help="Apply failed patch in the repo after upgrade is done")
     return parser.parse_args()
 
 def parse_config_file(config_file):
@@ -407,7 +409,10 @@ class Updater(object):
 
             if 'recipe' in pkg_ctx:
                 I(" %s: Auto commit changes ..." % pkg_ctx['PN'])
-                commit_msg = pkg_ctx['recipe'].commit_msg
+                if pkg_ctx['error']:
+                    commit_msg = "FAILED: %s" % pkg_ctx['recipe'].commit_msg
+                else:
+                    commit_msg = pkg_ctx['recipe'].commit_msg
                 if self.recipes:
                     self.git.commit(commit_msg)
                 else:
@@ -655,6 +660,11 @@ class Updater(object):
                     pkg_ctx['MAINTAINER'], pkg_ctx['error'])
             self.pkg_upgrade_handler(pkg_ctx)
 
+            if self.args.apply_failed and pkg_ctx in failed_pkgs_ctx:
+                if pkg_ctx['patch_file']:
+                    I(" %s: Applying failed patch" % pn)
+                    self.git.apply_patch(pkg_ctx['patch_file'])
+
         if attempted_pkgs > 0:
             publish_work_url = settings.get('publish_work_url', '')
             work_tarball = os.path.join(self.uh_base_work_dir,
-- 
2.7.4

[PATCH 11/12] upgradehelper.py: print info when recipe is skipped to upgrade

[Robert Yang]

This makes debug easier.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 upgradehelper.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 3ad33ae..c099445 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -826,11 +826,11 @@ class UniverseUpdater(Updater):
                     pkgs_list.append((pn, next_ver, maintainer))
                 else:
                     if no_upgrade_reason:
-                        D(" Skip package %s (status = %s, current version = %s," \
+                        I(" Skip package %s (status = %s, current version = %s," \
                             " next version = %s, no upgrade reason = %s)" %
                             (pn, status, cur_ver, next_ver, no_upgrade_reason))
                     else:
-                        D(" Skip package %s (status = %s, current version = %s," \
+                        I(" Skip package %s (status = %s, current version = %s," \
                             " next version = %s)" %
                             (pn, status, cur_ver, next_ver))
         return pkgs_list
-- 
2.7.4

[PATCH 12/12] upgradehelper.py: don't build gcc-runtime when --skip-compilation

[Robert Yang]

It doesn't make any sense to build it when skip compile.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 upgradehelper.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index c099445..6f100fe 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -565,7 +565,7 @@ class Updater(object):
             pkgs_ctx[p]['base_dir'] = self.uh_recipes_all_dir
         I(" ############################################################")
 
-        if pkgs_to_upgrade:
+        if pkgs_to_upgrade and not self.args.skip_compilation:
             I(" Building gcc runtimes ...")
             for machine in self.opts['machines']:
                 I("  building gcc runtime for %s" % machine)
-- 
2.7.4

Re: [PATCH 00/12] [auh] make it easy to use by recipe maintainer

[Robert Yang]

I had added alexander.kanavin@linux.intel.com in the cc list, but it was gone,
looks strange, now add him again.

// Robert

On 12/07/2017 03:37 PM, Robert Yang wrote:
> * Usages: (No settings is required by default)
>    - Upgrade one recipe:
>      $ upgradehelper.py less
> 
>    - Upgrade multiple recipes:
>      $ upgradehelper.py less bash
> 
>      The commit will be kept in the repo when *succeed*.
>      No commit when *failed* by default unless -f is used.
> 
>    - Upgrade recipes and apply failed patches after the upgrade is done:
>      $ upgradehelper.py less bash -f
> 
>    - Upgrade all recipes and send emails:
>      $ upgradehelper.py all -e
> 
> * Next:
>    - Auto fix do_fetch error when thera are more than one checksums, e.g., git repo.
>    - Add it into oe-core and suggest maintainer uses it ?
> 
> // Robert
> 
> The following changes since commit ccf93239c7def7b475fe20f8fcd91043bc59bd61:
> 
>    upgradehelper.py: Layer mode do a full cleanup of poky (2017-06-05 16:14:23 -0500)
> 
> are available in the git repository at:
> 
>    git://git.pokylinux.org/poky-contrib rbt/auh
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/auh
> 
> Robert Yang (12):
>    upgradehelper.py: fix checking for do_checkpkg
>    upgradehelper.py: support upgrade multiple recipes
>    upgradehelper.py: use UniverseUpdater for all cases
>    modules/steps.py: fix warn when skip compilation
>    modules/steps.py: fix warn when skip compilation
>    upgradehelper.py: only check email settings when -e is specified
>    upgradehelper.py: always do upgrade when recipes are specified
>    upgradehelper.py: clean repo only once when recipes are specified
>    upgradehelper.py: use git user in commit when recipes are specified
>    upgradehelper.py: add --apply-failed option
>    upgradehelper.py: print info when recipe is skipped to upgrade
>    upgradehelper.py: don't build gcc-runtime when --skip-compilation
> 
>   modules/steps.py |   4 --
>   upgradehelper.py | 118 ++++++++++++++++++++++++++++++++++---------------------
>   2 files changed, 73 insertions(+), 49 deletions(-)
> 

Re: [PATCH 00/12] [auh] make it easy to use by recipe maintainer

[Alexander Kanavin]

On 12/07/2017 09:37 AM, Robert Yang wrote:
> Robert Yang (12):
>    upgradehelper.py: fix checking for do_checkpkg
>    upgradehelper.py: support upgrade multiple recipes
>    upgradehelper.py: use UniverseUpdater for all cases
>    modules/steps.py: fix warn when skip compilation
>    modules/steps.py: fix warn when skip compilation
>    upgradehelper.py: only check email settings when -e is specified
>    upgradehelper.py: always do upgrade when recipes are specified
>    upgradehelper.py: clean repo only once when recipes are specified
>    upgradehelper.py: use git user in commit when recipes are specified
>    upgradehelper.py: add --apply-failed option
>    upgradehelper.py: print info when recipe is skipped to upgrade
>    upgradehelper.py: don't build gcc-runtime when --skip-compilation

Thanks, these do make sense. I'll integrate them into my devtool (and 
other fixes) work - hope to publish it in the next few days.

Alex

Re: [PATCH 05/12] modules/steps.py: fix warn when skip compilation

[Alexander Kanavin]

On 12/07/2017 09:37 AM, Robert Yang wrote:
>               try:
>                   I(" %s: Upgrading to %s" % (pkg_ctx['PN'], pkg_ctx['NPV']))
>                   for step, msg in upgrade_steps:
> +                    if step == compile and self.args.skip_compilation:
> +                        W(" %s: Skipping compile by user choice" % pkg_ctx['PN'])
> +                        continue
>                       if msg is not None:
>                           I(" %s: %s" % (pkg_ctx['PN'], msg))
>                       step(self.bb, self.git, self.opts, pkg_ctx)
> 

This one I have to reject. The loop that is executing the upgrade steps 
should not have to know or care what they are.

Alex

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Alexander Kanavin]

On 12/07/2017 09:37 AM, Robert Yang wrote:
> E.g.:
> $ upgradehelper.py less strace bash git
> 
> The commit is removed when failed, and kept when succeed, but it would be
> removed when next recipe runs, so only run clean_repo once can keep the commit,
> which is helpful for the user.

I have removed the logic that removes commits and manipulates branches 
altogether. AUH will always create a commit on the current branch, even 
if it caused a build or testimage failure, and otherwise won't touch git 
at all. If the user wants further repo manipulation, they can do it with 
an external script.

Alex

Re: [PATCH 09/12] upgradehelper.py: use git user in commit when recipes are specified

[Alexander Kanavin]

On 12/07/2017 09:37 AM, Robert Yang wrote:
> E.g.,:
> $ upgradehelper.py less
> 
> commit d9b551438037e105eebabbbcb2da6d8b8cae7504
> Author: Robert Yang <liezhi.yang@windriver.com>
> Date:   Wed Dec 6 16:38:55 2017 +0800
> 
>      less: upgrade to 529
> 
>      Signed-off-by: Robert Yang <liezhi.yang@windriver.com>

I'm not sure about this one. Commits shouldn't be created automatically 
on a person's behalf and then also signed-off - I'd say you need to 
first look at them.

Alex

Re: [PATCH 10/12] upgradehelper.py: add --apply-failed option

[Alexander Kanavin]

On 12/07/2017 09:37 AM, Robert Yang wrote:
>    -f, --apply-failed    Apply failed patch in the repo after upgrade is done
> 
> And add "FAILED:" in subject when failed. So that the user can go on
> working based on the commit.

AUH will simple leave all commits in the repo, succeessful and failed. 
Even successful upgrades may need further work before they can be 
submitted. It's useful to mark failed ones as problematic though, so 
this change can be partially taken.


Alex

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Robert Yang]

Hi Alexander,

Thanks for the quick reply.

On 12/07/2017 08:58 PM, Alexander Kanavin wrote:
> On 12/07/2017 09:37 AM, Robert Yang wrote:
>> E.g.:
>> $ upgradehelper.py less strace bash git
>>
>> The commit is removed when failed, and kept when succeed, but it would be
>> removed when next recipe runs, so only run clean_repo once can keep the commit,
>> which is helpful for the use
> 
> I have removed the logic that removes commits and manipulates branches 
> altogether. AUH will always create a commit on the current branch, even if it 
> caused a build or testimage failure, and otherwise won't touch git at all. If 
> the user wants further repo manipulation, they can do it with an external script
I'm not sure, the logical *without* this patch is:
1) Do the commit when succeed or failed.
2) git format-patch
3) Remove the commit if failed, and leave the commit there if succeed
4) The succeed commit will be removed by next recipe's run. For example:
    $ auh less git
    While less is succeed, git is failed, and less' commit will be cleaned
    by git's run.

What this patch does is on step 4, don't let the next recipe's run clean
the repo, Did I miss anything other patches, please ?

// Robert

> 
> Alex
> 

Re: [PATCH 09/12] upgradehelper.py: use git user in commit when recipes are specified

[Robert Yang]

Hello,

On 12/07/2017 09:02 PM, Alexander Kanavin wrote:
> On 12/07/2017 09:37 AM, Robert Yang wrote:
>> E.g.,:
>> $ upgradehelper.py less
>>
>> commit d9b551438037e105eebabbbcb2da6d8b8cae7504
>> Author: Robert Yang <liezhi.yang@windriver.com>
>> Date:   Wed Dec 6 16:38:55 2017 +0800
>>
>>      less: upgrade to 529
>>
>>      Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> 
> I'm not sure about this one. Commits shouldn't be created automatically on a 
> person's behalf and then also signed-off - I'd say you need to first look at them.

IMHO, Auh is a helper, when user run:
$ auh less

It helps create commits, should be the same as user run "git commit", and the
user should responsible for it. Or how about add a -S option:

-S, --SOB	Use git user as the author and add SOB.

// Robert

> 
> Alex
> 

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Robert Yang]

On 12/08/2017 09:43 AM, Robert Yang wrote:
> Hi Alexander,
> 
> Thanks for the quick reply.
> 
> On 12/07/2017 08:58 PM, Alexander Kanavin wrote:
>> On 12/07/2017 09:37 AM, Robert Yang wrote:
>>> E.g.:
>>> $ upgradehelper.py less strace bash git
>>>
>>> The commit is removed when failed, and kept when succeed, but it would be
>>> removed when next recipe runs, so only run clean_repo once can keep the commit,
>>> which is helpful for the use
>>
>> I have removed the logic that removes commits and manipulates branches 
>> altogether. AUH will always create a commit on the current branch, even if it 
>> caused a build or testimage failure, and otherwise won't touch git at all. If 
>> the user wants further repo manipulation, they can do it with an external script
> I'm not sure, the logical *without* this patch is:
> 1) Do the commit when succeed or failed.
> 2) git format-patch
> 3) Remove the commit if failed, and leave the commit there if succeed

Sorry, "remove the comit if failed" is added by this patch, and why I did this
was because:
$ auh glibc less bash gzip bzip2

All of them will be failed to upgrade if glibc fails since others depends on
glibc, so leave a failed commit in the repo would cause troubles when upgrade
multiple recipes, so we have to remove it when failed to let others can upgrade, 
and then apply the failed commit back (if -f is specified) after all of the
recipes are done.

// Robert

> 4) The succeed commit will be removed by next recipe's run. For example:
>     $ auh less git
>     While less is succeed, git is failed, and less' commit will be cleaned
>     by git's run.
> 
> What this patch does is on step 4, don't let the next recipe's run clean
> the repo, Did I miss anything other patches, please ?
> 
> // Robert
> 
>>
>> Alex
>>

Re: [PATCH 10/12] upgradehelper.py: add --apply-failed option

[Robert Yang]

On 12/07/2017 09:05 PM, Alexander Kanavin wrote:
> On 12/07/2017 09:37 AM, Robert Yang wrote:
>>    -f, --apply-failed    Apply failed patch in the repo after upgrade is done
>>
>> And add "FAILED:" in subject when failed. So that the user can go on
>> working based on the commit.
> 
> AUH will simple leave all commits in the repo, succeessful and failed. Even 
> successful upgrades may need further work before they can be submitted. It's 
> useful to mark failed ones as problematic though, so this change can be 
> partially taken.

Please see my reply to 8/12, the commit will be removed by the next recipe's
run when upgrade multiple recipes:

$ auh less git

less's commit will be removed by git's run.

// Robert

> 
> 
> Alex
> 

Re: [PATCH 09/12] upgradehelper.py: use git user in commit when recipes are specified

[Alexander Kanavin]

On 12/08/2017 03:51 AM, Robert Yang wrote:

>> I'm not sure about this one. Commits shouldn't be created 
>> automatically on a person's behalf and then also signed-off - I'd say 
>> you need to first look at them.
> 
> IMHO, Auh is a helper, when user run:
> $ auh less
> 
> It helps create commits, should be the same as user run "git commit", 
> and the
> user should responsible for it. Or how about add a -S option:
> 
> -S, --SOB    Use git user as the author and add SOB.

Yeah, as long as maintainers don't send those raw commits straight to 
mailing list, it's okay. It's rare that such an auto-created commit 
doesn't need further work (e.g. to explain changes in patches or 
license, or other adjustments).

Alex

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Alexander Kanavin]

On 12/08/2017 04:00 AM, Robert Yang wrote:
> 
>> I'm not sure, the logical *without* this patch is:
>> 1) Do the commit when succeed or failed.
>> 2) git format-patch
>> 3) Remove the commit if failed, and leave the commit there if succeed
> 
> Sorry, "remove the comit if failed" is added by this patch, and why I 
> did this
> was because:
> $ auh glibc less bash gzip bzip2
> 
> All of them will be failed to upgrade if glibc fails since others 
> depends on
> glibc, so leave a failed commit in the repo would cause troubles when 
> upgrade
> multiple recipes, so we have to remove it when failed to let others can 
> upgrade, and then apply the failed commit back (if -f is specified) 
> after all of the
> recipes are done.

In my branch I've changed the approach altogether. Here's how it works:

1) attempt to upgrade the recipe using 'devtool upgrade'.
2) If that failed (usually because custom patches couldn't be 
auto-rebased), save the diagnostic info and move to the next recipe.
3) if succeeded, create a commit with changes and try to build the recipe
4) save the build diagnostic info (regardless of whether build succeeded 
or failed) and move on to next recipe

So two things happen:
1) commits are never removed, only created, even if the recipe then 
fails to build
2) after AUH is done, it produces a sequence of update commits and 
information on what couldn't be updated, and what couldn't be build. The 
maintainer then can choose to fix the build failures, or take the 
commits that cause build failures out of the branch or revert them. AUH 
does not anymore manipulate branches or commits.

I think it's a simpler and easier to understand approach. Yes, this 
means that an updated recipe that is close to the root of dependency 
tree can cause a cascade of build failures (e.g. glibc), but the update 
commits for everything else will still be created, and the maintainer 
can easily revert the failing updates, and re-run the build. What you think?

I can publish the code for you to look at, even though it's still work 
in progress, and I'd like to fix up a few more things first.


Alex

Re: [PATCH 09/12] upgradehelper.py: use git user in commit when recipes are specified

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

On 8 December 2017 at 14:07, Alexander Kanavin <
alexander.kanavin@linux.intel.com> wrote:

> On 12/08/2017 03:51 AM, Robert Yang wrote:
>
> I'm not sure about this one. Commits shouldn't be created automatically on
>>> a person's behalf and then also signed-off - I'd say you need to first look
>>> at them.
>>>
>>
>> IMHO, Auh is a helper, when user run:
>> $ auh less
>>
>> It helps create commits, should be the same as user run "git commit", and
>> the
>> user should responsible for it. Or how about add a -S option:
>>
>> -S, --SOB    Use git user as the author and add SOB.
>>
>
> Yeah, as long as maintainers don't send those raw commits straight to
> mailing list, it's okay. It's rare that such an auto-created commit doesn't
> need further work (e.g. to explain changes in patches or license, or other
> adjustments).


The lack of a SOB was intentional for exactly this reason.  No SOB means a
human didn't look at it *at all*/

Ross

[-- Attachment #2: Type: text/html, Size: 1611 bytes --]

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Robert Yang]

On 12/08/2017 10:18 PM, Alexander Kanavin wrote:
> On 12/08/2017 04:00 AM, Robert Yang wrote:
>>
>>> I'm not sure, the logical *without* this patch is:
>>> 1) Do the commit when succeed or failed.
>>> 2) git format-patch
>>> 3) Remove the commit if failed, and leave the commit there if succeed
>>
>> Sorry, "remove the comit if failed" is added by this patch, and why I did this
>> was because:
>> $ auh glibc less bash gzip bzip2
>>
>> All of them will be failed to upgrade if glibc fails since others depends on
>> glibc, so leave a failed commit in the repo would cause troubles when upgrade
>> multiple recipes, so we have to remove it when failed to let others can 
>> upgrade, and then apply the failed commit back (if -f is specified) after all 
>> of the
>> recipes are done.
> 
> In my branch I've changed the approach altogether. Here's how it works:
> 
> 1) attempt to upgrade the recipe using 'devtool upgrade'.
> 2) If that failed (usually because custom patches couldn't be auto-rebased), 
> save the diagnostic info and move to the next recipe.
> 3) if succeeded, create a commit with changes and try to build the recipe
> 4) save the build diagnostic info (regardless of whether build succeeded or 
> failed) and move on to next recipe
> 
> So two things happen:
> 1) commits are never removed, only created, even if the recipe then fails to build
> 2) after AUH is done, it produces a sequence of update commits and information 
> on what couldn't be updated, and what couldn't be build. The maintainer then can 
> choose to fix the build failures, or take the commits that cause build failures 
> out of the branch or revert them. AUH does not anymore manipulate branches or 
> commits.
> 
> I think it's a simpler and easier to understand approach. Yes, this means that 
> an updated recipe that is close to the root of dependency tree can cause a 
> cascade of build failures (e.g. glibc), but the update commits for everything 
> else will still be created, and the maintainer can easily revert the failing 
> updates, and re-run the build. What you think?

Yes, since glibc would causes others failed to build, so I removed the commited
during upgrading, and then apply it back, I think that this is more helpful than
leave the failed commit there and causes others failed.

// Robert

> 
> I can publish the code for you to look at, even though it's still work in 
> progress, and I'd like to fix up a few more things first.
> 
> 
> Alex
> 

Re: [PATCH 09/12] upgradehelper.py: use git user in commit when recipes are specified

[Robert Yang]

On 12/09/2017 12:20 AM, Burton, Ross wrote:
> On 8 December 2017 at 14:07, Alexander Kanavin 
> <alexander.kanavin@linux.intel.com <mailto:alexander.kanavin@linux.intel.com>> 
> wrote:
> 
>     On 12/08/2017 03:51 AM, Robert Yang wrote:
> 
>             I'm not sure about this one. Commits shouldn't be created
>             automatically on a person's behalf and then also signed-off - I'd
>             say you need to first look at them.
> 
> 
>         IMHO, Auh is a helper, when user run:
>         $ auh less
> 
>         It helps create commits, should be the same as user run "git commit",
>         and the
>         user should responsible for it. Or how about add a -S option:
> 
>         -S, --SOB    Use git user as the author and add SOB.
> 
> 
>     Yeah, as long as maintainers don't send those raw commits straight to
>     mailing list, it's okay. It's rare that such an auto-created commit doesn't
>     need further work (e.g. to explain changes in patches or license, or other
>     adjustments).
> 
> 
> The lack of a SOB was intentional for exactly this reason.  No SOB means a human 
> didn't look at it *at all*/

Makes sense, so let's drop this patch.

// Robert


> 
> Ross

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Alexander Kanavin]

On 12/11/2017 08:13 AM, Robert Yang wrote:

>> I think it's a simpler and easier to understand approach. Yes, this 
>> means that an updated recipe that is close to the root of dependency 
>> tree can cause a cascade of build failures (e.g. glibc), but the 
>> update commits for everything else will still be created, and the 
>> maintainer can easily revert the failing updates, and re-run the 
>> build. What you think?
> 
> Yes, since glibc would causes others failed to build, so I removed the 
> commited
> during upgrading, and then apply it back, I think that this is more 
> helpful than
> leave the failed commit there and causes others failed.

How about simply issuing 'git revert' after a build has failed? That's 
easier to implement than rearranging the order of commits, and the 
commit message can include a link to the build failure logs.

Alex

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Robert Yang]

On 12/12/2017 08:26 PM, Alexander Kanavin wrote:
> On 12/11/2017 08:13 AM, Robert Yang wrote:
> 
>>> I think it's a simpler and easier to understand approach. Yes, this means 
>>> that an updated recipe that is close to the root of dependency tree can cause 
>>> a cascade of build failures (e.g. glibc), but the update commits for 
>>> everything else will still be created, and the maintainer can easily revert 
>>> the failing updates, and re-run the build. What you think?
>>
>> Yes, since glibc would causes others failed to build, so I removed the commited
>> during upgrading, and then apply it back, I think that this is more helpful than
>> leave the failed commit there and causes others failed.
> 
> How about simply issuing 'git revert' after a build has failed? That's easier to 
> implement than rearranging the order of commits, and the commit message can 
> include a link to the build failure logs.

Sounds reasonable to me, I will update the patch.

// Robert

> 
> Alex
> 

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Alexander Kanavin]

On 12/13/2017 03:16 AM, Robert Yang wrote:

>> How about simply issuing 'git revert' after a build has failed? That's 
>> easier to implement than rearranging the order of commits, and the 
>> commit message can include a link to the build failure logs.
> 
> Sounds reasonable to me, I will update the patch.

Wait until I publish my devtool port, then if you could rebase on that, 
that'd be great!

Alex

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Robert Yang]

On 12/13/2017 09:18 PM, Alexander Kanavin wrote:
> On 12/13/2017 03:16 AM, Robert Yang wrote:
> 
>>> How about simply issuing 'git revert' after a build has failed? That's easier 
>>> to implement than rearranging the order of commits, and the commit message 
>>> can include a link to the build failure logs.
>>
>> Sounds reasonable to me, I will update the patch.
> 
> Wait until I publish my devtool port, then if you could rebase on that, that'd 
> be great!

Got it, thanks.

// Robert

> 
> Alex
> 

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Alexander Kanavin]

On 12/14/2017 04:13 AM, Robert Yang wrote:

>>>> How about simply issuing 'git revert' after a build has failed? 
>>>> That's easier to implement than rearranging the order of commits, 
>>>> and the commit message can include a link to the build failure logs.
>>>
>>> Sounds reasonable to me, I will update the patch.
>>
>> Wait until I publish my devtool port, then if you could rebase on 
>> that, that'd be great!
> 
> Got it, thanks.

I just sent the patches to yocto@ ML and published them on github - 
please do try them out, and rebase your patchset on top (some of it may 
no longer be relevant).


Alex

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Alexander Kanavin]

On 12/14/2017 06:38 PM, Alexander Kanavin wrote:

>> Got it, thanks.
> 
> I just sent the patches to yocto@ ML and published them on github - 
> please do try them out, and rebase your patchset on top (some of it may 
> no longer be relevant).

I did the rebase myself, and pushed the result here - please check and 
feel free to add anything else that you deem useful:

http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/log/?h=devel

Alex

Re: [PATCH 08/12] upgradehelper.py: clean repo only once when recipes are specified

[Robert Yang]

Hi Alexander,

Thank you very much, I will try it later:-).

// Robert

On 12/15/2017 09:35 PM, Alexander Kanavin wrote:
> On 12/14/2017 06:38 PM, Alexander Kanavin wrote:
> 
>>> Got it, thanks.
>>
>> I just sent the patches to yocto@ ML and published them on github - please do 
>> try them out, and rebase your patchset on top (some of it may no longer be 
>> relevant).
> 
> I did the rebase myself, and pushed the result here - please check and feel free 
> to add anything else that you deem useful:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/log/?h=devel
> 
> Alex
>