[PATCH 00/10][auh] Add support for buildhistory and minor fixes

[PATCH 00/10][auh] Add support for buildhistory and minor fixes

[Aníbal Limón]

This changes can be found at,

http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/log/?h=alimon/devel

Aníbal Limón (10):
  upgradehelper.py: Only run with one recipe or all.
  upgradehelper.py: Validate if upgrade is needed
  upgradehelper.py: Only send emails when patch file exist
  bitbake.py: Improve performance on env() call
  upgradehelper.py: Add own step for loading environment
  upgradehelper.py: UniverseUpdate don't abort if recipe checkpkg fails.
  bitbake.py: Rename bitbake_log.txt to bitbake_error_log
  upgradehelper.py: Remove unused references to Buildhistory class
  upgradehelper.py: Move upstream versioning code to UniverseUpdater
  upgradehelper: Add support for generate buildhistory recipe diff's

 bitbake.py       |  52 ++------
 buildhistory.py  |  75 ++++++++++++
 errors.py        |   6 +
 upgradehelper.py | 359 ++++++++++++++++++++++++++++++-------------------------
 4 files changed, 290 insertions(+), 202 deletions(-)
 create mode 100644 buildhistory.py

-- 
1.9.1

[PATCH 01/10][auh] upgradehelper.py: Only run with one recipe or all.

[Aníbal Limón]

Remove ability to run with several recipes because isn't make sense.

Supported modes are,

	$ ./upgradehelper.py all
	$ ./upgradehelper.py recipe --to_version VERSION

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 34 ++++++++++------------------------
 1 file changed, 10 insertions(+), 24 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index d6ac7fd..63b30e4 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -65,7 +65,7 @@ def parse_cmdline():
     parser = argparse.ArgumentParser(description='Package Upgrade Helper',
                                      formatter_class=argparse.RawTextHelpFormatter,
                                      epilog=help_text)
-    parser.add_argument("recipe", nargs="+", help="recipe to be upgraded")
+    parser.add_argument("recipe", help="recipe to be upgraded")
 
     parser.add_argument("-t", "--to_version",
                         help="version to upgrade the recipe to")
@@ -507,14 +507,6 @@ class Updater(object):
 
             self.statistics.update(self.pn, self.new_ver, self.maintainer, error)
 
-            if self.interactive and attempted_pkgs < total_pkgs:
-                I(" %s: Proceed to next recipe? (Y/n)" % self.pn)
-                answer = sys.stdin.readline().strip().upper()
-
-                if answer != 'Y' and answer != '':
-                    I("Aborted by user!")
-                    exit(0)
-
         if (attempted_pkgs > 1):
             I("%s" % self.statistics.pkg_stats())
             if self.send_email:
@@ -700,24 +692,18 @@ if __name__ == "__main__":
 
     settings, maintainer_override = parse_config_file(args.config_file)
 
-    if len(args.recipe) == 1 and args.recipe[0] == "all":
+    if args.recipe == "all":
         updater = UniverseUpdater()
         updater.run()
-    elif len(args.recipe) >= 1:
-        if len(args.recipe) == 1:
-            if not args.to_version:
-                E(" For upgrade only one recipe you must specify --to_version\n")
-                exit(1)
-
-            if not args.maintainer and args.send_emails:
-                E(" For upgrade only one recipe and send email you must specify --maintainer\n")
-                exit(1)
+    else:
+        if not args.to_version:
+            E(" For upgrade only one recipe you must specify --to_version\n")
+            exit(1)
 
-            pkg_list = [(args.recipe[0], args.to_version, args.maintainer)]
-        else:
-            pkg_list = []
-            for pkg in args.recipe:
-                pkg_list.append((pkg, None, None))
+        if not args.maintainer and args.send_emails:
+            E(" For upgrade only one recipe and send email you must specify --maintainer\n")
+            exit(1)
 
+        pkg_list = [(args.recipe, args.to_version, args.maintainer)]
         updater = Updater(args.auto_mode, args.send_emails, args.skip_compilation)
         updater.run(pkg_list)
-- 
1.9.1

[PATCH 02/10][auh] upgradehelper.py: Validate if upgrade is needed

[Aníbal Limón]

When run upgradehelper only with one recipe upgrade needed
validation isn't did.

[YOCTO #7947]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 63b30e4..f10d974 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -207,6 +207,9 @@ class Updater(object):
     def _detect_repo(self):
         self._get_env()
 
+        if self.env['PV'] == self.new_ver:
+            raise UpgradeNotNeededError
+
         if self.git is not None:
             return
 
@@ -496,9 +499,24 @@ class Updater(object):
                     step()
 
                 I(" %s: Upgrade SUCCESSFUL! Please test!" % self.pn)
-            except Error as e:
-                E(" %s: %s" % (self.pn, e.message))
-                E(" %s: Upgrade FAILED! Logs and/or file diffs are available in %s" % (self.pn, self.workdir))
+            except Exception as e:
+                if isinstance(e, UpgradeNotNeededError):
+                    I(" %s: %s" % (self.pn, e.message))
+                elif isinstance(e, UnsupportedProtocolError):
+                    I(" %s: %s" % (self.pn, e.message))
+                else:
+                    if not isinstance(e, Error):
+                        import traceback
+                        msg = "Failed(unknown error)\n" + traceback.format_exc()
+                        e = Error(message=msg)
+                        error = e
+
+                    E(" %s: %s" % (self.pn, e.message))
+
+                    if os.listdir(self.workdir):
+                        E(" %s: Upgrade FAILED! Logs and/or file diffs are available in %s"
+                            % (self.pn, self.workdir))
+
                 error = e
 
             self._commit_changes()
-- 
1.9.1

[PATCH 03/10][auh] upgradehelper.py: Only send emails when patch file exist

[Aníbal Limón]

Remove old decision of send email based on error because only
makes sense to send email when patch file was created.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index f10d974..13fc7dd 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -358,7 +358,8 @@ class Updater(object):
             self.git.reset_hard(1)
             self.git.clean_untracked()
 
-        if self.send_email:
+        # only send email to Maintainer when patch file exist
+        if self.send_email and self.patch_file:
             mail_header = \
                 "Hello,\n\nYou are receiving this email because you are the maintainer\n" \
                 "of *%s* recipe and this is to let you know that the automatic attempt\n" \
@@ -383,13 +384,6 @@ class Updater(object):
                 "Any problem please contact Anibal Limon <anibal.limon@intel.com>.\n\n" \
                 "Regards,\nThe Upgrade Helper"
 
-            # only send email to Maintainer when is an error that can handle
-            if err and not isinstance(err, MaintainerError):
-                D( "%s: Don't send email to maintainer because the error was " \
-                   "%s and the information isn't useful, please review it." \
-                    % (self.pn, type(err).__name__))
-                return
-
             if self.maintainer in maintainer_override:
                 to_addr = maintainer_override[self.maintainer]
             else:
-- 
1.9.1

[PATCH 04/10][auh] bitbake.py: Improve performance on env() call

[Aníbal Limón]

When AUH load recipe enviroment use bitbake -e call and parses the
output using regex VAR=VALUE.

For improve performance discard all the comments on cmd output adding
a output_filter using grep.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 bitbake.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/bitbake.py b/bitbake.py
index 3631896..54389fb 100644
--- a/bitbake.py
+++ b/bitbake.py
@@ -42,7 +42,7 @@ class Bitbake(object):
         self.log_dir = None
         super(Bitbake, self).__init__()
 
-    def _cmd(self, recipe, options=None, env_var=None):
+    def _cmd(self, recipe, options=None, env_var=None, output_filter=None):
         cmd = ""
         if env_var is not None:
             cmd += env_var + " "
@@ -52,6 +52,9 @@ class Bitbake(object):
 
         cmd += recipe
 
+        if output_filter is not None:
+            cmd += ' |  grep ' + output_filter
+
         os.chdir(self.build_dir)
 
         try:
@@ -74,7 +77,7 @@ class Bitbake(object):
         return os.path.join(self.log_dir, "bitbake_log.txt")
 
     def env(self, recipe):
-        return self._cmd(recipe, "-e")
+        return self._cmd(recipe, "-e", output_filter="-v \"^#\"")
 
     def fetch(self, recipe):
         return self._cmd(recipe, "-c fetch")
-- 
1.9.1

[PATCH 05/10][auh] upgradehelper.py: Add own step for loading environment

[Aníbal Limón]

Instead of load environment in detecting git repository use own
function making easy to debug.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 13fc7dd..9119cb0 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -147,6 +147,7 @@ class Updater(object):
 
         self.upgrade_steps = [
             (self._create_workdir, None),
+            (self._get_env, "Loading environment ..."),
             (self._detect_repo, "Detecting git repository location ..."),
             (self._clean_repo, "Cleaning git repository of temporary branch ..."),
             (self._detect_recipe_type, None),
@@ -205,18 +206,16 @@ class Updater(object):
                 os.remove(os.path.join(self.workdir, f))
 
     def _detect_repo(self):
-        self._get_env()
-
-        if self.env['PV'] == self.new_ver:
+        if self.env['PKGV'] == self.new_ver:
             raise UpgradeNotNeededError
 
-        if self.git is not None:
+        # UniverseUpdater use git poky respository 
+        if isinstance(self.git, UniverseUpdater):
             return
 
         self.git = Git(self.recipe_dir)
 
         stdout = self.git.status()
-
         if stdout != "":
             if self.interactive:
                 W(" %s: git repository has uncommited work which will be dropped! Proceed? (y/N)" % self.pn)
@@ -527,6 +526,8 @@ class Updater(object):
 class UniverseUpdater(Updater):
     def __init__(self):
         Updater.__init__(self, True, True)
+
+        # XXX: assume that the poky directory is the first entry in the PATH
         self.git = Git(os.path.dirname(os.getenv('PATH', False).split(':')[0]))
 
         # read history file
-- 
1.9.1

[PATCH 06/10][auh] upgradehelper.py: UniverseUpdate don't abort if recipe checkpkg fails.

[Aníbal Limón]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 9119cb0..7bc8eed 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -285,10 +285,7 @@ class Updater(object):
                     C(" \"distrodata.bbclass\" not inherited. Consider adding "
                       "the following to your local.conf:\n\n"
                       "INHERIT =+ \"distrodata\"\n")
-                else:
-                    C(line)
-
-            exit(1)
+                    exit(1)
 
     def _parse_checkpkg_file(self, file_path):
         import csv
-- 
1.9.1

[PATCH 07/10][auh] bitbake.py: Rename bitbake_log.txt to bitbake_error_log

[Aníbal Limón]

Bitbake log only is written when error appears, rename it to be
consistent with information into the log.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 bitbake.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/bitbake.py b/bitbake.py
index 54389fb..e23dc28 100644
--- a/bitbake.py
+++ b/bitbake.py
@@ -36,6 +36,8 @@ for path in os.environ["PATH"].split(':'):
         sys.path.insert(0, os.path.join(path, "../lib"))
         import bb
 
+BITBAKE_ERROR_LOG = 'bitbake_error_log.txt'
+
 class Bitbake(object):
     def __init__(self, build_dir):
         self.build_dir = build_dir
@@ -63,7 +65,7 @@ class Bitbake(object):
             D("%s returned:\n%s" % (cmd, e.__str__()))
 
             if self.log_dir is not None and os.path.exists(self.log_dir):
-                with open(os.path.join(self.log_dir, "bitbake_log.txt"), "w+") as log:
+                with open(os.path.join(self.log_dir, BITBAKE_ERROR_LOG), "w+") as log:
                     log.write(e.stdout)
 
             raise Error("\'" + cmd + "\' failed", e.stdout, e.stderr)
@@ -74,7 +76,7 @@ class Bitbake(object):
         self.log_dir = dir
 
     def get_stdout_log(self):
-        return os.path.join(self.log_dir, "bitbake_log.txt")
+        return os.path.join(self.log_dir, BITBAKE_ERROR_LOG)
 
     def env(self, recipe):
         return self._cmd(recipe, "-e", output_filter="-v \"^#\"")
-- 
1.9.1

[PATCH 08/10][auh] upgradehelper.py: Remove unused references to Buildhistory class

[Aníbal Limón]

Buildhistory class isn't implemented so removed related code.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 11 +----------
 1 file changed, 1 insertion(+), 10 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 7bc8eed..a31d041 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -43,7 +43,7 @@ from datetime import date
 import shutil
 from errors import *
 from git import Git
-from bitbake import Bitbake, BuildHistory
+from bitbake import Bitbake
 from emailhandler import Email
 from statistics import Statistics
 from recipe import Recipe
@@ -135,7 +135,6 @@ class Updater(object):
             os.mkdir(self.uh_work_dir)
 
         self.bb = Bitbake(get_build_dir())
-        self.buildhistory = BuildHistory(get_build_dir())
         self.git = None
         self.author_email = settings.get('from', 'uh@not.set')
         self.author = "Upgrade Helper <%s>" % self.author_email
@@ -266,14 +265,6 @@ class Updater(object):
             I(" %s: compiling for %s ..." % (self.pn, machine))
             self.recipe.compile(machine)
 
-    def _review(self):
-        # Check build_history
-        if not self.skip_compilation:
-            I(" %s: Checking buildhistory ..." % self.pn)
-            self.buildhistory.set_work_dir(self.workdir)
-            if self.buildhistory.diff(len(self.machines)):
-               I(" %s: Wrote buildhistory-diff output ..." % self.pn)
-
     def _check_upstream_versions(self, packages=[("universe", None, None)]):
         I(" Fetching upstream version(s) ...")
 
-- 
1.9.1

[PATCH 09/10][auh] upgradehelper.py: Move upstream versioning code to UniverseUpdater

[Aníbal Limón]

Reorder code related to upstream versioning detection no makes sense
to have in Updater because only in UniverseUpdater is used.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 161 ++++++++++++++++++++++++++-----------------------------
 1 file changed, 77 insertions(+), 84 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index a31d041..4d8685c 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -265,63 +265,12 @@ class Updater(object):
             I(" %s: compiling for %s ..." % (self.pn, machine))
             self.recipe.compile(machine)
 
-    def _check_upstream_versions(self, packages=[("universe", None, None)]):
-        I(" Fetching upstream version(s) ...")
-
-        try:
-            self.bb.checkpkg(" ".join([p[0] for p in packages]))
-        except Error as e:
-            for line in e.stdout.split('\n'):
-                if line.find("ERROR: Task do_checkpkg does not exist") == 0:
-                    C(" \"distrodata.bbclass\" not inherited. Consider adding "
-                      "the following to your local.conf:\n\n"
-                      "INHERIT =+ \"distrodata\"\n")
-                    exit(1)
-
-    def _parse_checkpkg_file(self, file_path):
-        import csv
-
-        pkgs_list = []
-
-        with open(file_path, "r") as f:
-            reader = csv.reader(f, delimiter='\t')
-            for row in reader:
-                if reader.line_num == 1: # skip header line
-                    continue
-
-                pn = row[0]
-                cur_ver = row[1]
-                next_ver = row[2]
-                status = row[11]
-                maintainer = row[14]
-                no_upgrade_reason = row[15]
-
-                if status == 'UPDATE' and not no_upgrade_reason:
-                    pkgs_list.append((pn, next_ver, maintainer))
-                else:
-                    if no_upgrade_reason:
-                        D(" Skip package %s (status = %s, current version = %s," \
-                            " next version = %s, no upgrade reason = %s)" %
-                            (pn, status, cur_ver, next_ver, no_upgrade_reason))
-                    else:
-                        D(" Skip package %s (status = %s, current version = %s," \
-                            " next version = %s)" %
-                            (pn, status, cur_ver, next_ver))
-        return pkgs_list
-
     def _get_packages_to_upgrade(self, packages=None):
         if packages is None:
-            return []
-
-        if len(packages) == 1:
-            # if user specified the version to upgrade to, just return the
-            # tuple intact
-            if packages[0][1] is not None:
-                return packages
-
-        self._check_upstream_versions(packages)
-
-        return self._parse_checkpkg_file(get_build_dir() + "/tmp/log/checkpkg.csv")
+            I( "Nothing to upgrade")
+            exit(0)
+        else:
+            return packages
 
     # this function will be called at the end of each recipe upgrade
     def pkg_upgrade_handler(self, err):
@@ -530,6 +479,75 @@ class UniverseUpdater(Updater):
                                                         line.split(',')[3],
                                                         line.split(',')[4]]
 
+    def _update_master(self):
+        I(" Drop all uncommited changes (including untracked) ...")
+        self.git.reset_hard()
+        self.git.clean_untracked()
+
+        self.git.checkout_branch("master")
+        try:
+            self.git.delete_branch("upgrades")
+        except Error:
+            pass
+        I(" Sync master ...")
+        self.git.pull()
+        self.git.create_branch("upgrades")
+
+    def _prepare(self):
+        if settings.get("clean_sstate", "no") == "yes" and \
+                os.path.exists(os.path.join(get_build_dir(), "sstate-cache")):
+            I(" Removing sstate directory ...")
+            shutil.rmtree(os.path.join(get_build_dir(), "sstate-cache"))
+        if settings.get("clean_tmp", "no") == "yes" and \
+                os.path.exists(os.path.join(get_build_dir(), "tmp")):
+            I(" Removing tmp directory ...")
+            shutil.rmtree(os.path.join(get_build_dir(), "tmp"))
+
+
+    def _check_upstream_versions(self, packages=[("universe", None, None)]):
+        I(" Fetching upstream version(s) ...")
+
+        try:
+            self.bb.checkpkg(" ".join([p[0] for p in packages]))
+        except Error as e:
+            for line in e.stdout.split('\n'):
+                if line.find("ERROR: Task do_checkpkg does not exist") == 0:
+                    C(" \"distrodata.bbclass\" not inherited. Consider adding "
+                      "the following to your local.conf:\n\n"
+                      "INHERIT =+ \"distrodata\"\n")
+                    exit(1)
+
+    def _parse_checkpkg_file(self, file_path):
+        import csv
+
+        pkgs_list = []
+
+        with open(file_path, "r") as f:
+            reader = csv.reader(f, delimiter='\t')
+            for row in reader:
+                if reader.line_num == 1: # skip header line
+                    continue
+
+                pn = row[0]
+                cur_ver = row[1]
+                next_ver = row[2]
+                status = row[11]
+                maintainer = row[14]
+                no_upgrade_reason = row[15]
+
+                if status == 'UPDATE' and not no_upgrade_reason:
+                    pkgs_list.append((pn, next_ver, maintainer))
+                else:
+                    if no_upgrade_reason:
+                        D(" Skip package %s (status = %s, current version = %s," \
+                            " next version = %s, no upgrade reason = %s)" %
+                            (pn, status, cur_ver, next_ver, no_upgrade_reason))
+                    else:
+                        D(" Skip package %s (status = %s, current version = %s," \
+                            " next version = %s)" %
+                            (pn, status, cur_ver, next_ver))
+        return pkgs_list
+
     # checks if maintainer is in whitelist and that the recipe itself is not
     # blacklisted: python, gcc, etc. Also, check the history if the recipe
     # hasn't already been tried
@@ -580,30 +598,6 @@ class UniverseUpdater(Updater):
 
         return True
 
-    def update_master(self):
-        I(" Drop all uncommited changes (including untracked) ...")
-        self.git.reset_hard()
-        self.git.clean_untracked()
-
-        self.git.checkout_branch("master")
-        try:
-            self.git.delete_branch("upgrades")
-        except Error:
-            pass
-        I(" Sync master ...")
-        self.git.pull()
-        self.git.create_branch("upgrades")
-
-    def prepare(self):
-        if settings.get("clean_sstate", "no") == "yes" and \
-                os.path.exists(os.path.join(get_build_dir(), "sstate-cache")):
-            I(" Removing sstate directory ...")
-            shutil.rmtree(os.path.join(get_build_dir(), "sstate-cache"))
-        if settings.get("clean_tmp", "no") == "yes" and \
-                os.path.exists(os.path.join(get_build_dir(), "tmp")):
-            I(" Removing tmp directory ...")
-            shutil.rmtree(os.path.join(get_build_dir(), "tmp"))
-
     def _get_packages_to_upgrade(self, packages=None):
         last_date_checked = None
         last_master_commit = None
@@ -649,7 +643,7 @@ class UniverseUpdater(Updater):
 
         return pkgs_list
 
-    def update_history(self, pn, new_ver, maintainer, upgrade_status):
+    def _update_history(self, pn, new_ver, maintainer, upgrade_status):
         with open(self.history_file + ".tmp", "w+") as tmp_file:
             if os.path.exists(self.history_file):
                 with open(self.history_file) as history:
@@ -661,15 +655,14 @@ class UniverseUpdater(Updater):
                            upgrade_status + "\n")
         os.rename(self.history_file + ".tmp", self.history_file)
 
-    # overriding the base method
     def pkg_upgrade_handler(self, err):
         super(UniverseUpdater, self).pkg_upgrade_handler(err)
-        self.update_history(self.pn, self.new_ver, self.maintainer,
+        self._update_history(self.pn, self.new_ver, self.maintainer,
                 self._get_status_msg(err))
 
     def run(self):
-        self.update_master()
-        self.prepare()
+        self._update_master()
+        self._prepare()
         super(UniverseUpdater, self).run()
 
 def close_child_processes(signal_id, frame):
-- 
1.9.1

[PATCH 10/10][auh] upgradehelper: Add support for generate buildhistory recipe diff's

[Aníbal Limón]

Now AUH generates buildhistory diff when recipe was upgraded
successfully.

For enable this feature set into local.conf.

	INHERIT += "buildhistory"
	BUILDHISTORY_COMMIT = "1"

Summary of the changes,

	- bitbake.py: Enable environment generation without recipe,
          removes Buildhistory class.
	- buildhistory.py: Add buildhistory class that generates initial
          buildhistory revision and keeps track of build revisions for
          generate diff when build is successful.
        - upgradehelper.py: Add buildhistory steps for upgrade, add
          support for detect when buildhistory is enabled loading the
          environment without recipe.

[YOCTO #7175]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 bitbake.py       |  41 ++------------------
 buildhistory.py  |  75 ++++++++++++++++++++++++++++++++++++
 errors.py        |   6 +++
 upgradehelper.py | 115 ++++++++++++++++++++++++++++++++++++++++---------------
 4 files changed, 170 insertions(+), 67 deletions(-)
 create mode 100644 buildhistory.py

diff --git a/bitbake.py b/bitbake.py
index e23dc28..a1587ce 100644
--- a/bitbake.py
+++ b/bitbake.py
@@ -44,7 +44,7 @@ class Bitbake(object):
         self.log_dir = None
         super(Bitbake, self).__init__()
 
-    def _cmd(self, recipe, options=None, env_var=None, output_filter=None):
+    def _cmd(self, recipe=None, options=None, env_var=None, output_filter=None):
         cmd = ""
         if env_var is not None:
             cmd += env_var + " "
@@ -52,7 +52,8 @@ class Bitbake(object):
         if options is not None:
             cmd += options + " "
 
-        cmd += recipe
+        if recipe is not None:
+            cmd += recipe
 
         if output_filter is not None:
             cmd += ' |  grep ' + output_filter
@@ -78,7 +79,7 @@ class Bitbake(object):
     def get_stdout_log(self):
         return os.path.join(self.log_dir, BITBAKE_ERROR_LOG)
 
-    def env(self, recipe):
+    def env(self, recipe=None):
         return self._cmd(recipe, "-e", output_filter="-v \"^#\"")
 
     def fetch(self, recipe):
@@ -104,37 +105,3 @@ class Bitbake(object):
 
     def dependency_graph(self, package_list):
         return self._cmd(package_list, "-g")
-
-class BuildHistory(object):
-    def __init__(self, build_dir):
-        self.build_dir = build_dir
-        self.work_dir = None
-
-    def set_work_dir(self, work_dir):
-        self.work_dir = work_dir
-
-    # Return True if buildhistory-diff gives output
-    def diff(self, revision_steps):
-        os.chdir(self.build_dir)
-        cmd = "buildhistory-diff HEAD~" + str(revision_steps)
-
-        try:
-            stdout, stderr = bb.process.run(cmd)
-            # Write diff output to log file if there is any
-
-            if stdout and os.path.exists(self.work_dir):
-                with open(os.path.join(self.work_dir, "buildhistory.txt"), "w+") as log:
-                    log.write(stdout)
-                return True
-        except bb.process.ExecutionError as e:
-            for line in e.stdout.split('\n'):
-                if line.find("Buildhistory directory \"buildhistory/\" does not exist") == 0:
-                    C(" \"buildhistory.bbclass\" not inherited. Consider adding "
-                      "the following to your local.conf:\n\n"
-                      "INHERIT =+ \"buildhistory\"\n"
-                      "BUILDHISTORY_COMMIT = \"1\"\n\n"
-                      "Do not remove any other inherited class in the process (e.g. distrodata)\n")
-                    exit(1)
-
-        return False
-
diff --git a/buildhistory.py b/buildhistory.py
new file mode 100644
index 0000000..1732f23
--- /dev/null
+++ b/buildhistory.py
@@ -0,0 +1,75 @@
+#!/usr/bin/env python
+# vim: set ts=4 sw=4 et:
+#
+# Copyright (c) 2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+#
+
+import os
+import logging as log
+from logging import info as I
+from logging import debug as D
+from logging import error as E
+from logging import critical as C
+import sys
+from errors import *
+
+from bitbake import *
+from git import Git
+
+os.environ['BB_ENV_EXTRAWHITE'] = os.environ['BB_ENV_EXTRAWHITE'] + \
+                                    " BUILDHISTORY_DIR"
+
+class BuildHistory(object):
+    def __init__(self, bb, pn, workdir):
+        self.bb = bb
+        self.pn = pn
+        self.workdir = workdir
+        self.revs = []
+
+        self.buildhistory_dir = os.path.join(self.workdir, 'buildhistory')
+        if not os.path.exists(self.buildhistory_dir):
+            os.mkdir(self.buildhistory_dir)
+
+        self.git = Git(self.buildhistory_dir)
+
+        os.environ["BUILDHISTORY_DIR"] = self.buildhistory_dir
+
+    def init(self, machines):
+        self.bb.cleanall(self.pn)
+        for machine in machines:
+            self.bb.complete(self.pn, machine)
+            self.revs.append(self.git.last_commit("master"))
+
+    def add(self):
+        self.revs.append(self.git.last_commit("master"))
+
+    def diff(self):
+        rev_initial = self.revs[0]
+        rev_final = self.revs[-1]
+
+        cmd = "buildhistory-diff -a -p %s %s %s"  % (self.buildhistory_dir, 
+                rev_initial, rev_final)
+
+        try:
+            stdout, stderr = bb.process.run(cmd)
+
+            if stdout and os.path.exists(self.workdir):
+                with open(os.path.join(self.workdir, "buildhistory-diff.txt"),
+                        "w+") as log:
+                    log.write(stdout)
+        except bb.process.ExecutionError as e:
+            W( "%s: Buildhistory checking fails\n%s" % (self.pn, e.stdout))
diff --git a/errors.py b/errors.py
index 7194944..1504fa5 100644
--- a/errors.py
+++ b/errors.py
@@ -85,3 +85,9 @@ class UpgradeNotNeededError(Error):
     def __str__(self):
         return "Failed(up to date)"
 
+class EmptyEnvError(Error):
+    def __init__(self, stdout):
+        super(EmptyEnvError, self).__init__("Empty environment returned", stdout)
+
+    def __str__(self):
+        return "Failed(get_env)"
diff --git a/upgradehelper.py b/upgradehelper.py
index 4d8685c..a8bc5ec 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -44,6 +44,7 @@ import shutil
 from errors import *
 from git import Git
 from bitbake import Bitbake
+from buildhistory import BuildHistory
 from emailhandler import Email
 from statistics import Statistics
 from recipe import Recipe
@@ -145,23 +146,40 @@ class Updater(object):
         self.machines = settings.get('machines', 'qemux86 qemux86-64 qemuarm qemumips qemuppc').split()
 
         self.upgrade_steps = [
+            (self._load_env, "Loading environment ..."),
             (self._create_workdir, None),
-            (self._get_env, "Loading environment ..."),
             (self._detect_repo, "Detecting git repository location ..."),
             (self._clean_repo, "Cleaning git repository of temporary branch ..."),
             (self._detect_recipe_type, None),
+            (self._buildhistory_init, None),
             (self._unpack_original, "Fetch & unpack original version ..."),
             (self._rename, "Renaming recipes, reset PR (if exists) ..."),
             (self._cleanall, "Clean all ..."),
             (self._fetch, "Fetch new version (old checksums) ..."),
-            (self._compile, None)
+            (self._compile, None),
+            (self._buildhistory_diff, None)
         ]
 
+        try:
+            self.base_env = self._get_env()
+        except EmptyEnvError as e:
+            import traceback
+            E( " %s\n%s" % (e.message, traceback.format_exc()))
+            E( " Bitbake output:\n%s" % (e.stdout))
+            exit(1)
+        self.buildhistory_enabled = self._buildhistory_is_enabled()
+
         self.email_handler = Email(settings)
         self.statistics = Statistics()
 
-    def _get_env(self):
-        stdout = self.bb.env(self.pn)
+    def _get_status_msg(self, err):
+        if err:
+            return str(err)
+        else:
+            return "Succeeded"
+
+    def _get_env(self, pn=None):
+        stdout = self.bb.env(pn)
 
         assignment = re.compile("^([^ \t=]*)=(.*)")
         bb_env = dict()
@@ -173,38 +191,46 @@ class Updater(object):
 
                 bb_env[m.group(1)] = m.group(2).strip("\"")
 
-        self.env = bb_env
-        self.recipe_dir = os.path.dirname(self.env['FILE'])
+        if not bb_env:
+            raise EmptyEnvError(stdout)
 
-    def _detect_recipe_type(self):
-        if self.env['SRC_URI'].find("ftp://") != -1 or  \
-                self.env['SRC_URI'].find("http://") != -1 or \
-                self.env['SRC_URI'].find("https://") != -1:
-            recipe = Recipe
-        elif self.env['SRC_URI'].find("git://") != -1:
-            recipe = GitRecipe
-        else:
-            raise UnsupportedProtocolError
+        return bb_env
 
-        self.recipe = recipe(self.env, self.new_ver, self.interactive, self.workdir,
-                             self.recipe_dir, self.bb, self.git)
+    def _buildhistory_is_enabled(self):
+        enabled = False
 
-    def _get_status_msg(self, err):
-        if err:
-            return str(err)
-        else:
-            return "Succeeded"
+        if 'buildhistory' in self.base_env['INHERIT']:
+            if not 'BUILDHISTORY_COMMIT' in self.base_env:
+                E(" Buildhistory was enabled but need"\
+                        " BUILDHISTORY_COMMIT=1 please set.")
+                exit(1)
+
+            if not self.base_env['BUILDHISTORY_COMMIT'] == '1':
+                E(" Buildhistory was enabled but need"\
+                        " BUILDHISTORY_COMMIT=1 please set.")
+                exit(1)
+
+            if self.skip_compilation:
+                W(" Buildhistory disabled because user" \
+                        " skip compilation!")
+            else:
+                enabled = True
+
+        return enabled
+
+    def _load_env(self):
+        self.env = self._get_env(self.pn)
 
     def _create_workdir(self):
         self.workdir = os.path.join(self.uh_work_dir, self.pn)
 
-        if not os.path.exists(self.workdir):
-            os.mkdir(self.workdir)
-        else:
-            for f in os.listdir(self.workdir):
-                os.remove(os.path.join(self.workdir, f))
+        if os.path.exists(self.workdir):
+            shutil.rmtree(self.workdir)
+        os.mkdir(self.workdir)
 
     def _detect_repo(self):
+        self.recipe_dir = os.path.dirname(self.env['FILE'])
+
         if self.env['PKGV'] == self.new_ver:
             raise UpgradeNotNeededError
 
@@ -227,7 +253,7 @@ class Updater(object):
             self.git.reset_hard()
             self.git.clean_untracked()
 
-            self._get_env()
+            self.env = self._get_env(self.pn)
 
     def _clean_repo(self):
         try:
@@ -239,14 +265,34 @@ class Updater(object):
         except:
             pass
 
+    def _detect_recipe_type(self):
+        if self.env['SRC_URI'].find("ftp://") != -1 or  \
+                self.env['SRC_URI'].find("http://") != -1 or \
+                self.env['SRC_URI'].find("https://") != -1:
+            recipe = Recipe
+        elif self.env['SRC_URI'].find("git://") != -1:
+            recipe = GitRecipe
+        else:
+            raise UnsupportedProtocolError
+
+        self.recipe = recipe(self.env, self.new_ver, self.interactive, self.workdir,
+                             self.recipe_dir, self.bb, self.git)
+
+    def _buildhistory_init(self):
+        if self.buildhistory_enabled == False:
+            return
+
+        self.buildhistory = BuildHistory(self.bb, self.pn, self.workdir)
+        I(" %s: Initial buildhistory for %s ..." % (self.pn, self.machines))
+        self.buildhistory.init(self.machines)
+
     def _unpack_original(self):
         self.recipe.unpack()
 
     def _rename(self):
         self.recipe.rename()
 
-        # fetch new environment
-        self._get_env()
+        self.env = self._get_env(self.pn)
 
         self.recipe.update_env(self.env)
 
@@ -264,6 +310,15 @@ class Updater(object):
         for machine in self.machines:
             I(" %s: compiling for %s ..." % (self.pn, machine))
             self.recipe.compile(machine)
+            if self.buildhistory is not None:
+                self.buildhistory.add()
+
+    def _buildhistory_diff(self):
+        if self.buildhistory_enabled == False:
+            return
+
+        I(" %s: Checking buildhistory ..." % self.pn)
+        self.buildhistory.diff()
 
     def _get_packages_to_upgrade(self, packages=None):
         if packages is None:
-- 
1.9.1

Re: [PATCH 01/10][auh] upgradehelper.py: Only run with one recipe or all.

[Paul Eggleton]

On Wednesday 29 July 2015 15:50:44 Aníbal Limón wrote:
> Remove ability to run with several recipes because isn't make sense.
> 
> Supported modes are,
> 
> 	$ ./upgradehelper.py all
> 	$ ./upgradehelper.py recipe --to_version VERSION

I'm not sure I understand the reason for removing this - can you clarify?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 05/10][auh] upgradehelper.py: Add own step for loading environment

[Paul Eggleton]

On Wednesday 29 July 2015 15:50:48 Aníbal Limón wrote:
> Instead of load environment in detecting git repository use own
> function making easy to debug.

The changes here don't seem to entirely match up with the commit message.

> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> ---
>  upgradehelper.py | 11 ++++++-----
>  1 file changed, 6 insertions(+), 5 deletions(-)
> 
> diff --git a/upgradehelper.py b/upgradehelper.py
> index 13fc7dd..9119cb0 100755
> --- a/upgradehelper.py
> +++ b/upgradehelper.py
> @@ -147,6 +147,7 @@ class Updater(object):
> 
>          self.upgrade_steps = [
>              (self._create_workdir, None),
> +            (self._get_env, "Loading environment ..."),
>              (self._detect_repo, "Detecting git repository location ..."),
>              (self._clean_repo, "Cleaning git repository of temporary branch
> ..."), (self._detect_recipe_type, None),
> @@ -205,18 +206,16 @@ class Updater(object):
>                  os.remove(os.path.join(self.workdir, f))
> 
>      def _detect_repo(self):
> -        self._get_env()
> -
> -        if self.env['PV'] == self.new_ver:
> +        if self.env['PKGV'] == self.new_ver:
>              raise UpgradeNotNeededError
> 
> -        if self.git is not None:
> +        # UniverseUpdater use git poky respository
> +        if isinstance(self.git, UniverseUpdater):
>              return
> 
>          self.git = Git(self.recipe_dir)
> 
>          stdout = self.git.status()
> -
>          if stdout != "":
>              if self.interactive:
>                  W(" %s: git repository has uncommited work which will be
> dropped! Proceed? (y/N)" % self.pn) @@ -527,6 +526,8 @@ class
> Updater(object):
>  class UniverseUpdater(Updater):
>      def __init__(self):
>          Updater.__init__(self, True, True)
> +
> +        # XXX: assume that the poky directory is the first entry in the
> PATH self.git = Git(os.path.dirname(os.getenv('PATH',
> False).split(':')[0]))
> 
>          # read history file

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 10/10][auh] upgradehelper: Add support for generate buildhistory recipe diff's

[Paul Eggleton]

Hi Aníbal,

On Wednesday 29 July 2015 15:50:53 Aníbal Limón wrote:
> Now AUH generates buildhistory diff when recipe was upgraded
> successfully.
> 
> For enable this feature set into local.conf.
> 
> 	INHERIT += "buildhistory"
> 	BUILDHISTORY_COMMIT = "1"
> 
> Summary of the changes,
> 
> 	- bitbake.py: Enable environment generation without recipe,
>           removes Buildhistory class.
> 	- buildhistory.py: Add buildhistory class that generates initial
>           buildhistory revision and keeps track of build revisions for
>           generate diff when build is successful.
>         - upgradehelper.py: Add buildhistory steps for upgrade, add
>           support for detect when buildhistory is enabled loading the
>           environment without recipe.

So this looks OK, but I can't tell if it ensures you get the difference between 
successful builds rather than the difference to the last build - we need to 
ensure that a failed build of a recipe in between isn't considered as part of 
the comparison. In practice I guess this would mean storing the buildhistory 
revision of the last successful build for each recipe.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 10/10][auh] upgradehelper: Add support for generate buildhistory recipe diff's

[Aníbal Limón]

Hi Paul,

On 31/07/15 10:29, Paul Eggleton wrote:
> Hi Aníbal,
>
> On Wednesday 29 July 2015 15:50:53 Aníbal Limón wrote:
>> Now AUH generates buildhistory diff when recipe was upgraded
>> successfully.
>>
>> For enable this feature set into local.conf.
>>
>> 	INHERIT += "buildhistory"
>> 	BUILDHISTORY_COMMIT = "1"
>>
>> Summary of the changes,
>>
>> 	- bitbake.py: Enable environment generation without recipe,
>>            removes Buildhistory class.
>> 	- buildhistory.py: Add buildhistory class that generates initial
>>            buildhistory revision and keeps track of build revisions for
>>            generate diff when build is successful.
>>          - upgradehelper.py: Add buildhistory steps for upgrade, add
>>            support for detect when buildhistory is enabled loading the
>>            environment without recipe.
> So this looks OK, but I can't tell if it ensures you get the difference between
> successful builds rather than the difference to the last build - we need to
> ensure that a failed build of a recipe in between isn't considered as part of
> the comparison. In practice I guess this would mean storing the buildhistory
> revision of the last successful build for each recipe.

I only made a diff if the build success in all machines this guarantee 
that only
stores good information.

     alimon

>
> Cheers,
> Paul
>

Re: [PATCH 10/10][auh] upgradehelper: Add support for generate buildhistory recipe diff's

[Paul Eggleton]

On Friday 31 July 2015 10:39:37 Aníbal Limón wrote:
> Hi Paul,
> 
> On 31/07/15 10:29, Paul Eggleton wrote:
> > Hi Aníbal,
> > 
> > On Wednesday 29 July 2015 15:50:53 Aníbal Limón wrote:
> >> Now AUH generates buildhistory diff when recipe was upgraded
> >> successfully.
> >> 
> >> For enable this feature set into local.conf.
> >> 
> >> 	INHERIT += "buildhistory"
> >> 	BUILDHISTORY_COMMIT = "1"
> >> 
> >> Summary of the changes,
> >> 
> >> 	- bitbake.py: Enable environment generation without recipe,
> >> 	
> >>            removes Buildhistory class.
> >> 	
> >> 	- buildhistory.py: Add buildhistory class that generates initial
> >> 	
> >>            buildhistory revision and keeps track of build revisions for
> >>            generate diff when build is successful.
> >>          
> >>          - upgradehelper.py: Add buildhistory steps for upgrade, add
> >>          
> >>            support for detect when buildhistory is enabled loading the
> >>            environment without recipe.
> > 
> > So this looks OK, but I can't tell if it ensures you get the difference
> > between successful builds rather than the difference to the last build -
> > we need to ensure that a failed build of a recipe in between isn't
> > considered as part of the comparison. In practice I guess this would mean
> > storing the buildhistory revision of the last successful build for each
> > recipe.
> 
> I only made a diff if the build success in all machines this guarantee
> that only stores good information.

OK, I think I'm following now.

One other thing - I'm not particularly keen on the BB_ENV_EXTRAWHITE thing - 
can you perhaps query the build system for the path it's using instead?

CHeers,
Paul

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 10/10][auh] upgradehelper: Add support for generate buildhistory recipe diff's

[Aníbal Limón]

On 31/07/15 11:10, Paul Eggleton wrote:
> On Friday 31 July 2015 10:39:37 Aníbal Limón wrote:
>> Hi Paul,
>>
>> On 31/07/15 10:29, Paul Eggleton wrote:
>>> Hi Aníbal,
>>>
>>> On Wednesday 29 July 2015 15:50:53 Aníbal Limón wrote:
>>>> Now AUH generates buildhistory diff when recipe was upgraded
>>>> successfully.
>>>>
>>>> For enable this feature set into local.conf.
>>>>
>>>> 	INHERIT += "buildhistory"
>>>> 	BUILDHISTORY_COMMIT = "1"
>>>>
>>>> Summary of the changes,
>>>>
>>>> 	- bitbake.py: Enable environment generation without recipe,
>>>> 	
>>>>             removes Buildhistory class.
>>>> 	
>>>> 	- buildhistory.py: Add buildhistory class that generates initial
>>>> 	
>>>>             buildhistory revision and keeps track of build revisions for
>>>>             generate diff when build is successful.
>>>>           
>>>>           - upgradehelper.py: Add buildhistory steps for upgrade, add
>>>>           
>>>>             support for detect when buildhistory is enabled loading the
>>>>             environment without recipe.
>>> So this looks OK, but I can't tell if it ensures you get the difference
>>> between successful builds rather than the difference to the last build -
>>> we need to ensure that a failed build of a recipe in between isn't
>>> considered as part of the comparison. In practice I guess this would mean
>>> storing the buildhistory revision of the last successful build for each
>>> recipe.
>> I only made a diff if the build success in all machines this guarantee
>> that only stores good information.
> OK, I think I'm following now.
>
> One other thing - I'm not particularly keen on the BB_ENV_EXTRAWHITE thing -
> can you perhaps query the build system for the path it's using instead?

I modified this env var to pass BUILDHISTORY_DIR variable per recipe 
this enables
to save buildhistory into AUH workdir 
(build/upgrade-helper/work/recipe/buildhistory).

     alimon

>
> CHeers,
> Paul
>
> Cheers,
> Paul
>

Re: [PATCH 10/10][auh] upgradehelper: Add support for generate buildhistory recipe diff's

[Paul Eggleton]

On Friday 31 July 2015 11:19:04 Aníbal Limón wrote:
> On 31/07/15 11:10, Paul Eggleton wrote:
> > On Friday 31 July 2015 10:39:37 Aníbal Limón wrote:
> >> Hi Paul,
> >> 
> >> On 31/07/15 10:29, Paul Eggleton wrote:
> >>> Hi Aníbal,
> >>> 
> >>> On Wednesday 29 July 2015 15:50:53 Aníbal Limón wrote:
> >>>> Now AUH generates buildhistory diff when recipe was upgraded
> >>>> successfully.
> >>>> 
> >>>> For enable this feature set into local.conf.
> >>>> 
> >>>> 	INHERIT += "buildhistory"
> >>>> 	BUILDHISTORY_COMMIT = "1"
> >>>> 
> >>>> Summary of the changes,
> >>>> 
> >>>> 	- bitbake.py: Enable environment generation without recipe,
> >>>> 	
> >>>>             removes Buildhistory class.
> >>>> 	
> >>>> 	- buildhistory.py: Add buildhistory class that generates initial
> >>>> 	
> >>>>             buildhistory revision and keeps track of build revisions
> >>>>             for
> >>>>             generate diff when build is successful.
> >>>>           
> >>>>           - upgradehelper.py: Add buildhistory steps for upgrade, add
> >>>>           
> >>>>             support for detect when buildhistory is enabled loading the
> >>>>             environment without recipe.
> >>> 
> >>> So this looks OK, but I can't tell if it ensures you get the difference
> >>> between successful builds rather than the difference to the last build -
> >>> we need to ensure that a failed build of a recipe in between isn't
> >>> considered as part of the comparison. In practice I guess this would
> >>> mean
> >>> storing the buildhistory revision of the last successful build for each
> >>> recipe.
> >> 
> >> I only made a diff if the build success in all machines this guarantee
> >> that only stores good information.
> > 
> > OK, I think I'm following now.
> > 
> > One other thing - I'm not particularly keen on the BB_ENV_EXTRAWHITE thing
> > - can you perhaps query the build system for the path it's using instead?
> I modified this env var to pass BUILDHISTORY_DIR variable per recipe
> this enables
> to save buildhistory into AUH workdir
> (build/upgrade-helper/work/recipe/buildhistory).

Yes, I understand that, I just think using BB_ENV_EXTRAWHITE to pass this in 
is a horrible hack and I'd rather not do it that way.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre