[Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail

[Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail

[Victor Huesca]

This patch-series provides a new data for daily-mails in addition to
autobuild results: packages with a higher upstream version detected.
The goal is to provide a more information to developers by in the
daily-mail. This will help developers to know if packages they maintains
have a newer version and should help to have latest version of packages
in buildroot. Of course, this notification should not be included in
the actual daily-mails since new version are not release every-day but
it could be enabled for example every week.


Victor Huesca (4):
  utils/daily-mail: new data: outdated packages
  utils/daily-mail: add outdated packages to the developer notification
  utils/daily-mail: add outdated-packages email formating part
  utils/daily-mail: enable outdated packages emails

 utils/daily-mail | 106 ++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 100 insertions(+), 6 deletions(-)

-- 
2.21.0

[Buildroot] [PATCH buildroot-test v2 1/4] utils/daily-mail: new data: outdated packages

[Victor Huesca]

This patch uses the buildroot pkg-stats json output to list all outdated
packages. This will allow to ease the track of new upstream version of
buildroot packages.

This patch requires the non-standard python module 'packaging' in order
to compare the version numbers. This module can be install via the
distro package manager as 'python-packaging'.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/utils/daily-mail b/utils/daily-mail
index cf9ebd5..c16af54 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -14,11 +14,14 @@ from collections import defaultdict
 import math
 import argparse
 import re
+import json
+from packaging import version
 
 sys.path.append(os.path.join(localconfig.brbase, "utils"))
 import getdeveloperlib  # noqa: E402
 
 RE_DATE = re.compile(r'^\d\d\d\d-\d\d-\d\d$')
+RE_HASH_40 = re.compile(r'.*[a-fA-F0-9]{40}.*')
 
 baseurl = "autobuild.buildroot.net"
 http_baseurl = "http://" + baseurl
@@ -367,6 +370,30 @@ def get_build_results_grouped_by_reason(db, datestr, branches):
     return results_by_reason
 
 
+def get_outdated_pkg(path):
+    with open(path, 'r') as f:
+        stats = json.load(f)
+    s = []
+    for name, pkg in stats['packages'].items():
+        status, latest_ver, p_id = pkg['latest_version']
+        cur_ver = pkg['current_version']
+        if status not in (2, 3):
+            continue  # an upstream version exists
+        if not cur_ver or not latest_ver:
+            continue  # both version and latest version are actual strings
+        if RE_HASH_40.match(cur_ver):
+            continue  # the version is a hash
+        if version.parse(str(cur_ver)) >= version.parse(str(latest_ver)):
+            continue  # up to date
+
+        s.append({'name': str(name),
+                  'id': p_id,
+                  'version': str(cur_ver),
+                  'upstream': str(latest_ver),
+                  'from': 'DISTRO' if status == 2 else 'GUESS'})
+    return sorted(s, key=lambda pkg: pkg['name'])
+
+
 def calculate_notifications(results):
     '''
     Prepare the notifications{} dict for the notifications to individual
-- 
2.21.0

[Buildroot] [PATCH buildroot-test v2 2/4] utils/daily-mail: add outdated packages to the developer notification

[Victor Huesca]

This patch provides the logic to add outdated packages to the
notification structure.
This will allow future patches to add this new data to the per-developer
emails.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index c16af54..e99f526 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -72,6 +72,7 @@ class Notification:
     def __init__(self):
         self.arch_notifications = defaultdict(list)
         self.package_notifications = defaultdict(list)
+        self.package_version_notification = []
 
 
 def get_mklist(basepath):
@@ -160,6 +161,23 @@ def add_package_notification(branch, notifications, build_result):
     build_result['orphan'] = orphan
 
 
+def add_outdated_pkg_notification(notifications, package):
+    '''
+    Add to the notifications{} dict notifications that are related to
+    package "maintainers".
+    '''
+    orphan = True
+    for dev in developers:
+        if package['name'] in dev.packages:
+            orphan = False
+            notif = notifications.setdefault(dev, Notification())
+            notif.package_version_notification.append(package)
+    if orphan:
+        package['orphan'] = True
+        notif = notifications.setdefault(get_orphan_developer(), Notification())
+        notif.package_version_notification.append(package)
+
+
 def shrink_str(string, length, align='right', fill='...'):
     '''
     Returns the `string` shrinked to fit `length` characters and with `fill`
@@ -394,7 +412,7 @@ def get_outdated_pkg(path):
     return sorted(s, key=lambda pkg: pkg['name'])
 
 
-def calculate_notifications(results):
+def calculate_notifications(results, outdated_pkg):
     '''
     Prepare the notifications{} dict for the notifications to individual
     developers, based on architecture developers, package developers,
@@ -408,6 +426,10 @@ def calculate_notifications(results):
                 continue
             add_arch_notification(branch, notifications, result)
             add_package_notification(branch, notifications, result)
+
+    for pkg in outdated_pkg:
+        add_outdated_pkg_notification(notifications, pkg)
+
     return notifications
 
 
@@ -449,7 +471,7 @@ def __main__():
         overall_stats = {}
         results = {}
         results_by_reason = {}
-    notifications = calculate_notifications(results)
+    notifications = calculate_notifications(results, [])
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
     smtp.login(localconfig.smtpuser, localconfig.smtppass)
-- 
2.21.0

[Buildroot] [PATCH buildroot-test v2 3/4] utils/daily-mail: add outdated-packages email formating part

[Victor Huesca]

This patch provides the formating part for the outdated packages.
This adds a new table after the autobuild results to inform about
packages having a newer version.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 48 ++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 46 insertions(+), 2 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index e99f526..58c4b2a 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -245,6 +245,34 @@ def show_results(results, show_status, show_orphan=False):
     return contents
 
 
+def show_outdated(packages, show_orphan=False):
+    contents = ''
+    # Header
+    contents += '{:^30} | {:^8} | {:^44} | {:^12} | {:^12}'.format('name', 'found by',
+                                                                   'link to release-monitoring.org',
+                                                                   'version', 'upstream')
+    if show_orphan:
+        contents += ' | {:^5}'.format('orph?')
+    contents += '\n{0:-^30}-+-{0:-^8}-+-{0:-^44}-+-{0:-^12}-+-{0:-^12}-'.format('')
+    if show_orphan:
+        contents += '+-{:-^5}-'.format('')
+    contents += '\n'
+
+    # Actual data
+    for pkg in packages:
+        name = shrink_str(pkg['name'], 30)
+        version = shrink_str(pkg['version'], 12)
+        upstream = shrink_str(pkg['upstream'], 12)
+        orphan_str = 'ORPH' if pkg.get('orphan') else ''
+        url = 'https://release-monitoring.org/project/{:0>5}'.format(pkg['id'])
+
+        contents += '{:>30} | {:^8} | {:0>5} | {:12} | {:12}'.format(name, pkg['from'], url, version, upstream)
+        if show_orphan:
+            contents += ' | {:4}'.format(orphan_str)
+        contents += '\n'
+    return contents
+
+
 def developers_email(smtp, branches, notifications, datestr, dry_run):
     '''
     Send the e-mails to the individual developers
@@ -282,6 +310,17 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
 
             contents += "\n"
 
+        outdated = notif.package_version_notification
+        if len(outdated) != 0:
+            contents += '\n'
+            contents += textwrap.fill("This is the list of packages for which a new version has been "
+                                      "detected and for which you are a registered developer. Please "
+                                      "help us improving the quality of Buildroot by bumping these "
+                                      "packages to their latest version. Thanks!")
+            contents += '\n\n'
+            contents += show_outdated(outdated, show_orphan=show_orphan)
+            contents += '\n'
+
         contents += "-- \n"
         contents += http_baseurl
         if dry_run:
@@ -319,7 +358,7 @@ def global_email_branch_result(results, results_by_reason, branch):
     return contents
 
 
-def global_email(smtp, results, results_by_reason, datestr, overall, dry_run):
+def global_email(smtp, results, results_by_reason, datestr, overall, outdated_pkg, dry_run):
     '''
     Send the global e-mail to the mailing list
     '''
@@ -341,6 +380,11 @@ def global_email(smtp, results, results_by_reason, datestr, overall, dry_run):
         if len(results[branch]) == 0:
             continue
         contents += global_email_branch_result(results[branch], results_by_reason[branch], branch)
+    if outdated_pkg:
+        contents += '\n\n'
+        contents += "Packages having a newer version\n"
+        contents += "===============================\n\n"
+        contents += show_outdated(outdated_pkg, show_orphan=True)
     contents += "\n"
     contents += "-- \n"
     contents += http_baseurl
@@ -479,7 +523,7 @@ def __main__():
         developers_email(smtp, branches, notifications, date_str, args.dry_run)
     if 'global' in email_dest:
         global_email(smtp, results, results_by_reason, date_str,
-                     overall_stats, args.dry_run)
+                     overall_stats, [], args.dry_run)
     smtp.quit()
 
 
-- 
2.21.0

[Buildroot] [PATCH buildroot-test v2 4/4] utils/daily-mail: enable outdated packages emails

[Victor Huesca]

This patch adds an 'outdated' entry to the program data. This entry
adds information about the outdated packages in the global and
developers mails.

A new entry in the 'localconfig' file named 'pkg-stats' that contains
the location of the JSON output of the buildroot's 'pkg-stats' script.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index 58c4b2a..4f670fb 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -484,7 +484,7 @@ def parse_args():
     parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)')
     parser.add_argument('--dest', action='store', nargs='+', choices=['dev', 'global'],
                         help='List of emails type to send (ie. global, dev)')
-    parser.add_argument('--data', action='store', nargs='+', choices=['autobuild'],
+    parser.add_argument('--data', action='store', nargs='+', choices=['autobuild', 'outdated'],
                         help='List of information to add in emails (blank separated)')
     args = parser.parse_args()
     if args.date and not RE_DATE.match(args.date):
@@ -501,7 +501,7 @@ def __main__():
         date_str = yesterday.strftime('%Y-%m-%d')
     branches = args.branches if args.branches else get_branches()
     email_dest = set(args.dest) if args.dest else {'dev', 'global'}
-    email_data = set(args.data) if args.data else {'autobuild'}
+    email_data = set(args.data) if args.data else {'autobuild', 'outdated'}
 
     db = _mysql.connect(host=localconfig.host,
                         user=localconfig.user,
@@ -515,7 +515,8 @@ def __main__():
         overall_stats = {}
         results = {}
         results_by_reason = {}
-    notifications = calculate_notifications(results, [])
+    outdate_pkg = get_outdated_pkg(localconfig.pkg_stats) if 'outdated' in email_data else []
+    notifications = calculate_notifications(results, outdate_pkg)
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
     smtp.login(localconfig.smtpuser, localconfig.smtppass)
@@ -523,7 +524,7 @@ def __main__():
         developers_email(smtp, branches, notifications, date_str, args.dry_run)
     if 'global' in email_dest:
         global_email(smtp, results, results_by_reason, date_str,
-                     overall_stats, [], args.dry_run)
+                     overall_stats, outdate_pkg, args.dry_run)
     smtp.quit()
 
 
-- 
2.21.0

[Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail

[Thomas Petazzoni]

Hello Victor,

On Mon,  5 Aug 2019 15:15:40 +0200
Victor Huesca <victor.huesca@bootlin.com> wrote:

> Victor Huesca (4):
>   utils/daily-mail: new data: outdated packages
>   utils/daily-mail: add outdated packages to the developer notification
>   utils/daily-mail: add outdated-packages email formating part
>   utils/daily-mail: enable outdated packages emails

Thanks a lot for this work, I have applied all 4 patches, with some
minor changes. I also did a few formatting improvements:

  https://git.buildroot.org/buildroot-test/commit/?id=ba42c3bdce31581dc9c270165ebe59393e91c51f
  https://git.buildroot.org/buildroot-test/commit/?id=2e8ae4662d473d67398bf97eaf5e40bcbbc000be
  https://git.buildroot.org/buildroot-test/commit/?id=e7ed66b8a9383de60ab59e8604df4215b1753d30

The e-mail of today (both global and per developer) was sent with your
patches applied, so those notifications about outdated packages have
been sent to many developers.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com