* [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail
@ 2019-08-05 13:15 Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 1/4] utils/daily-mail: new data: outdated packages Victor Huesca
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Victor Huesca @ 2019-08-05 13:15 UTC (permalink / raw)
To: buildroot
This patch-series provides a new data for daily-mails in addition to
autobuild results: packages with a higher upstream version detected.
The goal is to provide a more information to developers by in the
daily-mail. This will help developers to know if packages they maintains
have a newer version and should help to have latest version of packages
in buildroot. Of course, this notification should not be included in
the actual daily-mails since new version are not release every-day but
it could be enabled for example every week.
Victor Huesca (4):
utils/daily-mail: new data: outdated packages
utils/daily-mail: add outdated packages to the developer notification
utils/daily-mail: add outdated-packages email formating part
utils/daily-mail: enable outdated packages emails
utils/daily-mail | 106 ++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 100 insertions(+), 6 deletions(-)
--
2.21.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH buildroot-test v2 1/4] utils/daily-mail: new data: outdated packages
2019-08-05 13:15 [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail Victor Huesca
@ 2019-08-05 13:15 ` Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 2/4] utils/daily-mail: add outdated packages to the developer notification Victor Huesca
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Victor Huesca @ 2019-08-05 13:15 UTC (permalink / raw)
To: buildroot
This patch uses the buildroot pkg-stats json output to list all outdated
packages. This will allow to ease the track of new upstream version of
buildroot packages.
This patch requires the non-standard python module 'packaging' in order
to compare the version numbers. This module can be install via the
distro package manager as 'python-packaging'.
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
utils/daily-mail | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/utils/daily-mail b/utils/daily-mail
index cf9ebd5..c16af54 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -14,11 +14,14 @@ from collections import defaultdict
import math
import argparse
import re
+import json
+from packaging import version
sys.path.append(os.path.join(localconfig.brbase, "utils"))
import getdeveloperlib # noqa: E402
RE_DATE = re.compile(r'^\d\d\d\d-\d\d-\d\d$')
+RE_HASH_40 = re.compile(r'.*[a-fA-F0-9]{40}.*')
baseurl = "autobuild.buildroot.net"
http_baseurl = "http://" + baseurl
@@ -367,6 +370,30 @@ def get_build_results_grouped_by_reason(db, datestr, branches):
return results_by_reason
+def get_outdated_pkg(path):
+ with open(path, 'r') as f:
+ stats = json.load(f)
+ s = []
+ for name, pkg in stats['packages'].items():
+ status, latest_ver, p_id = pkg['latest_version']
+ cur_ver = pkg['current_version']
+ if status not in (2, 3):
+ continue # an upstream version exists
+ if not cur_ver or not latest_ver:
+ continue # both version and latest version are actual strings
+ if RE_HASH_40.match(cur_ver):
+ continue # the version is a hash
+ if version.parse(str(cur_ver)) >= version.parse(str(latest_ver)):
+ continue # up to date
+
+ s.append({'name': str(name),
+ 'id': p_id,
+ 'version': str(cur_ver),
+ 'upstream': str(latest_ver),
+ 'from': 'DISTRO' if status == 2 else 'GUESS'})
+ return sorted(s, key=lambda pkg: pkg['name'])
+
+
def calculate_notifications(results):
'''
Prepare the notifications{} dict for the notifications to individual
--
2.21.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH buildroot-test v2 2/4] utils/daily-mail: add outdated packages to the developer notification
2019-08-05 13:15 [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 1/4] utils/daily-mail: new data: outdated packages Victor Huesca
@ 2019-08-05 13:15 ` Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 3/4] utils/daily-mail: add outdated-packages email formating part Victor Huesca
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Victor Huesca @ 2019-08-05 13:15 UTC (permalink / raw)
To: buildroot
This patch provides the logic to add outdated packages to the
notification structure.
This will allow future patches to add this new data to the per-developer
emails.
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
utils/daily-mail | 26 ++++++++++++++++++++++++--
1 file changed, 24 insertions(+), 2 deletions(-)
diff --git a/utils/daily-mail b/utils/daily-mail
index c16af54..e99f526 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -72,6 +72,7 @@ class Notification:
def __init__(self):
self.arch_notifications = defaultdict(list)
self.package_notifications = defaultdict(list)
+ self.package_version_notification = []
def get_mklist(basepath):
@@ -160,6 +161,23 @@ def add_package_notification(branch, notifications, build_result):
build_result['orphan'] = orphan
+def add_outdated_pkg_notification(notifications, package):
+ '''
+ Add to the notifications{} dict notifications that are related to
+ package "maintainers".
+ '''
+ orphan = True
+ for dev in developers:
+ if package['name'] in dev.packages:
+ orphan = False
+ notif = notifications.setdefault(dev, Notification())
+ notif.package_version_notification.append(package)
+ if orphan:
+ package['orphan'] = True
+ notif = notifications.setdefault(get_orphan_developer(), Notification())
+ notif.package_version_notification.append(package)
+
+
def shrink_str(string, length, align='right', fill='...'):
'''
Returns the `string` shrinked to fit `length` characters and with `fill`
@@ -394,7 +412,7 @@ def get_outdated_pkg(path):
return sorted(s, key=lambda pkg: pkg['name'])
-def calculate_notifications(results):
+def calculate_notifications(results, outdated_pkg):
'''
Prepare the notifications{} dict for the notifications to individual
developers, based on architecture developers, package developers,
@@ -408,6 +426,10 @@ def calculate_notifications(results):
continue
add_arch_notification(branch, notifications, result)
add_package_notification(branch, notifications, result)
+
+ for pkg in outdated_pkg:
+ add_outdated_pkg_notification(notifications, pkg)
+
return notifications
@@ -449,7 +471,7 @@ def __main__():
overall_stats = {}
results = {}
results_by_reason = {}
- notifications = calculate_notifications(results)
+ notifications = calculate_notifications(results, [])
smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
smtp.starttls()
smtp.login(localconfig.smtpuser, localconfig.smtppass)
--
2.21.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH buildroot-test v2 3/4] utils/daily-mail: add outdated-packages email formating part
2019-08-05 13:15 [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 1/4] utils/daily-mail: new data: outdated packages Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 2/4] utils/daily-mail: add outdated packages to the developer notification Victor Huesca
@ 2019-08-05 13:15 ` Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 4/4] utils/daily-mail: enable outdated packages emails Victor Huesca
2019-08-15 10:00 ` [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail Thomas Petazzoni
4 siblings, 0 replies; 6+ messages in thread
From: Victor Huesca @ 2019-08-05 13:15 UTC (permalink / raw)
To: buildroot
This patch provides the formating part for the outdated packages.
This adds a new table after the autobuild results to inform about
packages having a newer version.
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
utils/daily-mail | 48 ++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 46 insertions(+), 2 deletions(-)
diff --git a/utils/daily-mail b/utils/daily-mail
index e99f526..58c4b2a 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -245,6 +245,34 @@ def show_results(results, show_status, show_orphan=False):
return contents
+def show_outdated(packages, show_orphan=False):
+ contents = ''
+ # Header
+ contents += '{:^30} | {:^8} | {:^44} | {:^12} | {:^12}'.format('name', 'found by',
+ 'link to release-monitoring.org',
+ 'version', 'upstream')
+ if show_orphan:
+ contents += ' | {:^5}'.format('orph?')
+ contents += '\n{0:-^30}-+-{0:-^8}-+-{0:-^44}-+-{0:-^12}-+-{0:-^12}-'.format('')
+ if show_orphan:
+ contents += '+-{:-^5}-'.format('')
+ contents += '\n'
+
+ # Actual data
+ for pkg in packages:
+ name = shrink_str(pkg['name'], 30)
+ version = shrink_str(pkg['version'], 12)
+ upstream = shrink_str(pkg['upstream'], 12)
+ orphan_str = 'ORPH' if pkg.get('orphan') else ''
+ url = 'https://release-monitoring.org/project/{:0>5}'.format(pkg['id'])
+
+ contents += '{:>30} | {:^8} | {:0>5} | {:12} | {:12}'.format(name, pkg['from'], url, version, upstream)
+ if show_orphan:
+ contents += ' | {:4}'.format(orphan_str)
+ contents += '\n'
+ return contents
+
+
def developers_email(smtp, branches, notifications, datestr, dry_run):
'''
Send the e-mails to the individual developers
@@ -282,6 +310,17 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
contents += "\n"
+ outdated = notif.package_version_notification
+ if len(outdated) != 0:
+ contents += '\n'
+ contents += textwrap.fill("This is the list of packages for which a new version has been "
+ "detected and for which you are a registered developer. Please "
+ "help us improving the quality of Buildroot by bumping these "
+ "packages to their latest version. Thanks!")
+ contents += '\n\n'
+ contents += show_outdated(outdated, show_orphan=show_orphan)
+ contents += '\n'
+
contents += "-- \n"
contents += http_baseurl
if dry_run:
@@ -319,7 +358,7 @@ def global_email_branch_result(results, results_by_reason, branch):
return contents
-def global_email(smtp, results, results_by_reason, datestr, overall, dry_run):
+def global_email(smtp, results, results_by_reason, datestr, overall, outdated_pkg, dry_run):
'''
Send the global e-mail to the mailing list
'''
@@ -341,6 +380,11 @@ def global_email(smtp, results, results_by_reason, datestr, overall, dry_run):
if len(results[branch]) == 0:
continue
contents += global_email_branch_result(results[branch], results_by_reason[branch], branch)
+ if outdated_pkg:
+ contents += '\n\n'
+ contents += "Packages having a newer version\n"
+ contents += "===============================\n\n"
+ contents += show_outdated(outdated_pkg, show_orphan=True)
contents += "\n"
contents += "-- \n"
contents += http_baseurl
@@ -479,7 +523,7 @@ def __main__():
developers_email(smtp, branches, notifications, date_str, args.dry_run)
if 'global' in email_dest:
global_email(smtp, results, results_by_reason, date_str,
- overall_stats, args.dry_run)
+ overall_stats, [], args.dry_run)
smtp.quit()
--
2.21.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH buildroot-test v2 4/4] utils/daily-mail: enable outdated packages emails
2019-08-05 13:15 [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail Victor Huesca
` (2 preceding siblings ...)
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 3/4] utils/daily-mail: add outdated-packages email formating part Victor Huesca
@ 2019-08-05 13:15 ` Victor Huesca
2019-08-15 10:00 ` [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail Thomas Petazzoni
4 siblings, 0 replies; 6+ messages in thread
From: Victor Huesca @ 2019-08-05 13:15 UTC (permalink / raw)
To: buildroot
This patch adds an 'outdated' entry to the program data. This entry
adds information about the outdated packages in the global and
developers mails.
A new entry in the 'localconfig' file named 'pkg-stats' that contains
the location of the JSON output of the buildroot's 'pkg-stats' script.
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
utils/daily-mail | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/utils/daily-mail b/utils/daily-mail
index 58c4b2a..4f670fb 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -484,7 +484,7 @@ def parse_args():
parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)')
parser.add_argument('--dest', action='store', nargs='+', choices=['dev', 'global'],
help='List of emails type to send (ie. global, dev)')
- parser.add_argument('--data', action='store', nargs='+', choices=['autobuild'],
+ parser.add_argument('--data', action='store', nargs='+', choices=['autobuild', 'outdated'],
help='List of information to add in emails (blank separated)')
args = parser.parse_args()
if args.date and not RE_DATE.match(args.date):
@@ -501,7 +501,7 @@ def __main__():
date_str = yesterday.strftime('%Y-%m-%d')
branches = args.branches if args.branches else get_branches()
email_dest = set(args.dest) if args.dest else {'dev', 'global'}
- email_data = set(args.data) if args.data else {'autobuild'}
+ email_data = set(args.data) if args.data else {'autobuild', 'outdated'}
db = _mysql.connect(host=localconfig.host,
user=localconfig.user,
@@ -515,7 +515,8 @@ def __main__():
overall_stats = {}
results = {}
results_by_reason = {}
- notifications = calculate_notifications(results, [])
+ outdate_pkg = get_outdated_pkg(localconfig.pkg_stats) if 'outdated' in email_data else []
+ notifications = calculate_notifications(results, outdate_pkg)
smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
smtp.starttls()
smtp.login(localconfig.smtpuser, localconfig.smtppass)
@@ -523,7 +524,7 @@ def __main__():
developers_email(smtp, branches, notifications, date_str, args.dry_run)
if 'global' in email_dest:
global_email(smtp, results, results_by_reason, date_str,
- overall_stats, [], args.dry_run)
+ overall_stats, outdate_pkg, args.dry_run)
smtp.quit()
--
2.21.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail
2019-08-05 13:15 [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail Victor Huesca
` (3 preceding siblings ...)
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 4/4] utils/daily-mail: enable outdated packages emails Victor Huesca
@ 2019-08-15 10:00 ` Thomas Petazzoni
4 siblings, 0 replies; 6+ messages in thread
From: Thomas Petazzoni @ 2019-08-15 10:00 UTC (permalink / raw)
To: buildroot
Hello Victor,
On Mon, 5 Aug 2019 15:15:40 +0200
Victor Huesca <victor.huesca@bootlin.com> wrote:
> Victor Huesca (4):
> utils/daily-mail: new data: outdated packages
> utils/daily-mail: add outdated packages to the developer notification
> utils/daily-mail: add outdated-packages email formating part
> utils/daily-mail: enable outdated packages emails
Thanks a lot for this work, I have applied all 4 patches, with some
minor changes. I also did a few formatting improvements:
https://git.buildroot.org/buildroot-test/commit/?id=ba42c3bdce31581dc9c270165ebe59393e91c51f
https://git.buildroot.org/buildroot-test/commit/?id=2e8ae4662d473d67398bf97eaf5e40bcbbc000be
https://git.buildroot.org/buildroot-test/commit/?id=e7ed66b8a9383de60ab59e8604df4215b1753d30
The e-mail of today (both global and per developer) was sent with your
patches applied, so those notifications about outdated packages have
been sent to many developers.
Thanks!
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-08-15 10:00 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-05 13:15 [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 1/4] utils/daily-mail: new data: outdated packages Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 2/4] utils/daily-mail: add outdated packages to the developer notification Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 3/4] utils/daily-mail: add outdated-packages email formating part Victor Huesca
2019-08-05 13:15 ` [Buildroot] [PATCH buildroot-test v2 4/4] utils/daily-mail: enable outdated packages emails Victor Huesca
2019-08-15 10:00 ` [Buildroot] [PATCH buildroot-test v2 0/4] add support for outdated packages to daily-mail Thomas Petazzoni
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.