[Buildroot] [PATCH v3 0/3] support/script/pkg-stats: add support for json

[Buildroot] [PATCH v3 0/3] support/script/pkg-stats: add support for json

[Victor Huesca]

This patch serie adds support for json output in addition to the existing
HTML output.

Changes v1 --> v2:
  - Add comments in the dump_json function
  - Adjust the commit message
  - Factorize the git hash and the date (suggested by  Thomas)
  - Split into 3 commits:
     1. Prepare the argmument parser
     2. Add json support
     3. Factorize code du to adding the json output

Victor Huesca (3):
  support/scripts/pkg-stats: improve argparse usage
  support/scripts/pkg-stats: add support for json output
  support/scripts/pkg-stats: factorize date and commit

 support/scripts/pkg-stats | 71 +++++++++++++++++++++++++++++----------
 1 file changed, 54 insertions(+), 17 deletions(-)

-- 
2.21.0

[Buildroot] [PATCH v3 1/3] support/scripts/pkg-stats: improve argparse usage

[Victor Huesca]

Move the mutual exculsion of the '-n' and '-p' options to be part of the
parser instead of being checked in main.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 support/scripts/pkg-stats | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index b0be7d919b..d65f609d57 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -23,7 +23,6 @@ import os
 from collections import defaultdict
 import re
 import subprocess
-import sys
 import requests  # URL checking
 import json
 import certifi
@@ -700,18 +699,16 @@ def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument('-o', dest='output', action='store', required=True,
                         help='HTML output file')
-    parser.add_argument('-n', dest='npackages', type=int, action='store',
+    packages = parser.add_mutually_exclusive_group()
+    packages.add_argument('-n', dest='npackages', type=int, action='store',
                         help='Number of packages')
-    parser.add_argument('-p', dest='packages', action='store',
+    packages.add_argument('-p', dest='packages', action='store',
                         help='List of packages (comma separated)')
     return parser.parse_args()
 
 
 def __main__():
     args = parse_args()
-    if args.npackages and args.packages:
-        print("ERROR: -n and -p are mutually exclusive")
-        sys.exit(1)
     if args.packages:
         package_list = args.packages.split(",")
     else:
-- 
2.21.0

[Buildroot] [PATCH v3 2/3] support/scripts/pkg-stats: add support for json output

[Victor Huesca]

Pkg-stats is a great script that get a lot of interesting info from
buildroot packages. Unfortunately it is currently designed to output a
static HTML page only. While this is great to include on the
buildroot's website, the HTML is not designed to be easily parsable and
thus it is difficult to reuse it in other scripts.

This patch provide a new option to output a JSON file in addition to the
HTML one.

The old 'output' option has been renamed to 'html' to distinguish from
the new 'json' option.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
Changes v2 --> v3:
  - Use better names for the argparse group
  - Comment the tricky steps in dump_json
---
 support/scripts/pkg-stats | 48 +++++++++++++++++++++++++++++++++++----
 1 file changed, 44 insertions(+), 4 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index d65f609d57..478f98fbc2 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -695,16 +695,52 @@ def dump_html(packages, stats, output):
         f.write(html_footer)
 
 
+def dump_json(packages, stats, output):
+    # Format packages as a dictionnary instead of a list
+    # Exclude local field that does not contains real date
+    excluded_fields = ['url_worker', 'name']
+    pkgs = {
+        pkg.name: {
+            k: v
+            for k, v in pkg.__dict__.items()
+            if k not in excluded_fields
+        } for pkg in packages
+    }
+    # Aggregate infrastructures into a single dict entry
+    statistics = {
+        k: v
+        for k, v in stats.items()
+        if not k.startswith('infra-')
+    }
+    statistics['infra'] = {k[6:]: v for k, v in stats.items() if k.startswith('infra-')}
+    # The actual structure to dump, add commit and date to it
+    o = subprocess.check_output(["git", "log", "master", "-n", "1", "--pretty=format:%H"])
+    final = {'packages': pkgs,
+             'stats': statistics,
+             'commit': o.splitlines()[0],
+             'date': str(datetime.datetime.utcnow())}
+
+    with open(output, 'w') as f:
+        json.dump(final, f, indent=2, separators=(',', ': '))
+        f.write('\n')
+
+
 def parse_args():
     parser = argparse.ArgumentParser()
-    parser.add_argument('-o', dest='output', action='store', required=True,
+    output = parser.add_argument_group('output', 'Output file(s)')
+    output.add_argument('--html', dest='html', action='store',
                         help='HTML output file')
+    output.add_argument('--json', dest='json', action='store',
+                        help='JSON output file')
     packages = parser.add_mutually_exclusive_group()
     packages.add_argument('-n', dest='npackages', type=int, action='store',
                         help='Number of packages')
     packages.add_argument('-p', dest='packages', action='store',
                         help='List of packages (comma separated)')
-    return parser.parse_args()
+    args = parser.parse_args()
+    if not args.html and not args.json:
+        parser.error('at least one of --html or --json (or both) is required')
+    return args
 
 
 def __main__():
@@ -732,8 +768,12 @@ def __main__():
     check_package_latest_version(packages)
     print("Calculate stats")
     stats = calculate_stats(packages)
-    print("Write HTML")
-    dump_html(packages, stats, args.output)
+    if args.html:
+        print("Write HTML")
+        dump_html(packages, stats, args.html)
+    if args.json:
+        print("Write JSON")
+        dump_json(packages, stats, args.json)
 
 
 __main__()
-- 
2.21.0

[Buildroot] [PATCH v3 3/3] support/scripts/pkg-stats: factorize date and commit

[Victor Huesca]

The 'dump_html' and 'dump_json' both include commit infos as well as the
current date. It make more sense to retrieve these information once.
This patch simply does this factorization.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 support/scripts/pkg-stats | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 478f98fbc2..7dcb2b4cbd 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -678,24 +678,21 @@ def dump_html_stats(f, stats):
     f.write("</table>\n")
 
 
-def dump_gen_info(f):
+def dump_gen_info(f, date, git_commit):
     # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032
-    o = subprocess.check_output(["git", "log", "master", "-n", "1", "--pretty=format:%H"])
-    git_commit = o.splitlines()[0]
-    f.write("<p><i>Updated on %s, git commit %s</i></p>\n" %
-            (str(datetime.datetime.utcnow()), git_commit))
+    f.write("<p><i>Updated on %s, git commit %s</i></p>\n" % (str(date), git_commit))
 
 
-def dump_html(packages, stats, output):
+def dump_html(packages, stats, date, commit, output):
     with open(output, 'w') as f:
         f.write(html_header)
         dump_html_all_pkgs(f, packages)
         dump_html_stats(f, stats)
-        dump_gen_info(f)
+        dump_gen_info(f, date, commit)
         f.write(html_footer)
 
 
-def dump_json(packages, stats, output):
+def dump_json(packages, stats, date, commit, output):
     # Format packages as a dictionnary instead of a list
     # Exclude local field that does not contains real date
     excluded_fields = ['url_worker', 'name']
@@ -717,8 +714,8 @@ def dump_json(packages, stats, output):
     o = subprocess.check_output(["git", "log", "master", "-n", "1", "--pretty=format:%H"])
     final = {'packages': pkgs,
              'stats': statistics,
-             'commit': o.splitlines()[0],
-             'date': str(datetime.datetime.utcnow())}
+             'commit': commit,
+             'date': str(date)}
 
     with open(output, 'w') as f:
         json.dump(final, f, indent=2, separators=(',', ': '))
@@ -749,6 +746,9 @@ def __main__():
         package_list = args.packages.split(",")
     else:
         package_list = None
+    date = datetime.datetime.utcnow()
+    commit = subprocess.check_output(['git', 'log', 'master', '-n', '1',
+                                      '--pretty=format:%H']).splitlines()[0]
     print("Build package list ...")
     packages = get_pkglist(args.npackages, package_list)
     print("Getting package make info ...")
@@ -770,10 +770,10 @@ def __main__():
     stats = calculate_stats(packages)
     if args.html:
         print("Write HTML")
-        dump_html(packages, stats, args.html)
+        dump_html(packages, stats, date, commit, args.html)
     if args.json:
         print("Write JSON")
-        dump_json(packages, stats, args.json)
+        dump_json(packages, stats, date, commit, args.json)
 
 
 __main__()
-- 
2.21.0

[Buildroot] [PATCH v3 0/3] support/script/pkg-stats: add support for json

[Thomas Petazzoni]

Hello,

On Fri, 19 Jul 2019 15:06:27 +0200
Victor Huesca <victor.huesca@bootlin.com> wrote:

> Victor Huesca (3):
>   support/scripts/pkg-stats: improve argparse usage
>   support/scripts/pkg-stats: add support for json output
>   support/scripts/pkg-stats: factorize date and commit

I've applied all patches, with a minor change to the last patch (I'll
reply to the patch). Technically speaking, it would have been nicer to
do the date/commit rework before introducing the JSON output, but OK.

I also did a minor followup commit to rename dump_gen_info() to
dump_html_gen_info(), so that all HTML output related functions are
prefixed by dump_html.

Thanks for this work!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v3 3/3] support/scripts/pkg-stats: factorize date and commit

[Thomas Petazzoni]

Victor,

On Fri, 19 Jul 2019 15:06:30 +0200
Victor Huesca <victor.huesca@bootlin.com> wrote:

> -def dump_gen_info(f):
> +def dump_gen_info(f, date, git_commit):

I renamed the argument git_commit to "commit", to be consistent with
what you've done in the other functions.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com