* [Buildroot] [PATCH v4 01/12] support/scripts/pkg-stats: store latest version in a dict
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 02/12] support/scripts/pkg-stats: store patch files for the package Heiko Thiery
` (11 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
This patch changes the type of the latest_version variable to a dict.
This is for better readability/usability of the data. With this the json
output is more descriptive in later processing of the json output.
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 31 +++++++++++++++----------------
1 file changed, 15 insertions(+), 16 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 7721d98459..727b203032 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -71,7 +71,7 @@ class Package:
self.url_status = None
self.url_worker = None
self.cves = list()
- self.latest_version = (RM_API_STATUS_ERROR, None, None)
+ self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None}
def pkgvar(self):
return self.name.upper().replace("-", "_")
@@ -460,9 +460,8 @@ def check_package_latest_version(packages):
"""
Fills in the .latest_version field of all Package objects
- This field has a special format:
- (status, version, id)
- with:
+ This field is a dict and has the following keys:
+
- status: one of RM_API_STATUS_ERROR,
RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN,
RM_API_STATUS_NOT_FOUND
@@ -478,7 +477,7 @@ def check_package_latest_version(packages):
worker_pool = Pool(processes=64)
results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages))
for pkg, r in zip(packages, results):
- pkg.latest_version = r
+ pkg.latest_version = dict(zip(['status', 'version', 'id'], r))
del http_pool
@@ -516,13 +515,13 @@ def calculate_stats(packages):
stats["hash"] += 1
else:
stats["no-hash"] += 1
- if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO:
+ if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO:
stats["rmo-mapping"] += 1
else:
stats["rmo-no-mapping"] += 1
- if not pkg.latest_version[1]:
+ if not pkg.latest_version['version']:
stats["version-unknown"] += 1
- elif pkg.latest_version[1] == pkg.current_version:
+ elif pkg.latest_version['version'] == pkg.current_version:
stats["version-uptodate"] += 1
else:
stats["version-not-uptodate"] += 1
@@ -688,29 +687,29 @@ def dump_html_pkg(f, pkg):
f.write(" <td class=\"centered\">%s</td>\n" % current_version)
# Latest version
- if pkg.latest_version[0] == RM_API_STATUS_ERROR:
+ if pkg.latest_version['status'] == RM_API_STATUS_ERROR:
td_class.append("version-error")
- if pkg.latest_version[1] is None:
+ if pkg.latest_version['version'] is None:
td_class.append("version-unknown")
- elif pkg.latest_version[1] != pkg.current_version:
+ elif pkg.latest_version['version'] != pkg.current_version:
td_class.append("version-needs-update")
else:
td_class.append("version-good")
- if pkg.latest_version[0] == RM_API_STATUS_ERROR:
+ if pkg.latest_version['status'] == RM_API_STATUS_ERROR:
latest_version_text = "<b>Error</b>"
- elif pkg.latest_version[0] == RM_API_STATUS_NOT_FOUND:
+ elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND:
latest_version_text = "<b>Not found</b>"
else:
- if pkg.latest_version[1] is None:
+ if pkg.latest_version['version'] is None:
latest_version_text = "<b>Found, but no version</b>"
else:
latest_version_text = "<a href=\"https://release-monitoring.org/project/%s\"><b>%s</b></a>" % \
- (pkg.latest_version[2], str(pkg.latest_version[1]))
+ (pkg.latest_version['id'], str(pkg.latest_version['version']))
latest_version_text += "<br/>"
- if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO:
+ if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO:
latest_version_text += "found by <a href=\"https://release-monitoring.org/distro/Buildroot/\">distro</a>"
else:
latest_version_text += "found by guess"
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 02/12] support/scripts/pkg-stats: store patch files for the package
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 01/12] support/scripts/pkg-stats: store latest version in a dict Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 03/12] support/scripts/pkg-stats: set developers info Heiko Thiery
` (10 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
Remove the patch_count attribute and use a class property instead.
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 727b203032..4116622c98 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -64,7 +64,7 @@ class Package:
self.has_license = False
self.has_license_files = False
self.has_hash = False
- self.patch_count = 0
+ self.patch_files = []
self.warnings = 0
self.current_version = None
self.url = None
@@ -93,6 +93,10 @@ class Package:
self.url_status = "Missing"
fp.close()
+ @property
+ def patch_count(self):
+ return len(self.patch_files)
+
def set_infra(self):
"""
Fills in the .infras field
@@ -131,10 +135,9 @@ class Package:
"""
Fills in the .patch_count field
"""
- self.patch_count = 0
pkgdir = os.path.dirname(self.path)
for subdir, _, _ in os.walk(pkgdir):
- self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch'))
+ self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch')
def set_current_version(self):
"""
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 03/12] support/scripts/pkg-stats: set developers info
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 01/12] support/scripts/pkg-stats: store latest version in a dict Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 02/12] support/scripts/pkg-stats: store patch files for the package Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 04/12] support/scripts/pkg-stats: store licences of package Heiko Thiery
` (9 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
Use the function 'parse_developers' function from getdeveloperlib that
collect the information about the developers and the files they
maintain. Then set the maintainer(s) to each package.
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 4116622c98..c40fe61d9e 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -30,10 +30,14 @@ import certifi
import distutils.version
import time
import gzip
+import sys
from urllib3 import HTTPSConnectionPool
from urllib3.exceptions import HTTPError
from multiprocessing import Pool
+sys.path.append('utils/')
+from getdeveloperlib import parse_developers
+
NVD_START_YEAR = 2002
NVD_JSON_VERSION = "1.0"
NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
@@ -171,6 +175,16 @@ class Package:
"""
return cve in self.all_ignored_cves.get(self.pkgvar(), [])
+ def set_developers(self, developers):
+ """
+ Fills in the .developers field
+ """
+ self.developers = [
+ dev.name
+ for dev in developers
+ if dev.hasfile(self.path)
+ ]
+
def __eq__(self, other):
return self.path == other.path
@@ -891,6 +905,8 @@ def __main__():
'HEAD']).splitlines()[0]
print("Build package list ...")
packages = get_pkglist(args.npackages, package_list)
+ print("Getting developers ...")
+ developers = parse_developers()
print("Getting package make info ...")
package_init_make_info()
print("Getting package details ...")
@@ -902,6 +918,7 @@ def __main__():
pkg.set_check_package_warnings()
pkg.set_current_version()
pkg.set_url()
+ pkg.set_developers(developers)
print("Checking URL status")
check_package_urls(packages)
print("Getting latest versions ...")
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 04/12] support/scripts/pkg-stats: store licences of package
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
` (2 preceding siblings ...)
2020-03-02 14:50 ` [Buildroot] [PATCH v4 03/12] support/scripts/pkg-stats: set developers info Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 05/12] support/scripts/pkg-stats: add package status Heiko Thiery
` (8 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index c40fe61d9e..b0ee4f3bac 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -56,7 +56,7 @@ http_pool = None
class Package:
- all_licenses = list()
+ all_licenses = dict()
all_license_files = list()
all_versions = dict()
all_ignored_cves = dict()
@@ -65,6 +65,7 @@ class Package:
self.name = name
self.path = path
self.infras = None
+ self.license = None
self.has_license = False
self.has_license_files = False
self.has_hash = False
@@ -125,6 +126,7 @@ class Package:
var = self.pkgvar()
if var in self.all_licenses:
self.has_license = True
+ self.license = self.all_licenses[var]
if var in self.all_license_files:
self.has_license_files = True
@@ -387,7 +389,7 @@ def package_init_make_info():
if value == "unknown":
continue
pkgvar = pkgvar[:-8]
- Package.all_licenses.append(pkgvar)
+ Package.all_licenses[pkgvar] = value
elif pkgvar.endswith("_LICENSE_FILES"):
if pkgvar.endswith("_MANIFEST_LICENSE_FILES"):
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 05/12] support/scripts/pkg-stats: add package status
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
` (3 preceding siblings ...)
2020-03-02 14:50 ` [Buildroot] [PATCH v4 04/12] support/scripts/pkg-stats: store licences of package Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 06/12] support/scripts/pkg-stats: add package count to stats Heiko Thiery
` (7 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
Unify the status check information. The status is stored in a tuple. The
first entry is the status that can be 'ok', 'warning' or 'error'. The
second entry is a verbose message.
The following checks are performed:
- url: status of the URL check
- license: status of the license presence check
- license-files: status of the license file check
- hash: status of the hash file presence check
- patches: status of the patches count check
- pkg-check: status of the check-package script result
- developers: status if a package has developers in the DEVELOPERS file
- version: status of the version check
With that status information the following variables are replaced:
has_license, has_license_files, has_hash, url_status
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 101 ++++++++++++++++++++++++++------------
1 file changed, 69 insertions(+), 32 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index b0ee4f3bac..47f7507540 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -73,10 +73,10 @@ class Package:
self.warnings = 0
self.current_version = None
self.url = None
- self.url_status = None
self.url_worker = None
self.cves = list()
self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None}
+ self.status = {}
def pkgvar(self):
return self.name.upper().replace("-", "_")
@@ -85,17 +85,17 @@ class Package:
"""
Fills in the .url field
"""
- self.url_status = "No Config.in"
+ self.status['url'] = ("warning", "no Config.in")
for filename in os.listdir(os.path.dirname(self.path)):
if fnmatch.fnmatch(filename, 'Config.*'):
fp = open(os.path.join(os.path.dirname(self.path), filename), "r")
for config_line in fp:
if URL_RE.match(config_line):
self.url = config_line.strip()
- self.url_status = "Found"
+ self.status['url'] = ("ok", "found")
fp.close()
return
- self.url_status = "Missing"
+ self.status['url'] = ("error", "missing")
fp.close()
@property
@@ -121,30 +121,42 @@ class Package:
def set_license(self):
"""
- Fills in the .has_license and .has_license_files fields
+ Fills in the .status['license'] and .status['license-files'] fields
"""
var = self.pkgvar()
+ self.status['license'] = ("error", "missing")
+ self.status['license-files'] = ("error", "missing")
if var in self.all_licenses:
- self.has_license = True
self.license = self.all_licenses[var]
+ self.status['license'] = ("ok", "found")
if var in self.all_license_files:
- self.has_license_files = True
+ self.status['license-files'] = ("ok", "found")
def set_hash_info(self):
"""
- Fills in the .has_hash field
+ Fills in the .status['hash'] field
"""
hashpath = self.path.replace(".mk", ".hash")
- self.has_hash = os.path.exists(hashpath)
+ if os.path.exists(hashpath):
+ self.status['hash'] = ("ok", "found")
+ else:
+ self.status['hash'] = ("error", "missing")
def set_patch_count(self):
"""
- Fills in the .patch_count field
+ Fills in the .patch_count, .patch_files and .status['patches'] fields
"""
pkgdir = os.path.dirname(self.path)
for subdir, _, _ in os.walk(pkgdir):
self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch')
+ if self.patch_count == 0:
+ self.status['patches'] = ("ok", "no patches")
+ elif self.patch_count < 5:
+ self.status['patches'] = ("warning", "some patches")
+ else:
+ self.status['patches'] = ("error", "lots of patches")
+
def set_current_version(self):
"""
Fills in the .current_version field
@@ -155,10 +167,11 @@ class Package:
def set_check_package_warnings(self):
"""
- Fills in the .warnings field
+ Fills in the .warnings and .status['pkg-check'] fields
"""
cmd = ["./utils/check-package"]
pkgdir = os.path.dirname(self.path)
+ self.status['pkg-check'] = ("error", "Missing")
for root, dirs, files in os.walk(pkgdir):
for f in files:
if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
@@ -169,6 +182,10 @@ class Package:
m = re.match("^([0-9]*) warnings generated", line)
if m:
self.warnings = int(m.group(1))
+ if self.warnings == 0:
+ self.status['pkg-check'] = ("ok", "no warnings")
+ else:
+ self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings))
return
def is_cve_ignored(self, cve):
@@ -179,7 +196,7 @@ class Package:
def set_developers(self, developers):
"""
- Fills in the .developers field
+ Fills in the .developers and .status['developers'] field
"""
self.developers = [
dev.name
@@ -187,6 +204,14 @@ class Package:
if dev.hasfile(self.path)
]
+ if self.developers:
+ self.status['developers'] = ("ok", "{} developers".format(len(self.developers)))
+ else:
+ self.status['developers'] = ("warning", "no developers")
+
+ def is_status_ok(self, name):
+ return self.status[name][0] == 'ok'
+
def __eq__(self, other):
return self.path == other.path
@@ -195,7 +220,7 @@ class Package:
def __str__(self):
return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \
- (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count)
+ (self.name, self.path, self.is_status_ok('license'), self.is_status_ok('license-files'), self.status['hash'], self.patch_count)
class CVE:
@@ -409,23 +434,23 @@ def package_init_make_info():
def check_url_status_worker(url, url_status):
- if url_status != "Missing" and url_status != "No Config.in":
+ if url_status[0] == 'ok':
try:
url_status_code = requests.head(url, timeout=30).status_code
if url_status_code >= 400:
- return "Invalid(%s)" % str(url_status_code)
+ return ("error", "invalid {}".format(url_status_code))
except requests.exceptions.RequestException:
- return "Invalid(Err)"
- return "Ok"
+ return ("error", "invalid (err)")
+ return ("ok", "valid")
return url_status
def check_package_urls(packages):
Package.pool = Pool(processes=64)
for pkg in packages:
- pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status))
+ pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.status['url']))
for pkg in packages:
- pkg.url_status = pkg.url_worker.get(timeout=3600)
+ pkg.status['url'] = pkg.url_worker.get(timeout=3600)
def release_monitoring_get_latest_version_by_distro(pool, name):
@@ -497,6 +522,18 @@ def check_package_latest_version(packages):
results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages))
for pkg, r in zip(packages, results):
pkg.latest_version = dict(zip(['status', 'version', 'id'], r))
+
+ if pkg.latest_version['status'] == RM_API_STATUS_ERROR:
+ pkg.status['version'] = ('warning', "Release Monitoring API error")
+ elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND:
+ pkg.status['version'] = ('warning', "Package not found on Release Monitoring")
+
+ if pkg.latest_version['version'] is None:
+ pkg.status['version'] = ('warning', "No upstream version available on Release Monitoring")
+ elif pkg.latest_version['version'] != pkg.current_version:
+ pkg.status['version'] = ('error', "The newer version {} is available upstream".format(pkg.latest_version['version']))
+ else:
+ pkg.status['version'] = ('ok', 'up-to-date')
del http_pool
@@ -522,15 +559,15 @@ def calculate_stats(packages):
stats["infra-%s" % infra] += 1
else:
stats["infra-unknown"] += 1
- if pkg.has_license:
+ if pkg.is_status_ok('license'):
stats["license"] += 1
else:
stats["no-license"] += 1
- if pkg.has_license_files:
+ if pkg.is_status_ok('license-files'):
stats["license-files"] += 1
else:
stats["no-license-files"] += 1
- if pkg.has_hash:
+ if pkg.is_status_ok('hash'):
stats["hash"] += 1
else:
stats["no-hash"] += 1
@@ -673,30 +710,30 @@ def dump_html_pkg(f, pkg):
# License
td_class = ["centered"]
- if pkg.has_license:
+ if pkg.is_status_ok('license'):
td_class.append("correct")
else:
td_class.append("wrong")
f.write(" <td class=\"%s\">%s</td>\n" %
- (" ".join(td_class), boolean_str(pkg.has_license)))
+ (" ".join(td_class), boolean_str(pkg.is_status_ok('license'))))
# License files
td_class = ["centered"]
- if pkg.has_license_files:
+ if pkg.is_status_ok('license-files'):
td_class.append("correct")
else:
td_class.append("wrong")
f.write(" <td class=\"%s\">%s</td>\n" %
- (" ".join(td_class), boolean_str(pkg.has_license_files)))
+ (" ".join(td_class), boolean_str(pkg.is_status_ok('license-files'))))
# Hash
td_class = ["centered"]
- if pkg.has_hash:
+ if pkg.is_status_ok('hash'):
td_class.append("correct")
else:
td_class.append("wrong")
f.write(" <td class=\"%s\">%s</td>\n" %
- (" ".join(td_class), boolean_str(pkg.has_hash)))
+ (" ".join(td_class), boolean_str(pkg.is_status_ok('hash'))))
# Current version
if len(pkg.current_version) > 20:
@@ -747,12 +784,12 @@ def dump_html_pkg(f, pkg):
# URL status
td_class = ["centered"]
- url_str = pkg.url_status
- if pkg.url_status == "Missing" or pkg.url_status == "No Config.in":
+ url_str = pkg.status['url'][1]
+ if pkg.status['url'][0] in ("error", "warning"):
td_class.append("missing_url")
- elif pkg.url_status.startswith("Invalid"):
+ if pkg.status['url'][0] == "error":
td_class.append("invalid_url")
- url_str = "<a href=%s>%s</a>" % (pkg.url, pkg.url_status)
+ url_str = "<a href=%s>%s</a>" % (pkg.url, pkg.status['url'][1])
else:
td_class.append("good_url")
url_str = "<a href=%s>Link</a>" % pkg.url
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 06/12] support/scripts/pkg-stats: add package count to stats
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
` (4 preceding siblings ...)
2020-03-02 14:50 ` [Buildroot] [PATCH v4 05/12] support/scripts/pkg-stats: add package status Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 07/12] support/scripts/pkg-stats: store pkg dir path Heiko Thiery
` (6 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 1 +
1 file changed, 1 insertion(+)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 47f7507540..2847cb82f3 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -549,6 +549,7 @@ def check_package_cves(nvd_path, packages):
def calculate_stats(packages):
stats = defaultdict(int)
+ stats['packages'] = len(packages)
for pkg in packages:
# If packages have multiple infra, take the first one. For the
# vast majority of packages, the target and host infra are the
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 07/12] support/scripts/pkg-stats: store pkg dir path
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
` (5 preceding siblings ...)
2020-03-02 14:50 ` [Buildroot] [PATCH v4 06/12] support/scripts/pkg-stats: add package count to stats Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 08/12] support/scripts/pkg-stats: add defconfig support Heiko Thiery
` (5 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
This value can be used for later processing.
In the buildroot-stats application this is used to create links pointing
to the git repo of buildroot.
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 1 +
1 file changed, 1 insertion(+)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 2847cb82f3..1fd306a6d5 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -64,6 +64,7 @@ class Package:
def __init__(self, name, path):
self.name = name
self.path = path
+ self.pkg_path = os.path.dirname(path)
self.infras = None
self.license = None
self.has_license = False
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 08/12] support/scripts/pkg-stats: add defconfig support
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
` (6 preceding siblings ...)
2020-03-02 14:50 ` [Buildroot] [PATCH v4 07/12] support/scripts/pkg-stats: store pkg dir path Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 09/12] support/scripts/pkg-stats: add support for license hash check Heiko Thiery
` (4 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
Scan configs directory and create Defconfig objects.
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 43 +++++++++++++++++++++++++++++++++++++--
1 file changed, 41 insertions(+), 2 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 1fd306a6d5..432090c251 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -54,6 +54,34 @@ RM_API_STATUS_NOT_FOUND = 4
# because it's used by sub-processes.
http_pool = None
+class Defconfig:
+ def __init__(self, name, path):
+ self.name = name
+ self.path = path
+ self.developers = None
+
+ def set_developers(self, developers):
+ """
+ Fills in the .developers field
+ """
+ self.developers = [
+ developer.name
+ for developer in developers
+ if developer.hasfile(self.path)
+ ]
+
+
+def get_defconfig_list():
+ """
+ Builds the list of Buildroot defconfigs, returning a list of Defconfig
+ objects.
+ """
+ return [
+ Defconfig(name[:-len('_defconfig')], os.path.join('configs', name))
+ for name in os.listdir('configs')
+ if name.endswith('_defconfig')
+ ]
+
class Package:
all_licenses = dict()
@@ -886,7 +914,7 @@ def dump_html(packages, stats, date, commit, output):
f.write(html_footer)
-def dump_json(packages, stats, date, commit, output):
+def dump_json(packages, defconfigs, stats, date, commit, output):
# Format packages as a dictionnary instead of a list
# Exclude local field that does not contains real date
excluded_fields = ['url_worker', 'name']
@@ -897,6 +925,12 @@ def dump_json(packages, stats, date, commit, output):
if k not in excluded_fields
} for pkg in packages
}
+ defconfigs = {
+ d.name: {
+ k: v
+ for k, v in d.__dict__.items()
+ } for d in defconfigs
+ }
# Aggregate infrastructures into a single dict entry
statistics = {
k: v
@@ -907,6 +941,7 @@ def dump_json(packages, stats, date, commit, output):
# The actual structure to dump, add commit and date to it
final = {'packages': pkgs,
'stats': statistics,
+ 'defconfigs': defconfigs,
'commit': commit,
'date': str(date)}
@@ -948,6 +983,10 @@ def __main__():
packages = get_pkglist(args.npackages, package_list)
print("Getting developers ...")
developers = parse_developers()
+ print("Build defconfig list ...")
+ defconfigs = get_defconfig_list()
+ for d in defconfigs:
+ d.set_developers(developers)
print("Getting package make info ...")
package_init_make_info()
print("Getting package details ...")
@@ -974,7 +1013,7 @@ def __main__():
dump_html(packages, stats, date, commit, args.html)
if args.json:
print("Write JSON")
- dump_json(packages, stats, date, commit, args.json)
+ dump_json(packages, defconfigs, stats, date, commit, args.json)
__main__()
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 09/12] support/scripts/pkg-stats: add support for license hash check
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
` (7 preceding siblings ...)
2020-03-02 14:50 ` [Buildroot] [PATCH v4 08/12] support/scripts/pkg-stats: add defconfig support Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 10/12] support/scripts/pkg-stats: set status to 'na' for virtual packages Heiko Thiery
` (3 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
Store the names of license files and check if they are in the hash file.
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 432090c251..c9287b416e 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -85,7 +85,7 @@ def get_defconfig_list():
class Package:
all_licenses = dict()
- all_license_files = list()
+ all_license_files = dict()
all_versions = dict()
all_ignored_cves = dict()
@@ -98,6 +98,7 @@ class Package:
self.has_license = False
self.has_license_files = False
self.has_hash = False
+ self.license_files = None
self.patch_files = []
self.warnings = 0
self.current_version = None
@@ -159,6 +160,7 @@ class Package:
self.license = self.all_licenses[var]
self.status['license'] = ("ok", "found")
if var in self.all_license_files:
+ self.license_files = self.all_license_files[var].split(' ')
self.status['license-files'] = ("ok", "found")
def set_hash_info(self):
@@ -166,8 +168,18 @@ class Package:
Fills in the .status['hash'] field
"""
hashpath = self.path.replace(".mk", ".hash")
+ self.status['hash-license'] = ("na", "no hash file")
if os.path.exists(hashpath):
self.status['hash'] = ("ok", "found")
+ self.status['hash-license'] = ("error", "no license in hash file")
+ # check if license files are in hash file
+ if self.license_files is not None:
+ self.status['hash-license'] = ("ok", "found")
+ with open(hashpath) as f:
+ content = f.read()
+ for license in self.license_files:
+ if content.find(license) == -1:
+ self.status['hash-license'] = ("error", "license missing in hash file")
else:
self.status['hash'] = ("error", "missing")
@@ -449,7 +461,7 @@ def package_init_make_info():
if pkgvar.endswith("_MANIFEST_LICENSE_FILES"):
continue
pkgvar = pkgvar[:-14]
- Package.all_license_files.append(pkgvar)
+ Package.all_license_files[pkgvar] = value
elif pkgvar.endswith("_VERSION"):
if pkgvar.endswith("_DL_VERSION"):
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 10/12] support/scripts/pkg-stats: set status to 'na' for virtual packages
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
` (8 preceding siblings ...)
2020-03-02 14:50 ` [Buildroot] [PATCH v4 09/12] support/scripts/pkg-stats: add support for license hash check Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 11/12] support/scripts/pkg-stats: add list of status checks to the json output Heiko Thiery
` (2 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
If there is no infra set or infra is virtual the status is set to 'na'.
This is done for the follwing checks:
- license
- license-files
- hash
- hash-license
- patches
- version
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index c9287b416e..4cf1a919f3 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -132,6 +132,15 @@ class Package:
def patch_count(self):
return len(self.patch_files)
+ @property
+ def has_valid_infra(self):
+ try:
+ if self.infras[0][1] == 'virtual':
+ return False
+ except IndexError:
+ return False
+ return True
+
def set_infra(self):
"""
Fills in the .infras field
@@ -153,6 +162,11 @@ class Package:
"""
Fills in the .status['license'] and .status['license-files'] fields
"""
+ if not self.has_valid_infra:
+ self.status['license'] = ("na", "no valid package infra")
+ self.status['license-files'] = ("na", "no valid package infra")
+ return
+
var = self.pkgvar()
self.status['license'] = ("error", "missing")
self.status['license-files'] = ("error", "missing")
@@ -167,6 +181,11 @@ class Package:
"""
Fills in the .status['hash'] field
"""
+ if not self.has_valid_infra:
+ self.status['hash'] = ("na", "no valid package infra")
+ self.status['hash-license'] = ("na", "no valid package infra")
+ return
+
hashpath = self.path.replace(".mk", ".hash")
self.status['hash-license'] = ("na", "no hash file")
if os.path.exists(hashpath):
@@ -187,6 +206,10 @@ class Package:
"""
Fills in the .patch_count, .patch_files and .status['patches'] fields
"""
+ if not self.has_valid_infra:
+ self.status['patches'] = ("na", "no valid package infra")
+ return
+
pkgdir = os.path.dirname(self.path)
for subdir, _, _ in os.walk(pkgdir):
self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch')
@@ -564,6 +587,10 @@ def check_package_latest_version(packages):
for pkg, r in zip(packages, results):
pkg.latest_version = dict(zip(['status', 'version', 'id'], r))
+ if not pkg.has_valid_infra:
+ pkg.status['version'] = ("na", "no valid package infra")
+ continue
+
if pkg.latest_version['status'] == RM_API_STATUS_ERROR:
pkg.status['version'] = ('warning', "Release Monitoring API error")
elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND:
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 11/12] support/scripts/pkg-stats: add list of status checks to the json output
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
` (9 preceding siblings ...)
2020-03-02 14:50 ` [Buildroot] [PATCH v4 10/12] support/scripts/pkg-stats: set status to 'na' for virtual packages Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-02 14:50 ` [Buildroot] [PATCH v4 12/12] support/scripts/pkg-stats: add status for cve check Heiko Thiery
2020-03-06 16:46 ` [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Titouan Christophe
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 4cf1a919f3..9d058ca67d 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -88,6 +88,11 @@ class Package:
all_license_files = dict()
all_versions = dict()
all_ignored_cves = dict()
+ # This is the list of all possible checks. Add new checks to this list so
+ # a tool that post-processeds the json output knows the checks before
+ # iterating over the packages.
+ status_checks = ['cve', 'developers', 'hash', 'hash-license', 'license',
+ 'license-files', 'patches', 'pkg-check', 'url', 'version']
def __init__(self, name, path):
self.name = name
@@ -981,6 +986,7 @@ def dump_json(packages, defconfigs, stats, date, commit, output):
final = {'packages': pkgs,
'stats': statistics,
'defconfigs': defconfigs,
+ 'package_status_checks': Package.status_checks,
'commit': commit,
'date': str(date)}
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 12/12] support/scripts/pkg-stats: add status for cve check
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
` (10 preceding siblings ...)
2020-03-02 14:50 ` [Buildroot] [PATCH v4 11/12] support/scripts/pkg-stats: add list of status checks to the json output Heiko Thiery
@ 2020-03-02 14:50 ` Heiko Thiery
2020-03-06 16:46 ` [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Titouan Christophe
12 siblings, 0 replies; 15+ messages in thread
From: Heiko Thiery @ 2020-03-02 14:50 UTC (permalink / raw)
To: buildroot
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
---
support/scripts/pkg-stats | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 9d058ca67d..b0ae681c22 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -619,6 +619,12 @@ def check_package_cves(nvd_path, packages):
if pkg_name in packages and cve.affects(packages[pkg_name]):
packages[pkg_name].cves.append(cve.identifier)
+ for pkg_name, pkg in packages.items():
+ if len(pkg.cves) > 0:
+ pkg.status['cve'] = ('error', 'affected by cve')
+ else:
+ pkg.status['cve'] = ('ok', 'no cve found')
+
def calculate_stats(packages):
stats = defaultdict(int)
--
2.20.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements
2020-03-02 14:50 [Buildroot] [PATCH v4 00/12] pkg-stats json output improvements Heiko Thiery
` (11 preceding siblings ...)
2020-03-02 14:50 ` [Buildroot] [PATCH v4 12/12] support/scripts/pkg-stats: add status for cve check Heiko Thiery
@ 2020-03-06 16:46 ` Titouan Christophe
2020-03-06 18:32 ` Heiko Thiery
12 siblings, 1 reply; 15+ messages in thread
From: Titouan Christophe @ 2020-03-06 16:46 UTC (permalink / raw)
To: buildroot
Hello Heiko,
Thanks for sending this updated series.
However it does not apply nicely on top of master (probably because of
the last patch I sent to make pkg-stats compatible with Python3).
Would you care to resend ?
Titouan
On 3/2/20 3:50 PM, Heiko Thiery wrote:
> - add developers information to the packages
> - add supported defconfigs to json
> - add license information to json
> - add license file information to json
> - add patch files to json
> - add a more generic check status field to the packages for easier post
> processing. The following check status are availble: checks: cve, license,
> license-files, hash, hash-license,patches, version, url, developers,
> pkg-check
>
> ---
> v3 -> v4:
>
> Included the review comments from Titouan
>
> - use only one signed-off mail address
> - do some things in a more pythonic way
> - remove the patch_count attribute and use a class property instead
> (Note that the patch_count value will not go into the json output
> anymore,but then coutn can be computed by "len(patch_files)".)
> - remove the status initializer in the package class that was introduced
> in v3
> - update the cve status at the end of the cve check loop
> - change the is_valid_infra -> has_valid_infra and use it as property
>
> v2 -> v3:
> - keep variable latest_release but change format
> - add check for license file hashes
> - introduce a na status for the checks
> - add a list of all possible checks to the json output
> - add the cve check status
>
> v1 -> v2:
> - cleanup and recreation of patches
> - remove pkg name from dumping to json
> - use patch_files instead of combine count and files in dict
> - include getdevelopers.py to reuse Developers class
>
> ---
>
> Heiko Thiery (12):
> support/scripts/pkg-stats: store latest version in a dict
> support/scripts/pkg-stats: store patch files for the package
> support/scripts/pkg-stats: set developers info
> support/scripts/pkg-stats: store licences of package
> support/scripts/pkg-stats: add package status
> support/scripts/pkg-stats: add package count to stats
> support/scripts/pkg-stats: store pkg dir path
> support/scripts/pkg-stats: add defconfig support
> support/scripts/pkg-stats: add support for license hash check
> support/scripts/pkg-stats: set status to 'na' for virtual packages
> support/scripts/pkg-stats: initialize all package status checks
> support/scripts/pkg-stats: add status for cve check
>
> support/scripts/pkg-stats | 271 +++++++++++++++++++++++++++++---------
> 1 file changed, 211 insertions(+), 60 deletions(-)
>
^ permalink raw reply [flat|nested] 15+ messages in thread