[Buildroot] [PATCH v2 0/7] CPE validation

[Buildroot] [PATCH v2 0/7] CPE validation

[Thomas Petazzoni]

Hello,

We have recently merged support to allow packages to express their CPE
identifier, which is a unique identifier used by the NIST database to
identify software products, and associate CVEs to them.

This patch series extends that by ensuring that CPEs affected to
packages in Buildroot do exist in the official CPE dictionary from
NIST, and if they do not exist, assist in generating the XML snippet
that the CPE dictionary maintainers expect.

In more details:

 - PATCH 1 introduces a new "name" field in the show-info JSON output,
   with the raw name of the package. This is used in PATCH 2.

 - PATCH 2 fixes "make pkg-stats" to properly account for host
   packages, which were currently ignored. It makes use of the new
   "name" field introduced in PATCH 1 in the show-info JSON output.

 - PATCH 3 introduces support/scripts/cpedb.py, which is a new set of
   helper that is capable of downloading the CPE dictionary, parsing
   it, and generating XML snippets to submit new CPE entries.

 - PATCH 4 modifies support/scripts/pkg-stats to leverage the features
   of cpedb.py to include validation of CPE identifiers of Buildroot
   patches.

 - PATCH 5 adds support/scripts/gen-missing-cpe, which allows to
   generate the XML snippets expected by the CPE dictionary mainainers

 - PATCH 6 hooks up support/scripts/gen-missing-cpe as the "make
   missing-cpe" top-level make target

 - PATCH 7 extends the documentation with more details about
   Buildroot features related to vulnerability management.

Changes since v1:

 - Drop patches that have been merged upstream.

 - New patches adding the "name" property in the show-info JSON
   output, and using it so that "make pkg-stats" takes into account
   host packages.

 - Fix the parsing of the XML CPE databse for Python versions prior to
   3.7. This was reported by Matt, and fixed by Gr?gory Clement.

 - Improve how the namespaces are added to the XML file generated by
   "make missing-cpe", with the proper API instead of using a
   regexp. This was done by Gr?gory Clement.

 - Add encoding in the XML declaration found in the XML file generated
   by "make missing-cpe". This was done by Gr?gory Clement.

 - Reduced the size of the CPE database cache, with the side effect of
   also having prettier output for "make missing-cpe". This was done
   by Gr?gory Clement.

This is based on work from Matt Weber, which has been further
refactored/rewritten by Gr?gory Clement and myself.

To test this:

 $ make pkg-stats
 $ firefox output/pkg-stats.html
 $ make missing-cpe
 $ ls output/cpe-updates/

This work is also available at:

  https://github.com/tpetazzoni/buildroot/commits/cpe-validation

Thomas

Matt Weber (2):
  support/scripts/cpedb.py: new CPE XML helper
  support/scripts/gen-missing-cpe: add new script

Thomas Petazzoni (5):
  package/pkg-utils.mk: introduce "name" field in show-info output
  support/scripts/pkg-stats: properly handle host packages with -c
    option
  support/scripts/pkg-stats: check CPE existence in CPE dictionnary
  Makefile: add new missing-cpe target
  docs/manual: add details about vulnerability management

 Makefile                        |   9 ++
 docs/manual/common-usage.txt    |  42 +++++++
 package/pkg-utils.mk            |   1 +
 support/scripts/cpedb.py        | 203 ++++++++++++++++++++++++++++++++
 support/scripts/gen-missing-cpe |  65 ++++++++++
 support/scripts/pkg-stats       |  32 +++--
 6 files changed, 342 insertions(+), 10 deletions(-)
 create mode 100644 support/scripts/cpedb.py
 create mode 100755 support/scripts/gen-missing-cpe

-- 
2.29.2

[Buildroot] [PATCH v2 1/7] package/pkg-utils.mk: introduce "name" field in show-info output

[Thomas Petazzoni]

The keys of the JSON dict returned by "make show-info" is the package
name, including the "host-" prefix for host packages.

However, it is sometimes useful to get the actual name of the package,
without the "host-" prefix, so we add a "name" property that holds the
"raw name" of the package.

Suggested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-utils.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 11090cb628..d94041de24 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -96,6 +96,7 @@ endef
 # $(1): upper-case package or filesystem name
 define json-info
 	"$($(1)_NAME)": {
+		"name": "$($(1)_RAWNAME)",
 		"type": "$($(1)_TYPE)",
 		$(if $(filter rootfs,$($(1)_TYPE)), \
 			$(call _json-info-fs,$(1)), \
-- 
2.29.2

[Buildroot] [PATCH v2 2/7] support/scripts/pkg-stats: properly handle host packages with -c option

[Thomas Petazzoni]

In commit 7a607dab336e7f78ab069cff1b503d0688950583
("support/scripts/pkg-stats: support generating stats based on
configured packages"), we added a -c option to pkg-stats to generate a
report based on the list of packages enabled in the configuration,
rather than for all packages.

This is done based on the list of packages returned in JSON format by
"make show-info". However, we use the keys of the JSON dict returned
by "make show-info", which include the host- prefix of host
packages. Due to this, none of the host packages are currently
matching and therefore they are not reported in the pkg-stats -c
output.

This commit fixes that by using the recently introduced "name"
property in the "make show-info" JSON dict.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/pkg-stats | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index d222ba37c3..0d4e781e3b 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -358,7 +358,7 @@ def get_pkglist(npackages, package_list):
 def get_config_packages():
     cmd = ["make", "--no-print-directory", "show-info"]
     js = json.loads(subprocess.check_output(cmd))
-    return js.keys()
+    return set([v["name"] for k, v in js.items()])
 
 
 def package_init_make_info():
-- 
2.29.2

[Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper

[Thomas Petazzoni]

From: Matt Weber <matthew.weber@rockwellcollins.com>

Python class which consumes a NIST CPE XML and provides helper
functions to access and search the db's data.

 - Defines the CPE as a object with operations / formats
 - Processing of CPE dictionary

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>
Co-Developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 203 insertions(+)
 create mode 100644 support/scripts/cpedb.py

diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
new file mode 100644
index 0000000000..825ed6cb1e
--- /dev/null
+++ b/support/scripts/cpedb.py
@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+
+import xml.etree.ElementTree as ET
+from xml.etree.ElementTree import Element, SubElement
+import gzip
+import os
+import pickle
+import requests
+import time
+from xml.dom import minidom
+
+VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
+
+CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
+
+ns = {
+    '': 'http://cpe.mitre.org/dictionary/2.0',
+    'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
+    'xml': 'http://www.w3.org/XML/1998/namespace'
+}
+
+
+class CPE:
+    def __init__(self, cpe_str, titles, refs):
+        self.cpe_str = cpe_str
+        self.titles = titles
+        self.references = refs
+        self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
+
+    def update_xml_dict(self):
+        ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
+        cpes = Element('cpe-list')
+        cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")
+        cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
+        cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
+        cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
+        cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
+        cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
+        cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
+                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",
+                                                 "http://cpe.mitre.org/dictionary/2.0",
+                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
+                                                 "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
+                                                 "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
+                                                 "http://scap.nist.gov/schema/scap-core/0.3",
+                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
+                                                 "http://scap.nist.gov/schema/configuration/0.1",
+                                                 "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
+                                                 "http://scap.nist.gov/schema/scap-core/0.1",
+                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
+        item = SubElement(cpes, 'cpe-item')
+        cpe_short_name = CPE.short_name(self.cpe_str)
+        cpe_new_ver = CPE.version_update(self.cpe_str)
+
+        item.set('name', 'cpe:/' + cpe_short_name)
+        self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
+        for title in self.titles:
+            item.append(title)
+        if self.references:
+            item.append(self.references)
+        cpe23item = SubElement(item, 'cpe-23:cpe23-item')
+        cpe23item.set('name', self.cpe_str)
+
+        # Generate the XML as a string
+        xmlstr = ET.tostring(cpes)
+
+        # And use minidom to pretty print the XML
+        return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
+
+    @staticmethod
+    def version(cpe):
+        return cpe.split(":")[5]
+
+    @staticmethod
+    def product(cpe):
+        return cpe.split(":")[4]
+
+    @staticmethod
+    def short_name(cpe):
+        return ":".join(cpe.split(":")[2:6])
+
+    @staticmethod
+    def version_update(cpe):
+        return ":".join(cpe.split(":")[5:6])
+
+    @staticmethod
+    def no_version(cpe):
+        return ":".join(cpe.split(":")[:5])
+
+
+class CPEDB:
+    def __init__(self, nvd_path):
+        self.all_cpes = dict()
+        self.all_cpes_no_version = dict()
+        self.nvd_path = nvd_path
+
+    def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version):
+        print("CPE: Unzipping xml manifest...")
+        nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb'))
+        print("CPE: Converting xml manifest to dict...")
+        tree = ET.parse(nist_cpe_file)
+        all_cpedb = tree.getroot()
+        self.parse_dict(all_cpedb)
+
+        print("CPE: Caching dictionary")
+        cpes_file = open(cache_all_cpes, 'wb')
+        pickle.dump(self.all_cpes, cpes_file)
+        cpes_file.close()
+        cpes_file = open(cache_all_cpes_no_version, 'wb')
+        pickle.dump(self.all_cpes_no_version, cpes_file)
+        cpes_file.close()
+
+    def get_xml_dict(self):
+        print("CPE: Setting up NIST dictionary")
+        if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
+            os.makedirs(os.path.join(self.nvd_path, "cpe"))
+
+        cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
+        if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
+            print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
+            cpe_dict = requests.get(CPEDB_URL)
+            open(cpe_dict_local, "wb").write(cpe_dict.content)
+
+        cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl")
+        cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl")
+
+        if not os.path.exists(cache_all_cpes) or \
+           not os.path.exists(cache_all_cpes_no_version) or \
+           os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \
+           os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime:
+            self.gen_cached_cpedb(cpe_dict_local,
+                                  cache_all_cpes,
+                                  cache_all_cpes_no_version)
+
+        print("CPE: Loading CACHED dictionary")
+        cpe_file = open(cache_all_cpes, 'rb')
+        self.all_cpes = pickle.load(cpe_file)
+        cpe_file.close()
+        cpe_file = open(cache_all_cpes_no_version, 'rb')
+        self.all_cpes_no_version = pickle.load(cpe_file)
+        cpe_file.close()
+
+    def parse_dict(self, all_cpedb):
+        # Cycle through the dict and build two dict to be used for custom
+        # lookups of partial and complete CPE objects
+        # The objects are then used to create new proposed XML updates if
+        # if is determined one is required
+        # Out of the different language titles, select English
+        for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
+            cpe_titles = []
+            for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
+                title.tail = None
+                cpe_titles.append(title)
+
+            # Some older CPE don't include references, if they do, make
+            # sure we handle the case of one ref needing to be packed
+            # in a list
+            cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
+            if cpe_ref:
+                for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
+                    ref.tail = None
+                    ref.text = ref.text.upper()
+                    if ref.text not in VALID_REFS:
+                        ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
+                cpe_ref.tail = None
+                cpe_ref.text = None
+
+            cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
+            item = CPE(cpe_str, cpe_titles, cpe_ref)
+            cpe_str_no_version = CPE.no_version(cpe_str)
+            # This dict must have a unique key for every CPE version
+            # which allows matching to the specific obj data of that
+            # NIST dict entry
+            self.all_cpes.update({cpe_str: item})
+            # This dict has one entry for every CPE (w/o version) to allow
+            # partial match (no valid version) check (the obj is saved and
+            # used as seed for suggested xml updates. By updating the same
+            # non-version'd entry, it assumes the last update here is the
+            # latest version in the NIST dict)
+            self.all_cpes_no_version.update({cpe_str_no_version: item})
+
+    def find_partial(self, cpe_str):
+        cpe_str_no_version = CPE.no_version(cpe_str)
+        if cpe_str_no_version in self.all_cpes_no_version:
+            return cpe_str_no_version
+
+    def find_partial_obj(self, cpe_str):
+        cpe_str_no_version = CPE.no_version(cpe_str)
+        if cpe_str_no_version in self.all_cpes_no_version:
+            return self.all_cpes_no_version[cpe_str_no_version]
+
+    def find_partial_latest_version(self, cpe_str_partial):
+        cpe_obj = self.find_partial_obj(cpe_str_partial)
+        return cpe_obj.cpe_cur_ver
+
+    def find(self, cpe_str):
+        if self.find_partial(cpe_str):
+            if cpe_str in self.all_cpes:
+                return cpe_str
+
+    def gen_update_xml(self, cpe_str):
+        cpe = self.find_partial_obj(cpe_str)
+        return cpe.update_xml_dict()
-- 
2.29.2

[Buildroot] [PATCH v2 4/7] support/scripts/pkg-stats: check CPE existence in CPE dictionnary

[Thomas Petazzoni]

This commit extends pkg-stats to leverage the recently introduced
CPEDB class to verify that the CPEs provided by Buildroot packages are
indeed known in the official CPE dictionnary provided by NVD.

Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/pkg-stats | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 0d4e781e3b..f595026336 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -32,7 +32,7 @@ brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
 
 sys.path.append(os.path.join(brpath, "utils"))
 from getdeveloperlib import parse_developers  # noqa: E402
-
+from cpedb import CPEDB  # noqa: E402
 
 INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
 URL_RE = re.compile(r"\s*https?://\S*\s*$")
@@ -225,7 +225,8 @@ class Package:
 
         if var in self.all_cpeids:
             self.cpeid = self.all_cpeids[var]
-            self.status['cpe'] = ("ok", "verified CPE identifier")
+            # Set a preliminary status, it might be overridden by check_package_cpes()
+            self.status['cpe'] = ("warning", "not checked against CPE dictionnary")
         else:
             self.status['cpe'] = ("error", "no verified CPE identifier")
 
@@ -600,6 +601,18 @@ def check_package_cves(nvd_path, packages):
                 pkg.status['cve'] = ("ok", "not affected by CVEs")
 
 
+def check_package_cpes(nvd_path, packages):
+    cpedb = CPEDB(nvd_path)
+    cpedb.get_xml_dict()
+    for p in packages:
+        if not p.cpeid:
+            continue
+        if cpedb.find(p.cpeid):
+            p.status['cpe'] = ("ok", "verified CPE identifier")
+        else:
+            p.status['cpe'] = ("error", "CPE identifier unknown in CPE database")
+
+
 def calculate_stats(packages):
     stats = defaultdict(int)
     stats['packages'] = len(packages)
@@ -898,19 +911,17 @@ def dump_html_pkg(f, pkg):
 
     # CPE ID
     td_class = ["left"]
-    if pkg.status['cpe'][0] == "ok":
+    if pkg.is_status_ok("cpe"):
         td_class.append("cpe-ok")
-    elif pkg.status['cpe'][0] == "error":
+    elif pkg.is_status_error("cpe"):
         td_class.append("cpe-nok")
     else:
         td_class.append("cpe-unknown")
     f.write("  <td class=\"%s\">\n" % " ".join(td_class))
-    if pkg.status['cpe'][0] == "ok":
+    if pkg.cpeid:
         f.write("  <code>%s</code>\n" % pkg.cpeid)
-    elif pkg.status['cpe'][0] == "error":
-        f.write("  N/A\n")
-    else:
-        f.write("  %s\n" % pkg.status['cpe'][1])
+    if not pkg.is_status_ok("cpe"):
+        f.write("  %s%s\n" % ("<br/>" if pkg.cpeid else "", pkg.status['cpe'][1]))
     f.write("  </td>\n")
 
     f.write(" </tr>\n")
@@ -1106,6 +1117,7 @@ def __main__():
     if args.nvd_path:
         print("Checking packages CVEs")
         check_package_cves(args.nvd_path, packages)
+        check_package_cpes(args.nvd_path, packages)
     print("Calculate stats")
     stats = calculate_stats(packages)
     if args.html:
-- 
2.29.2

[Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script

[Thomas Petazzoni]

From: Matt Weber <matthew.weber@rockwellcollins.com>

This script queries the list of CPE IDs for the packages of the
current configuration (based on the "make show-info" output), and:

 - for CPE IDs that do not have any matching entry in the CPE
   database, it emits a warning

 - for CPE IDs that do have a matching entry, but not with the same
   version, it generates a snippet of XML that can be used to propose
   an updated version to NIST.

Ref: NIST has a group email (cpe_dictionary at nist.gov) used to
recieve these version update and new entry xml files.  They do
process the XML and provide feedback. In some cases they will
propose back something different where the vendor or version is
slightly different.

Limitations
 - Currently any use of non-number version identifiers isn't
   supported by NIST as they use ranges to determine impact
   of a CVE
 - Any Linux version from a non-upstream is also not supported
   without manually adjusting the information as the custom
   kernel will more then likely not match the upstream version
   used in the dictionary

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100755 support/scripts/gen-missing-cpe

diff --git a/support/scripts/gen-missing-cpe b/support/scripts/gen-missing-cpe
new file mode 100755
index 0000000000..ed7747295a
--- /dev/null
+++ b/support/scripts/gen-missing-cpe
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+
+import argparse
+import sys
+import json
+import subprocess
+import os
+from cpedb import CPEDB, CPE
+
+
+def gen_update_xml_reports(cpes, cpedb, output):
+    cpe_need_update = []
+
+    for cpe in cpes:
+        result = cpedb.find(cpe)
+        if not result:
+            result = cpedb.find_partial(CPE.no_version(cpe))
+            if result:
+                cpe_need_update.append(cpe)
+            else:
+                print("WARNING: no match found for '%s'" % cpe)
+
+    for cpe in cpe_need_update:
+        xml = cpedb.gen_update_xml(cpe)
+        fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
+        print("Generating %s" % fname)
+        fp = open(os.path.join(output, fname), 'w+')
+        fp.write(xml)
+        fp.close()
+
+    print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes)))
+
+
+def get_cpe_ids():
+    print("Getting list of CPE for enabled packages")
+    cmd = ["make", "--no-print-directory", "show-info"]
+    js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
+    return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
+
+
+def resolvepath(path):
+    return os.path.abspath(os.path.expanduser(path))
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output', dest='output',
+                        help='Path to the output CPE update files', type=resolvepath, required=True)
+    parser.add_argument('--nvd-path', dest='nvd_path',
+                        help='Path to the local NVD database', type=resolvepath, required=True)
+    return parser.parse_args()
+
+
+def __main__():
+    args = parse_args()
+    if not os.path.isdir(args.output):
+        print("ERROR: output directory %s does not exist" % args.output)
+        sys.exit(1)
+    cpedb = CPEDB(args.nvd_path)
+    cpedb.get_xml_dict()
+    cpes = get_cpe_ids()
+    gen_update_xml_reports(cpes, cpedb, args.output)
+
+
+__main__()
-- 
2.29.2

[Buildroot] [PATCH v2 6/7] Makefile: add new missing-cpe target

[Thomas Petazzoni]

It invokes the recently introduced gen-missing-cpe script.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 Makefile | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/Makefile b/Makefile
index 4d334adcd6..962931e0a5 100644
--- a/Makefile
+++ b/Makefile
@@ -945,6 +945,14 @@ pkg-stats:
 		--html $(O)/pkg-stats.html \
 		--nvd-path $(DL_DIR)/buildroot-nvd
 
+.PHONY: missing-cpe
+missing-cpe:
+	$(Q)mkdir -p $(O)/cpe-updates
+	$(Q)cd "$(CONFIG_DIR)" ; \
+	$(TOPDIR)/support/scripts/gen-missing-cpe \
+		--nvd-path $(DL_DIR)/buildroot-nvd \
+		--output $(O)/cpe-updates
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 # Some subdirectories are also package names. To avoid that "make linux"
@@ -1163,6 +1171,7 @@ help:
 	@echo '  legal-info             - generate info about license compliance'
 	@echo '  show-info              - generate info about packages, as a JSON blurb'
 	@echo '  pkg-stats              - generate info about packages as JSON and HTML'
+	@echo '  missing-cpe            - generate XML snippets for missing CPE identifiers'
 	@echo '  printvars              - dump internal variables selected with VARS=...'
 	@echo
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
-- 
2.29.2

[Buildroot] [PATCH v2 7/7] docs/manual: add details about vulnerability management

[Thomas Petazzoni]

This addition to the documentation gives some details about CPE/CVE,
pkg-stats and missing-cpe.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 docs/manual/common-usage.txt | 49 ++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/docs/manual/common-usage.txt b/docs/manual/common-usage.txt
index 9ba87a8339..e5d9d79420 100644
--- a/docs/manual/common-usage.txt
+++ b/docs/manual/common-usage.txt
@@ -417,6 +417,55 @@ effects:
   be empty and it's only at the very end of the build that they will
   be populated.
 
+[[vulnerability-management]]
+=== Vulnerability management
+
+Buildroot integrates a number of features that help tracking whether
+known security vulnerabilities are affecting some of the
+packages. Buildroot bases this logic on _CVEs_ (Common Vulnerabilities
+and Exposure) and the https://nvd.nist.gov/vuln[NVD database] provided
+by NIST.
+
+First, each package in Buildroot can be associated to a _CPE_
+identifier, by means of the various +FOO_CPE_ID_<something>+ variables
+that can be provided by each package. See
+xref:generic-package-reference[] for more details about those
+variables. When the _CPE_ identifier of a package is not defined, a
+default value is used, which may or may not be correct. This _CPE_
+identifier is used to find in the NVD database of vulnerabilities
+which _CVEs_ are affecting any given package.
+
+Based on this and the NIST databases, the +make pkg-stats+ target
+produces JSON and HTML formatted reports that indicate for each
+package of the current Buildroot configuration:
+
+* If there are known _CVEs_ for each package. Note that CVEs can
+  manually be ignored on a per-package basis using the
+  +FOO_IGNORED_CVES+ variable, for example if Buildroot has a patch
+  that fixes the vulnerability, or if the vulnerability is not
+  applicable to Buildroot for some reason.
+
+* If the _CPE_ identifier is valid and known in the
+  https://nvd.nist.gov/products/cpe[NIST CPE dictionary]. A _CPE_ is
+  valid is one of the +FOO_CPE_ID_<something>+ variable is defined, or
+  if +FOO_CPE_ID_VALID = YES+ validating that the default value of the
+  _CPE_ identifier is correct. In addition, _CPE_ are checked for
+  existence in the NIST CPE dictionary. If the _CPE_ identifier is not
+  known to the NIST CPE dictionary, then it is possible to contribute
+  to the NIST CPE dictionary: one need to submit a XML document that
+  describes the additional CPE identifier. Buildroot can help
+  generating such XML documents for missing CPE identifiers using the
+  +make missing-cpe+ target. The XML documents are generated in
+  +$(O)/cpe-updates+, will need to be edited before submission to
+  NIST. In particular, one should verify the version information and
+  check that the reference URLs are reachable with a valid type
+  (suggestions for types are provided in the generated file).  If a
+  previous _CPE_ entry was found in the NIST CPE dictionary, the XML
+  document generation tries to minimize the entry differences between
+  the existing versions in the dictionary. The XML document
+  generation uses the XML from the last version of that CPE as a
+  template for the new submission.
+
 include::eclipse-integration.txt[]
 
 include::advanced.txt[]
-- 
2.29.2

[Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper

[Yann E. MORIN]

Gr?gory, All,

On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:
> From: Matt Weber <matthew.weber@rockwellcollins.com>
> 
> Python class which consumes a NIST CPE XML and provides helper
> functions to access and search the db's data.
> 
>  - Defines the CPE as a object with operations / formats
>  - Processing of CPE dictionary
> 
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>

Gr?gory, you need to add your SOB line right after your CDB line.

Ditto for the following patch.

Regards,
Yann E. MORIN.

> Co-Developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 203 insertions(+)
>  create mode 100644 support/scripts/cpedb.py
> 
> diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
> new file mode 100644
> index 0000000000..825ed6cb1e
> --- /dev/null
> +++ b/support/scripts/cpedb.py
> @@ -0,0 +1,203 @@
> +#!/usr/bin/env python3
> +
> +import xml.etree.ElementTree as ET
> +from xml.etree.ElementTree import Element, SubElement
> +import gzip
> +import os
> +import pickle
> +import requests
> +import time
> +from xml.dom import minidom
> +
> +VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
> +
> +CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
> +
> +ns = {
> +    '': 'http://cpe.mitre.org/dictionary/2.0',
> +    'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
> +    'xml': 'http://www.w3.org/XML/1998/namespace'
> +}
> +
> +
> +class CPE:
> +    def __init__(self, cpe_str, titles, refs):
> +        self.cpe_str = cpe_str
> +        self.titles = titles
> +        self.references = refs
> +        self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
> +
> +    def update_xml_dict(self):
> +        ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
> +        cpes = Element('cpe-list')
> +        cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")
> +        cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
> +        cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
> +        cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
> +        cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
> +        cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
> +        cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
> +                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",
> +                                                 "http://cpe.mitre.org/dictionary/2.0",
> +                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
> +                                                 "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
> +                                                 "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
> +                                                 "http://scap.nist.gov/schema/scap-core/0.3",
> +                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
> +                                                 "http://scap.nist.gov/schema/configuration/0.1",
> +                                                 "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
> +                                                 "http://scap.nist.gov/schema/scap-core/0.1",
> +                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
> +        item = SubElement(cpes, 'cpe-item')
> +        cpe_short_name = CPE.short_name(self.cpe_str)
> +        cpe_new_ver = CPE.version_update(self.cpe_str)
> +
> +        item.set('name', 'cpe:/' + cpe_short_name)
> +        self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
> +        for title in self.titles:
> +            item.append(title)
> +        if self.references:
> +            item.append(self.references)
> +        cpe23item = SubElement(item, 'cpe-23:cpe23-item')
> +        cpe23item.set('name', self.cpe_str)
> +
> +        # Generate the XML as a string
> +        xmlstr = ET.tostring(cpes)
> +
> +        # And use minidom to pretty print the XML
> +        return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
> +
> +    @staticmethod
> +    def version(cpe):
> +        return cpe.split(":")[5]
> +
> +    @staticmethod
> +    def product(cpe):
> +        return cpe.split(":")[4]
> +
> +    @staticmethod
> +    def short_name(cpe):
> +        return ":".join(cpe.split(":")[2:6])
> +
> +    @staticmethod
> +    def version_update(cpe):
> +        return ":".join(cpe.split(":")[5:6])
> +
> +    @staticmethod
> +    def no_version(cpe):
> +        return ":".join(cpe.split(":")[:5])
> +
> +
> +class CPEDB:
> +    def __init__(self, nvd_path):
> +        self.all_cpes = dict()
> +        self.all_cpes_no_version = dict()
> +        self.nvd_path = nvd_path
> +
> +    def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version):
> +        print("CPE: Unzipping xml manifest...")
> +        nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb'))
> +        print("CPE: Converting xml manifest to dict...")
> +        tree = ET.parse(nist_cpe_file)
> +        all_cpedb = tree.getroot()
> +        self.parse_dict(all_cpedb)
> +
> +        print("CPE: Caching dictionary")
> +        cpes_file = open(cache_all_cpes, 'wb')
> +        pickle.dump(self.all_cpes, cpes_file)
> +        cpes_file.close()
> +        cpes_file = open(cache_all_cpes_no_version, 'wb')
> +        pickle.dump(self.all_cpes_no_version, cpes_file)
> +        cpes_file.close()
> +
> +    def get_xml_dict(self):
> +        print("CPE: Setting up NIST dictionary")
> +        if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
> +            os.makedirs(os.path.join(self.nvd_path, "cpe"))
> +
> +        cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
> +        if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
> +            print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
> +            cpe_dict = requests.get(CPEDB_URL)
> +            open(cpe_dict_local, "wb").write(cpe_dict.content)
> +
> +        cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl")
> +        cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl")
> +
> +        if not os.path.exists(cache_all_cpes) or \
> +           not os.path.exists(cache_all_cpes_no_version) or \
> +           os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \
> +           os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime:
> +            self.gen_cached_cpedb(cpe_dict_local,
> +                                  cache_all_cpes,
> +                                  cache_all_cpes_no_version)
> +
> +        print("CPE: Loading CACHED dictionary")
> +        cpe_file = open(cache_all_cpes, 'rb')
> +        self.all_cpes = pickle.load(cpe_file)
> +        cpe_file.close()
> +        cpe_file = open(cache_all_cpes_no_version, 'rb')
> +        self.all_cpes_no_version = pickle.load(cpe_file)
> +        cpe_file.close()
> +
> +    def parse_dict(self, all_cpedb):
> +        # Cycle through the dict and build two dict to be used for custom
> +        # lookups of partial and complete CPE objects
> +        # The objects are then used to create new proposed XML updates if
> +        # if is determined one is required
> +        # Out of the different language titles, select English
> +        for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
> +            cpe_titles = []
> +            for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
> +                title.tail = None
> +                cpe_titles.append(title)
> +
> +            # Some older CPE don't include references, if they do, make
> +            # sure we handle the case of one ref needing to be packed
> +            # in a list
> +            cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
> +            if cpe_ref:
> +                for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
> +                    ref.tail = None
> +                    ref.text = ref.text.upper()
> +                    if ref.text not in VALID_REFS:
> +                        ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
> +                cpe_ref.tail = None
> +                cpe_ref.text = None
> +
> +            cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
> +            item = CPE(cpe_str, cpe_titles, cpe_ref)
> +            cpe_str_no_version = CPE.no_version(cpe_str)
> +            # This dict must have a unique key for every CPE version
> +            # which allows matching to the specific obj data of that
> +            # NIST dict entry
> +            self.all_cpes.update({cpe_str: item})
> +            # This dict has one entry for every CPE (w/o version) to allow
> +            # partial match (no valid version) check (the obj is saved and
> +            # used as seed for suggested xml updates. By updating the same
> +            # non-version'd entry, it assumes the last update here is the
> +            # latest version in the NIST dict)
> +            self.all_cpes_no_version.update({cpe_str_no_version: item})
> +
> +    def find_partial(self, cpe_str):
> +        cpe_str_no_version = CPE.no_version(cpe_str)
> +        if cpe_str_no_version in self.all_cpes_no_version:
> +            return cpe_str_no_version
> +
> +    def find_partial_obj(self, cpe_str):
> +        cpe_str_no_version = CPE.no_version(cpe_str)
> +        if cpe_str_no_version in self.all_cpes_no_version:
> +            return self.all_cpes_no_version[cpe_str_no_version]
> +
> +    def find_partial_latest_version(self, cpe_str_partial):
> +        cpe_obj = self.find_partial_obj(cpe_str_partial)
> +        return cpe_obj.cpe_cur_ver
> +
> +    def find(self, cpe_str):
> +        if self.find_partial(cpe_str):
> +            if cpe_str in self.all_cpes:
> +                return cpe_str
> +
> +    def gen_update_xml(self, cpe_str):
> +        cpe = self.find_partial_obj(cpe_str)
> +        return cpe.update_xml_dict()
> -- 
> 2.29.2
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper

[Yann E. MORIN]

Gr?gory, All,

On 2021-01-31 23:47 +0100, Yann E. MORIN spake thusly:
> On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:
> > From: Matt Weber <matthew.weber@rockwellcollins.com>
> > 
> > Python class which consumes a NIST CPE XML and provides helper
> > functions to access and search the db's data.
> > 
> >  - Defines the CPE as a object with operations / formats
> >  - Processing of CPE dictionary
> > 
> > Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> > Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>
> 
> Gr?gory, you need to add your SOB line right after your CDB line.
> 
> Ditto for the following patch.

No need to resend the series, just reply to each patch with your SoB
line, I'll add them when applying.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2 2/7] support/scripts/pkg-stats: properly handle host packages with -c option

[Arnout Vandecappelle]

On 31/01/2021 14:38, Thomas Petazzoni wrote:
> In commit 7a607dab336e7f78ab069cff1b503d0688950583
> ("support/scripts/pkg-stats: support generating stats based on
> configured packages"), we added a -c option to pkg-stats to generate a
> report based on the list of packages enabled in the configuration,
> rather than for all packages.
> 
> This is done based on the list of packages returned in JSON format by
> "make show-info". However, we use the keys of the JSON dict returned
> by "make show-info", which include the host- prefix of host
> packages. Due to this, none of the host packages are currently
> matching and therefore they are not reported in the pkg-stats -c
> output.
> 
> This commit fixes that by using the recently introduced "name"
> property in the "make show-info" JSON dict.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

 This and the previous patch applied to master, thanks.

 The rest will have to wait for Gregory's SoB, but I'll review it.

 Regards,
 Arnout

> ---
>  support/scripts/pkg-stats | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index d222ba37c3..0d4e781e3b 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -358,7 +358,7 @@ def get_pkglist(npackages, package_list):
>  def get_config_packages():
>      cmd = ["make", "--no-print-directory", "show-info"]
>      js = json.loads(subprocess.check_output(cmd))
> -    return js.keys()
> +    return set([v["name"] for k, v in js.items()])
>  
>  
>  def package_init_make_info():
> 

[Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper

[Arnout Vandecappelle]

On 31/01/2021 14:38, Thomas Petazzoni wrote:
> From: Matt Weber <matthew.weber@rockwellcollins.com>
> 
> Python class which consumes a NIST CPE XML and provides helper
> functions to access and search the db's data.
> 
>  - Defines the CPE as a object with operations / formats
>  - Processing of CPE dictionary
> 
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>
> Co-Developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 203 insertions(+)
>  create mode 100644 support/scripts/cpedb.py
> 
> diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
> new file mode 100644
> index 0000000000..825ed6cb1e
> --- /dev/null
> +++ b/support/scripts/cpedb.py
> @@ -0,0 +1,203 @@
> +#!/usr/bin/env python3
> +
> +import xml.etree.ElementTree as ET
> +from xml.etree.ElementTree import Element, SubElement
> +import gzip
> +import os
> +import pickle
> +import requests
> +import time
> +from xml.dom import minidom
> +
> +VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
> +
> +CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
> +
> +ns = {
> +    '': 'http://cpe.mitre.org/dictionary/2.0',
> +    'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
> +    'xml': 'http://www.w3.org/XML/1998/namespace'
> +}
> +
> +
> +class CPE:
> +    def __init__(self, cpe_str, titles, refs):
> +        self.cpe_str = cpe_str
> +        self.titles = titles
> +        self.references = refs
> +        self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
> +
> +    def update_xml_dict(self):
> +        ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
> +        cpes = Element('cpe-list')
> +        cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")

 Note that instead of all these cpe.set calls, you can also do:

        cpes.attrib = {
            'xmlns:ns6': "http://scap.nist.gov/schema/scap-core/0.1",
            ...
        }

which IMO is more readable. But that's just nitpick.

 In fact, I think you can do it in the constructor:

        cpes = Element('cpe-list', attrib = { ...

but that is probably less readable.

> +        cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
> +        cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
> +        cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
> +        cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
> +        cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
> +        cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
> +                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",

 And here instead of join you can use:

        cpes.set('xsi:schemaLocation',
"http://scap.nist.gov/schema/cpe-extension/2.3 "
                                       "http://cpe.mitre.org/dictionary/2.0 "


(Python has string concatenation like C).


> +                                                 "http://cpe.mitre.org/dictionary/2.0",
> +                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
> +                                                 "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
> +                                                 "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
> +                                                 "http://scap.nist.gov/schema/scap-core/0.3",
> +                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
> +                                                 "http://scap.nist.gov/schema/configuration/0.1",
> +                                                 "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
> +                                                 "http://scap.nist.gov/schema/scap-core/0.1",
> +                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
> +        item = SubElement(cpes, 'cpe-item')
> +        cpe_short_name = CPE.short_name(self.cpe_str)
> +        cpe_new_ver = CPE.version_update(self.cpe_str)
> +
> +        item.set('name', 'cpe:/' + cpe_short_name)
> +        self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
> +        for title in self.titles:
> +            item.append(title)
> +        if self.references:
> +            item.append(self.references)
> +        cpe23item = SubElement(item, 'cpe-23:cpe23-item')
> +        cpe23item.set('name', self.cpe_str)
> +
> +        # Generate the XML as a string
> +        xmlstr = ET.tostring(cpes)
> +
> +        # And use minidom to pretty print the XML
> +        return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
> +
> +    @staticmethod
> +    def version(cpe):
> +        return cpe.split(":")[5]
> +
> +    @staticmethod
> +    def product(cpe):
> +        return cpe.split(":")[4]
> +
> +    @staticmethod
> +    def short_name(cpe):
> +        return ":".join(cpe.split(":")[2:6])
> +
> +    @staticmethod
> +    def version_update(cpe):

 It's weird to make this a staticmethod if the only caller takes self.cpe_str as
an argument, but OK.

> +        return ":".join(cpe.split(":")[5:6])
> +
> +    @staticmethod
> +    def no_version(cpe):
> +        return ":".join(cpe.split(":")[:5])
> +
> +
> +class CPEDB:
> +    def __init__(self, nvd_path):
> +        self.all_cpes = dict()
> +        self.all_cpes_no_version = dict()
> +        self.nvd_path = nvd_path
> +
> +    def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version):
> +        print("CPE: Unzipping xml manifest...")
> +        nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb'))
> +        print("CPE: Converting xml manifest to dict...")
> +        tree = ET.parse(nist_cpe_file)
> +        all_cpedb = tree.getroot()
> +        self.parse_dict(all_cpedb)
> +
> +        print("CPE: Caching dictionary")
> +        cpes_file = open(cache_all_cpes, 'wb')
> +        pickle.dump(self.all_cpes, cpes_file)
> +        cpes_file.close()
> +        cpes_file = open(cache_all_cpes_no_version, 'wb')
> +        pickle.dump(self.all_cpes_no_version, cpes_file)
> +        cpes_file.close()
> +
> +    def get_xml_dict(self):
> +        print("CPE: Setting up NIST dictionary")
> +        if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
> +            os.makedirs(os.path.join(self.nvd_path, "cpe"))
> +
> +        cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
> +        if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:

 This should be a parameter@the beginning of the file rather than a constant
here. Ideally expressed in hours, with *3600 here.

> +            print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
> +            cpe_dict = requests.get(CPEDB_URL)
> +            open(cpe_dict_local, "wb").write(cpe_dict.content)
> +
> +        cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl")
> +        cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl")
> +
> +        if not os.path.exists(cache_all_cpes) or \
> +           not os.path.exists(cache_all_cpes_no_version) or \
> +           os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \
> +           os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime:
> +            self.gen_cached_cpedb(cpe_dict_local,
> +                                  cache_all_cpes,
> +                                  cache_all_cpes_no_version)
> +
> +        print("CPE: Loading CACHED dictionary")
> +        cpe_file = open(cache_all_cpes, 'rb')
> +        self.all_cpes = pickle.load(cpe_file)
> +        cpe_file.close()

 The above is better done as

        with open(cache_all_cpes, 'rb') as cpe_file:
            self.all_cpes = pickle.load(cpe_file)

This do things properly in case of exception.

 However, I'm not so sure if the pickled cache is really worth it. I gave it a
go, and the ET.parse + parse_dict took roughly 20 seconds, while just loading
the pickle file also took about 20 seconds...


> +        cpe_file = open(cache_all_cpes_no_version, 'rb')
> +        self.all_cpes_no_version = pickle.load(cpe_file)
> +        cpe_file.close()
> +
> +    def parse_dict(self, all_cpedb):
> +        # Cycle through the dict and build two dict to be used for custom
> +        # lookups of partial and complete CPE objects
> +        # The objects are then used to create new proposed XML updates if
> +        # if is determined one is required
> +        # Out of the different language titles, select English

 This could have been a docstring.

 Actually, all "public" functions should really have a docstring.



 So, I accidentally applied this patch... So all the above comments are just
informational - feel free to send follow-up patches, or ignore my feedback.

 Regards,
 Arnout

> +        for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
> +            cpe_titles = []
> +            for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
> +                title.tail = None
> +                cpe_titles.append(title)
> +
> +            # Some older CPE don't include references, if they do, make
> +            # sure we handle the case of one ref needing to be packed
> +            # in a list
> +            cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
> +            if cpe_ref:
> +                for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
> +                    ref.tail = None
> +                    ref.text = ref.text.upper()
> +                    if ref.text not in VALID_REFS:
> +                        ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
> +                cpe_ref.tail = None
> +                cpe_ref.text = None
> +
> +            cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
> +            item = CPE(cpe_str, cpe_titles, cpe_ref)
> +            cpe_str_no_version = CPE.no_version(cpe_str)
> +            # This dict must have a unique key for every CPE version
> +            # which allows matching to the specific obj data of that
> +            # NIST dict entry
> +            self.all_cpes.update({cpe_str: item})
> +            # This dict has one entry for every CPE (w/o version) to allow
> +            # partial match (no valid version) check (the obj is saved and
> +            # used as seed for suggested xml updates. By updating the same
> +            # non-version'd entry, it assumes the last update here is the
> +            # latest version in the NIST dict)
> +            self.all_cpes_no_version.update({cpe_str_no_version: item})
> +
> +    def find_partial(self, cpe_str):
> +        cpe_str_no_version = CPE.no_version(cpe_str)
> +        if cpe_str_no_version in self.all_cpes_no_version:
> +            return cpe_str_no_version
> +
> +    def find_partial_obj(self, cpe_str):
> +        cpe_str_no_version = CPE.no_version(cpe_str)
> +        if cpe_str_no_version in self.all_cpes_no_version:
> +            return self.all_cpes_no_version[cpe_str_no_version]
> +
> +    def find_partial_latest_version(self, cpe_str_partial):
> +        cpe_obj = self.find_partial_obj(cpe_str_partial)
> +        return cpe_obj.cpe_cur_ver
> +
> +    def find(self, cpe_str):
> +        if self.find_partial(cpe_str):
> +            if cpe_str in self.all_cpes:
> +                return cpe_str
> +
> +    def gen_update_xml(self, cpe_str):
> +        cpe = self.find_partial_obj(cpe_str)
> +        return cpe.update_xml_dict()
> 

[Buildroot] [PATCH v2 4/7] support/scripts/pkg-stats: check CPE existence in CPE dictionnary

[Arnout Vandecappelle]

On 31/01/2021 14:38, Thomas Petazzoni wrote:
> This commit extends pkg-stats to leverage the recently introduced
> CPEDB class to verify that the CPEs provided by Buildroot packages are
> indeed known in the official CPE dictionnary provided by NVD.
> 
> Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

 This one was accidentally applied to master without Gre?gory's SoB as well.

 I didn't have any further comments on the patch. Imagine that!

 Regards,
 Arnout

[Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script

[Arnout Vandecappelle]

On 31/01/2021 14:38, Thomas Petazzoni wrote:
> From: Matt Weber <matthew.weber@rockwellcollins.com>
> 
> This script queries the list of CPE IDs for the packages of the
> current configuration (based on the "make show-info" output), and:
> 
>  - for CPE IDs that do not have any matching entry in the CPE
>    database, it emits a warning

 Maybe there should also be a warning for packages which don't have cpeid set at
all...

> 
>  - for CPE IDs that do have a matching entry, but not with the same
>    version, it generates a snippet of XML that can be used to propose
>    an updated version to NIST.
> 
> Ref: NIST has a group email (cpe_dictionary at nist.gov) used to
> recieve these version update and new entry xml files.  They do
> process the XML and provide feedback. In some cases they will
> propose back something different where the vendor or version is
> slightly different.

 It would be very useful if the script would also print a URL that describes the
submission process.

> 
> Limitations
>  - Currently any use of non-number version identifiers isn't
>    supported by NIST as they use ranges to determine impact
>    of a CVE
>  - Any Linux version from a non-upstream is also not supported
>    without manually adjusting the information as the custom
>    kernel will more then likely not match the upstream version
>    used in the dictionary
> 
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++
>  1 file changed, 65 insertions(+)
>  create mode 100755 support/scripts/gen-missing-cpe
> 
> diff --git a/support/scripts/gen-missing-cpe b/support/scripts/gen-missing-cpe
> new file mode 100755
> index 0000000000..ed7747295a
> --- /dev/null
> +++ b/support/scripts/gen-missing-cpe
> @@ -0,0 +1,65 @@
> +#!/usr/bin/env python3
> +
> +import argparse
> +import sys
> +import json
> +import subprocess
> +import os
> +from cpedb import CPEDB, CPE
> +
> +
> +def gen_update_xml_reports(cpes, cpedb, output):

 cpes should be cpeids (I first thought it was a list of CPE objects from cpedb).

> +    cpe_need_update = []
> +
> +    for cpe in cpes:

    for cpeid in cpeids:

> +        result = cpedb.find(cpe)
> +        if not result:
> +            result = cpedb.find_partial(CPE.no_version(cpe))
> +            if result:
> +                cpe_need_update.append(cpe)
> +            else:
> +                print("WARNING: no match found for '%s'" % cpe)
> +
> +    for cpe in cpe_need_update:
> +        xml = cpedb.gen_update_xml(cpe)
> +        fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
> +        print("Generating %s" % fname)
> +        fp = open(os.path.join(output, fname), 'w+')
> +        fp.write(xml)
> +        fp.close()

 This should be

        with open(...) as fp:
            fp.write(xml)

> +
> +    print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes)))
> +
> +
> +def get_cpe_ids():
> +    print("Getting list of CPE for enabled packages")
> +    cmd = ["make", "--no-print-directory", "show-info"]
> +    js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
> +    return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
> +
> +
> +def resolvepath(path):
> +    return os.path.abspath(os.path.expanduser(path))

 I don't understand this...

- expanduser should already have been done by the shell. If you call the script as

gen-missing-cpe --output \~/some-path

then I want the output to be in a directory called ~ in the current directory.
That's what the rest of the world does. (Yes, I get it, you want to be able to
do --output=~/some-path but that's just wrong.)

- abspath shouldn't be needed for anything, we're not doing any changedir or
anything like that.

In addition, this script is supposed to be called from `make missing-cpe` which
has already done those expansions.

> +
> +
> +def parse_args():
> +    parser = argparse.ArgumentParser()
> +    parser.add_argument('--output', dest='output',
> +                        help='Path to the output CPE update files', type=resolvepath, required=True)
> +    parser.add_argument('--nvd-path', dest='nvd_path',
> +                        help='Path to the local NVD database', type=resolvepath, required=True)
> +    return parser.parse_args()
> +
> +
> +def __main__():
> +    args = parse_args()
> +    if not os.path.isdir(args.output):
> +        print("ERROR: output directory %s does not exist" % args.output)
> +        sys.exit(1)
> +    cpedb = CPEDB(args.nvd_path)
> +    cpedb.get_xml_dict()
> +    cpes = get_cpe_ids()
> +    gen_update_xml_reports(cpes, cpedb, args.output)
> +
> +
> +__main__()

Usually it would be

if __name__ == '__main__':
    __main__()



 Nothing important here, but I want to give you the chance to rework if you
think it's worth it, so just

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>


 Regards,
 Arnout

[Buildroot] [PATCH v2 6/7] Makefile: add new missing-cpe target

[Arnout Vandecappelle]

On 31/01/2021 14:38, Thomas Petazzoni wrote:
> It invokes the recently introduced gen-missing-cpe script.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

> ---
>  Makefile | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 4d334adcd6..962931e0a5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -945,6 +945,14 @@ pkg-stats:
>  		--html $(O)/pkg-stats.html \
>  		--nvd-path $(DL_DIR)/buildroot-nvd
>  
> +.PHONY: missing-cpe
> +missing-cpe:
> +	$(Q)mkdir -p $(O)/cpe-updates
> +	$(Q)cd "$(CONFIG_DIR)" ; \
> +	$(TOPDIR)/support/scripts/gen-missing-cpe \
> +		--nvd-path $(DL_DIR)/buildroot-nvd \
> +		--output $(O)/cpe-updates
> +
>  else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
>  
>  # Some subdirectories are also package names. To avoid that "make linux"
> @@ -1163,6 +1171,7 @@ help:
>  	@echo '  legal-info             - generate info about license compliance'
>  	@echo '  show-info              - generate info about packages, as a JSON blurb'
>  	@echo '  pkg-stats              - generate info about packages as JSON and HTML'
> +	@echo '  missing-cpe            - generate XML snippets for missing CPE identifiers'
>  	@echo '  printvars              - dump internal variables selected with VARS=...'
>  	@echo
>  	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
> 

[Buildroot] [PATCH v2 7/7] docs/manual: add details about vulnerability management

[Arnout Vandecappelle]

On 31/01/2021 14:38, Thomas Petazzoni wrote:
> This addition to the documentation gives some details about CPE/CVE,
> pkg-stats and missing-cpe.

 That was sorely needed!

> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  docs/manual/common-usage.txt | 49 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 49 insertions(+)
> 
> diff --git a/docs/manual/common-usage.txt b/docs/manual/common-usage.txt
> index 9ba87a8339..e5d9d79420 100644
> --- a/docs/manual/common-usage.txt
> +++ b/docs/manual/common-usage.txt
> @@ -417,6 +417,55 @@ effects:
>    be empty and it's only at the very end of the build that they will
>    be populated.
>  
> +[[vulnerability-management]]
> +=== Vulnerability management
> +
> +Buildroot integrates a number of features that help tracking whether
> +known security vulnerabilities are affecting some of the
> +packages. Buildroot bases this logic on _CVEs_ (Common Vulnerabilities
> +and Exposure) and the https://nvd.nist.gov/vuln[NVD database] provided
> +by NIST.
> +
> +First, each package in Buildroot can be associated to a _CPE_
> +identifier, by means of the various +FOO_CPE_ID_<something>+ variables
> +that can be provided by each package. See
> +xref:generic-package-reference[] for more details about those
> +variables. When the _CPE_ identifier of a package is not defined, a
> +default value is used, which may or may not be correct.

 I don't think this reflects the reality very well. cpeid is only used if
*something* is set in the .mk file. Something like:

When no +FOO_CPE_ID_<something>+ variable is set for a package, it is not
considered as having a valid _CPE_ identifier.


> This _CPE_
> +identifier is used to find in the NVD database of vulnerabilities
> +which _CVEs_ are affecting any given package.

 ... If a package doesn't have a valid CPE identifier, a guess is made based on
the package name and version.

> +
> +Based on this and the NIST databases, the +make pkg-stats+ target
> +produces JSON and HTML formatted reports that indicate for each
> +package of the current Buildroot configuration:
> +
> +* If there are known _CVEs_ for each package. Note that CVEs can
> +  manually be ignored on a per-package basis using the
> +  +FOO_IGNORED_CVES+ variable, for example if Buildroot has a patch
> +  that fixes the vulnerability, or if the vulnerability is not
> +  applicable to Buildroot for some reason.
> +
> +* If the _CPE_ identifier is valid and known in the
> +  https://nvd.nist.gov/products/cpe[NIST CPE dictionary]. A _CPE_ is
> +  valid is one of the +FOO_CPE_ID_<something>+ variable is defined, or
> +  if +FOO_CPE_ID_VALID = YES+ validating that the default value of the
> +  _CPE_ identifier is correct. In addition, _CPE_ are checked for
> +  existence in the NIST CPE dictionary. If the _CPE_ identifier is not
> +  known to the NIST CPE dictionary, then it is possible to contribute
> +  to the NIST CPE dictionary: one need to submit a XML document that
                                     needs

> +  describes the additional CPE identifier. 

 Can you add a reference to a web page that describes the procedure? Otherwise
it's not terribly helpful IMHO.

> Buildroot can help
> +  generating such XML documents for missing CPE identifiers using the
> +  +make missing-cpe+ target. The XML documents are generated in
> +  +$(O)/cpe-updates+, will need to be edited before submission to
> +  NIST. In particular, one should verify the version information and
> +  check that the reference URLs are reachable with a valid type
> +  (suggestions for types are provided in the generated file).  If a
> +  previous _CPE_ entry was found in the NIST CPE dictionary, the XML

 If no previous CPE entry exists (i.e., the CPE_ID specified in the .mk file is
not valid), then no XML file is generated at all... AFAICS there is no help at
the moment for generating the XML for completely new entries.

 My brain is too tired ATM to try to formulate the documentation that expresses
that.

 I've marked this one changes requested in patchwork.

 Regards,
 Arnout


> +  document generation tries to minimize the entry differences between
> +  the existing versions in the dictionary. The XML document
> +  generation uses the XML from the last version of that CPE as a
> +  template for the new submission.
> +
>  include::eclipse-integration.txt[]
>  
>  include::advanced.txt[]
> 

[Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script

[Matthew Weber]

Thomas,


On Tue, Feb 2, 2021 at 3:29 PM Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
>
> On 31/01/2021 14:38, Thomas Petazzoni wrote:
> > From: Matt Weber <matthew.weber@rockwellcollins.com>
> >
> > This script queries the list of CPE IDs for the packages of the
> > current configuration (based on the "make show-info" output), and:
> >
> >  - for CPE IDs that do not have any matching entry in the CPE
> >    database, it emits a warning
>
>  Maybe there should also be a warning for packages which don't have cpeid set at
> all...

Agree, this would be something that people would miss if we don't list
the missing.  My original iteration had that as a category that was
printed to screen but no XML generated for it.

Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>

Regards,
Matt

[Buildroot] [PATCH v2 6/7] Makefile: add new missing-cpe target

[Matthew Weber]

Thomas,

On Tue, Feb 2, 2021 at 3:30 PM Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
>
> On 31/01/2021 14:38, Thomas Petazzoni wrote:
> > It invokes the recently introduced gen-missing-cpe script.
> >
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>

Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>

[Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script

[Yann E. MORIN]

Thomas, All,

On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:
> From: Matt Weber <matthew.weber@rockwellcollins.com>
> 
> This script queries the list of CPE IDs for the packages of the
> current configuration (based on the "make show-info" output), and:
> 
>  - for CPE IDs that do not have any matching entry in the CPE
>    database, it emits a warning
> 
>  - for CPE IDs that do have a matching entry, but not with the same
>    version, it generates a snippet of XML that can be used to propose
>    an updated version to NIST.
> 
> Ref: NIST has a group email (cpe_dictionary at nist.gov) used to
> recieve these version update and new entry xml files.  They do
> process the XML and provide feedback. In some cases they will
> propose back something different where the vendor or version is
> slightly different.
> 
> Limitations
>  - Currently any use of non-number version identifiers isn't
>    supported by NIST as they use ranges to determine impact
>    of a CVE
>  - Any Linux version from a non-upstream is also not supported
>    without manually adjusting the information as the custom
>    kernel will more then likely not match the upstream version
>    used in the dictionary
> 
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Applied to master, with most of the changes pointed out by Arnout,
thanks. I'll further reply to Arnout's comment...

Regards,
Yann E. MORIN.

> ---
>  support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++
>  1 file changed, 65 insertions(+)
>  create mode 100755 support/scripts/gen-missing-cpe
> 
> diff --git a/support/scripts/gen-missing-cpe b/support/scripts/gen-missing-cpe
> new file mode 100755
> index 0000000000..ed7747295a
> --- /dev/null
> +++ b/support/scripts/gen-missing-cpe
> @@ -0,0 +1,65 @@
> +#!/usr/bin/env python3
> +
> +import argparse
> +import sys
> +import json
> +import subprocess
> +import os
> +from cpedb import CPEDB, CPE
> +
> +
> +def gen_update_xml_reports(cpes, cpedb, output):
> +    cpe_need_update = []
> +
> +    for cpe in cpes:
> +        result = cpedb.find(cpe)
> +        if not result:
> +            result = cpedb.find_partial(CPE.no_version(cpe))
> +            if result:
> +                cpe_need_update.append(cpe)
> +            else:
> +                print("WARNING: no match found for '%s'" % cpe)
> +
> +    for cpe in cpe_need_update:
> +        xml = cpedb.gen_update_xml(cpe)
> +        fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
> +        print("Generating %s" % fname)
> +        fp = open(os.path.join(output, fname), 'w+')
> +        fp.write(xml)
> +        fp.close()
> +
> +    print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes)))
> +
> +
> +def get_cpe_ids():
> +    print("Getting list of CPE for enabled packages")
> +    cmd = ["make", "--no-print-directory", "show-info"]
> +    js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
> +    return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
> +
> +
> +def resolvepath(path):
> +    return os.path.abspath(os.path.expanduser(path))
> +
> +
> +def parse_args():
> +    parser = argparse.ArgumentParser()
> +    parser.add_argument('--output', dest='output',
> +                        help='Path to the output CPE update files', type=resolvepath, required=True)
> +    parser.add_argument('--nvd-path', dest='nvd_path',
> +                        help='Path to the local NVD database', type=resolvepath, required=True)
> +    return parser.parse_args()
> +
> +
> +def __main__():
> +    args = parse_args()
> +    if not os.path.isdir(args.output):
> +        print("ERROR: output directory %s does not exist" % args.output)
> +        sys.exit(1)
> +    cpedb = CPEDB(args.nvd_path)
> +    cpedb.get_xml_dict()
> +    cpes = get_cpe_ids()
> +    gen_update_xml_reports(cpes, cpedb, args.output)
> +
> +
> +__main__()
> -- 
> 2.29.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script

[Yann E. MORIN]

Arnout, All,

On 2021-02-02 22:29 +0100, Arnout Vandecappelle spake thusly:
> On 31/01/2021 14:38, Thomas Petazzoni wrote:
> > From: Matt Weber <matthew.weber@rockwellcollins.com>
> > 
> > This script queries the list of CPE IDs for the packages of the
> > current configuration (based on the "make show-info" output), and:
> > 
> >  - for CPE IDs that do not have any matching entry in the CPE
> >    database, it emits a warning
>  Maybe there should also be a warning for packages which don't have cpeid set at
> all...

I haven't done anything to do so; I though it was better to have at
least this script in its current state, rather than nothing at all.

[--SNIP--]
> > +def gen_update_xml_reports(cpes, cpedb, output):
>  cpes should be cpeids (I first thought it was a list of CPE objects from cpedb).

Done.

[--SNIP--]
> > +        fp = open(os.path.join(output, fname), 'w+')
> > +        fp.write(xml)
> > +        fp.close()
>  This should be
>         with open(...) as fp:
>             fp.write(xml)

Done.

[--SNIP--]
> > +def resolvepath(path):
> > +    return os.path.abspath(os.path.expanduser(path))
> 
>  I don't understand this...
> 
> - expanduser should already have been done by the shell. If you call the script as
> 
> gen-missing-cpe --output \~/some-path
> 
> then I want the output to be in a directory called ~ in the current directory.
> That's what the rest of the world does. (Yes, I get it, you want to be able to
> do --output=~/some-path but that's just wrong.)
> 
> - abspath shouldn't be needed for anything, we're not doing any changedir or
> anything like that.
> 
> In addition, this script is supposed to be called from `make missing-cpe` which
> has already done those expansions.

It was not obvious to me either that resolvepath() was needed, and as]
you said, the usage we have in Buildroot does not need expansion.
However, getting rid of it seemed too much, so I left it as-is.

Again, I thought it was better that we have this script rather than
nothing.

[--SNIP--]
> > +__main__()
> Usually it would be
> if __name__ == '__main__':
>     __main__()

Done.

>  Nothing important here, but I want to give you the chance to rework if you
> think it's worth it, so just
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

This was sitting in the backlog for quite a while, and we really needed
to push a bit on our CPE (and CVE) tooling, so I applied this to master
now.

Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'