* [Buildroot] [PATCH v2 1/7] package/pkg-utils.mk: introduce "name" field in show-info output
2021-01-31 13:38 [Buildroot] [PATCH v2 0/7] CPE validation Thomas Petazzoni
@ 2021-01-31 13:38 ` Thomas Petazzoni
2021-01-31 13:38 ` [Buildroot] [PATCH v2 2/7] support/scripts/pkg-stats: properly handle host packages with -c option Thomas Petazzoni
` (5 subsequent siblings)
6 siblings, 0 replies; 20+ messages in thread
From: Thomas Petazzoni @ 2021-01-31 13:38 UTC (permalink / raw)
To: buildroot
The keys of the JSON dict returned by "make show-info" is the package
name, including the "host-" prefix for host packages.
However, it is sometimes useful to get the actual name of the package,
without the "host-" prefix, so we add a "name" property that holds the
"raw name" of the package.
Suggested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/pkg-utils.mk | 1 +
1 file changed, 1 insertion(+)
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 11090cb628..d94041de24 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -96,6 +96,7 @@ endef
# $(1): upper-case package or filesystem name
define json-info
"$($(1)_NAME)": {
+ "name": "$($(1)_RAWNAME)",
"type": "$($(1)_TYPE)",
$(if $(filter rootfs,$($(1)_TYPE)), \
$(call _json-info-fs,$(1)), \
--
2.29.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* [Buildroot] [PATCH v2 2/7] support/scripts/pkg-stats: properly handle host packages with -c option
2021-01-31 13:38 [Buildroot] [PATCH v2 0/7] CPE validation Thomas Petazzoni
2021-01-31 13:38 ` [Buildroot] [PATCH v2 1/7] package/pkg-utils.mk: introduce "name" field in show-info output Thomas Petazzoni
@ 2021-01-31 13:38 ` Thomas Petazzoni
2021-02-02 19:29 ` Arnout Vandecappelle
2021-01-31 13:38 ` [Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper Thomas Petazzoni
` (4 subsequent siblings)
6 siblings, 1 reply; 20+ messages in thread
From: Thomas Petazzoni @ 2021-01-31 13:38 UTC (permalink / raw)
To: buildroot
In commit 7a607dab336e7f78ab069cff1b503d0688950583
("support/scripts/pkg-stats: support generating stats based on
configured packages"), we added a -c option to pkg-stats to generate a
report based on the list of packages enabled in the configuration,
rather than for all packages.
This is done based on the list of packages returned in JSON format by
"make show-info". However, we use the keys of the JSON dict returned
by "make show-info", which include the host- prefix of host
packages. Due to this, none of the host packages are currently
matching and therefore they are not reported in the pkg-stats -c
output.
This commit fixes that by using the recently introduced "name"
property in the "make show-info" JSON dict.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
support/scripts/pkg-stats | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index d222ba37c3..0d4e781e3b 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -358,7 +358,7 @@ def get_pkglist(npackages, package_list):
def get_config_packages():
cmd = ["make", "--no-print-directory", "show-info"]
js = json.loads(subprocess.check_output(cmd))
- return js.keys()
+ return set([v["name"] for k, v in js.items()])
def package_init_make_info():
--
2.29.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* [Buildroot] [PATCH v2 2/7] support/scripts/pkg-stats: properly handle host packages with -c option
2021-01-31 13:38 ` [Buildroot] [PATCH v2 2/7] support/scripts/pkg-stats: properly handle host packages with -c option Thomas Petazzoni
@ 2021-02-02 19:29 ` Arnout Vandecappelle
0 siblings, 0 replies; 20+ messages in thread
From: Arnout Vandecappelle @ 2021-02-02 19:29 UTC (permalink / raw)
To: buildroot
On 31/01/2021 14:38, Thomas Petazzoni wrote:
> In commit 7a607dab336e7f78ab069cff1b503d0688950583
> ("support/scripts/pkg-stats: support generating stats based on
> configured packages"), we added a -c option to pkg-stats to generate a
> report based on the list of packages enabled in the configuration,
> rather than for all packages.
>
> This is done based on the list of packages returned in JSON format by
> "make show-info". However, we use the keys of the JSON dict returned
> by "make show-info", which include the host- prefix of host
> packages. Due to this, none of the host packages are currently
> matching and therefore they are not reported in the pkg-stats -c
> output.
>
> This commit fixes that by using the recently introduced "name"
> property in the "make show-info" JSON dict.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This and the previous patch applied to master, thanks.
The rest will have to wait for Gregory's SoB, but I'll review it.
Regards,
Arnout
> ---
> support/scripts/pkg-stats | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index d222ba37c3..0d4e781e3b 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -358,7 +358,7 @@ def get_pkglist(npackages, package_list):
> def get_config_packages():
> cmd = ["make", "--no-print-directory", "show-info"]
> js = json.loads(subprocess.check_output(cmd))
> - return js.keys()
> + return set([v["name"] for k, v in js.items()])
>
>
> def package_init_make_info():
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper
2021-01-31 13:38 [Buildroot] [PATCH v2 0/7] CPE validation Thomas Petazzoni
2021-01-31 13:38 ` [Buildroot] [PATCH v2 1/7] package/pkg-utils.mk: introduce "name" field in show-info output Thomas Petazzoni
2021-01-31 13:38 ` [Buildroot] [PATCH v2 2/7] support/scripts/pkg-stats: properly handle host packages with -c option Thomas Petazzoni
@ 2021-01-31 13:38 ` Thomas Petazzoni
2021-01-31 22:47 ` Yann E. MORIN
2021-02-02 20:31 ` Arnout Vandecappelle
2021-01-31 13:38 ` [Buildroot] [PATCH v2 4/7] support/scripts/pkg-stats: check CPE existence in CPE dictionnary Thomas Petazzoni
` (3 subsequent siblings)
6 siblings, 2 replies; 20+ messages in thread
From: Thomas Petazzoni @ 2021-01-31 13:38 UTC (permalink / raw)
To: buildroot
From: Matt Weber <matthew.weber@rockwellcollins.com>
Python class which consumes a NIST CPE XML and provides helper
functions to access and search the db's data.
- Defines the CPE as a object with operations / formats
- Processing of CPE dictionary
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>
Co-Developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++
1 file changed, 203 insertions(+)
create mode 100644 support/scripts/cpedb.py
diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
new file mode 100644
index 0000000000..825ed6cb1e
--- /dev/null
+++ b/support/scripts/cpedb.py
@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+
+import xml.etree.ElementTree as ET
+from xml.etree.ElementTree import Element, SubElement
+import gzip
+import os
+import pickle
+import requests
+import time
+from xml.dom import minidom
+
+VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
+
+CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
+
+ns = {
+ '': 'http://cpe.mitre.org/dictionary/2.0',
+ 'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
+ 'xml': 'http://www.w3.org/XML/1998/namespace'
+}
+
+
+class CPE:
+ def __init__(self, cpe_str, titles, refs):
+ self.cpe_str = cpe_str
+ self.titles = titles
+ self.references = refs
+ self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
+
+ def update_xml_dict(self):
+ ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
+ cpes = Element('cpe-list')
+ cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")
+ cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
+ cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
+ cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
+ cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
+ cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
+ cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
+ "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",
+ "http://cpe.mitre.org/dictionary/2.0",
+ "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
+ "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
+ "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
+ "http://scap.nist.gov/schema/scap-core/0.3",
+ "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
+ "http://scap.nist.gov/schema/configuration/0.1",
+ "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
+ "http://scap.nist.gov/schema/scap-core/0.1",
+ "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
+ item = SubElement(cpes, 'cpe-item')
+ cpe_short_name = CPE.short_name(self.cpe_str)
+ cpe_new_ver = CPE.version_update(self.cpe_str)
+
+ item.set('name', 'cpe:/' + cpe_short_name)
+ self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
+ for title in self.titles:
+ item.append(title)
+ if self.references:
+ item.append(self.references)
+ cpe23item = SubElement(item, 'cpe-23:cpe23-item')
+ cpe23item.set('name', self.cpe_str)
+
+ # Generate the XML as a string
+ xmlstr = ET.tostring(cpes)
+
+ # And use minidom to pretty print the XML
+ return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
+
+ @staticmethod
+ def version(cpe):
+ return cpe.split(":")[5]
+
+ @staticmethod
+ def product(cpe):
+ return cpe.split(":")[4]
+
+ @staticmethod
+ def short_name(cpe):
+ return ":".join(cpe.split(":")[2:6])
+
+ @staticmethod
+ def version_update(cpe):
+ return ":".join(cpe.split(":")[5:6])
+
+ @staticmethod
+ def no_version(cpe):
+ return ":".join(cpe.split(":")[:5])
+
+
+class CPEDB:
+ def __init__(self, nvd_path):
+ self.all_cpes = dict()
+ self.all_cpes_no_version = dict()
+ self.nvd_path = nvd_path
+
+ def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version):
+ print("CPE: Unzipping xml manifest...")
+ nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb'))
+ print("CPE: Converting xml manifest to dict...")
+ tree = ET.parse(nist_cpe_file)
+ all_cpedb = tree.getroot()
+ self.parse_dict(all_cpedb)
+
+ print("CPE: Caching dictionary")
+ cpes_file = open(cache_all_cpes, 'wb')
+ pickle.dump(self.all_cpes, cpes_file)
+ cpes_file.close()
+ cpes_file = open(cache_all_cpes_no_version, 'wb')
+ pickle.dump(self.all_cpes_no_version, cpes_file)
+ cpes_file.close()
+
+ def get_xml_dict(self):
+ print("CPE: Setting up NIST dictionary")
+ if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
+ os.makedirs(os.path.join(self.nvd_path, "cpe"))
+
+ cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
+ if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
+ print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
+ cpe_dict = requests.get(CPEDB_URL)
+ open(cpe_dict_local, "wb").write(cpe_dict.content)
+
+ cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl")
+ cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl")
+
+ if not os.path.exists(cache_all_cpes) or \
+ not os.path.exists(cache_all_cpes_no_version) or \
+ os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \
+ os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime:
+ self.gen_cached_cpedb(cpe_dict_local,
+ cache_all_cpes,
+ cache_all_cpes_no_version)
+
+ print("CPE: Loading CACHED dictionary")
+ cpe_file = open(cache_all_cpes, 'rb')
+ self.all_cpes = pickle.load(cpe_file)
+ cpe_file.close()
+ cpe_file = open(cache_all_cpes_no_version, 'rb')
+ self.all_cpes_no_version = pickle.load(cpe_file)
+ cpe_file.close()
+
+ def parse_dict(self, all_cpedb):
+ # Cycle through the dict and build two dict to be used for custom
+ # lookups of partial and complete CPE objects
+ # The objects are then used to create new proposed XML updates if
+ # if is determined one is required
+ # Out of the different language titles, select English
+ for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
+ cpe_titles = []
+ for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
+ title.tail = None
+ cpe_titles.append(title)
+
+ # Some older CPE don't include references, if they do, make
+ # sure we handle the case of one ref needing to be packed
+ # in a list
+ cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
+ if cpe_ref:
+ for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
+ ref.tail = None
+ ref.text = ref.text.upper()
+ if ref.text not in VALID_REFS:
+ ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
+ cpe_ref.tail = None
+ cpe_ref.text = None
+
+ cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
+ item = CPE(cpe_str, cpe_titles, cpe_ref)
+ cpe_str_no_version = CPE.no_version(cpe_str)
+ # This dict must have a unique key for every CPE version
+ # which allows matching to the specific obj data of that
+ # NIST dict entry
+ self.all_cpes.update({cpe_str: item})
+ # This dict has one entry for every CPE (w/o version) to allow
+ # partial match (no valid version) check (the obj is saved and
+ # used as seed for suggested xml updates. By updating the same
+ # non-version'd entry, it assumes the last update here is the
+ # latest version in the NIST dict)
+ self.all_cpes_no_version.update({cpe_str_no_version: item})
+
+ def find_partial(self, cpe_str):
+ cpe_str_no_version = CPE.no_version(cpe_str)
+ if cpe_str_no_version in self.all_cpes_no_version:
+ return cpe_str_no_version
+
+ def find_partial_obj(self, cpe_str):
+ cpe_str_no_version = CPE.no_version(cpe_str)
+ if cpe_str_no_version in self.all_cpes_no_version:
+ return self.all_cpes_no_version[cpe_str_no_version]
+
+ def find_partial_latest_version(self, cpe_str_partial):
+ cpe_obj = self.find_partial_obj(cpe_str_partial)
+ return cpe_obj.cpe_cur_ver
+
+ def find(self, cpe_str):
+ if self.find_partial(cpe_str):
+ if cpe_str in self.all_cpes:
+ return cpe_str
+
+ def gen_update_xml(self, cpe_str):
+ cpe = self.find_partial_obj(cpe_str)
+ return cpe.update_xml_dict()
--
2.29.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* [Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper
2021-01-31 13:38 ` [Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper Thomas Petazzoni
@ 2021-01-31 22:47 ` Yann E. MORIN
2021-01-31 22:51 ` Yann E. MORIN
2021-02-02 20:31 ` Arnout Vandecappelle
1 sibling, 1 reply; 20+ messages in thread
From: Yann E. MORIN @ 2021-01-31 22:47 UTC (permalink / raw)
To: buildroot
Gr?gory, All,
On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:
> From: Matt Weber <matthew.weber@rockwellcollins.com>
>
> Python class which consumes a NIST CPE XML and provides helper
> functions to access and search the db's data.
>
> - Defines the CPE as a object with operations / formats
> - Processing of CPE dictionary
>
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>
Gr?gory, you need to add your SOB line right after your CDB line.
Ditto for the following patch.
Regards,
Yann E. MORIN.
> Co-Developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 203 insertions(+)
> create mode 100644 support/scripts/cpedb.py
>
> diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
> new file mode 100644
> index 0000000000..825ed6cb1e
> --- /dev/null
> +++ b/support/scripts/cpedb.py
> @@ -0,0 +1,203 @@
> +#!/usr/bin/env python3
> +
> +import xml.etree.ElementTree as ET
> +from xml.etree.ElementTree import Element, SubElement
> +import gzip
> +import os
> +import pickle
> +import requests
> +import time
> +from xml.dom import minidom
> +
> +VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
> +
> +CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
> +
> +ns = {
> + '': 'http://cpe.mitre.org/dictionary/2.0',
> + 'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
> + 'xml': 'http://www.w3.org/XML/1998/namespace'
> +}
> +
> +
> +class CPE:
> + def __init__(self, cpe_str, titles, refs):
> + self.cpe_str = cpe_str
> + self.titles = titles
> + self.references = refs
> + self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
> +
> + def update_xml_dict(self):
> + ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
> + cpes = Element('cpe-list')
> + cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")
> + cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
> + cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
> + cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
> + cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
> + cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
> + cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
> + "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",
> + "http://cpe.mitre.org/dictionary/2.0",
> + "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
> + "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
> + "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
> + "http://scap.nist.gov/schema/scap-core/0.3",
> + "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
> + "http://scap.nist.gov/schema/configuration/0.1",
> + "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
> + "http://scap.nist.gov/schema/scap-core/0.1",
> + "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
> + item = SubElement(cpes, 'cpe-item')
> + cpe_short_name = CPE.short_name(self.cpe_str)
> + cpe_new_ver = CPE.version_update(self.cpe_str)
> +
> + item.set('name', 'cpe:/' + cpe_short_name)
> + self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
> + for title in self.titles:
> + item.append(title)
> + if self.references:
> + item.append(self.references)
> + cpe23item = SubElement(item, 'cpe-23:cpe23-item')
> + cpe23item.set('name', self.cpe_str)
> +
> + # Generate the XML as a string
> + xmlstr = ET.tostring(cpes)
> +
> + # And use minidom to pretty print the XML
> + return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
> +
> + @staticmethod
> + def version(cpe):
> + return cpe.split(":")[5]
> +
> + @staticmethod
> + def product(cpe):
> + return cpe.split(":")[4]
> +
> + @staticmethod
> + def short_name(cpe):
> + return ":".join(cpe.split(":")[2:6])
> +
> + @staticmethod
> + def version_update(cpe):
> + return ":".join(cpe.split(":")[5:6])
> +
> + @staticmethod
> + def no_version(cpe):
> + return ":".join(cpe.split(":")[:5])
> +
> +
> +class CPEDB:
> + def __init__(self, nvd_path):
> + self.all_cpes = dict()
> + self.all_cpes_no_version = dict()
> + self.nvd_path = nvd_path
> +
> + def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version):
> + print("CPE: Unzipping xml manifest...")
> + nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb'))
> + print("CPE: Converting xml manifest to dict...")
> + tree = ET.parse(nist_cpe_file)
> + all_cpedb = tree.getroot()
> + self.parse_dict(all_cpedb)
> +
> + print("CPE: Caching dictionary")
> + cpes_file = open(cache_all_cpes, 'wb')
> + pickle.dump(self.all_cpes, cpes_file)
> + cpes_file.close()
> + cpes_file = open(cache_all_cpes_no_version, 'wb')
> + pickle.dump(self.all_cpes_no_version, cpes_file)
> + cpes_file.close()
> +
> + def get_xml_dict(self):
> + print("CPE: Setting up NIST dictionary")
> + if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
> + os.makedirs(os.path.join(self.nvd_path, "cpe"))
> +
> + cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
> + if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
> + print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
> + cpe_dict = requests.get(CPEDB_URL)
> + open(cpe_dict_local, "wb").write(cpe_dict.content)
> +
> + cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl")
> + cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl")
> +
> + if not os.path.exists(cache_all_cpes) or \
> + not os.path.exists(cache_all_cpes_no_version) or \
> + os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \
> + os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime:
> + self.gen_cached_cpedb(cpe_dict_local,
> + cache_all_cpes,
> + cache_all_cpes_no_version)
> +
> + print("CPE: Loading CACHED dictionary")
> + cpe_file = open(cache_all_cpes, 'rb')
> + self.all_cpes = pickle.load(cpe_file)
> + cpe_file.close()
> + cpe_file = open(cache_all_cpes_no_version, 'rb')
> + self.all_cpes_no_version = pickle.load(cpe_file)
> + cpe_file.close()
> +
> + def parse_dict(self, all_cpedb):
> + # Cycle through the dict and build two dict to be used for custom
> + # lookups of partial and complete CPE objects
> + # The objects are then used to create new proposed XML updates if
> + # if is determined one is required
> + # Out of the different language titles, select English
> + for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
> + cpe_titles = []
> + for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
> + title.tail = None
> + cpe_titles.append(title)
> +
> + # Some older CPE don't include references, if they do, make
> + # sure we handle the case of one ref needing to be packed
> + # in a list
> + cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
> + if cpe_ref:
> + for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
> + ref.tail = None
> + ref.text = ref.text.upper()
> + if ref.text not in VALID_REFS:
> + ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
> + cpe_ref.tail = None
> + cpe_ref.text = None
> +
> + cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
> + item = CPE(cpe_str, cpe_titles, cpe_ref)
> + cpe_str_no_version = CPE.no_version(cpe_str)
> + # This dict must have a unique key for every CPE version
> + # which allows matching to the specific obj data of that
> + # NIST dict entry
> + self.all_cpes.update({cpe_str: item})
> + # This dict has one entry for every CPE (w/o version) to allow
> + # partial match (no valid version) check (the obj is saved and
> + # used as seed for suggested xml updates. By updating the same
> + # non-version'd entry, it assumes the last update here is the
> + # latest version in the NIST dict)
> + self.all_cpes_no_version.update({cpe_str_no_version: item})
> +
> + def find_partial(self, cpe_str):
> + cpe_str_no_version = CPE.no_version(cpe_str)
> + if cpe_str_no_version in self.all_cpes_no_version:
> + return cpe_str_no_version
> +
> + def find_partial_obj(self, cpe_str):
> + cpe_str_no_version = CPE.no_version(cpe_str)
> + if cpe_str_no_version in self.all_cpes_no_version:
> + return self.all_cpes_no_version[cpe_str_no_version]
> +
> + def find_partial_latest_version(self, cpe_str_partial):
> + cpe_obj = self.find_partial_obj(cpe_str_partial)
> + return cpe_obj.cpe_cur_ver
> +
> + def find(self, cpe_str):
> + if self.find_partial(cpe_str):
> + if cpe_str in self.all_cpes:
> + return cpe_str
> +
> + def gen_update_xml(self, cpe_str):
> + cpe = self.find_partial_obj(cpe_str)
> + return cpe.update_xml_dict()
> --
> 2.29.2
>
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 20+ messages in thread* [Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper
2021-01-31 22:47 ` Yann E. MORIN
@ 2021-01-31 22:51 ` Yann E. MORIN
0 siblings, 0 replies; 20+ messages in thread
From: Yann E. MORIN @ 2021-01-31 22:51 UTC (permalink / raw)
To: buildroot
Gr?gory, All,
On 2021-01-31 23:47 +0100, Yann E. MORIN spake thusly:
> On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:
> > From: Matt Weber <matthew.weber@rockwellcollins.com>
> >
> > Python class which consumes a NIST CPE XML and provides helper
> > functions to access and search the db's data.
> >
> > - Defines the CPE as a object with operations / formats
> > - Processing of CPE dictionary
> >
> > Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> > Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>
>
> Gr?gory, you need to add your SOB line right after your CDB line.
>
> Ditto for the following patch.
No need to resend the series, just reply to each patch with your SoB
line, I'll add them when applying.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper
2021-01-31 13:38 ` [Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper Thomas Petazzoni
2021-01-31 22:47 ` Yann E. MORIN
@ 2021-02-02 20:31 ` Arnout Vandecappelle
1 sibling, 0 replies; 20+ messages in thread
From: Arnout Vandecappelle @ 2021-02-02 20:31 UTC (permalink / raw)
To: buildroot
On 31/01/2021 14:38, Thomas Petazzoni wrote:
> From: Matt Weber <matthew.weber@rockwellcollins.com>
>
> Python class which consumes a NIST CPE XML and provides helper
> functions to access and search the db's data.
>
> - Defines the CPE as a object with operations / formats
> - Processing of CPE dictionary
>
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>
> Co-Developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 203 insertions(+)
> create mode 100644 support/scripts/cpedb.py
>
> diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
> new file mode 100644
> index 0000000000..825ed6cb1e
> --- /dev/null
> +++ b/support/scripts/cpedb.py
> @@ -0,0 +1,203 @@
> +#!/usr/bin/env python3
> +
> +import xml.etree.ElementTree as ET
> +from xml.etree.ElementTree import Element, SubElement
> +import gzip
> +import os
> +import pickle
> +import requests
> +import time
> +from xml.dom import minidom
> +
> +VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
> +
> +CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
> +
> +ns = {
> + '': 'http://cpe.mitre.org/dictionary/2.0',
> + 'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
> + 'xml': 'http://www.w3.org/XML/1998/namespace'
> +}
> +
> +
> +class CPE:
> + def __init__(self, cpe_str, titles, refs):
> + self.cpe_str = cpe_str
> + self.titles = titles
> + self.references = refs
> + self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
> +
> + def update_xml_dict(self):
> + ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
> + cpes = Element('cpe-list')
> + cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")
Note that instead of all these cpe.set calls, you can also do:
cpes.attrib = {
'xmlns:ns6': "http://scap.nist.gov/schema/scap-core/0.1",
...
}
which IMO is more readable. But that's just nitpick.
In fact, I think you can do it in the constructor:
cpes = Element('cpe-list', attrib = { ...
but that is probably less readable.
> + cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
> + cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
> + cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
> + cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
> + cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
> + cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
> + "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",
And here instead of join you can use:
cpes.set('xsi:schemaLocation',
"http://scap.nist.gov/schema/cpe-extension/2.3 "
"http://cpe.mitre.org/dictionary/2.0 "
(Python has string concatenation like C).
> + "http://cpe.mitre.org/dictionary/2.0",
> + "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
> + "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
> + "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
> + "http://scap.nist.gov/schema/scap-core/0.3",
> + "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
> + "http://scap.nist.gov/schema/configuration/0.1",
> + "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
> + "http://scap.nist.gov/schema/scap-core/0.1",
> + "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
> + item = SubElement(cpes, 'cpe-item')
> + cpe_short_name = CPE.short_name(self.cpe_str)
> + cpe_new_ver = CPE.version_update(self.cpe_str)
> +
> + item.set('name', 'cpe:/' + cpe_short_name)
> + self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
> + for title in self.titles:
> + item.append(title)
> + if self.references:
> + item.append(self.references)
> + cpe23item = SubElement(item, 'cpe-23:cpe23-item')
> + cpe23item.set('name', self.cpe_str)
> +
> + # Generate the XML as a string
> + xmlstr = ET.tostring(cpes)
> +
> + # And use minidom to pretty print the XML
> + return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
> +
> + @staticmethod
> + def version(cpe):
> + return cpe.split(":")[5]
> +
> + @staticmethod
> + def product(cpe):
> + return cpe.split(":")[4]
> +
> + @staticmethod
> + def short_name(cpe):
> + return ":".join(cpe.split(":")[2:6])
> +
> + @staticmethod
> + def version_update(cpe):
It's weird to make this a staticmethod if the only caller takes self.cpe_str as
an argument, but OK.
> + return ":".join(cpe.split(":")[5:6])
> +
> + @staticmethod
> + def no_version(cpe):
> + return ":".join(cpe.split(":")[:5])
> +
> +
> +class CPEDB:
> + def __init__(self, nvd_path):
> + self.all_cpes = dict()
> + self.all_cpes_no_version = dict()
> + self.nvd_path = nvd_path
> +
> + def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version):
> + print("CPE: Unzipping xml manifest...")
> + nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb'))
> + print("CPE: Converting xml manifest to dict...")
> + tree = ET.parse(nist_cpe_file)
> + all_cpedb = tree.getroot()
> + self.parse_dict(all_cpedb)
> +
> + print("CPE: Caching dictionary")
> + cpes_file = open(cache_all_cpes, 'wb')
> + pickle.dump(self.all_cpes, cpes_file)
> + cpes_file.close()
> + cpes_file = open(cache_all_cpes_no_version, 'wb')
> + pickle.dump(self.all_cpes_no_version, cpes_file)
> + cpes_file.close()
> +
> + def get_xml_dict(self):
> + print("CPE: Setting up NIST dictionary")
> + if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
> + os.makedirs(os.path.join(self.nvd_path, "cpe"))
> +
> + cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
> + if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
This should be a parameter@the beginning of the file rather than a constant
here. Ideally expressed in hours, with *3600 here.
> + print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
> + cpe_dict = requests.get(CPEDB_URL)
> + open(cpe_dict_local, "wb").write(cpe_dict.content)
> +
> + cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl")
> + cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl")
> +
> + if not os.path.exists(cache_all_cpes) or \
> + not os.path.exists(cache_all_cpes_no_version) or \
> + os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \
> + os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime:
> + self.gen_cached_cpedb(cpe_dict_local,
> + cache_all_cpes,
> + cache_all_cpes_no_version)
> +
> + print("CPE: Loading CACHED dictionary")
> + cpe_file = open(cache_all_cpes, 'rb')
> + self.all_cpes = pickle.load(cpe_file)
> + cpe_file.close()
The above is better done as
with open(cache_all_cpes, 'rb') as cpe_file:
self.all_cpes = pickle.load(cpe_file)
This do things properly in case of exception.
However, I'm not so sure if the pickled cache is really worth it. I gave it a
go, and the ET.parse + parse_dict took roughly 20 seconds, while just loading
the pickle file also took about 20 seconds...
> + cpe_file = open(cache_all_cpes_no_version, 'rb')
> + self.all_cpes_no_version = pickle.load(cpe_file)
> + cpe_file.close()
> +
> + def parse_dict(self, all_cpedb):
> + # Cycle through the dict and build two dict to be used for custom
> + # lookups of partial and complete CPE objects
> + # The objects are then used to create new proposed XML updates if
> + # if is determined one is required
> + # Out of the different language titles, select English
This could have been a docstring.
Actually, all "public" functions should really have a docstring.
So, I accidentally applied this patch... So all the above comments are just
informational - feel free to send follow-up patches, or ignore my feedback.
Regards,
Arnout
> + for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
> + cpe_titles = []
> + for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
> + title.tail = None
> + cpe_titles.append(title)
> +
> + # Some older CPE don't include references, if they do, make
> + # sure we handle the case of one ref needing to be packed
> + # in a list
> + cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
> + if cpe_ref:
> + for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
> + ref.tail = None
> + ref.text = ref.text.upper()
> + if ref.text not in VALID_REFS:
> + ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
> + cpe_ref.tail = None
> + cpe_ref.text = None
> +
> + cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
> + item = CPE(cpe_str, cpe_titles, cpe_ref)
> + cpe_str_no_version = CPE.no_version(cpe_str)
> + # This dict must have a unique key for every CPE version
> + # which allows matching to the specific obj data of that
> + # NIST dict entry
> + self.all_cpes.update({cpe_str: item})
> + # This dict has one entry for every CPE (w/o version) to allow
> + # partial match (no valid version) check (the obj is saved and
> + # used as seed for suggested xml updates. By updating the same
> + # non-version'd entry, it assumes the last update here is the
> + # latest version in the NIST dict)
> + self.all_cpes_no_version.update({cpe_str_no_version: item})
> +
> + def find_partial(self, cpe_str):
> + cpe_str_no_version = CPE.no_version(cpe_str)
> + if cpe_str_no_version in self.all_cpes_no_version:
> + return cpe_str_no_version
> +
> + def find_partial_obj(self, cpe_str):
> + cpe_str_no_version = CPE.no_version(cpe_str)
> + if cpe_str_no_version in self.all_cpes_no_version:
> + return self.all_cpes_no_version[cpe_str_no_version]
> +
> + def find_partial_latest_version(self, cpe_str_partial):
> + cpe_obj = self.find_partial_obj(cpe_str_partial)
> + return cpe_obj.cpe_cur_ver
> +
> + def find(self, cpe_str):
> + if self.find_partial(cpe_str):
> + if cpe_str in self.all_cpes:
> + return cpe_str
> +
> + def gen_update_xml(self, cpe_str):
> + cpe = self.find_partial_obj(cpe_str)
> + return cpe.update_xml_dict()
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH v2 4/7] support/scripts/pkg-stats: check CPE existence in CPE dictionnary
2021-01-31 13:38 [Buildroot] [PATCH v2 0/7] CPE validation Thomas Petazzoni
` (2 preceding siblings ...)
2021-01-31 13:38 ` [Buildroot] [PATCH v2 3/7] support/scripts/cpedb.py: new CPE XML helper Thomas Petazzoni
@ 2021-01-31 13:38 ` Thomas Petazzoni
2021-02-02 20:49 ` Arnout Vandecappelle
2021-01-31 13:38 ` [Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script Thomas Petazzoni
` (2 subsequent siblings)
6 siblings, 1 reply; 20+ messages in thread
From: Thomas Petazzoni @ 2021-01-31 13:38 UTC (permalink / raw)
To: buildroot
This commit extends pkg-stats to leverage the recently introduced
CPEDB class to verify that the CPEs provided by Buildroot packages are
indeed known in the official CPE dictionnary provided by NVD.
Co-Developed-by: Gr?gory Clement <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
support/scripts/pkg-stats | 30 +++++++++++++++++++++---------
1 file changed, 21 insertions(+), 9 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 0d4e781e3b..f595026336 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -32,7 +32,7 @@ brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
sys.path.append(os.path.join(brpath, "utils"))
from getdeveloperlib import parse_developers # noqa: E402
-
+from cpedb import CPEDB # noqa: E402
INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
URL_RE = re.compile(r"\s*https?://\S*\s*$")
@@ -225,7 +225,8 @@ class Package:
if var in self.all_cpeids:
self.cpeid = self.all_cpeids[var]
- self.status['cpe'] = ("ok", "verified CPE identifier")
+ # Set a preliminary status, it might be overridden by check_package_cpes()
+ self.status['cpe'] = ("warning", "not checked against CPE dictionnary")
else:
self.status['cpe'] = ("error", "no verified CPE identifier")
@@ -600,6 +601,18 @@ def check_package_cves(nvd_path, packages):
pkg.status['cve'] = ("ok", "not affected by CVEs")
+def check_package_cpes(nvd_path, packages):
+ cpedb = CPEDB(nvd_path)
+ cpedb.get_xml_dict()
+ for p in packages:
+ if not p.cpeid:
+ continue
+ if cpedb.find(p.cpeid):
+ p.status['cpe'] = ("ok", "verified CPE identifier")
+ else:
+ p.status['cpe'] = ("error", "CPE identifier unknown in CPE database")
+
+
def calculate_stats(packages):
stats = defaultdict(int)
stats['packages'] = len(packages)
@@ -898,19 +911,17 @@ def dump_html_pkg(f, pkg):
# CPE ID
td_class = ["left"]
- if pkg.status['cpe'][0] == "ok":
+ if pkg.is_status_ok("cpe"):
td_class.append("cpe-ok")
- elif pkg.status['cpe'][0] == "error":
+ elif pkg.is_status_error("cpe"):
td_class.append("cpe-nok")
else:
td_class.append("cpe-unknown")
f.write(" <td class=\"%s\">\n" % " ".join(td_class))
- if pkg.status['cpe'][0] == "ok":
+ if pkg.cpeid:
f.write(" <code>%s</code>\n" % pkg.cpeid)
- elif pkg.status['cpe'][0] == "error":
- f.write(" N/A\n")
- else:
- f.write(" %s\n" % pkg.status['cpe'][1])
+ if not pkg.is_status_ok("cpe"):
+ f.write(" %s%s\n" % ("<br/>" if pkg.cpeid else "", pkg.status['cpe'][1]))
f.write(" </td>\n")
f.write(" </tr>\n")
@@ -1106,6 +1117,7 @@ def __main__():
if args.nvd_path:
print("Checking packages CVEs")
check_package_cves(args.nvd_path, packages)
+ check_package_cpes(args.nvd_path, packages)
print("Calculate stats")
stats = calculate_stats(packages)
if args.html:
--
2.29.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* [Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script
2021-01-31 13:38 [Buildroot] [PATCH v2 0/7] CPE validation Thomas Petazzoni
` (3 preceding siblings ...)
2021-01-31 13:38 ` [Buildroot] [PATCH v2 4/7] support/scripts/pkg-stats: check CPE existence in CPE dictionnary Thomas Petazzoni
@ 2021-01-31 13:38 ` Thomas Petazzoni
2021-02-02 21:29 ` Arnout Vandecappelle
2021-05-16 12:08 ` Yann E. MORIN
2021-01-31 13:38 ` [Buildroot] [PATCH v2 6/7] Makefile: add new missing-cpe target Thomas Petazzoni
2021-01-31 13:38 ` [Buildroot] [PATCH v2 7/7] docs/manual: add details about vulnerability management Thomas Petazzoni
6 siblings, 2 replies; 20+ messages in thread
From: Thomas Petazzoni @ 2021-01-31 13:38 UTC (permalink / raw)
To: buildroot
From: Matt Weber <matthew.weber@rockwellcollins.com>
This script queries the list of CPE IDs for the packages of the
current configuration (based on the "make show-info" output), and:
- for CPE IDs that do not have any matching entry in the CPE
database, it emits a warning
- for CPE IDs that do have a matching entry, but not with the same
version, it generates a snippet of XML that can be used to propose
an updated version to NIST.
Ref: NIST has a group email (cpe_dictionary at nist.gov) used to
recieve these version update and new entry xml files. They do
process the XML and provide feedback. In some cases they will
propose back something different where the vendor or version is
slightly different.
Limitations
- Currently any use of non-number version identifiers isn't
supported by NIST as they use ranges to determine impact
of a CVE
- Any Linux version from a non-upstream is also not supported
without manually adjusting the information as the custom
kernel will more then likely not match the upstream version
used in the dictionary
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++
1 file changed, 65 insertions(+)
create mode 100755 support/scripts/gen-missing-cpe
diff --git a/support/scripts/gen-missing-cpe b/support/scripts/gen-missing-cpe
new file mode 100755
index 0000000000..ed7747295a
--- /dev/null
+++ b/support/scripts/gen-missing-cpe
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+
+import argparse
+import sys
+import json
+import subprocess
+import os
+from cpedb import CPEDB, CPE
+
+
+def gen_update_xml_reports(cpes, cpedb, output):
+ cpe_need_update = []
+
+ for cpe in cpes:
+ result = cpedb.find(cpe)
+ if not result:
+ result = cpedb.find_partial(CPE.no_version(cpe))
+ if result:
+ cpe_need_update.append(cpe)
+ else:
+ print("WARNING: no match found for '%s'" % cpe)
+
+ for cpe in cpe_need_update:
+ xml = cpedb.gen_update_xml(cpe)
+ fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
+ print("Generating %s" % fname)
+ fp = open(os.path.join(output, fname), 'w+')
+ fp.write(xml)
+ fp.close()
+
+ print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes)))
+
+
+def get_cpe_ids():
+ print("Getting list of CPE for enabled packages")
+ cmd = ["make", "--no-print-directory", "show-info"]
+ js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
+ return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
+
+
+def resolvepath(path):
+ return os.path.abspath(os.path.expanduser(path))
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--output', dest='output',
+ help='Path to the output CPE update files', type=resolvepath, required=True)
+ parser.add_argument('--nvd-path', dest='nvd_path',
+ help='Path to the local NVD database', type=resolvepath, required=True)
+ return parser.parse_args()
+
+
+def __main__():
+ args = parse_args()
+ if not os.path.isdir(args.output):
+ print("ERROR: output directory %s does not exist" % args.output)
+ sys.exit(1)
+ cpedb = CPEDB(args.nvd_path)
+ cpedb.get_xml_dict()
+ cpes = get_cpe_ids()
+ gen_update_xml_reports(cpes, cpedb, args.output)
+
+
+__main__()
--
2.29.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* [Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script
2021-01-31 13:38 ` [Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script Thomas Petazzoni
@ 2021-02-02 21:29 ` Arnout Vandecappelle
2021-02-08 21:09 ` Matthew Weber
2021-05-16 12:13 ` Yann E. MORIN
2021-05-16 12:08 ` Yann E. MORIN
1 sibling, 2 replies; 20+ messages in thread
From: Arnout Vandecappelle @ 2021-02-02 21:29 UTC (permalink / raw)
To: buildroot
On 31/01/2021 14:38, Thomas Petazzoni wrote:
> From: Matt Weber <matthew.weber@rockwellcollins.com>
>
> This script queries the list of CPE IDs for the packages of the
> current configuration (based on the "make show-info" output), and:
>
> - for CPE IDs that do not have any matching entry in the CPE
> database, it emits a warning
Maybe there should also be a warning for packages which don't have cpeid set at
all...
>
> - for CPE IDs that do have a matching entry, but not with the same
> version, it generates a snippet of XML that can be used to propose
> an updated version to NIST.
>
> Ref: NIST has a group email (cpe_dictionary at nist.gov) used to
> recieve these version update and new entry xml files. They do
> process the XML and provide feedback. In some cases they will
> propose back something different where the vendor or version is
> slightly different.
It would be very useful if the script would also print a URL that describes the
submission process.
>
> Limitations
> - Currently any use of non-number version identifiers isn't
> supported by NIST as they use ranges to determine impact
> of a CVE
> - Any Linux version from a non-upstream is also not supported
> without manually adjusting the information as the custom
> kernel will more then likely not match the upstream version
> used in the dictionary
>
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++
> 1 file changed, 65 insertions(+)
> create mode 100755 support/scripts/gen-missing-cpe
>
> diff --git a/support/scripts/gen-missing-cpe b/support/scripts/gen-missing-cpe
> new file mode 100755
> index 0000000000..ed7747295a
> --- /dev/null
> +++ b/support/scripts/gen-missing-cpe
> @@ -0,0 +1,65 @@
> +#!/usr/bin/env python3
> +
> +import argparse
> +import sys
> +import json
> +import subprocess
> +import os
> +from cpedb import CPEDB, CPE
> +
> +
> +def gen_update_xml_reports(cpes, cpedb, output):
cpes should be cpeids (I first thought it was a list of CPE objects from cpedb).
> + cpe_need_update = []
> +
> + for cpe in cpes:
for cpeid in cpeids:
> + result = cpedb.find(cpe)
> + if not result:
> + result = cpedb.find_partial(CPE.no_version(cpe))
> + if result:
> + cpe_need_update.append(cpe)
> + else:
> + print("WARNING: no match found for '%s'" % cpe)
> +
> + for cpe in cpe_need_update:
> + xml = cpedb.gen_update_xml(cpe)
> + fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
> + print("Generating %s" % fname)
> + fp = open(os.path.join(output, fname), 'w+')
> + fp.write(xml)
> + fp.close()
This should be
with open(...) as fp:
fp.write(xml)
> +
> + print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes)))
> +
> +
> +def get_cpe_ids():
> + print("Getting list of CPE for enabled packages")
> + cmd = ["make", "--no-print-directory", "show-info"]
> + js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
> + return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
> +
> +
> +def resolvepath(path):
> + return os.path.abspath(os.path.expanduser(path))
I don't understand this...
- expanduser should already have been done by the shell. If you call the script as
gen-missing-cpe --output \~/some-path
then I want the output to be in a directory called ~ in the current directory.
That's what the rest of the world does. (Yes, I get it, you want to be able to
do --output=~/some-path but that's just wrong.)
- abspath shouldn't be needed for anything, we're not doing any changedir or
anything like that.
In addition, this script is supposed to be called from `make missing-cpe` which
has already done those expansions.
> +
> +
> +def parse_args():
> + parser = argparse.ArgumentParser()
> + parser.add_argument('--output', dest='output',
> + help='Path to the output CPE update files', type=resolvepath, required=True)
> + parser.add_argument('--nvd-path', dest='nvd_path',
> + help='Path to the local NVD database', type=resolvepath, required=True)
> + return parser.parse_args()
> +
> +
> +def __main__():
> + args = parse_args()
> + if not os.path.isdir(args.output):
> + print("ERROR: output directory %s does not exist" % args.output)
> + sys.exit(1)
> + cpedb = CPEDB(args.nvd_path)
> + cpedb.get_xml_dict()
> + cpes = get_cpe_ids()
> + gen_update_xml_reports(cpes, cpedb, args.output)
> +
> +
> +__main__()
Usually it would be
if __name__ == '__main__':
__main__()
Nothing important here, but I want to give you the chance to rework if you
think it's worth it, so just
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Regards,
Arnout
^ permalink raw reply [flat|nested] 20+ messages in thread* [Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script
2021-02-02 21:29 ` Arnout Vandecappelle
@ 2021-02-08 21:09 ` Matthew Weber
2021-05-16 12:13 ` Yann E. MORIN
1 sibling, 0 replies; 20+ messages in thread
From: Matthew Weber @ 2021-02-08 21:09 UTC (permalink / raw)
To: buildroot
Thomas,
On Tue, Feb 2, 2021 at 3:29 PM Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
>
> On 31/01/2021 14:38, Thomas Petazzoni wrote:
> > From: Matt Weber <matthew.weber@rockwellcollins.com>
> >
> > This script queries the list of CPE IDs for the packages of the
> > current configuration (based on the "make show-info" output), and:
> >
> > - for CPE IDs that do not have any matching entry in the CPE
> > database, it emits a warning
>
> Maybe there should also be a warning for packages which don't have cpeid set at
> all...
Agree, this would be something that people would miss if we don't list
the missing. My original iteration had that as a category that was
printed to screen but no XML generated for it.
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Regards,
Matt
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script
2021-02-02 21:29 ` Arnout Vandecappelle
2021-02-08 21:09 ` Matthew Weber
@ 2021-05-16 12:13 ` Yann E. MORIN
1 sibling, 0 replies; 20+ messages in thread
From: Yann E. MORIN @ 2021-05-16 12:13 UTC (permalink / raw)
To: buildroot
Arnout, All,
On 2021-02-02 22:29 +0100, Arnout Vandecappelle spake thusly:
> On 31/01/2021 14:38, Thomas Petazzoni wrote:
> > From: Matt Weber <matthew.weber@rockwellcollins.com>
> >
> > This script queries the list of CPE IDs for the packages of the
> > current configuration (based on the "make show-info" output), and:
> >
> > - for CPE IDs that do not have any matching entry in the CPE
> > database, it emits a warning
> Maybe there should also be a warning for packages which don't have cpeid set at
> all...
I haven't done anything to do so; I though it was better to have at
least this script in its current state, rather than nothing at all.
[--SNIP--]
> > +def gen_update_xml_reports(cpes, cpedb, output):
> cpes should be cpeids (I first thought it was a list of CPE objects from cpedb).
Done.
[--SNIP--]
> > + fp = open(os.path.join(output, fname), 'w+')
> > + fp.write(xml)
> > + fp.close()
> This should be
> with open(...) as fp:
> fp.write(xml)
Done.
[--SNIP--]
> > +def resolvepath(path):
> > + return os.path.abspath(os.path.expanduser(path))
>
> I don't understand this...
>
> - expanduser should already have been done by the shell. If you call the script as
>
> gen-missing-cpe --output \~/some-path
>
> then I want the output to be in a directory called ~ in the current directory.
> That's what the rest of the world does. (Yes, I get it, you want to be able to
> do --output=~/some-path but that's just wrong.)
>
> - abspath shouldn't be needed for anything, we're not doing any changedir or
> anything like that.
>
> In addition, this script is supposed to be called from `make missing-cpe` which
> has already done those expansions.
It was not obvious to me either that resolvepath() was needed, and as]
you said, the usage we have in Buildroot does not need expansion.
However, getting rid of it seemed too much, so I left it as-is.
Again, I thought it was better that we have this script rather than
nothing.
[--SNIP--]
> > +__main__()
> Usually it would be
> if __name__ == '__main__':
> __main__()
Done.
> Nothing important here, but I want to give you the chance to rework if you
> think it's worth it, so just
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This was sitting in the backlog for quite a while, and we really needed
to push a bit on our CPE (and CVE) tooling, so I applied this to master
now.
Thanks!
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script
2021-01-31 13:38 ` [Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script Thomas Petazzoni
2021-02-02 21:29 ` Arnout Vandecappelle
@ 2021-05-16 12:08 ` Yann E. MORIN
1 sibling, 0 replies; 20+ messages in thread
From: Yann E. MORIN @ 2021-05-16 12:08 UTC (permalink / raw)
To: buildroot
Thomas, All,
On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:
> From: Matt Weber <matthew.weber@rockwellcollins.com>
>
> This script queries the list of CPE IDs for the packages of the
> current configuration (based on the "make show-info" output), and:
>
> - for CPE IDs that do not have any matching entry in the CPE
> database, it emits a warning
>
> - for CPE IDs that do have a matching entry, but not with the same
> version, it generates a snippet of XML that can be used to propose
> an updated version to NIST.
>
> Ref: NIST has a group email (cpe_dictionary at nist.gov) used to
> recieve these version update and new entry xml files. They do
> process the XML and provide feedback. In some cases they will
> propose back something different where the vendor or version is
> slightly different.
>
> Limitations
> - Currently any use of non-number version identifiers isn't
> supported by NIST as they use ranges to determine impact
> of a CVE
> - Any Linux version from a non-upstream is also not supported
> without manually adjusting the information as the custom
> kernel will more then likely not match the upstream version
> used in the dictionary
>
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Applied to master, with most of the changes pointed out by Arnout,
thanks. I'll further reply to Arnout's comment...
Regards,
Yann E. MORIN.
> ---
> support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++
> 1 file changed, 65 insertions(+)
> create mode 100755 support/scripts/gen-missing-cpe
>
> diff --git a/support/scripts/gen-missing-cpe b/support/scripts/gen-missing-cpe
> new file mode 100755
> index 0000000000..ed7747295a
> --- /dev/null
> +++ b/support/scripts/gen-missing-cpe
> @@ -0,0 +1,65 @@
> +#!/usr/bin/env python3
> +
> +import argparse
> +import sys
> +import json
> +import subprocess
> +import os
> +from cpedb import CPEDB, CPE
> +
> +
> +def gen_update_xml_reports(cpes, cpedb, output):
> + cpe_need_update = []
> +
> + for cpe in cpes:
> + result = cpedb.find(cpe)
> + if not result:
> + result = cpedb.find_partial(CPE.no_version(cpe))
> + if result:
> + cpe_need_update.append(cpe)
> + else:
> + print("WARNING: no match found for '%s'" % cpe)
> +
> + for cpe in cpe_need_update:
> + xml = cpedb.gen_update_xml(cpe)
> + fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
> + print("Generating %s" % fname)
> + fp = open(os.path.join(output, fname), 'w+')
> + fp.write(xml)
> + fp.close()
> +
> + print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes)))
> +
> +
> +def get_cpe_ids():
> + print("Getting list of CPE for enabled packages")
> + cmd = ["make", "--no-print-directory", "show-info"]
> + js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
> + return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
> +
> +
> +def resolvepath(path):
> + return os.path.abspath(os.path.expanduser(path))
> +
> +
> +def parse_args():
> + parser = argparse.ArgumentParser()
> + parser.add_argument('--output', dest='output',
> + help='Path to the output CPE update files', type=resolvepath, required=True)
> + parser.add_argument('--nvd-path', dest='nvd_path',
> + help='Path to the local NVD database', type=resolvepath, required=True)
> + return parser.parse_args()
> +
> +
> +def __main__():
> + args = parse_args()
> + if not os.path.isdir(args.output):
> + print("ERROR: output directory %s does not exist" % args.output)
> + sys.exit(1)
> + cpedb = CPEDB(args.nvd_path)
> + cpedb.get_xml_dict()
> + cpes = get_cpe_ids()
> + gen_update_xml_reports(cpes, cpedb, args.output)
> +
> +
> +__main__()
> --
> 2.29.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH v2 6/7] Makefile: add new missing-cpe target
2021-01-31 13:38 [Buildroot] [PATCH v2 0/7] CPE validation Thomas Petazzoni
` (4 preceding siblings ...)
2021-01-31 13:38 ` [Buildroot] [PATCH v2 5/7] support/scripts/gen-missing-cpe: add new script Thomas Petazzoni
@ 2021-01-31 13:38 ` Thomas Petazzoni
2021-02-02 21:29 ` Arnout Vandecappelle
2021-01-31 13:38 ` [Buildroot] [PATCH v2 7/7] docs/manual: add details about vulnerability management Thomas Petazzoni
6 siblings, 1 reply; 20+ messages in thread
From: Thomas Petazzoni @ 2021-01-31 13:38 UTC (permalink / raw)
To: buildroot
It invokes the recently introduced gen-missing-cpe script.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
Makefile | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/Makefile b/Makefile
index 4d334adcd6..962931e0a5 100644
--- a/Makefile
+++ b/Makefile
@@ -945,6 +945,14 @@ pkg-stats:
--html $(O)/pkg-stats.html \
--nvd-path $(DL_DIR)/buildroot-nvd
+.PHONY: missing-cpe
+missing-cpe:
+ $(Q)mkdir -p $(O)/cpe-updates
+ $(Q)cd "$(CONFIG_DIR)" ; \
+ $(TOPDIR)/support/scripts/gen-missing-cpe \
+ --nvd-path $(DL_DIR)/buildroot-nvd \
+ --output $(O)/cpe-updates
+
else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
# Some subdirectories are also package names. To avoid that "make linux"
@@ -1163,6 +1171,7 @@ help:
@echo ' legal-info - generate info about license compliance'
@echo ' show-info - generate info about packages, as a JSON blurb'
@echo ' pkg-stats - generate info about packages as JSON and HTML'
+ @echo ' missing-cpe - generate XML snippets for missing CPE identifiers'
@echo ' printvars - dump internal variables selected with VARS=...'
@echo
@echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build'
--
2.29.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* [Buildroot] [PATCH v2 6/7] Makefile: add new missing-cpe target
2021-01-31 13:38 ` [Buildroot] [PATCH v2 6/7] Makefile: add new missing-cpe target Thomas Petazzoni
@ 2021-02-02 21:29 ` Arnout Vandecappelle
2021-02-08 21:10 ` Matthew Weber
0 siblings, 1 reply; 20+ messages in thread
From: Arnout Vandecappelle @ 2021-02-02 21:29 UTC (permalink / raw)
To: buildroot
On 31/01/2021 14:38, Thomas Petazzoni wrote:
> It invokes the recently introduced gen-missing-cpe script.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Regards,
Arnout
> ---
> Makefile | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index 4d334adcd6..962931e0a5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -945,6 +945,14 @@ pkg-stats:
> --html $(O)/pkg-stats.html \
> --nvd-path $(DL_DIR)/buildroot-nvd
>
> +.PHONY: missing-cpe
> +missing-cpe:
> + $(Q)mkdir -p $(O)/cpe-updates
> + $(Q)cd "$(CONFIG_DIR)" ; \
> + $(TOPDIR)/support/scripts/gen-missing-cpe \
> + --nvd-path $(DL_DIR)/buildroot-nvd \
> + --output $(O)/cpe-updates
> +
> else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
>
> # Some subdirectories are also package names. To avoid that "make linux"
> @@ -1163,6 +1171,7 @@ help:
> @echo ' legal-info - generate info about license compliance'
> @echo ' show-info - generate info about packages, as a JSON blurb'
> @echo ' pkg-stats - generate info about packages as JSON and HTML'
> + @echo ' missing-cpe - generate XML snippets for missing CPE identifiers'
> @echo ' printvars - dump internal variables selected with VARS=...'
> @echo
> @echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build'
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH v2 6/7] Makefile: add new missing-cpe target
2021-02-02 21:29 ` Arnout Vandecappelle
@ 2021-02-08 21:10 ` Matthew Weber
0 siblings, 0 replies; 20+ messages in thread
From: Matthew Weber @ 2021-02-08 21:10 UTC (permalink / raw)
To: buildroot
Thomas,
On Tue, Feb 2, 2021 at 3:30 PM Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
>
> On 31/01/2021 14:38, Thomas Petazzoni wrote:
> > It invokes the recently introduced gen-missing-cpe script.
> >
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH v2 7/7] docs/manual: add details about vulnerability management
2021-01-31 13:38 [Buildroot] [PATCH v2 0/7] CPE validation Thomas Petazzoni
` (5 preceding siblings ...)
2021-01-31 13:38 ` [Buildroot] [PATCH v2 6/7] Makefile: add new missing-cpe target Thomas Petazzoni
@ 2021-01-31 13:38 ` Thomas Petazzoni
2021-02-02 22:02 ` Arnout Vandecappelle
6 siblings, 1 reply; 20+ messages in thread
From: Thomas Petazzoni @ 2021-01-31 13:38 UTC (permalink / raw)
To: buildroot
This addition to the documentation gives some details about CPE/CVE,
pkg-stats and missing-cpe.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
docs/manual/common-usage.txt | 49 ++++++++++++++++++++++++++++++++++++
1 file changed, 49 insertions(+)
diff --git a/docs/manual/common-usage.txt b/docs/manual/common-usage.txt
index 9ba87a8339..e5d9d79420 100644
--- a/docs/manual/common-usage.txt
+++ b/docs/manual/common-usage.txt
@@ -417,6 +417,55 @@ effects:
be empty and it's only at the very end of the build that they will
be populated.
+[[vulnerability-management]]
+=== Vulnerability management
+
+Buildroot integrates a number of features that help tracking whether
+known security vulnerabilities are affecting some of the
+packages. Buildroot bases this logic on _CVEs_ (Common Vulnerabilities
+and Exposure) and the https://nvd.nist.gov/vuln[NVD database] provided
+by NIST.
+
+First, each package in Buildroot can be associated to a _CPE_
+identifier, by means of the various +FOO_CPE_ID_<something>+ variables
+that can be provided by each package. See
+xref:generic-package-reference[] for more details about those
+variables. When the _CPE_ identifier of a package is not defined, a
+default value is used, which may or may not be correct. This _CPE_
+identifier is used to find in the NVD database of vulnerabilities
+which _CVEs_ are affecting any given package.
+
+Based on this and the NIST databases, the +make pkg-stats+ target
+produces JSON and HTML formatted reports that indicate for each
+package of the current Buildroot configuration:
+
+* If there are known _CVEs_ for each package. Note that CVEs can
+ manually be ignored on a per-package basis using the
+ +FOO_IGNORED_CVES+ variable, for example if Buildroot has a patch
+ that fixes the vulnerability, or if the vulnerability is not
+ applicable to Buildroot for some reason.
+
+* If the _CPE_ identifier is valid and known in the
+ https://nvd.nist.gov/products/cpe[NIST CPE dictionary]. A _CPE_ is
+ valid is one of the +FOO_CPE_ID_<something>+ variable is defined, or
+ if +FOO_CPE_ID_VALID = YES+ validating that the default value of the
+ _CPE_ identifier is correct. In addition, _CPE_ are checked for
+ existence in the NIST CPE dictionary. If the _CPE_ identifier is not
+ known to the NIST CPE dictionary, then it is possible to contribute
+ to the NIST CPE dictionary: one need to submit a XML document that
+ describes the additional CPE identifier. Buildroot can help
+ generating such XML documents for missing CPE identifiers using the
+ +make missing-cpe+ target. The XML documents are generated in
+ +$(O)/cpe-updates+, will need to be edited before submission to
+ NIST. In particular, one should verify the version information and
+ check that the reference URLs are reachable with a valid type
+ (suggestions for types are provided in the generated file). If a
+ previous _CPE_ entry was found in the NIST CPE dictionary, the XML
+ document generation tries to minimize the entry differences between
+ the existing versions in the dictionary. The XML document
+ generation uses the XML from the last version of that CPE as a
+ template for the new submission.
+
include::eclipse-integration.txt[]
include::advanced.txt[]
--
2.29.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* [Buildroot] [PATCH v2 7/7] docs/manual: add details about vulnerability management
2021-01-31 13:38 ` [Buildroot] [PATCH v2 7/7] docs/manual: add details about vulnerability management Thomas Petazzoni
@ 2021-02-02 22:02 ` Arnout Vandecappelle
0 siblings, 0 replies; 20+ messages in thread
From: Arnout Vandecappelle @ 2021-02-02 22:02 UTC (permalink / raw)
To: buildroot
On 31/01/2021 14:38, Thomas Petazzoni wrote:
> This addition to the documentation gives some details about CPE/CVE,
> pkg-stats and missing-cpe.
That was sorely needed!
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> docs/manual/common-usage.txt | 49 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 49 insertions(+)
>
> diff --git a/docs/manual/common-usage.txt b/docs/manual/common-usage.txt
> index 9ba87a8339..e5d9d79420 100644
> --- a/docs/manual/common-usage.txt
> +++ b/docs/manual/common-usage.txt
> @@ -417,6 +417,55 @@ effects:
> be empty and it's only at the very end of the build that they will
> be populated.
>
> +[[vulnerability-management]]
> +=== Vulnerability management
> +
> +Buildroot integrates a number of features that help tracking whether
> +known security vulnerabilities are affecting some of the
> +packages. Buildroot bases this logic on _CVEs_ (Common Vulnerabilities
> +and Exposure) and the https://nvd.nist.gov/vuln[NVD database] provided
> +by NIST.
> +
> +First, each package in Buildroot can be associated to a _CPE_
> +identifier, by means of the various +FOO_CPE_ID_<something>+ variables
> +that can be provided by each package. See
> +xref:generic-package-reference[] for more details about those
> +variables. When the _CPE_ identifier of a package is not defined, a
> +default value is used, which may or may not be correct.
I don't think this reflects the reality very well. cpeid is only used if
*something* is set in the .mk file. Something like:
When no +FOO_CPE_ID_<something>+ variable is set for a package, it is not
considered as having a valid _CPE_ identifier.
> This _CPE_
> +identifier is used to find in the NVD database of vulnerabilities
> +which _CVEs_ are affecting any given package.
... If a package doesn't have a valid CPE identifier, a guess is made based on
the package name and version.
> +
> +Based on this and the NIST databases, the +make pkg-stats+ target
> +produces JSON and HTML formatted reports that indicate for each
> +package of the current Buildroot configuration:
> +
> +* If there are known _CVEs_ for each package. Note that CVEs can
> + manually be ignored on a per-package basis using the
> + +FOO_IGNORED_CVES+ variable, for example if Buildroot has a patch
> + that fixes the vulnerability, or if the vulnerability is not
> + applicable to Buildroot for some reason.
> +
> +* If the _CPE_ identifier is valid and known in the
> + https://nvd.nist.gov/products/cpe[NIST CPE dictionary]. A _CPE_ is
> + valid is one of the +FOO_CPE_ID_<something>+ variable is defined, or
> + if +FOO_CPE_ID_VALID = YES+ validating that the default value of the
> + _CPE_ identifier is correct. In addition, _CPE_ are checked for
> + existence in the NIST CPE dictionary. If the _CPE_ identifier is not
> + known to the NIST CPE dictionary, then it is possible to contribute
> + to the NIST CPE dictionary: one need to submit a XML document that
needs
> + describes the additional CPE identifier.
Can you add a reference to a web page that describes the procedure? Otherwise
it's not terribly helpful IMHO.
> Buildroot can help
> + generating such XML documents for missing CPE identifiers using the
> + +make missing-cpe+ target. The XML documents are generated in
> + +$(O)/cpe-updates+, will need to be edited before submission to
> + NIST. In particular, one should verify the version information and
> + check that the reference URLs are reachable with a valid type
> + (suggestions for types are provided in the generated file). If a
> + previous _CPE_ entry was found in the NIST CPE dictionary, the XML
If no previous CPE entry exists (i.e., the CPE_ID specified in the .mk file is
not valid), then no XML file is generated at all... AFAICS there is no help at
the moment for generating the XML for completely new entries.
My brain is too tired ATM to try to formulate the documentation that expresses
that.
I've marked this one changes requested in patchwork.
Regards,
Arnout
> + document generation tries to minimize the entry differences between
> + the existing versions in the dictionary. The XML document
> + generation uses the XML from the last version of that CPE as a
> + template for the new submission.
> +
> include::eclipse-integration.txt[]
>
> include::advanced.txt[]
>
^ permalink raw reply [flat|nested] 20+ messages in thread