* [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos
@ 2023-08-12 19:28 Daniel Lang
2023-08-12 19:28 ` [Buildroot] [PATCH v3 2/8] support/scripts/pkg-stats: ignore llvm-project.mk Daniel Lang
` (8 more replies)
0 siblings, 9 replies; 24+ messages in thread
From: Daniel Lang @ 2023-08-12 19:28 UTC (permalink / raw)
To: buildroot; +Cc: Sen Hastings, Thomas Petazzoni
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
support/scripts/pkg-stats | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 3248e3678d..eea900124c 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -84,7 +84,7 @@ class Package:
all_ignored_cves = dict()
all_cpeids = dict()
# This is the list of all possible checks. Add new checks to this list so
- # a tool that post-processeds the json output knows the checks before
+ # a tool that post-processes the json output knows the checks before
# iterating over the packages.
status_checks = ['cve', 'developers', 'hash', 'license',
'license-files', 'patches', 'pkg-check', 'url', 'version']
@@ -259,7 +259,7 @@ class Package:
if var in self.all_cpeids:
self.cpeid = self.all_cpeids[var]
# Set a preliminary status, it might be overridden by check_package_cpes()
- self.status['cpe'] = ("warning", "not checked against CPE dictionnary")
+ self.status['cpe'] = ("warning", "not checked against CPE dictionary")
else:
self.status['cpe'] = ("error", "no verified CPE identifier")
@@ -499,7 +499,7 @@ def check_package_latest_version_set_status(pkg, status, version, identifier):
async def check_package_get_latest_version_by_distro(session, pkg, retry=True):
- url = "https://release-monitoring.org//api/project/Buildroot/%s" % pkg.name
+ url = "https://release-monitoring.org/api/project/Buildroot/%s" % pkg.name
try:
async with session.get(url) as resp:
if resp.status != 200:
--
2.41.0
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 2/8] support/scripts/pkg-stats: ignore llvm-project.mk
2023-08-12 19:28 [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Daniel Lang
@ 2023-08-12 19:28 ` Daniel Lang
2023-08-30 20:31 ` Arnout Vandecappelle via buildroot
2023-08-12 19:28 ` [Buildroot] [PATCH v3 3/8] support/scripts/pkg-stats: check all files for warnings Daniel Lang
` (7 subsequent siblings)
8 siblings, 1 reply; 24+ messages in thread
From: Daniel Lang @ 2023-08-12 19:28 UTC (permalink / raw)
To: buildroot; +Cc: Sen Hastings, Thomas Petazzoni
Fixes:
f6eaf60 ("package/llvm-project: new group for llvm packages")
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
support/scripts/pkg-stats | 1 +
1 file changed, 1 insertion(+)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index eea900124c..1ac538f5f9 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -345,6 +345,7 @@ def get_pkglist(npackages, package_list):
"package/gstreamer/gstreamer.mk",
"package/gstreamer1/gstreamer1.mk",
"package/gtk2-themes/gtk2-themes.mk",
+ "package/llvm-project/llvm-project.mk",
"package/matchbox/matchbox.mk",
"package/opengl/opengl.mk",
"package/qt5/qt5.mk",
--
2.41.0
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 3/8] support/scripts/pkg-stats: check all files for warnings
2023-08-12 19:28 [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Daniel Lang
2023-08-12 19:28 ` [Buildroot] [PATCH v3 2/8] support/scripts/pkg-stats: ignore llvm-project.mk Daniel Lang
@ 2023-08-12 19:28 ` Daniel Lang
2023-08-30 20:36 ` Arnout Vandecappelle via buildroot
2023-08-12 19:28 ` [Buildroot] [PATCH v3 4/8] support/scripts/gen-missing-cpe: remove rarely used script Daniel Lang
` (6 subsequent siblings)
8 siblings, 1 reply; 24+ messages in thread
From: Daniel Lang @ 2023-08-12 19:28 UTC (permalink / raw)
To: buildroot; +Cc: Sen Hastings, Thomas Petazzoni
Instead of only checking .mk and Config.in{,.host}, check
all files in a package directory.
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v2 -> v3:
- new patch
---
support/scripts/pkg-stats | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 1ac538f5f9..c124b8a9cf 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -272,8 +272,7 @@ class Package:
self.status['pkg-check'] = ("error", "Missing")
for root, dirs, files in os.walk(pkgdir):
for f in files:
- if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
- cmd.append(os.path.join(root, f))
+ cmd.append(os.path.join(root, f))
o = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[1]
lines = o.splitlines()
for line in lines:
--
2.41.0
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 4/8] support/scripts/gen-missing-cpe: remove rarely used script
2023-08-12 19:28 [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Daniel Lang
2023-08-12 19:28 ` [Buildroot] [PATCH v3 2/8] support/scripts/pkg-stats: ignore llvm-project.mk Daniel Lang
2023-08-12 19:28 ` [Buildroot] [PATCH v3 3/8] support/scripts/pkg-stats: check all files for warnings Daniel Lang
@ 2023-08-12 19:28 ` Daniel Lang
2023-08-30 20:46 ` Arnout Vandecappelle via buildroot
2023-08-12 19:28 ` [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class Daniel Lang
` (5 subsequent siblings)
8 siblings, 1 reply; 24+ messages in thread
From: Daniel Lang @ 2023-08-12 19:28 UTC (permalink / raw)
To: buildroot; +Cc: Sen Hastings, Thomas Petazzoni
As discussed on the mailing list [0] keeping up with version numbers of
all registered CPE ID won't work.
In addition the feed used to generated the XML files will be retired
[1]. In the future an API needs to be used for fetching the data in
connection with a local database.
All of this works against keeping this script and porting it to the new
API.
As a last blow Matthew, the original author concluded [2]:
Makes sense to drop it. There never got to be enough momentum in the overall
software community to make CVE or even the new identifier really accurate.
[0]: https://lists.buildroot.org/pipermail/buildroot/2023-August/672620.html
[1]: https://nvd.nist.gov/General/News/change-timeline
[2]: https://lists.buildroot.org/pipermail/buildroot/2023-August/672651.html
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v2 -> v3:
- added this patch after feedback from Thomas
---
Makefile | 9 --
support/scripts/cpedb.py | 174 --------------------------------
support/scripts/gen-missing-cpe | 65 ------------
support/scripts/pkg-stats | 2 +-
4 files changed, 1 insertion(+), 249 deletions(-)
delete mode 100644 support/scripts/cpedb.py
delete mode 100755 support/scripts/gen-missing-cpe
diff --git a/Makefile b/Makefile
index 080136bc9a..fd807cc3cc 100644
--- a/Makefile
+++ b/Makefile
@@ -927,14 +927,6 @@ pkg-stats:
--html $(O)/pkg-stats.html \
--nvd-path $(DL_DIR)/buildroot-nvd
-.PHONY: missing-cpe
-missing-cpe:
- $(Q)mkdir -p $(CPE_UPDATES_DIR)
- $(Q)cd "$(CONFIG_DIR)" ; \
- $(TOPDIR)/support/scripts/gen-missing-cpe \
- --nvd-path $(DL_DIR)/buildroot-nvd \
- --output $(CPE_UPDATES_DIR)
-
else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
# Some subdirectories are also package names. To avoid that "make linux"
@@ -1191,7 +1183,6 @@ help:
@echo ' legal-info - generate info about license compliance'
@echo ' show-info - generate info about packages, as a JSON blurb'
@echo ' pkg-stats - generate info about packages as JSON and HTML'
- @echo ' missing-cpe - generate XML snippets for missing CPE identifiers'
@echo ' printvars - dump internal variables selected with VARS=...'
@echo ' show-vars - dump all internal variables as a JSON blurb; use VARS=...'
@echo ' to limit the list to variables names matching that pattern'
diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
deleted file mode 100644
index f4daf56124..0000000000
--- a/support/scripts/cpedb.py
+++ /dev/null
@@ -1,174 +0,0 @@
-#!/usr/bin/env python3
-
-import xml.etree.ElementTree as ET
-from xml.etree.ElementTree import Element, SubElement
-import gzip
-import os
-import requests
-import time
-from xml.dom import minidom
-
-VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
-
-CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
-
-ns = {
- '': 'http://cpe.mitre.org/dictionary/2.0',
- 'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
- 'xml': 'http://www.w3.org/XML/1998/namespace'
-}
-
-
-class CPE:
- def __init__(self, cpe_str, titles, refs):
- self.cpe_str = cpe_str
- self.titles = titles
- self.references = refs
- self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
-
- def update_xml_dict(self):
- ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
- cpes = Element('cpe-list')
- cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")
- cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
- cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
- cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
- cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
- cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
- cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
- "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",
- "http://cpe.mitre.org/dictionary/2.0",
- "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
- "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
- "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
- "http://scap.nist.gov/schema/scap-core/0.3",
- "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
- "http://scap.nist.gov/schema/configuration/0.1",
- "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
- "http://scap.nist.gov/schema/scap-core/0.1",
- "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
- item = SubElement(cpes, 'cpe-item')
- cpe_short_name = CPE.short_name(self.cpe_str)
- cpe_new_ver = CPE.version_update(self.cpe_str)
-
- item.set('name', 'cpe:/' + cpe_short_name)
- self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
- for title in self.titles:
- item.append(title)
- if self.references:
- item.append(self.references)
- cpe23item = SubElement(item, 'cpe-23:cpe23-item')
- cpe23item.set('name', self.cpe_str)
-
- # Generate the XML as a string
- xmlstr = ET.tostring(cpes)
-
- # And use minidom to pretty print the XML
- return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
-
- @staticmethod
- def version(cpe):
- return cpe.split(":")[5]
-
- @staticmethod
- def product(cpe):
- return cpe.split(":")[4]
-
- @staticmethod
- def short_name(cpe):
- return ":".join(cpe.split(":")[2:6])
-
- @staticmethod
- def version_update(cpe):
- return ":".join(cpe.split(":")[5:6])
-
- @staticmethod
- def no_version(cpe):
- return ":".join(cpe.split(":")[:5])
-
-
-class CPEDB:
- def __init__(self, nvd_path):
- self.all_cpes = dict()
- self.all_cpes_no_version = dict()
- self.nvd_path = nvd_path
-
- def get_xml_dict(self):
- print("CPE: Setting up NIST dictionary")
- if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
- os.makedirs(os.path.join(self.nvd_path, "cpe"))
-
- cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
- if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
- print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
- cpe_dict = requests.get(CPEDB_URL)
- open(cpe_dict_local, "wb").write(cpe_dict.content)
-
- print("CPE: Unzipping xml manifest...")
- nist_cpe_file = gzip.GzipFile(fileobj=open(cpe_dict_local, 'rb'))
- print("CPE: Converting xml manifest to dict...")
- tree = ET.parse(nist_cpe_file)
- all_cpedb = tree.getroot()
- self.parse_dict(all_cpedb)
-
- def parse_dict(self, all_cpedb):
- # Cycle through the dict and build two dict to be used for custom
- # lookups of partial and complete CPE objects
- # The objects are then used to create new proposed XML updates if
- # if is determined one is required
- # Out of the different language titles, select English
- for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
- cpe_titles = []
- for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
- title.tail = None
- cpe_titles.append(title)
-
- # Some older CPE don't include references, if they do, make
- # sure we handle the case of one ref needing to be packed
- # in a list
- cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
- if cpe_ref:
- for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
- ref.tail = None
- ref.text = ref.text.upper()
- if ref.text not in VALID_REFS:
- ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
- cpe_ref.tail = None
- cpe_ref.text = None
-
- cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
- item = CPE(cpe_str, cpe_titles, cpe_ref)
- cpe_str_no_version = CPE.no_version(cpe_str)
- # This dict must have a unique key for every CPE version
- # which allows matching to the specific obj data of that
- # NIST dict entry
- self.all_cpes.update({cpe_str: item})
- # This dict has one entry for every CPE (w/o version) to allow
- # partial match (no valid version) check (the obj is saved and
- # used as seed for suggested xml updates. By updating the same
- # non-version'd entry, it assumes the last update here is the
- # latest version in the NIST dict)
- self.all_cpes_no_version.update({cpe_str_no_version: item})
-
- def find_partial(self, cpe_str):
- cpe_str_no_version = CPE.no_version(cpe_str)
- if cpe_str_no_version in self.all_cpes_no_version:
- return cpe_str_no_version
-
- def find_partial_obj(self, cpe_str):
- cpe_str_no_version = CPE.no_version(cpe_str)
- if cpe_str_no_version in self.all_cpes_no_version:
- return self.all_cpes_no_version[cpe_str_no_version]
-
- def find_partial_latest_version(self, cpe_str_partial):
- cpe_obj = self.find_partial_obj(cpe_str_partial)
- return cpe_obj.cpe_cur_ver
-
- def find(self, cpe_str):
- if self.find_partial(cpe_str):
- if cpe_str in self.all_cpes:
- return cpe_str
-
- def gen_update_xml(self, cpe_str):
- cpe = self.find_partial_obj(cpe_str)
- return cpe.update_xml_dict()
diff --git a/support/scripts/gen-missing-cpe b/support/scripts/gen-missing-cpe
deleted file mode 100755
index 0b222f2659..0000000000
--- a/support/scripts/gen-missing-cpe
+++ /dev/null
@@ -1,65 +0,0 @@
-#!/usr/bin/env python3
-
-import argparse
-import sys
-import json
-import subprocess
-import os
-from cpedb import CPEDB, CPE
-
-
-def gen_update_xml_reports(cpeids, cpedb, output):
- cpe_need_update = []
-
- for cpe in cpeids:
- result = cpedb.find(cpe)
- if not result:
- result = cpedb.find_partial(CPE.no_version(cpe))
- if result:
- cpe_need_update.append(cpe)
- else:
- print("WARNING: no match found for '%s'" % cpe)
-
- for cpe in cpe_need_update:
- xml = cpedb.gen_update_xml(cpe)
- fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
- print("Generating %s" % fname)
- with open(os.path.join(output, fname), 'w+') as fp:
- fp.write(xml)
-
- print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpeids)))
-
-
-def get_cpe_ids():
- print("Getting list of CPE for enabled packages")
- cmd = ["make", "--no-print-directory", "show-info"]
- js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
- return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
-
-
-def resolvepath(path):
- return os.path.abspath(os.path.expanduser(path))
-
-
-def parse_args():
- parser = argparse.ArgumentParser()
- parser.add_argument('--output', dest='output',
- help='Path to the output CPE update files', type=resolvepath, required=True)
- parser.add_argument('--nvd-path', dest='nvd_path',
- help='Path to the local NVD database', type=resolvepath, required=True)
- return parser.parse_args()
-
-
-def __main__():
- args = parse_args()
- if not os.path.isdir(args.output):
- print("ERROR: output directory %s does not exist" % args.output)
- sys.exit(1)
- cpedb = CPEDB(args.nvd_path)
- cpedb.get_xml_dict()
- cpeids = get_cpe_ids()
- gen_update_xml_reports(cpeids, cpedb, args.output)
-
-
-if __name__ == "__main__":
- __main__()
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index c124b8a9cf..3cb9da6a0b 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -37,10 +37,10 @@ brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
sys.path.append(os.path.join(brpath, "utils"))
from getdeveloperlib import parse_developers # noqa: E402
-from cpedb import CPEDB_URL # noqa: E402
INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
URL_RE = re.compile(r"\s*https?://\S*\s*$")
+CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
RM_API_STATUS_ERROR = 1
RM_API_STATUS_FOUND_BY_DISTRO = 2
--
2.41.0
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class
2023-08-12 19:28 [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Daniel Lang
` (2 preceding siblings ...)
2023-08-12 19:28 ` [Buildroot] [PATCH v3 4/8] support/scripts/gen-missing-cpe: remove rarely used script Daniel Lang
@ 2023-08-12 19:28 ` Daniel Lang
2023-08-30 21:37 ` Arnout Vandecappelle via buildroot
2023-09-01 7:10 ` Arnout Vandecappelle via buildroot
2023-08-12 19:28 ` [Buildroot] [PATCH v3 6/8] support/scripts/cve.py: switch to NVD API v2 Daniel Lang
` (4 subsequent siblings)
8 siblings, 2 replies; 24+ messages in thread
From: Daniel Lang @ 2023-08-12 19:28 UTC (permalink / raw)
To: buildroot; +Cc: Thomas Petazzoni
The current NVD data feeds used for CVE and CPE checking will be retired
by 2023-12-05 [0]. Both have to be switched to the new v2 API. Since
fetching data from both sources workes the same, a common base class is
used to handle the API interaction.
To store the data locally a sqlite database is used.
[0]: https://nvd.nist.gov/General/News/change-timeline
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
DEVELOPERS | 1 +
support/scripts/nvd_api_v2.py | 138 ++++++++++++++++++++++++++++++++++
2 files changed, 139 insertions(+)
create mode 100755 support/scripts/nvd_api_v2.py
diff --git a/DEVELOPERS b/DEVELOPERS
index 6ffa3ee693..81f809a4c0 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -668,6 +668,7 @@ F: package/paho-mqtt-cpp/
F: package/pangomm/
F: package/pangomm2_46/
F: package/sam-ba/
+F: support/scripts/nvd_api_v2.py
N: Damien Lanson <damien@kal-host.com>
F: package/libvdpau/
diff --git a/support/scripts/nvd_api_v2.py b/support/scripts/nvd_api_v2.py
new file mode 100755
index 0000000000..3fdf32596f
--- /dev/null
+++ b/support/scripts/nvd_api_v2.py
@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+
+from datetime import datetime, timezone
+import os
+import requests
+import shutil
+import sqlite3
+import time
+
+NVD_API_VERSION = '2.0'
+NVD_API_BASE_URL = 'https://services.nvd.nist.gov/rest/json'
+
+
+class NVD_API:
+ """
+ A helper class that fetches data from a NVD API and
+ helps manage a sqlite database.
+ """
+ def __init__(self, nvd_path, service, database_file_prefix):
+ """
+ Initialize a new NVD API endpoint with service
+ as the URL postfix.
+ """
+ self.nvd_path = nvd_path
+ self.service = service
+ self.url = '%s/%s/%s' % (NVD_API_BASE_URL, service.lower(), NVD_API_VERSION)
+ self.db_file = os.path.join(nvd_path, '%s-%s.sqlite' % (database_file_prefix, NVD_API_VERSION))
+ self.db_file_tmp = '%s.tmp' % self.db_file
+
+ def init_db(self):
+ """
+ Needs to be implemented by derived classes.
+ Used to make sure that database tables exist.
+ """
+ pass
+
+ def save_to_db(self, start_index, total_results, content):
+ """
+ Needs to be implemented by derived classes.
+ Used to save the data given by a single API request
+ to the database.
+ """
+ pass
+
+ def cleanup_db(self):
+ """
+ Clean up any files that where left by previously
+ failed runs.
+ """
+ if os.path.exists(self.db_file_tmp):
+ os.remove(self.db_file_tmp)
+
+ def open_db(self, tmp=False):
+ """
+ Open and return a connection to the sqlite database.
+ """
+ if tmp:
+ return sqlite3.connect(self.db_file_tmp)
+ return sqlite3.connect(self.db_file)
+
+ def download(self, last_update):
+ """
+ Download all entries from NVD since last_update (if not None).
+ For each downloaded page save_to_db is called to together with
+ progress information.
+ NVD rate limiting allows 5 requests per 30 seconds or one every
+ 6 seconds.
+ """
+ args = {}
+ start_index = 0
+ total_results = 0
+ results_per_page = 0
+
+ print('Downloading new %s' % self.service)
+
+ if (last_update is not None):
+ args['lastModStartDate'] = last_update.isoformat()
+ args['lastModEndDate'] = datetime.now(tz=timezone.utc).isoformat()
+
+ while True:
+ args['startIndex'] = start_index
+
+ for attempt in range(5):
+ try:
+ page = requests.get(self.url, params=args)
+ page.raise_for_status()
+ content = page.json()
+ except Exception:
+ time.sleep(6)
+ else:
+ break
+
+ if content is None:
+ # Nothing was downloaded
+ return False
+
+ results_per_page = content['resultsPerPage']
+ total_results = content['totalResults']
+ start_index = content['startIndex']
+
+ start_index += results_per_page
+
+ if self.save_to_db(start_index, total_results, content) is not True:
+ return False
+
+ self.connection.commit()
+
+ if start_index >= total_results:
+ return True
+
+ # Otherwise rate limiting will be hit.
+ time.sleep(6)
+
+ def check_for_updates(self):
+ """
+ Check if the database file exists and if the last
+ update was more than 24 hours ago.
+ """
+ self.cleanup_db()
+ last_update = None
+ if os.path.exists(self.db_file):
+ last_update = os.stat(self.db_file).st_mtime
+ if last_update >= time.time() - 86400:
+ return []
+ # NVD uses UTC timestamps
+ last_update = datetime.fromtimestamp(last_update, tz=timezone.utc)
+ shutil.copy2(self.db_file, self.db_file_tmp)
+
+ self.connection = self.open_db(True)
+ self.init_db()
+
+ success = self.download(last_update)
+ self.connection.close()
+ if success:
+ shutil.move(self.db_file_tmp, self.db_file)
+ else:
+ print("Update failed!")
+ os.remove(self.db_file_tmp)
--
2.41.0
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 6/8] support/scripts/cve.py: switch to NVD API v2
2023-08-12 19:28 [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Daniel Lang
` (3 preceding siblings ...)
2023-08-12 19:28 ` [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class Daniel Lang
@ 2023-08-12 19:28 ` Daniel Lang
2023-08-12 19:28 ` [Buildroot] [PATCH v3 7/8] support/scripts/pkg-stats: switch CPEs " Daniel Lang
` (3 subsequent siblings)
8 siblings, 0 replies; 24+ messages in thread
From: Daniel Lang @ 2023-08-12 19:28 UTC (permalink / raw)
To: buildroot; +Cc: Sen Hastings, Thomas Petazzoni
The currently used feed will be retired in December 2023 [0].
As an alternative the new v2 API [1]. The new API allows downloading
sets of CVEs (2k at a time) that were modified after a given
UTC timestamp. Rate limiting of 5 requests in a sliding 30 seconds
window is used [2].
[0]: https://nvd.nist.gov/General/News/change-timeline
[1]: https://nvd.nist.gov/developers/vulnerabilities
[2]: https://nvd.nist.gov/developers/start-here
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v1 -> v2:
- switch to sqlite database for a more future proof storage
- CPE_ID class has been moved to a laster patch, therefore not used here
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
DEVELOPERS | 1 +
support/scripts/cve.py | 415 ++++++++++++++++++++------------------
support/scripts/pkg-stats | 31 ++-
3 files changed, 238 insertions(+), 209 deletions(-)
diff --git a/DEVELOPERS b/DEVELOPERS
index 81f809a4c0..cd4a58f2e6 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -668,6 +668,7 @@ F: package/paho-mqtt-cpp/
F: package/pangomm/
F: package/pangomm2_46/
F: package/sam-ba/
+F: support/scripts/cve.py
F: support/scripts/nvd_api_v2.py
N: Damien Lanson <damien@kal-host.com>
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index 7cd6fce4d8..4087df2ae3 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -17,40 +17,9 @@
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-import datetime
-import os
-import requests # URL checking
import distutils.version
-import time
-import gzip
-import sys
import operator
-
-try:
- import ijson
- # backend is a module in < 2.5, a string in >= 2.5
- if 'python' in getattr(ijson.backend, '__name__', ijson.backend):
- try:
- import ijson.backends.yajl2_cffi as ijson
- except ImportError:
- sys.stderr.write('Warning: Using slow ijson python backend\n')
-except ImportError:
- sys.stderr.write("You need ijson to parse NVD for CVE check\n")
- exit(1)
-
-sys.path.append('utils/')
-
-NVD_START_YEAR = 2002
-NVD_JSON_VERSION = "1.1"
-NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
-
-ops = {
- '>=': operator.ge,
- '>': operator.gt,
- '<=': operator.le,
- '<': operator.lt,
- '=': operator.eq
-}
+from nvd_api_v2 import NVD_API
# Check if two CPE IDs match each other
@@ -77,141 +46,32 @@ class CVE:
CVE_DOESNT_AFFECT = 2
CVE_UNKNOWN = 3
- def __init__(self, nvd_cve):
- """Initialize a CVE from its NVD JSON representation"""
- self.nvd_cve = nvd_cve
-
- @staticmethod
- def download_nvd_year(nvd_path, year):
- metaf = "nvdcve-%s-%s.meta" % (NVD_JSON_VERSION, year)
- path_metaf = os.path.join(nvd_path, metaf)
- jsonf_gz = "nvdcve-%s-%s.json.gz" % (NVD_JSON_VERSION, year)
- path_jsonf_gz = os.path.join(nvd_path, jsonf_gz)
-
- # If the database file is less than a day old, we assume the NVD data
- # locally available is recent enough.
- if os.path.exists(path_jsonf_gz) and os.stat(path_jsonf_gz).st_mtime >= time.time() - 86400:
- return path_jsonf_gz
-
- # If not, we download the meta file
- url = "%s/%s" % (NVD_BASE_URL, metaf)
- print("Getting %s" % url)
- page_meta = requests.get(url)
- page_meta.raise_for_status()
-
- # If the meta file already existed, we compare the existing
- # one with the data newly downloaded. If they are different,
- # we need to re-download the database.
- # If the database does not exist locally, we need to redownload it in
- # any case.
- if os.path.exists(path_metaf) and os.path.exists(path_jsonf_gz):
- meta_known = open(path_metaf, "r").read()
- if page_meta.text == meta_known:
- return path_jsonf_gz
-
- # Grab the compressed JSON NVD, and write files to disk
- url = "%s/%s" % (NVD_BASE_URL, jsonf_gz)
- print("Getting %s" % url)
- page_json = requests.get(url)
- page_json.raise_for_status()
- open(path_jsonf_gz, "wb").write(page_json.content)
- open(path_metaf, "w").write(page_meta.text)
- return path_jsonf_gz
-
- @classmethod
- def read_nvd_dir(cls, nvd_dir):
- """
- Iterate over all the CVEs contained in NIST Vulnerability Database
- feeds since NVD_START_YEAR. If the files are missing or outdated in
- nvd_dir, a fresh copy will be downloaded, and kept in .json.gz
- """
- for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1):
- filename = CVE.download_nvd_year(nvd_dir, year)
- try:
- content = ijson.items(gzip.GzipFile(filename), 'CVE_Items.item')
- except: # noqa: E722
- print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
- raise
- for cve in content:
- yield cls(cve)
-
- def each_product(self):
- """Iterate over each product section of this cve"""
- for vendor in self.nvd_cve['cve']['affects']['vendor']['vendor_data']:
- for product in vendor['product']['product_data']:
- yield product
-
- def parse_node(self, node):
- """
- Parse the node inside the configurations section to extract the
- cpe information usefull to know if a product is affected by
- the CVE. Actually only the product name and the version
- descriptor are needed, but we also provide the vendor name.
- """
+ ops = {
+ '>=': operator.ge,
+ '>': operator.gt,
+ '<=': operator.le,
+ '<': operator.lt,
+ '=': operator.eq
+ }
- # The node containing the cpe entries matching the CVE can also
- # contain sub-nodes, so we need to manage it.
- for child in node.get('children', ()):
- for parsed_node in self.parse_node(child):
- yield parsed_node
-
- for cpe in node.get('cpe_match', ()):
- if not cpe['vulnerable']:
- return
- product = cpe_product(cpe['cpe23Uri'])
- version = cpe_version(cpe['cpe23Uri'])
- # ignore when product is '-', which means N/A
- if product == '-':
- return
- op_start = ''
- op_end = ''
- v_start = ''
- v_end = ''
-
- if version != '*' and version != '-':
- # Version is defined, this is a '=' match
- op_start = '='
- v_start = version
- else:
- # Parse start version, end version and operators
- if 'versionStartIncluding' in cpe:
- op_start = '>='
- v_start = cpe['versionStartIncluding']
-
- if 'versionStartExcluding' in cpe:
- op_start = '>'
- v_start = cpe['versionStartExcluding']
-
- if 'versionEndIncluding' in cpe:
- op_end = '<='
- v_end = cpe['versionEndIncluding']
-
- if 'versionEndExcluding' in cpe:
- op_end = '<'
- v_end = cpe['versionEndExcluding']
-
- yield {
- 'id': cpe['cpe23Uri'],
- 'v_start': v_start,
- 'op_start': op_start,
- 'v_end': v_end,
- 'op_end': op_end
- }
-
- def each_cpe(self):
- for node in self.nvd_cve['configurations']['nodes']:
- for cpe in self.parse_node(node):
- yield cpe
+ def __init__(self, nvd_cve):
+ """Initialize a CVE from the database tuple representation"""
+ self.id = nvd_cve[0]
+ self.match_criteria = nvd_cve[2]
+ self.v_start = nvd_cve[3]
+ self.v_end = nvd_cve[4]
+ self.op_start = nvd_cve[5]
+ self.op_end = nvd_cve[6]
@property
def identifier(self):
"""The CVE unique identifier"""
- return self.nvd_cve['cve']['CVE_data_meta']['ID']
+ return self.id
@property
- def affected_products(self):
- """The set of CPE products referred by this CVE definition"""
- return set(cpe_product(p['id']) for p in self.each_cpe())
+ def affected_product(self):
+ """Name of the affected product"""
+ return cpe_product(self.match_criteria)
def affects(self, name, version, cve_ignore_list, cpeid=None):
"""
@@ -235,39 +95,208 @@ class CVE:
else:
pkg_version = distutils.version.LooseVersion(cpe_version(cpeid))
- for cpe in self.each_cpe():
- if not cpe_matches(cpe['id'], cpeid):
- continue
- if not cpe['v_start'] and not cpe['v_end']:
- return self.CVE_AFFECTS
- if not pkg_version:
+ if not cpe_matches(self.match_criteria, cpeid):
+ return self.CVE_DOESNT_AFFECT
+ if not self.v_start and not self.v_end:
+ return self.CVE_AFFECTS
+ if not pkg_version:
+ return self.CVE_DOESNT_AFFECT
+
+ if self.v_start:
+ try:
+ cve_affected_version = distutils.version.LooseVersion(self.v_start)
+ inrange = self.ops.get(self.op_start)(pkg_version, cve_affected_version)
+ except TypeError:
+ return self.CVE_UNKNOWN
+
+ # current package version is before v_start, so we're
+ # not affected by the CVE
+ if not inrange:
+ return self.CVE_DOESNT_AFFECT
+
+ if self.v_end:
+ try:
+ cve_affected_version = distutils.version.LooseVersion(self.v_end)
+ inrange = self.ops.get(self.op_end)(pkg_version, cve_affected_version)
+ except TypeError:
+ return self.CVE_UNKNOWN
+
+ # current package version is after v_end, so we're
+ # not affected by the CVE
+ if not inrange:
+ return self.CVE_DOESNT_AFFECT
+
+ # We're in the version range affected by this CVE
+ return self.CVE_AFFECTS
+
+
+class CVE_API(NVD_API):
+ """Download and manage CVEs in a sqlite database."""
+ def __init__(self, nvd_path):
+ """ Create a new API and database endpoint."""
+ NVD_API.__init__(self, nvd_path, 'CVEs', 'nvdcve')
+
+ def init_db(self):
+ """
+ Create all tables if the are missing.
+ """
+ cursor = self.connection.cursor()
+
+ cursor.execute('CREATE TABLE IF NOT EXISTS cves (\
+ id TEXT UNIQUE, \
+ description TEXT, \
+ metric2 REAL, \
+ metric3 REAL, \
+ severity TEXT)')
+
+ cursor.execute('CREATE TABLE IF NOT EXISTS cpe_matches (\
+ id TEXT UNIQUE, \
+ criteria TEXT, \
+ version_start TEXT, \
+ version_end TEXT, \
+ operator_start TEXT, \
+ operator_end TEXT)')
+
+ cursor.execute('CREATE TABLE IF NOT EXISTS configurations (\
+ cve_id TEXT, \
+ cpe_match_id TEXT, \
+ FOREIGN KEY (cve_id) REFERENCES cve (id) ON DELETE CASCADE, \
+ FOREIGN KEY (cpe_match_id) REFERENCES cpe_match (id) ON DELETE CASCADE, \
+ UNIQUE (cve_id, cpe_match_id))')
+
+ cursor.close()
+
+ def extract_cve_data(self, cve):
+ """Map CVE API data to database fields."""
+ description = ''
+ for d in cve['descriptions']:
+ if d['lang'] == 'en':
+ description = d['value']
+ metric2 = 0.0
+ metric3 = 0.0
+ severity = 'UNKNOWN'
+ if 'cvssMetricV31' in cve['metrics']:
+ metric3 = cve['metrics']['cvssMetricV31'][0]['cvssData']['baseScore']
+ severity = cve['metrics']['cvssMetricV31'][0]['cvssData']['baseSeverity']
+ elif 'cvssMetricV30' in cve['metrics']:
+ metric3 = cve['metrics']['cvssMetricV30'][0]['cvssData']['baseScore']
+ severity = cve['metrics']['cvssMetricV30'][0]['cvssData']['baseSeverity']
+ elif 'cvssMetricV2' in cve['metrics']:
+ metric2 = cve['metrics']['cvssMetricV2'][0]['cvssData']['baseScore']
+ severity = cve['metrics']['cvssMetricV2'][0]['baseSeverity']
+
+ return [cve['id'], description, metric2, metric3, severity]
+
+ def extract_cpe_match_data(self, cpe_match):
+ """Map CPE match information to database fields."""
+ product = cpe_product(cpe_match['criteria'])
+ version = cpe_version(cpe_match['criteria'])
+ # ignore when product is '-', which means N/A
+ if product == '-':
+ return
+ op_start = ''
+ op_end = ''
+ v_start = ''
+ v_end = ''
+
+ if version != '*' and version != '-':
+ # Version is defined, this is a '=' match
+ op_start = '='
+ v_start = version
+ else:
+ # Parse start version, end version and operators
+ if 'versionStartIncluding' in cpe_match:
+ op_start = '>='
+ v_start = cpe_match['versionStartIncluding']
+
+ if 'versionStartExcluding' in cpe_match:
+ op_start = '>'
+ v_start = cpe_match['versionStartExcluding']
+
+ if 'versionEndIncluding' in cpe_match:
+ op_end = '<='
+ v_end = cpe_match['versionEndIncluding']
+
+ if 'versionEndExcluding' in cpe_match:
+ op_end = '<'
+ v_end = cpe_match['versionEndExcluding']
+
+ return [
+ cpe_match['matchCriteriaId'],
+ cpe_match['criteria'],
+ v_start,
+ v_end,
+ op_start,
+ op_end
+ ]
+
+ def save_to_db(self, start_index, total_results, content):
+ """
+ Save the response of a single API request to the database
+ and report the progress.
+ """
+ cve_ids_changed = list()
+ cve_ids_dropped = list()
+ cves = list()
+ cpe_matches = list()
+ configurations = list()
+
+ for vul in content['vulnerabilities']:
+ if vul['cve']['vulnStatus'] == 'Rejected':
+ cve_ids_dropped.append((vul['cve']['id'],))
continue
- if cpe['v_start']:
- try:
- cve_affected_version = distutils.version.LooseVersion(cpe['v_start'])
- inrange = ops.get(cpe['op_start'])(pkg_version, cve_affected_version)
- except TypeError:
- return self.CVE_UNKNOWN
-
- # current package version is before v_start, so we're
- # not affected by the CVE
- if not inrange:
- continue
-
- if cpe['v_end']:
- try:
- cve_affected_version = distutils.version.LooseVersion(cpe['v_end'])
- inrange = ops.get(cpe['op_end'])(pkg_version, cve_affected_version)
- except TypeError:
- return self.CVE_UNKNOWN
-
- # current package version is after v_end, so we're
- # not affected by the CVE
- if not inrange:
- continue
-
- # We're in the version range affected by this CVE
- return self.CVE_AFFECTS
+ cve_ids_changed.append((vul['cve']['id'],))
+ cves.append(self.extract_cve_data(vul['cve']))
+
+ for config in vul['cve'].get('configurations', ()):
+ for node in config['nodes']:
+ for cpe_match in node['cpeMatch']:
+ if not cpe_match['vulnerable']:
+ continue
+ match_data = self.extract_cpe_match_data(cpe_match)
+ if not match_data:
+ continue
+ cpe_matches.append(match_data)
+ configurations.append([vul['cve']['id'], match_data[0]])
+
+ cursor = self.connection.cursor()
+
+ # Drop all CVEs that are rejected, status might have changed
+ cursor.executemany('DELETE FROM cves WHERE id = ?', cve_ids_dropped)
+ # Delete configuration mapping for included CVEs, otherwise we can't detect
+ # upstream dropping configurations.
+ cursor.executemany('DELETE FROM configurations WHERE cve_id = ?', cve_ids_changed)
+ cursor.executemany('INSERT OR REPLACE INTO cves VALUES (?, ?, ?, ?, ?)', cves)
+ cursor.executemany('INSERT OR REPLACE INTO cpe_matches VALUES (?, ?, ?, ?, ?, ?)', cpe_matches)
+ cursor.executemany('INSERT OR REPLACE INTO configurations VALUES (?, ?)', configurations)
+
+ cursor.close()
- return self.CVE_DOESNT_AFFECT
+ print("[%06d/%06d]" % (start_index, total_results))
+
+ return True
+
+ def load_all(self):
+ """
+ Load all entries from the database and use CVE class
+ to yield each result individually.
+ Each yielded object represents one configuration that
+ the included CVE is vulnerable for.
+ """
+ self.check_for_updates()
+
+ self.connection = self.open_db()
+ cursor = self.connection.cursor()
+ sql = 'SELECT c.id as cve_id, m.id, m.criteria, m.version_start, m.version_end, \
+ m.operator_start, m.operator_end \
+ FROM configurations \
+ INNER JOIN cves AS c ON c.id = configurations.cve_id \
+ INNER JOIN cpe_matches AS m ON m.id = configurations.cpe_match_id \
+ ORDER BY cve_id'
+
+ for row in cursor.execute(sql):
+ yield CVE(row)
+
+ cursor.close()
+ self.connection.close()
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 3cb9da6a0b..7be4ad9853 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -32,6 +32,7 @@ import time
import gzip
import xml.etree.ElementTree
import requests
+from cve import CVE_API, cpe_product
brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
@@ -607,15 +608,17 @@ async def check_package_latest_version(packages):
def check_package_cve_affects(cve, cpe_product_pkgs):
- for product in cve.affected_products:
- if product not in cpe_product_pkgs:
+ product = cve.affected_product
+ if product not in cpe_product_pkgs:
+ return
+ for pkg in cpe_product_pkgs[product]:
+ if cve.identifier in pkg.cves or cve.identifier in pkg.unsure_cves:
continue
- for pkg in cpe_product_pkgs[product]:
- cve_status = cve.affects(pkg.name, pkg.current_version, pkg.ignored_cves, pkg.cpeid)
- if cve_status == cve.CVE_AFFECTS:
- pkg.cves.append(cve.identifier)
- elif cve_status == cve.CVE_UNKNOWN:
- pkg.unsure_cves.append(cve.identifier)
+ cve_status = cve.affects(pkg.name, pkg.current_version, pkg.ignored_cves, pkg.cpeid)
+ if cve_status == cve.CVE_AFFECTS:
+ pkg.cves.append(cve.identifier)
+ elif cve_status == cve.CVE_UNKNOWN:
+ pkg.unsure_cves.append(cve.identifier)
def check_package_cves(nvd_path, packages):
@@ -631,12 +634,13 @@ def check_package_cves(nvd_path, packages):
pkg.status['cve'] = ("na", "no version information available")
continue
if pkg.cpeid:
- cpe_product = cvecheck.cpe_product(pkg.cpeid)
- cpe_product_pkgs[cpe_product].append(pkg)
+ product = cpe_product(pkg.cpeid)
+ cpe_product_pkgs[product].append(pkg)
else:
cpe_product_pkgs[pkg.name].append(pkg)
- for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
+ cve_api = CVE_API(nvd_path)
+ for cve in cve_api.load_all():
check_package_cve_affects(cve, cpe_product_pkgs)
for pkg in packages:
@@ -1285,13 +1289,8 @@ def parse_args():
def __main__():
- global cvecheck
-
args = parse_args()
- if args.nvd_path:
- import cve as cvecheck
-
show_info_js = None
if args.packages:
package_list = args.packages.split(",")
--
2.41.0
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 7/8] support/scripts/pkg-stats: switch CPEs to NVD API v2
2023-08-12 19:28 [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Daniel Lang
` (4 preceding siblings ...)
2023-08-12 19:28 ` [Buildroot] [PATCH v3 6/8] support/scripts/cve.py: switch to NVD API v2 Daniel Lang
@ 2023-08-12 19:28 ` Daniel Lang
2023-08-12 19:28 ` [Buildroot] [PATCH v3 8/8] support/scripts/pkg-stats: Only match CPE vendor and product Daniel Lang
` (2 subsequent siblings)
8 siblings, 0 replies; 24+ messages in thread
From: Daniel Lang @ 2023-08-12 19:28 UTC (permalink / raw)
To: buildroot; +Cc: Sen Hastings, Thomas Petazzoni
The currently used feed will be retired in December 2023 [0].
As an alternative the new v2 API [1]. The new API allows downloading
sets of CPEs (10k at a time) that were modified after a given
UTC timestamp. Rate limiting of 5 requests in a sliding 30 seconds
window is used [2].
[0]: https://nvd.nist.gov/General/News/change-timeline
[1]: https://nvd.nist.gov/developers/products
[2]: https://nvd.nist.gov/developers/start-here
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v2 -> v3:
- drop XML handling for gen-missing-cpe
- Move creation of CPE_ID class into this patch
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
support/scripts/cpe.py | 92 +++++++++++++++++++++++++++++++++++++++
support/scripts/cve.py | 29 +++---------
support/scripts/pkg-stats | 43 +++---------------
3 files changed, 103 insertions(+), 61 deletions(-)
create mode 100755 support/scripts/cpe.py
diff --git a/support/scripts/cpe.py b/support/scripts/cpe.py
new file mode 100755
index 0000000000..e452d8487b
--- /dev/null
+++ b/support/scripts/cpe.py
@@ -0,0 +1,92 @@
+#!/usr/bin/env python3
+
+from nvd_api_v2 import NVD_API
+
+
+class CPE_ID:
+ @staticmethod
+ def matches(cpe1, cpe2):
+ """Check if two CPE IDs match each other"""
+ cpe1_elems = cpe1.split(":")
+ cpe2_elems = cpe2.split(":")
+
+ remains = filter(lambda x: x[0] not in ["*", "-"] and x[1] not in ["*", "-"] and x[0] != x[1],
+ zip(cpe1_elems, cpe2_elems))
+ return len(list(remains)) == 0
+
+ @staticmethod
+ def product(cpe):
+ return cpe.split(':')[4]
+
+ @staticmethod
+ def version(cpe):
+ return cpe.split(':')[5]
+
+ @staticmethod
+ def no_version(cpe):
+ return ":".join(cpe.split(":")[:5])
+
+
+class CPE_API(NVD_API):
+ def __init__(self, nvd_path):
+ NVD_API.__init__(self, nvd_path, 'CPEs', 'nvdcpe')
+ self.cpes = list()
+ self.cpes_without_version = dict()
+
+ def init_db(self):
+ cursor = self.connection.cursor()
+
+ cursor.execute('CREATE TABLE IF NOT EXISTS products ( \
+ id TEXT UNIQUE, \
+ name TEXT)')
+
+ cursor.close()
+
+ def save_to_db(self, start_index, total_results, content):
+ cpe_ids_dropped = list()
+ products = list()
+
+ for product in content['products']:
+ if product['cpe']['deprecated']:
+ cpe_ids_dropped.append((product['cpe']['cpeNameId'],))
+ continue
+
+ cpe = product['cpe']
+
+ products.append([cpe['cpeNameId'], cpe['cpeName']])
+
+ cursor = self.connection.cursor()
+
+ # Drop all CPEs that are deprecated, status might have changed
+ cursor.executemany('DELETE FROM products WHERE id = ?', cpe_ids_dropped)
+ cursor.executemany('INSERT OR REPLACE INTO products VALUES (?, ?)', products)
+
+ print("[%07d/%07d]" % (start_index, total_results))
+
+ return True
+
+ def load_ids(self):
+ self.check_for_updates()
+
+ self.connection = self.open_db()
+ cursor = self.connection.cursor()
+
+ ids = list()
+ for row in cursor.execute('SELECT name FROM products'):
+ ids.append(row[0])
+
+ cursor.close()
+ self.connection.close()
+
+ self.cpes = ids
+ return ids
+
+ def generate_partials(self):
+ self.cpes_without_version = dict()
+ for cpe in self.cpes:
+ self.cpes_without_version[CPE_ID.no_version(cpe)] = cpe
+
+ def find_partial(self, cpe_id):
+ cpe_id_without_version = CPE_ID.no_version(cpe_id)
+ if cpe_id_without_version in self.cpes_without_version.keys():
+ return self.cpes_without_version[cpe_id_without_version]
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index 4087df2ae3..7af5786c15 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -20,24 +20,7 @@
import distutils.version
import operator
from nvd_api_v2 import NVD_API
-
-
-# Check if two CPE IDs match each other
-def cpe_matches(cpe1, cpe2):
- cpe1_elems = cpe1.split(":")
- cpe2_elems = cpe2.split(":")
-
- remains = filter(lambda x: x[0] not in ["*", "-"] and x[1] not in ["*", "-"] and x[0] != x[1],
- zip(cpe1_elems, cpe2_elems))
- return len(list(remains)) == 0
-
-
-def cpe_product(cpe):
- return cpe.split(':')[4]
-
-
-def cpe_version(cpe):
- return cpe.split(':')[5]
+from cpe import CPE_ID
class CVE:
@@ -71,7 +54,7 @@ class CVE:
@property
def affected_product(self):
"""Name of the affected product"""
- return cpe_product(self.match_criteria)
+ return CPE_ID.product(self.match_criteria)
def affects(self, name, version, cve_ignore_list, cpeid=None):
"""
@@ -93,9 +76,9 @@ class CVE:
# version, as they might be different due to
# <pkg>_CPE_ID_VERSION
else:
- pkg_version = distutils.version.LooseVersion(cpe_version(cpeid))
+ pkg_version = distutils.version.LooseVersion(CPE_ID.version(cpeid))
- if not cpe_matches(self.match_criteria, cpeid):
+ if not CPE_ID.matches(self.match_criteria, cpeid):
return self.CVE_DOESNT_AFFECT
if not self.v_start and not self.v_end:
return self.CVE_AFFECTS
@@ -189,8 +172,8 @@ class CVE_API(NVD_API):
def extract_cpe_match_data(self, cpe_match):
"""Map CPE match information to database fields."""
- product = cpe_product(cpe_match['criteria'])
- version = cpe_version(cpe_match['criteria'])
+ product = CPE_ID.product(cpe_match['criteria'])
+ version = CPE_ID.version(cpe_match['criteria'])
# ignore when product is '-', which means N/A
if product == '-':
return
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 7be4ad9853..196cbb660e 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -28,11 +28,8 @@ import re
import subprocess
import json
import sys
-import time
-import gzip
-import xml.etree.ElementTree
-import requests
-from cve import CVE_API, cpe_product
+from cpe import CPE_API, CPE_ID
+from cve import CVE_API
brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
@@ -41,7 +38,6 @@ from getdeveloperlib import parse_developers # noqa: E402
INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
URL_RE = re.compile(r"\s*https?://\S*\s*$")
-CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
RM_API_STATUS_ERROR = 1
RM_API_STATUS_FOUND_BY_DISTRO = 2
@@ -634,7 +630,7 @@ def check_package_cves(nvd_path, packages):
pkg.status['cve'] = ("na", "no version information available")
continue
if pkg.cpeid:
- product = cpe_product(pkg.cpeid)
+ product = CPE_ID.product(pkg.cpeid)
cpe_product_pkgs[product].append(pkg)
else:
cpe_product_pkgs[pkg.name].append(pkg)
@@ -652,37 +648,8 @@ def check_package_cves(nvd_path, packages):
def check_package_cpes(nvd_path, packages):
- class CpeXmlParser:
- cpes = []
-
- def start(self, tag, attrib):
- if tag == "{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item":
- self.cpes.append(attrib['name'])
-
- def close(self):
- return self.cpes
-
- print("CPE: Setting up NIST dictionary")
- if not os.path.exists(os.path.join(nvd_path, "cpe")):
- os.makedirs(os.path.join(nvd_path, "cpe"))
-
- cpe_dict_local = os.path.join(nvd_path, "cpe", os.path.basename(CPEDB_URL))
- if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
- print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
- cpe_dict = requests.get(CPEDB_URL)
- open(cpe_dict_local, "wb").write(cpe_dict.content)
-
- print("CPE: Unzipping xml manifest...")
- nist_cpe_file = gzip.GzipFile(fileobj=open(cpe_dict_local, 'rb'))
-
- parser = xml.etree.ElementTree.XMLParser(target=CpeXmlParser())
- while True:
- c = nist_cpe_file.read(1024*1024)
- if not c:
- break
- parser.feed(c)
- cpes = parser.close()
-
+ cpe_api = CPE_API(nvd_path)
+ cpes = cpe_api.load_ids()
for p in packages:
if not p.cpeid:
continue
--
2.41.0
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 8/8] support/scripts/pkg-stats: Only match CPE vendor and product
2023-08-12 19:28 [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Daniel Lang
` (5 preceding siblings ...)
2023-08-12 19:28 ` [Buildroot] [PATCH v3 7/8] support/scripts/pkg-stats: switch CPEs " Daniel Lang
@ 2023-08-12 19:28 ` Daniel Lang
2023-08-30 20:45 ` [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Arnout Vandecappelle via buildroot
2023-09-14 8:25 ` Peter Korsgaard
8 siblings, 0 replies; 24+ messages in thread
From: Daniel Lang @ 2023-08-12 19:28 UTC (permalink / raw)
To: buildroot; +Cc: Sen Hastings, Thomas Petazzoni
Instead of checking if the whole CPE string is known in the database,
only check if a string with the same vendor and product exists.
Reported-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v1 -> v2:
- patch added
---
support/scripts/pkg-stats | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 196cbb660e..06d3d483bf 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -649,11 +649,12 @@ def check_package_cves(nvd_path, packages):
def check_package_cpes(nvd_path, packages):
cpe_api = CPE_API(nvd_path)
- cpes = cpe_api.load_ids()
+ cpe_api.load_ids()
+ cpe_api.generate_partials()
for p in packages:
if not p.cpeid:
continue
- if p.cpeid in cpes:
+ if cpe_api.find_partial(p.cpeid):
p.status['cpe'] = ("ok", "verified CPE identifier")
else:
p.status['cpe'] = ("error", "CPE version unknown in CPE database")
--
2.41.0
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 2/8] support/scripts/pkg-stats: ignore llvm-project.mk
2023-08-12 19:28 ` [Buildroot] [PATCH v3 2/8] support/scripts/pkg-stats: ignore llvm-project.mk Daniel Lang
@ 2023-08-30 20:31 ` Arnout Vandecappelle via buildroot
2023-08-31 3:35 ` Daniel Lang
0 siblings, 1 reply; 24+ messages in thread
From: Arnout Vandecappelle via buildroot @ 2023-08-30 20:31 UTC (permalink / raw)
To: Daniel Lang, buildroot; +Cc: Sen Hastings, Thomas Petazzoni
On 12/08/2023 21:28, Daniel Lang wrote:
> Fixes:
> f6eaf60 ("package/llvm-project: new group for llvm packages")
>
> Signed-off-by: Daniel Lang <dalang@gmx.at>
> ---
> support/scripts/pkg-stats | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index eea900124c..1ac538f5f9 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -345,6 +345,7 @@ def get_pkglist(npackages, package_list):
> "package/gstreamer/gstreamer.mk",
> "package/gstreamer1/gstreamer1.mk",
> "package/gtk2-themes/gtk2-themes.mk",
> + "package/llvm-project/llvm-project.mk",
Why this specific one, and not the I-don't-know-how-many others that are in a
similar situation? E.g. qt6, barebox to take two examples.
Regards,
Arnout
> "package/matchbox/matchbox.mk",
> "package/opengl/opengl.mk",
> "package/qt5/qt5.mk",
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 3/8] support/scripts/pkg-stats: check all files for warnings
2023-08-12 19:28 ` [Buildroot] [PATCH v3 3/8] support/scripts/pkg-stats: check all files for warnings Daniel Lang
@ 2023-08-30 20:36 ` Arnout Vandecappelle via buildroot
2023-08-30 21:04 ` Thomas Petazzoni via buildroot
2023-08-31 19:52 ` Daniel Lang
0 siblings, 2 replies; 24+ messages in thread
From: Arnout Vandecappelle via buildroot @ 2023-08-30 20:36 UTC (permalink / raw)
To: Daniel Lang, buildroot
Cc: Romain Naour, Sen Hastings, Thomas Petazzoni, Yann E. MORIN
On 12/08/2023 21:28, Daniel Lang wrote:
> Instead of only checking .mk and Config.in{,.host}, check
> all files in a package directory.
("checking" here means "run check-package").
I think we should instead remove the .warnings and .status['pkg-check'] fields
entirely. It made sense back in the days, because we weren't running
check-package in CI. Now, however, we do, so there's little point to run
check-package as part of pkg-stats as well. On a.b.o the warnings column will
(almost) always be all 0.
There is the point of people using pkg-stats for their br2-external, but:
1. pkg-stats is pretty fragile so users shouldn't be too surprised b some breakage;
2. it's very easy for those users to run 'make check-package' instead.
Do other maintainers have the same opinion?
Regards,
Arnout
>
> Signed-off-by: Daniel Lang <dalang@gmx.at>
> ---
> v2 -> v3:
> - new patch
> ---
> support/scripts/pkg-stats | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 1ac538f5f9..c124b8a9cf 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -272,8 +272,7 @@ class Package:
> self.status['pkg-check'] = ("error", "Missing")
> for root, dirs, files in os.walk(pkgdir):
> for f in files:
> - if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
> - cmd.append(os.path.join(root, f))
> + cmd.append(os.path.join(root, f))
> o = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[1]
> lines = o.splitlines()
> for line in lines:
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos
2023-08-12 19:28 [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Daniel Lang
` (6 preceding siblings ...)
2023-08-12 19:28 ` [Buildroot] [PATCH v3 8/8] support/scripts/pkg-stats: Only match CPE vendor and product Daniel Lang
@ 2023-08-30 20:45 ` Arnout Vandecappelle via buildroot
2023-09-14 8:26 ` Peter Korsgaard
2023-09-14 8:25 ` Peter Korsgaard
8 siblings, 1 reply; 24+ messages in thread
From: Arnout Vandecappelle via buildroot @ 2023-08-30 20:45 UTC (permalink / raw)
To: Daniel Lang, buildroot; +Cc: Sen Hastings, Thomas Petazzoni
On 12/08/2023 21:28, Daniel Lang wrote:
> Signed-off-by: Daniel Lang <dalang@gmx.at>
Applied to master, thanks.
(Note: I'm applying this series to master instead of next, because we will
need the new NVD API on all stable branches.)
Regards,
Arnout
> ---
> support/scripts/pkg-stats | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 3248e3678d..eea900124c 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -84,7 +84,7 @@ class Package:
> all_ignored_cves = dict()
> all_cpeids = dict()
> # This is the list of all possible checks. Add new checks to this list so
> - # a tool that post-processeds the json output knows the checks before
> + # a tool that post-processes the json output knows the checks before
> # iterating over the packages.
> status_checks = ['cve', 'developers', 'hash', 'license',
> 'license-files', 'patches', 'pkg-check', 'url', 'version']
> @@ -259,7 +259,7 @@ class Package:
> if var in self.all_cpeids:
> self.cpeid = self.all_cpeids[var]
> # Set a preliminary status, it might be overridden by check_package_cpes()
> - self.status['cpe'] = ("warning", "not checked against CPE dictionnary")
> + self.status['cpe'] = ("warning", "not checked against CPE dictionary")
> else:
> self.status['cpe'] = ("error", "no verified CPE identifier")
>
> @@ -499,7 +499,7 @@ def check_package_latest_version_set_status(pkg, status, version, identifier):
>
>
> async def check_package_get_latest_version_by_distro(session, pkg, retry=True):
> - url = "https://release-monitoring.org//api/project/Buildroot/%s" % pkg.name
> + url = "https://release-monitoring.org/api/project/Buildroot/%s" % pkg.name
> try:
> async with session.get(url) as resp:
> if resp.status != 200:
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 4/8] support/scripts/gen-missing-cpe: remove rarely used script
2023-08-12 19:28 ` [Buildroot] [PATCH v3 4/8] support/scripts/gen-missing-cpe: remove rarely used script Daniel Lang
@ 2023-08-30 20:46 ` Arnout Vandecappelle via buildroot
0 siblings, 0 replies; 24+ messages in thread
From: Arnout Vandecappelle via buildroot @ 2023-08-30 20:46 UTC (permalink / raw)
To: Daniel Lang, buildroot; +Cc: Sen Hastings, Thomas Petazzoni
On 12/08/2023 21:28, Daniel Lang wrote:
> As discussed on the mailing list [0] keeping up with version numbers of
> all registered CPE ID won't work.
> In addition the feed used to generated the XML files will be retired
> [1]. In the future an API needs to be used for fetching the data in
> connection with a local database.
> All of this works against keeping this script and porting it to the new
> API.
> As a last blow Matthew, the original author concluded [2]:
> Makes sense to drop it. There never got to be enough momentum in the overall
> software community to make CVE or even the new identifier really accurate.
I have extended the commit log a bit to repeat a summary of the reasons for
the removal, and also to explain a bit what the patch does.
Applied to master, thanks.
Regards,
Arnout
>
> [0]: https://lists.buildroot.org/pipermail/buildroot/2023-August/672620.html
> [1]: https://nvd.nist.gov/General/News/change-timeline
> [2]: https://lists.buildroot.org/pipermail/buildroot/2023-August/672651.html
>
> Signed-off-by: Daniel Lang <dalang@gmx.at>
> ---
> v2 -> v3:
> - added this patch after feedback from Thomas
> ---
> Makefile | 9 --
> support/scripts/cpedb.py | 174 --------------------------------
> support/scripts/gen-missing-cpe | 65 ------------
> support/scripts/pkg-stats | 2 +-
> 4 files changed, 1 insertion(+), 249 deletions(-)
> delete mode 100644 support/scripts/cpedb.py
> delete mode 100755 support/scripts/gen-missing-cpe
>
> diff --git a/Makefile b/Makefile
> index 080136bc9a..fd807cc3cc 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -927,14 +927,6 @@ pkg-stats:
> --html $(O)/pkg-stats.html \
> --nvd-path $(DL_DIR)/buildroot-nvd
>
> -.PHONY: missing-cpe
> -missing-cpe:
> - $(Q)mkdir -p $(CPE_UPDATES_DIR)
> - $(Q)cd "$(CONFIG_DIR)" ; \
> - $(TOPDIR)/support/scripts/gen-missing-cpe \
> - --nvd-path $(DL_DIR)/buildroot-nvd \
> - --output $(CPE_UPDATES_DIR)
> -
> else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
>
> # Some subdirectories are also package names. To avoid that "make linux"
> @@ -1191,7 +1183,6 @@ help:
> @echo ' legal-info - generate info about license compliance'
> @echo ' show-info - generate info about packages, as a JSON blurb'
> @echo ' pkg-stats - generate info about packages as JSON and HTML'
> - @echo ' missing-cpe - generate XML snippets for missing CPE identifiers'
> @echo ' printvars - dump internal variables selected with VARS=...'
> @echo ' show-vars - dump all internal variables as a JSON blurb; use VARS=...'
> @echo ' to limit the list to variables names matching that pattern'
> diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
> deleted file mode 100644
> index f4daf56124..0000000000
> --- a/support/scripts/cpedb.py
> +++ /dev/null
> @@ -1,174 +0,0 @@
> -#!/usr/bin/env python3
> -
> -import xml.etree.ElementTree as ET
> -from xml.etree.ElementTree import Element, SubElement
> -import gzip
> -import os
> -import requests
> -import time
> -from xml.dom import minidom
> -
> -VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
> -
> -CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
> -
> -ns = {
> - '': 'http://cpe.mitre.org/dictionary/2.0',
> - 'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
> - 'xml': 'http://www.w3.org/XML/1998/namespace'
> -}
> -
> -
> -class CPE:
> - def __init__(self, cpe_str, titles, refs):
> - self.cpe_str = cpe_str
> - self.titles = titles
> - self.references = refs
> - self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
> -
> - def update_xml_dict(self):
> - ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
> - cpes = Element('cpe-list')
> - cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")
> - cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
> - cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
> - cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
> - cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
> - cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
> - cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
> - "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",
> - "http://cpe.mitre.org/dictionary/2.0",
> - "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
> - "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
> - "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
> - "http://scap.nist.gov/schema/scap-core/0.3",
> - "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
> - "http://scap.nist.gov/schema/configuration/0.1",
> - "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
> - "http://scap.nist.gov/schema/scap-core/0.1",
> - "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
> - item = SubElement(cpes, 'cpe-item')
> - cpe_short_name = CPE.short_name(self.cpe_str)
> - cpe_new_ver = CPE.version_update(self.cpe_str)
> -
> - item.set('name', 'cpe:/' + cpe_short_name)
> - self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
> - for title in self.titles:
> - item.append(title)
> - if self.references:
> - item.append(self.references)
> - cpe23item = SubElement(item, 'cpe-23:cpe23-item')
> - cpe23item.set('name', self.cpe_str)
> -
> - # Generate the XML as a string
> - xmlstr = ET.tostring(cpes)
> -
> - # And use minidom to pretty print the XML
> - return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
> -
> - @staticmethod
> - def version(cpe):
> - return cpe.split(":")[5]
> -
> - @staticmethod
> - def product(cpe):
> - return cpe.split(":")[4]
> -
> - @staticmethod
> - def short_name(cpe):
> - return ":".join(cpe.split(":")[2:6])
> -
> - @staticmethod
> - def version_update(cpe):
> - return ":".join(cpe.split(":")[5:6])
> -
> - @staticmethod
> - def no_version(cpe):
> - return ":".join(cpe.split(":")[:5])
> -
> -
> -class CPEDB:
> - def __init__(self, nvd_path):
> - self.all_cpes = dict()
> - self.all_cpes_no_version = dict()
> - self.nvd_path = nvd_path
> -
> - def get_xml_dict(self):
> - print("CPE: Setting up NIST dictionary")
> - if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
> - os.makedirs(os.path.join(self.nvd_path, "cpe"))
> -
> - cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
> - if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
> - print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
> - cpe_dict = requests.get(CPEDB_URL)
> - open(cpe_dict_local, "wb").write(cpe_dict.content)
> -
> - print("CPE: Unzipping xml manifest...")
> - nist_cpe_file = gzip.GzipFile(fileobj=open(cpe_dict_local, 'rb'))
> - print("CPE: Converting xml manifest to dict...")
> - tree = ET.parse(nist_cpe_file)
> - all_cpedb = tree.getroot()
> - self.parse_dict(all_cpedb)
> -
> - def parse_dict(self, all_cpedb):
> - # Cycle through the dict and build two dict to be used for custom
> - # lookups of partial and complete CPE objects
> - # The objects are then used to create new proposed XML updates if
> - # if is determined one is required
> - # Out of the different language titles, select English
> - for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
> - cpe_titles = []
> - for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
> - title.tail = None
> - cpe_titles.append(title)
> -
> - # Some older CPE don't include references, if they do, make
> - # sure we handle the case of one ref needing to be packed
> - # in a list
> - cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
> - if cpe_ref:
> - for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
> - ref.tail = None
> - ref.text = ref.text.upper()
> - if ref.text not in VALID_REFS:
> - ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
> - cpe_ref.tail = None
> - cpe_ref.text = None
> -
> - cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
> - item = CPE(cpe_str, cpe_titles, cpe_ref)
> - cpe_str_no_version = CPE.no_version(cpe_str)
> - # This dict must have a unique key for every CPE version
> - # which allows matching to the specific obj data of that
> - # NIST dict entry
> - self.all_cpes.update({cpe_str: item})
> - # This dict has one entry for every CPE (w/o version) to allow
> - # partial match (no valid version) check (the obj is saved and
> - # used as seed for suggested xml updates. By updating the same
> - # non-version'd entry, it assumes the last update here is the
> - # latest version in the NIST dict)
> - self.all_cpes_no_version.update({cpe_str_no_version: item})
> -
> - def find_partial(self, cpe_str):
> - cpe_str_no_version = CPE.no_version(cpe_str)
> - if cpe_str_no_version in self.all_cpes_no_version:
> - return cpe_str_no_version
> -
> - def find_partial_obj(self, cpe_str):
> - cpe_str_no_version = CPE.no_version(cpe_str)
> - if cpe_str_no_version in self.all_cpes_no_version:
> - return self.all_cpes_no_version[cpe_str_no_version]
> -
> - def find_partial_latest_version(self, cpe_str_partial):
> - cpe_obj = self.find_partial_obj(cpe_str_partial)
> - return cpe_obj.cpe_cur_ver
> -
> - def find(self, cpe_str):
> - if self.find_partial(cpe_str):
> - if cpe_str in self.all_cpes:
> - return cpe_str
> -
> - def gen_update_xml(self, cpe_str):
> - cpe = self.find_partial_obj(cpe_str)
> - return cpe.update_xml_dict()
> diff --git a/support/scripts/gen-missing-cpe b/support/scripts/gen-missing-cpe
> deleted file mode 100755
> index 0b222f2659..0000000000
> --- a/support/scripts/gen-missing-cpe
> +++ /dev/null
> @@ -1,65 +0,0 @@
> -#!/usr/bin/env python3
> -
> -import argparse
> -import sys
> -import json
> -import subprocess
> -import os
> -from cpedb import CPEDB, CPE
> -
> -
> -def gen_update_xml_reports(cpeids, cpedb, output):
> - cpe_need_update = []
> -
> - for cpe in cpeids:
> - result = cpedb.find(cpe)
> - if not result:
> - result = cpedb.find_partial(CPE.no_version(cpe))
> - if result:
> - cpe_need_update.append(cpe)
> - else:
> - print("WARNING: no match found for '%s'" % cpe)
> -
> - for cpe in cpe_need_update:
> - xml = cpedb.gen_update_xml(cpe)
> - fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
> - print("Generating %s" % fname)
> - with open(os.path.join(output, fname), 'w+') as fp:
> - fp.write(xml)
> -
> - print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpeids)))
> -
> -
> -def get_cpe_ids():
> - print("Getting list of CPE for enabled packages")
> - cmd = ["make", "--no-print-directory", "show-info"]
> - js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
> - return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
> -
> -
> -def resolvepath(path):
> - return os.path.abspath(os.path.expanduser(path))
> -
> -
> -def parse_args():
> - parser = argparse.ArgumentParser()
> - parser.add_argument('--output', dest='output',
> - help='Path to the output CPE update files', type=resolvepath, required=True)
> - parser.add_argument('--nvd-path', dest='nvd_path',
> - help='Path to the local NVD database', type=resolvepath, required=True)
> - return parser.parse_args()
> -
> -
> -def __main__():
> - args = parse_args()
> - if not os.path.isdir(args.output):
> - print("ERROR: output directory %s does not exist" % args.output)
> - sys.exit(1)
> - cpedb = CPEDB(args.nvd_path)
> - cpedb.get_xml_dict()
> - cpeids = get_cpe_ids()
> - gen_update_xml_reports(cpeids, cpedb, args.output)
> -
> -
> -if __name__ == "__main__":
> - __main__()
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index c124b8a9cf..3cb9da6a0b 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -37,10 +37,10 @@ brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
>
> sys.path.append(os.path.join(brpath, "utils"))
> from getdeveloperlib import parse_developers # noqa: E402
> -from cpedb import CPEDB_URL # noqa: E402
>
> INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
> URL_RE = re.compile(r"\s*https?://\S*\s*$")
> +CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
>
> RM_API_STATUS_ERROR = 1
> RM_API_STATUS_FOUND_BY_DISTRO = 2
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 3/8] support/scripts/pkg-stats: check all files for warnings
2023-08-30 20:36 ` Arnout Vandecappelle via buildroot
@ 2023-08-30 21:04 ` Thomas Petazzoni via buildroot
2023-08-31 19:52 ` Daniel Lang
1 sibling, 0 replies; 24+ messages in thread
From: Thomas Petazzoni via buildroot @ 2023-08-30 21:04 UTC (permalink / raw)
To: Arnout Vandecappelle
Cc: Daniel Lang, Sen Hastings, buildroot, Romain Naour, Yann E. MORIN
On Wed, 30 Aug 2023 22:36:10 +0200
Arnout Vandecappelle <arnout@mind.be> wrote:
> Do other maintainers have the same opinion?
Agreed.
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class
2023-08-12 19:28 ` [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class Daniel Lang
@ 2023-08-30 21:37 ` Arnout Vandecappelle via buildroot
2023-08-31 20:18 ` Daniel Lang
2023-09-01 7:10 ` Arnout Vandecappelle via buildroot
1 sibling, 1 reply; 24+ messages in thread
From: Arnout Vandecappelle via buildroot @ 2023-08-30 21:37 UTC (permalink / raw)
To: Daniel Lang, buildroot; +Cc: Thomas Petazzoni
Hi Daniel,
Since this is new code, I'm going to give some coding style comments that
differ a bit from existing code you see in pkg-stats.
I may actually end up applying the patch without those coding style changes,
because the new NVD API is more important than any coding style concerns...
We'll see how far I get.
This is also not a complete review of this patch because it's almost time for
me to go to bed.
On 12/08/2023 21:28, Daniel Lang wrote:
> The current NVD data feeds used for CVE and CPE checking will be retired
> by 2023-12-05 [0]. Both have to be switched to the new v2 API. Since
> fetching data from both sources workes the same, a common base class is
workes -> works
> used to handle the API interaction.
> To store the data locally a sqlite database is used.
Maybe explain a bit more than the new API doesn't allow to download files
anymore, so we need to invent our own file format for persisting it, and a
sqlite db is a convenient format that is always available in python.
>
> [0]: https://nvd.nist.gov/General/News/change-timeline
>
> Signed-off-by: Daniel Lang <dalang@gmx.at>
> ---
> DEVELOPERS | 1 +
> support/scripts/nvd_api_v2.py | 138 ++++++++++++++++++++++++++++++++++
> 2 files changed, 139 insertions(+)
> create mode 100755 support/scripts/nvd_api_v2.py
>
> diff --git a/DEVELOPERS b/DEVELOPERS
> index 6ffa3ee693..81f809a4c0 100644
> --- a/DEVELOPERS
> +++ b/DEVELOPERS
> @@ -668,6 +668,7 @@ F: package/paho-mqtt-cpp/
> F: package/pangomm/
> F: package/pangomm2_46/
> F: package/sam-ba/
> +F: support/scripts/nvd_api_v2.py
>
> N: Damien Lanson <damien@kal-host.com>
> F: package/libvdpau/
> diff --git a/support/scripts/nvd_api_v2.py b/support/scripts/nvd_api_v2.py
> new file mode 100755
> index 0000000000..3fdf32596f
> --- /dev/null
> +++ b/support/scripts/nvd_api_v2.py
> @@ -0,0 +1,138 @@
> +#!/usr/bin/env python3
> +
> +from datetime import datetime, timezone
> +import os
Personally, I think it's better to move away from the os module and use
pathlib.Path instead everywhere a path is used. We don't do that anywhere in
Buildroot at the moment, but I think we should.
> +import requests
> +import shutil
> +import sqlite3
> +import time
> +
> +NVD_API_VERSION = '2.0'
> +NVD_API_BASE_URL = 'https://services.nvd.nist.gov/rest/json'
> +
> +
> +class NVD_API:
> + """
> + A helper class that fetches data from a NVD API and
> + helps manage a sqlite database.
> + """
> + def __init__(self, nvd_path, service, database_file_prefix):
> + """
> + Initialize a new NVD API endpoint with service
> + as the URL postfix.
Could you add documentation for the arguments? I was initially confused by
nvd_path, it sounded like the path part of the URL...
> + """
> + self.nvd_path = nvd_path
> + self.service = service
> + self.url = '%s/%s/%s' % (NVD_API_BASE_URL, service.lower(), NVD_API_VERSION)
Prefer to use format strings:
self.url = f'{NVD_API_BASE_URL}/{service.lower()}/{NVD_API_VERSION}
> + self.db_file = os.path.join(nvd_path, '%s-%s.sqlite' % (database_file_prefix, NVD_API_VERSION))
> + self.db_file_tmp = '%s.tmp' % self.db_file
I haven't looked in too much detail at how the tmp file is used, but I wonder
if it is really needed. sqlite should be able to manage access to the database
so that it stays consistent. If we make sure to use transactions if more than
one row needs to be stored consistently (which I doubt is really needed anyway),
then we should always be able to read consistent data from the database. And if
we re-fetch information from the NVD database, then we should be able to update
the existing row if it actually was already downloaded before.
If I'm missing something here, please add a detailed explanation to the
beginning of this file or class to understand why the tmp file is needed.
> +
> + def init_db(self):
> + """
> + Needs to be implemented by derived classes.
> + Used to make sure that database tables exist.
> + """
> + pass
> +
> + def save_to_db(self, start_index, total_results, content):
> + """
> + Needs to be implemented by derived classes.
> + Used to save the data given by a single API request
> + to the database.
> + """
> + pass
> +
> + def cleanup_db(self):
The name of the function doesn't sound very good to me - it makes me think
you're deleting the db file itself, not the tmp file.
> + """
> + Clean up any files that where left by previously
> + failed runs.
> + """
> + if os.path.exists(self.db_file_tmp):
> + os.remove(self.db_file_tmp)
> +
> + def open_db(self, tmp=False):
> + """
> + Open and return a connection to the sqlite database.
> + """
> + if tmp:
> + return sqlite3.connect(self.db_file_tmp)
> + return sqlite3.connect(self.db_file)
I think putting this in a function is making things more complicated than
calling sqlite3.connect directly from the caller.
> +
> + def download(self, last_update):
> + """
> + Download all entries from NVD since last_update (if not None).
> + For each downloaded page save_to_db is called to together with
> + progress information.
> + NVD rate limiting allows 5 requests per 30 seconds or one every
> + 6 seconds.
Could we maybe define those numbers as class variables and calculate the sleep
etc. times that are used below based on those? Instead of having constant 5 and 6.
> + """
> + args = {}
> + start_index = 0
> + total_results = 0
> + results_per_page = 0
> +
> + print('Downloading new %s' % self.service)
> +
> + if (last_update is not None):
> + args['lastModStartDate'] = last_update.isoformat()
> + args['lastModEndDate'] = datetime.now(tz=timezone.utc).isoformat()
> +
> + while True:
> + args['startIndex'] = start_index
> +
> + for attempt in range(5):
> + try:
> + page = requests.get(self.url, params=args)
> + page.raise_for_status()
> + content = page.json()
> + except Exception:
This will also catch unrecoverable errors, e.g. DNS lookup failures, which
leads to an endless loop without any feedback. Would it be possible to instead
catch the exact exception that requests raises when we hit rate limiting?
> + time.sleep(6)
> + else:
> + break
> +
> + if content is None:
> + # Nothing was downloaded
> + return False
> +
> + results_per_page = content['resultsPerPage']
> + total_results = content['totalResults']
> + start_index = content['startIndex']
> +
> + start_index += results_per_page
> +
> + if self.save_to_db(start_index, total_results, content) is not True:
Instead of "is not True" just do "if not self.save_to_db".
I'm not sure what the start_index and total_results parameters are supposed to
be used for. Are they supposed to be persisted? If yes, then IMHO this should be
done by this class, in a separate table defined by this class.
> + return False
> +
> + self.connection.commit()
> +
> + if start_index >= total_results:
> + return True
> +
> + # Otherwise rate limiting will be hit.
> + time.sleep(6)
If the rate limit really is "5 requests per 30 seconds", then we can actually
do 3 requests back-to-back without any sleep without hitting the rate limit,
right? Assuming you do call the script pretty regularly (like, what we do on
a.b.o), the new results should be just a few pages, so we can avoid the sleep
entirely...
> +
> + def check_for_updates(self):
> + """
> + Check if the database file exists and if the last
> + update was more than 24 hours ago.
> + """
> + self.cleanup_db()
> + last_update = None
> + if os.path.exists(self.db_file):
> + last_update = os.stat(self.db_file).st_mtime
I don't think the mtime is a reliable way to determine the last update time.
Instead, IMHO it's better to add a table in the database that keeps track of the
last update. This would be a table that is added by the base class itself before
calling init_db(). After the download has completed entirely, this timestamp
would be updated. So if it's interrupted in the middle, the next try will
re-download with the previous timestamp. Also, the timestamp should be updated
with the timestamp that was used in the lastModEndDate (which can be quite a bit
earlier than the time at which the download actually finishes). Perhaps even
subtract one second from it to make sure we don't miss any updates that were
entered at the exact time we did the previous download.
> + if last_update >= time.time() - 86400:
> + return []
> + # NVD uses UTC timestamps
> + last_update = datetime.fromtimestamp(last_update, tz=timezone.utc)
> + shutil.copy2(self.db_file, self.db_file_tmp)
> +
> + self.connection = self.open_db(True)
self.connection = None should be added to __init__ to "declare" the attribute.
Also, db_connection would be a better name.
Now it's time for bed :-)
Regards,
Arnout
> + self.init_db()
> +
> + success = self.download(last_update)
> + self.connection.close()
> + if success:
> + shutil.move(self.db_file_tmp, self.db_file)
> + else:
> + print("Update failed!")
> + os.remove(self.db_file_tmp)
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 2/8] support/scripts/pkg-stats: ignore llvm-project.mk
2023-08-30 20:31 ` Arnout Vandecappelle via buildroot
@ 2023-08-31 3:35 ` Daniel Lang
0 siblings, 0 replies; 24+ messages in thread
From: Daniel Lang @ 2023-08-31 3:35 UTC (permalink / raw)
To: Arnout Vandecappelle, buildroot; +Cc: Sen Hastings, Thomas Petazzoni
Hi Arnout,
On 30.08.23 22:31, Arnout Vandecappelle via buildroot wrote:
>
>
> On 12/08/2023 21:28, Daniel Lang wrote:
>> Fixes:
>> f6eaf60 ("package/llvm-project: new group for llvm packages")
>>
>> Signed-off-by: Daniel Lang <dalang@gmx.at>
>> ---
>> support/scripts/pkg-stats | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
>> index eea900124c..1ac538f5f9 100755
>> --- a/support/scripts/pkg-stats
>> +++ b/support/scripts/pkg-stats
>> @@ -345,6 +345,7 @@ def get_pkglist(npackages, package_list):
>> "package/gstreamer/gstreamer.mk",
>> "package/gstreamer1/gstreamer1.mk",
>> "package/gtk2-themes/gtk2-themes.mk",
>> + "package/llvm-project/llvm-project.mk",
>
> Why this specific one, and not the I-don't-know-how-many others that are in a similar situation? E.g. qt6, barebox to take two examples.
I contributed the llvm-project sub-directory some time ago
and always wondered why it showed up in pkg-stats but not
qt5, which was my reference.
Working on this series I took a closer look at the script
and found the list of excluded packages, so I added llvm-project.
I will check all packages that are in a similar situation
for v4.
>
> Regards,
> Arnout
>
>> "package/matchbox/matchbox.mk",
>> "package/opengl/opengl.mk",
>> "package/qt5/qt5.mk",
Regards
Daniel
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 3/8] support/scripts/pkg-stats: check all files for warnings
2023-08-30 20:36 ` Arnout Vandecappelle via buildroot
2023-08-30 21:04 ` Thomas Petazzoni via buildroot
@ 2023-08-31 19:52 ` Daniel Lang
2023-09-01 6:47 ` Arnout Vandecappelle via buildroot
1 sibling, 1 reply; 24+ messages in thread
From: Daniel Lang @ 2023-08-31 19:52 UTC (permalink / raw)
To: Arnout Vandecappelle, buildroot
Cc: Romain Naour, Sen Hastings, Thomas Petazzoni, Yann E. MORIN
Hello Arnout,
On 30.08.23 22:36, Arnout Vandecappelle wrote:
>
>
> On 12/08/2023 21:28, Daniel Lang wrote:
>> Instead of only checking .mk and Config.in{,.host}, check
>> all files in a package directory.
>
> ("checking" here means "run check-package").
Correct.
>
> I think we should instead remove the .warnings and .status['pkg-check'] fields entirely. It made sense back in the days, because we weren't running check-package in CI. Now, however, we do, so there's little point to run check-package as part of pkg-stats as well. On a.b.o the warnings column will (almost) always be all 0.
It will always be 0 if .checkpackageignore is considered which isn't the case
here. Therefore the warning column would show the number of warnings including
those ignored.
The question of whether or not this information is needed is still relevant.
I just wanted to point out that in my testing the value wasn't 0 for all packages.
>
> There is the point of people using pkg-stats for their br2-external, but:
>
> 1. pkg-stats is pretty fragile so users shouldn't be too surprised b some breakage;
> 2. it's very easy for those users to run 'make check-package' instead.
>
>
> Do other maintainers have the same opinion?
Seeing as Thomas already agreed, I will prepare the removal, unless other opinions
are raised.
>
>
> Regards,
> Arnout
Regards,
Daniel
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class
2023-08-30 21:37 ` Arnout Vandecappelle via buildroot
@ 2023-08-31 20:18 ` Daniel Lang
2023-09-01 7:03 ` Arnout Vandecappelle via buildroot
0 siblings, 1 reply; 24+ messages in thread
From: Daniel Lang @ 2023-08-31 20:18 UTC (permalink / raw)
To: Arnout Vandecappelle, buildroot; +Cc: Thomas Petazzoni
Hello Arnout,
thanks for the detailed comments.
I will try to address them and send a new version quickly.
On 30.08.23 23:37, Arnout Vandecappelle wrote:
> Hi Daniel,
>
> Since this is new code, I'm going to give some coding style comments that differ a bit from existing code you see in pkg-stats.
>
> I may actually end up applying the patch without those coding style changes, because the new NVD API is more important than any coding style concerns... We'll see how far I get.
>
> This is also not a complete review of this patch because it's almost time for me to go to bed.
>
[-SNIP-]
>> + """
>> + self.nvd_path = nvd_path
>> + self.service = service
>> + self.url = '%s/%s/%s' % (NVD_API_BASE_URL, service.lower(), NVD_API_VERSION)
>
> Prefer to use format strings:
>
> self.url = f'{NVD_API_BASE_URL}/{service.lower()}/{NVD_API_VERSION}
This means that pkg-stats needs python >= 3.6.
I couldn't find anything about the minimum version that we currently require.
>
>> + self.db_file = os.path.join(nvd_path, '%s-%s.sqlite' % (database_file_prefix, NVD_API_VERSION))
>> + self.db_file_tmp = '%s.tmp' % self.db_file
>
> I haven't looked in too much detail at how the tmp file is used, but I wonder if it is really needed. sqlite should be able to manage access to the database so that it stays consistent. If we make sure to use transactions if more than one row needs to be stored consistently (which I doubt is really needed anyway), then we should always be able to read consistent data from the database. And if we re-fetch information from the NVD database, then we should be able to update the existing row if it actually was already downloaded before.
>
> If I'm missing something here, please add a detailed explanation to the beginning of this file or class to understand why the tmp file is needed.
The idea of the tmp file is that changes (inserts, updates) are only
done on the tmp file. If the update fails mid process we have an
unmodified version.
Inserting directly into the database would update the modification date
which is used to calculate the last update.
Considering your suggestion further down to add a metadata table
that stores the last update date, this becomes less relevant and
the tmp handling could probably be dropped.
>
>> +
>> + def init_db(self):
>> + """
>> + Needs to be implemented by derived classes.
>> + Used to make sure that database tables exist.
>> + """
>> + pass
[-SNIP-]
>> +
>> + def download(self, last_update):
>> + """
>> + Download all entries from NVD since last_update (if not None).
>> + For each downloaded page save_to_db is called to together with
>> + progress information.
>> + NVD rate limiting allows 5 requests per 30 seconds or one every
>> + 6 seconds.
>
> Could we maybe define those numbers as class variables and calculate the sleep etc. times that are used below based on those? Instead of having constant 5 and 6.
Sure, for the code the only relevant number is 6.
But I will add a variable that is defined as 30 / 5.
>
>> + """
>> + args = {}
>> + start_index = 0
>> + total_results = 0
>> + results_per_page = 0
>> +
>> + print('Downloading new %s' % self.service)
>> +
>> + if (last_update is not None):
>> + args['lastModStartDate'] = last_update.isoformat()
>> + args['lastModEndDate'] = datetime.now(tz=timezone.utc).isoformat()
>> +
>> + while True:
>> + args['startIndex'] = start_index
>> +
>> + for attempt in range(5):
>> + try:
>> + page = requests.get(self.url, params=args)
>> + page.raise_for_status()
>> + content = page.json()
>> + except Exception:
>
> This will also catch unrecoverable errors, e.g. DNS lookup failures, which leads to an endless loop without any feedback. Would it be possible to instead catch the exact exception that requests raises when we hit rate limiting?
If I remember correctly I encountered different errors while
downloading the complete set. But I will rerun it and see if
the errors can be narrowed down.
>
>> + time.sleep(6)
>> + else:
>> + break
>> +
>> + if content is None:
>> + # Nothing was downloaded
>> + return False
>> +
>> + results_per_page = content['resultsPerPage']
>> + total_results = content['totalResults']
>> + start_index = content['startIndex']
>> +
>> + start_index += results_per_page
>> +
>> + if self.save_to_db(start_index, total_results, content) is not True:
>
> Instead of "is not True" just do "if not self.save_to_db".
>
> I'm not sure what the start_index and total_results parameters are supposed to be used for. Are they supposed to be persisted? If yes, then IMHO this should be done by this class, in a separate table defined by this class.
start_index and total_results are used by the derived classes
to report the progress like [001000/200000].
I didn't want to do it in this class to be more flexible.
I decided that CVE progress is always reported with 6 digits
and CPE with 7 digits.
Does that make sense?
>
>
>> + return False
>> +
>> + self.connection.commit()
>> +
>> + if start_index >= total_results:
>> + return True
>> +
>> + # Otherwise rate limiting will be hit.
>> + time.sleep(6)
>
> If the rate limit really is "5 requests per 30 seconds", then we can actually do 3 requests back-to-back without any sleep without hitting the rate limit, right? Assuming you do call the script pretty regularly (like, what we do on a.b.o), the new results should be just a few pages, so we can avoid the sleep entirely...
The official wording [0] is "The public rate limit is 5 requests
in a rolling 30 second window; the rate limit with an API key is
50 requests in a rolling 30 seconds window. [...] However,
it is still recommended that your application sleeps for
several seconds between requests so that legitimate requests
are not denied, and all requests are responded to in sequence."
My understanding of this statement is that a sleep should always
be added.
In my experience testing this, timeouts/errors occur even with
the 6 second sleep.
>
>> +
>> + def check_for_updates(self):
>> + """
>> + Check if the database file exists and if the last
>> + update was more than 24 hours ago.
>> + """
>> + self.cleanup_db()
>> + last_update = None
>> + if os.path.exists(self.db_file):
>> + last_update = os.stat(self.db_file).st_mtime
>
> I don't think the mtime is a reliable way to determine the last update time. Instead, IMHO it's better to add a table in the database that keeps track of the last update. This would be a table that is added by the base class itself before calling init_db(). After the download has completed entirely, this timestamp would be updated. So if it's interrupted in the middle, the next try will re-download with the previous timestamp. Also, the timestamp should be updated with the timestamp that was used in the lastModEndDate (which can be quite a bit earlier than the time at which the download actually finishes). Perhaps even subtract one second from it to make sure we don't miss any updates that were entered at the exact time we did the previous download.
Good point.
As addressed above this would also make the tmp file irrelevant.
>
>
>> + if last_update >= time.time() - 86400:
>> + return []
>> + # NVD uses UTC timestamps
>> + last_update = datetime.fromtimestamp(last_update, tz=timezone.utc)
>> + shutil.copy2(self.db_file, self.db_file_tmp)
>> +
>> + self.connection = self.open_db(True)
>
> self.connection = None should be added to __init__ to "declare" the attribute.
>
> Also, db_connection would be a better name.
>
>
> Now it's time for bed :-)
>
> Regards,
> Arnout
>
>> + self.init_db()
>> +
>> + success = self.download(last_update)
>> + self.connection.close()
>> + if success:
>> + shutil.move(self.db_file_tmp, self.db_file)
>> + else:
>> + print("Update failed!")
>> + os.remove(self.db_file_tmp)
Regards,
Daniel
[0]: https://nvd.nist.gov/developers/start-here
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 3/8] support/scripts/pkg-stats: check all files for warnings
2023-08-31 19:52 ` Daniel Lang
@ 2023-09-01 6:47 ` Arnout Vandecappelle via buildroot
0 siblings, 0 replies; 24+ messages in thread
From: Arnout Vandecappelle via buildroot @ 2023-09-01 6:47 UTC (permalink / raw)
To: Daniel Lang, buildroot
Cc: Romain Naour, Sen Hastings, Thomas Petazzoni, Yann E. MORIN
On 31/08/2023 21:52, Daniel Lang wrote:
> Hello Arnout,
>
> On 30.08.23 22:36, Arnout Vandecappelle wrote:
>>
>>
>> On 12/08/2023 21:28, Daniel Lang wrote:
>>> Instead of only checking .mk and Config.in{,.host}, check
>>> all files in a package directory.
>>
>> ("checking" here means "run check-package").
>
> Correct.
>
>>
>> I think we should instead remove the .warnings and .status['pkg-check'] fields entirely. It made sense back in the days, because we weren't running check-package in CI. Now, however, we do, so there's little point to run check-package as part of pkg-stats as well. On a.b.o the warnings column will (almost) always be all 0.
>
> It will always be 0 if .checkpackageignore is considered which isn't the case
> here. Therefore the warning column would show the number of warnings including
> those ignored.
Oh, very good point, I didn't think of that...
On the short term: the reason we introduced .checkpackageignore to begin with
is that there are too many errors, so many that we're never going to fix them
all. In particular old patches for semi-dead projects. But also the 154
shellcheck errors are probably never going to be fixed.
That said, I do see the value in seeing the number of warnings for a package.
Right now, if you're looking at the pkg-stats output and decide to e.g.
version-bump a package, you'll probably not consider to also fix the shellcheck
warnings in the init script. However, if you also see that there's one warning,
that may trigger you to update the init script so you're package becomes fully
green.
In other words, I'm retracting my earlier suggestion of removing the Warnings
entirely.
Regards,
Arnout
> The question of whether or not this information is needed is still relevant.
> I just wanted to point out that in my testing the value wasn't 0 for all packages.
>
>>
>> There is the point of people using pkg-stats for their br2-external, but:
>>
>> 1. pkg-stats is pretty fragile so users shouldn't be too surprised b some breakage;
>> 2. it's very easy for those users to run 'make check-package' instead.
>>
>>
>> Do other maintainers have the same opinion?
>
> Seeing as Thomas already agreed, I will prepare the removal, unless other opinions
> are raised.
>
>>
>>
>> Regards,
>> Arnout
>
> Regards,
> Daniel
>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class
2023-08-31 20:18 ` Daniel Lang
@ 2023-09-01 7:03 ` Arnout Vandecappelle via buildroot
2023-09-01 8:10 ` Thomas Petazzoni via buildroot
2023-09-01 11:08 ` Daniel Lang
0 siblings, 2 replies; 24+ messages in thread
From: Arnout Vandecappelle via buildroot @ 2023-09-01 7:03 UTC (permalink / raw)
To: Daniel Lang, buildroot; +Cc: Thomas Petazzoni
On 31/08/2023 22:18, Daniel Lang wrote:
> Hello Arnout,
>
> thanks for the detailed comments.
> I will try to address them and send a new version quickly.
>
> On 30.08.23 23:37, Arnout Vandecappelle wrote:
>> Hi Daniel,
>>
>> Since this is new code, I'm going to give some coding style comments that differ a bit from existing code you see in pkg-stats.
>>
>> I may actually end up applying the patch without those coding style changes, because the new NVD API is more important than any coding style concerns... We'll see how far I get.
>>
>> This is also not a complete review of this patch because it's almost time for me to go to bed.
>>
>
> [-SNIP-]
>
>>> + """
>>> + self.nvd_path = nvd_path
>>> + self.service = service
>>> + self.url = '%s/%s/%s' % (NVD_API_BASE_URL, service.lower(), NVD_API_VERSION)
>>
>> Prefer to use format strings:
>>
>> self.url = f'{NVD_API_BASE_URL}/{service.lower()}/{NVD_API_VERSION}
>
> This means that pkg-stats needs python >= 3.6.
I think on a.b.o is running the script inside the buildroot/base container,
which has 3.9.2. Thomas?
Anyway, pkg-stats already uses f-strings in many places.
> I couldn't find anything about the minimum version that we currently require.
[snip]
>>> + results_per_page = content['resultsPerPage']
>>> + total_results = content['totalResults']
>>> + start_index = content['startIndex']
>>> +
>>> + start_index += results_per_page
>>> +
>>> + if self.save_to_db(start_index, total_results, content) is not True:
>>
>> Instead of "is not True" just do "if not self.save_to_db".
>>
>> I'm not sure what the start_index and total_results parameters are supposed to be used for. Are they supposed to be persisted? If yes, then IMHO this should be done by this class, in a separate table defined by this class.
>
> start_index and total_results are used by the derived classes
> to report the progress like [001000/200000].
> I didn't want to do it in this class to be more flexible.
Er, sounds like this would be the same in all derived classes though? But I
haven't looked at the rest yet so I could be wrong.
> I decided that CVE progress is always reported with 6 digits
> and CPE with 7 digits.
Ah, that's the flexibility you mean.
I would count the number of digits in total_results and use that. You can do
nested interpolation in an f-string, like:
f'[{start_index:0{len(str(total_results))}}/{total_results}]'
> Does that make sense?
>
>>
>>
>>> + return False
>>> +
>>> + self.connection.commit()
>>> +
>>> + if start_index >= total_results:
>>> + return True
>>> +
>>> + # Otherwise rate limiting will be hit.
>>> + time.sleep(6)
>>
>> If the rate limit really is "5 requests per 30 seconds", then we can actually do 3 requests back-to-back without any sleep without hitting the rate limit, right? Assuming you do call the script pretty regularly (like, what we do on a.b.o), the new results should be just a few pages, so we can avoid the sleep entirely...
In your experience, is it true that we only need to download a few pages if
you update every day? Or maybe it's even less than one page - in that case it's
not really relevant.
> The official wording [0] is "The public rate limit is 5 requests
> in a rolling 30 second window; the rate limit with an API key is
> 50 requests in a rolling 30 seconds window. [...] However,
> it is still recommended that your application sleeps for
> several seconds between requests so that legitimate requests
> are not denied, and all requests are responded to in sequence."
>
> My understanding of this statement is that a sleep should always
> be added.
> In my experience testing this, timeouts/errors occur even with
> the 6 second sleep.
My concern is just that a pkg-stats run which could be instantaneous (because
we have _mostly_ cached results) will now take 6 seconds more than needed.
But we really shouldn't make the code more complicated to cater to that
specific issue, so you can probably ignore my comment.
Regards,
Arnout
>>> +
>>> + def check_for_updates(self):
>>> + """
>>> + Check if the database file exists and if the last
>>> + update was more than 24 hours ago.
>>> + """
>>> + self.cleanup_db()
>>> + last_update = None
>>> + if os.path.exists(self.db_file):
>>> + last_update = os.stat(self.db_file).st_mtime
>>
>> I don't think the mtime is a reliable way to determine the last update time. Instead, IMHO it's better to add a table in the database that keeps track of the last update. This would be a table that is added by the base class itself before calling init_db(). After the download has completed entirely, this timestamp would be updated. So if it's interrupted in the middle, the next try will re-download with the previous timestamp. Also, the timestamp should be updated with the timestamp that was used in the lastModEndDate (which can be quite a bit earlier than the time at which the download actually finishes). Perhaps even subtract one second from it to make sure we don't miss any updates that were entered at the exact time we did the previous download.
>
> Good point.
> As addressed above this would also make the tmp file irrelevant.
>
>>
>>
>>> + if last_update >= time.time() - 86400:
>>> + return []
>>> + # NVD uses UTC timestamps
>>> + last_update = datetime.fromtimestamp(last_update, tz=timezone.utc)
>>> + shutil.copy2(self.db_file, self.db_file_tmp)
>>> +
>>> + self.connection = self.open_db(True)
>>
>> self.connection = None should be added to __init__ to "declare" the attribute.
>>
>> Also, db_connection would be a better name.
>>
>>
>> Now it's time for bed :-)
>>
>> Regards,
>> Arnout
>>
>>> + self.init_db()
>>> +
>>> + success = self.download(last_update)
>>> + self.connection.close()
>>> + if success:
>>> + shutil.move(self.db_file_tmp, self.db_file)
>>> + else:
>>> + print("Update failed!")
>>> + os.remove(self.db_file_tmp)
>
> Regards,
> Daniel
>
> [0]: https://nvd.nist.gov/developers/start-here
>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class
2023-08-12 19:28 ` [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class Daniel Lang
2023-08-30 21:37 ` Arnout Vandecappelle via buildroot
@ 2023-09-01 7:10 ` Arnout Vandecappelle via buildroot
1 sibling, 0 replies; 24+ messages in thread
From: Arnout Vandecappelle via buildroot @ 2023-09-01 7:10 UTC (permalink / raw)
To: Daniel Lang, buildroot; +Cc: Thomas Petazzoni
On 12/08/2023 21:28, Daniel Lang wrote:
[snip]
> + def check_for_updates(self):
> + """
> + Check if the database file exists and if the last
> + update was more than 24 hours ago.
> + """
> + self.cleanup_db()
> + last_update = None
> + if os.path.exists(self.db_file):
> + last_update = os.stat(self.db_file).st_mtime
> + if last_update >= time.time() - 86400:
> + return []
Actually, is this check really needed?
If it is, it really should be something smaller than 86400. If the script runs
daily, we really do want do update the database every day. If we use 86400, then
it will sometimes not update, because it runs just one second earlier than the
day before.
But I don't think it's needed. AFAIU, the check_for_updates() would be called
only once for a run of pkg-stats (well, once for CVE and once for CPE). So it's
only if you run pkg-stats several times in a row that it's needed to avoid
re-fetching. You normally don't run pkg-stats several times in a row. But even
so, it will just hit the rate limit, wait for 6 seconds, and all is well again.
Maybe your experiments show that that's not good either, e.g. because the rate
limit will hit harder and harder if you run pkg-stats all the time. In that
case, a time check is useful, but I'd drastically reduce it, to e.g. 2 hours.
Regards,
Arnout
> + # NVD uses UTC timestamps
> + last_update = datetime.fromtimestamp(last_update, tz=timezone.utc)
> + shutil.copy2(self.db_file, self.db_file_tmp)
> +
> + self.connection = self.open_db(True)
> + self.init_db()
> +
> + success = self.download(last_update)
> + self.connection.close()
> + if success:
> + shutil.move(self.db_file_tmp, self.db_file)
> + else:
> + print("Update failed!")
> + os.remove(self.db_file_tmp)
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class
2023-09-01 7:03 ` Arnout Vandecappelle via buildroot
@ 2023-09-01 8:10 ` Thomas Petazzoni via buildroot
2023-09-01 11:08 ` Daniel Lang
1 sibling, 0 replies; 24+ messages in thread
From: Thomas Petazzoni via buildroot @ 2023-09-01 8:10 UTC (permalink / raw)
To: Arnout Vandecappelle; +Cc: Daniel Lang, buildroot
On Fri, 1 Sep 2023 09:03:45 +0200
Arnout Vandecappelle <arnout@mind.be> wrote:
> I think on a.b.o is running the script inside the buildroot/base container,
> which has 3.9.2. Thomas?
No, the script is not run in the buildroot/base container. The machine
on which I run the script has Python 3.8.10.
Best regards,
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class
2023-09-01 7:03 ` Arnout Vandecappelle via buildroot
2023-09-01 8:10 ` Thomas Petazzoni via buildroot
@ 2023-09-01 11:08 ` Daniel Lang
1 sibling, 0 replies; 24+ messages in thread
From: Daniel Lang @ 2023-09-01 11:08 UTC (permalink / raw)
To: Arnout Vandecappelle, buildroot; +Cc: Thomas Petazzoni
On 01.09.23 09:03, Arnout Vandecappelle wrote:
>
>
> On 31/08/2023 22:18, Daniel Lang wrote:
>> Hello Arnout,
>>
>> thanks for the detailed comments.
>> I will try to address them and send a new version quickly.
>>
>> On 30.08.23 23:37, Arnout Vandecappelle wrote:
>>> Hi Daniel,
>>>
>>> Since this is new code, I'm going to give some coding style comments that differ a bit from existing code you see in pkg-stats.
>>>
>>> I may actually end up applying the patch without those coding style changes, because the new NVD API is more important than any coding style concerns... We'll see how far I get.
>>>
>>> This is also not a complete review of this patch because it's almost time for me to go to bed.
>>>
>>
[-SNIP-]
>>>> + return False
>>>> +
>>>> + self.connection.commit()
>>>> +
>>>> + if start_index >= total_results:
>>>> + return True
>>>> +
>>>> + # Otherwise rate limiting will be hit.
>>>> + time.sleep(6)
>>>
>>> If the rate limit really is "5 requests per 30 seconds", then we can actually do 3 requests back-to-back without any sleep without hitting the rate limit, right? Assuming you do call the script pretty regularly (like, what we do on a.b.o), the new results should be just a few pages, so we can avoid the sleep entirely...
>
> In your experience, is it true that we only need to download a few pages if you update every day? Or maybe it's even less than one page - in that case it's not really relevant.
214 CVEs where updated in the last 24h.
978 CVEs where updated in the last week.
Not sure those are representative numbers, but autobuilders
should never have to fetch more than 1-2 pages.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos
2023-08-12 19:28 [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Daniel Lang
` (7 preceding siblings ...)
2023-08-30 20:45 ` [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Arnout Vandecappelle via buildroot
@ 2023-09-14 8:25 ` Peter Korsgaard
8 siblings, 0 replies; 24+ messages in thread
From: Peter Korsgaard @ 2023-09-14 8:25 UTC (permalink / raw)
To: Daniel Lang; +Cc: Sen Hastings, Thomas Petazzoni, buildroot
>>>>> "Daniel" == Daniel Lang <dalang@gmx.at> writes:
> Signed-off-by: Daniel Lang <dalang@gmx.at>
Committed to 2023.02.x and 2023.05.x, thanks.
--
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos
2023-08-30 20:45 ` [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Arnout Vandecappelle via buildroot
@ 2023-09-14 8:26 ` Peter Korsgaard
0 siblings, 0 replies; 24+ messages in thread
From: Peter Korsgaard @ 2023-09-14 8:26 UTC (permalink / raw)
To: Arnout Vandecappelle via buildroot
Cc: Daniel Lang, Sen Hastings, Thomas Petazzoni
>>>>> "Arnout" == Arnout Vandecappelle via buildroot <buildroot@buildroot.org> writes:
> On 12/08/2023 21:28, Daniel Lang wrote:
>> Signed-off-by: Daniel Lang <dalang@gmx.at>
> Applied to master, thanks.
> (Note: I'm applying this series to master instead of next, because we
> will need the new NVD API on all stable branches.)
Notice: The EOL has recently been moved to December:
https://nvd.nist.gov/General/News/change-timeline
Making it quite close to 2024.02, so perhaps we don't need to worry
about backporting this after all?
--
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2023-09-14 8:27 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-08-12 19:28 [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Daniel Lang
2023-08-12 19:28 ` [Buildroot] [PATCH v3 2/8] support/scripts/pkg-stats: ignore llvm-project.mk Daniel Lang
2023-08-30 20:31 ` Arnout Vandecappelle via buildroot
2023-08-31 3:35 ` Daniel Lang
2023-08-12 19:28 ` [Buildroot] [PATCH v3 3/8] support/scripts/pkg-stats: check all files for warnings Daniel Lang
2023-08-30 20:36 ` Arnout Vandecappelle via buildroot
2023-08-30 21:04 ` Thomas Petazzoni via buildroot
2023-08-31 19:52 ` Daniel Lang
2023-09-01 6:47 ` Arnout Vandecappelle via buildroot
2023-08-12 19:28 ` [Buildroot] [PATCH v3 4/8] support/scripts/gen-missing-cpe: remove rarely used script Daniel Lang
2023-08-30 20:46 ` Arnout Vandecappelle via buildroot
2023-08-12 19:28 ` [Buildroot] [PATCH v3 5/8] support/scripts/nvd_api_v2.py: new helper class Daniel Lang
2023-08-30 21:37 ` Arnout Vandecappelle via buildroot
2023-08-31 20:18 ` Daniel Lang
2023-09-01 7:03 ` Arnout Vandecappelle via buildroot
2023-09-01 8:10 ` Thomas Petazzoni via buildroot
2023-09-01 11:08 ` Daniel Lang
2023-09-01 7:10 ` Arnout Vandecappelle via buildroot
2023-08-12 19:28 ` [Buildroot] [PATCH v3 6/8] support/scripts/cve.py: switch to NVD API v2 Daniel Lang
2023-08-12 19:28 ` [Buildroot] [PATCH v3 7/8] support/scripts/pkg-stats: switch CPEs " Daniel Lang
2023-08-12 19:28 ` [Buildroot] [PATCH v3 8/8] support/scripts/pkg-stats: Only match CPE vendor and product Daniel Lang
2023-08-30 20:45 ` [Buildroot] [PATCH v3 1/8] support/scripts/pkg-stats: fix typos Arnout Vandecappelle via buildroot
2023-09-14 8:26 ` Peter Korsgaard
2023-09-14 8:25 ` Peter Korsgaard
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).