* [Buildroot] [PATCH 0/9] Improving CVE reporting
@ 2020-07-08 16:39 Gregory CLEMENT
2020-07-08 16:39 ` [Buildroot] [PATCH 1/9] support/scripts: Turn CVE check into a module Gregory CLEMENT
` (8 more replies)
0 siblings, 9 replies; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-08 16:39 UTC (permalink / raw)
To: buildroot
Hello,
The purpose of this series is to improve the CVE reporting in order to
be usable for a project.
Until know the CVE affecting the packages were reported for the
buildroot project using pkg-stat. With this series it is now possible
to report the packages affected by CVEs for a given configuration.
While I was on CVE I switched to the support of the JSON 1.1 for the
NVDE database.
In this series I also added a new state for the CVE status of the
packages. This new state will be used to emphasize that the automatic
check has failed and it was needed to be verified manually. The idea
behind this was to be as much accurate as possible to avoid any false
positive. It will also help to improve the meta-data of the package.
The next step will be to reuse the works done by Matthew Weber [1] to
use the cpeid and only use the package name and the package version as
fall back.
In this series there is at least one open point about the packages
excluded from the cve check. For now I excluded the kernel and gcc as
there are also excluded by the pkg-stats script but this list could
(should ?) be extended or modified.
Gregory
[1]: http://patchwork.ozlabs.org/project/buildroot/list/?series=183798
Gregory CLEMENT (9):
support/scripts: Turn CVE check into a module
support/scripts/cve.py: Switch to JSON 1.1
package/pkg-utils: show-info: report the list of the CVEs ignored
package/pkg-utils: Make CVE class independent of the Pacakage class
support/scripts: Add a per configuration CVE checker
package/pkg-utils: cve.py: Handle exception when version comparison
fails
support/script/pkg-stats: Manage the CVEs that need to be check
support/script/cve-checker: Manage the CVEs that need to be check
package/pkg-utils/cve.py: Manage case when package version doesn't
exist
package/pkg-utils.mk | 5 +-
support/scripts/cve-checker | 308 ++++++++++++++++++++++++++++++++++++
support/scripts/cve.py | 229 +++++++++++++++++++++++++++
support/scripts/pkg-stats | 161 +++++--------------
4 files changed, 581 insertions(+), 122 deletions(-)
create mode 100755 support/scripts/cve-checker
create mode 100755 support/scripts/cve.py
--
2.27.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 1/9] support/scripts: Turn CVE check into a module
2020-07-08 16:39 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
@ 2020-07-08 16:39 ` Gregory CLEMENT
2020-07-08 16:54 ` Thomas Petazzoni
2020-07-08 16:39 ` [Buildroot] [PATCH 2/9] support/scripts/cve.py: Switch to JSON 1.1 Gregory CLEMENT
` (7 subsequent siblings)
8 siblings, 1 reply; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-08 16:39 UTC (permalink / raw)
To: buildroot
In order to be able to do CVE checking outside of pkg-stat, move the
CVE class in a module that can be used by other scripts.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
support/scripts/cve.py | 141 ++++++++++++++++++++++++++++++++++++++
support/scripts/pkg-stats | 115 +------------------------------
2 files changed, 144 insertions(+), 112 deletions(-)
create mode 100755 support/scripts/cve.py
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
new file mode 100755
index 0000000000..874ab4482d
--- /dev/null
+++ b/support/scripts/cve.py
@@ -0,0 +1,141 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+# Copyright (C) 2020 by Gregory CLEMENT <gregory.clement@bootlin.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import datetime
+import os
+import re
+import requests # URL checking
+import json
+import ijson
+import distutils.version
+import time
+import gzip
+import sys
+
+sys.path.append('utils/')
+
+NVD_START_YEAR = 2002
+NVD_JSON_VERSION = "1.0"
+NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
+
+class CVE:
+ """An accessor class for CVE Items in NVD files"""
+ def __init__(self, nvd_cve):
+ """Initialize a CVE from its NVD JSON representation"""
+ self.nvd_cve = nvd_cve
+
+ @staticmethod
+ def download_nvd_year(nvd_path, year):
+ metaf = "nvdcve-%s-%s.meta" % (NVD_JSON_VERSION, year)
+ path_metaf = os.path.join(nvd_path, metaf)
+ jsonf_gz = "nvdcve-%s-%s.json.gz" % (NVD_JSON_VERSION, year)
+ path_jsonf_gz = os.path.join(nvd_path, jsonf_gz)
+
+ # If the database file is less than a day old, we assume the NVD data
+ # locally available is recent enough.
+ if os.path.exists(path_jsonf_gz) and os.stat(path_jsonf_gz).st_mtime >= time.time() - 86400:
+ return path_jsonf_gz
+
+ # If not, we download the meta file
+ url = "%s/%s" % (NVD_BASE_URL, metaf)
+ print("Getting %s" % url)
+ page_meta = requests.get(url)
+ page_meta.raise_for_status()
+
+ # If the meta file already existed, we compare the existing
+ # one with the data newly downloaded. If they are different,
+ # we need to re-download the database.
+ # If the database does not exist locally, we need to redownload it in
+ # any case.
+ if os.path.exists(path_metaf) and os.path.exists(path_jsonf_gz):
+ meta_known = open(path_metaf, "r").read()
+ if page_meta.text == meta_known:
+ return path_jsonf_gz
+
+ # Grab the compressed JSON NVD, and write files to disk
+ url = "%s/%s" % (NVD_BASE_URL, jsonf_gz)
+ print("Getting %s" % url)
+ page_json = requests.get(url)
+ page_json.raise_for_status()
+ open(path_jsonf_gz, "wb").write(page_json.content)
+ open(path_metaf, "w").write(page_meta.text)
+ return path_jsonf_gz
+
+ @classmethod
+ def read_nvd_dir(cls, nvd_dir):
+ """
+ Iterate over all the CVEs contained in NIST Vulnerability Database
+ feeds since NVD_START_YEAR. If the files are missing or outdated in
+ nvd_dir, a fresh copy will be downloaded, and kept in .json.gz
+ """
+ for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1):
+ filename = CVE.download_nvd_year(nvd_dir, year)
+ try:
+ content = ijson.items(gzip.GzipFile(filename), 'CVE_Items.item')
+ except: # noqa: E722
+ print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
+ raise
+ for cve in content:
+ yield cls(cve['cve'])
+
+ def each_product(self):
+ """Iterate over each product section of this cve"""
+ for vendor in self.nvd_cve['affects']['vendor']['vendor_data']:
+ for product in vendor['product']['product_data']:
+ yield product
+
+ @property
+ def identifier(self):
+ """The CVE unique identifier"""
+ return self.nvd_cve['CVE_data_meta']['ID']
+
+ @property
+ def pkg_names(self):
+ """The set of package names referred by this CVE definition"""
+ return set(p['product_name'] for p in self.each_product())
+
+ def affects(self, br_pkg):
+ """
+ True if the Buildroot Package object passed as argument is affected
+ by this CVE.
+ """
+ if br_pkg.is_cve_ignored(self.identifier):
+ return False
+
+ for product in self.each_product():
+ if product['product_name'] != br_pkg.name:
+ continue
+
+ for v in product['version']['version_data']:
+ if v["version_affected"] == "=":
+ if br_pkg.current_version == v["version_value"]:
+ return True
+ elif v["version_affected"] == "<=":
+ pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
+ if not hasattr(pkg_version, "version"):
+ print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
+ continue
+ cve_affected_version = distutils.version.LooseVersion(v["version_value"])
+ if not hasattr(cve_affected_version, "version"):
+ print("Cannot parse CVE affected version '%s'" % v["version_value"])
+ continue
+ return pkg_version <= cve_affected_version
+ else:
+ print("version_affected: %s" % v['version_affected'])
+ return False
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index c1f41fc9e8..1c941104fe 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -38,9 +38,8 @@ from multiprocessing import Pool
sys.path.append('utils/')
from getdeveloperlib import parse_developers # noqa: E402
-NVD_START_YEAR = 2002
-NVD_JSON_VERSION = "1.0"
-NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
+import cve as cvecheck
+
INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
URL_RE = re.compile(r"\s*https?://\S*\s*$")
@@ -281,114 +280,6 @@ class Package:
(self.name, self.path, self.is_status_ok('license'),
self.is_status_ok('license-files'), self.status['hash'], self.patch_count)
-
-class CVE:
- """An accessor class for CVE Items in NVD files"""
- def __init__(self, nvd_cve):
- """Initialize a CVE from its NVD JSON representation"""
- self.nvd_cve = nvd_cve
-
- @staticmethod
- def download_nvd_year(nvd_path, year):
- metaf = "nvdcve-%s-%s.meta" % (NVD_JSON_VERSION, year)
- path_metaf = os.path.join(nvd_path, metaf)
- jsonf_gz = "nvdcve-%s-%s.json.gz" % (NVD_JSON_VERSION, year)
- path_jsonf_gz = os.path.join(nvd_path, jsonf_gz)
-
- # If the database file is less than a day old, we assume the NVD data
- # locally available is recent enough.
- if os.path.exists(path_jsonf_gz) and os.stat(path_jsonf_gz).st_mtime >= time.time() - 86400:
- return path_jsonf_gz
-
- # If not, we download the meta file
- url = "%s/%s" % (NVD_BASE_URL, metaf)
- print("Getting %s" % url)
- page_meta = requests.get(url)
- page_meta.raise_for_status()
-
- # If the meta file already existed, we compare the existing
- # one with the data newly downloaded. If they are different,
- # we need to re-download the database.
- # If the database does not exist locally, we need to redownload it in
- # any case.
- if os.path.exists(path_metaf) and os.path.exists(path_jsonf_gz):
- meta_known = open(path_metaf, "r").read()
- if page_meta.text == meta_known:
- return path_jsonf_gz
-
- # Grab the compressed JSON NVD, and write files to disk
- url = "%s/%s" % (NVD_BASE_URL, jsonf_gz)
- print("Getting %s" % url)
- page_json = requests.get(url)
- page_json.raise_for_status()
- open(path_jsonf_gz, "wb").write(page_json.content)
- open(path_metaf, "w").write(page_meta.text)
- return path_jsonf_gz
-
- @classmethod
- def read_nvd_dir(cls, nvd_dir):
- """
- Iterate over all the CVEs contained in NIST Vulnerability Database
- feeds since NVD_START_YEAR. If the files are missing or outdated in
- nvd_dir, a fresh copy will be downloaded, and kept in .json.gz
- """
- for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1):
- filename = CVE.download_nvd_year(nvd_dir, year)
- try:
- content = ijson.items(gzip.GzipFile(filename), 'CVE_Items.item')
- except: # noqa: E722
- print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
- raise
- for cve in content:
- yield cls(cve['cve'])
-
- def each_product(self):
- """Iterate over each product section of this cve"""
- for vendor in self.nvd_cve['affects']['vendor']['vendor_data']:
- for product in vendor['product']['product_data']:
- yield product
-
- @property
- def identifier(self):
- """The CVE unique identifier"""
- return self.nvd_cve['CVE_data_meta']['ID']
-
- @property
- def pkg_names(self):
- """The set of package names referred by this CVE definition"""
- return set(p['product_name'] for p in self.each_product())
-
- def affects(self, br_pkg):
- """
- True if the Buildroot Package object passed as argument is affected
- by this CVE.
- """
- if br_pkg.is_cve_ignored(self.identifier):
- return False
-
- for product in self.each_product():
- if product['product_name'] != br_pkg.name:
- continue
-
- for v in product['version']['version_data']:
- if v["version_affected"] == "=":
- if br_pkg.current_version == v["version_value"]:
- return True
- elif v["version_affected"] == "<=":
- pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
- if not hasattr(pkg_version, "version"):
- print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
- continue
- cve_affected_version = distutils.version.LooseVersion(v["version_value"])
- if not hasattr(cve_affected_version, "version"):
- print("Cannot parse CVE affected version '%s'" % v["version_value"])
- continue
- return pkg_version <= cve_affected_version
- else:
- print("version_affected: %s" % v['version_affected'])
- return False
-
-
def get_pkglist(npackages, package_list):
"""
Builds the list of Buildroot packages, returning a list of Package
@@ -608,7 +499,7 @@ def check_package_cves(nvd_path, packages):
if not os.path.isdir(nvd_path):
os.makedirs(nvd_path)
- for cve in CVE.read_nvd_dir(nvd_path):
+ for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
for pkg_name in cve.pkg_names:
if pkg_name in packages and cve.affects(packages[pkg_name]):
packages[pkg_name].cves.append(cve.identifier)
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 2/9] support/scripts/cve.py: Switch to JSON 1.1
2020-07-08 16:39 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
2020-07-08 16:39 ` [Buildroot] [PATCH 1/9] support/scripts: Turn CVE check into a module Gregory CLEMENT
@ 2020-07-08 16:39 ` Gregory CLEMENT
2020-07-08 16:40 ` [Buildroot] [PATCH 3/9] package/pkg-utils: show-info: report the list of the CVEs ignored Gregory CLEMENT
` (6 subsequent siblings)
8 siblings, 0 replies; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-08 16:39 UTC (permalink / raw)
To: buildroot
In 2019, the JSON vulnerability feeds switched from version 1.0 to
1.1.
The main difference is the removal of the affects element that was
used to check if a package was affected by a CVE.
This information is duplicated in the configuration element which
contains in the end the cpeid as well as properties about the versions
affected. Instead of having a list of the versions affected, with
these properties, it is possible to have a range of versions.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
support/scripts/cve.py | 119 +++++++++++++++++++++++++++++++++--------
1 file changed, 96 insertions(+), 23 deletions(-)
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index 874ab4482d..e911fe0c65 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -31,9 +31,19 @@ import sys
sys.path.append('utils/')
NVD_START_YEAR = 2002
-NVD_JSON_VERSION = "1.0"
+NVD_JSON_VERSION = "1.1"
NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
+import operator
+
+ops = {
+ '>=' : operator.ge,
+ '>' : operator.gt,
+ '<=' : operator.le,
+ '<' : operator.lt,
+ '=' : operator.eq
+}
+
class CVE:
"""An accessor class for CVE Items in NVD files"""
def __init__(self, nvd_cve):
@@ -92,23 +102,83 @@ class CVE:
print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
raise
for cve in content:
- yield cls(cve['cve'])
+ yield cls(cve)
def each_product(self):
"""Iterate over each product section of this cve"""
- for vendor in self.nvd_cve['affects']['vendor']['vendor_data']:
+ for vendor in self.nvd_cve['cve']['affects']['vendor']['vendor_data']:
for product in vendor['product']['product_data']:
yield product
+ def parse_node(self, node):
+ """
+ Parse the node inside the configurations section to extract the
+ cpe information usefull to know if a product is affected by
+ the CVE. Actually only the product name and the version
+ descriptor are needed, but we also provide the vendor name.
+ """
+
+ # The node containing the cpe entries matching the CVE can also
+ # contain sub-nodes, so we need to manage it.
+ for child in node.get('children', ()):
+ self.parse_node(child)
+
+ for cpe in node.get('cpe_match', ()):
+ if not cpe['vulnerable']:
+ return
+ cpe23 = cpe['cpe23Uri'].split(':')
+ vendor = cpe23[3]
+ product = cpe23[4]
+ version = cpe23[5]
+ op_start = ''
+ op_end = ''
+ v_start = ''
+ v_end = ''
+
+ if version != '*' and version != '-':
+ # Version is defined, this is a '=' match
+ op_start = '='
+ v_start = version
+ elif version == '-':
+ # no version information is available
+ op_start = '='
+ v_start = version
+ else:
+ # Parse start version, end version and operators
+ if 'versionStartIncluding' in cpe:
+ op_start = '>='
+ v_start = cpe['versionStartIncluding']
+
+ if 'versionStartExcluding' in cpe:
+ op_start = '>'
+ v_start = cpe['versionStartExcluding']
+
+ if 'versionEndIncluding' in cpe:
+ op_end = '<='
+ v_end = cpe['versionEndIncluding']
+
+ if 'versionEndExcluding' in cpe:
+ op_end = '<'
+ v_end = cpe['versionEndExcluding']
+
+ key =['vendor', 'product', 'v_start', 'op_start', 'v_end', 'op_end']
+ val = [vendor, product, v_start, op_start, v_end, op_end]
+ yield dict(zip(key, val))
+
+ def each_cpe(self):
+ for node in self.nvd_cve['configurations']['nodes']:
+ for cpe in self.parse_node(node):
+ yield cpe
+
@property
def identifier(self):
"""The CVE unique identifier"""
- return self.nvd_cve['CVE_data_meta']['ID']
+ return self.nvd_cve['cve']['CVE_data_meta']['ID']
@property
def pkg_names(self):
"""The set of package names referred by this CVE definition"""
- return set(p['product_name'] for p in self.each_product())
+ return set(p['product'] for p in self.each_cpe())
def affects(self, br_pkg):
"""
@@ -118,24 +188,27 @@ class CVE:
if br_pkg.is_cve_ignored(self.identifier):
return False
- for product in self.each_product():
- if product['product_name'] != br_pkg.name:
+ for cpe in self.each_cpe():
+ affected = True
+ if cpe['product'] != br_pkg.name:
+ continue
+ if cpe['v_start'] == '-':
+ return True
+ if not (cpe['v_start'] or cpe['v_end']):
+ print("No CVE affected version")
continue
+ pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
+ if not hasattr(pkg_version, "version"):
+ print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
+ continue
+
+ if cpe['v_start']:
+ cve_affected_version = distutils.version.LooseVersion(cpe['v_start'])
+ affected = ops.get(cpe['op_start'])(pkg_version, cve_affected_version)
- for v in product['version']['version_data']:
- if v["version_affected"] == "=":
- if br_pkg.current_version == v["version_value"]:
- return True
- elif v["version_affected"] == "<=":
- pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
- if not hasattr(pkg_version, "version"):
- print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
- continue
- cve_affected_version = distutils.version.LooseVersion(v["version_value"])
- if not hasattr(cve_affected_version, "version"):
- print("Cannot parse CVE affected version '%s'" % v["version_value"])
- continue
- return pkg_version <= cve_affected_version
- else:
- print("version_affected: %s" % v['version_affected'])
+ if (affected and cpe['v_end']):
+ cve_affected_version = distutils.version.LooseVersion(cpe['v_end'])
+ affected = ops.get(cpe['op_end'])(pkg_version, cve_affected_version)
+ if (affected):
+ return True
return False
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 3/9] package/pkg-utils: show-info: report the list of the CVEs ignored
2020-07-08 16:39 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
2020-07-08 16:39 ` [Buildroot] [PATCH 1/9] support/scripts: Turn CVE check into a module Gregory CLEMENT
2020-07-08 16:39 ` [Buildroot] [PATCH 2/9] support/scripts/cve.py: Switch to JSON 1.1 Gregory CLEMENT
@ 2020-07-08 16:40 ` Gregory CLEMENT
2020-07-08 16:53 ` Thomas Petazzoni
2020-07-08 16:40 ` [Buildroot] [PATCH 4/9] package/pkg-utils: Make CVE class independent of the Pacakage class Gregory CLEMENT
` (5 subsequent siblings)
8 siblings, 1 reply; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-08 16:40 UTC (permalink / raw)
To: buildroot
Add the list of the CVEs to ignore for each package because they
already have a fix for it.
This information will be useful for a cve-checker.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
package/pkg-utils.mk | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index d88a14ab0f..49ce6dc6f1 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -117,7 +117,10 @@ define _json-info-pkg
$(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
],
"reverse_dependencies": [
- $(call make-comma-list,$(sort $($(1)_RDEPENDENCIES)))
+ $(call make-comma-list,$(sort $($(1)_RDEPENDENCIES))),
+ ],
+ "ignored_cves": [
+ $(call make-comma-list,$(sort $($(1)_IGNORE_CVES)))
]
endef
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 4/9] package/pkg-utils: Make CVE class independent of the Pacakage class
2020-07-08 16:39 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
` (2 preceding siblings ...)
2020-07-08 16:40 ` [Buildroot] [PATCH 3/9] package/pkg-utils: show-info: report the list of the CVEs ignored Gregory CLEMENT
@ 2020-07-08 16:40 ` Gregory CLEMENT
2020-07-08 16:40 ` [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker Gregory CLEMENT
` (4 subsequent siblings)
8 siblings, 0 replies; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-08 16:40 UTC (permalink / raw)
To: buildroot
The affects method of the CVE use the Package class defined in
pkg-stats. The purpose of migrating the CVE class outside of pkg-stats
was to be able to reuse it from other scripts. So let's remove the
Package dependency and only use the needed information.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
support/scripts/cve.py | 10 +++++-----
support/scripts/pkg-stats | 14 ++++++++------
2 files changed, 13 insertions(+), 11 deletions(-)
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index e911fe0c65..b754a17991 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -180,26 +180,26 @@ class CVE:
"""The set of package names referred by this CVE definition"""
return set(p['product'] for p in self.each_cpe())
- def affects(self, br_pkg):
+ def affects(self, name, version, cve_ignore_list):
"""
True if the Buildroot Package object passed as argument is affected
by this CVE.
"""
- if br_pkg.is_cve_ignored(self.identifier):
+ if (self.identifier in cve_ignore_list):
return False
for cpe in self.each_cpe():
affected = True
- if cpe['product'] != br_pkg.name:
+ if cpe['product'] != name:
continue
if cpe['v_start'] == '-':
return True
if not (cpe['v_start'] or cpe['v_end']):
print("No CVE affected version")
continue
- pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
+ pkg_version = distutils.version.LooseVersion(version)
if not hasattr(pkg_version, "version"):
- print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
+ print("Cannot parse package '%s' version '%s'" % (name, version))
continue
if cpe['v_start']:
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 1c941104fe..883a5bd2be 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -245,11 +245,12 @@ class Package:
self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings))
return
- def is_cve_ignored(self, cve):
+ def cve_ignored_list(self):
"""
- Tells if the CVE is ignored by the package
+ Give the list of CVEs ignored by the package
"""
- return cve in self.all_ignored_cves.get(self.pkgvar(), [])
+ print(self.all_ignored_cves.get(self.pkgvar(), []))
+ return list(self.all_ignored_cves.get(self.pkgvar(), []))
def set_developers(self, developers):
"""
@@ -501,9 +502,10 @@ def check_package_cves(nvd_path, packages):
for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
for pkg_name in cve.pkg_names:
- if pkg_name in packages and cve.affects(packages[pkg_name]):
- packages[pkg_name].cves.append(cve.identifier)
-
+ if pkg_name in packages:
+ pkg = packages[pkg_name]
+ if cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list()):
+ pkg.cves.append(cve.identifier)
def calculate_stats(packages):
stats = defaultdict(int)
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker
2020-07-08 16:39 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
` (3 preceding siblings ...)
2020-07-08 16:40 ` [Buildroot] [PATCH 4/9] package/pkg-utils: Make CVE class independent of the Pacakage class Gregory CLEMENT
@ 2020-07-08 16:40 ` Gregory CLEMENT
2020-07-08 18:30 ` Matthew Weber
2020-07-09 11:46 ` Matthew Weber
2020-07-08 16:40 ` [Buildroot] [PATCH 6/9] package/pkg-utils: cve.py: Handle exception when version comparison fails Gregory CLEMENT
` (3 subsequent siblings)
8 siblings, 2 replies; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-08 16:40 UTC (permalink / raw)
To: buildroot
This scripts takes as entry on stdin a JSON description of the package
used for a given configuration. This description is the one generated
by "make show-info".
The script generates the list of all the package used and if they are
affected by a CVE. The output is either a JSON or an HTML file similar
to the one generated by pkg-stats.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
support/scripts/cve-checker | 291 ++++++++++++++++++++++++++++++++++++
1 file changed, 291 insertions(+)
create mode 100755 support/scripts/cve-checker
diff --git a/support/scripts/cve-checker b/support/scripts/cve-checker
new file mode 100755
index 0000000000..db8497d7aa
--- /dev/null
+++ b/support/scripts/cve-checker
@@ -0,0 +1,291 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+# Copyright (C) 2020 by Gregory CLEMENT <gregory.clement@bootlin.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import argparse
+import datetime
+import fnmatch
+import os
+from collections import defaultdict
+import re
+import subprocess
+import requests # URL checking
+import json
+import ijson
+import certifi
+import distutils.version
+import time
+import gzip
+import sys
+from urllib3 import HTTPSConnectionPool
+from urllib3.exceptions import HTTPError
+from multiprocessing import Pool
+
+sys.path.append('utils/')
+
+import cve as cvecheck
+
+
+INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
+URL_RE = re.compile(r"\s*https?://\S*\s*$")
+
+RM_API_STATUS_ERROR = 1
+RM_API_STATUS_FOUND_BY_DISTRO = 2
+RM_API_STATUS_FOUND_BY_PATTERN = 3
+RM_API_STATUS_NOT_FOUND = 4
+
+# Used to make multiple requests to the same host. It is global
+# because it's used by sub-processes.
+http_pool = None
+
+
+class Package:
+ def __init__(self, name, version, ignored_cves):
+ self.name = name
+ self.version = version
+ self.cves = list()
+ self.ignored_cves = ignored_cves
+
+def check_package_cves(nvd_path, packages):
+ if not os.path.isdir(nvd_path):
+ os.makedirs(nvd_path)
+
+ for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
+ for pkg_name in cve.pkg_names:
+ pkg = packages.get(pkg_name, '')
+ if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves):
+ pkg.cves.append(cve.identifier)
+
+html_header = """
+<head>
+<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
+<style type=\"text/css\">
+table {
+ width: 100%;
+}
+td {
+ border: 1px solid black;
+}
+td.centered {
+ text-align: center;
+}
+td.wrong {
+ background: #ff9a69;
+}
+td.correct {
+ background: #d2ffc4;
+}
+td.nopatches {
+ background: #d2ffc4;
+}
+td.somepatches {
+ background: #ffd870;
+}
+td.lotsofpatches {
+ background: #ff9a69;
+}
+
+td.good_url {
+ background: #d2ffc4;
+}
+td.missing_url {
+ background: #ffd870;
+}
+td.invalid_url {
+ background: #ff9a69;
+}
+
+td.version-good {
+ background: #d2ffc4;
+}
+td.version-needs-update {
+ background: #ff9a69;
+}
+td.version-unknown {
+ background: #ffd870;
+}
+td.version-error {
+ background: #ccc;
+}
+
+</style>
+<title>CVE status for Buildroot packages</title>
+</head>
+
+<a href=\"#results\">CVE Status</a><br/>
+
+<p id=\"sortable_hint\"></p>
+"""
+
+
+html_footer = """
+</body>
+<script>
+if (typeof sorttable === \"object\") {
+ document.getElementById(\"sortable_hint\").innerHTML =
+ \"hint: the table can be sorted by clicking the column headers\"
+}
+</script>
+</html>
+"""
+
+
+def infra_str(infra_list):
+ if not infra_list:
+ return "Unknown"
+ elif len(infra_list) == 1:
+ return "<b>%s</b><br/>%s" % (infra_list[0][1], infra_list[0][0])
+ elif infra_list[0][1] == infra_list[1][1]:
+ return "<b>%s</b><br/>%s + %s" % \
+ (infra_list[0][1], infra_list[0][0], infra_list[1][0])
+ else:
+ return "<b>%s</b> (%s)<br/><b>%s</b> (%s)" % \
+ (infra_list[0][1], infra_list[0][0],
+ infra_list[1][1], infra_list[1][0])
+
+
+def boolean_str(b):
+ if b:
+ return "Yes"
+ else:
+ return "No"
+
+
+def dump_html_pkg(f, pkg):
+ f.write(" <tr>\n")
+ f.write(" <td>%s</td>\n" % pkg.name)
+
+ # Current version
+ if len(pkg.version) > 20:
+ version = pkg.version[:20] + "..."
+ else:
+ version = pkg.version
+ f.write(" <td class=\"centered\">%s</td>\n" % version)
+
+ # CVEs
+ td_class = ["centered"]
+ if len(pkg.cves) == 0:
+ td_class.append("correct")
+ else:
+ td_class.append("wrong")
+ f.write(" <td class=\"%s\">\n" % " ".join(td_class))
+ for cve in pkg.cves:
+ f.write(" <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+ f.write(" </td>\n")
+
+ f.write(" </tr>\n")
+
+
+def dump_html_all_pkgs(f, packages):
+ f.write("""
+<table class=\"sortable\">
+<tr>
+<td>Package</td>
+<td class=\"centered\">Version</td>
+<td class=\"centered\">CVEs</td>
+</tr>
+""")
+ for pkg in packages:
+ dump_html_pkg(f, pkg)
+ f.write("</table>")
+
+
+def dump_html_gen_info(f, date, commit):
+ # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032
+ f.write("<p><i>Updated on %s, git commit %s</i></p>\n" % (str(date), commit))
+
+
+def dump_html(packages, date, commit, output):
+ with open(output, 'w') as f:
+ f.write(html_header)
+ dump_html_all_pkgs(f, packages)
+ dump_html_gen_info(f, date, commit)
+ f.write(html_footer)
+
+
+def dump_json(packages, date, commit, output):
+ # Format packages as a dictionnary instead of a list
+ # Exclude local field that does not contains real date
+ excluded_fields = ['url_worker', 'name']
+ pkgs = {
+ pkg.name: {
+ k: v
+ for k, v in pkg.__dict__.items()
+ if k not in excluded_fields
+ } for pkg in packages
+ }
+ # The actual structure to dump, add commit and date to it
+ final = {'packages': pkgs,
+ 'commit': commit,
+ 'date': str(date)}
+ with open(output, 'w') as f:
+ json.dump(final, f, indent=2, separators=(',', ': '))
+ f.write('\n')
+
+
+def resolvepath(path):
+ return os.path.abspath(os.path.expanduser(path))
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ output = parser.add_argument_group('output', 'Output file(s)')
+ output.add_argument('--html', dest='html', type=resolvepath,
+ help='HTML output file')
+ output.add_argument('--json', dest='json', type=resolvepath,
+ help='JSON output file')
+ packages = parser.add_mutually_exclusive_group()
+ packages.add_argument('-n', dest='npackages', type=int, action='store',
+ help='Number of packages')
+ packages.add_argument('-p', dest='packages', action='store',
+ help='List of packages (comma separated)')
+ parser.add_argument('--nvd-path', dest='nvd_path',
+ help='Path to the local NVD database', type=resolvepath)
+ args = parser.parse_args()
+ if not args.html and not args.json:
+ parser.error('at least one of --html or --json (or both) is required')
+ return args
+
+
+def __main__():
+ packages = list()
+ exclude_pacakges = ["linux", "gcc"]
+ content = json.load(sys.stdin)
+ for item in content:
+ if item in exclude_pacakges:
+ continue
+ pkg = content[item]
+ p = Package(item, pkg.get('version', ''), pkg.get('ignored_cves', ''))
+ packages.append(p)
+
+ args = parse_args()
+ date = datetime.datetime.utcnow()
+ commit = subprocess.check_output(['git', 'rev-parse',
+ 'HEAD']).splitlines()[0].decode()
+
+ if args.nvd_path:
+ print("Checking packages CVEs")
+ check_package_cves(args.nvd_path, {p.name: p for p in packages})
+ if args.html:
+ print("Write HTML")
+ dump_html(packages, date, commit, args.html)
+ if args.json:
+ print("Write JSON")
+ dump_json(packages, date, commit, args.json)
+
+__main__()
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 6/9] package/pkg-utils: cve.py: Handle exception when version comparison fails
2020-07-08 16:39 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
` (4 preceding siblings ...)
2020-07-08 16:40 ` [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker Gregory CLEMENT
@ 2020-07-08 16:40 ` Gregory CLEMENT
2020-07-09 8:52 ` Thomas Petazzoni
2020-07-08 16:40 ` [Buildroot] [PATCH 7/9] support/script/pkg-stats: Manage the CVEs that need to be check Gregory CLEMENT
` (2 subsequent siblings)
8 siblings, 1 reply; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-08 16:40 UTC (permalink / raw)
To: buildroot
With python 3, when a package has a version number x-y-z instead of
x.y.z, then the version returned by LooseVersion can't be compared
which raises an exception.
This patch handles this exception by adding a new return value when
the comparison can't be done.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
support/scripts/cve.py | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index b754a17991..03afdeb54a 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -203,12 +203,21 @@ class CVE:
continue
if cpe['v_start']:
+ try:
cve_affected_version = distutils.version.LooseVersion(cpe['v_start'])
affected = ops.get(cpe['op_start'])(pkg_version, cve_affected_version)
+ break
+ except:
+ return 'Unknown'
if (affected and cpe['v_end']):
+ try:
cve_affected_version = distutils.version.LooseVersion(cpe['v_end'])
affected = ops.get(cpe['op_end'])(pkg_version, cve_affected_version)
+ break
+ except:
+ return 'Unknown'
+
if (affected):
return True
return False
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 7/9] support/script/pkg-stats: Manage the CVEs that need to be check
2020-07-08 16:39 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
` (5 preceding siblings ...)
2020-07-08 16:40 ` [Buildroot] [PATCH 6/9] package/pkg-utils: cve.py: Handle exception when version comparison fails Gregory CLEMENT
@ 2020-07-08 16:40 ` Gregory CLEMENT
2020-07-09 9:00 ` Thomas Petazzoni
2020-07-08 16:40 ` [Buildroot] [PATCH 8/9] support/script/cve-checker: " Gregory CLEMENT
2020-07-08 16:40 ` [Buildroot] [PATCH 9/9] package/pkg-utils/cve.py: Manage case when package version doesn't exist Gregory CLEMENT
8 siblings, 1 reply; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-08 16:40 UTC (permalink / raw)
To: buildroot
When looking for if a package is affected, the version comparison can
fail. This means that we don't know if the version of the package used
is affected or not and we need to check manually the version.
This patch exposes this new information in json and html format.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
support/scripts/pkg-stats | 34 ++++++++++++++++++++++++++++++----
1 file changed, 30 insertions(+), 4 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 883a5bd2be..e033e15e07 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -106,9 +106,11 @@ class Package:
self.patch_files = []
self.warnings = 0
self.current_version = None
+ self.unknown_cve = False
self.url = None
self.url_worker = None
self.cves = list()
+ self.cves_to_check = list()
self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None}
self.status = {}
@@ -504,7 +506,12 @@ def check_package_cves(nvd_path, packages):
for pkg_name in cve.pkg_names:
if pkg_name in packages:
pkg = packages[pkg_name]
- if cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list()):
+ affected = cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list())
+ print(affected)
+ if (affected == 'Unknown'):
+ pkg.cves_to_check.append(cve.identifier)
+ elif affected == True:
+ print(cve.identifier)
pkg.cves.append(cve.identifier)
def calculate_stats(packages):
@@ -544,8 +551,11 @@ def calculate_stats(packages):
stats["version-not-uptodate"] += 1
stats["patches"] += pkg.patch_count
stats["total-cves"] += len(pkg.cves)
+ stats["total-cves-to-check"] += len(pkg.cves_to_check)
if len(pkg.cves) != 0:
stats["pkg-cves"] += 1
+ if len(pkg.cves_to_check) != 0:
+ stats["pkg-cves_to_check"] += 1
return stats
@@ -763,11 +773,22 @@ def dump_html_pkg(f, pkg):
td_class.append("correct")
else:
td_class.append("wrong")
- f.write(" <td class=\"%s\">\n" % " ".join(td_class))
+ f.write(" <td class=\"%s\">\n" % " ".join(td_class))
for cve in pkg.cves:
f.write(" <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
f.write(" </td>\n")
+ # CVEs to check
+ td_class = ["centered"]
+ if len(pkg.cves_to_check) == 0:
+ td_class.append("correct")
+ else:
+ td_class.append("wrong")
+ f.write(" <td class=\"%s\">\n" % " ".join(td_class))
+ for cve in pkg.cves_to_check:
+ f.write(" <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+ f.write(" </td>\n")
+
f.write(" </tr>\n")
@@ -786,6 +807,7 @@ def dump_html_all_pkgs(f, packages):
<td class=\"centered\">Warnings</td>
<td class=\"centered\">Upstream URL</td>
<td class=\"centered\">CVEs</td>
+<td class=\"centered\">CVEs to check</td>
</tr>
""")
for pkg in sorted(packages):
@@ -824,10 +846,14 @@ def dump_html_stats(f, stats):
stats["version-not-uptodate"])
f.write("<tr><td>Packages with no known upstream version</td><td>%s</td></tr>\n" %
stats["version-unknown"])
- f.write("<tr><td>Packages affected by CVEs</td><td>%s</td></tr>\n" %
+ f.write("<tr><td>Packages might affected by CVEs, where version needed to be checked</td><td>%s</td></tr>\n" %
stats["pkg-cves"])
- f.write("<tr><td>Total number of CVEs affecting all packages</td><td>%s</td></tr>\n" %
+ f.write("<tr><td>Total number of CVEs that might affect all packages, where version needed to be checked</td><td>%s</td></tr>\n" %
stats["total-cves"])
+ f.write("<tr><td>Packages affected by CVEs</td><td>%s</td></tr>\n" %
+ stats["pkg-cves_to_check"])
+ f.write("<tr><td>Total number of CVEs affecting all packages</td><td>%s</td></tr>\n" %
+ stats["total-cves_to_check"])
f.write("</table>\n")
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 8/9] support/script/cve-checker: Manage the CVEs that need to be check
2020-07-08 16:39 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
` (6 preceding siblings ...)
2020-07-08 16:40 ` [Buildroot] [PATCH 7/9] support/script/pkg-stats: Manage the CVEs that need to be check Gregory CLEMENT
@ 2020-07-08 16:40 ` Gregory CLEMENT
2020-07-08 16:40 ` [Buildroot] [PATCH 9/9] package/pkg-utils/cve.py: Manage case when package version doesn't exist Gregory CLEMENT
8 siblings, 0 replies; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-08 16:40 UTC (permalink / raw)
To: buildroot
When looking for if a package is affected, the version comparison can
fail. This means that we don't know if the version of the package used
is affected or not and we need to check manually the version.
This patch exposes this new information in json and html format.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
support/scripts/cve-checker | 21 +++++++++++++++++++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/support/scripts/cve-checker b/support/scripts/cve-checker
index db8497d7aa..2a8a2b4d18 100755
--- a/support/scripts/cve-checker
+++ b/support/scripts/cve-checker
@@ -59,6 +59,7 @@ class Package:
self.name = name
self.version = version
self.cves = list()
+ self.cves_to_check = list()
self.ignored_cves = ignored_cves
def check_package_cves(nvd_path, packages):
@@ -68,8 +69,12 @@ def check_package_cves(nvd_path, packages):
for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
for pkg_name in cve.pkg_names:
pkg = packages.get(pkg_name, '')
- if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves):
- pkg.cves.append(cve.identifier)
+ if pkg:
+ affected = cve.affects(pkg.name, pkg.version, pkg.ignored_cves)
+ if (affected == 'Unknown'):
+ pkg.cves_to_check.append(cve.identifier)
+ elif affected == True:
+ pkg.cves.append(cve.identifier)
html_header = """
<head>
@@ -188,6 +193,17 @@ def dump_html_pkg(f, pkg):
f.write(" <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
f.write(" </td>\n")
+ # CVEs to check
+ td_class = ["centered"]
+ if len(pkg.cves_to_check) == 0:
+ td_class.append("correct")
+ else:
+ td_class.append("wrong")
+ f.write(" <td class=\"%s\">\n" % " ".join(td_class))
+ for cve in pkg.cves_to_check:
+ f.write(" <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+ f.write(" </td>\n")
+
f.write(" </tr>\n")
@@ -198,6 +214,7 @@ def dump_html_all_pkgs(f, packages):
<td>Package</td>
<td class=\"centered\">Version</td>
<td class=\"centered\">CVEs</td>
+<td class=\"centered\">CVEs to check</td>
</tr>
""")
for pkg in packages:
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 9/9] package/pkg-utils/cve.py: Manage case when package version doesn't exist
2020-07-08 16:39 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
` (7 preceding siblings ...)
2020-07-08 16:40 ` [Buildroot] [PATCH 8/9] support/script/cve-checker: " Gregory CLEMENT
@ 2020-07-08 16:40 ` Gregory CLEMENT
8 siblings, 0 replies; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-08 16:40 UTC (permalink / raw)
To: buildroot
Until now, when a package didn't report a version, then the CVE
comparison was just skipped. It leads most of the time to declare the
package not affected by the CVE.
Instead of it, report the 'Unknown' status in order to be aware that
the CVE related to this package has to be checked.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
support/scripts/cve.py | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index 03afdeb54a..d3480d68dd 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -188,6 +188,7 @@ class CVE:
if (self.identifier in cve_ignore_list):
return False
+ unknown_pkg_version = False
for cpe in self.each_cpe():
affected = True
if cpe['product'] != name:
@@ -200,6 +201,7 @@ class CVE:
pkg_version = distutils.version.LooseVersion(version)
if not hasattr(pkg_version, "version"):
print("Cannot parse package '%s' version '%s'" % (name, version))
+ unknown_pkg_version = True
continue
if cpe['v_start']:
@@ -220,4 +222,8 @@ class CVE:
if (affected):
return True
- return False
+
+ if unknown_pkg_version:
+ return 'Unknown'
+ else:
+ return False
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 3/9] package/pkg-utils: show-info: report the list of the CVEs ignored
2020-07-08 16:40 ` [Buildroot] [PATCH 3/9] package/pkg-utils: show-info: report the list of the CVEs ignored Gregory CLEMENT
@ 2020-07-08 16:53 ` Thomas Petazzoni
0 siblings, 0 replies; 20+ messages in thread
From: Thomas Petazzoni @ 2020-07-08 16:53 UTC (permalink / raw)
To: buildroot
On Wed, 8 Jul 2020 18:40:00 +0200
Gregory CLEMENT <gregory.clement@bootlin.com> wrote:
> Add the list of the CVEs to ignore for each package because they
> already have a fix for it.
>
> This information will be useful for a cve-checker.
>
> Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
> ---
> package/pkg-utils.mk | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index d88a14ab0f..49ce6dc6f1 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -117,7 +117,10 @@ define _json-info-pkg
> $(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
> ],
> "reverse_dependencies": [
> - $(call make-comma-list,$(sort $($(1)_RDEPENDENCIES)))
> + $(call make-comma-list,$(sort $($(1)_RDEPENDENCIES))),
> + ],
> + "ignored_cves": [
> + $(call make-comma-list,$(sort $($(1)_IGNORE_CVES)))
While I understand the idea of labelling the property "ignored_cves", I
think it makes more sense to have a 1:1 mapping with the package
variable, i.e "ignore_cves".
This is a nitpick: do not resend the full series just for that, we can
fix it when applying.
Thanks!
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 1/9] support/scripts: Turn CVE check into a module
2020-07-08 16:39 ` [Buildroot] [PATCH 1/9] support/scripts: Turn CVE check into a module Gregory CLEMENT
@ 2020-07-08 16:54 ` Thomas Petazzoni
2020-07-09 7:34 ` Gregory CLEMENT
0 siblings, 1 reply; 20+ messages in thread
From: Thomas Petazzoni @ 2020-07-08 16:54 UTC (permalink / raw)
To: buildroot
On Wed, 8 Jul 2020 18:39:58 +0200
Gregory CLEMENT <gregory.clement@bootlin.com> wrote:
> In order to be able to do CVE checking outside of pkg-stat, move the
> CVE class in a module that can be used by other scripts.
>
> Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
> ---
> support/scripts/cve.py | 141 ++++++++++++++++++++++++++++++++++++++
> support/scripts/pkg-stats | 115 +------------------------------
> 2 files changed, 144 insertions(+), 112 deletions(-)
> create mode 100755 support/scripts/cve.py
>
> diff --git a/support/scripts/cve.py b/support/scripts/cve.py
> new file mode 100755
> index 0000000000..874ab4482d
> --- /dev/null
> +++ b/support/scripts/cve.py
> @@ -0,0 +1,141 @@
> +#!/usr/bin/env python
> +
> +# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> +# Copyright (C) 2020 by Gregory CLEMENT <gregory.clement@bootlin.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
> +
> +import datetime
> +import os
> +import re
> +import requests # URL checking
> +import json
> +import ijson
> +import distutils.version
> +import time
> +import gzip
> +import sys
Are you able to drop some of these imports from the pkg-stats script ?
For example, isn't ijson only used by cve.py now ? Same for gzip, for
distutils.version. Perhaps others ?
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker
2020-07-08 16:40 ` [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker Gregory CLEMENT
@ 2020-07-08 18:30 ` Matthew Weber
2020-07-09 8:41 ` Gregory CLEMENT
2020-07-09 11:46 ` Matthew Weber
1 sibling, 1 reply; 20+ messages in thread
From: Matthew Weber @ 2020-07-08 18:30 UTC (permalink / raw)
To: buildroot
Gregory,
On Wed, Jul 8, 2020 at 11:41 AM Gregory CLEMENT
<gregory.clement@bootlin.com> wrote:
>
> This scripts takes as entry on stdin a JSON description of the package
> used for a given configuration. This description is the one generated
> by "make show-info".
>
> The script generates the list of all the package used and if they are
> affected by a CVE. The output is either a JSON or an HTML file similar
> to the one generated by pkg-stats.
>
> Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
> ---
> support/scripts/cve-checker | 291 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 291 insertions(+)
> create mode 100755 support/scripts/cve-checker
>
> diff --git a/support/scripts/cve-checker b/support/scripts/cve-checker
> new file mode 100755
> index 0000000000..db8497d7aa
> --- /dev/null
> +++ b/support/scripts/cve-checker
> @@ -0,0 +1,291 @@
> +#!/usr/bin/env python
> +
> +# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> +# Copyright (C) 2020 by Gregory CLEMENT <gregory.clement@bootlin.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
> +
> +import argparse
> +import datetime
> +import fnmatch
> +import os
> +from collections import defaultdict
> +import re
> +import subprocess
> +import requests # URL checking
> +import json
> +import ijson
> +import certifi
> +import distutils.version
> +import time
> +import gzip
> +import sys
> +from urllib3 import HTTPSConnectionPool
> +from urllib3.exceptions import HTTPError
> +from multiprocessing import Pool
> +
> +sys.path.append('utils/')
> +
> +import cve as cvecheck
> +
> +
> +INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
> +URL_RE = re.compile(r"\s*https?://\S*\s*$")
> +
> +RM_API_STATUS_ERROR = 1
> +RM_API_STATUS_FOUND_BY_DISTRO = 2
> +RM_API_STATUS_FOUND_BY_PATTERN = 3
> +RM_API_STATUS_NOT_FOUND = 4
> +
> +# Used to make multiple requests to the same host. It is global
> +# because it's used by sub-processes.
> +http_pool = None
> +
> +
> +class Package:
> + def __init__(self, name, version, ignored_cves):
> + self.name = name
> + self.version = version
> + self.cves = list()
> + self.ignored_cves = ignored_cves
> +
> +def check_package_cves(nvd_path, packages):
> + if not os.path.isdir(nvd_path):
> + os.makedirs(nvd_path)
> +
> + for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
This read_nvd_dir call that does the dictionary download has a whole
bunch of parsing package output "Cannot parse package 'openssl'
version ''". I assume some of that output will get cleaned up when we
add the additional CPE matching
> + for pkg_name in cve.pkg_names:
> + pkg = packages.get(pkg_name, '')
> + if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves):
> + pkg.cves.append(cve.identifier)
> +
> +html_header = """
> +<head>
> +<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
> +<style type=\"text/css\">
> +table {
> + width: 100%;
> +}
> +td {
> + border: 1px solid black;
> +}
> +td.centered {
> + text-align: center;
> +}
> +td.wrong {
> + background: #ff9a69;
> +}
> +td.correct {
> + background: #d2ffc4;
> +}
> +td.nopatches {
> + background: #d2ffc4;
> +}
> +td.somepatches {
> + background: #ffd870;
> +}
> +td.lotsofpatches {
> + background: #ff9a69;
> +}
> +
> +td.good_url {
> + background: #d2ffc4;
> +}
> +td.missing_url {
> + background: #ffd870;
> +}
> +td.invalid_url {
> + background: #ff9a69;
> +}
> +
> +td.version-good {
> + background: #d2ffc4;
> +}
> +td.version-needs-update {
> + background: #ff9a69;
> +}
> +td.version-unknown {
> + background: #ffd870;
> +}
> +td.version-error {
> + background: #ccc;
> +}
> +
> +</style>
> +<title>CVE status for Buildroot packages</title>
> +</head>
> +
> +<a href=\"#results\">CVE Status</a><br/>
> +
> +<p id=\"sortable_hint\"></p>
> +"""
> +
> +
> +html_footer = """
> +</body>
> +<script>
> +if (typeof sorttable === \"object\") {
> + document.getElementById(\"sortable_hint\").innerHTML =
> + \"hint: the table can be sorted by clicking the column headers\"
> +}
> +</script>
> +</html>
> +"""
> +
> +
> +def infra_str(infra_list):
> + if not infra_list:
> + return "Unknown"
> + elif len(infra_list) == 1:
> + return "<b>%s</b><br/>%s" % (infra_list[0][1], infra_list[0][0])
> + elif infra_list[0][1] == infra_list[1][1]:
> + return "<b>%s</b><br/>%s + %s" % \
> + (infra_list[0][1], infra_list[0][0], infra_list[1][0])
> + else:
> + return "<b>%s</b> (%s)<br/><b>%s</b> (%s)" % \
> + (infra_list[0][1], infra_list[0][0],
> + infra_list[1][1], infra_list[1][0])
> +
> +
> +def boolean_str(b):
> + if b:
> + return "Yes"
> + else:
> + return "No"
> +
> +
> +def dump_html_pkg(f, pkg):
> + f.write(" <tr>\n")
> + f.write(" <td>%s</td>\n" % pkg.name)
> +
> + # Current version
> + if len(pkg.version) > 20:
> + version = pkg.version[:20] + "..."
> + else:
> + version = pkg.version
> + f.write(" <td class=\"centered\">%s</td>\n" % version)
> +
> + # CVEs
> + td_class = ["centered"]
> + if len(pkg.cves) == 0:
> + td_class.append("correct")
> + else:
> + td_class.append("wrong")
> + f.write(" <td class=\"%s\">\n" % " ".join(td_class))
> + for cve in pkg.cves:
> + f.write(" <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
> + f.write(" </td>\n")
> +
> + f.write(" </tr>\n")
> +
> +
> +def dump_html_all_pkgs(f, packages):
> + f.write("""
> +<table class=\"sortable\">
> +<tr>
> +<td>Package</td>
> +<td class=\"centered\">Version</td>
> +<td class=\"centered\">CVEs</td>
> +</tr>
> +""")
> + for pkg in packages:
> + dump_html_pkg(f, pkg)
> + f.write("</table>")
> +
> +
> +def dump_html_gen_info(f, date, commit):
> + # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032
> + f.write("<p><i>Updated on %s, git commit %s</i></p>\n" % (str(date), commit))
> +
> +
> +def dump_html(packages, date, commit, output):
> + with open(output, 'w') as f:
> + f.write(html_header)
> + dump_html_all_pkgs(f, packages)
> + dump_html_gen_info(f, date, commit)
> + f.write(html_footer)
> +
> +
> +def dump_json(packages, date, commit, output):
> + # Format packages as a dictionnary instead of a list
> + # Exclude local field that does not contains real date
> + excluded_fields = ['url_worker', 'name']
> + pkgs = {
> + pkg.name: {
> + k: v
> + for k, v in pkg.__dict__.items()
> + if k not in excluded_fields
> + } for pkg in packages
> + }
> + # The actual structure to dump, add commit and date to it
> + final = {'packages': pkgs,
> + 'commit': commit,
> + 'date': str(date)}
> + with open(output, 'w') as f:
> + json.dump(final, f, indent=2, separators=(',', ': '))
> + f.write('\n')
> +
> +
> +def resolvepath(path):
> + return os.path.abspath(os.path.expanduser(path))
> +
> +
> +def parse_args():
> + parser = argparse.ArgumentParser()
> + output = parser.add_argument_group('output', 'Output file(s)')
> + output.add_argument('--html', dest='html', type=resolvepath,
> + help='HTML output file')
> + output.add_argument('--json', dest='json', type=resolvepath,
> + help='JSON output file')
> + packages = parser.add_mutually_exclusive_group()
> + packages.add_argument('-n', dest='npackages', type=int, action='store',
> + help='Number of packages')
> + packages.add_argument('-p', dest='packages', action='store',
> + help='List of packages (comma separated)')
Are the -n and -p options left over from pulling this tool out of
pkg-stats? Since this report is based on a specific defconfig,
> + parser.add_argument('--nvd-path', dest='nvd_path',
> + help='Path to the local NVD database', type=resolvepath)
I noticed this was a required item, maybe default to a folder name in
the current folder when one isn't provided?
> + args = parser.parse_args()
> + if not args.html and not args.json:
> + parser.error('at least one of --html or --json (or both) is required')
plus nvd path unless you add a default
> + return args
> +
> +
> +def __main__():
> + packages = list()
> + exclude_pacakges = ["linux", "gcc"]
> + content = json.load(sys.stdin)
> + for item in content:
> + if item in exclude_pacakges:
> + continue
> + pkg = content[item]
> + p = Package(item, pkg.get('version', ''), pkg.get('ignored_cves', ''))
> + packages.append(p)
> +
> + args = parse_args()
> + date = datetime.datetime.utcnow()
> + commit = subprocess.check_output(['git', 'rev-parse',
> + 'HEAD']).splitlines()[0].decode()
This git commit check doesn't work when the tools are used with out of
tree buildroot builds.
To reproduce from within Buildroot clone:
make O=../foobar qemu_x86_64_defconfig
cd ../foobar
make show-info | support/scripts/cve-checker --html report.html --nvd-path nvd
> +
> + if args.nvd_path:
> + print("Checking packages CVEs")
> + check_package_cves(args.nvd_path, {p.name: p for p in packages})
> + if args.html:
> + print("Write HTML")
> + dump_html(packages, date, commit, args.html)
> + if args.json:
> + print("Write JSON")
> + dump_json(packages, date, commit, args.json)
> +
> +__main__()
> --
> 2.27.0
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 1/9] support/scripts: Turn CVE check into a module
2020-07-08 16:54 ` Thomas Petazzoni
@ 2020-07-09 7:34 ` Gregory CLEMENT
0 siblings, 0 replies; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-09 7:34 UTC (permalink / raw)
To: buildroot
Hello Thomas,
> On Wed, 8 Jul 2020 18:39:58 +0200
> Gregory CLEMENT <gregory.clement@bootlin.com> wrote:
>
>> In order to be able to do CVE checking outside of pkg-stat, move the
>> CVE class in a module that can be used by other scripts.
>>
>> Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
>> ---
>> support/scripts/cve.py | 141 ++++++++++++++++++++++++++++++++++++++
>> support/scripts/pkg-stats | 115 +------------------------------
>> 2 files changed, 144 insertions(+), 112 deletions(-)
>> create mode 100755 support/scripts/cve.py
>>
>> diff --git a/support/scripts/cve.py b/support/scripts/cve.py
>> new file mode 100755
>> index 0000000000..874ab4482d
>> --- /dev/null
>> +++ b/support/scripts/cve.py
>> @@ -0,0 +1,141 @@
>> +#!/usr/bin/env python
>> +
>> +# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
>> +# Copyright (C) 2020 by Gregory CLEMENT <gregory.clement@bootlin.com>
>> +#
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation; either version 2 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
>> +# General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write to the Free Software
>> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
>> +
>> +import datetime
>> +import os
>> +import re
>> +import requests # URL checking
>> +import json
>> +import ijson
>> +import distutils.version
>> +import time
>> +import gzip
>> +import sys
>
> Are you able to drop some of these imports from the pkg-stats script ?
> For example, isn't ijson only used by cve.py now ? Same for gzip, for
> distutils.version. Perhaps others ?
I removed some import in the cve.py but not in this file. I can do it,
indeed.
Gregory
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
--
Gregory Clement, Bootlin
Embedded Linux and Kernel engineering
http://bootlin.com
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker
2020-07-08 18:30 ` Matthew Weber
@ 2020-07-09 8:41 ` Gregory CLEMENT
2020-07-09 9:03 ` Gregory CLEMENT
0 siblings, 1 reply; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-09 8:41 UTC (permalink / raw)
To: buildroot
Hi Matt,
> Gregory,
>> +# Used to make multiple requests to the same host. It is global
>> +# because it's used by sub-processes.
>> +http_pool = None
>> +
>> +
>> +class Package:
>> + def __init__(self, name, version, ignored_cves):
>> + self.name = name
>> + self.version = version
>> + self.cves = list()
>> + self.ignored_cves = ignored_cves
>> +
>> +def check_package_cves(nvd_path, packages):
>> + if not os.path.isdir(nvd_path):
>> + os.makedirs(nvd_path)
>> +
>> + for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
>
> This read_nvd_dir call that does the dictionary download has a whole
> bunch of parsing package output "Cannot parse package 'openssl'
> version ''". I assume some of that output will get cleaned up when we
> add the additional CPE matching
Yes. Before this series, pkg-stat considered a pacakge as non- affected
by a CVE if it didn't manage to get the version of the package. Since
patch "package/pkg-utils/cve.py: Manage case when package version
doesn't exist" it is no more the case and the CVEs related to the package
are output in the colunm 'CVEs to check'.
However we might be less verbose during the parsing.
>
>> +def parse_args():
>> + parser = argparse.ArgumentParser()
>> + output = parser.add_argument_group('output', 'Output file(s)')
>> + output.add_argument('--html', dest='html', type=resolvepath,
>> + help='HTML output file')
>> + output.add_argument('--json', dest='json', type=resolvepath,
>> + help='JSON output file')
>> + packages = parser.add_mutually_exclusive_group()
>> + packages.add_argument('-n', dest='npackages', type=int, action='store',
>> + help='Number of packages')
>> + packages.add_argument('-p', dest='packages', action='store',
>> + help='List of packages (comma separated)')
>
> Are the -n and -p options left over from pulling this tool out of
> pkg-stats? Since this report is based on a specific defconfig,
Indeed they are left over and I will remove them
>
>> + parser.add_argument('--nvd-path', dest='nvd_path',
>> + help='Path to the local NVD database', type=resolvepath)
>
> I noticed this was a required item, maybe default to a folder name in
> the current folder when one isn't provided?
I will also add a default value.
>
>> + args = parser.parse_args()
>> + if not args.html and not args.json:
>> + parser.error('at least one of --html or --json (or both) is required')
>
> plus nvd path unless you add a default
>
>> + return args
>> +
>> +
>> +def __main__():
>> + packages = list()
>> + exclude_pacakges = ["linux", "gcc"]
>> + content = json.load(sys.stdin)
>> + for item in content:
>> + if item in exclude_pacakges:
>> + continue
>> + pkg = content[item]
>> + p = Package(item, pkg.get('version', ''), pkg.get('ignored_cves', ''))
>> + packages.append(p)
>> +
>> + args = parse_args()
>> + date = datetime.datetime.utcnow()
>> + commit = subprocess.check_output(['git', 'rev-parse',
>> + 'HEAD']).splitlines()[0].decode()
>
> This git commit check doesn't work when the tools are used with out of
> tree buildroot builds.
>
> To reproduce from within Buildroot clone:
> make O=../foobar qemu_x86_64_defconfig
> cd ../foobar
> make show-info | support/scripts/cve-checker --html report.html
> --nvd-path nvd
Thanks for the report I will check it.
Gregory
--
Gregory Clement, Bootlin
Embedded Linux and Kernel engineering
http://bootlin.com
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 6/9] package/pkg-utils: cve.py: Handle exception when version comparison fails
2020-07-08 16:40 ` [Buildroot] [PATCH 6/9] package/pkg-utils: cve.py: Handle exception when version comparison fails Gregory CLEMENT
@ 2020-07-09 8:52 ` Thomas Petazzoni
0 siblings, 0 replies; 20+ messages in thread
From: Thomas Petazzoni @ 2020-07-09 8:52 UTC (permalink / raw)
To: buildroot
On Wed, 8 Jul 2020 18:40:03 +0200
Gregory CLEMENT <gregory.clement@bootlin.com> wrote:
> With python 3, when a package has a version number x-y-z instead of
> x.y.z, then the version returned by LooseVersion can't be compared
> which raises an exception.
>
> This patch handles this exception by adding a new return value when
> the comparison can't be done.
>
> Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Could you re-organize the series to have this patch first ? Indeed,
this is a fix, we want to apply it both to master *and* backport it to
older branches as well.
I just ran pkg-stats this morning on a Python 3.x system, and it
exploded due to this exception.
However, are you sure just this patch as-is works fine ? The "affects"
function returned just a boolean, and now in some cases it returns a
string. How can this work without changes elsewhere in the code ?
I think it is not a great design to have a function that sometimes
returns a boolean type, sometimes returns a string. It probably needs
to be changed to return:
CVE_AFFECTS
CVE_DOESNT_AFFECT
CVE_UNKNOWN
or some other similar macros.
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 7/9] support/script/pkg-stats: Manage the CVEs that need to be check
2020-07-08 16:40 ` [Buildroot] [PATCH 7/9] support/script/pkg-stats: Manage the CVEs that need to be check Gregory CLEMENT
@ 2020-07-09 9:00 ` Thomas Petazzoni
0 siblings, 0 replies; 20+ messages in thread
From: Thomas Petazzoni @ 2020-07-09 9:00 UTC (permalink / raw)
To: buildroot
On Wed, 8 Jul 2020 18:40:04 +0200
Gregory CLEMENT <gregory.clement@bootlin.com> wrote:
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 883a5bd2be..e033e15e07 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -106,9 +106,11 @@ class Package:
> self.patch_files = []
> self.warnings = 0
> self.current_version = None
> + self.unknown_cve = False
Is this used in your patch ? I don't see it used anywhere.
> self.url = None
> self.url_worker = None
> self.cves = list()
> + self.cves_to_check = list()
> self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None}
> self.status = {}
>
> @@ -504,7 +506,12 @@ def check_package_cves(nvd_path, packages):
> for pkg_name in cve.pkg_names:
> if pkg_name in packages:
> pkg = packages[pkg_name]
> - if cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list()):
> + affected = cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list())
> + print(affected)
This is a debug message, probably not meant to be in your final patch.
> + if (affected == 'Unknown'):
> + pkg.cves_to_check.append(cve.identifier)
So this handling of the "Unknown" return value from cve.affects()
should be done together with the change in cve.affects() I guess.
> + elif affected == True:
> + print(cve.identifier)
Again another print, should it really be here ?
> pkg.cves.append(cve.identifier)
>
> def calculate_stats(packages):
> @@ -544,8 +551,11 @@ def calculate_stats(packages):
> stats["version-not-uptodate"] += 1
> stats["patches"] += pkg.patch_count
> stats["total-cves"] += len(pkg.cves)
> + stats["total-cves-to-check"] += len(pkg.cves_to_check)
> if len(pkg.cves) != 0:
> stats["pkg-cves"] += 1
> + if len(pkg.cves_to_check) != 0:
> + stats["pkg-cves_to_check"] += 1
> return stats
>
>
> @@ -763,11 +773,22 @@ def dump_html_pkg(f, pkg):
> td_class.append("correct")
> else:
> td_class.append("wrong")
> - f.write(" <td class=\"%s\">\n" % " ".join(td_class))
> + f.write(" <td class=\"%s\">\n" % " ".join(td_class))
Spurious change here.
> for cve in pkg.cves:
> f.write(" <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
> f.write(" </td>\n")
>
> + # CVEs to check
> + td_class = ["centered"]
> + if len(pkg.cves_to_check) == 0:
> + td_class.append("correct")
> + else:
> + td_class.append("wrong")
> + f.write(" <td class=\"%s\">\n" % " ".join(td_class))
so you're opening the <td> only in the else case
> + for cve in pkg.cves_to_check:
> + f.write(" <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
> + f.write(" </td>\n")
but closing it in both cases. Doesn't look good.
Also, if you're adding a column, you need to update the column header
as well, to give a title to this column.
> +
So you've added that to the HTML output. Has the JSON output also been
updated? Or perhaps it just works due to how the JSON output is
generated?
> f.write(" </tr>\n")
>
>
> @@ -786,6 +807,7 @@ def dump_html_all_pkgs(f, packages):
> <td class=\"centered\">Warnings</td>
> <td class=\"centered\">Upstream URL</td>
> <td class=\"centered\">CVEs</td>
> +<td class=\"centered\">CVEs to check</td>
> </tr>
> """)
> for pkg in sorted(packages):
> @@ -824,10 +846,14 @@ def dump_html_stats(f, stats):
> stats["version-not-uptodate"])
> f.write("<tr><td>Packages with no known upstream version</td><td>%s</td></tr>\n" %
> stats["version-unknown"])
> - f.write("<tr><td>Packages affected by CVEs</td><td>%s</td></tr>\n" %
> + f.write("<tr><td>Packages might affected by CVEs, where version needed to be checked</td><td>%s</td></tr>\n" %
"Packages might affected by CVEs" is not correct English I believe.
"Packages that might be affected by CVEs" sounds better.
"needed" -> "needs"
> stats["pkg-cves"])
> - f.write("<tr><td>Total number of CVEs affecting all packages</td><td>%s</td></tr>\n" %
> + f.write("<tr><td>Total number of CVEs that might affect all packages, where version needed to be checked</td><td>%s</td></tr>\n" %
version needed -> version needs
> stats["total-cves"])
> + f.write("<tr><td>Packages affected by CVEs</td><td>%s</td></tr>\n" %
> + stats["pkg-cves_to_check"])
> + f.write("<tr><td>Total number of CVEs affecting all packages</td><td>%s</td></tr>\n" %
> + stats["total-cves_to_check"])
> f.write("</table>\n")
>
>
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker
2020-07-09 8:41 ` Gregory CLEMENT
@ 2020-07-09 9:03 ` Gregory CLEMENT
0 siblings, 0 replies; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-09 9:03 UTC (permalink / raw)
To: buildroot
Hi,
>>> +
>>> + args = parse_args()
>>> + date = datetime.datetime.utcnow()
>>> + commit = subprocess.check_output(['git', 'rev-parse',
>>> + 'HEAD']).splitlines()[0].decode()
>>
>> This git commit check doesn't work when the tools are used with out of
>> tree buildroot builds.
>>
>> To reproduce from within Buildroot clone:
>> make O=../foobar qemu_x86_64_defconfig
>> cd ../foobar
>> make show-info | support/scripts/cve-checker --html report.html
>> --nvd-path nvd
>
>
> Thanks for the report I will check it.
I had a closer look on it, the way I tested was the following:
make O=../foobar qemu_x86_64_defconfig
make O=../foobar show-info | support/scripts/cve-checker --html report.html --nvd-path nvd
That's why I didn't see this issue. However, I kept this information from
pkg-stat, but actually I don't think it has big value for a given
configuration.
I will just remove it.
Gregory
>
> --
> Gregory Clement, Bootlin
> Embedded Linux and Kernel engineering
> http://bootlin.com
--
Gregory Clement, Bootlin
Embedded Linux and Kernel engineering
http://bootlin.com
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker
2020-07-08 16:40 ` [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker Gregory CLEMENT
2020-07-08 18:30 ` Matthew Weber
@ 2020-07-09 11:46 ` Matthew Weber
1 sibling, 0 replies; 20+ messages in thread
From: Matthew Weber @ 2020-07-09 11:46 UTC (permalink / raw)
To: buildroot
Gregory,
On Wed, Jul 8, 2020 at 11:41 AM Gregory CLEMENT
<gregory.clement@bootlin.com> wrote:
>
> This scripts takes as entry on stdin a JSON description of the package
> used for a given configuration. This description is the one generated
> by "make show-info".
>
> The script generates the list of all the package used and if they are
> affected by a CVE. The output is either a JSON or an HTML file similar
> to the one generated by pkg-stats.
>
> Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
> ---
> support/scripts/cve-checker | 291 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 291 insertions(+)
> create mode 100755 support/scripts/cve-checker
>
> diff --git a/support/scripts/cve-checker b/support/scripts/cve-checker
> new file mode 100755
> index 0000000000..db8497d7aa
> --- /dev/null
> +++ b/support/scripts/cve-checker
> @@ -0,0 +1,291 @@
> +#!/usr/bin/env python
> +
> +# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> +# Copyright (C) 2020 by Gregory CLEMENT <gregory.clement@bootlin.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
> +
> +import argparse
> +import datetime
> +import fnmatch
> +import os
> +from collections import defaultdict
> +import re
> +import subprocess
> +import requests # URL checking
> +import json
> +import ijson
I noticed ijson wasn't something I had already installed on my host
system. I'd suggest adding the dependency check similar to the
graph-size make target (support/scripts/size-stats), since this is a
user facing tool vs pkg-stats was for maintenance. An easy test to
see what dependencies a user might run into on a basic system could be
to use the base buildroot Docker
image(https://hub.docker.com/r/buildroot/base) or create one using
(support/docker/Dockerfile).
> +import certifi
> +import distutils.version
> +import time
> +import gzip
> +import sys
> +from urllib3 import HTTPSConnectionPool
> +from urllib3.exceptions import HTTPError
> +from multiprocessing import Pool
> +
Regards,
Matt
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker
2020-07-10 11:22 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
@ 2020-07-10 11:22 ` Gregory CLEMENT
0 siblings, 0 replies; 20+ messages in thread
From: Gregory CLEMENT @ 2020-07-10 11:22 UTC (permalink / raw)
To: buildroot
This scripts takes as entry on stdin a JSON description of the package
used for a given configuration. This description is the one generated
by "make show-info".
The script generates the list of all the package used and if they are
affected by a CVE. The output is either a JSON or an HTML file similar
to the one generated by pkg-stats.
Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
support/scripts/cve-checker | 291 ++++++++++++++++++++++++++++++++++++
1 file changed, 291 insertions(+)
create mode 100755 support/scripts/cve-checker
diff --git a/support/scripts/cve-checker b/support/scripts/cve-checker
new file mode 100755
index 0000000000..db8497d7aa
--- /dev/null
+++ b/support/scripts/cve-checker
@@ -0,0 +1,291 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+# Copyright (C) 2020 by Gregory CLEMENT <gregory.clement@bootlin.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import argparse
+import datetime
+import fnmatch
+import os
+from collections import defaultdict
+import re
+import subprocess
+import requests # URL checking
+import json
+import ijson
+import certifi
+import distutils.version
+import time
+import gzip
+import sys
+from urllib3 import HTTPSConnectionPool
+from urllib3.exceptions import HTTPError
+from multiprocessing import Pool
+
+sys.path.append('utils/')
+
+import cve as cvecheck
+
+
+INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
+URL_RE = re.compile(r"\s*https?://\S*\s*$")
+
+RM_API_STATUS_ERROR = 1
+RM_API_STATUS_FOUND_BY_DISTRO = 2
+RM_API_STATUS_FOUND_BY_PATTERN = 3
+RM_API_STATUS_NOT_FOUND = 4
+
+# Used to make multiple requests to the same host. It is global
+# because it's used by sub-processes.
+http_pool = None
+
+
+class Package:
+ def __init__(self, name, version, ignored_cves):
+ self.name = name
+ self.version = version
+ self.cves = list()
+ self.ignored_cves = ignored_cves
+
+def check_package_cves(nvd_path, packages):
+ if not os.path.isdir(nvd_path):
+ os.makedirs(nvd_path)
+
+ for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
+ for pkg_name in cve.pkg_names:
+ pkg = packages.get(pkg_name, '')
+ if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves):
+ pkg.cves.append(cve.identifier)
+
+html_header = """
+<head>
+<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
+<style type=\"text/css\">
+table {
+ width: 100%;
+}
+td {
+ border: 1px solid black;
+}
+td.centered {
+ text-align: center;
+}
+td.wrong {
+ background: #ff9a69;
+}
+td.correct {
+ background: #d2ffc4;
+}
+td.nopatches {
+ background: #d2ffc4;
+}
+td.somepatches {
+ background: #ffd870;
+}
+td.lotsofpatches {
+ background: #ff9a69;
+}
+
+td.good_url {
+ background: #d2ffc4;
+}
+td.missing_url {
+ background: #ffd870;
+}
+td.invalid_url {
+ background: #ff9a69;
+}
+
+td.version-good {
+ background: #d2ffc4;
+}
+td.version-needs-update {
+ background: #ff9a69;
+}
+td.version-unknown {
+ background: #ffd870;
+}
+td.version-error {
+ background: #ccc;
+}
+
+</style>
+<title>CVE status for Buildroot packages</title>
+</head>
+
+<a href=\"#results\">CVE Status</a><br/>
+
+<p id=\"sortable_hint\"></p>
+"""
+
+
+html_footer = """
+</body>
+<script>
+if (typeof sorttable === \"object\") {
+ document.getElementById(\"sortable_hint\").innerHTML =
+ \"hint: the table can be sorted by clicking the column headers\"
+}
+</script>
+</html>
+"""
+
+
+def infra_str(infra_list):
+ if not infra_list:
+ return "Unknown"
+ elif len(infra_list) == 1:
+ return "<b>%s</b><br/>%s" % (infra_list[0][1], infra_list[0][0])
+ elif infra_list[0][1] == infra_list[1][1]:
+ return "<b>%s</b><br/>%s + %s" % \
+ (infra_list[0][1], infra_list[0][0], infra_list[1][0])
+ else:
+ return "<b>%s</b> (%s)<br/><b>%s</b> (%s)" % \
+ (infra_list[0][1], infra_list[0][0],
+ infra_list[1][1], infra_list[1][0])
+
+
+def boolean_str(b):
+ if b:
+ return "Yes"
+ else:
+ return "No"
+
+
+def dump_html_pkg(f, pkg):
+ f.write(" <tr>\n")
+ f.write(" <td>%s</td>\n" % pkg.name)
+
+ # Current version
+ if len(pkg.version) > 20:
+ version = pkg.version[:20] + "..."
+ else:
+ version = pkg.version
+ f.write(" <td class=\"centered\">%s</td>\n" % version)
+
+ # CVEs
+ td_class = ["centered"]
+ if len(pkg.cves) == 0:
+ td_class.append("correct")
+ else:
+ td_class.append("wrong")
+ f.write(" <td class=\"%s\">\n" % " ".join(td_class))
+ for cve in pkg.cves:
+ f.write(" <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+ f.write(" </td>\n")
+
+ f.write(" </tr>\n")
+
+
+def dump_html_all_pkgs(f, packages):
+ f.write("""
+<table class=\"sortable\">
+<tr>
+<td>Package</td>
+<td class=\"centered\">Version</td>
+<td class=\"centered\">CVEs</td>
+</tr>
+""")
+ for pkg in packages:
+ dump_html_pkg(f, pkg)
+ f.write("</table>")
+
+
+def dump_html_gen_info(f, date, commit):
+ # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032
+ f.write("<p><i>Updated on %s, git commit %s</i></p>\n" % (str(date), commit))
+
+
+def dump_html(packages, date, commit, output):
+ with open(output, 'w') as f:
+ f.write(html_header)
+ dump_html_all_pkgs(f, packages)
+ dump_html_gen_info(f, date, commit)
+ f.write(html_footer)
+
+
+def dump_json(packages, date, commit, output):
+ # Format packages as a dictionnary instead of a list
+ # Exclude local field that does not contains real date
+ excluded_fields = ['url_worker', 'name']
+ pkgs = {
+ pkg.name: {
+ k: v
+ for k, v in pkg.__dict__.items()
+ if k not in excluded_fields
+ } for pkg in packages
+ }
+ # The actual structure to dump, add commit and date to it
+ final = {'packages': pkgs,
+ 'commit': commit,
+ 'date': str(date)}
+ with open(output, 'w') as f:
+ json.dump(final, f, indent=2, separators=(',', ': '))
+ f.write('\n')
+
+
+def resolvepath(path):
+ return os.path.abspath(os.path.expanduser(path))
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ output = parser.add_argument_group('output', 'Output file(s)')
+ output.add_argument('--html', dest='html', type=resolvepath,
+ help='HTML output file')
+ output.add_argument('--json', dest='json', type=resolvepath,
+ help='JSON output file')
+ packages = parser.add_mutually_exclusive_group()
+ packages.add_argument('-n', dest='npackages', type=int, action='store',
+ help='Number of packages')
+ packages.add_argument('-p', dest='packages', action='store',
+ help='List of packages (comma separated)')
+ parser.add_argument('--nvd-path', dest='nvd_path',
+ help='Path to the local NVD database', type=resolvepath)
+ args = parser.parse_args()
+ if not args.html and not args.json:
+ parser.error('at least one of --html or --json (or both) is required')
+ return args
+
+
+def __main__():
+ packages = list()
+ exclude_pacakges = ["linux", "gcc"]
+ content = json.load(sys.stdin)
+ for item in content:
+ if item in exclude_pacakges:
+ continue
+ pkg = content[item]
+ p = Package(item, pkg.get('version', ''), pkg.get('ignored_cves', ''))
+ packages.append(p)
+
+ args = parse_args()
+ date = datetime.datetime.utcnow()
+ commit = subprocess.check_output(['git', 'rev-parse',
+ 'HEAD']).splitlines()[0].decode()
+
+ if args.nvd_path:
+ print("Checking packages CVEs")
+ check_package_cves(args.nvd_path, {p.name: p for p in packages})
+ if args.html:
+ print("Write HTML")
+ dump_html(packages, date, commit, args.html)
+ if args.json:
+ print("Write JSON")
+ dump_json(packages, date, commit, args.json)
+
+__main__()
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
end of thread, other threads:[~2020-07-10 11:22 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-08 16:39 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
2020-07-08 16:39 ` [Buildroot] [PATCH 1/9] support/scripts: Turn CVE check into a module Gregory CLEMENT
2020-07-08 16:54 ` Thomas Petazzoni
2020-07-09 7:34 ` Gregory CLEMENT
2020-07-08 16:39 ` [Buildroot] [PATCH 2/9] support/scripts/cve.py: Switch to JSON 1.1 Gregory CLEMENT
2020-07-08 16:40 ` [Buildroot] [PATCH 3/9] package/pkg-utils: show-info: report the list of the CVEs ignored Gregory CLEMENT
2020-07-08 16:53 ` Thomas Petazzoni
2020-07-08 16:40 ` [Buildroot] [PATCH 4/9] package/pkg-utils: Make CVE class independent of the Pacakage class Gregory CLEMENT
2020-07-08 16:40 ` [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker Gregory CLEMENT
2020-07-08 18:30 ` Matthew Weber
2020-07-09 8:41 ` Gregory CLEMENT
2020-07-09 9:03 ` Gregory CLEMENT
2020-07-09 11:46 ` Matthew Weber
2020-07-08 16:40 ` [Buildroot] [PATCH 6/9] package/pkg-utils: cve.py: Handle exception when version comparison fails Gregory CLEMENT
2020-07-09 8:52 ` Thomas Petazzoni
2020-07-08 16:40 ` [Buildroot] [PATCH 7/9] support/script/pkg-stats: Manage the CVEs that need to be check Gregory CLEMENT
2020-07-09 9:00 ` Thomas Petazzoni
2020-07-08 16:40 ` [Buildroot] [PATCH 8/9] support/script/cve-checker: " Gregory CLEMENT
2020-07-08 16:40 ` [Buildroot] [PATCH 9/9] package/pkg-utils/cve.py: Manage case when package version doesn't exist Gregory CLEMENT
2020-07-10 11:22 [Buildroot] [PATCH 0/9] Improving CVE reporting Gregory CLEMENT
2020-07-10 11:22 ` [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker Gregory CLEMENT
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.