* [PATCH v3 1/3] cve-report: add scripts to generate CVE reports
@ 2018-10-10 16:25 grygorii tertychnyi
2018-10-29 23:29 ` akuster808
0 siblings, 1 reply; 3+ messages in thread
From: grygorii tertychnyi @ 2018-10-10 16:25 UTC (permalink / raw)
To: openembedded-core; +Cc: xe-linux-external
cvert-foss - generate CVE report for the list of packages.
Analyze the whole image manifest to align with the complex
CPE configurations.
cvert-update - update NVD feeds and store CVE structues dump.
CVE dump is a pickled representation of the cve_struct dictionary.
cvert.py - python library used by cvert-* scripts.
NVD JSON Vulnerability Feeds https://nvd.nist.gov/vuln/data-feeds#JSON_FEED
Usage examples:
o Download CVE feeds to "nvdfeed" directory
% cvert-update nvdfeed
o Update CVE feeds and store a dump in a file
% cvert-update --store cvedump nvdfeed
o Generate a CVE report
% cvert-foss --feed-dir nvdfeed --output report-foss.txt cve-manifest
o (faster) Use dump file to generate a CVE report
% cvert-foss --restore cvedump --output report-foss.txt cve-manifest
o Generate a full report
% cvert-foss --restore cvedump --show-description --show-reference \
--output report-foss-full.txt cve-manifest
Manifest example:
bash,4.2,CVE-2014-7187
python,2.7.35,
python,3.5.5,CVE-2017-17522 CVE-2018-1061
Report example:
patched | 7.5 | CVE-2018-1061 | python | 3.5.5
patched | 10.0 | CVE-2014-7187 | bash | 4.2
patched | 8.8 | CVE-2017-17522 | python | 3.5.5
unpatched | 10.0 | CVE-2014-6271 | bash | 4.2
unpatched | 10.0 | CVE-2014-6277 | bash | 4.2
unpatched | 10.0 | CVE-2014-6278 | bash | 4.2
unpatched | 10.0 | CVE-2014-7169 | bash | 4.2
unpatched | 10.0 | CVE-2014-7186 | bash | 4.2
unpatched | 4.6 | CVE-2012-3410 | bash | 4.2
unpatched | 8.4 | CVE-2016-7543 | bash | 4.2
unpatched | 5.0 | CVE-2010-3492 | python | 2.7.35
unpatched | 5.3 | CVE-2016-1494 | python | 2.7.35
unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5
unpatched | 6.5 | CVE-2017-18207 | python | 2.7.35
unpatched | 7.1 | CVE-2013-7338 | python | 2.7.35
unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5
unpatched | 8.8 | CVE-2017-17522 | python | 2.7.35
Signed-off-by: grygorii tertychnyi <gtertych@cisco.com>
---
Changes in v3:
o better logging: cvert.py lib log messages are controlled by cvert-* scripts
o add more examples
o add short params ("-o" "--output", "-f", "--feed-dir", etc)
o fix double entries in manifest
o fix pylint warnings
scripts/cvert-foss | 151 ++++++++++++++++
scripts/cvert-update | 79 +++++++++
scripts/cvert.py | 473 +++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 703 insertions(+)
create mode 100755 scripts/cvert-foss
create mode 100755 scripts/cvert-update
create mode 100644 scripts/cvert.py
diff --git a/scripts/cvert-foss b/scripts/cvert-foss
new file mode 100755
index 000000000000..00fbf2c0687b
--- /dev/null
+++ b/scripts/cvert-foss
@@ -0,0 +1,151 @@
+#!/usr/bin/env python3
+#
+# Copyright (c) 2018 by Cisco Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+""" Generate CVE report for the given CVE manifest
+"""
+
+import sys
+import textwrap
+import argparse
+import logging
+import logging.config
+import cvert
+
+def report_foss():
+ """Generate CVE report"""
+
+ parser = argparse.ArgumentParser(
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ description=textwrap.dedent("""
+ Generate CVE report for the given CVE manifest.
+ """),
+ epilog=textwrap.dedent("""
+ @ run examples:
+
+ # Download (update) NVD feeds in "nvdfeed" directory
+ # and prepare the report for the "cve-manifest" file
+ %% %(prog)s --feed-dir nvdfeed --output report-foss.txt cve-manifest
+
+ # Use existed NVD feeds in "nvdfeed" directory
+ # and prepare the report for the "cve-manifest" file
+ %% %(prog)s --offline --feed-dir nvdfeed --output report-foss.txt cve-manifest
+
+ # (faster) Restore CVE dump from "cvedump" (must exist)
+ # and prepare the report for the "cve-manifest" file
+ %% %(prog)s --restore cvedump --output report-foss.txt cve-manifest
+
+ # Restore CVE dump from "cvedump" (must exist)
+ # and prepare the extended report for the "cve-manifest" file
+ %% %(prog)s --restore cvedump --show-description --show-reference --output report-foss.txt cve-manifest
+
+ @ manifest example:
+
+ bash,4.2,CVE-2014-7187
+ python,2.7.35,
+ python,3.5.5,CVE-2017-17522 CVE-2018-1061
+
+ @ report example output:
+
+ . patched | 10.0 | CVE-2014-7187 | bash | 4.2
+ . patched | 7.5 | CVE-2018-1061 | python | 3.5.5
+ . patched | 8.8 | CVE-2017-17522 | python | 3.5.5
+ unpatched | 10.0 | CVE-2014-6271 | bash | 4.2
+ unpatched | 10.0 | CVE-2014-6277 | bash | 4.2
+ unpatched | 10.0 | CVE-2014-6278 | bash | 4.2
+ unpatched | 10.0 | CVE-2014-7169 | bash | 4.2
+ unpatched | 10.0 | CVE-2014-7186 | bash | 4.2
+ unpatched | 4.6 | CVE-2012-3410 | bash | 4.2
+ unpatched | 8.4 | CVE-2016-7543 | bash | 4.2
+ unpatched | 5.0 | CVE-2010-3492 | python | 2.7.35
+ unpatched | 5.3 | CVE-2016-1494 | python | 2.7.35
+ unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5
+ unpatched | 6.5 | CVE-2017-18207 | python | 2.7.35
+ unpatched | 7.1 | CVE-2013-7338 | python | 2.7.35
+ unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5
+ unpatched | 8.8 | CVE-2017-17522 | python | 2.7.35
+ """))
+
+ group = parser.add_mutually_exclusive_group(required=True)
+ group.add_argument("-f", "--feed-dir", help="feeds directory")
+ group.add_argument("-d", "--restore", help="load CVE data structures from file",
+ metavar="FILENAME")
+ parser.add_argument("--offline", help="do not update from NVD site",
+ action="store_true")
+ parser.add_argument("-o", "--output", help="save report to the file")
+ parser.add_argument("--show-description", help='show "Description" in the report',
+ action="store_true")
+ parser.add_argument("--show-reference", help='show "Reference" in the report',
+ action="store_true")
+ parser.add_argument("--debug", help="print debug messages",
+ action="store_true")
+
+ parser.add_argument("cve_manifest", help="file with a list of packages, "
+ "each line contains three comma separated values: name, "
+ "version and a space separated list of patched CVEs, "
+ "e.g.: python,3.5.5,CVE-2017-17522 CVE-2018-1061",
+ metavar="cve-manifest")
+
+ args = parser.parse_args()
+
+ logging.config.dictConfig(cvert.logconfig(args.debug))
+
+ cve_manifest = {}
+
+ with open(args.cve_manifest, "r") as fil:
+ for lin in fil:
+ lin = lin.rstrip()
+
+ # skip empty lines
+ if not lin:
+ continue
+
+ product, version, patched = lin.split(",", maxsplit=3)
+
+ if product in cve_manifest:
+ cve_manifest[product][version] = patched.split()
+ else:
+ cve_manifest[product] = {
+ version: patched.split()
+ }
+
+ if args.restore:
+ cve_struct = cvert.load_cve(args.restore)
+ elif args.feed_dir:
+ cve_struct = cvert.update_feeds(args.feed_dir, args.offline)
+
+ if not cve_struct and args.offline:
+ parser.error("No CVEs found. Try to turn off offline mode or use other file to restore.")
+
+ if args.output:
+ output = open(args.output, "w")
+ else:
+ output = sys.stdout
+
+ report = cvert.generate_report(cve_manifest, cve_struct)
+
+ cvert.print_report(report,
+ show_description=args.show_description,
+ show_reference=args.show_reference,
+ output=output)
+
+ if args.output:
+ output.close()
+
+
+if __name__ == "__main__":
+ report_foss()
diff --git a/scripts/cvert-update b/scripts/cvert-update
new file mode 100755
index 000000000000..3b3f5572a83c
--- /dev/null
+++ b/scripts/cvert-update
@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+#
+# Copyright (c) 2018 by Cisco Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+""" Update NVD feeds and store CVE blob locally
+"""
+
+
+import textwrap
+import argparse
+import logging
+import logging.config
+import cvert
+
+
+def update_cvert():
+ """Update CVE storage"""
+
+ parser = argparse.ArgumentParser(
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ description=textwrap.dedent("""
+ Update NVD feeds and store CVE blob locally.
+ """),
+ epilog=textwrap.dedent("""
+ examples:
+
+ # Download NVD feeds to "nvdfeed" directory.
+ # If there are meta files in the directory, they will be updated
+ # and only fresh archives will be downloaded
+ %% %(prog)s nvdfeed
+
+ # Inspect NVD feeds in "nvdfeed" directory
+ # and prepare a CVE dump python blob "cvedump".
+ # Use it later as input for cvert-* scripts (for speeding up)
+ %% %(prog)s --offline --store cvedump nvdfeed
+
+ # Download (update) NVD feeds and prepare the CVE dump
+ %% %(prog)s --store cvedump nvdfeed
+ """))
+
+ parser.add_argument("-d", "--store", help="save CVE data structures in file",
+ metavar="FILENAME")
+ parser.add_argument("--offline", help="do not update from NVD site",
+ action="store_true")
+ parser.add_argument("--debug", help="print debug messages",
+ action="store_true")
+
+ parser.add_argument("feed_dir", help="feeds directory",
+ metavar="feed-dir")
+
+ args = parser.parse_args()
+
+ logging.config.dictConfig(cvert.logconfig(args.debug))
+
+ cve_struct = cvert.update_feeds(args.feed_dir, args.offline)
+
+ if not cve_struct and args.offline:
+ parser.error("No CVEs found in {0}. Try turn off offline mode.".format(args.feed_dir))
+
+ if args.store:
+ cvert.save_cve(args.store, cve_struct)
+
+
+if __name__ == "__main__":
+ update_cvert()
diff --git a/scripts/cvert.py b/scripts/cvert.py
new file mode 100644
index 000000000000..f93b95c84965
--- /dev/null
+++ b/scripts/cvert.py
@@ -0,0 +1,473 @@
+#!/usr/bin/env python3
+#
+# Copyright (c) 2018 by Cisco Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+""" CVERT library: set of functions for CVE reports
+"""
+
+
+import os
+import re
+import sys
+import json
+import gzip
+import pickle
+import logging
+import hashlib
+import datetime
+import textwrap
+import urllib.request
+import distutils.version
+
+
+logging.getLogger(__name__).addHandler(logging.NullHandler())
+
+
+def generate_report(manifest, cve_struct):
+ """Generate CVE report"""
+
+ report = []
+
+ for cve in cve_struct:
+ affected = set()
+
+ for conf in cve_struct[cve]["nodes"]:
+ affected = affected.union(process_configuration(manifest, conf))
+
+ for key in affected:
+ product, version = key.split(",")
+ patched = manifest[product][version]
+
+ if cve in patched:
+ cve_item = {"status": "patched"}
+ else:
+ cve_item = {"status": "unpatched"}
+
+ cve_item["CVSS"] = "{0:.1f}".format(cve_struct[cve]["score"])
+ cve_item["CVE"] = cve
+ cve_item["product"] = product
+ cve_item["version"] = version
+ cve_item["description"] = cve_struct[cve]["description"]
+ cve_item["reference"] = [x["url"] for x in cve_struct[cve]["reference"]]
+
+ logging.debug("%9s %s %s,%s",
+ cve_item["status"], cve_item["CVE"],
+ cve_item["product"], cve_item["version"])
+
+ report.append(cve_item)
+
+ return sorted(report, key=lambda x: (x["status"], x["product"], x["CVSS"], x["CVE"]))
+
+
+def process_configuration(manifest, conf):
+ """Recursive call to process all CVE configurations"""
+
+ operator = conf["operator"]
+
+ if operator not in ["OR", "AND"]:
+ raise ValueError("operator {} is not supported".format(operator))
+
+ operator = True if operator == "AND" else False
+ match = False
+ affected = set()
+
+ if "cpe" in conf:
+ match = process_cpe(manifest, conf["cpe"][0], affected)
+
+ for cpe in conf["cpe"][1:]:
+ package_match = process_cpe(manifest, cpe, affected)
+
+ # match = match <operator> package_match
+ match = operator ^ ((operator ^ match) or (operator ^ package_match))
+ elif "children" in conf:
+ product_set = process_configuration(manifest, conf["children"][0])
+
+ if product_set:
+ match = True
+ affected = affected.union(product_set)
+
+ for child in conf["children"][1:]:
+ product_set = process_configuration(manifest, child)
+ package_match = True if product_set else False
+
+ # match = match OP package_match
+ match = operator ^ ((operator ^ match) or (operator ^ package_match))
+
+ if package_match:
+ affected = affected.union(product_set)
+
+ if match:
+ return affected
+
+ return ()
+
+
+def process_cpe(manifest, cpe, affected):
+ """Match CPE with all manifest packages"""
+
+ if not cpe["vulnerable"]:
+ # ignore non vulnerable part
+ return False
+
+ version_range = {}
+
+ for flag in ["versionStartIncluding",
+ "versionStartExcluding",
+ "versionEndIncluding",
+ "versionEndExcluding"]:
+ if flag in cpe:
+ version_range[flag] = cpe[flag]
+
+ # take only "product" and "version"
+ product, version = cpe["cpe23Uri"].split(":")[4:6]
+
+ if product not in manifest:
+ return False
+
+ if not version_range:
+ if version == "*":
+ # ignore CVEs that touches all versions of package,
+ # can not fix it anyway
+ logging.debug('ignore "*" in %s', cpe["cpe23Uri"])
+ return False
+ elif version == "-":
+ # "-" means NA
+ #
+ # NA (i.e. "not applicable/not used"). The logical value NA
+ # SHOULD be assigned when there is no legal or meaningful
+ # value for that attribute, or when that attribute is not
+ # used as part of the description.
+ # This includes the situation in which an attribute has
+ # an obtainable value that is null
+ #
+ # Ignores CVEs if version is not set
+ logging.debug('ignore "-" in %s', cpe["cpe23Uri"])
+ return False
+ else:
+ version_range["versionExactMatch"] = version
+
+ result = False
+
+ for version in manifest[product]:
+ try:
+ if match_version(version,
+ version_range):
+ logging.debug("match %s %s: %s", product, version, cpe["cpe23Uri"])
+ affected.add("{},{}".format(product, version))
+
+ result = True
+ except TypeError:
+ # version comparison is a very tricky
+ # sometimes provider changes product version in a strange manner
+ # and the above comparison just failed
+ # so here we try to make version string "more standard"
+
+ if match_version(twik_version(version),
+ [twik_version(v) for v in version_range]):
+ logging.debug("match %s %s (twiked): %s", product, twik_version(version),
+ cpe["cpe23Uri"])
+ affected.add("{},{}".format(product, version))
+
+ result = True
+
+ return result
+
+
+def match_version(version, vrange):
+ """Match version with the version range"""
+
+ result = False
+ version = util_version(version)
+
+ if "versionExactMatch" in vrange:
+ if version == util_version(vrange["versionExactMatch"]):
+ result = True
+ else:
+ result = True
+
+ if "versionStartIncluding" in vrange:
+ result = result and version >= util_version(vrange["versionStartIncluding"])
+
+ if "versionStartExcluding" in vrange:
+ result = result and version > util_version(vrange["versionStartExcluding"])
+
+ if "versionEndIncluding" in vrange:
+ result = result and version <= util_version(vrange["versionEndIncluding"])
+
+ if "versionEndExcluding" in vrange:
+ result = result and version < util_version(vrange["versionEndExcluding"])
+
+ return result
+
+
+def util_version(version):
+ """Simplify package version"""
+ return distutils.version.LooseVersion(version.split("+git")[0])
+
+
+def twik_version(version):
+ """Return "standard" version for complex cases"""
+ return "v1" + re.sub(r"^[a-zA-Z]+", "", version)
+
+
+def print_report(report, width=70, show_description=False, show_reference=False, output=sys.stdout):
+ """Print out final report"""
+
+ for cve in report:
+ print("{0:>9s} | {1:>4s} | {2:18s} | {3} | {4}".format(cve["status"], cve["CVSS"],
+ cve["CVE"], cve["product"],
+ cve["version"]),
+ file=output)
+
+ if show_description:
+ print("{0:>9s} + {1}".format(" ", "Description"), file=output)
+
+ for lin in textwrap.wrap(cve["description"], width=width):
+ print("{0:>9s} {1}".format(" ", lin), file=output)
+
+ if show_reference:
+ print("{0:>9s} + {1}".format(" ", "Reference"), file=output)
+
+ for url in cve["reference"]:
+ print("{0:>9s} {1}".format(" ", url), file=output)
+
+
+def update_feeds(feed_dir, offline=False, start=2002):
+ """Update all JSON feeds"""
+
+ feed_dir = os.path.realpath(feed_dir)
+ year_now = datetime.datetime.now().year
+ cve_struct = {}
+
+ for year in range(start, year_now + 1):
+ update_year(cve_struct, year, feed_dir, offline)
+
+ return cve_struct
+
+
+def update_year(cve_struct, year, feed_dir, offline):
+ """Update one JSON feed for the particular year"""
+
+ url_prefix = "https://static.nvd.nist.gov/feeds/json/cve/1.0"
+ file_prefix = "nvdcve-1.0-{0}".format(year)
+
+ meta = {
+ "url": "{0}/{1}.meta".format(url_prefix, file_prefix),
+ "file": os.path.join(feed_dir, "{0}.meta".format(file_prefix))
+ }
+
+ feed = {
+ "url": "{0}/{1}.json.gz".format(url_prefix, file_prefix),
+ "file": os.path.join(feed_dir, "{0}.json.gz".format(file_prefix))
+ }
+
+ ctx = {}
+
+ if not offline:
+ ctx = download_feed(meta, feed)
+
+ if not "meta" in ctx or not "feed" in ctx:
+ return
+
+ if not os.path.isfile(meta["file"]):
+ return
+
+ if not os.path.isfile(feed["file"]):
+ return
+
+ if not "meta" in ctx:
+ ctx["meta"] = ctx_meta(meta["file"])
+
+ if not "sha256" in ctx["meta"]:
+ return
+
+ if not "feed" in ctx:
+ ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"])
+
+ if not ctx["feed"]:
+ return
+
+ logging.debug("parsing year %s", year)
+
+ for cve_item in ctx["feed"]["CVE_Items"]:
+ iden, cve = parse_item(cve_item)
+
+ if not iden:
+ continue
+
+ if not cve:
+ logging.error("%s parse error", iden)
+ break
+
+ if iden in cve_struct:
+ logging.error("%s duplicated", iden)
+ break
+
+ cve_struct[iden] = cve
+
+ logging.debug("cve records: %d", len(cve_struct))
+
+
+def ctx_meta(filename):
+ """Parse feed meta file"""
+
+ if not os.path.isfile(filename):
+ return {}
+
+ ctx = {}
+
+ with open(filename) as fil:
+ for lin in fil:
+ pair = lin.split(":", maxsplit=1)
+ ctx[pair[0]] = pair[1].rstrip()
+
+ return ctx
+
+
+def ctx_gzip(filename, checksum=""):
+ """Parse feed archive file"""
+
+ if not os.path.isfile(filename):
+ return {}
+
+ with gzip.open(filename) as fil:
+ try:
+ ctx = fil.read()
+ except (EOFError, OSError):
+ logging.error("failed to process gz archive %s", filename, exc_info=True)
+ return {}
+
+ if checksum and checksum.upper() != hashlib.sha256(ctx).hexdigest().upper():
+ return {}
+
+ return json.loads(ctx.decode())
+
+
+def parse_item(cve_item):
+ """Parse one JSON CVE entry"""
+
+ cve_id = cve_item["cve"]["CVE_data_meta"]["ID"][:]
+ impact = cve_item["impact"]
+
+ if not impact:
+ # REJECTed CVE
+ return None, None
+
+ if "baseMetricV3" in impact:
+ score = impact["baseMetricV3"]["cvssV3"]["baseScore"]
+ elif "baseMetricV2" in impact:
+ score = impact["baseMetricV2"]["cvssV2"]["baseScore"]
+ else:
+ return cve_id, None
+
+ return cve_id, {
+ "score": score,
+ "nodes": cve_item["configurations"]["nodes"][:],
+ "reference": cve_item["cve"]["references"]["reference_data"][:],
+ "description": cve_item["cve"]["description"]["description_data"][0]["value"]
+ }
+
+
+def download_feed(meta, feed):
+ """Download and parse feed"""
+
+ ctx = {}
+
+ if not retrieve_url(meta["url"], meta["file"]):
+ return {}
+
+ ctx["meta"] = ctx_meta(meta["file"])
+
+ if not "sha256" in ctx["meta"]:
+ return {}
+
+ ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"])
+
+ if not ctx["feed"]:
+ if not retrieve_url(feed["url"], feed["file"]):
+ return {}
+
+ ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"])
+
+ return ctx
+
+
+def retrieve_url(url, filename=None):
+ """Download file by URL"""
+
+ if filename:
+ os.makedirs(os.path.dirname(filename), exist_ok=True)
+
+ logging.debug("downloading %s", url)
+
+ try:
+ urllib.request.urlretrieve(url, filename=filename)
+ except urllib.error.HTTPError:
+ logging.error("failed to download URL %s", url, exc_info=True)
+ return False
+
+ return True
+
+
+def logconfig(debug_flag=False):
+ """Return default log config"""
+
+ return {
+ "version": 1,
+ "formatters": {
+ "f": {
+ "format": "# %(asctime)s %% CVERT %% %(levelname)-8s %% %(message)s"
+ }
+ },
+ "handlers": {
+ "h": {
+ "class": "logging.StreamHandler",
+ "formatter": "f",
+ "level": logging.DEBUG if debug_flag else logging.INFO
+ }
+ },
+ "root": {
+ "handlers": ["h"],
+ "level": logging.DEBUG if debug_flag else logging.INFO
+ },
+ }
+
+
+def save_cve(filename, cve_struct):
+ """Save CVE structure in the file"""
+
+ filename = os.path.realpath(filename)
+
+ logging.debug("saving %d CVE records to %s", len(cve_struct), filename)
+
+ with open(filename, "wb") as fil:
+ pickle.dump(cve_struct, fil)
+
+
+def load_cve(filename):
+ """Load CVE structure from the file"""
+
+ filename = os.path.realpath(filename)
+
+ logging.debug("loading from %s", filename)
+
+ with open(filename, "rb") as fil:
+ cve_struct = pickle.load(fil)
+
+ logging.debug("cve records: %d", len(cve_struct))
+
+ return cve_struct
--
2.10.3.dirty
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH v3 1/3] cve-report: add scripts to generate CVE reports
2018-10-10 16:25 [PATCH v3 1/3] cve-report: add scripts to generate CVE reports grygorii tertychnyi
@ 2018-10-29 23:29 ` akuster808
2018-10-30 2:59 ` Grygorii Tertychnyi
0 siblings, 1 reply; 3+ messages in thread
From: akuster808 @ 2018-10-29 23:29 UTC (permalink / raw)
To: grygorii tertychnyi, openembedded-core; +Cc: xe-linux-external
Grygorii,
I was good to see you at OEDeM.
I have some feedback.
On 10/10/18 9:25 AM, grygorii tertychnyi via Openembedded-core wrote:
> cvert-foss - generate CVE report for the list of packages.
> Analyze the whole image manifest to align with the complex
> CPE configurations.
>
> cvert-update - update NVD feeds and store CVE structues dump.
> CVE dump is a pickled representation of the cve_struct dictionary.
>
> cvert.py - python library used by cvert-* scripts.
> NVD JSON Vulnerability Feeds https://nvd.nist.gov/vuln/data-feeds#JSON_FEED
>
> Usage examples:
>
> o Download CVE feeds to "nvdfeed" directory
> % cvert-update nvdfeed
> o Update CVE feeds and store a dump in a file
> % cvert-update --store cvedump nvdfeed
> o Generate a CVE report
> % cvert-foss --feed-dir nvdfeed --output report-foss.txt cve-manifest
> o (faster) Use dump file to generate a CVE report
> % cvert-foss --restore cvedump --output report-foss.txt cve-manifest
> o Generate a full report
> % cvert-foss --restore cvedump --show-description --show-reference \
> --output report-foss-full.txt cve-manifest report-foss-full.txt
The cve-manifest, I could not figure out a way to create this from the
above steps. I had your patches included when I did an image build then
I got the "report-foss-full.txt" created. I am guessing I missed some
steps somewhere. Can you clarify?
Also, is there a way to version the report file by image or standard
buildID so that I don't accidentally overwrite the txt file. Maybe a
future enhancement is to group the packages by layer. I ran this with
the meta-security layer and I now need to look up what packages belong
to what layer to go work on them.
Overall, I like this implementation and seems to be fast.
Thanks for the patches.
kind regards,
Armin
>
> Manifest example:
>
> bash,4.2,CVE-2014-7187
> python,2.7.35,
> python,3.5.5,CVE-2017-17522 CVE-2018-1061
>
> Report example:
>
> patched | 7.5 | CVE-2018-1061 | python | 3.5.5
> patched | 10.0 | CVE-2014-7187 | bash | 4.2
> patched | 8.8 | CVE-2017-17522 | python | 3.5.5
> unpatched | 10.0 | CVE-2014-6271 | bash | 4.2
> unpatched | 10.0 | CVE-2014-6277 | bash | 4.2
> unpatched | 10.0 | CVE-2014-6278 | bash | 4.2
> unpatched | 10.0 | CVE-2014-7169 | bash | 4.2
> unpatched | 10.0 | CVE-2014-7186 | bash | 4.2
> unpatched | 4.6 | CVE-2012-3410 | bash | 4.2
> unpatched | 8.4 | CVE-2016-7543 | bash | 4.2
> unpatched | 5.0 | CVE-2010-3492 | python | 2.7.35
> unpatched | 5.3 | CVE-2016-1494 | python | 2.7.35
> unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5
> unpatched | 6.5 | CVE-2017-18207 | python | 2.7.35
> unpatched | 7.1 | CVE-2013-7338 | python | 2.7.35
> unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5
> unpatched | 8.8 | CVE-2017-17522 | python | 2.7.35
>
> Signed-off-by: grygorii tertychnyi <gtertych@cisco.com>
> ---
>
> Changes in v3:
> o better logging: cvert.py lib log messages are controlled by cvert-* scripts
> o add more examples
> o add short params ("-o" "--output", "-f", "--feed-dir", etc)
> o fix double entries in manifest
> o fix pylint warnings
>
> scripts/cvert-foss | 151 ++++++++++++++++
> scripts/cvert-update | 79 +++++++++
> scripts/cvert.py | 473 +++++++++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 703 insertions(+)
> create mode 100755 scripts/cvert-foss
> create mode 100755 scripts/cvert-update
> create mode 100644 scripts/cvert.py
>
> diff --git a/scripts/cvert-foss b/scripts/cvert-foss
> new file mode 100755
> index 000000000000..00fbf2c0687b
> --- /dev/null
> +++ b/scripts/cvert-foss
> @@ -0,0 +1,151 @@
> +#!/usr/bin/env python3
> +#
> +# Copyright (c) 2018 by Cisco Systems, Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +#
> +
> +""" Generate CVE report for the given CVE manifest
> +"""
> +
> +import sys
> +import textwrap
> +import argparse
> +import logging
> +import logging.config
> +import cvert
> +
> +def report_foss():
> + """Generate CVE report"""
> +
> + parser = argparse.ArgumentParser(
> + formatter_class=argparse.RawDescriptionHelpFormatter,
> + description=textwrap.dedent("""
> + Generate CVE report for the given CVE manifest.
> + """),
> + epilog=textwrap.dedent("""
> + @ run examples:
> +
> + # Download (update) NVD feeds in "nvdfeed" directory
> + # and prepare the report for the "cve-manifest" file
> + %% %(prog)s --feed-dir nvdfeed --output report-foss.txt cve-manifest
> +
> + # Use existed NVD feeds in "nvdfeed" directory
> + # and prepare the report for the "cve-manifest" file
> + %% %(prog)s --offline --feed-dir nvdfeed --output report-foss.txt cve-manifest
> +
> + # (faster) Restore CVE dump from "cvedump" (must exist)
> + # and prepare the report for the "cve-manifest" file
> + %% %(prog)s --restore cvedump --output report-foss.txt cve-manifest
> +
> + # Restore CVE dump from "cvedump" (must exist)
> + # and prepare the extended report for the "cve-manifest" file
> + %% %(prog)s --restore cvedump --show-description --show-reference --output report-foss.txt cve-manifest
> +
> + @ manifest example:
> +
> + bash,4.2,CVE-2014-7187
> + python,2.7.35,
> + python,3.5.5,CVE-2017-17522 CVE-2018-1061
> +
> + @ report example output:
> +
> + . patched | 10.0 | CVE-2014-7187 | bash | 4.2
> + . patched | 7.5 | CVE-2018-1061 | python | 3.5.5
> + . patched | 8.8 | CVE-2017-17522 | python | 3.5.5
> + unpatched | 10.0 | CVE-2014-6271 | bash | 4.2
> + unpatched | 10.0 | CVE-2014-6277 | bash | 4.2
> + unpatched | 10.0 | CVE-2014-6278 | bash | 4.2
> + unpatched | 10.0 | CVE-2014-7169 | bash | 4.2
> + unpatched | 10.0 | CVE-2014-7186 | bash | 4.2
> + unpatched | 4.6 | CVE-2012-3410 | bash | 4.2
> + unpatched | 8.4 | CVE-2016-7543 | bash | 4.2
> + unpatched | 5.0 | CVE-2010-3492 | python | 2.7.35
> + unpatched | 5.3 | CVE-2016-1494 | python | 2.7.35
> + unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5
> + unpatched | 6.5 | CVE-2017-18207 | python | 2.7.35
> + unpatched | 7.1 | CVE-2013-7338 | python | 2.7.35
> + unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5
> + unpatched | 8.8 | CVE-2017-17522 | python | 2.7.35
> + """))
> +
> + group = parser.add_mutually_exclusive_group(required=True)
> + group.add_argument("-f", "--feed-dir", help="feeds directory")
> + group.add_argument("-d", "--restore", help="load CVE data structures from file",
> + metavar="FILENAME")
> + parser.add_argument("--offline", help="do not update from NVD site",
> + action="store_true")
> + parser.add_argument("-o", "--output", help="save report to the file")
> + parser.add_argument("--show-description", help='show "Description" in the report',
> + action="store_true")
> + parser.add_argument("--show-reference", help='show "Reference" in the report',
> + action="store_true")
> + parser.add_argument("--debug", help="print debug messages",
> + action="store_true")
> +
> + parser.add_argument("cve_manifest", help="file with a list of packages, "
> + "each line contains three comma separated values: name, "
> + "version and a space separated list of patched CVEs, "
> + "e.g.: python,3.5.5,CVE-2017-17522 CVE-2018-1061",
> + metavar="cve-manifest")
> +
> + args = parser.parse_args()
> +
> + logging.config.dictConfig(cvert.logconfig(args.debug))
> +
> + cve_manifest = {}
> +
> + with open(args.cve_manifest, "r") as fil:
> + for lin in fil:
> + lin = lin.rstrip()
> +
> + # skip empty lines
> + if not lin:
> + continue
> +
> + product, version, patched = lin.split(",", maxsplit=3)
> +
> + if product in cve_manifest:
> + cve_manifest[product][version] = patched.split()
> + else:
> + cve_manifest[product] = {
> + version: patched.split()
> + }
> +
> + if args.restore:
> + cve_struct = cvert.load_cve(args.restore)
> + elif args.feed_dir:
> + cve_struct = cvert.update_feeds(args.feed_dir, args.offline)
> +
> + if not cve_struct and args.offline:
> + parser.error("No CVEs found. Try to turn off offline mode or use other file to restore.")
> +
> + if args.output:
> + output = open(args.output, "w")
> + else:
> + output = sys.stdout
> +
> + report = cvert.generate_report(cve_manifest, cve_struct)
> +
> + cvert.print_report(report,
> + show_description=args.show_description,
> + show_reference=args.show_reference,
> + output=output)
> +
> + if args.output:
> + output.close()
> +
> +
> +if __name__ == "__main__":
> + report_foss()
> diff --git a/scripts/cvert-update b/scripts/cvert-update
> new file mode 100755
> index 000000000000..3b3f5572a83c
> --- /dev/null
> +++ b/scripts/cvert-update
> @@ -0,0 +1,79 @@
> +#!/usr/bin/env python3
> +#
> +# Copyright (c) 2018 by Cisco Systems, Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +#
> +
> +""" Update NVD feeds and store CVE blob locally
> +"""
> +
> +
> +import textwrap
> +import argparse
> +import logging
> +import logging.config
> +import cvert
> +
> +
> +def update_cvert():
> + """Update CVE storage"""
> +
> + parser = argparse.ArgumentParser(
> + formatter_class=argparse.RawDescriptionHelpFormatter,
> + description=textwrap.dedent("""
> + Update NVD feeds and store CVE blob locally.
> + """),
> + epilog=textwrap.dedent("""
> + examples:
> +
> + # Download NVD feeds to "nvdfeed" directory.
> + # If there are meta files in the directory, they will be updated
> + # and only fresh archives will be downloaded
> + %% %(prog)s nvdfeed
> +
> + # Inspect NVD feeds in "nvdfeed" directory
> + # and prepare a CVE dump python blob "cvedump".
> + # Use it later as input for cvert-* scripts (for speeding up)
> + %% %(prog)s --offline --store cvedump nvdfeed
> +
> + # Download (update) NVD feeds and prepare the CVE dump
> + %% %(prog)s --store cvedump nvdfeed
> + """))
> +
> + parser.add_argument("-d", "--store", help="save CVE data structures in file",
> + metavar="FILENAME")
> + parser.add_argument("--offline", help="do not update from NVD site",
> + action="store_true")
> + parser.add_argument("--debug", help="print debug messages",
> + action="store_true")
> +
> + parser.add_argument("feed_dir", help="feeds directory",
> + metavar="feed-dir")
> +
> + args = parser.parse_args()
> +
> + logging.config.dictConfig(cvert.logconfig(args.debug))
> +
> + cve_struct = cvert.update_feeds(args.feed_dir, args.offline)
> +
> + if not cve_struct and args.offline:
> + parser.error("No CVEs found in {0}. Try turn off offline mode.".format(args.feed_dir))
> +
> + if args.store:
> + cvert.save_cve(args.store, cve_struct)
> +
> +
> +if __name__ == "__main__":
> + update_cvert()
> diff --git a/scripts/cvert.py b/scripts/cvert.py
> new file mode 100644
> index 000000000000..f93b95c84965
> --- /dev/null
> +++ b/scripts/cvert.py
> @@ -0,0 +1,473 @@
> +#!/usr/bin/env python3
> +#
> +# Copyright (c) 2018 by Cisco Systems, Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +#
> +
> +""" CVERT library: set of functions for CVE reports
> +"""
> +
> +
> +import os
> +import re
> +import sys
> +import json
> +import gzip
> +import pickle
> +import logging
> +import hashlib
> +import datetime
> +import textwrap
> +import urllib.request
> +import distutils.version
> +
> +
> +logging.getLogger(__name__).addHandler(logging.NullHandler())
> +
> +
> +def generate_report(manifest, cve_struct):
> + """Generate CVE report"""
> +
> + report = []
> +
> + for cve in cve_struct:
> + affected = set()
> +
> + for conf in cve_struct[cve]["nodes"]:
> + affected = affected.union(process_configuration(manifest, conf))
> +
> + for key in affected:
> + product, version = key.split(",")
> + patched = manifest[product][version]
> +
> + if cve in patched:
> + cve_item = {"status": "patched"}
> + else:
> + cve_item = {"status": "unpatched"}
> +
> + cve_item["CVSS"] = "{0:.1f}".format(cve_struct[cve]["score"])
> + cve_item["CVE"] = cve
> + cve_item["product"] = product
> + cve_item["version"] = version
> + cve_item["description"] = cve_struct[cve]["description"]
> + cve_item["reference"] = [x["url"] for x in cve_struct[cve]["reference"]]
> +
> + logging.debug("%9s %s %s,%s",
> + cve_item["status"], cve_item["CVE"],
> + cve_item["product"], cve_item["version"])
> +
> + report.append(cve_item)
> +
> + return sorted(report, key=lambda x: (x["status"], x["product"], x["CVSS"], x["CVE"]))
> +
> +
> +def process_configuration(manifest, conf):
> + """Recursive call to process all CVE configurations"""
> +
> + operator = conf["operator"]
> +
> + if operator not in ["OR", "AND"]:
> + raise ValueError("operator {} is not supported".format(operator))
> +
> + operator = True if operator == "AND" else False
> + match = False
> + affected = set()
> +
> + if "cpe" in conf:
> + match = process_cpe(manifest, conf["cpe"][0], affected)
> +
> + for cpe in conf["cpe"][1:]:
> + package_match = process_cpe(manifest, cpe, affected)
> +
> + # match = match <operator> package_match
> + match = operator ^ ((operator ^ match) or (operator ^ package_match))
> + elif "children" in conf:
> + product_set = process_configuration(manifest, conf["children"][0])
> +
> + if product_set:
> + match = True
> + affected = affected.union(product_set)
> +
> + for child in conf["children"][1:]:
> + product_set = process_configuration(manifest, child)
> + package_match = True if product_set else False
> +
> + # match = match OP package_match
> + match = operator ^ ((operator ^ match) or (operator ^ package_match))
> +
> + if package_match:
> + affected = affected.union(product_set)
> +
> + if match:
> + return affected
> +
> + return ()
> +
> +
> +def process_cpe(manifest, cpe, affected):
> + """Match CPE with all manifest packages"""
> +
> + if not cpe["vulnerable"]:
> + # ignore non vulnerable part
> + return False
> +
> + version_range = {}
> +
> + for flag in ["versionStartIncluding",
> + "versionStartExcluding",
> + "versionEndIncluding",
> + "versionEndExcluding"]:
> + if flag in cpe:
> + version_range[flag] = cpe[flag]
> +
> + # take only "product" and "version"
> + product, version = cpe["cpe23Uri"].split(":")[4:6]
> +
> + if product not in manifest:
> + return False
> +
> + if not version_range:
> + if version == "*":
> + # ignore CVEs that touches all versions of package,
> + # can not fix it anyway
> + logging.debug('ignore "*" in %s', cpe["cpe23Uri"])
> + return False
> + elif version == "-":
> + # "-" means NA
> + #
> + # NA (i.e. "not applicable/not used"). The logical value NA
> + # SHOULD be assigned when there is no legal or meaningful
> + # value for that attribute, or when that attribute is not
> + # used as part of the description.
> + # This includes the situation in which an attribute has
> + # an obtainable value that is null
> + #
> + # Ignores CVEs if version is not set
> + logging.debug('ignore "-" in %s', cpe["cpe23Uri"])
> + return False
> + else:
> + version_range["versionExactMatch"] = version
> +
> + result = False
> +
> + for version in manifest[product]:
> + try:
> + if match_version(version,
> + version_range):
> + logging.debug("match %s %s: %s", product, version, cpe["cpe23Uri"])
> + affected.add("{},{}".format(product, version))
> +
> + result = True
> + except TypeError:
> + # version comparison is a very tricky
> + # sometimes provider changes product version in a strange manner
> + # and the above comparison just failed
> + # so here we try to make version string "more standard"
> +
> + if match_version(twik_version(version),
> + [twik_version(v) for v in version_range]):
> + logging.debug("match %s %s (twiked): %s", product, twik_version(version),
> + cpe["cpe23Uri"])
> + affected.add("{},{}".format(product, version))
> +
> + result = True
> +
> + return result
> +
> +
> +def match_version(version, vrange):
> + """Match version with the version range"""
> +
> + result = False
> + version = util_version(version)
> +
> + if "versionExactMatch" in vrange:
> + if version == util_version(vrange["versionExactMatch"]):
> + result = True
> + else:
> + result = True
> +
> + if "versionStartIncluding" in vrange:
> + result = result and version >= util_version(vrange["versionStartIncluding"])
> +
> + if "versionStartExcluding" in vrange:
> + result = result and version > util_version(vrange["versionStartExcluding"])
> +
> + if "versionEndIncluding" in vrange:
> + result = result and version <= util_version(vrange["versionEndIncluding"])
> +
> + if "versionEndExcluding" in vrange:
> + result = result and version < util_version(vrange["versionEndExcluding"])
> +
> + return result
> +
> +
> +def util_version(version):
> + """Simplify package version"""
> + return distutils.version.LooseVersion(version.split("+git")[0])
> +
> +
> +def twik_version(version):
> + """Return "standard" version for complex cases"""
> + return "v1" + re.sub(r"^[a-zA-Z]+", "", version)
> +
> +
> +def print_report(report, width=70, show_description=False, show_reference=False, output=sys.stdout):
> + """Print out final report"""
> +
> + for cve in report:
> + print("{0:>9s} | {1:>4s} | {2:18s} | {3} | {4}".format(cve["status"], cve["CVSS"],
> + cve["CVE"], cve["product"],
> + cve["version"]),
> + file=output)
> +
> + if show_description:
> + print("{0:>9s} + {1}".format(" ", "Description"), file=output)
> +
> + for lin in textwrap.wrap(cve["description"], width=width):
> + print("{0:>9s} {1}".format(" ", lin), file=output)
> +
> + if show_reference:
> + print("{0:>9s} + {1}".format(" ", "Reference"), file=output)
> +
> + for url in cve["reference"]:
> + print("{0:>9s} {1}".format(" ", url), file=output)
> +
> +
> +def update_feeds(feed_dir, offline=False, start=2002):
> + """Update all JSON feeds"""
> +
> + feed_dir = os.path.realpath(feed_dir)
> + year_now = datetime.datetime.now().year
> + cve_struct = {}
> +
> + for year in range(start, year_now + 1):
> + update_year(cve_struct, year, feed_dir, offline)
> +
> + return cve_struct
> +
> +
> +def update_year(cve_struct, year, feed_dir, offline):
> + """Update one JSON feed for the particular year"""
> +
> + url_prefix = "https://static.nvd.nist.gov/feeds/json/cve/1.0"
> + file_prefix = "nvdcve-1.0-{0}".format(year)
> +
> + meta = {
> + "url": "{0}/{1}.meta".format(url_prefix, file_prefix),
> + "file": os.path.join(feed_dir, "{0}.meta".format(file_prefix))
> + }
> +
> + feed = {
> + "url": "{0}/{1}.json.gz".format(url_prefix, file_prefix),
> + "file": os.path.join(feed_dir, "{0}.json.gz".format(file_prefix))
> + }
> +
> + ctx = {}
> +
> + if not offline:
> + ctx = download_feed(meta, feed)
> +
> + if not "meta" in ctx or not "feed" in ctx:
> + return
> +
> + if not os.path.isfile(meta["file"]):
> + return
> +
> + if not os.path.isfile(feed["file"]):
> + return
> +
> + if not "meta" in ctx:
> + ctx["meta"] = ctx_meta(meta["file"])
> +
> + if not "sha256" in ctx["meta"]:
> + return
> +
> + if not "feed" in ctx:
> + ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"])
> +
> + if not ctx["feed"]:
> + return
> +
> + logging.debug("parsing year %s", year)
> +
> + for cve_item in ctx["feed"]["CVE_Items"]:
> + iden, cve = parse_item(cve_item)
> +
> + if not iden:
> + continue
> +
> + if not cve:
> + logging.error("%s parse error", iden)
> + break
> +
> + if iden in cve_struct:
> + logging.error("%s duplicated", iden)
> + break
> +
> + cve_struct[iden] = cve
> +
> + logging.debug("cve records: %d", len(cve_struct))
> +
> +
> +def ctx_meta(filename):
> + """Parse feed meta file"""
> +
> + if not os.path.isfile(filename):
> + return {}
> +
> + ctx = {}
> +
> + with open(filename) as fil:
> + for lin in fil:
> + pair = lin.split(":", maxsplit=1)
> + ctx[pair[0]] = pair[1].rstrip()
> +
> + return ctx
> +
> +
> +def ctx_gzip(filename, checksum=""):
> + """Parse feed archive file"""
> +
> + if not os.path.isfile(filename):
> + return {}
> +
> + with gzip.open(filename) as fil:
> + try:
> + ctx = fil.read()
> + except (EOFError, OSError):
> + logging.error("failed to process gz archive %s", filename, exc_info=True)
> + return {}
> +
> + if checksum and checksum.upper() != hashlib.sha256(ctx).hexdigest().upper():
> + return {}
> +
> + return json.loads(ctx.decode())
> +
> +
> +def parse_item(cve_item):
> + """Parse one JSON CVE entry"""
> +
> + cve_id = cve_item["cve"]["CVE_data_meta"]["ID"][:]
> + impact = cve_item["impact"]
> +
> + if not impact:
> + # REJECTed CVE
> + return None, None
> +
> + if "baseMetricV3" in impact:
> + score = impact["baseMetricV3"]["cvssV3"]["baseScore"]
> + elif "baseMetricV2" in impact:
> + score = impact["baseMetricV2"]["cvssV2"]["baseScore"]
> + else:
> + return cve_id, None
> +
> + return cve_id, {
> + "score": score,
> + "nodes": cve_item["configurations"]["nodes"][:],
> + "reference": cve_item["cve"]["references"]["reference_data"][:],
> + "description": cve_item["cve"]["description"]["description_data"][0]["value"]
> + }
> +
> +
> +def download_feed(meta, feed):
> + """Download and parse feed"""
> +
> + ctx = {}
> +
> + if not retrieve_url(meta["url"], meta["file"]):
> + return {}
> +
> + ctx["meta"] = ctx_meta(meta["file"])
> +
> + if not "sha256" in ctx["meta"]:
> + return {}
> +
> + ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"])
> +
> + if not ctx["feed"]:
> + if not retrieve_url(feed["url"], feed["file"]):
> + return {}
> +
> + ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"])
> +
> + return ctx
> +
> +
> +def retrieve_url(url, filename=None):
> + """Download file by URL"""
> +
> + if filename:
> + os.makedirs(os.path.dirname(filename), exist_ok=True)
> +
> + logging.debug("downloading %s", url)
> +
> + try:
> + urllib.request.urlretrieve(url, filename=filename)
> + except urllib.error.HTTPError:
> + logging.error("failed to download URL %s", url, exc_info=True)
> + return False
> +
> + return True
> +
> +
> +def logconfig(debug_flag=False):
> + """Return default log config"""
> +
> + return {
> + "version": 1,
> + "formatters": {
> + "f": {
> + "format": "# %(asctime)s %% CVERT %% %(levelname)-8s %% %(message)s"
> + }
> + },
> + "handlers": {
> + "h": {
> + "class": "logging.StreamHandler",
> + "formatter": "f",
> + "level": logging.DEBUG if debug_flag else logging.INFO
> + }
> + },
> + "root": {
> + "handlers": ["h"],
> + "level": logging.DEBUG if debug_flag else logging.INFO
> + },
> + }
> +
> +
> +def save_cve(filename, cve_struct):
> + """Save CVE structure in the file"""
> +
> + filename = os.path.realpath(filename)
> +
> + logging.debug("saving %d CVE records to %s", len(cve_struct), filename)
> +
> + with open(filename, "wb") as fil:
> + pickle.dump(cve_struct, fil)
> +
> +
> +def load_cve(filename):
> + """Load CVE structure from the file"""
> +
> + filename = os.path.realpath(filename)
> +
> + logging.debug("loading from %s", filename)
> +
> + with open(filename, "rb") as fil:
> + cve_struct = pickle.load(fil)
> +
> + logging.debug("cve records: %d", len(cve_struct))
> +
> + return cve_struct
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v3 1/3] cve-report: add scripts to generate CVE reports
2018-10-29 23:29 ` akuster808
@ 2018-10-30 2:59 ` Grygorii Tertychnyi
0 siblings, 0 replies; 3+ messages in thread
From: Grygorii Tertychnyi @ 2018-10-30 2:59 UTC (permalink / raw)
To: akuster808; +Cc: xe-linux-external, openembedded-core
On Mon Oct29 2018 @ 23:29, akuster808 <akuster808@gmail.com>
wrote:
> Grygorii,
>
> I was good to see you at OEDeM.
Thanks Armin.
> I have some feedback.
>
> On 10/10/18 9:25 AM, grygorii tertychnyi via Openembedded-core
> wrote:
>> cvert-foss - generate CVE report for the list of packages.
>> Analyze the whole image manifest to align with the complex
>> CPE configurations.
>>
>> cvert-update - update NVD feeds and store CVE structues dump.
>> CVE dump is a pickled representation of the cve_struct
>> dictionary.
>>
>> cvert.py - python library used by cvert-* scripts.
>> NVD JSON Vulnerability Feeds
>> https://nvd.nist.gov/vuln/data-feeds#JSON_FEED
>>
>> Usage examples:
>>
>> o Download CVE feeds to "nvdfeed" directory
>> % cvert-update nvdfeed
>> o Update CVE feeds and store a dump in a file
>> % cvert-update --store cvedump nvdfeed
>> o Generate a CVE report
>> % cvert-foss --feed-dir nvdfeed --output report-foss.txt
>> cve-manifest
>> o (faster) Use dump file to generate a CVE report
>> % cvert-foss --restore cvedump --output report-foss.txt
>> cve-manifest
>> o Generate a full report
>> % cvert-foss --restore cvedump --show-description
>> --show-reference \
>> --output report-foss-full.txt cve-manifest
>> report-foss-full.txt
Oh, there is a typo. "cvert-foss" script requires only one
positional argument - "cve-manifest".
So, the "Generate a full report" command should look like this:
% cvert-foss --restore cvedump --show-description
--show-reference \
--output report-foss-full.txt cve-manifest
Is not a big deal. Just to be on the same page.
>
> The cve-manifest, I could not figure out a way to create this
> from the
> above steps. I had your patches included when I did an image
> build then
> I got the "report-foss-full.txt" created. I am guessing I missed
> some
> steps somewhere. Can you clarify?
I try to explain it this way:
There are three patches:
o 1st is "cve-report: add scripts to generate CVE reports"
"cvert-foss" takes the "cve-manifest" (input), and generates
the CVE report (output). So, you are right, these scripts
do not produce "cve-manifest" files.
o 2nd is "cvert-kernel - generate CVE report for the Linux
kernel"
"cvert-kernel" analyzes kernel git dir and generate the kernel
CVE report
o 3rd is "cve-report.bbclass: add class"
it generates "cve-manifest" file for the given <image> and
then
calls "cvert-foss" to produce the CVE report.
So, if you applied all three patches, modified "local.conf" and
run "bitbake -c report_cve <image>" the "cve-manifest" is
generated as part of the "generate_report_handler" function of
"cve-report.bbclass".
So, back to you question "how to generate cve-manifest", the
easiest way is:
echo 'INHERIT += "cve-report"' >> conf/local.conf
echo 'CVE_REPORT_MODE[packageonly] = "1"' >> conf/local.conf
bitbake -c report_cve <image>
It is described in the 3rd patch. Here, in the 1st patch I
described only the "cve-manifest" format. Same format is used by
"cve-check-tool", it is well-known and simply enough, so, no need
to change it.
Being a part of scripts directory it can be used outside of any
bbclass. You can create simple manifests and generate CVE reports
right on command line, simple example:
% echo "python,3.5.5,CVE-2017-17522 CVE-2018-1061" >
cve-manifest.txt
% ./cvert-foss --debug --feed-dir cve-feeds --output
report-foss.txt cve-manifest.txt
# 2018-10-29 19:27:48,673 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.meta
# 2018-10-29 19:27:50,250 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.json.gz
# 2018-10-29 19:27:53,537 % CVERT % DEBUG % parsing year 2002
# 2018-10-29 19:27:53,575 % CVERT % DEBUG % cve records: 6667
# 2018-10-29 19:27:53,605 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2003.meta
# 2018-10-29 19:27:55,121 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2003.json.gz
# 2018-10-29 19:27:57,360 % CVERT % DEBUG % parsing year 2003
# 2018-10-29 19:27:57,369 % CVERT % DEBUG % cve records: 8167
# 2018-10-29 19:27:57,382 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2004.meta
# 2018-10-29 19:27:58,825 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2004.json.gz
# 2018-10-29 19:28:01,595 % CVERT % DEBUG % parsing year 2004
# 2018-10-29 19:28:01,613 % CVERT % DEBUG % cve records: 10810
# 2018-10-29 19:28:01,636 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2005.meta
# 2018-10-29 19:28:03,008 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2005.json.gz
# 2018-10-29 19:28:06,088 % CVERT % DEBUG % parsing year 2005
# 2018-10-29 19:28:06,119 % CVERT % DEBUG % cve records: 15424
# 2018-10-29 19:28:06,152 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2006.meta
# 2018-10-29 19:28:07,627 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2006.json.gz
# 2018-10-29 19:28:11,392 % CVERT % DEBUG % parsing year 2006
# 2018-10-29 19:28:11,439 % CVERT % DEBUG % cve records: 22408
# 2018-10-29 19:28:11,489 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2007.meta
# 2018-10-29 19:28:13,163 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2007.json.gz
# 2018-10-29 19:28:17,432 % CVERT % DEBUG % parsing year 2007
# 2018-10-29 19:28:17,478 % CVERT % DEBUG % cve records: 28850
# 2018-10-29 19:28:17,530 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2008.meta
# 2018-10-29 19:28:19,195 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2008.json.gz
# 2018-10-29 19:28:22,937 % CVERT % DEBUG % parsing year 2008
# 2018-10-29 19:28:23,489 % CVERT % DEBUG % cve records: 35840
# 2018-10-29 19:28:23,556 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2009.meta
# 2018-10-29 19:28:25,127 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2009.json.gz
# 2018-10-29 19:28:29,013 % CVERT % DEBUG % parsing year 2009
# 2018-10-29 19:28:29,049 % CVERT % DEBUG % cve records: 40705
# 2018-10-29 19:28:29,104 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2010.meta
# 2018-10-29 19:28:31,082 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2010.json.gz
# 2018-10-29 19:28:35,571 % CVERT % DEBUG % parsing year 2010
# 2018-10-29 19:28:35,609 % CVERT % DEBUG % cve records: 45638
# 2018-10-29 19:28:35,677 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2011.meta
# 2018-10-29 19:28:37,263 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2011.json.gz
# 2018-10-29 19:28:43,598 % CVERT % DEBUG % parsing year 2011
# 2018-10-29 19:28:43,630 % CVERT % DEBUG % cve records: 50043
# 2018-10-29 19:28:43,739 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2012.meta
# 2018-10-29 19:28:45,264 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2012.json.gz
# 2018-10-29 19:28:49,718 % CVERT % DEBUG % parsing year 2012
# 2018-10-29 19:28:49,755 % CVERT % DEBUG % cve records: 55215
# 2018-10-29 19:28:49,830 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2013.meta
# 2018-10-29 19:28:51,455 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2013.json.gz
# 2018-10-29 19:28:56,101 % CVERT % DEBUG % parsing year 2013
# 2018-10-29 19:28:56,142 % CVERT % DEBUG % cve records: 60882
# 2018-10-29 19:28:56,210 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2014.meta
# 2018-10-29 19:28:57,977 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2014.json.gz
# 2018-10-29 19:29:02,677 % CVERT % DEBUG % parsing year 2014
# 2018-10-29 19:29:03,693 % CVERT % DEBUG % cve records: 68824
# 2018-10-29 19:29:03,783 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2015.meta
# 2018-10-29 19:29:05,820 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2015.json.gz
# 2018-10-29 19:29:09,577 % CVERT % DEBUG % parsing year 2015
# 2018-10-29 19:29:09,632 % CVERT % DEBUG % cve records: 76251
# 2018-10-29 19:29:09,720 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2016.meta
# 2018-10-29 19:29:11,362 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2016.json.gz
# 2018-10-29 19:29:17,063 % CVERT % DEBUG % parsing year 2016
# 2018-10-29 19:29:17,128 % CVERT % DEBUG % cve records: 84957
# 2018-10-29 19:29:17,238 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2017.meta
# 2018-10-29 19:29:19,102 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2017.json.gz
# 2018-10-29 19:29:25,282 % CVERT % DEBUG % parsing year 2017
# 2018-10-29 19:29:25,377 % CVERT % DEBUG % cve records: 98334
# 2018-10-29 19:29:25,564 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2018.meta
# 2018-10-29 19:29:27,195 % CVERT % DEBUG % downloading
https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2018.json.gz
# 2018-10-29 19:29:32,262 % CVERT % DEBUG % parsing year 2018
# 2018-10-29 19:29:32,323 % CVERT % DEBUG % cve records: 106349
# 2018-10-29 19:29:32,912 % CVERT % DEBUG % match python 3.5.5:
cpe:2.3:a:python:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:33,265 % CVERT % DEBUG % ignore "*" in
cpe:2.3:a:python_software_foundation:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:33,271 % CVERT % DEBUG % match python 3.5.5:
cpe:2.3:a:python:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:33,271 % CVERT % DEBUG % unpatched
CVE-2017-18207 python,3.5.5
# 2018-10-29 19:29:33,486 % CVERT % DEBUG % match python 3.5.5:
cpe:2.3:a:python:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:33,487 % CVERT % DEBUG % patched
CVE-2018-1061 python,3.5.5
# 2018-10-29 19:29:33,772 % CVERT % DEBUG % ignore "-" in
cpe:2.3:a:python:python:-:*:*:*:*:*:*:*
# 2018-10-29 19:29:34,585 % CVERT % DEBUG % match python 3.5.5:
cpe:2.3:a:python:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:34,585 % CVERT % DEBUG % unpatched
CVE-2018-1060 python,3.5.5
# 2018-10-29 19:29:36,668 % CVERT % DEBUG % match python 3.5.5:
cpe:2.3:a:python:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:36,668 % CVERT % DEBUG % patched
CVE-2017-17522 python,3.5.5
% cat report-foss.txt
patched | 7.5 | CVE-2018-1061 | python | 3.5.5
patched | 8.8 | CVE-2017-17522 | python | 3.5.5
unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5
unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5
Of course, it is better to use first "cvert-update --store ..."
and then multiple times "cvert-foss --restore ..." - much faster
on multiple runs.
> Also, is there a way to version the report file by image or
> standard
> buildID so that I don't accidentally overwrite the txt
> file. Maybe a
> future enhancement is to group the packages by layer. I ran this
> with
> the meta-security layer and I now need to look up what packages
> belong
> to what layer to go work on them.
I think it is possible, but again as part of bbclass
enhancement. If I understood Richard correctly on OEDeM, the
bbclass should be modified anyway ("do not copy
functionality"). But these cvert scripts do not depend on it. And
can be seen independently.
> Overall, I like this implementation and seems to be fast.
>
> Thanks for the patches.
Thanks. It is inspiring.
> kind regards,
>
> Armin
>
>>
>> Manifest example:
>>
>> bash,4.2,CVE-2014-7187
>> python,2.7.35,
>> python,3.5.5,CVE-2017-17522 CVE-2018-1061
>>
>> Report example:
>>
>> patched | 7.5 | CVE-2018-1061 | python | 3.5.5
>> patched | 10.0 | CVE-2014-7187 | bash | 4.2
>> patched | 8.8 | CVE-2017-17522 | python | 3.5.5
>> unpatched | 10.0 | CVE-2014-6271 | bash | 4.2
>> unpatched | 10.0 | CVE-2014-6277 | bash | 4.2
>> unpatched | 10.0 | CVE-2014-6278 | bash | 4.2
>> unpatched | 10.0 | CVE-2014-7169 | bash | 4.2
>> unpatched | 10.0 | CVE-2014-7186 | bash | 4.2
>> unpatched | 4.6 | CVE-2012-3410 | bash | 4.2
>> unpatched | 8.4 | CVE-2016-7543 | bash | 4.2
>> unpatched | 5.0 | CVE-2010-3492 | python | 2.7.35
>> unpatched | 5.3 | CVE-2016-1494 | python | 2.7.35
>> unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5
>> unpatched | 6.5 | CVE-2017-18207 | python | 2.7.35
>> unpatched | 7.1 | CVE-2013-7338 | python | 2.7.35
>> unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5
>> unpatched | 8.8 | CVE-2017-17522 | python | 2.7.35
>>
>> Signed-off-by: grygorii tertychnyi <gtertych@cisco.com>
>> ---
>>
>> Changes in v3:
>> o better logging: cvert.py lib log messages are controlled by
>> cvert-* scripts
>> o add more examples
>> o add short params ("-o" "--output", "-f", "--feed-dir", etc)
>> o fix double entries in manifest
>> o fix pylint warnings
>>
>> scripts/cvert-foss | 151 ++++++++++++++++
>> scripts/cvert-update | 79 +++++++++
>> scripts/cvert.py | 473
>> +++++++++++++++++++++++++++++++++++++++++++++++++++
>> 3 files changed, 703 insertions(+)
>> create mode 100755 scripts/cvert-foss
>> create mode 100755 scripts/cvert-update
>> create mode 100644 scripts/cvert.py
>>
>> diff --git a/scripts/cvert-foss b/scripts/cvert-foss
>> new file mode 100755
>> index 000000000000..00fbf2c0687b
>> --- /dev/null
>> +++ b/scripts/cvert-foss
>> @@ -0,0 +1,151 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Copyright (c) 2018 by Cisco Systems, Inc.
>> +#
>> +# This program is free software; you can redistribute it
>> and/or modify
>> +# it under the terms of the GNU General Public License version
>> 2 as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it will be
>> useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty
>> of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
>> the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public
>> License along
>> +# with this program; if not, write to the Free Software
>> Foundation, Inc.,
>> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>> +#
>> +
>> +""" Generate CVE report for the given CVE manifest
>> +"""
>> +
>> +import sys
>> +import textwrap
>> +import argparse
>> +import logging
>> +import logging.config
>> +import cvert
>> +
>> +def report_foss():
>> + """Generate CVE report"""
>> +
>> + parser = argparse.ArgumentParser(
>> + formatter_class=argparse.RawDescriptionHelpFormatter,
>> + description=textwrap.dedent("""
>> + Generate CVE report for the given CVE manifest.
>> + """),
>> + epilog=textwrap.dedent("""
>> + @ run examples:
>> +
>> + # Download (update) NVD feeds in "nvdfeed" directory
>> + # and prepare the report for the "cve-manifest" file
>> + %% %(prog)s --feed-dir nvdfeed --output
>> report-foss.txt cve-manifest
>> +
>> + # Use existed NVD feeds in "nvdfeed" directory
>> + # and prepare the report for the "cve-manifest" file
>> + %% %(prog)s --offline --feed-dir nvdfeed --output
>> report-foss.txt cve-manifest
>> +
>> + # (faster) Restore CVE dump from "cvedump" (must
>> exist)
>> + # and prepare the report for the "cve-manifest" file
>> + %% %(prog)s --restore cvedump --output report-foss.txt
>> cve-manifest
>> +
>> + # Restore CVE dump from "cvedump" (must exist)
>> + # and prepare the extended report for the
>> "cve-manifest" file
>> + %% %(prog)s --restore cvedump --show-description
>> --show-reference --output report-foss.txt cve-manifest
>> +
>> + @ manifest example:
>> +
>> + bash,4.2,CVE-2014-7187
>> + python,2.7.35,
>> + python,3.5.5,CVE-2017-17522 CVE-2018-1061
>> +
>> + @ report example output:
>> +
>> + . patched | 10.0 | CVE-2014-7187 | bash | 4.2
>> + . patched | 7.5 | CVE-2018-1061 | python | 3.5.5
>> + . patched | 8.8 | CVE-2017-17522 | python | 3.5.5
>> + unpatched | 10.0 | CVE-2014-6271 | bash | 4.2
>> + unpatched | 10.0 | CVE-2014-6277 | bash | 4.2
>> + unpatched | 10.0 | CVE-2014-6278 | bash | 4.2
>> + unpatched | 10.0 | CVE-2014-7169 | bash | 4.2
>> + unpatched | 10.0 | CVE-2014-7186 | bash | 4.2
>> + unpatched | 4.6 | CVE-2012-3410 | bash | 4.2
>> + unpatched | 8.4 | CVE-2016-7543 | bash | 4.2
>> + unpatched | 5.0 | CVE-2010-3492 | python |
>> 2.7.35
>> + unpatched | 5.3 | CVE-2016-1494 | python |
>> 2.7.35
>> + unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5
>> + unpatched | 6.5 | CVE-2017-18207 | python |
>> 2.7.35
>> + unpatched | 7.1 | CVE-2013-7338 | python |
>> 2.7.35
>> + unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5
>> + unpatched | 8.8 | CVE-2017-17522 | python |
>> 2.7.35
>> + """))
>> +
>> + group = parser.add_mutually_exclusive_group(required=True)
>> + group.add_argument("-f", "--feed-dir", help="feeds
>> directory")
>> + group.add_argument("-d", "--restore", help="load CVE data
>> structures from file",
>> + metavar="FILENAME")
>> + parser.add_argument("--offline", help="do not update from
>> NVD site",
>> + action="store_true")
>> + parser.add_argument("-o", "--output", help="save report to
>> the file")
>> + parser.add_argument("--show-description", help='show
>> "Description" in the report',
>> + action="store_true")
>> + parser.add_argument("--show-reference", help='show
>> "Reference" in the report',
>> + action="store_true")
>> + parser.add_argument("--debug", help="print debug
>> messages",
>> + action="store_true")
>> +
>> + parser.add_argument("cve_manifest", help="file with a list
>> of packages, "
>> + "each line contains three comma
>> separated values: name, "
>> + "version and a space separated list of
>> patched CVEs, "
>> + "e.g.: python,3.5.5,CVE-2017-17522
>> CVE-2018-1061",
>> + metavar="cve-manifest")
>> +
>> + args = parser.parse_args()
>> +
>> + logging.config.dictConfig(cvert.logconfig(args.debug))
>> +
>> + cve_manifest = {}
>> +
>> + with open(args.cve_manifest, "r") as fil:
>> + for lin in fil:
>> + lin = lin.rstrip()
>> +
>> + # skip empty lines
>> + if not lin:
>> + continue
>> +
>> + product, version, patched = lin.split(",",
>> maxsplit=3)
>> +
>> + if product in cve_manifest:
>> + cve_manifest[product][version] =
>> patched.split()
>> + else:
>> + cve_manifest[product] = {
>> + version: patched.split()
>> + }
>> +
>> + if args.restore:
>> + cve_struct = cvert.load_cve(args.restore)
>> + elif args.feed_dir:
>> + cve_struct = cvert.update_feeds(args.feed_dir,
>> args.offline)
>> +
>> + if not cve_struct and args.offline:
>> + parser.error("No CVEs found. Try to turn off offline
>> mode or use other file to restore.")
>> +
>> + if args.output:
>> + output = open(args.output, "w")
>> + else:
>> + output = sys.stdout
>> +
>> + report = cvert.generate_report(cve_manifest, cve_struct)
>> +
>> + cvert.print_report(report,
>> + show_description=args.show_description,
>> + show_reference=args.show_reference,
>> + output=output)
>> +
>> + if args.output:
>> + output.close()
>> +
>> +
>> +if __name__ == "__main__":
>> + report_foss()
>> diff --git a/scripts/cvert-update b/scripts/cvert-update
>> new file mode 100755
>> index 000000000000..3b3f5572a83c
>> --- /dev/null
>> +++ b/scripts/cvert-update
>> @@ -0,0 +1,79 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Copyright (c) 2018 by Cisco Systems, Inc.
>> +#
>> +# This program is free software; you can redistribute it
>> and/or modify
>> +# it under the terms of the GNU General Public License version
>> 2 as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it will be
>> useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty
>> of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
>> the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public
>> License along
>> +# with this program; if not, write to the Free Software
>> Foundation, Inc.,
>> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>> +#
>> +
>> +""" Update NVD feeds and store CVE blob locally
>> +"""
>> +
>> +
>> +import textwrap
>> +import argparse
>> +import logging
>> +import logging.config
>> +import cvert
>> +
>> +
>> +def update_cvert():
>> + """Update CVE storage"""
>> +
>> + parser = argparse.ArgumentParser(
>> + formatter_class=argparse.RawDescriptionHelpFormatter,
>> + description=textwrap.dedent("""
>> + Update NVD feeds and store CVE blob locally.
>> + """),
>> + epilog=textwrap.dedent("""
>> + examples:
>> +
>> + # Download NVD feeds to "nvdfeed" directory.
>> + # If there are meta files in the directory, they will
>> be updated
>> + # and only fresh archives will be downloaded
>> + %% %(prog)s nvdfeed
>> +
>> + # Inspect NVD feeds in "nvdfeed" directory
>> + # and prepare a CVE dump python blob "cvedump".
>> + # Use it later as input for cvert-* scripts (for
>> speeding up)
>> + %% %(prog)s --offline --store cvedump nvdfeed
>> +
>> + # Download (update) NVD feeds and prepare the CVE dump
>> + %% %(prog)s --store cvedump nvdfeed
>> + """))
>> +
>> + parser.add_argument("-d", "--store", help="save CVE data
>> structures in file",
>> + metavar="FILENAME")
>> + parser.add_argument("--offline", help="do not update from
>> NVD site",
>> + action="store_true")
>> + parser.add_argument("--debug", help="print debug
>> messages",
>> + action="store_true")
>> +
>> + parser.add_argument("feed_dir", help="feeds directory",
>> + metavar="feed-dir")
>> +
>> + args = parser.parse_args()
>> +
>> + logging.config.dictConfig(cvert.logconfig(args.debug))
>> +
>> + cve_struct = cvert.update_feeds(args.feed_dir,
>> args.offline)
>> +
>> + if not cve_struct and args.offline:
>> + parser.error("No CVEs found in {0}. Try turn off
>> offline mode.".format(args.feed_dir))
>> +
>> + if args.store:
>> + cvert.save_cve(args.store, cve_struct)
>> +
>> +
>> +if __name__ == "__main__":
>> + update_cvert()
>> diff --git a/scripts/cvert.py b/scripts/cvert.py
>> new file mode 100644
>> index 000000000000..f93b95c84965
>> --- /dev/null
>> +++ b/scripts/cvert.py
>> @@ -0,0 +1,473 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Copyright (c) 2018 by Cisco Systems, Inc.
>> +#
>> +# This program is free software; you can redistribute it
>> and/or modify
>> +# it under the terms of the GNU General Public License version
>> 2 as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it will be
>> useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty
>> of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
>> the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public
>> License along
>> +# with this program; if not, write to the Free Software
>> Foundation, Inc.,
>> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>> +#
>> +
>> +""" CVERT library: set of functions for CVE reports
>> +"""
>> +
>> +
>> +import os
>> +import re
>> +import sys
>> +import json
>> +import gzip
>> +import pickle
>> +import logging
>> +import hashlib
>> +import datetime
>> +import textwrap
>> +import urllib.request
>> +import distutils.version
>> +
>> +
>> +logging.getLogger(__name__).addHandler(logging.NullHandler())
>> +
>> +
>> +def generate_report(manifest, cve_struct):
>> + """Generate CVE report"""
>> +
>> + report = []
>> +
>> + for cve in cve_struct:
>> + affected = set()
>> +
>> + for conf in cve_struct[cve]["nodes"]:
>> + affected =
>> affected.union(process_configuration(manifest, conf))
>> +
>> + for key in affected:
>> + product, version = key.split(",")
>> + patched = manifest[product][version]
>> +
>> + if cve in patched:
>> + cve_item = {"status": "patched"}
>> + else:
>> + cve_item = {"status": "unpatched"}
>> +
>> + cve_item["CVSS"] =
>> "{0:.1f}".format(cve_struct[cve]["score"])
>> + cve_item["CVE"] = cve
>> + cve_item["product"] = product
>> + cve_item["version"] = version
>> + cve_item["description"] =
>> cve_struct[cve]["description"]
>> + cve_item["reference"] = [x["url"] for x in
>> cve_struct[cve]["reference"]]
>> +
>> + logging.debug("%9s %s %s,%s",
>> + cve_item["status"], cve_item["CVE"],
>> + cve_item["product"],
>> cve_item["version"])
>> +
>> + report.append(cve_item)
>> +
>> + return sorted(report, key=lambda x: (x["status"],
>> x["product"], x["CVSS"], x["CVE"]))
>> +
>> +
>> +def process_configuration(manifest, conf):
>> + """Recursive call to process all CVE configurations"""
>> +
>> + operator = conf["operator"]
>> +
>> + if operator not in ["OR", "AND"]:
>> + raise ValueError("operator {} is not
>> supported".format(operator))
>> +
>> + operator = True if operator == "AND" else False
>> + match = False
>> + affected = set()
>> +
>> + if "cpe" in conf:
>> + match = process_cpe(manifest, conf["cpe"][0],
>> affected)
>> +
>> + for cpe in conf["cpe"][1:]:
>> + package_match = process_cpe(manifest, cpe,
>> affected)
>> +
>> + # match = match <operator> package_match
>> + match = operator ^ ((operator ^ match) or
>> (operator ^ package_match))
>> + elif "children" in conf:
>> + product_set = process_configuration(manifest,
>> conf["children"][0])
>> +
>> + if product_set:
>> + match = True
>> + affected = affected.union(product_set)
>> +
>> + for child in conf["children"][1:]:
>> + product_set = process_configuration(manifest,
>> child)
>> + package_match = True if product_set else False
>> +
>> + # match = match OP package_match
>> + match = operator ^ ((operator ^ match) or
>> (operator ^ package_match))
>> +
>> + if package_match:
>> + affected = affected.union(product_set)
>> +
>> + if match:
>> + return affected
>> +
>> + return ()
>> +
>> +
>> +def process_cpe(manifest, cpe, affected):
>> + """Match CPE with all manifest packages"""
>> +
>> + if not cpe["vulnerable"]:
>> + # ignore non vulnerable part
>> + return False
>> +
>> + version_range = {}
>> +
>> + for flag in ["versionStartIncluding",
>> + "versionStartExcluding",
>> + "versionEndIncluding",
>> + "versionEndExcluding"]:
>> + if flag in cpe:
>> + version_range[flag] = cpe[flag]
>> +
>> + # take only "product" and "version"
>> + product, version = cpe["cpe23Uri"].split(":")[4:6]
>> +
>> + if product not in manifest:
>> + return False
>> +
>> + if not version_range:
>> + if version == "*":
>> + # ignore CVEs that touches all versions of
>> package,
>> + # can not fix it anyway
>> + logging.debug('ignore "*" in %s', cpe["cpe23Uri"])
>> + return False
>> + elif version == "-":
>> + # "-" means NA
>> + #
>> + # NA (i.e. "not applicable/not used"). The logical
>> value NA
>> + # SHOULD be assigned when there is no legal or
>> meaningful
>> + # value for that attribute, or when that attribute
>> is not
>> + # used as part of the description.
>> + # This includes the situation in which an
>> attribute has
>> + # an obtainable value that is null
>> + #
>> + # Ignores CVEs if version is not set
>> + logging.debug('ignore "-" in %s', cpe["cpe23Uri"])
>> + return False
>> + else:
>> + version_range["versionExactMatch"] = version
>> +
>> + result = False
>> +
>> + for version in manifest[product]:
>> + try:
>> + if match_version(version,
>> + version_range):
>> + logging.debug("match %s %s: %s", product,
>> version, cpe["cpe23Uri"])
>> + affected.add("{},{}".format(product, version))
>> +
>> + result = True
>> + except TypeError:
>> + # version comparison is a very tricky
>> + # sometimes provider changes product version in a
>> strange manner
>> + # and the above comparison just failed
>> + # so here we try to make version string "more
>> standard"
>> +
>> + if match_version(twik_version(version),
>> + [twik_version(v) for v in
>> version_range]):
>> + logging.debug("match %s %s (twiked): %s",
>> product, twik_version(version),
>> + cpe["cpe23Uri"])
>> + affected.add("{},{}".format(product, version))
>> +
>> + result = True
>> +
>> + return result
>> +
>> +
>> +def match_version(version, vrange):
>> + """Match version with the version range"""
>> +
>> + result = False
>> + version = util_version(version)
>> +
>> + if "versionExactMatch" in vrange:
>> + if version ==
>> util_version(vrange["versionExactMatch"]):
>> + result = True
>> + else:
>> + result = True
>> +
>> + if "versionStartIncluding" in vrange:
>> + result = result and version >=
>> util_version(vrange["versionStartIncluding"])
>> +
>> + if "versionStartExcluding" in vrange:
>> + result = result and version >
>> util_version(vrange["versionStartExcluding"])
>> +
>> + if "versionEndIncluding" in vrange:
>> + result = result and version <=
>> util_version(vrange["versionEndIncluding"])
>> +
>> + if "versionEndExcluding" in vrange:
>> + result = result and version <
>> util_version(vrange["versionEndExcluding"])
>> +
>> + return result
>> +
>> +
>> +def util_version(version):
>> + """Simplify package version"""
>> + return
>> distutils.version.LooseVersion(version.split("+git")[0])
>> +
>> +
>> +def twik_version(version):
>> + """Return "standard" version for complex cases"""
>> + return "v1" + re.sub(r"^[a-zA-Z]+", "", version)
>> +
>> +
>> +def print_report(report, width=70, show_description=False,
>> show_reference=False, output=sys.stdout):
>> + """Print out final report"""
>> +
>> + for cve in report:
>> + print("{0:>9s} | {1:>4s} | {2:18s} | {3} |
>> {4}".format(cve["status"], cve["CVSS"],
>> +
>> cve["CVE"], cve["product"],
>> +
>> cve["version"]),
>> + file=output)
>> +
>> + if show_description:
>> + print("{0:>9s} + {1}".format(" ", "Description"),
>> file=output)
>> +
>> + for lin in textwrap.wrap(cve["description"],
>> width=width):
>> + print("{0:>9s} {1}".format(" ", lin),
>> file=output)
>> +
>> + if show_reference:
>> + print("{0:>9s} + {1}".format(" ", "Reference"),
>> file=output)
>> +
>> + for url in cve["reference"]:
>> + print("{0:>9s} {1}".format(" ", url),
>> file=output)
>> +
>> +
>> +def update_feeds(feed_dir, offline=False, start=2002):
>> + """Update all JSON feeds"""
>> +
>> + feed_dir = os.path.realpath(feed_dir)
>> + year_now = datetime.datetime.now().year
>> + cve_struct = {}
>> +
>> + for year in range(start, year_now + 1):
>> + update_year(cve_struct, year, feed_dir, offline)
>> +
>> + return cve_struct
>> +
>> +
>> +def update_year(cve_struct, year, feed_dir, offline):
>> + """Update one JSON feed for the particular year"""
>> +
>> + url_prefix =
>> "https://static.nvd.nist.gov/feeds/json/cve/1.0"
>> + file_prefix = "nvdcve-1.0-{0}".format(year)
>> +
>> + meta = {
>> + "url": "{0}/{1}.meta".format(url_prefix, file_prefix),
>> + "file": os.path.join(feed_dir,
>> "{0}.meta".format(file_prefix))
>> + }
>> +
>> + feed = {
>> + "url": "{0}/{1}.json.gz".format(url_prefix,
>> file_prefix),
>> + "file": os.path.join(feed_dir,
>> "{0}.json.gz".format(file_prefix))
>> + }
>> +
>> + ctx = {}
>> +
>> + if not offline:
>> + ctx = download_feed(meta, feed)
>> +
>> + if not "meta" in ctx or not "feed" in ctx:
>> + return
>> +
>> + if not os.path.isfile(meta["file"]):
>> + return
>> +
>> + if not os.path.isfile(feed["file"]):
>> + return
>> +
>> + if not "meta" in ctx:
>> + ctx["meta"] = ctx_meta(meta["file"])
>> +
>> + if not "sha256" in ctx["meta"]:
>> + return
>> +
>> + if not "feed" in ctx:
>> + ctx["feed"] = ctx_gzip(feed["file"],
>> ctx["meta"]["sha256"])
>> +
>> + if not ctx["feed"]:
>> + return
>> +
>> + logging.debug("parsing year %s", year)
>> +
>> + for cve_item in ctx["feed"]["CVE_Items"]:
>> + iden, cve = parse_item(cve_item)
>> +
>> + if not iden:
>> + continue
>> +
>> + if not cve:
>> + logging.error("%s parse error", iden)
>> + break
>> +
>> + if iden in cve_struct:
>> + logging.error("%s duplicated", iden)
>> + break
>> +
>> + cve_struct[iden] = cve
>> +
>> + logging.debug("cve records: %d", len(cve_struct))
>> +
>> +
>> +def ctx_meta(filename):
>> + """Parse feed meta file"""
>> +
>> + if not os.path.isfile(filename):
>> + return {}
>> +
>> + ctx = {}
>> +
>> + with open(filename) as fil:
>> + for lin in fil:
>> + pair = lin.split(":", maxsplit=1)
>> + ctx[pair[0]] = pair[1].rstrip()
>> +
>> + return ctx
>> +
>> +
>> +def ctx_gzip(filename, checksum=""):
>> + """Parse feed archive file"""
>> +
>> + if not os.path.isfile(filename):
>> + return {}
>> +
>> + with gzip.open(filename) as fil:
>> + try:
>> + ctx = fil.read()
>> + except (EOFError, OSError):
>> + logging.error("failed to process gz archive %s",
>> filename, exc_info=True)
>> + return {}
>> +
>> + if checksum and checksum.upper() !=
>> hashlib.sha256(ctx).hexdigest().upper():
>> + return {}
>> +
>> + return json.loads(ctx.decode())
>> +
>> +
>> +def parse_item(cve_item):
>> + """Parse one JSON CVE entry"""
>> +
>> + cve_id = cve_item["cve"]["CVE_data_meta"]["ID"][:]
>> + impact = cve_item["impact"]
>> +
>> + if not impact:
>> + # REJECTed CVE
>> + return None, None
>> +
>> + if "baseMetricV3" in impact:
>> + score = impact["baseMetricV3"]["cvssV3"]["baseScore"]
>> + elif "baseMetricV2" in impact:
>> + score = impact["baseMetricV2"]["cvssV2"]["baseScore"]
>> + else:
>> + return cve_id, None
>> +
>> + return cve_id, {
>> + "score": score,
>> + "nodes": cve_item["configurations"]["nodes"][:],
>> + "reference":
>> cve_item["cve"]["references"]["reference_data"][:],
>> + "description":
>> cve_item["cve"]["description"]["description_data"][0]["value"]
>> + }
>> +
>> +
>> +def download_feed(meta, feed):
>> + """Download and parse feed"""
>> +
>> + ctx = {}
>> +
>> + if not retrieve_url(meta["url"], meta["file"]):
>> + return {}
>> +
>> + ctx["meta"] = ctx_meta(meta["file"])
>> +
>> + if not "sha256" in ctx["meta"]:
>> + return {}
>> +
>> + ctx["feed"] = ctx_gzip(feed["file"],
>> ctx["meta"]["sha256"])
>> +
>> + if not ctx["feed"]:
>> + if not retrieve_url(feed["url"], feed["file"]):
>> + return {}
>> +
>> + ctx["feed"] = ctx_gzip(feed["file"],
>> ctx["meta"]["sha256"])
>> +
>> + return ctx
>> +
>> +
>> +def retrieve_url(url, filename=None):
>> + """Download file by URL"""
>> +
>> + if filename:
>> + os.makedirs(os.path.dirname(filename), exist_ok=True)
>> +
>> + logging.debug("downloading %s", url)
>> +
>> + try:
>> + urllib.request.urlretrieve(url, filename=filename)
>> + except urllib.error.HTTPError:
>> + logging.error("failed to download URL %s", url,
>> exc_info=True)
>> + return False
>> +
>> + return True
>> +
>> +
>> +def logconfig(debug_flag=False):
>> + """Return default log config"""
>> +
>> + return {
>> + "version": 1,
>> + "formatters": {
>> + "f": {
>> + "format": "# %(asctime)s %% CVERT %%
>> %(levelname)-8s %% %(message)s"
>> + }
>> + },
>> + "handlers": {
>> + "h": {
>> + "class": "logging.StreamHandler",
>> + "formatter": "f",
>> + "level": logging.DEBUG if debug_flag else
>> logging.INFO
>> + }
>> + },
>> + "root": {
>> + "handlers": ["h"],
>> + "level": logging.DEBUG if debug_flag else
>> logging.INFO
>> + },
>> + }
>> +
>> +
>> +def save_cve(filename, cve_struct):
>> + """Save CVE structure in the file"""
>> +
>> + filename = os.path.realpath(filename)
>> +
>> + logging.debug("saving %d CVE records to %s",
>> len(cve_struct), filename)
>> +
>> + with open(filename, "wb") as fil:
>> + pickle.dump(cve_struct, fil)
>> +
>> +
>> +def load_cve(filename):
>> + """Load CVE structure from the file"""
>> +
>> + filename = os.path.realpath(filename)
>> +
>> + logging.debug("loading from %s", filename)
>> +
>> + with open(filename, "rb") as fil:
>> + cve_struct = pickle.load(fil)
>> +
>> + logging.debug("cve records: %d", len(cve_struct))
>> +
>> + return cve_struct
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2018-10-30 2:59 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-10 16:25 [PATCH v3 1/3] cve-report: add scripts to generate CVE reports grygorii tertychnyi
2018-10-29 23:29 ` akuster808
2018-10-30 2:59 ` Grygorii Tertychnyi
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.