* [PATCH v3 1/3] cve-report: add scripts to generate CVE reports @ 2018-10-10 16:25 grygorii tertychnyi 2018-10-29 23:29 ` akuster808 0 siblings, 1 reply; 3+ messages in thread From: grygorii tertychnyi @ 2018-10-10 16:25 UTC (permalink / raw) To: openembedded-core; +Cc: xe-linux-external cvert-foss - generate CVE report for the list of packages. Analyze the whole image manifest to align with the complex CPE configurations. cvert-update - update NVD feeds and store CVE structues dump. CVE dump is a pickled representation of the cve_struct dictionary. cvert.py - python library used by cvert-* scripts. NVD JSON Vulnerability Feeds https://nvd.nist.gov/vuln/data-feeds#JSON_FEED Usage examples: o Download CVE feeds to "nvdfeed" directory % cvert-update nvdfeed o Update CVE feeds and store a dump in a file % cvert-update --store cvedump nvdfeed o Generate a CVE report % cvert-foss --feed-dir nvdfeed --output report-foss.txt cve-manifest o (faster) Use dump file to generate a CVE report % cvert-foss --restore cvedump --output report-foss.txt cve-manifest o Generate a full report % cvert-foss --restore cvedump --show-description --show-reference \ --output report-foss-full.txt cve-manifest Manifest example: bash,4.2,CVE-2014-7187 python,2.7.35, python,3.5.5,CVE-2017-17522 CVE-2018-1061 Report example: patched | 7.5 | CVE-2018-1061 | python | 3.5.5 patched | 10.0 | CVE-2014-7187 | bash | 4.2 patched | 8.8 | CVE-2017-17522 | python | 3.5.5 unpatched | 10.0 | CVE-2014-6271 | bash | 4.2 unpatched | 10.0 | CVE-2014-6277 | bash | 4.2 unpatched | 10.0 | CVE-2014-6278 | bash | 4.2 unpatched | 10.0 | CVE-2014-7169 | bash | 4.2 unpatched | 10.0 | CVE-2014-7186 | bash | 4.2 unpatched | 4.6 | CVE-2012-3410 | bash | 4.2 unpatched | 8.4 | CVE-2016-7543 | bash | 4.2 unpatched | 5.0 | CVE-2010-3492 | python | 2.7.35 unpatched | 5.3 | CVE-2016-1494 | python | 2.7.35 unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5 unpatched | 6.5 | CVE-2017-18207 | python | 2.7.35 unpatched | 7.1 | CVE-2013-7338 | python | 2.7.35 unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5 unpatched | 8.8 | CVE-2017-17522 | python | 2.7.35 Signed-off-by: grygorii tertychnyi <gtertych@cisco.com> --- Changes in v3: o better logging: cvert.py lib log messages are controlled by cvert-* scripts o add more examples o add short params ("-o" "--output", "-f", "--feed-dir", etc) o fix double entries in manifest o fix pylint warnings scripts/cvert-foss | 151 ++++++++++++++++ scripts/cvert-update | 79 +++++++++ scripts/cvert.py | 473 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 703 insertions(+) create mode 100755 scripts/cvert-foss create mode 100755 scripts/cvert-update create mode 100644 scripts/cvert.py diff --git a/scripts/cvert-foss b/scripts/cvert-foss new file mode 100755 index 000000000000..00fbf2c0687b --- /dev/null +++ b/scripts/cvert-foss @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +# +# Copyright (c) 2018 by Cisco Systems, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" Generate CVE report for the given CVE manifest +""" + +import sys +import textwrap +import argparse +import logging +import logging.config +import cvert + +def report_foss(): + """Generate CVE report""" + + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + description=textwrap.dedent(""" + Generate CVE report for the given CVE manifest. + """), + epilog=textwrap.dedent(""" + @ run examples: + + # Download (update) NVD feeds in "nvdfeed" directory + # and prepare the report for the "cve-manifest" file + %% %(prog)s --feed-dir nvdfeed --output report-foss.txt cve-manifest + + # Use existed NVD feeds in "nvdfeed" directory + # and prepare the report for the "cve-manifest" file + %% %(prog)s --offline --feed-dir nvdfeed --output report-foss.txt cve-manifest + + # (faster) Restore CVE dump from "cvedump" (must exist) + # and prepare the report for the "cve-manifest" file + %% %(prog)s --restore cvedump --output report-foss.txt cve-manifest + + # Restore CVE dump from "cvedump" (must exist) + # and prepare the extended report for the "cve-manifest" file + %% %(prog)s --restore cvedump --show-description --show-reference --output report-foss.txt cve-manifest + + @ manifest example: + + bash,4.2,CVE-2014-7187 + python,2.7.35, + python,3.5.5,CVE-2017-17522 CVE-2018-1061 + + @ report example output: + + . patched | 10.0 | CVE-2014-7187 | bash | 4.2 + . patched | 7.5 | CVE-2018-1061 | python | 3.5.5 + . patched | 8.8 | CVE-2017-17522 | python | 3.5.5 + unpatched | 10.0 | CVE-2014-6271 | bash | 4.2 + unpatched | 10.0 | CVE-2014-6277 | bash | 4.2 + unpatched | 10.0 | CVE-2014-6278 | bash | 4.2 + unpatched | 10.0 | CVE-2014-7169 | bash | 4.2 + unpatched | 10.0 | CVE-2014-7186 | bash | 4.2 + unpatched | 4.6 | CVE-2012-3410 | bash | 4.2 + unpatched | 8.4 | CVE-2016-7543 | bash | 4.2 + unpatched | 5.0 | CVE-2010-3492 | python | 2.7.35 + unpatched | 5.3 | CVE-2016-1494 | python | 2.7.35 + unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5 + unpatched | 6.5 | CVE-2017-18207 | python | 2.7.35 + unpatched | 7.1 | CVE-2013-7338 | python | 2.7.35 + unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5 + unpatched | 8.8 | CVE-2017-17522 | python | 2.7.35 + """)) + + group = parser.add_mutually_exclusive_group(required=True) + group.add_argument("-f", "--feed-dir", help="feeds directory") + group.add_argument("-d", "--restore", help="load CVE data structures from file", + metavar="FILENAME") + parser.add_argument("--offline", help="do not update from NVD site", + action="store_true") + parser.add_argument("-o", "--output", help="save report to the file") + parser.add_argument("--show-description", help='show "Description" in the report', + action="store_true") + parser.add_argument("--show-reference", help='show "Reference" in the report', + action="store_true") + parser.add_argument("--debug", help="print debug messages", + action="store_true") + + parser.add_argument("cve_manifest", help="file with a list of packages, " + "each line contains three comma separated values: name, " + "version and a space separated list of patched CVEs, " + "e.g.: python,3.5.5,CVE-2017-17522 CVE-2018-1061", + metavar="cve-manifest") + + args = parser.parse_args() + + logging.config.dictConfig(cvert.logconfig(args.debug)) + + cve_manifest = {} + + with open(args.cve_manifest, "r") as fil: + for lin in fil: + lin = lin.rstrip() + + # skip empty lines + if not lin: + continue + + product, version, patched = lin.split(",", maxsplit=3) + + if product in cve_manifest: + cve_manifest[product][version] = patched.split() + else: + cve_manifest[product] = { + version: patched.split() + } + + if args.restore: + cve_struct = cvert.load_cve(args.restore) + elif args.feed_dir: + cve_struct = cvert.update_feeds(args.feed_dir, args.offline) + + if not cve_struct and args.offline: + parser.error("No CVEs found. Try to turn off offline mode or use other file to restore.") + + if args.output: + output = open(args.output, "w") + else: + output = sys.stdout + + report = cvert.generate_report(cve_manifest, cve_struct) + + cvert.print_report(report, + show_description=args.show_description, + show_reference=args.show_reference, + output=output) + + if args.output: + output.close() + + +if __name__ == "__main__": + report_foss() diff --git a/scripts/cvert-update b/scripts/cvert-update new file mode 100755 index 000000000000..3b3f5572a83c --- /dev/null +++ b/scripts/cvert-update @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +# +# Copyright (c) 2018 by Cisco Systems, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" Update NVD feeds and store CVE blob locally +""" + + +import textwrap +import argparse +import logging +import logging.config +import cvert + + +def update_cvert(): + """Update CVE storage""" + + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + description=textwrap.dedent(""" + Update NVD feeds and store CVE blob locally. + """), + epilog=textwrap.dedent(""" + examples: + + # Download NVD feeds to "nvdfeed" directory. + # If there are meta files in the directory, they will be updated + # and only fresh archives will be downloaded + %% %(prog)s nvdfeed + + # Inspect NVD feeds in "nvdfeed" directory + # and prepare a CVE dump python blob "cvedump". + # Use it later as input for cvert-* scripts (for speeding up) + %% %(prog)s --offline --store cvedump nvdfeed + + # Download (update) NVD feeds and prepare the CVE dump + %% %(prog)s --store cvedump nvdfeed + """)) + + parser.add_argument("-d", "--store", help="save CVE data structures in file", + metavar="FILENAME") + parser.add_argument("--offline", help="do not update from NVD site", + action="store_true") + parser.add_argument("--debug", help="print debug messages", + action="store_true") + + parser.add_argument("feed_dir", help="feeds directory", + metavar="feed-dir") + + args = parser.parse_args() + + logging.config.dictConfig(cvert.logconfig(args.debug)) + + cve_struct = cvert.update_feeds(args.feed_dir, args.offline) + + if not cve_struct and args.offline: + parser.error("No CVEs found in {0}. Try turn off offline mode.".format(args.feed_dir)) + + if args.store: + cvert.save_cve(args.store, cve_struct) + + +if __name__ == "__main__": + update_cvert() diff --git a/scripts/cvert.py b/scripts/cvert.py new file mode 100644 index 000000000000..f93b95c84965 --- /dev/null +++ b/scripts/cvert.py @@ -0,0 +1,473 @@ +#!/usr/bin/env python3 +# +# Copyright (c) 2018 by Cisco Systems, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" CVERT library: set of functions for CVE reports +""" + + +import os +import re +import sys +import json +import gzip +import pickle +import logging +import hashlib +import datetime +import textwrap +import urllib.request +import distutils.version + + +logging.getLogger(__name__).addHandler(logging.NullHandler()) + + +def generate_report(manifest, cve_struct): + """Generate CVE report""" + + report = [] + + for cve in cve_struct: + affected = set() + + for conf in cve_struct[cve]["nodes"]: + affected = affected.union(process_configuration(manifest, conf)) + + for key in affected: + product, version = key.split(",") + patched = manifest[product][version] + + if cve in patched: + cve_item = {"status": "patched"} + else: + cve_item = {"status": "unpatched"} + + cve_item["CVSS"] = "{0:.1f}".format(cve_struct[cve]["score"]) + cve_item["CVE"] = cve + cve_item["product"] = product + cve_item["version"] = version + cve_item["description"] = cve_struct[cve]["description"] + cve_item["reference"] = [x["url"] for x in cve_struct[cve]["reference"]] + + logging.debug("%9s %s %s,%s", + cve_item["status"], cve_item["CVE"], + cve_item["product"], cve_item["version"]) + + report.append(cve_item) + + return sorted(report, key=lambda x: (x["status"], x["product"], x["CVSS"], x["CVE"])) + + +def process_configuration(manifest, conf): + """Recursive call to process all CVE configurations""" + + operator = conf["operator"] + + if operator not in ["OR", "AND"]: + raise ValueError("operator {} is not supported".format(operator)) + + operator = True if operator == "AND" else False + match = False + affected = set() + + if "cpe" in conf: + match = process_cpe(manifest, conf["cpe"][0], affected) + + for cpe in conf["cpe"][1:]: + package_match = process_cpe(manifest, cpe, affected) + + # match = match <operator> package_match + match = operator ^ ((operator ^ match) or (operator ^ package_match)) + elif "children" in conf: + product_set = process_configuration(manifest, conf["children"][0]) + + if product_set: + match = True + affected = affected.union(product_set) + + for child in conf["children"][1:]: + product_set = process_configuration(manifest, child) + package_match = True if product_set else False + + # match = match OP package_match + match = operator ^ ((operator ^ match) or (operator ^ package_match)) + + if package_match: + affected = affected.union(product_set) + + if match: + return affected + + return () + + +def process_cpe(manifest, cpe, affected): + """Match CPE with all manifest packages""" + + if not cpe["vulnerable"]: + # ignore non vulnerable part + return False + + version_range = {} + + for flag in ["versionStartIncluding", + "versionStartExcluding", + "versionEndIncluding", + "versionEndExcluding"]: + if flag in cpe: + version_range[flag] = cpe[flag] + + # take only "product" and "version" + product, version = cpe["cpe23Uri"].split(":")[4:6] + + if product not in manifest: + return False + + if not version_range: + if version == "*": + # ignore CVEs that touches all versions of package, + # can not fix it anyway + logging.debug('ignore "*" in %s', cpe["cpe23Uri"]) + return False + elif version == "-": + # "-" means NA + # + # NA (i.e. "not applicable/not used"). The logical value NA + # SHOULD be assigned when there is no legal or meaningful + # value for that attribute, or when that attribute is not + # used as part of the description. + # This includes the situation in which an attribute has + # an obtainable value that is null + # + # Ignores CVEs if version is not set + logging.debug('ignore "-" in %s', cpe["cpe23Uri"]) + return False + else: + version_range["versionExactMatch"] = version + + result = False + + for version in manifest[product]: + try: + if match_version(version, + version_range): + logging.debug("match %s %s: %s", product, version, cpe["cpe23Uri"]) + affected.add("{},{}".format(product, version)) + + result = True + except TypeError: + # version comparison is a very tricky + # sometimes provider changes product version in a strange manner + # and the above comparison just failed + # so here we try to make version string "more standard" + + if match_version(twik_version(version), + [twik_version(v) for v in version_range]): + logging.debug("match %s %s (twiked): %s", product, twik_version(version), + cpe["cpe23Uri"]) + affected.add("{},{}".format(product, version)) + + result = True + + return result + + +def match_version(version, vrange): + """Match version with the version range""" + + result = False + version = util_version(version) + + if "versionExactMatch" in vrange: + if version == util_version(vrange["versionExactMatch"]): + result = True + else: + result = True + + if "versionStartIncluding" in vrange: + result = result and version >= util_version(vrange["versionStartIncluding"]) + + if "versionStartExcluding" in vrange: + result = result and version > util_version(vrange["versionStartExcluding"]) + + if "versionEndIncluding" in vrange: + result = result and version <= util_version(vrange["versionEndIncluding"]) + + if "versionEndExcluding" in vrange: + result = result and version < util_version(vrange["versionEndExcluding"]) + + return result + + +def util_version(version): + """Simplify package version""" + return distutils.version.LooseVersion(version.split("+git")[0]) + + +def twik_version(version): + """Return "standard" version for complex cases""" + return "v1" + re.sub(r"^[a-zA-Z]+", "", version) + + +def print_report(report, width=70, show_description=False, show_reference=False, output=sys.stdout): + """Print out final report""" + + for cve in report: + print("{0:>9s} | {1:>4s} | {2:18s} | {3} | {4}".format(cve["status"], cve["CVSS"], + cve["CVE"], cve["product"], + cve["version"]), + file=output) + + if show_description: + print("{0:>9s} + {1}".format(" ", "Description"), file=output) + + for lin in textwrap.wrap(cve["description"], width=width): + print("{0:>9s} {1}".format(" ", lin), file=output) + + if show_reference: + print("{0:>9s} + {1}".format(" ", "Reference"), file=output) + + for url in cve["reference"]: + print("{0:>9s} {1}".format(" ", url), file=output) + + +def update_feeds(feed_dir, offline=False, start=2002): + """Update all JSON feeds""" + + feed_dir = os.path.realpath(feed_dir) + year_now = datetime.datetime.now().year + cve_struct = {} + + for year in range(start, year_now + 1): + update_year(cve_struct, year, feed_dir, offline) + + return cve_struct + + +def update_year(cve_struct, year, feed_dir, offline): + """Update one JSON feed for the particular year""" + + url_prefix = "https://static.nvd.nist.gov/feeds/json/cve/1.0" + file_prefix = "nvdcve-1.0-{0}".format(year) + + meta = { + "url": "{0}/{1}.meta".format(url_prefix, file_prefix), + "file": os.path.join(feed_dir, "{0}.meta".format(file_prefix)) + } + + feed = { + "url": "{0}/{1}.json.gz".format(url_prefix, file_prefix), + "file": os.path.join(feed_dir, "{0}.json.gz".format(file_prefix)) + } + + ctx = {} + + if not offline: + ctx = download_feed(meta, feed) + + if not "meta" in ctx or not "feed" in ctx: + return + + if not os.path.isfile(meta["file"]): + return + + if not os.path.isfile(feed["file"]): + return + + if not "meta" in ctx: + ctx["meta"] = ctx_meta(meta["file"]) + + if not "sha256" in ctx["meta"]: + return + + if not "feed" in ctx: + ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"]) + + if not ctx["feed"]: + return + + logging.debug("parsing year %s", year) + + for cve_item in ctx["feed"]["CVE_Items"]: + iden, cve = parse_item(cve_item) + + if not iden: + continue + + if not cve: + logging.error("%s parse error", iden) + break + + if iden in cve_struct: + logging.error("%s duplicated", iden) + break + + cve_struct[iden] = cve + + logging.debug("cve records: %d", len(cve_struct)) + + +def ctx_meta(filename): + """Parse feed meta file""" + + if not os.path.isfile(filename): + return {} + + ctx = {} + + with open(filename) as fil: + for lin in fil: + pair = lin.split(":", maxsplit=1) + ctx[pair[0]] = pair[1].rstrip() + + return ctx + + +def ctx_gzip(filename, checksum=""): + """Parse feed archive file""" + + if not os.path.isfile(filename): + return {} + + with gzip.open(filename) as fil: + try: + ctx = fil.read() + except (EOFError, OSError): + logging.error("failed to process gz archive %s", filename, exc_info=True) + return {} + + if checksum and checksum.upper() != hashlib.sha256(ctx).hexdigest().upper(): + return {} + + return json.loads(ctx.decode()) + + +def parse_item(cve_item): + """Parse one JSON CVE entry""" + + cve_id = cve_item["cve"]["CVE_data_meta"]["ID"][:] + impact = cve_item["impact"] + + if not impact: + # REJECTed CVE + return None, None + + if "baseMetricV3" in impact: + score = impact["baseMetricV3"]["cvssV3"]["baseScore"] + elif "baseMetricV2" in impact: + score = impact["baseMetricV2"]["cvssV2"]["baseScore"] + else: + return cve_id, None + + return cve_id, { + "score": score, + "nodes": cve_item["configurations"]["nodes"][:], + "reference": cve_item["cve"]["references"]["reference_data"][:], + "description": cve_item["cve"]["description"]["description_data"][0]["value"] + } + + +def download_feed(meta, feed): + """Download and parse feed""" + + ctx = {} + + if not retrieve_url(meta["url"], meta["file"]): + return {} + + ctx["meta"] = ctx_meta(meta["file"]) + + if not "sha256" in ctx["meta"]: + return {} + + ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"]) + + if not ctx["feed"]: + if not retrieve_url(feed["url"], feed["file"]): + return {} + + ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"]) + + return ctx + + +def retrieve_url(url, filename=None): + """Download file by URL""" + + if filename: + os.makedirs(os.path.dirname(filename), exist_ok=True) + + logging.debug("downloading %s", url) + + try: + urllib.request.urlretrieve(url, filename=filename) + except urllib.error.HTTPError: + logging.error("failed to download URL %s", url, exc_info=True) + return False + + return True + + +def logconfig(debug_flag=False): + """Return default log config""" + + return { + "version": 1, + "formatters": { + "f": { + "format": "# %(asctime)s %% CVERT %% %(levelname)-8s %% %(message)s" + } + }, + "handlers": { + "h": { + "class": "logging.StreamHandler", + "formatter": "f", + "level": logging.DEBUG if debug_flag else logging.INFO + } + }, + "root": { + "handlers": ["h"], + "level": logging.DEBUG if debug_flag else logging.INFO + }, + } + + +def save_cve(filename, cve_struct): + """Save CVE structure in the file""" + + filename = os.path.realpath(filename) + + logging.debug("saving %d CVE records to %s", len(cve_struct), filename) + + with open(filename, "wb") as fil: + pickle.dump(cve_struct, fil) + + +def load_cve(filename): + """Load CVE structure from the file""" + + filename = os.path.realpath(filename) + + logging.debug("loading from %s", filename) + + with open(filename, "rb") as fil: + cve_struct = pickle.load(fil) + + logging.debug("cve records: %d", len(cve_struct)) + + return cve_struct -- 2.10.3.dirty ^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH v3 1/3] cve-report: add scripts to generate CVE reports 2018-10-10 16:25 [PATCH v3 1/3] cve-report: add scripts to generate CVE reports grygorii tertychnyi @ 2018-10-29 23:29 ` akuster808 2018-10-30 2:59 ` Grygorii Tertychnyi 0 siblings, 1 reply; 3+ messages in thread From: akuster808 @ 2018-10-29 23:29 UTC (permalink / raw) To: grygorii tertychnyi, openembedded-core; +Cc: xe-linux-external Grygorii, I was good to see you at OEDeM. I have some feedback. On 10/10/18 9:25 AM, grygorii tertychnyi via Openembedded-core wrote: > cvert-foss - generate CVE report for the list of packages. > Analyze the whole image manifest to align with the complex > CPE configurations. > > cvert-update - update NVD feeds and store CVE structues dump. > CVE dump is a pickled representation of the cve_struct dictionary. > > cvert.py - python library used by cvert-* scripts. > NVD JSON Vulnerability Feeds https://nvd.nist.gov/vuln/data-feeds#JSON_FEED > > Usage examples: > > o Download CVE feeds to "nvdfeed" directory > % cvert-update nvdfeed > o Update CVE feeds and store a dump in a file > % cvert-update --store cvedump nvdfeed > o Generate a CVE report > % cvert-foss --feed-dir nvdfeed --output report-foss.txt cve-manifest > o (faster) Use dump file to generate a CVE report > % cvert-foss --restore cvedump --output report-foss.txt cve-manifest > o Generate a full report > % cvert-foss --restore cvedump --show-description --show-reference \ > --output report-foss-full.txt cve-manifest report-foss-full.txt The cve-manifest, I could not figure out a way to create this from the above steps. I had your patches included when I did an image build then I got the "report-foss-full.txt" created. I am guessing I missed some steps somewhere. Can you clarify? Also, is there a way to version the report file by image or standard buildID so that I don't accidentally overwrite the txt file. Maybe a future enhancement is to group the packages by layer. I ran this with the meta-security layer and I now need to look up what packages belong to what layer to go work on them. Overall, I like this implementation and seems to be fast. Thanks for the patches. kind regards, Armin > > Manifest example: > > bash,4.2,CVE-2014-7187 > python,2.7.35, > python,3.5.5,CVE-2017-17522 CVE-2018-1061 > > Report example: > > patched | 7.5 | CVE-2018-1061 | python | 3.5.5 > patched | 10.0 | CVE-2014-7187 | bash | 4.2 > patched | 8.8 | CVE-2017-17522 | python | 3.5.5 > unpatched | 10.0 | CVE-2014-6271 | bash | 4.2 > unpatched | 10.0 | CVE-2014-6277 | bash | 4.2 > unpatched | 10.0 | CVE-2014-6278 | bash | 4.2 > unpatched | 10.0 | CVE-2014-7169 | bash | 4.2 > unpatched | 10.0 | CVE-2014-7186 | bash | 4.2 > unpatched | 4.6 | CVE-2012-3410 | bash | 4.2 > unpatched | 8.4 | CVE-2016-7543 | bash | 4.2 > unpatched | 5.0 | CVE-2010-3492 | python | 2.7.35 > unpatched | 5.3 | CVE-2016-1494 | python | 2.7.35 > unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5 > unpatched | 6.5 | CVE-2017-18207 | python | 2.7.35 > unpatched | 7.1 | CVE-2013-7338 | python | 2.7.35 > unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5 > unpatched | 8.8 | CVE-2017-17522 | python | 2.7.35 > > Signed-off-by: grygorii tertychnyi <gtertych@cisco.com> > --- > > Changes in v3: > o better logging: cvert.py lib log messages are controlled by cvert-* scripts > o add more examples > o add short params ("-o" "--output", "-f", "--feed-dir", etc) > o fix double entries in manifest > o fix pylint warnings > > scripts/cvert-foss | 151 ++++++++++++++++ > scripts/cvert-update | 79 +++++++++ > scripts/cvert.py | 473 +++++++++++++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 703 insertions(+) > create mode 100755 scripts/cvert-foss > create mode 100755 scripts/cvert-update > create mode 100644 scripts/cvert.py > > diff --git a/scripts/cvert-foss b/scripts/cvert-foss > new file mode 100755 > index 000000000000..00fbf2c0687b > --- /dev/null > +++ b/scripts/cvert-foss > @@ -0,0 +1,151 @@ > +#!/usr/bin/env python3 > +# > +# Copyright (c) 2018 by Cisco Systems, Inc. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License version 2 as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > +# > + > +""" Generate CVE report for the given CVE manifest > +""" > + > +import sys > +import textwrap > +import argparse > +import logging > +import logging.config > +import cvert > + > +def report_foss(): > + """Generate CVE report""" > + > + parser = argparse.ArgumentParser( > + formatter_class=argparse.RawDescriptionHelpFormatter, > + description=textwrap.dedent(""" > + Generate CVE report for the given CVE manifest. > + """), > + epilog=textwrap.dedent(""" > + @ run examples: > + > + # Download (update) NVD feeds in "nvdfeed" directory > + # and prepare the report for the "cve-manifest" file > + %% %(prog)s --feed-dir nvdfeed --output report-foss.txt cve-manifest > + > + # Use existed NVD feeds in "nvdfeed" directory > + # and prepare the report for the "cve-manifest" file > + %% %(prog)s --offline --feed-dir nvdfeed --output report-foss.txt cve-manifest > + > + # (faster) Restore CVE dump from "cvedump" (must exist) > + # and prepare the report for the "cve-manifest" file > + %% %(prog)s --restore cvedump --output report-foss.txt cve-manifest > + > + # Restore CVE dump from "cvedump" (must exist) > + # and prepare the extended report for the "cve-manifest" file > + %% %(prog)s --restore cvedump --show-description --show-reference --output report-foss.txt cve-manifest > + > + @ manifest example: > + > + bash,4.2,CVE-2014-7187 > + python,2.7.35, > + python,3.5.5,CVE-2017-17522 CVE-2018-1061 > + > + @ report example output: > + > + . patched | 10.0 | CVE-2014-7187 | bash | 4.2 > + . patched | 7.5 | CVE-2018-1061 | python | 3.5.5 > + . patched | 8.8 | CVE-2017-17522 | python | 3.5.5 > + unpatched | 10.0 | CVE-2014-6271 | bash | 4.2 > + unpatched | 10.0 | CVE-2014-6277 | bash | 4.2 > + unpatched | 10.0 | CVE-2014-6278 | bash | 4.2 > + unpatched | 10.0 | CVE-2014-7169 | bash | 4.2 > + unpatched | 10.0 | CVE-2014-7186 | bash | 4.2 > + unpatched | 4.6 | CVE-2012-3410 | bash | 4.2 > + unpatched | 8.4 | CVE-2016-7543 | bash | 4.2 > + unpatched | 5.0 | CVE-2010-3492 | python | 2.7.35 > + unpatched | 5.3 | CVE-2016-1494 | python | 2.7.35 > + unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5 > + unpatched | 6.5 | CVE-2017-18207 | python | 2.7.35 > + unpatched | 7.1 | CVE-2013-7338 | python | 2.7.35 > + unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5 > + unpatched | 8.8 | CVE-2017-17522 | python | 2.7.35 > + """)) > + > + group = parser.add_mutually_exclusive_group(required=True) > + group.add_argument("-f", "--feed-dir", help="feeds directory") > + group.add_argument("-d", "--restore", help="load CVE data structures from file", > + metavar="FILENAME") > + parser.add_argument("--offline", help="do not update from NVD site", > + action="store_true") > + parser.add_argument("-o", "--output", help="save report to the file") > + parser.add_argument("--show-description", help='show "Description" in the report', > + action="store_true") > + parser.add_argument("--show-reference", help='show "Reference" in the report', > + action="store_true") > + parser.add_argument("--debug", help="print debug messages", > + action="store_true") > + > + parser.add_argument("cve_manifest", help="file with a list of packages, " > + "each line contains three comma separated values: name, " > + "version and a space separated list of patched CVEs, " > + "e.g.: python,3.5.5,CVE-2017-17522 CVE-2018-1061", > + metavar="cve-manifest") > + > + args = parser.parse_args() > + > + logging.config.dictConfig(cvert.logconfig(args.debug)) > + > + cve_manifest = {} > + > + with open(args.cve_manifest, "r") as fil: > + for lin in fil: > + lin = lin.rstrip() > + > + # skip empty lines > + if not lin: > + continue > + > + product, version, patched = lin.split(",", maxsplit=3) > + > + if product in cve_manifest: > + cve_manifest[product][version] = patched.split() > + else: > + cve_manifest[product] = { > + version: patched.split() > + } > + > + if args.restore: > + cve_struct = cvert.load_cve(args.restore) > + elif args.feed_dir: > + cve_struct = cvert.update_feeds(args.feed_dir, args.offline) > + > + if not cve_struct and args.offline: > + parser.error("No CVEs found. Try to turn off offline mode or use other file to restore.") > + > + if args.output: > + output = open(args.output, "w") > + else: > + output = sys.stdout > + > + report = cvert.generate_report(cve_manifest, cve_struct) > + > + cvert.print_report(report, > + show_description=args.show_description, > + show_reference=args.show_reference, > + output=output) > + > + if args.output: > + output.close() > + > + > +if __name__ == "__main__": > + report_foss() > diff --git a/scripts/cvert-update b/scripts/cvert-update > new file mode 100755 > index 000000000000..3b3f5572a83c > --- /dev/null > +++ b/scripts/cvert-update > @@ -0,0 +1,79 @@ > +#!/usr/bin/env python3 > +# > +# Copyright (c) 2018 by Cisco Systems, Inc. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License version 2 as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > +# > + > +""" Update NVD feeds and store CVE blob locally > +""" > + > + > +import textwrap > +import argparse > +import logging > +import logging.config > +import cvert > + > + > +def update_cvert(): > + """Update CVE storage""" > + > + parser = argparse.ArgumentParser( > + formatter_class=argparse.RawDescriptionHelpFormatter, > + description=textwrap.dedent(""" > + Update NVD feeds and store CVE blob locally. > + """), > + epilog=textwrap.dedent(""" > + examples: > + > + # Download NVD feeds to "nvdfeed" directory. > + # If there are meta files in the directory, they will be updated > + # and only fresh archives will be downloaded > + %% %(prog)s nvdfeed > + > + # Inspect NVD feeds in "nvdfeed" directory > + # and prepare a CVE dump python blob "cvedump". > + # Use it later as input for cvert-* scripts (for speeding up) > + %% %(prog)s --offline --store cvedump nvdfeed > + > + # Download (update) NVD feeds and prepare the CVE dump > + %% %(prog)s --store cvedump nvdfeed > + """)) > + > + parser.add_argument("-d", "--store", help="save CVE data structures in file", > + metavar="FILENAME") > + parser.add_argument("--offline", help="do not update from NVD site", > + action="store_true") > + parser.add_argument("--debug", help="print debug messages", > + action="store_true") > + > + parser.add_argument("feed_dir", help="feeds directory", > + metavar="feed-dir") > + > + args = parser.parse_args() > + > + logging.config.dictConfig(cvert.logconfig(args.debug)) > + > + cve_struct = cvert.update_feeds(args.feed_dir, args.offline) > + > + if not cve_struct and args.offline: > + parser.error("No CVEs found in {0}. Try turn off offline mode.".format(args.feed_dir)) > + > + if args.store: > + cvert.save_cve(args.store, cve_struct) > + > + > +if __name__ == "__main__": > + update_cvert() > diff --git a/scripts/cvert.py b/scripts/cvert.py > new file mode 100644 > index 000000000000..f93b95c84965 > --- /dev/null > +++ b/scripts/cvert.py > @@ -0,0 +1,473 @@ > +#!/usr/bin/env python3 > +# > +# Copyright (c) 2018 by Cisco Systems, Inc. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License version 2 as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > +# > + > +""" CVERT library: set of functions for CVE reports > +""" > + > + > +import os > +import re > +import sys > +import json > +import gzip > +import pickle > +import logging > +import hashlib > +import datetime > +import textwrap > +import urllib.request > +import distutils.version > + > + > +logging.getLogger(__name__).addHandler(logging.NullHandler()) > + > + > +def generate_report(manifest, cve_struct): > + """Generate CVE report""" > + > + report = [] > + > + for cve in cve_struct: > + affected = set() > + > + for conf in cve_struct[cve]["nodes"]: > + affected = affected.union(process_configuration(manifest, conf)) > + > + for key in affected: > + product, version = key.split(",") > + patched = manifest[product][version] > + > + if cve in patched: > + cve_item = {"status": "patched"} > + else: > + cve_item = {"status": "unpatched"} > + > + cve_item["CVSS"] = "{0:.1f}".format(cve_struct[cve]["score"]) > + cve_item["CVE"] = cve > + cve_item["product"] = product > + cve_item["version"] = version > + cve_item["description"] = cve_struct[cve]["description"] > + cve_item["reference"] = [x["url"] for x in cve_struct[cve]["reference"]] > + > + logging.debug("%9s %s %s,%s", > + cve_item["status"], cve_item["CVE"], > + cve_item["product"], cve_item["version"]) > + > + report.append(cve_item) > + > + return sorted(report, key=lambda x: (x["status"], x["product"], x["CVSS"], x["CVE"])) > + > + > +def process_configuration(manifest, conf): > + """Recursive call to process all CVE configurations""" > + > + operator = conf["operator"] > + > + if operator not in ["OR", "AND"]: > + raise ValueError("operator {} is not supported".format(operator)) > + > + operator = True if operator == "AND" else False > + match = False > + affected = set() > + > + if "cpe" in conf: > + match = process_cpe(manifest, conf["cpe"][0], affected) > + > + for cpe in conf["cpe"][1:]: > + package_match = process_cpe(manifest, cpe, affected) > + > + # match = match <operator> package_match > + match = operator ^ ((operator ^ match) or (operator ^ package_match)) > + elif "children" in conf: > + product_set = process_configuration(manifest, conf["children"][0]) > + > + if product_set: > + match = True > + affected = affected.union(product_set) > + > + for child in conf["children"][1:]: > + product_set = process_configuration(manifest, child) > + package_match = True if product_set else False > + > + # match = match OP package_match > + match = operator ^ ((operator ^ match) or (operator ^ package_match)) > + > + if package_match: > + affected = affected.union(product_set) > + > + if match: > + return affected > + > + return () > + > + > +def process_cpe(manifest, cpe, affected): > + """Match CPE with all manifest packages""" > + > + if not cpe["vulnerable"]: > + # ignore non vulnerable part > + return False > + > + version_range = {} > + > + for flag in ["versionStartIncluding", > + "versionStartExcluding", > + "versionEndIncluding", > + "versionEndExcluding"]: > + if flag in cpe: > + version_range[flag] = cpe[flag] > + > + # take only "product" and "version" > + product, version = cpe["cpe23Uri"].split(":")[4:6] > + > + if product not in manifest: > + return False > + > + if not version_range: > + if version == "*": > + # ignore CVEs that touches all versions of package, > + # can not fix it anyway > + logging.debug('ignore "*" in %s', cpe["cpe23Uri"]) > + return False > + elif version == "-": > + # "-" means NA > + # > + # NA (i.e. "not applicable/not used"). The logical value NA > + # SHOULD be assigned when there is no legal or meaningful > + # value for that attribute, or when that attribute is not > + # used as part of the description. > + # This includes the situation in which an attribute has > + # an obtainable value that is null > + # > + # Ignores CVEs if version is not set > + logging.debug('ignore "-" in %s', cpe["cpe23Uri"]) > + return False > + else: > + version_range["versionExactMatch"] = version > + > + result = False > + > + for version in manifest[product]: > + try: > + if match_version(version, > + version_range): > + logging.debug("match %s %s: %s", product, version, cpe["cpe23Uri"]) > + affected.add("{},{}".format(product, version)) > + > + result = True > + except TypeError: > + # version comparison is a very tricky > + # sometimes provider changes product version in a strange manner > + # and the above comparison just failed > + # so here we try to make version string "more standard" > + > + if match_version(twik_version(version), > + [twik_version(v) for v in version_range]): > + logging.debug("match %s %s (twiked): %s", product, twik_version(version), > + cpe["cpe23Uri"]) > + affected.add("{},{}".format(product, version)) > + > + result = True > + > + return result > + > + > +def match_version(version, vrange): > + """Match version with the version range""" > + > + result = False > + version = util_version(version) > + > + if "versionExactMatch" in vrange: > + if version == util_version(vrange["versionExactMatch"]): > + result = True > + else: > + result = True > + > + if "versionStartIncluding" in vrange: > + result = result and version >= util_version(vrange["versionStartIncluding"]) > + > + if "versionStartExcluding" in vrange: > + result = result and version > util_version(vrange["versionStartExcluding"]) > + > + if "versionEndIncluding" in vrange: > + result = result and version <= util_version(vrange["versionEndIncluding"]) > + > + if "versionEndExcluding" in vrange: > + result = result and version < util_version(vrange["versionEndExcluding"]) > + > + return result > + > + > +def util_version(version): > + """Simplify package version""" > + return distutils.version.LooseVersion(version.split("+git")[0]) > + > + > +def twik_version(version): > + """Return "standard" version for complex cases""" > + return "v1" + re.sub(r"^[a-zA-Z]+", "", version) > + > + > +def print_report(report, width=70, show_description=False, show_reference=False, output=sys.stdout): > + """Print out final report""" > + > + for cve in report: > + print("{0:>9s} | {1:>4s} | {2:18s} | {3} | {4}".format(cve["status"], cve["CVSS"], > + cve["CVE"], cve["product"], > + cve["version"]), > + file=output) > + > + if show_description: > + print("{0:>9s} + {1}".format(" ", "Description"), file=output) > + > + for lin in textwrap.wrap(cve["description"], width=width): > + print("{0:>9s} {1}".format(" ", lin), file=output) > + > + if show_reference: > + print("{0:>9s} + {1}".format(" ", "Reference"), file=output) > + > + for url in cve["reference"]: > + print("{0:>9s} {1}".format(" ", url), file=output) > + > + > +def update_feeds(feed_dir, offline=False, start=2002): > + """Update all JSON feeds""" > + > + feed_dir = os.path.realpath(feed_dir) > + year_now = datetime.datetime.now().year > + cve_struct = {} > + > + for year in range(start, year_now + 1): > + update_year(cve_struct, year, feed_dir, offline) > + > + return cve_struct > + > + > +def update_year(cve_struct, year, feed_dir, offline): > + """Update one JSON feed for the particular year""" > + > + url_prefix = "https://static.nvd.nist.gov/feeds/json/cve/1.0" > + file_prefix = "nvdcve-1.0-{0}".format(year) > + > + meta = { > + "url": "{0}/{1}.meta".format(url_prefix, file_prefix), > + "file": os.path.join(feed_dir, "{0}.meta".format(file_prefix)) > + } > + > + feed = { > + "url": "{0}/{1}.json.gz".format(url_prefix, file_prefix), > + "file": os.path.join(feed_dir, "{0}.json.gz".format(file_prefix)) > + } > + > + ctx = {} > + > + if not offline: > + ctx = download_feed(meta, feed) > + > + if not "meta" in ctx or not "feed" in ctx: > + return > + > + if not os.path.isfile(meta["file"]): > + return > + > + if not os.path.isfile(feed["file"]): > + return > + > + if not "meta" in ctx: > + ctx["meta"] = ctx_meta(meta["file"]) > + > + if not "sha256" in ctx["meta"]: > + return > + > + if not "feed" in ctx: > + ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"]) > + > + if not ctx["feed"]: > + return > + > + logging.debug("parsing year %s", year) > + > + for cve_item in ctx["feed"]["CVE_Items"]: > + iden, cve = parse_item(cve_item) > + > + if not iden: > + continue > + > + if not cve: > + logging.error("%s parse error", iden) > + break > + > + if iden in cve_struct: > + logging.error("%s duplicated", iden) > + break > + > + cve_struct[iden] = cve > + > + logging.debug("cve records: %d", len(cve_struct)) > + > + > +def ctx_meta(filename): > + """Parse feed meta file""" > + > + if not os.path.isfile(filename): > + return {} > + > + ctx = {} > + > + with open(filename) as fil: > + for lin in fil: > + pair = lin.split(":", maxsplit=1) > + ctx[pair[0]] = pair[1].rstrip() > + > + return ctx > + > + > +def ctx_gzip(filename, checksum=""): > + """Parse feed archive file""" > + > + if not os.path.isfile(filename): > + return {} > + > + with gzip.open(filename) as fil: > + try: > + ctx = fil.read() > + except (EOFError, OSError): > + logging.error("failed to process gz archive %s", filename, exc_info=True) > + return {} > + > + if checksum and checksum.upper() != hashlib.sha256(ctx).hexdigest().upper(): > + return {} > + > + return json.loads(ctx.decode()) > + > + > +def parse_item(cve_item): > + """Parse one JSON CVE entry""" > + > + cve_id = cve_item["cve"]["CVE_data_meta"]["ID"][:] > + impact = cve_item["impact"] > + > + if not impact: > + # REJECTed CVE > + return None, None > + > + if "baseMetricV3" in impact: > + score = impact["baseMetricV3"]["cvssV3"]["baseScore"] > + elif "baseMetricV2" in impact: > + score = impact["baseMetricV2"]["cvssV2"]["baseScore"] > + else: > + return cve_id, None > + > + return cve_id, { > + "score": score, > + "nodes": cve_item["configurations"]["nodes"][:], > + "reference": cve_item["cve"]["references"]["reference_data"][:], > + "description": cve_item["cve"]["description"]["description_data"][0]["value"] > + } > + > + > +def download_feed(meta, feed): > + """Download and parse feed""" > + > + ctx = {} > + > + if not retrieve_url(meta["url"], meta["file"]): > + return {} > + > + ctx["meta"] = ctx_meta(meta["file"]) > + > + if not "sha256" in ctx["meta"]: > + return {} > + > + ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"]) > + > + if not ctx["feed"]: > + if not retrieve_url(feed["url"], feed["file"]): > + return {} > + > + ctx["feed"] = ctx_gzip(feed["file"], ctx["meta"]["sha256"]) > + > + return ctx > + > + > +def retrieve_url(url, filename=None): > + """Download file by URL""" > + > + if filename: > + os.makedirs(os.path.dirname(filename), exist_ok=True) > + > + logging.debug("downloading %s", url) > + > + try: > + urllib.request.urlretrieve(url, filename=filename) > + except urllib.error.HTTPError: > + logging.error("failed to download URL %s", url, exc_info=True) > + return False > + > + return True > + > + > +def logconfig(debug_flag=False): > + """Return default log config""" > + > + return { > + "version": 1, > + "formatters": { > + "f": { > + "format": "# %(asctime)s %% CVERT %% %(levelname)-8s %% %(message)s" > + } > + }, > + "handlers": { > + "h": { > + "class": "logging.StreamHandler", > + "formatter": "f", > + "level": logging.DEBUG if debug_flag else logging.INFO > + } > + }, > + "root": { > + "handlers": ["h"], > + "level": logging.DEBUG if debug_flag else logging.INFO > + }, > + } > + > + > +def save_cve(filename, cve_struct): > + """Save CVE structure in the file""" > + > + filename = os.path.realpath(filename) > + > + logging.debug("saving %d CVE records to %s", len(cve_struct), filename) > + > + with open(filename, "wb") as fil: > + pickle.dump(cve_struct, fil) > + > + > +def load_cve(filename): > + """Load CVE structure from the file""" > + > + filename = os.path.realpath(filename) > + > + logging.debug("loading from %s", filename) > + > + with open(filename, "rb") as fil: > + cve_struct = pickle.load(fil) > + > + logging.debug("cve records: %d", len(cve_struct)) > + > + return cve_struct ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v3 1/3] cve-report: add scripts to generate CVE reports 2018-10-29 23:29 ` akuster808 @ 2018-10-30 2:59 ` Grygorii Tertychnyi 0 siblings, 0 replies; 3+ messages in thread From: Grygorii Tertychnyi @ 2018-10-30 2:59 UTC (permalink / raw) To: akuster808; +Cc: xe-linux-external, openembedded-core On Mon Oct29 2018 @ 23:29, akuster808 <akuster808@gmail.com> wrote: > Grygorii, > > I was good to see you at OEDeM. Thanks Armin. > I have some feedback. > > On 10/10/18 9:25 AM, grygorii tertychnyi via Openembedded-core > wrote: >> cvert-foss - generate CVE report for the list of packages. >> Analyze the whole image manifest to align with the complex >> CPE configurations. >> >> cvert-update - update NVD feeds and store CVE structues dump. >> CVE dump is a pickled representation of the cve_struct >> dictionary. >> >> cvert.py - python library used by cvert-* scripts. >> NVD JSON Vulnerability Feeds >> https://nvd.nist.gov/vuln/data-feeds#JSON_FEED >> >> Usage examples: >> >> o Download CVE feeds to "nvdfeed" directory >> % cvert-update nvdfeed >> o Update CVE feeds and store a dump in a file >> % cvert-update --store cvedump nvdfeed >> o Generate a CVE report >> % cvert-foss --feed-dir nvdfeed --output report-foss.txt >> cve-manifest >> o (faster) Use dump file to generate a CVE report >> % cvert-foss --restore cvedump --output report-foss.txt >> cve-manifest >> o Generate a full report >> % cvert-foss --restore cvedump --show-description >> --show-reference \ >> --output report-foss-full.txt cve-manifest >> report-foss-full.txt Oh, there is a typo. "cvert-foss" script requires only one positional argument - "cve-manifest". So, the "Generate a full report" command should look like this: % cvert-foss --restore cvedump --show-description --show-reference \ --output report-foss-full.txt cve-manifest Is not a big deal. Just to be on the same page. > > The cve-manifest, I could not figure out a way to create this > from the > above steps. I had your patches included when I did an image > build then > I got the "report-foss-full.txt" created. I am guessing I missed > some > steps somewhere. Can you clarify? I try to explain it this way: There are three patches: o 1st is "cve-report: add scripts to generate CVE reports" "cvert-foss" takes the "cve-manifest" (input), and generates the CVE report (output). So, you are right, these scripts do not produce "cve-manifest" files. o 2nd is "cvert-kernel - generate CVE report for the Linux kernel" "cvert-kernel" analyzes kernel git dir and generate the kernel CVE report o 3rd is "cve-report.bbclass: add class" it generates "cve-manifest" file for the given <image> and then calls "cvert-foss" to produce the CVE report. So, if you applied all three patches, modified "local.conf" and run "bitbake -c report_cve <image>" the "cve-manifest" is generated as part of the "generate_report_handler" function of "cve-report.bbclass". So, back to you question "how to generate cve-manifest", the easiest way is: echo 'INHERIT += "cve-report"' >> conf/local.conf echo 'CVE_REPORT_MODE[packageonly] = "1"' >> conf/local.conf bitbake -c report_cve <image> It is described in the 3rd patch. Here, in the 1st patch I described only the "cve-manifest" format. Same format is used by "cve-check-tool", it is well-known and simply enough, so, no need to change it. Being a part of scripts directory it can be used outside of any bbclass. You can create simple manifests and generate CVE reports right on command line, simple example: % echo "python,3.5.5,CVE-2017-17522 CVE-2018-1061" > cve-manifest.txt % ./cvert-foss --debug --feed-dir cve-feeds --output report-foss.txt cve-manifest.txt # 2018-10-29 19:27:48,673 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.meta # 2018-10-29 19:27:50,250 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.json.gz # 2018-10-29 19:27:53,537 % CVERT % DEBUG % parsing year 2002 # 2018-10-29 19:27:53,575 % CVERT % DEBUG % cve records: 6667 # 2018-10-29 19:27:53,605 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2003.meta # 2018-10-29 19:27:55,121 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2003.json.gz # 2018-10-29 19:27:57,360 % CVERT % DEBUG % parsing year 2003 # 2018-10-29 19:27:57,369 % CVERT % DEBUG % cve records: 8167 # 2018-10-29 19:27:57,382 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2004.meta # 2018-10-29 19:27:58,825 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2004.json.gz # 2018-10-29 19:28:01,595 % CVERT % DEBUG % parsing year 2004 # 2018-10-29 19:28:01,613 % CVERT % DEBUG % cve records: 10810 # 2018-10-29 19:28:01,636 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2005.meta # 2018-10-29 19:28:03,008 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2005.json.gz # 2018-10-29 19:28:06,088 % CVERT % DEBUG % parsing year 2005 # 2018-10-29 19:28:06,119 % CVERT % DEBUG % cve records: 15424 # 2018-10-29 19:28:06,152 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2006.meta # 2018-10-29 19:28:07,627 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2006.json.gz # 2018-10-29 19:28:11,392 % CVERT % DEBUG % parsing year 2006 # 2018-10-29 19:28:11,439 % CVERT % DEBUG % cve records: 22408 # 2018-10-29 19:28:11,489 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2007.meta # 2018-10-29 19:28:13,163 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2007.json.gz # 2018-10-29 19:28:17,432 % CVERT % DEBUG % parsing year 2007 # 2018-10-29 19:28:17,478 % CVERT % DEBUG % cve records: 28850 # 2018-10-29 19:28:17,530 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2008.meta # 2018-10-29 19:28:19,195 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2008.json.gz # 2018-10-29 19:28:22,937 % CVERT % DEBUG % parsing year 2008 # 2018-10-29 19:28:23,489 % CVERT % DEBUG % cve records: 35840 # 2018-10-29 19:28:23,556 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2009.meta # 2018-10-29 19:28:25,127 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2009.json.gz # 2018-10-29 19:28:29,013 % CVERT % DEBUG % parsing year 2009 # 2018-10-29 19:28:29,049 % CVERT % DEBUG % cve records: 40705 # 2018-10-29 19:28:29,104 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2010.meta # 2018-10-29 19:28:31,082 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2010.json.gz # 2018-10-29 19:28:35,571 % CVERT % DEBUG % parsing year 2010 # 2018-10-29 19:28:35,609 % CVERT % DEBUG % cve records: 45638 # 2018-10-29 19:28:35,677 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2011.meta # 2018-10-29 19:28:37,263 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2011.json.gz # 2018-10-29 19:28:43,598 % CVERT % DEBUG % parsing year 2011 # 2018-10-29 19:28:43,630 % CVERT % DEBUG % cve records: 50043 # 2018-10-29 19:28:43,739 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2012.meta # 2018-10-29 19:28:45,264 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2012.json.gz # 2018-10-29 19:28:49,718 % CVERT % DEBUG % parsing year 2012 # 2018-10-29 19:28:49,755 % CVERT % DEBUG % cve records: 55215 # 2018-10-29 19:28:49,830 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2013.meta # 2018-10-29 19:28:51,455 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2013.json.gz # 2018-10-29 19:28:56,101 % CVERT % DEBUG % parsing year 2013 # 2018-10-29 19:28:56,142 % CVERT % DEBUG % cve records: 60882 # 2018-10-29 19:28:56,210 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2014.meta # 2018-10-29 19:28:57,977 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2014.json.gz # 2018-10-29 19:29:02,677 % CVERT % DEBUG % parsing year 2014 # 2018-10-29 19:29:03,693 % CVERT % DEBUG % cve records: 68824 # 2018-10-29 19:29:03,783 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2015.meta # 2018-10-29 19:29:05,820 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2015.json.gz # 2018-10-29 19:29:09,577 % CVERT % DEBUG % parsing year 2015 # 2018-10-29 19:29:09,632 % CVERT % DEBUG % cve records: 76251 # 2018-10-29 19:29:09,720 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2016.meta # 2018-10-29 19:29:11,362 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2016.json.gz # 2018-10-29 19:29:17,063 % CVERT % DEBUG % parsing year 2016 # 2018-10-29 19:29:17,128 % CVERT % DEBUG % cve records: 84957 # 2018-10-29 19:29:17,238 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2017.meta # 2018-10-29 19:29:19,102 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2017.json.gz # 2018-10-29 19:29:25,282 % CVERT % DEBUG % parsing year 2017 # 2018-10-29 19:29:25,377 % CVERT % DEBUG % cve records: 98334 # 2018-10-29 19:29:25,564 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2018.meta # 2018-10-29 19:29:27,195 % CVERT % DEBUG % downloading https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2018.json.gz # 2018-10-29 19:29:32,262 % CVERT % DEBUG % parsing year 2018 # 2018-10-29 19:29:32,323 % CVERT % DEBUG % cve records: 106349 # 2018-10-29 19:29:32,912 % CVERT % DEBUG % match python 3.5.5: cpe:2.3:a:python:python:*:*:*:*:*:*:*:* # 2018-10-29 19:29:33,265 % CVERT % DEBUG % ignore "*" in cpe:2.3:a:python_software_foundation:python:*:*:*:*:*:*:*:* # 2018-10-29 19:29:33,271 % CVERT % DEBUG % match python 3.5.5: cpe:2.3:a:python:python:*:*:*:*:*:*:*:* # 2018-10-29 19:29:33,271 % CVERT % DEBUG % unpatched CVE-2017-18207 python,3.5.5 # 2018-10-29 19:29:33,486 % CVERT % DEBUG % match python 3.5.5: cpe:2.3:a:python:python:*:*:*:*:*:*:*:* # 2018-10-29 19:29:33,487 % CVERT % DEBUG % patched CVE-2018-1061 python,3.5.5 # 2018-10-29 19:29:33,772 % CVERT % DEBUG % ignore "-" in cpe:2.3:a:python:python:-:*:*:*:*:*:*:* # 2018-10-29 19:29:34,585 % CVERT % DEBUG % match python 3.5.5: cpe:2.3:a:python:python:*:*:*:*:*:*:*:* # 2018-10-29 19:29:34,585 % CVERT % DEBUG % unpatched CVE-2018-1060 python,3.5.5 # 2018-10-29 19:29:36,668 % CVERT % DEBUG % match python 3.5.5: cpe:2.3:a:python:python:*:*:*:*:*:*:*:* # 2018-10-29 19:29:36,668 % CVERT % DEBUG % patched CVE-2017-17522 python,3.5.5 % cat report-foss.txt patched | 7.5 | CVE-2018-1061 | python | 3.5.5 patched | 8.8 | CVE-2017-17522 | python | 3.5.5 unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5 unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5 Of course, it is better to use first "cvert-update --store ..." and then multiple times "cvert-foss --restore ..." - much faster on multiple runs. > Also, is there a way to version the report file by image or > standard > buildID so that I don't accidentally overwrite the txt > file. Maybe a > future enhancement is to group the packages by layer. I ran this > with > the meta-security layer and I now need to look up what packages > belong > to what layer to go work on them. I think it is possible, but again as part of bbclass enhancement. If I understood Richard correctly on OEDeM, the bbclass should be modified anyway ("do not copy functionality"). But these cvert scripts do not depend on it. And can be seen independently. > Overall, I like this implementation and seems to be fast. > > Thanks for the patches. Thanks. It is inspiring. > kind regards, > > Armin > >> >> Manifest example: >> >> bash,4.2,CVE-2014-7187 >> python,2.7.35, >> python,3.5.5,CVE-2017-17522 CVE-2018-1061 >> >> Report example: >> >> patched | 7.5 | CVE-2018-1061 | python | 3.5.5 >> patched | 10.0 | CVE-2014-7187 | bash | 4.2 >> patched | 8.8 | CVE-2017-17522 | python | 3.5.5 >> unpatched | 10.0 | CVE-2014-6271 | bash | 4.2 >> unpatched | 10.0 | CVE-2014-6277 | bash | 4.2 >> unpatched | 10.0 | CVE-2014-6278 | bash | 4.2 >> unpatched | 10.0 | CVE-2014-7169 | bash | 4.2 >> unpatched | 10.0 | CVE-2014-7186 | bash | 4.2 >> unpatched | 4.6 | CVE-2012-3410 | bash | 4.2 >> unpatched | 8.4 | CVE-2016-7543 | bash | 4.2 >> unpatched | 5.0 | CVE-2010-3492 | python | 2.7.35 >> unpatched | 5.3 | CVE-2016-1494 | python | 2.7.35 >> unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5 >> unpatched | 6.5 | CVE-2017-18207 | python | 2.7.35 >> unpatched | 7.1 | CVE-2013-7338 | python | 2.7.35 >> unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5 >> unpatched | 8.8 | CVE-2017-17522 | python | 2.7.35 >> >> Signed-off-by: grygorii tertychnyi <gtertych@cisco.com> >> --- >> >> Changes in v3: >> o better logging: cvert.py lib log messages are controlled by >> cvert-* scripts >> o add more examples >> o add short params ("-o" "--output", "-f", "--feed-dir", etc) >> o fix double entries in manifest >> o fix pylint warnings >> >> scripts/cvert-foss | 151 ++++++++++++++++ >> scripts/cvert-update | 79 +++++++++ >> scripts/cvert.py | 473 >> +++++++++++++++++++++++++++++++++++++++++++++++++++ >> 3 files changed, 703 insertions(+) >> create mode 100755 scripts/cvert-foss >> create mode 100755 scripts/cvert-update >> create mode 100644 scripts/cvert.py >> >> diff --git a/scripts/cvert-foss b/scripts/cvert-foss >> new file mode 100755 >> index 000000000000..00fbf2c0687b >> --- /dev/null >> +++ b/scripts/cvert-foss >> @@ -0,0 +1,151 @@ >> +#!/usr/bin/env python3 >> +# >> +# Copyright (c) 2018 by Cisco Systems, Inc. >> +# >> +# This program is free software; you can redistribute it >> and/or modify >> +# it under the terms of the GNU General Public License version >> 2 as >> +# published by the Free Software Foundation. >> +# >> +# This program is distributed in the hope that it will be >> useful, >> +# but WITHOUT ANY WARRANTY; without even the implied warranty >> of >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See >> the >> +# GNU General Public License for more details. >> +# >> +# You should have received a copy of the GNU General Public >> License along >> +# with this program; if not, write to the Free Software >> Foundation, Inc., >> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. >> +# >> + >> +""" Generate CVE report for the given CVE manifest >> +""" >> + >> +import sys >> +import textwrap >> +import argparse >> +import logging >> +import logging.config >> +import cvert >> + >> +def report_foss(): >> + """Generate CVE report""" >> + >> + parser = argparse.ArgumentParser( >> + formatter_class=argparse.RawDescriptionHelpFormatter, >> + description=textwrap.dedent(""" >> + Generate CVE report for the given CVE manifest. >> + """), >> + epilog=textwrap.dedent(""" >> + @ run examples: >> + >> + # Download (update) NVD feeds in "nvdfeed" directory >> + # and prepare the report for the "cve-manifest" file >> + %% %(prog)s --feed-dir nvdfeed --output >> report-foss.txt cve-manifest >> + >> + # Use existed NVD feeds in "nvdfeed" directory >> + # and prepare the report for the "cve-manifest" file >> + %% %(prog)s --offline --feed-dir nvdfeed --output >> report-foss.txt cve-manifest >> + >> + # (faster) Restore CVE dump from "cvedump" (must >> exist) >> + # and prepare the report for the "cve-manifest" file >> + %% %(prog)s --restore cvedump --output report-foss.txt >> cve-manifest >> + >> + # Restore CVE dump from "cvedump" (must exist) >> + # and prepare the extended report for the >> "cve-manifest" file >> + %% %(prog)s --restore cvedump --show-description >> --show-reference --output report-foss.txt cve-manifest >> + >> + @ manifest example: >> + >> + bash,4.2,CVE-2014-7187 >> + python,2.7.35, >> + python,3.5.5,CVE-2017-17522 CVE-2018-1061 >> + >> + @ report example output: >> + >> + . patched | 10.0 | CVE-2014-7187 | bash | 4.2 >> + . patched | 7.5 | CVE-2018-1061 | python | 3.5.5 >> + . patched | 8.8 | CVE-2017-17522 | python | 3.5.5 >> + unpatched | 10.0 | CVE-2014-6271 | bash | 4.2 >> + unpatched | 10.0 | CVE-2014-6277 | bash | 4.2 >> + unpatched | 10.0 | CVE-2014-6278 | bash | 4.2 >> + unpatched | 10.0 | CVE-2014-7169 | bash | 4.2 >> + unpatched | 10.0 | CVE-2014-7186 | bash | 4.2 >> + unpatched | 4.6 | CVE-2012-3410 | bash | 4.2 >> + unpatched | 8.4 | CVE-2016-7543 | bash | 4.2 >> + unpatched | 5.0 | CVE-2010-3492 | python | >> 2.7.35 >> + unpatched | 5.3 | CVE-2016-1494 | python | >> 2.7.35 >> + unpatched | 6.5 | CVE-2017-18207 | python | 3.5.5 >> + unpatched | 6.5 | CVE-2017-18207 | python | >> 2.7.35 >> + unpatched | 7.1 | CVE-2013-7338 | python | >> 2.7.35 >> + unpatched | 7.5 | CVE-2018-1060 | python | 3.5.5 >> + unpatched | 8.8 | CVE-2017-17522 | python | >> 2.7.35 >> + """)) >> + >> + group = parser.add_mutually_exclusive_group(required=True) >> + group.add_argument("-f", "--feed-dir", help="feeds >> directory") >> + group.add_argument("-d", "--restore", help="load CVE data >> structures from file", >> + metavar="FILENAME") >> + parser.add_argument("--offline", help="do not update from >> NVD site", >> + action="store_true") >> + parser.add_argument("-o", "--output", help="save report to >> the file") >> + parser.add_argument("--show-description", help='show >> "Description" in the report', >> + action="store_true") >> + parser.add_argument("--show-reference", help='show >> "Reference" in the report', >> + action="store_true") >> + parser.add_argument("--debug", help="print debug >> messages", >> + action="store_true") >> + >> + parser.add_argument("cve_manifest", help="file with a list >> of packages, " >> + "each line contains three comma >> separated values: name, " >> + "version and a space separated list of >> patched CVEs, " >> + "e.g.: python,3.5.5,CVE-2017-17522 >> CVE-2018-1061", >> + metavar="cve-manifest") >> + >> + args = parser.parse_args() >> + >> + logging.config.dictConfig(cvert.logconfig(args.debug)) >> + >> + cve_manifest = {} >> + >> + with open(args.cve_manifest, "r") as fil: >> + for lin in fil: >> + lin = lin.rstrip() >> + >> + # skip empty lines >> + if not lin: >> + continue >> + >> + product, version, patched = lin.split(",", >> maxsplit=3) >> + >> + if product in cve_manifest: >> + cve_manifest[product][version] = >> patched.split() >> + else: >> + cve_manifest[product] = { >> + version: patched.split() >> + } >> + >> + if args.restore: >> + cve_struct = cvert.load_cve(args.restore) >> + elif args.feed_dir: >> + cve_struct = cvert.update_feeds(args.feed_dir, >> args.offline) >> + >> + if not cve_struct and args.offline: >> + parser.error("No CVEs found. Try to turn off offline >> mode or use other file to restore.") >> + >> + if args.output: >> + output = open(args.output, "w") >> + else: >> + output = sys.stdout >> + >> + report = cvert.generate_report(cve_manifest, cve_struct) >> + >> + cvert.print_report(report, >> + show_description=args.show_description, >> + show_reference=args.show_reference, >> + output=output) >> + >> + if args.output: >> + output.close() >> + >> + >> +if __name__ == "__main__": >> + report_foss() >> diff --git a/scripts/cvert-update b/scripts/cvert-update >> new file mode 100755 >> index 000000000000..3b3f5572a83c >> --- /dev/null >> +++ b/scripts/cvert-update >> @@ -0,0 +1,79 @@ >> +#!/usr/bin/env python3 >> +# >> +# Copyright (c) 2018 by Cisco Systems, Inc. >> +# >> +# This program is free software; you can redistribute it >> and/or modify >> +# it under the terms of the GNU General Public License version >> 2 as >> +# published by the Free Software Foundation. >> +# >> +# This program is distributed in the hope that it will be >> useful, >> +# but WITHOUT ANY WARRANTY; without even the implied warranty >> of >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See >> the >> +# GNU General Public License for more details. >> +# >> +# You should have received a copy of the GNU General Public >> License along >> +# with this program; if not, write to the Free Software >> Foundation, Inc., >> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. >> +# >> + >> +""" Update NVD feeds and store CVE blob locally >> +""" >> + >> + >> +import textwrap >> +import argparse >> +import logging >> +import logging.config >> +import cvert >> + >> + >> +def update_cvert(): >> + """Update CVE storage""" >> + >> + parser = argparse.ArgumentParser( >> + formatter_class=argparse.RawDescriptionHelpFormatter, >> + description=textwrap.dedent(""" >> + Update NVD feeds and store CVE blob locally. >> + """), >> + epilog=textwrap.dedent(""" >> + examples: >> + >> + # Download NVD feeds to "nvdfeed" directory. >> + # If there are meta files in the directory, they will >> be updated >> + # and only fresh archives will be downloaded >> + %% %(prog)s nvdfeed >> + >> + # Inspect NVD feeds in "nvdfeed" directory >> + # and prepare a CVE dump python blob "cvedump". >> + # Use it later as input for cvert-* scripts (for >> speeding up) >> + %% %(prog)s --offline --store cvedump nvdfeed >> + >> + # Download (update) NVD feeds and prepare the CVE dump >> + %% %(prog)s --store cvedump nvdfeed >> + """)) >> + >> + parser.add_argument("-d", "--store", help="save CVE data >> structures in file", >> + metavar="FILENAME") >> + parser.add_argument("--offline", help="do not update from >> NVD site", >> + action="store_true") >> + parser.add_argument("--debug", help="print debug >> messages", >> + action="store_true") >> + >> + parser.add_argument("feed_dir", help="feeds directory", >> + metavar="feed-dir") >> + >> + args = parser.parse_args() >> + >> + logging.config.dictConfig(cvert.logconfig(args.debug)) >> + >> + cve_struct = cvert.update_feeds(args.feed_dir, >> args.offline) >> + >> + if not cve_struct and args.offline: >> + parser.error("No CVEs found in {0}. Try turn off >> offline mode.".format(args.feed_dir)) >> + >> + if args.store: >> + cvert.save_cve(args.store, cve_struct) >> + >> + >> +if __name__ == "__main__": >> + update_cvert() >> diff --git a/scripts/cvert.py b/scripts/cvert.py >> new file mode 100644 >> index 000000000000..f93b95c84965 >> --- /dev/null >> +++ b/scripts/cvert.py >> @@ -0,0 +1,473 @@ >> +#!/usr/bin/env python3 >> +# >> +# Copyright (c) 2018 by Cisco Systems, Inc. >> +# >> +# This program is free software; you can redistribute it >> and/or modify >> +# it under the terms of the GNU General Public License version >> 2 as >> +# published by the Free Software Foundation. >> +# >> +# This program is distributed in the hope that it will be >> useful, >> +# but WITHOUT ANY WARRANTY; without even the implied warranty >> of >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See >> the >> +# GNU General Public License for more details. >> +# >> +# You should have received a copy of the GNU General Public >> License along >> +# with this program; if not, write to the Free Software >> Foundation, Inc., >> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. >> +# >> + >> +""" CVERT library: set of functions for CVE reports >> +""" >> + >> + >> +import os >> +import re >> +import sys >> +import json >> +import gzip >> +import pickle >> +import logging >> +import hashlib >> +import datetime >> +import textwrap >> +import urllib.request >> +import distutils.version >> + >> + >> +logging.getLogger(__name__).addHandler(logging.NullHandler()) >> + >> + >> +def generate_report(manifest, cve_struct): >> + """Generate CVE report""" >> + >> + report = [] >> + >> + for cve in cve_struct: >> + affected = set() >> + >> + for conf in cve_struct[cve]["nodes"]: >> + affected = >> affected.union(process_configuration(manifest, conf)) >> + >> + for key in affected: >> + product, version = key.split(",") >> + patched = manifest[product][version] >> + >> + if cve in patched: >> + cve_item = {"status": "patched"} >> + else: >> + cve_item = {"status": "unpatched"} >> + >> + cve_item["CVSS"] = >> "{0:.1f}".format(cve_struct[cve]["score"]) >> + cve_item["CVE"] = cve >> + cve_item["product"] = product >> + cve_item["version"] = version >> + cve_item["description"] = >> cve_struct[cve]["description"] >> + cve_item["reference"] = [x["url"] for x in >> cve_struct[cve]["reference"]] >> + >> + logging.debug("%9s %s %s,%s", >> + cve_item["status"], cve_item["CVE"], >> + cve_item["product"], >> cve_item["version"]) >> + >> + report.append(cve_item) >> + >> + return sorted(report, key=lambda x: (x["status"], >> x["product"], x["CVSS"], x["CVE"])) >> + >> + >> +def process_configuration(manifest, conf): >> + """Recursive call to process all CVE configurations""" >> + >> + operator = conf["operator"] >> + >> + if operator not in ["OR", "AND"]: >> + raise ValueError("operator {} is not >> supported".format(operator)) >> + >> + operator = True if operator == "AND" else False >> + match = False >> + affected = set() >> + >> + if "cpe" in conf: >> + match = process_cpe(manifest, conf["cpe"][0], >> affected) >> + >> + for cpe in conf["cpe"][1:]: >> + package_match = process_cpe(manifest, cpe, >> affected) >> + >> + # match = match <operator> package_match >> + match = operator ^ ((operator ^ match) or >> (operator ^ package_match)) >> + elif "children" in conf: >> + product_set = process_configuration(manifest, >> conf["children"][0]) >> + >> + if product_set: >> + match = True >> + affected = affected.union(product_set) >> + >> + for child in conf["children"][1:]: >> + product_set = process_configuration(manifest, >> child) >> + package_match = True if product_set else False >> + >> + # match = match OP package_match >> + match = operator ^ ((operator ^ match) or >> (operator ^ package_match)) >> + >> + if package_match: >> + affected = affected.union(product_set) >> + >> + if match: >> + return affected >> + >> + return () >> + >> + >> +def process_cpe(manifest, cpe, affected): >> + """Match CPE with all manifest packages""" >> + >> + if not cpe["vulnerable"]: >> + # ignore non vulnerable part >> + return False >> + >> + version_range = {} >> + >> + for flag in ["versionStartIncluding", >> + "versionStartExcluding", >> + "versionEndIncluding", >> + "versionEndExcluding"]: >> + if flag in cpe: >> + version_range[flag] = cpe[flag] >> + >> + # take only "product" and "version" >> + product, version = cpe["cpe23Uri"].split(":")[4:6] >> + >> + if product not in manifest: >> + return False >> + >> + if not version_range: >> + if version == "*": >> + # ignore CVEs that touches all versions of >> package, >> + # can not fix it anyway >> + logging.debug('ignore "*" in %s', cpe["cpe23Uri"]) >> + return False >> + elif version == "-": >> + # "-" means NA >> + # >> + # NA (i.e. "not applicable/not used"). The logical >> value NA >> + # SHOULD be assigned when there is no legal or >> meaningful >> + # value for that attribute, or when that attribute >> is not >> + # used as part of the description. >> + # This includes the situation in which an >> attribute has >> + # an obtainable value that is null >> + # >> + # Ignores CVEs if version is not set >> + logging.debug('ignore "-" in %s', cpe["cpe23Uri"]) >> + return False >> + else: >> + version_range["versionExactMatch"] = version >> + >> + result = False >> + >> + for version in manifest[product]: >> + try: >> + if match_version(version, >> + version_range): >> + logging.debug("match %s %s: %s", product, >> version, cpe["cpe23Uri"]) >> + affected.add("{},{}".format(product, version)) >> + >> + result = True >> + except TypeError: >> + # version comparison is a very tricky >> + # sometimes provider changes product version in a >> strange manner >> + # and the above comparison just failed >> + # so here we try to make version string "more >> standard" >> + >> + if match_version(twik_version(version), >> + [twik_version(v) for v in >> version_range]): >> + logging.debug("match %s %s (twiked): %s", >> product, twik_version(version), >> + cpe["cpe23Uri"]) >> + affected.add("{},{}".format(product, version)) >> + >> + result = True >> + >> + return result >> + >> + >> +def match_version(version, vrange): >> + """Match version with the version range""" >> + >> + result = False >> + version = util_version(version) >> + >> + if "versionExactMatch" in vrange: >> + if version == >> util_version(vrange["versionExactMatch"]): >> + result = True >> + else: >> + result = True >> + >> + if "versionStartIncluding" in vrange: >> + result = result and version >= >> util_version(vrange["versionStartIncluding"]) >> + >> + if "versionStartExcluding" in vrange: >> + result = result and version > >> util_version(vrange["versionStartExcluding"]) >> + >> + if "versionEndIncluding" in vrange: >> + result = result and version <= >> util_version(vrange["versionEndIncluding"]) >> + >> + if "versionEndExcluding" in vrange: >> + result = result and version < >> util_version(vrange["versionEndExcluding"]) >> + >> + return result >> + >> + >> +def util_version(version): >> + """Simplify package version""" >> + return >> distutils.version.LooseVersion(version.split("+git")[0]) >> + >> + >> +def twik_version(version): >> + """Return "standard" version for complex cases""" >> + return "v1" + re.sub(r"^[a-zA-Z]+", "", version) >> + >> + >> +def print_report(report, width=70, show_description=False, >> show_reference=False, output=sys.stdout): >> + """Print out final report""" >> + >> + for cve in report: >> + print("{0:>9s} | {1:>4s} | {2:18s} | {3} | >> {4}".format(cve["status"], cve["CVSS"], >> + >> cve["CVE"], cve["product"], >> + >> cve["version"]), >> + file=output) >> + >> + if show_description: >> + print("{0:>9s} + {1}".format(" ", "Description"), >> file=output) >> + >> + for lin in textwrap.wrap(cve["description"], >> width=width): >> + print("{0:>9s} {1}".format(" ", lin), >> file=output) >> + >> + if show_reference: >> + print("{0:>9s} + {1}".format(" ", "Reference"), >> file=output) >> + >> + for url in cve["reference"]: >> + print("{0:>9s} {1}".format(" ", url), >> file=output) >> + >> + >> +def update_feeds(feed_dir, offline=False, start=2002): >> + """Update all JSON feeds""" >> + >> + feed_dir = os.path.realpath(feed_dir) >> + year_now = datetime.datetime.now().year >> + cve_struct = {} >> + >> + for year in range(start, year_now + 1): >> + update_year(cve_struct, year, feed_dir, offline) >> + >> + return cve_struct >> + >> + >> +def update_year(cve_struct, year, feed_dir, offline): >> + """Update one JSON feed for the particular year""" >> + >> + url_prefix = >> "https://static.nvd.nist.gov/feeds/json/cve/1.0" >> + file_prefix = "nvdcve-1.0-{0}".format(year) >> + >> + meta = { >> + "url": "{0}/{1}.meta".format(url_prefix, file_prefix), >> + "file": os.path.join(feed_dir, >> "{0}.meta".format(file_prefix)) >> + } >> + >> + feed = { >> + "url": "{0}/{1}.json.gz".format(url_prefix, >> file_prefix), >> + "file": os.path.join(feed_dir, >> "{0}.json.gz".format(file_prefix)) >> + } >> + >> + ctx = {} >> + >> + if not offline: >> + ctx = download_feed(meta, feed) >> + >> + if not "meta" in ctx or not "feed" in ctx: >> + return >> + >> + if not os.path.isfile(meta["file"]): >> + return >> + >> + if not os.path.isfile(feed["file"]): >> + return >> + >> + if not "meta" in ctx: >> + ctx["meta"] = ctx_meta(meta["file"]) >> + >> + if not "sha256" in ctx["meta"]: >> + return >> + >> + if not "feed" in ctx: >> + ctx["feed"] = ctx_gzip(feed["file"], >> ctx["meta"]["sha256"]) >> + >> + if not ctx["feed"]: >> + return >> + >> + logging.debug("parsing year %s", year) >> + >> + for cve_item in ctx["feed"]["CVE_Items"]: >> + iden, cve = parse_item(cve_item) >> + >> + if not iden: >> + continue >> + >> + if not cve: >> + logging.error("%s parse error", iden) >> + break >> + >> + if iden in cve_struct: >> + logging.error("%s duplicated", iden) >> + break >> + >> + cve_struct[iden] = cve >> + >> + logging.debug("cve records: %d", len(cve_struct)) >> + >> + >> +def ctx_meta(filename): >> + """Parse feed meta file""" >> + >> + if not os.path.isfile(filename): >> + return {} >> + >> + ctx = {} >> + >> + with open(filename) as fil: >> + for lin in fil: >> + pair = lin.split(":", maxsplit=1) >> + ctx[pair[0]] = pair[1].rstrip() >> + >> + return ctx >> + >> + >> +def ctx_gzip(filename, checksum=""): >> + """Parse feed archive file""" >> + >> + if not os.path.isfile(filename): >> + return {} >> + >> + with gzip.open(filename) as fil: >> + try: >> + ctx = fil.read() >> + except (EOFError, OSError): >> + logging.error("failed to process gz archive %s", >> filename, exc_info=True) >> + return {} >> + >> + if checksum and checksum.upper() != >> hashlib.sha256(ctx).hexdigest().upper(): >> + return {} >> + >> + return json.loads(ctx.decode()) >> + >> + >> +def parse_item(cve_item): >> + """Parse one JSON CVE entry""" >> + >> + cve_id = cve_item["cve"]["CVE_data_meta"]["ID"][:] >> + impact = cve_item["impact"] >> + >> + if not impact: >> + # REJECTed CVE >> + return None, None >> + >> + if "baseMetricV3" in impact: >> + score = impact["baseMetricV3"]["cvssV3"]["baseScore"] >> + elif "baseMetricV2" in impact: >> + score = impact["baseMetricV2"]["cvssV2"]["baseScore"] >> + else: >> + return cve_id, None >> + >> + return cve_id, { >> + "score": score, >> + "nodes": cve_item["configurations"]["nodes"][:], >> + "reference": >> cve_item["cve"]["references"]["reference_data"][:], >> + "description": >> cve_item["cve"]["description"]["description_data"][0]["value"] >> + } >> + >> + >> +def download_feed(meta, feed): >> + """Download and parse feed""" >> + >> + ctx = {} >> + >> + if not retrieve_url(meta["url"], meta["file"]): >> + return {} >> + >> + ctx["meta"] = ctx_meta(meta["file"]) >> + >> + if not "sha256" in ctx["meta"]: >> + return {} >> + >> + ctx["feed"] = ctx_gzip(feed["file"], >> ctx["meta"]["sha256"]) >> + >> + if not ctx["feed"]: >> + if not retrieve_url(feed["url"], feed["file"]): >> + return {} >> + >> + ctx["feed"] = ctx_gzip(feed["file"], >> ctx["meta"]["sha256"]) >> + >> + return ctx >> + >> + >> +def retrieve_url(url, filename=None): >> + """Download file by URL""" >> + >> + if filename: >> + os.makedirs(os.path.dirname(filename), exist_ok=True) >> + >> + logging.debug("downloading %s", url) >> + >> + try: >> + urllib.request.urlretrieve(url, filename=filename) >> + except urllib.error.HTTPError: >> + logging.error("failed to download URL %s", url, >> exc_info=True) >> + return False >> + >> + return True >> + >> + >> +def logconfig(debug_flag=False): >> + """Return default log config""" >> + >> + return { >> + "version": 1, >> + "formatters": { >> + "f": { >> + "format": "# %(asctime)s %% CVERT %% >> %(levelname)-8s %% %(message)s" >> + } >> + }, >> + "handlers": { >> + "h": { >> + "class": "logging.StreamHandler", >> + "formatter": "f", >> + "level": logging.DEBUG if debug_flag else >> logging.INFO >> + } >> + }, >> + "root": { >> + "handlers": ["h"], >> + "level": logging.DEBUG if debug_flag else >> logging.INFO >> + }, >> + } >> + >> + >> +def save_cve(filename, cve_struct): >> + """Save CVE structure in the file""" >> + >> + filename = os.path.realpath(filename) >> + >> + logging.debug("saving %d CVE records to %s", >> len(cve_struct), filename) >> + >> + with open(filename, "wb") as fil: >> + pickle.dump(cve_struct, fil) >> + >> + >> +def load_cve(filename): >> + """Load CVE structure from the file""" >> + >> + filename = os.path.realpath(filename) >> + >> + logging.debug("loading from %s", filename) >> + >> + with open(filename, "rb") as fil: >> + cve_struct = pickle.load(fil) >> + >> + logging.debug("cve records: %d", len(cve_struct)) >> + >> + return cve_struct ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2018-10-30 2:59 UTC | newest] Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-10-10 16:25 [PATCH v3 1/3] cve-report: add scripts to generate CVE reports grygorii tertychnyi 2018-10-29 23:29 ` akuster808 2018-10-30 2:59 ` Grygorii Tertychnyi
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.