All of lore.kernel.org
 help / color / mirror / Atom feed
* [thud-next][PATCH 01/11] cve-check: backport rewrite from master
@ 2019-12-08 18:35 Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 02/11] cve-check: ensure all known CVEs are in the report Niko Mauno
                   ` (9 more replies)
  0 siblings, 10 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core; +Cc: Sergey Popovich, Jussi Kukkonen

From: Ross Burton <ross.burton@intel.com>

As detailed at [1] the XML feeds provided by NIST are being discontinued on
October 9th 2019.  As cve-check-tool uses these feeds, cve-check.bbclass will be
inoperable after this date.

To ensure that cve-check continues working, backport the following commits from
master to move away from the unmaintained cve-check-tool to our own Python code
that fetches the JSON:

546d14135c5 cve-update-db: New recipe to update CVE database
bc144b028f6 cve-check: Remove dependency to cve-check-tool-native
7f62a20b32a cve-check: Manage CVE_PRODUCT with more than one name
3bf63bc6084 cve-check: Consider CVE that affects versions with less than operator
c0eabd30d7b cve-update-db: Use std library instead of urllib3
27eb839ee65 cve-check: be idiomatic
09be21f4d17 cve-update-db: Manage proxy if needed.
975793e3825 cve-update-db: do_populate_cve_db depends on do_fetch
0325dd72714 cve-update-db: Catch request.urlopen errors.
4078da92b49 cve-check: Depends on cve-update-db-native
f7676e9a38d cve-update-db: Use NVD CPE data to populate PRODUCTS table
bc0195be1b1 cve-check: Update unpatched CVE matching
c807c2a6409 cve-update-db-native: Skip recipe when cve-check class is not loaded.
07bb8b25e17 cve-check: remove redundant readline CVE whitelisting
5388ed6d137 cve-check-tool: remove
270ac00cb43 cve-check.bbclass: initialize to_append
e6bf9000987 cve-check: allow comparison of Vendor as well as Product
91770338f76 cve-update-db-native: use SQL placeholders instead of format strings
7069302a4cc cve-check: Replace CVE_CHECK_CVE_WHITELIST by CVE_CHECK_WHITELIST
78de2cb39d7 cve-update-db-native: Remove hash column from database.
4b301030cf9 cve-update-db-native: use os.path.join instead of +
f0d822fad2a cve-update-db: actually inherit native
b309840b6aa cve-update-db-native: use executemany() to optimise CPE insertion
bb4e53af33d cve-update-db-native: improve metadata parsing
94227459792 cve-update-db-native: clean up JSON fetching
95438d52b73 cve-update-db-native: fix https proxy issues
1f9a963b9ff glibc: exclude child recipes from CVE scanning

[1] https://nvd.nist.gov/General/News/XML-Vulnerability-Feed-Retirement

(From OE-Core rev: 8c87e78547c598cada1bce92e7b25d85b994e2eb)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/cve-check.bbclass                | 142 +++++++-----
 meta/conf/distro/include/maintainers.inc      |   1 +
 meta/recipes-core/glibc/glibc-locale.inc      |   3 +
 meta/recipes-core/glibc/glibc-mtrace.inc      |   3 +
 meta/recipes-core/glibc/glibc-scripts.inc     |   3 +
 .../recipes-core/meta/cve-update-db-native.bb | 195 ++++++++++++++++
 .../cve-check-tool/cve-check-tool_5.6.4.bb    |  62 -----
 ...x-freeing-memory-allocated-by-sqlite.patch |  50 ----
 ...erriding-default-CA-certificate-file.patch | 215 ------------------
 ...s-in-percent-when-downloading-CVE-db.patch | 135 -----------
 ...omputed-vs-expected-sha256-digit-str.patch |  52 -----
 ...heck-for-malloc_trim-before-using-it.patch |  51 -----
 12 files changed, 292 insertions(+), 620 deletions(-)
 create mode 100644 meta/recipes-core/meta/cve-update-db-native.bb
 delete mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
 delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch
 delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch
 delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch
 delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch
 delete mode 100644 meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 743bc08a4f..c00d2910be 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}"
 CVE_VERSION ??= "${PV}"
 
 CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
-CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db"
+CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve_1.0.db"
 
 CVE_CHECK_LOG ?= "${T}/cve.log"
 CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
@@ -37,32 +37,33 @@ CVE_CHECK_COPY_FILES ??= "1"
 CVE_CHECK_CREATE_MANIFEST ??= "1"
 
 # Whitelist for packages (PN)
-CVE_CHECK_PN_WHITELIST = "\
-    glibc-locale \
-"
+CVE_CHECK_PN_WHITELIST ?= ""
 
-# Whitelist for CVE and version of package
-CVE_CHECK_CVE_WHITELIST = "{\
-    'CVE-2014-2524': ('6.3','5.2',), \
-}"
+# Whitelist for CVE. If a CVE is found, then it is considered patched.
+# The value is a string containing space separated CVE values:
+# 
+# CVE_CHECK_WHITELIST = 'CVE-2014-2524 CVE-2018-1234'
+# 
+CVE_CHECK_WHITELIST ?= ""
 
 python do_cve_check () {
     """
     Check recipe for patched and unpatched CVEs
     """
 
-    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE")):
+    if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
         patched_cves = get_patches_cves(d)
         patched, unpatched = check_cves(d, patched_cves)
         if patched or unpatched:
             cve_data = get_cve_info(d, patched + unpatched)
             cve_write_data(d, patched, unpatched, cve_data)
     else:
-        bb.note("Failed to update CVE database, skipping CVE check")
+        bb.note("No CVE database found, skipping CVE check")
+
 }
 
 addtask cve_check after do_unpack before do_build
-do_cve_check[depends] = "cve-check-tool-native:do_populate_sysroot cve-check-tool-native:do_populate_cve_db"
+do_cve_check[depends] = "cve-update-db-native:do_populate_cve_db"
 do_cve_check[nostamp] = "1"
 
 python cve_check_cleanup () {
@@ -163,65 +164,94 @@ def get_patches_cves(d):
 
 def check_cves(d, patched_cves):
     """
-    Run cve-check-tool looking for patched and unpatched CVEs.
+    Connect to the NVD database and find unpatched cves.
     """
-
     import ast, csv, tempfile, subprocess, io
+    from distutils.version import LooseVersion
 
-    cves_patched = []
     cves_unpatched = []
-    bpn = d.getVar("CVE_PRODUCT")
+    # CVE_PRODUCT can contain more than one product (eg. curl/libcurl)
+    products = d.getVar("CVE_PRODUCT").split()
     # If this has been unset then we're not scanning for CVEs here (for example, image recipes)
-    if not bpn:
+    if not products:
         return ([], [])
     pv = d.getVar("CVE_VERSION").split("+git")[0]
-    cves = " ".join(patched_cves)
-    cve_db_dir = d.getVar("CVE_CHECK_DB_DIR")
-    cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST"))
-    cve_cmd = "cve-check-tool"
-    cmd = [cve_cmd, "--no-html", "--skip-update", "--csv", "--not-affected", "-t", "faux", "-d", cve_db_dir]
 
     # If the recipe has been whitlisted we return empty lists
     if d.getVar("PN") in d.getVar("CVE_CHECK_PN_WHITELIST").split():
         bb.note("Recipe has been whitelisted, skipping check")
         return ([], [])
 
-    try:
-        # Write the faux CSV file to be used with cve-check-tool
-        fd, faux = tempfile.mkstemp(prefix="cve-faux-")
-        with os.fdopen(fd, "w") as f:
-            for pn in bpn.split():
-                f.write("%s,%s,%s,\n" % (pn, pv, cves))
-        cmd.append(faux)
-
-        output = subprocess.check_output(cmd).decode("utf-8")
-        bb.debug(2, "Output of command %s:\n%s" % ("\n".join(cmd), output))
-    except subprocess.CalledProcessError as e:
-        bb.warn("Couldn't check for CVEs: %s (output %s)" % (e, e.output))
-    finally:
-        os.remove(faux)
-
-    for row in csv.reader(io.StringIO(output)):
-        # Third row has the unpatched CVEs
-        if row[2]:
-            for cve in row[2].split():
-                # Skip if the CVE has been whitlisted for the current version
-                if pv in cve_whitelist.get(cve,[]):
-                    bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve))
+    old_cve_whitelist =  d.getVar("CVE_CHECK_CVE_WHITELIST")
+    if old_cve_whitelist:
+        bb.warn("CVE_CHECK_CVE_WHITELIST is deprecated, please use CVE_CHECK_WHITELIST.")
+    cve_whitelist = d.getVar("CVE_CHECK_WHITELIST").split()
+
+    import sqlite3
+    db_file = d.getVar("CVE_CHECK_DB_FILE")
+    conn = sqlite3.connect(db_file)
+
+    for product in products:
+        c = conn.cursor()
+        if ":" in product:
+            vendor, product = product.split(":", 1)
+            c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR IS ?", (product, vendor))
+        else:
+            c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,))
+
+        for row in c:
+            cve = row[0]
+            version_start = row[3]
+            operator_start = row[4]
+            version_end = row[5]
+            operator_end = row[6]
+
+            if cve in cve_whitelist:
+                bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve))
+            elif cve in patched_cves:
+                bb.note("%s has been patched" % (cve))
+            else:
+                to_append = False
+                if (operator_start == '=' and pv == version_start):
+                    cves_unpatched.append(cve)
                 else:
+                    if operator_start:
+                        try:
+                            to_append_start =  (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start))
+                            to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start))
+                        except:
+                            bb.note("%s: Failed to compare %s %s %s for %s" %
+                                    (product, pv, operator_start, version_start, cve))
+                            to_append_start = False
+                    else:
+                        to_append_start = False
+
+                    if operator_end:
+                        try:
+                            to_append_end  = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end))
+                            to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end))
+                        except:
+                            bb.note("%s: Failed to compare %s %s %s for %s" %
+                                    (product, pv, operator_end, version_end, cve))
+                            to_append_end = False
+                    else:
+                        to_append_end = False
+
+                    if operator_start and operator_end:
+                        to_append = to_append_start and to_append_end
+                    else:
+                        to_append = to_append_start or to_append_end
+
+                if to_append:
                     cves_unpatched.append(cve)
-                    bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve))
-        # Fourth row has patched CVEs
-        if row[3]:
-            for cve in row[3].split():
-                cves_patched.append(cve)
-                bb.debug(2, "%s-%s is patched for %s" % (bpn, pv, cve))
+                bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve))
+    conn.close()
 
-    return (cves_patched, cves_unpatched)
+    return (list(patched_cves), cves_unpatched)
 
 def get_cve_info(d, cves):
     """
-    Get CVE information from the database used by cve-check-tool.
+    Get CVE information from the database.
 
     Unfortunately the only way to get CVE info is set the output to
     html (hard to parse) or query directly the database.
@@ -241,9 +271,10 @@ def get_cve_info(d, cves):
     for row in cur.execute(query, tuple(cves)):
         cve_data[row[0]] = {}
         cve_data[row[0]]["summary"] = row[1]
-        cve_data[row[0]]["score"] = row[2]
-        cve_data[row[0]]["modified"] = row[3]
-        cve_data[row[0]]["vector"] = row[4]
+        cve_data[row[0]]["scorev2"] = row[2]
+        cve_data[row[0]]["scorev3"] = row[3]
+        cve_data[row[0]]["modified"] = row[4]
+        cve_data[row[0]]["vector"] = row[5]
     conn.close()
 
     return cve_data
@@ -270,7 +301,8 @@ def cve_write_data(d, patched, unpatched, cve_data):
             unpatched_cves.append(cve)
             write_string += "CVE STATUS: Unpatched\n"
         write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"]
-        write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"]
+        write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["scorev2"]
+        write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["scorev3"]
         write_string += "VECTOR: %s\n" % cve_data[cve]["vector"]
         write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve)
 
diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index 672f067792..c027901fdf 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -116,6 +116,7 @@ RECIPE_MAINTAINER_pn-cryptodev-tests = "Robert Yang <liezhi.yang@windriver.com>"
 RECIPE_MAINTAINER_pn-cups = "Chen Qi <Qi.Chen@windriver.com>"
 RECIPE_MAINTAINER_pn-curl = "Armin Kuster <akuster808@gmail.com>"
 RECIPE_MAINTAINER_pn-cve-check-tool = "Ross Burton <ross.burton@intel.com>"
+RECIPE_MAINTAINER_pn-cve-update-db-native = "Ross Burton <ross.burton@intel.com>"
 RECIPE_MAINTAINER_pn-cwautomacros = "Ross Burton <ross.burton@intel.com>"
 RECIPE_MAINTAINER_pn-db = "Mark Hatle <mark.hatle@windriver.com>"
 RECIPE_MAINTAINER_pn-dbus = "Chen Qi <Qi.Chen@windriver.com>"
diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
index 1b676dc26e..97d83cb856 100644
--- a/meta/recipes-core/glibc/glibc-locale.inc
+++ b/meta/recipes-core/glibc/glibc-locale.inc
@@ -95,3 +95,6 @@ do_install () {
 inherit libc-package
 
 BBCLASSEXTEND = "nativesdk"
+
+# Don't scan for CVEs as glibc will be scanned
+CVE_PRODUCT = ""
diff --git a/meta/recipes-core/glibc/glibc-mtrace.inc b/meta/recipes-core/glibc/glibc-mtrace.inc
index d703c14bdc..ef9d60ec23 100644
--- a/meta/recipes-core/glibc/glibc-mtrace.inc
+++ b/meta/recipes-core/glibc/glibc-mtrace.inc
@@ -11,3 +11,6 @@ do_install() {
 	install -d -m 0755 ${D}${bindir}
 	install -m 0755 ${SRC}/mtrace ${D}${bindir}/
 }
+
+# Don't scan for CVEs as glibc will be scanned
+CVE_PRODUCT = ""
diff --git a/meta/recipes-core/glibc/glibc-scripts.inc b/meta/recipes-core/glibc/glibc-scripts.inc
index 2a2b41507e..14a14e4512 100644
--- a/meta/recipes-core/glibc/glibc-scripts.inc
+++ b/meta/recipes-core/glibc/glibc-scripts.inc
@@ -18,3 +18,6 @@ do_install() {
 # sotruss script requires sotruss-lib.so (given by libsotruss package), 
 # to produce trace of the library calls.
 RDEPENDS_${PN} += "libsotruss"
+
+# Don't scan for CVEs as glibc will be scanned
+CVE_PRODUCT = ""
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
new file mode 100644
index 0000000000..2c427a5884
--- /dev/null
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -0,0 +1,195 @@
+SUMMARY = "Updates the NVD CVE database"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+inherit native
+
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+deltask do_populate_sysroot
+
+python () {
+    if not d.getVar("CVE_CHECK_DB_FILE"):
+        raise bb.parse.SkipRecipe("Skip recipe when cve-check class is not loaded.")
+}
+
+python do_populate_cve_db() {
+    """
+    Update NVD database with json data feed
+    """
+
+    import sqlite3, urllib, urllib.parse, shutil, gzip
+    from datetime import date
+
+    BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"
+    YEAR_START = 2002
+
+    db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK')
+    db_file = os.path.join(db_dir, 'nvdcve_1.0.db')
+    json_tmpfile = os.path.join(db_dir, 'nvd.json.gz')
+    proxy = d.getVar("https_proxy")
+
+    if proxy:
+        # instantiate an opener but do not install it as the global
+        # opener unless if we're really sure it's applicable for all
+        # urllib requests
+        proxy_handler = urllib.request.ProxyHandler({'https': proxy})
+        proxy_opener = urllib.request.build_opener(proxy_handler)
+    else:
+        proxy_opener = None
+
+    cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a')
+
+    if not os.path.isdir(db_dir):
+        os.mkdir(db_dir)
+
+    # Connect to database
+    conn = sqlite3.connect(db_file)
+    c = conn.cursor()
+
+    initialize_db(c)
+
+    for year in range(YEAR_START, date.today().year + 1):
+        year_url = BASE_URL + str(year)
+        meta_url = year_url + ".meta"
+        json_url = year_url + ".json.gz"
+
+        # Retrieve meta last modified date
+
+        response = None
+
+        if proxy_opener:
+            response = proxy_opener.open(meta_url)
+        else:
+            req = urllib.request.Request(meta_url)
+            response = urllib.request.urlopen(req)
+
+        if response:
+            for l in response.read().decode("utf-8").splitlines():
+                key, value = l.split(":", 1)
+                if key == "lastModifiedDate":
+                    last_modified = value
+                    break
+            else:
+                bb.warn("Cannot parse CVE metadata, update failed")
+                return
+
+        # Compare with current db last modified date
+        c.execute("select DATE from META where YEAR = ?", (year,))
+        meta = c.fetchone()
+        if not meta or meta[0] != last_modified:
+            # Clear products table entries corresponding to current year
+            c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,))
+
+            # Update db with current year json file
+            try:
+                if proxy_opener:
+                    response = proxy_opener.open(json_url)
+                else:
+                    req = urllib.request.Request(json_url)
+                    response = urllib.request.urlopen(req)
+
+                if response:
+                    update_db(c, gzip.decompress(response.read()).decode('utf-8'))
+                c.execute("insert or replace into META values (?, ?)", [year, last_modified])
+            except urllib.error.URLError as e:
+                cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n')
+                bb.warn("Cannot parse CVE data (%s), update failed" % e.reason)
+                return
+
+        # Update success, set the date to cve_check file.
+        if year == date.today().year:
+            cve_f.write('CVE database update : %s\n\n' % date.today())
+
+    cve_f.close()
+    conn.commit()
+    conn.close()
+}
+
+def initialize_db(c):
+    c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)")
+    c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \
+        SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)")
+    c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \
+        VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \
+        VERSION_END TEXT, OPERATOR_END TEXT)")
+
+def parse_node_and_insert(c, node, cveId):
+    # Parse children node if needed
+    for child in node.get('children', ()):
+        parse_node_and_insert(c, child, cveId)
+
+    def cpe_generator():
+        for cpe in node.get('cpe_match', ()):
+            if not cpe['vulnerable']:
+                return
+            cpe23 = cpe['cpe23Uri'].split(':')
+            vendor = cpe23[3]
+            product = cpe23[4]
+            version = cpe23[5]
+
+            if version != '*':
+                # Version is defined, this is a '=' match
+                yield [cveId, vendor, product, version, '=', '', '']
+            else:
+                # Parse start version, end version and operators
+                op_start = ''
+                op_end = ''
+                v_start = ''
+                v_end = ''
+
+                if 'versionStartIncluding' in cpe:
+                    op_start = '>='
+                    v_start = cpe['versionStartIncluding']
+
+                if 'versionStartExcluding' in cpe:
+                    op_start = '>'
+                    v_start = cpe['versionStartExcluding']
+
+                if 'versionEndIncluding' in cpe:
+                    op_end = '<='
+                    v_end = cpe['versionEndIncluding']
+
+                if 'versionEndExcluding' in cpe:
+                    op_end = '<'
+                    v_end = cpe['versionEndExcluding']
+
+                yield [cveId, vendor, product, v_start, op_start, v_end, op_end]
+
+    c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator())
+
+def update_db(c, jsondata):
+    import json
+    root = json.loads(jsondata)
+
+    for elt in root['CVE_Items']:
+        if not elt['impact']:
+            continue
+
+        cveId = elt['cve']['CVE_data_meta']['ID']
+        cveDesc = elt['cve']['description']['description_data'][0]['value']
+        date = elt['lastModifiedDate']
+        accessVector = elt['impact']['baseMetricV2']['cvssV2']['accessVector']
+        cvssv2 = elt['impact']['baseMetricV2']['cvssV2']['baseScore']
+
+        try:
+            cvssv3 = elt['impact']['baseMetricV3']['cvssV3']['baseScore']
+        except:
+            cvssv3 = 0.0
+
+        c.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)",
+                [cveId, cveDesc, cvssv2, cvssv3, date, accessVector])
+
+        configurations = elt['configurations']['nodes']
+        for config in configurations:
+            parse_node_and_insert(c, config, cveId)
+
+
+addtask do_populate_cve_db before do_fetch
+do_populate_cve_db[nostamp] = "1"
+
+EXCLUDE_FROM_WORLD = "1"
diff --git a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
deleted file mode 100644
index 1c84fb1cf2..0000000000
--- a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
+++ /dev/null
@@ -1,62 +0,0 @@
-SUMMARY = "cve-check-tool"
-DESCRIPTION = "cve-check-tool is a tool for checking known (public) CVEs.\
-The tool will identify potentially vunlnerable software packages within Linux distributions through version matching."
-HOMEPAGE = "https://github.com/ikeydoherty/cve-check-tool"
-SECTION = "Development/Tools"
-LICENSE = "GPL-2.0+"
-LIC_FILES_CHKSUM = "file://LICENSE;md5=e8c1458438ead3c34974bc0be3a03ed6"
-
-SRC_URI = "https://github.com/ikeydoherty/${BPN}/releases/download/v${PV}/${BP}.tar.xz \
-           file://check-for-malloc_trim-before-using-it.patch \
-           file://0001-print-progress-in-percent-when-downloading-CVE-db.patch \
-           file://0001-curl-allow-overriding-default-CA-certificate-file.patch \
-           file://0001-update-Compare-computed-vs-expected-sha256-digit-str.patch \
-           file://0001-Fix-freeing-memory-allocated-by-sqlite.patch \
-          "
-
-SRC_URI[md5sum] = "c5f4247140fc9be3bf41491d31a34155"
-SRC_URI[sha256sum] = "b8f283be718af8d31232ac1bfc10a0378fb958aaaa49af39168f8acf501e6a5b"
-
-UPSTREAM_CHECK_URI = "https://github.com/ikeydoherty/cve-check-tool/releases"
-
-DEPENDS = "libcheck glib-2.0 json-glib curl libxml2 sqlite3 openssl ca-certificates"
-
-RDEPENDS_${PN} = "ca-certificates"
-
-inherit pkgconfig autotools
-
-EXTRA_OECONF = "--disable-coverage --enable-relative-plugins"
-CFLAGS_append = " -Wno-error=pedantic"
-
-do_populate_cve_db() {
-    if [ "${BB_NO_NETWORK}" = "1" ] ; then
-        bbwarn "BB_NO_NETWORK is set; Can't update cve-check-tool database, new CVEs won't be detected"
-        return
-    fi
-
-    # In case we don't inherit cve-check class, use default values defined in the class.
-    cve_dir="${CVE_CHECK_DB_DIR}"
-    cve_file="${CVE_CHECK_TMP_FILE}"
-
-    [ -z "${cve_dir}" ] && cve_dir="${DL_DIR}/CVE_CHECK"
-    [ -z "${cve_file}" ] && cve_file="${TMPDIR}/cve_check"
-
-    unused="${@bb.utils.export_proxies(d)}"
-    bbdebug 2 "Updating cve-check-tool database located in $cve_dir"
-    # --cacert works around curl-native not finding the CA bundle
-    if cve-check-update --cacert ${sysconfdir}/ssl/certs/ca-certificates.crt -d "$cve_dir" ; then
-        printf "CVE database was updated on %s UTC\n\n" "$(LANG=C date --utc +'%F %T')" > "$cve_file"
-    else
-        bbwarn "Error in executing cve-check-update"
-        if [ "${@'1' if bb.data.inherits_class('cve-check', d) else '0'}" -ne 0 ] ; then
-            bbwarn "Failed to update cve-check-tool database, CVEs won't be checked"
-        fi
-    fi
-}
-
-addtask populate_cve_db after do_populate_sysroot
-do_populate_cve_db[depends] = "cve-check-tool-native:do_populate_sysroot"
-do_populate_cve_db[nostamp] = "1"
-do_populate_cve_db[progress] = "percent"
-
-BBCLASSEXTEND = "native nativesdk"
diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch b/meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch
deleted file mode 100644
index 4a82cf2dde..0000000000
--- a/meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch
+++ /dev/null
@@ -1,50 +0,0 @@
-From a3353429652f83bb8b0316500faa88fa2555542d Mon Sep 17 00:00:00 2001
-From: Peter Marko <peter.marko@siemens.com>
-Date: Thu, 13 Apr 2017 23:09:52 +0200
-Subject: [PATCH] Fix freeing memory allocated by sqlite
-
-Upstream-Status: Backport
-Signed-off-by: Peter Marko <peter.marko@siemens.com>
----
- src/core.c | 8 ++++----
- 1 file changed, 4 insertions(+), 4 deletions(-)
-
-diff --git a/src/core.c b/src/core.c
-index 6263031..6788f16 100644
---- a/src/core.c
-+++ b/src/core.c
-@@ -82,7 +82,7 @@ static bool ensure_table(CveDB *self)
-         rc = sqlite3_exec(self->db, query, NULL, NULL, &err);
-         if (rc != SQLITE_OK) {
-                 fprintf(stderr, "ensure_table(): %s\n", err);
--                free(err);
-+                sqlite3_free(err);
-                 return false;
-         }
-         
-@@ -91,7 +91,7 @@ static bool ensure_table(CveDB *self)
-         rc = sqlite3_exec(self->db, query, NULL, NULL, &err);
-         if (rc != SQLITE_OK) {
-                 fprintf(stderr, "ensure_table(): %s\n", err);
--                free(err);
-+                sqlite3_free(err);
-                 return false;
-         }
- 
-@@ -99,11 +99,11 @@ static bool ensure_table(CveDB *self)
-         rc = sqlite3_exec(self->db, query, NULL, NULL, &err);
-         if (rc != SQLITE_OK) {
-                 fprintf(stderr, "ensure_table(): %s\n", err);
--                free(err);
-+                sqlite3_free(err);
-                 return false;
-         }
-         if (err) {
--                free(err);
-+                sqlite3_free(err);
-         }
- 
-         return true;
--- 
-2.1.4
-
diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch b/meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch
deleted file mode 100644
index 3d8ebd1bd2..0000000000
--- a/meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch
+++ /dev/null
@@ -1,215 +0,0 @@
-From 825a9969dea052b02ba868bdf39e676349f10dce Mon Sep 17 00:00:00 2001
-From: Jussi Kukkonen <jussi.kukkonen@intel.com>
-Date: Thu, 9 Feb 2017 14:51:28 +0200
-Subject: [PATCH] curl: allow overriding default CA certificate file
-
-Similar to curl, --cacert can now be used in cve-check-tool and
-cve-check-update to override the default CA certificate file. Useful
-in cases where the system default is unsuitable (for example,
-out-dated) or broken (as in OE's current native libcurl, which embeds
-a path string from one build host and then uses it on another although
-the right path may have become something different).
-
-Upstream-Status: Submitted [https://github.com/ikeydoherty/cve-check-tool/pull/45]
-
-Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
-
-
-Took Patrick Ohlys original patch from meta-security-isafw, rebased
-on top of other patches.
-
-Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
----
- src/library/cve-check-tool.h |  1 +
- src/library/fetch.c          | 10 +++++++++-
- src/library/fetch.h          |  3 ++-
- src/main.c                   |  5 ++++-
- src/update-main.c            |  4 +++-
- src/update.c                 | 12 +++++++-----
- src/update.h                 |  2 +-
- 7 files changed, 27 insertions(+), 10 deletions(-)
-
-diff --git a/src/library/cve-check-tool.h b/src/library/cve-check-tool.h
-index e4bb5b1..f89eade 100644
---- a/src/library/cve-check-tool.h
-+++ b/src/library/cve-check-tool.h
-@@ -43,6 +43,7 @@ typedef struct CveCheckTool {
-     bool bugs;                          /**<Whether bug tracking is enabled */
-     GHashTable *mapping;                /**<CVE Mapping */
-     const char *output_file;            /**<Output file, if any */
-+    const char *cacert_file;            /**<Non-default SSL certificate file, if any */
- } CveCheckTool;
- 
- /**
-diff --git a/src/library/fetch.c b/src/library/fetch.c
-index 0fe6d76..8f998c3 100644
---- a/src/library/fetch.c
-+++ b/src/library/fetch.c
-@@ -60,7 +60,8 @@ static int progress_callback_new(void *ptr, curl_off_t dltotal, curl_off_t dlnow
- }
- 
- FetchStatus fetch_uri(const char *uri, const char *target, bool verbose,
--                      unsigned int start_percent, unsigned int end_percent)
-+                      unsigned int start_percent, unsigned int end_percent,
-+                      const char *cacert_file)
- {
-         FetchStatus ret = FETCH_STATUS_FAIL;
-         CURLcode res;
-@@ -74,6 +75,13 @@ FetchStatus fetch_uri(const char *uri, const char *target, bool verbose,
-                 return ret;
-         }
- 
-+        if (cacert_file) {
-+                res = curl_easy_setopt(curl, CURLOPT_CAINFO, cacert_file);
-+                if (res != CURLE_OK) {
-+                        goto bail;
-+                }
-+        }
-+
-         if (stat(target, &st) == 0) {
-                 res = curl_easy_setopt(curl, CURLOPT_TIMECONDITION, CURL_TIMECOND_IFMODSINCE);
-                 if (res != CURLE_OK) {
-diff --git a/src/library/fetch.h b/src/library/fetch.h
-index 4cce5d1..836c7d7 100644
---- a/src/library/fetch.h
-+++ b/src/library/fetch.h
-@@ -29,7 +29,8 @@ typedef enum {
-  * @return A FetchStatus, indicating the operation taken
-  */
- FetchStatus fetch_uri(const char *uri, const char *target, bool verbose,
--                      unsigned int this_percent, unsigned int next_percent);
-+                      unsigned int this_percent, unsigned int next_percent,
-+                      const char *cacert_file);
- 
- /**
-  * Attempt to extract the given gzipped file
-diff --git a/src/main.c b/src/main.c
-index 8e6f158..ae69d47 100644
---- a/src/main.c
-+++ b/src/main.c
-@@ -280,6 +280,7 @@ static bool csv_mode = false;
- static char *modified_stamp = NULL;
- static gchar *mapping_file = NULL;
- static gchar *output_file = NULL;
-+static gchar *cacert_file = NULL;
- 
- static GOptionEntry _entries[] = {
-         { "not-patched", 'n', 0, G_OPTION_ARG_NONE, &hide_patched, "Hide patched/addressed CVEs", NULL },
-@@ -294,6 +295,7 @@ static GOptionEntry _entries[] = {
-         { "csv", 'c', 0, G_OPTION_ARG_NONE, &csv_mode, "Output CSV formatted data only", NULL },
-         { "mapping", 'M', 0, G_OPTION_ARG_STRING, &mapping_file, "Path to a mapping file", NULL},
-         { "output-file", 'o', 0, G_OPTION_ARG_STRING, &output_file, "Path to the output file (output plugin specific)", NULL},
-+        { "cacert", 'C', 0, G_OPTION_ARG_STRING, &cacert_file, "Path to the combined SSL certificates file (system default is used if not set)", NULL},
-         { .short_name = 0 }
- };
- 
-@@ -492,6 +494,7 @@ int main(int argc, char **argv)
- 
-         quiet = csv_mode || !no_html;
-         self->output_file = output_file;
-+        self->cacert_file = cacert_file;
- 
-         if (!csv_mode && self->output_file) {
-                 quiet = false;
-@@ -530,7 +533,7 @@ int main(int argc, char **argv)
-                 if (status) {
-                         fprintf(stderr, "Update of db forced\n");
-                         cve_db_unlock();
--                        if (!update_db(quiet, db_path->str)) {
-+                        if (!update_db(quiet, db_path->str, self->cacert_file)) {
-                                 fprintf(stderr, "DB update failure\n");
-                                 goto cleanup;
-                         }
-diff --git a/src/update-main.c b/src/update-main.c
-index 2379cfa..c52d9d0 100644
---- a/src/update-main.c
-+++ b/src/update-main.c
-@@ -43,11 +43,13 @@ the Free Software Foundation; either version 2 of the License, or\n\
- static gchar *nvds = NULL;
- static bool _show_version = false;
- static bool _quiet = false;
-+static const char *_cacert_file = NULL;
- 
- static GOptionEntry _entries[] = {
-         { "nvd-dir", 'd', 0, G_OPTION_ARG_STRING, &nvds, "NVD directory in filesystem", NULL },
-         { "version", 'v', 0, G_OPTION_ARG_NONE, &_show_version, "Show version", NULL },
-         { "quiet", 'q', 0, G_OPTION_ARG_NONE, &_quiet, "Run silently", NULL },
-+        { "cacert", 'C', 0, G_OPTION_ARG_STRING, &_cacert_file, "Path to the combined SSL certificates file (system default is used if not set)", NULL},
-         { .short_name = 0 }
- };
- 
-@@ -88,7 +90,7 @@ int main(int argc, char **argv)
-                 goto end;
-         }
- 
--        if (update_db(_quiet, db_path->str)) {
-+        if (update_db(_quiet, db_path->str, _cacert_file)) {
-                 ret = EXIT_SUCCESS;
-         } else {
-                 fprintf(stderr, "Failed to update database\n");
-diff --git a/src/update.c b/src/update.c
-index 070560a..8cb4a39 100644
---- a/src/update.c
-+++ b/src/update.c
-@@ -267,7 +267,8 @@ static inline void update_end(int fd, const char *update_fname, bool ok)
- 
- static int do_fetch_update(int year, const char *db_dir, CveDB *cve_db,
-                            bool db_exist, bool verbose,
--                           unsigned int this_percent, unsigned int next_percent)
-+                           unsigned int this_percent, unsigned int next_percent,
-+                           const char *cacert_file)
- {
-         const char nvd_uri[] = URI_PREFIX;
-         autofree(cve_string) *uri_meta = NULL;
-@@ -331,14 +332,14 @@ refetch:
-         }
- 
-         /* Fetch NVD META file */
--        st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose, this_percent, this_percent);
-+        st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose, this_percent, this_percent, cacert_file);
-         if (st == FETCH_STATUS_FAIL) {
-                 fprintf(stderr, "Failed to fetch %s\n", uri_meta->str);
-                 return -1;
-         }
- 
-         /* Fetch NVD XML file */
--        st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose, this_percent, next_percent);
-+        st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose, this_percent, next_percent, cacert_file);
-         switch (st) {
-         case FETCH_STATUS_FAIL:
-                 fprintf(stderr, "Failed to fetch %s\n", uri_data_gz->str);
-@@ -391,7 +392,7 @@ refetch:
-         return 0;
- }
- 
--bool update_db(bool quiet, const char *db_file)
-+bool update_db(bool quiet, const char *db_file, const char *cacert_file)
- {
-         autofree(char) *db_dir = NULL;
-         autofree(CveDB) *cve_db = NULL;
-@@ -466,7 +467,8 @@ bool update_db(bool quiet, const char *db_file)
-                 if (!quiet)
-                         fprintf(stderr, "completed: %u%%\r", start_percent);
-                 rc = do_fetch_update(y, db_dir, cve_db, db_exist, !quiet,
--                                     start_percent, end_percent);
-+                                     start_percent, end_percent,
-+                                     cacert_file);
-                 switch (rc) {
-                 case 0:
-                         if (!quiet)
-diff --git a/src/update.h b/src/update.h
-index b8e9911..ceea0c3 100644
---- a/src/update.h
-+++ b/src/update.h
-@@ -15,7 +15,7 @@ cve_string *get_db_path(const char *path);
- 
- int update_required(const char *db_file);
- 
--bool update_db(bool quiet, const char *db_file);
-+bool update_db(bool quiet, const char *db_file, const char *cacert_file);
- 
- 
- /*
--- 
-2.1.4
-
diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch b/meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch
deleted file mode 100644
index 8ea6f686e3..0000000000
--- a/meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch
+++ /dev/null
@@ -1,135 +0,0 @@
-From e9ed26cde63f8ca7607a010a518329339f8c02d3 Mon Sep 17 00:00:00 2001
-From: =?UTF-8?q?Andr=C3=A9=20Draszik?= <git@andred.net>
-Date: Mon, 26 Sep 2016 12:12:41 +0100
-Subject: [PATCH] print progress in percent when downloading CVE db
-MIME-Version: 1.0
-Content-Type: text/plain; charset=UTF-8
-Content-Transfer-Encoding: 8bit
-
-Upstream-Status: Pending
-Signed-off-by: André Draszik <git@andred.net>
----
- src/library/fetch.c | 28 +++++++++++++++++++++++++++-
- src/library/fetch.h |  3 ++-
- src/update.c        | 16 ++++++++++++----
- 3 files changed, 41 insertions(+), 6 deletions(-)
-
-diff --git a/src/library/fetch.c b/src/library/fetch.c
-index 06d4b30..0fe6d76 100644
---- a/src/library/fetch.c
-+++ b/src/library/fetch.c
-@@ -37,13 +37,37 @@ static size_t write_func(void *ptr, size_t size, size_t nmemb, struct fetch_t *f
-         return fwrite(ptr, size, nmemb, f->f);
- }
- 
--FetchStatus fetch_uri(const char *uri, const char *target, bool verbose)
-+struct percent_t {
-+        unsigned int start;
-+        unsigned int end;
-+};
-+
-+static int progress_callback_new(void *ptr, curl_off_t dltotal, curl_off_t dlnow, curl_off_t ultotal, curl_off_t ulnow)
-+{
-+        (void) ultotal;
-+        (void) ulnow;
-+
-+        struct percent_t *percent = (struct percent_t *) ptr;
-+
-+        if (dltotal && percent && percent->end >= percent->start) {
-+                unsigned int diff = percent->end - percent->start;
-+                if (diff) {
-+                        fprintf(stderr,"completed: %"CURL_FORMAT_CURL_OFF_T"%%\r", percent->start + (diff * dlnow / dltotal));
-+                }
-+        }
-+
-+        return 0;
-+}
-+
-+FetchStatus fetch_uri(const char *uri, const char *target, bool verbose,
-+                      unsigned int start_percent, unsigned int end_percent)
- {
-         FetchStatus ret = FETCH_STATUS_FAIL;
-         CURLcode res;
-         struct stat st;
-         CURL *curl = NULL;
-         struct fetch_t *f = NULL;
-+        struct percent_t percent = { .start = start_percent, .end = end_percent };
- 
-         curl = curl_easy_init();
-         if (!curl) {
-@@ -67,6 +91,8 @@ FetchStatus fetch_uri(const char *uri, const char *target, bool verbose)
-         }
-         if (verbose) {
-                 (void)curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
-+                (void)curl_easy_setopt(curl, CURLOPT_XFERINFODATA, &percent);
-+                (void)curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, progress_callback_new);
-         }
-         res = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, (curl_write_callback)write_func);
-         if (res != CURLE_OK) {
-diff --git a/src/library/fetch.h b/src/library/fetch.h
-index 70c3779..4cce5d1 100644
---- a/src/library/fetch.h
-+++ b/src/library/fetch.h
-@@ -28,7 +28,8 @@ typedef enum {
-  * @param verbose Whether to be verbose
-  * @return A FetchStatus, indicating the operation taken
-  */
--FetchStatus fetch_uri(const char *uri, const char *target, bool verbose);
-+FetchStatus fetch_uri(const char *uri, const char *target, bool verbose,
-+                      unsigned int this_percent, unsigned int next_percent);
- 
- /**
-  * Attempt to extract the given gzipped file
-diff --git a/src/update.c b/src/update.c
-index 30fbe96..eaeeefd 100644
---- a/src/update.c
-+++ b/src/update.c
-@@ -266,7 +266,8 @@ static inline void update_end(int fd, const char *update_fname, bool ok)
- }
- 
- static int do_fetch_update(int year, const char *db_dir, CveDB *cve_db,
--                           bool db_exist, bool verbose)
-+                           bool db_exist, bool verbose,
-+                           unsigned int this_percent, unsigned int next_percent)
- {
-         const char nvd_uri[] = URI_PREFIX;
-         autofree(cve_string) *uri_meta = NULL;
-@@ -330,14 +331,14 @@ refetch:
-         }
- 
-         /* Fetch NVD META file */
--        st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose);
-+        st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose, this_percent, this_percent);
-         if (st == FETCH_STATUS_FAIL) {
-                 fprintf(stderr, "Failed to fetch %s\n", uri_meta->str);
-                 return -1;
-         }
- 
-         /* Fetch NVD XML file */
--        st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose);
-+        st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose, this_percent, next_percent);
-         switch (st) {
-         case FETCH_STATUS_FAIL:
-                 fprintf(stderr, "Failed to fetch %s\n", uri_data_gz->str);
-@@ -459,10 +460,17 @@ bool update_db(bool quiet, const char *db_file)
-         for (int i = YEAR_START; i <= year+1; i++) {
-                 int y = i > year ? -1 : i;
-                 int rc;
-+                unsigned int start_percent = ((i+0 - YEAR_START) * 100) / (year+2 - YEAR_START);
-+                unsigned int end_percent = ((i+1 - YEAR_START) * 100) / (year+2 - YEAR_START);
- 
--                rc = do_fetch_update(y, db_dir, cve_db, db_exist, !quiet);
-+                if (!quiet)
-+                        fprintf(stderr, "completed: %u%%\r", start_percent);
-+                rc = do_fetch_update(y, db_dir, cve_db, db_exist, !quiet,
-+                                     start_percent, end_percent);
-                 switch (rc) {
-                 case 0:
-+                        if (!quiet)
-+                                fprintf(stderr,"completed: %u%%\r", end_percent);
-                         continue;
-                 case ENOMEM:
-                         goto oom;
--- 
-2.9.3
-
diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch b/meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch
deleted file mode 100644
index 458c0cc84e..0000000000
--- a/meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch
+++ /dev/null
@@ -1,52 +0,0 @@
-From b0426e63c9ac61657e029f689bcb8dd051e752c6 Mon Sep 17 00:00:00 2001
-From: Sergey Popovich <popovich_sergei@mail.ua>
-Date: Fri, 21 Apr 2017 07:32:23 -0700
-Subject: [PATCH] update: Compare computed vs expected sha256 digit string
- ignoring case
-
-We produce sha256 digest string using %x snprintf()
-qualifier for each byte of digest which uses alphabetic
-characters from "a" to "f" in lower case to represent
-integer values from 10 to 15.
-
-Previously all of the NVD META files supply sha256
-digest string for corresponding XML file in lower case.
-
-However due to some reason this changed recently to
-provide digest digits in upper case causing fetched
-data consistency checks to fail. This prevents database
-from being updated periodically.
-
-While commit c4f6e94 (update: Do not treat sha256 failure
-as fatal if requested) adds useful option to skip
-digest validation at all and thus provides workaround for
-this situation, it might be unacceptable for some
-deployments where we need to ensure that downloaded
-data is consistent before start parsing it and update
-SQLite database.
-
-Use strcasecmp() to compare two digest strings case
-insensitively and addressing this case.
-
-Upstream-Status: Backport
-Signed-off-by: Sergey Popovich <popovich_sergei@mail.ua>
----
- src/update.c | 2 +-
- 1 file changed, 1 insertion(+), 1 deletion(-)
-
-diff --git a/src/update.c b/src/update.c
-index 8588f38..3cc6b67 100644
---- a/src/update.c
-+++ b/src/update.c
-@@ -187,7 +187,7 @@ static bool nvdcve_data_ok(const char *meta, const char *data)
-                 snprintf(&csum_data[idx], len, "%02hhx", digest[i]);
-         }
- 
--        ret = streq(csum_meta, csum_data);
-+        ret = !strcasecmp(csum_meta, csum_data);
- 
- err_unmap:
-         munmap(buffer, length);
--- 
-2.11.0
-
diff --git a/meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch b/meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch
deleted file mode 100644
index 0774ad946a..0000000000
--- a/meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch
+++ /dev/null
@@ -1,51 +0,0 @@
-From ce64633b9733e962b8d8482244301f614d8b5845 Mon Sep 17 00:00:00 2001
-From: Khem Raj <raj.khem@gmail.com>
-Date: Mon, 22 Aug 2016 22:54:24 -0700
-Subject: [PATCH] Check for malloc_trim before using it
-
-malloc_trim is gnu specific and not all libc
-implement it, threfore write a configure check
-to poke for it first and use the define to
-guard its use.
-
-Helps in compiling on musl based systems
-
-Signed-off-by: Khem Raj <raj.khem@gmail.com>
----
-Upstream-Status: Submitted [https://github.com/ikeydoherty/cve-check-tool/pull/48]
- configure.ac | 2 ++
- src/core.c   | 4 ++--
- 2 files changed, 4 insertions(+), 2 deletions(-)
-
-diff --git a/configure.ac b/configure.ac
-index d3b66ce..79c3542 100644
---- a/configure.ac
-+++ b/configure.ac
-@@ -19,6 +19,8 @@ m4_define([json_required_version], [0.16.0])
- m4_define([openssl_required_version],[1.0.0])
- # TODO: Set minimum sqlite
- 
-+AC_CHECK_FUNCS_ONCE(malloc_trim)
-+
- PKG_CHECK_MODULES(CVE_CHECK_TOOL,
-                  [
-                   glib-2.0 >= glib_required_version,
-diff --git a/src/core.c b/src/core.c
-index 6263031..0d5df29 100644
---- a/src/core.c
-+++ b/src/core.c
-@@ -498,9 +498,9 @@ bool cve_db_load(CveDB *self, const char *fname)
-         }
- 
-         b = true;
--
-+#ifdef HAVE_MALLOC_TRIM
-         malloc_trim(0);
--
-+#endif
-         xmlFreeTextReader(r);
-         if (fd) {
-                 close(fd);
--- 
-2.9.3
-
-- 
2.20.1




^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [thud-next][PATCH 02/11] cve-check: ensure all known CVEs are in the report
  2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
@ 2019-12-08 18:35 ` Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 03/11] cve-check: failure to parse versions should be more visible Niko Mauno
                   ` (8 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core

From: Ross Burton <ross.burton@intel.com>

CVEs that are whitelisted or were not vulnerable when there are version
comparisons were not included in the report, so alter the logic to ensure that
all relevant CVEs are in the report for completeness.

(From OE-Core rev: 98256ff05fcfe9d5ccad360582c36eafb577c264)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/cve-check.bbclass | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index c00d2910be..f87bcc9dc6 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -208,12 +208,14 @@ def check_cves(d, patched_cves):
 
             if cve in cve_whitelist:
                 bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve))
+                # TODO: this should be in the report as 'whitelisted'
+                patched_cves.add(cve)
             elif cve in patched_cves:
                 bb.note("%s has been patched" % (cve))
             else:
                 to_append = False
                 if (operator_start == '=' and pv == version_start):
-                    cves_unpatched.append(cve)
+                    to_append = True
                 else:
                     if operator_start:
                         try:
@@ -243,8 +245,11 @@ def check_cves(d, patched_cves):
                         to_append = to_append_start or to_append_end
 
                 if to_append:
+                    bb.note("%s-%s is vulnerable to %s" % (product, pv, cve))
                     cves_unpatched.append(cve)
-                bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve))
+                else:
+                    bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve))
+                    patched_cves.add(cve)
     conn.close()
 
     return (list(patched_cves), cves_unpatched)
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [thud-next][PATCH 03/11] cve-check: failure to parse versions should be more visible
  2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 02/11] cve-check: ensure all known CVEs are in the report Niko Mauno
@ 2019-12-08 18:35 ` Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 04/11] cve-check: we don't actually need to unpack to check Niko Mauno
                   ` (7 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core

From: Ross Burton <ross.burton@intel.com>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/cve-check.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index f87bcc9dc6..1c8b2223a2 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -222,7 +222,7 @@ def check_cves(d, patched_cves):
                             to_append_start =  (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start))
                             to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start))
                         except:
-                            bb.note("%s: Failed to compare %s %s %s for %s" %
+                            bb.warn("%s: Failed to compare %s %s %s for %s" %
                                     (product, pv, operator_start, version_start, cve))
                             to_append_start = False
                     else:
@@ -233,7 +233,7 @@ def check_cves(d, patched_cves):
                             to_append_end  = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end))
                             to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end))
                         except:
-                            bb.note("%s: Failed to compare %s %s %s for %s" %
+                            bb.warn("%s: Failed to compare %s %s %s for %s" %
                                     (product, pv, operator_end, version_end, cve))
                             to_append_end = False
                     else:
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [thud-next][PATCH 04/11] cve-check: we don't actually need to unpack to check
  2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 02/11] cve-check: ensure all known CVEs are in the report Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 03/11] cve-check: failure to parse versions should be more visible Niko Mauno
@ 2019-12-08 18:35 ` Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 05/11] cve-update-db-native: don't refresh more than once an hour Niko Mauno
                   ` (6 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core

From: Ross Burton <ross.burton@intel.com>

The patch scanner works with patch files in the layer, not in the workdir, so it
doesn't need to unpack.

(From OE-Core rev: 2cba6ada970deb5156e1ba0182f4f372851e3c17)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/cve-check.bbclass | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 1c8b2223a2..3326944d79 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -62,7 +62,7 @@ python do_cve_check () {
 
 }
 
-addtask cve_check after do_unpack before do_build
+addtask cve_check before do_build
 do_cve_check[depends] = "cve-update-db-native:do_populate_cve_db"
 do_cve_check[nostamp] = "1"
 
@@ -70,7 +70,6 @@ python cve_check_cleanup () {
     """
     Delete the file used to gather all the CVE information.
     """
-
     bb.utils.remove(e.data.getVar("CVE_CHECK_TMP_FILE"))
 }
 
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [thud-next][PATCH 05/11] cve-update-db-native: don't refresh more than once an hour
  2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
                   ` (2 preceding siblings ...)
  2019-12-08 18:35 ` [thud-next][PATCH 04/11] cve-check: we don't actually need to unpack to check Niko Mauno
@ 2019-12-08 18:35 ` Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 06/11] cve-update-db-native: don't hardcode the database name Niko Mauno
                   ` (5 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core

From: Ross Burton <ross.burton@intel.com>

We already fetch the yearly CVE metadata and check that for updates before
downloading the full data, but we can speed up CVE checking further by only
checking the CVE metadata once an hour.

(From OE-Core rev: 50d898fd360c58fe85460517d965f62b7654771a)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-core/meta/cve-update-db-native.bb | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
index 2c427a5884..19875a49b1 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -31,8 +31,16 @@ python do_populate_cve_db() {
     db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK')
     db_file = os.path.join(db_dir, 'nvdcve_1.0.db')
     json_tmpfile = os.path.join(db_dir, 'nvd.json.gz')
-    proxy = d.getVar("https_proxy")
 
+    # Don't refresh the database more than once an hour
+    try:
+        import time
+        if time.time() - os.path.getmtime(db_file) < (60*60):
+            return
+    except OSError:
+        pass
+
+    proxy = d.getVar("https_proxy")
     if proxy:
         # instantiate an opener but do not install it as the global
         # opener unless if we're really sure it's applicable for all
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [thud-next][PATCH 06/11] cve-update-db-native: don't hardcode the database name
  2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
                   ` (3 preceding siblings ...)
  2019-12-08 18:35 ` [thud-next][PATCH 05/11] cve-update-db-native: don't refresh more than once an hour Niko Mauno
@ 2019-12-08 18:35 ` Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 07/11] cve-update-db-native: add an index on the CVE ID column Niko Mauno
                   ` (4 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core

From: Ross Burton <ross.burton@intel.com>

Don't hardcode the database filename, there's a variable for this in
cve-check.bbclass.

(From OE-Core rev: 0d188a9dc4ae64c64cd661e9d9c3841e86f226ab)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-core/meta/cve-update-db-native.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
index 19875a49b1..c15534de08 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -28,8 +28,8 @@ python do_populate_cve_db() {
     BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"
     YEAR_START = 2002
 
-    db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK')
-    db_file = os.path.join(db_dir, 'nvdcve_1.0.db')
+    db_file = d.getVar("CVE_CHECK_DB_FILE")
+    db_dir = os.path.dirname(db_file)
     json_tmpfile = os.path.join(db_dir, 'nvd.json.gz')
 
     # Don't refresh the database more than once an hour
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [thud-next][PATCH 07/11] cve-update-db-native: add an index on the CVE ID column
  2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
                   ` (4 preceding siblings ...)
  2019-12-08 18:35 ` [thud-next][PATCH 06/11] cve-update-db-native: don't hardcode the database name Niko Mauno
@ 2019-12-08 18:35 ` Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 08/11] cve-update-db-native: clean up proxy handling Niko Mauno
                   ` (3 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core

From: Ross Burton <ross.burton@intel.com>

Create an index on the PRODUCTS table which contains a row for each CPE,
drastically increasing the performance of lookups for a specific CVE.

(From OE-Core rev: b4048b05b3a00d85c40d09961f846eadcebd812e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-core/meta/cve-update-db-native.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
index c15534de08..08b18f064f 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -120,11 +120,14 @@ python do_populate_cve_db() {
 
 def initialize_db(c):
     c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)")
+
     c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \
         SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)")
+
     c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \
         VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \
         VERSION_END TEXT, OPERATOR_END TEXT)")
+    c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_ID_IDX on PRODUCTS(ID);")
 
 def parse_node_and_insert(c, node, cveId):
     # Parse children node if needed
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [thud-next][PATCH 08/11] cve-update-db-native: clean up proxy handling
  2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
                   ` (5 preceding siblings ...)
  2019-12-08 18:35 ` [thud-next][PATCH 07/11] cve-update-db-native: add an index on the CVE ID column Niko Mauno
@ 2019-12-08 18:35 ` Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 09/11] cve-check: rewrite look to fix false negatives Niko Mauno
                   ` (2 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core

From: Ross Burton <ross.burton@intel.com>

urllib handles adding proxy handlers if the proxies are set in the environment,
so call bb.utils.export_proxies() to do that and remove the manual setup.

(From OE-Core rev: 6b73004668b3b71c9c38814b79fbb58c893ed434)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../recipes-core/meta/cve-update-db-native.bb | 31 +++----------------
 1 file changed, 5 insertions(+), 26 deletions(-)

diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
index 08b18f064f..db1d69a28e 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -21,10 +21,12 @@ python do_populate_cve_db() {
     """
     Update NVD database with json data feed
     """
-
+    import bb.utils
     import sqlite3, urllib, urllib.parse, shutil, gzip
     from datetime import date
 
+    bb.utils.export_proxies(d)
+
     BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"
     YEAR_START = 2002
 
@@ -40,16 +42,6 @@ python do_populate_cve_db() {
     except OSError:
         pass
 
-    proxy = d.getVar("https_proxy")
-    if proxy:
-        # instantiate an opener but do not install it as the global
-        # opener unless if we're really sure it's applicable for all
-        # urllib requests
-        proxy_handler = urllib.request.ProxyHandler({'https': proxy})
-        proxy_opener = urllib.request.build_opener(proxy_handler)
-    else:
-        proxy_opener = None
-
     cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a')
 
     if not os.path.isdir(db_dir):
@@ -67,15 +59,7 @@ python do_populate_cve_db() {
         json_url = year_url + ".json.gz"
 
         # Retrieve meta last modified date
-
-        response = None
-
-        if proxy_opener:
-            response = proxy_opener.open(meta_url)
-        else:
-            req = urllib.request.Request(meta_url)
-            response = urllib.request.urlopen(req)
-
+        response = urllib.request.urlopen(meta_url)
         if response:
             for l in response.read().decode("utf-8").splitlines():
                 key, value = l.split(":", 1)
@@ -95,12 +79,7 @@ python do_populate_cve_db() {
 
             # Update db with current year json file
             try:
-                if proxy_opener:
-                    response = proxy_opener.open(json_url)
-                else:
-                    req = urllib.request.Request(json_url)
-                    response = urllib.request.urlopen(req)
-
+                response = urllib.request.urlopen(json_url)
                 if response:
                     update_db(c, gzip.decompress(response.read()).decode('utf-8'))
                 c.execute("insert or replace into META values (?, ?)", [year, last_modified])
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [thud-next][PATCH 09/11] cve-check: rewrite look to fix false negatives
  2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
                   ` (6 preceding siblings ...)
  2019-12-08 18:35 ` [thud-next][PATCH 08/11] cve-update-db-native: clean up proxy handling Niko Mauno
@ 2019-12-08 18:35 ` Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 10/11] cve-check: neaten get_cve_info Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 11/11] cve-check: fetch CVE data once at a time instead of in a single call Niko Mauno
  9 siblings, 0 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core

From: Ross Burton <ross.burton@intel.com>

A previous optimisation was premature and resulted in false-negatives in the report.

Rewrite the checking algorithm to first get the list of potential CVEs by
vendor:product, then iterate through every matching CPE for that CVE to
determine if the bounds match or not.  By doing this in two stages we can know
if we've checked every CPE, instead of accidentally breaking out of the scan too
early.

(From OE-Core rev: d61aff9e22704ad69df1f7ab0f8784f4e7cc0c69)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/cve-check.bbclass | 63 ++++++++++++++++++----------------
 1 file changed, 34 insertions(+), 29 deletions(-)

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 3326944d79..c1cbdbde7b 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -165,7 +165,6 @@ def check_cves(d, patched_cves):
     """
     Connect to the NVD database and find unpatched cves.
     """
-    import ast, csv, tempfile, subprocess, io
     from distutils.version import LooseVersion
 
     cves_unpatched = []
@@ -187,68 +186,74 @@ def check_cves(d, patched_cves):
     cve_whitelist = d.getVar("CVE_CHECK_WHITELIST").split()
 
     import sqlite3
-    db_file = d.getVar("CVE_CHECK_DB_FILE")
-    conn = sqlite3.connect(db_file)
+    db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro")
+    conn = sqlite3.connect(db_file, uri=True)
 
+    # For each of the known product names (e.g. curl has CPEs using curl and libcurl)...
     for product in products:
-        c = conn.cursor()
         if ":" in product:
             vendor, product = product.split(":", 1)
-            c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR IS ?", (product, vendor))
         else:
-            c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,))
+            vendor = "%"
 
-        for row in c:
-            cve = row[0]
-            version_start = row[3]
-            operator_start = row[4]
-            version_end = row[5]
-            operator_end = row[6]
+        # Find all relevant CVE IDs.
+        for cverow in conn.execute("SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR LIKE ?", (product, vendor)):
+            cve = cverow[0]
 
             if cve in cve_whitelist:
                 bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve))
                 # TODO: this should be in the report as 'whitelisted'
                 patched_cves.add(cve)
+                continue
             elif cve in patched_cves:
                 bb.note("%s has been patched" % (cve))
-            else:
-                to_append = False
+                continue
+
+            vulnerable = False
+            for row in conn.execute("SELECT * FROM PRODUCTS WHERE ID IS ? AND PRODUCT IS ? AND VENDOR LIKE ?", (cve, product, vendor)):
+                (_, _, _, version_start, operator_start, version_end, operator_end) = row
+                #bb.debug(2, "Evaluating row " + str(row))
+
                 if (operator_start == '=' and pv == version_start):
-                    to_append = True
+                    vulnerable = True
                 else:
                     if operator_start:
                         try:
-                            to_append_start =  (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start))
-                            to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start))
+                            vulnerable_start =  (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start))
+                            vulnerable_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start))
                         except:
                             bb.warn("%s: Failed to compare %s %s %s for %s" %
                                     (product, pv, operator_start, version_start, cve))
-                            to_append_start = False
+                            vulnerable_start = False
                     else:
-                        to_append_start = False
+                        vulnerable_start = False
 
                     if operator_end:
                         try:
-                            to_append_end  = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end))
-                            to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end))
+                            vulnerable_end  = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end))
+                            vulnerable_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end))
                         except:
                             bb.warn("%s: Failed to compare %s %s %s for %s" %
                                     (product, pv, operator_end, version_end, cve))
-                            to_append_end = False
+                            vulnerable_end = False
                     else:
-                        to_append_end = False
+                        vulnerable_end = False
 
                     if operator_start and operator_end:
-                        to_append = to_append_start and to_append_end
+                        vulnerable = vulnerable_start and vulnerable_end
                     else:
-                        to_append = to_append_start or to_append_end
+                        vulnerable = vulnerable_start or vulnerable_end
 
-                if to_append:
+                if vulnerable:
                     bb.note("%s-%s is vulnerable to %s" % (product, pv, cve))
                     cves_unpatched.append(cve)
-                else:
-                    bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve))
-                    patched_cves.add(cve)
+                    break
+
+            if not vulnerable:
+                bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve))
+                # TODO: not patched but not vulnerable
+                patched_cves.add(cve)
+
     conn.close()
 
     return (list(patched_cves), cves_unpatched)
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [thud-next][PATCH 10/11] cve-check: neaten get_cve_info
  2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
                   ` (7 preceding siblings ...)
  2019-12-08 18:35 ` [thud-next][PATCH 09/11] cve-check: rewrite look to fix false negatives Niko Mauno
@ 2019-12-08 18:35 ` Niko Mauno
  2019-12-08 18:35 ` [thud-next][PATCH 11/11] cve-check: fetch CVE data once at a time instead of in a single call Niko Mauno
  9 siblings, 0 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core

From: Ross Burton <ross.burton@intel.com>

Remove obsolete Python 2 code, and use convenience methods for neatness.

(From OE-Core rev: f19253cc9e70c974a8e21a142086c13d7cde04ff)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/cve-check.bbclass | 18 +++++-------------
 1 file changed, 5 insertions(+), 13 deletions(-)

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index c1cbdbde7b..e95716d9de 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -261,23 +261,15 @@ def check_cves(d, patched_cves):
 def get_cve_info(d, cves):
     """
     Get CVE information from the database.
-
-    Unfortunately the only way to get CVE info is set the output to
-    html (hard to parse) or query directly the database.
     """
 
-    try:
-        import sqlite3
-    except ImportError:
-        from pysqlite2 import dbapi2 as sqlite3
+    import sqlite3
 
     cve_data = {}
-    db_file = d.getVar("CVE_CHECK_DB_FILE")
-    placeholder = ",".join("?" * len(cves))
-    query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholder
-    conn = sqlite3.connect(db_file)
-    cur = conn.cursor()
-    for row in cur.execute(query, tuple(cves)):
+    conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE"))
+    placeholders = ",".join("?" * len(cves))
+    query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders
+    for row in conn.execute(query, tuple(cves)):
         cve_data[row[0]] = {}
         cve_data[row[0]]["summary"] = row[1]
         cve_data[row[0]]["scorev2"] = row[2]
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [thud-next][PATCH 11/11] cve-check: fetch CVE data once at a time instead of in a single call
  2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
                   ` (8 preceding siblings ...)
  2019-12-08 18:35 ` [thud-next][PATCH 10/11] cve-check: neaten get_cve_info Niko Mauno
@ 2019-12-08 18:35 ` Niko Mauno
  9 siblings, 0 replies; 11+ messages in thread
From: Niko Mauno @ 2019-12-08 18:35 UTC (permalink / raw)
  To: openembedded-core

From: Ross Burton <ross.burton@intel.com>

This code used to construct a single SQL statement that fetched the NVD data for
every CVE requested.  For recipes such as the kernel where there are over 2000
CVEs to report this can hit the variable count limit and the query fails with
"sqlite3.OperationalError: too many SQL variables".  The default limit is 999
variables, but some distributions such as Debian set the default to 250000.

As the NVD table has an index on the ID column, whilst requesting the data
CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time
different is insignificant: 0.05s verses 0.01s on my machine.

(From OE-Core rev: 53d0cc1e9b7190fa66d7ff1c59518f91b0128d99)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/cve-check.bbclass | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index e95716d9de..19ed5548b3 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -267,17 +267,17 @@ def get_cve_info(d, cves):
 
     cve_data = {}
     conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE"))
-    placeholders = ",".join("?" * len(cves))
-    query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders
-    for row in conn.execute(query, tuple(cves)):
-        cve_data[row[0]] = {}
-        cve_data[row[0]]["summary"] = row[1]
-        cve_data[row[0]]["scorev2"] = row[2]
-        cve_data[row[0]]["scorev3"] = row[3]
-        cve_data[row[0]]["modified"] = row[4]
-        cve_data[row[0]]["vector"] = row[5]
-    conn.close()
 
+    for cve in cves:
+        for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)):
+            cve_data[row[0]] = {}
+            cve_data[row[0]]["summary"] = row[1]
+            cve_data[row[0]]["scorev2"] = row[2]
+            cve_data[row[0]]["scorev3"] = row[3]
+            cve_data[row[0]]["modified"] = row[4]
+            cve_data[row[0]]["vector"] = row[5]
+
+    conn.close()
     return cve_data
 
 def cve_write_data(d, patched, unpatched, cve_data):
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2019-12-08 18:41 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-12-08 18:35 [thud-next][PATCH 01/11] cve-check: backport rewrite from master Niko Mauno
2019-12-08 18:35 ` [thud-next][PATCH 02/11] cve-check: ensure all known CVEs are in the report Niko Mauno
2019-12-08 18:35 ` [thud-next][PATCH 03/11] cve-check: failure to parse versions should be more visible Niko Mauno
2019-12-08 18:35 ` [thud-next][PATCH 04/11] cve-check: we don't actually need to unpack to check Niko Mauno
2019-12-08 18:35 ` [thud-next][PATCH 05/11] cve-update-db-native: don't refresh more than once an hour Niko Mauno
2019-12-08 18:35 ` [thud-next][PATCH 06/11] cve-update-db-native: don't hardcode the database name Niko Mauno
2019-12-08 18:35 ` [thud-next][PATCH 07/11] cve-update-db-native: add an index on the CVE ID column Niko Mauno
2019-12-08 18:35 ` [thud-next][PATCH 08/11] cve-update-db-native: clean up proxy handling Niko Mauno
2019-12-08 18:35 ` [thud-next][PATCH 09/11] cve-check: rewrite look to fix false negatives Niko Mauno
2019-12-08 18:35 ` [thud-next][PATCH 10/11] cve-check: neaten get_cve_info Niko Mauno
2019-12-08 18:35 ` [thud-next][PATCH 11/11] cve-check: fetch CVE data once at a time instead of in a single call Niko Mauno

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.