[meta-oe][PATCH 1/3] cve-check: Depends on cve-update-db-native

[meta-oe][PATCH 1/3] cve-check: Depends on cve-update-db-native

[Pierre Le Magourou]

From: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>

do_populate_cve_db is a native task.

Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
---
 meta/classes/cve-check.bbclass          | 2 +-
 meta/recipes-core/meta/cve-update-db.bb | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 81071e3f19..6ffa0c4688 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -63,7 +63,7 @@ python do_cve_check () {
 }
 
 addtask cve_check after do_unpack before do_build
-do_cve_check[depends] = "cve-update-db:do_populate_cve_db"
+do_cve_check[depends] = "cve-update-db-native:do_populate_cve_db"
 do_cve_check[nostamp] = "1"
 
 python cve_check_cleanup () {
diff --git a/meta/recipes-core/meta/cve-update-db.bb b/meta/recipes-core/meta/cve-update-db.bb
index ae8f1a958b..8e553b4f9b 100644
--- a/meta/recipes-core/meta/cve-update-db.bb
+++ b/meta/recipes-core/meta/cve-update-db.bb
@@ -141,3 +141,4 @@ addtask do_populate_cve_db before do_fetch
 do_populate_cve_db[nostamp] = "1"
 
 EXCLUDE_FROM_WORLD = "1"
+BBCLASSEXTEND =+ "native"
-- 
2.11.0

[meta-oe][PATCH 2/3] cve-update-db: Use NVD CPE data to populate PRODUCTS table

[Pierre Le Magourou]

From: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>

Instead of using expanded list of affected versions that is not
reliable, use the 'cpe_match' node in the 'configurations' json node.

For cve-check to correctly match affected CVE, the sqlite database need to
contain operator_start, operator_end and the corresponding versions fields.

Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
---
 meta/recipes-core/meta/cve-update-db.bb | 88 +++++++++++++++++++++++++++------
 1 file changed, 74 insertions(+), 14 deletions(-)

diff --git a/meta/recipes-core/meta/cve-update-db.bb b/meta/recipes-core/meta/cve-update-db.bb
index 8e553b4f9b..3ba80a0d28 100644
--- a/meta/recipes-core/meta/cve-update-db.bb
+++ b/meta/recipes-core/meta/cve-update-db.bb
@@ -25,7 +25,7 @@ python do_populate_cve_db() {
     YEAR_START = 2002
 
     db_dir = d.getVar("DL_DIR") + '/CVE_CHECK'
-    db_file = db_dir + '/nvd-json.db'
+    db_file = db_dir + '/nvdcve.db'
     json_tmpfile = db_dir + '/nvd.json.gz'
     proxy = d.getVar("https_proxy")
     cve_f = open(d.getVar("TMPDIR") + '/cve_check', 'a')
@@ -99,9 +99,76 @@ def initialize_db(c):
     c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \
         SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)")
     c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (HASH INTEGER UNIQUE, ID TEXT, \
-        VENDOR TEXT, PRODUCT TEXT, VERSION TEXT, OPERATOR TEXT)")
-    c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_IDX ON PRODUCTS \
-        (PRODUCT, VERSION)")
+        VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \
+        VERSION_END TEXT, OPERATOR_END TEXT)")
+
+def insert_elt(c, db_values):
+    product_str = db_values[0] + db_values[1] + db_values[2] + db_values[3]
+    hashstr = hash_djb2(product_str)
+    db_values.insert(0, hashstr)
+    query = "insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?, ?, ?)"
+    c.execute(query, db_values)
+
+def parse_node_and_insert(c, node, cveId):
+    # Parse children node if needed
+    try:
+        for child in node['children']:
+            parse_node_and_insert(c, child, cveId)
+    except:
+        pass
+
+    # Exit if the cpe_match node does not exists
+    try:
+        cpe_match = node['cpe_match']
+    except:
+        return
+
+    for cpe in cpe_match:
+        if not cpe['vulnerable']:
+            return
+        cpe23 = cpe['cpe23Uri'].split(':')
+        vendor = cpe23[3]
+        product = cpe23[4]
+        version = cpe23[5]
+
+        if version != '*':
+            # Version is defined, this is a '=' match
+            db_values = [cveId, vendor, product, version, '=', '', '']
+            insert_elt(c, db_values)
+        else:
+            # Parse start version, end version and operators
+            op_start = ''
+            op_end = ''
+            v_start = ''
+            v_end = ''
+
+            try:
+                if cpe['versionStartIncluding']:
+                    op_start = '>='
+                    v_start = cpe['versionStartIncluding']
+            except:
+                pass
+            try:
+                if cpe['versionStartExcluding']:
+                    op_start = '>'
+                    v_start = cpe['versionStartExcluding']
+            except:
+                pass
+            try:
+                if cpe['versionEndIncluding']:
+                    op_end = '<='
+                    v_end = cpe['versionEndIncluding']
+            except:
+                pass
+            try:
+                if cpe['versionEndExcluding']:
+                    op_end = '<'
+                    v_end = cpe['versionEndExcluding']
+            except:
+                pass
+
+            db_values = [cveId, vendor, product, v_start, op_start, v_end, op_end]
+            insert_elt(c, db_values)
 
 def update_db(c, json_filename):
     import json
@@ -125,16 +192,9 @@ def update_db(c, json_filename):
         c.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)",
                 [cveId, cveDesc, cvssv2, cvssv3, date, accessVector])
 
-        for vendor in elt['cve']['affects']['vendor']['vendor_data']:
-            for product in vendor['product']['product_data']:
-                for version in product['version']['version_data']:
-                    product_str = cveId+vendor['vendor_name']+product['product_name']+version['version_value']
-                    hashstr = hash_djb2(product_str)
-                    c.execute("insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?)",
-                            [ hashstr, cveId, vendor['vendor_name'],
-                                product['product_name'], version['version_value'],
-                                version['version_affected']])
-
+        configurations = elt['configurations']['nodes']
+        for config in configurations:
+            parse_node_and_insert(c, config, cveId)
 
 
 addtask do_populate_cve_db before do_fetch
-- 
2.11.0

[meta-oe][PATCH 3/3] cve-check: Update unpatched CVE matching

[Pierre Le Magourou]

From: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>

Now that cve-update-db added CPE information to NVD database. We can
check for unpatched versions with operators '<', '<=', '>', and '>='.

Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
---
 meta/classes/cve-check.bbclass | 54 +++++++++++++++++++++++++++++++-----------
 1 file changed, 40 insertions(+), 14 deletions(-)

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 6ffa0c4688..ffd624333f 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}"
 CVE_VERSION ??= "${PV}"
 
 CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
-CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd-json.db"
+CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve.db"
 
 CVE_CHECK_LOG ?= "${T}/cve.log"
 CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
@@ -189,27 +189,53 @@ def check_cves(d, patched_cves):
     conn = sqlite3.connect(db_file)
     c = conn.cursor()
 
-    query = """SELECT * FROM PRODUCTS WHERE
-               (PRODUCT IS '{0}' AND VERSION = '{1}' AND OPERATOR IS '=') OR
-               (PRODUCT IS '{0}' AND OPERATOR IS '<=');"""
+    query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '{0}';"
+
     for product in products:
         for row in c.execute(query.format(product, pv)):
             cve = row[1]
-            version = row[4]
-
-            try:
-                discardVersion = LooseVersion(version) < LooseVersion(pv)
-            except:
-                discardVersion = True
+            version_start = row[4]
+            operator_start = row[5]
+            version_end = row[6]
+            operator_end = row[7]
 
             if pv in cve_whitelist.get(cve, []):
                 bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve))
             elif cve in patched_cves:
                 bb.note("%s has been patched" % (cve))
-            elif discardVersion:
-                bb.debug(2, "Do not consider version %s " % (version))
             else:
-                cves_unpatched.append(cve)
+                if (operator_start == '=' and pv == version_start):
+                    cves_unpatched.append(cve)
+                else:
+                    if operator_start:
+                        try:
+                            to_append_start =  (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start))
+                            to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start))
+                        except:
+                            bb.note("%s: Failed to compare %s %s %s for %s" %
+                                    (product, pv, operator_start, version_start, cve))
+                            to_append_start = False
+                    else:
+                        to_append_start = False
+
+                    if operator_end:
+                        try:
+                            to_append_end  = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end))
+                            to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end))
+                        except:
+                            bb.note("%s: Failed to compare %s %s %s for %s" %
+                                    (product, pv, operator_end, version_end, cve))
+                            to_append_end = False
+                    else:
+                        to_append_end = False
+
+                    if operator_start and operator_end:
+                        to_append = to_append_start and to_append_end
+                    else:
+                        to_append = to_append_start or to_append_end
+
+                if to_append:
+                    cves_unpatched.append(cve)
                 bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve))
     conn.close()
 
@@ -217,7 +243,7 @@ def check_cves(d, patched_cves):
 
 def get_cve_info(d, cves):
     """
-    Get CVE information from the database used by cve-check-tool.
+    Get CVE information from the database.
 
     Unfortunately the only way to get CVE info is set the output to
     html (hard to parse) or query directly the database.
-- 
2.11.0

✗ patchtest: failure for "[meta-oe] cve-check: Depends o..." and 2 more

[Patchwork]

== Series Details ==

Series: "[meta-oe] cve-check: Depends o..." and 2 more
Revision: 1
URL   : https://patchwork.openembedded.org/series/18529/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch            [meta-oe,1/3] cve-check: Depends on cve-update-db-native
 Issue             Series sent to the wrong mailing list [test_target_mailing_list] 
  Suggested fix    Check the project's README (meta-oe,1/3) and send the patch to the indicated list

* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at a0a96db236)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: [meta-oe][PATCH 1/3] cve-check: Depends on cve-update-db-native

[Richard Purdie]

On Thu, 2019-07-04 at 17:19 +0200, Pierre Le Magourou wrote:
> From: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
> 
> do_populate_cve_db is a native task.
> 
> Signed-off-by: Pierre Le Magourou <
> pierre.lemagourou@softbankrobotics.com>
> ---
>  meta/classes/cve-check.bbclass          | 2 +-
>  meta/recipes-core/meta/cve-update-db.bb | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)


https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/1015

:(

Cheers,

Richard

Re: [meta-oe][PATCH 1/3] cve-check: Depends on cve-update-db-native

[Pierre Le Magourou]

Hello,

> > From: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
> >
> > do_populate_cve_db is a native task.
> >
> > Signed-off-by: Pierre Le Magourou <
> > pierre.lemagourou@softbankrobotics.com>
> > ---
> >  meta/classes/cve-check.bbclass          | 2 +-
> >  meta/recipes-core/meta/cve-update-db.bb | 1 +
> >  2 files changed, 2 insertions(+), 1 deletion(-)
>
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/1015
>
> :(

The autobuilder job launches cve-update-db and cve-update-db-native in
parallel, this cannot work because they both try to update the same
sqlite db.
As cve-update-db is only used for the build machine (native only), we
only need a cve-update-db-native.bb recipe. I'll send a v2 patch to
fix that.

ps: Sorry I just realized I was sending the patches to the wrong
destination (meta-oe instead of oe-core)

Pierre