[PATCH 0/3] Add initial capability to check CVEs for recipes

[PATCH 0/3] Add initial capability to check CVEs for recipes

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This series add the cve-check-tool recipe, a tool used to identify
potentially vulnerable software through version matching. It will
check if a vulnerability has been addressed by a patch.

Also add the new cve-check class that will add a task for all recipes
to check for CVEs using cve-check-tool. This tool can be used by recipe,
image (will generate an image report in deploy dir), and with "world"
and "universe"

To run it just inherit the class and enter:

bitbake -c cve_check <recipe>

The following changes since commit 23056103c949b498c23b47579e8dd57ce78e6ed9:

  uclibc: Do not use immediate expansion operator (2016-02-22 20:42:48 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib mariano/bug7515
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug7515

Mariano Lopez (3):
  cve-check-tool: Add recipe
  cve-check-tool patch to allow select dir for the db
  cve-check.bbclass: Add class

 meta/classes/cve-check.bbclass                     | 229 +++++++++++++++++++++
 .../change_logic_cve_get_file_parent.patch         |  45 ++++
 .../cve-check-tool/cve-check-tool_5.6.bb           |  61 ++++++
 3 files changed, 335 insertions(+)
 create mode 100644 meta/classes/cve-check.bbclass
 create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch
 create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb

-- 
2.6.2

[PATCH 1/3] cve-check-tool: Add recipe

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

cve-check-tool is a program to for checking public CVEs.
This tool also seek to determine if a vulnerability has
been addressed by a patch.

The recipe also includes the do_populate_cve_db task
that will populate the database used by the tool. This
task is added when the cve-check class has been inherited.

[YOCTO #7515]

Co-authored by Elena Reshetova & Mariano Lopez

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 .../cve-check-tool/cve-check-tool_5.6.bb           | 61 ++++++++++++++++++++++
 1 file changed, 61 insertions(+)
 create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb

diff --git a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
new file mode 100644
index 0000000..b173eab
--- /dev/null
+++ b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
@@ -0,0 +1,61 @@
+SUMMARY = "cve-check-tool"
+DESCRIPTION = "cve-check-tool is a tool for checking known (public) CVEs.\
+The tool will identify potentially vunlnerable software packages within Linux distributions through version matching."
+HOMEPAGE = "https://github.com/ikeydoherty/cve-check-tool"
+SECTION = "Development/Tools"
+LICENSE = "GPL-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=e8c1458438ead3c34974bc0be3a03ed6"
+
+SRC_URI = "https://github.com/ikeydoherty/${BPN}/releases/download/v${PV}/${BP}.tar.xz \
+            file://change_logic_cve_get_file_parent.patch"
+
+SRC_URI[md5sum] = "30f32e6254580162eacfcc437a144463"
+SRC_URI[sha256sum] = "d35af2bfa014b9d7cdc9c59ec0bd7df40c22dfcd57244c9099c0aa9bdc9c0cb4"
+
+DEPENDS = "libcheck glib-2.0 json-glib curl libxml2 sqlite3 openssl"
+
+inherit pkgconfig autotools
+
+EXTRA_OECONF = "--disable-static"
+
+python () {
+    # If the cve-check class is inherited it is needed to populate the
+    # CVE database before checking the CVEs for all recipes.
+    pn = d.getVar("PN", True)
+    if pn.endswith("-native") and bb.data.inherits_class('cve-check', d):
+        bb.build.addtask("do_populate_cve_db", "do_build", "do_populate_sysroot", d)
+}
+
+python do_populate_cve_db () {
+    import subprocess
+    import time
+    from bb.utils import export_proxies
+
+    export_proxies(d)
+    fail_text = "Failed to update database"
+    error_str = fail_text
+    cve_dir = d.getVar("CVE_CHECK_DB_DIR", True)
+    cmd = "cve-check-update -d %s" % cve_dir
+    bb.debug(1, "Updating cve-check-tool database located in %s" % cve_dir)
+    try:
+        popen = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        output, error = popen.communicate()
+        bb.debug(2, "Command %s returned:\n%s" % (cmd, output.decode()))
+        error_str = error.decode()
+        bb.debug(2, "Command %s errors:\n%s" % (cmd, error_str))
+    except:
+        bb.warn("Error in executing cve-check-update: %s" % str(sys.exc_info()))
+
+    if fail_text in error_str:
+        bb.warn("Failed to update cve-check-tool database, CVEs won't be checked")
+    else:
+        utc_time = time.gmtime(time.time())
+        format_time = "%Y-%m-%d %H:%M:%S"
+        with open(d.getVar("CVE_CHECK_TMP_FILE", True), "w") as f:
+            f.write("CVE database was updated on %s UTC\n\n"
+                    % time.strftime(format_time, utc_time))
+}
+
+do_populate_cve_db[nostamp] = "1"
+
+BBCLASSEXTEND = "native"
-- 
2.6.2

Re: [PATCH 1/3] cve-check-tool: Add recipe

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

On 24 February 2016 at 15:27, <mariano.lopez@linux.intel.com> wrote:

> +python () {
> +    # If the cve-check class is inherited it is needed to populate the
> +    # CVE database before checking the CVEs for all recipes.
> +    pn = d.getVar("PN", True)
> +    if pn.endswith("-native") and bb.data.inherits_class('cve-check', d):
> +        bb.build.addtask("do_populate_cve_db", "do_build",
> "do_populate_sysroot", d)
> +}
>

Wouldn't this be better done as an explicit dependency of the class task?

Ross

[-- Attachment #2: Type: text/html, Size: 1022 bytes --]

[PATCH 2/3] cve-check-tool patch to allow select dir for the db

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This patch allows to select the directory for the
database used by cve-check-tool.

[YOCTO #7515]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 .../change_logic_cve_get_file_parent.patch         | 45 ++++++++++++++++++++++
 1 file changed, 45 insertions(+)
 create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch

diff --git a/meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch b/meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch
new file mode 100644
index 0000000..077de88
--- /dev/null
+++ b/meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch
@@ -0,0 +1,45 @@
+From 22cc9186909f98f024d78a08504d0bf532806de0 Mon Sep 17 00:00:00 2001
+From: Mariano Lopez <mariano.lopez@linux.intel.com>
+Date: Thu, 18 Feb 2016 14:26:02 +0000
+Subject: [PATCH] util.c: Change logic in cve_get_file_parent()
+
+Function cve_get_file_parent() will try to get the
+realpath and the get the dirname. If the file used
+to get parent doesn't exist the call will fail.
+
+This problem is present when using another directory
+for the database, realpath() won't find the nvd.db
+file and the program will exit quitely.
+
+This patch will first get the dirname and the get
+the realpath to avoid failing when the doesn't exist.
+
+Upstream-Status: Accepted [Release v5.6.3]
+
+Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
+---
+ src/library/util.c | 8 +++-----
+ 1 file changed, 3 insertions(+), 5 deletions(-)
+
+diff --git a/src/library/util.c b/src/library/util.c
+index 8a20728..4d4a576 100644
+--- a/src/library/util.c
++++ b/src/library/util.c
+@@ -184,11 +184,9 @@ bool cve_is_dir(const char *p)
+ 
+ char *cve_get_file_parent(const char *p)
+ {
+-        char *r = realpath(p, NULL);
+-        if (!r) {
+-                return NULL;
+-        }
+-        return dirname(r);
++        autofree(char) *d = strdup(p);
++        char *r = realpath(dirname(d), NULL);
++        return r;
+ }
+ 
+ bool cve_file_set_text(const char *path, char *text)
+-- 
+2.6.2
+
-- 
2.6.2

Re: [PATCH 2/3] cve-check-tool patch to allow select dir for the db

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 197 bytes --]

On 24 February 2016 at 15:27, <mariano.lopez@linux.intel.com> wrote:

> This patch allows to select the directory for the
> database used by cve-check-tool.
>

Squash this into 1/3.

Ross

[-- Attachment #2: Type: text/html, Size: 612 bytes --]

Re: [PATCH 2/3] cve-check-tool patch to allow select dir for the db

[Mariano Lopez]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]



On 02/25/2016 07:33 AM, Burton, Ross wrote:
>
> On 24 February 2016 at 15:27, <mariano.lopez@linux.intel.com 
> <mailto:mariano.lopez@linux.intel.com>> wrote:
>
>     This patch allows to select the directory for the
>     database used by cve-check-tool.
>
>
> Squash this into 1/3.
>
> Ross

This was the only patch that I made my own, that is why I sent it as 
separated patch. If you check the other patches there are co-authors in 
there. I don't want someone else take the blame if my patch bust 
something :)

Mariano

[-- Attachment #2: Type: text/html, Size: 1622 bytes --]

[PATCH 3/3] cve-check.bbclass: Add class

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This class adds a new task for all the recipes to use
cve-check-tool in order to look for public CVEs affecting
the packages generated.

It is possible to use this class when building an image,
building a recipe, or using the "world" or "universe" cases.

In order to use this class it must be inherited at some
point and it will add the task automatically to every recipe.

[YOCTO #7515]

Co-authored by Ross Burton & Mariano Lopez

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/cve-check.bbclass | 229 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 229 insertions(+)
 create mode 100644 meta/classes/cve-check.bbclass

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
new file mode 100644
index 0000000..69d90f3
--- /dev/null
+++ b/meta/classes/cve-check.bbclass
@@ -0,0 +1,229 @@
+#
+CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
+CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db"
+
+CVE_CHECK_LOCAL_DIR ?= "${WORKDIR}/cve"
+CVE_CHECK_LOCAL_FILE ?= "${CVE_CHECK_LOCAL_DIR}/cve.log"
+CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
+
+CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve"
+CVE_CHECK_MANIFEST ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cve"
+CVE_CHECK_COPY_FILES ??= "1"
+CVE_CHECK_CREATE_MANIFEST ??= "1"
+
+# Whitelist for packages (PN)
+cve_check_pn_whitelist () {
+    glibc-locale
+}
+
+# Whitelist for CVE and version of package
+python cve_check_cve_whitelist () {
+    {"CVE-2014-2524": ("6.3",), \
+    }
+}
+
+python do_cve_check () {
+    """
+    Check recipe for patched and unpatched CVEs
+    """
+
+    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
+        patched_cves = get_patches_cves(d)
+        patched, unpatched = check_cves(d, patched_cves)
+        if patched or unpatched:
+            cve_data = get_cve_info(d, patched + unpatched)
+            cve_write_data(d, patched, unpatched, cve_data)
+    else:
+        bb.note("Failed to update CVE database, skipping CVE check")
+}
+
+addtask cve_check before do_build
+do_cve_check[depends] = "cve-check-tool-native:do_populate_cve_db"
+do_cve_check[nostamp] = "1"
+
+python cve_check_cleanup () {
+    """
+    Delete the file used to gather all the CVE information.
+    """
+    import bb.utils
+
+    tmp_file = e.data.getVar("CVE_CHECK_TMP_FILE", True)
+    bb.utils.remove(tmp_file)
+}
+
+addhandler cve_check_cleanup
+cve_check_cleanup[eventmask] = "bb.cooker.CookerExit"
+
+python cve_check_write_rootfs_manifest () {
+    """
+    Create CVE manifest when building an image
+    """
+
+    import shutil
+    from bb.utils import mkdirhier
+
+    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)) and \
+            d.getVar("CVE_CHECK_CREATE_MANIFEST", True) == "1":
+        bb.note("Writing rootfs CVE manifest")
+        deploy_dir = d.getVar("DEPLOY_DIR_IMAGE", True)
+        link_name = d.getVar("IMAGE_LINK_NAME", True)
+        manifest_name = d.getVar("CVE_CHECK_MANIFEST", True)
+        cve_tmp_file = d.getVar("CVE_CHECK_TMP_FILE", True)
+
+        shutil.copyfile(cve_tmp_file, manifest_name)
+
+        if manifest_name is not None and os.path.exists(manifest_name):
+            manifest_link = os.path.join(deploy_dir, "%s.cve" % link_name)
+            if os.path.exists(manifest_link):
+                if d.getVar('RM_OLD_IMAGE', True) == "1" and \
+                        os.path.exists(os.path.realpath(manifest_link)):
+                    os.remove(os.path.realpath(manifest_link))
+                os.remove(manifest_link)
+            os.symlink(os.path.basename(manifest_name), manifest_link)
+            bb.plain("Image CVE report stored in: %s" % manifest_name)
+}
+
+ROOTFS_POSTPROCESS_COMMAND_prepend = "cve_check_write_rootfs_manifest; "
+
+
+def get_patches_cves(d):
+    """
+    Get patches that solve CVEs using the "CVE: " tag.
+    """
+
+    import re
+
+    pn = d.getVar("PN", True)
+    cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
+    patched_cves = set()
+    for url in src_patches(d):
+        patch_file = bb.fetch.decodeurl(url)[2]
+        with open(patch_file, "r") as f:
+            patch_text = f.read()
+
+        # Search for the "CVE: " line
+        match = cve_match.search(patch_text)
+        if match:
+            # Get only the CVEs without the "CVE: " tag
+            cves = patch_text[match.start()+5:match.end()]
+            for cve in cves.split():
+                patched_cves.add(cve)
+
+    return patched_cves
+
+def check_cves(d, patched_cves):
+    """
+    Run cve-check-tool looking for patched and unpatched CVEs.
+    """
+
+    from bb.utils import export_proxies
+    import ast, csv, tempfile, subprocess, StringIO
+
+    cves_patched = []
+    cves_unpatched = []
+    bpn = d.getVar("BPN", True)
+    pv = d.getVar("PV", True)
+    cves = " ".join(patched_cves)
+    cve_dir = d.getVar("CVE_CHECK_DB_DIR", True)
+    cve_whitelist = ast.literal_eval(d.getVar("cve_check_cve_whitelist", True).strip())
+    cmd = "cve-check-tool --no-html --csv --not-affected -t faux -d %s" % cve_dir
+
+    # If the recipe has been whitlisted we return empty lists
+    if d.getVar("PN", True) in d.getVar("cve_check_pn_whitelist", True).split():
+        return ([], [])
+
+    # It is needed to export the proxies to download the database using HTTP
+    export_proxies(d)
+    # Write the faux CSV file to be used with cve-check-tool
+    fd, faux = tempfile.mkstemp(prefix="cve-faux-")
+    with os.fdopen(fd, "w") as f:
+        f.write("%s,%s,%s," % (bpn, pv, cves))
+
+    cmd += " %s" % faux
+    try:
+        popen = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        output, error = popen.communicate()
+    except:
+        bb.warn("Couldn't check CVEs %s" % str(sys.exc_info()))
+    finally:
+        os.remove(faux)
+
+    for row in csv.reader(StringIO.StringIO(output)):
+        if row[2]:
+            for cve in row[2].split():
+                # Skip if the CVE has been whitlisted for the current version
+                if pv not in cve_whitelist.get(cve,[]):
+                    cves_unpatched.append(cve)
+        if row[3]:
+            for cve in row[3].split():
+                cves_patched.append(cve)
+
+    return (cves_patched, cves_unpatched)
+
+def get_cve_info(d, cves):
+    """
+    Get CVE information from the database used by cve-check-tool.
+    """
+
+    try:
+        import sqlite3
+    except ImportError:
+        from pysqlite2 import dbapi2 as sqlite3
+
+    cve_data = {}
+    db_file = d.getVar("CVE_CHECK_DB_FILE", True)
+    placeholder = ",".join("?" * len(cves))
+    query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholder
+    conn = sqlite3.connect(db_file)
+    cur = conn.cursor()
+    for row in cur.execute(query, tuple(cves)):
+        cve_data[row[0]] = {}
+        cve_data[row[0]]["summary"] = row[1]
+        cve_data[row[0]]["score"] = row[2]
+        cve_data[row[0]]["modified"] = row[3]
+        cve_data[row[0]]["vector"] = row[4]
+    conn.close()
+
+    return cve_data
+
+def cve_write_data(d, patched, unpatched, cve_data):
+    """
+    Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and
+    CVE manifest if enabled.
+    """
+
+    from bb.utils import mkdirhier
+
+    cve_file = d.getVar("CVE_CHECK_LOCAL_FILE", True)
+    nvd_link = "https://web.nvd.nist.gov/view/vuln/detail?vulnId="
+    write_string = ""
+    mkdirhier(d.getVar("CVE_CHECK_LOCAL_DIR", True))
+
+    for cve in sorted(cve_data):
+        write_string += "PACKAGE NAME: %s\n" % d.getVar("PN", True)
+        write_string += "PACKAGE VERSION: %s\n" % d.getVar("PV", True)
+        write_string += "CVE: %s\n" % cve
+        if cve in patched:
+            write_string += "CVE STATUS: Patched\n"
+        else:
+            write_string += "CVE STATUS: Unpatched\n"
+            bb.warn("Found unpatched CVE, for more information check %s" % cve_file)
+        write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"]
+        write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"]
+        write_string += "VECTOR: %s\n" % cve_data[cve]["vector"]
+        write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve)
+
+    with open(cve_file, "w") as f:
+        f.write(write_string)
+
+    if d.getVar("CVE_CHECK_COPY_FILES", True) == "1":
+        cve_dir = d.getVar("CVE_CHECK_DIR", True)
+        mkdirhier(cve_dir)
+        deploy_file = os.path.join(cve_dir, d.getVar("PN", True))
+        with open(deploy_file, "w") as f:
+            f.write(write_string)
+
+    if d.getVar("CVE_CHECK_CREATE_MANIFEST", True) == "1":
+        with open(d.getVar("CVE_CHECK_TMP_FILE", True), "a") as f:
+            f.write("%s" % write_string)
+
-- 
2.6.2

Re: [PATCH 3/3] cve-check.bbclass: Add class

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 4698 bytes --]

On 24 February 2016 at 15:27, <mariano.lopez@linux.intel.com> wrote:

> +# Whitelist for packages (PN)
> +cve_check_pn_whitelist () {
> +    glibc-locale
> +}


Why is this a shell function?  CVE_CHECK_PN_WHITELIST = "glibc-locale"
please.

+# Whitelist for CVE and version of package
> +python cve_check_cve_whitelist () {
> +    {"CVE-2014-2524": ("6.3",), \
> +    }
> +}


Why is this a Python function?  Make it a bare string with implied
formatting and it can be extended from outside this class, where as this
can't.


> +addtask cve_check before do_build
>

If you're expecting to look at the sources, you'll want to be after
do_fetch too.


> +do_cve_check[depends] = "cve-check-tool-native:do_populate_cve_db"
>


And cve-check-tool-native:do_populate_sysroot.


> +def get_patches_cves(d):
> +    """
> +    Get patches that solve CVEs using the "CVE: " tag.
> +    """
> +
> +    import re
> +
> +    pn = d.getVar("PN", True)
> +    cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
>

How does this work as the backslashes are escaping the - and d and d?  Use
r"" strings.

+    patched_cves = set()
> +    for url in src_patches(d):
> +        patch_file = bb.fetch.decodeurl(url)[2]
> +        with open(patch_file, "r") as f:
> +            patch_text = f.read()
> +
> +        # Search for the "CVE: " line
> +        match = cve_match.search(patch_text)
> +        if match:
> +            # Get only the CVEs without the "CVE: " tag
> +            cves = patch_text[match.start()+5:match.end()]
> +            for cve in cves.split():
> +                patched_cves.add(cve)
>
>
Breaks for patches such as this in glibc:

meta/recipes-core/glibc/glibc/CVE-2015-9761_1.patch:CVE: CVE-2015-9761
patch #1

I'd probably look for a line that starts with "CVE:" and the use re.findall
to find all strings matching r"CVE-\d{4}-\d+"


> +    # It is needed to export the proxies to download the database using
> HTTP
> +    export_proxies(d)
>

The database has already been downloaded hasn't it?


> +    # Write the faux CSV file to be used with cve-check-tool
> +    fd, faux = tempfile.mkstemp(prefix="cve-faux-")
> +    with os.fdopen(fd, "w") as f:
> +        f.write("%s,%s,%s," % (bpn, pv, cves))
>
>
Put this inside the try incase the write fails so the file will still be
deleted.


> +    cmd += " %s" % faux
> +    try:
> +        popen = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> +        output, error = popen.communicate()
>

Don't use the low-level function, the higher level helpers are clearer.

Always write cmd as a list unless you *need* the shell.

try:
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
except CommandCalledException as e:
    bb.warn("Couldn't check for CVEs: %s (output %s)" % (e, e.output))

+def get_cve_info(d, cves):
> +    """
> +    Get CVE information from the database used by cve-check-tool.
> +    """
> +
> +    try:
> +        import sqlite3
> +    except ImportError:
> +        from pysqlite2 import dbapi2 as sqlite3
>

Isn't the output from cve-check-tool good enough?  Would it be nicer to
extend the log instead of assuming that the database format won't ever
change?

+def cve_write_data(d, patched, unpatched, cve_data):
> +    """
> +    Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and
> +    CVE manifest if enabled.
> +    """
> +
> +    from bb.utils import mkdirhier
> +
> +    cve_file = d.getVar("CVE_CHECK_LOCAL_FILE", True)
> +    nvd_link = "https://web.nvd.nist.gov/view/vuln/detail?vulnId="
> +    write_string = ""
> +    mkdirhier(d.getVar("CVE_CHECK_LOCAL_DIR", True))
> +
> +    for cve in sorted(cve_data):
> +        write_string += "PACKAGE NAME: %s\n" % d.getVar("PN", True)
> +        write_string += "PACKAGE VERSION: %s\n" % d.getVar("PV", True)
> +        write_string += "CVE: %s\n" % cve
> +        if cve in patched:
> +            write_string += "CVE STATUS: Patched\n"
> +        else:
> +            write_string += "CVE STATUS: Unpatched\n"
> +            bb.warn("Found unpatched CVE, for more information check %s"
> % cve_file)
> +        write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"]
> +        write_string += "CVSS v2 BASE SCORE: %s\n" %
> cve_data[cve]["score"]
> +        write_string += "VECTOR: %s\n" % cve_data[cve]["vector"]
> +        write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve)
> +
> +    with open(cve_file, "w") as f:
> +        f.write(write_string)
>

Just write to the file instead of to a temporary string.

Ross

[-- Attachment #2: Type: text/html, Size: 8234 bytes --]

Re: [PATCH 3/3] cve-check.bbclass: Add class

[Mariano Lopez]

[-- Attachment #1: Type: text/plain, Size: 4391 bytes --]



On 02/29/2016 08:50 AM, Burton, Ross wrote:
> On 24 February 2016 at 15:27, <mariano.lopez@linux.intel.com 
> <mailto:mariano.lopez@linux.intel.com>> wrote:
>
>     +do_cve_check[depends] = "cve-check-tool-native:do_populate_cve_db"
>
>
>
> And cve-check-tool-native:do_populate_sysroot.

cve-check-tool-native:do_populate_cve_db depends on 
cve-check-tool-native:do_populate_sysroot, so adding it there would be 
redundant.
>
>     +def get_patches_cves(d):
>     +    """
>     +    Get patches that solve CVEs using the "CVE: " tag.
>     +    """
>     +
>     +    import re
>     +
>     +    pn = d.getVar("PN", True)
>     +    cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
>
>
> How does this work as the backslashes are escaping the - and d and d?  
> Use r"" strings.

The backslashes just escape the "-", the "d" is the same as with the raw 
string. I don't really see the need to use r"" here.

>
>     +   patched_cves = set()
>     +    for url in src_patches(d):
>     +        patch_file = bb.fetch.decodeurl(url)[2]
>     +        with open(patch_file, "r") as f:
>     +            patch_text = f.read()
>     +
>     +        # Search for the "CVE: " line
>     +        match = cve_match.search(patch_text)
>     +        if match:
>     +            # Get only the CVEs without the "CVE: " tag
>     +            cves = patch_text[match.start()+5:match.end()]
>     +            for cve in cves.split():
>     +                patched_cves.add(cve)
>
>
> Breaks for patches such as this in glibc:
>
> meta/recipes-core/glibc/glibc/CVE-2015-9761_1.patch:CVE: CVE-2015-9761 
> patch #1
>
> I'd probably look for a line that starts with "CVE:" and the use 
> re.findall to find all strings matching r"CVE-\d{4}-\d+"

What do you mean by break? It does catch the CVE just fine, to test it 
just revert the glibc 2.23 update. I find cleaner to match the string in 
a single operation instead of searching for the tag line by line and 
then match the CVEs.

>     +def get_cve_info(d, cves):
>     +    """
>     +    Get CVE information from the database used by cve-check-tool.
>     +    """
>     +
>     +    try:
>     +        import sqlite3
>     +    except ImportError:
>     +        from pysqlite2 import dbapi2 as sqlite3
>
>
> Isn't the output from cve-check-tool good enough? Would it be nicer to 
> extend the log instead of assuming that the database format won't ever 
> change?

The output from cve-check-tool is only the CVE number, if that is good 
enough, the query to the database can be removed.

>
>     +def cve_write_data(d, patched, unpatched, cve_data):
>     +    """
>     +    Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and
>     +    CVE manifest if enabled.
>     +    """
>     +
>     +    from bb.utils import mkdirhier
>     +
>     +    cve_file = d.getVar("CVE_CHECK_LOCAL_FILE", True)
>     +    nvd_link = "https://web.nvd.nist.gov/view/vuln/detail?vulnId="
>     +    write_string = ""
>     +    mkdirhier(d.getVar("CVE_CHECK_LOCAL_DIR", True))
>     +
>     +    for cve in sorted(cve_data):
>     +        write_string += "PACKAGE NAME: %s\n" % d.getVar("PN", True)
>     +        write_string += "PACKAGE VERSION: %s\n" % d.getVar("PV",
>     True)
>     +        write_string += "CVE: %s\n" % cve
>     +        if cve in patched:
>     +            write_string += "CVE STATUS: Patched\n"
>     +        else:
>     +            write_string += "CVE STATUS: Unpatched\n"
>     +            bb.warn("Found unpatched CVE, for more information
>     check %s" % cve_file)
>     +        write_string += "CVE SUMMARY: %s\n" %
>     cve_data[cve]["summary"]
>     +        write_string += "CVSS v2 BASE SCORE: %s\n" %
>     cve_data[cve]["score"]
>     +        write_string += "VECTOR: %s\n" % cve_data[cve]["vector"]
>     +        write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link,
>     cve)
>     +
>     +    with open(cve_file, "w") as f:
>     +        f.write(write_string)
>
>
> Just write to the file instead of to a temporary string.

The temporary string is used for other two files, one could be copied, 
but the other appends the string content.

>
> Ross

I have implemented the rest of the comments, just need your input before 
sending a new version.

Mariano

[-- Attachment #2: Type: text/html, Size: 9648 bytes --]

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Wed, Feb 24, 2016 at 03:27:05PM +0000, mariano.lopez@linux.intel.com wrote:
> From: Mariano Lopez <mariano.lopez@linux.intel.com>
> 
> This series add the cve-check-tool recipe, a tool used to identify
> potentially vulnerable software through version matching. It will
> check if a vulnerability has been addressed by a patch.
> 
> Also add the new cve-check class that will add a task for all recipes
> to check for CVEs using cve-check-tool. This tool can be used by recipe,
> image (will generate an image report in deploy dir), and with "world"
> and "universe"
> 
> To run it just inherit the class and enter:
> 
> bitbake -c cve_check <recipe>

I tried these on yocto/dizzy but:

ERROR: Task do_cve_check in /home/builder/src/base/poky/meta/recipes-core/busybox/busybox_1.22.1.bb depends upon non-existent task do_populate_cve_db in virtual:native:/home/builder/src/base/poky/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb

Is there some simple way to make this work there too?

For testing purposes I tried this only with busybox:

$ cat busybox_%.bbappend 
inherit cve-check

The cve-check-tool itself needed a few native backports/fixes:

diff --git a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
index 9df81cb..b98d991 100644
--- a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
+++ b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
@@ -21,3 +21,5 @@ FILES_${PN} += "${datadir}/icons"
 do_install_append () {
 	install -m 0644 ${WORKDIR}/index.theme ${D}/${datadir}/icons/hicolor
 }
+
+BBCLASSEXTEND = "native"
diff --git a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
index ce00709..26f8f7f 100644
--- a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
+++ b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
@@ -18,3 +18,5 @@ SRC_URI[archive.sha256sum] = "dbf558d2da989ab84a27e4e13daa51ceaa97eb959c2c2f8097
 inherit gnome gettext lib_package
 
 EXTRA_OECONF = "--disable-introspection"
+
+BBCLASSEXTEND = "native"

-Mikko

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Thu, Feb 25, 2016 at 01:14:21PM +0100, Mikko Rapeli wrote:
> On Wed, Feb 24, 2016 at 03:27:05PM +0000, mariano.lopez@linux.intel.com wrote:
> > From: Mariano Lopez <mariano.lopez@linux.intel.com>
> > 
> > This series add the cve-check-tool recipe, a tool used to identify
> > potentially vulnerable software through version matching. It will
> > check if a vulnerability has been addressed by a patch.
> > 
> > Also add the new cve-check class that will add a task for all recipes
> > to check for CVEs using cve-check-tool. This tool can be used by recipe,
> > image (will generate an image report in deploy dir), and with "world"
> > and "universe"
> > 
> > To run it just inherit the class and enter:
> > 
> > bitbake -c cve_check <recipe>
> 
> I tried these on yocto/dizzy but:
> 
> ERROR: Task do_cve_check in /home/builder/src/base/poky/meta/recipes-core/busybox/busybox_1.22.1.bb depends upon non-existent task do_populate_cve_db in virtual:native:/home/builder/src/base/poky/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
> 
> Is there some simple way to make this work there too?
> 
> For testing purposes I tried this only with busybox:
> 
> $ cat busybox_%.bbappend 
> inherit cve-check
> 
> The cve-check-tool itself needed a few native backports/fixes:
> 
> diff --git a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
> index 9df81cb..b98d991 100644
> --- a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
> +++ b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
> @@ -21,3 +21,5 @@ FILES_${PN} += "${datadir}/icons"
>  do_install_append () {
>  	install -m 0644 ${WORKDIR}/index.theme ${D}/${datadir}/icons/hicolor
>  }
> +
> +BBCLASSEXTEND = "native"
> diff --git a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
> index ce00709..26f8f7f 100644
> --- a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
> +++ b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
> @@ -18,3 +18,5 @@ SRC_URI[archive.sha256sum] = "dbf558d2da989ab84a27e4e13daa51ceaa97eb959c2c2f8097
>  inherit gnome gettext lib_package
>  
>  EXTRA_OECONF = "--disable-introspection"
> +
> +BBCLASSEXTEND = "native"

Sorry, I guess this is needed to enable the class properly:

$ grep cve-check conf/local.conf
INHERIT += "cve-check"

but there are some other backports needed in python modules...

-Mikko

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Thu, Feb 25, 2016 at 01:29:13PM +0100, Mikko Rapeli wrote:
> On Thu, Feb 25, 2016 at 01:14:21PM +0100, Mikko Rapeli wrote:
> > On Wed, Feb 24, 2016 at 03:27:05PM +0000, mariano.lopez@linux.intel.com wrote:
> > > From: Mariano Lopez <mariano.lopez@linux.intel.com>
> > > 
> > > This series add the cve-check-tool recipe, a tool used to identify
> > > potentially vulnerable software through version matching. It will
> > > check if a vulnerability has been addressed by a patch.
> > > 
> > > Also add the new cve-check class that will add a task for all recipes
> > > to check for CVEs using cve-check-tool. This tool can be used by recipe,
> > > image (will generate an image report in deploy dir), and with "world"
> > > and "universe"
> > > 
> > > To run it just inherit the class and enter:
> > > 
> > > bitbake -c cve_check <recipe>
> > 
> > I tried these on yocto/dizzy but:

Full changes needed in dizzy are:

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 670e592..f24a584 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -893,3 +893,21 @@ def multiprocessingpool(*args, **kwargs):
 
     return multiprocessing.Pool(*args, **kwargs)
 
+# export common proxies variables from datastore to environment
+def export_proxies(d):
+    import os
+
+    variables = ['http_proxy', 'HTTP_PROXY', 'https_proxy', 'HTTPS_PROXY',
+                    'ftp_proxy', 'FTP_PROXY', 'no_proxy', 'NO_PROXY']
+    exported = False
+
+    for v in variables:
+        if v in os.environ.keys():
+            exported = True
+        else:
+            v_proxy = d.getVar(v, True)
+            if v_proxy is not None:
+                os.environ[v] = v_proxy
+                exported = True
+
+    return exported
diff --git a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
index 9df81cb..b98d991 100644
--- a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
+++ b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
@@ -21,3 +21,5 @@ FILES_${PN} += "${datadir}/icons"
 do_install_append () {
 	install -m 0644 ${WORKDIR}/index.theme ${D}/${datadir}/icons/hicolor
 }
+
+BBCLASSEXTEND = "native"
diff --git a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
index ce00709..26f8f7f 100644
--- a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
+++ b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
@@ -18,3 +18,5 @@ SRC_URI[archive.sha256sum] = "dbf558d2da989ab84a27e4e13daa51ceaa97eb959c2c2f8097
 inherit gnome gettext lib_package
 
 EXTRA_OECONF = "--disable-introspection"
+
+BBCLASSEXTEND = "native"

And with this I get nice reports with "bitbake -c cve_check openssl" to
tmp/deploy/cve/openssl.

I'll try with a full image build next, but I really, really like this stuff.

Thanks!

-Mikko

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

For openssh there must be some bugs or tunings needed to match the version
numbers used in CVE to ones in yocto. openssh-6.6p1 has zero matches
with the check but I think there are several:

downloads/CVE_CHECK$ grep openssh *xml| grep 6\.6\:p1
nvdcve-2.0-2016.xml:        <cpe-lang:fact-ref name="cpe:/a:openbsd:openssh:6.6:p1"/>
nvdcve-2.0-2016.xml:      <vuln:product>cpe:/a:openbsd:openssh:6.6:p1</vuln:product>
nvdcve-2.0-2016.xml:        <cpe-lang:fact-ref name="cpe:/a:openbsd:openssh:6.6:p1"/>
nvdcve-2.0-2016.xml:      <vuln:product>cpe:/a:openbsd:openssh:6.6:p1</vuln:product>

How should these tunings be made?

-Mikko

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

Hi,

On my developer machine the cve-check ran ok for dizzy but on build server
with sstate-cache and rmwork enabled it failed with what looks like a race
condition when scanning the patch files:

17:45:36 ERROR: Error executing a python function in /home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
17:45:36 
17:45:36 The stack trace of python calls that resulted in this exception/failure was:
17:45:36 File: 'do_cve_check', lineno: 17, function: <module>
17:45:36      0013:    else:
17:45:36      0014:        bb.note("Failed to update CVE database, skipping CVE check")
17:45:36      0015:
17:45:36      0016:
17:45:36  *** 0017:do_cve_check(d)
17:45:36      0018:
17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check
17:45:37      0004:    Check recipe for patched and unpatched CVEs
17:45:37      0005:    """
17:45:37      0006:
17:45:37      0007:    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
17:45:37  *** 0008:        patched_cves = get_patches_cves(d)
17:45:37      0009:        patched, unpatched = check_cves(d, patched_cves)
17:45:37      0010:        if patched or unpatched:
17:45:37      0011:            cve_data = get_cve_info(d, patched + unpatched)
17:45:37      0012:            cve_write_data(d, patched, unpatched, cve_data)
17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves
17:45:37      0009:    cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
17:45:37      0010:    patched_cves = set()
17:45:37      0011:    for url in src_patches(d):
17:45:37      0012:        patch_file = bb.fetch.decodeurl(url)[2]
17:45:37  *** 0013:        with open(patch_file, "r") as f:
17:45:37      0014:            patch_text = f.read()
17:45:37      0015:
17:45:37      0016:        # Search for the "CVE: " line
17:45:37      0017:        match = cve_match.search(patch_text)
17:45:37 Exception: IOError: [Errno 2] No such file or directory: '/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
17:45:37 
17:45:37 ERROR: Function failed: do_cve_check

So could this be caused by cve-check changes or is this just a side effect
of some other recipe problems?

I could not see that kind of fixes in master.

-Mikko

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mariano Lopez]

On 02/26/2016 02:14 AM, Mikko.Rapeli@bmw.de wrote:
> Hi,
>
> On my developer machine the cve-check ran ok for dizzy but on build server
> with sstate-cache and rmwork enabled it failed with what looks like a race
> condition when scanning the patch files:
>
> 17:45:36 ERROR: Error executing a python function in /home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
> 17:45:36
> 17:45:36 The stack trace of python calls that resulted in this exception/failure was:
> 17:45:36 File: 'do_cve_check', lineno: 17, function: <module>
> 17:45:36      0013:    else:
> 17:45:36      0014:        bb.note("Failed to update CVE database, skipping CVE check")
> 17:45:36      0015:
> 17:45:36      0016:
> 17:45:36  *** 0017:do_cve_check(d)
> 17:45:36      0018:
> 17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check
> 17:45:37      0004:    Check recipe for patched and unpatched CVEs
> 17:45:37      0005:    """
> 17:45:37      0006:
> 17:45:37      0007:    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
> 17:45:37  *** 0008:        patched_cves = get_patches_cves(d)
> 17:45:37      0009:        patched, unpatched = check_cves(d, patched_cves)
> 17:45:37      0010:        if patched or unpatched:
> 17:45:37      0011:            cve_data = get_cve_info(d, patched + unpatched)
> 17:45:37      0012:            cve_write_data(d, patched, unpatched, cve_data)
> 17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves
> 17:45:37      0009:    cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
> 17:45:37      0010:    patched_cves = set()
> 17:45:37      0011:    for url in src_patches(d):
> 17:45:37      0012:        patch_file = bb.fetch.decodeurl(url)[2]
> 17:45:37  *** 0013:        with open(patch_file, "r") as f:
> 17:45:37      0014:            patch_text = f.read()
> 17:45:37      0015:
> 17:45:37      0016:        # Search for the "CVE: " line
> 17:45:37      0017:        match = cve_match.search(patch_text)
> 17:45:37 Exception: IOError: [Errno 2] No such file or directory: '/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
> 17:45:37
> 17:45:37 ERROR: Function failed: do_cve_check
>
> So could this be caused by cve-check changes or is this just a side effect
> of some other recipe problems?
>
> I could not see that kind of fixes in master.
>
> -Mikko

The changes in patch series were minimal and actually this part of the 
code wasn't touched at all. That part of the code will look for all the 
files in the SRC_URI variable and will look for the "CVE:" tag in order 
to find patches that solve CVEs.

It seems the problem is with the bitbake fetcher, or the recipe; 
unfortunately the fetcher is one of the components that most change 
between releases. Another thing to check is that if actually there is a 
heirloom-mailx_12.5-1.diff file in the paths that the fetcher look for. 
You can check this in the cve_check or patch log in the work directory 
of the recipe.

Mariano

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Fri, Feb 26, 2016 at 08:48:47AM -0600, Mariano Lopez wrote:
> On 02/26/2016 02:14 AM, Mikko.Rapeli@bmw.de wrote:
> >Hi,
> >
> >On my developer machine the cve-check ran ok for dizzy but on build server
> >with sstate-cache and rmwork enabled it failed with what looks like a race
> >condition when scanning the patch files:
> >
> >17:45:36 ERROR: Error executing a python function in /home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
> >17:45:36
> >17:45:36 The stack trace of python calls that resulted in this exception/failure was:
> >17:45:36 File: 'do_cve_check', lineno: 17, function: <module>
> >17:45:36      0013:    else:
> >17:45:36      0014:        bb.note("Failed to update CVE database, skipping CVE check")
> >17:45:36      0015:
> >17:45:36      0016:
> >17:45:36  *** 0017:do_cve_check(d)
> >17:45:36      0018:
> >17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check
> >17:45:37      0004:    Check recipe for patched and unpatched CVEs
> >17:45:37      0005:    """
> >17:45:37      0006:
> >17:45:37      0007:    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
> >17:45:37  *** 0008:        patched_cves = get_patches_cves(d)
> >17:45:37      0009:        patched, unpatched = check_cves(d, patched_cves)
> >17:45:37      0010:        if patched or unpatched:
> >17:45:37      0011:            cve_data = get_cve_info(d, patched + unpatched)
> >17:45:37      0012:            cve_write_data(d, patched, unpatched, cve_data)
> >17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves
> >17:45:37      0009:    cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
> >17:45:37      0010:    patched_cves = set()
> >17:45:37      0011:    for url in src_patches(d):
> >17:45:37      0012:        patch_file = bb.fetch.decodeurl(url)[2]
> >17:45:37  *** 0013:        with open(patch_file, "r") as f:
> >17:45:37      0014:            patch_text = f.read()
> >17:45:37      0015:
> >17:45:37      0016:        # Search for the "CVE: " line
> >17:45:37      0017:        match = cve_match.search(patch_text)
> >17:45:37 Exception: IOError: [Errno 2] No such file or directory: '/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
> >17:45:37
> >17:45:37 ERROR: Function failed: do_cve_check
> >
> >So could this be caused by cve-check changes or is this just a side effect
> >of some other recipe problems?
> >
> >I could not see that kind of fixes in master.
> >
> >-Mikko
> 
> The changes in patch series were minimal and actually this part of the code
> wasn't touched at all. That part of the code will look for all the files in
> the SRC_URI variable and will look for the "CVE:" tag in order to find
> patches that solve CVEs.

Yep, the code seems straight forward.

> It seems the problem is with the bitbake fetcher, or the recipe;
> unfortunately the fetcher is one of the components that most change between
> releases. Another thing to check is that if actually there is a
> heirloom-mailx_12.5-1.diff file in the paths that the fetcher look for. You
> can check this in the cve_check or patch log in the work directory of the
> recipe.

Unfortunately the file is there if I check with devshell but I have now
four different CI runs with this failure. Only difference to my developer
machine is sstate cache. Build machines maintain their own sstate cache.

-Mikko

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Fri, Feb 26, 2016 at 03:56:24PM +0100, Mikko Rapeli wrote:
> On Fri, Feb 26, 2016 at 08:48:47AM -0600, Mariano Lopez wrote:
> > On 02/26/2016 02:14 AM, Mikko.Rapeli@bmw.de wrote:
> > >Hi,
> > >
> > >On my developer machine the cve-check ran ok for dizzy but on build server
> > >with sstate-cache and rmwork enabled it failed with what looks like a race
> > >condition when scanning the patch files:
> > >
> > >17:45:36 ERROR: Error executing a python function in /home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
> > >17:45:36
> > >17:45:36 The stack trace of python calls that resulted in this exception/failure was:
> > >17:45:36 File: 'do_cve_check', lineno: 17, function: <module>
> > >17:45:36      0013:    else:
> > >17:45:36      0014:        bb.note("Failed to update CVE database, skipping CVE check")
> > >17:45:36      0015:
> > >17:45:36      0016:
> > >17:45:36  *** 0017:do_cve_check(d)
> > >17:45:36      0018:
> > >17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check
> > >17:45:37      0004:    Check recipe for patched and unpatched CVEs
> > >17:45:37      0005:    """
> > >17:45:37      0006:
> > >17:45:37      0007:    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
> > >17:45:37  *** 0008:        patched_cves = get_patches_cves(d)
> > >17:45:37      0009:        patched, unpatched = check_cves(d, patched_cves)
> > >17:45:37      0010:        if patched or unpatched:
> > >17:45:37      0011:            cve_data = get_cve_info(d, patched + unpatched)
> > >17:45:37      0012:            cve_write_data(d, patched, unpatched, cve_data)
> > >17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves
> > >17:45:37      0009:    cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
> > >17:45:37      0010:    patched_cves = set()
> > >17:45:37      0011:    for url in src_patches(d):
> > >17:45:37      0012:        patch_file = bb.fetch.decodeurl(url)[2]
> > >17:45:37  *** 0013:        with open(patch_file, "r") as f:
> > >17:45:37      0014:            patch_text = f.read()
> > >17:45:37      0015:
> > >17:45:37      0016:        # Search for the "CVE: " line
> > >17:45:37      0017:        match = cve_match.search(patch_text)
> > >17:45:37 Exception: IOError: [Errno 2] No such file or directory: '/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
> > >17:45:37
> > >17:45:37 ERROR: Function failed: do_cve_check
> > >
> > >So could this be caused by cve-check changes or is this just a side effect
> > >of some other recipe problems?
> > >
> > >I could not see that kind of fixes in master.
> > >
> > >-Mikko
> > 
> > The changes in patch series were minimal and actually this part of the code
> > wasn't touched at all. That part of the code will look for all the files in
> > the SRC_URI variable and will look for the "CVE:" tag in order to find
> > patches that solve CVEs.
> 
> Yep, the code seems straight forward.
> 
> > It seems the problem is with the bitbake fetcher, or the recipe;
> > unfortunately the fetcher is one of the components that most change between
> > releases. Another thing to check is that if actually there is a
> > heirloom-mailx_12.5-1.diff file in the paths that the fetcher look for. You
> > can check this in the cve_check or patch log in the work directory of the
> > recipe.
> 
> Unfortunately the file is there if I check with devshell but I have now
> four different CI runs with this failure. Only difference to my developer
> machine is sstate cache. Build machines maintain their own sstate cache.

Last two runs were with v2 patches.

-Mikko

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mariano Lopez]

On 02/26/2016 08:57 AM, Mikko.Rapeli@bmw.de wrote:
> On Fri, Feb 26, 2016 at 03:56:24PM +0100, Mikko Rapeli wrote:
>> On Fri, Feb 26, 2016 at 08:48:47AM -0600, Mariano Lopez wrote:
>>> On 02/26/2016 02:14 AM, Mikko.Rapeli@bmw.de wrote:
>>>> Hi,
>>>>
>>>> On my developer machine the cve-check ran ok for dizzy but on build server
>>>> with sstate-cache and rmwork enabled it failed with what looks like a race
>>>> condition when scanning the patch files:
>>>>
>>>> 17:45:36 ERROR: Error executing a python function in /home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
>>>> 17:45:36
>>>> 17:45:36 The stack trace of python calls that resulted in this exception/failure was:
>>>> 17:45:36 File: 'do_cve_check', lineno: 17, function: <module>
>>>> 17:45:36      0013:    else:
>>>> 17:45:36      0014:        bb.note("Failed to update CVE database, skipping CVE check")
>>>> 17:45:36      0015:
>>>> 17:45:36      0016:
>>>> 17:45:36  *** 0017:do_cve_check(d)
>>>> 17:45:36      0018:
>>>> 17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check
>>>> 17:45:37      0004:    Check recipe for patched and unpatched CVEs
>>>> 17:45:37      0005:    """
>>>> 17:45:37      0006:
>>>> 17:45:37      0007:    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
>>>> 17:45:37  *** 0008:        patched_cves = get_patches_cves(d)
>>>> 17:45:37      0009:        patched, unpatched = check_cves(d, patched_cves)
>>>> 17:45:37      0010:        if patched or unpatched:
>>>> 17:45:37      0011:            cve_data = get_cve_info(d, patched + unpatched)
>>>> 17:45:37      0012:            cve_write_data(d, patched, unpatched, cve_data)
>>>> 17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves
>>>> 17:45:37      0009:    cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
>>>> 17:45:37      0010:    patched_cves = set()
>>>> 17:45:37      0011:    for url in src_patches(d):
>>>> 17:45:37      0012:        patch_file = bb.fetch.decodeurl(url)[2]
>>>> 17:45:37  *** 0013:        with open(patch_file, "r") as f:
>>>> 17:45:37      0014:            patch_text = f.read()
>>>> 17:45:37      0015:
>>>> 17:45:37      0016:        # Search for the "CVE: " line
>>>> 17:45:37      0017:        match = cve_match.search(patch_text)
>>>> 17:45:37 Exception: IOError: [Errno 2] No such file or directory: '/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
>>>> 17:45:37
>>>> 17:45:37 ERROR: Function failed: do_cve_check
>>>>
>>>> So could this be caused by cve-check changes or is this just a side effect
>>>> of some other recipe problems?
>>>>
>>>> I could not see that kind of fixes in master.
>>>>
>>>> -Mikko
>>> The changes in patch series were minimal and actually this part of the code
>>> wasn't touched at all. That part of the code will look for all the files in
>>> the SRC_URI variable and will look for the "CVE:" tag in order to find
>>> patches that solve CVEs.
>> Yep, the code seems straight forward.
>>
>>> It seems the problem is with the bitbake fetcher, or the recipe;
>>> unfortunately the fetcher is one of the components that most change between
>>> releases. Another thing to check is that if actually there is a
>>> heirloom-mailx_12.5-1.diff file in the paths that the fetcher look for. You
>>> can check this in the cve_check or patch log in the work directory of the
>>> recipe.
>> Unfortunately the file is there if I check with devshell but I have now
>> four different CI runs with this failure. Only difference to my developer
>> machine is sstate cache. Build machines maintain their own sstate cache.
> Last two runs were with v2 patches.

Would be possible to run these CI with master to check if you see the 
error too?
Also, what you can do is to put try: except:, but this won't solve the 
problem, just will hide it so the build can finish.

>
> -Mikko

Mariano Lopez

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

On 26 February 2016 at 08:14, <Mikko.Rapeli@bmw.de> wrote:

> 17:45:37  *** 0013:        with open(patch_file, "r") as f:
> 17:45:37      0014:            patch_text = f.read()
> 17:45:37      0015:
> 17:45:37      0016:        # Search for the "CVE: " line
> 17:45:37      0017:        match = cve_match.search(patch_text)
> 17:45:37 Exception: IOError: [Errno 2] No such file or directory:
> '/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
> 17:45:37
> 17:45:37 ERROR: Function failed: do_cve_check
>
> So could this be caused by cve-check changes or is this just a side effect
> of some other recipe problems?
>

Do you have rm_work enabled?

Ross

[-- Attachment #2: Type: text/html, Size: 1247 bytes --]

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Mon, Feb 29, 2016 at 02:17:26PM +0000, Burton, Ross wrote:
> On 26 February 2016 at 08:14, <Mikko.Rapeli@bmw.de> wrote:
> 
> > 17:45:37  *** 0013:        with open(patch_file, "r") as f:
> > 17:45:37      0014:            patch_text = f.read()
> > 17:45:37      0015:
> > 17:45:37      0016:        # Search for the "CVE: " line
> > 17:45:37      0017:        match = cve_match.search(patch_text)
> > 17:45:37 Exception: IOError: [Errno 2] No such file or directory:
> > '/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
> > 17:45:37
> > 17:45:37 ERROR: Function failed: do_cve_check
> >
> > So could this be caused by cve-check changes or is this just a side effect
> > of some other recipe problems?
> >
> 
> Do you have rm_work enabled?

Yes.

-Mikko

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mariano Lopez]

On 02/29/2016 08:19 AM, Mikko.Rapeli@bmw.de wrote:
> On Mon, Feb 29, 2016 at 02:17:26PM +0000, Burton, Ross wrote:
>> On 26 February 2016 at 08:14, <Mikko.Rapeli@bmw.de> wrote:
>>
>>> 17:45:37  *** 0013:        with open(patch_file, "r") as f:
>>> 17:45:37      0014:            patch_text = f.read()
>>> 17:45:37      0015:
>>> 17:45:37      0016:        # Search for the "CVE: " line
>>> 17:45:37      0017:        match = cve_match.search(patch_text)
>>> 17:45:37 Exception: IOError: [Errno 2] No such file or directory:
>>> '/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
>>> 17:45:37
>>> 17:45:37 ERROR: Function failed: do_cve_check
>>>
>>> So could this be caused by cve-check changes or is this just a side effect
>>> of some other recipe problems?
>>>
>> Do you have rm_work enabled?
> Yes.
>
> -Mikko

I think I have found the problem, when you do devshell it will execute 
do_unpack and the cve_check task must run after that for some recipes. 
Try this:

addtask cve_check after do_unpack before do_build

Sorry, to no include a diff, the diff is way bigger than just this line 
at the moment.

Mariano

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Tue, Mar 01, 2016 at 09:15:37AM -0600, Mariano Lopez wrote:
> 
> 
> On 02/29/2016 08:19 AM, Mikko.Rapeli@bmw.de wrote:
> >On Mon, Feb 29, 2016 at 02:17:26PM +0000, Burton, Ross wrote:
> >>On 26 February 2016 at 08:14, <Mikko.Rapeli@bmw.de> wrote:
> >>
> >>>17:45:37  *** 0013:        with open(patch_file, "r") as f:
> >>>17:45:37      0014:            patch_text = f.read()
> >>>17:45:37      0015:
> >>>17:45:37      0016:        # Search for the "CVE: " line
> >>>17:45:37      0017:        match = cve_match.search(patch_text)
> >>>17:45:37 Exception: IOError: [Errno 2] No such file or directory:
> >>>'/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
> >>>17:45:37
> >>>17:45:37 ERROR: Function failed: do_cve_check
> >>>
> >>>So could this be caused by cve-check changes or is this just a side effect
> >>>of some other recipe problems?
> >>>
> >>Do you have rm_work enabled?
> >Yes.
> >
> >-Mikko
> 
> I think I have found the problem, when you do devshell it will execute
> do_unpack and the cve_check task must run after that for some recipes. Try
> this:
> 
> addtask cve_check after do_unpack before do_build

Thanks, with this change the scan builds pass on dizzy.

-Mikko

Re: [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mariano Lopez]

On 02/25/2016 10:27 AM, akuster wrote:
>   So my questions in the bugs go unanswered. If you don't ever intend on
> responding just say so, I do appreciate honestly. I then know where I stand.
>
> - armin

Sorry, for not answered the bug earlier, to be honest I hadn't seen your 
comment earlier.

I just replied to the bug.

Mariano

>
> On 02/24/2016 07:27 AM, mariano.lopez@linux.intel.com wrote:
>> From: Mariano Lopez <mariano.lopez@linux.intel.com>
>>
>> This series add the cve-check-tool recipe, a tool used to identify
>> potentially vulnerable software through version matching. It will
>> check if a vulnerability has been addressed by a patch.
>>
>> Also add the new cve-check class that will add a task for all recipes
>> to check for CVEs using cve-check-tool. This tool can be used by recipe,
>> image (will generate an image report in deploy dir), and with "world"
>> and "universe"
>>
>> To run it just inherit the class and enter:
>>
>> bitbake -c cve_check <recipe>
>>
>> The following changes since commit 23056103c949b498c23b47579e8dd57ce78e6ed9:
>>
>>    uclibc: Do not use immediate expansion operator (2016-02-22 20:42:48 +0000)
>>
>> are available in the git repository at:
>>
>>    git://git.yoctoproject.org/poky-contrib mariano/bug7515
>>    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug7515
>>
>> Mariano Lopez (3):
>>    cve-check-tool: Add recipe
>>    cve-check-tool patch to allow select dir for the db
>>    cve-check.bbclass: Add class
>>
>>   meta/classes/cve-check.bbclass                     | 229 +++++++++++++++++++++
>>   .../change_logic_cve_get_file_parent.patch         |  45 ++++
>>   .../cve-check-tool/cve-check-tool_5.6.bb           |  61 ++++++
>>   3 files changed, 335 insertions(+)
>>   create mode 100644 meta/classes/cve-check.bbclass
>>   create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch
>>   create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
>>

-- 
Mariano Lopez