Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Kevin Weng]

[-- Attachment #1: Type: text/plain, Size: 356 bytes --]

Hi Pierre,

I found that the hash function is causing collisions in the generated database such that some CVEs are being overwritten because of the UNIQUE constraint on the HASH column. For example, CVE-2018-1000873 has the same hash of 623198722 as CVE-2018-18338. This results in one of the two CVEs not appearing in the database.

--
Kevin Weng


[-- Attachment #2: Type: text/html, Size: 2242 bytes --]

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Pierre Le Magourou]

Hi Kevin,

> I found that the hash function is causing collisions in the generated database such that some CVEs are being overwritten because of the UNIQUE constraint on the HASH column. For example, CVE-2018-1000873 has the same hash of 623198722 as CVE-2018-18338. This results in one of the two CVEs not appearing in the database.

This is problematic. I kept using djb2 hash function, because it was
the one used in the previous cve-check-tool and it was fast. But it
might not be the right hash function to use. Do you have a better hash
function in mind ?
I can also drop hash function, remove everything from the database and
recreate all entries at each update but it will increase database
update time.

I don't have the same hash as you for CVE-2018-1000873 and
CVE-2018-18338, do you use my latest patches from master ? I did
several changes recently.

Pierre Le Magourou

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Pierre Le Magourou]

Hi,

> It looks like CVE_CHECK_DB_DIR has no default value which resulted in:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/988/steps/7/logs/step1b
>
> We only started seeing that error after your later patch to add back
> the do_fetch task. build-appliance is trying to collect up all the
> sources it may need.
>

I see the problem, it happens when cve-update-db do_fetch task is executed and
cve-check class is not inherited. I sent a v2 patch that sets default value to
CVE_CHECK_DB_DIR.


Pierre

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Richard Purdie]

On Wed, 2019-06-19 at 15:59 +0200, Pierre Le Magourou wrote:
> From: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
> 
> cve-check-tool-native do_populate_cve_db task was using deprecated
> NVD
> xml data feeds, cve-update-db uses NVD json data feeds.
> 
> Sqlite database schema was updated to take into account CVSSv3 CVE
> scores and operator in affected product versions.
> A new META table was added to store the last modification date of the
> NVD json data feeds.
> 
> Signed-off-by: Pierre Le Magourou <
> pierre.lemagourou@softbankrobotics.com>
> ---
>  meta/recipes-core/meta/cve-update-db.bb | 121
> ++++++++++++++++++++++++++++++++
>  1 file changed, 121 insertions(+)
>  create mode 100644 meta/recipes-core/meta/cve-update-db.bb
> 
> diff --git a/meta/recipes-core/meta/cve-update-db.bb b/meta/recipes-
> core/meta/cve-update-db.bb
> new file mode 100644
> index 0000000000..522fd23807
> --- /dev/null
> +++ b/meta/recipes-core/meta/cve-update-db.bb
> @@ -0,0 +1,121 @@
> +SUMMARY = "Updates the NVD CVE database"
> +LICENSE = "MIT"
> +
> +INHIBIT_DEFAULT_DEPS = "1"
> +PACKAGES = ""
> +
> +inherit nopackages
> +
> +deltask do_fetch
> +deltask do_unpack
> +deltask do_patch
> +deltask do_configure
> +deltask do_compile
> +deltask do_install
> +deltask do_populate_sysroot
> +
> +python do_populate_cve_db() {
> +    """
> +    Update NVD database with json data feed
> +    """
> +
> +    import sqlite3, urllib3, shutil, gzip, re
> +    from datetime import date
> +
> +    BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"
> +    YEAR_START = 2002
> +    JSON_TMPFILE = d.getVar("CVE_CHECK_DB_DIR") + '/nvd.json.gz'

It looks like CVE_CHECK_DB_DIR has no default value which resulted in:

https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/988/steps/7/logs/step1b

We only started seeing that error after your later patch to add back
the do_fetch task. build-appliance is trying to collect up all the
sources it may need.

Cheers,

Richard

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Burton, Ross]

On Tue, 25 Jun 2019 at 09:49, Pierre Le Magourou <lemagoup@gmail.com> wrote:
> > Also, the CVE db is updated using this custom task without link to
> > do_fetch, which means a fetchall task would not update the database for
> > off line NO_NETWORK builds.
> >
> > Could the task be added as dependency to do_fetch() or are there some other
> > side effects?
> >
>
> Yes I can do that, I don't think this will cause side effects.

Oh, you'll also want to set the proxies appropriately.

Ross

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Pierre Le Magourou]

Hi,

> Also, the CVE db is updated using this custom task without link to
> do_fetch, which means a fetchall task would not update the database for
> off line NO_NETWORK builds.
>
> Could the task be added as dependency to do_fetch() or are there some other
> side effects?
>

Yes I can do that, I don't think this will cause side effects.

Pierre

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Burton, Ross]

Thanks! :)

Ross

On Mon, 24 Jun 2019 at 09:33, Pierre Le Magourou <lemagoup@gmail.com> wrote:
>
> Hi,
>
> > > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment
> > > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe
> > > documentation could be updated too, e.g.
> > > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages
> >
> > Somehow I didn't notice it, thanks.
> >
> > Pierre, can you rewrite this to use standard library instead of urllib3?
>
> Yes of course,
> I only need urrlib to fetch the NVD json feeds, so I don't need
> urllib3, I can use standard library.
>
> Pierre

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Pierre Le Magourou]

Hi,

> > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment
> > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe
> > documentation could be updated too, e.g.
> > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages
>
> Somehow I didn't notice it, thanks.
>
> Pierre, can you rewrite this to use standard library instead of urllib3?

Yes of course,
I only need urrlib to fetch the NVD json feeds, so I don't need
urllib3, I can use standard library.

Pierre

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Mikko.Rapeli]

On Fri, Jun 21, 2019 at 01:29:18PM +0100, Burton, Ross wrote:
> On Fri, 21 Jun 2019 at 12:11, <Mikko.Rapeli@bmw.de> wrote:
> > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment
> > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe
> > documentation could be updated too, e.g.
> > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages
> 
> Somehow I didn't notice it, thanks.
> 
> Pierre, can you rewrite this to use standard library instead of urllib3?

Also, the CVE db is updated using this custom task without link to
do_fetch, which means a fetchall task would not update the database for
off line NO_NETWORK builds.

Could the task be added as dependency to do_fetch() or are there some other
side effects?

-Mikko

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Burton, Ross]

On Fri, 21 Jun 2019 at 12:11, <Mikko.Rapeli@bmw.de> wrote:
> This adds python3 urllib3 (python3-urllib3 in Debian) to build environment
> dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe
> documentation could be updated too, e.g.
> https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages

Somehow I didn't notice it, thanks.

Pierre, can you rewrite this to use standard library instead of urllib3?

Ross

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Mikko.Rapeli]

On Fri, Jun 21, 2019 at 02:03:36PM +0200, Alexander Kanavin wrote:
> On Fri, 21 Jun 2019 at 13:48, <Mikko.Rapeli@bmw.de> wrote:
>
> >
> > Hmm, possibly? I cherry-picked the patches to sumo and saw this missing
> > dependency in my container.
> >
> > Did poky master switch from using host python to native after sumo?
> >
> 
> poky uses host python for some things and native python for other things.
> Generally where 'core python' is enough then we use the host python, but
> when additional libraries are needed, then native python plus those
> libraries is a better choice, as this doesn't require adding to the
> required host packages list, and allows better control over how they're
> built.

Currently do_populate_cve_db is executed using host python, and
the DEPENDS field is empty, and INHIBIT_DEFAULT_DEPS = "1". python-urllib3
is also not available in poky, but only in meta-openembedded.

I'm fine with additional python3-urllib3 dependency to bitbake build host
tools when running CVE check builds. Just wanted to document it.

-Mikko

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

On Fri, 21 Jun 2019 at 13:48, <Mikko.Rapeli@bmw.de> wrote:

>
> Hmm, possibly? I cherry-picked the patches to sumo and saw this missing
> dependency in my container.
>
> Did poky master switch from using host python to native after sumo?
>

poky uses host python for some things and native python for other things.
Generally where 'core python' is enough then we use the host python, but
when additional libraries are needed, then native python plus those
libraries is a better choice, as this doesn't require adding to the
required host packages list, and allows better control over how they're
built.

Alex

[-- Attachment #2: Type: text/html, Size: 948 bytes --]

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Mikko.Rapeli]

On Fri, Jun 21, 2019 at 01:42:11PM +0200, Alexander Kanavin wrote:
> On Fri, 21 Jun 2019 at 13:11, <Mikko.Rapeli@bmw.de> wrote:
> 
> > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment
> > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe
> > documentation could be updated too, e.g.
> >
> > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages
> >
> > On my Debian 9 build container python3-urllib3 wasn't installed by default
> > or via other dependencies.
> >
> 
> Should this rather be provided via python3-native?

Hmm, possibly? I cherry-picked the patches to sumo and saw this missing
dependency in my container.

Did poky master switch from using host python to native after sumo?

-Mikko

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

On Fri, 21 Jun 2019 at 13:11, <Mikko.Rapeli@bmw.de> wrote:

> This adds python3 urllib3 (python3-urllib3 in Debian) to build environment
> dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe
> documentation could be updated too, e.g.
>
> https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages
>
> On my Debian 9 build container python3-urllib3 wasn't installed by default
> or via other dependencies.
>

Should this rather be provided via python3-native?

Alex

[-- Attachment #2: Type: text/html, Size: 998 bytes --]

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Mikko.Rapeli]

Hi,

This adds python3 urllib3 (python3-urllib3 in Debian) to build environment
dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe
documentation could be updated too, e.g.
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages

On my Debian 9 build container python3-urllib3 wasn't installed by default
or via other dependencies.

Thanks for these features!

-Mikko

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Pierre Le Magourou]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

> Not sure which of the changes is responsible, but this is new:
> WARNING: flex-native-2.6.0-r0 do_cve_check: Found unpatched CVE
> (CVE-2015-1773)
>
> https://nvd.nist.gov/vuln/detail/CVE-2015-1773
>
> Note that the flex tool is completely unrelated to Apache Flex.
>
>
I see, the 4/4 patch is responsible for that (Consider CVE that affects
versions with less than operator). It takes into account the comparison
operator in the json NVD file (new 'version_affected' field that was not in
the XML data feed). So this CVE matches because 2.6.0 <= 4.14.0. But it
should not match because it concerns another product (flex_project/flex vs
Apache/flex).

There is indeed a problem I didn't manage. The CVE_PRODUCT variable we use
in cve-check only takes the product name (here 'flex') into account, we
should also consider the vendor name (here 'flex_project').

Without this patch (4/4), the behaviour should be the same as before.

Pierre

[-- Attachment #2: Type: text/html, Size: 1418 bytes --]

Re: [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Adrian Bunk]

Not sure which of the changes is responsible, but this is new:
WARNING: flex-native-2.6.0-r0 do_cve_check: Found unpatched CVE (CVE-2015-1773)

https://nvd.nist.gov/vuln/detail/CVE-2015-1773

Note that the flex tool is completely unrelated to Apache Flex.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

[PATCH 1/4] cve-update-db: New recipe to update CVE database

[Pierre Le Magourou]

From: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>

cve-check-tool-native do_populate_cve_db task was using deprecated NVD
xml data feeds, cve-update-db uses NVD json data feeds.

Sqlite database schema was updated to take into account CVSSv3 CVE
scores and operator in affected product versions.
A new META table was added to store the last modification date of the
NVD json data feeds.

Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
---
 meta/recipes-core/meta/cve-update-db.bb | 121 ++++++++++++++++++++++++++++++++
 1 file changed, 121 insertions(+)
 create mode 100644 meta/recipes-core/meta/cve-update-db.bb

diff --git a/meta/recipes-core/meta/cve-update-db.bb b/meta/recipes-core/meta/cve-update-db.bb
new file mode 100644
index 0000000000..522fd23807
--- /dev/null
+++ b/meta/recipes-core/meta/cve-update-db.bb
@@ -0,0 +1,121 @@
+SUMMARY = "Updates the NVD CVE database"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+PACKAGES = ""
+
+inherit nopackages
+
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+deltask do_populate_sysroot
+
+python do_populate_cve_db() {
+    """
+    Update NVD database with json data feed
+    """
+
+    import sqlite3, urllib3, shutil, gzip, re
+    from datetime import date
+
+    BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"
+    YEAR_START = 2002
+    JSON_TMPFILE = d.getVar("CVE_CHECK_DB_DIR") + '/nvd.json.gz'
+
+    # Connect to database
+    db_file = d.getVar("CVE_CHECK_DB_FILE")
+    conn = sqlite3.connect(db_file)
+    c = conn.cursor()
+
+    initialize_db(c)
+
+    http = urllib3.PoolManager()
+
+    for year in range(YEAR_START, date.today().year + 1):
+        year_url = BASE_URL + str(year)
+        meta_url = year_url + ".meta"
+        json_url = year_url + ".json.gz"
+
+        # Retrieve meta last modified date
+        with http.request('GET', meta_url, preload_content=False) as r:
+            date_line = str(r.data.splitlines()[0])
+            last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1)
+
+        # Compare with current db last modified date
+        c.execute("select DATE from META where YEAR = '%d'" % year)
+        meta = c.fetchone()
+        if not meta or meta[0] != last_modified:
+            # Update db with current year json file
+            with http.request('GET', json_url, preload_content=False) as r, open(JSON_TMPFILE, 'wb') as tmpfile:
+                shutil.copyfileobj(r, tmpfile)
+            with gzip.open(JSON_TMPFILE, 'rt') as jsonfile:
+                update_db(c, jsonfile)
+            c.execute("insert or replace into META values (?, ?)",
+                    [year, last_modified])
+
+    conn.commit()
+    conn.close()
+
+    with open(d.getVar("CVE_CHECK_TMP_FILE"), 'a'):
+        os.utime(d.getVar("CVE_CHECK_TMP_FILE"), None)
+}
+
+# DJB2 hash algorithm
+def hash_djb2(s):
+    hash = 5381
+    for x in s:
+        hash = (( hash << 5) + hash) + ord(x)
+
+    return hash & 0xFFFFFFFF
+
+def initialize_db(c):
+    c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)")
+    c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \
+        SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)")
+    c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (HASH INTEGER UNIQUE, ID TEXT, \
+        VENDOR TEXT, PRODUCT TEXT, VERSION TEXT, OPERATOR TEXT)")
+    c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_IDX ON PRODUCTS \
+        (PRODUCT, VERSION)")
+
+def update_db(c, json_filename):
+    import json
+    root = json.load(json_filename)
+
+    for elt in root['CVE_Items']:
+        if not elt['impact']:
+            continue
+
+        cveId = elt['cve']['CVE_data_meta']['ID']
+        cveDesc = elt['cve']['description']['description_data'][0]['value']
+        date = elt['lastModifiedDate']
+        accessVector = elt['impact']['baseMetricV2']['cvssV2']['accessVector']
+        cvssv2 = elt['impact']['baseMetricV2']['cvssV2']['baseScore']
+
+        try:
+            cvssv3 = elt['impact']['baseMetricV3']['cvssV3']['baseScore']
+        except:
+            cvssv3 = 0.0
+
+        c.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)",
+                [cveId, cveDesc, cvssv2, cvssv3, date, accessVector])
+
+        for vendor in elt['cve']['affects']['vendor']['vendor_data']:
+            for product in vendor['product']['product_data']:
+                for version in product['version']['version_data']:
+                    product_str = cveId+vendor['vendor_name']+product['product_name']+version['version_value']
+                    hashstr = hash_djb2(product_str)
+                    c.execute("insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?)",
+                            [ hashstr, cveId, vendor['vendor_name'],
+                                product['product_name'], version['version_value'],
+                                version['version_affected']])
+
+
+
+addtask do_populate_cve_db before do_cve_check
+do_populate_cve_db[nostamp] = "1"
+
+EXCLUDE_FROM_WORLD = "1"
-- 
2.11.0