[poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

[poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

[Akash Hadke]

Add new method get_ignored_cves in cve_check.py
to get ignored CVEs from recipe by excluding distro-wide
ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc

While calling this method use below code to get argument values
paths = d.getVar('PATH').split(':')
cves = d.getVar('CVE_CHECK_IGNORE').split()

Signed-off-by: Akash Hadke <akash.hadke@kpit.com>
Signed-off-by: Akash Hadke <hadkeakash4@gmail.com>
---
 meta/lib/oe/cve_check.py | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/meta/lib/oe/cve_check.py b/meta/lib/oe/cve_check.py
index dc7d2e2826..d96d47b737 100644
--- a/meta/lib/oe/cve_check.py
+++ b/meta/lib/oe/cve_check.py
@@ -163,3 +163,41 @@ def cve_check_merge_jsons(output, data):
             return
 
     output["package"].append(data["package"][0])
+
+def get_ignored_cves(paths, cves):
+    """
+    Get ignored CVEs from the recipe and exlude the CVEs from
+    meta/conf/distro/include/cve-extra-exclusions.inc
+
+    While calling this method use below code to get argument values
+    paths = d.getVar('PATH').split(':')
+    cves = d.getVar('CVE_CHECK_IGNORE').split()
+    """
+    import os
+
+    cve_extra_exclusion_inc_file = "../meta/conf/distro/include/cve-extra-exclusions.inc"
+    for path in paths:
+        check_for_correct_file_path = os.path.join(path, cve_extra_exclusion_inc_file)
+        if os.path.isfile(check_for_correct_file_path):
+            inc_file = check_for_correct_file_path
+        else:
+            continue
+
+    cve_check_ignored = set()
+    ignored_cves = set()
+    with open(inc_file) as f:
+        lines = f.readlines()
+        for line in lines:
+            if line.strip():
+                # Ignore the comments from cve-extra-exclusions.inc
+                if not re.search("^#", line):
+                    cve_match = re.findall(r'CVE\-\d{4}\-\d+', line)
+                    for cve in cve_match:
+                        ignored_cves.add(cve)
+
+    for cve in cves:
+        if cve not in ignored_cves:
+            cve_check_ignored.add(cve)
+    ignored_cves_from_recipe = " ".join(cve_check_ignored)
+
+    return ignored_cves_from_recipe
-- 
2.17.1

[poky][master][PATCH 2/3] cve-export.bbclass: Add a new class to get patched and ignored CVEs from the build

[Akash Hadke]

This class executes an anonymous function which sets the
below variables
  CVE_IGNORED = CVEs that are ignored in recipes
  CVE_PATCHED = CVEs that are fixed by applying patches

It does not consider CVEs that are ignored in
poky/meta/conf/distro/include/cve-extra-exclusions.inc
and only provide CVEs that are ignored in the recipe.

Default values are set for CVE_PRODUCT and CVE_VERSION
to BPN and PV respectively.

Considered setting these values so that anyone can get
below information about the CVE from the build.
  CVE_PRODUCT
  CVE_VERSION
  CVE_IGNORED
  CVE_PATCHED

Signed-off-by: Akash Hadke <akash.hadke@kpit.com>
Signed-off-by: Akash Hadke <hadkeakash4@gmail.com>
---
 meta/classes/cve-export.bbclass | 37 +++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
 create mode 100644 meta/classes/cve-export.bbclass

diff --git a/meta/classes/cve-export.bbclass b/meta/classes/cve-export.bbclass
new file mode 100644
index 0000000000..5ed5760970
--- /dev/null
+++ b/meta/classes/cve-export.bbclass
@@ -0,0 +1,37 @@
+# This class is used to get patched and ignored CVEs from the build
+#
+# To use this class inherit it in the local.conf file.
+#
+# It executes an anonymous function which sets below variables
+#
+# CVE_IGNORED = CVEs those are ignored in recipes
+# CVE_PATCHED = CVEs those are fixed by applying patches
+#
+# It does not consider all the CVEs that are ignored in
+# poky/meta/conf/distro/include/cve-extra-exclusions.inc
+# and only provide CVEs that are ignored in the recipe.
+#
+# The product name sets default to BPN and version sets default to
+# PV but it can be overriden per recipe, to get the value of
+# product and version use d.getVar()
+
+CVE_PRODUCT ??= "${BPN}"
+CVE_VERSION ??= "${PV}"
+CVE_CHECK_IGNORE ?= ""
+
+python __anonymous () {
+    import re
+    from oe.cve_check import get_patched_cves
+    from oe.cve_check import get_ignored_cves
+
+    # Check if cve-extra-exclusions.inc file is included or not
+    if re.search('cve-extra-exclusions.inc', d.getVar('BBINCLUDED')):
+        paths = d.getVar('PATH').split(':')
+        cves = d.getVar('CVE_CHECK_IGNORE').split()
+        ignored_cves = get_ignored_cves(paths, cves)
+    else:
+        ignored_cves = " ".join(d.getVar('CVE_CHECK_IGNORE'))
+
+    d.setVar('CVE_IGNORED', ignored_cves)
+    d.setVar('CVE_PATCHED', " ".join(get_patched_cves(d)))
+}
-- 
2.17.1

[poky][master][PATCH 3/3] cve_export.py: Add new selftest for cve-export.bbclass

[Akash Hadke]

This test compares values of variables CVE_CHECK_IGNORE with
CVE_IGNORED that is exported from cve-export.bbclass

Signed-off-by: Akash Hadke <akash.hadke@kpit.com>
Signed-off-by: Akash Hadke <hadkeakash4@gmail.com>
---
 meta/lib/oeqa/selftest/cases/cve_export.py | 24 ++++++++++++++++++++++
 1 file changed, 24 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/cve_export.py

diff --git a/meta/lib/oeqa/selftest/cases/cve_export.py b/meta/lib/oeqa/selftest/cases/cve_export.py
new file mode 100644
index 0000000000..7c7fd07957
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/cve_export.py
@@ -0,0 +1,24 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_var
+from oe.cve_check import get_ignored_cves
+
+class CveExportTest (OESelftestTestCase):
+
+    def test_ignored_cves(self):
+        """
+        This test compares the values of variables CVE_CHECK_IGNORE with
+        CVE_IGNORED that is exported from cve-export.bbclass
+        """
+        target = 'unzip'
+        feature = 'INHERIT += "cve-export"\n'
+        feature += 'require conf/distro/include/cve-extra-exclusions.inc'
+        self.write_config(feature)
+
+        res = bitbake("%s -c fetch" % target, ignore_status=True)
+        self.assertEqual(res.status, 0, "\nCouldn't build.\nbitbake output %s" % res.output)
+
+        paths = get_bb_var('PATH', target).split(':')
+        cves = get_bb_var('CVE_CHECK_IGNORE', target).split()
+        final_cve = get_ignored_cves(paths, cves)
+        ignore_cves_from_exported_variable = get_bb_var('CVE_IGNORED', target)
+        self.assertEqual(final_cve, ignore_cves_from_exported_variable.strip(), "Ignored CVEs are not matching")
-- 
2.17.1

Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

[Marta Rybczynska]

On Wed, May 11, 2022 at 4:37 PM akash hadke via lists.openembedded.org
<akash.hadke=kpit.com@lists.openembedded.org> wrote:
>
> Add new method get_ignored_cves in cve_check.py
> to get ignored CVEs from recipe by excluding distro-wide
> ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc
>
> While calling this method use below code to get argument values
> paths = d.getVar('PATH').split(':')
> cves = d.getVar('CVE_CHECK_IGNORE').split()
>

Hello Akash,
While looking into this patch set I'm wondering what is your use case.
It seems to be to get a list
of ignored and patched CVEs. This is already available from the
cve-check output or from the create-spdx
output after some parsing. With the new JSON format for cve-check it
becomes very easy. If you could
elaborate more on the way you plan to use this data, I'm pretty sure
we can come with a simple
post-processing script to do the same.

BTW Why do assume people always include
meta/conf/distro/include/cve-extra-exclusions.inc ?
We don't do that at Oniro and we use our own judgement on outstanding CVEs.

Regards,
Marta

Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

[Akash Hadke]

[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]

Hello Marta,

Actually, I wanted to add the ignored and patched CVEs in buildhistory and for that purpose, I am exporting variables CVE_IGNORED and CVE_PATCHED with those values. I don't want to use cve-check.bbclass as it checks for the CVEs from the NVD database, and I only want to get ignored and patched CVEs from the recipe.

Regarding meta/conf/distro/include/cve-extra-exclusions.inc if any project includes it then CVEs that are ignored in cve-extra-exclusions.inc will get shown for each recipe in the CVE_CHECK_IGNORED list even though the CVEs are not related to that component recipe. Hence, I have did the changes to exclude CVEs from cve-extra-exclusions.inc ​

Best Regards,
Akash
________________________________
From: Marta Rybczynska <rybczynska@gmail.com>
Sent: 17 May 2022 14:42
To: Akash Hadke <Akash.Hadke@kpit.com>
Cc: OE-core <openembedded-core@lists.openembedded.org>; Ranjitsinh Rathod <Ranjitsinh.Rathod@kpit.com>; Akash Hadke <hadkeakash4@gmail.com>
Subject: Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

Caution: This email originated from outside of the KPIT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On Wed, May 11, 2022 at 4:37 PM akash hadke via lists.openembedded.org
<akash.hadke=kpit.com@lists.openembedded.org> wrote:
>
> Add new method get_ignored_cves in cve_check.py
> to get ignored CVEs from recipe by excluding distro-wide
> ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc
>
> While calling this method use below code to get argument values
> paths = d.getVar('PATH').split(':')
> cves = d.getVar('CVE_CHECK_IGNORE').split()
>

Hello Akash,
While looking into this patch set I'm wondering what is your use case.
It seems to be to get a list
of ignored and patched CVEs. This is already available from the
cve-check output or from the create-spdx
output after some parsing. With the new JSON format for cve-check it
becomes very easy. If you could
elaborate more on the way you plan to use this data, I'm pretty sure
we can come with a simple
post-processing script to do the same.

BTW Why do assume people always include
meta/conf/distro/include/cve-extra-exclusions.inc ?
We don't do that at Oniro and we use our own judgement on outstanding CVEs.

Regards,
Marta

[-- Attachment #2: Type: text/html, Size: 5309 bytes --]

Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

[Marta Rybczynska]

On Tue, May 17, 2022 at 1:42 PM Akash Hadke <Akash.Hadke@kpit.com> wrote:
>
> Hello Marta,
>
> Actually, I wanted to add the ignored and patched CVEs in buildhistory and for that purpose, I am exporting variables CVE_IGNORED and CVE_PATCHED with those values. I don't want to use cve-check.bbclass as it checks for the CVEs from the NVD database, and I only want to get ignored and patched CVEs from the recipe.

Hello again Akash,
What you'd like to do is to see the difference in ignored and patched
CVEs in buildhistory? Do I get it right?

>
> Regarding meta/conf/distro/include/cve-extra-exclusions.inc if any project includes it then CVEs that are ignored in cve-extra-exclusions.inc will get shown for each recipe in the CVE_CHECK_IGNORED list even though the CVEs are not related to that component recipe. Hence, I have did the changes to exclude CVEs from cve-extra-exclusions.inc

I think I understand the idea. The point I'm making is that if someone
does not include the cve-extra-exclusions.inc in their distro, the
code will still use it and filter out CVEs they still see when doing
cve-check.

Kind regards,
Marta

>
> Best Regards,
> Akash
> ________________________________
> From: Marta Rybczynska <rybczynska@gmail.com>
> Sent: 17 May 2022 14:42
> To: Akash Hadke <Akash.Hadke@kpit.com>
> Cc: OE-core <openembedded-core@lists.openembedded.org>; Ranjitsinh Rathod <Ranjitsinh.Rathod@kpit.com>; Akash Hadke <hadkeakash4@gmail.com>
> Subject: Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves
>
> Caution: This email originated from outside of the KPIT. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> On Wed, May 11, 2022 at 4:37 PM akash hadke via lists.openembedded.org
> <akash.hadke=kpit.com@lists.openembedded.org> wrote:
> >
> > Add new method get_ignored_cves in cve_check.py
> > to get ignored CVEs from recipe by excluding distro-wide
> > ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc
> >
> > While calling this method use below code to get argument values
> > paths = d.getVar('PATH').split(':')
> > cves = d.getVar('CVE_CHECK_IGNORE').split()
> >
>
> Hello Akash,
> While looking into this patch set I'm wondering what is your use case.
> It seems to be to get a list
> of ignored and patched CVEs. This is already available from the
> cve-check output or from the create-spdx
> output after some parsing. With the new JSON format for cve-check it
> becomes very easy. If you could
> elaborate more on the way you plan to use this data, I'm pretty sure
> we can come with a simple
> post-processing script to do the same.
>
> BTW Why do assume people always include
> meta/conf/distro/include/cve-extra-exclusions.inc ?
> We don't do that at Oniro and we use our own judgement on outstanding CVEs.
>
> Regards,
> Marta

Re: [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

[akash hadke]

[-- Attachment #1: Type: text/plain, Size: 261 bytes --]

Hi Marta,

If you see the code from the succeeding patch to this https://lists.openembedded.org/g/openembedded-core/message/165502 here I have checked if cve-extra-exclusions.inc is included or not. If it is not included then the code will not get executed.

[-- Attachment #2: Type: text/html, Size: 439 bytes --]

Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

[richard.purdie]

On Tue, 2022-05-17 at 11:42 +0000, akash hadke via lists.openembedded.org wrote:
> Actually, I wanted to add the ignored and patched CVEs in
> buildhistory and for that purpose, I am exporting variables
> CVE_IGNORED and CVE_PATCHED with those values. I don't want to use
> cve-check.bbclass as it checks for the CVEs from the NVD database,
> and I only want to get ignored and patched CVEs from the recipe.

I'd really prefer to have one cve handling class where we can configure
it to get the data different people need rather than multiple
difference cve classes which are going to confuse people. Could we have
a way to disable NVD data from the cve-check class?

Cheers,

Richard

Re: [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

[akash hadke]

[-- Attachment #1: Type: text/plain, Size: 291 bytes --]

Hi Richard,

I tried modifying the cve-check.bbclass but did not able to get the solution for disabling the NVD data, because when we inherit cve-check it executes the cve_check task that checks CVEs from NVD DB. So I am also not sure how to disable it hence I used the separate bbclass.

[-- Attachment #2: Type: text/html, Size: 299 bytes --]

Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

[richard.purdie]

On Wed, 2022-05-18 at 02:46 -0700, akash hadke via
lists.openembedded.org wrote:
> Hi Richard,
> 
> I tried modifying the cve-check.bbclass but did not able to get the
> solution for disabling the NVD data, because when we inherit cve-
> check it executes the cve_check task that checks CVEs from NVD DB. So
> I am also not sure how to disable it hence I used the separate
> bbclass.

I guess the task dependencies are a bit of an issue there. There are
probably ways to make those configurable, it would just take a little
work.

I really do want to encourage us to work together on common cve tooling
rather than having several partial implementations so I can't take this
patch series.

Cheers,

Richard

Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

[Marta Rybczynska]

On Wed, May 18, 2022 at 12:33 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2022-05-18 at 02:46 -0700, akash hadke via
> lists.openembedded.org wrote:
> > Hi Richard,
> >
> > I tried modifying the cve-check.bbclass but did not able to get the
> > solution for disabling the NVD data, because when we inherit cve-
> > check it executes the cve_check task that checks CVEs from NVD DB. So
> > I am also not sure how to disable it hence I used the separate
> > bbclass.
>
> I guess the task dependencies are a bit of an issue there. There are
> probably ways to make those configurable, it would just take a little
> work.
>
> I really do want to encourage us to work together on common cve tooling
> rather than having several partial implementations so I can't take this
> patch series.
>

I plan to cut those dependencies and make different stages optional.
In my case the first use
is to be able to run multiple cve-checks with the same database
(guaranteed without updates)
or run cve-check with some given known database (for testing).
We can add an option to make the actual check with the database optional.

Exporting the data to the buildhistory looks like a useful thing to do, too.

I can draft the split after YPS.

Kind regards,
Marta