[Buildroot] [PATCH 1/3] support/scripts/pkg-stats: fix flake8 E741 ambiguous variable name

[Buildroot] [PATCH 1/3] support/scripts/pkg-stats: fix flake8 E741 ambiguous variable name

[Matthew Weber]

Signed-off-by: Matthew Weber <matthew.weber@collins.com>
---
 support/scripts/pkg-stats | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index c7e30dfd2b..0cd3674c52 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -375,9 +375,9 @@ def package_init_make_info():
     variable_list = [x[5:] for x in variable_list if x.startswith("HOST_")] + \
                     [x for x in variable_list if not x.startswith("HOST_")]
 
-    for l in variable_list:
+    for item in variable_list:
         # Get variable name and value
-        pkgvar, value = l.split("=")
+        pkgvar, value = item.split("=")
 
         # Strip the suffix according to the variable
         if pkgvar.endswith("_LICENSE"):
-- 
2.17.1

[Buildroot] [PATCH 2/3] support/scripts/pkg-stats: add is_actual_package() and rework has_valid_infra()

[Matthew Weber]

has_valid_infra() is incorrectly named; it probably should be named
is_actual_package(), and has_valid_infra() would be changed to
actually represent having an actual infra.

This resolves packages reporting as having no valid package infra and
cleans up reporting cases of CPE and CVEs where there isn't a valid version
or package definition outside Buildroot

Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Matthew Weber <matthew.weber@collins.com>
---
Yann, hopefully I got this right :-)
---
 support/scripts/pkg-stats | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 0cd3674c52..ca55a301de 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -131,7 +131,15 @@ class Package:
 
     @property
     def has_valid_infra(self):
+        if self.infras is None:
+            return False
+        return len(self.infras) > 0
+
+    @property
+    def is_actual_package(self):
         try:
+            if not self.has_valid_infra:
+                return False
             if self.infras[0][1] == 'virtual':
                 return False
         except IndexError:
@@ -159,7 +167,7 @@ class Package:
         """
         Fills in the .status['license'] and .status['license-files'] fields
         """
-        if not self.has_valid_infra:
+        if not self.is_actual_package:
             self.status['license'] = ("na", "no valid package infra")
             self.status['license-files'] = ("na", "no valid package infra")
             return
@@ -177,7 +185,7 @@ class Package:
         """
         Fills in the .status['hash'] field
         """
-        if not self.has_valid_infra:
+        if not self.is_actual_package:
             self.status['hash'] = ("na", "no valid package infra")
             self.status['hash-license'] = ("na", "no valid package infra")
             return
@@ -192,7 +200,7 @@ class Package:
         """
         Fills in the .patch_count, .patch_files and .status['patches'] fields
         """
-        if not self.has_valid_infra:
+        if not self.is_actual_package:
             self.status['patches'] = ("na", "no valid package infra")
             return
 
@@ -220,7 +228,7 @@ class Package:
         Fills in the .cpeid field
         """
         var = self.pkgvar()
-        if not self.has_valid_infra:
+        if not self.is_actual_package:
             self.status['cpe'] = ("na", "no valid package infra")
             return
 
@@ -551,13 +559,13 @@ async def check_package_latest_version(packages):
       package, as known by release-monitoring.org
     """
 
-    for pkg in [p for p in packages if not p.has_valid_infra]:
+    for pkg in [p for p in packages if not p.is_actual_package]:
         pkg.status['version'] = ("na", "no valid package infra")
 
     tasks = []
     connector = aiohttp.TCPConnector(limit_per_host=5)
     async with aiohttp.ClientSession(connector=connector, trust_env=True) as sess:
-        packages = [p for p in packages if p.has_valid_infra]
+        packages = [p for p in packages if p.is_actual_package]
         for pkg in packages:
             tasks.append(asyncio.ensure_future(check_package_latest_version_get(sess, pkg, len(packages))))
         await asyncio.wait(tasks)
@@ -578,7 +586,7 @@ def check_package_cves(nvd_path, packages):
 
     cpe_product_pkgs = defaultdict(list)
     for pkg in packages:
-        if not pkg.has_valid_infra:
+        if not pkg.is_actual_package:
             pkg.status['cve'] = ("na", "no valid package infra")
             continue
         if not pkg.current_version:
-- 
2.17.1

[Buildroot] [PATCH 3/3] support/scripts/pkg-stats: clarify when a CVE/CPE should report as N/A

[Matthew Weber]

 - If a package doesn't have any versioning, ignore and state that
 - If a package is virtual, CVE=ignore and CPE state virtual
 - For any of these NA cases, don't provide search link

Signed-off-by: Matthew Weber <matthew.weber@collins.com>
---
 support/scripts/pkg-stats | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index ca55a301de..3aaf1169cb 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -229,7 +229,10 @@ class Package:
         """
         var = self.pkgvar()
         if not self.is_actual_package:
-            self.status['cpe'] = ("na", "no valid package infra")
+            self.status['cpe'] = ("na", "N/A - virtual pkg")
+            return
+        if not self.current_version:
+            self.status['cpe'] = ("na", "no version information available")
             return
 
         if var in self.all_cpeids:
@@ -587,7 +590,7 @@ def check_package_cves(nvd_path, packages):
     cpe_product_pkgs = defaultdict(list)
     for pkg in packages:
         if not pkg.is_actual_package:
-            pkg.status['cve'] = ("na", "no valid package infra")
+            pkg.status['cve'] = ("na", "N/A")
             continue
         if not pkg.current_version:
             pkg.status['cve'] = ("na", "no version information available")
@@ -942,12 +945,15 @@ def dump_html_pkg(f, pkg):
     if pkg.cpeid:
         f.write("  <code>%s</code>\n" % pkg.cpeid)
     if not pkg.is_status_ok("cpe"):
-        if pkg.cpeid:
-            f.write("  <br/>%s <a href=\"https://nvd.nist.gov/products/cpe/search/results?namingFormat=2.3&keyword=%s\">(Search)</a>\n" %  # noqa: E501
-                    (pkg.status['cpe'][1], ":".join(pkg.cpeid.split(":")[0:5])))
+        if pkg.is_actual_package and pkg.current_version:
+            if pkg.cpeid:
+                f.write("  <br/>%s <a href=\"https://nvd.nist.gov/products/cpe/search/results?namingFormat=2.3&keyword=%s\">(Search)</a>\n" %  # noqa: E501
+                        (pkg.status['cpe'][1], ":".join(pkg.cpeid.split(":")[0:5])))
+            else:
+                f.write("  %s <a href=\"https://nvd.nist.gov/products/cpe/search/results?namingFormat=2.3&keyword=%s\">(Search)</a>\n" %  # noqa: E501
+                        (pkg.status['cpe'][1], pkg.name))
         else:
-            f.write("  %s <a href=\"https://nvd.nist.gov/products/cpe/search/results?namingFormat=2.3&keyword=%s\">(Search)</a>\n" %
-                    (pkg.status['cpe'][1], pkg.name))
+            f.write("  %s\n" % pkg.status['cpe'][1])
 
     f.write("  </td>\n")
 
-- 
2.17.1

[Buildroot] [PATCH 3/3] support/scripts/pkg-stats: clarify when a CVE/CPE should report as N/A

[Yann E. MORIN]

Matthew, All,

On 2021-05-18 13:21 -0500, Matthew Weber via buildroot spake thusly:
>  - If a package doesn't have any versioning, ignore and state that
>  - If a package is virtual, CVE=ignore and CPE state virtual
>  - For any of these NA cases, don't provide search link
> 
> Signed-off-by: Matthew Weber <matthew.weber@collins.com>

Honestly, I get quickly lost in the coe of pkg-stats...

However, with this series applied, the cells for CVE and CPE for virtual
packages are not green, suggesting this is abnormal. However, this is
perfectly fine :the re are no CVE and no CPE for virtual packages, so
they should be green.

I managed to do that by adding the following on-top of this third patch:

    diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
    index 3aaf1169cb..d06778ab05 100755
    --- a/support/scripts/pkg-stats
    +++ b/support/scripts/pkg-stats
    @@ -910,6 +910,8 @@ def dump_html_pkg(f, pkg):
         td_class = ["centered"]
         if pkg.is_status_ok("cve"):
             td_class.append("cve-ok")
    +    elif pkg.is_status_na("cve") and not pkg.is_actual_package:
    +        td_class.append("cve-ok")
         elif pkg.is_status_error("cve"):
             td_class.append("cve-nok")
         else:
    @@ -937,6 +939,8 @@ def dump_html_pkg(f, pkg):
         td_class = ["left"]
         if pkg.is_status_ok("cpe"):
             td_class.append("cpe-ok")
    +    elif pkg.is_status_na("cpe") and not pkg.is_actual_package:
    +        td_class.append("cpe-ok")
         elif pkg.is_status_error("cpe"):
             td_class.append("cpe-nok")
         else:

But I am not usre this is the best solution... So, I've not applied
patches 2 and 3 in the series. Could you please respin with the above
(if it's OK for you, or with a better solution), please?

Additionally, as a further refinement, packages that have no version,
like urandom-scripts, makedevs, etc... are usually bundled with
Buildroot. Do you think for those we should:

 1. declare 'buildroot' to NVD, as the 'buildroot_project' vendor and
    the 'buildroot' product, as well as 'makedevs', 'urandom-scripts' et
    al. as products,

 2. add "MAKEDEVS_CPE_VENDOR = buildroot_project" to each of those
    bundled packages.

Thoughts?

Regards,
Yann E. MORIN.

> ---
>  support/scripts/pkg-stats | 20 +++++++++++++-------
>  1 file changed, 13 insertions(+), 7 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index ca55a301de..3aaf1169cb 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -229,7 +229,10 @@ class Package:
>          """
>          var = self.pkgvar()
>          if not self.is_actual_package:
> -            self.status['cpe'] = ("na", "no valid package infra")
> +            self.status['cpe'] = ("na", "N/A - virtual pkg")
> +            return
> +        if not self.current_version:
> +            self.status['cpe'] = ("na", "no version information available")
>              return
>  
>          if var in self.all_cpeids:
> @@ -587,7 +590,7 @@ def check_package_cves(nvd_path, packages):
>      cpe_product_pkgs = defaultdict(list)
>      for pkg in packages:
>          if not pkg.is_actual_package:
> -            pkg.status['cve'] = ("na", "no valid package infra")
> +            pkg.status['cve'] = ("na", "N/A")
>              continue
>          if not pkg.current_version:
>              pkg.status['cve'] = ("na", "no version information available")
> @@ -942,12 +945,15 @@ def dump_html_pkg(f, pkg):
>      if pkg.cpeid:
>          f.write("  <code>%s</code>\n" % pkg.cpeid)
>      if not pkg.is_status_ok("cpe"):
> -        if pkg.cpeid:
> -            f.write("  <br/>%s <a href=\"https://nvd.nist.gov/products/cpe/search/results?namingFormat=2.3&keyword=%s\">(Search)</a>\n" %  # noqa: E501
> -                    (pkg.status['cpe'][1], ":".join(pkg.cpeid.split(":")[0:5])))
> +        if pkg.is_actual_package and pkg.current_version:
> +            if pkg.cpeid:
> +                f.write("  <br/>%s <a href=\"https://nvd.nist.gov/products/cpe/search/results?namingFormat=2.3&keyword=%s\">(Search)</a>\n" %  # noqa: E501
> +                        (pkg.status['cpe'][1], ":".join(pkg.cpeid.split(":")[0:5])))
> +            else:
> +                f.write("  %s <a href=\"https://nvd.nist.gov/products/cpe/search/results?namingFormat=2.3&keyword=%s\">(Search)</a>\n" %  # noqa: E501
> +                        (pkg.status['cpe'][1], pkg.name))
>          else:
> -            f.write("  %s <a href=\"https://nvd.nist.gov/products/cpe/search/results?namingFormat=2.3&keyword=%s\">(Search)</a>\n" %
> -                    (pkg.status['cpe'][1], pkg.name))
> +            f.write("  %s\n" % pkg.status['cpe'][1])
>  
>      f.write("  </td>\n")
>  
> -- 
> 2.17.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [External] Re: [PATCH 3/3] support/scripts/pkg-stats: clarify when a CVE/CPE should report as N/A

[Weber, Matthew L Collins]

Yann,

> -----Original Message-----
> From: Yann E. MORIN <yann.morin.1998@free.fr>
> Sent: Tuesday, May 18, 2021 3:17 PM
> To: Weber, Matthew L Collins <Matthew.Weber@collins.com>
> Cc: buildroot at buildroot.org
> Subject: [External] Re: [Buildroot] [PATCH 3/3] support/scripts/pkg-stats:
> clarify when a CVE/CPE should report as N/A
> 
> Matthew, All,
> 
> On 2021-05-18 13:21 -0500, Matthew Weber via buildroot spake thusly:
> >  - If a package doesn't have any versioning, ignore and state that
> >  - If a package is virtual, CVE=ignore and CPE state virtual
> >  - For any of these NA cases, don't provide search link
> >
> > Signed-off-by: Matthew Weber <matthew.weber@collins.com>
> 
> Honestly, I get quickly lost in the coe of pkg-stats...
> 
> However, with this series applied, the cells for CVE and CPE for virtual
> packages are not green, suggesting this is abnormal. However, this is
> perfectly fine there are no CVE and no CPE for virtual packages, so they
> should be green.
> 
> I managed to do that by adding the following on-top of this third patch:
> 
>     diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
>     index 3aaf1169cb..d06778ab05 100755
>     --- a/support/scripts/pkg-stats
>     +++ b/support/scripts/pkg-stats
>     @@ -910,6 +910,8 @@ def dump_html_pkg(f, pkg):
>          td_class = ["centered"]
>          if pkg.is_status_ok("cve"):
>              td_class.append("cve-ok")
>     +    elif pkg.is_status_na("cve") and not pkg.is_actual_package:
>     +        td_class.append("cve-ok")
>          elif pkg.is_status_error("cve"):
>              td_class.append("cve-nok")
>          else:
>     @@ -937,6 +939,8 @@ def dump_html_pkg(f, pkg):
>          td_class = ["left"]
>          if pkg.is_status_ok("cpe"):
>              td_class.append("cpe-ok")
>     +    elif pkg.is_status_na("cpe") and not pkg.is_actual_package:
>     +        td_class.append("cpe-ok")
>          elif pkg.is_status_error("cpe"):
>              td_class.append("cpe-nok")
>          else:
> 
> But I am not usre this is the best solution... So, I've not applied patches 2 and
> 3 in the series. Could you please respin with the above (if it's OK for you, or
> with a better solution), please?

I'll take a look (I noticed this and didn't have a good feeling if they should be green).

> 
> Additionally, as a further refinement, packages that have no version, like
> urandom-scripts, makedevs, etc... are usually bundled with Buildroot. Do you
> think for those we should:
> 
>  1. declare 'buildroot' to NVD, as the 'buildroot_project' vendor and
>     the 'buildroot' product, as well as 'makedevs', 'urandom-scripts' et
>     al. as products,
> 
>  2. add "MAKEDEVS_CPE_VENDOR = buildroot_project" to each of those
>     bundled packages.
> 

I'd go with (2) where we use 'buildroot_project' as the vendor, and then each of those packages that have no version would use the Buildroot version and their name as the product.  I don't see any reason NIST won't support us adding those dictionary entries, and I know other distros do similar for things like ifupdown scripts, etc.

I can work on a series like this but would like to get the pending one merged before we open this topic up ?

Regards,
Matt

[Buildroot] [PATCH 1/3] support/scripts/pkg-stats: fix flake8 E741 ambiguous variable name

[Yann E. MORIN]

Matthew, All,

On 2021-05-18 13:21 -0500, Matthew Weber via buildroot spake thusly:
> Signed-off-by: Matthew Weber <matthew.weber@collins.com>

Applied to master, thanks.

Regards,
Yann E. MORIN.

> ---
>  support/scripts/pkg-stats | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index c7e30dfd2b..0cd3674c52 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -375,9 +375,9 @@ def package_init_make_info():
>      variable_list = [x[5:] for x in variable_list if x.startswith("HOST_")] + \
>                      [x for x in variable_list if not x.startswith("HOST_")]
>  
> -    for l in variable_list:
> +    for item in variable_list:
>          # Get variable name and value
> -        pkgvar, value = l.split("=")
> +        pkgvar, value = item.split("=")
>  
>          # Strip the suffix according to the variable
>          if pkgvar.endswith("_LICENSE"):
> -- 
> 2.17.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/3] support/scripts/pkg-stats: fix flake8 E741 ambiguous variable name

[Peter Korsgaard]

>>>>> "Matthew" == Matthew Weber via buildroot <buildroot@busybox.net> writes:

 > Signed-off-by: Matthew Weber <matthew.weber@collins.com>

Committed to 2021.02.x, thanks.

-- 
Bye, Peter Korsgaard