[Buildroot] [PATCH v5 0/7] CPE ID Support

[Buildroot] [PATCH v5 0/7] CPE ID Support

[Matt Weber]

This series begins adding CPE identifier support to Buildroot. The
intent is to establish and maintain a baseline of CPE IDs, one for each
package. Each of these IDs ties back to a NIST database entry for the
respective piece of software, which is linked to specific vunderabilities.

Within Buildroot, a CPE report can be generated (like legal-info) that
captures a target build's list of CPE IDs. This report can then be
checked for validity using the new support/scripts/cpe-report or another
third party tool. The script uses a new cpedb.py helper to search and
identify the validity of the CPE strings.

The pkgstats script has been extended to provide CPE ID checking of
matching/requires update/new as part of its html output.  It also uses
the new cpedb.py helper.

As part of testing this series, the following branch contains a series
of fixups required to make these specific packages match the database.
(I can submit these to the mailing list but there are ~70 of them)

https://github.com/rc-matthew-l-weber/buildroot/tree/cpe-info-github
Commit 14c3ee6 to 567732d

A follow-on patchset will be submitted adding support for pkgstat generation
of CPE updates in XML and Buildroot manual updates for guidance on submission
of those XML database updates to the NIST organization. (We'd like to get
feedback on this series first to save us effort on the update XML stuff)

Matt Weber (7):
  cpe-info: new make target
  cpe-info: id prefix/suffix
  cpe-info: only report target pkgs
  cpe-info: update manual for new pkg vars
  support/scripts/cpedb.py: new CPE XML helper
  support/scripts/pkg-stats: add CPE reporting
  support/scripts/cpe-report: new script

 Makefile                                |  17 ++++-
 docs/manual/adding-packages-generic.txt | 117 ++++++++++++++++++++------------
 package/Makefile.in                     |   4 ++
 package/pkg-generic.mk                  |  21 ++++++
 package/pkg-utils.mk                    |   8 +++
 support/scripts/cpe-report              |  53 +++++++++++++++
 support/scripts/cpedb.py                |  52 ++++++++++++++
 support/scripts/pkg-stats               |  80 ++++++++++++++++++++--
 8 files changed, 302 insertions(+), 50 deletions(-)
 create mode 100755 support/scripts/cpe-report
 create mode 100644 support/scripts/cpedb.py

-- 
1.9.1

[Buildroot] [PATCH v5 1/7] cpe-info: new make target

[Matt Weber]

Similar to make legal-info, produce a csv delimited file containing
all selected packages CPE identification.

Have the pkg infra define CPE_ID_* defaults using the package name
for the vendor and name as most CPE IDs seem to align with that
assumption. Also use the pkg version as the CPE ID's version field.

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
Changes
v2
[Thomas P
 - Moved comment on conditionals back to this patchset where
   the conditional is created vs later

v3
[Thomas P
 - Merged infra define CPE_ID_*  into this patch
 - Report all packages vs restricting to just allowing based on if
   the VENDOR was set (v2). This now represents Thomas P's original
   idea to report everything.  At first I felt I should restrict
   the reporting to those CPE IDs we had made sure were correct.
   Turns out we should have actually let the script handle fixing
   the CPEs and just make a complete design of this up front.

[Matt
 - Moved to using the _project on all vendors instead of just name
---
 Makefile               | 17 ++++++++++++++++-
 package/pkg-generic.mk | 13 +++++++++++++
 package/pkg-utils.mk   |  8 ++++++++
 3 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index c024c65..71632bb 100644
--- a/Makefile
+++ b/Makefile
@@ -146,7 +146,7 @@ nobuild_targets := source %-source \
 	clean distclean help show-targets graph-depends \
 	%-graph-depends %-show-depends %-show-version \
 	graph-build graph-size list-defconfigs \
-	savedefconfig printvars
+	savedefconfig printvars cpe-info %-cpe-info
 ifeq ($(MAKECMDGOALS),)
 BR_BUILDING = y
 else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),)
@@ -233,6 +233,7 @@ LEGAL_MANIFEST_CSV_TARGET = $(LEGAL_INFO_DIR)/manifest.csv
 LEGAL_MANIFEST_CSV_HOST = $(LEGAL_INFO_DIR)/host-manifest.csv
 LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings
 LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
+CPE_MANIFEST_CSV = $(BASE_DIR)/cpe-manifest.csv
 
 BR2_CONFIG = $(CONFIG_DIR)/.config
 
@@ -802,6 +803,19 @@ legal-info: dirs legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p
 		mv .legal-info.sha256 legal-info.sha256)
 	@echo "Legal info produced in $(LEGAL_INFO_DIR)"
 
+.PHONY: cpe-info-clean
+cpe-info-clean:
+	@rm -f $(CPE_MANIFEST_CSV)
+
+.PHONY: cpe-info-prepare
+cpe-info-prepare:
+	@$(call MESSAGE,"Gathering CPE info")
+	@$(call cpe-manifest,CPE ID,CVE PATCHED,PACKAGE,VERSION,SOURCE SITE)
+
+.PHONY: cpe-info
+cpe-info: cpe-info-clean cpe-info-prepare $(foreach p,$(PACKAGES),$(p)-cpe-info)
+	@echo "CPE info produced in $(CPE_MANIFEST_CSV)"
+
 .PHONY: show-targets
 show-targets:
 	@echo $(sort $(PACKAGES)) $(sort $(TARGETS_ROOTFS))
@@ -1070,6 +1084,7 @@ help:
 	@echo '  source                 - download all sources needed for offline-build'
 	@echo '  external-deps          - list external packages used'
 	@echo '  legal-info             - generate info about license compliance'
+	@echo '  cpe-info               - generate info about security CPE identification'
 	@echo '  printvars              - dump all the internal variables'
 	@echo
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 8a3b5f9..67ac436 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -861,6 +861,18 @@ else
 $(2)_KCONFIG_VAR = BR2_PACKAGE_$(2)
 endif
 
+$(2)_CPE_ID_VENDOR ?= $$($(2)_NAME)_project
+$(2)_CPE_ID_NAME ?= $$($(2)_NAME)
+$(2)_CPE_ID_VERSION ?= $$($(2)_VERSION)
+$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION)
+
+$(1)-cpe-info: PKG=$(2)
+$(1)-cpe-info:
+ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
+	@$$(call MESSAGE,"Collecting cpe info")
+	$(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
+endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
+
 # legal-info: declare dependencies and set values used later for the manifest
 ifneq ($$($(2)_LICENSE_FILES),)
 $(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES)
@@ -1002,6 +1014,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-clean-for-reconfigure \
 	$(1)-clean-for-reinstall \
 	$(1)-configure \
+	$(1)-cpe-info \
 	$(1)-depends \
 	$(1)-dirclean \
 	$(1)-external-deps \
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index c3acc22..11a2457 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -95,3 +95,11 @@ define legal-license-file # pkgname, pkgname-pkgver, pkgdir, filename, file-full
 	} && \
 	cp $(5) $(LICENSE_FILES_DIR_$(6))/$(2)/$(4)
 endef
+
+#
+# cpe-info helper functions
+#
+
+define cpe-manifest # cpe, cve patched, pkg name, version, url
+	echo '"$(1)","$(2)","$(3)","$(4)","$(5)"' >>$(CPE_MANIFEST_CSV)
+endef
-- 
1.9.1

[Buildroot] [PATCH v5 2/7] cpe-info: id prefix/suffix

[Matt Weber]

There are two types of software cpe prefixes, one for applications and
one for operating systems. Note: There is a third type for hardware.

This patchset determines which should be used and stores that
information with the package for later use when assembling the CPE
report.

There is also a suffix which we just default to wildcards at this
point.

Refs:
   https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7695.pdf
   https://cpe.mitre.org/specification/

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v1 -> v2
[Thomas P
 - Change to using a filter on pkg name value vs ifelse

v3
[Arnout
 - Moved CPE prefix and suffix defines to package/Makefile.in
---
 package/Makefile.in    | 4 ++++
 package/pkg-generic.mk | 8 +++++++-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/package/Makefile.in b/package/Makefile.in
index 4325f7b..ae69c4e 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -395,6 +395,10 @@ TARGET_CONFIGURE_ARGS = \
 
 ################################################################################
 
+CPE_PREFIX_OS = cpe:2.3:o
+CPE_PREFIX_APP = cpe:2.3:a
+CPE_SUFFIX = *:*:*:*:*:*:*
+
 ifeq ($(BR2_SYSTEM_ENABLE_NLS),y)
 NLS_OPTS = --enable-nls
 TARGET_NLS_DEPENDENCIES = host-gettext
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 67ac436..b56fefa 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -866,11 +866,17 @@ $(2)_CPE_ID_NAME ?= $$($(2)_NAME)
 $(2)_CPE_ID_VERSION ?= $$($(2)_VERSION)
 $(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION)
 
+ifneq ($(filter linux linux-headers,$(1)),)
+$(2)_CPE_PREFIX = $(CPE_PREFIX_OS)
+else
+$(2)_CPE_PREFIX = $(CPE_PREFIX_APP)
+endif
+
 $(1)-cpe-info: PKG=$(2)
 $(1)-cpe-info:
 ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
 	@$$(call MESSAGE,"Collecting cpe info")
-	$(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
+	$(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
 endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
 
 # legal-info: declare dependencies and set values used later for the manifest
-- 
1.9.1

[Buildroot] [PATCH v5 3/7] cpe-info: only report target pkgs

[Matt Weber]

The reporting of host packages causes some duplication and complicates
what is really in the targets configuration. For the purpose of the
first version of this patchset, its assumed that host packages aren't
relevant for the configuration and we only report the target's
contents.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v1 -> v2
[Thomas P
 - select if target vs selecting not host

v3
 - Fixed host build error because cpe-info wasn't defined
---
 package/pkg-generic.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index b56fefa..5d77c83 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -874,10 +874,12 @@ endif
 
 $(1)-cpe-info: PKG=$(2)
 $(1)-cpe-info:
+ifeq ($$($(2)_TYPE),target)
 ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
 	@$$(call MESSAGE,"Collecting cpe info")
 	$(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
 endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
+endif # ifeq ($$($(2)_TYPE),target)
 
 # legal-info: declare dependencies and set values used later for the manifest
 ifneq ($$($(2)_LICENSE_FILES),)
-- 
1.9.1

[Buildroot] [PATCH v5 4/7] cpe-info: update manual for new pkg vars

[Matt Weber]

Provide guidance on setting up the *_CPE_* and *_CVE_* variables.

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
Changes
v2
[Thomas P
 - Reworded LIBFOO_CVE_PATCHED description

[Matt W
 - Added definition for new preset variables to auto-gen the CPE ID
 - Added example LIBFOO_CPE_ID_VENDOR to LIBFOO

v3
 - Updated to make *_CPE_VENDOR optional
 - Changed wording around _CPE_ID as there is only one defined now
---
 docs/manual/adding-packages-generic.txt | 117 ++++++++++++++++++++------------
 1 file changed, 74 insertions(+), 43 deletions(-)

diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
index 7e1f246..45f0279 100644
--- a/docs/manual/adding-packages-generic.txt
+++ b/docs/manual/adding-packages-generic.txt
@@ -24,57 +24,59 @@ system is based on hand-written Makefiles or shell scripts.
 09: LIBFOO_SITE = http://www.foosoftware.org/download
 10: LIBFOO_LICENSE = GPL-3.0+
 11: LIBFOO_LICENSE_FILES = COPYING
-12: LIBFOO_INSTALL_STAGING = YES
-13: LIBFOO_CONFIG_SCRIPTS = libfoo-config
-14: LIBFOO_DEPENDENCIES = host-libaaa libbbb
-15:
-16: define LIBFOO_BUILD_CMDS
-17:	$(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all
-18: endef
-19:
-20: define LIBFOO_INSTALL_STAGING_CMDS
-21:	$(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a
-22:	$(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h
-23:	$(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib
-24: endef
-25:
-26: define LIBFOO_INSTALL_TARGET_CMDS
-27:	$(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib
-28:	$(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d
-29: endef
-30:
-31: define LIBFOO_USERS
-32:	foo -1 libfoo -1 * - - - LibFoo daemon
-33: endef
-34:
-35: define LIBFOO_DEVICES
-36:	/dev/foo  c  666  0  0	42  0  -  -  -
-37: endef
-38:
-39: define LIBFOO_PERMISSIONS
-40:	/bin/foo  f  4755  foo  libfoo	 -  -  -  -  -
-41: endef
-42:
-43: $(eval $(generic-package))
+12: LIBFOO_CPE_ID_VENDOR = foosoftware
+13: LIBFOO_INSTALL_STAGING = YES
+14: LIBFOO_CONFIG_SCRIPTS = libfoo-config
+15: LIBFOO_DEPENDENCIES = host-libaaa libbbb
+16:
+17: define LIBFOO_BUILD_CMDS
+18:	$(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all
+19: endef
+20:
+21: define LIBFOO_INSTALL_STAGING_CMDS
+22:	$(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a
+23:	$(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h
+24:	$(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib
+25: endef
+26:
+27: define LIBFOO_INSTALL_TARGET_CMDS
+28:	$(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib
+29:	$(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d
+30: endef
+31:
+32: define LIBFOO_USERS
+33:	foo -1 libfoo -1 * - - - LibFoo daemon
+34: endef
+35:
+36: define LIBFOO_DEVICES
+37:	/dev/foo  c  666  0  0	42  0  -  -  -
+38: endef
+39:
+40: define LIBFOO_PERMISSIONS
+41:	/bin/foo  f  4755  foo  libfoo	 -  -  -  -  -
+42: endef
+43:
+44: $(eval $(generic-package))
 --------------------------------
 
-The Makefile begins on line 7 to 11 with metadata information: the
+The Makefile begins on line 7 to 12 with metadata information: the
 version of the package (+LIBFOO_VERSION+), the name of the
 tarball containing the package (+LIBFOO_SOURCE+) (xz-ed tarball recommended)
 the Internet location at which the tarball can be downloaded from
-(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+) and file with the
-license text (+LIBFOO_LICENSE_FILES+). All variables must start with
+(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+), the file with the
+license text (+LIBFOO_LICENSE_FILES+) and the vendor for vunerability
+analysis (+LIBFOO_CPE_ID_VENDOR+). All variables must start with
 the same prefix, +LIBFOO_+ in this case. This prefix is always the
 uppercased version of the package name (see below to understand where
 the package name is defined).
 
-On line 12, we specify that this package wants to install something to
+On line 13, we specify that this package wants to install something to
 the staging space. This is often needed for libraries, since they must
 install header files and other development files in the staging space.
 This will ensure that the commands listed in the
 +LIBFOO_INSTALL_STAGING_CMDS+ variable will be executed.
 
-On line 13, we specify that there is some fixing to be done to some
+On line 14, we specify that there is some fixing to be done to some
 of the 'libfoo-config' files that were installed during
 +LIBFOO_INSTALL_STAGING_CMDS+ phase.
 These *-config files are executable shell script files that are
@@ -122,14 +124,14 @@ IMAGEMAGICK_CONFIG_SCRIPTS = \
 --------------------------------
 ================================
 
-On line 14, we specify the list of dependencies this package relies
+On line 15, we specify the list of dependencies this package relies
 on. These dependencies are listed in terms of lower-case package names,
 which can be packages for the target (without the +host-+
 prefix) or packages for the host (with the +host-+) prefix).
 Buildroot will ensure that all these packages are built and installed
 'before' the current package starts its configuration.
 
-The rest of the Makefile, lines 16..29, defines what should be done
+The rest of the Makefile, lines 17..29, defines what should be done
 at the different steps of the package configuration, compilation and
 installation.
 +LIBFOO_BUILD_CMDS+ tells what steps should be performed to
@@ -142,16 +144,16 @@ All these steps rely on the +$(@D)+ variable, which
 contains the directory where the source code of the package has been
 extracted.
 
-On lines 31..43, we define a user that is used by this package (e.g.
+On lines 32..44, we define a user that is used by this package (e.g.
 to run a daemon as non-root) (+LIBFOO_USERS+).
 
-On line 35..37, we define a device-node file used by this package
+On line 36..38, we define a device-node file used by this package
 (+LIBFOO_DEVICES+).
 
-On line 39..41, we define the permissions to set to specific files
+On line 40..42, we define the permissions to set to specific files
 installed by this package (+LIBFOO_PERMISSIONS+).
 
-Finally, on line 43, we call the +generic-package+ function, which
+Finally, on line 44, we call the +generic-package+ function, which
 generates, according to the variables defined previously, all the
 Makefile code necessary to make your package working.
 
@@ -469,6 +471,35 @@ information is (assuming the package name is +libfoo+) :
   locations, `/lib/firmware`, `/usr/lib/firmware`, `/lib/modules`,
   `/usr/lib/modules`, and `/usr/share`, which are automatically excluded.
 
+* +LIBFOO_CPE_ID_VENDOR+
+  This variable is optional. It only must be defined if the package name
+  does not match what the CPE ID uses for the vendor. By default it's set
+  to <pkg-name>_project.
+
+* +LIBFOO_CPE_ID_NAME+
+  This variable is optional. It only must be defined if the package name
+  does not match what the CPE ID uses for the name. By default it's set
+  to <pkg-name>.
+
+* +LIBFOO_CPE_ID_VERSION+
+  This variable is optional. By default it's set to <pkg-version>.
+
+* +LIBFOO_CPE_ID+ is optional, as the package infrastructure hangles the
+  default case of a single package's Common Product Enumeration (CPE)
+  identification string. +make cpe-info+ copies all of these into a
+  +cpe-manifest.csv+ file. To identify a package's possible CPE,
+  the National Vunerability Database can be searched at
+  https://nvd.nist.gov/products/cpe/search.
+
+* +LIBFOO_CVE_PATCHED+ is a space-separated list of the package's Common
+  Vunerability Enumeration (CVE) identification strings. This list
+  enumerates CVEs which are fixed by patches added in Buildroot. This
+  allows the CPE reporting to provide additional detail on CVEs which
+  have been fixed, even if Buildroot is not yet using an updated upstream
+  release including the fix. This variable is optional. If it is not
+  defined, the +CVE PATCHED+ field will appear empty in the manifest
+  file for this package.
+
 The recommended way to define these variables is to use the following
 syntax:
 
-- 
1.9.1

[Buildroot] [PATCH v5 5/7] support/scripts/cpedb.py: new CPE XML helper

[Matt Weber]

Python class which consumes a NIST CPE XML and provides helper
functions to access and search the db's data.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
v5
[Ricardo
 - Fixed typo in join/split of cpe str without version
 - Removed extra prints as they aren't needed when we have the
   output reports/stdout
 - Updated v4 comments about general flake formatting cleanup
 - Incorporated parts of patch 1/2 suggestions for optimizations

[Arnout
 - added pre-processing of cpe values into two sets, one with
   and one without version
 - Collectly with Ricardo, decided to move cpe class to this
   seperate script

v1 -> v4
 - No version
---
 support/scripts/cpedb.py | 52 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
 create mode 100644 support/scripts/cpedb.py

diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
new file mode 100644
index 0000000..77d1d17
--- /dev/null
+++ b/support/scripts/cpedb.py
@@ -0,0 +1,52 @@
+import sys
+import urllib2
+import xmltodict
+import gzip
+from StringIO import StringIO
+
+CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
+
+
+class CPEDB:
+    all_cpedb = dict()
+    all_cpes = set()
+    all_cpes_no_version = set()
+
+    def get_xml_dict(self):
+        print("CPE: Fetching xml manifest...")
+        try:
+            compressed_cpe_file = urllib2.urlopen(CPE_XML_URL)
+            print("CPE: Unzipping xml manifest...")
+            cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read()
+            print("CPE: Converting xml manifest to dict...")
+            self.all_cpedb = xmltodict.parse(cpe_file)
+
+            for cpe in self.all_cpedb['cpe-list']['cpe-item']:
+                cpe_str = cpe['cpe-23:cpe23-item']['@name']
+                cpe_str_no_version = self.get_cpe_no_version(cpe_str)
+                self.all_cpes.add(cpe_str)
+                self.all_cpes_no_version.add(cpe_str_no_version)
+
+        except urllib2.HTTPError:
+            print("CPE: HTTP Error: %s" % CPE_XML_URL)
+            sys.exit(1)
+        except urllib2.URLError:
+            print("CPE: URL Error: %s" % CPE_XML_URL)
+            sys.exit(1)
+
+    def find_partial(self, cpe_str):
+        cpe_str_no_version = self.get_cpe_no_version(cpe_str)
+        if cpe_str_no_version in self.all_cpes_no_version:
+            return cpe_str_no_version
+
+    def find(self, cpe_str):
+        if cpe_str in self.all_cpes:
+            return cpe_str
+
+    def get_cpe_no_version(self, cpe):
+        return ":".join(cpe.split(":")[:5])
+
+    def get_nvd_url(self, cpe_str):
+        return "https://nvd.nist.gov/products/cpe/search/results?keyword=" + \
+                urllib2.quote(cpe_str) + \
+                "&status=FINAL&orderBy=CPEURI&namingFormat=2.3"
-- 
1.9.1

[Buildroot] [PATCH v5 6/7] support/scripts/pkg-stats: add CPE reporting

[Matt Weber]

Pkg status now includes CPE as an item reported in the html
output (stat summary and for each pkg)

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes

v4 -> v5
[Ricardo
 - Renamed patch to correctly match file name
 - Removed extra prints as they aren't needed when we have the
   output reports/stdout
 - Updated v4 comments about general flake formatting cleanup

[Arnout
 - Collectly with Ricardo, decided to move cpe report analysis to
   a seperate script and breakout a module that's imported for the
   cpedb class
 - Rename cpe_dict to instead be cpedb

v3 -> v4
 - Collapsed patch 5 and 6 together into this single patch

[Eric
 - added except handling around file io
 - fixed condition where buildroot isn't generating a CPE
   string as part of the infra and output that is the case.
   (eventually these probably could be fixed but there aren't
   many at this point)

[Ricardo
 - fixed patch naming and resolved flake8 issues
 - took the opportunity to also fix other flake8 syntax
   update suggestions
 - added except handling to have proper exits
 - cleaned up csv file header skippin
 - condensed partial cve string split
 - updated help txt as suggested
 - reworked output file requirement.  Removed -o as required but
   added check if provided when -c isn't used

v3
 - New patch
---
 support/scripts/pkg-stats | 80 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 74 insertions(+), 6 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 43f7e8d..db58a75 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -24,6 +24,7 @@ from collections import defaultdict
 import re
 import subprocess
 import sys
+from cpedb import CPEDB
 
 INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
 
@@ -32,6 +33,7 @@ class Package:
     all_licenses = list()
     all_license_files = list()
     all_versions = dict()
+    all_cpe_id = dict()
 
     def __init__(self, name, path):
         self.name = name
@@ -43,6 +45,8 @@ class Package:
         self.patch_count = 0
         self.warnings = 0
         self.current_version = None
+        self.cpe_id = None
+        self.has_cpe = False
 
     def pkgvar(self):
         return self.name.upper().replace("-", "_")
@@ -116,6 +120,26 @@ class Package:
                 self.warnings = int(m.group(1))
                 return
 
+    def set_cpe_info(self, cpe_dict):
+        """
+        Fills in the .has_cpe field
+        """
+        var = self.pkgvar()
+        if var in self.all_cpe_id:
+            self.cpe_id = self.all_cpe_id[var]
+        if self.cpe_id is None:
+            # BR infra did not build a CPE ID for this pkg
+            # as it's most likely a host pkg
+            return
+        result = cpe_dict.find(self.cpe_id)
+        if not result:
+            result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(self.cpe_id))
+            if result:
+                self.has_cpe = "Update"
+            # Unset case for has_cpe is assumed missing/does not exist
+        else:
+            self.has_cpe = cpe_dict.get_nvd_url(self.cpe_id)
+
     def __eq__(self, other):
         return self.path == other.path
 
@@ -254,6 +278,20 @@ def package_init_make_info():
 
         Package.all_versions[pkgvar] = value
 
+    # CPE ID
+    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
+                                 "-s", "printvars", "VARS=%_CPE_ID"])
+    for l in o.splitlines():
+        # Get variable name and value
+        pkgvar, value = l.split("=")
+
+        # Strip _CPE_ID
+        pkgvar = pkgvar[:-7]
+        if pkgvar in ("LINUX", "LINUX_HEADERS"):
+            Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*"
+        else:
+            Package.all_cpe_id[pkgvar] = "cpe:2.3:a:" + value + ":*:*:*:*:*:*:*"
+
 
 def calculate_stats(packages):
     stats = defaultdict(int)
@@ -279,6 +317,12 @@ def calculate_stats(packages):
             stats["hash"] += 1
         else:
             stats["no-hash"] += 1
+        if pkg.has_cpe == "Update":
+            stats["update-cpe"] += 1
+        elif pkg.has_cpe:
+            stats["cpe"] += 1
+        else:
+            stats["no-cpe"] += 1
         stats["patches"] += pkg.patch_count
     return stats
 
@@ -422,6 +466,20 @@ def dump_html_pkg(f, pkg):
     f.write("  <td class=\"%s\">%d</td>\n" %
             (" ".join(td_class), pkg.warnings))
 
+    # CPE Valid
+    td_class = ["centered"]
+    if not pkg.has_cpe:
+        td_class.append("wrong")
+        f.write("  <td class=\"%s\">%s</td>\n" %
+                (" ".join(td_class), boolean_str(pkg.has_cpe)))
+    elif pkg.has_cpe == "Update":
+        td_class.append("wrong")
+        f.write("  <td class=\"%s\">Update</td>\n" %
+                (" ".join(td_class)))
+    else:
+        td_class.append("correct")
+        f.write("  <td class=\"%s\"><a href=\"%s\">%s</a></td>\n" %
+                (" ".join(td_class), pkg.has_cpe, boolean_str(pkg.has_cpe)))
     f.write(" </tr>\n")
 
 
@@ -437,6 +495,7 @@ def dump_html_all_pkgs(f, packages):
 <td class=\"centered\">Hash file</td>
 <td class=\"centered\">Current version</td>
 <td class=\"centered\">Warnings</td>
+<td class=\"centered\">CPE Valid</td>
 </tr>
 """)
     for pkg in sorted(packages):
@@ -463,6 +522,12 @@ def dump_html_stats(f, stats):
             stats["hash"])
     f.write(" <tr><td>Packages not having a hash file</td><td>%s</td></tr>\n" %
             stats["no-hash"])
+    f.write(" <tr><td>Packages having a registered CPE</td><td>%s</td></tr>\n" %
+            stats["cpe"])
+    f.write(" <tr><td>Packages needing CPE update</td><td>%s</td></tr>\n" %
+            stats["update-cpe"])
+    f.write(" <tr><td>Packages missing a registered CPE</td><td>%s</td></tr>\n" %
+            stats["no-cpe"])
     f.write(" <tr><td>Total number of patches</td><td>%s</td></tr>\n" %
             stats["patches"])
     f.write("</table>\n")
@@ -499,17 +564,19 @@ def parse_args():
 def __main__():
     args = parse_args()
     if args.npackages and args.packages:
-        print "ERROR: -n and -p are mutually exclusive"
+        print("ERROR: -n and -p are mutually exclusive")
         sys.exit(1)
     if args.packages:
         package_list = args.packages.split(",")
     else:
         package_list = None
-    print "Build package list ..."
+    cpedb = CPEDB()
+    cpedb.get_xml_dict()
+    print("Build package list ...")
     packages = get_pkglist(args.npackages, package_list)
-    print "Getting package make info ..."
+    print("Getting package make info ...")
     package_init_make_info()
-    print "Getting package details ..."
+    print("Getting package details ...")
     for pkg in packages:
         pkg.set_infra()
         pkg.set_license()
@@ -517,9 +584,10 @@ def __main__():
         pkg.set_patch_count()
         pkg.set_check_package_warnings()
         pkg.set_current_version()
-    print "Calculate stats"
+        pkg.set_cpe_info(cpedb)
+    print("Calculate stats")
     stats = calculate_stats(packages)
-    print "Write HTML"
+    print("Write HTML")
     dump_html(packages, stats, args.output)
 
 
-- 
1.9.1

[Buildroot] [PATCH v5 7/7] support/scripts/cpe-report: new script

[Matt Weber]

The script supports looking up all the CPEs provided in a
make cpe-info csv file export from a target Buildroot build.
It checks the current version and suggests a CPE needs update
or possibly initial submission to NIST.

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v5
[Ricardo
 - Updated v4 comments about general flake formatting cleanup
 - Incorporated parts of patch 1/2 suggestions for optimizations

[Ricardo/Arnout
 - Collectly, decided to move cpe report analysis to this
   script and use a seperate module cpedb class

[Arnout
 - Rename cpe_dict to instead be cpedb

v1 -> v4
 - Patch did not exist and was part of pkg-stats file
---
 support/scripts/cpe-report | 53 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)
 create mode 100755 support/scripts/cpe-report

diff --git a/support/scripts/cpe-report b/support/scripts/cpe-report
new file mode 100755
index 0000000..036eab2
--- /dev/null
+++ b/support/scripts/cpe-report
@@ -0,0 +1,53 @@
+#!/usr/bin/env python
+
+import argparse
+import sys
+import csv
+from cpedb import CPEDB
+
+
+def get_target_cpe_report(cpe_report_file, cpedb):
+    report_cpe_exact_match = ""
+    report_cpe_needing_update = ""
+    report_cpe_missing = ""
+
+    print("CPE: Checking for matches...")
+    try:
+        with open(cpe_report_file) as cpe_file:
+            cpe_list = csv.reader(cpe_file)
+            next(cpe_list)  # make cpe-info has a one line header
+            for cpe in cpe_list:
+                result = cpedb.find(cpe[0])
+                if not result:
+                    result = cpedb.find_partial(cpedb.get_cpe_no_version(cpe[0]))
+                    if not result:
+                        report_cpe_missing += cpe[0] + "\n"
+                    else:
+                        report_cpe_needing_update += cpe[0] + "\n"
+                else:
+                    report_cpe_exact_match += cpe[0] + "\n"
+    except (OSError, IOError) as e:
+        print("CPE: report csv file (%s): %s" % (e.errno, e.strerror))
+        sys.exit(1)
+
+    print("CPE: Found EXACT match:\n" + report_cpe_exact_match)
+    print("CPE: Found but REQUIRES UPDATE:\n" + report_cpe_needing_update)
+    print("CPE: Not found (proposing the following to be added):\n" + report_cpe_missing)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-c', dest='cpe_report', action='store', required=True,
+                        help='CPE Report generated by make cpe-info (csv format)')
+    return parser.parse_args()
+
+
+def __main__():
+    args = parse_args()
+    cpedb = CPEDB()
+    cpedb.get_xml_dict()
+    print("Performing Target CPE Report Analysis...")
+    get_target_cpe_report(args.cpe_report, cpedb)
+
+
+__main__()
-- 
1.9.1

[Buildroot] [PATCH v5 6/7] support/scripts/pkg-stats: add CPE reporting

[Thomas Petazzoni]

Hello,

On Thu, 17 May 2018 22:13:19 -0500, Matt Weber wrote:

>      packages = get_pkglist(args.npackages, package_list)
> -    print "Getting package make info ..."
> +    print("Getting package make info ...")
>      package_init_make_info()
> -    print "Getting package details ..."
> +    print("Getting package details ...")
>      for pkg in packages:
>          pkg.set_infra()
>          pkg.set_license()
> @@ -517,9 +584,10 @@ def __main__():
>          pkg.set_patch_count()
>          pkg.set_check_package_warnings()
>          pkg.set_current_version()
> -    print "Calculate stats"
> +        pkg.set_cpe_info(cpedb)
> +    print("Calculate stats")
>      stats = calculate_stats(packages)
> -    print "Write HTML"
> +    print("Write HTML")

I hate silly reviews, and this is a silly review, but this change to
use parenthesis in print calls is not related to introducing CPE
support, so it should be a separate patch (which I could merge sooner
and separately from the CPE discussion).

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v5 6/7] support/scripts/pkg-stats: add CPE reporting

[Matthew Weber]

Thomas,

On Fri, May 18, 2018 at 2:13 AM, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
> Hello,
>
> On Thu, 17 May 2018 22:13:19 -0500, Matt Weber wrote:
>
>>      packages = get_pkglist(args.npackages, package_list)
>> -    print "Getting package make info ..."
>> +    print("Getting package make info ...")
>>      package_init_make_info()
>> -    print "Getting package details ..."
>> +    print("Getting package details ...")
>>      for pkg in packages:
>>          pkg.set_infra()
>>          pkg.set_license()
>> @@ -517,9 +584,10 @@ def __main__():
>>          pkg.set_patch_count()
>>          pkg.set_check_package_warnings()
>>          pkg.set_current_version()
>> -    print "Calculate stats"
>> +        pkg.set_cpe_info(cpedb)
>> +    print("Calculate stats")
>>      stats = calculate_stats(packages)
>> -    print "Write HTML"
>> +    print("Write HTML")
>
> I hate silly reviews, and this is a silly review, but this change to
> use parenthesis in print calls is not related to introducing CPE
> support, so it should be a separate patch (which I could merge sooner
> and separately from the CPE discussion).

Noted, Ricardo had mentioned similar.  See below.
http://patchwork.ozlabs.org/patch/916300/

Matt