* [Buildroot] [PATCH v3 1/6] cpe-info: new make target
2018-05-07 20:29 [Buildroot] [PATCH v3 0/6] CPE ID Support Matt Weber
@ 2018-05-07 20:30 ` Matt Weber
2018-05-07 20:30 ` [Buildroot] [PATCH v3 2/6] cpe-info: id prefix/suffix Matt Weber
` (4 subsequent siblings)
5 siblings, 0 replies; 14+ messages in thread
From: Matt Weber @ 2018-05-07 20:30 UTC (permalink / raw)
To: buildroot
Similar to make legal-info, produce a csv delimited file containing
all selected packages CPE identification.
Have the pkg infra define CPE_ID_* defaults using the package name
for the vendor and name as most CPE IDs seem to align with that
assumption. Also use the pkg version as the CPE ID's version field.
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
Changes
v2
[Thomas P
- Moved comment on conditionals back to this patchset where
the conditional is created vs later
v3
[Thomas P
- Merged infra define CPE_ID_* into this patch
- Report all packages vs restricting to just allowing based on if
the VENDOR was set (v2). This now represents Thomas P's original
idea to report everything. At first I felt I should restrict
the reporting to those CPE IDs we had made sure were correct.
Turns out we should have actually let the script handle fixing
the CPEs and just make a complete design of this up front.
[Matt
- Moved to using the _project on all vendors instead of just name
---
Makefile | 17 ++++++++++++++++-
package/pkg-generic.mk | 13 +++++++++++++
package/pkg-utils.mk | 8 ++++++++
3 files changed, 37 insertions(+), 1 deletion(-)
diff --git a/Makefile b/Makefile
index c024c65..71632bb 100644
--- a/Makefile
+++ b/Makefile
@@ -146,7 +146,7 @@ nobuild_targets := source %-source \
clean distclean help show-targets graph-depends \
%-graph-depends %-show-depends %-show-version \
graph-build graph-size list-defconfigs \
- savedefconfig printvars
+ savedefconfig printvars cpe-info %-cpe-info
ifeq ($(MAKECMDGOALS),)
BR_BUILDING = y
else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),)
@@ -233,6 +233,7 @@ LEGAL_MANIFEST_CSV_TARGET = $(LEGAL_INFO_DIR)/manifest.csv
LEGAL_MANIFEST_CSV_HOST = $(LEGAL_INFO_DIR)/host-manifest.csv
LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings
LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
+CPE_MANIFEST_CSV = $(BASE_DIR)/cpe-manifest.csv
BR2_CONFIG = $(CONFIG_DIR)/.config
@@ -802,6 +803,19 @@ legal-info: dirs legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p
mv .legal-info.sha256 legal-info.sha256)
@echo "Legal info produced in $(LEGAL_INFO_DIR)"
+.PHONY: cpe-info-clean
+cpe-info-clean:
+ @rm -f $(CPE_MANIFEST_CSV)
+
+.PHONY: cpe-info-prepare
+cpe-info-prepare:
+ @$(call MESSAGE,"Gathering CPE info")
+ @$(call cpe-manifest,CPE ID,CVE PATCHED,PACKAGE,VERSION,SOURCE SITE)
+
+.PHONY: cpe-info
+cpe-info: cpe-info-clean cpe-info-prepare $(foreach p,$(PACKAGES),$(p)-cpe-info)
+ @echo "CPE info produced in $(CPE_MANIFEST_CSV)"
+
.PHONY: show-targets
show-targets:
@echo $(sort $(PACKAGES)) $(sort $(TARGETS_ROOTFS))
@@ -1070,6 +1084,7 @@ help:
@echo ' source - download all sources needed for offline-build'
@echo ' external-deps - list external packages used'
@echo ' legal-info - generate info about license compliance'
+ @echo ' cpe-info - generate info about security CPE identification'
@echo ' printvars - dump all the internal variables'
@echo
@echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build'
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 8a3b5f9..67ac436 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -861,6 +861,18 @@ else
$(2)_KCONFIG_VAR = BR2_PACKAGE_$(2)
endif
+$(2)_CPE_ID_VENDOR ?= $$($(2)_NAME)_project
+$(2)_CPE_ID_NAME ?= $$($(2)_NAME)
+$(2)_CPE_ID_VERSION ?= $$($(2)_VERSION)
+$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION)
+
+$(1)-cpe-info: PKG=$(2)
+$(1)-cpe-info:
+ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
+ @$$(call MESSAGE,"Collecting cpe info")
+ $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
+endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
+
# legal-info: declare dependencies and set values used later for the manifest
ifneq ($$($(2)_LICENSE_FILES),)
$(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES)
@@ -1002,6 +1014,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
$(1)-clean-for-reconfigure \
$(1)-clean-for-reinstall \
$(1)-configure \
+ $(1)-cpe-info \
$(1)-depends \
$(1)-dirclean \
$(1)-external-deps \
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index c3acc22..11a2457 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -95,3 +95,11 @@ define legal-license-file # pkgname, pkgname-pkgver, pkgdir, filename, file-full
} && \
cp $(5) $(LICENSE_FILES_DIR_$(6))/$(2)/$(4)
endef
+
+#
+# cpe-info helper functions
+#
+
+define cpe-manifest # cpe, cve patched, pkg name, version, url
+ echo '"$(1)","$(2)","$(3)","$(4)","$(5)"' >>$(CPE_MANIFEST_CSV)
+endef
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [Buildroot] [PATCH v3 2/6] cpe-info: id prefix/suffix
2018-05-07 20:29 [Buildroot] [PATCH v3 0/6] CPE ID Support Matt Weber
2018-05-07 20:30 ` [Buildroot] [PATCH v3 1/6] cpe-info: new make target Matt Weber
@ 2018-05-07 20:30 ` Matt Weber
2018-05-07 20:30 ` [Buildroot] [PATCH v3 3/6] cpe-info: only report target pkgs Matt Weber
` (3 subsequent siblings)
5 siblings, 0 replies; 14+ messages in thread
From: Matt Weber @ 2018-05-07 20:30 UTC (permalink / raw)
To: buildroot
There are two types of software cpe prefixes, one for applications and
one for operating systems. Note: There is a third type for hardware.
This patchset determines which should be used and stores that
information with the package for later use when assembling the CPE
report.
There is also a suffix which we just default to wildcards at this
point.
Refs:
https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7695.pdf
https://cpe.mitre.org/specification/
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v1 -> v2
[Thomas P
- Change to using a filter on pkg name value vs ifelse
v3
[Arnout
- Moved CPE prefix and suffix defines to package/Makefile.in
---
package/Makefile.in | 4 ++++
package/pkg-generic.mk | 8 +++++++-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/package/Makefile.in b/package/Makefile.in
index 4325f7b..ae69c4e 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -395,6 +395,10 @@ TARGET_CONFIGURE_ARGS = \
################################################################################
+CPE_PREFIX_OS = cpe:2.3:o
+CPE_PREFIX_APP = cpe:2.3:a
+CPE_SUFFIX = *:*:*:*:*:*:*
+
ifeq ($(BR2_SYSTEM_ENABLE_NLS),y)
NLS_OPTS = --enable-nls
TARGET_NLS_DEPENDENCIES = host-gettext
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 67ac436..b56fefa 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -866,11 +866,17 @@ $(2)_CPE_ID_NAME ?= $$($(2)_NAME)
$(2)_CPE_ID_VERSION ?= $$($(2)_VERSION)
$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION)
+ifneq ($(filter linux linux-headers,$(1)),)
+$(2)_CPE_PREFIX = $(CPE_PREFIX_OS)
+else
+$(2)_CPE_PREFIX = $(CPE_PREFIX_APP)
+endif
+
$(1)-cpe-info: PKG=$(2)
$(1)-cpe-info:
ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
@$$(call MESSAGE,"Collecting cpe info")
- $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
+ $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
# legal-info: declare dependencies and set values used later for the manifest
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [Buildroot] [PATCH v3 3/6] cpe-info: only report target pkgs
2018-05-07 20:29 [Buildroot] [PATCH v3 0/6] CPE ID Support Matt Weber
2018-05-07 20:30 ` [Buildroot] [PATCH v3 1/6] cpe-info: new make target Matt Weber
2018-05-07 20:30 ` [Buildroot] [PATCH v3 2/6] cpe-info: id prefix/suffix Matt Weber
@ 2018-05-07 20:30 ` Matt Weber
2018-05-07 20:30 ` [Buildroot] [PATCH v3 4/6] cpe-info: update manual for new pkg vars Matt Weber
` (2 subsequent siblings)
5 siblings, 0 replies; 14+ messages in thread
From: Matt Weber @ 2018-05-07 20:30 UTC (permalink / raw)
To: buildroot
The reporting of host packages causes some duplication and complicates
what is really in the targets configuration. For the purpose of the
first version of this patchset, its assumed that host packages aren't
relevant for the configuration and we only report the target's
contents.
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v1 -> v2
[Thomas P
- select if target vs selecting not host
v3
- Fixed host build error because cpe-info wasn't defined
---
package/pkg-generic.mk | 2 ++
1 file changed, 2 insertions(+)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index b56fefa..5d77c83 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -874,10 +874,12 @@ endif
$(1)-cpe-info: PKG=$(2)
$(1)-cpe-info:
+ifeq ($$($(2)_TYPE),target)
ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
@$$(call MESSAGE,"Collecting cpe info")
$(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
+endif # ifeq ($$($(2)_TYPE),target)
# legal-info: declare dependencies and set values used later for the manifest
ifneq ($$($(2)_LICENSE_FILES),)
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [Buildroot] [PATCH v3 4/6] cpe-info: update manual for new pkg vars
2018-05-07 20:29 [Buildroot] [PATCH v3 0/6] CPE ID Support Matt Weber
` (2 preceding siblings ...)
2018-05-07 20:30 ` [Buildroot] [PATCH v3 3/6] cpe-info: only report target pkgs Matt Weber
@ 2018-05-07 20:30 ` Matt Weber
2018-05-07 20:30 ` [Buildroot] [PATCH v3 5/6] support/scripts/pkgstat: add target CPE reporting Matt Weber
2018-05-07 20:30 ` [Buildroot] [PATCH v3 6/6] support/scripts/pkgstat: add CPE status reporting Matt Weber
5 siblings, 0 replies; 14+ messages in thread
From: Matt Weber @ 2018-05-07 20:30 UTC (permalink / raw)
To: buildroot
Provide guidance on setting up the *_CPE_* and *_CVE_* variables.
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
Changes
v2
[Thomas P
- Reworded LIBFOO_CVE_PATCHED description
[Matt W
- Added definition for new preset variables to auto-gen the CPE ID
- Added example LIBFOO_CPE_ID_VENDOR to LIBFOO
v3
- Updated to make *_CPE_VENDOR optional
- Changed wording around _CPE_ID as there is only one defined now
---
docs/manual/adding-packages-generic.txt | 117 ++++++++++++++++++++------------
1 file changed, 74 insertions(+), 43 deletions(-)
diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
index 7e1f246..45f0279 100644
--- a/docs/manual/adding-packages-generic.txt
+++ b/docs/manual/adding-packages-generic.txt
@@ -24,57 +24,59 @@ system is based on hand-written Makefiles or shell scripts.
09: LIBFOO_SITE = http://www.foosoftware.org/download
10: LIBFOO_LICENSE = GPL-3.0+
11: LIBFOO_LICENSE_FILES = COPYING
-12: LIBFOO_INSTALL_STAGING = YES
-13: LIBFOO_CONFIG_SCRIPTS = libfoo-config
-14: LIBFOO_DEPENDENCIES = host-libaaa libbbb
-15:
-16: define LIBFOO_BUILD_CMDS
-17: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all
-18: endef
-19:
-20: define LIBFOO_INSTALL_STAGING_CMDS
-21: $(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a
-22: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h
-23: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib
-24: endef
-25:
-26: define LIBFOO_INSTALL_TARGET_CMDS
-27: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib
-28: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d
-29: endef
-30:
-31: define LIBFOO_USERS
-32: foo -1 libfoo -1 * - - - LibFoo daemon
-33: endef
-34:
-35: define LIBFOO_DEVICES
-36: /dev/foo c 666 0 0 42 0 - - -
-37: endef
-38:
-39: define LIBFOO_PERMISSIONS
-40: /bin/foo f 4755 foo libfoo - - - - -
-41: endef
-42:
-43: $(eval $(generic-package))
+12: LIBFOO_CPE_ID_VENDOR = foosoftware
+13: LIBFOO_INSTALL_STAGING = YES
+14: LIBFOO_CONFIG_SCRIPTS = libfoo-config
+15: LIBFOO_DEPENDENCIES = host-libaaa libbbb
+16:
+17: define LIBFOO_BUILD_CMDS
+18: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all
+19: endef
+20:
+21: define LIBFOO_INSTALL_STAGING_CMDS
+22: $(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a
+23: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h
+24: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib
+25: endef
+26:
+27: define LIBFOO_INSTALL_TARGET_CMDS
+28: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib
+29: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d
+30: endef
+31:
+32: define LIBFOO_USERS
+33: foo -1 libfoo -1 * - - - LibFoo daemon
+34: endef
+35:
+36: define LIBFOO_DEVICES
+37: /dev/foo c 666 0 0 42 0 - - -
+38: endef
+39:
+40: define LIBFOO_PERMISSIONS
+41: /bin/foo f 4755 foo libfoo - - - - -
+42: endef
+43:
+44: $(eval $(generic-package))
--------------------------------
-The Makefile begins on line 7 to 11 with metadata information: the
+The Makefile begins on line 7 to 12 with metadata information: the
version of the package (+LIBFOO_VERSION+), the name of the
tarball containing the package (+LIBFOO_SOURCE+) (xz-ed tarball recommended)
the Internet location at which the tarball can be downloaded from
-(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+) and file with the
-license text (+LIBFOO_LICENSE_FILES+). All variables must start with
+(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+), the file with the
+license text (+LIBFOO_LICENSE_FILES+) and the vendor for vunerability
+analysis (+LIBFOO_CPE_ID_VENDOR+). All variables must start with
the same prefix, +LIBFOO_+ in this case. This prefix is always the
uppercased version of the package name (see below to understand where
the package name is defined).
-On line 12, we specify that this package wants to install something to
+On line 13, we specify that this package wants to install something to
the staging space. This is often needed for libraries, since they must
install header files and other development files in the staging space.
This will ensure that the commands listed in the
+LIBFOO_INSTALL_STAGING_CMDS+ variable will be executed.
-On line 13, we specify that there is some fixing to be done to some
+On line 14, we specify that there is some fixing to be done to some
of the 'libfoo-config' files that were installed during
+LIBFOO_INSTALL_STAGING_CMDS+ phase.
These *-config files are executable shell script files that are
@@ -122,14 +124,14 @@ IMAGEMAGICK_CONFIG_SCRIPTS = \
--------------------------------
================================
-On line 14, we specify the list of dependencies this package relies
+On line 15, we specify the list of dependencies this package relies
on. These dependencies are listed in terms of lower-case package names,
which can be packages for the target (without the +host-+
prefix) or packages for the host (with the +host-+) prefix).
Buildroot will ensure that all these packages are built and installed
'before' the current package starts its configuration.
-The rest of the Makefile, lines 16..29, defines what should be done
+The rest of the Makefile, lines 17..29, defines what should be done
at the different steps of the package configuration, compilation and
installation.
+LIBFOO_BUILD_CMDS+ tells what steps should be performed to
@@ -142,16 +144,16 @@ All these steps rely on the +$(@D)+ variable, which
contains the directory where the source code of the package has been
extracted.
-On lines 31..43, we define a user that is used by this package (e.g.
+On lines 32..44, we define a user that is used by this package (e.g.
to run a daemon as non-root) (+LIBFOO_USERS+).
-On line 35..37, we define a device-node file used by this package
+On line 36..38, we define a device-node file used by this package
(+LIBFOO_DEVICES+).
-On line 39..41, we define the permissions to set to specific files
+On line 40..42, we define the permissions to set to specific files
installed by this package (+LIBFOO_PERMISSIONS+).
-Finally, on line 43, we call the +generic-package+ function, which
+Finally, on line 44, we call the +generic-package+ function, which
generates, according to the variables defined previously, all the
Makefile code necessary to make your package working.
@@ -469,6 +471,35 @@ information is (assuming the package name is +libfoo+) :
locations, `/lib/firmware`, `/usr/lib/firmware`, `/lib/modules`,
`/usr/lib/modules`, and `/usr/share`, which are automatically excluded.
+* +LIBFOO_CPE_ID_VENDOR+
+ This variable is optional. It only must be defined if the package name
+ does not match what the CPE ID uses for the vendor. By default it's set
+ to <pkg-name>_project.
+
+* +LIBFOO_CPE_ID_NAME+
+ This variable is optional. It only must be defined if the package name
+ does not match what the CPE ID uses for the name. By default it's set
+ to <pkg-name>.
+
+* +LIBFOO_CPE_ID_VERSION+
+ This variable is optional. By default it's set to <pkg-version>.
+
+* +LIBFOO_CPE_ID+ is optional, as the package infrastructure hangles the
+ default case of a single package's Common Product Enumeration (CPE)
+ identification string. +make cpe-info+ copies all of these into a
+ +cpe-manifest.csv+ file. To identify a package's possible CPE,
+ the National Vunerability Database can be searched at
+ https://nvd.nist.gov/products/cpe/search.
+
+* +LIBFOO_CVE_PATCHED+ is a space-separated list of the package's Common
+ Vunerability Enumeration (CVE) identification strings. This list
+ enumerates CVEs which are fixed by patches added in Buildroot. This
+ allows the CPE reporting to provide additional detail on CVEs which
+ have been fixed, even if Buildroot is not yet using an updated upstream
+ release including the fix. This variable is optional. If it is not
+ defined, the +CVE PATCHED+ field will appear empty in the manifest
+ file for this package.
+
The recommended way to define these variables is to use the following
syntax:
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [Buildroot] [PATCH v3 5/6] support/scripts/pkgstat: add target CPE reporting
2018-05-07 20:29 [Buildroot] [PATCH v3 0/6] CPE ID Support Matt Weber
` (3 preceding siblings ...)
2018-05-07 20:30 ` [Buildroot] [PATCH v3 4/6] cpe-info: update manual for new pkg vars Matt Weber
@ 2018-05-07 20:30 ` Matt Weber
2018-05-09 21:00 ` Erik Larsson
2018-05-10 3:03 ` Ricardo Martincoski
2018-05-07 20:30 ` [Buildroot] [PATCH v3 6/6] support/scripts/pkgstat: add CPE status reporting Matt Weber
5 siblings, 2 replies; 14+ messages in thread
From: Matt Weber @ 2018-05-07 20:30 UTC (permalink / raw)
To: buildroot
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
support/scripts/pkg-stats | 103 ++++++++++++++++++++++++++++++++++++++--------
1 file changed, 86 insertions(+), 17 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 43f7e8d..144c00c 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -24,9 +24,15 @@ from collections import defaultdict
import re
import subprocess
import sys
+import urllib2
+import xmltodict
+import gzip
+from StringIO import StringIO
+import csv
INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
+CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
class Package:
all_licenses = list()
@@ -484,6 +490,62 @@ def dump_html(packages, stats, output):
dump_gen_info(f)
f.write(html_footer)
+class CPE:
+ all_cpes = dict()
+
+ def get_xml_dict(self):
+ print "CPE: Fetching xml manifest..."
+ try:
+ compressed_cpe_file = urllib2.urlopen(CPE_XML_URL)
+ print "CPE: Unzipping xml manifest..."
+ cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read()
+ print "CPE: Converting xml manifest to dict..."
+ self.all_cpes = xmltodict.parse(cpe_file)
+ except urllib2.HTTPError, e:
+ print "HTTP Error:", e.code, url
+ except urllib2.URLError, e:
+ print "URL Error:", e.reason, url
+
+ def find_partial(self,cpe_str):
+ print "CPE: Searching for partial [%s]" % cpe_str
+ for cpe in self.all_cpes['cpe-list']['cpe-item']:
+ if cpe_str in cpe['cpe-23:cpe23-item']['@name']:
+ return cpe['cpe-23:cpe23-item']['@name']
+
+ def find(self,cpe_str):
+ print "CPE: Searching for [%s]" % cpe_str
+ for cpe in self.all_cpes['cpe-list']['cpe-item']:
+ if cpe['cpe-23:cpe23-item']['@name'] == cpe_str:
+ return cpe['cpe-23:cpe23-item']['@name']
+
+def get_target_cpe_report(cpe_report_file, cpe_dict):
+ report_cpe_exact_match = ""
+ report_cpe_needing_update = ""
+ report_cpe_missing = ""
+
+ print "CPE: Checking for matches..."
+ with open(cpe_report_file) as cpe_file:
+ cpe_list = csv.reader(cpe_file)
+ for cpe in cpe_list:
+ if "CPE ID" not in cpe[0]:
+ result = cpe_dict.find(cpe[0])
+ if not result:
+ cpe_no_version = cpe[0].split(":")[0]+":"+cpe[0].split(":")[1]+":"+cpe[0].split(":")[2]+":"+cpe[0].split(":")[3]+":"+cpe[0].split(":")[4]
+ result = cpe_dict.find_partial(cpe_no_version)
+ if not result:
+ report_cpe_missing += cpe[0] + "\n"
+ else:
+ report_cpe_needing_update += cpe[0] + "\n"
+ else:
+ report_cpe_exact_match += cpe[0] + "\n"
+ print "CPE: Found EXACT match:"
+ print report_cpe_exact_match
+ print "CPE: Found but REQUIRES UPDATE:"
+ print report_cpe_needing_update
+ print "CPE: Not found (proposing the following to be added):"
+ print report_cpe_missing
+
+
def parse_args():
parser = argparse.ArgumentParser()
@@ -493,6 +555,8 @@ def parse_args():
help='Number of packages')
parser.add_argument('-p', dest='packages', action='store',
help='List of packages (comma separated)')
+ parser.add_argument('-c', dest='cpe_report', action='store',
+ help='Buildroot CPE Report (csv format)')
return parser.parse_args()
@@ -505,22 +569,27 @@ def __main__():
package_list = args.packages.split(",")
else:
package_list = None
- print "Build package list ..."
- packages = get_pkglist(args.npackages, package_list)
- print "Getting package make info ..."
- package_init_make_info()
- print "Getting package details ..."
- for pkg in packages:
- pkg.set_infra()
- pkg.set_license()
- pkg.set_hash_info()
- pkg.set_patch_count()
- pkg.set_check_package_warnings()
- pkg.set_current_version()
- print "Calculate stats"
- stats = calculate_stats(packages)
- print "Write HTML"
- dump_html(packages, stats, args.output)
-
+ if args.cpe_report:
+ print "Performing Target CPE Report Analysis..."
+ cpe_dict = CPE()
+ cpe_dict.get_xml_dict()
+ get_target_cpe_report(args.cpe_report,cpe_dict)
+ else:
+ print "Build package list ..."
+ packages = get_pkglist(args.npackages, package_list)
+ print "Getting package make info ..."
+ package_init_make_info()
+ print "Getting package details ..."
+ for pkg in packages:
+ pkg.set_infra()
+ pkg.set_license()
+ pkg.set_hash_info()
+ pkg.set_patch_count()
+ pkg.set_check_package_warnings()
+ pkg.set_current_version()
+ print "Calculate stats"
+ stats = calculate_stats(packages)
+ print "Write HTML"
+ dump_html(packages, stats, args.output)
__main__()
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [Buildroot] [PATCH v3 5/6] support/scripts/pkgstat: add target CPE reporting
2018-05-07 20:30 ` [Buildroot] [PATCH v3 5/6] support/scripts/pkgstat: add target CPE reporting Matt Weber
@ 2018-05-09 21:00 ` Erik Larsson
2018-05-09 21:36 ` Matthew Weber
2018-05-10 3:03 ` Ricardo Martincoski
1 sibling, 1 reply; 14+ messages in thread
From: Erik Larsson @ 2018-05-09 21:00 UTC (permalink / raw)
To: buildroot
Hi!
2018-05-07 22:30 GMT+02:00 Matt Weber <matthew.weber@rockwellcollins.com>:
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> ---
> support/scripts/pkg-stats | 103 ++++++++++++++++++++++++++++++++++++++--------
> 1 file changed, 86 insertions(+), 17 deletions(-)
>
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 43f7e8d..144c00c 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -24,9 +24,15 @@ from collections import defaultdict
> import re
> import subprocess
> import sys
> +import urllib2
> +import xmltodict
> +import gzip
> +from StringIO import StringIO
> +import csv
>
> INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
>
> +CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
>
> class Package:
> all_licenses = list()
> @@ -484,6 +490,62 @@ def dump_html(packages, stats, output):
> dump_gen_info(f)
> f.write(html_footer)
>
> +class CPE:
> + all_cpes = dict()
> +
> + def get_xml_dict(self):
> + print "CPE: Fetching xml manifest..."
> + try:
> + compressed_cpe_file = urllib2.urlopen(CPE_XML_URL)
> + print "CPE: Unzipping xml manifest..."
> + cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read()
> + print "CPE: Converting xml manifest to dict..."
> + self.all_cpes = xmltodict.parse(cpe_file)
> + except urllib2.HTTPError, e:
> + print "HTTP Error:", e.code, url
> + except urllib2.URLError, e:
> + print "URL Error:", e.reason, url
> +
> + def find_partial(self,cpe_str):
> + print "CPE: Searching for partial [%s]" % cpe_str
> + for cpe in self.all_cpes['cpe-list']['cpe-item']:
> + if cpe_str in cpe['cpe-23:cpe23-item']['@name']:
> + return cpe['cpe-23:cpe23-item']['@name']
> +
> + def find(self,cpe_str):
> + print "CPE: Searching for [%s]" % cpe_str
> + for cpe in self.all_cpes['cpe-list']['cpe-item']:
> + if cpe['cpe-23:cpe23-item']['@name'] == cpe_str:
> + return cpe['cpe-23:cpe23-item']['@name']
> +
> +def get_target_cpe_report(cpe_report_file, cpe_dict):
> + report_cpe_exact_match = ""
> + report_cpe_needing_update = ""
> + report_cpe_missing = ""
> +
> + print "CPE: Checking for matches..."
> + with open(cpe_report_file) as cpe_file:
"with open" will cause an exception if the file does not exist and
Python will print an ugly Traceback before stopping.
Maybe you can add a simple test if the file exists. The reason I found
it was because I made a simple typo when testing it :)
> + cpe_list = csv.reader(cpe_file)
> + for cpe in cpe_list:
> + if "CPE ID" not in cpe[0]:
> + result = cpe_dict.find(cpe[0])
> + if not result:
> + cpe_no_version = cpe[0].split(":")[0]+":"+cpe[0].split(":")[1]+":"+cpe[0].split(":")[2]+":"+cpe[0].split(":")[3]+":"+cpe[0].split(":")[4]
> + result = cpe_dict.find_partial(cpe_no_version)
> + if not result:
> + report_cpe_missing += cpe[0] + "\n"
> + else:
> + report_cpe_needing_update += cpe[0] + "\n"
> + else:
> + report_cpe_exact_match += cpe[0] + "\n"
> + print "CPE: Found EXACT match:"
> + print report_cpe_exact_match
> + print "CPE: Found but REQUIRES UPDATE:"
> + print report_cpe_needing_update
> + print "CPE: Not found (proposing the following to be added):"
> + print report_cpe_missing
> +
> +
>
> def parse_args():
> parser = argparse.ArgumentParser()
> @@ -493,6 +555,8 @@ def parse_args():
> help='Number of packages')
> parser.add_argument('-p', dest='packages', action='store',
> help='List of packages (comma separated)')
> + parser.add_argument('-c', dest='cpe_report', action='store',
> + help='Buildroot CPE Report (csv format)')
> return parser.parse_args()
>
>
> @@ -505,22 +569,27 @@ def __main__():
> package_list = args.packages.split(",")
> else:
> package_list = None
> - print "Build package list ..."
> - packages = get_pkglist(args.npackages, package_list)
> - print "Getting package make info ..."
> - package_init_make_info()
> - print "Getting package details ..."
> - for pkg in packages:
> - pkg.set_infra()
> - pkg.set_license()
> - pkg.set_hash_info()
> - pkg.set_patch_count()
> - pkg.set_check_package_warnings()
> - pkg.set_current_version()
> - print "Calculate stats"
> - stats = calculate_stats(packages)
> - print "Write HTML"
> - dump_html(packages, stats, args.output)
> -
> + if args.cpe_report:
> + print "Performing Target CPE Report Analysis..."
> + cpe_dict = CPE()
> + cpe_dict.get_xml_dict()
> + get_target_cpe_report(args.cpe_report,cpe_dict)
> + else:
> + print "Build package list ..."
> + packages = get_pkglist(args.npackages, package_list)
> + print "Getting package make info ..."
> + package_init_make_info()
> + print "Getting package details ..."
> + for pkg in packages:
> + pkg.set_infra()
> + pkg.set_license()
> + pkg.set_hash_info()
> + pkg.set_patch_count()
> + pkg.set_check_package_warnings()
> + pkg.set_current_version()
> + print "Calculate stats"
> + stats = calculate_stats(packages)
> + print "Write HTML"
> + dump_html(packages, stats, args.output)
>
> __main__()
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
Best regards,
Erik
^ permalink raw reply [flat|nested] 14+ messages in thread* [Buildroot] [PATCH v3 5/6] support/scripts/pkgstat: add target CPE reporting
2018-05-09 21:00 ` Erik Larsson
@ 2018-05-09 21:36 ` Matthew Weber
0 siblings, 0 replies; 14+ messages in thread
From: Matthew Weber @ 2018-05-09 21:36 UTC (permalink / raw)
To: buildroot
Erik,
On Wed, May 9, 2018 at 4:00 PM, Erik Larsson
<karl.erik.larsson@gmail.com> wrote:
> Hi!
>
> 2018-05-07 22:30 GMT+02:00 Matt Weber <matthew.weber@rockwellcollins.com>:
>> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
>> ---
>> support/scripts/pkg-stats | 103 ++++++++++++++++++++++++++++++++++++++--------
>> 1 file changed, 86 insertions(+), 17 deletions(-)
>>
[snip]
>> +def get_target_cpe_report(cpe_report_file, cpe_dict):
>> + report_cpe_exact_match = ""
>> + report_cpe_needing_update = ""
>> + report_cpe_missing = ""
>> +
>> + print "CPE: Checking for matches..."
>> + with open(cpe_report_file) as cpe_file:
>
> "with open" will cause an exception if the file does not exist and
> Python will print an ugly Traceback before stopping.
> Maybe you can add a simple test if the file exists. The reason I found
> it was because I made a simple typo when testing it :)
>
Noted for next rev of this series. Any other feedback on the behavior
of a target report? (csv content, pkgstat output, etc?)
Matt
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH v3 5/6] support/scripts/pkgstat: add target CPE reporting
2018-05-07 20:30 ` [Buildroot] [PATCH v3 5/6] support/scripts/pkgstat: add target CPE reporting Matt Weber
2018-05-09 21:00 ` Erik Larsson
@ 2018-05-10 3:03 ` Ricardo Martincoski
2018-05-10 18:00 ` Matthew Weber
1 sibling, 1 reply; 14+ messages in thread
From: Ricardo Martincoski @ 2018-05-10 3:03 UTC (permalink / raw)
To: buildroot
Hello,
This is not a complete code-review, but it seems you will send a new iteration.
When sending new iteration(s) of this series, could you run flake8 on patches 5
and 6 and fix all warnings?
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/67435667
On Mon, May 07, 2018 at 05:30 PM, Matt Weber wrote:
> support/scripts/pkgstat: add target CPE reporting
For this patch and also for patch 6, one of these prefixes would be better:
support/scripts/pkg-stats: ...
pkg-stats: ...
[snip]
> + def get_xml_dict(self):
> + print "CPE: Fetching xml manifest..."
> + try:
> + compressed_cpe_file = urllib2.urlopen(CPE_XML_URL)
> + print "CPE: Unzipping xml manifest..."
> + cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read()
> + print "CPE: Converting xml manifest to dict..."
> + self.all_cpes = xmltodict.parse(cpe_file)
> + except urllib2.HTTPError, e:
> + print "HTTP Error:", e.code, url
> + except urllib2.URLError, e:
> + print "URL Error:", e.reason, url
(after fixing the warning from flake8)
If the download fails the script continues and dies like this:
CPE: Fetching xml manifest...
URL Error: [Errno -2] Name or service not known
CPE: Checking for matches...
CPE: Searching for [cpe:2.3:a:gnu:bash:4.4.18:*:*:*:*:*:*:*]
Traceback (most recent call last):
File "support/scripts/pkg-stats", line 595, in <module>
__main__()
File "support/scripts/pkg-stats", line 576, in __main__
get_target_cpe_report(args.cpe_report,cpe_dict)
File "support/scripts/pkg-stats", line 531, in get_target_cpe_report
result = cpe_dict.find(cpe[0])
File "support/scripts/pkg-stats", line 517, in find
for cpe in self.all_cpes['cpe-list']['cpe-item']:
KeyError: 'cpe-list'
Maybe this is what you want:
except ...
print ...
sys.exit(1)
[snip]
> +def get_target_cpe_report(cpe_report_file, cpe_dict):
> + report_cpe_exact_match = ""
> + report_cpe_needing_update = ""
> + report_cpe_missing = ""
> +
> + print "CPE: Checking for matches..."
> + with open(cpe_report_file) as cpe_file:
> + cpe_list = csv.reader(cpe_file)
> + for cpe in cpe_list:
> + if "CPE ID" not in cpe[0]:
The .csv file contains always only one header, right?
If so, you could do like it is done on utils/size-stats-compare (code not
tested):
cpe_list = csv.reader(cpe_file)
header = next(cpe_list)
(... code to validate the header ...)
for cpe in cpe_list:
(... assume it is an entry, not a header...)
> + result = cpe_dict.find(cpe[0])
> + if not result:
> + cpe_no_version = cpe[0].split(":")[0]+":"+cpe[0].split(":")[1]+":"+cpe[0].split(":")[2]+":"+cpe[0].split(":")[3]+":"+cpe[0].split(":")[4]
Could this long line be replaced by this one?
cpe_no_version = ":".join(cpe[0].split(":", 5)[:5])
[snip]
> def parse_args():
> parser = argparse.ArgumentParser()
> @@ -493,6 +555,8 @@ def parse_args():
> help='Number of packages')
> parser.add_argument('-p', dest='packages', action='store',
> help='List of packages (comma separated)')
> + parser.add_argument('-c', dest='cpe_report', action='store',
> + help='Buildroot CPE Report (csv format)')
What do you think about this help instead:
help='CPE Report generated by make cpe-info (csv format)')
or something like this?
> return parser.parse_args()
>
>
> @@ -505,22 +569,27 @@ def __main__():
[snip]
> + if args.cpe_report:
> + print "Performing Target CPE Report Analysis..."
> + cpe_dict = CPE()
> + cpe_dict.get_xml_dict()
> + get_target_cpe_report(args.cpe_report,cpe_dict)
Is this code supposed to be used only to debug the script?
Or is its output to stdout supposed to used by the user to get a list of
packages without cpe info?
I ask this because this code branch still requires a '-o file' to be passed,
but it does not use it.
For the first case (only to debug the script) maybe it is acceptable as is.
But for the second case (script to be called by the end user), it is not nice to
require an argument that is not used. required=True could then be removed from
"parser.add_argument('-o'," and the check could be moved to __main__ (code not
tested):
def __main__():
args = parse_args()
if args.output is None and args.cpe_report is None:
print "ERROR: either argument -o or -c required"
sys.exit(1)
if args.output and args.cpe_report:
print "ERROR: -o and -c are mutually exclusive"
sys.exit(1)
Regards,
Ricardo
^ permalink raw reply [flat|nested] 14+ messages in thread* [Buildroot] [PATCH v3 5/6] support/scripts/pkgstat: add target CPE reporting
2018-05-10 3:03 ` Ricardo Martincoski
@ 2018-05-10 18:00 ` Matthew Weber
2018-05-14 21:43 ` Matthew Weber
0 siblings, 1 reply; 14+ messages in thread
From: Matthew Weber @ 2018-05-10 18:00 UTC (permalink / raw)
To: buildroot
Ricardo,
On Wed, May 9, 2018 at 10:03 PM, Ricardo Martincoski
<ricardo.martincoski@gmail.com> wrote:
> Hello,
>
> This is not a complete code-review, but it seems you will send a new iteration.
>
> When sending new iteration(s) of this series, could you run flake8 on patches 5
> and 6 and fix all warnings?
> https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/67435667
>
>
> On Mon, May 07, 2018 at 05:30 PM, Matt Weber wrote:
>
>> support/scripts/pkgstat: add target CPE reporting
>
> For this patch and also for patch 6, one of these prefixes would be better:
> support/scripts/pkg-stats: ...
> pkg-stats: ...
Noted, I'm actually just going to also collapse the patch 5 and 6
together (easier to manage with flake8/pep8 updates).
>
> [snip]
>> + def get_xml_dict(self):
>> + print "CPE: Fetching xml manifest..."
>> + try:
>> + compressed_cpe_file = urllib2.urlopen(CPE_XML_URL)
>> + print "CPE: Unzipping xml manifest..."
>> + cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read()
>> + print "CPE: Converting xml manifest to dict..."
>> + self.all_cpes = xmltodict.parse(cpe_file)
>> + except urllib2.HTTPError, e:
>> + print "HTTP Error:", e.code, url
>> + except urllib2.URLError, e:
>> + print "URL Error:", e.reason, url
>
> (after fixing the warning from flake8)
> If the download fails the script continues and dies like this:
>
> CPE: Fetching xml manifest...
> URL Error: [Errno -2] Name or service not known
> CPE: Checking for matches...
> CPE: Searching for [cpe:2.3:a:gnu:bash:4.4.18:*:*:*:*:*:*:*]
> Traceback (most recent call last):
> File "support/scripts/pkg-stats", line 595, in <module>
> __main__()
> File "support/scripts/pkg-stats", line 576, in __main__
> get_target_cpe_report(args.cpe_report,cpe_dict)
> File "support/scripts/pkg-stats", line 531, in get_target_cpe_report
> result = cpe_dict.find(cpe[0])
> File "support/scripts/pkg-stats", line 517, in find
> for cpe in self.all_cpes['cpe-list']['cpe-item']:
> KeyError: 'cpe-list'
>
> Maybe this is what you want:
> except ...
> print ...
> sys.exit(1)
Fixed by adding exit
>
> [snip]
>> +def get_target_cpe_report(cpe_report_file, cpe_dict):
>> + report_cpe_exact_match = ""
>> + report_cpe_needing_update = ""
>> + report_cpe_missing = ""
>> +
>> + print "CPE: Checking for matches..."
>> + with open(cpe_report_file) as cpe_file:
>> + cpe_list = csv.reader(cpe_file)
>> + for cpe in cpe_list:
>> + if "CPE ID" not in cpe[0]:
>
> The .csv file contains always only one header, right?
> If so, you could do like it is done on utils/size-stats-compare (code not
> tested):
> cpe_list = csv.reader(cpe_file)
> header = next(cpe_list)
> (... code to validate the header ...)
> for cpe in cpe_list:
> (... assume it is an entry, not a header...)
>
Yeah, that works and is cleaner.
>> + result = cpe_dict.find(cpe[0])
>> + if not result:
>> + cpe_no_version = cpe[0].split(":")[0]+":"+cpe[0].split(":")[1]+":"+cpe[0].split(":")[2]+":"+cpe[0].split(":")[3]+":"+cpe[0].split(":")[4]
>
> Could this long line be replaced by this one?
> cpe_no_version = ":".join(cpe[0].split(":", 5)[:5])
>
Yep. "".join(cpe.split(":")[:5])
> [snip]
>> def parse_args():
>> parser = argparse.ArgumentParser()
>> @@ -493,6 +555,8 @@ def parse_args():
>> help='Number of packages')
>> parser.add_argument('-p', dest='packages', action='store',
>> help='List of packages (comma separated)')
>> + parser.add_argument('-c', dest='cpe_report', action='store',
>> + help='Buildroot CPE Report (csv format)')
>
> What do you think about this help instead:
> help='CPE Report generated by make cpe-info (csv format)')
Sounds good.
> [snip]
>> + if args.cpe_report:
>> + print "Performing Target CPE Report Analysis..."
>> + cpe_dict = CPE()
>> + cpe_dict.get_xml_dict()
>> + get_target_cpe_report(args.cpe_report,cpe_dict)
>
> Is this code supposed to be used only to debug the script?
Debug. However we're working on a follow on patch that will add an
option to generate the xml files used to update a CPE entry. When
that's added we may optional this output unless a xml output file is
provided.
> I ask this because this code branch still requires a '-o file' to be passed,
> but it does not use it.
>
> For the first case (only to debug the script) maybe it is acceptable as is.
>
I will remove the -o and instead do something like the following?
elif args.output:
print "Build package list ..."
packages = get_pkglist(args.npackages, package_list)
print "Getting package make info ..."
package_init_make_info()
print "Getting package details ..."
for pkg in packages:
pkg.set_infra()
pkg.set_license()
pkg.set_hash_info()
pkg.set_patch_count()
pkg.set_check_package_warnings()
pkg.set_current_version()
pkg.set_cpe_info(cpe_dict)
print "Calculate stats"
stats = calculate_stats(packages)
print "Write HTML"
dump_html(packages, stats, args.output)
else:
print "Please provide the -o HTML output file arg"
Matt
^ permalink raw reply [flat|nested] 14+ messages in thread* [Buildroot] [PATCH v3 5/6] support/scripts/pkgstat: add target CPE reporting
2018-05-10 18:00 ` Matthew Weber
@ 2018-05-14 21:43 ` Matthew Weber
0 siblings, 0 replies; 14+ messages in thread
From: Matthew Weber @ 2018-05-14 21:43 UTC (permalink / raw)
To: buildroot
Ricardo,
On Thu, May 10, 2018 at 1:00 PM, Matthew Weber
<matthew.weber@rockwellcollins.com> wrote:
>
> Ricardo,
>
> On Wed, May 9, 2018 at 10:03 PM, Ricardo Martincoski
> <ricardo.martincoski@gmail.com> wrote:
> > Hello,
> >
> > This is not a complete code-review, but it seems you will send a new iteration.
> >
Here's v5
http://patchwork.ozlabs.org/project/buildroot/list/?series=43718
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH v3 6/6] support/scripts/pkgstat: add CPE status reporting
2018-05-07 20:29 [Buildroot] [PATCH v3 0/6] CPE ID Support Matt Weber
` (4 preceding siblings ...)
2018-05-07 20:30 ` [Buildroot] [PATCH v3 5/6] support/scripts/pkgstat: add target CPE reporting Matt Weber
@ 2018-05-07 20:30 ` Matt Weber
2018-05-09 21:03 ` Erik Larsson
5 siblings, 1 reply; 14+ messages in thread
From: Matt Weber @ 2018-05-07 20:30 UTC (permalink / raw)
To: buildroot
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
support/scripts/pkg-stats | 80 ++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 76 insertions(+), 4 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 144c00c..82057f1 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -38,6 +38,7 @@ class Package:
all_licenses = list()
all_license_files = list()
all_versions = dict()
+ all_cpe_id = dict()
def __init__(self, name, path):
self.name = name
@@ -49,6 +50,8 @@ class Package:
self.patch_count = 0
self.warnings = 0
self.current_version = None
+ self.cpe_id = None
+ self.has_cpe = False
def pkgvar(self):
return self.name.upper().replace("-", "_")
@@ -122,6 +125,22 @@ class Package:
self.warnings = int(m.group(1))
return
+ def set_cpe_info(self, cpe_dict):
+ """
+ Fills in the .has_cpe field
+ """
+ var = self.pkgvar()
+ if var in self.all_cpe_id:
+ self.cpe_id = self.all_cpe_id[var]
+ result = cpe_dict.find(self.cpe_id)
+ if not result:
+ result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(self.cpe_id))
+ if result:
+ self.has_cpe = "Update"
+ # Unset case for has_cpe is assumed missing/does not exist
+ else:
+ self.has_cpe = cpe_dict.get_nvd_url(self.cpe_id)
+
def __eq__(self, other):
return self.path == other.path
@@ -260,6 +279,22 @@ def package_init_make_info():
Package.all_versions[pkgvar] = value
+ # CPE ID
+ o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
+ "-s", "printvars", "VARS=%_CPE_ID"])
+ for l in o.splitlines():
+ # Get variable name and value
+ pkgvar, value = l.split("=")
+
+ # Strip _CPE_ID
+ pkgvar = pkgvar[:-7]
+
+ if pkgvar == "LINUX":
+ Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*"
+ elif pkgvar == "LINUX_HEADERS":
+ Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*"
+ else:
+ Package.all_cpe_id[pkgvar] = "cpe:2.3:a:" + value + ":*:*:*:*:*:*:*"
def calculate_stats(packages):
stats = defaultdict(int)
@@ -285,6 +320,12 @@ def calculate_stats(packages):
stats["hash"] += 1
else:
stats["no-hash"] += 1
+ if pkg.has_cpe == "Update":
+ stats["update-cpe"] += 1
+ elif pkg.has_cpe:
+ stats["cpe"] += 1
+ else:
+ stats["no-cpe"] += 1
stats["patches"] += pkg.patch_count
return stats
@@ -428,6 +469,20 @@ def dump_html_pkg(f, pkg):
f.write(" <td class=\"%s\">%d</td>\n" %
(" ".join(td_class), pkg.warnings))
+ # CPE Valid
+ td_class = ["centered"]
+ if not pkg.has_cpe:
+ td_class.append("wrong")
+ f.write(" <td class=\"%s\">%s</td>\n" %
+ (" ".join(td_class), boolean_str(pkg.has_cpe)))
+ elif pkg.has_cpe == "Update":
+ td_class.append("wrong")
+ f.write(" <td class=\"%s\">Update</td>\n" %
+ (" ".join(td_class)))
+ else:
+ td_class.append("correct")
+ f.write(" <td class=\"%s\"><a href=\"%s\">%s</a></td>\n" %
+ (" ".join(td_class), pkg.has_cpe, boolean_str(pkg.has_cpe)))
f.write(" </tr>\n")
@@ -443,6 +498,7 @@ def dump_html_all_pkgs(f, packages):
<td class=\"centered\">Hash file</td>
<td class=\"centered\">Current version</td>
<td class=\"centered\">Warnings</td>
+<td class=\"centered\">CPE Valid</td>
</tr>
""")
for pkg in sorted(packages):
@@ -469,6 +525,12 @@ def dump_html_stats(f, stats):
stats["hash"])
f.write(" <tr><td>Packages not having a hash file</td><td>%s</td></tr>\n" %
stats["no-hash"])
+ f.write(" <tr><td>Packages having a registered CPE</td><td>%s</td></tr>\n" %
+ stats["cpe"])
+ f.write(" <tr><td>Packages needing CPE update</td><td>%s</td></tr>\n" %
+ stats["update-cpe"])
+ f.write(" <tr><td>Packages missing a registered CPE</td><td>%s</td></tr>\n" %
+ stats["no-cpe"])
f.write(" <tr><td>Total number of patches</td><td>%s</td></tr>\n" %
stats["patches"])
f.write("</table>\n")
@@ -518,6 +580,16 @@ class CPE:
if cpe['cpe-23:cpe23-item']['@name'] == cpe_str:
return cpe['cpe-23:cpe23-item']['@name']
+ def get_cpe_no_version(self, cpe):
+ return cpe.split(":")[0]+":"+cpe.split(":")[1]+ \
+ ":"+cpe.split(":")[2]+":"+cpe.split(":")[3]+ \
+ ":"+cpe.split(":")[4]
+
+ def get_nvd_url(self, cpe_str):
+ return "https://nvd.nist.gov/products/cpe/search/results?keyword=" + \
+ urllib2.quote(cpe_str) + \
+ "&status=FINAL&orderBy=CPEURI&namingFormat=2.3"
+
def get_target_cpe_report(cpe_report_file, cpe_dict):
report_cpe_exact_match = ""
report_cpe_needing_update = ""
@@ -530,8 +602,7 @@ def get_target_cpe_report(cpe_report_file, cpe_dict):
if "CPE ID" not in cpe[0]:
result = cpe_dict.find(cpe[0])
if not result:
- cpe_no_version = cpe[0].split(":")[0]+":"+cpe[0].split(":")[1]+":"+cpe[0].split(":")[2]+":"+cpe[0].split(":")[3]+":"+cpe[0].split(":")[4]
- result = cpe_dict.find_partial(cpe_no_version)
+ result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(cpe[0]))
if not result:
report_cpe_missing += cpe[0] + "\n"
else:
@@ -569,10 +640,10 @@ def __main__():
package_list = args.packages.split(",")
else:
package_list = None
+ cpe_dict = CPE()
+ cpe_dict.get_xml_dict()
if args.cpe_report:
print "Performing Target CPE Report Analysis..."
- cpe_dict = CPE()
- cpe_dict.get_xml_dict()
get_target_cpe_report(args.cpe_report,cpe_dict)
else:
print "Build package list ..."
@@ -587,6 +658,7 @@ def __main__():
pkg.set_patch_count()
pkg.set_check_package_warnings()
pkg.set_current_version()
+ pkg.set_cpe_info(cpe_dict)
print "Calculate stats"
stats = calculate_stats(packages)
print "Write HTML"
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [Buildroot] [PATCH v3 6/6] support/scripts/pkgstat: add CPE status reporting
2018-05-07 20:30 ` [Buildroot] [PATCH v3 6/6] support/scripts/pkgstat: add CPE status reporting Matt Weber
@ 2018-05-09 21:03 ` Erik Larsson
2018-05-09 21:33 ` Matthew Weber
0 siblings, 1 reply; 14+ messages in thread
From: Erik Larsson @ 2018-05-09 21:03 UTC (permalink / raw)
To: buildroot
Hi!
2018-05-07 22:30 GMT+02:00 Matt Weber <matthew.weber@rockwellcollins.com>:
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> ---
> support/scripts/pkg-stats | 80 ++++++++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 76 insertions(+), 4 deletions(-)
>
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 144c00c..82057f1 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -38,6 +38,7 @@ class Package:
> all_licenses = list()
> all_license_files = list()
> all_versions = dict()
> + all_cpe_id = dict()
>
> def __init__(self, name, path):
> self.name = name
> @@ -49,6 +50,8 @@ class Package:
> self.patch_count = 0
> self.warnings = 0
> self.current_version = None
> + self.cpe_id = None
> + self.has_cpe = False
>
> def pkgvar(self):
> return self.name.upper().replace("-", "_")
> @@ -122,6 +125,22 @@ class Package:
> self.warnings = int(m.group(1))
> return
>
> + def set_cpe_info(self, cpe_dict):
> + """
> + Fills in the .has_cpe field
> + """
> + var = self.pkgvar()
> + if var in self.all_cpe_id:
> + self.cpe_id = self.all_cpe_id[var]
> + result = cpe_dict.find(self.cpe_id)
> + if not result:
> + result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(self.cpe_id))
> + if result:
> + self.has_cpe = "Update"
> + # Unset case for has_cpe is assumed missing/does not exist
> + else:
> + self.has_cpe = cpe_dict.get_nvd_url(self.cpe_id)
> +
> def __eq__(self, other):
> return self.path == other.path
>
> @@ -260,6 +279,22 @@ def package_init_make_info():
>
> Package.all_versions[pkgvar] = value
>
> + # CPE ID
> + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
> + "-s", "printvars", "VARS=%_CPE_ID"])
> + for l in o.splitlines():
> + # Get variable name and value
> + pkgvar, value = l.split("=")
> +
> + # Strip _CPE_ID
> + pkgvar = pkgvar[:-7]
> +
> + if pkgvar == "LINUX":
> + Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*"
> + elif pkgvar == "LINUX_HEADERS":
> + Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*"
> + else:
> + Package.all_cpe_id[pkgvar] = "cpe:2.3:a:" + value + ":*:*:*:*:*:*:*"
>
> def calculate_stats(packages):
> stats = defaultdict(int)
> @@ -285,6 +320,12 @@ def calculate_stats(packages):
> stats["hash"] += 1
> else:
> stats["no-hash"] += 1
> + if pkg.has_cpe == "Update":
> + stats["update-cpe"] += 1
> + elif pkg.has_cpe:
> + stats["cpe"] += 1
> + else:
> + stats["no-cpe"] += 1
> stats["patches"] += pkg.patch_count
> return stats
>
> @@ -428,6 +469,20 @@ def dump_html_pkg(f, pkg):
> f.write(" <td class=\"%s\">%d</td>\n" %
> (" ".join(td_class), pkg.warnings))
>
> + # CPE Valid
> + td_class = ["centered"]
> + if not pkg.has_cpe:
> + td_class.append("wrong")
> + f.write(" <td class=\"%s\">%s</td>\n" %
> + (" ".join(td_class), boolean_str(pkg.has_cpe)))
> + elif pkg.has_cpe == "Update":
> + td_class.append("wrong")
> + f.write(" <td class=\"%s\">Update</td>\n" %
> + (" ".join(td_class)))
> + else:
> + td_class.append("correct")
> + f.write(" <td class=\"%s\"><a href=\"%s\">%s</a></td>\n" %
> + (" ".join(td_class), pkg.has_cpe, boolean_str(pkg.has_cpe)))
> f.write(" </tr>\n")
>
>
> @@ -443,6 +498,7 @@ def dump_html_all_pkgs(f, packages):
> <td class=\"centered\">Hash file</td>
> <td class=\"centered\">Current version</td>
> <td class=\"centered\">Warnings</td>
> +<td class=\"centered\">CPE Valid</td>
> </tr>
> """)
> for pkg in sorted(packages):
> @@ -469,6 +525,12 @@ def dump_html_stats(f, stats):
> stats["hash"])
> f.write(" <tr><td>Packages not having a hash file</td><td>%s</td></tr>\n" %
> stats["no-hash"])
> + f.write(" <tr><td>Packages having a registered CPE</td><td>%s</td></tr>\n" %
> + stats["cpe"])
> + f.write(" <tr><td>Packages needing CPE update</td><td>%s</td></tr>\n" %
> + stats["update-cpe"])
> + f.write(" <tr><td>Packages missing a registered CPE</td><td>%s</td></tr>\n" %
> + stats["no-cpe"])
> f.write(" <tr><td>Total number of patches</td><td>%s</td></tr>\n" %
> stats["patches"])
> f.write("</table>\n")
> @@ -518,6 +580,16 @@ class CPE:
> if cpe['cpe-23:cpe23-item']['@name'] == cpe_str:
> return cpe['cpe-23:cpe23-item']['@name']
>
> + def get_cpe_no_version(self, cpe):
> + return cpe.split(":")[0]+":"+cpe.split(":")[1]+ \
> + ":"+cpe.split(":")[2]+":"+cpe.split(":")[3]+ \
> + ":"+cpe.split(":")[4]
I tested this script by running:
# ./support/scripts/pkg-stats -o /tmp/test.html
The result I got was a crash. Maybe you understand it better then me.
CPE: Searching for partial [cpe:2.3:a:copas_project:copas]
CPE: Searching for [None]
Traceback (most recent call last):
File "./support/scripts/pkg-stats", line 667, in <module>
__main__()
File "./support/scripts/pkg-stats", line 661, in __main__
pkg.set_cpe_info(cpe_dict)
File "./support/scripts/pkg-stats", line 137, in set_cpe_info
result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(self.cpe_id))
File "./support/scripts/pkg-stats", line 586, in get_cpe_no_version
":"+cpe.split(":")[4]
AttributeError: 'NoneType' object has no attribute 'split'
> +
> + def get_nvd_url(self, cpe_str):
> + return "https://nvd.nist.gov/products/cpe/search/results?keyword=" + \
> + urllib2.quote(cpe_str) + \
> + "&status=FINAL&orderBy=CPEURI&namingFormat=2.3"
> +
> def get_target_cpe_report(cpe_report_file, cpe_dict):
> report_cpe_exact_match = ""
> report_cpe_needing_update = ""
> @@ -530,8 +602,7 @@ def get_target_cpe_report(cpe_report_file, cpe_dict):
> if "CPE ID" not in cpe[0]:
> result = cpe_dict.find(cpe[0])
> if not result:
> - cpe_no_version = cpe[0].split(":")[0]+":"+cpe[0].split(":")[1]+":"+cpe[0].split(":")[2]+":"+cpe[0].split(":")[3]+":"+cpe[0].split(":")[4]
> - result = cpe_dict.find_partial(cpe_no_version)
> + result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(cpe[0]))
> if not result:
> report_cpe_missing += cpe[0] + "\n"
> else:
> @@ -569,10 +640,10 @@ def __main__():
> package_list = args.packages.split(",")
> else:
> package_list = None
> + cpe_dict = CPE()
> + cpe_dict.get_xml_dict()
> if args.cpe_report:
> print "Performing Target CPE Report Analysis..."
> - cpe_dict = CPE()
> - cpe_dict.get_xml_dict()
> get_target_cpe_report(args.cpe_report,cpe_dict)
> else:
> print "Build package list ..."
> @@ -587,6 +658,7 @@ def __main__():
> pkg.set_patch_count()
> pkg.set_check_package_warnings()
> pkg.set_current_version()
> + pkg.set_cpe_info(cpe_dict)
> print "Calculate stats"
> stats = calculate_stats(packages)
> print "Write HTML"
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
Best regards,
Erik
^ permalink raw reply [flat|nested] 14+ messages in thread* [Buildroot] [PATCH v3 6/6] support/scripts/pkgstat: add CPE status reporting
2018-05-09 21:03 ` Erik Larsson
@ 2018-05-09 21:33 ` Matthew Weber
0 siblings, 0 replies; 14+ messages in thread
From: Matthew Weber @ 2018-05-09 21:33 UTC (permalink / raw)
To: buildroot
Erik,
On Wed, May 9, 2018 at 4:03 PM, Erik Larsson
<karl.erik.larsson@gmail.com> wrote:
> Hi!
>
> 2018-05-07 22:30 GMT+02:00 Matt Weber <matthew.weber@rockwellcollins.com>:
>> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
>> ---
>> support/scripts/pkg-stats | 80 ++++++++++++++++++++++++++++++++++++++++++++---
>> 1 file changed, 76 insertions(+), 4 deletions(-)
>>
>> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
>> index 144c00c..82057f1 100755
>> --- a/support/scripts/pkg-stats
>> +++ b/support/scripts/pkg-stats
>> @@ -38,6 +38,7 @@ class Package:
>> all_licenses = list()
>> all_license_files = list()
>> all_versions = dict()
>> + all_cpe_id = dict()
>>
>> def __init__(self, name, path):
>> self.name = name
>> @@ -49,6 +50,8 @@ class Package:
>> self.patch_count = 0
>> self.warnings = 0
>> self.current_version = None
>> + self.cpe_id = None
>> + self.has_cpe = False
>>
>> def pkgvar(self):
>> return self.name.upper().replace("-", "_")
>> @@ -122,6 +125,22 @@ class Package:
>> self.warnings = int(m.group(1))
>> return
>>
>> + def set_cpe_info(self, cpe_dict):
>> + """
>> + Fills in the .has_cpe field
>> + """
>> + var = self.pkgvar()
>> + if var in self.all_cpe_id:
>> + self.cpe_id = self.all_cpe_id[var]
>> + result = cpe_dict.find(self.cpe_id)
>> + if not result:
>> + result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(self.cpe_id))
>> + if result:
>> + self.has_cpe = "Update"
>> + # Unset case for has_cpe is assumed missing/does not exist
>> + else:
>> + self.has_cpe = cpe_dict.get_nvd_url(self.cpe_id)
>> +
>> def __eq__(self, other):
>> return self.path == other.path
>>
>> @@ -260,6 +279,22 @@ def package_init_make_info():
>>
>> Package.all_versions[pkgvar] = value
>>
>> + # CPE ID
>> + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
>> + "-s", "printvars", "VARS=%_CPE_ID"])
>> + for l in o.splitlines():
>> + # Get variable name and value
>> + pkgvar, value = l.split("=")
>> +
>> + # Strip _CPE_ID
>> + pkgvar = pkgvar[:-7]
>> +
>> + if pkgvar == "LINUX":
>> + Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*"
>> + elif pkgvar == "LINUX_HEADERS":
>> + Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*"
>> + else:
>> + Package.all_cpe_id[pkgvar] = "cpe:2.3:a:" + value + ":*:*:*:*:*:*:*"
>>
>> def calculate_stats(packages):
>> stats = defaultdict(int)
>> @@ -285,6 +320,12 @@ def calculate_stats(packages):
>> stats["hash"] += 1
>> else:
>> stats["no-hash"] += 1
>> + if pkg.has_cpe == "Update":
>> + stats["update-cpe"] += 1
>> + elif pkg.has_cpe:
>> + stats["cpe"] += 1
>> + else:
>> + stats["no-cpe"] += 1
>> stats["patches"] += pkg.patch_count
>> return stats
>>
>> @@ -428,6 +469,20 @@ def dump_html_pkg(f, pkg):
>> f.write(" <td class=\"%s\">%d</td>\n" %
>> (" ".join(td_class), pkg.warnings))
>>
>> + # CPE Valid
>> + td_class = ["centered"]
>> + if not pkg.has_cpe:
>> + td_class.append("wrong")
>> + f.write(" <td class=\"%s\">%s</td>\n" %
>> + (" ".join(td_class), boolean_str(pkg.has_cpe)))
>> + elif pkg.has_cpe == "Update":
>> + td_class.append("wrong")
>> + f.write(" <td class=\"%s\">Update</td>\n" %
>> + (" ".join(td_class)))
>> + else:
>> + td_class.append("correct")
>> + f.write(" <td class=\"%s\"><a href=\"%s\">%s</a></td>\n" %
>> + (" ".join(td_class), pkg.has_cpe, boolean_str(pkg.has_cpe)))
>> f.write(" </tr>\n")
>>
>>
>> @@ -443,6 +498,7 @@ def dump_html_all_pkgs(f, packages):
>> <td class=\"centered\">Hash file</td>
>> <td class=\"centered\">Current version</td>
>> <td class=\"centered\">Warnings</td>
>> +<td class=\"centered\">CPE Valid</td>
>> </tr>
>> """)
>> for pkg in sorted(packages):
>> @@ -469,6 +525,12 @@ def dump_html_stats(f, stats):
>> stats["hash"])
>> f.write(" <tr><td>Packages not having a hash file</td><td>%s</td></tr>\n" %
>> stats["no-hash"])
>> + f.write(" <tr><td>Packages having a registered CPE</td><td>%s</td></tr>\n" %
>> + stats["cpe"])
>> + f.write(" <tr><td>Packages needing CPE update</td><td>%s</td></tr>\n" %
>> + stats["update-cpe"])
>> + f.write(" <tr><td>Packages missing a registered CPE</td><td>%s</td></tr>\n" %
>> + stats["no-cpe"])
>> f.write(" <tr><td>Total number of patches</td><td>%s</td></tr>\n" %
>> stats["patches"])
>> f.write("</table>\n")
>> @@ -518,6 +580,16 @@ class CPE:
>> if cpe['cpe-23:cpe23-item']['@name'] == cpe_str:
>> return cpe['cpe-23:cpe23-item']['@name']
>>
>> + def get_cpe_no_version(self, cpe):
>> + return cpe.split(":")[0]+":"+cpe.split(":")[1]+ \
>> + ":"+cpe.split(":")[2]+":"+cpe.split(":")[3]+ \
>> + ":"+cpe.split(":")[4]
>
> I tested this script by running:
> # ./support/scripts/pkg-stats -o /tmp/test.html
>
> The result I got was a crash. Maybe you understand it better then me.
>
> CPE: Searching for partial [cpe:2.3:a:copas_project:copas]
> CPE: Searching for [None]
> Traceback (most recent call last):
> File "./support/scripts/pkg-stats", line 667, in <module>
> __main__()
> File "./support/scripts/pkg-stats", line 661, in __main__
> pkg.set_cpe_info(cpe_dict)
> File "./support/scripts/pkg-stats", line 137, in set_cpe_info
> result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(self.cpe_id))
> File "./support/scripts/pkg-stats", line 586, in get_cpe_no_version
> ":"+cpe.split(":")[4]
> AttributeError: 'NoneType' object has no attribute 'split'
>
I was running pkgstat with a specific list of packages. I bet when
unbounded that error is produced. I'll retest, thanks for looking at
it!
Matt
^ permalink raw reply [flat|nested] 14+ messages in thread