* [Buildroot] [PATCH v8 0/9] Package CPE Reporting
@ 2019-03-08 22:04 Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 1/9] cpe-info: new make target Matt Weber
` (9 more replies)
0 siblings, 10 replies; 18+ messages in thread
From: Matt Weber @ 2019-03-08 22:04 UTC (permalink / raw)
To: buildroot
This series adds new infrastructure to report a target's set of package
CPE identifiers in a similar way to that of the legal info reporting.
The addition of CPE IDs to packages can be a manual process but there is
a default value which hopefully will match the string. A cpe-report tool
is provided to help manage the accuracy of the CPE strings. Some existing
fix-ups forthe CPE strings can be found in the following commit that
eventually needs to be split up and applied.
https://github.com/rc-matthew-l-weber/buildroot/commit/8ce8d4740b95672d8390799b611a35ea18a543e0
Changes from v7
- The pkg-stats CPE status update has been dropped as it seemed like a
better feature to be integrated at the release-monitoring.org level.
Or could be reopened with a new direction as part of GSoC.
(http://patchwork.ozlabs.org/patch/985550/)
- cpe-report script has been overhauled to provide preformatted xml
for version updates to help accelerate sending those to NIST
- A new security management section was added to the manual with details
on CPE upkeep and manual CVE analysis tools.
Matt Weber (8):
cpe-info: new make target
cpe-info: id prefix/suffix
cpe-info: only report target pkgs
cpe-info: cpe minor version support
cpe-info: update manual for new pkg vars
support/scripts/cpedb.py: new CPE XML helper
support/scripts/cpe-report: new script
docs/manual: new security management section
Shruthi Singh (1):
toolchain/toolchain-ext: glibc cpe-info support
Makefile | 17 +-
docs/manual/adding-packages-generic.txt | 111 ++++++++-----
docs/manual/cpe-reporting.txt | 103 ++++++++++++
docs/manual/manual.txt | 2 +
package/Makefile.in | 4 +
package/pkg-generic.mk | 36 ++++
package/pkg-utils.mk | 8 +
support/scripts/cpe-report | 70 ++++++++
support/scripts/cpedb.py | 185 +++++++++++++++++++++
.../toolchain-external/pkg-toolchain-external.mk | 7 +
10 files changed, 499 insertions(+), 44 deletions(-)
create mode 100644 docs/manual/cpe-reporting.txt
create mode 100755 support/scripts/cpe-report
create mode 100644 support/scripts/cpedb.py
--
1.9.1
^ permalink raw reply [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 1/9] cpe-info: new make target
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
@ 2019-03-08 22:04 ` Matt Weber
2019-04-14 14:49 ` Arnout Vandecappelle
2019-03-08 22:04 ` [Buildroot] [PATCH v8 2/9] cpe-info: id prefix/suffix Matt Weber
` (8 subsequent siblings)
9 siblings, 1 reply; 18+ messages in thread
From: Matt Weber @ 2019-03-08 22:04 UTC (permalink / raw)
To: buildroot
Similar to make legal-info, produce a csv delimited file containing
all selected packages CPE identification.
By default, support the pkg infra defining a set of CPE_ID_* defaults
using the package name for the vendor and name as most CPE IDs seem
to align with that assumption. Plus initially, use the pkg version as
the CPE ID's version field.
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v8
- Removed CVE patched list as this isn't implmented.
v4 -> v7
- No change
v3
[Thomas P
- Merged infra define CPE_ID_* into this patch
- Report all packages vs restricting to just allowing based on if
the VENDOR was set (v2). This now represents Thomas P's original
idea to report everything. At first I felt I should restrict
the reporting to those CPE IDs we had made sure were correct.
Turns out we should have actually let the script handle fixing
the CPEs and just make a complete design of this up front.
[Matt
- Moved to using the _project on all vendors instead of just name
v2
[Thomas P
- Moved comment on conditionals back to this patchset where
the conditional is created vs later
---
Makefile | 17 ++++++++++++++++-
package/pkg-generic.mk | 13 +++++++++++++
package/pkg-utils.mk | 8 ++++++++
3 files changed, 37 insertions(+), 1 deletion(-)
diff --git a/Makefile b/Makefile
index a64196f..cdf2cf9 100644
--- a/Makefile
+++ b/Makefile
@@ -135,7 +135,7 @@ nobuild_targets := source %-source \
clean distclean help show-targets graph-depends \
%-graph-depends %-show-depends %-show-version \
graph-build graph-size list-defconfigs \
- savedefconfig update-defconfig printvars
+ savedefconfig update-defconfig printvars cpe-info %-cpe-info
ifeq ($(MAKECMDGOALS),)
BR_BUILDING = y
else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),)
@@ -219,6 +219,7 @@ LEGAL_MANIFEST_CSV_TARGET = $(LEGAL_INFO_DIR)/manifest.csv
LEGAL_MANIFEST_CSV_HOST = $(LEGAL_INFO_DIR)/host-manifest.csv
LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings
LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
+CPE_MANIFEST_CSV = $(BASE_DIR)/cpe-manifest.csv
BR2_CONFIG = $(CONFIG_DIR)/.config
@@ -852,6 +853,19 @@ legal-info: legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p)-all
mv .legal-info.sha256 legal-info.sha256)
@echo "Legal info produced in $(LEGAL_INFO_DIR)"
+.PHONY: cpe-info-clean
+cpe-info-clean:
+ @rm -f $(CPE_MANIFEST_CSV)
+
+.PHONY: cpe-info-prepare
+cpe-info-prepare:
+ @$(call MESSAGE,"Gathering CPE info")
+ @$(call cpe-manifest,CPE ID,CVE PATCHED,PACKAGE,VERSION,SOURCE SITE)
+
+.PHONY: cpe-info
+cpe-info: cpe-info-clean cpe-info-prepare $(foreach p,$(PACKAGES),$(p)-cpe-info)
+ @echo "CPE info produced in $(CPE_MANIFEST_CSV)"
+
.PHONY: show-targets
show-targets:
@echo $(sort $(PACKAGES)) $(sort $(TARGETS_ROOTFS))
@@ -1124,6 +1138,7 @@ help:
@echo ' source - download all sources needed for offline-build'
@echo ' external-deps - list external packages used'
@echo ' legal-info - generate info about license compliance'
+ @echo ' cpe-info - generate info about security CPE identification'
@echo ' printvars - dump all the internal variables'
@echo
@echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build'
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 4353bd3..644128d 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -911,6 +911,18 @@ else
$(2)_KCONFIG_VAR = BR2_PACKAGE_$(2)
endif
+$(2)_CPE_ID_VENDOR ?= $$($(2)_NAME)_project
+$(2)_CPE_ID_NAME ?= $$($(2)_NAME)
+$(2)_CPE_ID_VERSION ?= $$($(2)_VERSION)
+$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION)
+
+$(1)-cpe-info: PKG=$(2)
+$(1)-cpe-info:
+ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
+ @$$(call MESSAGE,"Collecting cpe info")
+ $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
+endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
+
# legal-info: declare dependencies and set values used later for the manifest
ifneq ($$($(2)_LICENSE_FILES),)
$(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES)
@@ -1052,6 +1064,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
$(1)-clean-for-reconfigure \
$(1)-clean-for-reinstall \
$(1)-configure \
+ $(1)-cpe-info \
$(1)-depends \
$(1)-dirclean \
$(1)-external-deps \
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index bffd79d..0201632 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -102,3 +102,11 @@ legal-deps = \
$(filter-out $(if $(1:host-%=),host-%),\
$(call non-virtual-deps,\
$($(call UPPERCASE,$(1))_FINAL_RECURSIVE_DEPENDENCIES))),$(p) [$($(call UPPERCASE,$(p))_LICENSE)])
+
+#
+# cpe-info helper functions
+#
+
+define cpe-manifest # cpe, pkg name, version, url
+ echo '"$(1)","$(2)","$(3)","$(4)"' >>$(CPE_MANIFEST_CSV)
+endef
--
1.9.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 2/9] cpe-info: id prefix/suffix
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 1/9] cpe-info: new make target Matt Weber
@ 2019-03-08 22:04 ` Matt Weber
2019-04-14 14:53 ` Arnout Vandecappelle
2019-03-08 22:04 ` [Buildroot] [PATCH v8 3/9] cpe-info: only report target pkgs Matt Weber
` (7 subsequent siblings)
9 siblings, 1 reply; 18+ messages in thread
From: Matt Weber @ 2019-03-08 22:04 UTC (permalink / raw)
To: buildroot
There are two types of software CPE prefixes applicable for software,
one for applications and one for operating systems.
Note: The third type is for hardware.
This patchset determines which should be used and stores that
information with the package for later use when assembling the CPE
report.
Refs:
https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7695.pdf
https://cpe.mitre.org/specification/
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v8
- Rebase
v4 -> v7
- None
v3
[Arnout
- Moved CPE prefix and suffix defines to package/Makefile.in
v1 -> v2
[Thomas P
- Change to using a filter on pkg name value vs ifelse
---
package/Makefile.in | 4 ++++
package/pkg-generic.mk | 8 +++++++-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/package/Makefile.in b/package/Makefile.in
index dc818a2..644282f 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -389,6 +389,10 @@ TARGET_CONFIGURE_ARGS = \
################################################################################
+CPE_PREFIX_OS = cpe:2.3:o
+CPE_PREFIX_APP = cpe:2.3:a
+CPE_SUFFIX = *:*:*:*:*:*:*
+
ifeq ($(BR2_SYSTEM_ENABLE_NLS),y)
NLS_OPTS = --enable-nls
TARGET_NLS_DEPENDENCIES = host-gettext
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 644128d..a547c65 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -916,11 +916,17 @@ $(2)_CPE_ID_NAME ?= $$($(2)_NAME)
$(2)_CPE_ID_VERSION ?= $$($(2)_VERSION)
$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION)
+ifneq ($(filter linux linux-headers,$(1)),)
+$(2)_CPE_PREFIX = $(CPE_PREFIX_OS)
+else
+$(2)_CPE_PREFIX = $(CPE_PREFIX_APP)
+endif
+
$(1)-cpe-info: PKG=$(2)
$(1)-cpe-info:
ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
@$$(call MESSAGE,"Collecting cpe info")
- $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
+ $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
# legal-info: declare dependencies and set values used later for the manifest
--
1.9.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 3/9] cpe-info: only report target pkgs
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 1/9] cpe-info: new make target Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 2/9] cpe-info: id prefix/suffix Matt Weber
@ 2019-03-08 22:04 ` Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 4/9] cpe-info: cpe minor version support Matt Weber
` (6 subsequent siblings)
9 siblings, 0 replies; 18+ messages in thread
From: Matt Weber @ 2019-03-08 22:04 UTC (permalink / raw)
To: buildroot
The reporting of host packages causes some duplication and complicates
what is really in the targets configuration. For the purpose of the
first version of this patchset, its assumed that host packages aren't
relevant for the configuration and we only report the target's
contents.
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v4 -> v8
- None
v3
- Fixed host build error because cpe-info wasn't defined
v1 -> v2
[Thomas P
- select if target vs selecting not host
---
package/pkg-generic.mk | 2 ++
1 file changed, 2 insertions(+)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index a547c65..8ff389e 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -924,10 +924,12 @@ endif
$(1)-cpe-info: PKG=$(2)
$(1)-cpe-info:
+ifeq ($$($(2)_TYPE),target)
ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
@$$(call MESSAGE,"Collecting cpe info")
$(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
+endif # ifeq ($$($(2)_TYPE),target)
# legal-info: declare dependencies and set values used later for the manifest
ifneq ($$($(2)_LICENSE_FILES),)
--
1.9.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 4/9] cpe-info: cpe minor version support
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
` (2 preceding siblings ...)
2019-03-08 22:04 ` [Buildroot] [PATCH v8 3/9] cpe-info: only report target pkgs Matt Weber
@ 2019-03-08 22:04 ` Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 5/9] toolchain/toolchain-ext: glibc cpe-info support Matt Weber
` (5 subsequent siblings)
9 siblings, 0 replies; 18+ messages in thread
From: Matt Weber @ 2019-03-08 22:04 UTC (permalink / raw)
To: buildroot
Minor version are used to allow -rc, - and other variants of
representing a sub release in the CPE dictionary.
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v8
- New support to allow rc, -, and p# minor versions as the CPE dict
currently uses these on a number of packages.
---
package/Makefile.in | 2 +-
package/pkg-generic.mk | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/package/Makefile.in b/package/Makefile.in
index 644282f..3e93145 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -391,7 +391,7 @@ TARGET_CONFIGURE_ARGS = \
CPE_PREFIX_OS = cpe:2.3:o
CPE_PREFIX_APP = cpe:2.3:a
-CPE_SUFFIX = *:*:*:*:*:*:*
+CPE_SUFFIX = *:*:*:*:*:*
ifeq ($(BR2_SYSTEM_ENABLE_NLS),y)
NLS_OPTS = --enable-nls
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 8ff389e..1b895d0 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -914,7 +914,8 @@ endif
$(2)_CPE_ID_VENDOR ?= $$($(2)_NAME)_project
$(2)_CPE_ID_NAME ?= $$($(2)_NAME)
$(2)_CPE_ID_VERSION ?= $$($(2)_VERSION)
-$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION)
+$(2)_CPE_ID_VERSION_MINOR ?= *
+$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION):$$($(2)_CPE_ID_VERSION_MINOR)
ifneq ($(filter linux linux-headers,$(1)),)
$(2)_CPE_PREFIX = $(CPE_PREFIX_OS)
--
1.9.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 5/9] toolchain/toolchain-ext: glibc cpe-info support
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
` (3 preceding siblings ...)
2019-03-08 22:04 ` [Buildroot] [PATCH v8 4/9] cpe-info: cpe minor version support Matt Weber
@ 2019-03-08 22:04 ` Matt Weber
2019-04-14 15:27 ` Arnout Vandecappelle
2019-03-08 22:04 ` [Buildroot] [PATCH v8 6/9] cpe-info: update manual for new pkg vars Matt Weber
` (4 subsequent siblings)
9 siblings, 1 reply; 18+ messages in thread
From: Matt Weber @ 2019-03-08 22:04 UTC (permalink / raw)
To: buildroot
From: Shruthi Singh <shruthi.singh@rockwellcollins.com>
This commit adds the correct CPE string for glibc, describing CPE ID,
VERSION, PACKAGE NAME and URL.
Signed-off-by: Shruthi Singh shruthi.singh at rockwellcollins.com
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v8
- No change
v7
- New
---
package/pkg-generic.mk | 14 ++++++++++++++
toolchain/toolchain-external/pkg-toolchain-external.mk | 7 +++++++
2 files changed, 21 insertions(+)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 1b895d0..f9cedd2 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -924,8 +924,22 @@ $(2)_CPE_PREFIX = $(CPE_PREFIX_APP)
endif
$(1)-cpe-info: PKG=$(2)
+ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y)
+$(1)-cpe-info: toolchain
+endif
$(1)-cpe-info:
ifeq ($$($(2)_TYPE),target)
+ifneq ($$($(2)_NAME),toolchain-external)
+ifneq ($(findstring TOOLCHAIN_EXTERNAL, $(2)),)
+ifeq ($(BR2_TOOLCHAIN_EXTERNAL_GLIBC),y)
+ $$(eval $(2)_VERSION = $$(shell $$(call TOOLCHAIN_CPE_INFO)))
+ $$(eval $(2)_CPE_ID_VENDOR = gnu)
+ $$(eval $(2)_CPE_ID_NAME = glibc)
+ $$(eval $(2)_ACTUAL_SOURCE_SITE = https://github.com/bminor/glibc/releases)
+ $$(eval $(2)_RAWNAME = glibc)
+endif # ifeq ($(BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC),y)
+endif # ifneq ($(findstring TOOLCHAIN_EXTERNAL, $(2)),)
+endif # ifneq ($$($(2)_NAME),toolchain-external)
ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
@$$(call MESSAGE,"Collecting cpe info")
$(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
index db3570d..aed06c5 100644
--- a/toolchain/toolchain-external/pkg-toolchain-external.mk
+++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
@@ -440,6 +440,13 @@ define TOOLCHAIN_EXTERNAL_INSTALL_SYSROOT_LIBS
$(call copy_toolchain_sysroot,$${SYSROOT_DIR},$${ARCH_SYSROOT_DIR},$${ARCH_SUBDIR},$${ARCH_LIB_DIR},$${SUPPORT_LIB_DIR})
endef
+define TOOLCHAIN_CPE_INFO
+ ARCH_SYSROOT_DIR="$(call toolchain_find_sysroot,$(TOOLCHAIN_EXTERNAL_CC) $(TOOLCHAIN_EXTERNAL_CFLAGS))" ; \
+ MAJ=`awk '{ if ($$1 = /#define/ && ($$2= /__GLIBC__/)){printf $$3};}' $${ARCH_SYSROOT_DIR}/usr/include/features.h` ; \
+ MIN=`awk '{ if ($$1 = /#define/ && ($$2 = /_GLIBC_MINOR/)){printf $$3};}' $${ARCH_SYSROOT_DIR}/usr/include/features.h` ; \
+ echo $${MAJ}.$${MIN}
+endef
+
# Create a symlink from (usr/)$(ARCH_LIB_DIR) to lib.
# Note: the skeleton package additionally creates lib32->lib or lib64->lib
# (as appropriate)
--
1.9.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 6/9] cpe-info: update manual for new pkg vars
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
` (4 preceding siblings ...)
2019-03-08 22:04 ` [Buildroot] [PATCH v8 5/9] toolchain/toolchain-ext: glibc cpe-info support Matt Weber
@ 2019-03-08 22:04 ` Matt Weber
2019-04-14 15:32 ` Arnout Vandecappelle
2019-04-14 17:24 ` Thomas De Schampheleire
2019-03-08 22:04 ` [Buildroot] [PATCH v8 7/9] support/scripts/cpedb.py: new CPE XML helper Matt Weber
` (3 subsequent siblings)
9 siblings, 2 replies; 18+ messages in thread
From: Matt Weber @ 2019-03-08 22:04 UTC (permalink / raw)
To: buildroot
Provide guidance on setting up the *_CPE_* and *_CVE_* variables.
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v8
- Added note about minor version
- Removed CVE patch listing as the backend has not be implemented
(LIBFOO_CVE_PATCHED)
v4 -> v7
- None
v3
- Updated to make *_CPE_VENDOR optional
- Changed wording around _CPE_ID as there is only one defined now
v2
[Thomas P
- Reworded LIBFOO_CVE_PATCHED description
[Matt W
- Added definition for new preset variables to auto-gen the CPE ID
- Added example LIBFOO_CPE_ID_VENDOR to LIBFOO
---
docs/manual/adding-packages-generic.txt | 111 +++++++++++++++++++-------------
1 file changed, 68 insertions(+), 43 deletions(-)
diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
index 7be1754..ddf1b2e 100644
--- a/docs/manual/adding-packages-generic.txt
+++ b/docs/manual/adding-packages-generic.txt
@@ -24,57 +24,59 @@ system is based on hand-written Makefiles or shell scripts.
09: LIBFOO_SITE = http://www.foosoftware.org/download
10: LIBFOO_LICENSE = GPL-3.0+
11: LIBFOO_LICENSE_FILES = COPYING
-12: LIBFOO_INSTALL_STAGING = YES
-13: LIBFOO_CONFIG_SCRIPTS = libfoo-config
-14: LIBFOO_DEPENDENCIES = host-libaaa libbbb
-15:
-16: define LIBFOO_BUILD_CMDS
-17: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all
-18: endef
-19:
-20: define LIBFOO_INSTALL_STAGING_CMDS
-21: $(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a
-22: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h
-23: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib
-24: endef
-25:
-26: define LIBFOO_INSTALL_TARGET_CMDS
-27: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib
-28: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d
-29: endef
-30:
-31: define LIBFOO_USERS
-32: foo -1 libfoo -1 * - - - LibFoo daemon
-33: endef
-34:
-35: define LIBFOO_DEVICES
-36: /dev/foo c 666 0 0 42 0 - - -
-37: endef
-38:
-39: define LIBFOO_PERMISSIONS
-40: /bin/foo f 4755 foo libfoo - - - - -
-41: endef
-42:
-43: $(eval $(generic-package))
+12: LIBFOO_CPE_ID_VENDOR = foosoftware
+13: LIBFOO_INSTALL_STAGING = YES
+14: LIBFOO_CONFIG_SCRIPTS = libfoo-config
+15: LIBFOO_DEPENDENCIES = host-libaaa libbbb
+16:
+17: define LIBFOO_BUILD_CMDS
+18: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all
+19: endef
+20:
+21: define LIBFOO_INSTALL_STAGING_CMDS
+22: $(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a
+23: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h
+24: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib
+25: endef
+26:
+27: define LIBFOO_INSTALL_TARGET_CMDS
+28: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib
+29: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d
+30: endef
+31:
+32: define LIBFOO_USERS
+33: foo -1 libfoo -1 * - - - LibFoo daemon
+34: endef
+35:
+36: define LIBFOO_DEVICES
+37: /dev/foo c 666 0 0 42 0 - - -
+38: endef
+39:
+40: define LIBFOO_PERMISSIONS
+41: /bin/foo f 4755 foo libfoo - - - - -
+42: endef
+43:
+44: $(eval $(generic-package))
--------------------------------
-The Makefile begins on line 7 to 11 with metadata information: the
+The Makefile begins on line 7 to 12 with metadata information: the
version of the package (+LIBFOO_VERSION+), the name of the
tarball containing the package (+LIBFOO_SOURCE+) (xz-ed tarball recommended)
the Internet location at which the tarball can be downloaded from
-(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+) and file with the
-license text (+LIBFOO_LICENSE_FILES+). All variables must start with
+(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+), the file with the
+license text (+LIBFOO_LICENSE_FILES+) and the vendor for vunerability
+analysis (+LIBFOO_CPE_ID_VENDOR+). All variables must start with
the same prefix, +LIBFOO_+ in this case. This prefix is always the
uppercased version of the package name (see below to understand where
the package name is defined).
-On line 12, we specify that this package wants to install something to
+On line 13, we specify that this package wants to install something to
the staging space. This is often needed for libraries, since they must
install header files and other development files in the staging space.
This will ensure that the commands listed in the
+LIBFOO_INSTALL_STAGING_CMDS+ variable will be executed.
-On line 13, we specify that there is some fixing to be done to some
+On line 14, we specify that there is some fixing to be done to some
of the 'libfoo-config' files that were installed during
+LIBFOO_INSTALL_STAGING_CMDS+ phase.
These *-config files are executable shell script files that are
@@ -122,14 +124,14 @@ IMAGEMAGICK_CONFIG_SCRIPTS = \
--------------------------------
================================
-On line 14, we specify the list of dependencies this package relies
+On line 15, we specify the list of dependencies this package relies
on. These dependencies are listed in terms of lower-case package names,
which can be packages for the target (without the +host-+
prefix) or packages for the host (with the +host-+) prefix).
Buildroot will ensure that all these packages are built and installed
'before' the current package starts its configuration.
-The rest of the Makefile, lines 16..29, defines what should be done
+The rest of the Makefile, lines 17..29, defines what should be done
at the different steps of the package configuration, compilation and
installation.
+LIBFOO_BUILD_CMDS+ tells what steps should be performed to
@@ -142,16 +144,16 @@ All these steps rely on the +$(@D)+ variable, which
contains the directory where the source code of the package has been
extracted.
-On lines 31..43, we define a user that is used by this package (e.g.
+On lines 32..44, we define a user that is used by this package (e.g.
to run a daemon as non-root) (+LIBFOO_USERS+).
-On line 35..37, we define a device-node file used by this package
+On line 36..38, we define a device-node file used by this package
(+LIBFOO_DEVICES+).
-On line 39..41, we define the permissions to set to specific files
+On line 40..42, we define the permissions to set to specific files
installed by this package (+LIBFOO_PERMISSIONS+).
-Finally, on line 43, we call the +generic-package+ function, which
+Finally, on line 44, we call the +generic-package+ function, which
generates, according to the variables defined previously, all the
Makefile code necessary to make your package working.
@@ -482,6 +484,29 @@ not and can not work as people would expect it should:
locations, `/lib/firmware`, `/usr/lib/firmware`, `/lib/modules`,
`/usr/lib/modules`, and `/usr/share`, which are automatically excluded.
+* +LIBFOO_CPE_ID_VENDOR+
+ This variable is optional. It only must be defined if the package name
+ does not match what the CPE ID uses for the vendor. By default it's set
+ to <pkg-name>_project.
+
+* +LIBFOO_CPE_ID_NAME+
+ This variable is optional. It only must be defined if the package name
+ does not match what the CPE ID uses for the name. By default it's set
+ to <pkg-name>.
+
+* +LIBFOO_CPE_ID_VERSION+
+ This variable is optional. By default it's set to <pkg-version>.
+
+* +LIBFOO_CPE_ID_VERSION_MINOR+
+ This variable is optional. By default it's set to *.
+
+* +LIBFOO_CPE_ID+ is optional, as the package infrastructure hangles the
+ default case of a single package's Common Product Enumeration (CPE)
+ identification string. +make cpe-info+ copies all of these into a
+ +cpe-manifest.csv+ file. To identify a package's possible CPE,
+ the National Vunerability Database can be searched at
+ https://nvd.nist.gov/products/cpe/search.
+
The recommended way to define these variables is to use the following
syntax:
--
1.9.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 7/9] support/scripts/cpedb.py: new CPE XML helper
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
` (5 preceding siblings ...)
2019-03-08 22:04 ` [Buildroot] [PATCH v8 6/9] cpe-info: update manual for new pkg vars Matt Weber
@ 2019-03-08 22:04 ` Matt Weber
2019-04-14 18:05 ` Arnout Vandecappelle
2019-03-08 22:04 ` [Buildroot] [PATCH v8 8/9] support/scripts/cpe-report: new script Matt Weber
` (2 subsequent siblings)
9 siblings, 1 reply; 18+ messages in thread
From: Matt Weber @ 2019-03-08 22:04 UTC (permalink / raw)
To: buildroot
Python class which consumes a NIST CPE XML and provides helper
functions to access and search the db's data.
- Defines the CPE as a object with operations / formats
- Processing of CPE dictionary
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
v8
- Added support for generation of update xml to maintain the
NIST dictionary for any Buildroot package version bumps
- Dropped searching of the Config.in files for URLs, instead
assuming the first time a package is added to NIST, the xml is
manually filled out with reference urls. Any updates to versions
after that will use the proposed autogen xml that mines the URLS
from the NIST dict file.
- Caching support for a processed dictionary to speed up subsequent
runs when testing, as a db doesn't update more then once a day
v5 -> v7
- No change
v5
[Ricardo
- Fixed typo in join/split of cpe str without version
- Removed extra prints as they aren't needed when we have the
output reports/stdout
- Updated v4 comments about general flake formatting cleanup
- Incorporated parts of patch 1/2 suggestions for optimizations
[Arnout
- added pre-processing of cpe values into two sets, one with
and one without version
- Collectly with Ricardo, decided to move cpe class to this
seperate script
v1 -> v4
- No version
---
support/scripts/cpedb.py | 185 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 185 insertions(+)
create mode 100644 support/scripts/cpedb.py
diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
new file mode 100644
index 0000000..0369536
--- /dev/null
+++ b/support/scripts/cpedb.py
@@ -0,0 +1,185 @@
+import sys
+import urllib2
+from collections import OrderedDict
+import xmltodict
+import gzip
+from StringIO import StringIO
+import os
+import pickle
+
+VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
+
+
+class CPE:
+ cpe_str = None
+ cpe_str_short = None
+ cpe_desc = None
+ cpe_cur_ver = None
+ titles = {}
+ references = {}
+
+ def __init__(self, cpe_str, titles=None, refs=None):
+ self.cpe_str = cpe_str
+ self.cpe_str_short = ":".join(self.cpe_str.split(":")[:6])
+ self.titles = titles
+ self.references = refs
+ self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
+
+ def to_dict(self, cpe_str):
+ cpe_short_name = ":".join(cpe_str.split(":")[2:6])
+ cpe_new_ver = "".join(cpe_str.split(":")[5:6])
+ self.titles[0]['#text'] = self.titles[0]['#text'].replace(self.cpe_cur_ver, cpe_new_ver)
+ cpe_dict = OrderedDict([
+ ('cpe-item', OrderedDict([
+ ('@name', 'cpe:/' + cpe_short_name),
+ ('title', self.titles),
+ ('references', OrderedDict([('reference', self.references)])),
+ ('cpe-23:cpe23-item', OrderedDict([
+ ('@name', cpe_str)
+ ]))
+ ]))
+ ])
+ return cpe_dict
+
+
+class CPEDB:
+ all_cpes = dict()
+ all_cpes_no_version = dict()
+
+ def get_xml_dict(self, url):
+ print("CPE: Setting up NIST dictionary")
+ # Setup location to save dict and xmls, if it exists, assume we're
+ # reusing the previous dict
+ if not os.path.exists("cpe"):
+ os.makedirs("cpe")
+ self.get_new_xml_dict(url)
+ else:
+ print("CPE: Loading CACHED dictionary")
+ cpe_file = open('cpe/.all_cpes.pkl', 'rb')
+ self.all_cpes = pickle.load(cpe_file)
+ cpe_file.close()
+ cpe_file = open('cpe/.all_cpes_no_version.pkl', 'rb')
+ self.all_cpes_no_version = pickle.load(cpe_file)
+ cpe_file.close()
+
+ def get_new_xml_dict(self, url):
+ print("CPE: Fetching xml manifest from [" + url + "]")
+ try:
+ compressed_cpe_file = urllib2.urlopen(url)
+ print("CPE: Unzipping xml manifest...")
+ nist_cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read()
+ print("CPE: Converting xml manifest to dict...")
+ all_cpedb = xmltodict.parse(nist_cpe_file)
+
+ # Cycle through the dict and build two dict to be used for custom
+ # lookups of partial and complete CPE objects
+ # The objects are then used to create new proposed XML updates if
+ # if is determined one is required
+ for cpe in all_cpedb['cpe-list']['cpe-item']:
+ cpe_titles = cpe['title']
+ # There maybe multiple titles or one. Make sure this is
+ # always a list
+ if not isinstance(cpe_titles, (list,)):
+ cpe_titles = [cpe_titles]
+ # Out of the different language titles, select English
+ for title in cpe_titles:
+ if title['@xml:lang'] is 'en-US':
+ cpe_titles = [title]
+ # Some older CPE don't include references, if they do, make
+ # sure we handle the case of one ref needing to be packed
+ # in a list
+ if 'references' in cpe:
+ cpe_ref = cpe['references']['reference']
+ if not isinstance(cpe_ref, (list,)):
+ cpe_ref = [cpe_ref]
+ # The reference text has not been consistantly upper case
+ # in the NIST dict but they now require it. So force upper
+ # and then check for compliance to a specific tagging
+ for ref_href in cpe_ref:
+ ref_href['#text'] = ref_href['#text'].upper()
+ if ref_href['#text'] not in VALID_REFS:
+ ref_href['#text'] = ref_href['#text'] + "-- UPDATE this entry, here are some exmaples and just one word should be used -- " + ' '.join(VALID_REFS)
+ cpe_str = cpe['cpe-23:cpe23-item']['@name']
+ item = CPE(cpe_str, cpe_titles, cpe_ref)
+ cpe_str_no_version = self.get_cpe_no_version(cpe_str)
+ # This dict must have a unique key for every CPE version
+ # which allows matching to the specific obj data of that
+ # NIST dict entry
+ self.all_cpes.update({cpe_str: item})
+ # This dict has one entry for every CPE (w/o version) to allow
+ # partial match (no valid version) check (the obj is saved and
+ # used as seed for suggested xml updates. By updating the same
+ # non-version'd entry, it assumes the last update here is the
+ # latest version in the NIST dict)
+ self.all_cpes_no_version.update({cpe_str_no_version: item})
+
+ except urllib2.HTTPError:
+ print("CPE: HTTP Error: %s" % url)
+ sys.exit(1)
+ except urllib2.URLError:
+ print("CPE: URL Error: %s" % url)
+ sys.exit(1)
+
+ print("CPE: Caching dictionary")
+ cpes_file = open('cpe/.all_cpes.pkl', 'wb')
+ pickle.dump(self.all_cpes, cpes_file)
+ cpes_file.close()
+ cpes_file = open('cpe/.all_cpes_no_version.pkl', 'wb')
+ pickle.dump(self.all_cpes_no_version, cpes_file)
+ cpes_file.close()
+
+ def find_partial(self, cpe_str):
+ cpe_str_no_version = self.get_cpe_no_version(cpe_str)
+ if cpe_str_no_version in self.all_cpes_no_version:
+ return cpe_str_no_version
+
+ def find_partial_obj(self, cpe_str):
+ cpe_str_no_version = self.get_cpe_no_version(cpe_str)
+ if cpe_str_no_version in self.all_cpes_no_version:
+ return self.all_cpes_no_version[cpe_str_no_version]
+
+ def find_partial_latest_version(self, cpe_str_partial):
+ cpe_obj = self.find_partial_obj(cpe_str_partial)
+ return cpe_obj.cpe_cur_ver
+
+ def find(self, cpe_str):
+ if self.find_partial(cpe_str):
+ if cpe_str in self.all_cpes:
+ return cpe_str
+
+ def update(self, cpe_str):
+ to_update = self.find_partial_obj(cpe_str)
+ xml = self.__gen_xml__(to_update.to_dict(cpe_str))
+ fp = open(os.path.join('cpe', self.get_cpe_name(cpe_str) + '-' + self.get_cpe_version(cpe_str) + '.xml'), 'w+')
+ fp.write(xmltodict.unparse(xml, pretty=True))
+ fp.close()
+
+ def get_nvd_url(self, cpe_str):
+ return "https://nvd.nist.gov/products/cpe/search/results?keyword=" + \
+ urllib2.quote(cpe_str) + \
+ "&status=FINAL&orderBy=CPEURI&namingFormat=2.3"
+
+ def get_cpe_no_version(self, cpe):
+ return ":".join(cpe.split(":")[:5])
+
+ def get_cpe_name(self, cpe_str):
+ return "".join(cpe_str.split(":")[4])
+
+ def get_cpe_version(self, cpe_str):
+ return "".join(cpe_str.split(":")[5])
+
+ def __gen_xml__(self, cpe_list):
+ list_header = {
+ "cpe-list": {
+ "@xmlns:config": "http://scap.nist.gov/schema/configuration/0.1",
+ "@xmlns": "http://cpe.mitre.org/dictionary/2.0",
+ "@xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
+ "@xmlnsscap-core": "http://scap.nist.gov/schema/scap-core/0.3",
+ "@xmlns:cpe-23": "http://scap.nist.gov/schema/cpe-extension/2.3",
+ "@xmlns:ns6": "http://scap.nist.gov/schema/scap-core/0.1",
+ "@xmlns:meta": "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
+ "@xsi:schemaLocation": "http://scap.nist.gov/schema/cpe-extension/2.3 https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd http://cpe.mitre.org/dictionary/2.0 https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2 https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd http://scap.nist.gov/schema/scap-core/0.3 https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd http://scap.nist.gov/schema/configuration/0.1 https://scap.nist.gov/schema/nvd/configuration_0.1.xsd http://scap.nist.gov/schema/scap-core/0.1 https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"
+ }
+ }
+ list_header['cpe-list'].update(cpe_list)
+ return list_header
--
1.9.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 8/9] support/scripts/cpe-report: new script
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
` (6 preceding siblings ...)
2019-03-08 22:04 ` [Buildroot] [PATCH v8 7/9] support/scripts/cpedb.py: new CPE XML helper Matt Weber
@ 2019-03-08 22:04 ` Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 9/9] docs/manual: new security management section Matt Weber
2019-04-14 18:08 ` [Buildroot] [PATCH v8 0/9] Package CPE Reporting Arnout Vandecappelle
9 siblings, 0 replies; 18+ messages in thread
From: Matt Weber @ 2019-03-08 22:04 UTC (permalink / raw)
To: buildroot
The script supports looking up all the CPEs provided in a
make cpe-info csv file export from a target Buildroot build.
It checks the current version and suggests a CPE needs update
or possibly an initial submission is required to NIST.
Adds option to allow alternate locations for the dictionary
URL and caching of a processed dictionary to speed up execution.
Outputs a cpe/ folder with propsed xml generated from the
dictionary contents to propose updated versions to NIST.
For missing CPE matches, a cpe-report-missing.txt is created
by the script that can be used later to manually create proposed
new NIST dictionary entries.
Ref: NIST has a group email (cpe_dictionary at nist.gov) used to
recieve these version update and new entry xml files. They do
process the XML and provide feedback. In some cases they will
propose back something different where the vendor or version is
slightly different.
Limitations
- Currently any use of non-number version identifiers isn't
supported by NIST as they use ranges to determine impact
of a CVE
- Any Linux version from a non-upstream is also not supported
without manually adjusting the information as the custom
kernel will more then likely not match the upstream version
used in the dictionary
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
Changes
v8
- Updated to just output missing and needs version update
- Optional processed dictionary caching support
- Optional dictionary URL
- Creation of a missing status file (cpe-report-missing.txt)
- Adjusted index used in CSV for removal of CVE patched item
v5 -> v7
- No change
v5
[Ricardo
- Updated v4 comments about general flake formatting cleanup
- Incorporated parts of patch 1/2 suggestions for optimizations
[Ricardo/Arnout
- Collectly, decided to move cpe report analysis to this
script and use a seperate module cpedb class
[Arnout
- Rename cpe_dict to instead be cpedb
v1 -> v4
- Patch did not exist and was part of pkg-stats file
---
support/scripts/cpe-report | 70 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)
create mode 100755 support/scripts/cpe-report
diff --git a/support/scripts/cpe-report b/support/scripts/cpe-report
new file mode 100755
index 0000000..7242a37
--- /dev/null
+++ b/support/scripts/cpe-report
@@ -0,0 +1,70 @@
+#!/usr/bin/env python
+
+import argparse
+import sys
+import csv
+from cpedb import CPEDB
+
+CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
+
+
+def get_target_cpe_report(cpe_report_file, cpedb):
+ report_cpe_exact_match = ""
+ report_cpe_needing_update = ""
+ report_cpe_needing_update_list = ""
+ report_cpe_missing = ""
+
+ print("CPE: Checking for matches...")
+ try:
+ with open(cpe_report_file) as cpe_file:
+ cpe_list = csv.reader(cpe_file)
+ next(cpe_list) # make cpe-info has a one line header
+ for cpe in cpe_list:
+ result = cpedb.find(cpe[0])
+ if not result:
+ result = cpedb.find_partial(cpedb.get_cpe_no_version(cpe[0]))
+ if not result:
+ report_cpe_missing += cpe[0] + "," + cpe[1] + "," + cpe[3] + "\n"
+ else:
+ latest_version = cpedb.find_partial_latest_version(cpedb.get_cpe_no_version(cpe[0]))
+ report_cpe_needing_update += cpe[0] + ", Latest Version Guess from Dict[" + latest_version + "]\n"
+ report_cpe_needing_update_list += cpe[0] + "\n"
+ else:
+ report_cpe_exact_match += cpe[0] + "\n"
+ except (OSError, IOError) as e:
+ print("CPE: report csv file (%s): %s" % (e.errno, e.strerror))
+ sys.exit(1)
+
+ print("CPE: Found but may REQUIRE an UPDATE:\n" + report_cpe_needing_update)
+ print("CPE: Not found:\n" + report_cpe_missing)
+
+ fp = open('cpe-report-missing.txt', 'w+')
+ fp.write(report_cpe_missing)
+ fp.close()
+
+ for cpe in report_cpe_needing_update_list.splitlines():
+ cpedb.update(cpe)
+ print("XML Generation Complete of NIST update files, see ./cpe/*")
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('-c', dest='cpe_report', action='store', required=True,
+ help='CPE Report generated by make cpe-info (csv format)')
+ parser.add_argument('-u', dest='url', action='store', required=False,
+ help='(optional)URL to the NIST dict (official-cpe-dictionary_v2.3.xml.gz)')
+ return parser.parse_args()
+
+
+def __main__():
+ args = parse_args()
+ cpedb = CPEDB()
+ url = CPE_XML_URL
+ if args.url:
+ url = args.url
+ cpedb.get_xml_dict(url)
+ print("Performing Target CPE Report Analysis...")
+ get_target_cpe_report(args.cpe_report, cpedb)
+
+
+__main__()
--
1.9.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 9/9] docs/manual: new security management section
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
` (7 preceding siblings ...)
2019-03-08 22:04 ` [Buildroot] [PATCH v8 8/9] support/scripts/cpe-report: new script Matt Weber
@ 2019-03-08 22:04 ` Matt Weber
2019-04-14 14:39 ` Arnout Vandecappelle
2019-04-14 18:08 ` [Buildroot] [PATCH v8 0/9] Package CPE Reporting Arnout Vandecappelle
9 siblings, 1 reply; 18+ messages in thread
From: Matt Weber @ 2019-03-08 22:04 UTC (permalink / raw)
To: buildroot
This changeset captures an initial discussion on the use of CPE
reporting within a target build. It notes the reporting limitations
and provides actions a user could take to improve upon the current
report information.
There is also an example of how one might do CVE analysis using the
CPE report information.
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
---
Changes
v8
- Updated for cpe-report changes
- Added notes on doing CVE searches and submissions
v7
- New
---
docs/manual/cpe-reporting.txt | 103 ++++++++++++++++++++++++++++++++++++++++++
docs/manual/manual.txt | 2 +
2 files changed, 105 insertions(+)
create mode 100644 docs/manual/cpe-reporting.txt
diff --git a/docs/manual/cpe-reporting.txt b/docs/manual/cpe-reporting.txt
new file mode 100644
index 0000000..a6f32d5
--- /dev/null
+++ b/docs/manual/cpe-reporting.txt
@@ -0,0 +1,103 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[cpe-info]]
+
+== Security Vulnerability Management
+
+There are many different vulnerability databases (open/paid). This
+section documents the use of the National Vulnerability Database(NVD)
+provided by the National Institute of Standards and Technology (NIST).
+
+Within Buildroot, the intent is to provide good reporting of the build
+configuration's inventory of software. The vulnerability analysis is
+assumed to occur outside of the Buildroot environment (at this time).
+
+=== Common Platform Enumeration (CPE) Reporting
+
+Buildroot consists of a series of upstream packages. Each of those
+packages may have a CPE definition used to map vulnerabilities to Common
+Vulnerabilities and Exposures (CVE). A single package CPE has many versions
+and each version may have a suite of CVEs associated.
+
+To make the gathering of the software inventory of CPE easier, Buildroot can
+collect for you all the CPE related to the configured defconfig. To produce
+this material, after you have configured Buildroot with +make menuconfig+,
++make xconfig+ or +make gconfig+, run:
+
+--------------------
+make cpe-info
+--------------------
+
+Buildroot then collects and writes the +$(TOPDIR)/cpe-manifest.csv+. This file
+can be used for manual inspection against a CVE database or provided to
+external tools which perform CVE inventory/analysis.
+
+*CPE Maintenance*
+
+To maintain these CPE strings for version changes against the NIST dictionary,
+the manifest can be further processed. First, navigate to your Buildroot
+directory and execute the script below. The script has some optional arguments
+for providing a alternate dictionary URL or caching a processed dictionary.
+
+--------------------
+support/scripts/cpe-report -c $(TOPDIR)/cpe-manifest.csv
+--------------------
+
+This script retrieves the NIST dictionary and classifies each CPE as either
+matched, requires version update or missing. Based on this analysis, the script
+automatically uses the NIST dictionary entries to produce a draft of XML which
+can be submitted to NIST to update a version of an entry in the dictionary. It
+is important to review the generated xml files in the cpe folder as they may
+need refined reference tags and adjustments to how the version is represented
+in the title.
+
+In the case of missing items, a +cpe-report-missing.txt+ report is output by
+the script and can be used as a starting point to manually create a xml file
+to submit. Note, some manual analysis using the NIST search engine (https://nvd.nist.gov/products/cpe/search)
+is suggested for these missing item as the Buildroot +CPE_ID_+ variables maybe
+slightly incorrect and cause the cpe-report script to catagorize the package
+as missing. If that is the case, a change can be made by adjusting the default
+CPE variables in the specific package's +.mk+. See xref:_infrastructure_for_packages_with_specific_build_systems[]
+discussion on the use of +LIBFOO_CPE_*+.
+If the package is truely missing, the package's Kconfig help material and .mk
+should provide most of the information to construct a new NIST submission.
+
+To submit a new entry or updated entry to NIST, create an request email to the
+cpe_dictionary at nist.gov recipient and attach a individual xml file per package
+being added/updated. It is OK to have multiple version updates in a single
+file as long as they are all for the same package. For reference the guidance
+can be found on the NIST CPE site (https://nvd.nist.gov/products/cpe).
+
+*Limitations*
+
+Buildroot does not produce or accurately present some of the CPE material. Items
+such as any versions which are non-number/hash are not compliant with the CPE
+string specification and would require a manual analysis to update the CPE list
+before any external CVE analysis should occur. This is a similar situation for
+packages like the Linux kernel or U-Boot which may not have a version which
+directly maps to a CPE.
+
+There is an assumed default CPE string for each package which is auto-generated
+using existing package information. The output of +make cpe-info+ is based on
+this default information and the packages which have been individually tailored
+to match existing CPE strings. The Buildroot developers try to do their best to
+keep those declarative statements as accurate as possible, to the best of their
+knowledge. However, it is very well possible that those declarative statements
+are not all fully accurate nor exhaustive. Similar to legal-info, it is your
+responsibility to verify this information.
+
+=== Common Vulnerability and Exposures (CVE) Anaylsis
+The Common Vulnerabilities and Exposures (CVE) system provides a
+reference-method for publicly known information-security vulnerabilities and
+exposures. (https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures)
+
+Like previously stated, there are many tools and methods to perform this
+analysis. The most basic example is to do a manual CVE analysis by navigating
+to the NVD search engine (https://nvd.nist.gov/vuln/search) and using the CPE
+string identified in the first field of the +$(TOPDIR)/cpe-manifest.csv+.
+Here's an example for tcpdump (https://nvd.nist.gov/vuln/search/results?form_type=Basic&results_type=overview&query=cpe%3A2.3%3Aa%3Atcpdump%3Atcpdump%3A4.9.1%3A*%3A*%3A*%3A*%3A*%3A*%3A*&search_type=all).
+
+Beyond the manual search approach, the next step would be a more centralized
+shared database with multi-feed support (NVD+). The cve-search project aims
+to offer that type of solution (https://github.com/cve-search/cve-search).
diff --git a/docs/manual/manual.txt b/docs/manual/manual.txt
index 4eb4ba9..fad8bf2 100644
--- a/docs/manual/manual.txt
+++ b/docs/manual/manual.txt
@@ -46,6 +46,8 @@ include::legal-notice.txt[]
include::beyond-buildroot.txt[]
+include::cpe-reporting.txt[]
+
= Developer guide
include::how-buildroot-works.txt[]
--
1.9.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 9/9] docs/manual: new security management section
2019-03-08 22:04 ` [Buildroot] [PATCH v8 9/9] docs/manual: new security management section Matt Weber
@ 2019-04-14 14:39 ` Arnout Vandecappelle
0 siblings, 0 replies; 18+ messages in thread
From: Arnout Vandecappelle @ 2019-04-14 14:39 UTC (permalink / raw)
To: buildroot
Hi Matt,
I'm going to give most of my feedback here, because the docs really say where
we are going with this.
On 08/03/2019 23:04, Matt Weber wrote:
> This changeset captures an initial discussion on the use of CPE
> reporting within a target build. It notes the reporting limitations
> and provides actions a user could take to improve upon the current
> report information.
>
> There is also an example of how one might do CVE analysis using the
> CPE report information.
>
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> ---
> Changes
>
> v8
> - Updated for cpe-report changes
> - Added notes on doing CVE searches and submissions
>
> v7
> - New
> ---
> docs/manual/cpe-reporting.txt | 103 ++++++++++++++++++++++++++++++++++++++++++
> docs/manual/manual.txt | 2 +
> 2 files changed, 105 insertions(+)
> create mode 100644 docs/manual/cpe-reporting.txt
>
> diff --git a/docs/manual/cpe-reporting.txt b/docs/manual/cpe-reporting.txt
> new file mode 100644
> index 0000000..a6f32d5
> --- /dev/null
> +++ b/docs/manual/cpe-reporting.txt
> @@ -0,0 +1,103 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +[[cpe-info]]
> +
> +== Security Vulnerability Management
> +
> +There are many different vulnerability databases (open/paid). This
> +section documents the use of the National Vulnerability Database(NVD)
> +provided by the National Institute of Standards and Technology (NIST).
> +
> +Within Buildroot, the intent is to provide good reporting of the build
> +configuration's inventory of software. The vulnerability analysis is
> +assumed to occur outside of the Buildroot environment (at this time).
... But in the end, what we want is a list of CVE and other vulnerabilities
that apply to my particular Buildroot configuration, and preferably weeding out
the ones that have been patched within Buildroot.
The inventory is a step in the direction of getting there, because you can
search the NIST CVE database based on CPE strings. But that hinges on the
quality of the CPE strings. We can distinguish a few cases.
* Perfect match: CPE_ID is the same as the CPE dictionary entry.
* Imperfect match: CPE_ID is the same as a CPE dictionary entry, but it doesn't
actually refer to exactly the same software (e.g. because we have patched it, or
we use a git commit beyond the release version).
* Version missing: the package exists in the CPE dictionary, but our specific
version is missing. This could be converted into an imperfect match by using a
"wrong" version string, or by updating the CPE dictionary.
* Package missing: the package doesn't exist in the CPE dictionary.
* Wrong match: the matching package in the CPE dictionary doesn't actually refer
to the same software at all.
So, how bad are these imperfect, missing and wrong matches? In fact, nobody
cares about the match in the CPE dictionary, what we care about is the generated
vulnerability list.
For searching CVEs, we have the choice of using a full CPE_ID that is identical
to a string in the CPE dictionary, or to use wildcards to (potentially) match
more entries. The more wildcards, the more false hits you get when searching for
vulnerabilities. The less wildcards, the more vulnerabilities you will miss.
Since you probably prefer to have false hits rather than missing
vulnerabilities, I think we should tend to using more wildcards. (Note BTW that
you're going to have false hits and missing vulns even if an exact CPE_ID is
used, because sometimes the CVE info itself is inaccurate).
Therefore, I think the automatically generated CPE_ID should be more
wildcard-y. In particular, I think the vendor should be set to *. Conversely, I
think that when a package provides a CPE_ID, it should always specify the full
ID (which could still use $(FOO_VERSION), of course). Yes, that's a bit of
duplication, but it's trivial to do (supposedly you just cut and paste from the
existing dictionary entry) and it's nice and explicit.
Note that this means that IMO eventually all packages should have a CPE_ID (or,
if the CPE_ID is missing, it would imply that the package doesn't exist in the
CPE dictionary).
> +
> +=== Common Platform Enumeration (CPE) Reporting
> +
> +Buildroot consists of a series of upstream packages. Each of those
> +packages may have a CPE definition used to map vulnerabilities to Common
> +Vulnerabilities and Exposures (CVE). A single package CPE has many versions
> +and each version may have a suite of CVEs associated.
That last statement is not entirely correct. It's actualy the CVE that will
have a suite of CPE match strings associated with it.
Ideally, the CVE's match strings would just be compared to the query's match
string - this way, there would not be a need for a dictionary at all.
Unfortunately, that's not how it works - at least for the NIST lookup.
> +To make the gathering of the software inventory of CPE easier, Buildroot can
> +collect for you all the CPE related to the configured defconfig. To produce
> +this material, after you have configured Buildroot with +make menuconfig+,
> ++make xconfig+ or +make gconfig+, run:
> +
> +--------------------
> +make cpe-info
> +--------------------
> +
> +Buildroot then collects and writes the +$(TOPDIR)/cpe-manifest.csv+. This file
> +can be used for manual inspection against a CVE database or provided to
> +external tools which perform CVE inventory/analysis.
Now that we have show-info on its way in, I don't think we should add this
cpe-info target at all. Instead, we should rely on show-info and let tools parse
the JSON rather than the CSV.
> +*CPE Maintenance*
This part does not belong in the user guide, it should go to a section in the
developer guide. Either a new top-level section, or a subsection of Contributing
to Buildroot.
> +To maintain these CPE strings for version changes against the NIST dictionary,
> +the manifest can be further processed. First, navigate to your Buildroot
> +directory and execute the script below. The script has some optional arguments
> +for providing a alternate dictionary URL or caching a processed dictionary.
> +
> +--------------------
> +support/scripts/cpe-report -c $(TOPDIR)/cpe-manifest.csv
> +--------------------
> +
> +This script retrieves the NIST dictionary and classifies each CPE as either
> +matched, requires version update or missing. Based on this analysis, the script
> +automatically uses the NIST dictionary entries to produce a draft of XML which
> +can be submitted to NIST to update a version of an entry in the dictionary. It
> +is important to review the generated xml files in the cpe folder as they may
> +need refined reference tags and adjustments to how the version is represented
> +in the title.
> +
> +In the case of missing items, a +cpe-report-missing.txt+ report is output by
> +the script and can be used as a starting point to manually create a xml file
> +to submit. Note, some manual analysis using the NIST search engine (https://nvd.nist.gov/products/cpe/search)
> +is suggested for these missing item as the Buildroot +CPE_ID_+ variables maybe
> +slightly incorrect and cause the cpe-report script to catagorize the package
> +as missing. If that is the case, a change can be made by adjusting the default
> +CPE variables in the specific package's +.mk+. See xref:_infrastructure_for_packages_with_specific_build_systems[]
> +discussion on the use of +LIBFOO_CPE_*+.
> +If the package is truely missing, the package's Kconfig help material and .mk
> +should provide most of the information to construct a new NIST submission.
> +
> +To submit a new entry or updated entry to NIST, create an request email to the
> +cpe_dictionary at nist.gov recipient and attach a individual xml file per package
> +being added/updated. It is OK to have multiple version updates in a single
> +file as long as they are all for the same package.
This statement is a bit stupid: in Buildroot context, you're never going to
have multiple version updates in a single XML file, are you?
> For reference the guidance
> +can be found on the NIST CPE site (https://nvd.nist.gov/products/cpe).
I actually don't find anything there about how to contribute to the dictionary,
other than "contact the CPE team".
> +
> +*Limitations*
This part should again be in the user documentation, not the developer
documentation.
> +
> +Buildroot does not produce or accurately present some of the CPE material. Items
> +such as any versions which are non-number/hash are not compliant with the CPE
> +string specification and would require a manual analysis to update the CPE list
> +before any external CVE analysis should occur. This is a similar situation for
> +packages like the Linux kernel or U-Boot which may not have a version which
> +directly maps to a CPE.
> +
> +There is an assumed default CPE string for each package which is auto-generated
> +using existing package information. The output of +make cpe-info+ is based on
> +this default information and the packages which have been individually tailored
> +to match existing CPE strings. The Buildroot developers try to do their best to
> +keep those declarative statements as accurate as possible, to the best of their
> +knowledge. However, it is very well possible that those declarative statements
> +are not all fully accurate nor exhaustive. Similar to legal-info, it is your
> +responsibility to verify this information.
So all this would be a bit stronger if we can just say that autogenerated
string may or may not convey useful information, and that if the vendor is set,
you know that it's a validated string.
> +
> +=== Common Vulnerability and Exposures (CVE) Anaylsis
> +The Common Vulnerabilities and Exposures (CVE) system provides a
> +reference-method for publicly known information-security vulnerabilities and
> +exposures. (https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures)
> +
> +Like previously stated, there are many tools and methods to perform this
> +analysis. The most basic example is to do a manual CVE analysis by navigating
> +to the NVD search engine (https://nvd.nist.gov/vuln/search) and using the CPE
> +string identified in the first field of the +$(TOPDIR)/cpe-manifest.csv+.
> +Here's an example for tcpdump (https://nvd.nist.gov/vuln/search/results?form_type=Basic&results_type=overview&query=cpe%3A2.3%3Aa%3Atcpdump%3Atcpdump%3A4.9.1%3A*%3A*%3A*%3A*%3A*%3A*%3A*&search_type=all).
Could we make a simple script that takes show-info and generates a list of URLs?
> +
> +Beyond the manual search approach, the next step would be a more centralized
> +shared database with multi-feed support (NVD+). The cve-search project aims
> +to offer that type of solution (https://github.com/cve-search/cve-search).
I tried the public search at https://cve.circl.lu/ and apparently it (or at
least the web interface and API) only allows to search by exact CPE_ID, or by
vendor/product (but NOT version).
Are you actually using this stuff at the moment? I feel that it still has some
way to go before being actually useful. If this printing of CVE search links
would be there, that would be something already. And of course, the tool to add
new entries is nice as well.
Regards,
Arnout
> diff --git a/docs/manual/manual.txt b/docs/manual/manual.txt
> index 4eb4ba9..fad8bf2 100644
> --- a/docs/manual/manual.txt
> +++ b/docs/manual/manual.txt
> @@ -46,6 +46,8 @@ include::legal-notice.txt[]
>
> include::beyond-buildroot.txt[]
>
> +include::cpe-reporting.txt[]
> +
> = Developer guide
>
> include::how-buildroot-works.txt[]
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 1/9] cpe-info: new make target
2019-03-08 22:04 ` [Buildroot] [PATCH v8 1/9] cpe-info: new make target Matt Weber
@ 2019-04-14 14:49 ` Arnout Vandecappelle
0 siblings, 0 replies; 18+ messages in thread
From: Arnout Vandecappelle @ 2019-04-14 14:49 UTC (permalink / raw)
To: buildroot
I mentioned most of my feedback already in the response to patch 9, but I'll
repeat here for simplicity.
On 08/03/2019 23:04, Matt Weber wrote:
[snip]
> + savedefconfig update-defconfig printvars cpe-info %-cpe-info
So, I'd like at least the per-package cpe-info to be replaced by show-info. A
top-level cpe-info might still be useful, but even that I prefer as a script in
utils that calls make and pipes in jq to get the cpe-info.
I also question the usefulness of creating a CSV. JSON is probably easier.
If you do keep the cpe-info target, then I think the patch should be split up
into a first patch that just introduces CPE_ID (and adds it to show-info), then
a patch that documents it, and only then the patch that introduces the cpe-info
target.
[snip]
> +$(2)_CPE_ID_VENDOR ?= $$($(2)_NAME)_project
You can just use $(3) instead of $$($(2)_NAME). Note that we really want $(3)
here, not $(1) (i.e. not $(2)_NAME)) because for host packages (which admittedly
are not supported yet) we'd want the name without host-.
> +$(2)_CPE_ID_NAME ?= $$($(2)_NAME)
> +$(2)_CPE_ID_VERSION ?= $$($(2)_VERSION)
> +$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION)
I think there should be only CPE_ID, which has to be fully specified by the
caller. It should also contain the prefix and suffix already. IOW, I think this
should simply be:
$(2)_CPE_ID ?= cpe:2.3:a:*:$(3):$$($(2)_VERSION):*:*:*:*:*:*:*
Regards,
Arnout
> +
> +$(1)-cpe-info: PKG=$(2)
> +$(1)-cpe-info:
> +ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
> + @$$(call MESSAGE,"Collecting cpe info")
> + $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
> +endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
> +
> # legal-info: declare dependencies and set values used later for the manifest
> ifneq ($$($(2)_LICENSE_FILES),)
> $(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES)
> @@ -1052,6 +1064,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
> $(1)-clean-for-reconfigure \
> $(1)-clean-for-reinstall \
> $(1)-configure \
> + $(1)-cpe-info \
> $(1)-depends \
> $(1)-dirclean \
> $(1)-external-deps \
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index bffd79d..0201632 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -102,3 +102,11 @@ legal-deps = \
> $(filter-out $(if $(1:host-%=),host-%),\
> $(call non-virtual-deps,\
> $($(call UPPERCASE,$(1))_FINAL_RECURSIVE_DEPENDENCIES))),$(p) [$($(call UPPERCASE,$(p))_LICENSE)])
> +
> +#
> +# cpe-info helper functions
> +#
> +
> +define cpe-manifest # cpe, pkg name, version, url
> + echo '"$(1)","$(2)","$(3)","$(4)"' >>$(CPE_MANIFEST_CSV)
> +endef
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 2/9] cpe-info: id prefix/suffix
2019-03-08 22:04 ` [Buildroot] [PATCH v8 2/9] cpe-info: id prefix/suffix Matt Weber
@ 2019-04-14 14:53 ` Arnout Vandecappelle
0 siblings, 0 replies; 18+ messages in thread
From: Arnout Vandecappelle @ 2019-04-14 14:53 UTC (permalink / raw)
To: buildroot
On 08/03/2019 23:04, Matt Weber wrote:
> There are two types of software CPE prefixes applicable for software,
> one for applications and one for operating systems.
> Note: The third type is for hardware.
>
> This patchset determines which should be used and stores that
> information with the package for later use when assembling the CPE
> report.
>
> Refs:
> https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7695.pdf
> https://cpe.mitre.org/specification/
>
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> ---
> Changes
>
> v8
> - Rebase
>
> v4 -> v7
> - None
>
> v3
> [Arnout
> - Moved CPE prefix and suffix defines to package/Makefile.in
>
> v1 -> v2
> [Thomas P
> - Change to using a filter on pkg name value vs ifelse
> ---
> package/Makefile.in | 4 ++++
> package/pkg-generic.mk | 8 +++++++-
> 2 files changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/package/Makefile.in b/package/Makefile.in
> index dc818a2..644282f 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -389,6 +389,10 @@ TARGET_CONFIGURE_ARGS = \
>
> ################################################################################
>
> +CPE_PREFIX_OS = cpe:2.3:o
> +CPE_PREFIX_APP = cpe:2.3:a
> +CPE_SUFFIX = *:*:*:*:*:*:*
This is not correct. For Python packages, for instance, the language field is
set to python.
> +
> ifeq ($(BR2_SYSTEM_ENABLE_NLS),y)
> NLS_OPTS = --enable-nls
> TARGET_NLS_DEPENDENCIES = host-gettext
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 644128d..a547c65 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -916,11 +916,17 @@ $(2)_CPE_ID_NAME ?= $$($(2)_NAME)
> $(2)_CPE_ID_VERSION ?= $$($(2)_VERSION)
> $(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION)
>
> +ifneq ($(filter linux linux-headers,$(1)),)
> +$(2)_CPE_PREFIX = $(CPE_PREFIX_OS)
The linux and linux-headers packages are so special that I don't think it makes
sense to have a separate rule here. For example, the value of LINUX_VERSION is
probably NOT what you want to use in CPE_ID.
Regards,
Arnout
> +else
> +$(2)_CPE_PREFIX = $(CPE_PREFIX_APP)
> +endif
> +
> $(1)-cpe-info: PKG=$(2)
> $(1)-cpe-info:
> ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
> @$$(call MESSAGE,"Collecting cpe info")
> - $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
> + $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
> endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
>
> # legal-info: declare dependencies and set values used later for the manifest
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 5/9] toolchain/toolchain-ext: glibc cpe-info support
2019-03-08 22:04 ` [Buildroot] [PATCH v8 5/9] toolchain/toolchain-ext: glibc cpe-info support Matt Weber
@ 2019-04-14 15:27 ` Arnout Vandecappelle
0 siblings, 0 replies; 18+ messages in thread
From: Arnout Vandecappelle @ 2019-04-14 15:27 UTC (permalink / raw)
To: buildroot
On 08/03/2019 23:04, Matt Weber wrote:
> From: Shruthi Singh <shruthi.singh@rockwellcollins.com>
>
> This commit adds the correct CPE string for glibc, describing CPE ID,
> VERSION, PACKAGE NAME and URL.
>
> Signed-off-by: Shruthi Singh shruthi.singh at rockwellcollins.com
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
This handles just glibc. However, the toolchain consists of several components
that end up on the target: libgcc and libstdc++ coming from gcc,
glibc/uClibc/musl, linux-headers, gdbserver. So we might need to be able to
specify several CPE_IDs.
> ---
> Changes
>
> v8
> - No change
>
> v7
> - New
> ---
> package/pkg-generic.mk | 14 ++++++++++++++
> toolchain/toolchain-external/pkg-toolchain-external.mk | 7 +++++++
> 2 files changed, 21 insertions(+)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 1b895d0..f9cedd2 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -924,8 +924,22 @@ $(2)_CPE_PREFIX = $(CPE_PREFIX_APP)
> endif
>
> $(1)-cpe-info: PKG=$(2)
> +ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y)
> +$(1)-cpe-info: toolchain
> +endif
> $(1)-cpe-info:
> ifeq ($$($(2)_TYPE),target)
> +ifneq ($$($(2)_NAME),toolchain-external)
> +ifneq ($(findstring TOOLCHAIN_EXTERNAL, $(2)),)
> +ifeq ($(BR2_TOOLCHAIN_EXTERNAL_GLIBC),y)
> + $$(eval $(2)_VERSION = $$(shell $$(call TOOLCHAIN_CPE_INFO)))
That should have been _CPE_VERSION, obviously.
You don't need the $(call) construct because there are no arguments.
> + $$(eval $(2)_CPE_ID_VENDOR = gnu)
> + $$(eval $(2)_CPE_ID_NAME = glibc)
This kind of thing clearly does NOT belong in the generic package
infrastructure. It can easily be moved to pkg-toolchain-external.mk. This has
the additional advantage that it will go with the specific external toolchain.
> + $$(eval $(2)_ACTUAL_SOURCE_SITE = https://github.com/bminor/glibc/releases)
> + $$(eval $(2)_RAWNAME = glibc)
What are these two things doing here?
> +endif # ifeq ($(BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC),y)
> +endif # ifneq ($(findstring TOOLCHAIN_EXTERNAL, $(2)),)
> +endif # ifneq ($$($(2)_NAME),toolchain-external)
> ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
> @$$(call MESSAGE,"Collecting cpe info")
> $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE))
> diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
> index db3570d..aed06c5 100644
> --- a/toolchain/toolchain-external/pkg-toolchain-external.mk
> +++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
> @@ -440,6 +440,13 @@ define TOOLCHAIN_EXTERNAL_INSTALL_SYSROOT_LIBS
> $(call copy_toolchain_sysroot,$${SYSROOT_DIR},$${ARCH_SYSROOT_DIR},$${ARCH_SUBDIR},$${ARCH_LIB_DIR},$${SUPPORT_LIB_DIR})
> endef
>
> +define TOOLCHAIN_CPE_INFO
> + ARCH_SYSROOT_DIR="$(call toolchain_find_sysroot,$(TOOLCHAIN_EXTERNAL_CC) $(TOOLCHAIN_EXTERNAL_CFLAGS))" ; \
> + MAJ=`awk '{ if ($$1 = /#define/ && ($$2= /__GLIBC__/)){printf $$3};}' $${ARCH_SYSROOT_DIR}/usr/include/features.h` ; \
> + MIN=`awk '{ if ($$1 = /#define/ && ($$2 = /_GLIBC_MINOR/)){printf $$3};}' $${ARCH_SYSROOT_DIR}/usr/include/features.h` ; \
> + echo $${MAJ}.$${MIN}
This could be simplified into:
echo '__GLIBC__.__GLIBC_MINOR__' | \
cat $${ARCH_SYSROOT_DIR}/usr/include/features.h - | \
cpp -P | tr -d ' '
However, it brings up a new problem... This will only work if the toolchain has
been extracted already (or if it's a preinstalled toolchain, which is the same).
However, there is no dependency of cpe-info (or show-info) on
toolchain-external-foo-extract.
If that is anyway the case, we can simplify things even more by requiring the
toolchain to be installed. Then, we can use $(STAGING_DIR) instead of
$${ARCH_SYSROOT_DIR}.
But we should handle the case when the toolchain has not been
extracted/installed yet, and issue a warning (and return *) if that is the case.
So, this could become something like this (directly inside
inner-external-toolchain):
$(2)_GLIBC_VERSION = $$(if $$(wildcard $$(STAGING_DIR)/usr/include/features.h),\
$$(shell echo '__GLIBC__.__GLIBC_MINOR__' | \
cat $$(STAGING_DIR)/usr/include/features.h - | \
cpp -P | tr -d ' '),\
*$$(warning "External toolchain glibc version can only be detected after
toolchain installation"))
Please test both the case when it has and has not been extracted before
submitting :-)
Regards,
Arnout
> +endef
> +
> # Create a symlink from (usr/)$(ARCH_LIB_DIR) to lib.
> # Note: the skeleton package additionally creates lib32->lib or lib64->lib
> # (as appropriate)
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 6/9] cpe-info: update manual for new pkg vars
2019-03-08 22:04 ` [Buildroot] [PATCH v8 6/9] cpe-info: update manual for new pkg vars Matt Weber
@ 2019-04-14 15:32 ` Arnout Vandecappelle
2019-04-14 17:24 ` Thomas De Schampheleire
1 sibling, 0 replies; 18+ messages in thread
From: Arnout Vandecappelle @ 2019-04-14 15:32 UTC (permalink / raw)
To: buildroot
On 08/03/2019 23:04, Matt Weber wrote:
> +license text (+LIBFOO_LICENSE_FILES+) and the vendor for vunerability
> +analysis (+LIBFOO_CPE_ID_VENDOR+). All variables must start with
"for vulnerability analysis" is not entirely correct; it's for product
identification. And of course this would be a full CPE_ID not just a vendor. So
it should become something like
... and the CPE (Common Platform Enumeration) identifier.
There should be a reference to the cpe-info section as well, but of course that
can be added only after that section has been added.
Regards,
Arnout
^ permalink raw reply [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 6/9] cpe-info: update manual for new pkg vars
2019-03-08 22:04 ` [Buildroot] [PATCH v8 6/9] cpe-info: update manual for new pkg vars Matt Weber
2019-04-14 15:32 ` Arnout Vandecappelle
@ 2019-04-14 17:24 ` Thomas De Schampheleire
1 sibling, 0 replies; 18+ messages in thread
From: Thomas De Schampheleire @ 2019-04-14 17:24 UTC (permalink / raw)
To: buildroot
On Fri, Mar 8, 2019, 23:06 Matt Weber <matthew.weber@rockwellcollins.com>
wrote:
> Provide guidance on setting up the *_CPE_* and *_CVE_* variables.
>
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> ---
> Changes
>
> v8
> - Added note about minor version
> - Removed CVE patch listing as the backend has not be implemented
> (LIBFOO_CVE_PATCHED)
>
> v4 -> v7
> - None
>
> v3
> - Updated to make *_CPE_VENDOR optional
> - Changed wording around _CPE_ID as there is only one defined now
>
> v2
> [Thomas P
> - Reworded LIBFOO_CVE_PATCHED description
>
> [Matt W
> - Added definition for new preset variables to auto-gen the CPE ID
> - Added example LIBFOO_CPE_ID_VENDOR to LIBFOO
> ---
> docs/manual/adding-packages-generic.txt | 111
> +++++++++++++++++++-------------
> 1 file changed, 68 insertions(+), 43 deletions(-)
>
> diff --git a/docs/manual/adding-packages-generic.txt
> b/docs/manual/adding-packages-generic.txt
> index 7be1754..ddf1b2e 100644
> --- a/docs/manual/adding-packages-generic.txt
> +++ b/docs/manual/adding-packages-generic.txt
> @@ -24,57 +24,59 @@ system is based on hand-written Makefiles or shell
> scripts.
> 09: LIBFOO_SITE = http://www.foosoftware.org/download
> 10: LIBFOO_LICENSE = GPL-3.0+
> 11: LIBFOO_LICENSE_FILES = COPYING
> -12: LIBFOO_INSTALL_STAGING = YES
> -13: LIBFOO_CONFIG_SCRIPTS = libfoo-config
> -14: LIBFOO_DEPENDENCIES = host-libaaa libbbb
> -15:
> -16: define LIBFOO_BUILD_CMDS
> -17: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all
> -18: endef
> -19:
> -20: define LIBFOO_INSTALL_STAGING_CMDS
> -21: $(INSTALL) -D -m 0755 $(@D)/libfoo.a
> $(STAGING_DIR)/usr/lib/libfoo.a
> -22: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h
> -23: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib
> -24: endef
> -25:
> -26: define LIBFOO_INSTALL_TARGET_CMDS
> -27: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib
> -28: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d
> -29: endef
> -30:
> -31: define LIBFOO_USERS
> -32: foo -1 libfoo -1 * - - - LibFoo daemon
> -33: endef
> -34:
> -35: define LIBFOO_DEVICES
> -36: /dev/foo c 666 0 0 42 0 - - -
> -37: endef
> -38:
> -39: define LIBFOO_PERMISSIONS
> -40: /bin/foo f 4755 foo libfoo - - - - -
> -41: endef
> -42:
> -43: $(eval $(generic-package))
> +12: LIBFOO_CPE_ID_VENDOR = foosoftware
> +13: LIBFOO_INSTALL_STAGING = YES
> +14: LIBFOO_CONFIG_SCRIPTS = libfoo-config
> +15: LIBFOO_DEPENDENCIES = host-libaaa libbbb
> +16:
> +17: define LIBFOO_BUILD_CMDS
> +18: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all
> +19: endef
> +20:
> +21: define LIBFOO_INSTALL_STAGING_CMDS
> +22: $(INSTALL) -D -m 0755 $(@D)/libfoo.a
> $(STAGING_DIR)/usr/lib/libfoo.a
> +23: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h
> +24: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib
> +25: endef
> +26:
> +27: define LIBFOO_INSTALL_TARGET_CMDS
> +28: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib
> +29: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d
> +30: endef
> +31:
> +32: define LIBFOO_USERS
> +33: foo -1 libfoo -1 * - - - LibFoo daemon
> +34: endef
> +35:
> +36: define LIBFOO_DEVICES
> +37: /dev/foo c 666 0 0 42 0 - - -
> +38: endef
> +39:
> +40: define LIBFOO_PERMISSIONS
> +41: /bin/foo f 4755 foo libfoo - - - - -
> +42: endef
> +43:
> +44: $(eval $(generic-package))
> --------------------------------
>
> -The Makefile begins on line 7 to 11 with metadata information: the
> +The Makefile begins on line 7 to 12 with metadata information: the
> version of the package (+LIBFOO_VERSION+), the name of the
> tarball containing the package (+LIBFOO_SOURCE+) (xz-ed tarball
> recommended)
> the Internet location at which the tarball can be downloaded from
> -(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+) and file with the
> -license text (+LIBFOO_LICENSE_FILES+). All variables must start with
> +(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+), the file with the
> +license text (+LIBFOO_LICENSE_FILES+) and the vendor for vunerability
> +analysis (+LIBFOO_CPE_ID_VENDOR+). All variables must start with
> the same prefix, +LIBFOO_+ in this case. This prefix is always the
> uppercased version of the package name (see below to understand where
> the package name is defined).
>
> -On line 12, we specify that this package wants to install something to
> +On line 13, we specify that this package wants to install something to
> the staging space. This is often needed for libraries, since they must
> install header files and other development files in the staging space.
> This will ensure that the commands listed in the
> +LIBFOO_INSTALL_STAGING_CMDS+ variable will be executed.
>
> -On line 13, we specify that there is some fixing to be done to some
> +On line 14, we specify that there is some fixing to be done to some
> of the 'libfoo-config' files that were installed during
> +LIBFOO_INSTALL_STAGING_CMDS+ phase.
> These *-config files are executable shell script files that are
> @@ -122,14 +124,14 @@ IMAGEMAGICK_CONFIG_SCRIPTS = \
> --------------------------------
> ================================
>
> -On line 14, we specify the list of dependencies this package relies
> +On line 15, we specify the list of dependencies this package relies
> on. These dependencies are listed in terms of lower-case package names,
> which can be packages for the target (without the +host-+
> prefix) or packages for the host (with the +host-+) prefix).
> Buildroot will ensure that all these packages are built and installed
> 'before' the current package starts its configuration.
>
> -The rest of the Makefile, lines 16..29, defines what should be done
> +The rest of the Makefile, lines 17..29, defines what should be done
> at the different steps of the package configuration, compilation and
> installation.
> +LIBFOO_BUILD_CMDS+ tells what steps should be performed to
> @@ -142,16 +144,16 @@ All these steps rely on the +$(@D)+ variable, which
> contains the directory where the source code of the package has been
> extracted.
>
> -On lines 31..43, we define a user that is used by this package (e.g.
> +On lines 32..44, we define a user that is used by this package (e.g.
> to run a daemon as non-root) (+LIBFOO_USERS+).
>
> -On line 35..37, we define a device-node file used by this package
> +On line 36..38, we define a device-node file used by this package
> (+LIBFOO_DEVICES+).
>
> -On line 39..41, we define the permissions to set to specific files
> +On line 40..42, we define the permissions to set to specific files
> installed by this package (+LIBFOO_PERMISSIONS+).
>
> -Finally, on line 43, we call the +generic-package+ function, which
> +Finally, on line 44, we call the +generic-package+ function, which
> generates, according to the variables defined previously, all the
> Makefile code necessary to make your package working.
>
> @@ -482,6 +484,29 @@ not and can not work as people would expect it should:
> locations, `/lib/firmware`, `/usr/lib/firmware`, `/lib/modules`,
> `/usr/lib/modules`, and `/usr/share`, which are automatically excluded.
>
> +* +LIBFOO_CPE_ID_VENDOR+
> + This variable is optional. It only must be defined if the package name
> + does not match what the CPE ID uses for the vendor. By default it's set
> + to <pkg-name>_project.
> +
> +* +LIBFOO_CPE_ID_NAME+
> + This variable is optional. It only must be defined if the package name
> + does not match what the CPE ID uses for the name. By default it's set
> + to <pkg-name>.
> +
> +* +LIBFOO_CPE_ID_VERSION+
> + This variable is optional. By default it's set to <pkg-version>.
> +
> +* +LIBFOO_CPE_ID_VERSION_MINOR+
> + This variable is optional. By default it's set to *.
> +
> +* +LIBFOO_CPE_ID+ is optional, as the package infrastructure hangles the
> + default case of a single package's Common Product Enumeration (CPE)
> + identification string. +make cpe-info+ copies all of these into a
> + +cpe-manifest.csv+ file. To identify a package's possible CPE,
> + the National Vunerability Database can be searched at
>
Also here: Vulnerability with l
+ https://nvd.nist.gov/products/cpe/search.
> +
> The recommended way to define these variables is to use the following
> syntax:
>
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190414/a74e7d61/attachment.html>
^ permalink raw reply [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 7/9] support/scripts/cpedb.py: new CPE XML helper
2019-03-08 22:04 ` [Buildroot] [PATCH v8 7/9] support/scripts/cpedb.py: new CPE XML helper Matt Weber
@ 2019-04-14 18:05 ` Arnout Vandecappelle
0 siblings, 0 replies; 18+ messages in thread
From: Arnout Vandecappelle @ 2019-04-14 18:05 UTC (permalink / raw)
To: buildroot
On 08/03/2019 23:04, Matt Weber wrote:
> Python class which consumes a NIST CPE XML and provides helper
> functions to access and search the db's data.
>
> - Defines the CPE as a object with operations / formats
> - Processing of CPE dictionary
>
> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
> ---
>
> v8
> - Added support for generation of update xml to maintain the
> NIST dictionary for any Buildroot package version bumps
> - Dropped searching of the Config.in files for URLs, instead
> assuming the first time a package is added to NIST, the xml is
> manually filled out with reference urls. Any updates to versions
> after that will use the proposed autogen xml that mines the URLS
> from the NIST dict file.
> - Caching support for a processed dictionary to speed up subsequent
> runs when testing, as a db doesn't update more then once a day
Can we split this into three patches:
1. basic CPEDB and cpe-report.
2. Add caching
3. Add XML generation.
For the cache, it's worth mentioning how much you save be storing the pickled
object rather than the XML.
>
> v5 -> v7
> - No change
>
> v5
> [Ricardo
> - Fixed typo in join/split of cpe str without version
> - Removed extra prints as they aren't needed when we have the
> output reports/stdout
> - Updated v4 comments about general flake formatting cleanup
> - Incorporated parts of patch 1/2 suggestions for optimizations
>
> [Arnout
> - added pre-processing of cpe values into two sets, one with
> and one without version
> - Collectly with Ricardo, decided to move cpe class to this
> seperate script
>
> v1 -> v4
> - No version
> ---
> support/scripts/cpedb.py | 185 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 185 insertions(+)
> create mode 100644 support/scripts/cpedb.py
>
> diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py
> new file mode 100644
> index 0000000..0369536
> --- /dev/null
> +++ b/support/scripts/cpedb.py
> @@ -0,0 +1,185 @@
> +import sys
> +import urllib2
For python3, this should be urllib.request. So:
try:
import urllib.request as urllib
except ImportError:
import urllib2 as urllib
> +from collections import OrderedDict
> +import xmltodict
This is not a standard module, so maybe we should add an exception handler (in
cpe-report) that says which non-standard packages must be installed.
> +import gzip
> +from StringIO import StringIO
> +import os
> +import pickle
> +
> +VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
> +
> +
> +class CPE:
CPE is a bit too generic. CPEEntry would be better.
> + cpe_str = None
> + cpe_str_short = None
> + cpe_desc = None
> + cpe_cur_ver = None
> + titles = {}
> + references = {}
> +
> + def __init__(self, cpe_str, titles=None, refs=None):
> + self.cpe_str = cpe_str
> + self.cpe_str_short = ":".join(self.cpe_str.split(":")[:6])
Generating the cpe-2.2 name is actually a little bit more complicated than
that, so I would pass it explicitly in the constructor as cpe_2_2_str (instead
of str_short). Or, to keep it consistent with the database, cpe_name for the 2.2
string, and cpe_23_name for the 2.3 string.
> + self.titles = titles
> + self.references = refs
> + self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
This is just
self.cpe_cur_ver = self.cpe_str.split(":")[5]
> +
> + def to_dict(self, cpe_str):
Why pass a cpe_str here? I would expect to convert self to a dict. That way,
most of the below can be removed (after setting cpe_cur_ver to a new version in
the object, of course).
> + cpe_short_name = ":".join(cpe_str.split(":")[2:6])
> + cpe_new_ver = "".join(cpe_str.split(":")[5:6])
> + self.titles[0]['#text'] = self.titles[0]['#text'].replace(self.cpe_cur_ver, cpe_new_ver)
> + cpe_dict = OrderedDict([
> + ('cpe-item', OrderedDict([
> + ('@name', 'cpe:/' + cpe_short_name),
> + ('title', self.titles),
> + ('references', OrderedDict([('reference', self.references)])),
> + ('cpe-23:cpe23-item', OrderedDict([
> + ('@name', cpe_str)
> + ]))
> + ]))
> + ])
> + return cpe_dict
> +
> +
> +class CPEDB:
They call it the dictionary, so maybe CPEDictionary is better.
> + all_cpes = dict()
> + all_cpes_no_version = dict()
> +
> + def get_xml_dict(self, url):
> + print("CPE: Setting up NIST dictionary")
> + # Setup location to save dict and xmls, if it exists, assume we're
> + # reusing the previous dict
> + if not os.path.exists("cpe"):
The cache path should be settable with a command line option. You'll probably
want to share the cache between different buildroot trees.
> + os.makedirs("cpe")
> + self.get_new_xml_dict(url)
> + else:
> + print("CPE: Loading CACHED dictionary")
> + cpe_file = open('cpe/.all_cpes.pkl', 'rb')
> + self.all_cpes = pickle.load(cpe_file)
> + cpe_file.close()
> + cpe_file = open('cpe/.all_cpes_no_version.pkl', 'rb')
> + self.all_cpes_no_version = pickle.load(cpe_file)
> + cpe_file.close()
> +
> + def get_new_xml_dict(self, url):
> + print("CPE: Fetching xml manifest from [" + url + "]")
> + try:
> + compressed_cpe_file = urllib2.urlopen(url)
> + print("CPE: Unzipping xml manifest...")
I believe this is not correct, the unzipping will only start when you read from
it in the xmltodict.parse() call.
> + nist_cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read()
Is all this StringIO stuff needed? Why not just
nist_cpe_file = gzip.GzipFile(fileobj=compressed_cpe_file)
?
> + print("CPE: Converting xml manifest to dict...")
> + all_cpedb = xmltodict.parse(nist_cpe_file)
FYI, this bit takes about 7 seconds on my laptop.
> +
> + # Cycle through the dict and build two dict to be used for custom
> + # lookups of partial and complete CPE objects
> + # The objects are then used to create new proposed XML updates if
> + # if is determined one is required
> + for cpe in all_cpedb['cpe-list']['cpe-item']:
> + cpe_titles = cpe['title']
> + # There maybe multiple titles or one. Make sure this is
> + # always a list
> + if not isinstance(cpe_titles, (list,)):
> + cpe_titles = [cpe_titles]
> + # Out of the different language titles, select English
> + for title in cpe_titles:
> + if title['@xml:lang'] is 'en-US':
> + cpe_titles = [title]
You'll probably want to break here.
> + # Some older CPE don't include references, if they do, make
> + # sure we handle the case of one ref needing to be packed
> + # in a list
> + if 'references' in cpe:
> + cpe_ref = cpe['references']['reference']
> + if not isinstance(cpe_ref, (list,)):
> + cpe_ref = [cpe_ref]
> + # The reference text has not been consistantly upper case
To be completely exact: it is NEVER uppercase. A lot of the existing references
are some free text. And AFAICS, the official labels are all title-cased
(Advisory, Version, ...). Where do you get this information that it should be
uppercase?
> + # in the NIST dict but they now require it. So force upper
> + # and then check for compliance to a specific tagging
> + for ref_href in cpe_ref:
> + ref_href['#text'] = ref_href['#text'].upper()
> + if ref_href['#text'] not in VALID_REFS:
> + ref_href['#text'] = ref_href['#text'] + "-- UPDATE this entry, here are some exmaples and just one word should be used -- " + ' '.join(VALID_REFS)
I think if a current entry contains some invalid ref, we should just drop it.
However, IMO any updates like this (dropping invalid refs, uppercasing them,
...) should be done only when the version is updated, not when the CPEDB is created.
In other words, I think the CPEEntry should get a method create_new_version
that creates a new CPEEntry (or maybe a duck-typed similar class) with the new
version.
> + cpe_str = cpe['cpe-23:cpe23-item']['@name']
> + item = CPE(cpe_str, cpe_titles, cpe_ref)
> + cpe_str_no_version = self.get_cpe_no_version(cpe_str)
> + # This dict must have a unique key for every CPE version
> + # which allows matching to the specific obj data of that
> + # NIST dict entry
> + self.all_cpes.update({cpe_str: item})
> + # This dict has one entry for every CPE (w/o version) to allow
> + # partial match (no valid version) check (the obj is saved and
> + # used as seed for suggested xml updates. By updating the same
> + # non-version'd entry, it assumes the last update here is the
> + # latest version in the NIST dict)
> + self.all_cpes_no_version.update({cpe_str_no_version: item})
> +
> + except urllib2.HTTPError:
> + print("CPE: HTTP Error: %s" % url)
> + sys.exit(1)
> + except urllib2.URLError:
> + print("CPE: URL Error: %s" % url)
> + sys.exit(1)
> +
> + print("CPE: Caching dictionary")
> + cpes_file = open('cpe/.all_cpes.pkl', 'wb')
I'm a bit afraid of this pickling, because it risks getting inconsistent if you
update this file. Could we put some versioning mechanism in it? Maybe just in
the file name?
> + pickle.dump(self.all_cpes, cpes_file)
> + cpes_file.close()
> + cpes_file = open('cpe/.all_cpes_no_version.pkl', 'wb')
> + pickle.dump(self.all_cpes_no_version, cpes_file)
> + cpes_file.close()
> +
> + def find_partial(self, cpe_str):
> + cpe_str_no_version = self.get_cpe_no_version(cpe_str)
> + if cpe_str_no_version in self.all_cpes_no_version:
> + return cpe_str_no_version
This will not fly with my idea of defaulting vendor to *...
Actually, I realize now: if we require setting a full CPE_ID rather than the
vendor, product, version fields, and if it would be part of show-info rather
than cpe-info, then it may be better to leave the CPE_ID empty by default, and
let cpe-report do the lookup. The cpe-report script would then be responsible
for constructing a guessed CPE_ID, and it has more flexibility to do that based
on package name and version. For example, it can try with the package name and
it can try to strip the perl-/python-/... prefix.
To make that easy, it would be better to make the all_cpes_no_version keyed on
product only instead of keyed on generated CPE string.
> +
> + def find_partial_obj(self, cpe_str):
> + cpe_str_no_version = self.get_cpe_no_version(cpe_str)
> + if cpe_str_no_version in self.all_cpes_no_version:
> + return self.all_cpes_no_version[cpe_str_no_version]
> +
> + def find_partial_latest_version(self, cpe_str_partial):
> + cpe_obj = self.find_partial_obj(cpe_str_partial)
> + return cpe_obj.cpe_cur_ver
> +
> + def find(self, cpe_str):
> + if self.find_partial(cpe_str):
> + if cpe_str in self.all_cpes:
> + return cpe_str
> +
> + def update(self, cpe_str):
> + to_update = self.find_partial_obj(cpe_str)
> + xml = self.__gen_xml__(to_update.to_dict(cpe_str))
> + fp = open(os.path.join('cpe', self.get_cpe_name(cpe_str) + '-' + self.get_cpe_version(cpe_str) + '.xml'), 'w+')
> + fp.write(xmltodict.unparse(xml, pretty=True))
> + fp.close()
> +
> + def get_nvd_url(self, cpe_str):
This function is not used AFAICS, and probably a few others as well. Please
remove them.
> + return "https://nvd.nist.gov/products/cpe/search/results?keyword=" + \
> + urllib2.quote(cpe_str) + \
> + "&status=FINAL&orderBy=CPEURI&namingFormat=2.3"
> +
> + def get_cpe_no_version(self, cpe):
> + return ":".join(cpe.split(":")[:5])
> +
> + def get_cpe_name(self, cpe_str):
> + return "".join(cpe_str.split(":")[4])
> +
> + def get_cpe_version(self, cpe_str):
> + return "".join(cpe_str.split(":")[5])
> +
> + def __gen_xml__(self, cpe_list):
This function is only used internally, right? Then don't give it a dunder name.
dunder names should be reserved for official Python semantics. You can indicate
that it is private by starting it with underscore, so _gen_xml().
Regards,
Arnout
> + list_header = {
> + "cpe-list": {
> + "@xmlns:config": "http://scap.nist.gov/schema/configuration/0.1",
> + "@xmlns": "http://cpe.mitre.org/dictionary/2.0",
> + "@xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
> + "@xmlnsscap-core": "http://scap.nist.gov/schema/scap-core/0.3",
> + "@xmlns:cpe-23": "http://scap.nist.gov/schema/cpe-extension/2.3",
> + "@xmlns:ns6": "http://scap.nist.gov/schema/scap-core/0.1",
> + "@xmlns:meta": "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
> + "@xsi:schemaLocation": "http://scap.nist.gov/schema/cpe-extension/2.3 https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd http://cpe.mitre.org/dictionary/2.0 https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2 https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd http://scap.nist.gov/schema/scap-core/0.3 https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd http://scap.nist.gov/schema/configuration/0.1 https://scap.nist.gov/schema/nvd/configuration_0.1.xsd http://scap.nist.gov/schema/scap-core/0.1 https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"
> + }
> + }
> + list_header['cpe-list'].update(cpe_list)
> + return list_header
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* [Buildroot] [PATCH v8 0/9] Package CPE Reporting
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
` (8 preceding siblings ...)
2019-03-08 22:04 ` [Buildroot] [PATCH v8 9/9] docs/manual: new security management section Matt Weber
@ 2019-04-14 18:08 ` Arnout Vandecappelle
9 siblings, 0 replies; 18+ messages in thread
From: Arnout Vandecappelle @ 2019-04-14 18:08 UTC (permalink / raw)
To: buildroot
On 08/03/2019 23:04, Matt Weber wrote:
> Some existing
> fix-ups forthe CPE strings can be found in the following commit that
> eventually needs to be split up and applied.
>
> https://github.com/rc-matthew-l-weber/buildroot/commit/8ce8d4740b95672d8390799b611a35ea18a543e0
I don't really think it needs to be split up. Only if it would touch more than
100 packages it might be better to split it up in chunks of 50-ish packages.
Regards,
Arnout
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2019-04-14 18:08 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-08 22:04 [Buildroot] [PATCH v8 0/9] Package CPE Reporting Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 1/9] cpe-info: new make target Matt Weber
2019-04-14 14:49 ` Arnout Vandecappelle
2019-03-08 22:04 ` [Buildroot] [PATCH v8 2/9] cpe-info: id prefix/suffix Matt Weber
2019-04-14 14:53 ` Arnout Vandecappelle
2019-03-08 22:04 ` [Buildroot] [PATCH v8 3/9] cpe-info: only report target pkgs Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 4/9] cpe-info: cpe minor version support Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 5/9] toolchain/toolchain-ext: glibc cpe-info support Matt Weber
2019-04-14 15:27 ` Arnout Vandecappelle
2019-03-08 22:04 ` [Buildroot] [PATCH v8 6/9] cpe-info: update manual for new pkg vars Matt Weber
2019-04-14 15:32 ` Arnout Vandecappelle
2019-04-14 17:24 ` Thomas De Schampheleire
2019-03-08 22:04 ` [Buildroot] [PATCH v8 7/9] support/scripts/cpedb.py: new CPE XML helper Matt Weber
2019-04-14 18:05 ` Arnout Vandecappelle
2019-03-08 22:04 ` [Buildroot] [PATCH v8 8/9] support/scripts/cpe-report: new script Matt Weber
2019-03-08 22:04 ` [Buildroot] [PATCH v8 9/9] docs/manual: new security management section Matt Weber
2019-04-14 14:39 ` Arnout Vandecappelle
2019-04-14 18:08 ` [Buildroot] [PATCH v8 0/9] Package CPE Reporting Arnout Vandecappelle
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.