[Buildroot] [PATCH 01/16 v2] core/show-info: report build_dir relative to CONFIG_DIR

[Buildroot] [PATCH 01/16 v2] core/show-info: report build_dir relative to CONFIG_DIR

[Yann E. MORIN]

Currently, the build_dir field is reported relative to $(BASE_DIR), to
avoid leaking local paths.

However, BASE_DIR is not a directory that is very convenient: for
in-tree builds, it is $(CONFIG_DIR)/output/, while for out-of-tree
builds, it is $(CONFIG_DIR). This difference is purely an idiosyncracy
of how out-of-tree builds have been implemented in Buildroot, and is
not under the control of the user.

What the user is in control of, however, is where the .config file is
located. This, really, is the directory we should base relative paths
on.

Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 package/pkg-utils.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index ae3c7f9da9..b2f0b8cebf 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -113,7 +113,7 @@ define _json-info-pkg
 		"virtual": false$(comma)
 		$(call _json-info-pkg-details,$(1)) \
 	)
-	"build_dir": "$(patsubst $(BASE_DIR)/%,%,$($(1)_BUILDDIR))",
+	"build_dir": "$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))",
 	$(if $(filter target,$($(1)_TYPE)), \
 		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
 		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 02/16 v2] package/kodi-*: run pkg-config at build time, not at parse time

[Yann E. MORIN]

When calling 'printvars' on a just-configured tree, the output contains
many spurious warnings about pkg-config being not found:

    make[1]: [...]/host/bin/pkg-config: Command not found

This is partly because a few packages call pkg-config at the time the
Makefile is parsed (really, right when make evaluates the recipe before
executing it).

It is customary, instead, to defer the call to pkg-config to the actual
shell running the recipe's commands, like all our other packages do.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Bernd Kuhls <bernd.kuhls@t-online.de>
---
 .../kodi-screensaver-asterwave/kodi-screensaver-asterwave.mk  | 4 ++--
 package/kodi-screensaver-cpblobs/kodi-screensaver-cpblobs.mk  | 4 ++--
 .../kodi-screensaver-matrixtrails.mk                          | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/package/kodi-screensaver-asterwave/kodi-screensaver-asterwave.mk b/package/kodi-screensaver-asterwave/kodi-screensaver-asterwave.mk
index b78646b9a9..21def8ca4c 100644
--- a/package/kodi-screensaver-asterwave/kodi-screensaver-asterwave.mk
+++ b/package/kodi-screensaver-asterwave/kodi-screensaver-asterwave.mk
@@ -11,7 +11,7 @@ KODI_SCREENSAVER_ASTERWAVE_LICENSE_FILES = LICENSE.md
 KODI_SCREENSAVER_ASTERWAVE_DEPENDENCIES = glm kodi
 
 KODI_SCREENSAVER_ASTERWAVE_CONF_OPTS += \
-	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)" \
-	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)"
+	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`" \
+	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`"
 
 $(eval $(cmake-package))
diff --git a/package/kodi-screensaver-cpblobs/kodi-screensaver-cpblobs.mk b/package/kodi-screensaver-cpblobs/kodi-screensaver-cpblobs.mk
index c6dd26a89b..6f462e7d27 100644
--- a/package/kodi-screensaver-cpblobs/kodi-screensaver-cpblobs.mk
+++ b/package/kodi-screensaver-cpblobs/kodi-screensaver-cpblobs.mk
@@ -11,7 +11,7 @@ KODI_SCREENSAVER_CPBLOBS_LICENSE_FILES = LICENSE.md
 KODI_SCREENSAVER_CPBLOBS_DEPENDENCIES = glm kodi
 
 KODI_SCREENSAVER_CPBLOBS_CONF_OPTS += \
-	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)" \
-	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)"
+	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`" \
+	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`"
 
 $(eval $(cmake-package))
diff --git a/package/kodi-screensaver-matrixtrails/kodi-screensaver-matrixtrails.mk b/package/kodi-screensaver-matrixtrails/kodi-screensaver-matrixtrails.mk
index ac4303a719..45d3e46b4a 100644
--- a/package/kodi-screensaver-matrixtrails/kodi-screensaver-matrixtrails.mk
+++ b/package/kodi-screensaver-matrixtrails/kodi-screensaver-matrixtrails.mk
@@ -11,7 +11,7 @@ KODI_SCREENSAVER_MATRIXTRAILS_LICENSE_FILES = LICENSE.md
 KODI_SCREENSAVER_MATRIXTRAILS_DEPENDENCIES = kodi
 
 KODI_SCREENSAVER_MATRIXTRAILS_CONF_OPTS += \
-	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)" \
-	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)"
+	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`" \
+	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`"
 
 $(eval $(cmake-package))
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 03/16 v2] support/dependencies: avoid spurious warning on print-vars

[Yann E. MORIN]

When calling 'printvars', the 'suitable-host-package' macro is printed
(a macro is just a variable like the others, after all, just with some
parameters). Because it is printed as a variable, it is missing its
parameters, but it still tries to evaluate the $(shell) construct.

This causes spurious warning:

    make[1]: support/dependencies/check-host-.sh: Command not found

Only try and call the script if there is actually a tool to check for.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 support/dependencies/dependencies.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/dependencies/dependencies.mk b/support/dependencies/dependencies.mk
index 4fac5c731b..a7bb1a6080 100644
--- a/support/dependencies/dependencies.mk
+++ b/support/dependencies/dependencies.mk
@@ -15,7 +15,7 @@ else
 # script should use 'which' to find a candidate. The script should return
 # the path to the suitable host tool, or nothing if no suitable tool was found.
 define suitable-host-package
-$(shell support/dependencies/check-host-$(1).sh $(2))
+$(if $(1),$(shell support/dependencies/check-host-$(1).sh $(2)))
 endef
 endif
 # host utilities needs host-tar to extract the source code tarballs, so
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 04/16 v2] package/infras: do not conditionally set empty variables

[Yann E. MORIN]

Setting an unset variable to an empty value is useless in make; an unset
variable just expands to an empty string anyway. So what we do currently
has no side effect:

    variable set and not empty   -> variable not modified
    variable set and empty       -> variable not modified
    variable unset               -> set to an empty string

However, additional variables do have an impact on the parsing time of
the Makefiles, and the more variables, the more collisions in the hash
table used internally by make, which slows down the parsing.

By dropping those conditionally-set-empty variables, we gain about 3%:

    Run     Before  After
    1       5.572   5.325
    2       5.434   5.354
    3       5.490   5.320
    4       5.525   5.330
    5       5.476   5.330
    6       5.511   5.434
    7       5.498   5.388
    8       5.524   5.371
    9       5.479   5.346
    10      5.637   5.324
    Mean:   5.515   5.352

Yeah, 0.163s does not look like much, and this does not make
autocompletion any more usable. Still, that 3% gain is not to be
ashamed of either.

Note that there are 3 others case where we do set empty variables, but
those are unconditional and serve other purposes:

  - pkg-virtual: this is done on purpose to avoid a bug when the
    environment may have TOOLCHAIN_VERSION or _SOURCE set, and we really
    want those to be empty, so the assignment is not conditional;

  - pkg-python: the reason for setting those to empty is dubious at
    best; it's been there since the inception of the python infra, back
    in 2013; still, the case is different than this patch addresses;

  - pkg-toolchain-external: this is the case for a toolchain already
    installed, so indeed we want to set _SOURCE and _VERSION to empty.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-autotools.mk |  4 ----
 package/pkg-cmake.mk     |  4 ----
 package/pkg-generic.mk   | 27 ---------------------------
 package/pkg-kconfig.mk   |  3 ---
 package/pkg-meson.mk     |  4 ----
 package/pkg-python.mk    |  4 ----
 package/pkg-qmake.mk     |  4 ----
 package/pkg-waf.mk       |  5 -----
 8 files changed, 55 deletions(-)

diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
index daa688dab6..717ff81593 100644
--- a/package/pkg-autotools.mk
+++ b/package/pkg-autotools.mk
@@ -160,10 +160,6 @@ ifeq ($(4),host)
  $(2)_AUTORECONF_OPTS ?= $$($(3)_AUTORECONF_OPTS)
 endif
 
-$(2)_CONF_ENV			?=
-$(2)_CONF_OPTS			?=
-$(2)_MAKE_ENV			?=
-$(2)_MAKE_OPTS			?=
 $(2)_INSTALL_OPTS                ?= install
 $(2)_INSTALL_STAGING_OPTS	?= DESTDIR=$$(STAGING_DIR) install
 $(2)_INSTALL_TARGET_OPTS		?= DESTDIR=$$(TARGET_DIR) install
diff --git a/package/pkg-cmake.mk b/package/pkg-cmake.mk
index 4ee100a0c6..3b1db35fb6 100644
--- a/package/pkg-cmake.mk
+++ b/package/pkg-cmake.mk
@@ -51,11 +51,7 @@ endif
 
 define inner-cmake-package
 
-$(2)_CONF_ENV			?=
-$(2)_CONF_OPTS			?=
 $(2)_MAKE			?= $$(MAKE)
-$(2)_MAKE_ENV			?=
-$(2)_MAKE_OPTS			?=
 $(2)_INSTALL_OPTS		?= install
 $(2)_INSTALL_STAGING_OPTS	?= DESTDIR=$$(STAGING_DIR) install/fast
 $(2)_INSTALL_TARGET_OPTS	?= DESTDIR=$$(TARGET_DIR) install/fast
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index ded5176428..ab48240283 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -538,8 +538,6 @@ $(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASENAME)
 ifndef $(2)_SUBDIR
  ifdef $(3)_SUBDIR
   $(2)_SUBDIR = $$($(3)_SUBDIR)
- else
-  $(2)_SUBDIR ?=
  endif
 endif
 
@@ -836,31 +834,6 @@ $(2)_EXTRACT_CMDS ?= \
 		$$(TAR_OPTIONS) -)
 
 # pre/post-steps hooks
-$(2)_PRE_DOWNLOAD_HOOKS         ?=
-$(2)_POST_DOWNLOAD_HOOKS        ?=
-$(2)_PRE_EXTRACT_HOOKS          ?=
-$(2)_POST_EXTRACT_HOOKS         ?=
-$(2)_PRE_RSYNC_HOOKS            ?=
-$(2)_POST_RSYNC_HOOKS           ?=
-$(2)_PRE_PATCH_HOOKS            ?=
-$(2)_POST_PATCH_HOOKS           ?=
-$(2)_PRE_CONFIGURE_HOOKS        ?=
-$(2)_POST_CONFIGURE_HOOKS       ?=
-$(2)_PRE_BUILD_HOOKS            ?=
-$(2)_POST_BUILD_HOOKS           ?=
-$(2)_PRE_INSTALL_HOOKS          ?=
-$(2)_POST_INSTALL_HOOKS         ?=
-$(2)_PRE_INSTALL_STAGING_HOOKS  ?=
-$(2)_POST_INSTALL_STAGING_HOOKS ?=
-$(2)_PRE_INSTALL_TARGET_HOOKS   ?=
-$(2)_POST_INSTALL_TARGET_HOOKS  ?=
-$(2)_PRE_INSTALL_IMAGES_HOOKS   ?=
-$(2)_POST_INSTALL_IMAGES_HOOKS  ?=
-$(2)_PRE_LEGAL_INFO_HOOKS       ?=
-$(2)_POST_LEGAL_INFO_HOOKS      ?=
-$(2)_TARGET_FINALIZE_HOOKS      ?=
-$(2)_ROOTFS_PRE_CMD_HOOKS       ?=
-
 $(2)_POST_PREPARE_HOOKS += FIXUP_PYTHON_SYSCONFIGDATA
 
 ifeq ($$($(2)_TYPE),target)
diff --git a/package/pkg-kconfig.mk b/package/pkg-kconfig.mk
index 23a291fccf..32dcfea0bc 100644
--- a/package/pkg-kconfig.mk
+++ b/package/pkg-kconfig.mk
@@ -81,9 +81,6 @@ define inner-kconfig-package
 # Default values
 $(2)_MAKE ?= $$(MAKE)
 $(2)_KCONFIG_EDITORS ?= menuconfig
-$(2)_KCONFIG_OPTS ?=
-$(2)_KCONFIG_FIXUP_CMDS ?=
-$(2)_KCONFIG_FRAGMENT_FILES ?=
 $(2)_KCONFIG_DOTCONFIG ?= .config
 $(2)_KCONFIG_SUPPORTS_DEFCONFIG ?= YES
 
diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
index 16a90d157b..ec535174b6 100644
--- a/package/pkg-meson.mk
+++ b/package/pkg-meson.mk
@@ -45,10 +45,6 @@ NINJA_OPTS	= $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
 define inner-meson-package
 
-$(2)_CONF_ENV		?=
-$(2)_CONF_OPTS		?=
-$(2)_NINJA_ENV		?=
-
 #
 # Configure step. Only define it if not already defined by the package
 # .mk file. And take care of the differences between host and target
diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index e6b81bdfd3..ce0fc97283 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -108,10 +108,6 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
 
 define inner-python-package
 
-$(2)_ENV         ?=
-$(2)_BUILD_OPTS   ?=
-$(2)_INSTALL_OPTS ?=
-
 ifndef $(2)_SETUP_TYPE
  ifdef $(3)_SETUP_TYPE
   $(2)_SETUP_TYPE = $$($(3)_SETUP_TYPE)
diff --git a/package/pkg-qmake.mk b/package/pkg-qmake.mk
index deb033c1d6..4dcaf6dd62 100644
--- a/package/pkg-qmake.mk
+++ b/package/pkg-qmake.mk
@@ -40,10 +40,6 @@ endef
 
 define inner-qmake-package
 
-$(2)_CONF_ENV			?=
-$(2)_CONF_OPTS			?=
-$(2)_MAKE_ENV			?=
-$(2)_MAKE_OPTS			?=
 $(2)_INSTALL_STAGING_OPTS	?= install
 $(2)_INSTALL_TARGET_OPTS	?= $$($(2)_INSTALL_STAGING_OPTS)
 
diff --git a/package/pkg-waf.mk b/package/pkg-waf.mk
index 038d89454e..a593476a69 100644
--- a/package/pkg-waf.mk
+++ b/package/pkg-waf.mk
@@ -49,11 +49,6 @@ else
 $(2)_WAF ?= ./waf
 endif
 
-$(2)_BUILD_OPTS				?=
-$(2)_INSTALL_STAGING_OPTS		?=
-$(2)_INSTALL_TARGET_OPTS		?=
-$(2)_WAF_OPTS				?=
-
 #
 # Configure step. Only define it if not already defined by the package
 # .mk file.
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 05/16 v2] package/pkg-python: do not set empty variables

[Yann E. MORIN]

Similarly to what we just did for conditionally-set-empty variables,
unconditionally setting empty variables serves no purpose in Makefiles,
as unset variables are just expanded as empty...

Even though we have numerous python packages, the speed gain was not
measurable, with delta much less than the noise.

Still, for consistency, we just do not set those variables.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-python.mk | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index ce0fc97283..4cc416cea0 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -127,7 +127,6 @@ $(2)_BASE_INSTALL_STAGING_OPTS = $$(PKG_PYTHON_DISTUTILS_INSTALL_STAGING_OPTS)
 else
 $(2)_BASE_ENV         = $$(HOST_PKG_PYTHON_DISTUTILS_ENV)
 $(2)_BASE_BUILD_TGT   = build
-$(2)_BASE_BUILD_OPTS   =
 $(2)_BASE_INSTALL_OPTS = $$(HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPTS)
 endif
 # Setuptools
@@ -135,13 +134,11 @@ else ifeq ($$($(2)_SETUP_TYPE),setuptools)
 ifeq ($(4),target)
 $(2)_BASE_ENV         = $$(PKG_PYTHON_SETUPTOOLS_ENV)
 $(2)_BASE_BUILD_TGT   = build
-$(2)_BASE_BUILD_OPTS   =
 $(2)_BASE_INSTALL_TARGET_OPTS  = $$(PKG_PYTHON_SETUPTOOLS_INSTALL_TARGET_OPTS)
 $(2)_BASE_INSTALL_STAGING_OPTS = $$(PKG_PYTHON_SETUPTOOLS_INSTALL_STAGING_OPTS)
 else
 $(2)_BASE_ENV         = $$(HOST_PKG_PYTHON_SETUPTOOLS_ENV)
 $(2)_BASE_BUILD_TGT   = build
-$(2)_BASE_BUILD_OPTS   =
 $(2)_BASE_INSTALL_OPTS = $$(HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS)
 endif
 else
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 06/16 v2] core/pkg-generic: also list files installed in images/

[Yann E. MORIN]

Some packages install nothing in target nor staging, but install images
(like the kernel vmlinux, or a bootloader boot blob...)

If we want to appropriately account for the files installed by each
package, we also need to take images/ into account.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Herve Codina <herve.codina@bootlin.com>

---
Changes v1 -> v2:
  - fix typo in commit log (Hervé)
---
 package/pkg-generic.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index ab48240283..5626af1d87 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -276,6 +276,7 @@ $(BUILD_DIR)/%/.stamp_configured:
 	$(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES))
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
+	@$(call pkg_size_before,$(BINARIES_DIR),-images)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
 	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
@@ -402,6 +403,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(TARGET_DIR))
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
+	@$(call pkg_size_after,$(BINARIES_DIR),-images)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
 	$(Q)touch $@
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 07/16 v2] Makefile: really comment syntax colouring

[Yann E. MORIN]

The unmatched escaped single-quote lies in the middle of a few
function calls, so they too must be fake-closed to properly fix
colour highlighting in some editors.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 4062883cc2..6f6bb72de3 100644
--- a/Makefile
+++ b/Makefile
@@ -1068,7 +1068,7 @@ printvars:
 		$(if $(QUOTED_VARS),\
 			$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
 			$(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))
-# ' Syntax colouring...
+# ')))) # Syntax colouring...
 
 .PHONY: clean
 clean:
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 08/16 v2] core/show-info: 'name' only applies to packages

[Yann E. MORIN]

Commit 0cfa1659487f (package/pkg-utils.mk: introduce "name" field in
show-info output) did what it said, but did so in the generic show-info
part, thus it was also added to filesystems (rootfs), the other kind of
entity that show-info reports on.

Only packages have a "name"; filesystems do not. Instead, they already
have an 'image_name'.

Move the 'name' field to the package-related part of show-info.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-utils.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index b2f0b8cebf..973eabe437 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -96,7 +96,6 @@ endef
 # $(1): upper-case package or filesystem name
 define json-info
 	"$($(1)_NAME)": {
-		"name": "$($(1)_RAWNAME)",
 		"type": "$($(1)_TYPE)",
 		$(if $(filter rootfs,$($(1)_TYPE)), \
 			$(call _json-info-fs,$(1)), \
@@ -108,6 +107,7 @@ endef
 # _json-info-pkg, _json-info-pkg-details, _json-info-fs: private helpers
 # for json-info, above
 define _json-info-pkg
+	"name": "$($(1)_RAWNAME)",
 	$(if $($(1)_IS_VIRTUAL), \
 		"virtual": true$(comma),
 		"virtual": false$(comma)
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 09/16 v2] support/utils: make-comma-list does just that, not quoting

[Yann E. MORIN]

Currently, we have two functions that build a comma-separated list
of items; one is double-quoting the items, while the other is
single-quoting them. Their naming is not very consistent.

Besides, in a followup change, we will need to build a comma-separated
list of items that are already double-quoted.

Introduce a macro that does just build a comma-separated list, and
use that in the two other macros; rename the existing macro so the
naming is consistent.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 package/pkg-utils.mk  | 10 +++++-----
 support/misc/utils.mk | 12 ++++++++----
 2 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 973eabe437..94e806963b 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -120,17 +120,17 @@ define _json-info-pkg
 		"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
 	)
 	"dependencies": [
-		$(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
+		$(call make-dq-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
 	],
 	"reverse_dependencies": [
-		$(call make-comma-list,$(sort $($(1)_RDEPENDENCIES)))
+		$(call make-dq-comma-list,$(sort $($(1)_RDEPENDENCIES)))
 	]
 	$(if $($(1)_CPE_ID_VALID), \
 		$(comma) "cpe-id": "$($(1)_CPE_ID)" \
 	)
 	$(if $($(1)_IGNORE_CVES),
 		$(comma) "ignore_cves": [
-			$(call make-comma-list,$(sort $($(1)_IGNORE_CVES)))
+			$(call make-dq-comma-list,$(sort $($(1)_IGNORE_CVES)))
 		]
 	)
 endef
@@ -144,7 +144,7 @@ define _json-info-pkg-details
 		{
 			"source": "$(notdir $(dl))",
 			"uris": [
-				$(call make-comma-list,
+				$(call make-dq-comma-list,
 					$(subst \|,|,
 						$(call DOWNLOAD_URIS,$(dl),$(1))
 					)
@@ -161,7 +161,7 @@ define _json-info-fs
 				null \
 			),
 	"dependencies": [
-		$(call make-comma-list,$(sort $($(1)_DEPENDENCIES)))
+		$(call make-dq-comma-list,$(sort $($(1)_DEPENDENCIES)))
 	]
 endef
 
diff --git a/support/misc/utils.mk b/support/misc/utils.mk
index dc60cad979..fdc22a2eef 100644
--- a/support/misc/utils.mk
+++ b/support/misc/utils.mk
@@ -84,13 +84,17 @@ finddirclauses = $(call notfirstword,$(patsubst %,-o -path '$(1)/%',$(2)))
 # notfirstword(wordlist): returns all but the first word in wordlist
 notfirstword = $(wordlist 2,$(words $(1)),$(1))
 
-# build a comma-separated list of quoted items, from a space-separated
+# build a comma-separated list of items, from a space-separated
+# list of items:   a b c d  -->  a, b, c, d
+make-comma-list = $(subst $(space),$(comma)$(space),$(strip $(1)))
+
+# build a comma-separated list of double-quoted items, from a space-separated
 # list of unquoted items:   a b c d  -->  "a", "b", "c", "d"
-make-comma-list = $(subst $(space),$(comma)$(space),$(patsubst %,"%",$(strip $(1))))
+make-dq-comma-list = $(call make-comma-list,$(patsubst %,"%",$(strip $(1))))
 
-# build a comma-separated list of single quoted items, from a space-separated
+# build a comma-separated list of single-quoted items, from a space-separated
 # list of unquoted items:   a b c d  -->  'a', 'b', 'c', 'd'
-make-sq-comma-list = $(subst $(space),$(comma)$(space),$(patsubst %,'%',$(strip $(1))))
+make-sq-comma-list = $(call make-comma-list,$(patsubst %,'%',$(strip $(1))))
 
 # Needed for the foreach loops to loop over the list of hooks, so that
 # each hook call is properly separated by a newline.
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 10/16 v2] support/misc/utils: introduce $(tab)=\t and $(escape)=\x1b

[Yann E. MORIN]

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 support/misc/utils.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/support/misc/utils.mk b/support/misc/utils.mk
index fdc22a2eef..4d0efcced0 100644
--- a/support/misc/utils.mk
+++ b/support/misc/utils.mk
@@ -13,6 +13,8 @@ qstrip = $(strip $(subst ",,$(1)))
 comma := ,
 empty :=
 space := $(empty) $(empty)
+tab := $(empty)	$(empty)
+escape := $(shell printf '\x1b')
 
 # make 4.3:
 # https://lwn.net/Articles/810071/
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 11/16 v2] package/pkg-utils: introduce helper to properly json-escape a string

[Yann E. MORIN]

In quite a few places, we need to generate string that are proper JSON
values or keys.

However, JSON is very strict on what constitute a string, and most JSON
parsers (like jq or python3's json module) are very picky when parsing a
string; any deviation from the spec is immediately sanctioned by a hard
error (jq aborts, python3's json module raise an exception).

Introduce a macro that properly prepares a Makefile value into a valid
JSON string:

  - backslash '\' must be escaped;

  - double-quotes need to be escaped of course, as they are the string
    delimiter in JSON;

  - anything in the range [0x00..0x1F] must be escaped; in practice, we
    only ever need to escape \n, \t, and ESC (we could add more in the
    future if need be);

  - finally, we also escape the space, \x20, so that we can call
    $(strip) on a JSON blurb (like we do for example do build a
    comma-separated list, or when we sanitise the JSON) without losing
    multiple spaces where they make sense.

It would have been nice if we had been able to split the macro on
multiple lines, but spaces creep in from everywhere in that case, and
getting rid of them is getting quite nasty... We could introduce
intermediate macros, but meh...

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 package/pkg-utils.mk | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 94e806963b..15e3e14e1b 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -176,6 +176,17 @@ clean-json = $(strip \
 	))))) \
 )
 
+# mk-json-str -- escape and double-quote a string to make it a valid json string
+#  - escape \
+#  - escape "
+#  - escape \n
+#  - escape \t
+#  - escape ESC
+#  - escape SPACE (so that we can $(strip) a JSON blurb without squashing multiple spaces)
+# This unfortunately has to be on a single line...
+mk-json-str = "$(subst $(space),\u0020,$(subst $(escape),\u001b,$(subst $(tab),\t,$(subst $(sep),\n,$(subst ",\",$(subst \,\\,$(1)))))))"
+# )))))" # Syntax colouring
+
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 # rsync the contents of per-package directories
 # $1: space-separated list of packages to rsync from
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 12/16 v2] package/pkg-utils: generate proper JSON strings where they are emitted

[Yann E. MORIN]

Currently, our clean-json macro does two things:
  - remove excessive commas before a closing list or dictionary;
  - escape the backslash.

We are going to need the comma-drop feature of clean-json, but on a
JSON blurb where all the characters, backslash included, are already
all properly escaped, so that we do not need further escape.

So, we drop the backslash escaping from clean-json, and use the newly
introduced mk-json-str helper in every locations where we turn a
Makefile variable into a JSON string.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 package/pkg-utils.mk | 53 ++++++++++++++++++++++++++++----------------
 1 file changed, 34 insertions(+), 19 deletions(-)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 15e3e14e1b..0b0ff77419 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -96,7 +96,7 @@ endef
 # $(1): upper-case package or filesystem name
 define json-info
 	"$($(1)_NAME)": {
-		"type": "$($(1)_TYPE)",
+		"type": $(call mk-json-str,$($(1)_TYPE)),
 		$(if $(filter rootfs,$($(1)_TYPE)), \
 			$(call _json-info-fs,$(1)), \
 			$(call _json-info-pkg,$(1)), \
@@ -107,47 +107,59 @@ endef
 # _json-info-pkg, _json-info-pkg-details, _json-info-fs: private helpers
 # for json-info, above
 define _json-info-pkg
-	"name": "$($(1)_RAWNAME)",
+	"name": $(call mk-json-str,$($(1)_RAWNAME)),
 	$(if $($(1)_IS_VIRTUAL), \
 		"virtual": true$(comma),
 		"virtual": false$(comma)
 		$(call _json-info-pkg-details,$(1)) \
 	)
-	"build_dir": "$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))",
+	"build_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))),
 	$(if $(filter target,$($(1)_TYPE)), \
 		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
 		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
 		"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
 	)
 	"dependencies": [
-		$(call make-dq-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
+		$(call make-comma-list, \
+			$(foreach dep,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)), \
+				$(call mk-json-str,$(dep)) \
+			) \
+		)
 	],
 	"reverse_dependencies": [
-		$(call make-dq-comma-list,$(sort $($(1)_RDEPENDENCIES)))
+		$(call make-comma-list, \
+			$(foreach dep,$(sort $($(1)_RDEPENDENCIES)), \
+				$(call mk-json-str,$(dep)) \
+			) \
+		)
 	]
 	$(if $($(1)_CPE_ID_VALID), \
-		$(comma) "cpe-id": "$($(1)_CPE_ID)" \
+		$(comma) "cpe-id": $(call mk-json-str,$($(1)_CPE_ID)) \
 	)
 	$(if $($(1)_IGNORE_CVES),
 		$(comma) "ignore_cves": [
-			$(call make-dq-comma-list,$(sort $($(1)_IGNORE_CVES)))
+			$(call make-comma-list, \
+				$(foreach cve,$(sort $($(1)_IGNORE_CVES)), \
+					$(call mk-json-str,$(cve)) \
+				) \
+			)
 		]
 	)
 endef
 
 define _json-info-pkg-details
-	"version": "$($(1)_DL_VERSION)",
-	"licenses": "$($(1)_LICENSE)",
-	"dl_dir": "$($(1)_DL_SUBDIR)",
+	"version": $(call mk-json-str,$($(1)_DL_VERSION)),
+	"licenses": $(call mk-json-str,$($(1)_LICENSE)),
+	"dl_dir": $(call mk-json-str,$($(1)_DL_SUBDIR)),
 	"downloads": [
 	$(foreach dl,$(sort $($(1)_ALL_DOWNLOADS)),
 		{
-			"source": "$(notdir $(dl))",
+			"source": $(call mk-json-str,$(notdir $(dl))),
 			"uris": [
-				$(call make-dq-comma-list,
-					$(subst \|,|,
-						$(call DOWNLOAD_URIS,$(dl),$(1))
-					)
+				$(call make-comma-list, \
+					$(foreach uri,$(call DOWNLOAD_URIS,$(dl),$(1)), \
+						$(call mk-json-str,$(subst \|,|,$(uri))) \
+					) \
 				)
 			]
 		},
@@ -157,11 +169,15 @@ endef
 
 define _json-info-fs
 	"image_name": $(if $($(1)_FINAL_IMAGE_NAME), \
-				"$($(1)_FINAL_IMAGE_NAME)", \
+				$(call mk-json-str,$($(1)_FINAL_IMAGE_NAME)), \
 				null \
 			),
 	"dependencies": [
-		$(call make-dq-comma-list,$(sort $($(1)_DEPENDENCIES)))
+		$(call make-comma-list, \
+			$(foreach dep,$(sort $($(1)_DEPENDENCIES)), \
+				$(call mk-json-str,$(dep)) \
+			) \
+		)
 	]
 endef
 
@@ -171,9 +187,8 @@ endef
 clean-json = $(strip \
 	$(subst $(comma)},}, $(subst $(comma)$(space)},$(space)}, \
 	$(subst $(comma)],], $(subst $(comma)$(space)],$(space)], \
-	$(subst \,\\, \
 		$(strip $(1)) \
-	))))) \
+	)))) \
 )
 
 # mk-json-str -- escape and double-quote a string to make it a valid json string
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 00/16 v2] core: add show-vars, a json-formatted equivalent to printvars (branch yem/show-vars)

[Yann E. MORIN]

Hello All!

This series introduces a new top-level command, show-vars, which like
printvars, prints all the variables of Buildroot, and like show-info,
does so in a JSON-formatted output.

The first 8 patches are preparatory cleanups in the core infras. They
were previously posted separately, but their raison d'être is only
really obvious with the introduction of this new show-vars.

Then the following 4 patches prepare the existing show-info helpers for
more genericity.

The 3 next patches add a few more items to show-info (I'm sneaking them
in this series, am I not devious?). There are hints in the commit logs
about something that will come in the future, that will make use of
those new fields (and now am I not teasing too much?).

Eventually, we introduce show-vars, which is pretty trivial in and off
itelf once all the preparatory work has been done...

Regards,
Yann E. MORIN.


----------------------------------------------------------------
Yann E. MORIN (16):
      core/show-info: report build_dir relative to CONFIG_DIR
      package/kodi-*: run pkg-config at build time, not at parse time
      support/dependencies: avoid spurious warning on print-vars
      package/infras: do not conditionally set empty variables
      package/pkg-python: do not set empty variables
      core/pkg-generic: also list files installed in images/
      Makefile: really comment syntax colouring
      core/show-info: 'name' only applies to packages
      support/utils: make-comma-list does just that, not quoting
      support/misc/utils: introduce $(tab)=\t and $(escape)=\x1b
      package/pkg-utils: introduce helper to properly json-escape a string
      package/pkg-utils: generate proper JSON strings where they are emitted
      core/show-info: report whether a package is overriden
      core/show-info: also export source and stamp directories in show-info
      core/show-info: report package build steps and stamp files
      Makefile: introduce show-vars, a json-formatted equivalent to printvars

 Makefile                                           | 23 +++++-
 .../kodi-screensaver-asterwave.mk                  |  4 +-
 .../kodi-screensaver-cpblobs.mk                    |  4 +-
 .../kodi-screensaver-matrixtrails.mk               |  4 +-
 package/pkg-autotools.mk                           |  4 -
 package/pkg-cmake.mk                               |  4 -
 package/pkg-generic.mk                             | 29 +------
 package/pkg-kconfig.mk                             |  3 -
 package/pkg-meson.mk                               |  4 -
 package/pkg-python.mk                              |  7 --
 package/pkg-qmake.mk                               |  4 -
 package/pkg-utils.mk                               | 90 +++++++++++++++++-----
 package/pkg-waf.mk                                 |  5 --
 support/dependencies/dependencies.mk               |  2 +-
 support/misc/utils.mk                              | 14 +++-
 15 files changed, 111 insertions(+), 90 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 13/16 v2] core/show-info: report whether a package is overriden

[Yann E. MORIN]

The show-info output can be used in quite some ways, like to assess the
validity of the current configuration, like a production build in a CI
would do for example. One such assessment would be to ensure that no
package is overriden.

Additionally, when we will (later!) want to check whether we can re-use
cached artefacts of a package, we will not want to do so for packages
that are overridden (nor would we want to actually cache such a result
either).

We do not expose the override location, because this is a purely-local
information, and show info should never contain such local information
(the output of show-info can be shared). Besides, that information is
already known by the user, as they passed it to Buildroot via local.mk
to begin with.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop@gmail.com
---
 package/pkg-utils.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 0b0ff77419..da123a6ed2 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -113,6 +113,7 @@ define _json-info-pkg
 		"virtual": false$(comma)
 		$(call _json-info-pkg-details,$(1)) \
 	)
+	"overriden": $(if $($(1)_OVERRIDE_SRCDIR),true,false),
 	"build_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))),
 	$(if $(filter target,$($(1)_TYPE)), \
 		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 14/16 v2] core/show-info: also export source and stamp directories in show-info

[Yann E. MORIN]

External tools may need to peek into the source tree (to check what the
list of patches that were applied), or in the stamp directory (to check
and report on the progress of a build)

Currently, both locations are identical, but semantically different and
an internal implementation detail. Exposing both separately will llow us
to change either without breaking users' scripts. Hopefully.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 package/pkg-utils.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index da123a6ed2..37a4d244d7 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -114,6 +114,8 @@ define _json-info-pkg
 		$(call _json-info-pkg-details,$(1)) \
 	)
 	"overriden": $(if $($(1)_OVERRIDE_SRCDIR),true,false),
+	"stamp_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_DIR))),
+	"source_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_DIR))),
 	"build_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))),
 	$(if $(filter target,$($(1)_TYPE)), \
 		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 15/16 v2] core/show-info: report package build steps and stamp files

[Yann E. MORIN]

People (and their scripts!) whi want to report on the progress of the
build, will need to know what steps a package follows in what order.

This is handled internally by Buildroot itself already, by use of stamp
files.

However, the names of those stamp files are purely an implementation
detail (even if they are very unlikely to ever change).

Expose the build steps in a list, which is guaranteed to be ordered, ad
for each step, its name and stamp file. The output (once filtered
through jq, elided for bevity) will look like:

    [...]
    "install_target": true,
    "install_staging": false,
    "install_images": false,
    "build_steps": [
      {
        "step": "download",
        "stamp_file": ".stamp_downloaded"
      },
      {
        "step": "extract",
        "stamp_file": ".stamp_extracted"
      },
      {
        "step": "patch",
        "stamp_file": ".stamp_patched"
      },
      {
        "step": "configure",
        "stamp_file": ".stamp_configured"
      },
      {
        "step": "build",
        "stamp_file": ".stamp_built"
      },
      {
        "step": "install_target",
        "stamp_file": ".stamp_target_installed"
      }
    ],
    [...]

(Note: the first three fields already existed before this patch.)

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop@gmail.com
---
 package/pkg-utils.mk | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 37a4d244d7..d7cd8f9775 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -122,6 +122,29 @@ define _json-info-pkg
 		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
 		"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
 	)
+	"build_steps": [
+		$(if $($(1)_OVERRIDE_SRCDIR), \
+			{ "step": "rsync"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_RSYNC))) }$(comma) \
+			, \
+			{ "step": "download"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_SOURCE))) }$(comma) \
+			{ "step": "extract"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_EXTRACT))) }$(comma) \
+			{ "step": "patch"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_PATCH))) }$(comma) \
+		)
+		{ "step": "configure", "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_CONFIGURE))) },
+		{ "step": "build", "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_BUILD))) },
+		$(if $(filter YES,$($(1)_INSTALL_STAGING)), \
+			{ "step": "install_staging"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_INSTALL_STAGING))) }$(comma) \
+		)
+		$(if $(filter YES,$($(1)_INSTALL_TARGET)), \
+			{ "step": "install_target"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_INSTALL_TARGET))) }$(comma) \
+		)
+		$(if $(filter YES,$($(1)_INSTALL_IMAGES)), \
+			{ "step": "install_images"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_INSTALL_IMAGES))) }$(comma) \
+		)
+		$(if $(filter host,$($(1)_TYPE)), \
+			{ "step": "install"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_INSTALL_HOST))) } \
+		)
+	],
 	"dependencies": [
 		$(call make-comma-list, \
 			$(foreach dep,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)), \
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 16/16 v2] Makefile: introduce show-vars, a json-formatted equivalent to printvars

[Yann E. MORIN]

The current printvars output suffers from a serious design flaw:
variables are not delimited, which makes it impossible to reliably
retrieve the value of variables; only variables that are known to
not contain a \n can be relatively safely extracted.

However, in some cases, it is important to be able to retrieve the
multi-line value of a variable, notably the CMDS or the hooks. One
such use-case (to follow in an unscheduled future) would be to hash
the variables that make up a package "configuration", and cache or
extract the files for that package to speed up the build.

Modeled after printvars and show-info, we introduce show-vars (what a
lack of imagination here) that outputs a json dictionary which keys are
the variable names, and for each variable, provides the raw and expanded
values.

Unlike printvars, we do not provide a way to get either the raw or
expanded value; both are systematically printed; a user will get just
the one is needs. Additionally, strings in JSON are quoted, so there is
no need to provide a way to quote variables; that would not make sense.

Note: for printvars, we require that the user provides an explicit
pattern to filter variables on. This is historical (see fd5bd12379dc,
Makefile: printvars: don't print anything when VARS is not set). The
underlying reasoning was that printvars is too "raw", and variables are
not well delimited, so printvars was mostly used to extract a few values
here and there, from scripts, or to quickly inspect a specific package's
variables during debugging.

But show-vars, although technically plain-text, being JSON, is not very
human-readable, and is mostly aimed at tools that will parse it with a
real JSON parser, and which will want to have a complete view of a lot
of variables at once. As such, and contrary to printvars, it makes sense
to report on all variables by default, unless the user explicitly
requested a subset.

As a final note: a lot of our variables only make sense in the context
of an actual make target. For example, a variable of package foo, that
contains $(@D)/bar, would expand to .../build/FOO-VERSION/bar. This is
because our CMDS and hooks are expanded as the recipe of a stamp file
that lies in the package build directory.

But for show-info, this falls flat on its face: it is not the stamp file
of a package, so there is no package directory, and show-info itself has
not directory part, so $(@D) expands to '.' (dot).

Additionally, some variables may contain calls to $(shell) (e.g. to call
pkg-config), and this also does not work with show-info.

These two issues make it impossible to emit the correct expanded value
of variables. To be noted: printvars has the exact same limitations for
the exact same reasons.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 Makefile | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 6f6bb72de3..0a24f7c5be 100644
--- a/Makefile
+++ b/Makefile
@@ -141,7 +141,7 @@ nobuild_targets := source %-source \
 	clean distclean help show-targets graph-depends \
 	%-graph-depends %-show-depends %-show-version \
 	graph-build graph-size list-defconfigs \
-	savedefconfig update-defconfig printvars
+	savedefconfig update-defconfig printvars show-vars
 ifeq ($(MAKECMDGOALS),)
 BR_BUILDING = y
 else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),)
@@ -1058,6 +1058,7 @@ endif
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
 # only the variables matching the make pattern passed in VARS are
 # displayed.
+# show-vars does the same, but as a JSON dictionnary.
 .PHONY: printvars
 printvars:
 	@:
@@ -1070,6 +1071,22 @@ printvars:
 			$(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))
 # ')))) # Syntax colouring...
 
+.PHONY: show-vars
+show-vars: VARS?=%
+show-vars:
+	@:
+	$(info $(call clean-json, { \
+			$(foreach V, \
+				$(sort $(filter $(VARS),$(.VARIABLES))), \
+				$(if $(filter-out environment% default automatic, $(origin $V)), \
+					"$V": { \
+						"expanded": $(call mk-json-str,$($V))$(comma) \
+						"raw": $(call mk-json-str,$(value $V)) \
+					}$(comma) \
+				) \
+			) \
+	} ))
+
 .PHONY: clean
 clean:
 	rm -rf $(BASE_TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) $(HOST_DIR_SYMLINK) \
@@ -1162,6 +1179,8 @@ help:
 	@echo '  pkg-stats              - generate info about packages as JSON and HTML'
 	@echo '  missing-cpe            - generate XML snippets for missing CPE identifiers'
 	@echo '  printvars              - dump internal variables selected with VARS=...'
+	@echo '  show-vars              - dump all internal variables as a JSON blurb; use VARS=...'
+	@echo '                             to limit the list to variables names matching that pattern'
 	@echo
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
 	@echo '  make O=dir             - Locate all output files in "dir", including .config'
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 00/16 v2] core: add show-vars, a json-formatted equivalent to printvars (branch yem/show-vars)

[Yann E. MORIN]

All,

On 2021-11-13 14:28 +0100, Yann E. MORIN spake thusly:
> Hello All!
> 
> This series introduces a new top-level command, show-vars, which like
> printvars, prints all the variables of Buildroot, and like show-info,
> does so in a JSON-formatted output.
> 
> The first 8 patches are preparatory cleanups in the core infras. They
> were previously posted separately, but their raison d'être is only
> really obvious with the introduction of this new show-vars.
> 
> Then the following 4 patches prepare the existing show-info helpers for
> more genericity.
> 
> The 3 next patches add a few more items to show-info (I'm sneaking them
> in this series, am I not devious?). There are hints in the commit logs
> about something that will come in the future, that will make use of
> those new fields (and now am I not teasing too much?).
> 
> Eventually, we introduce show-vars, which is pretty trivial in and off
> itelf once all the preparatory work has been done...
> 
> Regards,
> Yann E. MORIN.
> 
> 
> ----------------------------------------------------------------
> Yann E. MORIN (16):
>       core/show-info: report build_dir relative to CONFIG_DIR
>       package/kodi-*: run pkg-config at build time, not at parse time
>       support/dependencies: avoid spurious warning on print-vars

Those three patches are fixes, so applied to master.

>       package/infras: do not conditionally set empty variables
>       package/pkg-python: do not set empty variables

Those two patches, not applied

>       core/pkg-generic: also list files installed in images/
>       Makefile: really comment syntax colouring
>       core/show-info: 'name' only applies to packages

Those three patches are fixes, so applied to master.

>       support/utils: make-comma-list does just that, not quoting
>       support/misc/utils: introduce $(tab)=\t and $(escape)=\x1b
>       package/pkg-utils: introduce helper to properly json-escape a string
>       package/pkg-utils: generate proper JSON strings where they are emitted
>       core/show-info: report whether a package is overriden
>       core/show-info: also export source and stamp directories in show-info
>       core/show-info: report package build steps and stamp files
>       Makefile: introduce show-vars, a json-formatted equivalent to printvars

The rest of the series is not about fixes, so will have to wait for
another round.

Regards,
Yann E. MORIN.

> 
>  Makefile                                           | 23 +++++-
>  .../kodi-screensaver-asterwave.mk                  |  4 +-
>  .../kodi-screensaver-cpblobs.mk                    |  4 +-
>  .../kodi-screensaver-matrixtrails.mk               |  4 +-
>  package/pkg-autotools.mk                           |  4 -
>  package/pkg-cmake.mk                               |  4 -
>  package/pkg-generic.mk                             | 29 +------
>  package/pkg-kconfig.mk                             |  3 -
>  package/pkg-meson.mk                               |  4 -
>  package/pkg-python.mk                              |  7 --
>  package/pkg-qmake.mk                               |  4 -
>  package/pkg-utils.mk                               | 90 +++++++++++++++++-----
>  package/pkg-waf.mk                                 |  5 --
>  support/dependencies/dependencies.mk               |  2 +-
>  support/misc/utils.mk                              | 14 +++-
>  15 files changed, 111 insertions(+), 90 deletions(-)
> 
> -- 
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 08/16 v2] core/show-info: 'name' only applies to packages

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Commit 0cfa1659487f (package/pkg-utils.mk: introduce "name" field in
 > show-info output) did what it said, but did so in the generic show-info
 > part, thus it was also added to filesystems (rootfs), the other kind of
 > entity that show-info reports on.

 > Only packages have a "name"; filesystems do not. Instead, they already
 > have an 'image_name'.

 > Move the 'name' field to the package-related part of show-info.

 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
 > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Committed to 2021.02.x and 2021.08.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 04/16 v2] package/infras: do not conditionally set empty variables

[Thomas Petazzoni]

On Sat, 13 Nov 2021 14:28:15 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Setting an unset variable to an empty value is useless in make; an unset
> variable just expands to an empty string anyway. So what we do currently
> has no side effect:
> 
>     variable set and not empty   -> variable not modified
>     variable set and empty       -> variable not modified
>     variable unset               -> set to an empty string
> 
> However, additional variables do have an impact on the parsing time of
> the Makefiles, and the more variables, the more collisions in the hash
> table used internally by make, which slows down the parsing.
> 
> By dropping those conditionally-set-empty variables, we gain about 3%:
> 
>     Run     Before  After
>     1       5.572   5.325
>     2       5.434   5.354
>     3       5.490   5.320
>     4       5.525   5.330
>     5       5.476   5.330
>     6       5.511   5.434
>     7       5.498   5.388
>     8       5.524   5.371
>     9       5.479   5.346
>     10      5.637   5.324
>     Mean:   5.515   5.352
> 
> Yeah, 0.163s does not look like much, and this does not make
> autocompletion any more usable. Still, that 3% gain is not to be
> ashamed of either.
> 
> Note that there are 3 others case where we do set empty variables, but
> those are unconditional and serve other purposes:
> 
>   - pkg-virtual: this is done on purpose to avoid a bug when the
>     environment may have TOOLCHAIN_VERSION or _SOURCE set, and we really
>     want those to be empty, so the assignment is not conditional;
> 
>   - pkg-python: the reason for setting those to empty is dubious at
>     best; it's been there since the inception of the python infra, back
>     in 2013; still, the case is different than this patch addresses;
> 
>   - pkg-toolchain-external: this is the case for a toolchain already
>     installed, so indeed we want to set _SOURCE and _VERSION to empty.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/pkg-autotools.mk |  4 ----
>  package/pkg-cmake.mk     |  4 ----
>  package/pkg-generic.mk   | 27 ---------------------------
>  package/pkg-kconfig.mk   |  3 ---
>  package/pkg-meson.mk     |  4 ----
>  package/pkg-python.mk    |  4 ----
>  package/pkg-qmake.mk     |  4 ----
>  package/pkg-waf.mk       |  5 -----
>  8 files changed, 55 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 05/16 v2] package/pkg-python: do not set empty variables

[Thomas Petazzoni]

On Sat, 13 Nov 2021 14:28:16 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Similarly to what we just did for conditionally-set-empty variables,
> unconditionally setting empty variables serves no purpose in Makefiles,
> as unset variables are just expanded as empty...
> 
> Even though we have numerous python packages, the speed gain was not
> measurable, with delta much less than the noise.
> 
> Still, for consistency, we just do not set those variables.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/pkg-python.mk | 3 ---
>  1 file changed, 3 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 09/16 v2] support/utils: make-comma-list does just that, not quoting

[Thomas Petazzoni]

On Sat, 13 Nov 2021 14:28:20 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Currently, we have two functions that build a comma-separated list
> of items; one is double-quoting the items, while the other is
> single-quoting them. Their naming is not very consistent.
> 
> Besides, in a followup change, we will need to build a comma-separated
> list of items that are already double-quoted.
> 
> Introduce a macro that does just build a comma-separated list, and
> use that in the two other macros; rename the existing macro so the
> naming is consistent.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
>  package/pkg-utils.mk  | 10 +++++-----
>  support/misc/utils.mk | 12 ++++++++----
>  2 files changed, 13 insertions(+), 9 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 10/16 v2] support/misc/utils: introduce $(tab)=\t and $(escape)=\x1b

[Thomas Petazzoni]

On Sat, 13 Nov 2021 14:28:21 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
>  support/misc/utils.mk | 2 ++
>  1 file changed, 2 insertions(+)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 11/16 v2] package/pkg-utils: introduce helper to properly json-escape a string

[Thomas Petazzoni]

On Sat, 13 Nov 2021 14:28:22 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> In quite a few places, we need to generate string that are proper JSON
> values or keys.
> 
> However, JSON is very strict on what constitute a string, and most JSON
> parsers (like jq or python3's json module) are very picky when parsing a
> string; any deviation from the spec is immediately sanctioned by a hard
> error (jq aborts, python3's json module raise an exception).
> 
> Introduce a macro that properly prepares a Makefile value into a valid
> JSON string:
> 
>   - backslash '\' must be escaped;
> 
>   - double-quotes need to be escaped of course, as they are the string
>     delimiter in JSON;
> 
>   - anything in the range [0x00..0x1F] must be escaped; in practice, we
>     only ever need to escape \n, \t, and ESC (we could add more in the
>     future if need be);
> 
>   - finally, we also escape the space, \x20, so that we can call
>     $(strip) on a JSON blurb (like we do for example do build a
>     comma-separated list, or when we sanitise the JSON) without losing
>     multiple spaces where they make sense.
> 
> It would have been nice if we had been able to split the macro on
> multiple lines, but spaces creep in from everywhere in that case, and
> getting rid of them is getting quite nasty... We could introduce
> intermediate macros, but meh...
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
>  package/pkg-utils.mk | 11 +++++++++++
>  1 file changed, 11 insertions(+)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 12/16 v2] package/pkg-utils: generate proper JSON strings where they are emitted

[Thomas Petazzoni]

On Sat, 13 Nov 2021 14:28:23 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Currently, our clean-json macro does two things:
>   - remove excessive commas before a closing list or dictionary;
>   - escape the backslash.
> 
> We are going to need the comma-drop feature of clean-json, but on a
> JSON blurb where all the characters, backslash included, are already
> all properly escaped, so that we do not need further escape.
> 
> So, we drop the backslash escaping from clean-json, and use the newly
> introduced mk-json-str helper in every locations where we turn a
> Makefile variable into a JSON string.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
>  package/pkg-utils.mk | 53 ++++++++++++++++++++++++++++----------------
>  1 file changed, 34 insertions(+), 19 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 13/16 v2] core/show-info: report whether a package is overriden

[Thomas Petazzoni]

On Sat, 13 Nov 2021 14:28:24 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> The show-info output can be used in quite some ways, like to assess the
> validity of the current configuration, like a production build in a CI
> would do for example. One such assessment would be to ensure that no
> package is overriden.
> 
> Additionally, when we will (later!) want to check whether we can re-use
> cached artefacts of a package, we will not want to do so for packages
> that are overridden (nor would we want to actually cache such a result
> either).
> 
> We do not expose the override location, because this is a purely-local
> information, and show info should never contain such local information
> (the output of show-info can be shared). Besides, that information is
> already known by the user, as they passed it to Buildroot via local.mk
> to begin with.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vadim Kochan <vadim4j@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: eeppeliteloop@gmail.com

I did not apply this one, because when you set <pkg>_SITE_METHOD =
local, then <pkg>_OVERRIDE_SRCDIR will be assigned, but the package
isn't really "overridden". It's using locally available source code, as
part of the package definition itself.

So I found that advertising this package as "overridden" was not
semantically correct.

I'm of course ready to discuss this, and see other arguments :)

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 14/16 v2] core/show-info: also export source and stamp directories in show-info

[Thomas Petazzoni]

On Sat, 13 Nov 2021 14:28:25 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> External tools may need to peek into the source tree (to check what the
> list of patches that were applied), or in the stamp directory (to check
> and report on the progress of a build)
> 
> Currently, both locations are identical, but semantically different and
> an internal implementation detail. Exposing both separately will llow us
> to change either without breaking users' scripts. Hopefully.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

I was not too sure on this one, but applied anyway.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 15/16 v2] core/show-info: report package build steps and stamp files

[Thomas Petazzoni]

On Sat, 13 Nov 2021 14:28:26 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> People (and their scripts!) whi want to report on the progress of the
> build, will need to know what steps a package follows in what order.
> 
> This is handled internally by Buildroot itself already, by use of stamp
> files.
> 
> However, the names of those stamp files are purely an implementation
> detail (even if they are very unlikely to ever change).
> 
> Expose the build steps in a list, which is guaranteed to be ordered, ad
> for each step, its name and stamp file. The output (once filtered
> through jq, elided for bevity) will look like:
> 
>     [...]
>     "install_target": true,
>     "install_staging": false,
>     "install_images": false,
>     "build_steps": [
>       {
>         "step": "download",
>         "stamp_file": ".stamp_downloaded"
>       },
>       {
>         "step": "extract",
>         "stamp_file": ".stamp_extracted"
>       },
>       {
>         "step": "patch",
>         "stamp_file": ".stamp_patched"
>       },
>       {
>         "step": "configure",
>         "stamp_file": ".stamp_configured"
>       },
>       {
>         "step": "build",
>         "stamp_file": ".stamp_built"
>       },
>       {
>         "step": "install_target",
>         "stamp_file": ".stamp_target_installed"
>       }
>     ],
>     [...]
> 
> (Note: the first three fields already existed before this patch.)
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vadim Kochan <vadim4j@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: eeppeliteloop@gmail.com

This one I really found excessive. There's pretty a 1:1 relationship
between the steps and the stamp files, so anything that has knowledge
of the internal steps of a package can also know about the stamp files.

As it's more controversial than the rest of the series, I left it
aside. I'll for now mark it as Rejected, but I'm also ready to revisit
that if need be.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 16/16 v2] Makefile: introduce show-vars, a json-formatted equivalent to printvars

[Thomas Petazzoni]

On Sat, 13 Nov 2021 14:28:27 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> The current printvars output suffers from a serious design flaw:
> variables are not delimited, which makes it impossible to reliably
> retrieve the value of variables; only variables that are known to
> not contain a \n can be relatively safely extracted.
> 
> However, in some cases, it is important to be able to retrieve the
> multi-line value of a variable, notably the CMDS or the hooks. One
> such use-case (to follow in an unscheduled future) would be to hash
> the variables that make up a package "configuration", and cache or
> extract the files for that package to speed up the build.
> 
> Modeled after printvars and show-info, we introduce show-vars (what a
> lack of imagination here) that outputs a json dictionary which keys are
> the variable names, and for each variable, provides the raw and expanded
> values.
> 
> Unlike printvars, we do not provide a way to get either the raw or
> expanded value; both are systematically printed; a user will get just
> the one is needs. Additionally, strings in JSON are quoted, so there is
> no need to provide a way to quote variables; that would not make sense.
> 
> Note: for printvars, we require that the user provides an explicit
> pattern to filter variables on. This is historical (see fd5bd12379dc,
> Makefile: printvars: don't print anything when VARS is not set). The
> underlying reasoning was that printvars is too "raw", and variables are
> not well delimited, so printvars was mostly used to extract a few values
> here and there, from scripts, or to quickly inspect a specific package's
> variables during debugging.
> 
> But show-vars, although technically plain-text, being JSON, is not very
> human-readable, and is mostly aimed at tools that will parse it with a
> real JSON parser, and which will want to have a complete view of a lot
> of variables at once. As such, and contrary to printvars, it makes sense
> to report on all variables by default, unless the user explicitly
> requested a subset.
> 
> As a final note: a lot of our variables only make sense in the context
> of an actual make target. For example, a variable of package foo, that
> contains $(@D)/bar, would expand to .../build/FOO-VERSION/bar. This is
> because our CMDS and hooks are expanded as the recipe of a stamp file
> that lies in the package build directory.
> 
> But for show-info, this falls flat on its face: it is not the stamp file
> of a package, so there is no package directory, and show-info itself has
> not directory part, so $(@D) expands to '.' (dot).
> 
> Additionally, some variables may contain calls to $(shell) (e.g. to call
> pkg-config), and this also does not work with show-info.
> 
> These two issues make it impossible to emit the correct expanded value
> of variables. To be noted: printvars has the exact same limitations for
> the exact same reasons.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
>  Makefile | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)

Applied to master, thanks.

However, there are two things that are missing:

 (1) Update to the Buildroot manual

 (2) At least one test case in supporting/testing/

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 13/16 v2] core/show-info: report whether a package is overriden

[Yann E. MORIN]

Thomas, All,

On 2021-12-30 21:45 +0100, Thomas Petazzoni spake thusly:
> On Sat, 13 Nov 2021 14:28:24 +0100
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > The show-info output can be used in quite some ways, like to assess the
> > validity of the current configuration, like a production build in a CI
> > would do for example. One such assessment would be to ensure that no
> > package is overriden.
[--SNIP--]
> I did not apply this one, because when you set <pkg>_SITE_METHOD =
> local, then <pkg>_OVERRIDE_SRCDIR will be assigned, but the package
> isn't really "overridden". It's using locally available source code, as
> part of the package definition itself.
> 
> So I found that advertising this package as "overridden" was not
> semantically correct.

Indeed, I'll see if we can easily discriminate the two, to really report
overriden non-local packages.

Regards,
Yann E. MORIN.

> I'm of course ready to discuss this, and see other arguments :)
> 
> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 15/16 v2] core/show-info: report package build steps and stamp files

[Yann E. MORIN]

Thomas, All,

On 2021-12-30 21:46 +0100, Thomas Petazzoni spake thusly:
> On Sat, 13 Nov 2021 14:28:26 +0100
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > People (and their scripts!) whi want to report on the progress of the
> > build, will need to know what steps a package follows in what order.
> > 
> > This is handled internally by Buildroot itself already, by use of stamp
> > files.
> > 
> > However, the names of those stamp files are purely an implementation
> > detail (even if they are very unlikely to ever change).
[--SNIP--]
> This one I really found excessive. There's pretty a 1:1 relationship
> between the steps and the stamp files, so anything that has knowledge
> of the internal steps of a package can also know about the stamp files.

The thing is, those utilities do not even need to have the knowledge of
the internal steps, because the steps are all listed explicitly, in
order, with their names and associated stamp-files.

So, such a tool could just do (python-esque):

    for step in packages[pkg_name]['build_steps']:
        print('{}: {}'.format(step['step'], os.path.exists(step['stamp_file'])))

and they do not even have to concern themselves with whether we add or
remove steps, or whether we change the ordering or what.

(Yeah, the key for the step name could be changed from 'step' to 'name';
naming is hard.)

The only nitpick however, is that with TLPB, installation in images/ is
not ordered wrt the installation in target/ or staging/ (target/ is
always after staging/ even with TLPB).

Regards,
Yann E. MORIN.

> As it's more controversial than the rest of the series, I left it
> aside. I'll for now mark it as Rejected, but I'm also ready to revisit
> that if need be.
> 
> Thomas
> -- 
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot