[Buildroot] [PATCH 1/8] core/show-info: do not show install types for host packages

[Buildroot] [PATCH 1/8] core/show-info: do not show install types for host packages

[Yann E. MORIN]

By their very nature, host packages have no target, staging, or
images install steps; they have a single install step, that is
always performed.

However, we currently report those three types of install, which
is misleading at best, and really incorrect.

If we were to report any install type for host package, that would
be a single one, and it would always be true.

So, do not report any install type for host packages, as it does
not make sense to report anything that is always true.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop at gmail.com
---
 package/pkg-utils.mk | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index d324934dba..14101fcc2a 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -112,9 +112,11 @@ define _json-info-pkg-details
 	"version": "$($(1)_DL_VERSION)",
 	"licenses": "$($(1)_LICENSE)",
 	"dl_dir": "$($(1)_DL_SUBDIR)",
-	"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET)),
-	"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING)),
-	"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES)),
+	$(if $(filter target,$($(1)_TYPE)), \
+		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
+		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
+		"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
+	)
 	"downloads": [
 	$(foreach dl,$(sort $($(1)_ALL_DOWNLOADS)),
 		{
-- 
2.20.1

[Buildroot] [PATCH 2/8] infra/pkg-generic: don't set INSTALL_{TARGET, STAGING, IMAGES} for host

[Yann E. MORIN]

By their very nature, host packages have no target, staging, or
images install steps; they have a single install step, that is
always performed.

As sucj, setting the corresponding _INSTALL_{TARGET,STAGING,IMAGES}
variables does not make sense for host packages.

However, people (and scripts) may get confused when they process the
output of printvars, e.g.:

    $ make printvars VARS=HOST_LIBTOOL_INSTALL_TARGET
    HOST_LIBTOOL_INSTALL_TARGET=YES

Only set those variables for target packages. There is no corresponding
variable for host packages, as they are always installed (and only
once).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop at gmail.com
---
 package/pkg-generic.mk | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 7679c41556..8cd5a7ff62 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -706,10 +706,6 @@ $(2)_FINAL_RECURSIVE_RDEPENDENCIES = $$(sort \
 	) \
 	$$($(2)_FINAL_RECURSIVE_RDEPENDENCIES__X))
 
-$(2)_INSTALL_STAGING		?= NO
-$(2)_INSTALL_IMAGES		?= NO
-$(2)_INSTALL_TARGET		?= YES
-
 # define sub-target stamps
 $(2)_TARGET_INSTALL_TARGET =	$$($(2)_DIR)/.stamp_target_installed
 $(2)_TARGET_INSTALL_STAGING =	$$($(2)_DIR)/.stamp_staging_installed
@@ -770,6 +766,9 @@ $(1):			$(1)-install
 ifeq ($$($(2)_TYPE),host)
 $(1)-install:	        $(1)-install-host
 else
+$(2)_INSTALL_STAGING	?= NO
+$(2)_INSTALL_IMAGES	?= NO
+$(2)_INSTALL_TARGET	?= YES
 $(1)-install:		$(1)-install-staging $(1)-install-target $(1)-install-images
 endif
 
-- 
2.20.1

[Buildroot] [PATCH 3/8] core/show-info: report install types for virtual packages too

[Yann E. MORIN]

Although virtual packages do not usually install anything, they do
participate in the dependency chain, and they also transition smoothly
from 'not handled yet' all the way down to 'all set'. And in fact, it
happens that some virtual packages do install something; for example,
the virtual package 'toolchain' does tweak the musl headers during a
post staging hook.

As such, people may want to know when those packages have been handled.
To that goal, they will need to know what stampfile to look at (the
target, staging, or images ones) to identify what step the virtual
package is at.

So, report the type of install steps for virtual packages too, which
means for all packages.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop at gmail.com
---
 package/pkg-utils.mk | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 14101fcc2a..b6c6aeeb40 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -100,6 +100,11 @@ define _json-info-pkg
 		"virtual": false$(comma)
 		$(call _json-info-pkg-details,$(1)) \
 	)
+	$(if $(filter target,$($(1)_TYPE)), \
+		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
+		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
+		"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
+	)
 	"dependencies": [
 		$(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
 	],
@@ -112,11 +117,6 @@ define _json-info-pkg-details
 	"version": "$($(1)_DL_VERSION)",
 	"licenses": "$($(1)_LICENSE)",
 	"dl_dir": "$($(1)_DL_SUBDIR)",
-	$(if $(filter target,$($(1)_TYPE)), \
-		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
-		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
-		"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
-	)
 	"downloads": [
 	$(foreach dl,$(sort $($(1)_ALL_DOWNLOADS)),
 		{
-- 
2.20.1

[Buildroot] [PATCH 4/8] core/show-info: report the package build directory

[Yann E. MORIN]

Some people want to be able to inspect the current state of the build,
and report information about it, like:
  - report the number of packages already built out of the total,
  - list the packages being actually built (e.g. for TLPB)
  - etc...

However, the location where a package is built is inherently an internal
detail, so expose that to the user in the output of show-info. We only
expose the location relative to the base directory (basically, either
output/ or the user-suppiled $(O)), so that show-info does not contain
local information (the output of show-info can be shared).

Interested parties will be able to poke in there to identify the stamp
files and deduce the package's state.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop at gmail.com
---
 package/pkg-utils.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index b6c6aeeb40..66504d0be2 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -100,6 +100,7 @@ define _json-info-pkg
 		"virtual": false$(comma)
 		$(call _json-info-pkg-details,$(1)) \
 	)
+	"build_dir": "$(patsubst $(BASE_DIR)/%,%,$($(1)_BUILDDIR))",
 	$(if $(filter target,$($(1)_TYPE)), \
 		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
 		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
-- 
2.20.1

[Buildroot] [PATCH 0/8] core/show-info: export extra information about the package build (branch yem/show-info-extras)

[Yann E. MORIN]

Hello All!

People (and scripts) who want to report on the build progress will need
ato peek inside the internal details of the package build steps, so we
expose this extra information in show-info.

Suggested-by: eeppeliteloop at gmail.com


Regards,
Yann E. MORIN.


The following changes since commit 91238c2605babd1a3d262bacdf04e14d39e59810

  package/uftp: bump to version 4.10.2 (2020-04-11 10:07:02 +0200)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 0493cfafbdc9dd03d461d8707de2358156f9ad5d

  core/show-info: report image name of filesystems (2020-04-11 10:10:18 +0200)


----------------------------------------------------------------
Yann E. MORIN (8):
      core/show-info: do not show install types for host packages
      infra/pkg-generic: don't set INSTALL_{TARGET,STAGING,IMAGES} for host
      core/show-info: report install types for virtual packages too
      core/show-info: report the package build directory
      core/show-info: report whether a package is overriden
      core/show-info: report package stamp files
      core/show-info: report the ordered list of build steps
      core/show-info: report image name of filesystems

 package/pkg-generic.mk |  7 +++----
 package/pkg-utils.mk   | 52 +++++++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 52 insertions(+), 7 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 5/8] core/show-info: report whether a package is overriden

[Yann E. MORIN]

When a package is overridden, the download, extract, and patch
stampfiles are replaced by a single rsync stampfile.

People (and scripts) who want to report on the current status of the
build, will want to know what stamp file to look at.

Expose in show-info whether the package is overridden or not.

We do not expose the override location, because this is a purely-local
information, and show info should never contain such local information
(the output of show-info can be shared). Besides, that information is
already known by the user, as they passed it to Buildroot via local.mk
to begin with.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop at gmail.com
---
 package/pkg-utils.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 66504d0be2..98e3ede3d1 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -100,6 +100,7 @@ define _json-info-pkg
 		"virtual": false$(comma)
 		$(call _json-info-pkg-details,$(1)) \
 	)
+	"overriden": $(if $($(1)_OVERRIDE_SRCDIR),true,false),
 	"build_dir": "$(patsubst $(BASE_DIR)/%,%,$($(1)_BUILDDIR))",
 	$(if $(filter target,$($(1)_TYPE)), \
 		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
-- 
2.20.1

[Buildroot] [PATCH 6/8] core/show-info: report package stamp files

[Yann E. MORIN]

People (and scripts) who want to report on the build progress, will want
to look at the various stamp files.

However, the names of those stamp files are purely an implementation
detail (even if they are very unlikely to ever change).

So, expose the names of those stampfiles in show-info, so that
interested parties do not have to hard-code them.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop at gmail.com
---
 package/pkg-utils.mk | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 98e3ede3d1..e7bfc39283 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -107,6 +107,29 @@ define _json-info-pkg
 		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
 		"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
 	)
+	"stamp_files": {
+		$(if $($(1)_OVERRIDE_SRCDIR), \
+			"rsync": "$(notdir $($(1)_TARGET_RSYNC))"$(comma) \
+			, \
+			"download": "$(notdir $($(1)_TARGET_SOURCE))"$(comma) \
+			"extract": "$(notdir $($(1)_TARGET_EXTRACT))"$(comma) \
+			"patch": "$(notdir $($(1)_TARGET_PATCH))"$(comma) \
+		)
+		"configure": "$(notdir $($(1)_TARGET_CONFIGURE))",
+		"build": "$(notdir $($(1)_TARGET_BUILD))",
+		$(if $(filter YES,$($(1)_INSTALL_STAGING)),\
+			"install_staging": "$(notdir $($(1)_TARGET_INSTALL_STAGING))"$(comma) \
+		)
+		$(if $(filter YES,$($(1)_INSTALL_TARGET)),\
+			"install_target": "$(notdir $($(1)_TARGET_INSTALL_TARGET))"$(comma) \
+		)
+		$(if $(filter YES,$($(1)_INSTALL_IMAGES)),\
+			"install_images": "$(notdir $($(1)_TARGET_INSTALL_IMAGES))"$(comma) \
+		)
+		$(if $(filter host,$($(1)_TYPE)), \
+			"install":  "$(notdir $($(1)_TARGET_INSTALL_HOST))"$(comma) \
+		)
+	},
 	"dependencies": [
 		$(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
 	],
-- 
2.20.1

[Buildroot] [PATCH 7/8] core/show-info: report the ordered list of build steps

[Yann E. MORIN]

People (and scritps) who want to report on the build progress, now know
where and what to look for (build directory and stampfiles), but they
also need to know the ordering of those stampfiles.

The stampfiles item is a dictionary; dictionaries are guaranteed to not
be ordered, so even if we output the stampfiles in order, we can expect
users of that dictionary to get them back in order.

Expose the build steps in a list, which is guaranteed to be ordered.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop at gmail.com
---
 package/pkg-utils.mk | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index e7bfc39283..bcdbf568fe 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -130,6 +130,21 @@ define _json-info-pkg
 			"install":  "$(notdir $($(1)_TARGET_INSTALL_HOST))"$(comma) \
 		)
 	},
+	"build_steps": [
+		$(if $($(1)_OVERRIDE_SRCDIR), \
+			"rsync"$(comma) \
+			, \
+			"download"$(comma) \
+			"extract"$(comma) \
+			"patch"$(comma) \
+		)
+		"build",
+		"configure",
+		$(if $(filter YES,$($(1)_INSTALL_STAGING)),"install_staging"$(comma))
+		$(if $(filter YES,$($(1)_INSTALL_TARGET)),"install_target"$(comma))
+		$(if $(filter YES,$($(1)_INSTALL_IMAGES)),"install_images"$(comma))
+		$(if $(filter host,$($(1)_TYPE)),"install"$(comma))
+	],
 	"dependencies": [
 		$(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
 	],
-- 
2.20.1

[Buildroot] [PATCH 8/8] core/show-info: report image name of filesystems

[Yann E. MORIN]

The rootfses, unlike packages, do not have stampfiles to represent
whether they are built or not. Indeed, we always rebuild the rootfs, and
there is a single step to do.

Hpwever, people (and scripts) who want to report on the build progress,
will want to know whether each rootfs has been built already or not.

Expose in show-info the name of the file that wil contain the rootfs
image.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop at gmail.com
---
 package/pkg-utils.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index bcdbf568fe..a464b660ef 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -174,6 +174,10 @@ define _json-info-pkg-details
 endef
 
 define _json-info-fs
+	"image_name": $(if $($(1)_FINAL_IMAGE_NAME), \
+				"$($(1)_FINAL_IMAGE_NAME)", \
+				null \
+			),
 	"dependencies": [
 		$(call make-comma-list,$(sort $($(1)_DEPENDENCIES)))
 	]
-- 
2.20.1

[Buildroot] [PATCH 5/8] core/show-info: report whether a package is overriden

[Thomas Petazzoni]

On Sat, 11 Apr 2020 10:12:30 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> When a package is overridden, the download, extract, and patch
> stampfiles are replaced by a single rsync stampfile.
> 
> People (and scripts) who want to report on the current status of the
> build, will want to know what stamp file to look at.
> 
> Expose in show-info whether the package is overridden or not.
> 
> We do not expose the override location, because this is a purely-local
> information, and show info should never contain such local information
> (the output of show-info can be shared). Besides, that information is
> already known by the user, as they passed it to Buildroot via local.mk
> to begin with.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vadim Kochan <vadim4j@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: eeppeliteloop at gmail.com
> ---
>  package/pkg-utils.mk | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index 66504d0be2..98e3ede3d1 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -100,6 +100,7 @@ define _json-info-pkg
>  		"virtual": false$(comma)
>  		$(call _json-info-pkg-details,$(1)) \
>  	)
> +	"overriden": $(if $($(1)_OVERRIDE_SRCDIR),true,false),

I don't have a good suggestion, but I'm not sure "overriden" is the
most appropriate term. Indeed, the download/extract/patch steps are
also replaced by a rsync step for packages that use _SITE_METHOD =
local, and such packages are not "overriden".

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 6/8] core/show-info: report package stamp files

[Thomas Petazzoni]

On Sat, 11 Apr 2020 10:12:31 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> +	"stamp_files": {
> +		$(if $($(1)_OVERRIDE_SRCDIR), \
> +			"rsync": "$(notdir $($(1)_TARGET_RSYNC))"$(comma) \
> +			, \
> +			"download": "$(notdir $($(1)_TARGET_SOURCE))"$(comma) \
> +			"extract": "$(notdir $($(1)_TARGET_EXTRACT))"$(comma) \
> +			"patch": "$(notdir $($(1)_TARGET_PATCH))"$(comma) \
> +		)
> +		"configure": "$(notdir $($(1)_TARGET_CONFIGURE))",
> +		"build": "$(notdir $($(1)_TARGET_BUILD))",
> +		$(if $(filter YES,$($(1)_INSTALL_STAGING)),\
> +			"install_staging": "$(notdir $($(1)_TARGET_INSTALL_STAGING))"$(comma) \
> +		)
> +		$(if $(filter YES,$($(1)_INSTALL_TARGET)),\
> +			"install_target": "$(notdir $($(1)_TARGET_INSTALL_TARGET))"$(comma) \
> +		)
> +		$(if $(filter YES,$($(1)_INSTALL_IMAGES)),\
> +			"install_images": "$(notdir $($(1)_TARGET_INSTALL_IMAGES))"$(comma) \
> +		)
> +		$(if $(filter host,$($(1)_TYPE)), \
> +			"install":  "$(notdir $($(1)_TARGET_INSTALL_HOST))"$(comma) \
> +		)

Isn't this going to add for each package exactly the same information,
over and over again ? I.e, just the names of stamp files, again, again,
and again. This seems like a big waste of space, making the JSON output
very noisy. I think the tool showing the progress should simply know
what the stamp files are.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 7/8] core/show-info: report the ordered list of build steps

[Thomas Petazzoni]

On Sat, 11 Apr 2020 10:12:32 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> People (and scritps) who want to report on the build progress, now know
> where and what to look for (build directory and stampfiles), but they
> also need to know the ordering of those stampfiles.
> 
> The stampfiles item is a dictionary; dictionaries are guaranteed to not
> be ordered, so even if we output the stampfiles in order, we can expect
> users of that dictionary to get them back in order.
> 
> Expose the build steps in a list, which is guaranteed to be ordered.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vadim Kochan <vadim4j@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: eeppeliteloop at gmail.com
> ---
>  package/pkg-utils.mk | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)

Meh, just like PATCH 6/8, I'm not sure. This is going to be a lot of
additional information in the JSON, almost duplicated for every
package, and there is really nothing bad in the "progress showing" tool
to have knowledge about these steps. It's not like they are changing
all the time.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 3/8] core/show-info: report install types for virtual packages too

[Thomas Petazzoni]

On Sat, 11 Apr 2020 10:12:28 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Although virtual packages do not usually install anything, they do
> participate in the dependency chain, and they also transition smoothly
> from 'not handled yet' all the way down to 'all set'.

I'm having a hard time to understand what you mean by:

  they also transition smoothly from 'not handled yet' all the way down
  to 'all set'.

Could you clarify ?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 5/8] core/show-info: report whether a package is overriden

[Yann E. MORIN]

Thomas, All,

On 2020-04-11 10:36 +0200, Thomas Petazzoni spake thusly:
> On Sat, 11 Apr 2020 10:12:30 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
> > When a package is overridden, the download, extract, and patch
> > stampfiles are replaced by a single rsync stampfile.
> > 
> > People (and scripts) who want to report on the current status of the
> > build, will want to know what stamp file to look at.
> > 
> > Expose in show-info whether the package is overridden or not.
> > 
> > We do not expose the override location, because this is a purely-local
> > information, and show info should never contain such local information
> > (the output of show-info can be shared). Besides, that information is
> > already known by the user, as they passed it to Buildroot via local.mk
> > to begin with.
> > 
> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> > Cc: Vadim Kochan <vadim4j@gmail.com>
> > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > Cc: eeppeliteloop at gmail.com
> > ---
> >  package/pkg-utils.mk | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> > index 66504d0be2..98e3ede3d1 100644
> > --- a/package/pkg-utils.mk
> > +++ b/package/pkg-utils.mk
> > @@ -100,6 +100,7 @@ define _json-info-pkg
> >  		"virtual": false$(comma)
> >  		$(call _json-info-pkg-details,$(1)) \
> >  	)
> > +	"overriden": $(if $($(1)_OVERRIDE_SRCDIR),true,false),
> 
> I don't have a good suggestion, but I'm not sure "overriden" is the
> most appropriate term. Indeed, the download/extract/patch steps are
> also replaced by a rsync step for packages that use _SITE_METHOD =
> local, and such packages are not "overriden".

It is very unfortunate that we conflate the two conditions.

We can't even reconstruct the override by looking at whether
_SITE_METHOD == local, because even local packages may be overriden...

So, is it worth that I try and untangle the tow notions? Given the
feedback on the rest of this eries, I don;t want to invest too much time
if there is no chance of it getting in...

Regards,
Yann E. MORIN.

> Thomas
> -- 
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 3/8] core/show-info: report install types for virtual packages too

[Yann E. MORIN]

Thomas, All,

On 2020-04-11 10:41 +0200, Thomas Petazzoni spake thusly:
> On Sat, 11 Apr 2020 10:12:28 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
> > Although virtual packages do not usually install anything, they do
> > participate in the dependency chain, and they also transition smoothly
> > from 'not handled yet' all the way down to 'all set'.
> I'm having a hard time to understand what you mean by:
>   they also transition smoothly from 'not handled yet' all the way down
>   to 'all set'.
> Could you clarify ?

I mean that virtual package, like all other packages, have a sequence of
a few steps that transition the virtual package from "not handled yet"
to "installed and dependencies can start", i.e. virtual packages are:

 1. configured
 2. built
 3. installed in staging (optional)
 4. installed in target (optional)
 5. isntalled in images (optional)

Virtual packages are just packages in this respect. It's just that their
deoendencies are treated specially.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 5/8] core/show-info: report whether a package is overriden

[Thomas Petazzoni]

On Sat, 11 Apr 2020 11:44:36 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> > I don't have a good suggestion, but I'm not sure "overriden" is the
> > most appropriate term. Indeed, the download/extract/patch steps are
> > also replaced by a rsync step for packages that use _SITE_METHOD =
> > local, and such packages are not "overriden".  
> 
> It is very unfortunate that we conflate the two conditions.

Well, from an internal implementation point of view, SITE_METHOD =
local and OVERRIDE_SRCDIR are just exactly the same thing.

Perhaps you could use:

	"rsynced": $(if $($(1)_OVERRIDE_SRCDIR),true,false),

as this is really annotates the fact that the package source code is
rsynced. You could even make it clearer with "source-rsynced" or
something like that.

> We can't even reconstruct the override by looking at whether
> _SITE_METHOD == local, because even local packages may be overriden...

Indeed.

> So, is it worth that I try and untangle the tow notions? Given the
> feedback on the rest of this eries, I don;t want to invest too much time
> if there is no chance of it getting in...

I think this particular patch is OK, even though admittedly the
external tool could just watch for the correct stamp files to show up:
if .stamp_downloaded shows up, we're downloading it normally, if
.stamp_rsynced shows up, we have a local or overridden package.

In the design of the tool, it would be good to make sure that top-level
parallel build is taken into account.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 5/8] core/show-info: report whether a package is overriden

[Yann E. MORIN]

Thomas, All,

On 2020-04-11 14:42 +0200, Thomas Petazzoni spake thusly:
> On Sat, 11 Apr 2020 11:44:36 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
> > > I don't have a good suggestion, but I'm not sure "overriden" is the
> > > most appropriate term. Indeed, the download/extract/patch steps are
> > > also replaced by a rsync step for packages that use _SITE_METHOD =
> > > local, and such packages are not "overriden".  
> > 
> > It is very unfortunate that we conflate the two conditions.
> 
> Well, from an internal implementation point of view, SITE_METHOD =
> local and OVERRIDE_SRCDIR are just exactly the same thing.

But that is an implementation detail, indeed.

In fact, we should have had local support without override support. And
then we should have grafted override suport on top of the local one,
while the code as it is makes local re-use the override code...

I'll try to hack a cleanup in that area...

> Perhaps you could use:
> 
> 	"rsynced": $(if $($(1)_OVERRIDE_SRCDIR),true,false),

Fact is, that still does not reflect what I want to expose. An overriden
package is de-facto not reproducible, while a local package may.

But I don't care enough to argue further. "resynced" gets the same value
as "overriden", and since we can;t get the semantic of "overriden", it
is as good as it is.

> as this is really annotates the fact that the package source code is
> rsynced. You could even make it clearer with "source-rsynced" or
> something like that.
> 
> > We can't even reconstruct the override by looking at whether
> > _SITE_METHOD == local, because even local packages may be overriden...
> 
> Indeed.
> 
> > So, is it worth that I try and untangle the tow notions? Given the
> > feedback on the rest of this eries, I don;t want to invest too much time
> > if there is no chance of it getting in...
> 
> I think this particular patch is OK, even though admittedly the
> external tool could just watch for the correct stamp files to show up:
> if .stamp_downloaded shows up, we're downloading it normally, if
> .stamp_rsynced shows up, we have a local or overridden package.
> 
> In the design of the tool, it would be good to make sure that top-level
> parallel build is taken into account.

Which is exactly what I am trying to achieve here: provide all the info
about the internal details, so that the tool does not have to guess.

But personally I don't care about such a tool; I'm just trying to make
life easier for those (like eeppeliteloop in Cc) who want such tools.

Regards,
Yann E. MORIN.

> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 7/8] core/show-info: report the ordered list of build steps

[Philippe Proulx]

On Sat, Apr 11, 2020 at 4:39 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> On Sat, 11 Apr 2020 10:12:32 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
>
> > People (and scritps) who want to report on the build progress, now know
> > where and what to look for (build directory and stampfiles), but they
> > also need to know the ordering of those stampfiles.
> >
> > The stampfiles item is a dictionary; dictionaries are guaranteed to not
> > be ordered, so even if we output the stampfiles in order, we can expect
> > users of that dictionary to get them back in order.
> >
> > Expose the build steps in a list, which is guaranteed to be ordered.
> >
> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> > Cc: Vadim Kochan <vadim4j@gmail.com>
> > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > Cc: eeppeliteloop at gmail.com
> > ---
> >  package/pkg-utils.mk | 15 +++++++++++++++
> >  1 file changed, 15 insertions(+)
>
> Meh, just like PATCH 6/8, I'm not sure. This is going to be a lot of
> additional information in the JSON, almost duplicated for every
> package, and there is really nothing bad in the "progress showing" tool
> to have knowledge about these steps. It's not like they are changing
> all the time.

Replying for patch 6/8 and this one.

Is the `show-info` target considered a public API? If so, without a
version, it means it can never break. So let me suggest another layout
which breaks `show-info` (name it `show-info-2` if you will).

Output example (using YAML only for clarity here):

    version: 2
    build_steps:
      target:
        - name: download
          stamp_file: .stamp_downloaded
        - name: extract
          stamp_file: .stamp_extracted
        - name: patch
          stamp_file: .stamp_patched
        - name: build
          stamp_file: .stamp_configured
        - name: configure
          stamp_file: .stamp_built
        - name: install_staging
          stamp_file: .stamp_staging_installed
        - name: install_target
          stamp_file: .stamp_target_installed
        - name: install_image
          stamp_file: .stamp_image_installed
        ...
      host:
        - ...
      rootfs:
        - ...
    packages:
      target:
        libglib2:
          virtual: false
          version: 2.64.1
          licenses: LGPL-2.1+
          dl_dir: libglib2
          downloads:
            - source: glib-2.64.1.tar.xz
              uris:
                - http+http://ftp.gnome.org/pub/gnome/sources/glib/2.64
                - http|urlencode+http://sources.buildroot.net/libglib2
                - http|urlencode+http://sources.buildroot.net
          overriden: false
          build_dir: build/libglib2-2.64.1/
          build_steps:
            - download
            - extract
            - patch
            - build
            - configure
            - install_staging
            - install_target
          dependencies:
            - host-libglib2
            - host-meson
            - host-pkgconf
            - host-skeleton
            - host-tar
            - libffi
            - pcre
            - skeleton
            - toolchain
            - zlib
          reverse_dependencies:
            - cairo
        ...
      host:
        ...
      rootfs:
        ...

So, changes:

* A root `version` entry indicates the API version.

* The root `build_steps` object indicates all the _possible_ (sorted)
  build steps for each package type.

  Each build step in the array has its name and its corresponding
  stamp file to indicate when it's completed.

  As Yann wrote:

  > However, the names of those stamp files are purely an implementation
  > detail (even if they are very unlikely to ever change).

* The root `packages` object contains the package information for each
  package type.

  Not strictly related to this patch or the previous one, but instead
  of repeating the type in each object, you can just group packages
  by type.

* For each package object, the `build_steps` array indicate what are the
  build steps for this package.

  Each item is the name of a build step in the corresponding root
  `build_steps` object's array.

* For a target package, the `install_staging`, `install_target`, and
  `install_images` entries are removed.

  They are not needed anymore because `build_steps` offers the same
  information.

  I understand some build steps could be implicit and not listed here,
  but with my suggestion, you already have what's needed for any build
  monitoring tool to watch a specific package build process if:

  * Some of the implicit build steps become optional in the future for
    any reason that you don't know now.

  * You add new types of build steps: simply describe them in the
    `build_steps` root arrays.

  Both scenarios won't require an API version bump.

If you're not willing to break `show-info` and don't want `show-info-2`,
then you can:

* Keep `show-info` as is:

  * Augment a package information with `build_steps` (like above).

  * Keep the `install_*` entries which become redundant.

* Have another make target, say `show-build-steps`, which only prints
  the root `build_steps` object above.

What do you think?

Phil

>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

[Buildroot] [PATCH 5/8] core/show-info: report whether a package is overriden

[Thomas Petazzoni]

Hello,

On Sat, 11 Apr 2020 15:22:58 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> > Well, from an internal implementation point of view, SITE_METHOD =
> > local and OVERRIDE_SRCDIR are just exactly the same thing.  
> 
> But that is an implementation detail, indeed.
> 
> In fact, we should have had local support without override support. And
> then we should have grafted override suport on top of the local one,
> while the code as it is makes local re-use the override code...

I am not sure what that would bring, and how that would make things
better/clearer. Ultimately, what happens for both overridden packages
and local packages is that they are rsynced.

> > Perhaps you could use:
> > 
> > 	"rsynced": $(if $($(1)_OVERRIDE_SRCDIR),true,false),  
> 
> Fact is, that still does not reflect what I want to expose. An overriden
> package is de-facto not reproducible, while a local package may.

I don't understand why a local package is more reproducible than an
overridden package. In both cases, Buildroot cannot guarantee the
reproducibility.

> But I don't care enough to argue further. "resynced" gets the same value
> as "overriden", and since we can;t get the semantic of "overriden", it
> is as good as it is.

I don't follow your reasoning here. I am fine with having the boolean
this patch proposes to add, I was just suggesting another term than
"overridden", as that was not really covering the case of local
packages.

> > I think this particular patch is OK, even though admittedly the
> > external tool could just watch for the correct stamp files to show up:
> > if .stamp_downloaded shows up, we're downloading it normally, if
> > .stamp_rsynced shows up, we have a local or overridden package.
> > 
> > In the design of the tool, it would be good to make sure that top-level
> > parallel build is taken into account.  
> 
> Which is exactly what I am trying to achieve here: provide all the info
> about the internal details, so that the tool does not have to guess.

And I'm not sure it's a worthwhile goal. I think the tool should just
have the knowledge of what possible stamp files exist, and what are the
possible sequences of stamp file creation.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 7/8] core/show-info: report the ordered list of build steps

[Thomas Petazzoni]

On Sat, 11 Apr 2020 09:41:47 -0400
Philippe Proulx <eeppeliteloop@gmail.com> wrote:

> Replying for patch 6/8 and this one.
> 
> Is the `show-info` target considered a public API?

Somewhat yes: the goal of show-info is that people can craft their own
tooling on top of it. Breaking the show-info output would break such
tools.

> If so, without a
> version, it means it can never break. So let me suggest another layout
> which breaks `show-info` (name it `show-info-2` if you will).
> 
> Output example (using YAML only for clarity here):

[...]

This could certainly work, and reduces a bit the duplication of
information.

However, I still don't understand why your tool is not capable of
having this knowledge about the stamp files. This is something that
rarely changes. I think we haven't changed the stamp files since... 5
years? More? If your tool is in the Buildroot tree itself, it can
simply be updated if there is ever a change in the sequencing of steps.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 7/8] core/show-info: report the ordered list of build steps

[Philippe Proulx]

On Sat, Apr 11, 2020 at 10:19 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> On Sat, 11 Apr 2020 09:41:47 -0400
> Philippe Proulx <eeppeliteloop@gmail.com> wrote:
>
> > Replying for patch 6/8 and this one.
> >
> > Is the `show-info` target considered a public API?
>
> Somewhat yes: the goal of show-info is that people can craft their own
> tooling on top of it. Breaking the show-info output would break such
> tools.
>
> > If so, without a
> > version, it means it can never break. So let me suggest another layout
> > which breaks `show-info` (name it `show-info-2` if you will).
> >
> > Output example (using YAML only for clarity here):
>
> [...]
>
> This could certainly work, and reduces a bit the duplication of
> information.
>
> However, I still don't understand why your tool is not capable of
> having this knowledge about the stamp files. This is something that
> rarely changes. I think we haven't changed the stamp files since... 5
> years? More? If your tool is in the Buildroot tree itself, it can
> simply be updated if there is ever a change in the sequencing of steps.

Yes of course.

I was only trying to improve Yann's solution considering your comments
about data redundancy.

If you tell me that things such build step order and stamp file names do
not change for many years and as such can be considered as interfaces,
I'm fine with this.

It's always a balance between adding the same logic to all the viewers
and generating less data.

My tool could be part of the Buildroot project, but I doubt you'll want
something that depends on PyQt5 in there. However, I could certainly
contribute a Python module which offers utilities over the `show-info`
output so as to have a versioned, object-oriented API, and then import
it from an external tool. This way, even if `show-info` changes, the
Python module can continue to offer the same (non-breaking) interface.

I still think we somewhat need the `build_dir` entry though.

Phil

>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

[Buildroot] [PATCH 7/8] core/show-info: report the ordered list of build steps

[Thomas Petazzoni]

Hello,

On Sat, 11 Apr 2020 11:06:05 -0400
Philippe Proulx <eeppeliteloop@gmail.com> wrote:

> > However, I still don't understand why your tool is not capable of
> > having this knowledge about the stamp files. This is something that
> > rarely changes. I think we haven't changed the stamp files since... 5
> > years? More? If your tool is in the Buildroot tree itself, it can
> > simply be updated if there is ever a change in the sequencing of steps.  
> 
> Yes of course.
> 
> I was only trying to improve Yann's solution considering your comments
> about data redundancy.

Sure, and your proposal makes sense if we believe we need to have this
build sequencing information exposed in the JSON output.

> If you tell me that things such build step order and stamp file names do
> not change for many years and as such can be considered as interfaces,
> I'm fine with this.

I don't think we can really consider these as stable interfaces, but it
is sufficiently stable to rely on it.

> It's always a balance between adding the same logic to all the viewers
> and generating less data.
> 
> My tool could be part of the Buildroot project, but I doubt you'll want
> something that depends on PyQt5 in there.

I don't see why we couldn't have it in the tree. It's not a mandatory
tool, so it's fine to have something that depends on PyQt5.

> However, I could certainly contribute a Python module which offers
> utilities over the `show-info` output so as to have a versioned,
> object-oriented API, and then import it from an external tool. This
> way, even if `show-info` changes, the Python module can continue to
> offer the same (non-breaking) interface.
> 
> I still think we somewhat need the `build_dir` entry though.

Yes, for the build_dir entry, I definitely agree, it is useful and
needed.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 5/8] core/show-info: report whether a package is overriden

[Yann E. MORIN]

Thomas, All,

On 2020-04-11 16:14 +0200, Thomas Petazzoni spake thusly:
> On Sat, 11 Apr 2020 15:22:58 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > > Well, from an internal implementation point of view, SITE_METHOD =
> > > local and OVERRIDE_SRCDIR are just exactly the same thing.  
> > But that is an implementation detail, indeed.
> > In fact, we should have had local support without override support. And
> > then we should have grafted override suport on top of the local one,
> > while the code as it is makes local re-use the override code...
> I am not sure what that would bring, and how that would make things
> better/clearer. Ultimately, what happens for both overridden packages
> and local packages is that they are rsynced.

I'd still like to be able to tell the two apart, but that is eventually
orthogonal to the current topic.

> > > Perhaps you could use:
> > > 	"rsynced": $(if $($(1)_OVERRIDE_SRCDIR),true,false),  
> > Fact is, that still does not reflect what I want to expose. An overriden
> > package is de-facto not reproducible, while a local package may.
> I don't understand why a local package is more reproducible than an
> overridden package. In both cases, Buildroot cannot guarantee the
> reproducibility.

If the package is in the same sourec tree as Buildroot (or a git-submodule
or something like that), then it is reproducible (as it is in VCS and
de-facto referenced by a specific commit that brings both Buildroot and
the package sources. That's why a Isaid that a local packge "may" be
reproducible.

> > But I don't care enough to argue further. "resynced" gets the same value
> > as "overriden", and since we can;t get the semantic of "overriden", it
> > is as good as it is.
> I don't follow your reasoning here. I am fine with having the boolean
> this patch proposes to add, I was just suggesting another term than
> "overridden", as that was not really covering the case of local
> packages.

Yes, you are right here; I got side-tracked. For the stampfiles, we
don't need to discrimninate between local or overriden: we just need to
know it is rsynced (or downlaoded-extracted-patched).

> > > I think this particular patch is OK, even though admittedly the
> > > external tool could just watch for the correct stamp files to show up:
> > > if .stamp_downloaded shows up, we're downloading it normally, if
> > > .stamp_rsynced shows up, we have a local or overridden package.
> > > In the design of the tool, it would be good to make sure that top-level
> > > parallel build is taken into account.  
> > Which is exactly what I am trying to achieve here: provide all the info
> > about the internal details, so that the tool does not have to guess.
> And I'm not sure it's a worthwhile goal. I think the tool should just
> have the knowledge of what possible stamp files exist, and what are the
> possible sequences of stamp file creation.

OK. I'll drop from my series the parts related to that...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 7/8] core/show-info: report the ordered list of build steps

[Yann E. MORIN]

Philippe, All,

On 2020-04-11 09:41 -0400, Philippe Proulx spake thusly:
> On Sat, Apr 11, 2020 at 4:39 AM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> > On Sat, 11 Apr 2020 10:12:32 +0200
> > "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > > People (and scritps) who want to report on the build progress, now know
> > > where and what to look for (build directory and stampfiles), but they
> > > also need to know the ordering of those stampfiles.
> > >
> > > The stampfiles item is a dictionary; dictionaries are guaranteed to not
> > > be ordered, so even if we output the stampfiles in order, we can expect
> > > users of that dictionary to get them back in order.
> > >
> > > Expose the build steps in a list, which is guaranteed to be ordered.
> > >
> > > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> > > Cc: Vadim Kochan <vadim4j@gmail.com>
> > > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > > Cc: Arnout Vandecappelle <arnout@mind.be>
> > > Cc: eeppeliteloop at gmail.com
> > > ---
> > >  package/pkg-utils.mk | 15 +++++++++++++++
> > >  1 file changed, 15 insertions(+)
> >
> > Meh, just like PATCH 6/8, I'm not sure. This is going to be a lot of
> > additional information in the JSON, almost duplicated for every
> > package, and there is really nothing bad in the "progress showing" tool
> > to have knowledge about these steps. It's not like they are changing
> > all the time.
> 
> Replying for patch 6/8 and this one.
> 
> Is the `show-info` target considered a public API?

Somewhat, yes. We can't guaraenntee its stability, though. If there are
keys that no longer make sense, then we would no longer be able to
expose them. Ditto, if a key semantics were to change, we would have to
reflect that in show-info.

For the former case, whem/if we eventually merge target/ and staging/
into one (or rather, we derive target/ from staging/) then there would
be no point in reportign the two, as they would both be always true, so
we would only need to keep "install_target" and "install_images"
(obviously we could have a transition period during which we still
report "isntall_staging", but it would be slated for removal long term).

For the latter case, if we eventually have a way to express the licenses
in a format more complex that jsut the list of licenses, then the
"licenses" key would have to evolve from a simple string to a more
complex object.

(Note: the above is just two hypotetical examples, the first being still
probable medium-term.)

> If so, without a
> version, it means it can never break.

I disagree: we can't guarantee its stability. We can make it as stable
as possible, but we can't leave it in stone. Probably what's going to
happen is that old keys will stay for as long as they still exist, and
only new stuff will be added. But when something no longer exist, we
won't be able to expose it anymore...

So, I am totally fine with breaking the output of show-info when we
can't do otherwise.

> So let me suggest another layout
> which breaks `show-info` (name it `show-info-2` if you will).
> 
> Output example (using YAML only for clarity here):
> 
>     version: 2
>     build_steps:
>       target:
>         - name: download
>           stamp_file: .stamp_downloaded
>         - name: extract
>           stamp_file: .stamp_extracted
>         - name: patch
>           stamp_file: .stamp_patched
>         - name: build
>           stamp_file: .stamp_configured
>         - name: configure
>           stamp_file: .stamp_built
>         - name: install_staging
>           stamp_file: .stamp_staging_installed
>         - name: install_target
>           stamp_file: .stamp_target_installed
>         - name: install_image
>           stamp_file: .stamp_image_installed

What you're doing here is indirection; I don't like indirection.

By having the information directly in the nodes, it is readily
available; the data can be parsed as a stream, with little to no
memorisation.

With indirection, it means you have to memorise some data so it can be
reused later. With your example, it is very limited, indeed, but it
opens a can of worm: except for lists, json is not guaranteed to be
ordered, so there is no reason that we can expect the "build_steps"
definitions to be read before the rest, so we'd have to buffer the
entirety of show-info before we can even start doing anything with it.

[--SNIP--]
> * The root `packages` object contains the package information for each
>   package type.
>   Not strictly related to this patch or the previous one, but instead
>   of repeating the type in each object, you can just group packages
>   by type.
> 
> * For each package object, the `build_steps` array indicate what are the
>   build steps for this package.
>   Each item is the name of a build step in the corresponding root
>   `build_steps` object's array.
> 
> * For a target package, the `install_staging`, `install_target`, and
>   `install_images` entries are removed.
>   They are not needed anymore because `build_steps` offers the same
>   information.
>   I understand some build steps could be implicit and not listed here,
>   but with my suggestion, you already have what's needed for any build
>   monitoring tool to watch a specific package build process if:
> 
>   * Some of the implicit build steps become optional in the future for
>     any reason that you don't know now.
> 
>   * You add new types of build steps: simply describe them in the
>     `build_steps` root arrays.
> 
>   Both scenarios won't require an API version bump.

As I understand your proposal, your tool would not have to hard-code the
build steps, because they would defined globally, and each package would
reference (in an ordered list0 the steps that apply to it.

Well, no need to define the steps globally, since you already have the
information at the package level: just construct the build sequence for
that package from the build steps listed in that package.

> If you're not willing to break `show-info` and don't want `show-info-2`,
> then you can:
> 
> * Keep `show-info` as is:
>   * Augment a package information with `build_steps` (like above).
>   * Keep the `install_*` entries which become redundant.

That's exactly what I did: I just augment show-info with new
informnation. leaving the legacy install{target,staging,images} keys in
place for those that used to use them.

> * Have another make target, say `show-build-steps`, which only prints
>   the root `build_steps` object above.
> What do you think?

I don't like indirection. ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 7/8] core/show-info: report the ordered list of build steps

[Yann E. MORIN]

Thomas, All,

On 2020-04-11 16:19 +0200, Thomas Petazzoni spake thusly:
> On Sat, 11 Apr 2020 09:41:47 -0400
> Philippe Proulx <eeppeliteloop@gmail.com> wrote:
> > Replying for patch 6/8 and this one.
> > Is the `show-info` target considered a public API?
> Somewhat yes: the goal of show-info is that people can craft their own
> tooling on top of it. Breaking the show-info output would break such
> tools.

Which in some cases wil not be avoidable, see my previous reply.

> > If so, without a
> > version, it means it can never break. So let me suggest another layout
> > which breaks `show-info` (name it `show-info-2` if you will).
> > 
> > Output example (using YAML only for clarity here):
> This could certainly work, and reduces a bit the duplication of
> information.

The show-info is not for human consumption, I don't see a problem with
its content growing.

Also, as I explained in my previous reply to Philippe, having the
information where it is needed (i.e at the package level) allows to
process show-info in a stream, without having to memorise anything.

OTOH, having metadata description means the whole of show-info as to be
loaded in memory before any rocessing can outcur, since the content of
json is not guaranteeed to be ordered.

Yes, we could try and get our bare show-info output would be ordered,
but as soon as it gets processed (e.g. with json_pp or whatever), the
ordering would get lost.

> However, I still don't understand why your tool is not capable of
> having this knowledge about the stamp files. This is something that
> rarely changes. I think we haven't changed the stamp files since... 5
> years? More? If your tool is in the Buildroot tree itself, it can
> simply be updated if there is ever a change in the sequencing of steps.

That's true. But if you want to consider show-info to be a public API
and thus be stable,m then you would have to consider the stampfiles to
also be public API and thus stable, which is definitely what we want.

But that's also true that it changes so rarely that it would be
acceptable to break the tools.

And if we have such tools in-tree, we can fix them as we change the
internal details.

Regards,
Yann E. MORIN.

> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 7/8] core/show-info: report the ordered list of build steps

[Yann E. MORIN]

Philippe, All,

On 2020-04-11 11:06 -0400, Philippe Proulx spake thusly:
> On Sat, Apr 11, 2020 at 10:19 AM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
[--SNIP--]
> If you tell me that things such build step order and stamp file names do
> not change for many years and as such can be considered as interfaces,
> I'm fine with this.

They are not an interface, definitely not (IMHO). But they are stable
enough that you can build something on them and not expect a breakage
that often.

But an interface? No, we can't commit to that.

> It's always a balance between adding the same logic to all the viewers
> and generating less data.

Since the output of show-info is for machine consumption, and is not
geared at human consuption without first being processed, I don;t see a
problem with generating "more" data.

> My tool could be part of the Buildroot project, but I doubt you'll want
> something that depends on PyQt5 in there.

Not a problem, as it is not a mandatory tool. We already have
utils/brmake that uses unbuffer, which is not available on most distros
by default.

> However, I could certainly
> contribute a Python module which offers utilities over the `show-info`
> output so as to have a versioned, object-oriented API, and then import
> it from an external tool. This way, even if `show-info` changes, the
> Python module can continue to offer the same (non-breaking) interface.

You can't guarantee that either: some keys may disapear, so such a
module could not expose them any more either, so not a stable API.

All you could provide as a stable API, is a way to rerieve basic
information, like the list of packages and such. Or a way to query
keys, which would not provide much more than what 'import json' would.

Regards,
Yann E. MORIN.

> I still think we somewhat need the `build_dir` entry though.
> 
> Phil
> 
> >
> > Best regards,
> >
> > Thomas
> > --
> > Thomas Petazzoni, CTO, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/8] core/show-info: do not show install types for host packages

[Thomas Petazzoni]

On Sat, 11 Apr 2020 10:12:26 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> By their very nature, host packages have no target, staging, or
> images install steps; they have a single install step, that is
> always performed.
> 
> However, we currently report those three types of install, which
> is misleading at best, and really incorrect.
> 
> If we were to report any install type for host package, that would
> be a single one, and it would always be true.
> 
> So, do not report any install type for host packages, as it does
> not make sense to report anything that is always true.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vadim Kochan <vadim4j@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: eeppeliteloop at gmail.com
> ---
>  package/pkg-utils.mk | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/8] infra/pkg-generic: don't set INSTALL_{TARGET, STAGING, IMAGES} for host

[Thomas Petazzoni]

On Sat, 11 Apr 2020 10:12:27 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> By their very nature, host packages have no target, staging, or
> images install steps; they have a single install step, that is
> always performed.
> 
> As sucj, setting the corresponding _INSTALL_{TARGET,STAGING,IMAGES}
> variables does not make sense for host packages.
> 
> However, people (and scripts) may get confused when they process the
> output of printvars, e.g.:
> 
>     $ make printvars VARS=HOST_LIBTOOL_INSTALL_TARGET
>     HOST_LIBTOOL_INSTALL_TARGET=YES
> 
> Only set those variables for target packages. There is no corresponding
> variable for host packages, as they are always installed (and only
> once).
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vadim Kochan <vadim4j@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: eeppeliteloop at gmail.com
> ---
>  package/pkg-generic.mk | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 3/8] core/show-info: report install types for virtual packages too

[Thomas Petazzoni]

On Sat, 11 Apr 2020 10:12:28 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Although virtual packages do not usually install anything, they do
> participate in the dependency chain, and they also transition smoothly
> from 'not handled yet' all the way down to 'all set'. And in fact, it
> happens that some virtual packages do install something; for example,
> the virtual package 'toolchain' does tweak the musl headers during a
> post staging hook.
> 
> As such, people may want to know when those packages have been handled.
> To that goal, they will need to know what stampfile to look at (the
> target, staging, or images ones) to identify what step the virtual
> package is at.
> 
> So, report the type of install steps for virtual packages too, which
> means for all packages.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vadim Kochan <vadim4j@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: eeppeliteloop at gmail.com
> ---
>  package/pkg-utils.mk | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)

I've applied, but with a significantly reworked commit log to clarify
the intention. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 4/8] core/show-info: report the package build directory

[Thomas Petazzoni]

On Sat, 11 Apr 2020 10:12:29 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Some people want to be able to inspect the current state of the build,
> and report information about it, like:
>   - report the number of packages already built out of the total,
>   - list the packages being actually built (e.g. for TLPB)
>   - etc...
> 
> However, the location where a package is built is inherently an internal
> detail, so expose that to the user in the output of show-info. We only
> expose the location relative to the base directory (basically, either
> output/ or the user-suppiled $(O)), so that show-info does not contain
> local information (the output of show-info can be shared).
> 
> Interested parties will be able to poke in there to identify the stamp
> files and deduce the package's state.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vadim Kochan <vadim4j@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: eeppeliteloop at gmail.com
> ---
>  package/pkg-utils.mk | 1 +
>  1 file changed, 1 insertion(+)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 8/8] core/show-info: report image name of filesystems

[Thomas Petazzoni]

On Sat, 11 Apr 2020 10:12:33 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> The rootfses, unlike packages, do not have stampfiles to represent
> whether they are built or not. Indeed, we always rebuild the rootfs, and
> there is a single step to do.
> 
> Hpwever, people (and scripts) who want to report on the build progress,
> will want to know whether each rootfs has been built already or not.
> 
> Expose in show-info the name of the file that wil contain the rootfs
> image.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vadim Kochan <vadim4j@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: eeppeliteloop at gmail.com
> ---
>  package/pkg-utils.mk | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index bcdbf568fe..a464b660ef 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -174,6 +174,10 @@ define _json-info-pkg-details
>  endef
>  
>  define _json-info-fs
> +	"image_name": $(if $($(1)_FINAL_IMAGE_NAME), \
> +				"$($(1)_FINAL_IMAGE_NAME)", \
> +				null \
> +			),

Is the condition needed because of the initramfs case?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 0/8] core/show-info: export extra information about the package build (branch yem/show-info-extras)

[Thomas Petazzoni]

On Sat, 11 Apr 2020 10:12:30 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Yann E. MORIN (8):
>       core/show-info: do not show install types for host packages
>       infra/pkg-generic: don't set INSTALL_{TARGET,STAGING,IMAGES} for host
>       core/show-info: report install types for virtual packages too
>       core/show-info: report the package build directory

So, I've applied these first four patches.

>       core/show-info: report whether a package is overriden
>       core/show-info: report package stamp files
>       core/show-info: report the ordered list of build steps

Regarding these 3, as we have discussed, I am not in favor of these
changes. I think the "progress" tool should simply have the knowledge
of the stamp files and the possible order of steps. It's not like we
change that very often.

>       core/show-info: report image name of filesystems

I just replied to this one to ask a simple question about it.
Otherwise, I think it is OK.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 8/8] core/show-info: report image name of filesystems

[Yann E. MORIN]

Thomas, All,

On 2020-04-25 15:12 +0200, Thomas Petazzoni spake thusly:
> On Sat, 11 Apr 2020 10:12:33 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
> > The rootfses, unlike packages, do not have stampfiles to represent
> > whether they are built or not. Indeed, we always rebuild the rootfs, and
> > there is a single step to do.
> > 
> > Hpwever, people (and scripts) who want to report on the build progress,

*However

> > will want to know whether each rootfs has been built already or not.
> > 
> > Expose in show-info the name of the file that wil contain the rootfs
> > image.
> > 
> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> > Cc: Vadim Kochan <vadim4j@gmail.com>
> > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > Cc: eeppeliteloop at gmail.com
> > ---
> >  package/pkg-utils.mk | 4 ++++
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> > index bcdbf568fe..a464b660ef 100644
> > --- a/package/pkg-utils.mk
> > +++ b/package/pkg-utils.mk
> > @@ -174,6 +174,10 @@ define _json-info-pkg-details
> >  endef
> >  
> >  define _json-info-fs
> > +	"image_name": $(if $($(1)_FINAL_IMAGE_NAME), \
> > +				"$($(1)_FINAL_IMAGE_NAME)", \
> > +				null \
> > +			),
> 
> Is the condition needed because of the initramfs case?

As far as I remember, yes. I forgot about it when writing the commit
log, it seems...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Re: [Buildroot] [PATCH 8/8] core/show-info: report image name of filesystems

[Arnout Vandecappelle]

On 11/04/2020 10:12, Yann E. MORIN wrote:
> The rootfses, unlike packages, do not have stampfiles to represent
> whether they are built or not. Indeed, we always rebuild the rootfs, and
> there is a single step to do.
> 
> Hpwever, people (and scripts) who want to report on the build progress,
> will want to know whether each rootfs has been built already or not.
> 
> Expose in show-info the name of the file that wil contain the rootfs
> image.
> 

 Applied to master, thanks.

 I extended the commit message with an explanation of the _FINAL_IMAGE_NAME
condition.

 Regards,
 Arnout

> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vadim Kochan <vadim4j@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: eeppeliteloop@gmail.com
> ---
>  package/pkg-utils.mk | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index bcdbf568fe..a464b660ef 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -174,6 +174,10 @@ define _json-info-pkg-details
>  endef
>  
>  define _json-info-fs
> +	"image_name": $(if $($(1)_FINAL_IMAGE_NAME), \
> +				"$($(1)_FINAL_IMAGE_NAME)", \
> +				null \
> +			),
>  	"dependencies": [
>  		$(call make-comma-list,$(sort $($(1)_DEPENDENCIES)))
>  	]
> 
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot