[Buildroot] [pull request v4] Pull request for branch yem/instrument-build

[Buildroot] [pull request v4] Pull request for branch yem/instrument-build

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Hello All!

This series adds support for instrumenting the build process, reworked
after discussion both on the ML and during the developpers' day.

It works as thus:
  - we register a set of hooks
  - hooks are called before and after each steps

There are currently two hooks defined:
  - log steps with timings (as an example of how to do a hook)
  - call a user-supplied scripts

More hooks can be added later on. Specifically, we may want to add a
script (in support/scripts or wherever) that runs some verifications
after packages are installed (in target/ and staging/), for example
to check for RPATH, or to check scripts do not have build-dir paths
in them (eg python scripts)...

I'm planning on adding at least one other hook, that runs all scripts
present in support/step-hooks/ (or whatever its name).

Changes v3 -> v4:
  - rename user-supplied hook  (Thomas DS)
  - typoes  (Thomas DS)

Changes v2 -> v3:
  - rename the internal variable that holds the hooks  (Thomas DS)
  - misc typo fixes  (Thomas DS)

Changes v1 -> v2:
  - removed the intra-steps instrumentation  (Thomas, Peter)
  - accept a list of user-supplied hooks, not only one hook  (Thomas)

Regards,
Yann E. MORIN.


The following changes since commit 6d93f061eccad3b81a27e8cf02cb8b29e9dba44e:

  ethtool: bump version to 3.12.1 (2013-11-11 15:06:57 +0100)

are available in the git repository at:

  git://gitorious.org/buildroot/buildroot.git yem/instrument-build

for you to fetch changes up to 8e74486ebbb0c14cd011d2143b988d261f592f7a:

  pkg-infra: add user-supplied step-hooks (2013-11-11 15:59:16 +0100)

----------------------------------------------------------------
Yann E. MORIN (4):
      post-{build,images} hooks: export BUILD_DIR too
      pkg-infra: introduce pre/post-step hooks
      pkg-infra: add hook to log timing of steps
      pkg-infra: add user-supplied step-hooks

 Makefile                            |  4 +--
 docs/manual/customize-rootfs.txt    |  5 ++--
 docs/manual/debugging-buildroot.txt | 33 ++++++++++++++++++++
 docs/manual/developer-guide.txt     |  2 ++
 package/Makefile.in                 |  6 ++++
 package/pkg-generic.mk              | 60 +++++++++++++++++++++++++++++++++++++
 6 files changed, 106 insertions(+), 4 deletions(-)
 create mode 100644 docs/manual/debugging-buildroot.txt

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] post-{build, images} hooks: export BUILD_DIR too

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Also export BUILD_DIR for post-{build,images} hooks, so they do have
a place to store generated files.

Note: this will be more einteresting for the instrumentation of steps,
to come in a later patch.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 Makefile                         | 4 ++--
 docs/manual/customize-rootfs.txt | 5 +++--
 package/Makefile.in              | 6 ++++++
 3 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index 2f7db59..634151c 100644
--- a/Makefile
+++ b/Makefile
@@ -513,7 +513,7 @@ endif
 
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_BUILD_SCRIPT)), \
 		$(call MESSAGE,"Executing post-build script $(s)"); \
-		$(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+		$(USER_HOOKS_EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
 LOCALE_WHITELIST=$(BUILD_DIR)/locales.nopurge
@@ -559,7 +559,7 @@ endif
 target-post-image:
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \
-		$(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+		$(USER_HOOKS_EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
 toolchain-eclipse-register:
 	./support/scripts/eclipse-register-toolchain `readlink -f $(O)` $(notdir $(TARGET_CROSS)) $(BR2_ARCH)
diff --git a/docs/manual/customize-rootfs.txt b/docs/manual/customize-rootfs.txt
index 49a6b4b..450b3d5 100644
--- a/docs/manual/customize-rootfs.txt
+++ b/docs/manual/customize-rootfs.txt
@@ -41,6 +41,7 @@ there are a few ways to customize the resulting target filesystem.
     - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
     - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see
       xref:generic-package-reference[]
+    - +BUILD_DIR+: the directory where packages are extracted and built
     - +BINARIES_DIR+: the place where all binary files (aka images) are
       stored
     - +BASE_DIR+: the base output directory
@@ -78,8 +79,8 @@ in one of these _post-image scripts_ will require special handling
 
 Just like for the _post-build scripts_ mentioned above, you also have
 access to the following environment variables from your _post-image
-scripts_: +BUILDROOT_CONFIG+, +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+,
-+BINARIES_DIR+ and +BASE_DIR+.
+scripts_: +BUILDROOT_CONFIG+, +BUILD_DIR+, +HOST_DIR+, +STAGING_DIR+,
++TARGET_DIR+, +BINARIES_DIR+ and +BASE_DIR+.
 
 Additionally, each of the +BR2_ROOTFS_POST_BUILD_SCRIPT+ and
 +BR2_ROOTFS_POST_IMAGE_SCRIPT+ scripts will be passed the arguments
diff --git a/package/Makefile.in b/package/Makefile.in
index 612f3c7..5d52fbd 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -275,6 +275,12 @@ HOST_MAKE_ENV=PATH=$(HOST_PATH) \
 		PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig" \
 		PERLLIB="$(HOST_DIR)/usr/lib/perl"
 
+# This extra environment we can not export ourselves (eg. because some
+# packages use that variable internally, eg. uboot), so we have to
+# explicitly pass it to user-supplied external hooks (eg. post-build,
+# post-images)
+USER_HOOKS_EXTRA_ENV=\
+	BUILD_DIR=$(BUILD_DIR)
 
 ################################################################################
 # settings we need to pass to configure
-- 
1.8.1.2

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

This hooks will let us instrument the build process in many ways:
  - log current step to see what broke
  - time each step to see what is worth optimising
  - sanity-check installed files (rpath, overwritten files...)
  - call user-provided script
  - ...

The steps are coarse-grain, and all have a 'start' and a 'end' hooks.
Here is the list of available steps (8 total):
  - extract
  - patch
  - configure
  - build
  - install-host
  - install-staging
  - install-image
  - install-target

The download, clean and uninstall steps are not instrumented on purpose.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 package/pkg-generic.mk | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 75fda02..4519223 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -21,6 +21,30 @@
 ################################################################################
 
 ################################################################################
+# Helper functions to catch start/end of each step
+################################################################################
+
+# Those two functions are called by each step below.
+# They are responsible for calling all hooks defined in
+# $(GLOBAL_INSTRUMENTATION_HOOKS) and pass each of them
+# three arguments:
+#   $1: either 'start' or 'end'
+#   $2: the name of the step
+#   $3: the name of the package
+
+# Start step
+# $1: step name
+define step_start
+	$(foreach hook,$(GLOBAL_INSTRUMENTATION_HOOKS),$(call $(hook),start,$(1),$($(PKG)_NAME))$(sep))
+endef
+
+# End step
+# $1: step name
+define step_end
+	$(foreach hook,$(GLOBAL_INSTRUMENTATION_HOOKS),$(call $(hook),end,$(1),$($(PKG)_NAME))$(sep))
+endef
+
+################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
 
@@ -55,6 +79,7 @@ endif
 
 # Unpack the archive
 $(BUILD_DIR)/%/.stamp_extracted:
+	@$(call step_start,extract)
 	@$(call MESSAGE,"Extracting")
 	$(Q)mkdir -p $(@D)
 	$($(PKG)_EXTRACT_CMDS)
@@ -62,6 +87,7 @@ $(BUILD_DIR)/%/.stamp_extracted:
 	$(Q)chmod -R +rw $(@D)
 	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,extract)
 
 # Rsync the source directory if the <pkg>_OVERRIDE_SRCDIR feature is
 # used.
@@ -91,6 +117,7 @@ endif
 $(BUILD_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
 $(BUILD_DIR)/%/.stamp_patched: PATCH_BASE_DIRS = $($(PKG)_DIR_PREFIX)/$(RAWNAME) $(call qstrip,$(BR2_GLOBAL_PATCH_DIR))/$(RAWNAME)
 $(BUILD_DIR)/%/.stamp_patched:
+	@$(call step_start,patch)
 	@$(call MESSAGE,"Patching")
 	$(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(foreach p,$($(PKG)_PATCH),support/scripts/apply-patches.sh $(@D) $(DL_DIR) $(notdir $(p))$(sep))
@@ -107,31 +134,39 @@ $(BUILD_DIR)/%/.stamp_patched:
 	)
 	$(foreach hook,$($(PKG)_POST_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,patch)
 
 # Configure
 $(BUILD_DIR)/%/.stamp_configured:
+	@$(call step_start,configure)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	@$(call MESSAGE,"Configuring")
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,configure)
 
 # Build
 $(BUILD_DIR)/%/.stamp_built::
+	@$(call step_start,build)
 	@$(call MESSAGE,"Building")
 	$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,build)
 
 # Install to host dir
 $(BUILD_DIR)/%/.stamp_host_installed:
+	@$(call step_start,install-host)
 	@$(call MESSAGE,"Installing to host directory")
 	$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,install-host)
 
 # Install to staging dir
 $(BUILD_DIR)/%/.stamp_staging_installed:
+	@$(call step_start,install-staging)
 	@$(call MESSAGE,"Installing to staging directory")
 	$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
@@ -143,16 +178,20 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 				$(addprefix $(STAGING_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ;\
 	fi
 	$(Q)touch $@
+	@$(call step_end,install-staging)
 
 # Install to images dir
 $(BUILD_DIR)/%/.stamp_images_installed:
+	@$(call step_start,install-image)
 	@$(call MESSAGE,"Installing to images directory")
 	$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,install-image)
 
 # Install to target dir
 $(BUILD_DIR)/%/.stamp_target_installed:
+	@$(call step_start,install-target)
 	@$(call MESSAGE,"Installing to target")
 	$(if $(BR2_INIT_SYSTEMD),\
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
@@ -164,6 +203,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 		$(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
 	fi
 	$(Q)touch $@
+	@$(call step_end,install-target)
 
 # Clean package
 $(BUILD_DIR)/%/.stamp_cleaned:
-- 
1.8.1.2

[Buildroot] [PATCH 3/4] pkg-infra: add hook to log timing of steps

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

The timing information is stored in the file $(O)/build-time.log

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/pkg-generic.mk | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 4519223..1cf23a0 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -44,6 +44,17 @@ define step_end
 	$(foreach hook,$(GLOBAL_INSTRUMENTATION_HOOKS),$(call $(hook),end,$(1),$($(PKG)_NAME))$(sep))
 endef
 
+#######################################
+# Actual steps hooks
+
+# Time steps
+define step_time
+	printf "%s:%-5.5s:%-20.20s: %s\n"           \
+	       "$$(date +%s)" "$(1)" "$(2)" "$(3)"  \
+	       >>"$(BUILD_DIR)/build-time.log"
+endef
+GLOBAL_INSTRUMENTATION_HOOKS += step_time
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
-- 
1.8.1.2

[Buildroot] [PATCH 4/4] pkg-infra: add user-supplied step-hooks

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Allow user to supply their own step-hooks by passing a variable
on the make command-line:
    make BR2_INSTRUMENTATION_SCRIPTS=/path/to/my/script

This can be usefull to run site-specific actions at each step of the
build process, such as logging installed, removed or modified files,
do sanity checks on installed files...

It is possible to call more than one script, by passing a space-separated
lists of scripts to call.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 docs/manual/debugging-buildroot.txt | 33 +++++++++++++++++++++++++++++++++
 docs/manual/developer-guide.txt     |  2 ++
 package/pkg-generic.mk              |  9 +++++++++
 3 files changed, 44 insertions(+)
 create mode 100644 docs/manual/debugging-buildroot.txt

diff --git a/docs/manual/debugging-buildroot.txt b/docs/manual/debugging-buildroot.txt
new file mode 100644
index 0000000..5fa05b0
--- /dev/null
+++ b/docs/manual/debugging-buildroot.txt
@@ -0,0 +1,33 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[debugging-buildroot]]
+
+Debugging Buildroot
+-------------------
+
+It is possible to instrument the steps +Buildroot+ does when building
+packages. Define the variable +BR2_INSTRUMENTATION_SCRIPTS+ to contain
+the path of one or more scripts (or other executables), in a
+space-separated list, you want called before and after each step. The
+scripts are called in sequence, with three parameters:
+
+  - +start+ or +end+ to denote the start (resp. the end) of a step;
+  - the name of the step about to be started, or which just ended.
+  - the name of the package
+
+For example :
+
+----
+make BR2_INSTRUMENTATION_SCRIPTS="/path/to/my/script1 /path/to/my/script2"
+----
+
+That script has access to the following variables:
+
+  - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
+  - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see
+    xref:generic-package-reference[]
+  - +BUILD_DIR+: the directory where packages are extracted and built
+  - +BINARIES_DIR+: the place where all binary files (aka images) are
+    stored
+  - +BASE_DIR+: the base output directory
diff --git a/docs/manual/developer-guide.txt b/docs/manual/developer-guide.txt
index 8125ad5..9054dee 100644
--- a/docs/manual/developer-guide.txt
+++ b/docs/manual/developer-guide.txt
@@ -11,3 +11,5 @@ include::adding-packages.txt[]
 include::patch-policy.txt[]
 
 include::download-infra.txt[]
+
+include::debugging-buildroot.txt[]
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 1cf23a0..2822bd7 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -55,6 +55,15 @@ define step_time
 endef
 GLOBAL_INSTRUMENTATION_HOOKS += step_time
 
+# User-supplied script
+define step_user
+	@$(foreach user_hook, $(BR2_INSTRUMENTATION_SCRIPTS), \
+		$(USER_HOOKS_EXTRA_ENV) $(user_hook) "$(1)" "$(2)" "$(3)"$(sep))
+endef
+ifneq ($(BR2_INSTRUMENTATION_SCRIPTS),)
+GLOBAL_INSTRUMENTATION_HOOKS += step_user
+endif
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
-- 
1.8.1.2

[Buildroot] [PATCH 1/4] post-{build, images} hooks: export BUILD_DIR too

[Thomas De Schampheleire]

On Mon, Nov 11, 2013 at 4:03 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Also export BUILD_DIR for post-{build,images} hooks, so they do have
> a place to store generated files.
>
> Note: this will be more einteresting for the instrumentation of steps,
> to come in a later patch.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  Makefile                         | 4 ++--
>  docs/manual/customize-rootfs.txt | 5 +++--
>  package/Makefile.in              | 6 ++++++
>  3 files changed, 11 insertions(+), 4 deletions(-)

Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Thomas De Schampheleire]

On Mon, Nov 11, 2013 at 4:03 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> This hooks will let us instrument the build process in many ways:
>   - log current step to see what broke
>   - time each step to see what is worth optimising
>   - sanity-check installed files (rpath, overwritten files...)
>   - call user-provided script
>   - ...
>
> The steps are coarse-grain, and all have a 'start' and a 'end' hooks.
> Here is the list of available steps (8 total):
>   - extract
>   - patch
>   - configure
>   - build
>   - install-host
>   - install-staging
>   - install-image
>   - install-target
>
> The download, clean and uninstall steps are not instrumented on purpose.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  package/pkg-generic.mk | 40 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 40 insertions(+)

Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

[Buildroot] [PATCH 3/4] pkg-infra: add hook to log timing of steps

[Thomas De Schampheleire]

On Mon, Nov 11, 2013 at 4:03 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> The timing information is stored in the file $(O)/build-time.log
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
>  package/pkg-generic.mk | 11 +++++++++++
>  1 file changed, 11 insertions(+)

Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

[Buildroot] [PATCH 4/4] pkg-infra: add user-supplied step-hooks

[Thomas De Schampheleire]

On Mon, Nov 11, 2013 at 4:03 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Allow user to supply their own step-hooks by passing a variable
> on the make command-line:
>     make BR2_INSTRUMENTATION_SCRIPTS=/path/to/my/script
>
> This can be usefull to run site-specific actions at each step of the
> build process, such as logging installed, removed or modified files,
> do sanity checks on installed files...
>
> It is possible to call more than one script, by passing a space-separated
> lists of scripts to call.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  docs/manual/debugging-buildroot.txt | 33 +++++++++++++++++++++++++++++++++
>  docs/manual/developer-guide.txt     |  2 ++
>  package/pkg-generic.mk              |  9 +++++++++
>  3 files changed, 44 insertions(+)
>  create mode 100644 docs/manual/debugging-buildroot.txt

Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

[Buildroot] [PATCH 1/4] post-{build, images} hooks: export BUILD_DIR too

[Samuel Martin]

2013/11/11 Thomas De Schampheleire <patrickdepinguin@gmail.com>

> On Mon, Nov 11, 2013 at 4:03 PM, Yann E. MORIN <yann.morin.1998@free.fr>
> wrote:
> > From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >
> > Also export BUILD_DIR for post-{build,images} hooks, so they do have
> > a place to store generated files.
> >
> > Note: this will be more einteresting for the instrumentation of steps,
> > to come in a later patch.
> >
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> > ---
> >  Makefile                         | 4 ++--
> >  docs/manual/customize-rootfs.txt | 5 +++--
> >  package/Makefile.in              | 6 ++++++
> >  3 files changed, 11 insertions(+), 4 deletions(-)
>
> Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>

-- 
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131111/59f104f9/attachment.html>

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Samuel Martin]

2013/11/11 Thomas De Schampheleire <patrickdepinguin@gmail.com>

> On Mon, Nov 11, 2013 at 4:03 PM, Yann E. MORIN <yann.morin.1998@free.fr>
> wrote:
> > From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >
> > This hooks will let us instrument the build process in many ways:
> >   - log current step to see what broke
> >   - time each step to see what is worth optimising
> >   - sanity-check installed files (rpath, overwritten files...)
> >   - call user-provided script
> >   - ...
> >
> > The steps are coarse-grain, and all have a 'start' and a 'end' hooks.
> > Here is the list of available steps (8 total):
> >   - extract
> >   - patch
> >   - configure
> >   - build
> >   - install-host
> >   - install-staging
> >   - install-image
> >   - install-target
> >
> > The download, clean and uninstall steps are not instrumented on purpose.
> >
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> > ---
> >  package/pkg-generic.mk | 40 ++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 40 insertions(+)
>
> Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

-- 
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131111/172daead/attachment.html>

[Buildroot] [PATCH 3/4] pkg-infra: add hook to log timing of steps

[Samuel Martin]

2013/11/11 Thomas De Schampheleire <patrickdepinguin@gmail.com>

> On Mon, Nov 11, 2013 at 4:03 PM, Yann E. MORIN <yann.morin.1998@free.fr>
> wrote:
> > From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >
> > The timing information is stored in the file $(O)/build-time.log
> >
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > ---
> >  package/pkg-generic.mk | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
>
> Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>

-- 
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131111/7f5e82f3/attachment.html>

[Buildroot] [PATCH 4/4] pkg-infra: add user-supplied step-hooks

[Samuel Martin]

2013/11/11 Thomas De Schampheleire <patrickdepinguin@gmail.com>

> On Mon, Nov 11, 2013 at 4:03 PM, Yann E. MORIN <yann.morin.1998@free.fr>
> wrote:
> > From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >
> > Allow user to supply their own step-hooks by passing a variable
> > on the make command-line:
> >     make BR2_INSTRUMENTATION_SCRIPTS=/path/to/my/script
> >
> > This can be usefull to run site-specific actions at each step of the
> > build process, such as logging installed, removed or modified files,
> > do sanity checks on installed files...
> >
> > It is possible to call more than one script, by passing a space-separated
> > lists of scripts to call.
> >
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> > ---
> >  docs/manual/debugging-buildroot.txt | 33
> +++++++++++++++++++++++++++++++++
> >  docs/manual/developer-guide.txt     |  2 ++
> >  package/pkg-generic.mk              |  9 +++++++++
> >  3 files changed, 44 insertions(+)
> >  create mode 100644 docs/manual/debugging-buildroot.txt
>
> Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>

-- 
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131111/9fafa53d/attachment.html>

[Buildroot] [pull request v4] Pull request for branch yem/instrument-build

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Hello All!

> This series adds support for instrumenting the build process, reworked
> after discussion both on the ML and during the developpers' day.

> It works as thus:
>   - we register a set of hooks
>   - hooks are called before and after each steps

> There are currently two hooks defined:
>   - log steps with timings (as an example of how to do a hook)
>   - call a user-supplied scripts

> More hooks can be added later on. Specifically, we may want to add a
> script (in support/scripts or wherever) that runs some verifications
> after packages are installed (in target/ and staging/), for example
> to check for RPATH, or to check scripts do not have build-dir paths
> in them (eg python scripts)...

> I'm planning on adding at least one other hook, that runs all scripts
> present in support/step-hooks/ (or whatever its name).

> Changes v3 -> v4:
>   - rename user-supplied hook  (Thomas DS)
>   - typoes  (Thomas DS)

> Changes v2 -> v3:
>   - rename the internal variable that holds the hooks  (Thomas DS)
>   - misc typo fixes  (Thomas DS)

> Changes v1 -> v2:
>   - removed the intra-steps instrumentation  (Thomas, Peter)
>   - accept a list of user-supplied hooks, not only one hook  (Thomas)

> Regards,
> Yann E. MORIN.

Committed series to next, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 4/4] pkg-infra: add user-supplied step-hooks

[Thomas De Schampheleire]

Hi Yann,

On Mon, Nov 11, 2013 at 12:51 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Allow user to supply their own step-hooks by passing a variable
> on the make command-line:
>     make BR2_STEP_USER_HOOK=/path/to/my/script
>
> This can be usefull to run site-specific actions at each step of the

useful

> build process, such as logging installed, removed or modified files,
> do sanity checks on installed files...
>
> It is possible to call more than one script, by passing a space-separated
> lists of scripts to call.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
>  docs/manual/debugging-buildroot.txt | 33 +++++++++++++++++++++++++++++++++
>  docs/manual/developer-guide.txt     |  2 ++
>  package/pkg-generic.mk              |  9 +++++++++
>  3 files changed, 44 insertions(+)
>  create mode 100644 docs/manual/debugging-buildroot.txt
>
> diff --git a/docs/manual/debugging-buildroot.txt b/docs/manual/debugging-buildroot.txt
> new file mode 100644
> index 0000000..7e66208
> --- /dev/null
> +++ b/docs/manual/debugging-buildroot.txt
> @@ -0,0 +1,33 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +[[debugging-buildroot]]
> +
> +Debugging Buildroot
> +-------------------
> +
> +It is possible to instrument the steps +Buildroot+ does when building
> +packages. Define the variable +BR2_STEPS_USER_HOOKS+ to contain the path
> +of one or more scripts (or other executables), in a space-separated list,
> +you want called before and after each step. The script is called with
> +three parameters:
> +
> +  - +start+ or +end+ to denote the start (resp. the end) of a step;
> +  - the name of the step about to be started, or which just ended.
> +  - the name of the package
> +
> +For example :
> +
> +----
> +make BR2_STEPS_USER_HOOKS="/path/to/my/script1 /path/to/my/script2"
> +----

I think this variable should be named more in accordance with the name
GLOBAL_INSTRUMENTATION_HOOKS. For example:
BR2_USER_INSTRUMENTATION_HOOKS
or plainly:
BR2_INSTRUMENTATION_HOOKS
(or something else)

Best regards,
Thomas

[Buildroot] [PATCH 4/4] pkg-infra: add user-supplied step-hooks

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Allow user to supply their own step-hooks by passing a variable
on the make command-line:
    make BR2_STEP_USER_HOOK=/path/to/my/script

This can be usefull to run site-specific actions at each step of the
build process, such as logging installed, removed or modified files,
do sanity checks on installed files...

It is possible to call more than one script, by passing a space-separated
lists of scripts to call.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 docs/manual/debugging-buildroot.txt | 33 +++++++++++++++++++++++++++++++++
 docs/manual/developer-guide.txt     |  2 ++
 package/pkg-generic.mk              |  9 +++++++++
 3 files changed, 44 insertions(+)
 create mode 100644 docs/manual/debugging-buildroot.txt

diff --git a/docs/manual/debugging-buildroot.txt b/docs/manual/debugging-buildroot.txt
new file mode 100644
index 0000000..7e66208
--- /dev/null
+++ b/docs/manual/debugging-buildroot.txt
@@ -0,0 +1,33 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[debugging-buildroot]]
+
+Debugging Buildroot
+-------------------
+
+It is possible to instrument the steps +Buildroot+ does when building
+packages. Define the variable +BR2_STEPS_USER_HOOKS+ to contain the path
+of one or more scripts (or other executables), in a space-separated list,
+you want called before and after each step. The script is called with
+three parameters:
+
+  - +start+ or +end+ to denote the start (resp. the end) of a step;
+  - the name of the step about to be started, or which just ended.
+  - the name of the package
+
+For example :
+
+----
+make BR2_STEPS_USER_HOOKS="/path/to/my/script1 /path/to/my/script2"
+----
+
+That script has access to the following variables:
+
+  - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
+  - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see
+    xref:generic-package-reference[]
+  - +BUILD_DIR+: the directory where packages are extracted and built
+  - +BINARIES_DIR+: the place where all binary files (aka images) are
+    stored
+  - +BASE_DIR+: the base output directory
diff --git a/docs/manual/developer-guide.txt b/docs/manual/developer-guide.txt
index 8125ad5..9054dee 100644
--- a/docs/manual/developer-guide.txt
+++ b/docs/manual/developer-guide.txt
@@ -11,3 +11,5 @@ include::adding-packages.txt[]
 include::patch-policy.txt[]
 
 include::download-infra.txt[]
+
+include::debugging-buildroot.txt[]
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 5dba216..a8dd7da 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -54,6 +54,15 @@ define step_time
 endef
 GLOBAL_INSTRUMENTATION_HOOKS += step_time
 
+# User-supplied script
+define step_user
+	@$(foreach user_hook, $(BR2_STEPS_USER_HOOKS), \
+		$(USER_HOOKS_EXTRA_ENV) $(user_hook) "$(1)" "$(2)" "$(3)"$(sep))
+endef
+ifneq ($(BR2_STEPS_USER_HOOKS),)
+GLOBAL_INSTRUMENTATION_HOOKS += step_user
+endif
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
-- 
1.8.1.2

[Buildroot] [PATCH 4/4] pkg-infra: add user-supplied step-hooks

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Allow user to supply their own step-hooks by passing a variable
on the make command-line:
    make BR2_STEP_USER_HOOK=/path/to/my/script

This can be usefull to run site-specific actions at each step of the
build process, such as logging installed, removed or modified files,
do sanity checks on installed files...

It is possible to call more than one script, by passing a space-separated
lists of scripts to call.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 docs/manual/debugging-buildroot.txt | 33 +++++++++++++++++++++++++++++++++
 docs/manual/developer-guide.txt     |  2 ++
 package/pkg-generic.mk              |  9 +++++++++
 3 files changed, 44 insertions(+)
 create mode 100644 docs/manual/debugging-buildroot.txt

diff --git a/docs/manual/debugging-buildroot.txt b/docs/manual/debugging-buildroot.txt
new file mode 100644
index 0000000..7e66208
--- /dev/null
+++ b/docs/manual/debugging-buildroot.txt
@@ -0,0 +1,33 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[debugging-buildroot]]
+
+Debugging Buildroot
+-------------------
+
+It is possible to instrument the steps +Buildroot+ does when building
+packages. Define the variable +BR2_STEPS_USER_HOOKS+ to contain the path
+of one or more scripts (or other executables), in a space-separated list,
+you want called before and after each step. The script is called with
+three parameters:
+
+  - +start+ or +end+ to denote the start (resp. the end) of a step;
+  - the name of the step about to be started, or which just ended.
+  - the name of the package
+
+For example :
+
+----
+make BR2_STEPS_USER_HOOKS="/path/to/my/script1 /path/to/my/script2"
+----
+
+That script has access to the following variables:
+
+  - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
+  - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see
+    xref:generic-package-reference[]
+  - +BUILD_DIR+: the directory where packages are extracted and built
+  - +BINARIES_DIR+: the place where all binary files (aka images) are
+    stored
+  - +BASE_DIR+: the base output directory
diff --git a/docs/manual/developer-guide.txt b/docs/manual/developer-guide.txt
index 43272f5..2e6b7aa 100644
--- a/docs/manual/developer-guide.txt
+++ b/docs/manual/developer-guide.txt
@@ -13,3 +13,5 @@ include::patch-policy.txt[]
 include::download-infra.txt[]
 
 include::board-support.txt[]
+
+include::debugging-buildroot.txt[]
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 72e0e82..625dabc 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -54,6 +54,15 @@ define step_time
 endef
 STEP_HOOKS += step_time
 
+# User-supplied script
+define step_user
+	@$(foreach user_hook, $(BR2_STEPS_USER_HOOKS), \
+		$(USER_HOOKS_EXTRA_ENV) $(user_hook) "$(1)" "$(2)" "$(3)"$(sep))
+endef
+ifneq ($(BR2_STEPS_USER_HOOKS),)
+STEP_HOOKS += step_user
+endif
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
-- 
1.8.1.2

[Buildroot] [PATCH 4/4] pkg-infra: add user-supplied step-hooks

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Allow user to supply their own step-hooks by passing a variable
on the make command-line:
    make BR2_STEP_USER_HOOK=/path/to/my/script

This can be usefull to run site-specific actions at each step of the
build process, such as logging installed, removed or modified files,
do sanity checks on installed files...

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/pkg-generic.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 162c9ed..f5c33b2 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -42,6 +42,14 @@ define step_time
 endef
 STEP_HOOKS += step_time
 
+# User-supplied script
+define step_user
+	"$(BR2_STEP_USER_HOOK)" "$(1)" "$(2)" "$($(PKG)_NAME)"
+endef
+ifneq ($(BR2_STEP_USER_HOOK),)
+STEP_HOOKS += step_user
+endif
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
-- 
1.8.1.2