[Buildroot] [PATCH 0 of 2 v3-resend] Add find utility functions / exclude files from strip

[Buildroot] [PATCH 0 of 2 v3-resend] Add find utility functions / exclude files from strip

[Thomas De Schampheleire]

This patch series adds a few utility functions to Makefile.package.in that
help in composing find commands, and uses these functions to exclude
user-configured files and directories from being stripped in target-finalize.

The previous version of this series also modified the Python configuration to
allow for a 'reduced Python'. These patches were left out for now, as Thomas
Petazzoni raised some objections regarding the two-faced configuration. I hope
we can continue the discussion about this topic in that thread.

Thomas, you also had comments on the complexity of the find*clauses functions.
If you have alternative suggestions, please let me know. Thanks.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
v3: update based on review comments, remove Python changes for now
v2: simplify find utilities

 Config.in                   |  19 +++++++++++++++++++
 Makefile                    |   9 +++++++--
 package/Makefile.package.in |  11 +++++++++++
 3 files changed, 37 insertions(+), 2 deletions(-)

[Buildroot] [PATCH 1 of 2 v3-resend] Makefile.package.in: add utility functions find*clauses and notfirstword

[Thomas De Schampheleire]

This patch adds a few utility functions to Makefile.package.in.

Functions finddirclauses and findfileclauses help in building a find command
that skips a set of directories and performs operations on a set of files.
This pattern can for example be used to keep certain files or directories from
being stripped, or to remove certain files from a package installation.

The notfirstword function is the inverse of the 'firstword' function in make:
it returns all but the first word.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

---
v3: use -path instead of -wholename; fix a bug in finddirclauses. Thanks, Cam.
v2: integrate _cont variants in main find*clauses functions. Thanks, Arnout.

 package/Makefile.package.in |  11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -87,6 +87,17 @@ MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG)
 TERM_BOLD := $(shell tput smso)
 TERM_RESET := $(shell tput rmso)
 
+# Utility functions for 'find'
+# findfileclauses: creates "-name 'X' -o -name 'Y'"
+# [1:namelist]
+findfileclauses=-name '$(firstword $(1))' $(patsubst %,-o -name '%',$(call notfirstword,$(1)))
+# finddirclauses: creates "-path 'basedir/dirX' -o -path 'basedir/dirY'"
+# [1:basedir, 2:namelist]
+finddirclauses=-path '$(1)/$(firstword $(2))' $(patsubst %,-o -path '$(1)/%',$(call notfirstword,$(2)))
+
+# Miscellaneous utility functions
+notfirstword=$(wordlist 2,$(words $(1)),$(1))
+
 # Download method commands
 WGET:=$(call qstrip,$(BR2_WGET)) $(QUIET)
 SVN:=$(call qstrip,$(BR2_SVN))

[Buildroot] [PATCH 2 of 2 v3-resend] build: add option to exclude executables/dirs from being stripped

[Thomas De Schampheleire]

Sometimes it may be desirable to keep debug symbols for some binaries and
libraries on the target. This commit introduces the config option
BR2_STRIP_EXCLUDE_FILES, which is interpreted as a list of such binaries
and libraries, and the option BR2_STRIP_EXCLUDE_DIRS, which indicates
directories excluded from stripping entirely.
These exclusions are passed to the find command in the target-finalize step.

Signed-off-by: Sven Neumann <s.neumann@raumfeld.com>
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
v3: merge with redundant patch 'stripping: use findfileclauses utility function'
v2: add the possibility to exclude entire directories

 Config.in |  19 +++++++++++++++++++
 Makefile  |   9 +++++++--
 2 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/Config.in b/Config.in
--- a/Config.in
+++ b/Config.in
@@ -262,6 +262,25 @@ config BR2_STRIP_none
 	  none    do not strip (only for debugging!)
 endchoice
 
+config BR2_STRIP_EXCLUDE_FILES
+	string "executables that should not be stripped"
+	depends on !BR2_STRIP_none
+	default ""
+	help
+	  You may specify a space-separated list of binaries and libraries
+	  here that should not be stripped on the target.
+
+config BR2_STRIP_EXCLUDE_DIRS
+	string "directories that should be skipped when stripping"
+	depends on !BR2_STRIP_none
+	default ""
+	help
+	  You may specify a space-separated list of directories that should
+	  be skipped when stripping. Binaries and libraries in these
+	  directories will not be touched.
+	  The directories should be specified relative to the target directory,
+	  without leading slash.
+
 choice
 	prompt "gcc optimization level"
 	default BR2_OPTIMIZE_S
diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -429,6 +429,12 @@ endif
 erase-fakeroots:
 	rm -f $(BUILD_DIR)/.fakeroot*
 
+STRIP_FIND_CMD=find $(TARGET_DIR)
+ifneq (,$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS)))
+STRIP_FIND_CMD+=\( $(call finddirclauses,$(TARGET_DIR),$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS))) \) -prune -o
+endif
+STRIP_FIND_CMD+=-type f -perm +111 -not \( $(call findfileclauses,libthread_db*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print
+
 target-finalize:
 ifeq ($(BR2_HAVE_DEVFILES),y)
 	( support/scripts/copy.sh $(STAGING_DIR) $(TARGET_DIR) )
@@ -453,8 +459,7 @@ endif
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
 	find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f
 endif
-	find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' | \
-		xargs $(STRIPCMD) 2>/dev/null || true
+	$(STRIP_FIND_CMD) | xargs $(STRIPCMD) 2>/dev/null || true
 	find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
 		xargs -r $(KSTRIPCMD) || true
 

[Buildroot] [PATCH 1 of 2 v3-resend] Makefile.package.in: add utility functions find*clauses and notfirstword

[Arnout Vandecappelle]

On Thursday 09 February 2012 21:09:29 Thomas De Schampheleire wrote:
> +# Utility functions for 'find'
> +# findfileclauses: creates "-name 'X' -o -name 'Y'"
> +# [1:namelist]
> +findfileclauses=-name '$(firstword $(1))' $(patsubst %,-o -name '%',$(call notfirstword,$(1)))
> +# finddirclauses: creates "-path 'basedir/dirX' -o -path 'basedir/dirY'"
> +# [1:basedir, 2:namelist]
> +finddirclauses=-path '$(1)/$(firstword $(2))' $(patsubst %,-o -path '$(1)/%',$(call notfirstword,$(2)))
> +
> +# Miscellaneous utility functions
> +notfirstword=$(wordlist 2,$(words $(1)),$(1))
> 

 You asked for inspiration on how to simplify this; here are two (untested) ideas:

- a function 'join':

# join,<joinstr>,<wordlist>: puts <joinstr> between the words of <wordlist>
# e.g.: $(call join,+,a b c) -> a + b + c
join = $(firstword $(2)) $(addprefix $(1) ,$(call notfirstword,$(2))

findfileclauses = $(call join,-o,$(patsubst %,-name '%',$(1))

- strip the first -o after joining:
findfileclauses = $(call notfirstword,$(patsubst %, -o -name '%',$(1)))
finddirclauses = $(call notfirstword,$(patsubst %, -o -path '$(1)/%',$(2)))


 Regards,
 Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 2 of 2 v3-resend] build: add option to exclude executables/dirs from being stripped

[Arnout Vandecappelle]

On Thursday 09 February 2012 21:09:30 Thomas De Schampheleire wrote:
> Sometimes it may be desirable to keep debug symbols for some binaries and
> libraries on the target. This commit introduces the config option
> BR2_STRIP_EXCLUDE_FILES, which is interpreted as a list of such binaries
> and libraries, and the option BR2_STRIP_EXCLUDE_DIRS, which indicates
> directories excluded from stripping entirely.
> These exclusions are passed to the find command in the target-finalize step.
> 
> Signed-off-by: Sven Neumann <s.neumann@raumfeld.com>
> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>


[snip]
> +STRIP_FIND_CMD=find $(TARGET_DIR)
> +ifneq (,$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS)))
> +STRIP_FIND_CMD+=\( $(call finddirclauses,$(TARGET_DIR),$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS))) \) -prune -o
> +endif
> +STRIP_FIND_CMD+=-type f -perm +111 -not \( $(call findfileclauses,libthread_db*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print

 If for whatever reason this patch has to go through another cycle, I'd
like the following whitespace changes:

- space before and after the =

- split up the last line in two, e.g.

STRIP_FIND_CMD += -type f -perm +111 
STRIP_FIND_CMD += -not \( $(call findfileclauses,libthread_db*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print

 (still 120 columns wide, but better already).

 Regards,
 Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 1 of 2 v3-resend] Makefile.package.in: add utility functions find*clauses and notfirstword

[Thomas De Schampheleire]

Hi Arnout,

On Sat, Feb 11, 2012 at 5:57 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On Thursday 09 February 2012 21:09:29 Thomas De Schampheleire wrote:
>> +# Utility functions for 'find'
>> +# findfileclauses: creates "-name 'X' -o -name 'Y'"
>> +# [1:namelist]
>> +findfileclauses=-name '$(firstword $(1))' $(patsubst %,-o -name '%',$(call notfirstword,$(1)))
>> +# finddirclauses: creates "-path 'basedir/dirX' -o -path 'basedir/dirY'"
>> +# [1:basedir, 2:namelist]
>> +finddirclauses=-path '$(1)/$(firstword $(2))' $(patsubst %,-o -path '$(1)/%',$(call notfirstword,$(2)))
>> +
>> +# Miscellaneous utility functions
>> +notfirstword=$(wordlist 2,$(words $(1)),$(1))
>>
>
> ?You asked for inspiration on how to simplify this; here are two (untested) ideas:
>
> - a function 'join':
>
> # join,<joinstr>,<wordlist>: puts <joinstr> between the words of <wordlist>
> # e.g.: $(call join,+,a b c) -> a + b + c
> join = $(firstword $(2)) $(addprefix $(1) ,$(call notfirstword,$(2))
>
> findfileclauses = $(call join,-o,$(patsubst %,-name '%',$(1))
>
> - strip the first -o after joining:
> findfileclauses = $(call notfirstword,$(patsubst %, -o -name '%',$(1)))
> finddirclauses = $(call notfirstword,$(patsubst %, -o -path '$(1)/%',$(2)))

Thanks for these suggestions. I eventually used the second one.
I tried the first one, because the join function could have been
useful in other situations. However, there were two issues.
1. 'join' already is a make function (of course another name would suffice).
2. Because make uses whitespace to separate words, join cannot be used
here: the words are in fact groups of words: '-name X'. As a result,
'firstword $(2)' would be '-name' and 'notfirstword $(2)' would be
'X'. This conflicts with our purpose.

The second suggestion is also very nice and easily understandable, so
I took that as alternative.

Best regards,
Thomas