[Buildroot] [RFC 0/4] add genimage Kconfig entry

[Buildroot] [RFC 0/4] add genimage Kconfig entry

[Etienne Phelip]

Many boards specific postimage scripts are almost use only to execute
the genimage host tool.

The purpose of this RFC is to provide a generic wrapper for the
genimage tool, in order to reduce boilerplate.

The first two patches add the generic genimage plumbing.
The next two patches are examples of post-image scripts simplification.

The new target-genimage targer is called after the target-post-image
target since that is most of the board use cases. odroidc2 is an
exception, we can still execute genimage in its post-image script.

If there is no objection to the first two patches, I will submit a new
patch series introducing this change to all boards using genimage.

Etienne Phelip (4):
  support/scripts: add generic genimage script
  system: add option to pass genimage config files
  configs/raspberrypi3_defconfig: set BR2_GENIMAGE_CFG_FILES
  board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

 Makefile                                     |  9 +++++++--
 board/acmesystems/aria-g25/post-image.sh     | 14 --------------
 board/raspberrypi/post-image.sh              | 16 ----------------
 configs/acmesystems_aria_g25_128mb_defconfig |  2 +-
 configs/acmesystems_aria_g25_256mb_defconfig |  2 +-
 configs/raspberrypi3_defconfig               |  1 +
 support/scripts/genimage.sh                  | 27 +++++++++++++++++++++++++++
 system/Config.in                             | 11 +++++++++++
 8 files changed, 48 insertions(+), 34 deletions(-)
 delete mode 100755 board/acmesystems/aria-g25/post-image.sh
 create mode 100755 support/scripts/genimage.sh

-- 
2.9.3

[Buildroot] [RFC 1/4] support/scripts: add generic genimage script

[Etienne Phelip]

This script is a wrapper for the genimage tool used by most boards.
The board postimage script can now call this script instead of invoking
genimage command themselves.

Signed-off-by: Etienne Phelip <etienne.phelip@savoirfairelinux.com>
---
 support/scripts/genimage.sh | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
 create mode 100755 support/scripts/genimage.sh

diff --git a/support/scripts/genimage.sh b/support/scripts/genimage.sh
new file mode 100755
index 0000000..0ed0e8b
--- /dev/null
+++ b/support/scripts/genimage.sh
@@ -0,0 +1,27 @@
+#!/bin/bash
+
+die() {
+  echo "Error: $@" >&2
+  exit 1
+}
+
+GENIMAGE_TMP="${BUILD_DIR}/genimage.tmp"
+
+while getopts c: OPT ; do
+	case "${OPT}" in
+	c) GENIMAGE_CFG="${OPTARG}";;
+	:) die "option '${OPTARG}' expects a mandatory argument\n";;
+	\?) die "unknown option '${OPTARG}'\n";;
+	esac
+done
+
+[ -n "${GENIMAGE_CFG}" ] || die "Missing argument"
+
+rm -rf "${GENIMAGE_TMP}"
+
+genimage \
+	--rootpath "${TARGET_DIR}"     \
+	--tmppath "${GENIMAGE_TMP}"    \
+	--inputpath "${BINARIES_DIR}"  \
+	--outputpath "${BINARIES_DIR}" \
+	--config "${GENIMAGE_CFG}"
-- 
2.9.3

[Buildroot] [RFC 1/4] support/scripts: add generic genimage script

[Arnout Vandecappelle]

On 29-03-17 16:51, Etienne Phelip wrote:
> This script is a wrapper for the genimage tool used by most boards.
> The board postimage script can now call this script instead of invoking
> genimage command themselves.
> 
> Signed-off-by: Etienne Phelip <etienne.phelip@savoirfairelinux.com>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

> ---
>  support/scripts/genimage.sh | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
>  create mode 100755 support/scripts/genimage.sh
> 
> diff --git a/support/scripts/genimage.sh b/support/scripts/genimage.sh
> new file mode 100755
> index 0000000..0ed0e8b
> --- /dev/null
> +++ b/support/scripts/genimage.sh
> @@ -0,0 +1,27 @@
> +#!/bin/bash
> +
> +die() {
> +  echo "Error: $@" >&2
> +  exit 1
> +}
> +
> +GENIMAGE_TMP="${BUILD_DIR}/genimage.tmp"
> +
> +while getopts c: OPT ; do
> +	case "${OPT}" in
> +	c) GENIMAGE_CFG="${OPTARG}";;
> +	:) die "option '${OPTARG}' expects a mandatory argument\n";;
> +	\?) die "unknown option '${OPTARG}'\n";;
> +	esac
> +done
> +
> +[ -n "${GENIMAGE_CFG}" ] || die "Missing argument"
> +
> +rm -rf "${GENIMAGE_TMP}"
> +
> +genimage \
> +	--rootpath "${TARGET_DIR}"     \
> +	--tmppath "${GENIMAGE_TMP}"    \
> +	--inputpath "${BINARIES_DIR}"  \
> +	--outputpath "${BINARIES_DIR}" \
> +	--config "${GENIMAGE_CFG}"
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFC 1/4] support/scripts: add generic genimage script

[Thomas Petazzoni]

Hello,

On Wed, 29 Mar 2017 10:51:17 -0400, Etienne Phelip wrote:
> This script is a wrapper for the genimage tool used by most boards.
> The board postimage script can now call this script instead of invoking
> genimage command themselves.
> 
> Signed-off-by: Etienne Phelip <etienne.phelip@savoirfairelinux.com>
> ---
>  support/scripts/genimage.sh | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
>  create mode 100755 support/scripts/genimage.sh

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC 2/4] system: add option to pass genimage config files

[Etienne Phelip]

Add a new Kconfig entry to allow users to provide a list of genimage
config files.

A new target is added to the Makefile in order to execute the
support/scripts/genimage.sh script for each provided config files.

The new target-genimage target depends on target-post-image, thus
update the world target dependency.

Signed-off-by: Etienne Phelip <etienne.phelip@savoirfairelinux.com>
---
 Makefile         |  9 +++++++--
 system/Config.in | 11 +++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index cceae92..71c750c 100644
--- a/Makefile
+++ b/Makefile
@@ -546,11 +546,11 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: target-post-image
+world: target-genimage
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars help \
-	list-defconfigs target-finalize target-post-image source-check
+	list-defconfigs target-finalize target-post-image target-genimage source-check
 
 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR):
@@ -719,6 +719,11 @@ target-post-image: $(TARGETS_ROOTFS) target-finalize
 		$(call MESSAGE,"Executing post-image script $(s)"); \
 		$(EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
+target-genimage: target-post-image
+	@$(foreach c, $(call qstrip,$(BR2_GENIMAGE_CFG_FILES)), \
+		$(call MESSAGE,"Executing genimage with config $(c)"); \
+		$(EXTRA_ENV) ./support/scripts/genimage.sh -c $(c)$(sep))
+
 source: $(foreach p,$(PACKAGES),$(p)-all-source)
 
 _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps)
diff --git a/system/Config.in b/system/Config.in
index 3ddf843..d95ba43 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -564,4 +564,15 @@ config BR2_ROOTFS_POST_SCRIPT_ARGS
 	  directory / images directory. The arguments in this option will be
 	  passed *after* those.
 
+config BR2_GENIMAGE_CFG_FILES
+	string "genimage config files passed to genimage"
+	depends on BR2_PACKAGE_HOST_GENIMAGE
+	default ""
+	help
+	  Specify a space-separated list of genimage config files to used by
+	  genimage after the post-image script has been executed.
+
+	  This can be used to generate multiple binary image for one or more
+	  mediums (e.g. SD card and flash).
+
 endmenu
-- 
2.9.3

[Buildroot] [RFC 3/4] configs/raspberrypi3_defconfig: set BR2_GENIMAGE_CFG_FILES

[Etienne Phelip]

Use the new BR2_GENIMAGE_CFG_FILES Kconfig symbol to reduce
boilerplate and use the generic genimage wrapper.

Signed-off-by: Etienne Phelip <etienne.phelip@savoirfairelinux.com>
---
 board/raspberrypi/post-image.sh | 16 ----------------
 configs/raspberrypi3_defconfig  |  1 +
 2 files changed, 1 insertion(+), 16 deletions(-)

diff --git a/board/raspberrypi/post-image.sh b/board/raspberrypi/post-image.sh
index b2bb070..6d8e62c 100755
--- a/board/raspberrypi/post-image.sh
+++ b/board/raspberrypi/post-image.sh
@@ -1,10 +1,5 @@
 #!/bin/sh
 
-BOARD_DIR="$(dirname $0)"
-BOARD_NAME="$(basename ${BOARD_DIR})"
-GENIMAGE_CFG="${BOARD_DIR}/genimage-${BOARD_NAME}.cfg"
-GENIMAGE_TMP="${BUILD_DIR}/genimage.tmp"
-
 case "${2}" in
 	--add-pi3-miniuart-bt-overlay)
 	if ! grep -qE '^dtoverlay=' "${BINARIES_DIR}/rpi-firmware/config.txt"; then
@@ -17,14 +12,3 @@ __EOF__
 	fi
 	;;
 esac
-
-rm -rf "${GENIMAGE_TMP}"
-
-genimage                           \
-	--rootpath "${TARGET_DIR}"     \
-	--tmppath "${GENIMAGE_TMP}"    \
-	--inputpath "${BINARIES_DIR}"  \
-	--outputpath "${BINARIES_DIR}" \
-	--config "${GENIMAGE_CFG}"
-
-exit $?
diff --git a/configs/raspberrypi3_defconfig b/configs/raspberrypi3_defconfig
index de0c42d..a6deff4 100644
--- a/configs/raspberrypi3_defconfig
+++ b/configs/raspberrypi3_defconfig
@@ -35,3 +35,4 @@ BR2_TARGET_ROOTFS_EXT2_4=y
 BR2_ROOTFS_POST_BUILD_SCRIPT="board/raspberrypi3/post-build.sh"
 BR2_ROOTFS_POST_IMAGE_SCRIPT="board/raspberrypi3/post-image.sh"
 BR2_ROOTFS_POST_SCRIPT_ARGS="--add-pi3-miniuart-bt-overlay"
+BR2_GENIMAGE_CFG_FILES="board/raspberrypi3/genimage-raspberrypi3.cfg"
-- 
2.9.3

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Etienne Phelip]

Use the new BR2_GENIMAGE_CFG_FILES Kconfig symbol to reduce
boilerplate and use the generic genimage wrapper.

Signed-off-by: Etienne Phelip <etienne.phelip@savoirfairelinux.com>
---
 board/acmesystems/aria-g25/post-image.sh     | 14 --------------
 configs/acmesystems_aria_g25_128mb_defconfig |  2 +-
 configs/acmesystems_aria_g25_256mb_defconfig |  2 +-
 3 files changed, 2 insertions(+), 16 deletions(-)
 delete mode 100755 board/acmesystems/aria-g25/post-image.sh

diff --git a/board/acmesystems/aria-g25/post-image.sh b/board/acmesystems/aria-g25/post-image.sh
deleted file mode 100755
index 2846f56..0000000
--- a/board/acmesystems/aria-g25/post-image.sh
+++ /dev/null
@@ -1,14 +0,0 @@
-#!/bin/sh
-
-BOARD_DIR="$(dirname $0)"
-GENIMAGE_CFG="${BOARD_DIR}/genimage.cfg"
-GENIMAGE_TMP="${BUILD_DIR}/genimage.tmp"
-
-rm -rf "${GENIMAGE_TMP}"
-
-genimage  \
-	--rootpath "${TARGET_DIR}"     \
-	--tmppath "${GENIMAGE_TMP}"    \
-	--inputpath "${BINARIES_DIR}"  \
-	--outputpath "${BINARIES_DIR}" \
-	--config "${GENIMAGE_CFG}"
diff --git a/configs/acmesystems_aria_g25_128mb_defconfig b/configs/acmesystems_aria_g25_128mb_defconfig
index 0ec210f..23bb8b9 100644
--- a/configs/acmesystems_aria_g25_128mb_defconfig
+++ b/configs/acmesystems_aria_g25_128mb_defconfig
@@ -8,7 +8,6 @@ BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_4_7=y
 BR2_TARGET_GENERIC_ISSUE="Welcome to Aria-G25 Buildroot"
 BR2_TARGET_GENERIC_GETTY_PORT="ttyS0"
 BR2_TARGET_GENERIC_GETTY_BAUDRATE_115200=y
-BR2_ROOTFS_POST_IMAGE_SCRIPT="board/acmesystems/aria-g25/post-image.sh"
 
 # Kernel configuration
 BR2_LINUX_KERNEL=y
@@ -34,3 +33,4 @@ BR2_TARGET_AT91BOOTSTRAP3_DEFCONFIG="aria-128m"
 BR2_PACKAGE_HOST_DOSFSTOOLS=y
 BR2_PACKAGE_HOST_GENIMAGE=y
 BR2_PACKAGE_HOST_MTOOLS=y
+BR2_GENIMAGE_CFG_FILES="board/acmesystems/aria-g25/genimage.cfg"
diff --git a/configs/acmesystems_aria_g25_256mb_defconfig b/configs/acmesystems_aria_g25_256mb_defconfig
index a480287..9b44bbe 100644
--- a/configs/acmesystems_aria_g25_256mb_defconfig
+++ b/configs/acmesystems_aria_g25_256mb_defconfig
@@ -8,7 +8,6 @@ BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_4_7=y
 BR2_TARGET_GENERIC_ISSUE="Welcome to Aria-G25 Buildroot"
 BR2_TARGET_GENERIC_GETTY_PORT="ttyS0"
 BR2_TARGET_GENERIC_GETTY_BAUDRATE_115200=y
-BR2_ROOTFS_POST_IMAGE_SCRIPT="board/acmesystems/aria-g25/post-image.sh"
 
 # Kernel configuration
 BR2_LINUX_KERNEL=y
@@ -34,3 +33,4 @@ BR2_TARGET_AT91BOOTSTRAP3_DEFCONFIG="aria-256m"
 BR2_PACKAGE_HOST_DOSFSTOOLS=y
 BR2_PACKAGE_HOST_GENIMAGE=y
 BR2_PACKAGE_HOST_MTOOLS=y
+BR2_GENIMAGE_CFG_FILES="board/acmesystems/aria-g25/genimage.cfg"
-- 
2.9.3

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Arnout Vandecappelle]

On 29-03-17 16:51, Etienne Phelip wrote:
> diff --git a/configs/acmesystems_aria_g25_128mb_defconfig b/configs/acmesystems_aria_g25_128mb_defconfig
> index 0ec210f..23bb8b9 100644
> --- a/configs/acmesystems_aria_g25_128mb_defconfig
> +++ b/configs/acmesystems_aria_g25_128mb_defconfig
> @@ -8,7 +8,6 @@ BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_4_7=y
>  BR2_TARGET_GENERIC_ISSUE="Welcome to Aria-G25 Buildroot"
>  BR2_TARGET_GENERIC_GETTY_PORT="ttyS0"
>  BR2_TARGET_GENERIC_GETTY_BAUDRATE_115200=y
> -BR2_ROOTFS_POST_IMAGE_SCRIPT="board/acmesystems/aria-g25/post-image.sh"
>  
>  # Kernel configuration
>  BR2_LINUX_KERNEL=y
> @@ -34,3 +33,4 @@ BR2_TARGET_AT91BOOTSTRAP3_DEFCONFIG="aria-128m"
>  BR2_PACKAGE_HOST_DOSFSTOOLS=y
>  BR2_PACKAGE_HOST_GENIMAGE=y
>  BR2_PACKAGE_HOST_MTOOLS=y
> +BR2_GENIMAGE_CFG_FILES="board/acmesystems/aria-g25/genimage.cfg"

 I'm still not entirely sure if this new option is worthwhile. Without it, you
would instead need

BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
BR2_ROOTFS_POST_SCRIPT_ARGS="board/acmesystems/aria-g25/genimage.cfg"

Not really complicated either... The only real advantage I see is that it
motivates people more to use genimage (which is otherwise a bit hidden). But for
that, a paragraph or two in docs/manual/customize-post-image.txt would also
work. By the way, even for this series an explanation of the option would be
required in that file.

 What do the others think?


 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Andreas Naumann]

Hi Phelip,

Am 31.03.2017 um 00:51 schrieb Arnout Vandecappelle:
>
>  I'm still not entirely sure if this new option is worthwhile. Without it, you
> would instead need
>
> BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
> BR2_ROOTFS_POST_SCRIPT_ARGS="board/acmesystems/aria-g25/genimage.cfg"
>
> Not really complicated either... The only real advantage I see is that it
> motivates people more to use genimage (which is otherwise a bit hidden). But for
> that, a paragraph or two in docs/manual/customize-post-image.txt would also
> work. By the way, even for this series an explanation of the option would be
> required in that file.
>
>  What do the others think?

I have a slightly different use case than most genimage configs in 
buildroot right now. This is, I use genimage's ability to split my 
customized rootfs into multiple filesystems in order to give them them 
ro/writeable and other capabilities as appropriate.
Thus to preserve ownership and such I need to run genimage under 
fakeroot. Currently I do this explicitely in a post image script. An 
improvement I think about is switching to using the post fakeroot 
integration which buildroot provides since a while.

So unless this becomes some kind of option, your proposed change 
probably wouldnt work for me.

What I was thinking about in the past would be a deeper integration of 
genimage in buildroots filesystem configuration, actually generating the 
genimage config. But so far it was not worth the effort for me.


regards,
Andreas

>
>
>  Regards,
>  Arnout
>

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Arnout Vandecappelle]

On 31-03-17 09:46, Andreas Naumann wrote:
> Hi Phelip,
> 
> Am 31.03.2017 um 00:51 schrieb Arnout Vandecappelle:
>>
>>  I'm still not entirely sure if this new option is worthwhile. Without it, you
>> would instead need
>>
>> BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
>> BR2_ROOTFS_POST_SCRIPT_ARGS="board/acmesystems/aria-g25/genimage.cfg"
>>
>> Not really complicated either... The only real advantage I see is that it
>> motivates people more to use genimage (which is otherwise a bit hidden). But for
>> that, a paragraph or two in docs/manual/customize-post-image.txt would also
>> work. By the way, even for this series an explanation of the option would be
>> required in that file.
>>
>>  What do the others think?
> 
> I have a slightly different use case than most genimage configs in buildroot
> right now. This is, I use genimage's ability to split my customized rootfs into
> multiple filesystems in order to give them them ro/writeable and other
> capabilities as appropriate.
> Thus to preserve ownership and such I need to run genimage under fakeroot.

 It would indeed make sense to run genimage under fakeroot, but then it'd need
to have some option to extract the tarball first. Actually, that's something
that could be added to the genimage.sh script at some point...

> Currently I do this explicitely in a post image script. An improvement I think
> about is switching to using the post fakeroot integration which buildroot
> provides since a while.
> 
> So unless this becomes some kind of option, your proposed change probably
> wouldnt work for me.

 It is certainly optional, you can always call genimage directly from you
post-image script. You can also call the genimage.sh script but I don't know if
that's useful in your case.

> What I was thinking about in the past would be a deeper integration of genimage
> in buildroots filesystem configuration, actually generating the genimage config.
> But so far it was not worth the effort for me.

 We have considered that in the past. The problem is that genimage doesn't
support some of the options that we do. Otherwise, however, it would be really
nice to remove all our messy filesystem handling code and just generate a
genimage.cfg file...

 OTOH we also considered that a full image typically needs more than just a
single filesystem, so generating a genimage.cfg for part of the image seems a
bit pointless.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Andreas Naumann]

Hi,

Am 31.03.2017 um 18:34 schrieb Arnout Vandecappelle:
>
>
> On 31-03-17 09:46, Andreas Naumann wrote:
>> Hi Phelip,
>>
>> Am 31.03.2017 um 00:51 schrieb Arnout Vandecappelle:
>>>
>>>  I'm still not entirely sure if this new option is worthwhile. Without it, you
>>> would instead need
>>>
>>> BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
>>> BR2_ROOTFS_POST_SCRIPT_ARGS="board/acmesystems/aria-g25/genimage.cfg"
>>>
>>> Not really complicated either... The only real advantage I see is that it
>>> motivates people more to use genimage (which is otherwise a bit hidden). But for
>>> that, a paragraph or two in docs/manual/customize-post-image.txt would also
>>> work. By the way, even for this series an explanation of the option would be
>>> required in that file.
>>>
>>>  What do the others think?
>>
>> I have a slightly different use case than most genimage configs in buildroot
>> right now. This is, I use genimage's ability to split my customized rootfs into
>> multiple filesystems in order to give them them ro/writeable and other
>> capabilities as appropriate.
>> Thus to preserve ownership and such I need to run genimage under fakeroot.
>
>  It would indeed make sense to run genimage under fakeroot, but then it'd need
> to have some option to extract the tarball first. Actually, that's something
> that could be added to the genimage.sh script at some point...
>
>> Currently I do this explicitely in a post image script. An improvement I think
>> about is switching to using the post fakeroot integration which buildroot
>> provides since a while.
>>
>> So unless this becomes some kind of option, your proposed change probably
>> wouldnt work for me.
>
>  It is certainly optional, you can always call genimage directly from you
> post-image script. You can also call the genimage.sh script but I don't know if
> that's useful in your case.

True, Phelip's option [RFC 2/4] doesnt take away other options.

>
>> What I was thinking about in the past would be a deeper integration of genimage
>> in buildroots filesystem configuration, actually generating the genimage config.
>> But so far it was not worth the effort for me.
>
>  We have considered that in the past. The problem is that genimage doesn't
> support some of the options that we do. Otherwise, however, it would be really

Are you refering to filesystem-specific options or missing ones like 
axfs, cloop, cramfs?


> nice to remove all our messy filesystem handling code and just generate a
> genimage.cfg file...
>
>  OTOH we also considered that a full image typically needs more than just a
> single filesystem, so generating a genimage.cfg for part of the image seems a
> bit pointless.

I dont quite understand. Do you mean combining e.g. a standard and a 
rescue system?


regards,
Andreas


>
>  Regards,
>  Arnout
>

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Arnout Vandecappelle]

On 03-04-17 15:01, Andreas Naumann wrote:
>>> What I was thinking about in the past would be a deeper integration of genimage
>>> in buildroots filesystem configuration, actually generating the genimage config.
>>> But so far it was not worth the effort for me.
>>
>>  We have considered that in the past. The problem is that genimage doesn't
>> support some of the options that we do. Otherwise, however, it would be really
> 
> Are you refering to filesystem-specific options or missing ones like axfs,
> cloop, cramfs?

 I'm refering to filesystem-specific options. E.g. AFAIK genimage doesn't allow
configuration of ext2 #inodes or reserved blocks, or UBIFS max LEB count or
compression.


>> nice to remove all our messy filesystem handling code and just generate a
>> genimage.cfg file...

 And my point here is: we could just extend genimage with all those options.
Then we can use genimage instead of our rootfs handling. For the most part, the
rootfs infra could be reduced to a single genimage.cfg file.

 But as usual: all that would be nice in a way, but doesn't give us any direct
advantage over what we have now, and what we have now works.


>>
>>  OTOH we also considered that a full image typically needs more than just a
>> single filesystem, so generating a genimage.cfg for part of the image seems a
>> bit pointless.
> 
> I dont quite understand. Do you mean combining e.g. a standard and a rescue system?

 In practice, you rarely use the .ext2 or .ubifs image produced by Buildroot
directly; instead, you'll embed it into a larger image that is a lot more
complicated than e.g. the .ubi image offered by Buildroot (because you'd want a
writeable data volume in addition to the ubifs rootfs).

 So if we would use genimage for generating the .ext2 rootfs itself, then in
practice you'll often end up running genimage twice: once for the .ext2 rootfs,
and a second time to integrate it in a larger image.


 Coming back to the BR2_GENIMAGE_CFG_FILES option: actually it would make a
whole lot more sense if the genimage were executed under fakeroot, because then
we no longer need to generate the .ext2 subimage.

 Still, there is hardly any advantage over having this as an explicit option
rather than configuring

BR2_ROOTFS_POST_FAKEROOT_SCRIPT="support/scripts/genimage.sh"
BR2_ROOTFS_POST_SCRIPT_ARGS="-c path/to/my/genimage.cfg"

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Andreas Naumann]

Hi,

Am 03.04.2017 um 15:54 schrieb Arnout Vandecappelle:
>
>
> On 03-04-17 15:01, Andreas Naumann wrote:
>>>> What I was thinking about in the past would be a deeper integration of genimage
>>>> in buildroots filesystem configuration, actually generating the genimage config.
>>>> But so far it was not worth the effort for me.
>>>
>>>  We have considered that in the past. The problem is that genimage doesn't
>>> support some of the options that we do. Otherwise, however, it would be really
>>
>> Are you refering to filesystem-specific options or missing ones like axfs,
>> cloop, cramfs?
>
>  I'm refering to filesystem-specific options. E.g. AFAIK genimage doesn't allow
> configuration of ext2 #inodes or reserved blocks, or UBIFS max LEB count or
> compression.

There is the option to supply extraargs to the mkfs tools of most of the 
filesystems. However some sizes and LEBs may remain 
hardcoded/calculated. Dont know if that would suffice.
>
>
>>> nice to remove all our messy filesystem handling code and just generate a
>>> genimage.cfg file...
>
>  And my point here is: we could just extend genimage with all those options.
> Then we can use genimage instead of our rootfs handling. For the most part, the
> rootfs infra could be reduced to a single genimage.cfg file.
>
>  But as usual: all that would be nice in a way, but doesn't give us any direct
> advantage over what we have now, and what we have now works.
>

ditto

>
>>>
>>>  OTOH we also considered that a full image typically needs more than just a
>>> single filesystem, so generating a genimage.cfg for part of the image seems a
>>> bit pointless.
>>
>> I dont quite understand. Do you mean combining e.g. a standard and a rescue system?
>
>  In practice, you rarely use the .ext2 or .ubifs image produced by Buildroot
> directly; instead, you'll embed it into a larger image that is a lot more
> complicated than e.g. the .ubi image offered by Buildroot (because you'd want a
> writeable data volume in addition to the ubifs rootfs).
>
>  So if we would use genimage for generating the .ext2 rootfs itself, then in
> practice you'll often end up running genimage twice: once for the .ext2 rootfs,
> and a second time to integrate it in a larger image.

I guess a good integration could avoid that, but I assume it would be 
difficult to predict the many ways people might want to customize this.

In addition I also think the practical problems of somehow having to 
convert Kconfig options to a genimage config are not worth it (even 
though augeas, which with I played lately, seems like it might provide 
an elegant way to do this.)

>
>
>  Coming back to the BR2_GENIMAGE_CFG_FILES option: actually it would make a
> whole lot more sense if the genimage were executed under fakeroot, because then
> we no longer need to generate the .ext2 subimage.
>
>  Still, there is hardly any advantage over having this as an explicit option
> rather than configuring
>
> BR2_ROOTFS_POST_FAKEROOT_SCRIPT="support/scripts/genimage.sh"
> BR2_ROOTFS_POST_SCRIPT_ARGS="-c path/to/my/genimage.cfg"

In fact, this is almost what i do right now. Just a rootfs.tar is 
created and subsequently processed by genimage (I have a small patch 
that allows genimage to be fed with a tar in additin to the input dir).


regards,
Andreas

>
>  Regards,
>  Arnout
>
>

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Arnout Vandecappelle]

On 05-04-17 08:38, Andreas Naumann wrote:
> Hi,
> 
> Am 03.04.2017 um 15:54 schrieb Arnout Vandecappelle:
[snip]
>>  Coming back to the BR2_GENIMAGE_CFG_FILES option: actually it would make a
>> whole lot more sense if the genimage were executed under fakeroot, because then
>> we no longer need to generate the .ext2 subimage.
>>
>>  Still, there is hardly any advantage over having this as an explicit option
>> rather than configuring
>>
>> BR2_ROOTFS_POST_FAKEROOT_SCRIPT="support/scripts/genimage.sh"
>> BR2_ROOTFS_POST_SCRIPT_ARGS="-c path/to/my/genimage.cfg"
> 
> In fact, this is almost what i do right now. Just a rootfs.tar is created and
> subsequently processed by genimage (I have a small patch that allows genimage to
> be fed with a tar in additin to the input dir).

 I do think it's nicer to run it under fakeroot, rather than creating a tarball
that's not going to be used in the end. Especially because it ties in nicely
with any rootfs fixups that need to be done, e.g. splitting over several
filesystems.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Thomas Petazzoni]

Hello,

On Fri, 31 Mar 2017 00:51:08 +0200, Arnout Vandecappelle wrote:

>  I'm still not entirely sure if this new option is worthwhile. Without it, you
> would instead need
> 
> BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
> BR2_ROOTFS_POST_SCRIPT_ARGS="board/acmesystems/aria-g25/genimage.cfg"
> 
> Not really complicated either... The only real advantage I see is that it
> motivates people more to use genimage (which is otherwise a bit hidden). But for
> that, a paragraph or two in docs/manual/customize-post-image.txt would also
> work. By the way, even for this series an explanation of the option would be
> required in that file.
> 
>  What do the others think?

I agree that the new option doesn't seem really useful.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Hello,
 > On Fri, 31 Mar 2017 00:51:08 +0200, Arnout Vandecappelle wrote:

 >> I'm still not entirely sure if this new option is worthwhile. Without it, you
 >> would instead need
 >> 
 >> BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
 >> BR2_ROOTFS_POST_SCRIPT_ARGS="board/acmesystems/aria-g25/genimage.cfg"
 >> 
 >> Not really complicated either... The only real advantage I see is that it
 >> motivates people more to use genimage (which is otherwise a bit hidden). But for
 >> that, a paragraph or two in docs/manual/customize-post-image.txt would also
 >> work. By the way, even for this series an explanation of the option would be
 >> required in that file.
 >> 
 >> What do the others think?

 > I agree that the new option doesn't seem really useful.

I'm not sure. One of the problems with genimage is that it calls a
number of external utilities depending on the content of
genimage.cfg, and it isn't directly clear what host packages you need to
enable to ensure it works. If we add explicit BR2_GENIMAGE_FAT / CPIO /
EXT2 / ISO / JFFS2 / UBI / .. options selecting the needed host packages
then that would solve that issue.

I think the option should select BR2_HOST_PACKAGE_GENIMAGE instead of
depending on it though.

-- 
Bye, Peter Korsgaard

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Arnout Vandecappelle]

On 03-04-17 11:06, Peter Korsgaard wrote:
>>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> 
>  > Hello,
>  > On Fri, 31 Mar 2017 00:51:08 +0200, Arnout Vandecappelle wrote:
> 
>  >> I'm still not entirely sure if this new option is worthwhile. Without it, you
>  >> would instead need
>  >> 
>  >> BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
>  >> BR2_ROOTFS_POST_SCRIPT_ARGS="board/acmesystems/aria-g25/genimage.cfg"
>  >> 
>  >> Not really complicated either... The only real advantage I see is that it
>  >> motivates people more to use genimage (which is otherwise a bit hidden). But for
>  >> that, a paragraph or two in docs/manual/customize-post-image.txt would also
>  >> work. By the way, even for this series an explanation of the option would be
>  >> required in that file.
>  >> 
>  >> What do the others think?
> 
>  > I agree that the new option doesn't seem really useful.
> 
> I'm not sure. One of the problems with genimage is that it calls a
> number of external utilities depending on the content of
> genimage.cfg, and it isn't directly clear what host packages you need to
> enable to ensure it works. If we add explicit BR2_GENIMAGE_FAT / CPIO /
> EXT2 / ISO / JFFS2 / UBI / .. options selecting the needed host packages
> then that would solve that issue.

 So, how would you do that? To know which host packages to select, we need to
parse the genimage.cfg file. This is not possible from Kconfig....

 A long time ago (back in the days we still had ct-ng support), someone posted a
huge hack to Kconfig to allow it to call an external program while constructing
the Kconfig model. Something like that might work, but it's really a huge change
to Kconfig semantics.


 Regards,
 Arnout

> I think the option should select BR2_HOST_PACKAGE_GENIMAGE instead of
> depending on it though.
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

 >> >> What do the others think?
 >> 
 >> > I agree that the new option doesn't seem really useful.
 >> 
 >> I'm not sure. One of the problems with genimage is that it calls a
 >> number of external utilities depending on the content of
 >> genimage.cfg, and it isn't directly clear what host packages you need to
 >> enable to ensure it works. If we add explicit BR2_GENIMAGE_FAT / CPIO /
 >> EXT2 / ISO / JFFS2 / UBI / .. options selecting the needed host packages
 >> then that would solve that issue.

 >  So, how would you do that? To know which host packages to select, we need to
 > parse the genimage.cfg file. This is not possible from Kconfig....

I would simply leave it to the user to select the correct sub
options. The point is just that explicit sub options are clearer than
having to reverse engineer genimage to figure out what programs it
executes (and then figure out what host packages provide them).


 >  A long time ago (back in the days we still had ct-ng support), someone posted a
 > huge hack to Kconfig to allow it to call an external program while constructing
 > the Kconfig model. Something like that might work, but it's really a huge change
 > to Kconfig semantics.

Argh, lets not go there!

-- 
Bye, Peter Korsgaard

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Arnout Vandecappelle]

On 04-04-17 23:34, Peter Korsgaard wrote:
>>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:
> Hi,
> 
>  >> >> What do the others think?
>  >> 
>  >> > I agree that the new option doesn't seem really useful.
>  >> 
>  >> I'm not sure. One of the problems with genimage is that it calls a
>  >> number of external utilities depending on the content of
>  >> genimage.cfg, and it isn't directly clear what host packages you need to
>  >> enable to ensure it works. If we add explicit BR2_GENIMAGE_FAT / CPIO /
>  >> EXT2 / ISO / JFFS2 / UBI / .. options selecting the needed host packages
>  >> then that would solve that issue.
> 
>  >  So, how would you do that? To know which host packages to select, we need to
>  > parse the genimage.cfg file. This is not possible from Kconfig....
> 
> I would simply leave it to the user to select the correct sub
> options. The point is just that explicit sub options are clearer than
> having to reverse engineer genimage to figure out what programs it
> executes (and then figure out what host packages provide them).

 OK, makes sense.

 Combined with the other discussion in this thread, long-term I see us evolving
towards:

- deprecate some of the finer-grain fs tuning options we have now;
- always using genimage to generate the target filesystems;
- position our fs targets more as a 'quick fix' solution than for production;
- make host-e2fsprogs etc. blind options again.

 With that in mind, I'd tend to move the BR2_GENIMAGE_CFG_FILES option to top of
the filesystem menu rather than the Build menu. Although perhaps even better
would be to disable the filesystem menu entirely if BR2_GENIMAGE_CFG_FILES is
not empty (though that would only be if everything can be built with genimage).

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Thomas Petazzoni]

Hello,

On Wed, 5 Apr 2017 17:02:36 +0200, Arnout Vandecappelle wrote:

>  Combined with the other discussion in this thread, long-term I see us evolving
> towards:
> 
> - deprecate some of the finer-grain fs tuning options we have now;
> - always using genimage to generate the target filesystems;
> - position our fs targets more as a 'quick fix' solution than for production;
> - make host-e2fsprogs etc. blind options again.

To be honest, I am not sure I share this long-term view. I like the way
things are done today, very modular: we can generate just a filesystem
image, optionally use genimage afterwards, etc. So the "always using
genimage to generate the target filesystem" is not something that I see
as an improvement, for example.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Étienne Phélip]

Good morning everyone,

----- Le 5 Avr 17, ? 12:14, Thomas Petazzoni thomas.petazzoni at free-electrons.com a ?crit :

> Hello,
> 
> On Wed, 5 Apr 2017 17:02:36 +0200, Arnout Vandecappelle wrote:
> 
>>  Combined with the other discussion in this thread, long-term I see us evolving
>> towards:
>> 
>> - deprecate some of the finer-grain fs tuning options we have now;
>> - always using genimage to generate the target filesystems;
>> - position our fs targets more as a 'quick fix' solution than for production;
>> - make host-e2fsprogs etc. blind options again.
> 
> To be honest, I am not sure I share this long-term view. I like the way
> things are done today, very modular: we can generate just a filesystem
> image, optionally use genimage afterwards, etc. So the "always using
> genimage to generate the target filesystem" is not something that I see
> as an improvement, for example.

To sum things up:
 - Should I add BR2_GENIMAGE_CFG_FILES or reuse the script in the
postimage script
 - if BR2_GENIMAGE_CFG_FILES is choosen, should it 'depends on' or
'select' BR2_PACKAGE_HOST_GENIMAGE?
 - should I convert all boards or just the one already in this RFC?

Best regards
-Etienne

[Buildroot] [RFC 4/4] board/acmesystems/aria-g25: set BR2_GENIMAGE_CFG_FILES

[Arnout Vandecappelle]

On 10-04-17 15:44, ?tienne Ph?lip wrote:
> Good morning everyone,
> 
> ----- Le 5 Avr 17, ? 12:14, Thomas Petazzoni thomas.petazzoni at free-electrons.com a ?crit :
> 
>> Hello,
>>
>> On Wed, 5 Apr 2017 17:02:36 +0200, Arnout Vandecappelle wrote:
>>
>>>  Combined with the other discussion in this thread, long-term I see us evolving
>>> towards:
>>>
>>> - deprecate some of the finer-grain fs tuning options we have now;
>>> - always using genimage to generate the target filesystems;
>>> - position our fs targets more as a 'quick fix' solution than for production;
>>> - make host-e2fsprogs etc. blind options again.
>>
>> To be honest, I am not sure I share this long-term view. I like the way
>> things are done today, very modular: we can generate just a filesystem
>> image, optionally use genimage afterwards, etc. So the "always using
>> genimage to generate the target filesystem" is not something that I see
>> as an improvement, for example.
> 
> To sum things up:

 Peter has the final word in things, and nobody objected anymore to introducing
BR2_GENIMAGE_CFG_FILES, so:

>  - Should I add BR2_GENIMAGE_CFG_FILES or reuse the script in the
> postimage script

 Introduce BR2_GENIMAGE_CFG_FILES.

>  - if BR2_GENIMAGE_CFG_FILES is choosen, should it 'depends on' or
> 'select' BR2_PACKAGE_HOST_GENIMAGE?

 select BR2_PACKAGE_HOST_GENIMAGE

 Note that since BR2_GENIMAGE_CFG_FILES is a string option, you can't do that
directly. You need to introduce a blind helper option:

config BR2_GENIMAGE_CFG_FILES_SELECT
	bool
	default y if BR2_GENIMAGE_CFG_FILES != ""
	select BR2_PACKAGE_HOST_GENIMAGE

>  - should I convert all boards or just the one already in this RFC?

 I think you can start converting more boards. But you can already stop before
all boards are converted.

 For each board you should verify that the build result with and without your
change is identical! To save build time, you may want to use an external
toolchain though...

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF