openembedded-core.lists.openembedded.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size
@ 2023-06-04 12:37 Charles-Antoine Couret
  2023-06-04 12:37 ` [PATCH 1/3] image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types Charles-Antoine Couret
                   ` (4 more replies)
  0 siblings, 5 replies; 7+ messages in thread
From: Charles-Antoine Couret @ 2023-06-04 12:37 UTC (permalink / raw)
  To: openembedded-core; +Cc: Charles-Antoine Couret

In case of fixed partitionning where the rootfs partition can't exceed an
amount of bytes, there is currently no automatic and no generic way to have
this requirement met in any case.

Until now, ROOTFS_SIZE value got from directory_size() does not takes into account
the size of required metadata for the filesystem itself (and does not work well
for other block size than 4k BTW).
Obviously it's a difficult task which depends on rootfs size and filesystem type.

The workaround was to set IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
to add the required extra margins. But when the final rootfs is closed to the
maximum size, it's difficult to adjust them correctly  And if you remove
or add new recipes in your image, you've to recompute these margins to have enough
space for these metadata when the rootfs is small, and to not have too big final
file when the rootfs is big.

It's cumbersome and error prone to just have a build failure when the final output
can't be flashed into the partition.

The solution is to follow how it's implemented in buildroot by having a
specific variable, here IMAGE_FILE_MAXSIZE, to create the final sparse file
and trying to fill it with the content of rootfs. If there is enough space,
margins are well compressed and does not consume space in the filesystem.
If there is no enough space, an error is triggered to warm the developer before
trying to use it in the device.

If IMAGE_FILE_MAXSIZE is not set, the idea is to keep the previous behaviour
for compatibility reason and to met other requirements.

Charles-Antoine Couret (3):
  image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types
  image_types: use IMAGE_FILE_MAXSIZE variable for btrfs image types
  image_types: use IMAGE_FILE_MAXSIZE variable for f2fs image types

 meta/classes-recipe/image_types.bbclass | 34 +++++++++++++++++++------
 1 file changed, 26 insertions(+), 8 deletions(-)

-- 
2.40.1



^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 1/3] image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types
  2023-06-04 12:37 [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size Charles-Antoine Couret
@ 2023-06-04 12:37 ` Charles-Antoine Couret
  2023-06-07 12:30   ` [OE-core] " Alexandre Belloni
  2023-06-04 12:37 ` [PATCH 2/3] image_types: use IMAGE_FILE_MAXSIZE variable for btrfs " Charles-Antoine Couret
                   ` (3 subsequent siblings)
  4 siblings, 1 reply; 7+ messages in thread
From: Charles-Antoine Couret @ 2023-06-04 12:37 UTC (permalink / raw)
  To: openembedded-core; +Cc: Charles-Antoine Couret

If defined, this variable value overrides the size of ext* partition file created by mkfs.
Otherwise previous logic based on ROOTFS_SIZE variable is used.

It should be set when the final file size would not be above a specific value due to fixed
partitionning for example.

Signed-off-by: Charles-Antoine Couret <charles-antoine.couret@mind.be>
---
 meta/classes-recipe/image_types.bbclass | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
index e4939af459..cebbb61545 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -68,24 +68,33 @@ IMAGE_CMD:cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${I
 
 oe_mkext234fs () {
 	fstype=$1
+	image_file_maxsize=$2
 	extra_imagecmd=""
+	rootfs_file_size=$ROOTFS_SIZE
 
-	if [ $# -gt 1 ]; then
-		shift
+	if [ $# -gt 2 ]; then
+		shift 2
 		extra_imagecmd=$@
 	fi
 
+
+	if [[ "${image_file_maxsize}" != "\"\"" ]]; then
+		# Remove quotes to get numbers only
+		rootfs_file_size=$(echo "${image_file_maxsize}" | tr -d '"')
+	fi
+
 	# If generating an empty image the size of the sparse block should be large
 	# enough to allocate an ext4 filesystem using 4096 bytes per inode, this is
 	# about 60K, so dd needs a minimum count of 60, with bs=1024 (bytes per IO)
 	eval local COUNT=\"0\"
 	eval local MIN_COUNT=\"60\"
-	if [ $ROOTFS_SIZE -lt $MIN_COUNT ]; then
+	if [ $rootfs_file_size -lt $MIN_COUNT ]; then
 		eval COUNT=\"$MIN_COUNT\"
 	fi
+
 	# Create a sparse image block
-	bbdebug 1 Executing "dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype seek=$ROOTFS_SIZE count=$COUNT bs=1024"
-	dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype seek=$ROOTFS_SIZE count=$COUNT bs=1024
+	bbdebug 1 Executing "dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype seek=$rootfs_file_size count=$COUNT bs=1024"
+	dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype seek=$rootfs_file_size count=$COUNT bs=1024
 	bbdebug 1 "Actual Rootfs size:  `du -s ${IMAGE_ROOTFS}`"
 	bbdebug 1 "Actual Partition size: `stat -c '%s' ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype`"
 	bbdebug 1 Executing "mkfs.$fstype -F $extra_imagecmd ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype -d ${IMAGE_ROOTFS}"
@@ -94,9 +103,9 @@ oe_mkext234fs () {
 	fsck.$fstype -pvfD ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype || [ $? -le 3 ]
 }
 
-IMAGE_CMD:ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
-IMAGE_CMD:ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
-IMAGE_CMD:ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
+IMAGE_CMD:ext2 = "oe_mkext234fs ext2 \"${IMAGE_FILE_MAXSIZE:ext2}\" ${EXTRA_IMAGECMD}"
+IMAGE_CMD:ext3 = "oe_mkext234fs ext3 \"${IMAGE_FILE_MAXSIZE:ext3}\" ${EXTRA_IMAGECMD}"
+IMAGE_CMD:ext4 = "oe_mkext234fs ext4 \"${IMAGE_FILE_MAXSIZE:ext4}\" ${EXTRA_IMAGECMD}"
 
 MIN_BTRFS_SIZE ?= "16384"
 IMAGE_CMD:btrfs () {
-- 
2.40.1



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 2/3] image_types: use IMAGE_FILE_MAXSIZE variable for btrfs image types
  2023-06-04 12:37 [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size Charles-Antoine Couret
  2023-06-04 12:37 ` [PATCH 1/3] image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types Charles-Antoine Couret
@ 2023-06-04 12:37 ` Charles-Antoine Couret
  2023-06-04 12:37 ` [PATCH 3/3] image_types: use IMAGE_FILE_MAXSIZE variable for f2fs " Charles-Antoine Couret
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 7+ messages in thread
From: Charles-Antoine Couret @ 2023-06-04 12:37 UTC (permalink / raw)
  To: openembedded-core; +Cc: Charles-Antoine Couret

If defined, this variable value overrides the size of btrfs partition file created by mkfs.
Otherwise previous logic based on ROOTFS_SIZE variable is used.

It should be set when the final file size would not be above a specific value due to fixed
partitionning for example.

Signed-off-by: Charles-Antoine Couret <charles-antoine.couret@mind.be>
---
 meta/classes-recipe/image_types.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
index cebbb61545..f157a84b2e 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -110,6 +110,10 @@ IMAGE_CMD:ext4 = "oe_mkext234fs ext4 \"${IMAGE_FILE_MAXSIZE:ext4}\" ${EXTRA_IMAG
 MIN_BTRFS_SIZE ?= "16384"
 IMAGE_CMD:btrfs () {
 	size=${ROOTFS_SIZE}
+	if [ -n "${IMAGE_FILE_MAXSIZE:btrfs}" ]; then
+		size=${IMAGE_FILE_MAXSIZE:btrfs}
+	fi
+
 	if [ ${size} -lt ${MIN_BTRFS_SIZE} ] ; then
 		size=${MIN_BTRFS_SIZE}
 		bbwarn "Rootfs size is too small for BTRFS. Filesystem will be extended to ${size}K"
-- 
2.40.1



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 3/3] image_types: use IMAGE_FILE_MAXSIZE variable for f2fs image types
  2023-06-04 12:37 [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size Charles-Antoine Couret
  2023-06-04 12:37 ` [PATCH 1/3] image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types Charles-Antoine Couret
  2023-06-04 12:37 ` [PATCH 2/3] image_types: use IMAGE_FILE_MAXSIZE variable for btrfs " Charles-Antoine Couret
@ 2023-06-04 12:37 ` Charles-Antoine Couret
  2023-06-07  9:52 ` [OE-core] [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size Alexandre Belloni
  2023-07-06 11:57 ` Ross Burton
  4 siblings, 0 replies; 7+ messages in thread
From: Charles-Antoine Couret @ 2023-06-04 12:37 UTC (permalink / raw)
  To: openembedded-core; +Cc: Charles-Antoine Couret

If defined, this variable value overrides the size of f2fs partition file created by mkfs.
Otherwise previous logic based on ROOTFS_SIZE variable is used.

It should be set when the final file size would not be above a specific value due to fixed
partitionning for example.

Signed-off-by: Charles-Antoine Couret <charles-antoine.couret@mind.be>
---
 meta/classes-recipe/image_types.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
index f157a84b2e..b5d32b7622 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -250,6 +250,11 @@ IMAGE_CMD:f2fs () {
         # 500M the standard IMAGE_OVERHEAD_FACTOR does not work, so add additional
         # space here when under 500M
 	size=${ROOTFS_SIZE}
+
+	if [ -n "${IMAGE_FILE_MAXSIZE:f2fs}" ]; then
+		size=${IMAGE_FILE_MAXSIZE:f2fs}
+	fi
+
 	if [ ${size} -lt ${MIN_F2FS_SIZE} ] ; then
 		size=${MIN_F2FS_SIZE}
 		bbwarn "Rootfs size is too small for F2FS. Filesystem will be extended to ${size}K"
-- 
2.40.1



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size
  2023-06-04 12:37 [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size Charles-Antoine Couret
                   ` (2 preceding siblings ...)
  2023-06-04 12:37 ` [PATCH 3/3] image_types: use IMAGE_FILE_MAXSIZE variable for f2fs " Charles-Antoine Couret
@ 2023-06-07  9:52 ` Alexandre Belloni
  2023-07-06 11:57 ` Ross Burton
  4 siblings, 0 replies; 7+ messages in thread
From: Alexandre Belloni @ 2023-06-07  9:52 UTC (permalink / raw)
  To: charles-antoine.couret; +Cc: openembedded-core

Hello,

On 04/06/2023 14:37:52+0200, Charles-Antoine Couret via lists.openembedded.org wrote:
> In case of fixed partitionning where the rootfs partition can't exceed an
> amount of bytes, there is currently no automatic and no generic way to have
> this requirement met in any case.
> 
> Until now, ROOTFS_SIZE value got from directory_size() does not takes into account
> the size of required metadata for the filesystem itself (and does not work well
> for other block size than 4k BTW).
> Obviously it's a difficult task which depends on rootfs size and filesystem type.
> 
> The workaround was to set IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
> to add the required extra margins. But when the final rootfs is closed to the
> maximum size, it's difficult to adjust them correctly  And if you remove
> or add new recipes in your image, you've to recompute these margins to have enough
> space for these metadata when the rootfs is small, and to not have too big final
> file when the rootfs is big.
> 
> It's cumbersome and error prone to just have a build failure when the final output
> can't be flashed into the partition.
> 
> The solution is to follow how it's implemented in buildroot by having a
> specific variable, here IMAGE_FILE_MAXSIZE, to create the final sparse file
> and trying to fill it with the content of rootfs. If there is enough space,
> margins are well compressed and does not consume space in the filesystem.
> If there is no enough space, an error is triggered to warm the developer before
> trying to use it in the device.
> 
> If IMAGE_FILE_MAXSIZE is not set, the idea is to keep the previous behaviour
> for compatibility reason and to met other requirements.

It would be great if you could add test cases that ensure that the
generated images are indeed fitting the size.

> 
> Charles-Antoine Couret (3):
>   image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types
>   image_types: use IMAGE_FILE_MAXSIZE variable for btrfs image types
>   image_types: use IMAGE_FILE_MAXSIZE variable for f2fs image types
> 
>  meta/classes-recipe/image_types.bbclass | 34 +++++++++++++++++++------
>  1 file changed, 26 insertions(+), 8 deletions(-)
> 
> -- 
> 2.40.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#182359): https://lists.openembedded.org/g/openembedded-core/message/182359
> Mute This Topic: https://lists.openembedded.org/mt/99320002/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH 1/3] image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types
  2023-06-04 12:37 ` [PATCH 1/3] image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types Charles-Antoine Couret
@ 2023-06-07 12:30   ` Alexandre Belloni
  0 siblings, 0 replies; 7+ messages in thread
From: Alexandre Belloni @ 2023-06-07 12:30 UTC (permalink / raw)
  To: charles-antoine.couret; +Cc: openembedded-core

This fails on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/69/builds/7219/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/120/builds/2859/steps/14/logs/stdio

ERROR: core-image-minimal-1.0-r0 do_image_ext4: ExecutionError('/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/temp/run.do_image_ext4.266360', 2, None, None)
NOTE: recipe core-image-minimal-1.0-r0: task do_image_tar: Succeeded
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/temp/log.do_image_ext4.266360
Log data follows:
| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-core/glibc/cross-localedef-native_2.37.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-core/glibc/ldconfig-native_2.12.1.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/qemu/qemuwrapper-cross_1.0.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-kernel/kmod/depmodwrapper-cross_1.0.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/createrepo-c/createrepo-c_0.21.1.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/dnf/dnf_4.14.0.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.47.0.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/makedevs/makedevs_1.0.1.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/opkg-utils/opkg-utils_0.5.0.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/opkg/opkg_0.6.1.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/rpm/rpm_4.18.1.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-extended/pbzip2/pbzip2_1.1.13.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-extended/pigz/pigz_2.7.bb:do_populate_sysroot']
| NOTE: Installed into sysroot: ['e2fsprogs-native']
| NOTE: Skipping as already exists in sysroot: ['cross-localedef-native', 'ldconfig-native', 'qemuwrapper-cross', 'depmodwrapper-cross', 'update-rc.d-native', 'createrepo-c-native', 'dnf-native', 'makedevs-native', 'opkg-utils-native', 'opkg-native', 'pseudo-native', 'rpm-native', 'pbzip2-native', 'pigz-native', 'librepo-native', 'libcomps-native', 'gettext-minimal-native', 'python3-native', 'python3-iniparse-native', 'libdnf-native', 'cmake-native', 'kmod-native', 'openssl-native', 'debianutils-native', 'libtool-native', 'attr-native', 'util-linux-native', 'texinfo-dummy-native', 'bzip2-native', 'sqlite3-native', 'file-native', 'xz-native', 'zstd-native', 'popt-native', 'lua-native', 'elfutils-native', 'libgcrypt-native', 'qemu-native', 'zlib-native', 'libxml2-native', 'expat-native', 'curl-native', 'glib-2.0-native', 'perl-native', 'shadow-native', 'libarchive-native', 'libsolv-native', 'gpgme-native', 'libffi-native', 'libtirpc-native', 'libedit-native', 'ncurses-native', 'util-linux-libuuid-native', 'libnsl2-native', 'gdbm-native', 'python3-wheel-native', 'python3-six-native', 'python3-build-native', 'python3-installer-native', 'python3-setuptools-native', 'libcheck-native', 'gobject-introspection-native', 'json-c-native', 'swig-native', 'libmodulemd-native', 'libpcre2-native', 'libcap-ng-native', 'readline-native', 'libmicrohttpd-native', 'libgpg-error-native', 'libcap-native', 'gettext-native', 'make-native', 'libassuan-native', 'python3-flit-core-native', 'python3-pyproject-hooks-native', 'python3-packaging-native', 'flex-native', 'libyaml-native', 'gnutls-native', 'unzip-native', 'm4-native', 'libtasn1-native', 'gmp-native', 'libidn2-native', 'libunistring-native', 'nettle-native', 'gnu-config-native']
| DEBUG: sed -e 's:^[^/]*/:/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/:g' /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/sysroots-components/x86_64/e2fsprogs-native/fixmepath | xargs sed -i -e 's:FIXMESTAGINGDIRTARGET:/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot:g; s:FIXMESTAGINGDIRHOST:/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native:g' -e 's:FIXME_PSEUDO_SYSROOT:/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/sysroots-components/x86_64/pseudo-native:g' -e 's:FIXME_HOSTTOOLS_DIR:/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/hosttools:g' -e 's:FIXME_PKGDATA_DIR:/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/pkgdata/qemux86-64:g' -e 's:FIXME_PSEUDO_LOCALSTATEDIR:/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/pseudo/:g' -e 's:FIXME_LOGFIFO:/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/temp/fifo.266360:g'
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing python function set_image_size
| DEBUG: 33737.600000 = 25952 * 1.300000
| DEBUG: 33737.600000 = max(33737.600000, 8192)[33737.600000] + 0
| DEBUG: 33738.000000 = int(33737.600000)
| DEBUG: 33738 = aligned(33738)
| DEBUG: returning 33738
| DEBUG: Python function set_image_size finished
| DEBUG: Executing shell function do_image_ext4
| /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/temp/run.do_image_ext4.266360: 151: Bad substitution
| WARNING: exit code 2 from a shell command.
NOTE: recipe core-image-minimal-1.0-r0: task do_image_ext4: Failed

On 04/06/2023 14:37:53+0200, Charles-Antoine Couret via lists.openembedded.org wrote:
> If defined, this variable value overrides the size of ext* partition file created by mkfs.
> Otherwise previous logic based on ROOTFS_SIZE variable is used.
> 
> It should be set when the final file size would not be above a specific value due to fixed
> partitionning for example.
> 
> Signed-off-by: Charles-Antoine Couret <charles-antoine.couret@mind.be>
> ---
>  meta/classes-recipe/image_types.bbclass | 25 +++++++++++++++++--------
>  1 file changed, 17 insertions(+), 8 deletions(-)
> 
> diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
> index e4939af459..cebbb61545 100644
> --- a/meta/classes-recipe/image_types.bbclass
> +++ b/meta/classes-recipe/image_types.bbclass
> @@ -68,24 +68,33 @@ IMAGE_CMD:cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${I
>  
>  oe_mkext234fs () {
>  	fstype=$1
> +	image_file_maxsize=$2
>  	extra_imagecmd=""
> +	rootfs_file_size=$ROOTFS_SIZE
>  
> -	if [ $# -gt 1 ]; then
> -		shift
> +	if [ $# -gt 2 ]; then
> +		shift 2
>  		extra_imagecmd=$@
>  	fi
>  
> +
> +	if [[ "${image_file_maxsize}" != "\"\"" ]]; then
> +		# Remove quotes to get numbers only
> +		rootfs_file_size=$(echo "${image_file_maxsize}" | tr -d '"')
> +	fi
> +
>  	# If generating an empty image the size of the sparse block should be large
>  	# enough to allocate an ext4 filesystem using 4096 bytes per inode, this is
>  	# about 60K, so dd needs a minimum count of 60, with bs=1024 (bytes per IO)
>  	eval local COUNT=\"0\"
>  	eval local MIN_COUNT=\"60\"
> -	if [ $ROOTFS_SIZE -lt $MIN_COUNT ]; then
> +	if [ $rootfs_file_size -lt $MIN_COUNT ]; then
>  		eval COUNT=\"$MIN_COUNT\"
>  	fi
> +
>  	# Create a sparse image block
> -	bbdebug 1 Executing "dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype seek=$ROOTFS_SIZE count=$COUNT bs=1024"
> -	dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype seek=$ROOTFS_SIZE count=$COUNT bs=1024
> +	bbdebug 1 Executing "dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype seek=$rootfs_file_size count=$COUNT bs=1024"
> +	dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype seek=$rootfs_file_size count=$COUNT bs=1024
>  	bbdebug 1 "Actual Rootfs size:  `du -s ${IMAGE_ROOTFS}`"
>  	bbdebug 1 "Actual Partition size: `stat -c '%s' ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype`"
>  	bbdebug 1 Executing "mkfs.$fstype -F $extra_imagecmd ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype -d ${IMAGE_ROOTFS}"
> @@ -94,9 +103,9 @@ oe_mkext234fs () {
>  	fsck.$fstype -pvfD ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype || [ $? -le 3 ]
>  }
>  
> -IMAGE_CMD:ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
> -IMAGE_CMD:ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
> -IMAGE_CMD:ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
> +IMAGE_CMD:ext2 = "oe_mkext234fs ext2 \"${IMAGE_FILE_MAXSIZE:ext2}\" ${EXTRA_IMAGECMD}"
> +IMAGE_CMD:ext3 = "oe_mkext234fs ext3 \"${IMAGE_FILE_MAXSIZE:ext3}\" ${EXTRA_IMAGECMD}"
> +IMAGE_CMD:ext4 = "oe_mkext234fs ext4 \"${IMAGE_FILE_MAXSIZE:ext4}\" ${EXTRA_IMAGECMD}"
>  
>  MIN_BTRFS_SIZE ?= "16384"
>  IMAGE_CMD:btrfs () {
> -- 
> 2.40.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#182360): https://lists.openembedded.org/g/openembedded-core/message/182360
> Mute This Topic: https://lists.openembedded.org/mt/99320004/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size
  2023-06-04 12:37 [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size Charles-Antoine Couret
                   ` (3 preceding siblings ...)
  2023-06-07  9:52 ` [OE-core] [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size Alexandre Belloni
@ 2023-07-06 11:57 ` Ross Burton
  4 siblings, 0 replies; 7+ messages in thread
From: Ross Burton @ 2023-07-06 11:57 UTC (permalink / raw)
  To: charles-antoine.couret; +Cc: openembedded-core

On 4 Jun 2023, at 13:37, Charles-Antoine Couret via lists.openembedded.org <charles-antoine.couret=mind.be@lists.openembedded.org> wrote:
> The solution is to follow how it's implemented in buildroot by having a
> specific variable, here IMAGE_FILE_MAXSIZE, to create the final sparse file
> and trying to fill it with the content of rootfs. If there is enough space,
> margins are well compressed and does not consume space in the filesystem.
> If there is no enough space, an error is triggered to warm the developer before
> trying to use it in the device.

This seems like a useful feature but the autobuilder failed and now the patches don’t apply to master.  Will you be fixing the issues the AB exposed and rebasing?

Ross

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2023-07-06 11:57 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-06-04 12:37 [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size Charles-Antoine Couret
2023-06-04 12:37 ` [PATCH 1/3] image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types Charles-Antoine Couret
2023-06-07 12:30   ` [OE-core] " Alexandre Belloni
2023-06-04 12:37 ` [PATCH 2/3] image_types: use IMAGE_FILE_MAXSIZE variable for btrfs " Charles-Antoine Couret
2023-06-04 12:37 ` [PATCH 3/3] image_types: use IMAGE_FILE_MAXSIZE variable for f2fs " Charles-Antoine Couret
2023-06-07  9:52 ` [OE-core] [PATCH 0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size Alexandre Belloni
2023-07-06 11:57 ` Ross Burton

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).