Re: [Buildroot] [PATCH 1/1] fs/squashfs: add more options to customize creation

["Yann E. MORIN" <yann.morin.1998@free.fr>]

Linus, All,

On 2022-05-15 14:54 +0000, Linus Kaschulla via buildroot spake thusly:
> Squashfs seems to be the best way to shipping a highly compressed
> rootfs that can easily be mounted without any extraction overhead.
> While trying to get the squashfs as small as possible, I found out
> that buildroot only allows rather basic customizations and many
> options to improve the compression ratio are not available.
> 
> Notably, squashfs doesn't use the maximum available blocksize by
> default and compression options offered by `mksquashfs` are not
> changable from buildroot without patching the source code.
> 
> This patch adds 2 new options to fs/squashfs:
> 
>  - Changing the blocksize (e.g. to improve compression ratio)
>  - Appending any custom arguments (e.g. for adding compression options)

Two changes, two patches. ;-)

> Using these options, specifying the max blocksize of 1M and adding
> the arguments `-Xbcj arm` allows for a notably smaller rootfs when
> using xz compression and building for an arm target.

So, I think this is good to have those additional options, however I
think we should restrict them. For example, arbitrary string options is
not very nice.

Instead, here's what I suggest:

  - for the blocksize: add a choice, for the only possible and
    meaningful values (1L is probably too low, maybe 4K should be the
    minimum?)

        choice
            bool "blocksize"
            default BR2_TARGET_ROOTFS_BS_128K

        config BR2_TARGET_ROOTFS_BS_1K
            bool "1k"

        config BR2_TARGET_ROOTFS_BS_2K
            bool "1k"

        [...]

        config BR2_TARGET_ROOTFS_BS_1M
            bool "1k"

        endchoice

        config BR2_TARGET_ROOTFS_BS
            string
            default "1K"   if BR2_TARGET_ROOTFS_BS_1K
            default "2K"   if BR2_TARGET_ROOTFS_BS_2K
            ...
            default "1M"   if BR2_TARGET_ROOTFS_BS_1M

    and in squashfs.mk:

        ROOTFS_SQUASHFS_ARGS += -b $(call qstrip,$(BR2_TARGET_ROOTFS_SQUASHFS4_BLOCKSIZE))

  - for the comnpression options, I think a generic boolean would be
    better:

        config BR2_TARGET_ROOTFS_EXTREME_COMP
            bool "extreme compression if possible"
            help
              Use options to increase compression ration as much as
              possible, like using architecture-specific options, at
              the cost of time when assmebline the filesystem image.

              For example:
                - with gzip, use -Xcompression-level 9
                - with lzo, use -Xcompression-level 9
                - with xz on ARM, use -Xbcj arm.thumb

        config BR2_TARGET_ROOTFS_COMP_OPTS
            string
            default "-Xcompression-level 9" if BR2_TARGET_ROOTFS_SQUASHFS4_GZIP
            default "-Xcompression-level 9" if BR2_TARGET_ROOTFS_SQUASHFS4_LZO
            default "-Xbcj arm,armthumb" if BR2_TARGET_ROOTFS_SQUASHFS4_XZ && (BR2_arm || BR2_armeb)
            default "-Xbcj x86" if BR2_TARGET_ROOTFS_SQUASHFS4_XZ && (BR2_i386 || BR2_x86_64)
            ...

    and in squashfs.mk:

        ROOTFS_SQUASHFS_ARGS += $(call qstrip,$(BR2_TARGET_ROOTFS_COMP_OPTS))

    Also note that lz4 already uses -Xhc unconditionally, and commit
    07e37bcc42f notes that an extra option was dropped, so maybe we
    should just unconditionally enable all those options when possible?

And of course, this should be in two patches: one to introudce the
blocksize, one to introduce the extreme compression optiopns

Regards,
Yann E. MORIN.

> Signed-off-by: Linus Kaschulla <linus@cosmos-ink.net>
> ---
>  fs/squashfs/Config.in   | 20 ++++++++++++++++++++
>  fs/squashfs/squashfs.mk |  8 ++++++++
>  2 files changed, 28 insertions(+)
> 
> diff --git a/fs/squashfs/Config.in b/fs/squashfs/Config.in
> index 70d4a20cf0..7ae2743961 100644
> --- a/fs/squashfs/Config.in
> +++ b/fs/squashfs/Config.in
> @@ -12,6 +12,16 @@ config BR2_TARGET_ROOTFS_SQUASHFS_PAD
>  	  Say 'y' here (the default) to pad the the filesystem image
>  	  to a 4K boundary. Say 'n' to disable padding.
>  
> +config BR2_TARGET_ROOTFS_SQUASHFS4_BLOCKSIZE
> +	string "block size"
> +	help
> +	  Define a specific block size to use when creating the
> +	  squashfs. Higher values can improve compression ratios.
> +	  The suffixes K and M are supported. The value should not
> +	  be above 1 MiB.
> +
> +	  If unsure, leave this empty to use the default blocksize.
> +
>  choice
>  	prompt "Compression algorithm"
>  	default BR2_TARGET_ROOTFS_SQUASHFS4_GZIP
> @@ -39,4 +49,14 @@ config BR2_TARGET_ROOTFS_SQUASHFS4_ZSTD
>  
>  endchoice
>  
> +config BR2_TARGET_ROOTFS_SQUASHFS4_CUSTOM_ARGS
> +	string "additional arguments to pass to the squashfs creation"
> +	help
> +	  Append extra commandline arguments to the mksquashfs command.
> +	  This can be useful for finetuning a selected compression or
> +	  enable/disable other features of squashfs.
> +	  See "$HOST_DIR/bin/mksquashfs -h" for available arguments.
> +
> +	  If unsure, leave empty.
> +
>  endif
> diff --git a/fs/squashfs/squashfs.mk b/fs/squashfs/squashfs.mk
> index 7a5e3e313e..42343b1b40 100644
> --- a/fs/squashfs/squashfs.mk
> +++ b/fs/squashfs/squashfs.mk
> @@ -12,6 +12,10 @@ ifeq ($(BR2_TARGET_ROOTFS_SQUASHFS_PAD),)
>  ROOTFS_SQUASHFS_ARGS += -nopad
>  endif
>  
> +ifneq ($(BR2_TARGET_ROOTFS_SQUASHFS4_BLOCKSIZE),"")
> +ROOTFS_SQUASHFS_ARGS += -b "$(BR2_TARGET_ROOTFS_SQUASHFS4_BLOCKSIZE)"
> +endif
> +
>  ifeq ($(BR2_TARGET_ROOTFS_SQUASHFS4_LZ4),y)
>  ROOTFS_SQUASHFS_ARGS += -comp lz4 -Xhc
>  else ifeq ($(BR2_TARGET_ROOTFS_SQUASHFS4_LZO),y)
> @@ -26,6 +30,10 @@ else
>  ROOTFS_SQUASHFS_ARGS += -comp gzip
>  endif
>  
> +ifneq ($(BR2_TARGET_ROOTFS_SQUASHFS4_CUSTOM_ARGS),"")
> +ROOTFS_SQUASHFS_ARGS += $(call qstrip,$(BR2_TARGET_ROOTFS_SQUASHFS4_CUSTOM_ARGS))
> +endif
> +
>  define ROOTFS_SQUASHFS_CMD
>  	$(HOST_DIR)/bin/mksquashfs $(TARGET_DIR) $@ $(ROOTFS_SQUASHFS_ARGS)
>  endef
> -- 
> 2.35.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot