* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-21 22:55 ` [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods Gustavo Zacarias
@ 2013-01-21 23:38 ` Stefan Fröberg
2013-01-21 23:44 ` Gustavo Zacarias
2013-01-22 17:05 ` Thomas Petazzoni
2013-01-24 7:09 ` Arnout Vandecappelle
2 siblings, 1 reply; 15+ messages in thread
From: Stefan Fröberg @ 2013-01-21 23:38 UTC (permalink / raw)
To: buildroot
Hi Gustavo
22.1.2013 0:55, Gustavo Zacarias kirjoitti:
> Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
> ---
> fs/common.mk | 8 +++++++-
> fs/cpio/Config.in | 18 ++++++++++++++----
> 2 files changed, 21 insertions(+), 5 deletions(-)
>
> diff --git a/fs/common.mk b/fs/common.mk
> index 8b5b2f2..0c171fc 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -39,7 +39,7 @@ ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE)) \
> define ROOTFS_TARGET_INTERNAL
>
> # extra deps
> -ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma)
> +ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
>
> $$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
> @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
> @@ -70,6 +70,12 @@ endif
> ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZMA),y)
> $$(LZMA) -9 -c $$@ > $$@.lzma
> endif
> +ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZO),y)
> + $$(LZOP) -9 -c $$@ > $$@.lzo
> +endif
> +ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
> + $$(XZ) -9 -c $$@ > $$@.xz
> +endif
You will probably need --check=crc32 with that xz-compression.
I have had difficulties booting linux 3.3.1 kernel (don't know about
later versions)
with xz-compressed initramfs without that option.
Regards
Stefan
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-21 23:38 ` Stefan Fröberg
@ 2013-01-21 23:44 ` Gustavo Zacarias
2013-01-22 0:34 ` Stefan Fröberg
0 siblings, 1 reply; 15+ messages in thread
From: Gustavo Zacarias @ 2013-01-21 23:44 UTC (permalink / raw)
To: buildroot
On 01/21/2013 08:38 PM, Stefan Fr?berg wrote:
> You will probably need --check=crc32 with that xz-compression.
> I have had difficulties booting linux 3.3.1 kernel (don't know about
> later versions)
> with xz-compressed initramfs without that option.
You are 100% correct.
It didn't fail me during testing because buildroot is never using any
compressed cpio archive, thus the previous options (gzip, bzip2) were
never used.
I noticed when checking the bzImage size in a tweaked qemu_x86_defconfig
test run that it never changed much.
New patchset will follow that fixes this and the previous issue.
Regards.
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-21 23:44 ` Gustavo Zacarias
@ 2013-01-22 0:34 ` Stefan Fröberg
0 siblings, 0 replies; 15+ messages in thread
From: Stefan Fröberg @ 2013-01-22 0:34 UTC (permalink / raw)
To: buildroot
22.1.2013 1:44, Gustavo Zacarias kirjoitti:
> On 01/21/2013 08:38 PM, Stefan Fr?berg wrote:
>
>> You will probably need --check=crc32 with that xz-compression.
>> I have had difficulties booting linux 3.3.1 kernel (don't know about
>> later versions)
>> with xz-compressed initramfs without that option.
> You are 100% correct.
> It didn't fail me during testing because buildroot is never using any
> compressed cpio archive, thus the previous options (gzip, bzip2) were
> never used.
Yeah.
It really doesn't make sense to compress if the initramfs file in
question is embedded directly into
kernel (which at least version 2012.05 and earlier buildroot did)
because usually the kernel
itself would be also compressed. So in that specific case doing double
compression
(once for the initramfs file embedded into kernel and then again for the
whole kernel+initramfs thing)
would be waste of time.
But for initramfs files *outside* kernel these two new compression
methods are very usefull.
lzo being just barely better in compression than gzip but fastest
decompression of all,
while xz being the best compression.
(So lzo replaces gzip and xz replaces bzip2 and lzma)
I use xz for kernel itself and also for outside initramfs
> I noticed when checking the bzImage size in a tweaked qemu_x86_defconfig
> test run that it never changed much.
> New patchset will follow that fixes this and the previous issue.
> Regards.
Stefan
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-21 22:55 ` [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods Gustavo Zacarias
2013-01-21 23:38 ` Stefan Fröberg
@ 2013-01-22 17:05 ` Thomas Petazzoni
2013-01-22 18:17 ` Stefan Fröberg
2013-01-24 7:09 ` Arnout Vandecappelle
2 siblings, 1 reply; 15+ messages in thread
From: Thomas Petazzoni @ 2013-01-22 17:05 UTC (permalink / raw)
To: buildroot
Dear Gustavo Zacarias,
On Mon, 21 Jan 2013 19:55:43 -0300, Gustavo Zacarias wrote:
> +config BR2_TARGET_ROOTFS_CPIO_LZO
> + bool "lzo"
> + help
> + Do compress the cpio filesystem with lzo.
> +
> +config BR2_TARGET_ROOTFS_CPIO_XZ
> + bool "xz"
> + help
> + Do compress the cpio filesystem with xz.
I don't remember exactly, but isn't the kernel build process itself
capable of compressing the initramfs already? I remember we had some
discussion about this a while ago, but I don't remember the outcome of
these. Arnout was leading this discussion, IIRC.
Best regards,
Thomas
--
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-22 17:05 ` Thomas Petazzoni
@ 2013-01-22 18:17 ` Stefan Fröberg
2013-01-24 7:11 ` Arnout Vandecappelle
2013-01-24 10:48 ` Gustavo Zacarias
0 siblings, 2 replies; 15+ messages in thread
From: Stefan Fröberg @ 2013-01-22 18:17 UTC (permalink / raw)
To: buildroot
22.1.2013 19:05, Thomas Petazzoni kirjoitti:
> Dear Gustavo Zacarias,
>
> On Mon, 21 Jan 2013 19:55:43 -0300, Gustavo Zacarias wrote:
>
>> +config BR2_TARGET_ROOTFS_CPIO_LZO
>> + bool "lzo"
>> + help
>> + Do compress the cpio filesystem with lzo.
>> +
>> +config BR2_TARGET_ROOTFS_CPIO_XZ
>> + bool "xz"
>> + help
>> + Do compress the cpio filesystem with xz.
> I don't remember exactly, but isn't the kernel build process itself
> capable of compressing the initramfs already? I remember we had some
> discussion about this a while ago, but I don't remember the outcome of
> these. Arnout was leading this discussion, IIRC.
If it's embedded initramfs then yes, kernel can handle it all itself
just nicely.
without needing to compress inside initramfs.
(images kernel.png and kernel2.png)
But if outside cpio-archive (aka initramfs) then no. It has to be done
by hand.
And like I said to Gustavo it's bad to do double compression.
Here's are the different combinations:
1 ) compressed initramfs inside compressed kernel = bad (waste of time
decompressing twice, and also waste of package file header size overhead)
2 ) compressed initramfs inside uncompressed kernel = better (no time
waste but still space waste)
3 ) uncompressed initramfs inside compressed kernel = best solution for
embedded initramfs (no time waste & time space waste)
4 ) uncompressed outside initramfs and uncompressed kernel (waste of
space and even according to some benchmarks, waste of time)
5 ) compressed outside initramfs and compressed kernel (no wasted time
and no wasted space)
So option 3 is best for embedded initramfs and option 5 for otherwise
for those who want more flexibility with their enivronment.
For example I could have this:
-------------------------------------------------------------------------------------------------------------------------------------------------
| embedded, uncompressed initramfs insize xz-compressed kernel, with
just busybox and minimal stuff |
--------------------------------------------------------------------------------------------------------------------------------------------------
|
---------------------------------------------------------------------------------------------------------------------------------
| compressed outside initramfs with ncurses & directfb application,
like links web-browser |
---------------------------------------------------------------------------------------------------------------------------------
|
----------------------------------------------------------------------------------------------------------------------------------
| still another compressed initramfs with Xorg, firefox and all the
heavy stuff |
----------------------------------------------------------------------------------------------------------------------------------
All stackable one top over another (at least grub legacy allows giving
several initramfs).
And in Grub conf I could have boot entries like:
- Minimal system (just busybox and it's apps)
- Light system (ncurses, directfb and other non-Xorg "graphical" apps)
- Full system (full Xorg stuff and apps)
Without outside initramfs option I would have to stick kernel (and it's
modules) to each of those initramfs files which would be a waste of space.
Now I have only one kernel (with or without initramfs) and 2 outside
initramfs which I can mix and match like lego blocks.
Stefan
> Best regards,
>
> Thomas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kernel.png
Type: image/png
Size: 24654 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130122/3fb45fea/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kernel2.png
Type: image/png
Size: 24921 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130122/3fb45fea/attachment-0003.png>
^ permalink raw reply [flat|nested] 15+ messages in thread* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-22 18:17 ` Stefan Fröberg
@ 2013-01-24 7:11 ` Arnout Vandecappelle
2013-01-24 11:55 ` Stefan Fröberg
2013-01-24 10:48 ` Gustavo Zacarias
1 sibling, 1 reply; 15+ messages in thread
From: Arnout Vandecappelle @ 2013-01-24 7:11 UTC (permalink / raw)
To: buildroot
On 01/22/13 19:17, Stefan Fr?berg wrote:
[snip]
> Here's are the different combinations:
Excellent explanation! How about adding this to the documentation?
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-24 7:11 ` Arnout Vandecappelle
@ 2013-01-24 11:55 ` Stefan Fröberg
0 siblings, 0 replies; 15+ messages in thread
From: Stefan Fröberg @ 2013-01-24 11:55 UTC (permalink / raw)
To: buildroot
24.1.2013 9:11, Arnout Vandecappelle kirjoitti:
> On 01/22/13 19:17, Stefan Fr?berg wrote:
> [snip]
>> Here's are the different combinations:
>
> Excellent explanation! How about adding this to the documentation?
>
> Regards,
> Arnout
Well, uh...ok.
But before that I have to finish that <pkg>_CONFIG_FIXUP thing first.
I think ill just make one final, v3 version, of it with fixed doc and
your sed changes
as one single combined patch.
And then Im going to test it one final time before resubmit.
Regards
Stefan
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-22 18:17 ` Stefan Fröberg
2013-01-24 7:11 ` Arnout Vandecappelle
@ 2013-01-24 10:48 ` Gustavo Zacarias
2013-01-24 12:08 ` Stefan Fröberg
1 sibling, 1 reply; 15+ messages in thread
From: Gustavo Zacarias @ 2013-01-24 10:48 UTC (permalink / raw)
To: buildroot
On 01/22/2013 03:17 PM, Stefan Fr?berg wrote:
>> I don't remember exactly, but isn't the kernel build process itself
>> capable of compressing the initramfs already? I remember we had some
>> discussion about this a while ago, but I don't remember the outcome of
>> these. Arnout was leading this discussion, IIRC.
>
> If it's embedded initramfs then yes, kernel can handle it all itself
> just nicely.
> without needing to compress inside initramfs.
> (images kernel.png and kernel2.png)
>
> But if outside cpio-archive (aka initramfs) then no. It has to be done
> by hand.
> And like I said to Gustavo it's bad to do double compression.
Well, it's not always bad to do double compression.
On some systems where touching the bootloader isn't an option with say,
uboot, if uboot doesn't understand the newer compression method you're
most likely stuck with gzip or none for the uImage.
So doing xz for the initramfs in that case would be a space saver - even
if you're doing double (de)compression.
Granted, you'll pay it dearly in cpu time.
I started looking at this because someone in #elinux started asking
about making the tinyest possible kernel+initramfs for x86 for pxeboot
with some packages and i saw we were lacking options to go for the best.
Also noticed we weren't doing anything about the compressed cpio archive
so the resulting combined kernel was roughly always the same size.
For some unknown reason he wanted everything to fit in a 1.8 MB file -
maybe some PXE limitation, i don't know.
Question is, should we care about those uses cases or not?
Regards.
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-24 10:48 ` Gustavo Zacarias
@ 2013-01-24 12:08 ` Stefan Fröberg
2013-01-24 12:13 ` Gustavo Zacarias
0 siblings, 1 reply; 15+ messages in thread
From: Stefan Fröberg @ 2013-01-24 12:08 UTC (permalink / raw)
To: buildroot
24.1.2013 12:48, Gustavo Zacarias kirjoitti:
> On 01/22/2013 03:17 PM, Stefan Fr?berg wrote:
>>> I don't remember exactly, but isn't the kernel build process itself
>>> capable of compressing the initramfs already? I remember we had some
>>> discussion about this a while ago, but I don't remember the outcome of
>>> these. Arnout was leading this discussion, IIRC.
>> If it's embedded initramfs then yes, kernel can handle it all itself
>> just nicely.
>> without needing to compress inside initramfs.
>> (images kernel.png and kernel2.png)
>>
>> But if outside cpio-archive (aka initramfs) then no. It has to be done
>> by hand.
>> And like I said to Gustavo it's bad to do double compression.
> Well, it's not always bad to do double compression.
> On some systems where touching the bootloader isn't an option with say,
> uboot, if uboot doesn't understand the newer compression method you're
> most likely stuck with gzip or none for the uImage.
> So doing xz for the initramfs in that case would be a space saver - even
> if you're doing double (de)compression.
> Granted, you'll pay it dearly in cpu time.
> I started looking at this because someone in #elinux started asking
> about making the tinyest possible kernel+initramfs for x86 for pxeboot
> with some packages and i saw we were lacking options to go for the best.
> Also noticed we weren't doing anything about the compressed cpio archive
> so the resulting combined kernel was roughly always the same size.
> For some unknown reason he wanted everything to fit in a 1.8 MB file -
> maybe some PXE limitation, i don't know.
> Question is, should we care about those uses cases or not?
> Regards.
I think users should be allowed to do every possible combination they
want to.
But also give them big fat warning in cases when their choices might
drop performance
so that they are perfectly aware of consequences of the choices they
made in config.
So yeah, double (de)compression is option too but with big fat warning
added that might
affect your system speed.
User is the King.
Stefan
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-24 12:08 ` Stefan Fröberg
@ 2013-01-24 12:13 ` Gustavo Zacarias
0 siblings, 0 replies; 15+ messages in thread
From: Gustavo Zacarias @ 2013-01-24 12:13 UTC (permalink / raw)
To: buildroot
On 01/24/2013 09:08 AM, Stefan Fr?berg wrote:
> I think users should be allowed to do every possible combination they
> want to.
> But also give them big fat warning in cases when their choices might
> drop performance
> so that they are perfectly aware of consequences of the choices they
> made in config.
> So yeah, double (de)compression is option too but with big fat warning
> added that might
> affect your system speed.
>
> User is the King.
At this point my patchset (v2) does nothing to address the issue, so
it's safe to commit.
I was leaving that functionality for a separate one afterwards if the
consensus is "go for it".
Regards.
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-21 22:55 ` [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods Gustavo Zacarias
2013-01-21 23:38 ` Stefan Fröberg
2013-01-22 17:05 ` Thomas Petazzoni
@ 2013-01-24 7:09 ` Arnout Vandecappelle
2013-01-24 10:57 ` Gustavo Zacarias
2 siblings, 1 reply; 15+ messages in thread
From: Arnout Vandecappelle @ 2013-01-24 7:09 UTC (permalink / raw)
To: buildroot
On 01/21/13 23:55, Gustavo Zacarias wrote:
> Signed-off-by: Gustavo Zacarias<gustavo@zacarias.com.ar>
> ---
> fs/common.mk | 8 +++++++-
> fs/cpio/Config.in | 18 ++++++++++++++----
> 2 files changed, 21 insertions(+), 5 deletions(-)
>
> diff --git a/fs/common.mk b/fs/common.mk
> index 8b5b2f2..0c171fc 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -39,7 +39,7 @@ ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE)) \
> define ROOTFS_TARGET_INTERNAL
>
> # extra deps
> -ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma)
> +ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
Gustavo,
If you were to submit a v4, could you split this very long line?
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] initramfs: add support for LZO and XZ compression methods
2013-01-24 7:09 ` Arnout Vandecappelle
@ 2013-01-24 10:57 ` Gustavo Zacarias
0 siblings, 0 replies; 15+ messages in thread
From: Gustavo Zacarias @ 2013-01-24 10:57 UTC (permalink / raw)
To: buildroot
On 01/24/2013 04:09 AM, Arnout Vandecappelle wrote:
> Gustavo,
>
> If you were to submit a v4, could you split this very long line?
>
> Regards,
> Arnout
v4? There's not even a v3 yet! ;-)
There are many fixable lines inside fs/ that are easily wrappable.
For example in ubifs/ubifs.mk, cloop/cloop.mk, initramfs/initramfs.mk
(if we're going for the strict 80 column standard) and others.
But yes, on a new patchset if necessary i could.
Regards.
^ permalink raw reply [flat|nested] 15+ messages in thread
