[Buildroot] [PATCH 0/2] use host compressor for rootfs

[Buildroot] [PATCH 0/2] use host compressor for rootfs

[Philippe Reynes]

When the rootfs is compressed with lzma, lzo or xz, we
use the binary compiled by buildroot. But with gzip
or bzip, we use the tools of the machine. This serie
change this behaviour for gzip and bzip2, it add
a dependancy on host-<compressor> and use it to
compress the rootfs.

Philippe Reynes (2):
  rootfs-gzip: use host-gzip instead of local gzip
  rootfs-bzip: use host-bzip2 instead of local bzip2

 fs/common.mk           |    6 ++++--
 package/bzip2/bzip2.mk |    2 ++
 package/gzip/gzip.mk   |    3 +++
 3 files changed, 9 insertions(+), 2 deletions(-)

-- 
1.7.4.4

[Buildroot] [PATCH 1/2] rootfs-gzip: use host-gzip instead of local gzip

[Philippe Reynes]

Signed-off-by: Philippe Reynes <tremyfr@yahoo.fr>
---
 fs/common.mk         |    3 ++-
 package/gzip/gzip.mk |    3 +++
 2 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/fs/common.mk b/fs/common.mk
index d95c26b..57bcd2d 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -41,8 +41,9 @@ ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \
 	$(if $(PACKAGES_USERS),host-mkpasswd)
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_GZIP),y)
+ROOTFS_$(2)_DEPENDENCIES += host-gzip
 ROOTFS_$(2)_COMPRESS_EXT = .gz
-ROOTFS_$(2)_COMPRESS_CMD = gzip -9 -c
+ROOTFS_$(2)_COMPRESS_CMD = $$(GZIP) -9 -c
 endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_BZIP2),y)
 ROOTFS_$(2)_COMPRESS_EXT = .bz2
diff --git a/package/gzip/gzip.mk b/package/gzip/gzip.mk
index e1a7bef..ced6485 100644
--- a/package/gzip/gzip.mk
+++ b/package/gzip/gzip.mk
@@ -11,3 +11,6 @@ GZIP_LICENSE = GPLv3+
 GZIP_LICENSE_FILES = COPYING
 
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))
+
+GZIP = $(HOST_DIR)/usr/bin/gzip
-- 
1.7.4.4

[Buildroot] [PATCH 2/2] rootfs-bzip: use host-bzip2 instead of local bzip2

[Philippe Reynes]

Signed-off-by: Philippe Reynes <tremyfr@yahoo.fr>
---
 fs/common.mk           |    3 ++-
 package/bzip2/bzip2.mk |    2 ++
 2 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/fs/common.mk b/fs/common.mk
index 57bcd2d..8e11b46 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -46,8 +46,9 @@ ROOTFS_$(2)_COMPRESS_EXT = .gz
 ROOTFS_$(2)_COMPRESS_CMD = $$(GZIP) -9 -c
 endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_BZIP2),y)
+ROOTFS_$(2)_DEPENDENCIES += host-bzip2
 ROOTFS_$(2)_COMPRESS_EXT = .bz2
-ROOTFS_$(2)_COMPRESS_CMD = bzip2 -9 -c
+ROOTFS_$(2)_COMPRESS_CMD = $$(BZIP2) -9 -c
 endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZMA),y)
 ROOTFS_$(2)_DEPENDENCIES += host-lzma
diff --git a/package/bzip2/bzip2.mk b/package/bzip2/bzip2.mk
index fcddb55..469f72b 100644
--- a/package/bzip2/bzip2.mk
+++ b/package/bzip2/bzip2.mk
@@ -65,3 +65,5 @@ endef
 
 $(eval $(generic-package))
 $(eval $(host-generic-package))
+
+BZIP2 = $(HOST_DIR)/usr/bin/bzip2
-- 
1.7.4.4

[Buildroot] [PATCH 0/2] use host compressor for rootfs

[Baruch Siach]

Hi Philippe,

On Sun, Apr 06, 2014 at 09:48:19AM +0200, Philippe Reynes wrote:
> When the rootfs is compressed with lzma, lzo or xz, we
> use the binary compiled by buildroot. But with gzip
> or bzip, we use the tools of the machine. This serie
> change this behaviour for gzip and bzip2, it add
> a dependancy on host-<compressor> and use it to
> compress the rootfs.

gzip and bzip2 are mandatory dependencies of buildroot 
(http://nightly.buildroot.org/manual.html#requirement-mandatory). These 
packages are shipped by default with virtually all general purpose distros. 
What problem are you trying to fix besides consistency?

baruch

> Philippe Reynes (2):
>   rootfs-gzip: use host-gzip instead of local gzip
>   rootfs-bzip: use host-bzip2 instead of local bzip2
> 
>  fs/common.mk           |    6 ++++--
>  package/bzip2/bzip2.mk |    2 ++
>  package/gzip/gzip.mk   |    3 +++
>  3 files changed, 9 insertions(+), 2 deletions(-)
> 
> -- 
> 1.7.4.4

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [PATCH 0/2] use host compressor for rootfs

[Philippe Reynes]

Hi Baruch,

I've missed this "mandatory dependancy".
I suppose that this dependancy is usefull to uncompress tarball.
But I think that the compilation of firmware should be done with
tools compiled by buildroot.

If lzma is used to compress the rootfs, then lzma is compile
by builtoot. If gzip is used to compress the rootfs, then we use
local gzip. It's not very "coherent".
And using compiled binary with buildroot increase chance to
regenerate firmware on (very) different machine.

And I agree with you, it's not a "real" issue, it's more a "nice to have".

Regards,
Philippe



Le Dimanche 6 avril 2014 9h59, Baruch Siach <baruch@tkos.co.il> a ?crit :
 
Hi Philippe,

On Sun, Apr 06, 2014 at 09:48:19AM +0200, Philippe Reynes wrote:
> When the rootfs is compressed with lzma, lzo or xz, we
> use the binary compiled by buildroot. But with gzip
> or bzip, we use the tools of the machine. This serie
> change this behaviour for gzip and bzip2, it add
> a dependancy on host-<compressor> and use it to
> compress the rootfs.

gzip and bzip2 are mandatory dependencies of buildroot 
(http://nightly.buildroot.org/manual.html#requirement-mandatory). These 
packages are shipped by default with virtually all general purpose distros. 
What problem are you trying to fix besides consistency?

baruch


> Philippe Reynes (2):
>?  rootfs-gzip: use host-gzip instead of local gzip
>?  rootfs-bzip: use host-bzip2 instead of local bzip2
> 
>? fs/common.mk? ? ? ? ?  |? ? 6 ++++--
>? package/bzip2/bzip2.mk |? ? 2 ++
>? package/gzip/gzip.mk?  |? ? 3 +++
>? 3 files changed, 9 insertions(+), 2 deletions(-)
> 
> -- 
> 1.7.4.4

-- 
? ? http://baruch.siach.name/blog/ ? ? ? ? ? ? ? ?  ~. .~?  Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
?  - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il 
-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140406/9ef8f90e/attachment.html>

[Buildroot] [PATCH 0/2] use host compressor for rootfs

[Thomas De Schampheleire]

Philippe Reynes <tremyfr@yahoo.fr> schreef:
>Hi Baruch,
>
>I've missed this "mandatory dependancy".
>I suppose that this dependancy is usefull to uncompress tarball.
>But I think that the compilation of firmware should be done with
>tools compiled by buildroot.
>
>If lzma is used to compress the rootfs, then lzma is compile
>by builtoot. If gzip is used to compress the rootfs, then we use
>local gzip. It's not very "coherent".
>And using compiled binary with buildroot increase chance to
>regenerate firmware on (very) different machine.
>
>And I agree with you, it's not a "real" issue, it's more a "nice to have".

The problem is that the build time is increased without good reason.
If there is a problem with a specific version of gzip/bzip2, we could always use the suitable-host-package mechanism as we do for tar. In fact, you could start using it for lzma too and avoid building host-lzma if it's already present...

Best regards,
Thomas

[Buildroot] [PATCH 0/2] use host compressor for rootfs

[Thomas Petazzoni]

Dear Philippe Reynes,

On Sun, 6 Apr 2014 10:25:24 +0100 (BST), Philippe Reynes wrote:

> I've missed this "mandatory dependancy".
> I suppose that this dependancy is usefull to uncompress tarball.
> But I think that the compilation of firmware should be done with
> tools compiled by buildroot.
> 
> If lzma is used to compress the rootfs, then lzma is compile
> by builtoot. If gzip is used to compress the rootfs, then we use
> local gzip. It's not very "coherent".
> And using compiled binary with buildroot increase chance to
> regenerate firmware on (very) different machine.
> 
> And I agree with you, it's not a "real" issue, it's more a "nice to have".

Well, I guess there are two important things here:

 1) Buildroot tries to avoid building native utilities needlessly. So
    when possible, we try to use the tools available on the machine, in
    order to keep the overall build time as reasonable as possible.

 2) gzip and bzip2 are widely available tools, which haven't caused any
    compatibility issue in years. On the other hands, tools like lzma,
    xz and so on are a lot less widely available. They may not even be
    available at all on old Linux distributions that some companies are
    still using.

Both of these points combined explain why we handle gzip/bzip2
differently than lzma/xz.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 0/2] use host compressor for rootfs

[Baruch Siach]

Hi Thomas, Thomas,

On Sun, Apr 06, 2014 at 11:32:48AM +0200, Thomas Petazzoni wrote:
> On Sun, 6 Apr 2014 10:25:24 +0100 (BST), Philippe Reynes wrote:
> 
> > I've missed this "mandatory dependancy".
> > I suppose that this dependancy is usefull to uncompress tarball.
> > But I think that the compilation of firmware should be done with
> > tools compiled by buildroot.
> > 
> > If lzma is used to compress the rootfs, then lzma is compile
> > by builtoot. If gzip is used to compress the rootfs, then we use
> > local gzip. It's not very "coherent".
> > And using compiled binary with buildroot increase chance to
> > regenerate firmware on (very) different machine.
> > 
> > And I agree with you, it's not a "real" issue, it's more a "nice to have".
> 
> Well, I guess there are two important things here:
> 
>  1) Buildroot tries to avoid building native utilities needlessly. So
>     when possible, we try to use the tools available on the machine, in
>     order to keep the overall build time as reasonable as possible.
> 
>  2) gzip and bzip2 are widely available tools, which haven't caused any
>     compatibility issue in years. On the other hands, tools like lzma,
>     xz and so on are a lot less widely available. They may not even be
>     available at all on old Linux distributions that some companies are
>     still using.
> 
> Both of these points combined explain why we handle gzip/bzip2
> differently than lzma/xz.

Since commit d1f325f554cab (xzcat: treat as host prerequisite and build if 
needed) xz is always built when the host does not have it. Can we remove 
host-xz from the dependencies of HOST_SQUASHFS and ROOTFS_*_XZ?

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [PATCH 0/2] use host compressor for rootfs

[Philippe Reynes]

Hi Thomas,

I've experienced a weird behavior with lzma.
We are using lzma to compress initramfs.

On an old buildroot where the local lzma was used.

It was working fine on some machine, but it fails
on others. The fails was a "not enought memory"
at boot time. Of course, it was also an old kernel.
I never understood why using local lzma raise
this issue.


After we add a dependency to "host-lzma",
it works everywhere.

So using host-<compressor> fix this issue, and
now we have the same behavior on all desktop.

This serie is based on : if it could happen on 

lzma, it could happen on gzip/bzip2.


Regards,
Philippe


Le Dimanche 6 avril 2014 11h32, Thomas De Schampheleire <patrickdepinguin@gmail.com> a ?crit :
 
Philippe Reynes <tremyfr@yahoo.fr> schreef:

>Hi Baruch,
>
>I've missed this "mandatory dependancy".
>I suppose that this dependancy is usefull to uncompress tarball.
>But I think that the compilation of firmware should be done with
>tools compiled by buildroot.
>
>If lzma is used to compress the rootfs, then lzma is compile
>by builtoot. If gzip is used to compress the rootfs, then we use
>local gzip. It's not very "coherent".
>And using compiled binary with buildroot increase chance to
>regenerate firmware on (very) different machine.
>
>And I agree with you, it's not a "real" issue, it's more a "nice to have".

The problem is that the build time is increased without good reason.
If there is a problem with a specific version of gzip/bzip2, we could always use the suitable-host-package mechanism as we do for tar. In fact, you could start using it for lzma too and avoid building host-lzma if it's already present...

Best regards,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140406/7000f352/attachment-0001.html>

[Buildroot] [PATCH 0/2] use host compressor for rootfs

[Thomas Petazzoni]

Dear Philippe Reynes,

On Sun,  6 Apr 2014 09:48:19 +0200, Philippe Reynes wrote:
> When the rootfs is compressed with lzma, lzo or xz, we
> use the binary compiled by buildroot. But with gzip
> or bzip, we use the tools of the machine. This serie
> change this behaviour for gzip and bzip2, it add
> a dependancy on host-<compressor> and use it to
> compress the rootfs.
> 
> Philippe Reynes (2):
>   rootfs-gzip: use host-gzip instead of local gzip
>   rootfs-bzip: use host-bzip2 instead of local bzip2

Since the general reception about these patches has been fairly
negative, I've marked them as Rejected in patchwork.

Thanks for your contribution!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com