[Buildroot] [PATCH] package/lzma-alone: new package

[Buildroot] [PATCH] package/lzma-alone: new package

[Rafał Miłecki]

From: Rafa? Mi?ecki <rafal@milecki.pl>

This new package provides "lzma_alone" host binary based on the original
LZMA SDK. It provides few extra options when compared to the LZMA Utils
project "lzma" binary (already packaged as the "lzma").

This tool is required for preparing firmware images for some home
routers with picky bootloaders. It's main advantage is support for
specifying dictionary size and lc/lp/pb LZMA values.

Broadcom CFE bootloader for MIPS BCM47xx devices is known to fail to
boot images using LZMA compression with dictionary.
Broadcom cferam bootloader for ARM64 BCM4908 devices is known to fail to
boot images without uncompressed size specified.
Broadcom CFE for BCM63xx DSL devices has even different requirements.

Packaging LZMA SDK "lzma" as "lzma_alone" is how it's handled in the
Arch, Debian and Ubuntu.

Signed-off-by: Rafa? Mi?ecki <rafal@milecki.pl>
---
 package/lzma-alone/lzma-alone.hash |  1 +
 package/lzma-alone/lzma-alone.mk   | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 package/lzma-alone/lzma-alone.hash
 create mode 100644 package/lzma-alone/lzma-alone.mk

diff --git a/package/lzma-alone/lzma-alone.hash b/package/lzma-alone/lzma-alone.hash
new file mode 100644
index 0000000000..39759c4e3b
--- /dev/null
+++ b/package/lzma-alone/lzma-alone.hash
@@ -0,0 +1 @@
+sha256	9aade84f229fb25f7aef39d8866b375fe6d35a9e18098d7cd86a99e294902944	lzma922.tar.bz2
diff --git a/package/lzma-alone/lzma-alone.mk b/package/lzma-alone/lzma-alone.mk
new file mode 100644
index 0000000000..6a1d8dd3c0
--- /dev/null
+++ b/package/lzma-alone/lzma-alone.mk
@@ -0,0 +1,23 @@
+################################################################################
+#
+# lzma-alone
+#
+################################################################################
+
+LZMA_ALONE_VERSION = 9.22
+LZMA_ALONE_SITE = https://sourceforge.net/projects/sevenzip/files/LZMA%20SDK/
+LZMA_ALONE_SOURCE = lzma922.tar.bz2
+LZMA_ALONE_STRIP_COMPONENTS = 0
+LZMA_ALONE_LICENSE = LGPL-2.1-or-later, BSD-3-Clause
+
+define HOST_LZMA_ALONE_BUILD_CMDS
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)/C/Util/Lzma -f makefile.gcc
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)/CPP/7zip/Bundles/LzmaCon -f makefile.gcc
+endef
+
+define HOST_LZMA_ALONE_INSTALL_CMDS
+	$(INSTALL) -d -m 0755 $(HOST_DIR)/bin
+	$(INSTALL) -D -m 0755 $(@D)/CPP/7zip/Bundles/LzmaCon/lzma $(HOST_DIR)/bin/lzma_alone
+endef
+
+$(eval $(host-generic-package))
-- 
2.27.0

[Buildroot] [PATCH] package/lzma-alone: new package

[Thomas Petazzoni]

Hello Rafa?,

On Mon, 26 Oct 2020 22:34:33 +0100
Rafa? Mi?ecki <zajec5@gmail.com> wrote:

> From: Rafa? Mi?ecki <rafal@milecki.pl>
> 
> This new package provides "lzma_alone" host binary based on the original
> LZMA SDK. It provides few extra options when compared to the LZMA Utils
> project "lzma" binary (already packaged as the "lzma").
> 
> This tool is required for preparing firmware images for some home
> routers with picky bootloaders. It's main advantage is support for
> specifying dictionary size and lc/lp/pb LZMA values.
> 
> Broadcom CFE bootloader for MIPS BCM47xx devices is known to fail to
> boot images using LZMA compression with dictionary.
> Broadcom cferam bootloader for ARM64 BCM4908 devices is known to fail to
> boot images without uncompressed size specified.
> Broadcom CFE for BCM63xx DSL devices has even different requirements.
> 
> Packaging LZMA SDK "lzma" as "lzma_alone" is how it's handled in the
> Arch, Debian and Ubuntu.
> 
> Signed-off-by: Rafa? Mi?ecki <rafal@milecki.pl>

I am a bit confused by all the different lzma projects out there.

The Debian source package lzma (https://tracker.debian.org/pkg/lzma)
produces the lzma, lzma-alone and lzma-dev packages, and its source
package is apparently referring to what you suggest to use here.

The Buildroot package lzma points to https://tukaani.org/lzma/, which
says "Users of LZMA Utils should move to XZ Utils." and where the
latest release is from 2008.

And then, there is the LZMA SDK at https://www.7-zip.org/sdk.html,
where the latest release is 19.00, from 2019, at
https://www.7-zip.org/a/lzma1900.7z. This seems to be the same files
provided at
https://sourceforge.net/projects/sevenzip/files/LZMA%20SDK/, but you're
using the lzma922 version.

Do you have some clarifications about this (apparent) mess ?

>  package/lzma-alone/lzma-alone.hash |  1 +
>  package/lzma-alone/lzma-alone.mk   | 23 +++++++++++++++++++++++
>  2 files changed, 24 insertions(+)
>  create mode 100644 package/lzma-alone/lzma-alone.hash
>  create mode 100644 package/lzma-alone/lzma-alone.mk

We probably want a Config.in.host entry to this package.

> +define HOST_LZMA_ALONE_INSTALL_CMDS
> +	$(INSTALL) -d -m 0755 $(HOST_DIR)/bin

Not needed, the following $(INSTALL) -D will create the destination
folder for you.

> +	$(INSTALL) -D -m 0755 $(@D)/CPP/7zip/Bundles/LzmaCon/lzma $(HOST_DIR)/bin/lzma_alone
> +endef
> +
> +$(eval $(host-generic-package))

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] package/lzma-alone: new package

[Rafał Miłecki]

On 27.10.2020 10:18, Thomas Petazzoni wrote:
> On Mon, 26 Oct 2020 22:34:33 +0100
> Rafa? Mi?ecki <zajec5@gmail.com> wrote:
> 
>> From: Rafa? Mi?ecki <rafal@milecki.pl>
>>
>> This new package provides "lzma_alone" host binary based on the original
>> LZMA SDK. It provides few extra options when compared to the LZMA Utils
>> project "lzma" binary (already packaged as the "lzma").
>>
>> This tool is required for preparing firmware images for some home
>> routers with picky bootloaders. It's main advantage is support for
>> specifying dictionary size and lc/lp/pb LZMA values.
>>
>> Broadcom CFE bootloader for MIPS BCM47xx devices is known to fail to
>> boot images using LZMA compression with dictionary.
>> Broadcom cferam bootloader for ARM64 BCM4908 devices is known to fail to
>> boot images without uncompressed size specified.
>> Broadcom CFE for BCM63xx DSL devices has even different requirements.
>>
>> Packaging LZMA SDK "lzma" as "lzma_alone" is how it's handled in the
>> Arch, Debian and Ubuntu.
>>
>> Signed-off-by: Rafa? Mi?ecki <rafal@milecki.pl>
> 
> I am a bit confused by all the different lzma projects out there.
> 
> The Debian source package lzma (https://tracker.debian.org/pkg/lzma)
> produces the lzma, lzma-alone and lzma-dev packages, and its source
> package is apparently referring to what you suggest to use here.
> 
> The Buildroot package lzma points to https://tukaani.org/lzma/, which
> says "Users of LZMA Utils should move to XZ Utils." and where the
> latest release is from 2008.
> 
> And then, there is the LZMA SDK at https://www.7-zip.org/sdk.html,
> where the latest release is 19.00, from 2019, at
> https://www.7-zip.org/a/lzma1900.7z. This seems to be the same files
> provided at
> https://sourceforge.net/projects/sevenzip/files/LZMA%20SDK/, but you're
> using the lzma922 version.
> 
> Do you have some clarifications about this (apparent) mess ?

I wasn't fully aware of Debian solution. So it seems that Debian
developers:
1. Switched from LZMA Utils to the LZMA SDK
2. Kept "lzma" CLI from the LZMA Utils project

They did that by patching LZMA SDK sources (see
lzma_9.22-2.1.debian.tar-1.xz and its "patches"). They ported lzmp.cpp
from the LZMA Utils.

Advantages:
1. Single code base
2. Using the latest LZMA core code

Disadvantages:
1. Some missing tools like lzmainfo

To make things worse, there is also a XZ Utils project that also
provides binaries lzma (with CLI like in LZMA Utils), unlzma & lzmainfo.


What strategy should buildroot follow in this case?


As for version choise I used 9.22 as that is what Debian uses and it's
the latest release with sources available as tar.bz2 (newer use 7z).


>>   package/lzma-alone/lzma-alone.hash |  1 +
>>   package/lzma-alone/lzma-alone.mk   | 23 +++++++++++++++++++++++
>>   2 files changed, 24 insertions(+)
>>   create mode 100644 package/lzma-alone/lzma-alone.hash
>>   create mode 100644 package/lzma-alone/lzma-alone.mk
> 
> We probably want a Config.in.host entry to this package.

What extra options should I put in Config.in.host?


>> +define HOST_LZMA_ALONE_INSTALL_CMDS
>> +	$(INSTALL) -d -m 0755 $(HOST_DIR)/bin
> 
> Not needed, the following $(INSTALL) -D will create the destination
> folder for you.

Thanks!

[Buildroot] [PATCH] package/lzma-alone: new package

[Rafał Miłecki]

Hi Thomas,

On 27.10.2020 15:33, Rafa? Mi?ecki wrote:
> On 27.10.2020 10:18, Thomas Petazzoni wrote:
>> On Mon, 26 Oct 2020 22:34:33 +0100
>> Rafa? Mi?ecki <zajec5@gmail.com> wrote:
>>
>>> From: Rafa? Mi?ecki <rafal@milecki.pl>
>>>
>>> This new package provides "lzma_alone" host binary based on the original
>>> LZMA SDK. It provides few extra options when compared to the LZMA Utils
>>> project "lzma" binary (already packaged as the "lzma").
>>>
>>> This tool is required for preparing firmware images for some home
>>> routers with picky bootloaders. It's main advantage is support for
>>> specifying dictionary size and lc/lp/pb LZMA values.
>>>
>>> Broadcom CFE bootloader for MIPS BCM47xx devices is known to fail to
>>> boot images using LZMA compression with dictionary.
>>> Broadcom cferam bootloader for ARM64 BCM4908 devices is known to fail to
>>> boot images without uncompressed size specified.
>>> Broadcom CFE for BCM63xx DSL devices has even different requirements.
>>>
>>> Packaging LZMA SDK "lzma" as "lzma_alone" is how it's handled in the
>>> Arch, Debian and Ubuntu.
>>>
>>> Signed-off-by: Rafa? Mi?ecki <rafal@milecki.pl>
>>
>> I am a bit confused by all the different lzma projects out there.
>>
>> The Debian source package lzma (https://tracker.debian.org/pkg/lzma)
>> produces the lzma, lzma-alone and lzma-dev packages, and its source
>> package is apparently referring to what you suggest to use here.
>>
>> The Buildroot package lzma points to https://tukaani.org/lzma/, which
>> says "Users of LZMA Utils should move to XZ Utils." and where the
>> latest release is from 2008.
>>
>> And then, there is the LZMA SDK at https://www.7-zip.org/sdk.html,
>> where the latest release is 19.00, from 2019, at
>> https://www.7-zip.org/a/lzma1900.7z. This seems to be the same files
>> provided at
>> https://sourceforge.net/projects/sevenzip/files/LZMA%20SDK/, but you're
>> using the lzma922 version.
>>
>> Do you have some clarifications about this (apparent) mess ?
> 
> I wasn't fully aware of Debian solution. So it seems that Debian
> developers:
> 1. Switched from LZMA Utils to the LZMA SDK
> 2. Kept "lzma" CLI from the LZMA Utils project
> 
> They did that by patching LZMA SDK sources (see
> lzma_9.22-2.1.debian.tar-1.xz and its "patches"). They ported lzmp.cpp
> from the LZMA Utils.
> 
> Advantages:
> 1. Single code base
> 2. Using the latest LZMA core code
> 
> Disadvantages:
> 1. Some missing tools like lzmainfo
> 
> To make things worse, there is also a XZ Utils project that also
> provides binaries lzma (with CLI like in LZMA Utils), unlzma & lzmainfo.
> 
> 
> What strategy should buildroot follow in this case?

Any opinion on that?