[Buildroot] [PATCH v4] mtree: new package

[Buildroot] [PATCH v4] mtree: new package

[Esben Haabendal]

From: Esben Haabendal <esben@haabendal.dk>

Rebased against latest master, fixing merge conflict with DEVELOPERS file.

Signed-off-by: Esben Haabendal <esben@haabendal.dk>
---
 DEVELOPERS               |  1 +
 package/Config.in        |  1 +
 package/mtree/Config.in  | 15 +++++++++++++++
 package/mtree/mtree.hash |  5 +++++
 package/mtree/mtree.mk   | 17 +++++++++++++++++
 5 files changed, 39 insertions(+)
 create mode 100644 package/mtree/Config.in
 create mode 100644 package/mtree/mtree.hash
 create mode 100644 package/mtree/mtree.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index d21e494b1426..9496b30d3429 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -700,6 +700,7 @@ F:	package/szip/
 
 N:	Esben Haabendal <esben@haabendal.dk>
 F:	boot/gummiboot/
+F:	package/mtree/
 
 N:	Etienne Carriere <etienne.carriere@linaro.org>
 F:	boot/optee-os/
diff --git a/package/Config.in b/package/Config.in
index e372e0c43009..747eb39435f8 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -197,6 +197,7 @@ menu "Filesystem and flash utilities"
 	source "package/mmc-utils/Config.in"
 	source "package/mtd/Config.in"
 	source "package/mtools/Config.in"
+	source "package/mtree/Config.in"
 	source "package/nfs-utils/Config.in"
 	source "package/nilfs-utils/Config.in"
 	source "package/ntfs-3g/Config.in"
diff --git a/package/mtree/Config.in b/package/mtree/Config.in
new file mode 100644
index 000000000000..175d05d3bb0d
--- /dev/null
+++ b/package/mtree/Config.in
@@ -0,0 +1,15 @@
+config BR2_PACKAGE_MTREE
+	bool "mtree"
+	# mtree uses <fts.h> which is not included by default in uClibc
+	depends on BR2_TOOLCHAIN_USES_GLIBC
+	# fts.h does not support LFS (_FILE_OFFSET_BITS==64) in glibc versions
+	# older than 2.23, and codesourcery-arm is currently using glibc 2.18
+	depends on !BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM
+	select BR2_PACKAGE_OPENSSL
+	help
+	  mtree - utility for creating and verifying file hierarchies.
+
+	  Port of the mtree utility, originally only available in BSD
+	  distributions, to Linux and other operating systems.
+
+	  https://github.com/archiecobbs/mtree-port
diff --git a/package/mtree/mtree.hash b/package/mtree/mtree.hash
new file mode 100644
index 000000000000..8dc54a201133
--- /dev/null
+++ b/package/mtree/mtree.hash
@@ -0,0 +1,5 @@
+# Locally computed
+sha256 fb5e99b28e270ab9a4622bd7142e9a7e9703fa08db3086398c52d33bb0505327  mtree-1.0.4.tar.gz
+
+# License file (locally computed as well)
+sha256 024360ed813a773260cee77ae84196da44a0842f45cd9bddad869de13d6c4bd2  COPYING
diff --git a/package/mtree/mtree.mk b/package/mtree/mtree.mk
new file mode 100644
index 000000000000..6227508106c7
--- /dev/null
+++ b/package/mtree/mtree.mk
@@ -0,0 +1,17 @@
+################################################################################
+#
+# mtree
+#
+################################################################################
+
+MTREE_VERSION = 1.0.4
+MTREE_SOURCE = mtree-$(MTREE_VERSION).tar.gz
+MTREE_SITE = https://s3.amazonaws.com/archie-public/mtree-port
+MTREE_LICENSE = BSD-3-Clause
+MTREE_LICENSE_FILES = COPYING
+
+MTREE_DEPENDENCIES = openssl
+
+MTREE_INSTALL_TARGET_OPTS = DESTDIR=$(TARGET_DIR) install-exec
+
+$(eval $(autotools-package))
-- 
2.21.0

[Buildroot] [PATCH v4] mtree: new package

[Thomas Petazzoni]

Hello Esben,

On Tue, 26 Mar 2019 15:24:11 +0100
Esben Haabendal <esben.haabendal@gmail.com> wrote:

> diff --git a/package/mtree/Config.in b/package/mtree/Config.in
> new file mode 100644
> index 000000000000..175d05d3bb0d
> --- /dev/null
> +++ b/package/mtree/Config.in
> @@ -0,0 +1,15 @@
> +config BR2_PACKAGE_MTREE
> +	bool "mtree"
> +	# mtree uses <fts.h> which is not included by default in uClibc
> +	depends on BR2_TOOLCHAIN_USES_GLIBC
> +	# fts.h does not support LFS (_FILE_OFFSET_BITS==64) in glibc versions
> +	# older than 2.23, and codesourcery-arm is currently using glibc 2.18
> +	depends on !BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM

My memory is a bit fuzzy on this, but we have some workarounds for this
issue in other packages:

# Undefining _FILE_OFFSET_BITS here because of a "bug" with glibc fts.h
# large file support. See https://bugzilla.redhat.com/show_bug.cgi?id=574992
# for more information.
LIBCGROUP_CONF_ENV = \
        CXXFLAGS="$(TARGET_CXXFLAGS) -U_FILE_OFFSET_BITS" \
        CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS"


# Undefining _FILE_OFFSET_BITS here because of a "bug" with glibc fts.h
# large file support.
# See https://bugzilla.redhat.com/show_bug.cgi?id=574992 for more information
RESTORECOND_MAKE_OPTS += \
        $(TARGET_CONFIGURE_OPTS) \
        CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS" \
        CPPFLAGS="$(TARGET_CPPFLAGS) -U_FILE_OFFSET_BITS" \
        ARCH="$(BR2_ARCH)"

and in a few other places.

Is it the same problem ? If it has been resolved in glibc 2.23, should
we remove those work-arounds ?

I just would like to make sure we don't work around this problem in
different ways in different packages.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v4] mtree: new package

[Esben Haabendal]

Hi Thomas

Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

> On Tue, 26 Mar 2019 15:24:11 +0100
> Esben Haabendal <esben.haabendal@gmail.com> wrote:
>
>> diff --git a/package/mtree/Config.in b/package/mtree/Config.in
>> new file mode 100644
>> index 000000000000..175d05d3bb0d
>> --- /dev/null
>> +++ b/package/mtree/Config.in
>> @@ -0,0 +1,15 @@
>> +config BR2_PACKAGE_MTREE
>> +	bool "mtree"
>> +	# mtree uses <fts.h> which is not included by default in uClibc
>> +	depends on BR2_TOOLCHAIN_USES_GLIBC
>> +	# fts.h does not support LFS (_FILE_OFFSET_BITS==64) in glibc versions
>> +	# older than 2.23, and codesourcery-arm is currently using glibc 2.18
>> +	depends on !BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM
>
> My memory is a bit fuzzy on this, but we have some workarounds for this
> issue in other packages:
>
> # Undefining _FILE_OFFSET_BITS here because of a "bug" with glibc fts.h
> # large file support. See https://bugzilla.redhat.com/show_bug.cgi?id=574992
> # for more information.
> LIBCGROUP_CONF_ENV = \
>         CXXFLAGS="$(TARGET_CXXFLAGS) -U_FILE_OFFSET_BITS" \
>         CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS"
>
>
> # Undefining _FILE_OFFSET_BITS here because of a "bug" with glibc fts.h
> # large file support.
> # See https://bugzilla.redhat.com/show_bug.cgi?id=574992 for more information
> RESTORECOND_MAKE_OPTS += \
>         $(TARGET_CONFIGURE_OPTS) \
>         CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS" \
>         CPPFLAGS="$(TARGET_CPPFLAGS) -U_FILE_OFFSET_BITS" \
>         ARCH="$(BR2_ARCH)"
>
> and in a few other places.
>
> Is it the same problem ?

Yes, sounds very much like the same problem.

> If it has been resolved in glibc 2.23, should
> we remove those work-arounds ?

It depends.  If we do that, people using external toolchains with glibc
older than 2.23 would see breakage.

> I just would like to make sure we don't work around this problem in
> different ways in different packages.

Sounds like a good plan.  Unfortunately, I don't see that any of the
proposed workaround generally good.

For packages where large file support is not deemed an important
feature, The -U_FILE_OFFSET_BITS workaround is good.  But for packages
where large file support could be relevant, it seems like a bad idea to
disable it just to stay compatible with old glibc versions that only a
few people use together with new buildroot versions.

If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree
package.  I personally don't need large file support...

/Esben

[Buildroot] [PATCH v4] mtree: new package

[Thomas Petazzoni]

Hello Esben,

On Wed, 27 Mar 2019 23:28:58 +0100
Esben Haabendal <esben.haabendal@gmail.com> wrote:

> > If it has been resolved in glibc 2.23, should
> > we remove those work-arounds ?  
> 
> It depends.  If we do that, people using external toolchains with glibc
> older than 2.23 would see breakage.
> 
> > I just would like to make sure we don't work around this problem in
> > different ways in different packages.  
> 
> Sounds like a good plan.  Unfortunately, I don't see that any of the
> proposed workaround generally good.
> 
> For packages where large file support is not deemed an important
> feature, The -U_FILE_OFFSET_BITS workaround is good.  But for packages
> where large file support could be relevant, it seems like a bad idea to
> disable it just to stay compatible with old glibc versions that only a
> few people use together with new buildroot versions.
> 
> If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree
> package.  I personally don't need large file support...

OK, I did a bit of testing, taking as example the libcgroup package. I
confirmed that without -U_FILE_OFFSET_BITS, libcgroup fails to build
with the Sourcery ARM toolchain (old glibc), but builds perfectly fine
with the Linaro toolchain (recent glibc).

So, from there, you have two options:

 (1) You don't have much time and you're not willing to not much
     additional work. In this case, use the -U_FILE_OFFSET_BITS trick in
     mtree. This way, we have the same quirk/workaround everywhere for this
     fts.h problem.

 (2) You have more time. In this case, add a hidden option in
     toolchain/Config.in called BR2_TOOLCHAIN_HAS_BROKEN_FTS. This hidden
     option should be selected by the broken ARM Codesourcery toolchain.
     Then all packages that do the -U_FILE_OFFSET_BITS trick can be changed
     to only do it when BR2_TOOLCHAIN_HAS_BROKEN_FTS is enabled
     (including mtree, of course).

     Obviously, this leaves old custom external toolchains potentially
     broken, as for these, the work around will not be applied. If this
     turns out to really be a problem in practice, we can always add an
     option BR2_TOOLCHAIN_EXTERNAL_CUSTOM_HAS_BROKEN_FTS, and let the
     user enable it + a check that if this option is not enabled, the
     external toolchain really does not have broken fts.h.

Let me know which option you chose :-)

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v4] mtree: new package

[Esben Haabendal]

Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

> Hello Esben,
>
> On Wed, 27 Mar 2019 23:28:58 +0100
> Esben Haabendal <esben.haabendal@gmail.com> wrote:
>
>> > If it has been resolved in glibc 2.23, should
>> > we remove those work-arounds ?  
>> 
>> It depends.  If we do that, people using external toolchains with glibc
>> older than 2.23 would see breakage.
>> 
>> > I just would like to make sure we don't work around this problem in
>> > different ways in different packages.  
>> 
>> Sounds like a good plan.  Unfortunately, I don't see that any of the
>> proposed workaround generally good.
>> 
>> For packages where large file support is not deemed an important
>> feature, The -U_FILE_OFFSET_BITS workaround is good.  But for packages
>> where large file support could be relevant, it seems like a bad idea to
>> disable it just to stay compatible with old glibc versions that only a
>> few people use together with new buildroot versions.
>> 
>> If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree
>> package.  I personally don't need large file support...
>
> OK, I did a bit of testing, taking as example the libcgroup package. I
> confirmed that without -U_FILE_OFFSET_BITS, libcgroup fails to build
> with the Sourcery ARM toolchain (old glibc), but builds perfectly fine
> with the Linaro toolchain (recent glibc).
>
> So, from there, you have two options:
>
>  (1) You don't have much time and you're not willing to not much
>      additional work. In this case, use the -U_FILE_OFFSET_BITS trick in
>      mtree. This way, we have the same quirk/workaround everywhere for this
>      fts.h problem.
>
>  (2) You have more time. In this case, add a hidden option in
>      toolchain/Config.in called BR2_TOOLCHAIN_HAS_BROKEN_FTS. This hidden
>      option should be selected by the broken ARM Codesourcery toolchain.
>      Then all packages that do the -U_FILE_OFFSET_BITS trick can be changed
>      to only do it when BR2_TOOLCHAIN_HAS_BROKEN_FTS is enabled
>      (including mtree, of course).
>
>      Obviously, this leaves old custom external toolchains potentially
>      broken, as for these, the work around will not be applied. If this
>      turns out to really be a problem in practice, we can always add an
>      option BR2_TOOLCHAIN_EXTERNAL_CUSTOM_HAS_BROKEN_FTS, and let the
>      user enable it + a check that if this option is not enabled, the
>      external toolchain really does not have broken fts.h.
>
> Let me know which option you chose :-)

I think I will give option 2 a try.  Sounds like a nice solution :-)

/Esben

[Buildroot] [PATCH v4] mtree: new package

[Arnout Vandecappelle]

On 28/03/2019 16:14, Esben Haabendal wrote:
> Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:
> 
>> Hello Esben,
>>
>> On Wed, 27 Mar 2019 23:28:58 +0100
>> Esben Haabendal <esben.haabendal@gmail.com> wrote:
>>
>>>> If it has been resolved in glibc 2.23, should
>>>> we remove those work-arounds ?  
>>>
>>> It depends.  If we do that, people using external toolchains with glibc
>>> older than 2.23 would see breakage.
>>>
>>>> I just would like to make sure we don't work around this problem in
>>>> different ways in different packages.  
>>>
>>> Sounds like a good plan.  Unfortunately, I don't see that any of the
>>> proposed workaround generally good.
>>>
>>> For packages where large file support is not deemed an important
>>> feature, The -U_FILE_OFFSET_BITS workaround is good.  But for packages
>>> where large file support could be relevant, it seems like a bad idea to
>>> disable it just to stay compatible with old glibc versions that only a
>>> few people use together with new buildroot versions.
>>>
>>> If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree
>>> package.  I personally don't need large file support...
>>
>> OK, I did a bit of testing, taking as example the libcgroup package. I
>> confirmed that without -U_FILE_OFFSET_BITS, libcgroup fails to build
>> with the Sourcery ARM toolchain (old glibc), but builds perfectly fine
>> with the Linaro toolchain (recent glibc).
>>
>> So, from there, you have two options:
>>
>>  (1) You don't have much time and you're not willing to not much
>>      additional work. In this case, use the -U_FILE_OFFSET_BITS trick in
>>      mtree. This way, we have the same quirk/workaround everywhere for this
>>      fts.h problem.
>>
>>  (2) You have more time. In this case, add a hidden option in
>>      toolchain/Config.in called BR2_TOOLCHAIN_HAS_BROKEN_FTS. This hidden
>>      option should be selected by the broken ARM Codesourcery toolchain.
>>      Then all packages that do the -U_FILE_OFFSET_BITS trick can be changed
>>      to only do it when BR2_TOOLCHAIN_HAS_BROKEN_FTS is enabled
>>      (including mtree, of course).
>>
>>      Obviously, this leaves old custom external toolchains potentially
>>      broken, as for these, the work around will not be applied. If this
>>      turns out to really be a problem in practice, we can always add an
>>      option BR2_TOOLCHAIN_EXTERNAL_CUSTOM_HAS_BROKEN_FTS, and let the
>>      user enable it + a check that if this option is not enabled, the
>>      external toolchain really does not have broken fts.h.
>>
>> Let me know which option you chose :-)
> 
> I think I will give option 2 a try.  Sounds like a nice solution :-)

 To be honest, I'm more inclined to remove the -U completely and add an
autobuilder exception... I don't believe we have to go too far to work around
bugs in old toolchains.

 Regards,
 Arnout

[Buildroot] [PATCH v4] mtree: new package

[Thomas Petazzoni]

On Thu, 28 Mar 2019 19:13:16 +0100
Arnout Vandecappelle <arnout@mind.be> wrote:

> > I think I will give option 2 a try.  Sounds like a nice solution :-)  
> 
>  To be honest, I'm more inclined to remove the -U completely and add an
> autobuilder exception... I don't believe we have to go too far to work around
> bugs in old toolchains.

I'm fine with this as well. It means we would no longer support glibc
2.19 anymore.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v4] mtree: new package

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

 > On Thu, 28 Mar 2019 19:13:16 +0100
 > Arnout Vandecappelle <arnout@mind.be> wrote:

 >> > I think I will give option 2 a try.  Sounds like a nice solution :-)  
 >> 
 >> To be honest, I'm more inclined to remove the -U completely and add an
 >> autobuilder exception... I don't believe we have to go too far to work around
 >> bugs in old toolchains.

 > I'm fine with this as well. It means we would no longer support glibc
 > 2.19 anymore.

Didn't you just last month argue against removing support for glibc <
2.19 support (the runc security fix):

http://lists.busybox.net/pipermail/buildroot/2019-February/244056.html

I don't have a problem dropping support for ancient toolchains when they
add too much complexity, but the lack of large file support is mtree is
probably not really reason enough.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v4] mtree: new package

[Thomas Petazzoni]

On Thu, 28 Mar 2019 21:59:59 +0100
Peter Korsgaard <peter@korsgaard.com> wrote:

>  > I'm fine with this as well. It means we would no longer support glibc
>  > 2.19 anymore.  
> 
> Didn't you just last month argue against removing support for glibc <
> 2.19 support (the runc security fix):

In fact, my last sentence lacked an ending question mark. I wanted to
ask if we were dropping support entirely for glibc 2.19, or just saying
"glibc 2.19 is not long important enough to worry too much about it,
especially when it's just for the few packages that use the <fts.h>
interface".

But if your point is to have me say that I don't have a clear and
well-defined opinion, then yes it's the case. I do see a number of
companies/customers continue to use old toolchains and therefore
dropping support too quickly tends to be annoying. On the other side,
we clearly see the maintenance burden that keeping support for old
toolchains creates.

> I don't have a problem dropping support for ancient toolchains when they
> add too much complexity, but the lack of large file support is mtree is
> probably not really reason enough.

So, what do you suggest ? What I proposed with the hidden option ?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v4] mtree: new package

[Arnout Vandecappelle]

On 28/03/2019 22:53, Thomas Petazzoni wrote:
> On Thu, 28 Mar 2019 21:59:59 +0100
> Peter Korsgaard <peter@korsgaard.com> wrote:
> 
>>  > I'm fine with this as well. It means we would no longer support glibc
>>  > 2.19 anymore.  
>>
>> Didn't you just last month argue against removing support for glibc <
>> 2.19 support (the runc security fix):
> 
> In fact, my last sentence lacked an ending question mark. I wanted to
> ask if we were dropping support entirely for glibc 2.19, or just saying
> "glibc 2.19 is not long important enough to worry too much about it,
> especially when it's just for the few packages that use the <fts.h>
> interface".

 Just to be clear: we don't remove support for these old toolchains. We remove
the "guarantee" (which was anyway never much of a guarantee) that all packages
that you can select in menuconfig will build with it.


> But if your point is to have me say that I don't have a clear and
> well-defined opinion, then yes it's the case. I do see a number of
> companies/customers continue to use old toolchains and therefore
> dropping support too quickly tends to be annoying.

 But these will typically *not* be one of our external toolchains, but rather a
custom external toolchain. So the solution of a hidden option is not much of a
solution.

 Of course, it can be added to the custom external toolchain options as well.
But then, that's one more option that *everybody* has to set for their custom
external toolchains, while it only really applies to the few people that use
such an old toolchain *and* that want to select packages that were not even
available to them at the time that they started using this toolchain. In my
opinion, the "few" here is 0, maybe 1.

 A really fundamental solution could be that at build time we detect in these
packages whether the toolchain has a broken fts.h, and if yes we add
-Ulargefile. But that's also complicated.

 That is actually my main point: I don't think adding such an option is worth
the bother, because it can only really solve the issue by adding a lot of
complexity.

> On the other side,
> we clearly see the maintenance burden that keeping support for old
> toolchains creates.
> 
>> I don't have a problem dropping support for ancient toolchains when they

 The important thing to note is: we drop support for such ancient toolchains
trying to build the mtree package.

 Regards,
 Arnout

>> add too much complexity, but the lack of large file support is mtree is
>> probably not really reason enough.
> 
> So, what do you suggest ? What I proposed with the hidden option ?
> 
> Thomas
> 

[Buildroot] [PATCH v4] mtree: new package

[Thomas Petazzoni]

On Fri, 29 Mar 2019 10:31:22 +0100
Arnout Vandecappelle <arnout@mind.be> wrote:

> >> I don't have a problem dropping support for ancient toolchains when they  
> 
>  The important thing to note is: we drop support for such ancient toolchains
> trying to build the mtree package.

Fair enough. So let's drop all these -U_FILE_OFFSET_BITS from all
packages that use this workaround, and add the corresponding
autobuilder exceptions in utils/genrandconfig.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com