[OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Joshua Watt]

Adds the command line compression tools which are needed by bitbake

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/conf/bitbake.conf | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index f451ba6a47..3801db7d42 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -484,12 +484,12 @@ HOSTTOOLS += " \
     [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \
+    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
-    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
+    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \
     sleep sort split stat strings strip tail tar tee test touch tr true uname \
-    uniq wc wget which xargs \
+    uniq wc wget which xargs zstd \
 "
 
 # Tools needed to run testimage runtime image testing
-- 
2.32.0

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1682 bytes --]

While I welcome the transition to zstd, I think the commit message needs to
be more descriptive. Needed by bitbake where and how? Is there some
patchset upcoming?

Alex

On Wed, 14 Jul 2021 at 15:42, Joshua Watt <JPEWhacker@gmail.com> wrote:

> Adds the command line compression tools which are needed by bitbake
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>  meta/conf/bitbake.conf | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index f451ba6a47..3801db7d42 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -484,12 +484,12 @@ HOSTTOOLS += " \
>      [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp
> comm cp cpio \
>      cpp cut date dd diff diffstat dirname du echo egrep env expand expr
> false \
>      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip
> gzip \
> -    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo
> mknod \
> +    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir
> mkfifo mknod \
>      mktemp mv nm objcopy objdump od patch perl pr printf pwd \
> -    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
> +    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed
> seq sh \
>      sha1sum sha224sum sha256sum sha384sum sha512sum \
>      sleep sort split stat strings strip tail tar tee test touch tr true
> uname \
> -    uniq wc wget which xargs \
> +    uniq wc wget which xargs zstd \
>  "
>
>  # Tools needed to run testimage runtime image testing
> --
> 2.32.0
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 2154 bytes --]

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 2125 bytes --]

On 7/14/21 8:45 AM, Alexander Kanavin wrote:
> While I welcome the transition to zstd, I think the commit message 
> needs to be more descriptive. Needed by bitbake where and how? Is 
> there some patchset upcoming?


Ya, this is necessary for the patch: 
https://lists.openembedded.org/g/bitbake-devel/message/12385, which 
isn't merged yet.


>
> Alex
>
> On Wed, 14 Jul 2021 at 15:42, Joshua Watt <JPEWhacker@gmail.com 
> <mailto:JPEWhacker@gmail.com>> wrote:
>
>     Adds the command line compression tools which are needed by bitbake
>
>     Signed-off-by: Joshua Watt <JPEWhacker@gmail.com
>     <mailto:JPEWhacker@gmail.com>>
>     ---
>      meta/conf/bitbake.conf | 6 +++---
>      1 file changed, 3 insertions(+), 3 deletions(-)
>
>     diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>     index f451ba6a47..3801db7d42 100644
>     --- a/meta/conf/bitbake.conf
>     +++ b/meta/conf/bitbake.conf
>     @@ -484,12 +484,12 @@ HOSTTOOLS += " \
>          [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath
>     cmp comm cp cpio \
>          cpp cut date dd diff diffstat dirname du echo egrep env
>     expand expr false \
>          fgrep file find flock g++ gawk gcc getconf getopt git grep
>     gunzip gzip \
>     -    head hostname iconv id install ld ldd ln ls make md5sum mkdir
>     mkfifo mknod \
>     +    head hostname iconv id install ld ldd ln ls lz4c make md5sum
>     mkdir mkfifo mknod \
>          mktemp mv nm objcopy objdump od patch perl pr printf pwd \
>     -    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed
>     seq sh \
>     +    python3 pzstd ranlib readelf readlink realpath rm rmdir
>     rpcgen sed seq sh \
>          sha1sum sha224sum sha256sum sha384sum sha512sum \
>          sleep sort split stat strings strip tail tar tee test touch
>     tr true uname \
>     -    uniq wc wget which xargs \
>     +    uniq wc wget which xargs zstd \
>      "
>
>      # Tools needed to run testimage runtime image testing
>     -- 
>     2.32.0
>
>
>     
>

[-- Attachment #2: Type: text/html, Size: 3751 bytes --]

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2093 bytes --]

Right, but that still only sets up the infra and tests :) What will use it?

Alex

On Wed, 14 Jul 2021 at 15:51, Joshua Watt <jpewhacker@gmail.com> wrote:

> On 7/14/21 8:45 AM, Alexander Kanavin wrote:
>
> While I welcome the transition to zstd, I think the commit message needs
> to be more descriptive. Needed by bitbake where and how? Is there some
> patchset upcoming?
>
>
> Ya, this is necessary for the patch:
> https://lists.openembedded.org/g/bitbake-devel/message/12385, which isn't
> merged yet.
>
>
>
> Alex
>
> On Wed, 14 Jul 2021 at 15:42, Joshua Watt <JPEWhacker@gmail.com> wrote:
>
>> Adds the command line compression tools which are needed by bitbake
>>
>> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
>> ---
>>  meta/conf/bitbake.conf | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index f451ba6a47..3801db7d42 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -484,12 +484,12 @@ HOSTTOOLS += " \
>>      [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp
>> comm cp cpio \
>>      cpp cut date dd diff diffstat dirname du echo egrep env expand expr
>> false \
>>      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip
>> gzip \
>> -    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo
>> mknod \
>> +    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir
>> mkfifo mknod \
>>      mktemp mv nm objcopy objdump od patch perl pr printf pwd \
>> -    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
>> +    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed
>> seq sh \
>>      sha1sum sha224sum sha256sum sha384sum sha512sum \
>>      sleep sort split stat strings strip tail tar tee test touch tr true
>> uname \
>> -    uniq wc wget which xargs \
>> +    uniq wc wget which xargs zstd \
>>  "
>>
>>  # Tools needed to run testimage runtime image testing
>> --
>> 2.32.0
>>
>>
>> 
>>
>>

[-- Attachment #2: Type: text/html, Size: 3792 bytes --]

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Alexandre Belloni]

Hello,

On 14/07/2021 08:42:48-0500, Joshua Watt wrote:
> Adds the command line compression tools which are needed by bitbake
> 
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>  meta/conf/bitbake.conf | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index f451ba6a47..3801db7d42 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -484,12 +484,12 @@ HOSTTOOLS += " \
>      [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
>      cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
>      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
> -    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \
> +    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
>      mktemp mv nm objcopy objdump od patch perl pr printf pwd \
> -    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
> +    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
>      sha1sum sha224sum sha256sum sha384sum sha512sum \
>      sleep sort split stat strings strip tail tar tee test touch tr true uname \
> -    uniq wc wget which xargs \
> +    uniq wc wget which xargs zstd \

zstd seems to hurt reproducibility:

AssertionError: The following deb packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/elfutils-ptest_0.185-r0_amd64.deb
/home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db-dbg_2.9.4-r0_amd64.deb
/home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db-src_2.9.4-r0_amd64.deb
/home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db_2.9.4-r0_amd64.deb
The following ipk packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/elfutils-ptest_0.185-r0_core2-64.ipk
/home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db-dbg_2.9.4-r0_core2-64.ipk
/home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db-src_2.9.4-r0_core2-64.ipk
/home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db_2.9.4-r0_core2-64.ipk
The following rpm packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/elfutils-ptest-0.185-r0.core2_64.rpm
/home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-2.9.4-r0.core2_64.rpm
/home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-dbg-2.9.4-r0.core2_64.rpm
/home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-src-2.9.4-r0.core2_64.rpm

diffoscope output here:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210814-qe8khapn/packages/diff-html/

I would think this was expected...

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Khem Raj]

On Wed, Jul 14, 2021 at 6:42 AM Joshua Watt <JPEWhacker@gmail.com> wrote:
>
> Adds the command line compression tools which are needed by bitbake
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>  meta/conf/bitbake.conf | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index f451ba6a47..3801db7d42 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -484,12 +484,12 @@ HOSTTOOLS += " \
>      [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
>      cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
>      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
> -    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \
> +    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
>      mktemp mv nm objcopy objdump od patch perl pr printf pwd \
> -    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
> +    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
>      sha1sum sha224sum sha256sum sha384sum sha512sum \
>      sleep sort split stat strings strip tail tar tee test touch tr true uname \
> -    uniq wc wget which xargs \
> +    uniq wc wget which xargs zstd \
>  "
>

this should also be reflecting on the distro prerequisites for
supported distros.

>  # Tools needed to run testimage runtime image testing
> --
> 2.32.0
>
>
> 
>

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Khem Raj]

On Wed, Jul 14, 2021 at 6:42 AM Joshua Watt <JPEWhacker@gmail.com> wrote:
>
> Adds the command line compression tools which are needed by bitbake
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>  meta/conf/bitbake.conf | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index f451ba6a47..3801db7d42 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -484,12 +484,12 @@ HOSTTOOLS += " \
>      [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
>      cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
>      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
> -    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \
> +    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
>      mktemp mv nm objcopy objdump od patch perl pr printf pwd \
> -    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
> +    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
>      sha1sum sha224sum sha256sum sha384sum sha512sum \
>      sleep sort split stat strings strip tail tar tee test touch tr true uname \
> -    uniq wc wget which xargs \
> +    uniq wc wget which xargs zstd \
>  "
>

on ubuntu 18.04 we will need

sudo apt install zstd liblz4-tool

>  # Tools needed to run testimage runtime image testing
> --
> 2.32.0
>
>
> 
>

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Randy MacLeod]

On 2021-08-16 12:14 p.m., Khem Raj wrote:
> On Wed, Jul 14, 2021 at 6:42 AM Joshua Watt <JPEWhacker@gmail.com> wrote:
>>
>> Adds the command line compression tools which are needed by bitbake
>>
>> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
>> ---
>>   meta/conf/bitbake.conf | 6 +++---
>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index f451ba6a47..3801db7d42 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -484,12 +484,12 @@ HOSTTOOLS += " \
>>       [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
>>       cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
>>       fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
>> -    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \
>> +    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
>>       mktemp mv nm objcopy objdump od patch perl pr printf pwd \
>> -    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
>> +    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
>>       sha1sum sha224sum sha256sum sha384sum sha512sum \
>>       sleep sort split stat strings strip tail tar tee test touch tr true uname \
>> -    uniq wc wget which xargs \
>> +    uniq wc wget which xargs zstd \
>>   "
>>
> 
> on ubuntu 18.04 we will need
> 
> sudo apt install zstd liblz4-tool

Michael,

Can you update the docs for this change?

http://docs.yoctoproject.org/ref-manual/system-requirements.html?highlight=bzip2

I don't know what the package names are for non-debian distros.

This must have been done for the YP AB cluster but
maybe that's done manually?

../Randy

> 
>>   # Tools needed to run testimage runtime image testing
>> --
>> 2.32.0
>>
>>
>>
>>
>>
>>
>> 
>>


-- 
# Randy MacLeod
# Wind River Linux

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Konrad Weihmann]

Hi,

I kind of missed out on this change... but I got caught up by reality 
hard this morning :-(.

I currently don't see a reason why zstd is required on the host - only 
systemd is having a hard dependency against zstd atm, while the other 
lately added references are just optional PACKAGECONFIGs (and there are 
plenty of them which have loose ends - in poky and meta-oe)

So the question I want to raise: why a hard dependency? wouldn't have 
been HOSTTOOLS_NONFATAL enough?

I mean this is the second hard cut in the project within just weeks and 
this time it was host related, which is even harder to fix in a timely 
manner in a corporate environment (basically rolling out changes to all 
dev installations isn't something I fancy on a Wednesday morning :-( )

None of the commit messages somehow mentions, why we all of a sudden 
need that on the host, instead of building it from sources (which should 
be possible judging from the latest master sources).

And on that note: that the documentation wasn't updated in the same 
changeset is very unfortunate (not even the crops container where updated)

/rant

Konrad

On 19.08.21 00:19, Randy MacLeod wrote:
> On 2021-08-16 12:14 p.m., Khem Raj wrote:
>> On Wed, Jul 14, 2021 at 6:42 AM Joshua Watt <JPEWhacker@gmail.com> wrote:
>>>
>>> Adds the command line compression tools which are needed by bitbake
>>>
>>> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
>>> ---
>>>   meta/conf/bitbake.conf | 6 +++---
>>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>>> index f451ba6a47..3801db7d42 100644
>>> --- a/meta/conf/bitbake.conf
>>> +++ b/meta/conf/bitbake.conf
>>> @@ -484,12 +484,12 @@ HOSTTOOLS += " \
>>>       [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath 
>>> cmp comm cp cpio \
>>>       cpp cut date dd diff diffstat dirname du echo egrep env expand 
>>> expr false \
>>>       fgrep file find flock g++ gawk gcc getconf getopt git grep 
>>> gunzip gzip \
>>> -    head hostname iconv id install ld ldd ln ls make md5sum mkdir 
>>> mkfifo mknod \
>>> +    head hostname iconv id install ld ldd ln ls lz4c make md5sum 
>>> mkdir mkfifo mknod \
>>>       mktemp mv nm objcopy objdump od patch perl pr printf pwd \
>>> -    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq 
>>> sh \
>>> +    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen 
>>> sed seq sh \
>>>       sha1sum sha224sum sha256sum sha384sum sha512sum \
>>>       sleep sort split stat strings strip tail tar tee test touch tr 
>>> true uname \
>>> -    uniq wc wget which xargs \
>>> +    uniq wc wget which xargs zstd \
>>>   "
>>>
>>
>> on ubuntu 18.04 we will need
>>
>> sudo apt install zstd liblz4-tool
> 
> Michael,
> 
> Can you update the docs for this change?
> 
> http://docs.yoctoproject.org/ref-manual/system-requirements.html?highlight=bzip2 
> 
> 
> I don't know what the package names are for non-debian distros.
> 
> This must have been done for the YP AB cluster but
> maybe that's done manually?
> 
> ../Randy
> 
>>
>>>   # Tools needed to run testimage runtime image testing
>>> -- 
>>> 2.32.0
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
> 
> 
> 
> 
> 
> 

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Leon Woestenberg]

Hi Konrad,

On Thu, Aug 19, 2021 at 4:51 PM Konrad Weihmann <kweihmann@outlook.com> wrote:
>
> I mean this is the second hard cut in the project within just weeks and
> this time it was host related, which is even harder to fix in a timely
> manner in a corporate environment (basically rolling out changes to all
> dev installations isn't something I fancy on a Wednesday morning :-( )
>
The choice to follow bleeding edge, means bleeding at some points in time.

Why do you follow master on all dev installations if you are not able
to match (slightly) disruptive changes?

I would recommend to prevalidate changes on one instance before
spreading across all (inflexible) dev installations anyway.

I did not follow the rationale closely, but I think it was related to
CMake dependencies as well.

Regards,

Leon.

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Richard Purdie]

On Thu, 2021-08-19 at 16:50 +0200, Konrad Weihmann wrote:
> I kind of missed out on this change... but I got caught up by reality 
> hard this morning :-(.
> 
> I currently don't see a reason why zstd is required on the host - only 
> systemd is having a hard dependency against zstd atm, while the other 
> lately added references are just optional PACKAGECONFIGs (and there are 
> plenty of them which have loose ends - in poky and meta-oe)
> 
> So the question I want to raise: why a hard dependency? wouldn't have 
> been HOSTTOOLS_NONFATAL enough?
> 
> I mean this is the second hard cut in the project within just weeks and 
> this time it was host related, which is even harder to fix in a timely 
> manner in a corporate environment (basically rolling out changes to all 
> dev installations isn't something I fancy on a Wednesday morning :-( )
> 
> None of the commit messages somehow mentions, why we all of a sudden 
> need that on the host, instead of building it from sources (which should 
> be possible judging from the latest master sources).
> 
> And on that note: that the documentation wasn't updated in the same 
> changeset is very unfortunate (not even the crops container where updated)

Sorry the change is a bit of a pain. It has been in master-next for a while
and we did have to jump through a number of hoops as a project to enable this
(e.g. adding to buildtools and our autobuilder infrastructure).

The reason we're doing this is that we need to use compression within bitbake
itself for some of our output files. We've not done this in the past as it is
a pain to have the host prerequisites however there are multiple places we can
potentially benefit, such as compressed pkgdata and compressed siginfo files.
It is also possible we may be able to store a compressed parsed datastore.

The siginfo files and datastore could mean we can improve debugging of a 
number of currently hard to solve problems our users face.

Without this we're unlikely to be able to move forward in these areas. If we
do it, we open up new development opportunities and potentially improved
usability. We can't "just build it" as that would mean we can't use it within
bitbake itself.

Sorry this isn't well described anywhere, it is mainly as we can't yet
"guarantee" that this will do these things. We do know it won't happen
unless we try though.

Master *is* a development branch, not a stable release so things can change.

Cheers,

Richard

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Konrad Weihmann]

On 19.08.21 18:06, Richard Purdie wrote:
> On Thu, 2021-08-19 at 16:50 +0200, Konrad Weihmann wrote:
>> I kind of missed out on this change... but I got caught up by reality
>> hard this morning :-(.
>>
>> I currently don't see a reason why zstd is required on the host - only
>> systemd is having a hard dependency against zstd atm, while the other
>> lately added references are just optional PACKAGECONFIGs (and there are
>> plenty of them which have loose ends - in poky and meta-oe)
>>
>> So the question I want to raise: why a hard dependency? wouldn't have
>> been HOSTTOOLS_NONFATAL enough?
>>
>> I mean this is the second hard cut in the project within just weeks and
>> this time it was host related, which is even harder to fix in a timely
>> manner in a corporate environment (basically rolling out changes to all
>> dev installations isn't something I fancy on a Wednesday morning :-( )
>>
>> None of the commit messages somehow mentions, why we all of a sudden
>> need that on the host, instead of building it from sources (which should
>> be possible judging from the latest master sources).
>>
>> And on that note: that the documentation wasn't updated in the same
>> changeset is very unfortunate (not even the crops container where updated)
> 
> Sorry the change is a bit of a pain. It has been in master-next for a while
> and we did have to jump through a number of hoops as a project to enable this
> (e.g. adding to buildtools and our autobuilder infrastructure).
> 
> The reason we're doing this is that we need to use compression within bitbake
> itself for some of our output files. We've not done this in the past as it is
> a pain to have the host prerequisites however there are multiple places we can
> potentially benefit, such as compressed pkgdata and compressed siginfo files.
> It is also possible we may be able to store a compressed parsed datastore.
> 

Thanks Richard, that's explains why it's needed

> The siginfo files and datastore could mean we can improve debugging of a
> number of currently hard to solve problems our users face.
> 
> Without this we're unlikely to be able to move forward in these areas. If we
> do it, we open up new development opportunities and potentially improved
> usability. We can't "just build it" as that would mean we can't use it within
> bitbake itself.
> 
> Sorry this isn't well described anywhere, it is mainly as we can't yet
> "guarantee" that this will do these things. We do know it won't happen
> unless we try though.

Agree - btw maybe a good opportunity to insist on better commit messages 
;-)- just out of interest: is there any guideline, maybe it's worth to 
at least link to a commit message guide at 
https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded 
or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines.
And slightly off-topic, but maybe also worth to do some automation, as 
I've seen multiple patches lately, that don't 100% follow the patch 
guideline in place.

> 
> Master *is* a development branch, not a stable release so things can change.

I've been brave enough to enforce a multi-branch development (just to 
keep the bar low when jumping from release to release), maybe I take 
that as a personal learning :)

> 
> Cheers,
> 
> Richard
> 

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 3834 bytes --]

On Thu, Aug 19, 2021, 11:31 AM Konrad Weihmann <kweihmann@outlook.com>
wrote:

>
>
> On 19.08.21 18:06, Richard Purdie wrote:
> > On Thu, 2021-08-19 at 16:50 +0200, Konrad Weihmann wrote:
> >> I kind of missed out on this change... but I got caught up by reality
> >> hard this morning :-(.
> >>
> >> I currently don't see a reason why zstd is required on the host - only
> >> systemd is having a hard dependency against zstd atm, while the other
> >> lately added references are just optional PACKAGECONFIGs (and there are
> >> plenty of them which have loose ends - in poky and meta-oe)
> >>
> >> So the question I want to raise: why a hard dependency? wouldn't have
> >> been HOSTTOOLS_NONFATAL enough?
> >>
> >> I mean this is the second hard cut in the project within just weeks and
> >> this time it was host related, which is even harder to fix in a timely
> >> manner in a corporate environment (basically rolling out changes to all
> >> dev installations isn't something I fancy on a Wednesday morning :-( )
> >>
> >> None of the commit messages somehow mentions, why we all of a sudden
> >> need that on the host, instead of building it from sources (which should
> >> be possible judging from the latest master sources).
> >>
> >> And on that note: that the documentation wasn't updated in the same
> >> changeset is very unfortunate (not even the crops container where
> updated)
> >
> > Sorry the change is a bit of a pain. It has been in master-next for a
> while
> > and we did have to jump through a number of hoops as a project to enable
> this
> > (e.g. adding to buildtools and our autobuilder infrastructure).
> >
> > The reason we're doing this is that we need to use compression within
> bitbake
> > itself for some of our output files. We've not done this in the past as
> it is
> > a pain to have the host prerequisites however there are multiple places
> we can
> > potentially benefit, such as compressed pkgdata and compressed siginfo
> files.
> > It is also possible we may be able to store a compressed parsed
> datastore.
> >
>
> Thanks Richard, that's explains why it's needed
>
> > The siginfo files and datastore could mean we can improve debugging of a
> > number of currently hard to solve problems our users face.
> >
> > Without this we're unlikely to be able to move forward in these areas.
> If we
> > do it, we open up new development opportunities and potentially improved
> > usability. We can't "just build it" as that would mean we can't use it
> within
> > bitbake itself.
> >
> > Sorry this isn't well described anywhere, it is mainly as we can't yet
> > "guarantee" that this will do these things. We do know it won't happen
> > unless we try though.
>
> Agree - btw maybe a good opportunity to insist on better commit messages
> ;-)- just out of interest: is there any guideline, maybe it's worth to
> at least link to a commit message guide at
> https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded
> or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines.
> And slightly off-topic, but maybe also worth to do some automation, as
> I've seen multiple patches lately, that don't 100% follow the patch
> guideline in place.
>

Sorry, that was my fault. I was preoccupied and didn't fix it when it was
pointed out earlier.

FWIW, I do have some WIP changes that use this functionality that will
hopefully land soon so it will be used :)


> >
> > Master *is* a development branch, not a stable release so things can
> change.
>
> I've been brave enough to enforce a multi-branch development (just to
> keep the bar low when jumping from release to release), maybe I take
> that as a personal learning :)
>
> >
> > Cheers,
> >
> > Richard
> >
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 5102 bytes --]

Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd

[Richard Purdie]

On Thu, 2021-08-19 at 18:31 +0200, Konrad Weihmann wrote:
> 
> On 19.08.21 18:06, Richard Purdie wrote:
> > On Thu, 2021-08-19 at 16:50 +0200, Konrad Weihmann wrote:
> > > I kind of missed out on this change... but I got caught up by reality
> > > hard this morning :-(.
> > > 
> > > I currently don't see a reason why zstd is required on the host - only
> > > systemd is having a hard dependency against zstd atm, while the other
> > > lately added references are just optional PACKAGECONFIGs (and there are
> > > plenty of them which have loose ends - in poky and meta-oe)
> > > 
> > > So the question I want to raise: why a hard dependency? wouldn't have
> > > been HOSTTOOLS_NONFATAL enough?
> > > 
> > > I mean this is the second hard cut in the project within just weeks and
> > > this time it was host related, which is even harder to fix in a timely
> > > manner in a corporate environment (basically rolling out changes to all
> > > dev installations isn't something I fancy on a Wednesday morning :-( )
> > > 
> > > None of the commit messages somehow mentions, why we all of a sudden
> > > need that on the host, instead of building it from sources (which should
> > > be possible judging from the latest master sources).
> > > 
> > > And on that note: that the documentation wasn't updated in the same
> > > changeset is very unfortunate (not even the crops container where updated)
> > 
> > Sorry the change is a bit of a pain. It has been in master-next for a while
> > and we did have to jump through a number of hoops as a project to enable this
> > (e.g. adding to buildtools and our autobuilder infrastructure).
> > 
> > The reason we're doing this is that we need to use compression within bitbake
> > itself for some of our output files. We've not done this in the past as it is
> > a pain to have the host prerequisites however there are multiple places we can
> > potentially benefit, such as compressed pkgdata and compressed siginfo files.
> > It is also possible we may be able to store a compressed parsed datastore.
> > 
> 
> Thanks Richard, that's explains why it's needed
> 
> > The siginfo files and datastore could mean we can improve debugging of a
> > number of currently hard to solve problems our users face.
> > 
> > Without this we're unlikely to be able to move forward in these areas. If we
> > do it, we open up new development opportunities and potentially improved
> > usability. We can't "just build it" as that would mean we can't use it within
> > bitbake itself.
> > 
> > Sorry this isn't well described anywhere, it is mainly as we can't yet
> > "guarantee" that this will do these things. We do know it won't happen
> > unless we try though.
> 
> Agree - btw maybe a good opportunity to insist on better commit messages 
> ;-)- just out of interest: is there any guideline, maybe it's worth to 
> at least link to a commit message guide at 
> https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded 
> or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines.
> And slightly off-topic, but maybe also worth to do some automation, as 
> I've seen multiple patches lately, that don't 100% follow the patch 
> guideline in place.

We had automation to test some of these things. It broke and we simply don't
have the maintainers to look after it and keep it working, it is simply too much
for Michael and I to manage to hold together.

I'm doing the best I can but I can't do this alone. Until we get more people
stepping up to help with things like the automated tests, we'll have to manage
with what I can do.

FWIW I had to spend a ton of time fixing buildtools, getting the workers to have
the right binaries and so on to merge this change. I tried to get the docs
updated but I didn't micromanage it and it didn't happen as yet. I'm aware of
it, I don't know if I'll just have to go and fix that next too.

So yes, I agree nice commit messages would be nice. Please feel free to help
review patches.

Cheers,

Richard