* [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd
@ 2021-07-14 13:42 Joshua Watt
2021-07-14 13:45 ` Alexander Kanavin
` (3 more replies)
0 siblings, 4 replies; 14+ messages in thread
From: Joshua Watt @ 2021-07-14 13:42 UTC (permalink / raw)
To: openembedded-core; +Cc: Joshua Watt
Adds the command line compression tools which are needed by bitbake
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
meta/conf/bitbake.conf | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index f451ba6a47..3801db7d42 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -484,12 +484,12 @@ HOSTTOOLS += " \
[ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
- head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \
+ head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
mktemp mv nm objcopy objdump od patch perl pr printf pwd \
- python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
+ python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
sha1sum sha224sum sha256sum sha384sum sha512sum \
sleep sort split stat strings strip tail tar tee test touch tr true uname \
- uniq wc wget which xargs \
+ uniq wc wget which xargs zstd \
"
# Tools needed to run testimage runtime image testing
--
2.32.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-07-14 13:42 [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd Joshua Watt @ 2021-07-14 13:45 ` Alexander Kanavin 2021-07-14 13:51 ` Joshua Watt 2021-08-14 18:35 ` Alexandre Belloni ` (2 subsequent siblings) 3 siblings, 1 reply; 14+ messages in thread From: Alexander Kanavin @ 2021-07-14 13:45 UTC (permalink / raw) To: Joshua Watt; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 1682 bytes --] While I welcome the transition to zstd, I think the commit message needs to be more descriptive. Needed by bitbake where and how? Is there some patchset upcoming? Alex On Wed, 14 Jul 2021 at 15:42, Joshua Watt <JPEWhacker@gmail.com> wrote: > Adds the command line compression tools which are needed by bitbake > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- > meta/conf/bitbake.conf | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index f451ba6a47..3801db7d42 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -484,12 +484,12 @@ HOSTTOOLS += " \ > [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp > comm cp cpio \ > cpp cut date dd diff diffstat dirname du echo egrep env expand expr > false \ > fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip > gzip \ > - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo > mknod \ > + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir > mkfifo mknod \ > mktemp mv nm objcopy objdump od patch perl pr printf pwd \ > - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ > + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed > seq sh \ > sha1sum sha224sum sha256sum sha384sum sha512sum \ > sleep sort split stat strings strip tail tar tee test touch tr true > uname \ > - uniq wc wget which xargs \ > + uniq wc wget which xargs zstd \ > " > > # Tools needed to run testimage runtime image testing > -- > 2.32.0 > > > > > [-- Attachment #2: Type: text/html, Size: 2154 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-07-14 13:45 ` Alexander Kanavin @ 2021-07-14 13:51 ` Joshua Watt 2021-07-14 13:56 ` Alexander Kanavin 0 siblings, 1 reply; 14+ messages in thread From: Joshua Watt @ 2021-07-14 13:51 UTC (permalink / raw) To: Alexander Kanavin; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 2125 bytes --] On 7/14/21 8:45 AM, Alexander Kanavin wrote: > While I welcome the transition to zstd, I think the commit message > needs to be more descriptive. Needed by bitbake where and how? Is > there some patchset upcoming? Ya, this is necessary for the patch: https://lists.openembedded.org/g/bitbake-devel/message/12385, which isn't merged yet. > > Alex > > On Wed, 14 Jul 2021 at 15:42, Joshua Watt <JPEWhacker@gmail.com > <mailto:JPEWhacker@gmail.com>> wrote: > > Adds the command line compression tools which are needed by bitbake > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com > <mailto:JPEWhacker@gmail.com>> > --- > meta/conf/bitbake.conf | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index f451ba6a47..3801db7d42 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -484,12 +484,12 @@ HOSTTOOLS += " \ > [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath > cmp comm cp cpio \ > cpp cut date dd diff diffstat dirname du echo egrep env > expand expr false \ > fgrep file find flock g++ gawk gcc getconf getopt git grep > gunzip gzip \ > - head hostname iconv id install ld ldd ln ls make md5sum mkdir > mkfifo mknod \ > + head hostname iconv id install ld ldd ln ls lz4c make md5sum > mkdir mkfifo mknod \ > mktemp mv nm objcopy objdump od patch perl pr printf pwd \ > - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed > seq sh \ > + python3 pzstd ranlib readelf readlink realpath rm rmdir > rpcgen sed seq sh \ > sha1sum sha224sum sha256sum sha384sum sha512sum \ > sleep sort split stat strings strip tail tar tee test touch > tr true uname \ > - uniq wc wget which xargs \ > + uniq wc wget which xargs zstd \ > " > > # Tools needed to run testimage runtime image testing > -- > 2.32.0 > > > > [-- Attachment #2: Type: text/html, Size: 3751 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-07-14 13:51 ` Joshua Watt @ 2021-07-14 13:56 ` Alexander Kanavin 0 siblings, 0 replies; 14+ messages in thread From: Alexander Kanavin @ 2021-07-14 13:56 UTC (permalink / raw) To: Joshua Watt; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 2093 bytes --] Right, but that still only sets up the infra and tests :) What will use it? Alex On Wed, 14 Jul 2021 at 15:51, Joshua Watt <jpewhacker@gmail.com> wrote: > On 7/14/21 8:45 AM, Alexander Kanavin wrote: > > While I welcome the transition to zstd, I think the commit message needs > to be more descriptive. Needed by bitbake where and how? Is there some > patchset upcoming? > > > Ya, this is necessary for the patch: > https://lists.openembedded.org/g/bitbake-devel/message/12385, which isn't > merged yet. > > > > Alex > > On Wed, 14 Jul 2021 at 15:42, Joshua Watt <JPEWhacker@gmail.com> wrote: > >> Adds the command line compression tools which are needed by bitbake >> >> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> >> --- >> meta/conf/bitbake.conf | 6 +++--- >> 1 file changed, 3 insertions(+), 3 deletions(-) >> >> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf >> index f451ba6a47..3801db7d42 100644 >> --- a/meta/conf/bitbake.conf >> +++ b/meta/conf/bitbake.conf >> @@ -484,12 +484,12 @@ HOSTTOOLS += " \ >> [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp >> comm cp cpio \ >> cpp cut date dd diff diffstat dirname du echo egrep env expand expr >> false \ >> fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip >> gzip \ >> - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo >> mknod \ >> + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir >> mkfifo mknod \ >> mktemp mv nm objcopy objdump od patch perl pr printf pwd \ >> - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ >> + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed >> seq sh \ >> sha1sum sha224sum sha256sum sha384sum sha512sum \ >> sleep sort split stat strings strip tail tar tee test touch tr true >> uname \ >> - uniq wc wget which xargs \ >> + uniq wc wget which xargs zstd \ >> " >> >> # Tools needed to run testimage runtime image testing >> -- >> 2.32.0 >> >> >> >> >> [-- Attachment #2: Type: text/html, Size: 3792 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-07-14 13:42 [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd Joshua Watt 2021-07-14 13:45 ` Alexander Kanavin @ 2021-08-14 18:35 ` Alexandre Belloni 2021-08-16 15:49 ` Khem Raj 2021-08-16 16:14 ` Khem Raj 3 siblings, 0 replies; 14+ messages in thread From: Alexandre Belloni @ 2021-08-14 18:35 UTC (permalink / raw) To: Joshua Watt, Richard Purdie; +Cc: openembedded-core Hello, On 14/07/2021 08:42:48-0500, Joshua Watt wrote: > Adds the command line compression tools which are needed by bitbake > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- > meta/conf/bitbake.conf | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index f451ba6a47..3801db7d42 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -484,12 +484,12 @@ HOSTTOOLS += " \ > [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ > cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ > fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ > - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \ > + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ > mktemp mv nm objcopy objdump od patch perl pr printf pwd \ > - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ > + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ > sha1sum sha224sum sha256sum sha384sum sha512sum \ > sleep sort split stat strings strip tail tar tee test touch tr true uname \ > - uniq wc wget which xargs \ > + uniq wc wget which xargs zstd \ zstd seems to hurt reproducibility: AssertionError: The following deb packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/elfutils-ptest_0.185-r0_amd64.deb /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db-dbg_2.9.4-r0_amd64.deb /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db-src_2.9.4-r0_amd64.deb /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db_2.9.4-r0_amd64.deb The following ipk packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/elfutils-ptest_0.185-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db-dbg_2.9.4-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db-src_2.9.4-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db_2.9.4-r0_core2-64.ipk The following rpm packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/elfutils-ptest-0.185-r0.core2_64.rpm /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-2.9.4-r0.core2_64.rpm /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-dbg-2.9.4-r0.core2_64.rpm /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-src-2.9.4-r0.core2_64.rpm diffoscope output here: https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210814-qe8khapn/packages/diff-html/ I would think this was expected... -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-07-14 13:42 [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd Joshua Watt 2021-07-14 13:45 ` Alexander Kanavin 2021-08-14 18:35 ` Alexandre Belloni @ 2021-08-16 15:49 ` Khem Raj 2021-08-16 16:14 ` Khem Raj 3 siblings, 0 replies; 14+ messages in thread From: Khem Raj @ 2021-08-16 15:49 UTC (permalink / raw) To: Joshua Watt; +Cc: Patches and discussions about the oe-core layer On Wed, Jul 14, 2021 at 6:42 AM Joshua Watt <JPEWhacker@gmail.com> wrote: > > Adds the command line compression tools which are needed by bitbake > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- > meta/conf/bitbake.conf | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index f451ba6a47..3801db7d42 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -484,12 +484,12 @@ HOSTTOOLS += " \ > [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ > cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ > fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ > - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \ > + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ > mktemp mv nm objcopy objdump od patch perl pr printf pwd \ > - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ > + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ > sha1sum sha224sum sha256sum sha384sum sha512sum \ > sleep sort split stat strings strip tail tar tee test touch tr true uname \ > - uniq wc wget which xargs \ > + uniq wc wget which xargs zstd \ > " > this should also be reflecting on the distro prerequisites for supported distros. > # Tools needed to run testimage runtime image testing > -- > 2.32.0 > > > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-07-14 13:42 [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd Joshua Watt ` (2 preceding siblings ...) 2021-08-16 15:49 ` Khem Raj @ 2021-08-16 16:14 ` Khem Raj 2021-08-18 22:19 ` Randy MacLeod 3 siblings, 1 reply; 14+ messages in thread From: Khem Raj @ 2021-08-16 16:14 UTC (permalink / raw) To: Joshua Watt; +Cc: Patches and discussions about the oe-core layer On Wed, Jul 14, 2021 at 6:42 AM Joshua Watt <JPEWhacker@gmail.com> wrote: > > Adds the command line compression tools which are needed by bitbake > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- > meta/conf/bitbake.conf | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index f451ba6a47..3801db7d42 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -484,12 +484,12 @@ HOSTTOOLS += " \ > [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ > cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ > fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ > - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \ > + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ > mktemp mv nm objcopy objdump od patch perl pr printf pwd \ > - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ > + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ > sha1sum sha224sum sha256sum sha384sum sha512sum \ > sleep sort split stat strings strip tail tar tee test touch tr true uname \ > - uniq wc wget which xargs \ > + uniq wc wget which xargs zstd \ > " > on ubuntu 18.04 we will need sudo apt install zstd liblz4-tool > # Tools needed to run testimage runtime image testing > -- > 2.32.0 > > > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-08-16 16:14 ` Khem Raj @ 2021-08-18 22:19 ` Randy MacLeod 2021-08-19 14:50 ` Konrad Weihmann 0 siblings, 1 reply; 14+ messages in thread From: Randy MacLeod @ 2021-08-18 22:19 UTC (permalink / raw) To: Khem Raj, Joshua Watt, Michael Opdenacker Cc: Patches and discussions about the oe-core layer On 2021-08-16 12:14 p.m., Khem Raj wrote: > On Wed, Jul 14, 2021 at 6:42 AM Joshua Watt <JPEWhacker@gmail.com> wrote: >> >> Adds the command line compression tools which are needed by bitbake >> >> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> >> --- >> meta/conf/bitbake.conf | 6 +++--- >> 1 file changed, 3 insertions(+), 3 deletions(-) >> >> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf >> index f451ba6a47..3801db7d42 100644 >> --- a/meta/conf/bitbake.conf >> +++ b/meta/conf/bitbake.conf >> @@ -484,12 +484,12 @@ HOSTTOOLS += " \ >> [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ >> cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ >> fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ >> - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \ >> + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ >> mktemp mv nm objcopy objdump od patch perl pr printf pwd \ >> - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ >> + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ >> sha1sum sha224sum sha256sum sha384sum sha512sum \ >> sleep sort split stat strings strip tail tar tee test touch tr true uname \ >> - uniq wc wget which xargs \ >> + uniq wc wget which xargs zstd \ >> " >> > > on ubuntu 18.04 we will need > > sudo apt install zstd liblz4-tool Michael, Can you update the docs for this change? http://docs.yoctoproject.org/ref-manual/system-requirements.html?highlight=bzip2 I don't know what the package names are for non-debian distros. This must have been done for the YP AB cluster but maybe that's done manually? ../Randy > >> # Tools needed to run testimage runtime image testing >> -- >> 2.32.0 >> >> >> >> >> >> >> >> -- # Randy MacLeod # Wind River Linux ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-08-18 22:19 ` Randy MacLeod @ 2021-08-19 14:50 ` Konrad Weihmann 2021-08-19 15:07 ` Leon Woestenberg 2021-08-19 16:06 ` Richard Purdie 0 siblings, 2 replies; 14+ messages in thread From: Konrad Weihmann @ 2021-08-19 14:50 UTC (permalink / raw) To: openembedded-core Hi, I kind of missed out on this change... but I got caught up by reality hard this morning :-(. I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe) So the question I want to raise: why a hard dependency? wouldn't have been HOSTTOOLS_NONFATAL enough? I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( ) None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources). And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where updated) /rant Konrad On 19.08.21 00:19, Randy MacLeod wrote: > On 2021-08-16 12:14 p.m., Khem Raj wrote: >> On Wed, Jul 14, 2021 at 6:42 AM Joshua Watt <JPEWhacker@gmail.com> wrote: >>> >>> Adds the command line compression tools which are needed by bitbake >>> >>> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> >>> --- >>> meta/conf/bitbake.conf | 6 +++--- >>> 1 file changed, 3 insertions(+), 3 deletions(-) >>> >>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf >>> index f451ba6a47..3801db7d42 100644 >>> --- a/meta/conf/bitbake.conf >>> +++ b/meta/conf/bitbake.conf >>> @@ -484,12 +484,12 @@ HOSTTOOLS += " \ >>> [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath >>> cmp comm cp cpio \ >>> cpp cut date dd diff diffstat dirname du echo egrep env expand >>> expr false \ >>> fgrep file find flock g++ gawk gcc getconf getopt git grep >>> gunzip gzip \ >>> - head hostname iconv id install ld ldd ln ls make md5sum mkdir >>> mkfifo mknod \ >>> + head hostname iconv id install ld ldd ln ls lz4c make md5sum >>> mkdir mkfifo mknod \ >>> mktemp mv nm objcopy objdump od patch perl pr printf pwd \ >>> - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq >>> sh \ >>> + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen >>> sed seq sh \ >>> sha1sum sha224sum sha256sum sha384sum sha512sum \ >>> sleep sort split stat strings strip tail tar tee test touch tr >>> true uname \ >>> - uniq wc wget which xargs \ >>> + uniq wc wget which xargs zstd \ >>> " >>> >> >> on ubuntu 18.04 we will need >> >> sudo apt install zstd liblz4-tool > > Michael, > > Can you update the docs for this change? > > http://docs.yoctoproject.org/ref-manual/system-requirements.html?highlight=bzip2 > > > I don't know what the package names are for non-debian distros. > > This must have been done for the YP AB cluster but > maybe that's done manually? > > ../Randy > >> >>> # Tools needed to run testimage runtime image testing >>> -- >>> 2.32.0 >>> >>> >>> >>> >>> >>> >>> >>> > > > > > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-08-19 14:50 ` Konrad Weihmann @ 2021-08-19 15:07 ` Leon Woestenberg 2021-08-19 16:06 ` Richard Purdie 1 sibling, 0 replies; 14+ messages in thread From: Leon Woestenberg @ 2021-08-19 15:07 UTC (permalink / raw) To: Konrad Weihmann; +Cc: OE Core mailing list Hi Konrad, On Thu, Aug 19, 2021 at 4:51 PM Konrad Weihmann <kweihmann@outlook.com> wrote: > > I mean this is the second hard cut in the project within just weeks and > this time it was host related, which is even harder to fix in a timely > manner in a corporate environment (basically rolling out changes to all > dev installations isn't something I fancy on a Wednesday morning :-( ) > The choice to follow bleeding edge, means bleeding at some points in time. Why do you follow master on all dev installations if you are not able to match (slightly) disruptive changes? I would recommend to prevalidate changes on one instance before spreading across all (inflexible) dev installations anyway. I did not follow the rationale closely, but I think it was related to CMake dependencies as well. Regards, Leon. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-08-19 14:50 ` Konrad Weihmann 2021-08-19 15:07 ` Leon Woestenberg @ 2021-08-19 16:06 ` Richard Purdie 2021-08-19 16:31 ` Konrad Weihmann 1 sibling, 1 reply; 14+ messages in thread From: Richard Purdie @ 2021-08-19 16:06 UTC (permalink / raw) To: Konrad Weihmann, openembedded-core On Thu, 2021-08-19 at 16:50 +0200, Konrad Weihmann wrote: > I kind of missed out on this change... but I got caught up by reality > hard this morning :-(. > > I currently don't see a reason why zstd is required on the host - only > systemd is having a hard dependency against zstd atm, while the other > lately added references are just optional PACKAGECONFIGs (and there are > plenty of them which have loose ends - in poky and meta-oe) > > So the question I want to raise: why a hard dependency? wouldn't have > been HOSTTOOLS_NONFATAL enough? > > I mean this is the second hard cut in the project within just weeks and > this time it was host related, which is even harder to fix in a timely > manner in a corporate environment (basically rolling out changes to all > dev installations isn't something I fancy on a Wednesday morning :-( ) > > None of the commit messages somehow mentions, why we all of a sudden > need that on the host, instead of building it from sources (which should > be possible judging from the latest master sources). > > And on that note: that the documentation wasn't updated in the same > changeset is very unfortunate (not even the crops container where updated) Sorry the change is a bit of a pain. It has been in master-next for a while and we did have to jump through a number of hoops as a project to enable this (e.g. adding to buildtools and our autobuilder infrastructure). The reason we're doing this is that we need to use compression within bitbake itself for some of our output files. We've not done this in the past as it is a pain to have the host prerequisites however there are multiple places we can potentially benefit, such as compressed pkgdata and compressed siginfo files. It is also possible we may be able to store a compressed parsed datastore. The siginfo files and datastore could mean we can improve debugging of a number of currently hard to solve problems our users face. Without this we're unlikely to be able to move forward in these areas. If we do it, we open up new development opportunities and potentially improved usability. We can't "just build it" as that would mean we can't use it within bitbake itself. Sorry this isn't well described anywhere, it is mainly as we can't yet "guarantee" that this will do these things. We do know it won't happen unless we try though. Master *is* a development branch, not a stable release so things can change. Cheers, Richard ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-08-19 16:06 ` Richard Purdie @ 2021-08-19 16:31 ` Konrad Weihmann 2021-08-19 16:34 ` Joshua Watt 2021-08-19 16:40 ` Richard Purdie 0 siblings, 2 replies; 14+ messages in thread From: Konrad Weihmann @ 2021-08-19 16:31 UTC (permalink / raw) To: Richard Purdie, openembedded-core On 19.08.21 18:06, Richard Purdie wrote: > On Thu, 2021-08-19 at 16:50 +0200, Konrad Weihmann wrote: >> I kind of missed out on this change... but I got caught up by reality >> hard this morning :-(. >> >> I currently don't see a reason why zstd is required on the host - only >> systemd is having a hard dependency against zstd atm, while the other >> lately added references are just optional PACKAGECONFIGs (and there are >> plenty of them which have loose ends - in poky and meta-oe) >> >> So the question I want to raise: why a hard dependency? wouldn't have >> been HOSTTOOLS_NONFATAL enough? >> >> I mean this is the second hard cut in the project within just weeks and >> this time it was host related, which is even harder to fix in a timely >> manner in a corporate environment (basically rolling out changes to all >> dev installations isn't something I fancy on a Wednesday morning :-( ) >> >> None of the commit messages somehow mentions, why we all of a sudden >> need that on the host, instead of building it from sources (which should >> be possible judging from the latest master sources). >> >> And on that note: that the documentation wasn't updated in the same >> changeset is very unfortunate (not even the crops container where updated) > > Sorry the change is a bit of a pain. It has been in master-next for a while > and we did have to jump through a number of hoops as a project to enable this > (e.g. adding to buildtools and our autobuilder infrastructure). > > The reason we're doing this is that we need to use compression within bitbake > itself for some of our output files. We've not done this in the past as it is > a pain to have the host prerequisites however there are multiple places we can > potentially benefit, such as compressed pkgdata and compressed siginfo files. > It is also possible we may be able to store a compressed parsed datastore. > Thanks Richard, that's explains why it's needed > The siginfo files and datastore could mean we can improve debugging of a > number of currently hard to solve problems our users face. > > Without this we're unlikely to be able to move forward in these areas. If we > do it, we open up new development opportunities and potentially improved > usability. We can't "just build it" as that would mean we can't use it within > bitbake itself. > > Sorry this isn't well described anywhere, it is mainly as we can't yet > "guarantee" that this will do these things. We do know it won't happen > unless we try though. Agree - btw maybe a good opportunity to insist on better commit messages ;-)- just out of interest: is there any guideline, maybe it's worth to at least link to a commit message guide at https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines. And slightly off-topic, but maybe also worth to do some automation, as I've seen multiple patches lately, that don't 100% follow the patch guideline in place. > > Master *is* a development branch, not a stable release so things can change. I've been brave enough to enforce a multi-branch development (just to keep the bar low when jumping from release to release), maybe I take that as a personal learning :) > > Cheers, > > Richard > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-08-19 16:31 ` Konrad Weihmann @ 2021-08-19 16:34 ` Joshua Watt 2021-08-19 16:40 ` Richard Purdie 1 sibling, 0 replies; 14+ messages in thread From: Joshua Watt @ 2021-08-19 16:34 UTC (permalink / raw) To: Konrad Weihmann; +Cc: Richard Purdie, OE-core [-- Attachment #1: Type: text/plain, Size: 3834 bytes --] On Thu, Aug 19, 2021, 11:31 AM Konrad Weihmann <kweihmann@outlook.com> wrote: > > > On 19.08.21 18:06, Richard Purdie wrote: > > On Thu, 2021-08-19 at 16:50 +0200, Konrad Weihmann wrote: > >> I kind of missed out on this change... but I got caught up by reality > >> hard this morning :-(. > >> > >> I currently don't see a reason why zstd is required on the host - only > >> systemd is having a hard dependency against zstd atm, while the other > >> lately added references are just optional PACKAGECONFIGs (and there are > >> plenty of them which have loose ends - in poky and meta-oe) > >> > >> So the question I want to raise: why a hard dependency? wouldn't have > >> been HOSTTOOLS_NONFATAL enough? > >> > >> I mean this is the second hard cut in the project within just weeks and > >> this time it was host related, which is even harder to fix in a timely > >> manner in a corporate environment (basically rolling out changes to all > >> dev installations isn't something I fancy on a Wednesday morning :-( ) > >> > >> None of the commit messages somehow mentions, why we all of a sudden > >> need that on the host, instead of building it from sources (which should > >> be possible judging from the latest master sources). > >> > >> And on that note: that the documentation wasn't updated in the same > >> changeset is very unfortunate (not even the crops container where > updated) > > > > Sorry the change is a bit of a pain. It has been in master-next for a > while > > and we did have to jump through a number of hoops as a project to enable > this > > (e.g. adding to buildtools and our autobuilder infrastructure). > > > > The reason we're doing this is that we need to use compression within > bitbake > > itself for some of our output files. We've not done this in the past as > it is > > a pain to have the host prerequisites however there are multiple places > we can > > potentially benefit, such as compressed pkgdata and compressed siginfo > files. > > It is also possible we may be able to store a compressed parsed > datastore. > > > > Thanks Richard, that's explains why it's needed > > > The siginfo files and datastore could mean we can improve debugging of a > > number of currently hard to solve problems our users face. > > > > Without this we're unlikely to be able to move forward in these areas. > If we > > do it, we open up new development opportunities and potentially improved > > usability. We can't "just build it" as that would mean we can't use it > within > > bitbake itself. > > > > Sorry this isn't well described anywhere, it is mainly as we can't yet > > "guarantee" that this will do these things. We do know it won't happen > > unless we try though. > > Agree - btw maybe a good opportunity to insist on better commit messages > ;-)- just out of interest: is there any guideline, maybe it's worth to > at least link to a commit message guide at > https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded > or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines. > And slightly off-topic, but maybe also worth to do some automation, as > I've seen multiple patches lately, that don't 100% follow the patch > guideline in place. > Sorry, that was my fault. I was preoccupied and didn't fix it when it was pointed out earlier. FWIW, I do have some WIP changes that use this functionality that will hopefully land soon so it will be used :) > > > > Master *is* a development branch, not a stable release so things can > change. > > I've been brave enough to enforce a multi-branch development (just to > keep the bar low when jumping from release to release), maybe I take > that as a personal learning :) > > > > > Cheers, > > > > Richard > > > > > > [-- Attachment #2: Type: text/html, Size: 5102 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd 2021-08-19 16:31 ` Konrad Weihmann 2021-08-19 16:34 ` Joshua Watt @ 2021-08-19 16:40 ` Richard Purdie 1 sibling, 0 replies; 14+ messages in thread From: Richard Purdie @ 2021-08-19 16:40 UTC (permalink / raw) To: Konrad Weihmann, openembedded-core On Thu, 2021-08-19 at 18:31 +0200, Konrad Weihmann wrote: > > On 19.08.21 18:06, Richard Purdie wrote: > > On Thu, 2021-08-19 at 16:50 +0200, Konrad Weihmann wrote: > > > I kind of missed out on this change... but I got caught up by reality > > > hard this morning :-(. > > > > > > I currently don't see a reason why zstd is required on the host - only > > > systemd is having a hard dependency against zstd atm, while the other > > > lately added references are just optional PACKAGECONFIGs (and there are > > > plenty of them which have loose ends - in poky and meta-oe) > > > > > > So the question I want to raise: why a hard dependency? wouldn't have > > > been HOSTTOOLS_NONFATAL enough? > > > > > > I mean this is the second hard cut in the project within just weeks and > > > this time it was host related, which is even harder to fix in a timely > > > manner in a corporate environment (basically rolling out changes to all > > > dev installations isn't something I fancy on a Wednesday morning :-( ) > > > > > > None of the commit messages somehow mentions, why we all of a sudden > > > need that on the host, instead of building it from sources (which should > > > be possible judging from the latest master sources). > > > > > > And on that note: that the documentation wasn't updated in the same > > > changeset is very unfortunate (not even the crops container where updated) > > > > Sorry the change is a bit of a pain. It has been in master-next for a while > > and we did have to jump through a number of hoops as a project to enable this > > (e.g. adding to buildtools and our autobuilder infrastructure). > > > > The reason we're doing this is that we need to use compression within bitbake > > itself for some of our output files. We've not done this in the past as it is > > a pain to have the host prerequisites however there are multiple places we can > > potentially benefit, such as compressed pkgdata and compressed siginfo files. > > It is also possible we may be able to store a compressed parsed datastore. > > > > Thanks Richard, that's explains why it's needed > > > The siginfo files and datastore could mean we can improve debugging of a > > number of currently hard to solve problems our users face. > > > > Without this we're unlikely to be able to move forward in these areas. If we > > do it, we open up new development opportunities and potentially improved > > usability. We can't "just build it" as that would mean we can't use it within > > bitbake itself. > > > > Sorry this isn't well described anywhere, it is mainly as we can't yet > > "guarantee" that this will do these things. We do know it won't happen > > unless we try though. > > Agree - btw maybe a good opportunity to insist on better commit messages > ;-)- just out of interest: is there any guideline, maybe it's worth to > at least link to a commit message guide at > https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded > or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines. > And slightly off-topic, but maybe also worth to do some automation, as > I've seen multiple patches lately, that don't 100% follow the patch > guideline in place. We had automation to test some of these things. It broke and we simply don't have the maintainers to look after it and keep it working, it is simply too much for Michael and I to manage to hold together. I'm doing the best I can but I can't do this alone. Until we get more people stepping up to help with things like the automated tests, we'll have to manage with what I can do. FWIW I had to spend a ton of time fixing buildtools, getting the workers to have the right binaries and so on to merge this change. I tried to get the docs updated but I didn't micromanage it and it didn't happen as yet. I'm aware of it, I don't know if I'll just have to go and fix that next too. So yes, I agree nice commit messages would be nice. Please feel free to help review patches. Cheers, Richard ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-08-19 16:40 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-07-14 13:42 [OE-core][PATCH] bitbake.conf: Add lz4c, pzstd and zstd Joshua Watt 2021-07-14 13:45 ` Alexander Kanavin 2021-07-14 13:51 ` Joshua Watt 2021-07-14 13:56 ` Alexander Kanavin 2021-08-14 18:35 ` Alexandre Belloni 2021-08-16 15:49 ` Khem Raj 2021-08-16 16:14 ` Khem Raj 2021-08-18 22:19 ` Randy MacLeod 2021-08-19 14:50 ` Konrad Weihmann 2021-08-19 15:07 ` Leon Woestenberg 2021-08-19 16:06 ` Richard Purdie 2021-08-19 16:31 ` Konrad Weihmann 2021-08-19 16:34 ` Joshua Watt 2021-08-19 16:40 ` Richard Purdie
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.