[PATCH v2 1/2] rpm2cpio.sh: update to the last 4.x version

[PATCH v2 1/2] rpm2cpio.sh: update to the last 4.x version

[aplanas]

From: Alberto Planas <aplanas@suse.com>

openSUSE RPMs are compressing the RPM payload using zstd, that
correspond to the magic ID 0x28, 0xb5, 0x2f.

This patch update the script to the last version from the rpm project,
and add support to this compression format, and extract the cpio payload
using the "unzstd" binary.

Signed-off-by: Alberto Planas <aplanas@suse.com>
---
 scripts/rpm2cpio.sh | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
 mode change 100755 => 100644 scripts/rpm2cpio.sh

diff --git a/scripts/rpm2cpio.sh b/scripts/rpm2cpio.sh
old mode 100755
new mode 100644
index 2034373fe4..8199b43784
--- a/scripts/rpm2cpio.sh
+++ b/scripts/rpm2cpio.sh
@@ -7,7 +7,7 @@ fatal() {
 }
 
 pkg="$1"
-[ -n "$pkg" -a -e "$pkg" ] ||
+[ -n "$pkg" ] && [ -e "$pkg" ] ||
 	fatal "No package supplied"
 
 _dd() {
@@ -16,14 +16,23 @@ _dd() {
 }
 
 calcsize() {
+
+	case "$(_dd $1 bs=4 count=1 | tr -d '\0')" in
+		"$(printf '\216\255\350')"*) ;; # '\x8e\xad\xe8'
+		*) fatal "File doesn't look like rpm: $pkg" ;;
+	esac
+
 	offset=$(($1 + 8))
 
 	local i b b0 b1 b2 b3 b4 b5 b6 b7
 
 	i=0
 	while [ $i -lt 8 ]; do
- 		b=$(_dd $(($offset + $i)) bs=1 count=1; echo X)
- 		b=${b%X}
+		# add . to not loose \n
+		# strip \0 as it gets dropped with warning otherwise
+		b="$(_dd $(($offset + $i)) bs=1 count=1 | tr -d '\0' ; echo .)"
+		b=${b%.}    # strip . again
+
 		[ -z "$b" ] &&
 			b="0" ||
 			b="$(exec printf '%u\n' "'$b")"
@@ -35,7 +44,7 @@ calcsize() {
 	offset=$(($offset + $rsize))
 }
 
-case "$(_dd 0 bs=8 count=1)" in
+case "$(_dd 0 bs=4 count=1 | tr -d '\0')" in
 	"$(printf '\355\253\356\333')"*) ;; # '\xed\xab\xee\xdb'
 	*) fatal "File doesn't look like rpm: $pkg" ;;
 esac
@@ -46,10 +55,11 @@ sigsize=$rsize
 calcsize $(($offset + (8 - ($sigsize % 8)) % 8))
 hdrsize=$rsize
 
-case "$(_dd $offset bs=3 count=1)" in
-	"$(printf '\102\132')"*) _dd $offset | bzip2 -d ;; # '\x42\x5a'
-	"$(printf '\037\213')"*) _dd $offset | gunzip  ;; # '\x1f\x8b'
-	"$(printf '\375\067')"*) _dd $offset | xzcat   ;; # '\xfd\x37'
-	"$(printf '\135\000')"*) _dd $offset | unlzma  ;; # '\x5d\x00'
-	*) fatal "Unrecognized rpm file: $pkg" ;;
+case "$(_dd $offset bs=2 count=1 | tr -d '\0')" in
+	"$(printf '\102\132')") _dd $offset | bunzip2 ;; # '\x42\x5a'
+	"$(printf '\037\213')") _dd $offset | gunzip  ;; # '\x1f\x8b'
+	"$(printf '\375\067')") _dd $offset | xzcat   ;; # '\xfd\x37'
+	"$(printf '\135')") _dd $offset | unlzma      ;; # '\x5d\x00'
+	"$(printf '\050\265')") _dd $offset | unzstd  ;; # '\x28\xb5'
+	*) fatal "Unrecognized payload compression format in rpm file: $pkg" ;;
 esac
-- 
2.41.0

[PATCH v2 2/2] bitbake.conf: add unzstd in HOSTTOOLS

[aplanas]

From: Alberto Planas <aplanas@suse.com>

rpm2cpio.sh can make calls to unzstd to uncompress the RPM payload that
conform the cpio file.

zstd is already part of HOSTTOOLS, as a link to the system installed
zstd.

This patch add unzstd in HOSTOOLS list as a non-optional binary, so is
available to rpm2cpio.sh when it is required.

Signed-off-by: Alberto Planas <aplanas@suse.com>
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index e28df7478f..702881144e 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -524,7 +524,7 @@ HOSTTOOLS += " \
     python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \
     sleep sort split stat strings strip tail tar tee test touch tr true uname \
-    uniq wc wget which xargs zstd \
+    uniq unzstd wc wget which xargs zstd \
 "
 
 # Tools needed to run testimage runtime image testing
-- 
2.41.0

Re: [OE-core] [PATCH v2 2/2] bitbake.conf: add unzstd in HOSTTOOLS

[Richard Purdie]

On Thu, 2023-06-15 at 12:54 +0200, Alberto Planas via
lists.openembedded.org wrote:
> From: Alberto Planas <aplanas@suse.com>
> 
> rpm2cpio.sh can make calls to unzstd to uncompress the RPM payload that
> conform the cpio file.
> 
> zstd is already part of HOSTTOOLS, as a link to the system installed
> zstd.
> 
> This patch add unzstd in HOSTOOLS list as a non-optional binary, so is
> available to rpm2cpio.sh when it is required.
> 
> Signed-off-by: Alberto Planas <aplanas@suse.com>
> ---
>  meta/conf/bitbake.conf | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index e28df7478f..702881144e 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -524,7 +524,7 @@ HOSTTOOLS += " \
>      python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
>      sha1sum sha224sum sha256sum sha384sum sha512sum \
>      sleep sort split stat strings strip tail tar tee test touch tr true uname \
> -    uniq wc wget which xargs zstd \
> +    uniq unzstd wc wget which xargs zstd \
>  "
>  
>  # Tools needed to run testimage runtime image testing

Couldn't we just use zstd -d in the script? They're always the same
binary anyway as far as I understand it...

Cheers,

Richard

Re: [OE-core] [PATCH v2 2/2] bitbake.conf: add unzstd in HOSTTOOLS

[Alberto Planas Dominguez]

> From: Richard Purdie <richard.purdie@linuxfoundation.org>

> Couldn't we just use zstd -d in the script? They're always the same
> binary anyway as far as I understand it...

Yes, this should work. I decided to include the second patch to avoid changing the rpm2cpio.sh code, that is a verbatim copy of the upstream script (minus the added comment from the old version, that reference where to find the script)

If this is the preferred way I can do a v3 that drop this patch and changes rpm2cpio.sh.

Re: [OE-core] [PATCH v2 2/2] bitbake.conf: add unzstd in HOSTTOOLS

[Richard Purdie]

On Thu, 2023-06-15 at 12:58 +0000, Alberto Planas Dominguez wrote:
> > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> > Couldn't we just use zstd -d in the script? They're always the same
> > binary anyway as far as I understand it...
> 
> Yes, this should work. I decided to include the second patch to avoid
> changing the rpm2cpio.sh code, that is a verbatim copy of the
> upstream script (minus the added comment from the old version, that
> reference where to find the script)
> 
> If this is the preferred way I can do a v3 that drop this patch and
> changes rpm2cpio.sh.

I guess we can stay in sync then, I was just hoping not to add more to
HOSTTOOLS. Thanks for the context.

Cheers,

Richard

Re: [OE-core] [PATCH v2 2/2] bitbake.conf: add unzstd in HOSTTOOLS

[Khem Raj]

I am seeing some race condition see

https://errors.yoctoproject.org/Errors/Details/708923/


On Thu, Jun 15, 2023 at 6:34 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2023-06-15 at 12:58 +0000, Alberto Planas Dominguez wrote:
> > > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> >
> > > Couldn't we just use zstd -d in the script? They're always the same
> > > binary anyway as far as I understand it...
> >
> > Yes, this should work. I decided to include the second patch to avoid
> > changing the rpm2cpio.sh code, that is a verbatim copy of the
> > upstream script (minus the added comment from the old version, that
> > reference where to find the script)
> >
> > If this is the preferred way I can do a v3 that drop this patch and
> > changes rpm2cpio.sh.
>
> I guess we can stay in sync then, I was just hoping not to add more to
> HOSTTOOLS. Thanks for the context.
>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#182861): https://lists.openembedded.org/g/openembedded-core/message/182861
> Mute This Topic: https://lists.openembedded.org/mt/99546093/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2 1/2] rpm2cpio.sh: update to the last 4.x version

[Khem Raj]

I am seeing some race condition see

https://errors.yoctoproject.org/Errors/Details/708923/

On Thu, Jun 15, 2023 at 3:55 AM Alberto Planas via
lists.openembedded.org <aplanas=suse.com@lists.openembedded.org>
wrote:
>
> From: Alberto Planas <aplanas@suse.com>
>
> openSUSE RPMs are compressing the RPM payload using zstd, that
> correspond to the magic ID 0x28, 0xb5, 0x2f.
>
> This patch update the script to the last version from the rpm project,
> and add support to this compression format, and extract the cpio payload
> using the "unzstd" binary.
>
> Signed-off-by: Alberto Planas <aplanas@suse.com>
> ---
>  scripts/rpm2cpio.sh | 30 ++++++++++++++++++++----------
>  1 file changed, 20 insertions(+), 10 deletions(-)
>  mode change 100755 => 100644 scripts/rpm2cpio.sh
>
> diff --git a/scripts/rpm2cpio.sh b/scripts/rpm2cpio.sh
> old mode 100755
> new mode 100644
> index 2034373fe4..8199b43784
> --- a/scripts/rpm2cpio.sh
> +++ b/scripts/rpm2cpio.sh
> @@ -7,7 +7,7 @@ fatal() {
>  }
>
>  pkg="$1"
> -[ -n "$pkg" -a -e "$pkg" ] ||
> +[ -n "$pkg" ] && [ -e "$pkg" ] ||
>         fatal "No package supplied"
>
>  _dd() {
> @@ -16,14 +16,23 @@ _dd() {
>  }
>
>  calcsize() {
> +
> +       case "$(_dd $1 bs=4 count=1 | tr -d '\0')" in
> +               "$(printf '\216\255\350')"*) ;; # '\x8e\xad\xe8'
> +               *) fatal "File doesn't look like rpm: $pkg" ;;
> +       esac
> +
>         offset=$(($1 + 8))
>
>         local i b b0 b1 b2 b3 b4 b5 b6 b7
>
>         i=0
>         while [ $i -lt 8 ]; do
> -               b=$(_dd $(($offset + $i)) bs=1 count=1; echo X)
> -               b=${b%X}
> +               # add . to not loose \n
> +               # strip \0 as it gets dropped with warning otherwise
> +               b="$(_dd $(($offset + $i)) bs=1 count=1 | tr -d '\0' ; echo .)"
> +               b=${b%.}    # strip . again
> +
>                 [ -z "$b" ] &&
>                         b="0" ||
>                         b="$(exec printf '%u\n' "'$b")"
> @@ -35,7 +44,7 @@ calcsize() {
>         offset=$(($offset + $rsize))
>  }
>
> -case "$(_dd 0 bs=8 count=1)" in
> +case "$(_dd 0 bs=4 count=1 | tr -d '\0')" in
>         "$(printf '\355\253\356\333')"*) ;; # '\xed\xab\xee\xdb'
>         *) fatal "File doesn't look like rpm: $pkg" ;;
>  esac
> @@ -46,10 +55,11 @@ sigsize=$rsize
>  calcsize $(($offset + (8 - ($sigsize % 8)) % 8))
>  hdrsize=$rsize
>
> -case "$(_dd $offset bs=3 count=1)" in
> -       "$(printf '\102\132')"*) _dd $offset | bzip2 -d ;; # '\x42\x5a'
> -       "$(printf '\037\213')"*) _dd $offset | gunzip  ;; # '\x1f\x8b'
> -       "$(printf '\375\067')"*) _dd $offset | xzcat   ;; # '\xfd\x37'
> -       "$(printf '\135\000')"*) _dd $offset | unlzma  ;; # '\x5d\x00'
> -       *) fatal "Unrecognized rpm file: $pkg" ;;
> +case "$(_dd $offset bs=2 count=1 | tr -d '\0')" in
> +       "$(printf '\102\132')") _dd $offset | bunzip2 ;; # '\x42\x5a'
> +       "$(printf '\037\213')") _dd $offset | gunzip  ;; # '\x1f\x8b'
> +       "$(printf '\375\067')") _dd $offset | xzcat   ;; # '\xfd\x37'
> +       "$(printf '\135')") _dd $offset | unlzma      ;; # '\x5d\x00'
> +       "$(printf '\050\265')") _dd $offset | unzstd  ;; # '\x28\xb5'
> +       *) fatal "Unrecognized payload compression format in rpm file: $pkg" ;;
>  esac
> --
> 2.41.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#182844): https://lists.openembedded.org/g/openembedded-core/message/182844
> Mute This Topic: https://lists.openembedded.org/mt/99546092/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2 1/2] rpm2cpio.sh: update to the last 4.x version

[Alberto Planas Dominguez]

From: Khem Raj <raj.khem@gmail.com>
 
> I am seeing some race condition see
> https://errors.yoctoproject.org/Errors/Details/708923/

Confirmed. It is the second time that I do the same mistake. I hope v3 finally fix it.