All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] ipk ar/tar unpack
@ 2020-07-15 13:20 chris.minnoy_ext
  2020-07-19 15:25 ` [bitbake-devel] " Richard Purdie
  0 siblings, 1 reply; 5+ messages in thread
From: chris.minnoy_ext @ 2020-07-15 13:20 UTC (permalink / raw)
  To: bitbake-devel

From: Chris Minnoy <chris.minnoy_ext@softathome.com>
Date: Wed, 15 Jul 2020 15:02:35 +0200
Subject: [PATCH] ipk packages can be formed with ar or tar. ar/tar can't 
read
  each other archives. This fixes the unpack to call the correct 
archiver based
  on the format.

---
  bitbake/lib/bb/fetch2/__init__.py | 40 ++++++++++++++++++++++++++++---
  1 file changed, 37 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/__init__.py 
b/bitbake/lib/bb/fetch2/__init__.py
index 2ada43befc..e79149a208 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -1467,7 +1467,41 @@ class FetchMethod(object):
                      iterate_file = unpack_file
                  else:
                      cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
-            elif file.endswith('.deb') or file.endswith('.ipk'):
+            elif file.endswith('.ipk'):
+                output = subprocess.check_output(['file', file], 
preexec_fn=subprocess_setup)
+                if output:
+                    if output.decode().find("current ar archive") != -1:
+                        output = subprocess.check_output(['ar', '-t', 
file], preexec_fn=subprocess_setup)
+                        datafile = None
+                        if output:
+                            for line in output.decode().splitlines():
+                                if line.startswith('data.tar.'):
+                                    datafile = line
+                                    break
+                            else:
+                                raise UnpackError("Unable to unpack ipk 
package - does not contain data.tar.* file", urldata.url)
+                        else:
+                            raise UnpackError("Unable to unpack ipk 
package - could not list contents", urldata.url)
+                        cmd = 'ar x %s %s && tar --no-same-owner -xpf 
%s && rm %s' % (file, datafile, datafile, datafile)
+                    elif output.decode().find("gzip compressed data") 
!= -1:
+                        output = subprocess.check_output(['tar', '-tf', 
file], preexec_fn=subprocess_setup)
+                        datafile = None
+                        if output:
+                            for line in output.decode().splitlines():
+                                if line.startswith('./data.tar.'):
+                                    datafile = line
+                                    break
+                            else:
+                                raise UnpackError("Unable to unpack ipk 
package - does not contain data.tar.* file", urldata.url)
+                        else:
+                            raise UnpackError("Unable to unpack ipk 
package - could not list contents", urldata.url)
+                        cmd = 'tar -xf %s %s && tar --no-same-owner 
-xpf %s && rm %s' % (file, datafile, datafile, datafile)
+
+                    else:
+                        raise UnpackError("Unable to unpack ipk package 
- unknown format", urldata.url)
+                else:
+                    raise UnpackError("Unable to detect .ipk file 
format - unknown format", urldata.url)
+            elif file.endswith('.deb'):
                  output = subprocess.check_output(['ar', '-t', file], 
preexec_fn=subprocess_setup)
                  datafile = None
                  if output:
@@ -1476,9 +1510,9 @@ class FetchMethod(object):
                              datafile = line
                              break
                      else:
-                        raise UnpackError("Unable to unpack deb/ipk 
package - does not contain data.tar.* file", urldata.url)
+                        raise UnpackError("Unable to unpack deb package 
- does not contain data.tar.* file", urldata.url)
                  else:
-                    raise UnpackError("Unable to unpack deb/ipk package 
- could not list contents", urldata.url)
+                    raise UnpackError("Unable to unpack deb package - 
could not list contents", urldata.url)
                  cmd = 'ar x %s %s && tar --no-same-owner -xpf %s && rm 
%s' % (file, datafile, datafile, datafile)

          # If 'subdir' param exists, create a dir and use it as 
destination for unpack cmd
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [bitbake-devel] [PATCH] ipk ar/tar unpack
  2020-07-15 13:20 [PATCH] ipk ar/tar unpack chris.minnoy_ext
@ 2020-07-19 15:25 ` Richard Purdie
  2020-08-07 11:54   ` Chris Minnoy
  0 siblings, 1 reply; 5+ messages in thread
From: Richard Purdie @ 2020-07-19 15:25 UTC (permalink / raw)
  To: Chris Minnoy, bitbake-devel

Hi,

On Wed, 2020-07-15 at 15:20 +0200, Chris Minnoy wrote:
> From: Chris Minnoy <chris.minnoy_ext@softathome.com>
> Date: Wed, 15 Jul 2020 15:02:35 +0200
> Subject: [PATCH] ipk packages can be formed with ar or tar. ar/tar
> can't 
> read
>   each other archives. This fixes the unpack to call the correct 
> archiver based
>   on the format.
> 
> ---
>   bitbake/lib/bb/fetch2/__init__.py | 40
> ++++++++++++++++++++++++++++---
>   1 file changed, 37 insertions(+), 3 deletions(-)
> 
> diff --git a/bitbake/lib/bb/fetch2/__init__.py 
> b/bitbake/lib/bb/fetch2/__init__.py
> index 2ada43befc..e79149a208 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -1467,7 +1467,41 @@ class FetchMethod(object):
>                       iterate_file = unpack_file
>                   else:
>                       cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
> -            elif file.endswith('.deb') or file.endswith('.ipk'):
> +            elif file.endswith('.ipk'):
> +                output = subprocess.check_output(['file', file], 
> preexec_fn=subprocess_setup)
> +                if output:

Thanks for the patch. Unfortunately it came through line wrapped. The
short/long commit message also doesn't match our patch format
guidelines.

https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines

Could you fix those issues and resubmit please?

Cheers,

Richard


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] ipk ar/tar unpack
  2020-07-19 15:25 ` [bitbake-devel] " Richard Purdie
@ 2020-08-07 11:54   ` Chris Minnoy
  2020-08-07 13:01     ` [bitbake-devel] " Richard Purdie
  0 siblings, 1 reply; 5+ messages in thread
From: Chris Minnoy @ 2020-08-07 11:54 UTC (permalink / raw)
  To: bitbake-devel

[-- Attachment #1: Type: text/plain, Size: 5147 bytes --]

From d21155c4b55ac96d3c13a3cef1f020df67b62641 Mon Sep 17 00:00:00 2001
From: Chris Minnoy <chris.minnoy_ext@softathome.com>
Date: Fri, 7 Aug 2020 11:35:27 +0000
Subject: [PATCH 1/1] bitbake fetch2: fetcher can now unpack .ipk files from ar and tar format

Ipk packages can be formed with ar or tar. ar/tar can't read each others archives.
This fixes the unpack to call the correct archiver based on the format.
---
bitbake/lib/bb/fetch2/__init__.py | 39 ++++++++++++++++++++++++++++++++++++---
1 file changed, 36 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 2ada43befc..d66d677434 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -1467,7 +1467,40 @@ class FetchMethod(object):
iterate_file = unpack_file
else:
cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
-            elif file.endswith('.deb') or file.endswith('.ipk'):
+            elif file.endswith('.ipk'):
+                output = subprocess.check_output(['file', file], preexec_fn=subprocess_setup)
+                if output:
+                    if output.decode().find("current ar archive") != -1:
+                        output = subprocess.check_output(['ar', '-t', file], preexec_fn=subprocess_setup)
+                        datafile = None
+                        if output:
+                            for line in output.decode().splitlines():
+                                if line.startswith('data.tar.'):
+                                    datafile = line
+                                    break
+                                else:
+                                    raise UnpackError("Unable to unpack ipk package - does not contain data.tar.* file", urldata.url)
+                        else:
+                            raise UnpackError("Unable to unpack ipk package - could not list contents", urldata.url)
+                        cmd = 'ar x %s %s && tar --no-same-owner -xpf %s && rm %s' % (file, datafile, datafile, datafile)
+                    elif output.decode().find("gzip compressed data") != -1:
+                        output = subprocess.check_output(['tar', '-tf', file], preexec_fn=subprocess_setup)
+                        datafile = None
+                        if output:
+                            for line in output.decode().splitlines():
+                                if line.startswith('./data.tar.'):
+                                    datafile = line
+                                    break
+                                else:
+                                    raise UnpackError("Unable to unpack ipk package - does not contain data.tar.* file", urldata.url)
+                        else:
+                            raise UnpackError("Unable to unpack ipk package - could not list contents", urldata.url)
+                        cmd = 'tar -xf %s %s && tar --no-same-owner -xpf %s && rm %s' % (file, datafile, datafile, datafile)
+                    else:
+                        raise UnpackError("Unable to unpack ipk package - unknown format", urldata.url)
+                else:
+                    raise UnpackError("Unable to detect .ipk file format - unknown format", urldata.url)
+            elif file.endswith('.deb'):
output = subprocess.check_output(['ar', '-t', file], preexec_fn=subprocess_setup)
datafile = None
if output:
@@ -1476,9 +1509,9 @@ class FetchMethod(object):
datafile = line
break
else:
-                        raise UnpackError("Unable to unpack deb/ipk package - does not contain data.tar.* file", urldata.url)
+                        raise UnpackError("Unable to unpack deb package - does not contain data.tar.* file", urldata.url)
else:
-                    raise UnpackError("Unable to unpack deb/ipk package - could not list contents", urldata.url)
+                    raise UnpackError("Unable to unpack deb package - could not list contents", urldata.url)
cmd = 'ar x %s %s && tar --no-same-owner -xpf %s && rm %s' % (file, datafile, datafile, datafile)

# If 'subdir' param exists, create a dir and use it as destination for unpack cmd
--
2.11.0

[-- Attachment #2: Type: text/html, Size: 10564 bytes --]

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [bitbake-devel] [PATCH] ipk ar/tar unpack
  2020-08-07 11:54   ` Chris Minnoy
@ 2020-08-07 13:01     ` Richard Purdie
  2020-08-24  9:41       ` Chris Minnoy
  0 siblings, 1 reply; 5+ messages in thread
From: Richard Purdie @ 2020-08-07 13:01 UTC (permalink / raw)
  To: Chris Minnoy, bitbake-devel

On Fri, 2020-08-07 at 04:54 -0700, Chris Minnoy wrote:
> From d21155c4b55ac96d3c13a3cef1f020df67b62641 Mon Sep 17 00:00:00
> 2001
> From: Chris Minnoy <chris.minnoy_ext@softathome.com>
> Date: Fri, 7 Aug 2020 11:35:27 +0000
> Subject: [PATCH 1/1] bitbake fetch2: fetcher can now unpack .ipk
> files from ar and tar format
> 
> Ipk packages can be formed with ar or tar. ar/tar can't read each
> others archives.
> This fixes the unpack to call the correct archiver based on the
> format.

What is creating ipk files using tar instead of ar?

I know they did once exist but I've not seen real world use of those
for many years?

Cheers,

Richard


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [bitbake-devel] [PATCH] ipk ar/tar unpack
  2020-08-07 13:01     ` [bitbake-devel] " Richard Purdie
@ 2020-08-24  9:41       ` Chris Minnoy
  0 siblings, 0 replies; 5+ messages in thread
From: Chris Minnoy @ 2020-08-24  9:41 UTC (permalink / raw)
  To: Richard Purdie, bitbake-devel

We sometimes import ipk packages from OpenWRT into Yocto.
Unfortunatly they use the other archiver; hence the patch.

Cheers,

Chris

On 07/08/2020 15:01, Richard Purdie wrote:
> On Fri, 2020-08-07 at 04:54 -0700, Chris Minnoy wrote:
>>  From d21155c4b55ac96d3c13a3cef1f020df67b62641 Mon Sep 17 00:00:00
>> 2001
>> From: Chris Minnoy <chris.minnoy_ext@softathome.com>
>> Date: Fri, 7 Aug 2020 11:35:27 +0000
>> Subject: [PATCH 1/1] bitbake fetch2: fetcher can now unpack .ipk
>> files from ar and tar format
>>
>> Ipk packages can be formed with ar or tar. ar/tar can't read each
>> others archives.
>> This fixes the unpack to call the correct archiver based on the
>> format.
> What is creating ipk files using tar instead of ar?
>
> I know they did once exist but I've not seen real world use of those
> for many years?
>
> Cheers,
>
> Richard
>
-- 
With kind regards,

Chris Minnoy


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2020-08-24  9:41 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-15 13:20 [PATCH] ipk ar/tar unpack chris.minnoy_ext
2020-07-19 15:25 ` [bitbake-devel] " Richard Purdie
2020-08-07 11:54   ` Chris Minnoy
2020-08-07 13:01     ` [bitbake-devel] " Richard Purdie
2020-08-24  9:41       ` Chris Minnoy

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.