[PATCH] fetch2/wget: do not hardcode tarball compressors in version check

[PATCH] fetch2/wget: do not hardcode tarball compressors in version check

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 bitbake/lib/bb/fetch2/wget.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index fd9b304961..3021ef359e 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -583,7 +583,7 @@ class Wget(FetchMethod):
 
         # src.rpm extension was added only for rpm package. Can be removed if the rpm
         # packaged will always be considered as having to be manually upgraded
-        psuffix_regex = r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
+        psuffix_regex = r"(tar\.\w+|tgz|zip|xz|rpm|bz2|orig\.tar\.\w+|src\.tar\.\w+|src\.tgz|svnr\d+\.tar\.\w+|stable\.tar\.\w+|src\.rpm)"
 
         # match name, version and archive type of a package
         package_regex_comp = re.compile(r"(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
-- 
2.20.1

Re: [bitbake-devel] [PATCH] fetch2/wget: do not hardcode tarball compressors in version check

[Konrad Weihmann]

On 06.12.21 17:41, Alexander Kanavin wrote:
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   bitbake/lib/bb/fetch2/wget.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
> index fd9b304961..3021ef359e 100644
> --- a/bitbake/lib/bb/fetch2/wget.py
> +++ b/bitbake/lib/bb/fetch2/wget.py
> @@ -583,7 +583,7 @@ class Wget(FetchMethod):
>   
>           # src.rpm extension was added only for rpm package. Can be removed if the rpm
>           # packaged will always be considered as having to be manually upgraded
> -        psuffix_regex = r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
> +        psuffix_regex = r"(tar\.\w+|tgz|zip|xz|rpm|bz2|orig\.tar\.\w+|src\.tar\.\w+|src\.tgz|svnr\d+\.tar\.\w+|stable\.tar\.\w+|src\.rpm)"

I've seen tar.gz.md5, tar.gz.sig, tar.siz or even tar.sum files on a 
bunch of server - unfortunately those would now be considered as well. 
Not sure if that causes any negative effect on the actual version 
parsing...

>   
>           # match name, version and archive type of a package
>           package_regex_comp = re.compile(r"(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13133): https://lists.openembedded.org/g/bitbake-devel/message/13133
> Mute This Topic: https://lists.openembedded.org/mt/87544042/3647476
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [kweihmann@outlook.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [bitbake-devel] [PATCH] fetch2/wget: do not hardcode tarball compressors in version check

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2045 bytes --]

I'd hope that those always come in tandem with the actual tarballs, if that
proves to be not the case, we can tweak things further.

Alex

On Mon, 6 Dec 2021 at 17:50, Konrad Weihmann <kweihmann@outlook.com> wrote:

>
>
> On 06.12.21 17:41, Alexander Kanavin wrote:
> > Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> > ---
> >   bitbake/lib/bb/fetch2/wget.py | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/bitbake/lib/bb/fetch2/wget.py
> b/bitbake/lib/bb/fetch2/wget.py
> > index fd9b304961..3021ef359e 100644
> > --- a/bitbake/lib/bb/fetch2/wget.py
> > +++ b/bitbake/lib/bb/fetch2/wget.py
> > @@ -583,7 +583,7 @@ class Wget(FetchMethod):
> >
> >           # src.rpm extension was added only for rpm package. Can be
> removed if the rpm
> >           # packaged will always be considered as having to be manually
> upgraded
> > -        psuffix_regex =
> r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
> > +        psuffix_regex =
> r"(tar\.\w+|tgz|zip|xz|rpm|bz2|orig\.tar\.\w+|src\.tar\.\w+|src\.tgz|svnr\d+\.tar\.\w+|stable\.tar\.\w+|src\.rpm)"
>
> I've seen tar.gz.md5, tar.gz.sig, tar.siz or even tar.sum files on a
> bunch of server - unfortunately those would now be considered as well.
> Not sure if that causes any negative effect on the actual version
> parsing...
>
> >
> >           # match name, version and archive type of a package
> >           package_regex_comp =
> re.compile(r"(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
> >
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#13133):
> https://lists.openembedded.org/g/bitbake-devel/message/13133
> > Mute This Topic: https://lists.openembedded.org/mt/87544042/3647476
> > Group Owner: bitbake-devel+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> kweihmann@outlook.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>

[-- Attachment #2: Type: text/html, Size: 3180 bytes --]

Re: [bitbake-devel] [PATCH] fetch2/wget: do not hardcode tarball compressors in version check

[Richard Purdie]

On Mon, 2021-12-06 at 17:41 +0100, Alexander Kanavin wrote:
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  bitbake/lib/bb/fetch2/wget.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
> index fd9b304961..3021ef359e 100644
> --- a/bitbake/lib/bb/fetch2/wget.py
> +++ b/bitbake/lib/bb/fetch2/wget.py
> @@ -583,7 +583,7 @@ class Wget(FetchMethod):
>  
>          # src.rpm extension was added only for rpm package. Can be removed if the rpm
>          # packaged will always be considered as having to be manually upgraded
> -        psuffix_regex = r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
> +        psuffix_regex = r"(tar\.\w+|tgz|zip|xz|rpm|bz2|orig\.tar\.\w+|src\.tar\.\w+|src\.tgz|svnr\d+\.tar\.\w+|stable\.tar\.\w+|src\.rpm)"
>  
>          # match name, version and archive type of a package
>          package_regex_comp = re.compile(r"(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"

Looks good, is there something we should be adding to lib/bb/tests/fetch.py so
that we don't forget this lesson?

Cheers,

Richard

Re: [bitbake-devel] [PATCH] fetch2/wget: do not hardcode tarball compressors in version check

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

On Mon, 6 Dec 2021 at 23:47, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> >          # match name, version and archive type of a package
> >          package_regex_comp =
> re.compile(r"(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
>
> Looks good, is there something we should be adding to
> lib/bb/tests/fetch.py so
> that we don't forget this lesson?
>

I don't think so, 'test that various tar.something suffixes are ok' is
already covered, and it's not possible to make a test for 'any tar.suffix
is okay'.

Alex

[-- Attachment #2: Type: text/html, Size: 1061 bytes --]

Re: [bitbake-devel] [PATCH] fetch2/wget: do not hardcode tarball compressors in version check

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

On Tue, 7 Dec 2021 at 11:24, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> The issue was a .orig.tar.XXX so perhaps we're missing a test for that
> though?
>
> I appreciate we can't cover everything, my worry is that if we change this
> code
> again, we may lose this corner case...
>

The issue was that orig.tar.gz was covered by the regex (as it was
explicitly listed), but orig.tar.bz2 was not. The test already checks
various .tar.XXX suffixes, so I don't think there's much value in also
adding .orig.tar.XXX and stable.tar.XXX and src.tar.XXX checks. They've all
been changed to use \w (i.e. alphanumeric suffix).

Alex

[-- Attachment #2: Type: text/html, Size: 1098 bytes --]

Re: [bitbake-devel] [PATCH] fetch2/wget: do not hardcode tarball compressors in version check

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 688 bytes --]

On Tue, 7 Dec 2021 at 12:28, Alexander Kanavin via lists.openembedded.org
<alex.kanavin=gmail.com@lists.openembedded.org> wrote:

> I appreciate we can't cover everything, my worry is that if we change this
> code
>
>> again, we may lose this corner case...
>>
>
> The issue was that orig.tar.gz was covered by the regex (as it was
> explicitly listed), but orig.tar.bz2 was not. The test already checks
> various .tar.XXX suffixes, so I don't think there's much value in also
> adding .orig.tar.XXX and stable.tar.XXX and src.tar.XXX checks. They've all
> been changed to use \w (i.e. alphanumeric suffix).
>

Let me look again actually,  maybe I misunderstood what the test does.

Alex

[-- Attachment #2: Type: text/html, Size: 1408 bytes --]