[bitbake-devel][PATCH V2 1/3] fetch2: fix downloadfilename issue with premirror

[bitbake-devel][PATCH V2 1/3] fetch2: fix downloadfilename issue with premirror

[Chen Qi]

The following commit to fix [Yocto #13039] causes regression of
the behavior of PREMIRRORS.

  "bitbake: fetch2: fix premirror URI when downloadfilename defined"

Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb
as an example.
SRC_URI = "\
    http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \
    ...
"
With the above commit, it now tries to fetch 1.4.0.tar.gz instead of
freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources
not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz.

The commit above tries to avoid fetching from invalid url such as:
https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz.
And its solution is to make basename to be 1.4.0.tar.gz, thus causing the
regression.

This patch fixes the above regression. For Yocto #13039, it now tries
to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 bitbake/lib/bb/fetch2/__init__.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 666cc1306a..000b49a500 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -466,9 +466,13 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
                     # Kill parameters, they make no sense for mirror tarballs
                     uri_decoded[5] = {}
                 elif ud.localpath and ud.method.supports_checksum(ud):
-                    basename = os.path.basename(uri_decoded[loc])
-                if basename and not result_decoded[loc].endswith(basename):
-                    result_decoded[loc] = os.path.join(result_decoded[loc], basename)
+                    basename = os.path.basename(ud.localpath)
+                if basename:
+                    uri_basename = os.path.basename(uri_decoded[loc])
+                    if basename != uri_basename and result_decoded[loc].endswith(uri_basename):
+                        result_decoded[loc] = result_decoded[loc].replace(uri_basename, basename)
+                    elif not result_decoded[loc].endswith(basename):
+                        result_decoded[loc] = os.path.join(result_decoded[loc], basename)
         else:
             return None
     result = encodeurl(result_decoded)
-- 
2.33.0

[bitbake-devel][PATCH V2 2/3] tests/fetch.py: fix premirror test cases

[Chen Qi]

When downloadfilename is specified, it is used to fetch from premirror.
So fix the test cases accordingly.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 bitbake/lib/bb/tests/fetch.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
index 8ad1c85990..d06580db88 100644
--- a/bitbake/lib/bb/tests/fetch.py
+++ b/bitbake/lib/bb/tests/fetch.py
@@ -875,17 +875,17 @@ class FetcherNetworkTest(FetcherTest):
     @skipIfNoNetwork()
     def test_fetch_premirror_specify_downloadfilename_regex_uri(self):
         self.d.setVar("PREMIRRORS", "http://.*/.* https://downloads.yoctoproject.org/releases/bitbake/")
-        fetcher = bb.fetch.Fetch(["http://invalid.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz;downloadfilename=bitbake-v1.0.0.tar.gz"], self.d)
+        fetcher = bb.fetch.Fetch(["http://invalid.yoctoproject.org/releases/bitbake/1.0.tar.gz;downloadfilename=bitbake-1.0.tar.gz"], self.d)
         fetcher.download()
-        self.assertEqual(os.path.getsize(self.dldir + "/bitbake-v1.0.0.tar.gz"), 57749)
+        self.assertEqual(os.path.getsize(self.dldir + "/bitbake-1.0.tar.gz"), 57749)
 
     @skipIfNoNetwork()
     # BZ13039
     def test_fetch_premirror_specify_downloadfilename_specific_uri(self):
         self.d.setVar("PREMIRRORS", "http://invalid.yoctoproject.org/releases/bitbake https://downloads.yoctoproject.org/releases/bitbake")
-        fetcher = bb.fetch.Fetch(["http://invalid.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz;downloadfilename=bitbake-v1.0.0.tar.gz"], self.d)
+        fetcher = bb.fetch.Fetch(["http://invalid.yoctoproject.org/releases/bitbake/1.0.tar.gz;downloadfilename=bitbake-1.0.tar.gz"], self.d)
         fetcher.download()
-        self.assertEqual(os.path.getsize(self.dldir + "/bitbake-v1.0.0.tar.gz"), 57749)
+        self.assertEqual(os.path.getsize(self.dldir + "/bitbake-1.0.tar.gz"), 57749)
 
     @skipIfNoNetwork()
     def gitfetcher(self, url1, url2):
-- 
2.33.0

[bitbake-devel][PATCH V2 3/3] tests/fetch.py: add test case to ensure downloadfilename is used for premirror

[Chen Qi]

Add a test case test_fetch_premirror_use_downloadfilename_to_fetch to ensure
that 'downloadfilename' is used when fetching from premirror.

Although the other two previous test cases, test_fetch_premirror_specify_downloadfilename_regex_uri
and test_fetch_premirror_specify_downloadfilename_specific_uri already
implicitly contain such verification, we still need to add a very clear
case to ensure no regression.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 bitbake/lib/bb/tests/fetch.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
index d06580db88..a008e7cc5b 100644
--- a/bitbake/lib/bb/tests/fetch.py
+++ b/bitbake/lib/bb/tests/fetch.py
@@ -887,6 +887,14 @@ class FetcherNetworkTest(FetcherTest):
         fetcher.download()
         self.assertEqual(os.path.getsize(self.dldir + "/bitbake-1.0.tar.gz"), 57749)
 
+    @skipIfNoNetwork()
+    def test_fetch_premirror_use_downloadfilename_to_fetch(self):
+        # Ensure downloadfilename is used when fetching from premirror.
+        self.d.setVar("PREMIRRORS", "http://.*/.* https://downloads.yoctoproject.org/releases/bitbake")
+        fetcher = bb.fetch.Fetch(["http://invalid.yoctoproject.org/releases/bitbake/bitbake-1.1.tar.gz;downloadfilename=bitbake-1.0.tar.gz"], self.d)
+        fetcher.download()
+        self.assertEqual(os.path.getsize(self.dldir + "/bitbake-1.0.tar.gz"), 57749)
+
     @skipIfNoNetwork()
     def gitfetcher(self, url1, url2):
         def checkrevision(self, fetcher):
-- 
2.33.0

Re: [bitbake-devel][PATCH V2 1/3] fetch2: fix downloadfilename issue with premirror

[ChenQi]

[-- Attachment #1: Type: text/plain, Size: 3073 bytes --]

Gentle ping on this series.

Regards,
Qi

On 10/18/2021 11:45 AM, Chen Qi wrote:
> The following commit to fix [Yocto #13039] causes regression of
> the behavior of PREMIRRORS.
>
>    "bitbake: fetch2: fix premirror URI when downloadfilename defined"
>
> Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb
> as an example.
> SRC_URI = "\
>      http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \
>      ...
> "
> With the above commit, it now tries to fetch 1.4.0.tar.gz instead of
> freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources
> not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz.
>
> The commit above tries to avoid fetching from invalid url such as:
> https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz.
> And its solution is to make basename to be 1.4.0.tar.gz, thus causing the
> regression.
>
> This patch fixes the above regression. For Yocto #13039, it now tries
> to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz.
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>   bitbake/lib/bb/fetch2/__init__.py | 10 +++++++---
>   1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index 666cc1306a..000b49a500 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -466,9 +466,13 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
>                       # Kill parameters, they make no sense for mirror tarballs
>                       uri_decoded[5] = {}
>                   elif ud.localpath and ud.method.supports_checksum(ud):
> -                    basename = os.path.basename(uri_decoded[loc])
> -                if basename and not result_decoded[loc].endswith(basename):
> -                    result_decoded[loc] = os.path.join(result_decoded[loc], basename)
> +                    basename = os.path.basename(ud.localpath)
> +                if basename:
> +                    uri_basename = os.path.basename(uri_decoded[loc])
> +                    if basename != uri_basename and result_decoded[loc].endswith(uri_basename):
> +                        result_decoded[loc] = result_decoded[loc].replace(uri_basename, basename)
> +                    elif not result_decoded[loc].endswith(basename):
> +                        result_decoded[loc] = os.path.join(result_decoded[loc], basename)
>           else:
>               return None
>       result = encodeurl(result_decoded)
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#12790): https://lists.openembedded.org/g/bitbake-devel/message/12790
> Mute This Topic: https://lists.openembedded.org/mt/86406437/3618072
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [Qi.Chen@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


[-- Attachment #2: Type: text/html, Size: 4314 bytes --]

Re: [bitbake-devel][PATCH V2 1/3] fetch2: fix downloadfilename issue with premirror

[Alexandre Belloni]

Hi,

On 27/10/2021 09:59:54+0800, Chen Qi wrote:
> Gentle ping on this series.
> 

It is currently under testing and will be in the next queue for Richard
to review and apply.

> Regards,
> Qi
> 
> On 10/18/2021 11:45 AM, Chen Qi wrote:
> > The following commit to fix [Yocto #13039] causes regression of
> > the behavior of PREMIRRORS.
> > 
> >    "bitbake: fetch2: fix premirror URI when downloadfilename defined"
> > 
> > Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb
> > as an example.
> > SRC_URI = "\
> >      http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \
> >      ...
> > "
> > With the above commit, it now tries to fetch 1.4.0.tar.gz instead of
> > freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources
> > not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz.
> > 
> > The commit above tries to avoid fetching from invalid url such as:
> > https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz.
> > And its solution is to make basename to be 1.4.0.tar.gz, thus causing the
> > regression.
> > 
> > This patch fixes the above regression. For Yocto #13039, it now tries
> > to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz.
> > 
> > Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> > ---
> >   bitbake/lib/bb/fetch2/__init__.py | 10 +++++++---
> >   1 file changed, 7 insertions(+), 3 deletions(-)
> > 
> > diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> > index 666cc1306a..000b49a500 100644
> > --- a/bitbake/lib/bb/fetch2/__init__.py
> > +++ b/bitbake/lib/bb/fetch2/__init__.py
> > @@ -466,9 +466,13 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
> >                       # Kill parameters, they make no sense for mirror tarballs
> >                       uri_decoded[5] = {}
> >                   elif ud.localpath and ud.method.supports_checksum(ud):
> > -                    basename = os.path.basename(uri_decoded[loc])
> > -                if basename and not result_decoded[loc].endswith(basename):
> > -                    result_decoded[loc] = os.path.join(result_decoded[loc], basename)
> > +                    basename = os.path.basename(ud.localpath)
> > +                if basename:
> > +                    uri_basename = os.path.basename(uri_decoded[loc])
> > +                    if basename != uri_basename and result_decoded[loc].endswith(uri_basename):
> > +                        result_decoded[loc] = result_decoded[loc].replace(uri_basename, basename)
> > +                    elif not result_decoded[loc].endswith(basename):
> > +                        result_decoded[loc] = os.path.join(result_decoded[loc], basename)
> >           else:
> >               return None
> >       result = encodeurl(result_decoded)
> > 
> > 
> > 
> > 
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#12840): https://lists.openembedded.org/g/bitbake-devel/message/12840
> Mute This Topic: https://lists.openembedded.org/mt/86406437/3617179
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com