bitbake-devel.lists.openembedded.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v3] fetch2: make sure the uri added as expected
       [not found] <168714731971476B.24733@lists.openembedded.org>
@ 2021-09-13  5:48 ` Yu, Mingli
  2021-10-26 23:24   ` [bitbake-devel] " Alexandre Belloni
  0 siblings, 1 reply; 3+ messages in thread
From: Yu, Mingli @ 2021-09-13  5:48 UTC (permalink / raw)
  To: bitbake-devel

From: Mingli Yu <mingli.yu@windriver.com>

When call uri_replace to get the new uris, we should guarantee the
new uri is added when the replace uri begins with file if the find
uri begin with wildcard which mapped as below.
PREMIRRORS_append = " \
     .*://.*/.*  git://somewhere.org/somedir/MIRRORNAME;protocol=http \n \
"    |           |
     |           |
     |           |
     find        replace

Before the patch:
the original uri
gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https

PREMIRRORS_append = " \
     .*://.*/.*  git://somewhere.org/somedir/MIRRORNAME;protocol=http \n \
"
The new uri is git://somewhere.org/somedir/github.com.tianocore.edk2.git/git2_github.com.tianocore.edk2.git.tar.gz;protocol=http

After the patch:
the original uri
gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https

PREMIRRORS_append = " \
     .*://.*/.*  git://somewhere.org/somedir/MIRRORNAME;protocol=http \n \
"
The new uri is None as the find begins with wildcard but the replace
doesn't begin with file.

Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
---
 lib/bb/fetch2/__init__.py | 2 ++
 lib/bb/tests/fetch.py     | 8 +++++++-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index d9e1599a..1c5242c7 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -430,6 +430,8 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
     uri_replace_decoded = list(decodeurl(uri_replace))
     logger.debug2("For url %s comparing %s to %s" % (uri_decoded, uri_find_decoded, uri_replace_decoded))
     result_decoded = ['', '', '', '', '', {}]
+    if uri_find_decoded[0] == '.*' and uri_replace_decoded[0] != 'file':
+        uri_find_decoded[0] = uri_replace_decoded[0]
     for loc, i in enumerate(uri_find_decoded):
         result_decoded[loc] = uri_decoded[loc]
         regexp = i
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 242be368..25893a34 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -406,6 +406,8 @@ class MirrorUriTest(FetcherTest):
             : "git://somewhere.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
         ("git://someserver.org/bitbake;tag=1234567890123456789012345678901234567890", "git://someserver.org/bitbake", "git://git.openembedded.org/bitbake")
             : "git://git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890",
+        ("gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https", ".*://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http")
+            : None,
         ("file://sstate-xyz.tgz", "file://.*", "file:///somewhere/1234/sstate-cache") 
             : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
         ("file://sstate-xyz.tgz", "file://.*", "file:///somewhere/1234/sstate-cache/") 
@@ -447,7 +449,11 @@ class MirrorUriTest(FetcherTest):
             ud.setup_localpath(self.d)
             mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2]))
             newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d)
-            self.assertEqual([v], newuris)
+            if not v:
+                 self.assertEqual([v], newuris)
+            else:
+                 self.assertEqual([], newuris)
+
 
     def test_urilist1(self):
         fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 3+ messages in thread

* Re: [bitbake-devel] [PATCH v3] fetch2: make sure the uri added as expected
  2021-09-13  5:48 ` [PATCH v3] fetch2: make sure the uri added as expected Yu, Mingli
@ 2021-10-26 23:24   ` Alexandre Belloni
  2021-10-27  6:45     ` Yu, Mingli
  0 siblings, 1 reply; 3+ messages in thread
From: Alexandre Belloni @ 2021-10-26 23:24 UTC (permalink / raw)
  To: Yu, Mingli; +Cc: bitbake-devel

Hello,

I'm sorry, it took me a while to properly test this patch but I'm pretty
sure it is the cause of:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2773/steps/11/logs/stdio

On 13/09/2021 13:48:10+0800, Yu, Mingli wrote:
> From: Mingli Yu <mingli.yu@windriver.com>
> 
> When call uri_replace to get the new uris, we should guarantee the
> new uri is added when the replace uri begins with file if the find
> uri begin with wildcard which mapped as below.
> PREMIRRORS_append = " \
>      .*://.*/.*  git://somewhere.org/somedir/MIRRORNAME;protocol=http \n \
> "    |           |
>      |           |
>      |           |
>      find        replace
> 
> Before the patch:
> the original uri
> gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https
> 
> PREMIRRORS_append = " \
>      .*://.*/.*  git://somewhere.org/somedir/MIRRORNAME;protocol=http \n \
> "
> The new uri is git://somewhere.org/somedir/github.com.tianocore.edk2.git/git2_github.com.tianocore.edk2.git.tar.gz;protocol=http
> 
> After the patch:
> the original uri
> gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https
> 
> PREMIRRORS_append = " \
>      .*://.*/.*  git://somewhere.org/somedir/MIRRORNAME;protocol=http \n \
> "
> The new uri is None as the find begins with wildcard but the replace
> doesn't begin with file.
> 
> Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
> ---
>  lib/bb/fetch2/__init__.py | 2 ++
>  lib/bb/tests/fetch.py     | 8 +++++++-
>  2 files changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index d9e1599a..1c5242c7 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -430,6 +430,8 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
>      uri_replace_decoded = list(decodeurl(uri_replace))
>      logger.debug2("For url %s comparing %s to %s" % (uri_decoded, uri_find_decoded, uri_replace_decoded))
>      result_decoded = ['', '', '', '', '', {}]
> +    if uri_find_decoded[0] == '.*' and uri_replace_decoded[0] != 'file':
> +        uri_find_decoded[0] = uri_replace_decoded[0]
>      for loc, i in enumerate(uri_find_decoded):
>          result_decoded[loc] = uri_decoded[loc]
>          regexp = i
> diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
> index 242be368..25893a34 100644
> --- a/lib/bb/tests/fetch.py
> +++ b/lib/bb/tests/fetch.py
> @@ -406,6 +406,8 @@ class MirrorUriTest(FetcherTest):
>              : "git://somewhere.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
>          ("git://someserver.org/bitbake;tag=1234567890123456789012345678901234567890", "git://someserver.org/bitbake", "git://git.openembedded.org/bitbake")
>              : "git://git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890",
> +        ("gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https", ".*://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http")
> +            : None,
>          ("file://sstate-xyz.tgz", "file://.*", "file:///somewhere/1234/sstate-cache") 
>              : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
>          ("file://sstate-xyz.tgz", "file://.*", "file:///somewhere/1234/sstate-cache/") 
> @@ -447,7 +449,11 @@ class MirrorUriTest(FetcherTest):
>              ud.setup_localpath(self.d)
>              mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2]))
>              newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d)
> -            self.assertEqual([v], newuris)
> +            if not v:
> +                 self.assertEqual([v], newuris)
> +            else:
> +                 self.assertEqual([], newuris)
> +
>  
>      def test_urilist1(self):
>          fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
> -- 
> 2.32.0
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#12637): https://lists.openembedded.org/g/bitbake-devel/message/12637
> Mute This Topic: https://lists.openembedded.org/mt/85568847/3617179
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [bitbake-devel] [PATCH v3] fetch2: make sure the uri added as expected
  2021-10-26 23:24   ` [bitbake-devel] " Alexandre Belloni
@ 2021-10-27  6:45     ` Yu, Mingli
  0 siblings, 0 replies; 3+ messages in thread
From: Yu, Mingli @ 2021-10-27  6:45 UTC (permalink / raw)
  To: Alexandre Belloni; +Cc: bitbake-devel



On 10/27/21 7:24 AM, Alexandre Belloni wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
> 
> Hello,
> 
> I'm sorry, it took me a while to properly test this patch but I'm pretty
> sure it is the cause of:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2773/steps/11/logs/stdio

Thanks for your following up! I will look into this.

Thanks,

> 
> On 13/09/2021 13:48:10+0800, Yu, Mingli wrote:
>> From: Mingli Yu <mingli.yu@windriver.com>
>>
>> When call uri_replace to get the new uris, we should guarantee the
>> new uri is added when the replace uri begins with file if the find
>> uri begin with wildcard which mapped as below.
>> PREMIRRORS_append = " \
>>       .*://.*/.*  git://somewhere.org/somedir/MIRRORNAME;protocol=http \n \
>> "    |           |
>>       |           |
>>       |           |
>>       find        replace
>>
>> Before the patch:
>> the original uri
>> gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https
>>
>> PREMIRRORS_append = " \
>>       .*://.*/.*  git://somewhere.org/somedir/MIRRORNAME;protocol=http \n \
>> "
>> The new uri is git://somewhere.org/somedir/github.com.tianocore.edk2.git/git2_github.com.tianocore.edk2.git.tar.gz;protocol=http
>>
>> After the patch:
>> the original uri
>> gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https
>>
>> PREMIRRORS_append = " \
>>       .*://.*/.*  git://somewhere.org/somedir/MIRRORNAME;protocol=http \n \
>> "
>> The new uri is None as the find begins with wildcard but the replace
>> doesn't begin with file.
>>
>> Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
>> ---
>>   lib/bb/fetch2/__init__.py | 2 ++
>>   lib/bb/tests/fetch.py     | 8 +++++++-
>>   2 files changed, 9 insertions(+), 1 deletion(-)
>>
>> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
>> index d9e1599a..1c5242c7 100644
>> --- a/lib/bb/fetch2/__init__.py
>> +++ b/lib/bb/fetch2/__init__.py
>> @@ -430,6 +430,8 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
>>       uri_replace_decoded = list(decodeurl(uri_replace))
>>       logger.debug2("For url %s comparing %s to %s" % (uri_decoded, uri_find_decoded, uri_replace_decoded))
>>       result_decoded = ['', '', '', '', '', {}]
>> +    if uri_find_decoded[0] == '.*' and uri_replace_decoded[0] != 'file':
>> +        uri_find_decoded[0] = uri_replace_decoded[0]
>>       for loc, i in enumerate(uri_find_decoded):
>>           result_decoded[loc] = uri_decoded[loc]
>>           regexp = i
>> diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
>> index 242be368..25893a34 100644
>> --- a/lib/bb/tests/fetch.py
>> +++ b/lib/bb/tests/fetch.py
>> @@ -406,6 +406,8 @@ class MirrorUriTest(FetcherTest):
>>               : "git://somewhere.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http",
>>           ("git://someserver.org/bitbake;tag=1234567890123456789012345678901234567890", "git://someserver.org/bitbake", "git://git.openembedded.org/bitbake")
>>               : "git://git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890",
>> +        ("gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https", ".*://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http")
>> +            : None,
>>           ("file://sstate-xyz.tgz", "file://.*", "file:///somewhere/1234/sstate-cache")
>>               : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
>>           ("file://sstate-xyz.tgz", "file://.*", "file:///somewhere/1234/sstate-cache/")
>> @@ -447,7 +449,11 @@ class MirrorUriTest(FetcherTest):
>>               ud.setup_localpath(self.d)
>>               mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2]))
>>               newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d)
>> -            self.assertEqual([v], newuris)
>> +            if not v:
>> +                 self.assertEqual([v], newuris)
>> +            else:
>> +                 self.assertEqual([], newuris)
>> +
>>
>>       def test_urilist1(self):
>>           fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
>> --
>> 2.32.0
>>
> 
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#12637): https://lists.openembedded.org/g/bitbake-devel/message/12637
>> Mute This Topic: https://lists.openembedded.org/mt/85568847/3617179
>> Group Owner: bitbake-devel+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
> 
> 
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
> 


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2021-10-27  6:39 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <168714731971476B.24733@lists.openembedded.org>
2021-09-13  5:48 ` [PATCH v3] fetch2: make sure the uri added as expected Yu, Mingli
2021-10-26 23:24   ` [bitbake-devel] " Alexandre Belloni
2021-10-27  6:45     ` Yu, Mingli

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).