[PATCH 0/1] RFC Bug in PREMIRROR

[PATCH 0/1] RFC Bug in PREMIRROR

[Vishal Patel]

I think we found a bug in the PREMIRROR system where if both a remote and local
PREMIRROR is defined, as well as BB_NO_NETWORK = '1', the system will fail with

bb.fetch2.NetworkAccess: Network access disabled through BB_NO_NETWORK (or set indirectly due to use of BB_FETCH_PREMIRRORONLY)

instead of trying the next PREMIRROR which is a local file:// based one.
The attached patch only adds a test case for the issue, but does not fix the problem.
If anyone can point me to where to fix the problem, I can work on this.

Vishal Patel (1):
  tests/fetch.py: Add test case to verify PREMIRROR with mix of remote
    and local

 lib/bb/tests/fetch.py | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

--
2.25.1

This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

[PATCH 1/1] tests/fetch.py: Add test case to verify PREMIRROR with mix of remote and local

[Vishal Patel]

If the PREMIRROR lists both remote and local mirrors, and has BB_NO_NETWORK
enabled, it will fail on the first network attempt without falling back to
the local (file://) PREMIRROR.

Signed-off-by: Vishal Patel <vishal.patel@xilinx.com>
---
 lib/bb/tests/fetch.py | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 9291ce4a..559c678b 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -823,6 +823,26 @@ class FetcherNoNetworkTest(FetcherTest):
         self.assertFalse(os.path.exists(os.path.join(self.dldir, "test-file.tar.gz")))
         self.assertFalse(os.path.exists(os.path.join(self.dldir, "test-file.tar.gz.done")))

+    def test_fetch_premirror(self):
+        mirrordir = os.path.join(self.tempdir, 'mirror')
+        os.mkdir(mirrordir)
+        # create the file in a mirror directory with correct hash
+        string = "this is a test file\n".encode("utf-8")
+        with open(os.path.join(mirrordir, "test-file.tar.gz"), "wb") as f:
+            f.write(string)
+        self.d.setVarFlag("SRC_URI", "sha256sum", "b6668cf8c46c7075e18215d922e7812ca082fa6cc34668d00a6c20aee4551fb6")
+        self.d.setVar("PREMIRRORS", "http://.*/.* file://%s/test-file.tar.gz" % mirrordir)
+        fetcher = bb.fetch.Fetch(["http://invalid.yoctoproject.org/release/test-file.tar.gz"], self.d)
+        fetcher.download()
+        self.assertEqual(os.path.getsize(self.dldir + "/test-file.tar.gz"), 20)
+        bb.utils.remove(self.dldir, recurse=True)
+
+        self.d.setVar("BB_NO_NETWORK", "1")
+        self.d.setVar("PREMIRRORS", "http://.*/.* http://downloads.yoctoproject.org/releases/bitbake \nhttp://.*/.* file://%s/test-file.tar.gz" % mirrordir)
+        fetcher = bb.fetch.Fetch(["http://invalid.yoctoproject.org/release/test-file.tar.gz"], self.d)
+        fetcher.download()
+        self.assertEqual(os.path.getsize(self.dldir + "/test-file.tar.gz"), 20)
+
 class FetcherNetworkTest(FetcherTest):
     @skipIfNoNetwork()
     def test_fetch(self):
--
2.25.1

This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

Re: [bitbake-devel] [PATCH 0/1] RFC Bug in PREMIRROR

[Mark Hatle]

On 9/1/21 2:44 PM, Vishal Patel wrote:
> I think we found a bug in the PREMIRROR system where if both a remote and local
> PREMIRROR is defined, as well as BB_NO_NETWORK = '1', the system will fail with
> 
> bb.fetch2.NetworkAccess: Network access disabled through BB_NO_NETWORK (or set indirectly due to use of BB_FETCH_PREMIRRORONLY)
> 
> instead of trying the next PREMIRROR which is a local file:// based one.
> The attached patch only adds a test case for the issue, but does not fix the problem.
> If anyone can point me to where to fix the problem, I can work on this.

What we're attempting to do is specify both a network and local disk premirror
location.  The network one seems to get added in the list first (for whatever
reason), but it never falls back to the disk version.

As the patch shows:

PREMIRROR = "http://.*/.* http://downloads.yoctoproject.org/releases/bitbake \n
http://.*/.* file://%s/test-file.tar.gz \n"

appears to fail when it gets to the first one, and never tries the second entry.

> Vishal Patel (1):
>   tests/fetch.py: Add test case to verify PREMIRROR with mix of remote
>     and local
> 
>  lib/bb/tests/fetch.py | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> --
> 2.25.1
> 
> This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

BTW we apologize for the footer, it shouldn't have been on this and the
associate patch email.  We're working to get this removed in future conversations.

Re: [bitbake-devel] [PATCH 0/1] RFC Bug in PREMIRROR

[Richard Purdie]

On Wed, 2021-09-01 at 12:44 -0700, Vishal Patel wrote:
> I think we found a bug in the PREMIRROR system where if both a remote and local
> PREMIRROR is defined, as well as BB_NO_NETWORK = '1', the system will fail with
> 
> bb.fetch2.NetworkAccess: Network access disabled through BB_NO_NETWORK (or set indirectly due to use of BB_FETCH_PREMIRRORONLY)
> 
> instead of trying the next PREMIRROR which is a local file:// based one.
> The attached patch only adds a test case for the issue, but does not fix the problem.
> If anyone can point me to where to fix the problem, I can work on this.

Thanks for the testcase, that does at least make this easy to see.

Running the test, it is clear from the trace that after trying a mirror, the
bb.fetch2.NetworkAccess exception is fatal and stops any other url from being
tried. That does seem not to be what the user would want.

I was going to let you debug that but I did quickly try this:


diff --git a/bitbake/lib/bb/fetch2/__init__.py
b/bitbake/lib/bb/fetch2/__init__.py
index 914fa5c0243..65d8c38af76 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -1034,7 +1034,7 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
         return ud.localpath
 
     except bb.fetch2.NetworkAccess:
-        raise
+        return False
 
     except IOError as e:
         if e.errno in [errno.ESTALE]:

and found that does make your test case pass. It also doesn't regress any other
test that I spotted in quick testing.

I'm not sure if that is the correct fix, I'd need to think about it a bit more. 

The question is whether if you disable the networking, you want errors on
network access or want the accesses disabled. Some people do want to know if the
network is used when disabled and BB_NO_NETWORK may mean different things to
different people.

Cheers,

Richard

Re: [bitbake-devel] [PATCH 0/1] RFC Bug in PREMIRROR

[Mark Hatle]

On 9/1/21 4:40 PM, Richard Purdie wrote:
> On Wed, 2021-09-01 at 12:44 -0700, Vishal Patel wrote:
>> I think we found a bug in the PREMIRROR system where if both a remote and local
>> PREMIRROR is defined, as well as BB_NO_NETWORK = '1', the system will fail with
>>
>> bb.fetch2.NetworkAccess: Network access disabled through BB_NO_NETWORK (or set indirectly due to use of BB_FETCH_PREMIRRORONLY)
>>
>> instead of trying the next PREMIRROR which is a local file:// based one.
>> The attached patch only adds a test case for the issue, but does not fix the problem.
>> If anyone can point me to where to fix the problem, I can work on this.
> 
> Thanks for the testcase, that does at least make this easy to see.
> 
> Running the test, it is clear from the trace that after trying a mirror, the
> bb.fetch2.NetworkAccess exception is fatal and stops any other url from being
> tried. That does seem not to be what the user would want.
> 
> I was going to let you debug that but I did quickly try this:
> 
> 
> diff --git a/bitbake/lib/bb/fetch2/__init__.py
> b/bitbake/lib/bb/fetch2/__init__.py
> index 914fa5c0243..65d8c38af76 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -1034,7 +1034,7 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
>          return ud.localpath
>  
>      except bb.fetch2.NetworkAccess:
> -        raise
> +        return False
>  
>      except IOError as e:
>          if e.errno in [errno.ESTALE]:
> 
> and found that does make your test case pass. It also doesn't regress any other
> test that I spotted in quick testing.
> 
> I'm not sure if that is the correct fix, I'd need to think about it a bit more. 
> 
> The question is whether if you disable the networking, you want errors on
> network access or want the accesses disabled. Some people do want to know if the
> network is used when disabled and BB_NO_NETWORK may mean different things to
> different people.

I thought in the past it just skipped them but stored a message or something
about the networkaccess.. then if it couldn't find anything it THEN displayed
the message.

roughly:

try... ohh network?  capture exception
try next...
try next...
nothing found, print exception message (and since it's PREMIRROR don't fail,
continue to regular download and then MIRROR)

--Mark

> Cheers,
> 
> Richard
> 
> 
> 
> 
> 

Re: [bitbake-devel] [PATCH 0/1] RFC Bug in PREMIRROR

[Richard Purdie]

On Wed, 2021-09-01 at 16:46 -0500, Mark Hatle wrote:
> 
> On 9/1/21 4:40 PM, Richard Purdie wrote:
> > On Wed, 2021-09-01 at 12:44 -0700, Vishal Patel wrote:
> > > I think we found a bug in the PREMIRROR system where if both a remote and local
> > > PREMIRROR is defined, as well as BB_NO_NETWORK = '1', the system will fail with
> > > 
> > > bb.fetch2.NetworkAccess: Network access disabled through BB_NO_NETWORK (or set indirectly due to use of BB_FETCH_PREMIRRORONLY)
> > > 
> > > instead of trying the next PREMIRROR which is a local file:// based one.
> > > The attached patch only adds a test case for the issue, but does not fix the problem.
> > > If anyone can point me to where to fix the problem, I can work on this.
> > 
> > Thanks for the testcase, that does at least make this easy to see.
> > 
> > Running the test, it is clear from the trace that after trying a mirror, the
> > bb.fetch2.NetworkAccess exception is fatal and stops any other url from being
> > tried. That does seem not to be what the user would want.
> > 
> > I was going to let you debug that but I did quickly try this:
> > 
> > 
> > diff --git a/bitbake/lib/bb/fetch2/__init__.py
> > b/bitbake/lib/bb/fetch2/__init__.py
> > index 914fa5c0243..65d8c38af76 100644
> > --- a/bitbake/lib/bb/fetch2/__init__.py
> > +++ b/bitbake/lib/bb/fetch2/__init__.py
> > @@ -1034,7 +1034,7 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
> >          return ud.localpath
> >  
> >      except bb.fetch2.NetworkAccess:
> > -        raise
> > +        return False
> >  
> >      except IOError as e:
> >          if e.errno in [errno.ESTALE]:
> > 
> > and found that does make your test case pass. It also doesn't regress any other
> > test that I spotted in quick testing.
> > 
> > I'm not sure if that is the correct fix, I'd need to think about it a bit more. 
> > 
> > The question is whether if you disable the networking, you want errors on
> > network access or want the accesses disabled. Some people do want to know if the
> > network is used when disabled and BB_NO_NETWORK may mean different things to
> > different people.
> 
> I thought in the past it just skipped them but stored a message or something
> about the networkaccess.. then if it couldn't find anything it THEN displayed
> the message.
> 
> roughly:
> 
> try... ohh network?  capture exception
> try next...
> try next...
> nothing found, print exception message (and since it's PREMIRROR don't fail,
> continue to regular download and then MIRROR)

It does that about general failures but network access is different. Some users
expect it to fail immediately for any access.

Cheers,

Richard