[RFC PATCH] Allow remote sstate cache fetches when BB_NO_NETWORK set

[RFC PATCH] Allow remote sstate cache fetches when BB_NO_NETWORK set

[Joe MacDonald]

My objective here is to support a very specific scenario.  I have a well-fed
sstate cache on my internal network.  I want to use it for accelerating builds
across a variety of machines where NFS is not really an option.  I don't want
bitbake to ever attempt to fetch sources for packages, though, so I'm always
using BB_NO_NETWORK.

The proposal I have is adding a new local.conf variable that will allow cache
fetches only to be exempt from BB_NO_NETWORK restrictions.  If it is set and
BB_NO_NETWORK is not set, there's no change in behaviour.  If it is not set and
BB_NO_NETWORK is set, again, no change in today's behaviour.  If both are set,
however, then any of the regular fetch2 fetchers will get an "okay" from
check_network_access() if they're invoked by a sstate cache fetch attempt.  Any
other fetch attempt will still hit raise a NetworkAccess exception.

-- 
-Joe MacDonald.
:wq

[bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors

[Joe MacDonald]

BB_NO_NETWORK disables any fetching, however if we're using an external
sstate cache, we may want to be able to fetch those objects even if we are
not fetching the upstream sources.

Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com>
---
 bitbake/lib/bb/fetch2/__init__.py |   18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 6211cd7..edd290b3 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -714,10 +714,13 @@ def check_network_access(d, info = "", url = None):
     """
     log remote network access, and error if BB_NO_NETWORK is set
     """
-    if d.getVar("BB_NO_NETWORK", True) == "1":
-        raise NetworkAccess(url, info)
+    if d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == "1" and d.getVar("LOCAL_SSTATE_CACHE", True) == True:
+        logger.debug(1, "Fetcher accessed the network for sstate cache data with the command %s" % info)
     else:
-        logger.debug(1, "Fetcher accessed the network with the command %s" % info)
+        if d.getVar("BB_NO_NETWORK", True) == "1":
+            raise NetworkAccess(url, info)
+        else:
+            logger.debug(1, "Fetcher accessed the network with the command %s" % info)
 
 def build_mirroruris(origud, mirrors, ld):
     uris = []
@@ -1301,7 +1304,7 @@ class FetchMethod(object):
         return "%s-%s" % (key, d.getVar("PN", True) or "")
 
 class Fetch(object):
-    def __init__(self, urls, d, cache = True, localonly = False):
+    def __init__(self, urls, d, cache = True, localonly = False, sstate_cache = False):
         if localonly and cache:
             raise Exception("bb.fetch2.Fetch.__init__: cannot set cache and localonly at same time")
 
@@ -1310,6 +1313,8 @@ class Fetch(object):
         self.urls = urls
         self.d = d
         self.ud = {}
+        self.sstate_cache = sstate_cache
+        d.setVar("LOCAL_SSTATE_CACHE", sstate_cache)
 
         fn = d.getVar('FILE', True)
         if cache and fn and fn in urldata_cache:
@@ -1354,7 +1359,10 @@ class Fetch(object):
         if len(urls) == 0:
             urls = self.urls
 
-        network = self.d.getVar("BB_NO_NETWORK", True)
+        if self.sstate_cache == True and self.d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == True:
+            network = True
+        else:
+            network = self.d.getVar("BB_NO_NETWORK", True)
         premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY", True) == "1")
 
         for u in urls:
-- 
1.7.10.4

[bitbake][oe-core][RFC PATCH 2/2] sstate.bbclass: Add an extra check for sstate_mirrors

[Joe MacDonald]

BB_NO_NETWORK disables any fetching, however if we're using an external
sstate cache, we may want to be able to fetch those objects even if we are
not fetching the upstream sources.

Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com>
---
 meta/classes/sstate.bbclass |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 28dc312..759fb93 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -515,7 +515,7 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
                    'file://{0}.siginfo'.format(sstatefetch)]:
         localdata.setVar('SRC_URI', srcuri)
         try:
-            fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+            fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False, sstate_cache=True)
             fetcher.download()
 
             # Need to optimise this, if using file:// urls, the fetcher just changes the local path
@@ -632,7 +632,7 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
             bb.debug(2, "SState: Attempting to fetch %s" % srcuri)
 
             try:
-                fetcher = bb.fetch2.Fetch(srcuri.split(), localdata)
+                fetcher = bb.fetch2.Fetch(srcuri.split(), localdata, sstate_cache=True)
                 fetcher.checkstatus()
                 bb.debug(2, "SState: Successful fetch test for %s" % srcuri)
                 ret.append(task)
-- 
1.7.10.4

Re: [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors

[Richard Purdie]

On Fri, 2013-06-28 at 11:05 -0400, Joe MacDonald wrote:
> BB_NO_NETWORK disables any fetching, however if we're using an external
> sstate cache, we may want to be able to fetch those objects even if we are
> not fetching the upstream sources.
> 
> Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com>
> ---
>  bitbake/lib/bb/fetch2/__init__.py |   18 +++++++++++++-----
>  1 file changed, 13 insertions(+), 5 deletions(-)


Bitbake's fetch module knows nothing about sstate right now and I don't
see any reason to teach it about it. Why can't you just change
BB_NO_NETWORK in sstate.bbclass before we call into the fetcher?

Cheers,

Richard

> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index 6211cd7..edd290b3 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -714,10 +714,13 @@ def check_network_access(d, info = "", url = None):
>      """
>      log remote network access, and error if BB_NO_NETWORK is set
>      """
> -    if d.getVar("BB_NO_NETWORK", True) == "1":
> -        raise NetworkAccess(url, info)
> +    if d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == "1" and d.getVar("LOCAL_SSTATE_CACHE", True) == True:
> +        logger.debug(1, "Fetcher accessed the network for sstate cache data with the command %s" % info)
>      else:
> -        logger.debug(1, "Fetcher accessed the network with the command %s" % info)
> +        if d.getVar("BB_NO_NETWORK", True) == "1":
> +            raise NetworkAccess(url, info)
> +        else:
> +            logger.debug(1, "Fetcher accessed the network with the command %s" % info)
>  
>  def build_mirroruris(origud, mirrors, ld):
>      uris = []
> @@ -1301,7 +1304,7 @@ class FetchMethod(object):
>          return "%s-%s" % (key, d.getVar("PN", True) or "")
>  
>  class Fetch(object):
> -    def __init__(self, urls, d, cache = True, localonly = False):
> +    def __init__(self, urls, d, cache = True, localonly = False, sstate_cache = False):
>          if localonly and cache:
>              raise Exception("bb.fetch2.Fetch.__init__: cannot set cache and localonly at same time")
>  
> @@ -1310,6 +1313,8 @@ class Fetch(object):
>          self.urls = urls
>          self.d = d
>          self.ud = {}
> +        self.sstate_cache = sstate_cache
> +        d.setVar("LOCAL_SSTATE_CACHE", sstate_cache)
>  
>          fn = d.getVar('FILE', True)
>          if cache and fn and fn in urldata_cache:
> @@ -1354,7 +1359,10 @@ class Fetch(object):
>          if len(urls) == 0:
>              urls = self.urls
>  
> -        network = self.d.getVar("BB_NO_NETWORK", True)
> +        if self.sstate_cache == True and self.d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == True:
> +            network = True
> +        else:
> +            network = self.d.getVar("BB_NO_NETWORK", True)
>          premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY", True) == "1")
>  
>          for u in urls:

Re: [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors

[Mark Hatle]

On 6/28/13 11:56 AM, Richard Purdie wrote:
> On Fri, 2013-06-28 at 11:05 -0400, Joe MacDonald wrote:
>> BB_NO_NETWORK disables any fetching, however if we're using an external
>> sstate cache, we may want to be able to fetch those objects even if we are
>> not fetching the upstream sources.
>>
>> Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com>
>> ---
>>   bitbake/lib/bb/fetch2/__init__.py |   18 +++++++++++++-----
>>   1 file changed, 13 insertions(+), 5 deletions(-)
>
>
> Bitbake's fetch module knows nothing about sstate right now and I don't
> see any reason to teach it about it. Why can't you just change
> BB_NO_NETWORK in sstate.bbclass before we call into the fetcher?

We considered that, but won't that then change the value potentially for other 
things that could be running in parallel?  Or is the namespace protection good 
enough for that?

I.e. in sstate.bbclass do a d.setVar ... call the fetch .. d.setVar (restore)?

--Mark

> Cheers,
>
> Richard
>
>> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
>> index 6211cd7..edd290b3 100644
>> --- a/bitbake/lib/bb/fetch2/__init__.py
>> +++ b/bitbake/lib/bb/fetch2/__init__.py
>> @@ -714,10 +714,13 @@ def check_network_access(d, info = "", url = None):
>>       """
>>       log remote network access, and error if BB_NO_NETWORK is set
>>       """
>> -    if d.getVar("BB_NO_NETWORK", True) == "1":
>> -        raise NetworkAccess(url, info)
>> +    if d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == "1" and d.getVar("LOCAL_SSTATE_CACHE", True) == True:
>> +        logger.debug(1, "Fetcher accessed the network for sstate cache data with the command %s" % info)
>>       else:
>> -        logger.debug(1, "Fetcher accessed the network with the command %s" % info)
>> +        if d.getVar("BB_NO_NETWORK", True) == "1":
>> +            raise NetworkAccess(url, info)
>> +        else:
>> +            logger.debug(1, "Fetcher accessed the network with the command %s" % info)
>>
>>   def build_mirroruris(origud, mirrors, ld):
>>       uris = []
>> @@ -1301,7 +1304,7 @@ class FetchMethod(object):
>>           return "%s-%s" % (key, d.getVar("PN", True) or "")
>>
>>   class Fetch(object):
>> -    def __init__(self, urls, d, cache = True, localonly = False):
>> +    def __init__(self, urls, d, cache = True, localonly = False, sstate_cache = False):
>>           if localonly and cache:
>>               raise Exception("bb.fetch2.Fetch.__init__: cannot set cache and localonly at same time")
>>
>> @@ -1310,6 +1313,8 @@ class Fetch(object):
>>           self.urls = urls
>>           self.d = d
>>           self.ud = {}
>> +        self.sstate_cache = sstate_cache
>> +        d.setVar("LOCAL_SSTATE_CACHE", sstate_cache)
>>
>>           fn = d.getVar('FILE', True)
>>           if cache and fn and fn in urldata_cache:
>> @@ -1354,7 +1359,10 @@ class Fetch(object):
>>           if len(urls) == 0:
>>               urls = self.urls
>>
>> -        network = self.d.getVar("BB_NO_NETWORK", True)
>> +        if self.sstate_cache == True and self.d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == True:
>> +            network = True
>> +        else:
>> +            network = self.d.getVar("BB_NO_NETWORK", True)
>>           premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY", True) == "1")
>>
>>           for u in urls:
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>

Re: [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors

[Richard Purdie]

On Fri, 2013-06-28 at 12:00 -0500, Mark Hatle wrote:
> On 6/28/13 11:56 AM, Richard Purdie wrote:
> > On Fri, 2013-06-28 at 11:05 -0400, Joe MacDonald wrote:
> >> BB_NO_NETWORK disables any fetching, however if we're using an external
> >> sstate cache, we may want to be able to fetch those objects even if we are
> >> not fetching the upstream sources.
> >>
> >> Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com>
> >> ---
> >>   bitbake/lib/bb/fetch2/__init__.py |   18 +++++++++++++-----
> >>   1 file changed, 13 insertions(+), 5 deletions(-)
> >
> >
> > Bitbake's fetch module knows nothing about sstate right now and I don't
> > see any reason to teach it about it. Why can't you just change
> > BB_NO_NETWORK in sstate.bbclass before we call into the fetcher?
> 
> We considered that, but won't that then change the value potentially for other 
> things that could be running in parallel?  Or is the namespace protection good 
> enough for that?
> 
> I.e. in sstate.bbclass do a d.setVar ... call the fetch .. d.setVar (restore)?

If you look carefully you'll see sstate.bbclass does a
bb.data.createCopy() of the datastore and only changes that before
calling into the fetcher module. The change is only local to that copy
and the copy only lives as long as its referenced so it should all be
fine.

Cheers,

Richard

Re: [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 2025 bytes --]

[Re: [bitbake-devel] [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors] On 13.06.28 (Fri 18:04) Richard Purdie wrote:

> On Fri, 2013-06-28 at 12:00 -0500, Mark Hatle wrote:
> > On 6/28/13 11:56 AM, Richard Purdie wrote:
> > > On Fri, 2013-06-28 at 11:05 -0400, Joe MacDonald wrote:
> > >> BB_NO_NETWORK disables any fetching, however if we're using an external
> > >> sstate cache, we may want to be able to fetch those objects even if we are
> > >> not fetching the upstream sources.
> > >>
> > >> Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com>
> > >> ---
> > >>   bitbake/lib/bb/fetch2/__init__.py |   18 +++++++++++++-----
> > >>   1 file changed, 13 insertions(+), 5 deletions(-)
> > >
> > >
> > > Bitbake's fetch module knows nothing about sstate right now and I don't
> > > see any reason to teach it about it. Why can't you just change
> > > BB_NO_NETWORK in sstate.bbclass before we call into the fetcher?
> > 
> > We considered that, but won't that then change the value potentially for other 
> > things that could be running in parallel?  Or is the namespace protection good 
> > enough for that?
> > 
> > I.e. in sstate.bbclass do a d.setVar ... call the fetch .. d.setVar (restore)?
> 
> If you look carefully you'll see sstate.bbclass does a
> bb.data.createCopy() of the datastore and only changes that before
> calling into the fetcher module. The change is only local to that copy
> and the copy only lives as long as its referenced so it should all be
> fine.

Okay, thanks for the feedback, guys.  I've re-worked this into a purely
sstate.bbclass change and verified it is still behaving nicely, I'll
mail that out to openembedded-core@lists.openembedded.org now.

-J.

> 
> Cheers,
> 
> Richard
> 
> 
> 
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel

-- 
-Joe MacDonald.
:wq

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]