Fetch problems with PREMIRRORS

Fetch problems with PREMIRRORS

[Gary Thomas]

Per Richard's suggestion, I have added this to my DISTRO.conf:

# Provide pre-staged sources
PREMIRRORS = "\
http://.*/.*   file:///work/misc/Poky/sources/ \n \
https://.*/.*   file:///work/misc/Poky/sources/ \n \
ftp://.*/.*   file:///work/misc/Poky/sources/ \n \
bzr://.*/.*   file:///work/misc/Poky/sources/ \n \
cvs://.*/.*   file:///work/misc/Poky/sources/ \n \
git://.*/.*   file:///work/misc/Poky/sources/ \n \
hg://.*/.*    file:///work/misc/Poky/sources/ \n \
osc://.*/.*   file:///work/misc/Poky/sources/ \n \
p4://.*/.*    file:///work/misc/Poky/sources/ \n \
svk://.*/.*   file:///work/misc/Poky/sources/ \n \
svn://.*/.*   file:///work/misc/Poky/sources/ \n"

The tree '/work/misc/Poky/sources' is a fully populated source
cache.

This all works great as long as the needed sources are in
the cache.  However, there seems to be no fallback if they
are not present.  I get errors like this:
| NOTE: fetch http://pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
| NOTE: fetch http://autobuilder.pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
| ERROR: Function 'Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any 
source.' failed
NOTE: package pseudo-native-0.0+git1+5434325fc887817ebb2bad36313d8277992cef1d-r18: task Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to 
fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any source.: Failed

In order to actually fetch, I have to remove the PREMIRRORS
from my conf.

Is there some way to get it to use the source cache but
still fetch when needed?

Thanks

Note: I also normally have BB_NO_NETWORK="1", but that is not
applicable here (no message about it) and setting it false does
not change the behaviour.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Fetch problems with PREMIRRORS

[Richard Purdie]

On Wed, 2011-02-09 at 06:40 -0700, Gary Thomas wrote:
> Per Richard's suggestion, I have added this to my DISTRO.conf:
> 
> # Provide pre-staged sources
> PREMIRRORS = "\
> http://.*/.*   file:///work/misc/Poky/sources/ \n \
> https://.*/.*   file:///work/misc/Poky/sources/ \n \
> ftp://.*/.*   file:///work/misc/Poky/sources/ \n \
> bzr://.*/.*   file:///work/misc/Poky/sources/ \n \
> cvs://.*/.*   file:///work/misc/Poky/sources/ \n \
> git://.*/.*   file:///work/misc/Poky/sources/ \n \
> hg://.*/.*    file:///work/misc/Poky/sources/ \n \
> osc://.*/.*   file:///work/misc/Poky/sources/ \n \
> p4://.*/.*    file:///work/misc/Poky/sources/ \n \
> svk://.*/.*   file:///work/misc/Poky/sources/ \n \
> svn://.*/.*   file:///work/misc/Poky/sources/ \n"
> 
> The tree '/work/misc/Poky/sources' is a fully populated source
> cache.
> 
> This all works great as long as the needed sources are in
> the cache.  However, there seems to be no fallback if they
> are not present.  I get errors like this:
> | NOTE: fetch http://pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
> | NOTE: fetch http://autobuilder.pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
> | ERROR: Function 'Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any 
> source.' failed
> NOTE: package pseudo-native-0.0+git1+5434325fc887817ebb2bad36313d8277992cef1d-r18: task Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to 
> fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any source.: Failed
> 
> In order to actually fetch, I have to remove the PREMIRRORS
> from my conf.
> 
> Is there some way to get it to use the source cache but
> still fetch when needed?
> 
> Thanks
> 
> Note: I also normally have BB_NO_NETWORK="1", but that is not
> applicable here (no message about it) and setting it false does
> not change the behaviour.

It is applicable.

Try not setting that variable at all, not setting it false. I suspect
there are two issues here:

a) BB_NO_NETWORK is having an effect but the log messages are being 
   swallowed...
b) "false" handling for the variable might not be working properly so 
   leave it unset.

We can fix those things but I'd like to confirm these are the problems
first.

Cheers,

Richard

Re: Fetch problems with PREMIRRORS

[Gary Thomas]

On 02/10/2011 07:03 AM, Richard Purdie wrote:
> On Wed, 2011-02-09 at 06:40 -0700, Gary Thomas wrote:
>> Per Richard's suggestion, I have added this to my DISTRO.conf:
>>
>> # Provide pre-staged sources
>> PREMIRRORS = "\
>> http://.*/.*   file:///work/misc/Poky/sources/ \n \
>> https://.*/.*   file:///work/misc/Poky/sources/ \n \
>> ftp://.*/.*   file:///work/misc/Poky/sources/ \n \
>> bzr://.*/.*   file:///work/misc/Poky/sources/ \n \
>> cvs://.*/.*   file:///work/misc/Poky/sources/ \n \
>> git://.*/.*   file:///work/misc/Poky/sources/ \n \
>> hg://.*/.*    file:///work/misc/Poky/sources/ \n \
>> osc://.*/.*   file:///work/misc/Poky/sources/ \n \
>> p4://.*/.*    file:///work/misc/Poky/sources/ \n \
>> svk://.*/.*   file:///work/misc/Poky/sources/ \n \
>> svn://.*/.*   file:///work/misc/Poky/sources/ \n"
>>
>> The tree '/work/misc/Poky/sources' is a fully populated source
>> cache.
>>
>> This all works great as long as the needed sources are in
>> the cache.  However, there seems to be no fallback if they
>> are not present.  I get errors like this:
>> | NOTE: fetch http://pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
>> | NOTE: fetch http://autobuilder.pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
>> | ERROR: Function 'Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any
>> source.' failed
>> NOTE: package pseudo-native-0.0+git1+5434325fc887817ebb2bad36313d8277992cef1d-r18: task Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to
>> fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any source.: Failed
>>
>> In order to actually fetch, I have to remove the PREMIRRORS
>> from my conf.
>>
>> Is there some way to get it to use the source cache but
>> still fetch when needed?
>>
>> Thanks
>>
>> Note: I also normally have BB_NO_NETWORK="1", but that is not
>> applicable here (no message about it) and setting it false does
>> not change the behaviour.
>
> It is applicable.
>
> Try not setting that variable at all, not setting it false. I suspect
> there are two issues here:
>
> a) BB_NO_NETWORK is having an effect but the log messages are being
>     swallowed...
> b) "false" handling for the variable might not be working properly so
>     leave it unset.
>
> We can fix those things but I'd like to confirm these are the problems
> first.

Correct, it does seem to be the case.  When BB_NO_NETWORK="0",
the fetcher works properly.  When set to "1", it fails, but
no explicit error about why is generated, neither on the console
nor in the log.do_fetch file.

I made this small change to check it out:
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index ef9d75f..57c4f20 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -395,6 +395,7 @@ def check_network_access(d, info = ""):
      log remote network access, and error if BB_NO_NETWORK is set
      """
      if bb.data.getVar("BB_NO_NETWORK", d, True) == "1":
+        bb.error("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % info)
          raise FetchError("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % inf
      else:
          logger.debug(1, "Fetcher accessed the network with the command %s" % info)

This shows that this _is_ the error case, with the messages making it to
the console and log file.  I can see that this exception is being thrown
on line 886, but the reason is lost in the exception data & I'm unclear
how to retrieve it and present it to the user.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Fetch problems with PREMIRRORS

[Richard Purdie]

On Thu, 2011-02-10 at 07:51 -0700, Gary Thomas wrote:
> > Try not setting that variable at all, not setting it false. I suspect
> > there are two issues here:
> >
> > a) BB_NO_NETWORK is having an effect but the log messages are being
> >     swallowed...
> > b) "false" handling for the variable might not be working properly so
> >     leave it unset.
> >
> > We can fix those things but I'd like to confirm these are the problems
> > first.
> 
> Correct, it does seem to be the case.  When BB_NO_NETWORK="0",
> the fetcher works properly.  When set to "1", it fails, but
> no explicit error about why is generated, neither on the console
> nor in the log.do_fetch file.
> 
> I made this small change to check it out:
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index ef9d75f..57c4f20 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -395,6 +395,7 @@ def check_network_access(d, info = ""):
>       log remote network access, and error if BB_NO_NETWORK is set
>       """
>       if bb.data.getVar("BB_NO_NETWORK", d, True) == "1":
> +        bb.error("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % info)
>           raise FetchError("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % inf
>       else:
>           logger.debug(1, "Fetcher accessed the network with the command %s" % info)
> 
> This shows that this _is_ the error case, with the messages making it to
> the console and log file.  I can see that this exception is being thrown
> on line 886, but the reason is lost in the exception data & I'm unclear
> how to retrieve it and present it to the user.

This should be at least partly addressed with some of the recent commits
such as:

http://git.pokylinux.org/cgit.cgi/poky/commit/?id=ea70c4362fdb81bc9467975c145c48196c45e3af

so the debug logs (with -D) at least show whats happening.

Not perfect but better at least...

Cheers,

Richard