Re: [bitbake-devel] [PATCH v3 3/4] fetch2/wget: ensure all variables are set when calling urllib

["Enrico Scholz" <enrico.scholz@sigma-chemnitz.de>]

"Enrico Scholz via lists.openembedded.org"
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> writes:

> "Ross Burton" <ross@burtonini.com> writes:
>
>> Instead of just exporting the proxy variables when calling into urllib,
>> use bb.utils.environment() to export all of the known variables that are
>> needed for proper connectivity.
>
> This patch breaks fetching from sstate servers[1].
> 
> | https_proxy=http://www-cache:3128/
> | no_proxy='.sigma-chemnitz.de,...'
> 
> I can reproduce it here e.g. by
>
> | $ bitbake gettext-minimal-native -ccleansstate
> | $ bitbake gettext-minimal-native
> | ERROR: gettext-minimal-native-0.21-r0 do_populate_lic_setscene: No suitable staging package found
> |
> | DEBUG: checkstatus() urlopen failed: <urlopen error Tunnel connection failed: 403 Forbidden>

I guess this happens because in

| +        with bb.utils.environment(**newenv):
| +            handlers = [...
| +                        urllib.request.ProxyHandler(),

the urllib.request.ProxyHandler constructor takes 'https_proxy' from
environment (but not 'no_proxy').

Real open happens later (when the 'with bb.utils.environment...' has
been left) and there,

| class ProxyHandler(BaseHandler):
|     def proxy_open(self, req, proxy, type):
|         ...
|         if req.host and proxy_bypass(req.host):
|             return None

is called.  'proxy_bypass()' checks 'no_proxy' from env which is not
available anymore.


Normale fetching does not seem to be affected because proxy vars are in
BB_ENV_EXTRAWHITE and exported to jobs.  But 'sstate' fetching does not
seem to use this mechanism.



Enrico