[PATCH v3 1/4] utils: add environment updating context manager

[PATCH v3 1/4] utils: add environment updating context manager

[Ross Burton]

bb.utils.environment() is a context manager to alter os.environ inside
a specific block, restoring it after the block is closed.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 bitbake/lib/bb/tests/utils.py | 18 ++++++++++++++++++
 bitbake/lib/bb/utils.py       | 16 ++++++++++++++++
 2 files changed, 34 insertions(+)

diff --git a/bitbake/lib/bb/tests/utils.py b/bitbake/lib/bb/tests/utils.py
index a7ff33db527..4d5e21b99e1 100644
--- a/bitbake/lib/bb/tests/utils.py
+++ b/bitbake/lib/bb/tests/utils.py
@@ -666,3 +666,21 @@ class GetReferencedVars(unittest.TestCase):
 
         layers = [{"SRC_URI"}, {"QT_GIT", "QT_MODULE", "QT_MODULE_BRANCH_PARAM", "QT_GIT_PROTOCOL"}, {"QT_GIT_PROJECT", "QT_MODULE_BRANCH", "BPN"}, {"PN", "SPECIAL_PKGSUFFIX"}]
         self.check_referenced("${SRC_URI}", layers)
+
+
+class EnvironmentTests(unittest.TestCase):
+    def test_environment(self):
+        os.environ["A"] = "this is A"
+        self.assertIn("A", os.environ)
+        self.assertEqual(os.environ["A"], "this is A")
+        self.assertNotIn("B", os.environ)
+
+        with bb.utils.environment(B="this is B"):
+            self.assertIn("A", os.environ)
+            self.assertEqual(os.environ["A"], "this is A")
+            self.assertIn("B", os.environ)
+            self.assertEqual(os.environ["B"], "this is B")
+
+        self.assertIn("A", os.environ)
+        self.assertEqual(os.environ["A"], "this is A")
+        self.assertNotIn("B", os.environ)
diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index e6e82d11182..70634910f7b 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1681,3 +1681,19 @@ def rename(src, dst):
             shutil.move(src, dst)
         else:
             raise err
+
+@contextmanager
+def environment(**envvars):
+    """
+    Context manager to selectively update the environment with the specified mapping.
+    """
+    backup = dict(os.environ)
+    try:
+        os.environ.update(envvars)
+        yield
+    finally:
+        for var in envvars:
+            if var in backup:
+                os.environ[var] = backup[var]
+            else:
+                del os.environ[var]
-- 
2.25.1

[PATCH v3 2/4] fetch2: expose environment variable names that need to be exported

[Ross Burton]

There is a list of environment variable names that need to be exported
when executing external commands, such as 'http_proxy'.  To avoid
duplication, make this a top-level variable.

Also add SSL_CERT_FILE, which is used by OpenSSL to locate the
certificate bundle. This is needed in buildtools environments where the
default path isn't valid.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 bitbake/lib/bb/fetch2/__init__.py | 46 ++++++++++++++++---------------
 1 file changed, 24 insertions(+), 22 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index ad898680ff7..914fa5c0243 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -808,6 +808,29 @@ def localpath(url, d):
     fetcher = bb.fetch2.Fetch([url], d)
     return fetcher.localpath(url)
 
+# Need to export PATH as binary could be in metadata paths
+# rather than host provided
+# Also include some other variables.
+FETCH_EXPORT_VARS = ['HOME', 'PATH',
+                     'HTTP_PROXY', 'http_proxy',
+                     'HTTPS_PROXY', 'https_proxy',
+                     'FTP_PROXY', 'ftp_proxy',
+                     'FTPS_PROXY', 'ftps_proxy',
+                     'NO_PROXY', 'no_proxy',
+                     'ALL_PROXY', 'all_proxy',
+                     'GIT_PROXY_COMMAND',
+                     'GIT_SSH',
+                     'GIT_SSL_CAINFO',
+                     'GIT_SMART_HTTP',
+                     'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
+                     'SOCKS5_USER', 'SOCKS5_PASSWD',
+                     'DBUS_SESSION_BUS_ADDRESS',
+                     'P4CONFIG',
+                     'SSL_CERT_FILE',
+                     'AWS_ACCESS_KEY_ID',
+                     'AWS_SECRET_ACCESS_KEY',
+                     'AWS_DEFAULT_REGION']
+
 def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
     """
     Run cmd returning the command output
@@ -816,28 +839,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
     Optionally remove the files/directories listed in cleanup upon failure
     """
 
-    # Need to export PATH as binary could be in metadata paths
-    # rather than host provided
-    # Also include some other variables.
-    # FIXME: Should really include all export varaiables?
-    exportvars = ['HOME', 'PATH',
-                  'HTTP_PROXY', 'http_proxy',
-                  'HTTPS_PROXY', 'https_proxy',
-                  'FTP_PROXY', 'ftp_proxy',
-                  'FTPS_PROXY', 'ftps_proxy',
-                  'NO_PROXY', 'no_proxy',
-                  'ALL_PROXY', 'all_proxy',
-                  'GIT_PROXY_COMMAND',
-                  'GIT_SSH',
-                  'GIT_SSL_CAINFO',
-                  'GIT_SMART_HTTP',
-                  'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
-                  'SOCKS5_USER', 'SOCKS5_PASSWD',
-                  'DBUS_SESSION_BUS_ADDRESS',
-                  'P4CONFIG',
-                  'AWS_ACCESS_KEY_ID',
-                  'AWS_SECRET_ACCESS_KEY',
-                  'AWS_DEFAULT_REGION']
+    exportvars = FETCH_EXPORT_VARS
 
     if not cleanup:
         cleanup = []
-- 
2.25.1

[PATCH v3 3/4] fetch2/wget: ensure all variables are set when calling urllib

[Ross Burton]

Instead of just exporting the proxy variables when calling into urllib,
use bb.utils.environment() to export all of the known variables that are
needed for proper connectivity.

Specifically, this ensures that SSL_CERT_FILE is set, so that libssl can
find the certificates in buildtools environments

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 bitbake/lib/bb/fetch2/wget.py | 43 ++++++++++++++++++++++++-----------
 1 file changed, 30 insertions(+), 13 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 784df70c9f6..d67f9b889cc 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -282,19 +282,36 @@ class Wget(FetchMethod):
                 newreq = urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, headers, newurl)
                 newreq.get_method = req.get_method
                 return newreq
-        exported_proxies = export_proxies(d)
-
-        handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback]
-        if exported_proxies:
-            handlers.append(urllib.request.ProxyHandler())
-        handlers.append(CacheHTTPHandler())
-        # Since Python 2.7.9 ssl cert validation is enabled by default
-        # see PEP-0476, this causes verification errors on some https servers
-        # so disable by default.
-        import ssl
-        if hasattr(ssl, '_create_unverified_context'):
-            handlers.append(urllib.request.HTTPSHandler(context=ssl._create_unverified_context()))
-        opener = urllib.request.build_opener(*handlers)
+
+        # We need to update the environment here as both the proxy and HTTPS
+        # handlers need variables set. The proxy needs http_proxy and friends to
+        # be set, and HTTPSHandler ends up calling into openssl to load the
+        # certificates. In buildtools configurations this will be looking at the
+        # wrong place for certificates by default: we set SSL_CERT_FILE to the
+        # right location in the buildtools environment script but as BitBake
+        # prunes prunes the environment this is lost. When binaries are executed
+        # runfetchcmd ensures these values are in the environment, but this is
+        # pure Python so we need to update the environment.
+        #
+        # Avoid tramping the environment too much by using bb.utils.environment
+        # to scope the changes to the build_opener request, which is when the
+        # environment lookups happen.
+        newenv = {}
+        for name in bb.fetch2.FETCH_EXPORT_VARS:
+            value = d.getVar(name) or d.getVar("BB_ORIGENV").getVar(name)
+            if value:
+                newenv[name] = value
+
+        with bb.utils.environment(**newenv):
+            import ssl
+
+            context = ssl._create_unverified_context()
+            handlers = [FixedHTTPRedirectHandler,
+                        HTTPMethodFallback,
+                        urllib.request.ProxyHandler(),
+                        CacheHTTPHandler(),
+                        urllib.request.HTTPSHandler(context=context)]
+            opener = urllib.request.build_opener(*handlers)
 
         try:
             uri = ud.url.split(";")[0]
-- 
2.25.1

[PATCH v3 4/4] fetch2/wget: fetch securely by default

[Ross Burton]

From: Ross Burton <ross@burtonini.com>

The days of broken certificates are behind us now, so instead of always
passing --no-check-certificate to wget, don't pass it by default and
instead only pass it BB_CHECK_SSL_CERTS = "0".

[ YOCTO #14108 ]

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 .../bitbake-user-manual-fetching.rst          |  4 ++++
 .../bitbake-user-manual-ref-variables.rst     |  4 ++++
 bitbake/lib/bb/fetch2/wget.py                 | 19 ++++++++++++++++---
 3 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 593de61f242..40b245b6d30 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -144,6 +144,10 @@ download without a checksum triggers an error message. The
 make any attempted network access a fatal error, which is useful for
 checking that mirrors are complete as well as other things.
 
+If :term:`BB_CHECK_SSL_CERTS` is set to ``0`` then SSL certificate checking will
+be disabled. This variable defaults to ``1`` so SSL certificates are normally
+checked.
+
 .. _bb-the-unpack:
 
 The Unpack
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
index 6283c2654c8..2392ec42563 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
@@ -93,6 +93,10 @@ overview of their function and contents.
       fetcher does not attempt to use the host listed in :term:`SRC_URI` after
       a successful fetch from the :term:`PREMIRRORS` occurs.
 
+   :term:`BB_CHECK_SSL_CERTS`
+      Specifies if SSL certificates should be checked when fetching. The default
+      value is ``1`` and certificates are not checked if the value is set to ``0``.
+
    :term:`BB_CONSOLELOG`
       Specifies the path to a log file into which BitBake's user interface
       writes output during the build.
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index d67f9b889cc..81c377ae6f2 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -52,13 +52,19 @@ class WgetProgressHandler(bb.progress.LineFilterProgressHandler):
 
 
 class Wget(FetchMethod):
+    """Class to fetch urls via 'wget'"""
 
     # CDNs like CloudFlare may do a 'browser integrity test' which can fail
     # with the standard wget/urllib User-Agent, so pretend to be a modern
     # browser.
     user_agent = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
 
-    """Class to fetch urls via 'wget'"""
+    def check_certs(self, d):
+        """
+        Should certificates be checked?
+        """
+        return (d.getVar("BB_CHECK_SSL_CERTS") or "1") != "0"
+
     def supports(self, ud, d):
         """
         Check to see if a given url can be fetched with wget.
@@ -82,7 +88,10 @@ class Wget(FetchMethod):
         if not ud.localfile:
             ud.localfile = d.expand(urllib.parse.unquote(ud.host + ud.path).replace("/", "."))
 
-        self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate"
+        self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t 2 -T 30 --passive-ftp"
+
+        if not self.check_certs(d):
+            self.basecmd += " --no-check-certificate"
 
     def _runwget(self, ud, d, command, quiet, workdir=None):
 
@@ -305,7 +314,11 @@ class Wget(FetchMethod):
         with bb.utils.environment(**newenv):
             import ssl
 
-            context = ssl._create_unverified_context()
+            if self.check_certs(d):
+                context = ssl.create_default_context()
+            else:
+                context = ssl._create_unverified_context()
+
             handlers = [FixedHTTPRedirectHandler,
                         HTTPMethodFallback,
                         urllib.request.ProxyHandler(),
-- 
2.25.1

Re: [bitbake-devel] [PATCH v3 4/4] fetch2/wget: fetch securely by default

[Michael Opdenacker]

Hi Ross,

Thanks for the patches!

On 8/10/21 6:55 PM, Ross Burton wrote:
> From: Ross Burton <ross@burtonini.com>
>
> The days of broken certificates are behind us now, so instead of always
> passing --no-check-certificate to wget, don't pass it by default and
> instead only pass it BB_CHECK_SSL_CERTS = "0".
>
> [ YOCTO #14108 ]
>
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  .../bitbake-user-manual-fetching.rst          |  4 ++++
>  .../bitbake-user-manual-ref-variables.rst     |  4 ++++
>  bitbake/lib/bb/fetch2/wget.py                 | 19 ++++++++++++++++---
>  3 files changed, 24 insertions(+), 3 deletions(-)
>
> diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst

This patch looks good, but your patches seem to apply to the Poky repo,
not BitBake. Wouldn't be easier for Richard if you submitted patches for
the BitBake repo instead? That's where the patches need to go, if I
understood correctly. Unless there's a workflow I'm not familiar with...

Cheers,
Michael.
-- 

Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [bitbake-devel] [PATCH v3 4/4] fetch2/wget: fetch securely by default

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel@lists.openembedded.org <bitbake-
> devel@lists.openembedded.org> On Behalf Of Michael Opdenacker
> Sent: den 11 augusti 2021 15:46
> To: Ross Burton <ross@burtonini.com>; bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH v3 4/4] fetch2/wget: fetch securely by
> default
> 
> Hi Ross,
> 
> Thanks for the patches!
> 
> On 8/10/21 6:55 PM, Ross Burton wrote:
> > From: Ross Burton <ross@burtonini.com>
> >
> > The days of broken certificates are behind us now, so instead of always
> > passing --no-check-certificate to wget, don't pass it by default and
> > instead only pass it BB_CHECK_SSL_CERTS = "0".
> >
> > [ YOCTO #14108 ]
> >
> > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > ---
> >  .../bitbake-user-manual-fetching.rst          |  4 ++++
> >  .../bitbake-user-manual-ref-variables.rst     |  4 ++++
> >  bitbake/lib/bb/fetch2/wget.py                 | 19 ++++++++++++++++---
> >  3 files changed, 24 insertions(+), 3 deletions(-)
> >
> > diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-
> fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-
> fetching.rst
> 
> This patch looks good, but your patches seem to apply to the Poky repo,
> not BitBake. Wouldn't be easier for Richard if you submitted patches for
> the BitBake repo instead? That's where the patches need to go, if I
> understood correctly. Unless there's a workflow I'm not familiar with...
> 
> Cheers,
> Michael.

For those of us that actually use Poky, it is a lot easier to generate patches 
vs Poky, than the underlying repositories (as they are not available). This is 
normally not a problem, and RP handles it. The only caveat is to remember to 
split the patches and send them to different mail addresses when a bigger change 
requires changes to both OE-Core and Bitbake.

//Peter

Re: [bitbake-devel] [PATCH v3 4/4] fetch2/wget: fetch securely by default

[Ross Burton]

In my experience git has been clever enough to notice that the base is
different and will apply the patches anyway.

Ross

On Wed, 11 Aug 2021 at 14:46, Michael Opdenacker
<michael.opdenacker@bootlin.com> wrote:
>
> Hi Ross,
>
> Thanks for the patches!
>
> On 8/10/21 6:55 PM, Ross Burton wrote:
> > From: Ross Burton <ross@burtonini.com>
> >
> > The days of broken certificates are behind us now, so instead of always
> > passing --no-check-certificate to wget, don't pass it by default and
> > instead only pass it BB_CHECK_SSL_CERTS = "0".
> >
> > [ YOCTO #14108 ]
> >
> > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > ---
> >  .../bitbake-user-manual-fetching.rst          |  4 ++++
> >  .../bitbake-user-manual-ref-variables.rst     |  4 ++++
> >  bitbake/lib/bb/fetch2/wget.py                 | 19 ++++++++++++++++---
> >  3 files changed, 24 insertions(+), 3 deletions(-)
> >
> > diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
>
> This patch looks good, but your patches seem to apply to the Poky repo,
> not BitBake. Wouldn't be easier for Richard if you submitted patches for
> the BitBake repo instead? That's where the patches need to go, if I
> understood correctly. Unless there's a workflow I'm not familiar with...
>
> Cheers,
> Michael.
> --
>
> Michael Opdenacker, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>

Re: [bitbake-devel] [PATCH v3 4/4] fetch2/wget: fetch securely by default

[Richard Purdie]

On Fri, 2021-08-13 at 17:41 +0100, Ross Burton wrote:
> On Wed, 11 Aug 2021 at 14:46, Michael Opdenacker
> <michael.opdenacker@bootlin.com> wrote:
> > 
> > Hi Ross,
> > 
> > Thanks for the patches!
> > 
> > On 8/10/21 6:55 PM, Ross Burton wrote:
> > > From: Ross Burton <ross@burtonini.com>
> > > 
> > > The days of broken certificates are behind us now, so instead of always
> > > passing --no-check-certificate to wget, don't pass it by default and
> > > instead only pass it BB_CHECK_SSL_CERTS = "0".
> > > 
> > > [ YOCTO #14108 ]
> > > 
> > > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > > ---
> > >  .../bitbake-user-manual-fetching.rst          |  4 ++++
> > >  .../bitbake-user-manual-ref-variables.rst     |  4 ++++
> > >  bitbake/lib/bb/fetch2/wget.py                 | 19 ++++++++++++++++---
> > >  3 files changed, 24 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
> > 
> > This patch looks good, but your patches seem to apply to the Poky repo,
> > not BitBake. Wouldn't be easier for Richard if you submitted patches for
> > the BitBake repo instead? That's where the patches need to go, if I
> > understood correctly. Unless there's a workflow I'm not familiar with...
>
> In my experience git has been clever enough to notice that the base is
> different and will apply the patches anyway.

Sadly not, I just fix this stuff up locally and deal with it. If I were to
do anything more, I'll get told the 101 other ways we should be doing it 
and/or put people off sending patches and we really don't have the bandwidth 
to change right now.

Cheers,

Richard

Re: [bitbake-devel] [PATCH v3 3/4] fetch2/wget: ensure all variables are set when calling urllib

[Enrico Scholz]

"Ross Burton" <ross@burtonini.com> writes:

> Instead of just exporting the proxy variables when calling into urllib,
> use bb.utils.environment() to export all of the known variables that are
> needed for proper connectivity.

This patch breaks fetching from sstate servers[1].

We have

|  PREMIRRORS="\
|    ftp://.*/.*    https://sstate.intern.sigma-chemnitz.de:30011/api/v1/download/irdjqumJ5ImHIGG5/source/defaultpkgname/ \n
|    https?://.*/.* https://sstate.intern.sigma-chemnitz.de:30011/api/v1/download/irdjqumJ5ImHIGG5/source/defaultpkgname/ \n
|    git://.*/.*    https://sstate.intern.sigma-chemnitz.de:30011/api/v1/download/irdjqumJ5ImHIGG5/scm/defaultpkgname/INVALID \n"

      [lines splitted for readability]
      [there is no secret in the log... token is not reusable]

and

| https_proxy=http://www-cache:3128/
| no_proxy='.sigma-chemnitz.de,sigma-chemnitz.de,cvg.de,.cvg.de,andto.de,.andto.de,localhost,<local>'

Before this patch, the sstate server was connected directly but now, the
proxy is tried.


I can reproduce it here e.g. by

| $ bitbake gettext-minimal-native -ccleansstate
| $ bitbake gettext-minimal-native
| ERROR: gettext-minimal-native-0.21-r0 do_populate_lic_setscene: No suitable staging package found
|
| DEBUG: checkstatus() urlopen failed: <urlopen error Tunnel connection failed: 403 Forbidden>

  (proxy logs show a 'CONNECT' to 'sstate.intern.sigma-chemnitz.de'
   which is reject with 403)



'git bisect' points to this commit.



Enrico

Footnotes:
[1]  fwiw, https://gitlab-ext.sigma-chemnitz.de/ensc/sstate-server

Re: [bitbake-devel] [PATCH v3 3/4] fetch2/wget: ensure all variables are set when calling urllib

[Enrico Scholz]

"Enrico Scholz via lists.openembedded.org"
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> writes:

>> Instead of just exporting the proxy variables when calling into urllib,
>> use bb.utils.environment() to export all of the known variables that are
>> needed for proper connectivity.
>
> This patch breaks fetching from sstate servers[1].
>
> We have
>
> |  PREMIRRORS="\
> | ftp://.*/.*
> https://sstate.intern.sigma-chemnitz.de:30011/api/v1/download/irdjqumJ5ImHIGG5/source/defaultpkgname/

wrong variable;

| SSTATE_MIRRORS="    file://.*    https://sstate.intern.sigma-chemnitz.de:30011/api/v1/download/irdjqumJ5ImHIGG5/sstate/defaultpkgname/PATH "

is the correct one



Enrico

Re: [bitbake-devel] [PATCH v3 3/4] fetch2/wget: ensure all variables are set when calling urllib

[Enrico Scholz]

"Enrico Scholz via lists.openembedded.org"
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> writes:

> "Ross Burton" <ross@burtonini.com> writes:
>
>> Instead of just exporting the proxy variables when calling into urllib,
>> use bb.utils.environment() to export all of the known variables that are
>> needed for proper connectivity.
>
> This patch breaks fetching from sstate servers[1].
> 
> | https_proxy=http://www-cache:3128/
> | no_proxy='.sigma-chemnitz.de,...'
> 
> I can reproduce it here e.g. by
>
> | $ bitbake gettext-minimal-native -ccleansstate
> | $ bitbake gettext-minimal-native
> | ERROR: gettext-minimal-native-0.21-r0 do_populate_lic_setscene: No suitable staging package found
> |
> | DEBUG: checkstatus() urlopen failed: <urlopen error Tunnel connection failed: 403 Forbidden>

I guess this happens because in

| +        with bb.utils.environment(**newenv):
| +            handlers = [...
| +                        urllib.request.ProxyHandler(),

the urllib.request.ProxyHandler constructor takes 'https_proxy' from
environment (but not 'no_proxy').

Real open happens later (when the 'with bb.utils.environment...' has
been left) and there,

| class ProxyHandler(BaseHandler):
|     def proxy_open(self, req, proxy, type):
|         ...
|         if req.host and proxy_bypass(req.host):
|             return None

is called.  'proxy_bypass()' checks 'no_proxy' from env which is not
available anymore.


Normale fetching does not seem to be affected because proxy vars are in
BB_ENV_EXTRAWHITE and exported to jobs.  But 'sstate' fetching does not
seem to use this mechanism.



Enrico

Re: [bitbake-devel] [PATCH v3 3/4] fetch2/wget: ensure all variables are set when calling urllib

[Enrico Scholz]

"Enrico Scholz via lists.openembedded.org"
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> writes:

> "Ross Burton" <ross@burtonini.com> writes:
>
>> Instead of just exporting the proxy variables when calling into urllib,
>> use bb.utils.environment() to export all of the known variables that are
>> needed for proper connectivity.
>
> This patch breaks fetching from sstate servers[1].

| --- a/bitbake/lib/bb/fetch2/wget.py
| +++ b/bitbake/lib/bb/fetch2/wget.py
| ...
| +        with bb.utils.environment(**newenv):
| +            import ssl
|          ...
|          try:
|              uri = ud.url.split(";")[0]

Moving this 'try' block (with the `opener.open(r)` call) into the 'with'
context seems to solve the problem.



Enrico

Re: [bitbake-devel] [PATCH v3 3/4] fetch2/wget: ensure all variables are set when calling urllib

[Ross Burton]

As you've chased this, please send a patch.  Apologies, I thought I
checked that all the variables were already checked by the time the
with block closes.

Ross

On Wed, 18 Aug 2021 at 10:28, Enrico Scholz
<enrico.scholz@sigma-chemnitz.de> wrote:
>
> "Enrico Scholz via lists.openembedded.org"
> <enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> writes:
>
> > "Ross Burton" <ross@burtonini.com> writes:
> >
> >> Instead of just exporting the proxy variables when calling into urllib,
> >> use bb.utils.environment() to export all of the known variables that are
> >> needed for proper connectivity.
> >
> > This patch breaks fetching from sstate servers[1].
>
> | --- a/bitbake/lib/bb/fetch2/wget.py
> | +++ b/bitbake/lib/bb/fetch2/wget.py
> | ...
> | +        with bb.utils.environment(**newenv):
> | +            import ssl
> |          ...
> |          try:
> |              uri = ud.url.split(";")[0]
>
> Moving this 'try' block (with the `opener.open(r)` call) into the 'with'
> context seems to solve the problem.
>
>
>
> Enrico