bitbake-devel.lists.openembedded.org archive mirror
 help / color / mirror / Atom feed
* [bitbake-devel][PATCH 0/2] Add fetch2 wget exception handler
@ 2023-03-09 22:52 Mark Hatle
  2023-03-09 22:52 ` [bitbake-devel][PATCH 1/2] wget.py: Add catch TimeoutError exception Mark Hatle
  2023-03-09 22:52 ` [bitbake-devel][PATCH 2/2] wget.py: Combine urlopener exceptions Mark Hatle
  0 siblings, 2 replies; 3+ messages in thread
From: Mark Hatle @ 2023-03-09 22:52 UTC (permalink / raw)
  To: bitbake-devel

While rare we were getting TimeoutError exceptions in our CI infrastructure.
It is unclear what the cause is (still being investigated), but from what
I can tell, it appears connection opens, we do a number of invalid requests
and the remote side just drops further requests (not closing the connection)
which results in a socket TimeoutError.

Backtrace we receieved follows:

Checking sstate mirror object availability...ERROR: SState: cannot test file://13/8b/sstate:base-passwd:cortexa72-cortexa53-xilinx-linux:3.5.52:r0:cortexa72-cortexa53:10:138b29e1fe5981198dbca2fecdef5712826e63f2593e4d604ef54d6d5d51bd6d_package.tar.zst: TimeoutError('timed out')
Traceback (most recent call last):
  File "/poky/meta/classes-global/sstate.bbclass", line 1010, in checkstatus
    fetcher.checkstatus()
  File "/poky/bitbake/lib/bb/fetch2/__init__.py", line 1833, in checkstatus
    ret = m.try_mirrors(self, ud, self.d, mirrors, True)
  File "/poky/bitbake/lib/bb/fetch2/__init__.py", line 1606, in try_mirrors
    return bool(try_mirrors(fetch, d, urldata, mirrors, check))
  File "/poky/bitbake/lib/bb/fetch2/__init__.py", line 1134, in try_mirrors
    ret = try_mirror_url(fetch, origud, uds[index], ld, check)
  File "/poky/bitbake/lib/bb/fetch2/__init__.py", line 1027, in try_mirror_url
    found = ud.method.checkstatus(fetch, ud, ld)
  File "/poky/bitbake/lib/bb/fetch2/wget.py", line 369, in checkstatus
    with opener.open(r, timeout=30) as response:
  File "/buildtools/sysroots/x86_64-petalinux-linux/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/buildtools/sysroots/x86_64-petalinux-linux/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/buildtools/sysroots/x86_64-petalinux-linux/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/poky/bitbake/lib/bb/fetch2/wget.py", line 174, in http_open
    return self.do_open(HTTPConnectionCache, req)
  File "/poky/bitbake/lib/bb/fetch2/wget.py", line 244, in do_open
    r = h.getresponse()
  File "/buildtools/sysroots/x86_64-petalinux-linux/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/buildtools/sysroots/x86_64-petalinux-linux/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/buildtools/sysroots/x86_64-petalinux-linux/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/buildtools/sysroots/x86_64-petalinux-linux/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
TimeoutError: timed out


Mark Hatle (2):
  wget.py: Add catch TimeoutError exception
  wget.py: Combine urlopener exceptions

 bitbake/lib/bb/fetch2/wget.py | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

-- 
2.25.1



^ permalink raw reply	[flat|nested] 3+ messages in thread

* [bitbake-devel][PATCH 1/2] wget.py: Add catch TimeoutError exception
  2023-03-09 22:52 [bitbake-devel][PATCH 0/2] Add fetch2 wget exception handler Mark Hatle
@ 2023-03-09 22:52 ` Mark Hatle
  2023-03-09 22:52 ` [bitbake-devel][PATCH 2/2] wget.py: Combine urlopener exceptions Mark Hatle
  1 sibling, 0 replies; 3+ messages in thread
From: Mark Hatle @ 2023-03-09 22:52 UTC (permalink / raw)
  To: bitbake-devel

We've observed TimeoutError exceptions during the sstate-cache mirror fetch,
it appears that due to the number of (invalid) files requested the remote
side is eventually dropping the connection (not closing it) which can result
in a TimeoutError exception being sent, while rate it is different from the
urllib.error.URLError or ConnectionResetError.

Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
---
 bitbake/lib/bb/fetch2/wget.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 859b4f99e2d..927487a0b66 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -385,6 +385,14 @@ class Wget(FetchMethod):
                     # debug for now to avoid spamming the logs in e.g. remote sstate searches
                     logger.debug2("checkstatus() urlopen failed: %s" % e)
                     return False
+            except TimeoutError as e:
+                if try_again:
+                    logger.debug2("checkstatus: trying again")
+                    return self.checkstatus(fetch, ud, d, False)
+                else:
+                    # debug for now to avoid spamming the logs in e.g. remote sstate searches
+                    logger.debug2("checkstatus() urlopen TimeoutError: %s" % e)
+                    return False
 
         return True
 
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 3+ messages in thread

* [bitbake-devel][PATCH 2/2] wget.py: Combine urlopener exceptions
  2023-03-09 22:52 [bitbake-devel][PATCH 0/2] Add fetch2 wget exception handler Mark Hatle
  2023-03-09 22:52 ` [bitbake-devel][PATCH 1/2] wget.py: Add catch TimeoutError exception Mark Hatle
@ 2023-03-09 22:52 ` Mark Hatle
  1 sibling, 0 replies; 3+ messages in thread
From: Mark Hatle @ 2023-03-09 22:52 UTC (permalink / raw)
  To: bitbake-devel

No reason to have three identical exception handles, refactor to catch any
of the exceptions with the same block of code.

Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
---
 bitbake/lib/bb/fetch2/wget.py | 18 +-----------------
 1 file changed, 1 insertion(+), 17 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 927487a0b66..dc883176104 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -369,7 +369,7 @@ class Wget(FetchMethod):
 
                 with opener.open(r, timeout=30) as response:
                     pass
-            except urllib.error.URLError as e:
+            except (urllib.error.URLError, ConnectionResetError, TimeoutError) as e:
                 if try_again:
                     logger.debug2("checkstatus: trying again")
                     return self.checkstatus(fetch, ud, d, False)
@@ -377,22 +377,6 @@ class Wget(FetchMethod):
                     # debug for now to avoid spamming the logs in e.g. remote sstate searches
                     logger.debug2("checkstatus() urlopen failed: %s" % e)
                     return False
-            except ConnectionResetError as e:
-                if try_again:
-                    logger.debug2("checkstatus: trying again")
-                    return self.checkstatus(fetch, ud, d, False)
-                else:
-                    # debug for now to avoid spamming the logs in e.g. remote sstate searches
-                    logger.debug2("checkstatus() urlopen failed: %s" % e)
-                    return False
-            except TimeoutError as e:
-                if try_again:
-                    logger.debug2("checkstatus: trying again")
-                    return self.checkstatus(fetch, ud, d, False)
-                else:
-                    # debug for now to avoid spamming the logs in e.g. remote sstate searches
-                    logger.debug2("checkstatus() urlopen TimeoutError: %s" % e)
-                    return False
 
         return True
 
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2023-03-09 22:52 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-09 22:52 [bitbake-devel][PATCH 0/2] Add fetch2 wget exception handler Mark Hatle
2023-03-09 22:52 ` [bitbake-devel][PATCH 1/2] wget.py: Add catch TimeoutError exception Mark Hatle
2023-03-09 22:52 ` [bitbake-devel][PATCH 2/2] wget.py: Combine urlopener exceptions Mark Hatle

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).