All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jose Quaresma <quaresma.jose@gmail.com>
To: Jose Quaresma <quaresma.jose@gmail.com>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>,
	OE-core <openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core] [PATCH 1/2] sstate: add a LockedSet class to be used on the mirror thread pool
Date: Fri, 29 Apr 2022 14:37:56 +0100	[thread overview]
Message-ID: <CANPvuRnY6N0BwD-=ttTyCwTNTwvXLzx=TR-9xJo2WnAvR_KoYQ@mail.gmail.com> (raw)
In-Reply-To: <16EA614949E42DAD.4798@lists.openembedded.org>

[-- Attachment #1: Type: text/plain, Size: 10661 bytes --]

Jose Quaresma via lists.openembedded.org <quaresma.jose=
gmail.com@lists.openembedded.org> escreveu no dia sexta, 29/04/2022 à(s)
14:33:

>
>
> Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia
> quinta, 28/04/2022 à(s) 13:30:
>
>> On Tue, 2022-04-19 at 10:46 +0100, Jose Quaresma wrote:
>> > The python set() is not thread safe and we use it on the ThreadedPool.
>> > With this LockedSet python class we can call the 'add' and 'remove'
>> > safely inside the ThreadedPool.
>> >
>> > This piece of code is taken from the stackoverflow
>> >
>> https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe
>> >
>> > May be related with [YOCTO #14775] --
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775
>> >
>> > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
>>
>> For info,
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5083
>> was a failure with master-next which included the changes in this
>> discussion. It
>> doesn't seem to address that issue :(
>>
>> Cheers,
>>
>> Richard
>>
>>
> Unfortunately this solution still doesn't solve the problem.
>
> I'm going to analyze the log to try to understand what's going on,
> however I leave here one of the tracebacks in case anyone can see an
> anomaly here.
>
> The mirror thread pool is considerably different in this patch series
> which may imply that the problem may be on the server side but it is still
> too early to draw conclusions.
>
> I still haven't had time to think about how to do a test for this case.
>
> Jose
>
> ERROR: SState: cannot test
> file://universal/f3/69/sstate:python3-wheel-native:x86_64-linux:0.37.1:r0:x86_64:8:f36956da8ee13cebb1fb6df615d0eb548a4bf637c50021c213d8c6f0973674bf_deploy_source_date_epoch.tar.zst:
> TimeoutError('timed out')
> Traceback (most recent call last):
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/meta/classes/sstate.bbclass",
> line 1023, in checkstatus
> fetcher.checkstatus()
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py",
> line 1829, in checkstatus
> ret = m.try_mirrors(self, ud, self.d, mirrors, True)
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py",
> line 1602, in try_mirrors
> return bool(try_mirrors(fetch, d, urldata, mirrors, check))
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py",
> line 1128, in try_mirrors
> ret = try_mirror_url(fetch, origud, uds[index], ld, check)
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py",
> line 1023, in try_mirror_url
> found = ud.method.checkstatus(fetch, ud, ld)
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py",
> line 361, in checkstatus
> with opener.open(r, timeout=30) as response:
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py",
> line 519, in open
> response = self._open(req, data)
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py",
> line 536, in _open
> result = self._call_chain(self.handle_open, protocol, protocol +
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py",
> line 496, in _call_chain
> result = func(*args)
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py",
> line 166, in http_open
> return self.do_open(HTTPConnectionCache, req)
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py",
> line 236, in do_open
> r = h.getresponse()
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py",
> line 1374, in getresponse
> response.begin()
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py",
> line 318, in begin
> version, status, reason = self._read_status()
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py",
> line 279, in _read_status
> line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
> File
> "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/socket.py",
> line 705, in readinto
> return self._sock.recv_into(b)
> TimeoutError: timed out
>
>
Latest tracebacks are messed up so here is a copy of it:


ERROR: SState: cannot test
file://universal/f3/69/sstate:python3-wheel-native:x86_64-linux:0.37.1:r0:x86_64:8:f36956da8ee13cebb1fb6df615d0eb548a4bf637c50021c213d8c6f0973674bf_deploy_source_date_epoch.tar.zst:
TimeoutError('timed out')
Traceback (most recent call last):
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/meta/classes/sstate.bbclass",
line 1023, in checkstatus
    fetcher.checkstatus()
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py",
line 1829, in checkstatus
    ret = m.try_mirrors(self, ud, self.d, mirrors, True)
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py",
line 1602, in try_mirrors
    return bool(try_mirrors(fetch, d, urldata, mirrors, check))
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py",
line 1128, in try_mirrors
    ret = try_mirror_url(fetch, origud, uds[index], ld, check)
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py",
line 1023, in try_mirror_url
    found = ud.method.checkstatus(fetch, ud, ld)
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py",
line 361, in checkstatus
    with opener.open(r, timeout=30) as response:
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py",
line 519, in open
    response = self._open(req, data)
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py",
line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py",
line 496, in _call_chain
    result = func(*args)
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py",
line 166, in http_open
    return self.do_open(HTTPConnectionCache, req)
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py",
line 236, in do_open
    r = h.getresponse()
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py",
line 1374, in getresponse
    response.begin()
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py",
line 318, in begin
    version, status, reason = self._read_status()
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py",
line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File
"/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/socket.py",
line 705, in readinto
    return self._sock.recv_into(b)
TimeoutError: timed out


> --
> Best regards,
>
> José Quaresma
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#165013):
> https://lists.openembedded.org/g/openembedded-core/message/165013
> Mute This Topic: https://lists.openembedded.org/mt/90558156/5052612
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> quaresma.jose@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

-- 
Best regards,

José Quaresma

[-- Attachment #2: Type: text/html, Size: 22883 bytes --]

      parent reply	other threads:[~2022-04-29 13:38 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-04-19  9:46 [PATCH 1/2] sstate: add a LockedSet class to be used on the mirror thread pool Jose Quaresma
2022-04-19  9:46 ` [PATCH 2/2] sstate: reuse the localdata copy " Jose Quaresma
2022-04-19 10:57   ` [OE-core] " Richard Purdie
2022-04-19 11:55     ` Jose Quaresma
2022-04-28 12:30 ` [OE-core] [PATCH 1/2] sstate: add a LockedSet class to be used " Richard Purdie
2022-04-29 13:32   ` Jose Quaresma
     [not found]   ` <16EA614949E42DAD.4798@lists.openembedded.org>
2022-04-29 13:37     ` Jose Quaresma [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CANPvuRnY6N0BwD-=ttTyCwTNTwvXLzx=TR-9xJo2WnAvR_KoYQ@mail.gmail.com' \
    --to=quaresma.jose@gmail.com \
    --cc=openembedded-core@lists.openembedded.org \
    --cc=richard.purdie@linuxfoundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.