* [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter @ 2021-09-19 20:51 Jose Quaresma 2021-09-19 20:51 ` [PATCH 2/2] sstate.bbclass: cleanup the sstate mirror progress check Jose Quaresma 2021-09-21 10:43 ` [OE-core] [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter Alexandre Belloni 0 siblings, 2 replies; 5+ messages in thread From: Jose Quaresma @ 2021-09-19 20:51 UTC (permalink / raw) To: openembedded-core; +Cc: Jose Quaresma We don't need extra python collections to count the found files on the sstate cache and sstate mirrors. The main found collections provides all the files that were found, then we only need to count the files on sstate mirror Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> --- meta/classes/sstate.bbclass | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 29679e6a5e..fc156ac81a 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -893,8 +893,6 @@ BB_HASHCHECK_FUNCTION = "sstate_checkhashes" def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs): found = set() - foundLocal = set() - foundNet = set() missed = set() def gethash(task): @@ -927,12 +925,12 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, if os.path.exists(sstatefile): bb.debug(2, "SState: Found valid sstate file %s" % sstatefile) found.add(tid) - foundLocal.add(tid) continue else: - missed.add(tid) bb.debug(2, "SState: Looked for but didn't find file %s" % sstatefile) + missed.add(tid) + foundMirrors = 0 mirrors = d.getVar("SSTATE_MIRRORS") if mirrors: # Copy the data object and override DL_DIR and SRC_URI @@ -972,8 +970,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, connection_cache=thread_worker.connection_cache) fetcher.checkstatus() bb.debug(2, "SState: Successful fetch test for %s" % srcuri) + foundMirrors += 1 found.add(tid) - foundNet.add(tid) if tid in missed: missed.remove(tid) except: @@ -1034,7 +1032,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, match = 0 if total: match = len(found) / total * 100 - bb.plain("Sstate summary: Wanted %d Local %d Network %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(foundLocal), len(foundNet),len(missed), currentcount, match, complete)) + bb.plain("Sstate summary: Wanted %d Local %d Mirrors %d Missed %d Current %d (%d%% match, %d%% complete)" % + (total, len(found)-foundMirrors, foundMirrors, len(missed), currentcount, match, complete)) if hasattr(bb.parse.siggen, "checkhashes"): bb.parse.siggen.checkhashes(sq_data, missed, found, d) -- 2.33.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] sstate.bbclass: cleanup the sstate mirror progress check 2021-09-19 20:51 [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter Jose Quaresma @ 2021-09-19 20:51 ` Jose Quaresma 2021-09-21 10:43 ` [OE-core] [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter Alexandre Belloni 1 sibling, 0 replies; 5+ messages in thread From: Jose Quaresma @ 2021-09-19 20:51 UTC (permalink / raw) To: openembedded-core; +Cc: Jose Quaresma We only has the progress bar when we have more than 100 objects. So check for this and store the result to show the progress bar. Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> --- meta/classes/sstate.bbclass | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index fc156ac81a..1be20e4055 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -978,11 +978,11 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, missed.add(tid) bb.debug(2, "SState: Unsuccessful fetch test for %s" % srcuri) pass - if len(tasklist) >= min_tasks: + + if progress: bb.event.fire(bb.event.ProcessProgress(msg, len(tasklist) - thread_worker.tasks.qsize()), d) tasklist = [] - min_tasks = 100 for tid in sq_data['hash']: if tid in found: continue @@ -993,7 +993,11 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, if tasklist: nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist)) - if len(tasklist) >= min_tasks: + progress = False + if len(tasklist) >= 100: + progress = True + + if progress: msg = "Checking sstate mirror object availability" bb.event.fire(bb.event.ProcessStarted(msg, len(tasklist)), d) @@ -1006,7 +1010,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, pool.wait_completion() bb.event.disable_threadlock() - if len(tasklist) >= min_tasks: + if progress: bb.event.fire(bb.event.ProcessFinished(msg), d) inheritlist = d.getVar("INHERIT") -- 2.33.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter 2021-09-19 20:51 [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter Jose Quaresma 2021-09-19 20:51 ` [PATCH 2/2] sstate.bbclass: cleanup the sstate mirror progress check Jose Quaresma @ 2021-09-21 10:43 ` Alexandre Belloni 2021-09-21 14:43 ` Jose Quaresma 2021-10-03 22:01 ` Jose Quaresma 1 sibling, 2 replies; 5+ messages in thread From: Alexandre Belloni @ 2021-09-21 10:43 UTC (permalink / raw) To: Jose Quaresma; +Cc: openembedded-core Hello Jose, On 19/09/2021 21:51:11+0100, Jose Quaresma wrote: > We don't need extra python collections to count the found files > on the sstate cache and sstate mirrors. > The main found collections provides all the files that were found, > then we only need to count the files on sstate mirror > I believe this is the cause of: https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/3999 https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/4039 https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/4019 https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/3338 https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/4012 https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/4004 https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/4009 https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/4004 I removed both patches and all the builds passed > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> > --- > meta/classes/sstate.bbclass | 11 +++++------ > 1 file changed, 5 insertions(+), 6 deletions(-) > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass > index 29679e6a5e..fc156ac81a 100644 > --- a/meta/classes/sstate.bbclass > +++ b/meta/classes/sstate.bbclass > @@ -893,8 +893,6 @@ BB_HASHCHECK_FUNCTION = "sstate_checkhashes" > > def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs): > found = set() > - foundLocal = set() > - foundNet = set() > missed = set() > > def gethash(task): > @@ -927,12 +925,12 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, > if os.path.exists(sstatefile): > bb.debug(2, "SState: Found valid sstate file %s" % sstatefile) > found.add(tid) > - foundLocal.add(tid) > continue > else: > - missed.add(tid) > bb.debug(2, "SState: Looked for but didn't find file %s" % sstatefile) > + missed.add(tid) > > + foundMirrors = 0 > mirrors = d.getVar("SSTATE_MIRRORS") > if mirrors: > # Copy the data object and override DL_DIR and SRC_URI > @@ -972,8 +970,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, > connection_cache=thread_worker.connection_cache) > fetcher.checkstatus() > bb.debug(2, "SState: Successful fetch test for %s" % srcuri) > + foundMirrors += 1 > found.add(tid) > - foundNet.add(tid) > if tid in missed: > missed.remove(tid) > except: > @@ -1034,7 +1032,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, > match = 0 > if total: > match = len(found) / total * 100 > - bb.plain("Sstate summary: Wanted %d Local %d Network %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(foundLocal), len(foundNet),len(missed), currentcount, match, complete)) > + bb.plain("Sstate summary: Wanted %d Local %d Mirrors %d Missed %d Current %d (%d%% match, %d%% complete)" % > + (total, len(found)-foundMirrors, foundMirrors, len(missed), currentcount, match, complete)) > > if hasattr(bb.parse.siggen, "checkhashes"): > bb.parse.siggen.checkhashes(sq_data, missed, found, d) > -- > 2.33.0 > > > > -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter 2021-09-21 10:43 ` [OE-core] [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter Alexandre Belloni @ 2021-09-21 14:43 ` Jose Quaresma 2021-10-03 22:01 ` Jose Quaresma 1 sibling, 0 replies; 5+ messages in thread From: Jose Quaresma @ 2021-09-21 14:43 UTC (permalink / raw) To: Alexandre Belloni; +Cc: OE-core Hi Alexandre, I've had some problems in the past with this patchset and this time I removed what I considered the most problematic. I can't see the problems that are present here. Any help in the review is welcome. Jose Alexandre Belloni <alexandre.belloni@bootlin.com> escreveu no dia terça, 21/09/2021 à(s) 11:43: > > Hello Jose, > > On 19/09/2021 21:51:11+0100, Jose Quaresma wrote: > > We don't need extra python collections to count the found files > > on the sstate cache and sstate mirrors. > > The main found collections provides all the files that were found, > > then we only need to count the files on sstate mirror > > > > I believe this is the cause of: > https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/3999 > https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/4039 > https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/4019 > https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/3338 > https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/4012 > https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/4004 > https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/4009 > https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/4004 > > I removed both patches and all the builds passed > > > > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> > > --- > > meta/classes/sstate.bbclass | 11 +++++------ > > 1 file changed, 5 insertions(+), 6 deletions(-) > > > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass > > index 29679e6a5e..fc156ac81a 100644 > > --- a/meta/classes/sstate.bbclass > > +++ b/meta/classes/sstate.bbclass > > @@ -893,8 +893,6 @@ BB_HASHCHECK_FUNCTION = "sstate_checkhashes" > > > > def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs): > > found = set() > > - foundLocal = set() > > - foundNet = set() > > missed = set() > > > > def gethash(task): > > @@ -927,12 +925,12 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, > > if os.path.exists(sstatefile): > > bb.debug(2, "SState: Found valid sstate file %s" % sstatefile) > > found.add(tid) > > - foundLocal.add(tid) > > continue > > else: > > - missed.add(tid) > > bb.debug(2, "SState: Looked for but didn't find file %s" % sstatefile) > > + missed.add(tid) > > > > + foundMirrors = 0 > > mirrors = d.getVar("SSTATE_MIRRORS") > > if mirrors: > > # Copy the data object and override DL_DIR and SRC_URI > > @@ -972,8 +970,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, > > connection_cache=thread_worker.connection_cache) > > fetcher.checkstatus() > > bb.debug(2, "SState: Successful fetch test for %s" % srcuri) > > + foundMirrors += 1 > > found.add(tid) > > - foundNet.add(tid) > > if tid in missed: > > missed.remove(tid) > > except: > > @@ -1034,7 +1032,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, > > match = 0 > > if total: > > match = len(found) / total * 100 > > - bb.plain("Sstate summary: Wanted %d Local %d Network %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(foundLocal), len(foundNet),len(missed), currentcount, match, complete)) > > + bb.plain("Sstate summary: Wanted %d Local %d Mirrors %d Missed %d Current %d (%d%% match, %d%% complete)" % > > + (total, len(found)-foundMirrors, foundMirrors, len(missed), currentcount, match, complete)) > > > > if hasattr(bb.parse.siggen, "checkhashes"): > > bb.parse.siggen.checkhashes(sq_data, missed, found, d) > > -- > > 2.33.0 > > > > > > > > > > > > -- > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com -- Best regards, José Quaresma ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter 2021-09-21 10:43 ` [OE-core] [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter Alexandre Belloni 2021-09-21 14:43 ` Jose Quaresma @ 2021-10-03 22:01 ` Jose Quaresma 1 sibling, 0 replies; 5+ messages in thread From: Jose Quaresma @ 2021-10-03 22:01 UTC (permalink / raw) To: Alexandre Belloni; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 5005 bytes --] Hi Alexandre, Alexandre Belloni <alexandre.belloni@bootlin.com> escreveu no dia terça, 21/09/2021 à(s) 11:43: > Hello Jose, > > On 19/09/2021 21:51:11+0100, Jose Quaresma wrote: > > We don't need extra python collections to count the found files > > on the sstate cache and sstate mirrors. > > The main found collections provides all the files that were found, > > then we only need to count the files on sstate mirror > > > > I believe this is the cause of: > https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/3999 > https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/4039 > https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/4019 > https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/3338 > https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/4012 > https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/4004 > https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/4009 > https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/4004 > > I removed both patches and all the builds passed > > > > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> > > --- > > meta/classes/sstate.bbclass | 11 +++++------ > > 1 file changed, 5 insertions(+), 6 deletions(-) > > > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass > > index 29679e6a5e..fc156ac81a 100644 > > --- a/meta/classes/sstate.bbclass > > +++ b/meta/classes/sstate.bbclass > > @@ -893,8 +893,6 @@ BB_HASHCHECK_FUNCTION = "sstate_checkhashes" > > > > def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, > summary=True, **kwargs): > > found = set() > > - foundLocal = set() > > - foundNet = set() > > missed = set() > > > > def gethash(task): > > @@ -927,12 +925,12 @@ def sstate_checkhashes(sq_data, d, siginfo=False, > currentcount=0, summary=True, > > if os.path.exists(sstatefile): > > bb.debug(2, "SState: Found valid sstate file %s" % > sstatefile) > > found.add(tid) > > - foundLocal.add(tid) > > continue > > else: > > - missed.add(tid) > > bb.debug(2, "SState: Looked for but didn't find file %s" % > sstatefile) > > + missed.add(tid) > > > > + foundMirrors = 0 > > mirrors = d.getVar("SSTATE_MIRRORS") > > if mirrors: > > # Copy the data object and override DL_DIR and SRC_URI > > @@ -972,8 +970,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, > currentcount=0, summary=True, > > > connection_cache=thread_worker.connection_cache) > > fetcher.checkstatus() > > bb.debug(2, "SState: Successful fetch test for %s" % > srcuri) > > + foundMirrors += 1 > After looking again at that patch I think the problem may be here as it increments the counter inside of the thread pool without any lock. I sent a new version that uses a different approach to count the files that was found on sstate mirror. https://lists.openembedded.org/g/openembedded-core/topic/patch_sstate_bbclass_count/86052575?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,0,86052575,previd=1633298061762412833,nextid=1633195213330301948&previd=1633298061762412833&nextid=1633195213330301948 > > found.add(tid) > > - foundNet.add(tid) > > if tid in missed: > > missed.remove(tid) > > except: > > @@ -1034,7 +1032,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, > currentcount=0, summary=True, > > match = 0 > > if total: > > match = len(found) / total * 100 > > - bb.plain("Sstate summary: Wanted %d Local %d Network %d Missed > %d Current %d (%d%% match, %d%% complete)" % (total, len(foundLocal), > len(foundNet),len(missed), currentcount, match, complete)) > > + bb.plain("Sstate summary: Wanted %d Local %d Mirrors %d Missed > %d Current %d (%d%% match, %d%% complete)" % > > + (total, len(found)-foundMirrors, foundMirrors, len(missed), > currentcount, match, complete)) > > > > if hasattr(bb.parse.siggen, "checkhashes"): > > bb.parse.siggen.checkhashes(sq_data, missed, found, d) > > -- > > 2.33.0 > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#156164): > https://lists.openembedded.org/g/openembedded-core/message/156164 > > Mute This Topic: https://lists.openembedded.org/mt/85726949/3617179 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alexandre.belloni@bootlin.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > -- > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > -- Best regards, José Quaresma [-- Attachment #2: Type: text/html, Size: 8172 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-10-03 22:01 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-19 20:51 [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter Jose Quaresma 2021-09-19 20:51 ` [PATCH 2/2] sstate.bbclass: cleanup the sstate mirror progress check Jose Quaresma 2021-09-21 10:43 ` [OE-core] [PATCH 1/2] sstate.bbclass: track found files on mirrors with a counter Alexandre Belloni 2021-09-21 14:43 ` Jose Quaresma 2021-10-03 22:01 ` Jose Quaresma
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.