[PATCH] externalsrc: write task locks into ${T}, not into ${S}

[PATCH] externalsrc: write task locks into ${T}, not into ${S}

[Alexander Kanavin]

Otherwise lock files appearing and disappearing can interfere
with reproducibility code enumerating source files in do_unpack():

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3745/steps/14/logs/stdio

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/classes/externalsrc.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 90792a737b..b326f10050 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -88,7 +88,7 @@ python () {
                 bb.build.deltask(task, d)
             elif os.path.realpath(d.getVar('S')) == os.path.realpath(d.getVar('B')):
                 # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time
-                d.appendVarFlag(task, "lockfiles", " ${S}/singletask.lock")
+                d.appendVarFlag(task, "lockfiles", " ${T}/singletask.lock")
 
             for funcname in [task, "base_" + task, "kernel_" + task]:
                 # We do not want our source to be wiped out, ever (kernel.bbclass does this for do_clean)
-- 
2.30.2

Re: [OE-core] [PATCH] externalsrc: write task locks into ${T}, not into ${S}

[Richard Purdie]

On Tue, 2022-07-05 at 18:39 +0200, Alexander Kanavin wrote:
> Otherwise lock files appearing and disappearing can interfere
> with reproducibility code enumerating source files in do_unpack():
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3745/steps/14/logs/stdio
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  meta/classes/externalsrc.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
> index 90792a737b..b326f10050 100644
> --- a/meta/classes/externalsrc.bbclass
> +++ b/meta/classes/externalsrc.bbclass
> @@ -88,7 +88,7 @@ python () {
>                  bb.build.deltask(task, d)
>              elif os.path.realpath(d.getVar('S')) == os.path.realpath(d.getVar('B')):
>                  # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time
> -                d.appendVarFlag(task, "lockfiles", " ${S}/singletask.lock")
> +                d.appendVarFlag(task, "lockfiles", " ${T}/singletask.lock")
>  
>              for funcname in [task, "base_" + task, "kernel_" + task]:
>                  # We do not want our source to be wiped out, ever (kernel.bbclass does this for do_clean)


Not an option I'm afraid as it is in ${S} for a reason. This is two
recipes both pointing at the same source (e.g. a recipe and it's
BBCLASSEXTEND native version) can work.

Cheers,

Richard

Re: [OE-core] [PATCH] externalsrc: write task locks into ${T}, not into ${S}

[Alexander Kanavin]

On Tue, 5 Jul 2022 at 19:03, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Not an option I'm afraid as it is in ${S} for a reason. This is two
> recipes both pointing at the same source (e.g. a recipe and it's
> BBCLASSEXTEND native version) can work.


I considered adding an exclusion for the lock filename to the code
that tracebacks, but special casing like that is pretty ugh. Today
here, tomorrow elsewhere.

Alex

Re: [OE-core] [PATCH] externalsrc: write task locks into ${T}, not into ${S}

[Richard Purdie]

On Tue, 2022-07-05 at 19:07 +0200, Alexander Kanavin wrote:
> On Tue, 5 Jul 2022 at 19:03, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > Not an option I'm afraid as it is in ${S} for a reason. This is two
> > recipes both pointing at the same source (e.g. a recipe and it's
> > BBCLASSEXTEND native version) can work.
> 
> 
> I considered adding an exclusion for the lock filename to the code
> that tracebacks, but special casing like that is pretty ugh. Today
> here, tomorrow elsewhere.

Right, but the lock was also added there for a reason so we're going to
have to work out some solution :/.

Cheers,

Richard

Re: [OE-core] [PATCH] externalsrc: write task locks into ${T}, not into ${S}

[Alexander Kanavin]

On Tue, 5 Jul 2022 at 23:48, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > I considered adding an exclusion for the lock filename to the code
> > that tracebacks, but special casing like that is pretty ugh. Today
> > here, tomorrow elsewhere.
>
> Right, but the lock was also added there for a reason so we're going to
> have to work out some solution :/.

One thing I don't quite understand is why the lock file is appearing
and disappearing while the task itself is executing? Shouldn't it be
happening when the task queue is being processed?

Alex

Re: [OE-core] [PATCH] externalsrc: write task locks into ${T}, not into ${S}

[Richard Purdie]

On Tue, 2022-07-05 at 23:51 +0200, Alexander Kanavin wrote:
> On Tue, 5 Jul 2022 at 23:48, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > > I considered adding an exclusion for the lock filename to the code
> > > that tracebacks, but special casing like that is pretty ugh. Today
> > > here, tomorrow elsewhere.
> > 
> > Right, but the lock was also added there for a reason so we're going to
> > have to work out some solution :/.
> 
> One thing I don't quite understand is why the lock file is appearing
> and disappearing while the task itself is executing? Shouldn't it be
> happening when the task queue is being processed?

I think we do need to understand why that is happening. I don't
remember the specifics of the lock but it is possible it is being held
and then released by another task running in parallel?

Cheers,

Richard