Ordering of anonymous Python functions and task signature calculations

Ordering of anonymous Python functions and task signature calculations

[Matt Hoosier]

[-- Attachment #1: Type: text/plain, Size: 1365 bytes --]

In order to support a use-case that embeds information about the Git
revision of Yocto itself that was used to make a build, I would like to run
some arbitrary Python code and dump the results (Git SHA1's, tag names,
etc) into a Bitbake variable.

The idea is that the variable would get consumed in a do_install() task to
populate some cookie-crumb file in the package payload.

Something like:

  DEPENDS = "git-native"

  python () {
    # This pseudocode isn't strictly functional for invoking Git, but you
get the idea
    d.setVar('GIT_INFO', subprocess.Popen(['git', 'rev-list',
...]).communicate().stdout)
  }

  do_install () {
    install -d ${D}/etc
    echo "${GIT_INFO}" > ${D}/etc/git-info.txt
  }

This all works on a fresh run, but Bitbake appears not to be evaluating the
anonymous Python function on each execution. This leads it to have
out-of-date information about the Git working copy when global state
changes but happens not to impact the particular recipe holding this logic.

Is there any trick available to cause the Python function to execute (and
hence update the value of ${GIT_INFO} each Bitbake execution so that the
up-to-the-moment contents of it filter into the calculation of the
signature for do_install()? Or am I just trying a wrong-headed approach and
doing something illegitimate?

-Matt

[-- Attachment #2: Type: text/html, Size: 1671 bytes --]

Re: Ordering of anonymous Python functions and task signature calculations

[Khem Raj]

On Tue, Jan 24, 2017 at 9:55 AM, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> In order to support a use-case that embeds information about the Git
> revision of Yocto itself that was used to make a build, I would like to run
> some arbitrary Python code and dump the results (Git SHA1's, tag names, etc)
> into a Bitbake variable.
>
> The idea is that the variable would get consumed in a do_install() task to
> populate some cookie-crumb file in the package payload.
>
> Something like:
>
>   DEPENDS = "git-native"
>
>   python () {
>     # This pseudocode isn't strictly functional for invoking Git, but you
> get the idea
>     d.setVar('GIT_INFO', subprocess.Popen(['git', 'rev-list',
> ...]).communicate().stdout)
>   }
>
>   do_install () {
>     install -d ${D}/etc
>     echo "${GIT_INFO}" > ${D}/etc/git-info.txt
>   }
>
> This all works on a fresh run, but Bitbake appears not to be evaluating the
> anonymous Python function on each execution. This leads it to have
> out-of-date information about the Git working copy when global state changes
> but happens not to impact the particular recipe holding this logic.
>
> Is there any trick available to cause the Python function to execute (and
> hence update the value of ${GIT_INFO} each Bitbake execution so that the
> up-to-the-moment contents of it filter into the calculation of the signature
> for do_install()? Or am I just trying a wrong-headed approach and doing
> something illegitimate?
>

May be you can do it as a post processing step at image build time.

> -Matt
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

Re: Ordering of anonymous Python functions and task signature calculations

[André Draszik]

On Tue, 2017-01-24 at 11:55 -0600, Matt Hoosier wrote:
> In order to support a use-case that embeds information about the Git
> revision of Yocto itself that was used to make a build, I would like to
> run
> some arbitrary Python code and dump the results (Git SHA1's, tag names,
> etc) into a Bitbake variable.

Have you looked at image-buildinfo.bbclass? From your description, it seems
to do what you want.

INHERIT += "image-buildinfo"

works well for me...


Cheers,
Andre'

Re: Ordering of anonymous Python functions and task signature calculations

[Matt Hoosier]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

On Wed, Jan 25, 2017 at 3:03 AM, André Draszik <git@andred.net> wrote:

> On Tue, 2017-01-24 at 11:55 -0600, Matt Hoosier wrote:
> > In order to support a use-case that embeds information about the Git
> > revision of Yocto itself that was used to make a build, I would like to
> > run
> > some arbitrary Python code and dump the results (Git SHA1's, tag names,
> > etc) into a Bitbake variable.
>
> Have you looked at image-buildinfo.bbclass? From your description, it seems
> to do what you want.
>
> INHERIT += "image-buildinfo"
>
> works well for me...
>

Yes, that's close. I have a few extra bits of information from Git that I'd
like to include, but the basic approach you and Khem pointed out to do this
during image construction works well enough.

Thanks.

[-- Attachment #2: Type: text/html, Size: 1228 bytes --]

Re: Ordering of anonymous Python functions and task signature calculations

[Matt Hoosier]

[-- Attachment #1: Type: text/plain, Size: 3039 bytes --]

(Cross-posting to bitbake-devel.)

Quite a while ago I wrote the following message:

On Tue, Jan 24, 2017 at 11:55 AM Matt Hoosier <matt.hoosier@gmail.com>
wrote:

> In order to support a use-case that embeds information about the Git
> revision of Yocto itself that was used to make a build, I would like to run
> some arbitrary Python code and dump the results (Git SHA1's, tag names,
> etc) into a Bitbake variable.
>
> The idea is that the variable would get consumed in a do_install() task to
> populate some cookie-crumb file in the package payload.
>
> Something like:
>
>   DEPENDS = "git-native"
>
>   python () {
>     # This pseudocode isn't strictly functional for invoking Git, but you
> get the idea
>     d.setVar('GIT_INFO', subprocess.Popen(['git', 'rev-list',
> ...]).communicate().stdout)
>   }
>
>   do_install () {
>     install -d ${D}/etc
>     echo "${GIT_INFO}" > ${D}/etc/git-info.txt
>   }
>
> This all works on a fresh run, but Bitbake appears not to be evaluating
> the anonymous Python function on each execution. This leads it to have
> out-of-date information about the Git working copy when global state
> changes but happens not to impact the particular recipe holding this logic.
>
>
The replies given at the time were very helpful in letting me accomplish my
immediate goals, but I never really did get to the bottom of the real
question:


> Is there any trick available to cause the Python function to execute (and
> hence update the value of ${GIT_INFO} each Bitbake execution so that the
> up-to-the-moment contents of it filter into the calculation of the
> signature for do_install()? Or am I just trying a wrong-headed approach and
> doing something illegitimate?
>

Some build systems I know of [1] that are based on hashes rather than
timestamps have a way that you can inform the task execution engine to
always poison the inputs to a certain task's up-to-dateness check. Is there
anything in Bitbake that would allow a declaration to force such-and-such
variable in a recipe always to be re-evaluated:

  EXTERNALLY_INFLUENCED_VARIABLE = "${@int(random.random() * 10)}"
  do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "

  do_install() {
    echo '${ EXTERNALLY_INFLUENCED_VARIABLE }' > ${D}/etc/stuff.txt
  }

Don't read too much into the example calling random() here. That's not
really what I want to do; the point is just that the variable's value
(though deterministic) isn't computed strictly from Bitbake metadata. So
I'm searching for a way to get the value of EXTERNALLY_INFLUENCED_VARIABLE
to be re-computed each time Bitbake starts. If the computed value ends up
matching what was seen on the last execution, then any dependent tasks
wouldn't need re-run.

Any ideas?

[1] I'm thinking particularly of Waf's hash-based task graph, which
supports annotations that can be used to always cause a task to be re-run
even if all the inputs feeding into it appear up-to-date.

Thanks,
Matt

[-- Attachment #2: Type: text/html, Size: 4820 bytes --]

Re: Ordering of anonymous Python functions and task signature calculations

[Richard Purdie]

On Wed, 2018-07-25 at 08:47 -0500, Matt Hoosier wrote:
> Quite a while ago I wrote the following message:
> On Tue, Jan 24, 2017 at 11:55 AM Matt Hoosier <matt.hoosier@gmail.com
> > wrote:
> > In order to support a use-case that embeds information about the
> > Git revision of Yocto itself that was used to make a build, I would
> > like to run some arbitrary Python code and dump the results (Git
> > SHA1's, tag names, etc) into a Bitbake variable.
> > 
> > The idea is that the variable would get consumed in a do_install()
> > task to populate some cookie-crumb file in the package payload.
> > 
> > Something like:
> > 
> >   DEPENDS = "git-native"
> > 
> >   python () {
> >     # This pseudocode isn't strictly functional for invoking Git,
> > but you get the idea
> >     d.setVar('GIT_INFO', subprocess.Popen(['git', 'rev-list',
> > ...]).communicate().stdout)
> >   }
> > 
> >   do_install () {
> >     install -d ${D}/etc
> >     echo "${GIT_INFO}" > ${D}/etc/git-info.txt
> >   }
> > 
> > This all works on a fresh run, but Bitbake appears not to be
> > evaluating the anonymous Python function on each execution. This
> > leads it to have out-of-date information about the Git working copy
> > when global state changes but happens not to impact the particular
> > recipe holding this logic.
> > 
> > 
> 
> The replies given at the time were very helpful in letting me
> accomplish my immediate goals, but I never really did get to the
> bottom of the real question:
>  
> > Is there any trick available to cause the Python function to
> > execute (and hence update the value of ${GIT_INFO} each Bitbake
> > execution so that the up-to-the-moment contents of it filter into
> > the calculation of the signature for do_install()? Or am I just
> > trying a wrong-headed approach and doing something illegitimate? 
> > 
> 
> Some build systems I know of [1] that are based on hashes rather than
> timestamps have a way that you can inform the task execution engine
> to always poison the inputs to a certain task's up-to-dateness check.
> Is there anything in Bitbake that would allow a declaration to force
> such-and-such variable in a recipe always to be re-evaluated:
> 
>   EXTERNALLY_INFLUENCED_VARIABLE = "${@int(random.random() * 10)}"
>   do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "
> 
>   do_install() {
>     echo '${ EXTERNALLY_INFLUENCED_VARIABLE }' > ${D}/etc/stuff.txt
>   }
> 
> Don't read too much into the example calling random() here. That's
> not really what I want to do; the point is just that the variable's
> value (though deterministic) isn't computed strictly from Bitbake
> metadata. So I'm searching for a way to get the value of
> EXTERNALLY_INFLUENCED_VARIABLE to be re-computed each time Bitbake
> starts. If the computed value ends up matching what was seen on the
> last execution, then any dependent tasks wouldn't need re-run.
> 
> Any ideas?
> 
> [1] I'm thinking particularly of Waf's hash-based task graph, which
> supports annotations that can be used to always cause a task to be
> re-run even if all the inputs feeding into it appear up-to-date.

From a bitbake perspective there are two things you need to ensure
here, that the recipe gets reparsed at every execution and that the
task dependency exists. You have the dependency above and that is the
easier piece to fix.

If the recipe is reparsed, the anon python reruns and the task hash
recalculated.

If a recipe sets BB_DONT_CACHE, it will be reparsed upon each run and
the hashes will dynamically change as you want. BB_SRCREV_POLICY !=
"cache" uses that behind the scenes so that it can dynamically update
srcrevs.

Hope that helps...

Cheers,

Richard

Re: [yocto] Ordering of anonymous Python functions and task signature calculations

[Richard Purdie]

On Wed, 2018-07-25 at 08:47 -0500, Matt Hoosier wrote:
> Quite a while ago I wrote the following message:
> On Tue, Jan 24, 2017 at 11:55 AM Matt Hoosier <matt.hoosier@gmail.com
> > wrote:
> > In order to support a use-case that embeds information about the
> > Git revision of Yocto itself that was used to make a build, I would
> > like to run some arbitrary Python code and dump the results (Git
> > SHA1's, tag names, etc) into a Bitbake variable.
> > 
> > The idea is that the variable would get consumed in a do_install()
> > task to populate some cookie-crumb file in the package payload.
> > 
> > Something like:
> > 
> >   DEPENDS = "git-native"
> > 
> >   python () {
> >     # This pseudocode isn't strictly functional for invoking Git,
> > but you get the idea
> >     d.setVar('GIT_INFO', subprocess.Popen(['git', 'rev-list',
> > ...]).communicate().stdout)
> >   }
> > 
> >   do_install () {
> >     install -d ${D}/etc
> >     echo "${GIT_INFO}" > ${D}/etc/git-info.txt
> >   }
> > 
> > This all works on a fresh run, but Bitbake appears not to be
> > evaluating the anonymous Python function on each execution. This
> > leads it to have out-of-date information about the Git working copy
> > when global state changes but happens not to impact the particular
> > recipe holding this logic.
> > 
> > 
> 
> The replies given at the time were very helpful in letting me
> accomplish my immediate goals, but I never really did get to the
> bottom of the real question:
>  
> > Is there any trick available to cause the Python function to
> > execute (and hence update the value of ${GIT_INFO} each Bitbake
> > execution so that the up-to-the-moment contents of it filter into
> > the calculation of the signature for do_install()? Or am I just
> > trying a wrong-headed approach and doing something illegitimate? 
> > 
> 
> Some build systems I know of [1] that are based on hashes rather than
> timestamps have a way that you can inform the task execution engine
> to always poison the inputs to a certain task's up-to-dateness check.
> Is there anything in Bitbake that would allow a declaration to force
> such-and-such variable in a recipe always to be re-evaluated:
> 
>   EXTERNALLY_INFLUENCED_VARIABLE = "${@int(random.random() * 10)}"
>   do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "
> 
>   do_install() {
>     echo '${ EXTERNALLY_INFLUENCED_VARIABLE }' > ${D}/etc/stuff.txt
>   }
> 
> Don't read too much into the example calling random() here. That's
> not really what I want to do; the point is just that the variable's
> value (though deterministic) isn't computed strictly from Bitbake
> metadata. So I'm searching for a way to get the value of
> EXTERNALLY_INFLUENCED_VARIABLE to be re-computed each time Bitbake
> starts. If the computed value ends up matching what was seen on the
> last execution, then any dependent tasks wouldn't need re-run.
> 
> Any ideas?
> 
> [1] I'm thinking particularly of Waf's hash-based task graph, which
> supports annotations that can be used to always cause a task to be
> re-run even if all the inputs feeding into it appear up-to-date.

From a bitbake perspective there are two things you need to ensure
here, that the recipe gets reparsed at every execution and that the
task dependency exists. You have the dependency above and that is the
easier piece to fix.

If the recipe is reparsed, the anon python reruns and the task hash
recalculated.

If a recipe sets BB_DONT_CACHE, it will be reparsed upon each run and
the hashes will dynamically change as you want. BB_SRCREV_POLICY !=
"cache" uses that behind the scenes so that it can dynamically update
srcrevs.

Hope that helps...

Cheers,

Richard

Re: Ordering of anonymous Python functions and task signature calculations

[Matt Hoosier]

[-- Attachment #1: Type: text/plain, Size: 5067 bytes --]

Thanks for the reply.

On Wed, Jul 25, 2018 at 4:58 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2018-07-25 at 08:47 -0500, Matt Hoosier wrote:
> > Quite a while ago I wrote the following message:
> > On Tue, Jan 24, 2017 at 11:55 AM Matt Hoosier <matt.hoosier@gmail.com
> > > wrote:
> > > In order to support a use-case that embeds information about the
> > > Git revision of Yocto itself that was used to make a build, I would
> > > like to run some arbitrary Python code and dump the results (Git
> > > SHA1's, tag names, etc) into a Bitbake variable.
> > >
> > > The idea is that the variable would get consumed in a do_install()
> > > task to populate some cookie-crumb file in the package payload.
> > >
> > > Something like:
> > >
> > >   DEPENDS = "git-native"
> > >
> > >   python () {
> > >     # This pseudocode isn't strictly functional for invoking Git,
> > > but you get the idea
> > >     d.setVar('GIT_INFO', subprocess.Popen(['git', 'rev-list',
> > > ...]).communicate().stdout)
> > >   }
> > >
> > >   do_install () {
> > >     install -d ${D}/etc
> > >     echo "${GIT_INFO}" > ${D}/etc/git-info.txt
> > >   }
> > >
> > > This all works on a fresh run, but Bitbake appears not to be
> > > evaluating the anonymous Python function on each execution. This
> > > leads it to have out-of-date information about the Git working copy
> > > when global state changes but happens not to impact the particular
> > > recipe holding this logic.
> > >
> > >
> >
> > The replies given at the time were very helpful in letting me
> > accomplish my immediate goals, but I never really did get to the
> > bottom of the real question:
> >
> > > Is there any trick available to cause the Python function to
> > > execute (and hence update the value of ${GIT_INFO} each Bitbake
> > > execution so that the up-to-the-moment contents of it filter into
> > > the calculation of the signature for do_install()? Or am I just
> > > trying a wrong-headed approach and doing something illegitimate?
> > >
> >
> > Some build systems I know of [1] that are based on hashes rather than
> > timestamps have a way that you can inform the task execution engine
> > to always poison the inputs to a certain task's up-to-dateness check.
> > Is there anything in Bitbake that would allow a declaration to force
> > such-and-such variable in a recipe always to be re-evaluated:
> >
> >   EXTERNALLY_INFLUENCED_VARIABLE = "${@int(random.random() * 10)}"
> >   do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "
> >
> >   do_install() {
> >     echo '${ EXTERNALLY_INFLUENCED_VARIABLE }' > ${D}/etc/stuff.txt
> >   }
> >
> > Don't read too much into the example calling random() here. That's
> > not really what I want to do; the point is just that the variable's
> > value (though deterministic) isn't computed strictly from Bitbake
> > metadata. So I'm searching for a way to get the value of
> > EXTERNALLY_INFLUENCED_VARIABLE to be re-computed each time Bitbake
> > starts. If the computed value ends up matching what was seen on the
> > last execution, then any dependent tasks wouldn't need re-run.
> >
> > Any ideas?
> >
> > [1] I'm thinking particularly of Waf's hash-based task graph, which
> > supports annotations that can be used to always cause a task to be
> > re-run even if all the inputs feeding into it appear up-to-date.
>
> From a bitbake perspective there are two things you need to ensure
> here, that the recipe gets reparsed at every execution


Your advice about BB_DONT_CACHE does cause reparsing to get triggered. I
can see that in logs.


> and that the
> task dependency exists. You have the dependency above and that is the
> easier piece to fix.
>

But this part I'm not seeing. The variable's value is always recalculated
(sometimes more than once during a Bitbake run, but that's okay because
it's idempotent), but the task only re-executes if the payload of the .bb
file itself has changed. The blurb I wrote above:

  do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "

is just some nonsense syntax I wrote there to illustrate my point. I don't
think there's any var flag called "vardepsalwaysdirty". Did you have this
line in mind when you were saying that I'd already gotten the task made
dependent on the variable? Or were you just noting that the syntactic use
of the variable inside of the task should engage the normal hash-based
task-dependency calculation logic?


>
> If the recipe is reparsed, the anon python reruns and the task hash
> recalculated.
>

Is there anything extra I need to do to cause the reparsed variable's hash
to cascade to the task?


>
> If a recipe sets BB_DONT_CACHE, it will be reparsed upon each run and
> the hashes will dynamically change as you want. BB_SRCREV_POLICY !=
> "cache" uses that behind the scenes so that it can dynamically update
> srcrevs.
>
> Hope that helps...
>
> Cheers,
>
> Richard
>
>

Thanks! -Matt

[-- Attachment #2: Type: text/html, Size: 7482 bytes --]

Re: [yocto] Ordering of anonymous Python functions and task signature calculations

[Matt Hoosier]

[-- Attachment #1: Type: text/plain, Size: 5067 bytes --]

Thanks for the reply.

On Wed, Jul 25, 2018 at 4:58 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2018-07-25 at 08:47 -0500, Matt Hoosier wrote:
> > Quite a while ago I wrote the following message:
> > On Tue, Jan 24, 2017 at 11:55 AM Matt Hoosier <matt.hoosier@gmail.com
> > > wrote:
> > > In order to support a use-case that embeds information about the
> > > Git revision of Yocto itself that was used to make a build, I would
> > > like to run some arbitrary Python code and dump the results (Git
> > > SHA1's, tag names, etc) into a Bitbake variable.
> > >
> > > The idea is that the variable would get consumed in a do_install()
> > > task to populate some cookie-crumb file in the package payload.
> > >
> > > Something like:
> > >
> > >   DEPENDS = "git-native"
> > >
> > >   python () {
> > >     # This pseudocode isn't strictly functional for invoking Git,
> > > but you get the idea
> > >     d.setVar('GIT_INFO', subprocess.Popen(['git', 'rev-list',
> > > ...]).communicate().stdout)
> > >   }
> > >
> > >   do_install () {
> > >     install -d ${D}/etc
> > >     echo "${GIT_INFO}" > ${D}/etc/git-info.txt
> > >   }
> > >
> > > This all works on a fresh run, but Bitbake appears not to be
> > > evaluating the anonymous Python function on each execution. This
> > > leads it to have out-of-date information about the Git working copy
> > > when global state changes but happens not to impact the particular
> > > recipe holding this logic.
> > >
> > >
> >
> > The replies given at the time were very helpful in letting me
> > accomplish my immediate goals, but I never really did get to the
> > bottom of the real question:
> >
> > > Is there any trick available to cause the Python function to
> > > execute (and hence update the value of ${GIT_INFO} each Bitbake
> > > execution so that the up-to-the-moment contents of it filter into
> > > the calculation of the signature for do_install()? Or am I just
> > > trying a wrong-headed approach and doing something illegitimate?
> > >
> >
> > Some build systems I know of [1] that are based on hashes rather than
> > timestamps have a way that you can inform the task execution engine
> > to always poison the inputs to a certain task's up-to-dateness check.
> > Is there anything in Bitbake that would allow a declaration to force
> > such-and-such variable in a recipe always to be re-evaluated:
> >
> >   EXTERNALLY_INFLUENCED_VARIABLE = "${@int(random.random() * 10)}"
> >   do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "
> >
> >   do_install() {
> >     echo '${ EXTERNALLY_INFLUENCED_VARIABLE }' > ${D}/etc/stuff.txt
> >   }
> >
> > Don't read too much into the example calling random() here. That's
> > not really what I want to do; the point is just that the variable's
> > value (though deterministic) isn't computed strictly from Bitbake
> > metadata. So I'm searching for a way to get the value of
> > EXTERNALLY_INFLUENCED_VARIABLE to be re-computed each time Bitbake
> > starts. If the computed value ends up matching what was seen on the
> > last execution, then any dependent tasks wouldn't need re-run.
> >
> > Any ideas?
> >
> > [1] I'm thinking particularly of Waf's hash-based task graph, which
> > supports annotations that can be used to always cause a task to be
> > re-run even if all the inputs feeding into it appear up-to-date.
>
> From a bitbake perspective there are two things you need to ensure
> here, that the recipe gets reparsed at every execution


Your advice about BB_DONT_CACHE does cause reparsing to get triggered. I
can see that in logs.


> and that the
> task dependency exists. You have the dependency above and that is the
> easier piece to fix.
>

But this part I'm not seeing. The variable's value is always recalculated
(sometimes more than once during a Bitbake run, but that's okay because
it's idempotent), but the task only re-executes if the payload of the .bb
file itself has changed. The blurb I wrote above:

  do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "

is just some nonsense syntax I wrote there to illustrate my point. I don't
think there's any var flag called "vardepsalwaysdirty". Did you have this
line in mind when you were saying that I'd already gotten the task made
dependent on the variable? Or were you just noting that the syntactic use
of the variable inside of the task should engage the normal hash-based
task-dependency calculation logic?


>
> If the recipe is reparsed, the anon python reruns and the task hash
> recalculated.
>

Is there anything extra I need to do to cause the reparsed variable's hash
to cascade to the task?


>
> If a recipe sets BB_DONT_CACHE, it will be reparsed upon each run and
> the hashes will dynamically change as you want. BB_SRCREV_POLICY !=
> "cache" uses that behind the scenes so that it can dynamically update
> srcrevs.
>
> Hope that helps...
>
> Cheers,
>
> Richard
>
>

Thanks! -Matt

[-- Attachment #2: Type: text/html, Size: 7482 bytes --]

Re: Ordering of anonymous Python functions and task signature calculations

[richard.purdie]

On Thu, 2018-07-26 at 09:22 -0500, Matt Hoosier wrote:
> On Wed, Jul 25, 2018 at 4:58 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> Your advice about BB_DONT_CACHE does cause reparsing to get
> triggered. I can see that in logs.
>  
> > and that the
> > task dependency exists. You have the dependency above and that is
> > the
> > easier piece to fix.
> 
> But this part I'm not seeing. The variable's value is always
> recalculated (sometimes more than once during a Bitbake run, but
> that's okay because it's idempotent), but the task only re-executes
> if the payload of the .bb file itself has changed. The blurb I wrote
> above:
> 
>   do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "
> 
> is just some nonsense syntax I wrote there to illustrate my point. I
> don't think there's any var flag called "vardepsalwaysdirty". Did you
> have this line in mind when you were saying that I'd already gotten
> the task made dependent on the variable? Or were you just noting that
> the syntactic use of the variable inside of the task should engage
> the normal hash-based task-dependency calculation logic?
>  
> > If the recipe is reparsed, the anon python reruns and the task hash
> > recalculated.
> 
> Is there anything extra I need to do to cause the reparsed variable's
> hash to cascade to the task?

I think I understand what you're asking now. The key thing is:

conf/bitbake.conf:AUTOREV[vardepvalue] = "${SRCPV}"
conf/bitbake.conf:SRCPV[vardepvalue] = "${SRCPV}"

which makes SRCREV/AUTOREV work.

For your specific example you'd want:

EXTERNALLY_INFLUENCED_VARIABLE[vardepvalue] = "${EXTERNALLY_INFLUENCED_VARIABLE}"

which will add the value of the variable to the task hash.

Cheers,

Richard

Re: [yocto] Ordering of anonymous Python functions and task signature calculations

[richard.purdie]

On Thu, 2018-07-26 at 09:22 -0500, Matt Hoosier wrote:
> On Wed, Jul 25, 2018 at 4:58 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> Your advice about BB_DONT_CACHE does cause reparsing to get
> triggered. I can see that in logs.
>  
> > and that the
> > task dependency exists. You have the dependency above and that is
> > the
> > easier piece to fix.
> 
> But this part I'm not seeing. The variable's value is always
> recalculated (sometimes more than once during a Bitbake run, but
> that's okay because it's idempotent), but the task only re-executes
> if the payload of the .bb file itself has changed. The blurb I wrote
> above:
> 
>   do_install[vardepsalwaysdirty] = " EXTERNALLY_INFLUENCED_VARIABLE "
> 
> is just some nonsense syntax I wrote there to illustrate my point. I
> don't think there's any var flag called "vardepsalwaysdirty". Did you
> have this line in mind when you were saying that I'd already gotten
> the task made dependent on the variable? Or were you just noting that
> the syntactic use of the variable inside of the task should engage
> the normal hash-based task-dependency calculation logic?
>  
> > If the recipe is reparsed, the anon python reruns and the task hash
> > recalculated.
> 
> Is there anything extra I need to do to cause the reparsed variable's
> hash to cascade to the task?

I think I understand what you're asking now. The key thing is:

conf/bitbake.conf:AUTOREV[vardepvalue] = "${SRCPV}"
conf/bitbake.conf:SRCPV[vardepvalue] = "${SRCPV}"

which makes SRCREV/AUTOREV work.

For your specific example you'd want:

EXTERNALLY_INFLUENCED_VARIABLE[vardepvalue] = "${EXTERNALLY_INFLUENCED_VARIABLE}"

which will add the value of the variable to the task hash.

Cheers,

Richard