All of lore.kernel.org
 help / color / mirror / Atom feed
* Forcing the rebuild of dependent tasks across recipes
@ 2019-11-22 15:09 Matthias Schiffer
  2019-11-22 16:34 ` Richard Purdie
  2019-11-22 17:15 ` Mark Hatle
  0 siblings, 2 replies; 8+ messages in thread
From: Matthias Schiffer @ 2019-11-22 15:09 UTC (permalink / raw)
  To: bitbake-devel

Hi all,

is there a robust way to enforce the automatic rebuild of dependent
tasks across recipes? I'm having the following issue:

- Recipe my-core-image has an output file that is copied to the deploy
dir. The output file is not fully reproducible, so cleaning the recipe
and rebuilding it will yield a different file even through the sstate
signature of the recipe is unchanged (the file I'm interested is
DMVerity hash data that I generate as an additional filesystem type)

- Task do_compile of recipe my-initramfs-scripts depends on my-core-
image:do_image_complete. I need to enforce rebuilding of this task
whenever the deployed files of my-core-image have changed (the recipe
uses the "root hash" generated by DMVerity as input, so that the
filesystem can be verified on mount) - and of course also of all
dependent tasks like the initramfs image and FIT kernel image

I've experimented with things like file-checksums and BB_DONT_CACHE,
but everything I've tried fails when the rebuilds of my-core-image and
my-initramfs-scripts are supposed to happen in the same run of Bitbake:

- file-checksums only seems to be taken into account at initial recipe
parse time, not when the task I'm interested in is actually executed? I
need to run bitbake twice to get the correct results.
- When I use an anonymous Python function to put the root hash I'm
interested in into a variable, Bitbake complains about nondeterministic
metadata.

Using nostamp would be an option, but obviously an ugly one.

I'm currently on Yocto Thud / Bitbake 1.40.


Thanks in advance,

Matthias



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Forcing the rebuild of dependent tasks across recipes
  2019-11-22 15:09 Forcing the rebuild of dependent tasks across recipes Matthias Schiffer
@ 2019-11-22 16:34 ` Richard Purdie
  2019-11-25  9:46   ` (EXT) " Matthias Schiffer
  2019-11-22 17:15 ` Mark Hatle
  1 sibling, 1 reply; 8+ messages in thread
From: Richard Purdie @ 2019-11-22 16:34 UTC (permalink / raw)
  To: Matthias Schiffer, bitbake-devel

On Fri, 2019-11-22 at 16:09 +0100, Matthias Schiffer wrote:
> is there a robust way to enforce the automatic rebuild of dependent
> tasks across recipes? I'm having the following issue:
> 
> - Recipe my-core-image has an output file that is copied to the
> deploy
> dir. The output file is not fully reproducible, so cleaning the
> recipe
> and rebuilding it will yield a different file even through the sstate
> signature of the recipe is unchanged (the file I'm interested is
> DMVerity hash data that I generate as an additional filesystem type)
> 
> - Task do_compile of recipe my-initramfs-scripts depends on my-core-
> image:do_image_complete. I need to enforce rebuilding of this task
> whenever the deployed files of my-core-image have changed (the recipe
> uses the "root hash" generated by DMVerity as input, so that the
> filesystem can be verified on mount) - and of course also of all
> dependent tasks like the initramfs image and FIT kernel image
> 
> I've experimented with things like file-checksums and BB_DONT_CACHE,
> but everything I've tried fails when the rebuilds of my-core-image
> and
> my-initramfs-scripts are supposed to happen in the same run of
> Bitbake:
> 
> - file-checksums only seems to be taken into account at initial
> recipe
> parse time, not when the task I'm interested in is actually executed?
> I
> need to run bitbake twice to get the correct results.
> - When I use an anonymous Python function to put the root hash I'm
> interested in into a variable, Bitbake complains about
> nondeterministic
> metadata.
> 
> Using nostamp would be an option, but obviously an ugly one.
> 
> I'm currently on Yocto Thud / Bitbake 1.40.

If you look behind the scenes at how "bitbake x -f" works, it writes
.taint files into ${STAMPS}.

If A depends on B and B taints A's stamp each time it runs, it will
rerun that task and all tasks that depend upon it. See
bb.build.write_taint(). That function is not designed to be called
externally so you may need your own version. 

Ugly, but your problem is ugly. We don't have a standard mechanism as
we don't really want to encourage this.

Cheers,

Richard




^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Forcing the rebuild of dependent tasks across recipes
  2019-11-22 15:09 Forcing the rebuild of dependent tasks across recipes Matthias Schiffer
  2019-11-22 16:34 ` Richard Purdie
@ 2019-11-22 17:15 ` Mark Hatle
  2019-11-25  9:58   ` (EXT) " Matthias Schiffer
  1 sibling, 1 reply; 8+ messages in thread
From: Mark Hatle @ 2019-11-22 17:15 UTC (permalink / raw)
  To: Matthias Schiffer, bitbake-devel



On 11/22/19 9:09 AM, Matthias Schiffer wrote:
> Hi all,
> 
> is there a robust way to enforce the automatic rebuild of dependent
> tasks across recipes? I'm having the following issue:
> 
> - Recipe my-core-image has an output file that is copied to the deploy
> dir. The output file is not fully reproducible, so cleaning the recipe
> and rebuilding it will yield a different file even through the sstate
> signature of the recipe is unchanged (the file I'm interested is
> DMVerity hash data that I generate as an additional filesystem type)
> 
> - Task do_compile of recipe my-initramfs-scripts depends on my-core-
> image:do_image_complete. I need to enforce rebuilding of this task
> whenever the deployed files of my-core-image have changed (the recipe
> uses the "root hash" generated by DMVerity as input, so that the
> filesystem can be verified on mount) - and of course also of all
> dependent tasks like the initramfs image and FIT kernel image
> 
> I've experimented with things like file-checksums and BB_DONT_CACHE,
> but everything I've tried fails when the rebuilds of my-core-image and
> my-initramfs-scripts are supposed to happen in the same run of Bitbake:
> 
> - file-checksums only seems to be taken into account at initial recipe
> parse time, not when the task I'm interested in is actually executed? I
> need to run bitbake twice to get the correct results.
> - When I use an anonymous Python function to put the root hash I'm
> interested in into a variable, Bitbake complains about nondeterministic
> metadata.

There is a way of doing this, I've had to do it in the past.

The anonymous python function needs to product the same output on each and every
run.  If it doesn't this will cause the non-deterministic behavior.  (It can't
rely on another items to finish building for instance.)

So you end up having to have a recipe (with a file-checksum and/or anon python)
that sets up variables that will keep the hash up-to-date for THAT recipe.. then
your image (or other) recipes need to depend on that one.

Unfortunately I no longer have access to the example where I had to do this in
the past, or I could try to give you something more concrete to look at.

--Mark

> 
> Using nostamp would be an option, but obviously an ugly one.
> 
> I'm currently on Yocto Thud / Bitbake 1.40.
> 
> 
> Thanks in advance,
> 
> Matthias
> 


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: (EXT) Re: Forcing the rebuild of dependent tasks across recipes
  2019-11-22 16:34 ` Richard Purdie
@ 2019-11-25  9:46   ` Matthias Schiffer
  0 siblings, 0 replies; 8+ messages in thread
From: Matthias Schiffer @ 2019-11-25  9:46 UTC (permalink / raw)
  To: Richard Purdie; +Cc: bitbake-devel

On Fri, 2019-11-22 at 16:34 +0000, Richard Purdie wrote:
> On Fri, 2019-11-22 at 16:09 +0100, Matthias Schiffer wrote:
> > is there a robust way to enforce the automatic rebuild of dependent
> > tasks across recipes? I'm having the following issue:
> > 
> > - Recipe my-core-image has an output file that is copied to the
> > deploy
> > dir. The output file is not fully reproducible, so cleaning the
> > recipe
> > and rebuilding it will yield a different file even through the
> > sstate
> > signature of the recipe is unchanged (the file I'm interested is
> > DMVerity hash data that I generate as an additional filesystem
> > type)
> > 
> > - Task do_compile of recipe my-initramfs-scripts depends on my-
> > core-
> > image:do_image_complete. I need to enforce rebuilding of this task
> > whenever the deployed files of my-core-image have changed (the
> > recipe
> > uses the "root hash" generated by DMVerity as input, so that the
> > filesystem can be verified on mount) - and of course also of all
> > dependent tasks like the initramfs image and FIT kernel image
> > 
> > I've experimented with things like file-checksums and
> > BB_DONT_CACHE,
> > but everything I've tried fails when the rebuilds of my-core-image
> > and
> > my-initramfs-scripts are supposed to happen in the same run of
> > Bitbake:
> > 
> > - file-checksums only seems to be taken into account at initial
> > recipe
> > parse time, not when the task I'm interested in is actually
> > executed?
> > I
> > need to run bitbake twice to get the correct results.
> > - When I use an anonymous Python function to put the root hash I'm
> > interested in into a variable, Bitbake complains about
> > nondeterministic
> > metadata.
> > 
> > Using nostamp would be an option, but obviously an ugly one.
> > 
> > I'm currently on Yocto Thud / Bitbake 1.40.
> 
> If you look behind the scenes at how "bitbake x -f" works, it writes
> .taint files into ${STAMPS}.
> 
> If A depends on B and B taints A's stamp each time it runs, it will
> rerun that task and all tasks that depend upon it. See
> bb.build.write_taint(). That function is not designed to be called
> externally so you may need your own version. 

Hello Richard,

I don't think this will help me. If I'm interpreting the Bitbake code
correctly, the taint will only be interpreted at initial parse time
(which it must, as it would lead to non-deterministic task signatures
otherwise). So I can either set the taint early (always invalidating
the signature, leading to nostamp behaviour), or during my core image
creation (leading to the file-checksums behaviour, requiring a second
bitbake run before the taint has any effect). Or am I overlooking
something?


Thanks,

Matthias


> 
> Ugly, but your problem is ugly. We don't have a standard mechanism as
> we don't really want to encourage this.
> 
> Cheers,
> 
> Richard
> 
> 



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: (EXT) Re: Forcing the rebuild of dependent tasks across recipes
  2019-11-22 17:15 ` Mark Hatle
@ 2019-11-25  9:58   ` Matthias Schiffer
  2019-11-25 14:54     ` Mark Hatle
  0 siblings, 1 reply; 8+ messages in thread
From: Matthias Schiffer @ 2019-11-25  9:58 UTC (permalink / raw)
  To: Mark Hatle; +Cc: bitbake-devel

On Fri, 2019-11-22 at 11:15 -0600, Mark Hatle wrote:
> 
> On 11/22/19 9:09 AM, Matthias Schiffer wrote:
> > Hi all,
> > 
> > is there a robust way to enforce the automatic rebuild of dependent
> > tasks across recipes? I'm having the following issue:
> > 
> > - Recipe my-core-image has an output file that is copied to the
> > deploy
> > dir. The output file is not fully reproducible, so cleaning the
> > recipe
> > and rebuilding it will yield a different file even through the
> > sstate
> > signature of the recipe is unchanged (the file I'm interested is
> > DMVerity hash data that I generate as an additional filesystem
> > type)
> > 
> > - Task do_compile of recipe my-initramfs-scripts depends on my-
> > core-
> > image:do_image_complete. I need to enforce rebuilding of this task
> > whenever the deployed files of my-core-image have changed (the
> > recipe
> > uses the "root hash" generated by DMVerity as input, so that the
> > filesystem can be verified on mount) - and of course also of all
> > dependent tasks like the initramfs image and FIT kernel image
> > 
> > I've experimented with things like file-checksums and
> > BB_DONT_CACHE,
> > but everything I've tried fails when the rebuilds of my-core-image
> > and
> > my-initramfs-scripts are supposed to happen in the same run of
> > Bitbake:
> > 
> > - file-checksums only seems to be taken into account at initial
> > recipe
> > parse time, not when the task I'm interested in is actually
> > executed? I
> > need to run bitbake twice to get the correct results.
> > - When I use an anonymous Python function to put the root hash I'm
> > interested in into a variable, Bitbake complains about
> > nondeterministic
> > metadata.
> 
> There is a way of doing this, I've had to do it in the past.
> 
> The anonymous python function needs to product the same output on
> each and every
> run.  If it doesn't this will cause the non-deterministic
> behavior.  (It can't
> rely on another items to finish building for instance.)
> 
> So you end up having to have a recipe (with a file-checksum and/or
> anon python)
> that sets up variables that will keep the hash up-to-date for THAT
> recipe.. then
> your image (or other) recipes need to depend on that one.

Hello Mark,

I'm afraid I don't follow. How does doing something in another recipe
fix the non-deterministic behaviour?

I'd think that any solution based on the task signatures will have the
same issue - that a hash correctly reflecting the inputs that go into
my initramfs task would depend on the build results of my core image
task, which is not available at recipe parse time.

I think I'll have to go with nostamp after all - the unnessesary
rebuilds it causes in this case are ugly, but rather negligible in
terms of build time.


Thanks,

Matthias



> 
> Unfortunately I no longer have access to the example where I had to
> do this in
> the past, or I could try to give you something more concrete to look
> at.
> 
> --Mark
> 
> > 
> > Using nostamp would be an option, but obviously an ugly one.
> > 
> > I'm currently on Yocto Thud / Bitbake 1.40.
> > 
> > 
> > Thanks in advance,
> > 
> > Matthias
> > 



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: (EXT) Re: Forcing the rebuild of dependent tasks across recipes
  2019-11-25  9:58   ` (EXT) " Matthias Schiffer
@ 2019-11-25 14:54     ` Mark Hatle
  2019-11-26  8:43       ` (EXT) " Matthias Schiffer
  0 siblings, 1 reply; 8+ messages in thread
From: Mark Hatle @ 2019-11-25 14:54 UTC (permalink / raw)
  To: Matthias Schiffer; +Cc: bitbake-devel



On 11/25/19 3:58 AM, Matthias Schiffer wrote:
> On Fri, 2019-11-22 at 11:15 -0600, Mark Hatle wrote:
>>
>> On 11/22/19 9:09 AM, Matthias Schiffer wrote:
>>> Hi all,
>>>
>>> is there a robust way to enforce the automatic rebuild of dependent
>>> tasks across recipes? I'm having the following issue:
>>>
>>> - Recipe my-core-image has an output file that is copied to the
>>> deploy
>>> dir. The output file is not fully reproducible, so cleaning the
>>> recipe
>>> and rebuilding it will yield a different file even through the
>>> sstate
>>> signature of the recipe is unchanged (the file I'm interested is
>>> DMVerity hash data that I generate as an additional filesystem
>>> type)
>>>
>>> - Task do_compile of recipe my-initramfs-scripts depends on my-
>>> core-
>>> image:do_image_complete. I need to enforce rebuilding of this task
>>> whenever the deployed files of my-core-image have changed (the
>>> recipe
>>> uses the "root hash" generated by DMVerity as input, so that the
>>> filesystem can be verified on mount) - and of course also of all
>>> dependent tasks like the initramfs image and FIT kernel image
>>>
>>> I've experimented with things like file-checksums and
>>> BB_DONT_CACHE,
>>> but everything I've tried fails when the rebuilds of my-core-image
>>> and
>>> my-initramfs-scripts are supposed to happen in the same run of
>>> Bitbake:
>>>
>>> - file-checksums only seems to be taken into account at initial
>>> recipe
>>> parse time, not when the task I'm interested in is actually
>>> executed? I
>>> need to run bitbake twice to get the correct results.
>>> - When I use an anonymous Python function to put the root hash I'm
>>> interested in into a variable, Bitbake complains about
>>> nondeterministic
>>> metadata.
>>
>> There is a way of doing this, I've had to do it in the past.
>>
>> The anonymous python function needs to product the same output on
>> each and every
>> run.  If it doesn't this will cause the non-deterministic
>> behavior.  (It can't
>> rely on another items to finish building for instance.)
>>
>> So you end up having to have a recipe (with a file-checksum and/or
>> anon python)
>> that sets up variables that will keep the hash up-to-date for THAT
>> recipe.. then
>> your image (or other) recipes need to depend on that one.
> 
> Hello Mark,
> 
> I'm afraid I don't follow. How does doing something in another recipe
> fix the non-deterministic behaviour?

You must always compute your internal variables in -exactly- the same way.  You
were saying you were passing in an external object and needed to have that
influence the values.  (Unless I am understanding incorrectly.)

> I'd think that any solution based on the task signatures will have the
> same issue - that a hash correctly reflecting the inputs that go into
> my initramfs task would depend on the build results of my core image
> task, which is not available at recipe parse time.

All signatures are computed at parse time.  They either need to be computed
based on external (non-changing) input, or other recipes (which themselves would
have an static hash, but can change from build to build.)

> I think I'll have to go with nostamp after all - the unnessesary
> rebuilds it causes in this case are ugly, but rather negligible in
> terms of build time.

So is the input changing out from under the system?  I.e. during the build
process did it change, or are you trying to use the output of one recipe to
create input from another without standard recipe dependency behavior?

--Mark

> 
> Thanks,
> 
> Matthias
> 
> 
> 
>>
>> Unfortunately I no longer have access to the example where I had to
>> do this in
>> the past, or I could try to give you something more concrete to look
>> at.
>>
>> --Mark
>>
>>>
>>> Using nostamp would be an option, but obviously an ugly one.
>>>
>>> I'm currently on Yocto Thud / Bitbake 1.40.
>>>
>>>
>>> Thanks in advance,
>>>
>>> Matthias
>>>


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: (EXT) Re: (EXT) Re: Forcing the rebuild of dependent tasks across recipes
  2019-11-25 14:54     ` Mark Hatle
@ 2019-11-26  8:43       ` Matthias Schiffer
  2019-11-26 15:40         ` Mark Hatle
  0 siblings, 1 reply; 8+ messages in thread
From: Matthias Schiffer @ 2019-11-26  8:43 UTC (permalink / raw)
  To: Mark Hatle; +Cc: bitbake-devel

On Mon, 2019-11-25 at 08:54 -0600, Mark Hatle wrote:
> 
> On 11/25/19 3:58 AM, Matthias Schiffer wrote:
> > On Fri, 2019-11-22 at 11:15 -0600, Mark Hatle wrote:
> > > 
> > > On 11/22/19 9:09 AM, Matthias Schiffer wrote:
> > > > Hi all,
> > > > 
> > > > is there a robust way to enforce the automatic rebuild of
> > > > dependent
> > > > tasks across recipes? I'm having the following issue:
> > > > 
> > > > - Recipe my-core-image has an output file that is copied to the
> > > > deploy
> > > > dir. The output file is not fully reproducible, so cleaning the
> > > > recipe
> > > > and rebuilding it will yield a different file even through the
> > > > sstate
> > > > signature of the recipe is unchanged (the file I'm interested
> > > > is
> > > > DMVerity hash data that I generate as an additional filesystem
> > > > type)
> > > > 
> > > > - Task do_compile of recipe my-initramfs-scripts depends on my-
> > > > core-
> > > > image:do_image_complete. I need to enforce rebuilding of this
> > > > task
> > > > whenever the deployed files of my-core-image have changed (the
> > > > recipe
> > > > uses the "root hash" generated by DMVerity as input, so that
> > > > the
> > > > filesystem can be verified on mount) - and of course also of
> > > > all
> > > > dependent tasks like the initramfs image and FIT kernel image
> > > > 
> > > > I've experimented with things like file-checksums and
> > > > BB_DONT_CACHE,
> > > > but everything I've tried fails when the rebuilds of my-core-
> > > > image
> > > > and
> > > > my-initramfs-scripts are supposed to happen in the same run of
> > > > Bitbake:
> > > > 
> > > > - file-checksums only seems to be taken into account at initial
> > > > recipe
> > > > parse time, not when the task I'm interested in is actually
> > > > executed? I
> > > > need to run bitbake twice to get the correct results.
> > > > - When I use an anonymous Python function to put the root hash
> > > > I'm
> > > > interested in into a variable, Bitbake complains about
> > > > nondeterministic
> > > > metadata.
> > > 
> > > There is a way of doing this, I've had to do it in the past.
> > > 
> > > The anonymous python function needs to product the same output on
> > > each and every
> > > run.  If it doesn't this will cause the non-deterministic
> > > behavior.  (It can't
> > > rely on another items to finish building for instance.)
> > > 
> > > So you end up having to have a recipe (with a file-checksum
> > > and/or
> > > anon python)
> > > that sets up variables that will keep the hash up-to-date for
> > > THAT
> > > recipe.. then
> > > your image (or other) recipes need to depend on that one.
> > 
> > Hello Mark,
> > 
> > I'm afraid I don't follow. How does doing something in another
> > recipe
> > fix the non-deterministic behaviour?
> 
> You must always compute your internal variables in -exactly- the same
> way.  You
> were saying you were passing in an external object and needed to have
> that
> influence the values.  (Unless I am understanding incorrectly.)
> 
> > I'd think that any solution based on the task signatures will have
> > the
> > same issue - that a hash correctly reflecting the inputs that go
> > into
> > my initramfs task would depend on the build results of my core
> > image
> > task, which is not available at recipe parse time.
> 
> All signatures are computed at parse time.  They either need to be
> computed
> based on external (non-changing) input, or other recipes (which
> themselves would
> have an static hash, but can change from build to build.)
> 
> > I think I'll have to go with nostamp after all - the unnessesary
> > rebuilds it causes in this case are ugly, but rather negligible in
> > terms of build time.
> 
> So is the input changing out from under the system?  I.e. during the
> build
> process did it change, or are you trying to use the output of one
> recipe to
> create input from another without standard recipe dependency
> behavior?

The latter. I have a regular core image built by Yocto, and I don't
want to care about full reproducibility - so rebuilding the core image
will give me a different image regarding timestamps etc. The second
recipe (and its dependents) needs to rerun when the hash of the built
image changes. This does not happen without nostamp, because no
variable known to Bitbake can contain this hash at parse time.

Matthias


> 
> --Mark
> 
> > 
> > Thanks,
> > 
> > Matthias
> > 
> > 
> > 
> > > 
> > > Unfortunately I no longer have access to the example where I had
> > > to
> > > do this in
> > > the past, or I could try to give you something more concrete to
> > > look
> > > at.
> > > 
> > > --Mark
> > > 
> > > > 
> > > > Using nostamp would be an option, but obviously an ugly one.
> > > > 
> > > > I'm currently on Yocto Thud / Bitbake 1.40.
> > > > 
> > > > 
> > > > Thanks in advance,
> > > > 
> > > > Matthias
> > > > 



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: (EXT) Re: (EXT) Re: Forcing the rebuild of dependent tasks across recipes
  2019-11-26  8:43       ` (EXT) " Matthias Schiffer
@ 2019-11-26 15:40         ` Mark Hatle
  0 siblings, 0 replies; 8+ messages in thread
From: Mark Hatle @ 2019-11-26 15:40 UTC (permalink / raw)
  To: Matthias Schiffer; +Cc: bitbake-devel



On 11/26/19 2:43 AM, Matthias Schiffer wrote:
> On Mon, 2019-11-25 at 08:54 -0600, Mark Hatle wrote:
>>
>> On 11/25/19 3:58 AM, Matthias Schiffer wrote:
>>> On Fri, 2019-11-22 at 11:15 -0600, Mark Hatle wrote:
>>>>
>>>> On 11/22/19 9:09 AM, Matthias Schiffer wrote:
>>>>> Hi all,
>>>>>
>>>>> is there a robust way to enforce the automatic rebuild of
>>>>> dependent
>>>>> tasks across recipes? I'm having the following issue:
>>>>>
>>>>> - Recipe my-core-image has an output file that is copied to the
>>>>> deploy
>>>>> dir. The output file is not fully reproducible, so cleaning the
>>>>> recipe
>>>>> and rebuilding it will yield a different file even through the
>>>>> sstate
>>>>> signature of the recipe is unchanged (the file I'm interested
>>>>> is
>>>>> DMVerity hash data that I generate as an additional filesystem
>>>>> type)
>>>>>
>>>>> - Task do_compile of recipe my-initramfs-scripts depends on my-
>>>>> core-
>>>>> image:do_image_complete. I need to enforce rebuilding of this
>>>>> task
>>>>> whenever the deployed files of my-core-image have changed (the
>>>>> recipe
>>>>> uses the "root hash" generated by DMVerity as input, so that
>>>>> the
>>>>> filesystem can be verified on mount) - and of course also of
>>>>> all
>>>>> dependent tasks like the initramfs image and FIT kernel image
>>>>>
>>>>> I've experimented with things like file-checksums and
>>>>> BB_DONT_CACHE,
>>>>> but everything I've tried fails when the rebuilds of my-core-
>>>>> image
>>>>> and
>>>>> my-initramfs-scripts are supposed to happen in the same run of
>>>>> Bitbake:
>>>>>
>>>>> - file-checksums only seems to be taken into account at initial
>>>>> recipe
>>>>> parse time, not when the task I'm interested in is actually
>>>>> executed? I
>>>>> need to run bitbake twice to get the correct results.
>>>>> - When I use an anonymous Python function to put the root hash
>>>>> I'm
>>>>> interested in into a variable, Bitbake complains about
>>>>> nondeterministic
>>>>> metadata.
>>>>
>>>> There is a way of doing this, I've had to do it in the past.
>>>>
>>>> The anonymous python function needs to product the same output on
>>>> each and every
>>>> run.  If it doesn't this will cause the non-deterministic
>>>> behavior.  (It can't
>>>> rely on another items to finish building for instance.)
>>>>
>>>> So you end up having to have a recipe (with a file-checksum
>>>> and/or
>>>> anon python)
>>>> that sets up variables that will keep the hash up-to-date for
>>>> THAT
>>>> recipe.. then
>>>> your image (or other) recipes need to depend on that one.
>>>
>>> Hello Mark,
>>>
>>> I'm afraid I don't follow. How does doing something in another
>>> recipe
>>> fix the non-deterministic behaviour?
>>
>> You must always compute your internal variables in -exactly- the same
>> way.  You
>> were saying you were passing in an external object and needed to have
>> that
>> influence the values.  (Unless I am understanding incorrectly.)
>>
>>> I'd think that any solution based on the task signatures will have
>>> the
>>> same issue - that a hash correctly reflecting the inputs that go
>>> into
>>> my initramfs task would depend on the build results of my core
>>> image
>>> task, which is not available at recipe parse time.
>>
>> All signatures are computed at parse time.  They either need to be
>> computed
>> based on external (non-changing) input, or other recipes (which
>> themselves would
>> have an static hash, but can change from build to build.)
>>
>>> I think I'll have to go with nostamp after all - the unnessesary
>>> rebuilds it causes in this case are ugly, but rather negligible in
>>> terms of build time.
>>
>> So is the input changing out from under the system?  I.e. during the
>> build
>> process did it change, or are you trying to use the output of one
>> recipe to
>> create input from another without standard recipe dependency
>> behavior?
> 
> The latter. I have a regular core image built by Yocto, and I don't
> want to care about full reproducibility - so rebuilding the core image
> will give me a different image regarding timestamps etc. The second
> recipe (and its dependents) needs to rerun when the hash of the built
> image changes. This does not happen without nostamp, because no
> variable known to Bitbake can contain this hash at parse time.

Bitbake DOES know the hash of the produced image -recipe-, and that should be
enough to know if you need to rebuild.

So you'd end up with a recipe dependency set that looks something like:

final_recipe -> image_recipe -> glibc, bash, etc...

(with modern OE, you can do a multiconfig build if your final_recipe and
image_recipe are different architectures..)


or you build as two separate steps (two different bitbake commands)..


--Mark

> Matthias
> 
> 
>>
>> --Mark
>>
>>>
>>> Thanks,
>>>
>>> Matthias
>>>
>>>
>>>
>>>>
>>>> Unfortunately I no longer have access to the example where I had
>>>> to
>>>> do this in
>>>> the past, or I could try to give you something more concrete to
>>>> look
>>>> at.
>>>>
>>>> --Mark
>>>>
>>>>>
>>>>> Using nostamp would be an option, but obviously an ugly one.
>>>>>
>>>>> I'm currently on Yocto Thud / Bitbake 1.40.
>>>>>
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Matthias
>>>>>


^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2019-11-26 15:40 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-22 15:09 Forcing the rebuild of dependent tasks across recipes Matthias Schiffer
2019-11-22 16:34 ` Richard Purdie
2019-11-25  9:46   ` (EXT) " Matthias Schiffer
2019-11-22 17:15 ` Mark Hatle
2019-11-25  9:58   ` (EXT) " Matthias Schiffer
2019-11-25 14:54     ` Mark Hatle
2019-11-26  8:43       ` (EXT) " Matthias Schiffer
2019-11-26 15:40         ` Mark Hatle

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.