[PATCH] meta-ti: Set the --jobs based on host cpu_count

[PATCH] meta-ti: Set the --jobs based on host cpu_count

[Luca Ceresoli]

From: "Kemal R. Shakir" <kemal@ti.com>

Using BB_NUMBER_THREADS to set variables causes an "basehash value changed"
error when BB_NUMBER_THREADS is changed between successive builds.

Fixes:

  ERROR: When reparsing .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_configure, the basehash value changed from 63897a188012cedde03f756b5dbbfa70 to 13b80bf4f4327c367bd201eb4aa32ad7. The metadata is not deterministic and this needs to be fixed.
  ERROR: When reparsing .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_compile, the basehash value changed from 2fb59a7db2a2f69d2c25c06c1b6a4245 to 314c664cc040afae2800fcc378dd9ae0. The metadata is not deterministic and this needs to be fixed.

Steps to reproduce:

  BB_NUMBER_THREADS="3" bitbake ti-ipc-rtos
  BB_NUMBER_THREADS="4" bitbake -c cleansstate osal-rtos
  BB_NUMBER_THREADS="5" bitbake ti-ipc-rtos

Reported here: https://e2e.ti.com/support/processors/f/791/p/825142/3059419
Patch from Kemal R. Shakir
Reported-by: Matthew McKee (e-mail unknown)
[Luca: add detailed commit message]
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>

---

I'm having the same problem and found this patch solving it on e2e.ti.com,
but it does not appear on any git branch nor in the meta-ti mailing list. I
tested it on the thud branch and fixes the problem. Can it be applied to
both thud and future branches?

Thanks
---
 classes/ti-pdk.bbclass         | 2 +-
 recipes-ti/ipc/ti-ipc-rtos.inc | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass
index f0e6f2d2738a..5f78bafa94b5 100644
--- a/classes/ti-pdk.bbclass
+++ b/classes/ti-pdk.bbclass
@@ -68,7 +68,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1"
 
 TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}"
 
-PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
+PARALLEL_XDC = "--jobs=${@oe.utils.cpu_count()}"
 PARALLEL_MAKE = ""
 
 def get_doxygen_support(d):
diff --git a/recipes-ti/ipc/ti-ipc-rtos.inc b/recipes-ti/ipc/ti-ipc-rtos.inc
index dcd6309fe2ab..1b21375edd33 100644
--- a/recipes-ti/ipc/ti-ipc-rtos.inc
+++ b/recipes-ti/ipc/ti-ipc-rtos.inc
@@ -89,7 +89,7 @@ EXTRA_OEMAKE = "\
     XDC_INSTALL_DIR="${XDC_INSTALL_DIR}" \
     BIOS_INSTALL_DIR="${SYSBIOS_INSTALL_DIR}" \
     ${IPC_TARGETS} \
-    JOBS="--jobs=${BB_NUMBER_THREADS}" \
+    JOBS="--jobs=${@oe.utils.cpu_count()}" \
     JAVA_TOOL_OPTIONS=-Xss2560k \
 "
 EXTRA_OEMAKE_append_am65xx = " PDK_INSTALL_DIR=${PDK_INSTALL_DIR}"
-- 
2.24.0

Re: [meta-ti] [PATCH] meta-ti: Set the --jobs based on host cpu_count

[Khem Raj]

On Fri, Nov 29, 2019 at 6:48 AM Luca Ceresoli <luca@lucaceresoli.net> wrote:
>
> From: "Kemal R. Shakir" <kemal@ti.com>
>
> Using BB_NUMBER_THREADS to set variables causes an "basehash value changed"
> error when BB_NUMBER_THREADS is changed between successive builds.
>
> Fixes:
>
>   ERROR: When reparsing .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_configure, the basehash value changed from 63897a188012cedde03f756b5dbbfa70 to 13b80bf4f4327c367bd201eb4aa32ad7. The metadata is not deterministic and this needs to be fixed.
>   ERROR: When reparsing .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_compile, the basehash value changed from 2fb59a7db2a2f69d2c25c06c1b6a4245 to 314c664cc040afae2800fcc378dd9ae0. The metadata is not deterministic and this needs to be fixed.
>
> Steps to reproduce:
>
>   BB_NUMBER_THREADS="3" bitbake ti-ipc-rtos
>   BB_NUMBER_THREADS="4" bitbake -c cleansstate osal-rtos
>   BB_NUMBER_THREADS="5" bitbake ti-ipc-rtos
>
> Reported here: https://e2e.ti.com/support/processors/f/791/p/825142/3059419
> Patch from Kemal R. Shakir
> Reported-by: Matthew McKee (e-mail unknown)
> [Luca: add detailed commit message]
> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
> Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
>
> ---
>
> I'm having the same problem and found this patch solving it on e2e.ti.com,
> but it does not appear on any git branch nor in the meta-ti mailing list. I
> tested it on the thud branch and fixes the problem. Can it be applied to
> both thud and future branches?
>
> Thanks
> ---
>  classes/ti-pdk.bbclass         | 2 +-
>  recipes-ti/ipc/ti-ipc-rtos.inc | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass
> index f0e6f2d2738a..5f78bafa94b5 100644
> --- a/classes/ti-pdk.bbclass
> +++ b/classes/ti-pdk.bbclass
> @@ -68,7 +68,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1"
>
>  TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}"
>
> -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
> +PARALLEL_XDC = "--jobs=${@oe.utils.cpu_count()}"

this may create another variableto control when limting parallelism so
expressing it in
terms of BB_NUMBER_THREAD s desired

perhaps you can try adding

do_configure[vardepsexclude] = "PARALLEL_XDC"
do_compile[vardepsexclude] = "PARALLEL_XDC"

>  PARALLEL_MAKE = ""
>
>  def get_doxygen_support(d):
> diff --git a/recipes-ti/ipc/ti-ipc-rtos.inc b/recipes-ti/ipc/ti-ipc-rtos.inc
> index dcd6309fe2ab..1b21375edd33 100644
> --- a/recipes-ti/ipc/ti-ipc-rtos.inc
> +++ b/recipes-ti/ipc/ti-ipc-rtos.inc
> @@ -89,7 +89,7 @@ EXTRA_OEMAKE = "\
>      XDC_INSTALL_DIR="${XDC_INSTALL_DIR}" \
>      BIOS_INSTALL_DIR="${SYSBIOS_INSTALL_DIR}" \
>      ${IPC_TARGETS} \
> -    JOBS="--jobs=${BB_NUMBER_THREADS}" \
> +    JOBS="--jobs=${@oe.utils.cpu_count()}" \

something similar here.

>      JAVA_TOOL_OPTIONS=-Xss2560k \
>  "
>  EXTRA_OEMAKE_append_am65xx = " PDK_INSTALL_DIR=${PDK_INSTALL_DIR}"
> --
> 2.24.0
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#12534): https://lists.yoctoproject.org/g/meta-ti/message/12534
> Mute This Topic: https://lists.yoctoproject.org/mt/64369908/1997914
> Group Owner: meta-ti+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/meta-ti/unsub  [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-

Re: [meta-ti] [PATCH] meta-ti: Set the --jobs based on host cpu_count

[Luca Ceresoli]

Hi Khem,

On 29/11/19 17:17, Khem Raj wrote:
> On Fri, Nov 29, 2019 at 6:48 AM Luca Ceresoli <luca@lucaceresoli.net> wrote:
>>
>> From: "Kemal R. Shakir" <kemal@ti.com>
>>
>> Using BB_NUMBER_THREADS to set variables causes an "basehash value changed"
>> error when BB_NUMBER_THREADS is changed between successive builds.
>>
>> Fixes:
>>
>>   ERROR: When reparsing .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_configure, the basehash value changed from 63897a188012cedde03f756b5dbbfa70 to 13b80bf4f4327c367bd201eb4aa32ad7. The metadata is not deterministic and this needs to be fixed.
>>   ERROR: When reparsing .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_compile, the basehash value changed from 2fb59a7db2a2f69d2c25c06c1b6a4245 to 314c664cc040afae2800fcc378dd9ae0. The metadata is not deterministic and this needs to be fixed.
>>
>> Steps to reproduce:
>>
>>   BB_NUMBER_THREADS="3" bitbake ti-ipc-rtos
>>   BB_NUMBER_THREADS="4" bitbake -c cleansstate osal-rtos
>>   BB_NUMBER_THREADS="5" bitbake ti-ipc-rtos
>>
>> Reported here: https://e2e.ti.com/support/processors/f/791/p/825142/3059419
>> Patch from Kemal R. Shakir
>> Reported-by: Matthew McKee (e-mail unknown)
>> [Luca: add detailed commit message]
>> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
>> Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
>>
>> ---
>>
>> I'm having the same problem and found this patch solving it on e2e.ti.com,
>> but it does not appear on any git branch nor in the meta-ti mailing list. I
>> tested it on the thud branch and fixes the problem. Can it be applied to
>> both thud and future branches?
>>
>> Thanks
>> ---
>>  classes/ti-pdk.bbclass         | 2 +-
>>  recipes-ti/ipc/ti-ipc-rtos.inc | 2 +-
>>  2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass
>> index f0e6f2d2738a..5f78bafa94b5 100644
>> --- a/classes/ti-pdk.bbclass
>> +++ b/classes/ti-pdk.bbclass
>> @@ -68,7 +68,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1"
>>
>>  TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}"
>>
>> -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
>> +PARALLEL_XDC = "--jobs=${@oe.utils.cpu_count()}"
> 
> this may create another variableto control when limting parallelism so
> expressing it in
> terms of BB_NUMBER_THREAD s desired
> 
> perhaps you can try adding
> 
> do_configure[vardepsexclude] = "PARALLEL_XDC"
> do_compile[vardepsexclude] = "PARALLEL_XDC"

Thank you for your review.

I agree the solution in this patch is not ideal as it removes ability to
control the build load.

However using BB_NUMBER_THREADS (with our without vardepsexclude) is
also not quite correct. PARALLEL_XDC should have to do with
PARALLEL_MAKE, not BB_NUMBER_THREADS.

But PARALLEL_MAKE is not a number, so using PARALLEL_MAKE would be
correct but it's not doable, and using BB_NUMBER_THREADS is doable but
not quite correct... :(

Bottom line: for lack of better ideas, I'm sending a new patch that
keeps using BB_NUMBER_THREADS and takes your suggestion above. I guess
it should be OK for most users. At least it's not worse than the current
behavior and can still be improved later.

-- 
Luca

Re: [meta-ti] [PATCH] meta-ti: Set the --jobs based on host cpu_count

[Khem Raj]

On Mon, Dec 2, 2019 at 4:20 AM Luca Ceresoli <luca@lucaceresoli.net> wrote:
>
> Hi Khem,
>
> On 29/11/19 17:17, Khem Raj wrote:
> > On Fri, Nov 29, 2019 at 6:48 AM Luca Ceresoli <luca@lucaceresoli.net> wrote:
> >>
> >> From: "Kemal R. Shakir" <kemal@ti.com>
> >>
> >> Using BB_NUMBER_THREADS to set variables causes an "basehash value changed"
> >> error when BB_NUMBER_THREADS is changed between successive builds.
> >>
> >> Fixes:
> >>
> >>   ERROR: When reparsing .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_configure, the basehash value changed from 63897a188012cedde03f756b5dbbfa70 to 13b80bf4f4327c367bd201eb4aa32ad7. The metadata is not deterministic and this needs to be fixed.
> >>   ERROR: When reparsing .../meta-ti/recipes-ti/ipc/ti-ipc-rtos_git.bb.do_compile, the basehash value changed from 2fb59a7db2a2f69d2c25c06c1b6a4245 to 314c664cc040afae2800fcc378dd9ae0. The metadata is not deterministic and this needs to be fixed.
> >>
> >> Steps to reproduce:
> >>
> >>   BB_NUMBER_THREADS="3" bitbake ti-ipc-rtos
> >>   BB_NUMBER_THREADS="4" bitbake -c cleansstate osal-rtos
> >>   BB_NUMBER_THREADS="5" bitbake ti-ipc-rtos
> >>
> >> Reported here: https://e2e.ti.com/support/processors/f/791/p/825142/3059419
> >> Patch from Kemal R. Shakir
> >> Reported-by: Matthew McKee (e-mail unknown)
> >> [Luca: add detailed commit message]
> >> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
> >> Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
> >>
> >> ---
> >>
> >> I'm having the same problem and found this patch solving it on e2e.ti.com,
> >> but it does not appear on any git branch nor in the meta-ti mailing list. I
> >> tested it on the thud branch and fixes the problem. Can it be applied to
> >> both thud and future branches?
> >>
> >> Thanks
> >> ---
> >>  classes/ti-pdk.bbclass         | 2 +-
> >>  recipes-ti/ipc/ti-ipc-rtos.inc | 2 +-
> >>  2 files changed, 2 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass
> >> index f0e6f2d2738a..5f78bafa94b5 100644
> >> --- a/classes/ti-pdk.bbclass
> >> +++ b/classes/ti-pdk.bbclass
> >> @@ -68,7 +68,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1"
> >>
> >>  TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}"
> >>
> >> -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
> >> +PARALLEL_XDC = "--jobs=${@oe.utils.cpu_count()}"
> >
> > this may create another variableto control when limting parallelism so
> > expressing it in
> > terms of BB_NUMBER_THREAD s desired
> >
> > perhaps you can try adding
> >
> > do_configure[vardepsexclude] = "PARALLEL_XDC"
> > do_compile[vardepsexclude] = "PARALLEL_XDC"
>
> Thank you for your review.
>
> I agree the solution in this patch is not ideal as it removes ability to
> control the build load.
>
> However using BB_NUMBER_THREADS (with our without vardepsexclude) is
> also not quite correct. PARALLEL_XDC should have to do with
> PARALLEL_MAKE, not BB_NUMBER_THREADS.
>
> But PARALLEL_MAKE is not a number, so using PARALLEL_MAKE would be
> correct but it's not doable, and using BB_NUMBER_THREADS is doable but
> not quite correct... :(
>
> Bottom line: for lack of better ideas, I'm sending a new patch that
> keeps using BB_NUMBER_THREADS and takes your suggestion above. I guess
> it should be OK for most users. At least it's not worse than the current
> behavior and can still be improved later.
>

another option would be to use parallel_make_argument to contruct what you want
e.g. see how boost does it

https://git.openembedded.org/openembedded-core/tree/meta/recipes-support/boost/boost.inc#n141

> --
> Luca

Re: [meta-ti] [PATCH] meta-ti: Set the --jobs based on host cpu_count

[Luca Ceresoli]

Hi Khem,

On 02/12/19 18:09, Khem Raj wrote:
[...]
>>>> --- a/classes/ti-pdk.bbclass
>>>> +++ b/classes/ti-pdk.bbclass
>>>> @@ -68,7 +68,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1"
>>>>
>>>>  TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}"
>>>>
>>>> -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
>>>> +PARALLEL_XDC = "--jobs=${@oe.utils.cpu_count()}"
>>>
>>> this may create another variableto control when limting parallelism so
>>> expressing it in
>>> terms of BB_NUMBER_THREAD s desired
>>>
>>> perhaps you can try adding
>>>
>>> do_configure[vardepsexclude] = "PARALLEL_XDC"
>>> do_compile[vardepsexclude] = "PARALLEL_XDC"
>>
>> Thank you for your review.
>>
>> I agree the solution in this patch is not ideal as it removes ability to
>> control the build load.
>>
>> However using BB_NUMBER_THREADS (with our without vardepsexclude) is
>> also not quite correct. PARALLEL_XDC should have to do with
>> PARALLEL_MAKE, not BB_NUMBER_THREADS.
>>
>> But PARALLEL_MAKE is not a number, so using PARALLEL_MAKE would be
>> correct but it's not doable, and using BB_NUMBER_THREADS is doable but
>> not quite correct... :(
>>
>> Bottom line: for lack of better ideas, I'm sending a new patch that
>> keeps using BB_NUMBER_THREADS and takes your suggestion above. I guess
>> it should be OK for most users. At least it's not worse than the current
>> behavior and can still be improved later.
>>
> 
> another option would be to use parallel_make_argument to contruct what you want
> e.g. see how boost does it
> 
> https://git.openembedded.org/openembedded-core/tree/meta/recipes-support/boost/boost.inc#n141

Oh, good idea, thanks! It looks like the correct solution to use the
proper variable (PARALLEL_MAKE) in place of BB_NUMBER_THREADS.

But surprisingly it also fixes the "basehash value changed" problem. How
can this happen? My patch is simply:

--- a/classes/ti-pdk.bbclass
+++ b/classes/ti-pdk.bbclass
-PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
+PARALLEL_XDC = "${@oe.utils.parallel_make_argument(d, '--jobs=%d')}"

and

--- a/recipes-ti/ipc/ti-ipc-rtos.inc
+++ b/recipes-ti/ipc/ti-ipc-rtos.inc
+PARALLEL_XDC = "${@oe.utils.parallel_make_argument(d, '--jobs=%d')}"
 EXTRA_OEMAKE = "\
     PLATFORM=${PLATFORM} \
     XDC_INSTALL_DIR="${XDC_INSTALL_DIR}" \
     BIOS_INSTALL_DIR="${SYSBIOS_INSTALL_DIR}" \
     ${IPC_TARGETS} \
-    JOBS="--jobs=${BB_NUMBER_THREADS}" \
+    ${PARALLEL_XDC} \
     JAVA_TOOL_OPTIONS=-Xss2560k \
 "

So I wonder why the signature is not considered as changing since
EXTRA_OEMAKE *is* changing from one build to another with a different
PARALLEL_MAKE.

Any hints?

Whatever the reason, this version definitely supersedes v2, and after
some more testing I'll send v3 with the above implementation.

-- 
Luca