[yocto-autobuilder-helper][PATCH] config.json: track system load with PARALLEL_MAKE

[yocto-autobuilder-helper][PATCH] config.json: track system load with PARALLEL_MAKE

[Trevor Gamblin]

This adds the "-l" option to PARALLEL_MAKE in config.json with an
initial testing value of 100 (100% system load). This option is supported
by both Make and Ninja. However, we also require the "--debug=j" option
to be passed to Make in order for the latter to report perceived system
load in the do_compile logs, and since this option is not supported by
Ninja, also add EXTRA_OEMAKE to the EXTRAVARS so that we can determine if
the target load percentage needs to be adjusted.

Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
---
 config.json | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/config.json b/config.json
index f54081b..7fc89ea 100644
--- a/config.json
+++ b/config.json
@@ -44,7 +44,7 @@
             "PREMIRRORS = ''",
             "BB_GENERATE_MIRROR_TARBALLS = '1'",
             "BB_NUMBER_THREADS = '16'",
-            "PARALLEL_MAKE = '-j 16'",
+            "PARALLEL_MAKE = '-j 16 -l 100'",
             "XZ_MEMLIMIT = '5%'",
             "XZ_THREADS = '8'",
             "BB_TASK_NICE_LEVEL = '5'",
@@ -61,7 +61,8 @@
             "RUNQEMU_TMPFS_DIR = '/home/pokybuild/tmp'",
             "BB_HEARTBEAT_EVENT = '60'",
             "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
-            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh 100'"
+            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh 100'",
+            "EXTRA_OEMAKE = ' --debug=j'"
         ]
     },
     "templates" : {
-- 
2.31.1

Re: [yocto] [yocto-autobuilder-helper][PATCH] config.json: track system load with PARALLEL_MAKE

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]

I seem to vaguely remember that -l is not actually taking a percentage, but
an absolute value that is specific to the amount of CPU cores a system has.
Have you verified your assumption?

Alex

On Mon, 12 Jul 2021 at 14:11, Trevor Gamblin <trevor.gamblin@windriver.com>
wrote:

> This adds the "-l" option to PARALLEL_MAKE in config.json with an
> initial testing value of 100 (100% system load). This option is supported
> by both Make and Ninja. However, we also require the "--debug=j" option
> to be passed to Make in order for the latter to report perceived system
> load in the do_compile logs, and since this option is not supported by
> Ninja, also add EXTRA_OEMAKE to the EXTRAVARS so that we can determine if
> the target load percentage needs to be adjusted.
>
> Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
> ---
>  config.json | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/config.json b/config.json
> index f54081b..7fc89ea 100644
> --- a/config.json
> +++ b/config.json
> @@ -44,7 +44,7 @@
>              "PREMIRRORS = ''",
>              "BB_GENERATE_MIRROR_TARBALLS = '1'",
>              "BB_NUMBER_THREADS = '16'",
> -            "PARALLEL_MAKE = '-j 16'",
> +            "PARALLEL_MAKE = '-j 16 -l 100'",
>              "XZ_MEMLIMIT = '5%'",
>              "XZ_THREADS = '8'",
>              "BB_TASK_NICE_LEVEL = '5'",
> @@ -61,7 +61,8 @@
>              "RUNQEMU_TMPFS_DIR = '/home/pokybuild/tmp'",
>              "BB_HEARTBEAT_EVENT = '60'",
>              "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
> -            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh 100'"
> +            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh 100'",
> +            "EXTRA_OEMAKE = ' --debug=j'"
>          ]
>      },
>      "templates" : {
> --
> 2.31.1
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 2752 bytes --]

Re: [yocto] [yocto-autobuilder-helper][PATCH] config.json: track system load with PARALLEL_MAKE

[Trevor Gamblin]

[-- Attachment #1: Type: text/plain, Size: 3003 bytes --]


On 2021-07-12 12:04 p.m., Alexander Kanavin wrote:
>
> **[Please note: This e-mail is from an EXTERNAL e-mail address]
>
> I seem to vaguely remember that -l is not actually taking a 
> percentage, but an absolute value that is specific to the amount of 
> CPU cores a system has. Have you verified your assumption?
You are right that it's not actually a percentage and is instead tied to 
the number of cores. I'll correct the patch body to reflect this when I 
inevitably send a v2. With the latest testing on the AB, my suspicion 
that "-l 100" is too generous for most machines is being confirmed...
>
> Alex
>
> On Mon, 12 Jul 2021 at 14:11, Trevor Gamblin 
> <trevor.gamblin@windriver.com <mailto:trevor.gamblin@windriver.com>> 
> wrote:
>
>     This adds the "-l" option to PARALLEL_MAKE in config.json with an
>     initial testing value of 100 (100% system load). This option is
>     supported
>     by both Make and Ninja. However, we also require the "--debug=j"
>     option
>     to be passed to Make in order for the latter to report perceived
>     system
>     load in the do_compile logs, and since this option is not supported by
>     Ninja, also add EXTRA_OEMAKE to the EXTRAVARS so that we can
>     determine if
>     the target load percentage needs to be adjusted.
>
>     Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com
>     <mailto:trevor.gamblin@windriver.com>>
>     ---
>      config.json | 5 +++--
>      1 file changed, 3 insertions(+), 2 deletions(-)
>
>     diff --git a/config.json b/config.json
>     index f54081b..7fc89ea 100644
>     --- a/config.json
>     +++ b/config.json
>     @@ -44,7 +44,7 @@
>                  "PREMIRRORS = ''",
>                  "BB_GENERATE_MIRROR_TARBALLS = '1'",
>                  "BB_NUMBER_THREADS = '16'",
>     -            "PARALLEL_MAKE = '-j 16'",
>     +            "PARALLEL_MAKE = '-j 16 -l 100'",
>                  "XZ_MEMLIMIT = '5%'",
>                  "XZ_THREADS = '8'",
>                  "BB_TASK_NICE_LEVEL = '5'",
>     @@ -61,7 +61,8 @@
>                  "RUNQEMU_TMPFS_DIR = '/home/pokybuild/tmp'",
>                  "BB_HEARTBEAT_EVENT = '60'",
>                  "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
>     -            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh
>     <https://urldefense.com/v3/__http://oe-time-dd-test.sh__;!!AjveYdw8EvQ!IMv8JyMfE5mhi_q22gXmTjUg883rq2tckayxOn-9FWThX9U5fJRJRkJYL6L8m9A1NvmBBg$>
>     100'"
>     +            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh
>     <https://urldefense.com/v3/__http://oe-time-dd-test.sh__;!!AjveYdw8EvQ!IMv8JyMfE5mhi_q22gXmTjUg883rq2tckayxOn-9FWThX9U5fJRJRkJYL6L8m9A1NvmBBg$>
>     100'",
>     +            "EXTRA_OEMAKE = ' --debug=j'"
>              ]
>          },
>          "templates" : {
>     -- 
>     2.31.1
>
>
>     
>

[-- Attachment #2: Type: text/html, Size: 5656 bytes --]