Re: [yocto] [] [PATCH yocto-autobuilder-helper] config.json: set oe-time-dd-test.sh timeout to 3 seconds

Re: [yocto] [] [PATCH yocto-autobuilder-helper] config.json: set oe-time-dd-test.sh timeout to 3 seconds

[Randy MacLeod]

[-- Attachment #1.1: Type: text/plain, Size: 2955 bytes --]

On 2023-02-02 17:00, Randy MacLeod via lists.yoctoproject.org wrote:
> For the month of January 2023, the distribution of dd times has a long
> tail that extends to 13 seconds with 2 events exceeding the current
> limit of 30 seconds.
>
> Reduce the timeout to 3 seconds based on the observed distribution of
> dd times, which would result in the timout triggering about 20 times a month.
> That's enough data to be useful but not so much that it's overwhelming the
> logging or the people who will analyze it. It also avoids the rapid increase
> in the tail of the distribution which starts to rise exponentially under 2 seconds.
> It's sensible response time for people to expect the system to have.


See attached graphs!

I don't know why there are two peaks that you can easily see on the 
linear scale distribution
but the 3 graphs show why I picked the 3 second cutoff.

Below is histogram data for a 0.1 second bin of  the tail of the 
distribution.

I can share the raw data or the 0.001 ms binned version if anyone is 
interested.

../Randy

      53 1.0
      39 1.1
      31 1.2
      22 1.3
      14 1.4
      23 1.5
      10 1.6
      14 1.7
       7 1.8
       5 1.9
       6 2.0
       7 2.1
       3 2.2
       3 2.3
       2 2.4
       5 2.5
       1 2.6
       2 2.7
       1 2.9
       1 3.1
       3 3.3
       1 3.4
       4 3.6
       1 3.8
       1 3.9
       1 4.1
       2 4.2
       2 4.5
       1 5.0
       1 5.1
       1 5.4
       1 6.3
       1 13.

>
> Signed-off-by: Randy MacLeod<Randy.MacLeod@windriver.com>
> ---
>   config.json | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/config.json b/config.json
> index 446528a..e50ec44 100644
> --- a/config.json
> +++ b/config.json
> @@ -68,7 +68,7 @@
>               "RUNQEMU_TMPFS_DIR = '/home/pokybuild/tmp'",
>               "BB_HEARTBEAT_EVENT = '60'",
>               "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
> -            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh -c 100 -t 15'",
> +            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh -c 100 -t 3'",
>               "BB_LOG_HOST_STAT_ON_FAILURE = '1'",
>               "BB_LOG_HOST_STAT_CMDS_FAILURE = 'oe-time-dd-test.sh -l'",
>               "SDK_TOOLCHAIN_LANGS += 'rust'",
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#59149):https://lists.yoctoproject.org/g/yocto/message/59149
> Mute This Topic:https://lists.yoctoproject.org/mt/96710817/3616765
> Group Owner:yocto+owner@lists.yoctoproject.org
> Unsubscribe:https://lists.yoctoproject.org/g/yocto/unsub  [randy.macleod@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

-- 
# Randy MacLeod
# Wind River Linux

[-- Attachment #1.2: Type: text/html, Size: 5129 bytes --]

[-- Attachment #2: yp-ab-dd-time-distrib--0-6s--jan-2023.png --]
[-- Type: image/png, Size: 7295 bytes --]

[-- Attachment #3: yp-ab-dd-time-distrib--0-1.1s--jan-2023.png --]
[-- Type: image/png, Size: 9274 bytes --]

[-- Attachment #4: yp-ab-dd-time-distrib--0-0.2s-lin--jan-2023.png --]
[-- Type: image/png, Size: 7807 bytes --]

Re: [yocto] [] [PATCH yocto-autobuilder-helper] config.json: set oe-time-dd-test.sh timeout to 3 seconds

[Randy MacLeod]

[-- Attachment #1: Type: text/plain, Size: 507 bytes --]

On 2023-02-02 17:10, Randy MacLeod via lists.yoctoproject.org wrote:
> I don't know why there are two peaks that you can easily see on the 
> linear scale distribution
> but the 3 graphs show why I picked the 3 second cutoff.

I'm guessing but I suspect that the lower latency distribution is just 
from when

the Yocto AB workers are idle. The builders in the cluster at WR are always
busy, sometimes needlessly,  so I didn't consider that initially.

-- 
# Randy MacLeod
# Wind River Linux

[-- Attachment #2: Type: text/html, Size: 878 bytes --]

Re: [yocto] [] [PATCH yocto-autobuilder-helper] config.json: set oe-time-dd-test.sh timeout to 3 seconds

[Randy MacLeod]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On 2023-02-02 18:32, Randy MacLeod via lists.yoctoproject.org wrote:
> On 2023-02-02 17:10, Randy MacLeod via lists.yoctoproject.org wrote:
>> I don't know why there are two peaks that you can easily see on the 
>> linear scale distribution
>> but the 3 graphs show why I picked the 3 second cutoff.
>
> I'm guessing but I suspect that the lower latency distribution is just 
> from when
>
> the Yocto AB workers are idle. The builders in the cluster at WR are 
> always
> busy, sometimes needlessly,  so I didn't consider that initially.
>

Sigh, that's obviously wrong since this data is only collected when 
running bitbake!

Maybe it's the arm vs the intel workers?

I'll stop speculating until I look at the data a bit more...


-- 

# Randy MacLeod
# Wind River Linux

[-- Attachment #2: Type: text/html, Size: 1427 bytes --]