Re: [bitbake-devel] [2.0][PATCH 1/5] runqueue: fix PSI check calculation

* Re: [bitbake-devel] [2.0][PATCH 1/5] runqueue: fix PSI check calculation
       [not found] <17964777D58C012A.3202@lists.openembedded.org>
@ 2023-11-10 14:06 ` Martin Jansa
  2024-02-15 23:06   ` Randy MacLeod
  0 siblings, 1 reply; 4+ messages in thread
From: Martin Jansa @ 2023-11-10 14:06 UTC (permalink / raw)
  To: martin.jansa; +Cc: bitbake-devel, steve, Chen Qi, Richard Purdie, Randy Macleod

[-- Attachment #1: Type: text/plain, Size: 5116 bytes --]

I forgot to include [RFC] tag in the subject, because these changes might
be a bit controversial.

From my testing:
https://github.com/shr-project/test-oe-build-time/commit/d5111f4472ac397c0f1197eb6366ac7d2e56453f
the reasonable pressure values for 2.0 (in kirkstone) and 2.6 (in nanbield)
are significantly different due to these changes in PSI calculation and
logic so while 1000 was reasonable value for my system in 2.0, I need
100000 with 2.6.

I've backported these changes long time ago for my local builds to use the
same PSI values for all builds.

And the controversy is that backporting these will change the expected
values (so kind of change in behavior which might not be suitable for
stable release). But the commits make it sound as a bug in the logic, so we
can also consider them as bug fixes.

The change in expected pressure values was reported in:
https://lists.openembedded.org/g/bitbake-devel/message/14942

If it's not a bug to be fixed in 2.0 then maybe we should mention different
PSI values in release notes for nanbield (I know I'm too late for that).

Regards,

On Fri, Nov 10, 2023 at 2:54 PM Martin Jansa via lists.openembedded.org
<martin.jansa=gmail.com@lists.openembedded.org> wrote:

> From: Chen Qi <Qi.Chen@windriver.com>
>
> The current PSI check calculation does not take into consideration
> the possibility of the time interval between last check and current
> check being much larger than 1s. In fact, the current behavior does
> not match what the manual says about BB_PRESSURE_MAX_XXX, even if
> the value is set to upper limit, 1000000, we still get many blocks
> on new task launch. The difference between 'total' should be divided
> by the time interval if it's larger than 1s.
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  lib/bb/runqueue.py | 13 +++++++++----
>  1 file changed, 9 insertions(+), 4 deletions(-)
>
> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> index 8605c46c8..3e89c38bc 100644
> --- a/lib/bb/runqueue.py
> +++ b/lib/bb/runqueue.py
> @@ -198,15 +198,20 @@ class RunQueueScheduler(object):
>                  curr_cpu_pressure =
> cpu_pressure_fds.readline().split()[4].split("=")[1]
>                  curr_io_pressure =
> io_pressure_fds.readline().split()[4].split("=")[1]
>                  curr_memory_pressure =
> memory_pressure_fds.readline().split()[4].split("=")[1]
> -                exceeds_cpu_pressure =  self.rq.max_cpu_pressure and
> (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) >
> self.rq.max_cpu_pressure
> -                exceeds_io_pressure =  self.rq.max_io_pressure and
> (float(curr_io_pressure) - float(self.prev_io_pressure)) >
> self.rq.max_io_pressure
> -                exceeds_memory_pressure = self.rq.max_memory_pressure and
> (float(curr_memory_pressure) - float(self.prev_memory_pressure)) >
> self.rq.max_memory_pressure
>                  now = time.time()
> -                if now - self.prev_pressure_time > 1.0:
> +                tdiff = now - self.prev_pressure_time
> +                if tdiff > 1.0:
> +                    exceeds_cpu_pressure =  self.rq.max_cpu_pressure and
> (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) / tdiff >
> self.rq.max_cpu_pressure
> +                    exceeds_io_pressure =  self.rq.max_io_pressure and
> (float(curr_io_pressure) - float(self.prev_io_pressure)) / tdiff >
> self.rq.max_io_pressure
> +                    exceeds_memory_pressure = self.rq.max_memory_pressure
> and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) /
> tdiff > self.rq.max_memory_pressure
>                      self.prev_cpu_pressure = curr_cpu_pressure
>                      self.prev_io_pressure = curr_io_pressure
>                      self.prev_memory_pressure = curr_memory_pressure
>                      self.prev_pressure_time = now
> +                else:
> +                    exceeds_cpu_pressure =  self.rq.max_cpu_pressure and
> (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) >
> self.rq.max_cpu_pressure
> +                    exceeds_io_pressure =  self.rq.max_io_pressure and
> (float(curr_io_pressure) - float(self.prev_io_pressure)) >
> self.rq.max_io_pressure
> +                    exceeds_memory_pressure = self.rq.max_memory_pressure
> and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) >
> self.rq.max_memory_pressure
>              return (exceeds_cpu_pressure or exceeds_io_pressure or
> exceeds_memory_pressure)
>          return False
>
> --
> 2.42.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15494):
> https://lists.openembedded.org/g/bitbake-devel/message/15494
> Mute This Topic: https://lists.openembedded.org/mt/102506693/3617156
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> martin.jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 6826 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread