From: changhuaixin <changhuaixin@linux.alibaba.com>
To: changhuaixin <changhuaixin@linux.alibaba.com>
Cc: bsegall@google.com, dietmar.eggemann@arm.com,
juri.lelli@redhat.com, khlebnikov@yandex-team.ru,
linux-kernel@vger.kernel.org, mgorman@suse.de, mingo@redhat.com,
pauld@redhead.com, peterz@infradead.org, pjt@google.com,
rostedt@goodmis.org, shanpeic@linux.alibaba.com,
vincent.guittot@linaro.org, xiyou.wangcong@gmail.com
Subject: Re: [PATCH v3 0/4] sched/fair: Burstable CFS bandwidth controller
Date: Tue, 26 Jan 2021 18:18:59 +0800 [thread overview]
Message-ID: <9FD4A7E9-B545-40AB-A5B5-66DF37991474@linux.alibaba.com> (raw)
In-Reply-To: <20210121110453.18899-1-changhuaixin@linux.alibaba.com>
> On Jan 21, 2021, at 7:04 PM, Huaixin Chang <changhuaixin@linux.alibaba.com> wrote:
>
> Changelog
>
> v3:
> 1. Fix another issue reported by test robot.
> 2. Update docs as Randy Dunlap suggested.
>
> v2:
> 1. Fix an issue reported by test robot.
> 2. Rewriting docs. Appreciate any further suggestions or help.
>
> The CFS bandwidth controller limits CPU requests of a task group to
> quota during each period. However, parallel workloads might be bursty
> so that they get throttled. And they are latency sensitive at the same
> time so that throttling them is undesired.
>
> Scaling up period and quota allows greater burst capacity. But it might
> cause longer stuck till next refill. We introduce "burst" to allow
> accumulating unused quota from previous periods, and to be assigned when
> a task group requests more CPU than quota during a specific period. Thus
> allowing CPU time requests as long as the average requested CPU time is
> below quota on the long run. The maximum accumulation is capped by burst
> and is set 0 by default, thus the traditional behaviour remains.
>
> A huge drop of 99th tail latency from more than 500ms to 27ms is seen for
> real java workloads when using burst. Similar drops are seen when
> testing with schbench too:
>
> echo $$ > /sys/fs/cgroup/cpu/test/cgroup.procs
> echo 700000 > /sys/fs/cgroup/cpu/test/cpu.cfs_quota_us
> echo 100000 > /sys/fs/cgroup/cpu/test/cpu.cfs_period_us
> echo 400000 > /sys/fs/cgroup/cpu/test/cpu.cfs_burst_us
>
> # The average CPU usage is around 500%, which is 200ms CPU time
> # every 40ms.
> ./schbench -m 1 -t 30 -r 60 -c 10000 -R 500
>
> Without burst:
>
> Latency percentiles (usec)
> 50.0000th: 7
> 75.0000th: 8
> 90.0000th: 9
> 95.0000th: 10
> *99.0000th: 933
> 99.5000th: 981
> 99.9000th: 3068
> min=0, max=20054
> rps: 498.31 p95 (usec) 10 p99 (usec) 933 p95/cputime 0.10% p99/cputime 9.33%
>
> With burst:
>
> Latency percentiles (usec)
> 50.0000th: 7
> 75.0000th: 8
> 90.0000th: 9
> 95.0000th: 9
> *99.0000th: 12
> 99.5000th: 13
> 99.9000th: 19
> min=0, max=406
> rps: 498.36 p95 (usec) 9 p99 (usec) 12 p95/cputime 0.09% p99/cputime 0.12%
>
> How much workloads with benefit from burstable CFS bandwidth control
> depends on how bursty and how latency sensitive they are.
>
> Previously, Cong Wang and Konstantin Khlebnikov proposed similar
> feature:
> https://lore.kernel.org/lkml/20180522062017.5193-1-xiyou.wangcong@gmail.com/
> https://lore.kernel.org/lkml/157476581065.5793.4518979877345136813.stgit@buzz/
>
> This time we present more latency statistics and handle overflow while
> accumulating.
>
> Huaixin Chang (4):
> sched/fair: Introduce primitives for CFS bandwidth burst
> sched/fair: Make CFS bandwidth controller burstable
> sched/fair: Add cfs bandwidth burst statistics
> sched/fair: Add document for burstable CFS bandwidth control
>
> Documentation/scheduler/sched-bwc.rst | 49 +++++++++++--
> include/linux/sched/sysctl.h | 2 +
> kernel/sched/core.c | 126 +++++++++++++++++++++++++++++-----
> kernel/sched/fair.c | 58 +++++++++++++---
> kernel/sched/sched.h | 9 ++-
> kernel/sysctl.c | 18 +++++
> 6 files changed, 232 insertions(+), 30 deletions(-)
>
> --
> 2.14.4.44.g2045bb6
Ping, any new comments on this patchset? If there're no other concerns, I think it's ready to be merged?
next prev parent reply other threads:[~2021-01-26 11:36 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-17 7:46 [PATCH 0/4] sched/fair: Burstable CFS bandwidth controller Huaixin Chang
2020-12-17 7:46 ` [PATCH 1/4] sched/fair: Introduce primitives for CFS bandwidth burst Huaixin Chang
2020-12-17 13:36 ` Peter Zijlstra
2020-12-21 13:53 ` changhuaixin
2021-01-12 9:21 ` changhuaixin
2020-12-17 7:46 ` [PATCH 2/4] sched/fair: Make CFS bandwidth controller burstable Huaixin Chang
2020-12-18 9:53 ` kernel test robot
2020-12-17 7:46 ` [PATCH 3/4] sched/fair: Add cfs bandwidth burst statistics Huaixin Chang
2020-12-17 7:46 ` [PATCH 4/4] sched/fair: Add document for burstable CFS bandwidth control Huaixin Chang
2020-12-17 21:25 ` [PATCH 0/4] sched/fair: Burstable CFS bandwidth controller Benjamin Segall
2021-01-20 12:27 ` [PATCH v2 " Huaixin Chang
2021-01-20 12:27 ` [PATCH 1/4] sched/fair: Introduce primitives for CFS bandwidth burst Huaixin Chang
2021-01-20 12:27 ` [PATCH 2/4] sched/fair: Make CFS bandwidth controller burstable Huaixin Chang
2021-01-20 17:06 ` kernel test robot
2021-01-20 18:33 ` kernel test robot
2021-01-20 22:01 ` kernel test robot
2021-01-20 22:01 ` [RFC PATCH] sched/fair: __refill_cfs_bandwidth_runtime() can be static kernel test robot
2021-01-20 12:27 ` [PATCH 3/4] sched/fair: Add cfs bandwidth burst statistics Huaixin Chang
2021-01-20 12:27 ` [PATCH 4/4] sched/fair: Add document for burstable CFS bandwidth control Huaixin Chang
2021-01-20 19:48 ` Randy Dunlap
2021-01-21 11:04 ` [PATCH v3 0/4] sched/fair: Burstable CFS bandwidth controller Huaixin Chang
2021-01-21 11:04 ` [PATCH v3 1/4] sched/fair: Introduce primitives for CFS bandwidth burst Huaixin Chang
2021-03-10 12:39 ` Peter Zijlstra
2021-01-21 11:04 ` [PATCH v3 2/4] sched/fair: Make CFS bandwidth controller burstable Huaixin Chang
2021-03-10 13:04 ` Peter Zijlstra
2021-03-12 13:54 ` changhuaixin
2021-03-16 10:55 ` Peter Zijlstra
2021-01-21 11:04 ` [PATCH v3 3/4] sched/fair: Add cfs bandwidth burst statistics Huaixin Chang
2021-03-10 13:10 ` Peter Zijlstra
2021-01-21 11:04 ` [PATCH v3 4/4] sched/fair: Add document for burstable CFS bandwidth control Huaixin Chang
2021-03-10 13:17 ` Peter Zijlstra
2021-01-26 10:18 ` changhuaixin [this message]
2021-03-10 12:26 ` [PATCH v3 0/4] sched/fair: Burstable CFS bandwidth controller Peter Zijlstra
2021-02-09 13:17 ` Odin Ugedal
2021-02-09 10:28 ` Tejun Heo
2021-02-27 13:48 ` changhuaixin
2021-03-10 11:11 ` Odin Ugedal
2021-03-12 13:26 ` changhuaixin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9FD4A7E9-B545-40AB-A5B5-66DF37991474@linux.alibaba.com \
--to=changhuaixin@linux.alibaba.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=khlebnikov@yandex-team.ru \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=pauld@redhead.com \
--cc=peterz@infradead.org \
--cc=pjt@google.com \
--cc=rostedt@goodmis.org \
--cc=shanpeic@linux.alibaba.com \
--cc=vincent.guittot@linaro.org \
--cc=xiyou.wangcong@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).