From: changhuaixin <changhuaixin@linux.alibaba.com>
To: Phil Auld <pauld@redhat.com>
Cc: changhuaixin <changhuaixin@linux.alibaba.com>,
Peter Zijlstra <peterz@infradead.org>,
Benjamin Segall <bsegall@google.com>,
dietmar.eggemann@arm.com, juri.lelli@redhat.com,
khlebnikov@yandex-team.ru,
open list <linux-kernel@vger.kernel.org>,
mgorman@suse.de, mingo@redhat.com, Odin Ugedal <odin@uged.al>,
Odin Ugedal <odin@ugedal.com>, Paul Turner <pjt@google.com>,
rostedt@goodmis.org, Shanpei Chen <shanpeic@linux.alibaba.com>,
Tejun Heo <tj@kernel.org>,
Vincent Guittot <vincent.guittot@linaro.org>,
xiyou.wangcong@gmail.com
Subject: Re: [PATCH v4 1/4] sched/fair: Introduce primitives for CFS bandwidth burst
Date: Fri, 19 Mar 2021 20:51:59 +0800 [thread overview]
Message-ID: <CECB0471-7D8E-4FE0-9144-795553A3700B@linux.alibaba.com> (raw)
In-Reply-To: <YFNNWumXTSa3Bssl@lorien.usersys.redhat.com>
> On Mar 18, 2021, at 8:59 PM, Phil Auld <pauld@redhat.com> wrote:
>
> On Thu, Mar 18, 2021 at 09:26:58AM +0800 changhuaixin wrote:
>>
>>
>>> On Mar 17, 2021, at 4:06 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>>>
>>> On Wed, Mar 17, 2021 at 03:16:18PM +0800, changhuaixin wrote:
>>>
>>>>> Why do you allow such a large burst? I would expect something like:
>>>>>
>>>>> if (burst > quote)
>>>>> return -EINVAL;
>>>>>
>>>>> That limits the variance in the system. Allowing super long bursts seems
>>>>> to defeat the entire purpose of bandwidth control.
>>>>
>>>> I understand your concern. Surely large burst value might allow super
>>>> long bursts thus preventing bandwidth control entirely for a long
>>>> time.
>>>>
>>>> However, I am afraid it is hard to decide what the maximum burst
>>>> should be from the bandwidth control mechanism itself. Allowing some
>>>> burst to the maximum of quota is helpful, but not enough. There are
>>>> cases where workloads are bursty that they need many times more than
>>>> quota in a single period. In such cases, limiting burst to the maximum
>>>> of quota fails to meet the needs.
>>>>
>>>> Thus, I wonder whether is it acceptable to leave the maximum burst to
>>>> users. If the desired behavior is to allow some burst, configure burst
>>>> accordingly. If that is causing variance, use share or other fairness
>>>> mechanism. And if fairness mechanism still fails to coordinate, do not
>>>> use burst maybe.
>>>
>>> It's not fairness, bandwidth control is about isolation, and burst
>>> introduces interference.
>>>
>>>> In this way, cfs_b->buffer can be removed while cfs_b->max_overrun is
>>>> still needed maybe.
>>>
>>> So what is the typical avg,stdev,max and mode for the workloads where you find
>>> you need this?
>>>
>>> I would really like to put a limit on the burst. IMO a workload that has
>>> a burst many times longer than the quota is plain broken.
>>
>> I see. Then the problem comes down to how large the limit on burst shall be.
>>
>> I have sampled the CPU usage of a bursty container in 100ms periods. The statistics are:
>> average : 42.2%
>> stddev : 81.5%
>> max : 844.5%
>> P95 : 183.3%
>> P99 : 437.0%
>>
>> If quota is 100000ms, burst buffer needs to be 8 times more in order for this workload not to be throttled.
>> I can't say this is typical, but these workloads exist. On a machine running Kubernetes containers,
>> where there is often room for such burst and the interference is hard to notice, users would prefer
>> allowing such burst to being throttled occasionally.
>>
>
> I admit to not having followed all the history of this patch set. That said, when I see the above I just
> think your quota is too low for your workload.
>
Yeah, more quota is helpful for this workload. But that usually prevents us from improving the total CPU
usage by putting more work onto a single machine.
> The burst (mis?)feature seems to be a way to bypass the quota. And it sort of assumes cooperative
> containers that will only burst when they need it and then go back to normal.
>
>> In this sense, I suggest limit burst buffer to 16 times of quota or around. That should be enough for users to
>> improve tail latency caused by throttling. And users might choose a smaller one or even none, if the interference
>> is unacceptable. What do you think?
>>
>
> Having quotas that can regularly be exceeded by 16 times seems to make the concept of a quota
> meaningless. I'd have thought a burst would be some small percentage.
>
> What if several such containers burst at the same time? Can't that lead to overcommit that can effect
> other well-behaved containers?
>
I see. Maybe there should be some calculation on the probabilities of that, as Peter has replied.
>
> Cheers,
> Phil
>
> --
next prev parent reply other threads:[~2021-03-19 12:50 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-16 4:49 [PATCH v4 0/4] sched/fair: Burstable CFS bandwidth controller Huaixin Chang
2021-03-16 4:49 ` [PATCH v4 1/4] sched/fair: Introduce primitives for CFS bandwidth burst Huaixin Chang
2021-03-16 9:27 ` Peter Zijlstra
2021-03-16 9:41 ` Peter Zijlstra
2021-03-16 9:54 ` Peter Zijlstra
2021-03-17 7:16 ` changhuaixin
2021-03-17 8:06 ` Peter Zijlstra
2021-03-18 1:26 ` changhuaixin
2021-03-18 12:59 ` Phil Auld
2021-03-18 15:10 ` Peter Zijlstra
2021-04-19 8:18 ` changhuaixin
2021-03-19 12:51 ` changhuaixin [this message]
2021-03-18 15:05 ` Peter Zijlstra
2021-03-19 12:39 ` changhuaixin
2021-03-20 2:06 ` changhuaixin
2021-05-12 12:41 ` changhuaixin
2021-03-16 10:40 ` Peter Zijlstra
2021-03-16 4:49 ` [PATCH v4 2/4] sched/fair: Make CFS bandwidth controller burstable Huaixin Chang
2021-03-16 9:52 ` Peter Zijlstra
2021-03-16 4:49 ` [PATCH v4 3/4] sched/fair: Add cfs bandwidth burst statistics Huaixin Chang
2021-03-16 4:49 ` [PATCH v4 4/4] sched/fair: Add document for burstable CFS bandwidth control Huaixin Chang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CECB0471-7D8E-4FE0-9144-795553A3700B@linux.alibaba.com \
--to=changhuaixin@linux.alibaba.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=khlebnikov@yandex-team.ru \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=odin@uged.al \
--cc=odin@ugedal.com \
--cc=pauld@redhat.com \
--cc=peterz@infradead.org \
--cc=pjt@google.com \
--cc=rostedt@goodmis.org \
--cc=shanpeic@linux.alibaba.com \
--cc=tj@kernel.org \
--cc=vincent.guittot@linaro.org \
--cc=xiyou.wangcong@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).