From: Paul Turner <pjt@google.com>
To: linux-kernel@vger.kernel.org
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>,
Bharata B Rao <bharata@linux.vnet.ibm.com>,
Dhaval Giani <dhaval.giani@gmail.com>,
Balbir Singh <balbir@linux.vnet.ibm.com>,
Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>,
Srivatsa Vaddagiri <vatsa@in.ibm.com>,
Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>,
Hidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>,
Ingo Molnar <mingo@elte.hu>, Pavel Emelyanov <xemul@openvz.org>,
Hu Tao <hutao@cn.fujitsu.com>
Subject: [patch 17/17] sched: add documentation for bandwidth control
Date: Wed, 06 Jul 2011 22:30:53 -0700 [thread overview]
Message-ID: <20110707053101.311634860@google.com> (raw)
In-Reply-To: 20110707053036.173186930@google.com
[-- Attachment #1: sched-bwc-documentation.patch --]
[-- Type: text/plain, Size: 5489 bytes --]
From: Bharata B Rao <bharata@linux.vnet.ibm.com>
Basic description of usage and effect for CFS Bandwidth Control.
Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
Signed-off-by: Paul Turner <pjt@google.com>
---
Documentation/scheduler/sched-bwc.txt | 98
++++++++++++++++++++++++++++++++++
Documentation/scheduler/sched-bwc.txt | 122 ++++++++++++++++++++++++++++++++++
1 file changed, 122 insertions(+)
Index: tip/Documentation/scheduler/sched-bwc.txt
===================================================================
--- /dev/null
+++ tip/Documentation/scheduler/sched-bwc.txt
@@ -0,0 +1,122 @@
+CFS Bandwidth Control
+=====================
+
+[ This document only discusses CPU bandwidth control for SCHED_NORMAL.
+ The SCHED_RT case is covered in Documentation/scheduler/sched-rt-group.txt ]
+
+CFS bandwidth control is a CONFIG_FAIR_GROUP_SCHED extension which allows the
+specification of the maximum CPU bandwidth available to a group or hierarchy.
+
+The bandwidth allowed for a group is specified using a quota and period. Within
+each given "period" (microseconds), a group is allowed to consume only up to
+"quota" microseconds of CPU time. When the CPU bandwidth consumption of a
+group exceeds this limit (for that period), the tasks belonging to its
+hierarchy will be throttled and are not allowed to run again until the next
+period.
+
+A group's unused runtime is globally tracked, being refreshed with quota units
+above at each period boundary. As threads consume this bandwidth it is
+transferred to cpu-local "silos" on a demand basis. The amount transferred
+within each of these updates is tunable and described as the "slice".
+
+Management
+----------
+Quota and period are managed within the cpu subsystem via cgroupfs.
+
+cpu.cfs_quota_us: the total available run-time within a period (in microseconds)
+cpu.cfs_period_us: the length of a period (in microseconds)
+cpu.stat: exports throttling statistics [explained further below]
+
+The default values are:
+ cpu.cfs_period_us=100ms
+ cpu.cfs_quota=-1
+
+A value of -1 for cpu.cfs_quota_us indicates that the group does not have any
+bandwidth restriction in place, such a group is described as an unconstrained
+bandwidth group. This represents the traditional work-conserving behavior for
+CFS.
+
+Writing any (valid) positive value(s) will enact the specified bandwidth limit.
+The minimum quota allowed for the quota or period is 1ms. There is also an
+upper bound on the period length of 1s. Additional restrictions exist when
+bandwidth limits are used in a hierarchical fashion, these are explained in
+more detail below.
+
+Writing any negative value to cpu.cfs_quota_us will remove the bandwidth limit
+and return the group to an unconstrained state once more.
+
+Any updates to a group's bandwidth specification will result in it becoming
+unthrottled if it is in a constrained state.
+
+System wide settings
+--------------------
+For efficiency run-time is transferred between the global pool and CPU local
+"silos" in a batch fashion. This greatly reduces global accounting pressure
+on large systems. The amount transferred each time such an update is required
+is described as the "slice".
+
+This is tunable via procfs:
+ /proc/sys/kernel/sched_cfs_bandwidth_slice_us (default=5ms)
+
+Larger slice values will reduce transfer overheads, while smaller values allow
+for more fine-grained consumption.
+
+Statistics
+----------
+A group's bandwidth statistics are exported via 3 fields in cpu.stat.
+
+cpu.stat:
+- nr_periods: Number of enforcement intervals that have elapsed.
+- nr_throttled: Number of times the group has been throttled/limited.
+- throttled_time: The total time duration (in nanoseconds) for which entities
+ of the group have been throttled.
+
+This interface is read-only.
+
+Hierarchical considerations
+---------------------------
+The interface enforces that an individual entity's bandwidth is always
+attainable, that is: max(c_i) <= C. However, over-subscription in the
+aggregate case is explicitly allowed to enable work-conserving semantics
+within a hierarchy.
+ e.g. \Sum (c_i) may exceed C
+[ Where C is the parent's bandwidth, and c_i its children ]
+
+
+There are two ways in which a group may become throttled:
+ a. it fully consumes its own quota within a period
+ b. a parent's quota is fully consumed within its period
+
+In case b) above, even though the child may have runtime remaining it will not
+be allowed to until the parent's runtime is refreshed.
+
+Examples
+--------
+1. Limit a group to 1 CPU worth of runtime.
+
+ If period is 250ms and quota is also 250ms, the group will get
+ 1 CPU worth of runtime every 250ms.
+
+ # echo 250000 > cpu.cfs_quota_us /* quota = 250ms */
+ # echo 250000 > cpu.cfs_period_us /* period = 250ms */
+
+2. Limit a group to 2 CPUs worth of runtime on a multi-CPU machine.
+
+ With 500ms period and 1000ms quota, the group can get 2 CPUs worth of
+ runtime every 500ms.
+
+ # echo 1000000 > cpu.cfs_quota_us /* quota = 1000ms */
+ # echo 500000 > cpu.cfs_period_us /* period = 500ms */
+
+ The larger period here allows for increased burst capacity.
+
+3. Limit a group to 20% of 1 CPU.
+
+ With 50ms period, 10ms quota will be equivalent to 20% of 1 CPU.
+
+ # echo 10000 > cpu.cfs_quota_us /* quota = 10ms */
+ # echo 50000 > cpu.cfs_period_us /* period = 50ms */
+
+ By using a small period here we are ensuring a consistent latency
+ response at the expense of burst capacity.
+
next prev parent reply other threads:[~2011-07-07 5:36 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-07-07 5:30 [patch 00/17] CFS Bandwidth Control v7.1 Paul Turner
2011-07-07 5:30 ` [patch 01/17] sched: (fixlet) dont update shares twice on on_rq parent Paul Turner
2011-07-21 18:28 ` [tip:sched/core] sched: Don't " tip-bot for Paul Turner
2011-07-07 5:30 ` [patch 02/17] sched: hierarchical task accounting for SCHED_OTHER Paul Turner
2011-07-07 5:30 ` [patch 03/17] sched: introduce primitives to account for CFS bandwidth tracking Paul Turner
2011-07-07 13:48 ` Peter Zijlstra
2011-07-07 21:30 ` Paul Turner
2011-07-07 5:30 ` [patch 04/17] sched: validate CFS quota hierarchies Paul Turner
2011-07-07 5:30 ` [patch 05/17] sched: accumulate per-cfs_rq cpu usage and charge against bandwidth Paul Turner
2011-07-07 5:30 ` [patch 06/17] sched: add a timer to handle CFS bandwidth refresh Paul Turner
2011-07-07 5:30 ` [patch 07/17] sched: expire invalid runtime Paul Turner
2011-07-07 5:30 ` [patch 08/17] sched: add support for throttling group entities Paul Turner
2011-07-07 5:30 ` [patch 09/17] sched: add support for unthrottling " Paul Turner
2011-07-07 5:30 ` [patch 10/17] sched: allow for positional tg_tree walks Paul Turner
2011-07-07 5:30 ` [patch 11/17] sched: prevent interactions with throttled entities Paul Turner
2011-07-07 5:30 ` [patch 12/17] sched: prevent buddy " Paul Turner
2011-07-07 5:30 ` [patch 13/17] sched: migrate throttled tasks on HOTPLUG Paul Turner
2011-07-07 5:30 ` [patch 14/17] sched: throttle entities exceeding their allowed bandwidth Paul Turner
2011-07-07 5:30 ` [patch 15/17] sched: add exports tracking cfs bandwidth control statistics Paul Turner
2011-07-07 5:30 ` [patch 16/17] sched: return unused runtime on group dequeue Paul Turner
2011-07-07 5:30 ` Paul Turner [this message]
2011-07-07 11:13 ` [patch 00/17] CFS Bandwidth Control v7.1 Peter Zijlstra
2011-07-11 1:22 ` Hu Tao
2011-07-07 11:23 ` Ingo Molnar
2011-07-07 11:28 ` Peter Zijlstra
2011-07-07 14:38 ` Peter Zijlstra
2011-07-07 14:51 ` Ingo Molnar
2011-07-07 14:54 ` Peter Zijlstra
2011-07-07 14:56 ` Ingo Molnar
2011-07-07 16:23 ` Jason Baron
2011-07-07 17:20 ` Peter Zijlstra
2011-07-07 18:15 ` Jason Baron
2011-07-07 20:36 ` jump_label defaults (was Re: [patch 00/17] CFS Bandwidth Control v7.1) Peter Zijlstra
2011-07-08 9:20 ` Peter Zijlstra
2011-07-08 15:47 ` Jason Baron
2011-07-07 16:52 ` [patch 00/17] CFS Bandwidth Control v7.1 Andi Kleen
2011-07-07 17:08 ` Peter Zijlstra
2011-07-07 17:59 ` Peter Zijlstra
2011-07-07 19:36 ` Jason Baron
2011-07-08 7:45 ` Paul Turner
2011-07-08 7:39 ` Paul Turner
2011-07-08 10:32 ` Peter Zijlstra
2011-07-09 7:34 ` Paul Turner
2011-07-10 18:12 ` Ingo Molnar
2011-07-07 14:06 ` Peter Zijlstra
2011-07-08 7:35 ` Paul Turner
2011-07-11 1:22 ` Hu Tao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110707053101.311634860@google.com \
--to=pjt@google.com \
--cc=a.p.zijlstra@chello.nl \
--cc=balbir@linux.vnet.ibm.com \
--cc=bharata@linux.vnet.ibm.com \
--cc=dhaval.giani@gmail.com \
--cc=hutao@cn.fujitsu.com \
--cc=kamalesh@linux.vnet.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=seto.hidetoshi@jp.fujitsu.com \
--cc=svaidy@linux.vnet.ibm.com \
--cc=vatsa@in.ibm.com \
--cc=xemul@openvz.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).