* [PATCHSET] blkcg: unify blkgs for different policies
@ 2012-02-01 21:19 Tejun Heo
2012-02-01 21:19 ` [PATCH 01/11] blkcg: let blkio_group point to blkio_cgroup directly Tejun Heo
` (11 more replies)
0 siblings, 12 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel
Hey, again.
Currently, blkcg policies have and manage their own blkgs, so blkgs
are per cgroup-queue-policy combination instead of cgroup-queue
combination. This leads to nasty problems. It isn't clear which part
are common to both policies. There are unused duplicates in common
part of blkg and it isn't trivial to tell which part is being used.
The separation also leads to duplicate logic in both policies which
makes the code difficult to follow, prone to subtle bugs and, most
importantly, hinders proper layering between blkcg core and policy
implementations.
Because locking, blkg management, elvswitch and policy
[de]registration are tightly woven, it is challenging to untangle -
doing proper in-place policy data replacement requires locking
improvements which in turn is painful to do when policy
implementations are doing their own things with blkgs.
As a transitional step, all blkgs other than root one are shot down on
policy [de]registration and root blkg is updated in place. This is
hackish but should get us through locking update after which we can
implement in-place update for all blkgs safely. While this does
introduce race window while policies are being [de]registered, this
isn't anything new (e.g. none of stat update functions synchronize
against policy update) and shouldn't cause any actual problem given
blk-throttle can't be built as module and cfq-iosched is default
iosched on most installations.
This patchset was pretty painful but I think/hope things will be
eaiser from here on. Note that this patchset does add ~180 LOC. Some
of them are comments and it's expected to shrink again with further
cleanups and removal of transitional stuff.
Changes to come are:
* locking simplification
* proper in-place update of policy data for all blkgs on policy
change
* fix broken blkcg switch after throttling.
* use unified stats updated under queue lock and drop percpu stats
which should fix locking / context bug across percpu allocation.
* make set of applied policies per-queue
* move stats and conf into their owning policies and let blkcg core
provide generic framework / helper instead of hard coding all the
possible ones. This should be accompanied by cgroup updates to
allow changing files in cgroupfs. Not sure how this will turn out
yet.
This patchset contains the following 11 patches.
0001-blkcg-let-blkio_group-point-to-blkio_cgroup-directly.patch
0002-block-relocate-elevator-initialized-test-from-blk_cl.patch
0003-blkcg-add-blkcg_-init-drain-exit-_queue.patch
0004-blkcg-clear-all-request_queues-on-blkcg-policy-un-re.patch
0005-blkcg-let-blkcg-core-handle-policy-private-data-allo.patch
0006-blkcg-move-refcnt-to-blkcg-core.patch
0007-blkcg-make-blkg-pd-an-array-and-move-configuration-a.patch
0008-blkcg-don-t-use-blkg-plid-in-stat-related-functions.patch
0009-blkcg-move-per-queue-blkg-list-heads-and-counters-to.patch
0010-blkcg-let-blkcg-core-manage-per-queue-blkg-list-and-.patch
0011-blkcg-unify-blkg-s-for-blkcg-policies.patch
0001-0003 are prep patches.
0004 shoots down all non-root blkgs on policy [de]registration.
0005 separates per-policy data from common part of blkg and allocate
them separately from blkcg core.
0006-0010 collect common data fields and logic from policy
implementations into blkcg core.
0011 unifies blkgs so that there's one blkg per cgroup-queue pair.
This patchset is on top of
v3.3-rc2 62aa2b537c6f5957afd98e29f96897419ed5ebab
+ [1] blkcg: kill policy node and blkg->dev, take#4
and is also available in the following git branch.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git blkcg-unified-blkg
diffstat follows.
block/blk-cgroup.c | 674 +++++++++++++++++++++++++++++++++++--------------
block/blk-cgroup.h | 211 +++++++++++----
block/blk-core.c | 26 +
block/blk-sysfs.c | 6
block/blk-throttle.c | 232 +++-------------
block/blk.h | 2
block/cfq-iosched.c | 274 +++++--------------
block/cfq.h | 96 ++++--
block/elevator.c | 2
include/linux/blkdev.h | 7
10 files changed, 853 insertions(+), 677 deletions(-)
Thanks.
--
tejun
[1] http://thread.gmane.org/gmane.linux.kernel/1247152
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH 01/11] blkcg: let blkio_group point to blkio_cgroup directly
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-02 20:03 ` Vivek Goyal
2012-02-01 21:19 ` [PATCH 02/11] block: relocate elevator initialized test from blk_cleanup_queue() to blk_drain_queue() Tejun Heo
` (10 subsequent siblings)
11 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
Currently, blkg points to the associated blkcg via its css_id. This
unnecessarily complicates dereferencing blkcg. Let blkg hold a
reference to the associated blkcg and point directly to it and disable
css_id on blkio_subsys.
This change requires splitting blkiocg_destroy() into
blkiocg_pre_destroy() and blkiocg_destroy() so that all blkg's can be
destroyed and all the blkcg references held by them dropped during
cgroup removal.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-cgroup.c | 43 ++++++++++++++++++++++++-------------------
block/blk-cgroup.h | 2 +-
block/blk-throttle.c | 3 +++
block/cfq-iosched.c | 4 ++++
4 files changed, 32 insertions(+), 20 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 10eedb3..b4380a0 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -37,6 +37,7 @@ static int blkiocg_can_attach(struct cgroup_subsys *, struct cgroup *,
struct cgroup_taskset *);
static void blkiocg_attach(struct cgroup_subsys *, struct cgroup *,
struct cgroup_taskset *);
+static int blkiocg_pre_destroy(struct cgroup_subsys *, struct cgroup *);
static void blkiocg_destroy(struct cgroup_subsys *, struct cgroup *);
static int blkiocg_populate(struct cgroup_subsys *, struct cgroup *);
@@ -51,10 +52,10 @@ struct cgroup_subsys blkio_subsys = {
.create = blkiocg_create,
.can_attach = blkiocg_can_attach,
.attach = blkiocg_attach,
+ .pre_destroy = blkiocg_pre_destroy,
.destroy = blkiocg_destroy,
.populate = blkiocg_populate,
.subsys_id = blkio_subsys_id,
- .use_id = 1,
.module = THIS_MODULE,
};
EXPORT_SYMBOL_GPL(blkio_subsys);
@@ -442,6 +443,7 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
if (blkg)
return blkg;
+ /* blkg holds a reference to blkcg */
if (!css_tryget(&blkcg->css))
return ERR_PTR(-EINVAL);
@@ -463,15 +465,16 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
spin_lock_init(&new_blkg->stats_lock);
rcu_assign_pointer(new_blkg->q, q);
- new_blkg->blkcg_id = css_id(&blkcg->css);
+ new_blkg->blkcg = blkcg;
new_blkg->plid = plid;
cgroup_path(blkcg->css.cgroup, new_blkg->path,
sizeof(new_blkg->path));
+ } else {
+ css_put(&blkcg->css);
}
rcu_read_lock();
spin_lock_irq(q->queue_lock);
- css_put(&blkcg->css);
/* did bypass get turned on inbetween? */
if (unlikely(blk_queue_bypass(q)) && !for_root) {
@@ -500,6 +503,7 @@ out:
if (new_blkg) {
free_percpu(new_blkg->stats_cpu);
kfree(new_blkg);
+ css_put(&blkcg->css);
}
return blkg;
}
@@ -508,7 +512,6 @@ EXPORT_SYMBOL_GPL(blkg_lookup_create);
static void __blkiocg_del_blkio_group(struct blkio_group *blkg)
{
hlist_del_init_rcu(&blkg->blkcg_node);
- blkg->blkcg_id = 0;
}
/*
@@ -517,24 +520,17 @@ static void __blkiocg_del_blkio_group(struct blkio_group *blkg)
*/
int blkiocg_del_blkio_group(struct blkio_group *blkg)
{
- struct blkio_cgroup *blkcg;
+ struct blkio_cgroup *blkcg = blkg->blkcg;
unsigned long flags;
- struct cgroup_subsys_state *css;
int ret = 1;
- rcu_read_lock();
- css = css_lookup(&blkio_subsys, blkg->blkcg_id);
- if (css) {
- blkcg = container_of(css, struct blkio_cgroup, css);
- spin_lock_irqsave(&blkcg->lock, flags);
- if (!hlist_unhashed(&blkg->blkcg_node)) {
- __blkiocg_del_blkio_group(blkg);
- ret = 0;
- }
- spin_unlock_irqrestore(&blkcg->lock, flags);
+ spin_lock_irqsave(&blkcg->lock, flags);
+ if (!hlist_unhashed(&blkg->blkcg_node)) {
+ __blkiocg_del_blkio_group(blkg);
+ ret = 0;
}
+ spin_unlock_irqrestore(&blkcg->lock, flags);
- rcu_read_unlock();
return ret;
}
EXPORT_SYMBOL_GPL(blkiocg_del_blkio_group);
@@ -1376,7 +1372,8 @@ static int blkiocg_populate(struct cgroup_subsys *subsys, struct cgroup *cgroup)
ARRAY_SIZE(blkio_files));
}
-static void blkiocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)
+static int blkiocg_pre_destroy(struct cgroup_subsys *subsys,
+ struct cgroup *cgroup)
{
struct blkio_cgroup *blkcg = cgroup_to_blkio_cgroup(cgroup);
unsigned long flags;
@@ -1385,6 +1382,7 @@ static void blkiocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)
struct blkio_policy_type *blkiop;
rcu_read_lock();
+
do {
spin_lock_irqsave(&blkcg->lock, flags);
@@ -1414,8 +1412,15 @@ static void blkiocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)
spin_unlock(&blkio_list_lock);
} while (1);
- free_css_id(&blkio_subsys, &blkcg->css);
rcu_read_unlock();
+
+ return 0;
+}
+
+static void blkiocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)
+{
+ struct blkio_cgroup *blkcg = cgroup_to_blkio_cgroup(cgroup);
+
if (blkcg != &blkio_root_cgroup)
kfree(blkcg);
}
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 7ebecf6..ca1fc63 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -163,7 +163,7 @@ struct blkio_group {
/* Pointer to the associated request_queue, RCU protected */
struct request_queue __rcu *q;
struct hlist_node blkcg_node;
- unsigned short blkcg_id;
+ struct blkio_cgroup *blkcg;
/* Store cgroup path */
char path[128];
/* policy which owns this blk group */
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 52a4293..fe6a442 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -169,6 +169,9 @@ static void throtl_put_tg(struct throtl_grp *tg)
if (!atomic_dec_and_test(&tg->ref))
return;
+ /* release the extra blkcg reference this blkg has been holding */
+ css_put(&tg->blkg.blkcg->css);
+
/*
* A group is freed in rcu manner. But having an rcu lock does not
* mean that one can access all the fields of blkg and assume these
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 09d61ad..c2e1645 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -1133,6 +1133,10 @@ static void cfq_put_cfqg(struct cfq_group *cfqg)
cfqg->ref--;
if (cfqg->ref)
return;
+
+ /* release the extra blkcg reference this blkg has been holding */
+ css_put(&cfqg->blkg.blkcg->css);
+
for_each_cfqg_st(cfqg, i, j, st)
BUG_ON(!RB_EMPTY_ROOT(&st->rb));
free_percpu(cfqg->blkg.stats_cpu);
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH 02/11] block: relocate elevator initialized test from blk_cleanup_queue() to blk_drain_queue()
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
2012-02-01 21:19 ` [PATCH 01/11] blkcg: let blkio_group point to blkio_cgroup directly Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-02 20:20 ` Vivek Goyal
2012-02-01 21:19 ` [PATCH 03/11] blkcg: add blkcg_{init|drain|exit}_queue() Tejun Heo
` (9 subsequent siblings)
11 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
blk_cleanup_queue() may be called for a queue which doesn't have
elevator initialized to skip invoking blk_drain_queue().
blk_drain_queue() will be used for other purposes too and may be
called for such half-initialized queues from other paths. Move
q->elevator test into blk_drain_queue().
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-core.c | 16 +++++++++-------
1 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/block/blk-core.c b/block/blk-core.c
index 1b73d06..ddda1cc 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -359,6 +359,13 @@ EXPORT_SYMBOL(blk_put_queue);
*/
void blk_drain_queue(struct request_queue *q, bool drain_all)
{
+ /*
+ * The caller might be trying to tear down @q before its elevator
+ * is initialized, in which case we don't want to call into it.
+ */
+ if (!q->elevator)
+ return;
+
while (true) {
bool drain = false;
int i;
@@ -468,13 +475,8 @@ void blk_cleanup_queue(struct request_queue *q)
spin_unlock_irq(lock);
mutex_unlock(&q->sysfs_lock);
- /*
- * Drain all requests queued before DEAD marking. The caller might
- * be trying to tear down @q before its elevator is initialized, in
- * which case we don't want to call into draining.
- */
- if (q->elevator)
- blk_drain_queue(q, true);
+ /* drain all requests queued before DEAD marking */
+ blk_drain_queue(q, true);
/* @q won't process any more request, flush async actions */
del_timer_sync(&q->backing_dev_info.laptop_mode_wb_timer);
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH 03/11] blkcg: add blkcg_{init|drain|exit}_queue()
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
2012-02-01 21:19 ` [PATCH 01/11] blkcg: let blkio_group point to blkio_cgroup directly Tejun Heo
2012-02-01 21:19 ` [PATCH 02/11] block: relocate elevator initialized test from blk_cleanup_queue() to blk_drain_queue() Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-01 21:19 ` [PATCH 04/11] blkcg: clear all request_queues on blkcg policy [un]registrations Tejun Heo
` (8 subsequent siblings)
11 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
Currently block core calls directly into blk-throttle for init, drain
and exit. This patch adds blkcg_{init|drain|exit}_queue() which wraps
the blk-throttle functions. This is to give more control and
visiblity to blkcg core layer for proper layering. Further patches
will add logic common to blkcg policies to the functions.
While at it, collapse blk_throtl_release() into blk_throtl_exit().
There's no reason to keep them separate.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-cgroup.c | 42 ++++++++++++++++++++++++++++++++++++++++++
block/blk-cgroup.h | 7 +++++++
block/blk-core.c | 7 ++++---
block/blk-sysfs.c | 4 ++--
block/blk-throttle.c | 3 ---
block/blk.h | 2 --
6 files changed, 55 insertions(+), 10 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index b4380a0..15264d0 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -20,6 +20,7 @@
#include <linux/genhd.h>
#include <linux/delay.h>
#include "blk-cgroup.h"
+#include "blk.h"
#define MAX_KEY_LEN 100
@@ -1448,6 +1449,47 @@ done:
return &blkcg->css;
}
+/**
+ * blkcg_init_queue - initialize blkcg part of request queue
+ * @q: request_queue to initialize
+ *
+ * Called from blk_alloc_queue_node(). Responsible for initializing blkcg
+ * part of new request_queue @q.
+ *
+ * RETURNS:
+ * 0 on success, -errno on failure.
+ */
+int blkcg_init_queue(struct request_queue *q)
+{
+ might_sleep();
+
+ return blk_throtl_init(q);
+}
+
+/**
+ * blkcg_drain_queue - drain blkcg part of request_queue
+ * @q: request_queue to drain
+ *
+ * Called from blk_drain_queue(). Responsible for draining blkcg part.
+ */
+void blkcg_drain_queue(struct request_queue *q)
+{
+ lockdep_assert_held(q->queue_lock);
+
+ blk_throtl_drain(q);
+}
+
+/**
+ * blkcg_exit_queue - exit and release blkcg part of request_queue
+ * @q: request_queue being released
+ *
+ * Called from blk_release_queue(). Responsible for exiting blkcg part.
+ */
+void blkcg_exit_queue(struct request_queue *q)
+{
+ blk_throtl_exit(q);
+}
+
/*
* We cannot support shared io contexts, as we have no mean to support
* two tasks with the same ioc in two different groups without major rework
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index ca1fc63..3bc1710 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -215,6 +215,10 @@ struct blkio_policy_type {
enum blkio_policy_id plid;
};
+extern int blkcg_init_queue(struct request_queue *q);
+extern void blkcg_drain_queue(struct request_queue *q);
+extern void blkcg_exit_queue(struct request_queue *q);
+
/* Blkio controller policy registration */
extern void blkio_policy_register(struct blkio_policy_type *);
extern void blkio_policy_unregister(struct blkio_policy_type *);
@@ -233,6 +237,9 @@ struct blkio_group {
struct blkio_policy_type {
};
+static inline int blkcg_init_queue(struct request_queue *q) { return 0; }
+static inline void blkcg_drain_queue(struct request_queue *q) { }
+static inline void blkcg_exit_queue(struct request_queue *q) { }
static inline void blkio_policy_register(struct blkio_policy_type *blkiop) { }
static inline void blkio_policy_unregister(struct blkio_policy_type *blkiop) { }
static inline void blkg_destroy_all(struct request_queue *q) { }
diff --git a/block/blk-core.c b/block/blk-core.c
index ddda1cc..ab0a11b 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -34,6 +34,7 @@
#include <trace/events/block.h>
#include "blk.h"
+#include "blk-cgroup.h"
EXPORT_TRACEPOINT_SYMBOL_GPL(block_bio_remap);
EXPORT_TRACEPOINT_SYMBOL_GPL(block_rq_remap);
@@ -280,7 +281,7 @@ EXPORT_SYMBOL(blk_stop_queue);
*
* This function does not cancel any asynchronous activity arising
* out of elevator or throttling code. That would require elevaotor_exit()
- * and blk_throtl_exit() to be called with queue lock initialized.
+ * and blkcg_exit_queue() to be called with queue lock initialized.
*
*/
void blk_sync_queue(struct request_queue *q)
@@ -373,7 +374,7 @@ void blk_drain_queue(struct request_queue *q, bool drain_all)
spin_lock_irq(q->queue_lock);
elv_drain_elevator(q);
- blk_throtl_drain(q);
+ blkcg_drain_queue(q);
/*
* This function might be called on a queue which failed
@@ -561,7 +562,7 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id)
*/
q->queue_lock = &q->__queue_lock;
- if (blk_throtl_init(q))
+ if (blkcg_init_queue(q))
goto fail_id;
return q;
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index cf15001..00cdc98 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -9,6 +9,7 @@
#include <linux/blktrace_api.h>
#include "blk.h"
+#include "blk-cgroup.h"
struct queue_sysfs_entry {
struct attribute attr;
@@ -486,7 +487,7 @@ static void blk_release_queue(struct kobject *kobj)
elevator_exit(q->elevator);
}
- blk_throtl_exit(q);
+ blkcg_exit_queue(q);
if (rl->rq_pool)
mempool_destroy(rl->rq_pool);
@@ -494,7 +495,6 @@ static void blk_release_queue(struct kobject *kobj)
if (q->queue_tags)
__blk_queue_free_tags(q);
- blk_throtl_release(q);
blk_trace_shutdown(q);
bdi_destroy(&q->backing_dev_info);
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index fe6a442..ac6d0fe 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1226,10 +1226,7 @@ void blk_throtl_exit(struct request_queue *q)
* it.
*/
throtl_shutdown_wq(q);
-}
-void blk_throtl_release(struct request_queue *q)
-{
kfree(q->td);
}
diff --git a/block/blk.h b/block/blk.h
index 33897f6..bf04a6f 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -234,7 +234,6 @@ extern bool blk_throtl_bio(struct request_queue *q, struct bio *bio);
extern void blk_throtl_drain(struct request_queue *q);
extern int blk_throtl_init(struct request_queue *q);
extern void blk_throtl_exit(struct request_queue *q);
-extern void blk_throtl_release(struct request_queue *q);
#else /* CONFIG_BLK_DEV_THROTTLING */
static inline bool blk_throtl_bio(struct request_queue *q, struct bio *bio)
{
@@ -243,7 +242,6 @@ static inline bool blk_throtl_bio(struct request_queue *q, struct bio *bio)
static inline void blk_throtl_drain(struct request_queue *q) { }
static inline int blk_throtl_init(struct request_queue *q) { return 0; }
static inline void blk_throtl_exit(struct request_queue *q) { }
-static inline void blk_throtl_release(struct request_queue *q) { }
#endif /* CONFIG_BLK_DEV_THROTTLING */
#endif /* BLK_INTERNAL_H */
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH 04/11] blkcg: clear all request_queues on blkcg policy [un]registrations
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
` (2 preceding siblings ...)
2012-02-01 21:19 ` [PATCH 03/11] blkcg: add blkcg_{init|drain|exit}_queue() Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-01 21:19 ` [PATCH 05/11] blkcg: let blkcg core handle policy private data allocation Tejun Heo
` (7 subsequent siblings)
11 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
Keep track of all request_queues which have blkcg initialized and turn
on bypass and invoke blkcg_clear_queue() on all before making changes
to blkcg policies.
This is to prepare for moving blkg management into blkcg core. Note
that this uses more brute force than necessary. Finer grained shoot
down will be implemented later and given that policy [un]registration
almost never happens on running systems (blk-throtl can't be built as
a module and cfq usually is the builtin default iosched), this
shouldn't be a problem for the time being.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-cgroup.c | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
include/linux/blkdev.h | 3 +++
2 files changed, 50 insertions(+), 1 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 15264d0..10878a4 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -27,6 +27,9 @@
static DEFINE_SPINLOCK(blkio_list_lock);
static LIST_HEAD(blkio_list);
+static DEFINE_MUTEX(all_q_mutex);
+static LIST_HEAD(all_q_list);
+
struct blkio_cgroup blkio_root_cgroup = { .weight = 2*BLKIO_WEIGHT_DEFAULT };
EXPORT_SYMBOL_GPL(blkio_root_cgroup);
@@ -1461,9 +1464,20 @@ done:
*/
int blkcg_init_queue(struct request_queue *q)
{
+ int ret;
+
might_sleep();
- return blk_throtl_init(q);
+ ret = blk_throtl_init(q);
+ if (ret)
+ return ret;
+
+ mutex_lock(&all_q_mutex);
+ INIT_LIST_HEAD(&q->all_q_node);
+ list_add_tail(&q->all_q_node, &all_q_list);
+ mutex_unlock(&all_q_mutex);
+
+ return 0;
}
/**
@@ -1487,6 +1501,10 @@ void blkcg_drain_queue(struct request_queue *q)
*/
void blkcg_exit_queue(struct request_queue *q)
{
+ mutex_lock(&all_q_mutex);
+ list_del_init(&q->all_q_node);
+ mutex_unlock(&all_q_mutex);
+
blk_throtl_exit(q);
}
@@ -1532,8 +1550,33 @@ static void blkiocg_attach(struct cgroup_subsys *ss, struct cgroup *cgrp,
}
}
+static void blkcg_bypass_start(void)
+ __acquires(&all_q_mutex)
+{
+ struct request_queue *q;
+
+ mutex_lock(&all_q_mutex);
+
+ list_for_each_entry(q, &all_q_list, all_q_node) {
+ blk_queue_bypass_start(q);
+ blkg_destroy_all(q);
+ }
+}
+
+static void blkcg_bypass_end(void)
+ __releases(&all_q_mutex)
+{
+ struct request_queue *q;
+
+ list_for_each_entry(q, &all_q_list, all_q_node)
+ blk_queue_bypass_end(q);
+
+ mutex_unlock(&all_q_mutex);
+}
+
void blkio_policy_register(struct blkio_policy_type *blkiop)
{
+ blkcg_bypass_start();
spin_lock(&blkio_list_lock);
BUG_ON(blkio_policy[blkiop->plid]);
@@ -1541,11 +1584,13 @@ void blkio_policy_register(struct blkio_policy_type *blkiop)
list_add_tail(&blkiop->list, &blkio_list);
spin_unlock(&blkio_list_lock);
+ blkcg_bypass_end();
}
EXPORT_SYMBOL_GPL(blkio_policy_register);
void blkio_policy_unregister(struct blkio_policy_type *blkiop)
{
+ blkcg_bypass_start();
spin_lock(&blkio_list_lock);
BUG_ON(blkio_policy[blkiop->plid] != blkiop);
@@ -1553,5 +1598,6 @@ void blkio_policy_unregister(struct blkio_policy_type *blkiop)
list_del_init(&blkiop->list);
spin_unlock(&blkio_list_lock);
+ blkcg_bypass_end();
}
EXPORT_SYMBOL_GPL(blkio_policy_unregister);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index f10958b..e38e4d0 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -397,6 +397,9 @@ struct request_queue {
struct bsg_class_device bsg_dev;
#endif
+#ifdef CONFIG_BLK_CGROUP
+ struct list_head all_q_node;
+#endif
#ifdef CONFIG_BLK_DEV_THROTTLING
/* Throttle data */
struct throtl_data *td;
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH 05/11] blkcg: let blkcg core handle policy private data allocation
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
` (3 preceding siblings ...)
2012-02-01 21:19 ` [PATCH 04/11] blkcg: clear all request_queues on blkcg policy [un]registrations Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-01 21:19 ` [PATCH 06/11] blkcg: move refcnt to blkcg core Tejun Heo
` (6 subsequent siblings)
11 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
Currently, blkg's are embedded in private data blkcg policy private
data structure and thus allocated and freed by policies. This leads
to duplicate codes in policies, hinders implementing common part in
blkcg core with strong semantics, and forces duplicate blkg's for the
same cgroup-q association.
This patch introduces struct blkg_policy_data which is a separate data
structure chained from blkg. Policies specifies the amount of private
data it needs in its blkio_policy_type->pdata_size and blkcg core
takes care of allocating them along with blkg which can be accessed
using blkg_to_pdata(). blkg can be determined from pdata using
pdata_to_blkg(). blkio_alloc_group_fn() method is accordingly updated
to blkio_init_group_fn().
For consistency, tg_of_blkg() and cfqg_of_blkg() are replaced with
blkg_to_tg() and blkg_to_cfqg() respectively, and functions to map in
the reverse direction are added.
Except that policy specific data now lives in a separate data
structure from blkg, this patch doesn't introduce any functional
difference.
This will be used to unify blkg's for different policies.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-cgroup.c | 86 +++++++++++++++++++++++++++++++++---------
block/blk-cgroup.h | 53 ++++++++++++++++++++++++-
block/blk-throttle.c | 79 +++++++++++++++++++-------------------
block/cfq-iosched.c | 102 +++++++++++++++++++++++++------------------------
4 files changed, 209 insertions(+), 111 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 10878a4..e42da84 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -422,6 +422,70 @@ void blkiocg_update_io_merged_stats(struct blkio_group *blkg, bool direction,
}
EXPORT_SYMBOL_GPL(blkiocg_update_io_merged_stats);
+/**
+ * blkg_free - free a blkg
+ * @blkg: blkg to free
+ *
+ * Free @blkg which may be partially allocated.
+ */
+static void blkg_free(struct blkio_group *blkg)
+{
+ if (blkg) {
+ free_percpu(blkg->stats_cpu);
+ kfree(blkg->pd);
+ kfree(blkg);
+ }
+}
+
+/**
+ * blkg_alloc - allocate a blkg
+ * @blkcg: block cgroup the new blkg is associated with
+ * @q: request_queue the new blkg is associated with
+ * @pol: policy the new blkg is associated with
+ *
+ * Allocate a new blkg assocating @blkcg and @q for @pol.
+ *
+ * FIXME: Should be called with queue locked but currently isn't due to
+ * percpu stat breakage.
+ */
+static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
+ struct request_queue *q,
+ struct blkio_policy_type *pol)
+{
+ struct blkio_group *blkg;
+
+ /* alloc and init base part */
+ blkg = kzalloc_node(sizeof(*blkg), GFP_ATOMIC, q->node);
+ if (!blkg)
+ return NULL;
+
+ spin_lock_init(&blkg->stats_lock);
+ rcu_assign_pointer(blkg->q, q);
+ blkg->blkcg = blkcg;
+ blkg->plid = pol->plid;
+ cgroup_path(blkcg->css.cgroup, blkg->path, sizeof(blkg->path));
+
+ /* alloc per-policy data */
+ blkg->pd = kzalloc_node(sizeof(*blkg->pd) + pol->pdata_size, GFP_ATOMIC,
+ q->node);
+ if (!blkg->pd) {
+ blkg_free(blkg);
+ return NULL;
+ }
+
+ /* broken, read comment in the callsite */
+ blkg->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
+ if (!blkg->stats_cpu) {
+ blkg_free(blkg);
+ return NULL;
+ }
+
+ /* attach pd to blkg and invoke per-policy init */
+ blkg->pd->blkg = blkg;
+ pol->ops.blkio_init_group_fn(blkg);
+ return blkg;
+}
+
struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
struct request_queue *q,
enum blkio_policy_id plid,
@@ -463,19 +527,7 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
spin_unlock_irq(q->queue_lock);
rcu_read_unlock();
- new_blkg = pol->ops.blkio_alloc_group_fn(q, blkcg);
- if (new_blkg) {
- new_blkg->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
-
- spin_lock_init(&new_blkg->stats_lock);
- rcu_assign_pointer(new_blkg->q, q);
- new_blkg->blkcg = blkcg;
- new_blkg->plid = plid;
- cgroup_path(blkcg->css.cgroup, new_blkg->path,
- sizeof(new_blkg->path));
- } else {
- css_put(&blkcg->css);
- }
+ new_blkg = blkg_alloc(blkcg, q, pol);
rcu_read_lock();
spin_lock_irq(q->queue_lock);
@@ -492,7 +544,7 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
goto out;
/* did alloc fail? */
- if (unlikely(!new_blkg || !new_blkg->stats_cpu)) {
+ if (unlikely(!new_blkg)) {
blkg = ERR_PTR(-ENOMEM);
goto out;
}
@@ -504,11 +556,7 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
pol->ops.blkio_link_group_fn(q, blkg);
spin_unlock(&blkcg->lock);
out:
- if (new_blkg) {
- free_percpu(new_blkg->stats_cpu);
- kfree(new_blkg);
- css_put(&blkcg->css);
- }
+ blkg_free(new_blkg);
return blkg;
}
EXPORT_SYMBOL_GPL(blkg_lookup_create);
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 3bc1710..9537819 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -159,6 +159,15 @@ struct blkio_group_conf {
u64 bps[2];
};
+/* per-blkg per-policy data */
+struct blkg_policy_data {
+ /* the blkg this per-policy data belongs to */
+ struct blkio_group *blkg;
+
+ /* pol->pdata_size bytes of private data used by policy impl */
+ char pdata[] __aligned(__alignof__(unsigned long long));
+};
+
struct blkio_group {
/* Pointer to the associated request_queue, RCU protected */
struct request_queue __rcu *q;
@@ -177,10 +186,11 @@ struct blkio_group {
struct blkio_group_stats stats;
/* Per cpu stats pointer */
struct blkio_group_stats_cpu __percpu *stats_cpu;
+
+ struct blkg_policy_data *pd;
};
-typedef struct blkio_group *(blkio_alloc_group_fn)(struct request_queue *q,
- struct blkio_cgroup *blkcg);
+typedef void (blkio_init_group_fn)(struct blkio_group *blkg);
typedef void (blkio_link_group_fn)(struct request_queue *q,
struct blkio_group *blkg);
typedef void (blkio_unlink_group_fn)(struct request_queue *q,
@@ -198,7 +208,7 @@ typedef void (blkio_update_group_write_iops_fn)(struct request_queue *q,
struct blkio_group *blkg, unsigned int write_iops);
struct blkio_policy_ops {
- blkio_alloc_group_fn *blkio_alloc_group_fn;
+ blkio_init_group_fn *blkio_init_group_fn;
blkio_link_group_fn *blkio_link_group_fn;
blkio_unlink_group_fn *blkio_unlink_group_fn;
blkio_clear_queue_fn *blkio_clear_queue_fn;
@@ -213,6 +223,7 @@ struct blkio_policy_type {
struct list_head list;
struct blkio_policy_ops ops;
enum blkio_policy_id plid;
+ size_t pdata_size; /* policy specific private data size */
};
extern int blkcg_init_queue(struct request_queue *q);
@@ -224,6 +235,38 @@ extern void blkio_policy_register(struct blkio_policy_type *);
extern void blkio_policy_unregister(struct blkio_policy_type *);
extern void blkg_destroy_all(struct request_queue *q);
+/**
+ * blkg_to_pdata - get policy private data
+ * @blkg: blkg of interest
+ * @pol: policy of interest
+ *
+ * Return pointer to private data associated with the @blkg-@pol pair.
+ */
+static inline void *blkg_to_pdata(struct blkio_group *blkg,
+ struct blkio_policy_type *pol)
+{
+ return blkg ? blkg->pd->pdata : NULL;
+}
+
+/**
+ * pdata_to_blkg - get blkg associated with policy private data
+ * @pdata: policy private data of interest
+ * @pol: policy @pdata is for
+ *
+ * @pdata is policy private data for @pol. Determine the blkg it's
+ * associated with.
+ */
+static inline struct blkio_group *pdata_to_blkg(void *pdata,
+ struct blkio_policy_type *pol)
+{
+ if (pdata) {
+ struct blkg_policy_data *pd =
+ container_of(pdata, struct blkg_policy_data, pdata);
+ return pd->blkg;
+ }
+ return NULL;
+}
+
static inline char *blkg_path(struct blkio_group *blkg)
{
return blkg->path;
@@ -244,6 +287,10 @@ static inline void blkio_policy_register(struct blkio_policy_type *blkiop) { }
static inline void blkio_policy_unregister(struct blkio_policy_type *blkiop) { }
static inline void blkg_destroy_all(struct request_queue *q) { }
+static inline void *blkg_to_pdata(struct blkio_group *blkg,
+ struct blkio_policy_type *pol) { return NULL; }
+static inline struct blkio_group *pdata_to_blkg(void *pdata,
+ struct blkio_policy_type *pol) { return NULL; }
static inline char *blkg_path(struct blkio_group *blkg) { return NULL; }
#endif
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index ac6d0fe..9c8a124 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -21,6 +21,8 @@ static int throtl_quantum = 32;
/* Throttling is performed over 100ms slice and after that slice is renewed */
static unsigned long throtl_slice = HZ/10; /* 100 ms */
+static struct blkio_policy_type blkio_policy_throtl;
+
/* A workqueue to queue throttle related work */
static struct workqueue_struct *kthrotld_workqueue;
static void throtl_schedule_delayed_work(struct throtl_data *td,
@@ -52,7 +54,6 @@ struct throtl_grp {
*/
unsigned long disptime;
- struct blkio_group blkg;
atomic_t ref;
unsigned int flags;
@@ -108,6 +109,16 @@ struct throtl_data
int limits_changed;
};
+static inline struct throtl_grp *blkg_to_tg(struct blkio_group *blkg)
+{
+ return blkg_to_pdata(blkg, &blkio_policy_throtl);
+}
+
+static inline struct blkio_group *tg_to_blkg(struct throtl_grp *tg)
+{
+ return pdata_to_blkg(tg, &blkio_policy_throtl);
+}
+
enum tg_state_flags {
THROTL_TG_FLAG_on_rr = 0, /* on round-robin busy list */
};
@@ -130,19 +141,11 @@ THROTL_TG_FNS(on_rr);
#define throtl_log_tg(td, tg, fmt, args...) \
blk_add_trace_msg((td)->queue, "throtl %s " fmt, \
- blkg_path(&(tg)->blkg), ##args); \
+ blkg_path(tg_to_blkg(tg)), ##args); \
#define throtl_log(td, fmt, args...) \
blk_add_trace_msg((td)->queue, "throtl " fmt, ##args)
-static inline struct throtl_grp *tg_of_blkg(struct blkio_group *blkg)
-{
- if (blkg)
- return container_of(blkg, struct throtl_grp, blkg);
-
- return NULL;
-}
-
static inline unsigned int total_nr_queued(struct throtl_data *td)
{
return td->nr_queued[0] + td->nr_queued[1];
@@ -156,21 +159,24 @@ static inline struct throtl_grp *throtl_ref_get_tg(struct throtl_grp *tg)
static void throtl_free_tg(struct rcu_head *head)
{
- struct throtl_grp *tg;
+ struct throtl_grp *tg = container_of(head, struct throtl_grp, rcu_head);
+ struct blkio_group *blkg = tg_to_blkg(tg);
- tg = container_of(head, struct throtl_grp, rcu_head);
- free_percpu(tg->blkg.stats_cpu);
- kfree(tg);
+ free_percpu(blkg->stats_cpu);
+ kfree(blkg->pd);
+ kfree(blkg);
}
static void throtl_put_tg(struct throtl_grp *tg)
{
+ struct blkio_group *blkg = tg_to_blkg(tg);
+
BUG_ON(atomic_read(&tg->ref) <= 0);
if (!atomic_dec_and_test(&tg->ref))
return;
/* release the extra blkcg reference this blkg has been holding */
- css_put(&tg->blkg.blkcg->css);
+ css_put(&blkg->blkcg->css);
/*
* A group is freed in rcu manner. But having an rcu lock does not
@@ -184,14 +190,9 @@ static void throtl_put_tg(struct throtl_grp *tg)
call_rcu(&tg->rcu_head, throtl_free_tg);
}
-static struct blkio_group *throtl_alloc_blkio_group(struct request_queue *q,
- struct blkio_cgroup *blkcg)
+static void throtl_init_blkio_group(struct blkio_group *blkg)
{
- struct throtl_grp *tg;
-
- tg = kzalloc_node(sizeof(*tg), GFP_ATOMIC, q->node);
- if (!tg)
- return NULL;
+ struct throtl_grp *tg = blkg_to_tg(blkg);
INIT_HLIST_NODE(&tg->tg_node);
RB_CLEAR_NODE(&tg->rb_node);
@@ -211,15 +212,13 @@ static struct blkio_group *throtl_alloc_blkio_group(struct request_queue *q,
* exit or cgroup deletion path depending on who is exiting first.
*/
atomic_set(&tg->ref, 1);
-
- return &tg->blkg;
}
static void throtl_link_blkio_group(struct request_queue *q,
struct blkio_group *blkg)
{
struct throtl_data *td = q->td;
- struct throtl_grp *tg = tg_of_blkg(blkg);
+ struct throtl_grp *tg = blkg_to_tg(blkg);
hlist_add_head(&tg->tg_node, &td->tg_list);
td->nr_undestroyed_grps++;
@@ -235,7 +234,7 @@ throtl_grp *throtl_lookup_tg(struct throtl_data *td, struct blkio_cgroup *blkcg)
if (blkcg == &blkio_root_cgroup)
return td->root_tg;
- return tg_of_blkg(blkg_lookup(blkcg, td->queue, BLKIO_POLICY_THROTL));
+ return blkg_to_tg(blkg_lookup(blkcg, td->queue, BLKIO_POLICY_THROTL));
}
static struct throtl_grp *throtl_lookup_create_tg(struct throtl_data *td,
@@ -257,7 +256,7 @@ static struct throtl_grp *throtl_lookup_create_tg(struct throtl_data *td,
/* if %NULL and @q is alive, fall back to root_tg */
if (!IS_ERR(blkg))
- tg = tg_of_blkg(blkg);
+ tg = blkg_to_tg(blkg);
else if (!blk_queue_dead(q))
tg = td->root_tg;
}
@@ -639,7 +638,7 @@ static void throtl_charge_bio(struct throtl_grp *tg, struct bio *bio)
tg->bytes_disp[rw] += bio->bi_size;
tg->io_disp[rw]++;
- blkiocg_update_dispatch_stats(&tg->blkg, bio->bi_size, rw, sync);
+ blkiocg_update_dispatch_stats(tg_to_blkg(tg), bio->bi_size, rw, sync);
}
static void throtl_add_bio_tg(struct throtl_data *td, struct throtl_grp *tg,
@@ -901,7 +900,7 @@ static bool throtl_release_tgs(struct throtl_data *td, bool release_root)
* it from cgroup list, then it will take care of destroying
* cfqg also.
*/
- if (!blkiocg_del_blkio_group(&tg->blkg))
+ if (!blkiocg_del_blkio_group(tg_to_blkg(tg)))
throtl_destroy_tg(td, tg);
else
empty = false;
@@ -929,7 +928,7 @@ void throtl_unlink_blkio_group(struct request_queue *q,
unsigned long flags;
spin_lock_irqsave(q->queue_lock, flags);
- throtl_destroy_tg(q->td, tg_of_blkg(blkg));
+ throtl_destroy_tg(q->td, blkg_to_tg(blkg));
spin_unlock_irqrestore(q->queue_lock, flags);
}
@@ -968,7 +967,7 @@ static void throtl_update_blkio_group_common(struct throtl_data *td,
static void throtl_update_blkio_group_read_bps(struct request_queue *q,
struct blkio_group *blkg, u64 read_bps)
{
- struct throtl_grp *tg = tg_of_blkg(blkg);
+ struct throtl_grp *tg = blkg_to_tg(blkg);
tg->bps[READ] = read_bps;
throtl_update_blkio_group_common(q->td, tg);
@@ -977,7 +976,7 @@ static void throtl_update_blkio_group_read_bps(struct request_queue *q,
static void throtl_update_blkio_group_write_bps(struct request_queue *q,
struct blkio_group *blkg, u64 write_bps)
{
- struct throtl_grp *tg = tg_of_blkg(blkg);
+ struct throtl_grp *tg = blkg_to_tg(blkg);
tg->bps[WRITE] = write_bps;
throtl_update_blkio_group_common(q->td, tg);
@@ -986,7 +985,7 @@ static void throtl_update_blkio_group_write_bps(struct request_queue *q,
static void throtl_update_blkio_group_read_iops(struct request_queue *q,
struct blkio_group *blkg, unsigned int read_iops)
{
- struct throtl_grp *tg = tg_of_blkg(blkg);
+ struct throtl_grp *tg = blkg_to_tg(blkg);
tg->iops[READ] = read_iops;
throtl_update_blkio_group_common(q->td, tg);
@@ -995,7 +994,7 @@ static void throtl_update_blkio_group_read_iops(struct request_queue *q,
static void throtl_update_blkio_group_write_iops(struct request_queue *q,
struct blkio_group *blkg, unsigned int write_iops)
{
- struct throtl_grp *tg = tg_of_blkg(blkg);
+ struct throtl_grp *tg = blkg_to_tg(blkg);
tg->iops[WRITE] = write_iops;
throtl_update_blkio_group_common(q->td, tg);
@@ -1010,7 +1009,7 @@ static void throtl_shutdown_wq(struct request_queue *q)
static struct blkio_policy_type blkio_policy_throtl = {
.ops = {
- .blkio_alloc_group_fn = throtl_alloc_blkio_group,
+ .blkio_init_group_fn = throtl_init_blkio_group,
.blkio_link_group_fn = throtl_link_blkio_group,
.blkio_unlink_group_fn = throtl_unlink_blkio_group,
.blkio_clear_queue_fn = throtl_clear_queue,
@@ -1024,6 +1023,7 @@ static struct blkio_policy_type blkio_policy_throtl = {
throtl_update_blkio_group_write_iops,
},
.plid = BLKIO_POLICY_THROTL,
+ .pdata_size = sizeof(struct throtl_grp),
};
bool blk_throtl_bio(struct request_queue *q, struct bio *bio)
@@ -1049,8 +1049,9 @@ bool blk_throtl_bio(struct request_queue *q, struct bio *bio)
tg = throtl_lookup_tg(td, blkcg);
if (tg) {
if (tg_no_rule_group(tg, rw)) {
- blkiocg_update_dispatch_stats(&tg->blkg, bio->bi_size,
- rw, rw_is_sync(bio->bi_rw));
+ blkiocg_update_dispatch_stats(tg_to_blkg(tg),
+ bio->bi_size, rw,
+ rw_is_sync(bio->bi_rw));
goto out_unlock_rcu;
}
}
@@ -1176,7 +1177,7 @@ int blk_throtl_init(struct request_queue *q)
blkg = blkg_lookup_create(&blkio_root_cgroup, q, BLKIO_POLICY_THROTL,
true);
if (!IS_ERR(blkg))
- td->root_tg = tg_of_blkg(blkg);
+ td->root_tg = blkg_to_tg(blkg);
spin_unlock_irq(q->queue_lock);
rcu_read_unlock();
@@ -1207,7 +1208,7 @@ void blk_throtl_exit(struct request_queue *q)
spin_unlock_irq(q->queue_lock);
/*
- * Wait for tg->blkg->q accessors to exit their grace periods.
+ * Wait for tg_to_blkg(tg)->q accessors to exit their grace periods.
* Do this wait only if there are other undestroyed groups out
* there (other than root group). This can happen if cgroup deletion
* path claimed the responsibility of cleaning up a group before
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index c2e1645..0875c7f 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -17,6 +17,8 @@
#include "blk.h"
#include "cfq.h"
+static struct blkio_policy_type blkio_policy_cfq;
+
/*
* tunables
*/
@@ -206,7 +208,6 @@ struct cfq_group {
unsigned long saved_workload_slice;
enum wl_type_t saved_workload;
enum wl_prio_t saved_serving_prio;
- struct blkio_group blkg;
#ifdef CONFIG_CFQ_GROUP_IOSCHED
struct hlist_node cfqd_node;
int ref;
@@ -310,6 +311,16 @@ struct cfq_data {
unsigned int nr_blkcg_linked_grps;
};
+static inline struct cfq_group *blkg_to_cfqg(struct blkio_group *blkg)
+{
+ return blkg_to_pdata(blkg, &blkio_policy_cfq);
+}
+
+static inline struct blkio_group *cfqg_to_blkg(struct cfq_group *cfqg)
+{
+ return pdata_to_blkg(cfqg, &blkio_policy_cfq);
+}
+
static struct cfq_group *cfq_get_next_cfqg(struct cfq_data *cfqd);
static struct cfq_rb_root *service_tree_for(struct cfq_group *cfqg,
@@ -374,11 +385,11 @@ CFQ_CFQQ_FNS(wait_busy);
#define cfq_log_cfqq(cfqd, cfqq, fmt, args...) \
blk_add_trace_msg((cfqd)->queue, "cfq%d%c %s " fmt, (cfqq)->pid, \
cfq_cfqq_sync((cfqq)) ? 'S' : 'A', \
- blkg_path(&(cfqq)->cfqg->blkg), ##args)
+ blkg_path(cfqg_to_blkg((cfqq))), ##args)
#define cfq_log_cfqg(cfqd, cfqg, fmt, args...) \
blk_add_trace_msg((cfqd)->queue, "%s " fmt, \
- blkg_path(&(cfqg)->blkg), ##args) \
+ blkg_path(cfqg_to_blkg((cfqg))), ##args) \
#else
#define cfq_log_cfqq(cfqd, cfqq, fmt, args...) \
@@ -935,7 +946,7 @@ cfq_group_notify_queue_del(struct cfq_data *cfqd, struct cfq_group *cfqg)
cfq_log_cfqg(cfqd, cfqg, "del_from_rr group");
cfq_group_service_tree_del(st, cfqg);
cfqg->saved_workload_slice = 0;
- cfq_blkiocg_update_dequeue_stats(&cfqg->blkg, 1);
+ cfq_blkiocg_update_dequeue_stats(cfqg_to_blkg(cfqg), 1);
}
static inline unsigned int cfq_cfqq_slice_usage(struct cfq_queue *cfqq,
@@ -1007,9 +1018,9 @@ static void cfq_group_served(struct cfq_data *cfqd, struct cfq_group *cfqg,
"sl_used=%u disp=%u charge=%u iops=%u sect=%lu",
used_sl, cfqq->slice_dispatch, charge,
iops_mode(cfqd), cfqq->nr_sectors);
- cfq_blkiocg_update_timeslice_used(&cfqg->blkg, used_sl,
+ cfq_blkiocg_update_timeslice_used(cfqg_to_blkg(cfqg), used_sl,
unaccounted_sl);
- cfq_blkiocg_set_start_empty_time(&cfqg->blkg);
+ cfq_blkiocg_set_start_empty_time(cfqg_to_blkg(cfqg));
}
/**
@@ -1032,18 +1043,12 @@ static void cfq_init_cfqg_base(struct cfq_group *cfqg)
}
#ifdef CONFIG_CFQ_GROUP_IOSCHED
-static inline struct cfq_group *cfqg_of_blkg(struct blkio_group *blkg)
-{
- if (blkg)
- return container_of(blkg, struct cfq_group, blkg);
- return NULL;
-}
-
static void cfq_update_blkio_group_weight(struct request_queue *q,
struct blkio_group *blkg,
unsigned int weight)
{
- struct cfq_group *cfqg = cfqg_of_blkg(blkg);
+ struct cfq_group *cfqg = blkg_to_cfqg(blkg);
+
cfqg->new_weight = weight;
cfqg->needs_update = true;
}
@@ -1052,7 +1057,7 @@ static void cfq_link_blkio_group(struct request_queue *q,
struct blkio_group *blkg)
{
struct cfq_data *cfqd = q->elevator->elevator_data;
- struct cfq_group *cfqg = cfqg_of_blkg(blkg);
+ struct cfq_group *cfqg = blkg_to_cfqg(blkg);
cfqd->nr_blkcg_linked_grps++;
@@ -1060,17 +1065,12 @@ static void cfq_link_blkio_group(struct request_queue *q,
hlist_add_head(&cfqg->cfqd_node, &cfqd->cfqg_list);
}
-static struct blkio_group *cfq_alloc_blkio_group(struct request_queue *q,
- struct blkio_cgroup *blkcg)
+static void cfq_init_blkio_group(struct blkio_group *blkg)
{
- struct cfq_group *cfqg;
-
- cfqg = kzalloc_node(sizeof(*cfqg), GFP_ATOMIC, q->node);
- if (!cfqg)
- return NULL;
+ struct cfq_group *cfqg = blkg_to_cfqg(blkg);
cfq_init_cfqg_base(cfqg);
- cfqg->weight = blkcg->weight;
+ cfqg->weight = blkg->blkcg->weight;
/*
* Take the initial reference that will be released on destroy
@@ -1079,8 +1079,6 @@ static struct blkio_group *cfq_alloc_blkio_group(struct request_queue *q,
* or cgroup deletion path depending on who is exiting first.
*/
cfqg->ref = 1;
-
- return &cfqg->blkg;
}
/*
@@ -1101,7 +1099,7 @@ static struct cfq_group *cfq_lookup_create_cfqg(struct cfq_data *cfqd,
blkg = blkg_lookup_create(blkcg, q, BLKIO_POLICY_PROP, false);
if (!IS_ERR(blkg))
- cfqg = cfqg_of_blkg(blkg);
+ cfqg = blkg_to_cfqg(blkg);
}
return cfqg;
@@ -1126,6 +1124,7 @@ static void cfq_link_cfqq_cfqg(struct cfq_queue *cfqq, struct cfq_group *cfqg)
static void cfq_put_cfqg(struct cfq_group *cfqg)
{
+ struct blkio_group *blkg = cfqg_to_blkg(cfqg);
struct cfq_rb_root *st;
int i, j;
@@ -1135,12 +1134,13 @@ static void cfq_put_cfqg(struct cfq_group *cfqg)
return;
/* release the extra blkcg reference this blkg has been holding */
- css_put(&cfqg->blkg.blkcg->css);
+ css_put(&blkg->blkcg->css);
for_each_cfqg_st(cfqg, i, j, st)
BUG_ON(!RB_EMPTY_ROOT(&st->rb));
- free_percpu(cfqg->blkg.stats_cpu);
- kfree(cfqg);
+ free_percpu(blkg->stats_cpu);
+ kfree(blkg->pd);
+ kfree(blkg);
}
static void cfq_destroy_cfqg(struct cfq_data *cfqd, struct cfq_group *cfqg)
@@ -1172,7 +1172,7 @@ static bool cfq_release_cfq_groups(struct cfq_data *cfqd)
* it from cgroup list, then it will take care of destroying
* cfqg also.
*/
- if (!cfq_blkiocg_del_blkio_group(&cfqg->blkg))
+ if (!cfq_blkiocg_del_blkio_group(cfqg_to_blkg(cfqg)))
cfq_destroy_cfqg(cfqd, cfqg);
else
empty = false;
@@ -1201,7 +1201,7 @@ static void cfq_unlink_blkio_group(struct request_queue *q,
unsigned long flags;
spin_lock_irqsave(q->queue_lock, flags);
- cfq_destroy_cfqg(cfqd, cfqg_of_blkg(blkg));
+ cfq_destroy_cfqg(cfqd, blkg_to_cfqg(blkg));
spin_unlock_irqrestore(q->queue_lock, flags);
}
@@ -1504,12 +1504,12 @@ static void cfq_reposition_rq_rb(struct cfq_queue *cfqq, struct request *rq)
{
elv_rb_del(&cfqq->sort_list, rq);
cfqq->queued[rq_is_sync(rq)]--;
- cfq_blkiocg_update_io_remove_stats(&(RQ_CFQG(rq))->blkg,
+ cfq_blkiocg_update_io_remove_stats(cfqg_to_blkg(RQ_CFQG(rq)),
rq_data_dir(rq), rq_is_sync(rq));
cfq_add_rq_rb(rq);
- cfq_blkiocg_update_io_add_stats(&(RQ_CFQG(rq))->blkg,
- &cfqq->cfqd->serving_group->blkg, rq_data_dir(rq),
- rq_is_sync(rq));
+ cfq_blkiocg_update_io_add_stats(cfqg_to_blkg(RQ_CFQG(rq)),
+ cfqg_to_blkg(cfqq->cfqd->serving_group),
+ rq_data_dir(rq), rq_is_sync(rq));
}
static struct request *
@@ -1565,7 +1565,7 @@ static void cfq_remove_request(struct request *rq)
cfq_del_rq_rb(rq);
cfqq->cfqd->rq_queued--;
- cfq_blkiocg_update_io_remove_stats(&(RQ_CFQG(rq))->blkg,
+ cfq_blkiocg_update_io_remove_stats(cfqg_to_blkg(RQ_CFQG(rq)),
rq_data_dir(rq), rq_is_sync(rq));
if (rq->cmd_flags & REQ_PRIO) {
WARN_ON(!cfqq->prio_pending);
@@ -1601,7 +1601,7 @@ static void cfq_merged_request(struct request_queue *q, struct request *req,
static void cfq_bio_merged(struct request_queue *q, struct request *req,
struct bio *bio)
{
- cfq_blkiocg_update_io_merged_stats(&(RQ_CFQG(req))->blkg,
+ cfq_blkiocg_update_io_merged_stats(cfqg_to_blkg(RQ_CFQG(req)),
bio_data_dir(bio), cfq_bio_sync(bio));
}
@@ -1624,7 +1624,7 @@ cfq_merged_requests(struct request_queue *q, struct request *rq,
if (cfqq->next_rq == next)
cfqq->next_rq = rq;
cfq_remove_request(next);
- cfq_blkiocg_update_io_merged_stats(&(RQ_CFQG(rq))->blkg,
+ cfq_blkiocg_update_io_merged_stats(cfqg_to_blkg(RQ_CFQG(rq)),
rq_data_dir(next), rq_is_sync(next));
cfqq = RQ_CFQQ(next);
@@ -1673,7 +1673,7 @@ static int cfq_allow_merge(struct request_queue *q, struct request *rq,
static inline void cfq_del_timer(struct cfq_data *cfqd, struct cfq_queue *cfqq)
{
del_timer(&cfqd->idle_slice_timer);
- cfq_blkiocg_update_idle_time_stats(&cfqq->cfqg->blkg);
+ cfq_blkiocg_update_idle_time_stats(cfqg_to_blkg(cfqq->cfqg));
}
static void __cfq_set_active_queue(struct cfq_data *cfqd,
@@ -1682,7 +1682,7 @@ static void __cfq_set_active_queue(struct cfq_data *cfqd,
if (cfqq) {
cfq_log_cfqq(cfqd, cfqq, "set_active wl_prio:%d wl_type:%d",
cfqd->serving_prio, cfqd->serving_type);
- cfq_blkiocg_update_avg_queue_size_stats(&cfqq->cfqg->blkg);
+ cfq_blkiocg_update_avg_queue_size_stats(cfqg_to_blkg(cfqq->cfqg));
cfqq->slice_start = 0;
cfqq->dispatch_start = jiffies;
cfqq->allocated_slice = 0;
@@ -2030,7 +2030,7 @@ static void cfq_arm_slice_timer(struct cfq_data *cfqd)
sl = cfqd->cfq_slice_idle;
mod_timer(&cfqd->idle_slice_timer, jiffies + sl);
- cfq_blkiocg_update_set_idle_time_stats(&cfqq->cfqg->blkg);
+ cfq_blkiocg_update_set_idle_time_stats(cfqg_to_blkg(cfqq->cfqg));
cfq_log_cfqq(cfqd, cfqq, "arm_idle: %lu group_idle: %d", sl,
group_idle ? 1 : 0);
}
@@ -2053,8 +2053,9 @@ static void cfq_dispatch_insert(struct request_queue *q, struct request *rq)
cfqd->rq_in_flight[cfq_cfqq_sync(cfqq)]++;
cfqq->nr_sectors += blk_rq_sectors(rq);
- cfq_blkiocg_update_dispatch_stats(&cfqq->cfqg->blkg, blk_rq_bytes(rq),
- rq_data_dir(rq), rq_is_sync(rq));
+ cfq_blkiocg_update_dispatch_stats(cfqg_to_blkg(cfqq->cfqg),
+ blk_rq_bytes(rq), rq_data_dir(rq),
+ rq_is_sync(rq));
}
/*
@@ -3141,7 +3142,7 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq,
__blk_run_queue(cfqd->queue);
} else {
cfq_blkiocg_update_idle_time_stats(
- &cfqq->cfqg->blkg);
+ cfqg_to_blkg(cfqq->cfqg));
cfq_mark_cfqq_must_dispatch(cfqq);
}
}
@@ -3168,9 +3169,9 @@ static void cfq_insert_request(struct request_queue *q, struct request *rq)
rq_set_fifo_time(rq, jiffies + cfqd->cfq_fifo_expire[rq_is_sync(rq)]);
list_add_tail(&rq->queuelist, &cfqq->fifo);
cfq_add_rq_rb(rq);
- cfq_blkiocg_update_io_add_stats(&(RQ_CFQG(rq))->blkg,
- &cfqd->serving_group->blkg, rq_data_dir(rq),
- rq_is_sync(rq));
+ cfq_blkiocg_update_io_add_stats(cfqg_to_blkg(RQ_CFQG(rq)),
+ cfqg_to_blkg(cfqd->serving_group),
+ rq_data_dir(rq), rq_is_sync(rq));
cfq_rq_enqueued(cfqd, cfqq, rq);
}
@@ -3266,7 +3267,7 @@ static void cfq_completed_request(struct request_queue *q, struct request *rq)
cfqd->rq_in_driver--;
cfqq->dispatched--;
(RQ_CFQG(rq))->dispatched--;
- cfq_blkiocg_update_completion_stats(&cfqq->cfqg->blkg,
+ cfq_blkiocg_update_completion_stats(cfqg_to_blkg(cfqq->cfqg),
rq_start_time_ns(rq), rq_io_start_time_ns(rq),
rq_data_dir(rq), rq_is_sync(rq));
@@ -3647,7 +3648,7 @@ static int cfq_init_queue(struct request_queue *q)
blkg = blkg_lookup_create(&blkio_root_cgroup, q, BLKIO_POLICY_PROP,
true);
if (!IS_ERR(blkg))
- cfqd->root_group = cfqg_of_blkg(blkg);
+ cfqd->root_group = blkg_to_cfqg(blkg);
spin_unlock_irq(q->queue_lock);
rcu_read_unlock();
@@ -3833,13 +3834,14 @@ static struct elevator_type iosched_cfq = {
#ifdef CONFIG_CFQ_GROUP_IOSCHED
static struct blkio_policy_type blkio_policy_cfq = {
.ops = {
- .blkio_alloc_group_fn = cfq_alloc_blkio_group,
+ .blkio_init_group_fn = cfq_init_blkio_group,
.blkio_link_group_fn = cfq_link_blkio_group,
.blkio_unlink_group_fn = cfq_unlink_blkio_group,
.blkio_clear_queue_fn = cfq_clear_queue,
.blkio_update_group_weight_fn = cfq_update_blkio_group_weight,
},
.plid = BLKIO_POLICY_PROP,
+ .pdata_size = sizeof(struct cfq_group),
};
#endif
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH 06/11] blkcg: move refcnt to blkcg core
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
` (4 preceding siblings ...)
2012-02-01 21:19 ` [PATCH 05/11] blkcg: let blkcg core handle policy private data allocation Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-02 22:07 ` Vivek Goyal
2012-02-01 21:19 ` [PATCH 07/11] blkcg: make blkg->pd an array and move configuration and stats into it Tejun Heo
` (5 subsequent siblings)
11 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
Currently, blkcg policy implementations manage blkg refcnt duplicating
mostly identical code in both policies. This patch moves refcnt to
blkg and let blkcg core handle refcnt and freeing of blkgs.
* cfq blkgs now also get freed via RCU.
* cfq blkgs lose RB_EMPTY_ROOT() sanity check on blkg free. If
necessary, we can add blkio_exit_group_fn() to resurrect this.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-cgroup.c | 24 ++++++++++++++++++++
block/blk-cgroup.h | 35 ++++++++++++++++++++++++++++++
block/blk-throttle.c | 58 +++----------------------------------------------
block/cfq-iosched.c | 58 ++++++++-----------------------------------------
4 files changed, 73 insertions(+), 102 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index e42da84..ba49af3 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -463,6 +463,7 @@ static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
rcu_assign_pointer(blkg->q, q);
blkg->blkcg = blkcg;
blkg->plid = pol->plid;
+ blkg->refcnt = 1;
cgroup_path(blkcg->css.cgroup, blkg->path, sizeof(blkg->path));
/* alloc per-policy data */
@@ -633,6 +634,29 @@ void blkg_destroy_all(struct request_queue *q)
}
}
+static void blkg_rcu_free(struct rcu_head *rcu_head)
+{
+ blkg_free(container_of(rcu_head, struct blkio_group, rcu_head));
+}
+
+void __blkg_release(struct blkio_group *blkg)
+{
+ /* release the extra blkcg reference this blkg has been holding */
+ css_put(&blkg->blkcg->css);
+
+ /*
+ * A group is freed in rcu manner. But having an rcu lock does not
+ * mean that one can access all the fields of blkg and assume these
+ * are valid. For example, don't try to follow throtl_data and
+ * request queue links.
+ *
+ * Having a reference to blkg under an rcu allows acess to only
+ * values local to groups like group stats and group rate limits
+ */
+ call_rcu(&blkg->rcu_head, blkg_rcu_free);
+}
+EXPORT_SYMBOL_GPL(__blkg_release);
+
static void blkio_reset_stats_cpu(struct blkio_group *blkg)
{
struct blkio_group_stats_cpu *stats_cpu;
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 9537819..7da1068 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -177,6 +177,8 @@ struct blkio_group {
char path[128];
/* policy which owns this blk group */
enum blkio_policy_id plid;
+ /* reference count */
+ int refcnt;
/* Configuration */
struct blkio_group_conf conf;
@@ -188,6 +190,8 @@ struct blkio_group {
struct blkio_group_stats_cpu __percpu *stats_cpu;
struct blkg_policy_data *pd;
+
+ struct rcu_head rcu_head;
};
typedef void (blkio_init_group_fn)(struct blkio_group *blkg);
@@ -272,6 +276,35 @@ static inline char *blkg_path(struct blkio_group *blkg)
return blkg->path;
}
+/**
+ * blkg_get - get a blkg reference
+ * @blkg: blkg to get
+ *
+ * The caller should be holding queue_lock and an existing reference.
+ */
+static inline void blkg_get(struct blkio_group *blkg)
+{
+ lockdep_assert_held(blkg->q->queue_lock);
+ WARN_ON_ONCE(!blkg->refcnt);
+ blkg->refcnt++;
+}
+
+void __blkg_release(struct blkio_group *blkg);
+
+/**
+ * blkg_put - put a blkg reference
+ * @blkg: blkg to put
+ *
+ * The caller should be holding queue_lock.
+ */
+static inline void blkg_put(struct blkio_group *blkg)
+{
+ lockdep_assert_held(blkg->q->queue_lock);
+ WARN_ON_ONCE(blkg->refcnt <= 0);
+ if (!--blkg->refcnt)
+ __blkg_release(blkg);
+}
+
#else
struct blkio_group {
@@ -292,6 +325,8 @@ static inline void *blkg_to_pdata(struct blkio_group *blkg,
static inline struct blkio_group *pdata_to_blkg(void *pdata,
struct blkio_policy_type *pol) { return NULL; }
static inline char *blkg_path(struct blkio_group *blkg) { return NULL; }
+static inline void blkg_get(struct blkio_group *blkg) { }
+static inline void blkg_put(struct blkio_group *blkg) { }
#endif
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 9c8a124..153ba50 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -54,7 +54,6 @@ struct throtl_grp {
*/
unsigned long disptime;
- atomic_t ref;
unsigned int flags;
/* Two lists for READ and WRITE */
@@ -80,8 +79,6 @@ struct throtl_grp {
/* Some throttle limits got updated for the group */
int limits_changed;
-
- struct rcu_head rcu_head;
};
struct throtl_data
@@ -151,45 +148,6 @@ static inline unsigned int total_nr_queued(struct throtl_data *td)
return td->nr_queued[0] + td->nr_queued[1];
}
-static inline struct throtl_grp *throtl_ref_get_tg(struct throtl_grp *tg)
-{
- atomic_inc(&tg->ref);
- return tg;
-}
-
-static void throtl_free_tg(struct rcu_head *head)
-{
- struct throtl_grp *tg = container_of(head, struct throtl_grp, rcu_head);
- struct blkio_group *blkg = tg_to_blkg(tg);
-
- free_percpu(blkg->stats_cpu);
- kfree(blkg->pd);
- kfree(blkg);
-}
-
-static void throtl_put_tg(struct throtl_grp *tg)
-{
- struct blkio_group *blkg = tg_to_blkg(tg);
-
- BUG_ON(atomic_read(&tg->ref) <= 0);
- if (!atomic_dec_and_test(&tg->ref))
- return;
-
- /* release the extra blkcg reference this blkg has been holding */
- css_put(&blkg->blkcg->css);
-
- /*
- * A group is freed in rcu manner. But having an rcu lock does not
- * mean that one can access all the fields of blkg and assume these
- * are valid. For example, don't try to follow throtl_data and
- * request queue links.
- *
- * Having a reference to blkg under an rcu allows acess to only
- * values local to groups like group stats and group rate limits
- */
- call_rcu(&tg->rcu_head, throtl_free_tg);
-}
-
static void throtl_init_blkio_group(struct blkio_group *blkg)
{
struct throtl_grp *tg = blkg_to_tg(blkg);
@@ -204,14 +162,6 @@ static void throtl_init_blkio_group(struct blkio_group *blkg)
tg->bps[WRITE] = -1;
tg->iops[READ] = -1;
tg->iops[WRITE] = -1;
-
- /*
- * Take the initial reference that will be released on destroy
- * This can be thought of a joint reference by cgroup and
- * request queue which will be dropped by either request queue
- * exit or cgroup deletion path depending on who is exiting first.
- */
- atomic_set(&tg->ref, 1);
}
static void throtl_link_blkio_group(struct request_queue *q,
@@ -648,7 +598,7 @@ static void throtl_add_bio_tg(struct throtl_data *td, struct throtl_grp *tg,
bio_list_add(&tg->bio_lists[rw], bio);
/* Take a bio reference on tg */
- throtl_ref_get_tg(tg);
+ blkg_get(tg_to_blkg(tg));
tg->nr_queued[rw]++;
td->nr_queued[rw]++;
throtl_enqueue_tg(td, tg);
@@ -681,8 +631,8 @@ static void tg_dispatch_one_bio(struct throtl_data *td, struct throtl_grp *tg,
bio = bio_list_pop(&tg->bio_lists[rw]);
tg->nr_queued[rw]--;
- /* Drop bio reference on tg */
- throtl_put_tg(tg);
+ /* Drop bio reference on blkg */
+ blkg_put(tg_to_blkg(tg));
BUG_ON(td->nr_queued[rw] <= 0);
td->nr_queued[rw]--;
@@ -880,7 +830,7 @@ throtl_destroy_tg(struct throtl_data *td, struct throtl_grp *tg)
* Put the reference taken at the time of creation so that when all
* queues are gone, group can be destroyed.
*/
- throtl_put_tg(tg);
+ blkg_put(tg_to_blkg(tg));
td->nr_undestroyed_grps--;
}
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 0875c7f..5ce81a8 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -210,7 +210,6 @@ struct cfq_group {
enum wl_prio_t saved_serving_prio;
#ifdef CONFIG_CFQ_GROUP_IOSCHED
struct hlist_node cfqd_node;
- int ref;
#endif
/* number of requests that are on the dispatch list or inside driver */
int dispatched;
@@ -1071,14 +1070,6 @@ static void cfq_init_blkio_group(struct blkio_group *blkg)
cfq_init_cfqg_base(cfqg);
cfqg->weight = blkg->blkcg->weight;
-
- /*
- * Take the initial reference that will be released on destroy
- * This can be thought of a joint reference by cgroup and
- * elevator which will be dropped by either elevator exit
- * or cgroup deletion path depending on who is exiting first.
- */
- cfqg->ref = 1;
}
/*
@@ -1105,12 +1096,6 @@ static struct cfq_group *cfq_lookup_create_cfqg(struct cfq_data *cfqd,
return cfqg;
}
-static inline struct cfq_group *cfq_ref_get_cfqg(struct cfq_group *cfqg)
-{
- cfqg->ref++;
- return cfqg;
-}
-
static void cfq_link_cfqq_cfqg(struct cfq_queue *cfqq, struct cfq_group *cfqg)
{
/* Currently, all async queues are mapped to root group */
@@ -1119,28 +1104,7 @@ static void cfq_link_cfqq_cfqg(struct cfq_queue *cfqq, struct cfq_group *cfqg)
cfqq->cfqg = cfqg;
/* cfqq reference on cfqg */
- cfqq->cfqg->ref++;
-}
-
-static void cfq_put_cfqg(struct cfq_group *cfqg)
-{
- struct blkio_group *blkg = cfqg_to_blkg(cfqg);
- struct cfq_rb_root *st;
- int i, j;
-
- BUG_ON(cfqg->ref <= 0);
- cfqg->ref--;
- if (cfqg->ref)
- return;
-
- /* release the extra blkcg reference this blkg has been holding */
- css_put(&blkg->blkcg->css);
-
- for_each_cfqg_st(cfqg, i, j, st)
- BUG_ON(!RB_EMPTY_ROOT(&st->rb));
- free_percpu(blkg->stats_cpu);
- kfree(blkg->pd);
- kfree(blkg);
+ blkg_get(cfqg_to_blkg(cfqg));
}
static void cfq_destroy_cfqg(struct cfq_data *cfqd, struct cfq_group *cfqg)
@@ -1157,7 +1121,7 @@ static void cfq_destroy_cfqg(struct cfq_data *cfqd, struct cfq_group *cfqg)
* Put the reference taken at the time of creation so that when all
* queues are gone, group can be destroyed.
*/
- cfq_put_cfqg(cfqg);
+ blkg_put(cfqg_to_blkg(cfqg));
}
static bool cfq_release_cfq_groups(struct cfq_data *cfqd)
@@ -1225,18 +1189,12 @@ static struct cfq_group *cfq_lookup_create_cfqg(struct cfq_data *cfqd,
return cfqd->root_group;
}
-static inline struct cfq_group *cfq_ref_get_cfqg(struct cfq_group *cfqg)
-{
- return cfqg;
-}
-
static inline void
cfq_link_cfqq_cfqg(struct cfq_queue *cfqq, struct cfq_group *cfqg) {
cfqq->cfqg = cfqg;
}
static void cfq_release_cfq_groups(struct cfq_data *cfqd) {}
-static inline void cfq_put_cfqg(struct cfq_group *cfqg) {}
#endif /* GROUP_IOSCHED */
@@ -2637,7 +2595,7 @@ static void cfq_put_queue(struct cfq_queue *cfqq)
BUG_ON(cfq_cfqq_on_rr(cfqq));
kmem_cache_free(cfq_pool, cfqq);
- cfq_put_cfqg(cfqg);
+ blkg_put(cfqg_to_blkg(cfqg));
}
static void cfq_put_cooperator(struct cfq_queue *cfqq)
@@ -3388,7 +3346,7 @@ static void cfq_put_request(struct request *rq)
cfqq->allocated[rw]--;
/* Put down rq reference on cfqg */
- cfq_put_cfqg(RQ_CFQG(rq));
+ blkg_put(cfqg_to_blkg(RQ_CFQG(rq)));
rq->elv.priv[0] = NULL;
rq->elv.priv[1] = NULL;
@@ -3483,8 +3441,9 @@ new_queue:
cfqq->allocated[rw]++;
cfqq->ref++;
+ blkg_get(cfqg_to_blkg(cfqq->cfqg));
rq->elv.priv[0] = cfqq;
- rq->elv.priv[1] = cfq_ref_get_cfqg(cfqq->cfqg);
+ rq->elv.priv[1] = cfqq->cfqg;
spin_unlock_irq(q->queue_lock);
return 0;
}
@@ -3682,8 +3641,11 @@ static int cfq_init_queue(struct request_queue *q)
*/
cfq_init_cfqq(cfqd, &cfqd->oom_cfqq, 1, 0);
cfqd->oom_cfqq.ref++;
+
+ spin_lock_irq(q->queue_lock);
cfq_link_cfqq_cfqg(&cfqd->oom_cfqq, cfqd->root_group);
- cfq_put_cfqg(cfqd->root_group);
+ blkg_put(cfqg_to_blkg(cfqd->root_group));
+ spin_unlock_irq(q->queue_lock);
init_timer(&cfqd->idle_slice_timer);
cfqd->idle_slice_timer.function = cfq_idle_slice_timer;
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH 07/11] blkcg: make blkg->pd an array and move configuration and stats into it
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
` (5 preceding siblings ...)
2012-02-01 21:19 ` [PATCH 06/11] blkcg: move refcnt to blkcg core Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-01 21:19 ` [PATCH 08/11] blkcg: don't use blkg->plid in stat related functions Tejun Heo
` (4 subsequent siblings)
11 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
To prepare for unifying blkgs for different policies, make blkg->pd an
array with BLKIO_NR_POLICIES elements and move blkg->conf, ->stats,
and ->stats_cpu into blkg_policy_data.
This patch doesn't introduce any functional difference.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-cgroup.c | 150 ++++++++++++++++++++++++++++++++--------------------
block/blk-cgroup.h | 18 +++---
2 files changed, 102 insertions(+), 66 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index ba49af3..5696a8d 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -184,12 +184,14 @@ static void blkio_check_and_dec_stat(uint64_t *stat, bool direction, bool sync)
static void blkio_set_start_group_wait_time(struct blkio_group *blkg,
struct blkio_group *curr_blkg)
{
- if (blkio_blkg_waiting(&blkg->stats))
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+
+ if (blkio_blkg_waiting(&pd->stats))
return;
if (blkg == curr_blkg)
return;
- blkg->stats.start_group_wait_time = sched_clock();
- blkio_mark_blkg_waiting(&blkg->stats);
+ pd->stats.start_group_wait_time = sched_clock();
+ blkio_mark_blkg_waiting(&pd->stats);
}
/* This should be called with the blkg->stats_lock held. */
@@ -222,24 +224,26 @@ static void blkio_end_empty_time(struct blkio_group_stats *stats)
void blkiocg_update_set_idle_time_stats(struct blkio_group *blkg)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
unsigned long flags;
spin_lock_irqsave(&blkg->stats_lock, flags);
- BUG_ON(blkio_blkg_idling(&blkg->stats));
- blkg->stats.start_idle_time = sched_clock();
- blkio_mark_blkg_idling(&blkg->stats);
+ BUG_ON(blkio_blkg_idling(&pd->stats));
+ pd->stats.start_idle_time = sched_clock();
+ blkio_mark_blkg_idling(&pd->stats);
spin_unlock_irqrestore(&blkg->stats_lock, flags);
}
EXPORT_SYMBOL_GPL(blkiocg_update_set_idle_time_stats);
void blkiocg_update_idle_time_stats(struct blkio_group *blkg)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
unsigned long flags;
unsigned long long now;
struct blkio_group_stats *stats;
spin_lock_irqsave(&blkg->stats_lock, flags);
- stats = &blkg->stats;
+ stats = &pd->stats;
if (blkio_blkg_idling(stats)) {
now = sched_clock();
if (time_after64(now, stats->start_idle_time))
@@ -252,11 +256,12 @@ EXPORT_SYMBOL_GPL(blkiocg_update_idle_time_stats);
void blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
unsigned long flags;
struct blkio_group_stats *stats;
spin_lock_irqsave(&blkg->stats_lock, flags);
- stats = &blkg->stats;
+ stats = &pd->stats;
stats->avg_queue_size_sum +=
stats->stat_arr[BLKIO_STAT_QUEUED][BLKIO_STAT_READ] +
stats->stat_arr[BLKIO_STAT_QUEUED][BLKIO_STAT_WRITE];
@@ -268,11 +273,12 @@ EXPORT_SYMBOL_GPL(blkiocg_update_avg_queue_size_stats);
void blkiocg_set_start_empty_time(struct blkio_group *blkg)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
unsigned long flags;
struct blkio_group_stats *stats;
spin_lock_irqsave(&blkg->stats_lock, flags);
- stats = &blkg->stats;
+ stats = &pd->stats;
if (stats->stat_arr[BLKIO_STAT_QUEUED][BLKIO_STAT_READ] ||
stats->stat_arr[BLKIO_STAT_QUEUED][BLKIO_STAT_WRITE]) {
@@ -299,7 +305,9 @@ EXPORT_SYMBOL_GPL(blkiocg_set_start_empty_time);
void blkiocg_update_dequeue_stats(struct blkio_group *blkg,
unsigned long dequeue)
{
- blkg->stats.dequeue += dequeue;
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+
+ pd->stats.dequeue += dequeue;
}
EXPORT_SYMBOL_GPL(blkiocg_update_dequeue_stats);
#else
@@ -312,12 +320,13 @@ void blkiocg_update_io_add_stats(struct blkio_group *blkg,
struct blkio_group *curr_blkg, bool direction,
bool sync)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
unsigned long flags;
spin_lock_irqsave(&blkg->stats_lock, flags);
- blkio_add_stat(blkg->stats.stat_arr[BLKIO_STAT_QUEUED], 1, direction,
+ blkio_add_stat(pd->stats.stat_arr[BLKIO_STAT_QUEUED], 1, direction,
sync);
- blkio_end_empty_time(&blkg->stats);
+ blkio_end_empty_time(&pd->stats);
blkio_set_start_group_wait_time(blkg, curr_blkg);
spin_unlock_irqrestore(&blkg->stats_lock, flags);
}
@@ -326,10 +335,11 @@ EXPORT_SYMBOL_GPL(blkiocg_update_io_add_stats);
void blkiocg_update_io_remove_stats(struct blkio_group *blkg,
bool direction, bool sync)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
unsigned long flags;
spin_lock_irqsave(&blkg->stats_lock, flags);
- blkio_check_and_dec_stat(blkg->stats.stat_arr[BLKIO_STAT_QUEUED],
+ blkio_check_and_dec_stat(pd->stats.stat_arr[BLKIO_STAT_QUEUED],
direction, sync);
spin_unlock_irqrestore(&blkg->stats_lock, flags);
}
@@ -338,12 +348,13 @@ EXPORT_SYMBOL_GPL(blkiocg_update_io_remove_stats);
void blkiocg_update_timeslice_used(struct blkio_group *blkg, unsigned long time,
unsigned long unaccounted_time)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
unsigned long flags;
spin_lock_irqsave(&blkg->stats_lock, flags);
- blkg->stats.time += time;
+ pd->stats.time += time;
#ifdef CONFIG_DEBUG_BLK_CGROUP
- blkg->stats.unaccounted_time += unaccounted_time;
+ pd->stats.unaccounted_time += unaccounted_time;
#endif
spin_unlock_irqrestore(&blkg->stats_lock, flags);
}
@@ -356,6 +367,7 @@ EXPORT_SYMBOL_GPL(blkiocg_update_timeslice_used);
void blkiocg_update_dispatch_stats(struct blkio_group *blkg,
uint64_t bytes, bool direction, bool sync)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
struct blkio_group_stats_cpu *stats_cpu;
unsigned long flags;
@@ -366,7 +378,7 @@ void blkiocg_update_dispatch_stats(struct blkio_group *blkg,
*/
local_irq_save(flags);
- stats_cpu = this_cpu_ptr(blkg->stats_cpu);
+ stats_cpu = this_cpu_ptr(pd->stats_cpu);
u64_stats_update_begin(&stats_cpu->syncp);
stats_cpu->sectors += bytes >> 9;
@@ -382,12 +394,13 @@ EXPORT_SYMBOL_GPL(blkiocg_update_dispatch_stats);
void blkiocg_update_completion_stats(struct blkio_group *blkg,
uint64_t start_time, uint64_t io_start_time, bool direction, bool sync)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
struct blkio_group_stats *stats;
unsigned long flags;
unsigned long long now = sched_clock();
spin_lock_irqsave(&blkg->stats_lock, flags);
- stats = &blkg->stats;
+ stats = &pd->stats;
if (time_after64(now, io_start_time))
blkio_add_stat(stats->stat_arr[BLKIO_STAT_SERVICE_TIME],
now - io_start_time, direction, sync);
@@ -402,6 +415,7 @@ EXPORT_SYMBOL_GPL(blkiocg_update_completion_stats);
void blkiocg_update_io_merged_stats(struct blkio_group *blkg, bool direction,
bool sync)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
struct blkio_group_stats_cpu *stats_cpu;
unsigned long flags;
@@ -412,7 +426,7 @@ void blkiocg_update_io_merged_stats(struct blkio_group *blkg, bool direction,
*/
local_irq_save(flags);
- stats_cpu = this_cpu_ptr(blkg->stats_cpu);
+ stats_cpu = this_cpu_ptr(pd->stats_cpu);
u64_stats_update_begin(&stats_cpu->syncp);
blkio_add_stat(stats_cpu->stat_arr_cpu[BLKIO_STAT_CPU_MERGED], 1,
@@ -430,11 +444,17 @@ EXPORT_SYMBOL_GPL(blkiocg_update_io_merged_stats);
*/
static void blkg_free(struct blkio_group *blkg)
{
- if (blkg) {
- free_percpu(blkg->stats_cpu);
- kfree(blkg->pd);
- kfree(blkg);
+ struct blkg_policy_data *pd;
+
+ if (!blkg)
+ return;
+
+ pd = blkg->pd[blkg->plid];
+ if (pd) {
+ free_percpu(pd->stats_cpu);
+ kfree(pd);
}
+ kfree(blkg);
}
/**
@@ -453,6 +473,7 @@ static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
struct blkio_policy_type *pol)
{
struct blkio_group *blkg;
+ struct blkg_policy_data *pd;
/* alloc and init base part */
blkg = kzalloc_node(sizeof(*blkg), GFP_ATOMIC, q->node);
@@ -466,23 +487,26 @@ static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
blkg->refcnt = 1;
cgroup_path(blkcg->css.cgroup, blkg->path, sizeof(blkg->path));
- /* alloc per-policy data */
- blkg->pd = kzalloc_node(sizeof(*blkg->pd) + pol->pdata_size, GFP_ATOMIC,
- q->node);
- if (!blkg->pd) {
+ /* alloc per-policy data and attach it to blkg */
+ pd = kzalloc_node(sizeof(*pd) + pol->pdata_size, GFP_ATOMIC,
+ q->node);
+ if (!pd) {
blkg_free(blkg);
return NULL;
}
+ blkg->pd[pol->plid] = pd;
+ pd->blkg = blkg;
+
/* broken, read comment in the callsite */
- blkg->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
- if (!blkg->stats_cpu) {
+
+ pd->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
+ if (!pd->stats_cpu) {
blkg_free(blkg);
return NULL;
}
- /* attach pd to blkg and invoke per-policy init */
- blkg->pd->blkg = blkg;
+ /* invoke per-policy init */
pol->ops.blkio_init_group_fn(blkg);
return blkg;
}
@@ -659,6 +683,7 @@ EXPORT_SYMBOL_GPL(__blkg_release);
static void blkio_reset_stats_cpu(struct blkio_group *blkg)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
struct blkio_group_stats_cpu *stats_cpu;
int i, j, k;
/*
@@ -673,7 +698,7 @@ static void blkio_reset_stats_cpu(struct blkio_group *blkg)
* unless this becomes a real issue.
*/
for_each_possible_cpu(i) {
- stats_cpu = per_cpu_ptr(blkg->stats_cpu, i);
+ stats_cpu = per_cpu_ptr(pd->stats_cpu, i);
stats_cpu->sectors = 0;
for(j = 0; j < BLKIO_STAT_CPU_NR; j++)
for (k = 0; k < BLKIO_STAT_TOTAL; k++)
@@ -698,8 +723,10 @@ blkiocg_reset_stats(struct cgroup *cgroup, struct cftype *cftype, u64 val)
blkcg = cgroup_to_blkio_cgroup(cgroup);
spin_lock_irq(&blkcg->lock);
hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node) {
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+
spin_lock(&blkg->stats_lock);
- stats = &blkg->stats;
+ stats = &pd->stats;
#ifdef CONFIG_DEBUG_BLK_CGROUP
idling = blkio_blkg_idling(stats);
waiting = blkio_blkg_waiting(stats);
@@ -779,13 +806,14 @@ static uint64_t blkio_fill_stat(char *str, int chars_left, uint64_t val,
static uint64_t blkio_read_stat_cpu(struct blkio_group *blkg,
enum stat_type_cpu type, enum stat_sub_type sub_type)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
int cpu;
struct blkio_group_stats_cpu *stats_cpu;
u64 val = 0, tval;
for_each_possible_cpu(cpu) {
unsigned int start;
- stats_cpu = per_cpu_ptr(blkg->stats_cpu, cpu);
+ stats_cpu = per_cpu_ptr(pd->stats_cpu, cpu);
do {
start = u64_stats_fetch_begin(&stats_cpu->syncp);
@@ -837,20 +865,21 @@ static uint64_t blkio_get_stat(struct blkio_group *blkg,
struct cgroup_map_cb *cb, const char *dname,
enum stat_type type)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
uint64_t disk_total;
char key_str[MAX_KEY_LEN];
enum stat_sub_type sub_type;
if (type == BLKIO_STAT_TIME)
return blkio_fill_stat(key_str, MAX_KEY_LEN - 1,
- blkg->stats.time, cb, dname);
+ pd->stats.time, cb, dname);
#ifdef CONFIG_DEBUG_BLK_CGROUP
if (type == BLKIO_STAT_UNACCOUNTED_TIME)
return blkio_fill_stat(key_str, MAX_KEY_LEN - 1,
- blkg->stats.unaccounted_time, cb, dname);
+ pd->stats.unaccounted_time, cb, dname);
if (type == BLKIO_STAT_AVG_QUEUE_SIZE) {
- uint64_t sum = blkg->stats.avg_queue_size_sum;
- uint64_t samples = blkg->stats.avg_queue_size_samples;
+ uint64_t sum = pd->stats.avg_queue_size_sum;
+ uint64_t samples = pd->stats.avg_queue_size_samples;
if (samples)
do_div(sum, samples);
else
@@ -860,26 +889,26 @@ static uint64_t blkio_get_stat(struct blkio_group *blkg,
}
if (type == BLKIO_STAT_GROUP_WAIT_TIME)
return blkio_fill_stat(key_str, MAX_KEY_LEN - 1,
- blkg->stats.group_wait_time, cb, dname);
+ pd->stats.group_wait_time, cb, dname);
if (type == BLKIO_STAT_IDLE_TIME)
return blkio_fill_stat(key_str, MAX_KEY_LEN - 1,
- blkg->stats.idle_time, cb, dname);
+ pd->stats.idle_time, cb, dname);
if (type == BLKIO_STAT_EMPTY_TIME)
return blkio_fill_stat(key_str, MAX_KEY_LEN - 1,
- blkg->stats.empty_time, cb, dname);
+ pd->stats.empty_time, cb, dname);
if (type == BLKIO_STAT_DEQUEUE)
return blkio_fill_stat(key_str, MAX_KEY_LEN - 1,
- blkg->stats.dequeue, cb, dname);
+ pd->stats.dequeue, cb, dname);
#endif
for (sub_type = BLKIO_STAT_READ; sub_type < BLKIO_STAT_TOTAL;
sub_type++) {
blkio_get_key_name(sub_type, dname, key_str, MAX_KEY_LEN,
false);
- cb->fill(cb, key_str, blkg->stats.stat_arr[type][sub_type]);
+ cb->fill(cb, key_str, pd->stats.stat_arr[type][sub_type]);
}
- disk_total = blkg->stats.stat_arr[type][BLKIO_STAT_READ] +
- blkg->stats.stat_arr[type][BLKIO_STAT_WRITE];
+ disk_total = pd->stats.stat_arr[type][BLKIO_STAT_READ] +
+ pd->stats.stat_arr[type][BLKIO_STAT_WRITE];
blkio_get_key_name(BLKIO_STAT_TOTAL, dname, key_str, MAX_KEY_LEN,
false);
cb->fill(cb, key_str, disk_total);
@@ -891,6 +920,7 @@ static int blkio_policy_parse_and_set(char *buf, enum blkio_policy_id plid,
{
struct gendisk *disk = NULL;
struct blkio_group *blkg = NULL;
+ struct blkg_policy_data *pd;
char *s[4], *p, *major_s = NULL, *minor_s = NULL;
unsigned long major, minor;
int i = 0, ret = -EINVAL;
@@ -950,35 +980,37 @@ static int blkio_policy_parse_and_set(char *buf, enum blkio_policy_id plid,
goto out_unlock;
}
+ pd = blkg->pd[plid];
+
switch (plid) {
case BLKIO_POLICY_PROP:
if ((temp < BLKIO_WEIGHT_MIN && temp > 0) ||
temp > BLKIO_WEIGHT_MAX)
goto out_unlock;
- blkg->conf.weight = temp;
+ pd->conf.weight = temp;
blkio_update_group_weight(blkg, temp ?: blkcg->weight);
break;
case BLKIO_POLICY_THROTL:
switch(fileid) {
case BLKIO_THROTL_read_bps_device:
- blkg->conf.bps[READ] = temp;
+ pd->conf.bps[READ] = temp;
blkio_update_group_bps(blkg, temp ?: -1, fileid);
break;
case BLKIO_THROTL_write_bps_device:
- blkg->conf.bps[WRITE] = temp;
+ pd->conf.bps[WRITE] = temp;
blkio_update_group_bps(blkg, temp ?: -1, fileid);
break;
case BLKIO_THROTL_read_iops_device:
if (temp > THROTL_IOPS_MAX)
goto out_unlock;
- blkg->conf.iops[READ] = temp;
+ pd->conf.iops[READ] = temp;
blkio_update_group_iops(blkg, temp ?: -1, fileid);
break;
case BLKIO_THROTL_write_iops_device:
if (temp > THROTL_IOPS_MAX)
goto out_unlock;
- blkg->conf.iops[WRITE] = temp;
+ pd->conf.iops[WRITE] = temp;
blkio_update_group_iops(blkg, temp ?: -1, fileid);
break;
}
@@ -1026,31 +1058,32 @@ static int blkiocg_file_write(struct cgroup *cgrp, struct cftype *cft,
static void blkio_print_group_conf(struct cftype *cft, struct blkio_group *blkg,
struct seq_file *m)
{
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
const char *dname = dev_name(blkg->q->backing_dev_info.dev);
int fileid = BLKIOFILE_ATTR(cft->private);
int rw = WRITE;
switch (blkg->plid) {
case BLKIO_POLICY_PROP:
- if (blkg->conf.weight)
+ if (pd->conf.weight)
seq_printf(m, "%s\t%u\n",
- dname, blkg->conf.weight);
+ dname, pd->conf.weight);
break;
case BLKIO_POLICY_THROTL:
switch (fileid) {
case BLKIO_THROTL_read_bps_device:
rw = READ;
case BLKIO_THROTL_write_bps_device:
- if (blkg->conf.bps[rw])
+ if (pd->conf.bps[rw])
seq_printf(m, "%s\t%llu\n",
- dname, blkg->conf.bps[rw]);
+ dname, pd->conf.bps[rw]);
break;
case BLKIO_THROTL_read_iops_device:
rw = READ;
case BLKIO_THROTL_write_iops_device:
- if (blkg->conf.iops[rw])
+ if (pd->conf.iops[rw])
seq_printf(m, "%s\t%u\n",
- dname, blkg->conf.iops[rw]);
+ dname, pd->conf.iops[rw]);
break;
}
break;
@@ -1232,9 +1265,12 @@ static int blkio_weight_write(struct blkio_cgroup *blkcg, int plid, u64 val)
spin_lock_irq(&blkcg->lock);
blkcg->weight = (unsigned int)val;
- hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node)
- if (blkg->plid == plid && !blkg->conf.weight)
+ hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node) {
+ struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+
+ if (blkg->plid == plid && !pd->conf.weight)
blkio_update_group_weight(blkg, blkcg->weight);
+ }
spin_unlock_irq(&blkcg->lock);
spin_unlock(&blkio_list_lock);
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 7da1068..5dffd43 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -164,6 +164,13 @@ struct blkg_policy_data {
/* the blkg this per-policy data belongs to */
struct blkio_group *blkg;
+ /* Configuration */
+ struct blkio_group_conf conf;
+
+ struct blkio_group_stats stats;
+ /* Per cpu stats pointer */
+ struct blkio_group_stats_cpu __percpu *stats_cpu;
+
/* pol->pdata_size bytes of private data used by policy impl */
char pdata[] __aligned(__alignof__(unsigned long long));
};
@@ -180,16 +187,9 @@ struct blkio_group {
/* reference count */
int refcnt;
- /* Configuration */
- struct blkio_group_conf conf;
-
/* Need to serialize the stats in the case of reset/update */
spinlock_t stats_lock;
- struct blkio_group_stats stats;
- /* Per cpu stats pointer */
- struct blkio_group_stats_cpu __percpu *stats_cpu;
-
- struct blkg_policy_data *pd;
+ struct blkg_policy_data *pd[BLKIO_NR_POLICIES];
struct rcu_head rcu_head;
};
@@ -249,7 +249,7 @@ extern void blkg_destroy_all(struct request_queue *q);
static inline void *blkg_to_pdata(struct blkio_group *blkg,
struct blkio_policy_type *pol)
{
- return blkg ? blkg->pd->pdata : NULL;
+ return blkg ? blkg->pd[pol->plid]->pdata : NULL;
}
/**
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH 08/11] blkcg: don't use blkg->plid in stat related functions
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
` (6 preceding siblings ...)
2012-02-01 21:19 ` [PATCH 07/11] blkcg: make blkg->pd an array and move configuration and stats into it Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-01 21:19 ` [PATCH 09/11] blkcg: move per-queue blkg list heads and counters to queue and blkg Tejun Heo
` (3 subsequent siblings)
11 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
blkg is scheduled to be unified for all policies and thus there won't
be one-to-one mapping from blkg to policy. Update stat related
functions to take explicit @pol or @plid arguments and not use
blkg->plid.
This is painful for now but most of specific stat interface functions
will be replaced with a handful of generic helpers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-cgroup.c | 150 ++++++++++++++++++++++++++++----------------------
block/blk-cgroup.h | 80 +++++++++++++++++----------
block/blk-throttle.c | 4 +-
block/cfq-iosched.c | 44 +++++++++-----
block/cfq.h | 96 +++++++++++++++++++-------------
5 files changed, 224 insertions(+), 150 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 5696a8d..c121189 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -78,14 +78,14 @@ struct blkio_cgroup *task_blkio_cgroup(struct task_struct *tsk)
}
EXPORT_SYMBOL_GPL(task_blkio_cgroup);
-static inline void
-blkio_update_group_weight(struct blkio_group *blkg, unsigned int weight)
+static inline void blkio_update_group_weight(struct blkio_group *blkg,
+ int plid, unsigned int weight)
{
struct blkio_policy_type *blkiop;
list_for_each_entry(blkiop, &blkio_list, list) {
/* If this policy does not own the blkg, do not send updates */
- if (blkiop->plid != blkg->plid)
+ if (blkiop->plid != plid)
continue;
if (blkiop->ops.blkio_update_group_weight_fn)
blkiop->ops.blkio_update_group_weight_fn(blkg->q,
@@ -93,15 +93,15 @@ blkio_update_group_weight(struct blkio_group *blkg, unsigned int weight)
}
}
-static inline void blkio_update_group_bps(struct blkio_group *blkg, u64 bps,
- int fileid)
+static inline void blkio_update_group_bps(struct blkio_group *blkg, int plid,
+ u64 bps, int fileid)
{
struct blkio_policy_type *blkiop;
list_for_each_entry(blkiop, &blkio_list, list) {
/* If this policy does not own the blkg, do not send updates */
- if (blkiop->plid != blkg->plid)
+ if (blkiop->plid != plid)
continue;
if (fileid == BLKIO_THROTL_read_bps_device
@@ -117,14 +117,15 @@ static inline void blkio_update_group_bps(struct blkio_group *blkg, u64 bps,
}
static inline void blkio_update_group_iops(struct blkio_group *blkg,
- unsigned int iops, int fileid)
+ int plid, unsigned int iops,
+ int fileid)
{
struct blkio_policy_type *blkiop;
list_for_each_entry(blkiop, &blkio_list, list) {
/* If this policy does not own the blkg, do not send updates */
- if (blkiop->plid != blkg->plid)
+ if (blkiop->plid != plid)
continue;
if (fileid == BLKIO_THROTL_read_iops_device
@@ -182,9 +183,10 @@ static void blkio_check_and_dec_stat(uint64_t *stat, bool direction, bool sync)
#ifdef CONFIG_DEBUG_BLK_CGROUP
/* This should be called with the blkg->stats_lock held. */
static void blkio_set_start_group_wait_time(struct blkio_group *blkg,
- struct blkio_group *curr_blkg)
+ struct blkio_policy_type *pol,
+ struct blkio_group *curr_blkg)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
if (blkio_blkg_waiting(&pd->stats))
return;
@@ -222,9 +224,10 @@ static void blkio_end_empty_time(struct blkio_group_stats *stats)
blkio_clear_blkg_empty(stats);
}
-void blkiocg_update_set_idle_time_stats(struct blkio_group *blkg)
+void blkiocg_update_set_idle_time_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
unsigned long flags;
spin_lock_irqsave(&blkg->stats_lock, flags);
@@ -235,9 +238,10 @@ void blkiocg_update_set_idle_time_stats(struct blkio_group *blkg)
}
EXPORT_SYMBOL_GPL(blkiocg_update_set_idle_time_stats);
-void blkiocg_update_idle_time_stats(struct blkio_group *blkg)
+void blkiocg_update_idle_time_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
unsigned long flags;
unsigned long long now;
struct blkio_group_stats *stats;
@@ -254,9 +258,10 @@ void blkiocg_update_idle_time_stats(struct blkio_group *blkg)
}
EXPORT_SYMBOL_GPL(blkiocg_update_idle_time_stats);
-void blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg)
+void blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
unsigned long flags;
struct blkio_group_stats *stats;
@@ -271,9 +276,10 @@ void blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg)
}
EXPORT_SYMBOL_GPL(blkiocg_update_avg_queue_size_stats);
-void blkiocg_set_start_empty_time(struct blkio_group *blkg)
+void blkiocg_set_start_empty_time(struct blkio_group *blkg,
+ struct blkio_policy_type *pol)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
unsigned long flags;
struct blkio_group_stats *stats;
@@ -303,39 +309,43 @@ void blkiocg_set_start_empty_time(struct blkio_group *blkg)
EXPORT_SYMBOL_GPL(blkiocg_set_start_empty_time);
void blkiocg_update_dequeue_stats(struct blkio_group *blkg,
- unsigned long dequeue)
+ struct blkio_policy_type *pol,
+ unsigned long dequeue)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
pd->stats.dequeue += dequeue;
}
EXPORT_SYMBOL_GPL(blkiocg_update_dequeue_stats);
#else
static inline void blkio_set_start_group_wait_time(struct blkio_group *blkg,
- struct blkio_group *curr_blkg) {}
-static inline void blkio_end_empty_time(struct blkio_group_stats *stats) {}
+ struct blkio_policy_type *pol,
+ struct blkio_group *curr_blkg) { }
+static inline void blkio_end_empty_time(struct blkio_group_stats *stats) { }
#endif
void blkiocg_update_io_add_stats(struct blkio_group *blkg,
- struct blkio_group *curr_blkg, bool direction,
- bool sync)
+ struct blkio_policy_type *pol,
+ struct blkio_group *curr_blkg, bool direction,
+ bool sync)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
unsigned long flags;
spin_lock_irqsave(&blkg->stats_lock, flags);
blkio_add_stat(pd->stats.stat_arr[BLKIO_STAT_QUEUED], 1, direction,
sync);
blkio_end_empty_time(&pd->stats);
- blkio_set_start_group_wait_time(blkg, curr_blkg);
+ blkio_set_start_group_wait_time(blkg, pol, curr_blkg);
spin_unlock_irqrestore(&blkg->stats_lock, flags);
}
EXPORT_SYMBOL_GPL(blkiocg_update_io_add_stats);
void blkiocg_update_io_remove_stats(struct blkio_group *blkg,
- bool direction, bool sync)
+ struct blkio_policy_type *pol,
+ bool direction, bool sync)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
unsigned long flags;
spin_lock_irqsave(&blkg->stats_lock, flags);
@@ -345,10 +355,12 @@ void blkiocg_update_io_remove_stats(struct blkio_group *blkg,
}
EXPORT_SYMBOL_GPL(blkiocg_update_io_remove_stats);
-void blkiocg_update_timeslice_used(struct blkio_group *blkg, unsigned long time,
- unsigned long unaccounted_time)
+void blkiocg_update_timeslice_used(struct blkio_group *blkg,
+ struct blkio_policy_type *pol,
+ unsigned long time,
+ unsigned long unaccounted_time)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
unsigned long flags;
spin_lock_irqsave(&blkg->stats_lock, flags);
@@ -365,9 +377,10 @@ EXPORT_SYMBOL_GPL(blkiocg_update_timeslice_used);
* is valid.
*/
void blkiocg_update_dispatch_stats(struct blkio_group *blkg,
- uint64_t bytes, bool direction, bool sync)
+ struct blkio_policy_type *pol,
+ uint64_t bytes, bool direction, bool sync)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
struct blkio_group_stats_cpu *stats_cpu;
unsigned long flags;
@@ -392,9 +405,12 @@ void blkiocg_update_dispatch_stats(struct blkio_group *blkg,
EXPORT_SYMBOL_GPL(blkiocg_update_dispatch_stats);
void blkiocg_update_completion_stats(struct blkio_group *blkg,
- uint64_t start_time, uint64_t io_start_time, bool direction, bool sync)
+ struct blkio_policy_type *pol,
+ uint64_t start_time,
+ uint64_t io_start_time, bool direction,
+ bool sync)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
struct blkio_group_stats *stats;
unsigned long flags;
unsigned long long now = sched_clock();
@@ -412,10 +428,11 @@ void blkiocg_update_completion_stats(struct blkio_group *blkg,
EXPORT_SYMBOL_GPL(blkiocg_update_completion_stats);
/* Merged stats are per cpu. */
-void blkiocg_update_io_merged_stats(struct blkio_group *blkg, bool direction,
- bool sync)
+void blkiocg_update_io_merged_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol,
+ bool direction, bool sync)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
struct blkio_group_stats_cpu *stats_cpu;
unsigned long flags;
@@ -681,9 +698,9 @@ void __blkg_release(struct blkio_group *blkg)
}
EXPORT_SYMBOL_GPL(__blkg_release);
-static void blkio_reset_stats_cpu(struct blkio_group *blkg)
+static void blkio_reset_stats_cpu(struct blkio_group *blkg, int plid)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[plid];
struct blkio_group_stats_cpu *stats_cpu;
int i, j, k;
/*
@@ -754,7 +771,7 @@ blkiocg_reset_stats(struct cgroup *cgroup, struct cftype *cftype, u64 val)
spin_unlock(&blkg->stats_lock);
/* Reset Per cpu stats which don't take blkg->stats_lock */
- blkio_reset_stats_cpu(blkg);
+ blkio_reset_stats_cpu(blkg, blkg->plid);
}
spin_unlock_irq(&blkcg->lock);
@@ -803,10 +820,10 @@ static uint64_t blkio_fill_stat(char *str, int chars_left, uint64_t val,
}
-static uint64_t blkio_read_stat_cpu(struct blkio_group *blkg,
+static uint64_t blkio_read_stat_cpu(struct blkio_group *blkg, int plid,
enum stat_type_cpu type, enum stat_sub_type sub_type)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[plid];
int cpu;
struct blkio_group_stats_cpu *stats_cpu;
u64 val = 0, tval;
@@ -829,7 +846,7 @@ static uint64_t blkio_read_stat_cpu(struct blkio_group *blkg,
return val;
}
-static uint64_t blkio_get_stat_cpu(struct blkio_group *blkg,
+static uint64_t blkio_get_stat_cpu(struct blkio_group *blkg, int plid,
struct cgroup_map_cb *cb, const char *dname,
enum stat_type_cpu type)
{
@@ -838,7 +855,7 @@ static uint64_t blkio_get_stat_cpu(struct blkio_group *blkg,
enum stat_sub_type sub_type;
if (type == BLKIO_STAT_CPU_SECTORS) {
- val = blkio_read_stat_cpu(blkg, type, 0);
+ val = blkio_read_stat_cpu(blkg, plid, type, 0);
return blkio_fill_stat(key_str, MAX_KEY_LEN - 1, val, cb,
dname);
}
@@ -847,12 +864,12 @@ static uint64_t blkio_get_stat_cpu(struct blkio_group *blkg,
sub_type++) {
blkio_get_key_name(sub_type, dname, key_str, MAX_KEY_LEN,
false);
- val = blkio_read_stat_cpu(blkg, type, sub_type);
+ val = blkio_read_stat_cpu(blkg, plid, type, sub_type);
cb->fill(cb, key_str, val);
}
- disk_total = blkio_read_stat_cpu(blkg, type, BLKIO_STAT_READ) +
- blkio_read_stat_cpu(blkg, type, BLKIO_STAT_WRITE);
+ disk_total = blkio_read_stat_cpu(blkg, plid, type, BLKIO_STAT_READ) +
+ blkio_read_stat_cpu(blkg, plid, type, BLKIO_STAT_WRITE);
blkio_get_key_name(BLKIO_STAT_TOTAL, dname, key_str, MAX_KEY_LEN,
false);
@@ -861,11 +878,11 @@ static uint64_t blkio_get_stat_cpu(struct blkio_group *blkg,
}
/* This should be called with blkg->stats_lock held */
-static uint64_t blkio_get_stat(struct blkio_group *blkg,
+static uint64_t blkio_get_stat(struct blkio_group *blkg, int plid,
struct cgroup_map_cb *cb, const char *dname,
enum stat_type type)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[plid];
uint64_t disk_total;
char key_str[MAX_KEY_LEN];
enum stat_sub_type sub_type;
@@ -989,29 +1006,29 @@ static int blkio_policy_parse_and_set(char *buf, enum blkio_policy_id plid,
goto out_unlock;
pd->conf.weight = temp;
- blkio_update_group_weight(blkg, temp ?: blkcg->weight);
+ blkio_update_group_weight(blkg, plid, temp ?: blkcg->weight);
break;
case BLKIO_POLICY_THROTL:
switch(fileid) {
case BLKIO_THROTL_read_bps_device:
pd->conf.bps[READ] = temp;
- blkio_update_group_bps(blkg, temp ?: -1, fileid);
+ blkio_update_group_bps(blkg, plid, temp ?: -1, fileid);
break;
case BLKIO_THROTL_write_bps_device:
pd->conf.bps[WRITE] = temp;
- blkio_update_group_bps(blkg, temp ?: -1, fileid);
+ blkio_update_group_bps(blkg, plid, temp ?: -1, fileid);
break;
case BLKIO_THROTL_read_iops_device:
if (temp > THROTL_IOPS_MAX)
goto out_unlock;
pd->conf.iops[READ] = temp;
- blkio_update_group_iops(blkg, temp ?: -1, fileid);
+ blkio_update_group_iops(blkg, plid, temp ?: -1, fileid);
break;
case BLKIO_THROTL_write_iops_device:
if (temp > THROTL_IOPS_MAX)
goto out_unlock;
pd->conf.iops[WRITE] = temp;
- blkio_update_group_iops(blkg, temp ?: -1, fileid);
+ blkio_update_group_iops(blkg, plid, temp ?: -1, fileid);
break;
}
break;
@@ -1058,12 +1075,13 @@ static int blkiocg_file_write(struct cgroup *cgrp, struct cftype *cft,
static void blkio_print_group_conf(struct cftype *cft, struct blkio_group *blkg,
struct seq_file *m)
{
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
- const char *dname = dev_name(blkg->q->backing_dev_info.dev);
+ int plid = BLKIOFILE_POLICY(cft->private);
int fileid = BLKIOFILE_ATTR(cft->private);
+ struct blkg_policy_data *pd = blkg->pd[plid];
+ const char *dname = dev_name(blkg->q->backing_dev_info.dev);
int rw = WRITE;
- switch (blkg->plid) {
+ switch (plid) {
case BLKIO_POLICY_PROP:
if (pd->conf.weight)
seq_printf(m, "%s\t%u\n",
@@ -1155,15 +1173,17 @@ static int blkio_read_blkg_stats(struct blkio_cgroup *blkcg,
rcu_read_lock();
hlist_for_each_entry_rcu(blkg, n, &blkcg->blkg_list, blkcg_node) {
const char *dname = dev_name(blkg->q->backing_dev_info.dev);
+ int plid = BLKIOFILE_POLICY(cft->private);
- if (BLKIOFILE_POLICY(cft->private) != blkg->plid)
+ if (plid != blkg->plid)
continue;
- if (pcpu)
- cgroup_total += blkio_get_stat_cpu(blkg, cb, dname,
- type);
- else {
+ if (pcpu) {
+ cgroup_total += blkio_get_stat_cpu(blkg, plid,
+ cb, dname, type);
+ } else {
spin_lock_irq(&blkg->stats_lock);
- cgroup_total += blkio_get_stat(blkg, cb, dname, type);
+ cgroup_total += blkio_get_stat(blkg, plid,
+ cb, dname, type);
spin_unlock_irq(&blkg->stats_lock);
}
}
@@ -1269,7 +1289,7 @@ static int blkio_weight_write(struct blkio_cgroup *blkcg, int plid, u64 val)
struct blkg_policy_data *pd = blkg->pd[blkg->plid];
if (blkg->plid == plid && !pd->conf.weight)
- blkio_update_group_weight(blkg, blkcg->weight);
+ blkio_update_group_weight(blkg, plid, blkcg->weight);
}
spin_unlock_irq(&blkcg->lock);
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 5dffd43..60e96b4 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -335,12 +335,17 @@ static inline void blkg_put(struct blkio_group *blkg) { }
#define BLKIO_WEIGHT_DEFAULT 500
#ifdef CONFIG_DEBUG_BLK_CGROUP
-void blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg);
+void blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol);
void blkiocg_update_dequeue_stats(struct blkio_group *blkg,
- unsigned long dequeue);
-void blkiocg_update_set_idle_time_stats(struct blkio_group *blkg);
-void blkiocg_update_idle_time_stats(struct blkio_group *blkg);
-void blkiocg_set_start_empty_time(struct blkio_group *blkg);
+ struct blkio_policy_type *pol,
+ unsigned long dequeue);
+void blkiocg_update_set_idle_time_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol);
+void blkiocg_update_idle_time_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol);
+void blkiocg_set_start_empty_time(struct blkio_group *blkg,
+ struct blkio_policy_type *pol);
#define BLKG_FLAG_FNS(name) \
static inline void blkio_mark_blkg_##name( \
@@ -363,14 +368,16 @@ BLKG_FLAG_FNS(idling)
BLKG_FLAG_FNS(empty)
#undef BLKG_FLAG_FNS
#else
-static inline void blkiocg_update_avg_queue_size_stats(
- struct blkio_group *blkg) {}
+static inline void blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol) { }
static inline void blkiocg_update_dequeue_stats(struct blkio_group *blkg,
- unsigned long dequeue) {}
-static inline void blkiocg_update_set_idle_time_stats(struct blkio_group *blkg)
-{}
-static inline void blkiocg_update_idle_time_stats(struct blkio_group *blkg) {}
-static inline void blkiocg_set_start_empty_time(struct blkio_group *blkg) {}
+ struct blkio_policy_type *pol, unsigned long dequeue) { }
+static inline void blkiocg_update_set_idle_time_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol) { }
+static inline void blkiocg_update_idle_time_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol) { }
+static inline void blkiocg_set_start_empty_time(struct blkio_group *blkg,
+ struct blkio_policy_type *pol) { }
#endif
#ifdef CONFIG_BLK_CGROUP
@@ -386,18 +393,27 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
enum blkio_policy_id plid,
bool for_root);
void blkiocg_update_timeslice_used(struct blkio_group *blkg,
- unsigned long time,
- unsigned long unaccounted_time);
-void blkiocg_update_dispatch_stats(struct blkio_group *blkg, uint64_t bytes,
- bool direction, bool sync);
+ struct blkio_policy_type *pol,
+ unsigned long time,
+ unsigned long unaccounted_time);
+void blkiocg_update_dispatch_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol,
+ uint64_t bytes, bool direction, bool sync);
void blkiocg_update_completion_stats(struct blkio_group *blkg,
- uint64_t start_time, uint64_t io_start_time, bool direction, bool sync);
-void blkiocg_update_io_merged_stats(struct blkio_group *blkg, bool direction,
- bool sync);
+ struct blkio_policy_type *pol,
+ uint64_t start_time,
+ uint64_t io_start_time, bool direction,
+ bool sync);
+void blkiocg_update_io_merged_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol,
+ bool direction, bool sync);
void blkiocg_update_io_add_stats(struct blkio_group *blkg,
- struct blkio_group *curr_blkg, bool direction, bool sync);
+ struct blkio_policy_type *pol,
+ struct blkio_group *curr_blkg, bool direction,
+ bool sync);
void blkiocg_update_io_remove_stats(struct blkio_group *blkg,
- bool direction, bool sync);
+ struct blkio_policy_type *pol,
+ bool direction, bool sync);
#else
struct cgroup;
static inline struct blkio_cgroup *
@@ -411,19 +427,23 @@ blkiocg_del_blkio_group(struct blkio_group *blkg) { return 0; }
static inline struct blkio_group *blkg_lookup(struct blkio_cgroup *blkcg,
void *key) { return NULL; }
static inline void blkiocg_update_timeslice_used(struct blkio_group *blkg,
- unsigned long time,
- unsigned long unaccounted_time)
-{}
+ struct blkio_policy_type *pol, unsigned long time,
+ unsigned long unaccounted_time) { }
static inline void blkiocg_update_dispatch_stats(struct blkio_group *blkg,
- uint64_t bytes, bool direction, bool sync) {}
+ struct blkio_policy_type *pol, uint64_t bytes,
+ bool direction, bool sync) { }
static inline void blkiocg_update_completion_stats(struct blkio_group *blkg,
- uint64_t start_time, uint64_t io_start_time, bool direction,
- bool sync) {}
+ struct blkio_policy_type *pol, uint64_t start_time,
+ uint64_t io_start_time, bool direction, bool sync) { }
static inline void blkiocg_update_io_merged_stats(struct blkio_group *blkg,
- bool direction, bool sync) {}
+ struct blkio_policy_type *pol, bool direction,
+ bool sync) { }
static inline void blkiocg_update_io_add_stats(struct blkio_group *blkg,
- struct blkio_group *curr_blkg, bool direction, bool sync) {}
+ struct blkio_policy_type *pol,
+ struct blkio_group *curr_blkg, bool direction,
+ bool sync) { }
static inline void blkiocg_update_io_remove_stats(struct blkio_group *blkg,
- bool direction, bool sync) {}
+ struct blkio_policy_type *pol, bool direction,
+ bool sync) { }
#endif
#endif /* _BLK_CGROUP_H */
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 153ba50..b2fddaf 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -588,7 +588,8 @@ static void throtl_charge_bio(struct throtl_grp *tg, struct bio *bio)
tg->bytes_disp[rw] += bio->bi_size;
tg->io_disp[rw]++;
- blkiocg_update_dispatch_stats(tg_to_blkg(tg), bio->bi_size, rw, sync);
+ blkiocg_update_dispatch_stats(tg_to_blkg(tg), &blkio_policy_throtl,
+ bio->bi_size, rw, sync);
}
static void throtl_add_bio_tg(struct throtl_data *td, struct throtl_grp *tg,
@@ -1000,6 +1001,7 @@ bool blk_throtl_bio(struct request_queue *q, struct bio *bio)
if (tg) {
if (tg_no_rule_group(tg, rw)) {
blkiocg_update_dispatch_stats(tg_to_blkg(tg),
+ &blkio_policy_throtl,
bio->bi_size, rw,
rw_is_sync(bio->bi_rw));
goto out_unlock_rcu;
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 5ce81a8..0cb3908 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -945,7 +945,8 @@ cfq_group_notify_queue_del(struct cfq_data *cfqd, struct cfq_group *cfqg)
cfq_log_cfqg(cfqd, cfqg, "del_from_rr group");
cfq_group_service_tree_del(st, cfqg);
cfqg->saved_workload_slice = 0;
- cfq_blkiocg_update_dequeue_stats(cfqg_to_blkg(cfqg), 1);
+ cfq_blkiocg_update_dequeue_stats(cfqg_to_blkg(cfqg),
+ &blkio_policy_cfq, 1);
}
static inline unsigned int cfq_cfqq_slice_usage(struct cfq_queue *cfqq,
@@ -1017,9 +1018,9 @@ static void cfq_group_served(struct cfq_data *cfqd, struct cfq_group *cfqg,
"sl_used=%u disp=%u charge=%u iops=%u sect=%lu",
used_sl, cfqq->slice_dispatch, charge,
iops_mode(cfqd), cfqq->nr_sectors);
- cfq_blkiocg_update_timeslice_used(cfqg_to_blkg(cfqg), used_sl,
- unaccounted_sl);
- cfq_blkiocg_set_start_empty_time(cfqg_to_blkg(cfqg));
+ cfq_blkiocg_update_timeslice_used(cfqg_to_blkg(cfqg), &blkio_policy_cfq,
+ used_sl, unaccounted_sl);
+ cfq_blkiocg_set_start_empty_time(cfqg_to_blkg(cfqg), &blkio_policy_cfq);
}
/**
@@ -1463,9 +1464,11 @@ static void cfq_reposition_rq_rb(struct cfq_queue *cfqq, struct request *rq)
elv_rb_del(&cfqq->sort_list, rq);
cfqq->queued[rq_is_sync(rq)]--;
cfq_blkiocg_update_io_remove_stats(cfqg_to_blkg(RQ_CFQG(rq)),
- rq_data_dir(rq), rq_is_sync(rq));
+ &blkio_policy_cfq, rq_data_dir(rq),
+ rq_is_sync(rq));
cfq_add_rq_rb(rq);
cfq_blkiocg_update_io_add_stats(cfqg_to_blkg(RQ_CFQG(rq)),
+ &blkio_policy_cfq,
cfqg_to_blkg(cfqq->cfqd->serving_group),
rq_data_dir(rq), rq_is_sync(rq));
}
@@ -1524,7 +1527,8 @@ static void cfq_remove_request(struct request *rq)
cfqq->cfqd->rq_queued--;
cfq_blkiocg_update_io_remove_stats(cfqg_to_blkg(RQ_CFQG(rq)),
- rq_data_dir(rq), rq_is_sync(rq));
+ &blkio_policy_cfq, rq_data_dir(rq),
+ rq_is_sync(rq));
if (rq->cmd_flags & REQ_PRIO) {
WARN_ON(!cfqq->prio_pending);
cfqq->prio_pending--;
@@ -1560,7 +1564,8 @@ static void cfq_bio_merged(struct request_queue *q, struct request *req,
struct bio *bio)
{
cfq_blkiocg_update_io_merged_stats(cfqg_to_blkg(RQ_CFQG(req)),
- bio_data_dir(bio), cfq_bio_sync(bio));
+ &blkio_policy_cfq, bio_data_dir(bio),
+ cfq_bio_sync(bio));
}
static void
@@ -1583,7 +1588,8 @@ cfq_merged_requests(struct request_queue *q, struct request *rq,
cfqq->next_rq = rq;
cfq_remove_request(next);
cfq_blkiocg_update_io_merged_stats(cfqg_to_blkg(RQ_CFQG(rq)),
- rq_data_dir(next), rq_is_sync(next));
+ &blkio_policy_cfq, rq_data_dir(next),
+ rq_is_sync(next));
cfqq = RQ_CFQQ(next);
/*
@@ -1631,7 +1637,8 @@ static int cfq_allow_merge(struct request_queue *q, struct request *rq,
static inline void cfq_del_timer(struct cfq_data *cfqd, struct cfq_queue *cfqq)
{
del_timer(&cfqd->idle_slice_timer);
- cfq_blkiocg_update_idle_time_stats(cfqg_to_blkg(cfqq->cfqg));
+ cfq_blkiocg_update_idle_time_stats(cfqg_to_blkg(cfqq->cfqg),
+ &blkio_policy_cfq);
}
static void __cfq_set_active_queue(struct cfq_data *cfqd,
@@ -1640,7 +1647,8 @@ static void __cfq_set_active_queue(struct cfq_data *cfqd,
if (cfqq) {
cfq_log_cfqq(cfqd, cfqq, "set_active wl_prio:%d wl_type:%d",
cfqd->serving_prio, cfqd->serving_type);
- cfq_blkiocg_update_avg_queue_size_stats(cfqg_to_blkg(cfqq->cfqg));
+ cfq_blkiocg_update_avg_queue_size_stats(cfqg_to_blkg(cfqq->cfqg),
+ &blkio_policy_cfq);
cfqq->slice_start = 0;
cfqq->dispatch_start = jiffies;
cfqq->allocated_slice = 0;
@@ -1988,7 +1996,8 @@ static void cfq_arm_slice_timer(struct cfq_data *cfqd)
sl = cfqd->cfq_slice_idle;
mod_timer(&cfqd->idle_slice_timer, jiffies + sl);
- cfq_blkiocg_update_set_idle_time_stats(cfqg_to_blkg(cfqq->cfqg));
+ cfq_blkiocg_update_set_idle_time_stats(cfqg_to_blkg(cfqq->cfqg),
+ &blkio_policy_cfq);
cfq_log_cfqq(cfqd, cfqq, "arm_idle: %lu group_idle: %d", sl,
group_idle ? 1 : 0);
}
@@ -2012,8 +2021,8 @@ static void cfq_dispatch_insert(struct request_queue *q, struct request *rq)
cfqd->rq_in_flight[cfq_cfqq_sync(cfqq)]++;
cfqq->nr_sectors += blk_rq_sectors(rq);
cfq_blkiocg_update_dispatch_stats(cfqg_to_blkg(cfqq->cfqg),
- blk_rq_bytes(rq), rq_data_dir(rq),
- rq_is_sync(rq));
+ &blkio_policy_cfq, blk_rq_bytes(rq),
+ rq_data_dir(rq), rq_is_sync(rq));
}
/*
@@ -3100,7 +3109,8 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq,
__blk_run_queue(cfqd->queue);
} else {
cfq_blkiocg_update_idle_time_stats(
- cfqg_to_blkg(cfqq->cfqg));
+ cfqg_to_blkg(cfqq->cfqg),
+ &blkio_policy_cfq);
cfq_mark_cfqq_must_dispatch(cfqq);
}
}
@@ -3128,6 +3138,7 @@ static void cfq_insert_request(struct request_queue *q, struct request *rq)
list_add_tail(&rq->queuelist, &cfqq->fifo);
cfq_add_rq_rb(rq);
cfq_blkiocg_update_io_add_stats(cfqg_to_blkg(RQ_CFQG(rq)),
+ &blkio_policy_cfq,
cfqg_to_blkg(cfqd->serving_group),
rq_data_dir(rq), rq_is_sync(rq));
cfq_rq_enqueued(cfqd, cfqq, rq);
@@ -3226,8 +3237,9 @@ static void cfq_completed_request(struct request_queue *q, struct request *rq)
cfqq->dispatched--;
(RQ_CFQG(rq))->dispatched--;
cfq_blkiocg_update_completion_stats(cfqg_to_blkg(cfqq->cfqg),
- rq_start_time_ns(rq), rq_io_start_time_ns(rq),
- rq_data_dir(rq), rq_is_sync(rq));
+ &blkio_policy_cfq, rq_start_time_ns(rq),
+ rq_io_start_time_ns(rq), rq_data_dir(rq),
+ rq_is_sync(rq));
cfqd->rq_in_flight[cfq_cfqq_sync(cfqq)]--;
diff --git a/block/cfq.h b/block/cfq.h
index 3987601..5584e1b 100644
--- a/block/cfq.h
+++ b/block/cfq.h
@@ -4,67 +4,79 @@
#ifdef CONFIG_CFQ_GROUP_IOSCHED
static inline void cfq_blkiocg_update_io_add_stats(struct blkio_group *blkg,
- struct blkio_group *curr_blkg, bool direction, bool sync)
+ struct blkio_policy_type *pol,
+ struct blkio_group *curr_blkg,
+ bool direction, bool sync)
{
- blkiocg_update_io_add_stats(blkg, curr_blkg, direction, sync);
+ blkiocg_update_io_add_stats(blkg, pol, curr_blkg, direction, sync);
}
static inline void cfq_blkiocg_update_dequeue_stats(struct blkio_group *blkg,
- unsigned long dequeue)
+ struct blkio_policy_type *pol, unsigned long dequeue)
{
- blkiocg_update_dequeue_stats(blkg, dequeue);
+ blkiocg_update_dequeue_stats(blkg, pol, dequeue);
}
static inline void cfq_blkiocg_update_timeslice_used(struct blkio_group *blkg,
- unsigned long time, unsigned long unaccounted_time)
+ struct blkio_policy_type *pol, unsigned long time,
+ unsigned long unaccounted_time)
{
- blkiocg_update_timeslice_used(blkg, time, unaccounted_time);
+ blkiocg_update_timeslice_used(blkg, pol, time, unaccounted_time);
}
-static inline void cfq_blkiocg_set_start_empty_time(struct blkio_group *blkg)
+static inline void cfq_blkiocg_set_start_empty_time(struct blkio_group *blkg,
+ struct blkio_policy_type *pol)
{
- blkiocg_set_start_empty_time(blkg);
+ blkiocg_set_start_empty_time(blkg, pol);
}
static inline void cfq_blkiocg_update_io_remove_stats(struct blkio_group *blkg,
- bool direction, bool sync)
+ struct blkio_policy_type *pol, bool direction,
+ bool sync)
{
- blkiocg_update_io_remove_stats(blkg, direction, sync);
+ blkiocg_update_io_remove_stats(blkg, pol, direction, sync);
}
static inline void cfq_blkiocg_update_io_merged_stats(struct blkio_group *blkg,
- bool direction, bool sync)
+ struct blkio_policy_type *pol, bool direction,
+ bool sync)
{
- blkiocg_update_io_merged_stats(blkg, direction, sync);
+ blkiocg_update_io_merged_stats(blkg, pol, direction, sync);
}
-static inline void cfq_blkiocg_update_idle_time_stats(struct blkio_group *blkg)
+static inline void cfq_blkiocg_update_idle_time_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol)
{
- blkiocg_update_idle_time_stats(blkg);
+ blkiocg_update_idle_time_stats(blkg, pol);
}
static inline void
-cfq_blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg)
+cfq_blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol)
{
- blkiocg_update_avg_queue_size_stats(blkg);
+ blkiocg_update_avg_queue_size_stats(blkg, pol);
}
static inline void
-cfq_blkiocg_update_set_idle_time_stats(struct blkio_group *blkg)
+cfq_blkiocg_update_set_idle_time_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol)
{
- blkiocg_update_set_idle_time_stats(blkg);
+ blkiocg_update_set_idle_time_stats(blkg, pol);
}
static inline void cfq_blkiocg_update_dispatch_stats(struct blkio_group *blkg,
- uint64_t bytes, bool direction, bool sync)
+ struct blkio_policy_type *pol, uint64_t bytes,
+ bool direction, bool sync)
{
- blkiocg_update_dispatch_stats(blkg, bytes, direction, sync);
+ blkiocg_update_dispatch_stats(blkg, pol, bytes, direction, sync);
}
-static inline void cfq_blkiocg_update_completion_stats(struct blkio_group *blkg, uint64_t start_time, uint64_t io_start_time, bool direction, bool sync)
+static inline void cfq_blkiocg_update_completion_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol, uint64_t start_time,
+ uint64_t io_start_time, bool direction, bool sync)
{
- blkiocg_update_completion_stats(blkg, start_time, io_start_time,
- direction, sync);
+ blkiocg_update_completion_stats(blkg, pol, start_time, io_start_time,
+ direction, sync);
}
static inline int cfq_blkiocg_del_blkio_group(struct blkio_group *blkg)
@@ -74,30 +86,38 @@ static inline int cfq_blkiocg_del_blkio_group(struct blkio_group *blkg)
#else /* CFQ_GROUP_IOSCHED */
static inline void cfq_blkiocg_update_io_add_stats(struct blkio_group *blkg,
- struct blkio_group *curr_blkg, bool direction, bool sync) {}
-
+ struct blkio_policy_type *pol,
+ struct blkio_group *curr_blkg, bool direction,
+ bool sync) { }
static inline void cfq_blkiocg_update_dequeue_stats(struct blkio_group *blkg,
- unsigned long dequeue) {}
-
+ struct blkio_policy_type *pol, unsigned long dequeue) { }
static inline void cfq_blkiocg_update_timeslice_used(struct blkio_group *blkg,
- unsigned long time, unsigned long unaccounted_time) {}
-static inline void cfq_blkiocg_set_start_empty_time(struct blkio_group *blkg) {}
+ struct blkio_policy_type *pol, unsigned long time,
+ unsigned long unaccounted_time) { }
+static inline void cfq_blkiocg_set_start_empty_time(struct blkio_group *blkg,
+ struct blkio_policy_type *pol) { }
static inline void cfq_blkiocg_update_io_remove_stats(struct blkio_group *blkg,
- bool direction, bool sync) {}
+ struct blkio_policy_type *pol, bool direction,
+ bool sync) { }
static inline void cfq_blkiocg_update_io_merged_stats(struct blkio_group *blkg,
- bool direction, bool sync) {}
-static inline void cfq_blkiocg_update_idle_time_stats(struct blkio_group *blkg)
-{
-}
+ struct blkio_policy_type *pol, bool direction,
+ bool sync) { }
+static inline void cfq_blkiocg_update_idle_time_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol) { }
static inline void
-cfq_blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg) {}
+cfq_blkiocg_update_avg_queue_size_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol) { }
static inline void
-cfq_blkiocg_update_set_idle_time_stats(struct blkio_group *blkg) {}
+cfq_blkiocg_update_set_idle_time_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol) { }
static inline void cfq_blkiocg_update_dispatch_stats(struct blkio_group *blkg,
- uint64_t bytes, bool direction, bool sync) {}
-static inline void cfq_blkiocg_update_completion_stats(struct blkio_group *blkg, uint64_t start_time, uint64_t io_start_time, bool direction, bool sync) {}
+ struct blkio_policy_type *pol, uint64_t bytes,
+ bool direction, bool sync) { }
+static inline void cfq_blkiocg_update_completion_stats(struct blkio_group *blkg,
+ struct blkio_policy_type *pol, uint64_t start_time,
+ uint64_t io_start_time, bool direction, bool sync) { }
static inline int cfq_blkiocg_del_blkio_group(struct blkio_group *blkg)
{
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH 09/11] blkcg: move per-queue blkg list heads and counters to queue and blkg
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
` (7 preceding siblings ...)
2012-02-01 21:19 ` [PATCH 08/11] blkcg: don't use blkg->plid in stat related functions Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-02 22:47 ` Vivek Goyal
2012-02-01 21:19 ` [PATCH 10/11] blkcg: let blkcg core manage per-queue blkg list and counter Tejun Heo
` (2 subsequent siblings)
11 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
Currently, specific policy implementations are responsible for
maintaining list and number of blkgs. This duplicates code
unnecessarily, and hinders factoring common code and providing blkcg
API with better defined semantics.
After this patch, request_queue hosts list heads and counters and blkg
has list nodes for both policies. This patch only relocates the
necessary fields and the next patch will actually move management code
into blkcg core.
Note that request_queue->blkg_list[] and ->nr_blkgs[] are hardcoded to
have 2 elements. This is to avoid include dependency and will be
removed by the next patch.
This patch doesn't introduce any behavior change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-cgroup.c | 2 +
block/blk-cgroup.h | 1 +
block/blk-core.c | 4 +++
block/blk-throttle.c | 49 ++++++++++++++++++++++-------------------------
block/cfq-iosched.c | 46 +++++++++++++++++---------------------------
include/linux/blkdev.h | 5 ++++
6 files changed, 53 insertions(+), 54 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index c121189..0513ed5 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -499,6 +499,8 @@ static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
spin_lock_init(&blkg->stats_lock);
rcu_assign_pointer(blkg->q, q);
+ INIT_LIST_HEAD(&blkg->q_node[0]);
+ INIT_LIST_HEAD(&blkg->q_node[1]);
blkg->blkcg = blkcg;
blkg->plid = pol->plid;
blkg->refcnt = 1;
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 60e96b4..ae96f19 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -178,6 +178,7 @@ struct blkg_policy_data {
struct blkio_group {
/* Pointer to the associated request_queue, RCU protected */
struct request_queue __rcu *q;
+ struct list_head q_node[BLKIO_NR_POLICIES];
struct hlist_node blkcg_node;
struct blkio_cgroup *blkcg;
/* Store cgroup path */
diff --git a/block/blk-core.c b/block/blk-core.c
index ab0a11b..025ef60 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -546,6 +546,10 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id)
setup_timer(&q->timeout, blk_rq_timed_out_timer, (unsigned long) q);
INIT_LIST_HEAD(&q->timeout_list);
INIT_LIST_HEAD(&q->icq_list);
+#if defined(CONFIG_BLK_CGROUP) || defined(CONFIG_BLK_CGROUP_MODULE)
+ INIT_LIST_HEAD(&q->blkg_list[0]);
+ INIT_LIST_HEAD(&q->blkg_list[1]);
+#endif
INIT_LIST_HEAD(&q->flush_queue[0]);
INIT_LIST_HEAD(&q->flush_queue[1]);
INIT_LIST_HEAD(&q->flush_data_in_flight);
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index b2fddaf..c15d383 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -41,9 +41,6 @@ struct throtl_rb_root {
#define rb_entry_tg(node) rb_entry((node), struct throtl_grp, rb_node)
struct throtl_grp {
- /* List of throtl groups on the request queue*/
- struct hlist_node tg_node;
-
/* active throtl group service_tree member */
struct rb_node rb_node;
@@ -83,9 +80,6 @@ struct throtl_grp {
struct throtl_data
{
- /* List of throtl groups */
- struct hlist_head tg_list;
-
/* service tree for active throtl groups */
struct throtl_rb_root tg_service_tree;
@@ -152,7 +146,6 @@ static void throtl_init_blkio_group(struct blkio_group *blkg)
{
struct throtl_grp *tg = blkg_to_tg(blkg);
- INIT_HLIST_NODE(&tg->tg_node);
RB_CLEAR_NODE(&tg->rb_node);
bio_list_init(&tg->bio_lists[0]);
bio_list_init(&tg->bio_lists[1]);
@@ -167,11 +160,9 @@ static void throtl_init_blkio_group(struct blkio_group *blkg)
static void throtl_link_blkio_group(struct request_queue *q,
struct blkio_group *blkg)
{
- struct throtl_data *td = q->td;
- struct throtl_grp *tg = blkg_to_tg(blkg);
-
- hlist_add_head(&tg->tg_node, &td->tg_list);
- td->nr_undestroyed_grps++;
+ list_add(&blkg->q_node[BLKIO_POLICY_THROTL],
+ &q->blkg_list[BLKIO_POLICY_THROTL]);
+ q->nr_blkgs[BLKIO_POLICY_THROTL]++;
}
static struct
@@ -711,8 +702,8 @@ static int throtl_select_dispatch(struct throtl_data *td, struct bio_list *bl)
static void throtl_process_limit_change(struct throtl_data *td)
{
- struct throtl_grp *tg;
- struct hlist_node *pos, *n;
+ struct request_queue *q = td->queue;
+ struct blkio_group *blkg, *n;
if (!td->limits_changed)
return;
@@ -721,7 +712,10 @@ static void throtl_process_limit_change(struct throtl_data *td)
throtl_log(td, "limits changed");
- hlist_for_each_entry_safe(tg, pos, n, &td->tg_list, tg_node) {
+ list_for_each_entry_safe(blkg, n, &q->blkg_list[BLKIO_POLICY_THROTL],
+ q_node[BLKIO_POLICY_THROTL]) {
+ struct throtl_grp *tg = blkg_to_tg(blkg);
+
if (!tg->limits_changed)
continue;
@@ -822,26 +816,31 @@ throtl_schedule_delayed_work(struct throtl_data *td, unsigned long delay)
static void
throtl_destroy_tg(struct throtl_data *td, struct throtl_grp *tg)
{
+ struct blkio_group *blkg = tg_to_blkg(tg);
+
/* Something wrong if we are trying to remove same group twice */
- BUG_ON(hlist_unhashed(&tg->tg_node));
+ WARN_ON_ONCE(list_empty(&blkg->q_node[BLKIO_POLICY_THROTL]));
- hlist_del_init(&tg->tg_node);
+ list_del_init(&blkg->q_node[BLKIO_POLICY_THROTL]);
/*
* Put the reference taken at the time of creation so that when all
* queues are gone, group can be destroyed.
*/
blkg_put(tg_to_blkg(tg));
- td->nr_undestroyed_grps--;
+ td->queue->nr_blkgs[BLKIO_POLICY_THROTL]--;
}
static bool throtl_release_tgs(struct throtl_data *td, bool release_root)
{
- struct hlist_node *pos, *n;
- struct throtl_grp *tg;
+ struct request_queue *q = td->queue;
+ struct blkio_group *blkg, *n;
bool empty = true;
- hlist_for_each_entry_safe(tg, pos, n, &td->tg_list, tg_node) {
+ list_for_each_entry_safe(blkg, n, &q->blkg_list[BLKIO_POLICY_THROTL],
+ q_node[BLKIO_POLICY_THROTL]) {
+ struct throtl_grp *tg = blkg_to_tg(blkg);
+
/* skip root? */
if (!release_root && tg == td->root_tg)
continue;
@@ -851,7 +850,7 @@ static bool throtl_release_tgs(struct throtl_data *td, bool release_root)
* it from cgroup list, then it will take care of destroying
* cfqg also.
*/
- if (!blkiocg_del_blkio_group(tg_to_blkg(tg)))
+ if (!blkiocg_del_blkio_group(blkg))
throtl_destroy_tg(td, tg);
else
empty = false;
@@ -1114,7 +1113,6 @@ int blk_throtl_init(struct request_queue *q)
if (!td)
return -ENOMEM;
- INIT_HLIST_HEAD(&td->tg_list);
td->tg_service_tree = THROTL_RB_ROOT;
td->limits_changed = false;
INIT_DELAYED_WORK(&td->throtl_work, blk_throtl_work);
@@ -1144,7 +1142,7 @@ int blk_throtl_init(struct request_queue *q)
void blk_throtl_exit(struct request_queue *q)
{
struct throtl_data *td = q->td;
- bool wait = false;
+ bool wait;
BUG_ON(!td);
@@ -1154,8 +1152,7 @@ void blk_throtl_exit(struct request_queue *q)
throtl_release_tgs(td, true);
/* If there are other groups */
- if (td->nr_undestroyed_grps > 0)
- wait = true;
+ wait = q->nr_blkgs[BLKIO_POLICY_THROTL];
spin_unlock_irq(q->queue_lock);
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 0cb3908..c13e376 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -208,9 +208,7 @@ struct cfq_group {
unsigned long saved_workload_slice;
enum wl_type_t saved_workload;
enum wl_prio_t saved_serving_prio;
-#ifdef CONFIG_CFQ_GROUP_IOSCHED
- struct hlist_node cfqd_node;
-#endif
+
/* number of requests that are on the dispatch list or inside driver */
int dispatched;
struct cfq_ttime ttime;
@@ -302,12 +300,6 @@ struct cfq_data {
struct cfq_queue oom_cfqq;
unsigned long last_delayed_sync;
-
- /* List of cfq groups being managed on this device*/
- struct hlist_head cfqg_list;
-
- /* Number of groups which are on blkcg->blkg_list */
- unsigned int nr_blkcg_linked_grps;
};
static inline struct cfq_group *blkg_to_cfqg(struct blkio_group *blkg)
@@ -1056,13 +1048,9 @@ static void cfq_update_blkio_group_weight(struct request_queue *q,
static void cfq_link_blkio_group(struct request_queue *q,
struct blkio_group *blkg)
{
- struct cfq_data *cfqd = q->elevator->elevator_data;
- struct cfq_group *cfqg = blkg_to_cfqg(blkg);
-
- cfqd->nr_blkcg_linked_grps++;
-
- /* Add group on cfqd list */
- hlist_add_head(&cfqg->cfqd_node, &cfqd->cfqg_list);
+ list_add(&blkg->q_node[BLKIO_POLICY_PROP],
+ &q->blkg_list[BLKIO_POLICY_PROP]);
+ q->nr_blkgs[BLKIO_POLICY_PROP]++;
}
static void cfq_init_blkio_group(struct blkio_group *blkg)
@@ -1110,13 +1098,15 @@ static void cfq_link_cfqq_cfqg(struct cfq_queue *cfqq, struct cfq_group *cfqg)
static void cfq_destroy_cfqg(struct cfq_data *cfqd, struct cfq_group *cfqg)
{
+ struct blkio_group *blkg = cfqg_to_blkg(cfqg);
+
/* Something wrong if we are trying to remove same group twice */
- BUG_ON(hlist_unhashed(&cfqg->cfqd_node));
+ BUG_ON(list_empty(&blkg->q_node[BLKIO_POLICY_PROP]));
- hlist_del_init(&cfqg->cfqd_node);
+ list_del_init(&blkg->q_node[BLKIO_POLICY_PROP]);
- BUG_ON(cfqd->nr_blkcg_linked_grps <= 0);
- cfqd->nr_blkcg_linked_grps--;
+ BUG_ON(cfqd->queue->nr_blkgs[BLKIO_POLICY_PROP] <= 0);
+ cfqd->queue->nr_blkgs[BLKIO_POLICY_PROP]--;
/*
* Put the reference taken at the time of creation so that when all
@@ -1127,18 +1117,19 @@ static void cfq_destroy_cfqg(struct cfq_data *cfqd, struct cfq_group *cfqg)
static bool cfq_release_cfq_groups(struct cfq_data *cfqd)
{
- struct hlist_node *pos, *n;
- struct cfq_group *cfqg;
+ struct request_queue *q = cfqd->queue;
+ struct blkio_group *blkg, *n;
bool empty = true;
- hlist_for_each_entry_safe(cfqg, pos, n, &cfqd->cfqg_list, cfqd_node) {
+ list_for_each_entry_safe(blkg, n, &q->blkg_list[BLKIO_POLICY_PROP],
+ q_node[BLKIO_POLICY_PROP]) {
/*
* If cgroup removal path got to blk_group first and removed
* it from cgroup list, then it will take care of destroying
* cfqg also.
*/
- if (!cfq_blkiocg_del_blkio_group(cfqg_to_blkg(cfqg)))
- cfq_destroy_cfqg(cfqd, cfqg);
+ if (!cfq_blkiocg_del_blkio_group(blkg))
+ cfq_destroy_cfqg(cfqd, blkg_to_cfqg(blkg));
else
empty = false;
}
@@ -3552,7 +3543,7 @@ static void cfq_exit_queue(struct elevator_queue *e)
{
struct cfq_data *cfqd = e->elevator_data;
struct request_queue *q = cfqd->queue;
- bool wait = false;
+ bool wait;
cfq_shutdown_timer_wq(cfqd);
@@ -3568,8 +3559,7 @@ static void cfq_exit_queue(struct elevator_queue *e)
* If there are groups which we could not unlink from blkcg list,
* wait for a rcu period for them to be freed.
*/
- if (cfqd->nr_blkcg_linked_grps)
- wait = true;
+ wait = q->nr_blkgs[BLKIO_POLICY_PROP];
spin_unlock_irq(q->queue_lock);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index e38e4d0..5eb8a93 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -362,6 +362,11 @@ struct request_queue {
struct list_head timeout_list;
struct list_head icq_list;
+#if defined(CONFIG_BLK_CGROUP) || defined(CONFIG_BLK_CGROUP_MODULE)
+ /* XXX: array size hardcoded to avoid include dependency (temporary) */
+ struct list_head blkg_list[2];
+ int nr_blkgs[2];
+#endif
struct queue_limits limits;
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH 10/11] blkcg: let blkcg core manage per-queue blkg list and counter
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
` (8 preceding siblings ...)
2012-02-01 21:19 ` [PATCH 09/11] blkcg: move per-queue blkg list heads and counters to queue and blkg Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-01 21:19 ` [PATCH 11/11] blkcg: unify blkg's for blkcg policies Tejun Heo
2012-02-02 19:29 ` [PATCHSET] blkcg: unify blkgs for different policies Vivek Goyal
11 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
With the previous patch to move blkg list heads and counters to
request_queue and blkg, logic to manage them in both policies are
almost identical and can be moved to blkcg core.
This patch moves blkg link logic into blkg_lookup_create(), implements
common blkg unlink code in blkg_destroy(), and updates
blkg_destory_all() so that it's policy specific and can skip root
group. The updated blkg_destroy_all() is now used to both clear queue
for bypassing and elv switching, and release all blkgs on q exit.
This patch introduces a race window where policy [de]registration may
race against queue blkg clearing. This can only be a problem on cfq
unload and shouldn't be a real problem in practice (and we have many
other places where this race already exists). Future patches will
remove these unlikely races.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-cgroup.c | 72 ++++++++++++++++++++++++++++--------
block/blk-cgroup.h | 15 +++-----
block/blk-throttle.c | 99 +-------------------------------------------------
block/cfq-iosched.c | 98 +++-----------------------------------------------
block/elevator.c | 5 ++-
5 files changed, 71 insertions(+), 218 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 0513ed5..a7f9363 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -596,8 +596,11 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
/* insert */
spin_lock(&blkcg->lock);
swap(blkg, new_blkg);
+
hlist_add_head_rcu(&blkg->blkcg_node, &blkcg->blkg_list);
- pol->ops.blkio_link_group_fn(q, blkg);
+ list_add(&blkg->q_node[plid], &q->blkg_list[plid]);
+ q->nr_blkgs[plid]++;
+
spin_unlock(&blkcg->lock);
out:
blkg_free(new_blkg);
@@ -646,36 +649,69 @@ struct blkio_group *blkg_lookup(struct blkio_cgroup *blkcg,
}
EXPORT_SYMBOL_GPL(blkg_lookup);
-void blkg_destroy_all(struct request_queue *q)
+static void blkg_destroy(struct blkio_group *blkg, enum blkio_policy_id plid)
+{
+ struct request_queue *q = blkg->q;
+
+ lockdep_assert_held(q->queue_lock);
+
+ /* Something wrong if we are trying to remove same group twice */
+ WARN_ON_ONCE(list_empty(&blkg->q_node[plid]));
+ list_del_init(&blkg->q_node[plid]);
+
+ WARN_ON_ONCE(q->nr_blkgs[plid] <= 0);
+ q->nr_blkgs[plid]--;
+
+ /*
+ * Put the reference taken at the time of creation so that when all
+ * queues are gone, group can be destroyed.
+ */
+ blkg_put(blkg);
+}
+
+void blkg_destroy_all(struct request_queue *q, enum blkio_policy_id plid,
+ bool destroy_root)
{
- struct blkio_policy_type *pol;
+ struct blkio_group *blkg, *n;
while (true) {
bool done = true;
- spin_lock(&blkio_list_lock);
spin_lock_irq(q->queue_lock);
- /*
- * clear_queue_fn() might return with non-empty group list
- * if it raced cgroup removal and lost. cgroup removal is
- * guaranteed to make forward progress and retrying after a
- * while is enough. This ugliness is scheduled to be
- * removed after locking update.
- */
- list_for_each_entry(pol, &blkio_list, list)
- if (!pol->ops.blkio_clear_queue_fn(q))
+ list_for_each_entry_safe(blkg, n, &q->blkg_list[plid],
+ q_node[plid]) {
+ /* skip root? */
+ if (!destroy_root && blkg->blkcg == &blkio_root_cgroup)
+ continue;
+
+ /*
+ * If cgroup removal path got to blk_group first
+ * and removed it from cgroup list, then it will
+ * take care of destroying cfqg also.
+ */
+ if (!blkiocg_del_blkio_group(blkg))
+ blkg_destroy(blkg, plid);
+ else
done = false;
+ }
spin_unlock_irq(q->queue_lock);
- spin_unlock(&blkio_list_lock);
+ /*
+ * Group list may not be empty if we raced cgroup removal
+ * and lost. cgroup removal is guaranteed to make forward
+ * progress and retrying after a while is enough. This
+ * ugliness is scheduled to be removed after locking
+ * update.
+ */
if (done)
break;
msleep(10); /* just some random duration I like */
}
}
+EXPORT_SYMBOL_GPL(blkg_destroy_all);
static void blkg_rcu_free(struct rcu_head *rcu_head)
{
@@ -1538,11 +1574,13 @@ static int blkiocg_pre_destroy(struct cgroup_subsys *subsys,
* this event.
*/
spin_lock(&blkio_list_lock);
+ spin_lock_irqsave(q->queue_lock, flags);
list_for_each_entry(blkiop, &blkio_list, list) {
if (blkiop->plid != blkg->plid)
continue;
- blkiop->ops.blkio_unlink_group_fn(q, blkg);
+ blkg_destroy(blkg, blkiop->plid);
}
+ spin_unlock_irqrestore(q->queue_lock, flags);
spin_unlock(&blkio_list_lock);
} while (1);
@@ -1684,12 +1722,14 @@ static void blkcg_bypass_start(void)
__acquires(&all_q_mutex)
{
struct request_queue *q;
+ int i;
mutex_lock(&all_q_mutex);
list_for_each_entry(q, &all_q_list, all_q_node) {
blk_queue_bypass_start(q);
- blkg_destroy_all(q);
+ for (i = 0; i < BLKIO_NR_POLICIES; i++)
+ blkg_destroy_all(q, i, false);
}
}
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index ae96f19..83ce5fa 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -196,11 +196,6 @@ struct blkio_group {
};
typedef void (blkio_init_group_fn)(struct blkio_group *blkg);
-typedef void (blkio_link_group_fn)(struct request_queue *q,
- struct blkio_group *blkg);
-typedef void (blkio_unlink_group_fn)(struct request_queue *q,
- struct blkio_group *blkg);
-typedef bool (blkio_clear_queue_fn)(struct request_queue *q);
typedef void (blkio_update_group_weight_fn)(struct request_queue *q,
struct blkio_group *blkg, unsigned int weight);
typedef void (blkio_update_group_read_bps_fn)(struct request_queue *q,
@@ -214,9 +209,6 @@ typedef void (blkio_update_group_write_iops_fn)(struct request_queue *q,
struct blkio_policy_ops {
blkio_init_group_fn *blkio_init_group_fn;
- blkio_link_group_fn *blkio_link_group_fn;
- blkio_unlink_group_fn *blkio_unlink_group_fn;
- blkio_clear_queue_fn *blkio_clear_queue_fn;
blkio_update_group_weight_fn *blkio_update_group_weight_fn;
blkio_update_group_read_bps_fn *blkio_update_group_read_bps_fn;
blkio_update_group_write_bps_fn *blkio_update_group_write_bps_fn;
@@ -238,7 +230,8 @@ extern void blkcg_exit_queue(struct request_queue *q);
/* Blkio controller policy registration */
extern void blkio_policy_register(struct blkio_policy_type *);
extern void blkio_policy_unregister(struct blkio_policy_type *);
-extern void blkg_destroy_all(struct request_queue *q);
+extern void blkg_destroy_all(struct request_queue *q,
+ enum blkio_policy_id plid, bool destroy_root);
/**
* blkg_to_pdata - get policy private data
@@ -319,7 +312,9 @@ static inline void blkcg_drain_queue(struct request_queue *q) { }
static inline void blkcg_exit_queue(struct request_queue *q) { }
static inline void blkio_policy_register(struct blkio_policy_type *blkiop) { }
static inline void blkio_policy_unregister(struct blkio_policy_type *blkiop) { }
-static inline void blkg_destroy_all(struct request_queue *q) { }
+static inline void blkg_destroy_all(struct request_queue *q,
+ enum blkio_policy_id plid,
+ bool destory_root) { }
static inline void *blkg_to_pdata(struct blkio_group *blkg,
struct blkio_policy_type *pol) { return NULL; }
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index c15d383..1329412 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -157,14 +157,6 @@ static void throtl_init_blkio_group(struct blkio_group *blkg)
tg->iops[WRITE] = -1;
}
-static void throtl_link_blkio_group(struct request_queue *q,
- struct blkio_group *blkg)
-{
- list_add(&blkg->q_node[BLKIO_POLICY_THROTL],
- &q->blkg_list[BLKIO_POLICY_THROTL]);
- q->nr_blkgs[BLKIO_POLICY_THROTL]++;
-}
-
static struct
throtl_grp *throtl_lookup_tg(struct throtl_data *td, struct blkio_cgroup *blkcg)
{
@@ -813,89 +805,6 @@ throtl_schedule_delayed_work(struct throtl_data *td, unsigned long delay)
}
}
-static void
-throtl_destroy_tg(struct throtl_data *td, struct throtl_grp *tg)
-{
- struct blkio_group *blkg = tg_to_blkg(tg);
-
- /* Something wrong if we are trying to remove same group twice */
- WARN_ON_ONCE(list_empty(&blkg->q_node[BLKIO_POLICY_THROTL]));
-
- list_del_init(&blkg->q_node[BLKIO_POLICY_THROTL]);
-
- /*
- * Put the reference taken at the time of creation so that when all
- * queues are gone, group can be destroyed.
- */
- blkg_put(tg_to_blkg(tg));
- td->queue->nr_blkgs[BLKIO_POLICY_THROTL]--;
-}
-
-static bool throtl_release_tgs(struct throtl_data *td, bool release_root)
-{
- struct request_queue *q = td->queue;
- struct blkio_group *blkg, *n;
- bool empty = true;
-
- list_for_each_entry_safe(blkg, n, &q->blkg_list[BLKIO_POLICY_THROTL],
- q_node[BLKIO_POLICY_THROTL]) {
- struct throtl_grp *tg = blkg_to_tg(blkg);
-
- /* skip root? */
- if (!release_root && tg == td->root_tg)
- continue;
-
- /*
- * If cgroup removal path got to blk_group first and removed
- * it from cgroup list, then it will take care of destroying
- * cfqg also.
- */
- if (!blkiocg_del_blkio_group(blkg))
- throtl_destroy_tg(td, tg);
- else
- empty = false;
- }
- return empty;
-}
-
-/*
- * Blk cgroup controller notification saying that blkio_group object is being
- * delinked as associated cgroup object is going away. That also means that
- * no new IO will come in this group. So get rid of this group as soon as
- * any pending IO in the group is finished.
- *
- * This function is called under rcu_read_lock(). @q is the rcu protected
- * pointer. That means @q is a valid request_queue pointer as long as we
- * are rcu read lock.
- *
- * @q was fetched from blkio_group under blkio_cgroup->lock. That means
- * it should not be NULL as even if queue was going away, cgroup deltion
- * path got to it first.
- */
-void throtl_unlink_blkio_group(struct request_queue *q,
- struct blkio_group *blkg)
-{
- unsigned long flags;
-
- spin_lock_irqsave(q->queue_lock, flags);
- throtl_destroy_tg(q->td, blkg_to_tg(blkg));
- spin_unlock_irqrestore(q->queue_lock, flags);
-}
-
-static bool throtl_clear_queue(struct request_queue *q)
-{
- lockdep_assert_held(q->queue_lock);
-
- /*
- * Clear tgs but leave the root one alone. This is necessary
- * because root_tg is expected to be persistent and safe because
- * blk-throtl can never be disabled while @q is alive. This is a
- * kludge to prepare for unified blkg. This whole function will be
- * removed soon.
- */
- return throtl_release_tgs(q->td, false);
-}
-
static void throtl_update_blkio_group_common(struct throtl_data *td,
struct throtl_grp *tg)
{
@@ -960,9 +869,6 @@ static void throtl_shutdown_wq(struct request_queue *q)
static struct blkio_policy_type blkio_policy_throtl = {
.ops = {
.blkio_init_group_fn = throtl_init_blkio_group,
- .blkio_link_group_fn = throtl_link_blkio_group,
- .blkio_unlink_group_fn = throtl_unlink_blkio_group,
- .blkio_clear_queue_fn = throtl_clear_queue,
.blkio_update_group_read_bps_fn =
throtl_update_blkio_group_read_bps,
.blkio_update_group_write_bps_fn =
@@ -1148,12 +1054,11 @@ void blk_throtl_exit(struct request_queue *q)
throtl_shutdown_wq(q);
- spin_lock_irq(q->queue_lock);
- throtl_release_tgs(td, true);
+ blkg_destroy_all(q, BLKIO_POLICY_THROTL, true);
/* If there are other groups */
+ spin_lock_irq(q->queue_lock);
wait = q->nr_blkgs[BLKIO_POLICY_THROTL];
-
spin_unlock_irq(q->queue_lock);
/*
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index c13e376..7093309 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -1045,14 +1045,6 @@ static void cfq_update_blkio_group_weight(struct request_queue *q,
cfqg->needs_update = true;
}
-static void cfq_link_blkio_group(struct request_queue *q,
- struct blkio_group *blkg)
-{
- list_add(&blkg->q_node[BLKIO_POLICY_PROP],
- &q->blkg_list[BLKIO_POLICY_PROP]);
- q->nr_blkgs[BLKIO_POLICY_PROP]++;
-}
-
static void cfq_init_blkio_group(struct blkio_group *blkg)
{
struct cfq_group *cfqg = blkg_to_cfqg(blkg);
@@ -1096,84 +1088,6 @@ static void cfq_link_cfqq_cfqg(struct cfq_queue *cfqq, struct cfq_group *cfqg)
blkg_get(cfqg_to_blkg(cfqg));
}
-static void cfq_destroy_cfqg(struct cfq_data *cfqd, struct cfq_group *cfqg)
-{
- struct blkio_group *blkg = cfqg_to_blkg(cfqg);
-
- /* Something wrong if we are trying to remove same group twice */
- BUG_ON(list_empty(&blkg->q_node[BLKIO_POLICY_PROP]));
-
- list_del_init(&blkg->q_node[BLKIO_POLICY_PROP]);
-
- BUG_ON(cfqd->queue->nr_blkgs[BLKIO_POLICY_PROP] <= 0);
- cfqd->queue->nr_blkgs[BLKIO_POLICY_PROP]--;
-
- /*
- * Put the reference taken at the time of creation so that when all
- * queues are gone, group can be destroyed.
- */
- blkg_put(cfqg_to_blkg(cfqg));
-}
-
-static bool cfq_release_cfq_groups(struct cfq_data *cfqd)
-{
- struct request_queue *q = cfqd->queue;
- struct blkio_group *blkg, *n;
- bool empty = true;
-
- list_for_each_entry_safe(blkg, n, &q->blkg_list[BLKIO_POLICY_PROP],
- q_node[BLKIO_POLICY_PROP]) {
- /*
- * If cgroup removal path got to blk_group first and removed
- * it from cgroup list, then it will take care of destroying
- * cfqg also.
- */
- if (!cfq_blkiocg_del_blkio_group(blkg))
- cfq_destroy_cfqg(cfqd, blkg_to_cfqg(blkg));
- else
- empty = false;
- }
- return empty;
-}
-
-/*
- * Blk cgroup controller notification saying that blkio_group object is being
- * delinked as associated cgroup object is going away. That also means that
- * no new IO will come in this group. So get rid of this group as soon as
- * any pending IO in the group is finished.
- *
- * This function is called under rcu_read_lock(). key is the rcu protected
- * pointer. That means @q is a valid request_queue pointer as long as we
- * are rcu read lock.
- *
- * @q was fetched from blkio_group under blkio_cgroup->lock. That means
- * it should not be NULL as even if elevator was exiting, cgroup deltion
- * path got to it first.
- */
-static void cfq_unlink_blkio_group(struct request_queue *q,
- struct blkio_group *blkg)
-{
- struct cfq_data *cfqd = q->elevator->elevator_data;
- unsigned long flags;
-
- spin_lock_irqsave(q->queue_lock, flags);
- cfq_destroy_cfqg(cfqd, blkg_to_cfqg(blkg));
- spin_unlock_irqrestore(q->queue_lock, flags);
-}
-
-static struct elevator_type iosched_cfq;
-
-static bool cfq_clear_queue(struct request_queue *q)
-{
- lockdep_assert_held(q->queue_lock);
-
- /* shoot down blkgs iff the current elevator is cfq */
- if (!q->elevator || q->elevator->type != &iosched_cfq)
- return true;
-
- return cfq_release_cfq_groups(q->elevator->elevator_data);
-}
-
#else /* GROUP_IOSCHED */
static struct cfq_group *cfq_lookup_create_cfqg(struct cfq_data *cfqd,
struct blkio_cgroup *blkcg)
@@ -1186,8 +1100,6 @@ cfq_link_cfqq_cfqg(struct cfq_queue *cfqq, struct cfq_group *cfqg) {
cfqq->cfqg = cfqg;
}
-static void cfq_release_cfq_groups(struct cfq_data *cfqd) {}
-
#endif /* GROUP_IOSCHED */
/*
@@ -3553,14 +3465,17 @@ static void cfq_exit_queue(struct elevator_queue *e)
__cfq_slice_expired(cfqd, cfqd->active_queue, 0);
cfq_put_async_queues(cfqd);
- cfq_release_cfq_groups(cfqd);
+
+ spin_unlock_irq(q->queue_lock);
+
+ blkg_destroy_all(q, BLKIO_POLICY_PROP, true);
/*
* If there are groups which we could not unlink from blkcg list,
* wait for a rcu period for them to be freed.
*/
+ spin_lock_irq(q->queue_lock);
wait = q->nr_blkgs[BLKIO_POLICY_PROP];
-
spin_unlock_irq(q->queue_lock);
cfq_shutdown_timer_wq(cfqd);
@@ -3799,9 +3714,6 @@ static struct elevator_type iosched_cfq = {
static struct blkio_policy_type blkio_policy_cfq = {
.ops = {
.blkio_init_group_fn = cfq_init_blkio_group,
- .blkio_link_group_fn = cfq_link_blkio_group,
- .blkio_unlink_group_fn = cfq_unlink_blkio_group,
- .blkio_clear_queue_fn = cfq_clear_queue,
.blkio_update_group_weight_fn = cfq_update_blkio_group_weight,
},
.plid = BLKIO_POLICY_PROP,
diff --git a/block/elevator.c b/block/elevator.c
index d6116af..4599615 100644
--- a/block/elevator.c
+++ b/block/elevator.c
@@ -923,7 +923,7 @@ static int elevator_switch(struct request_queue *q, struct elevator_type *new_e)
{
struct elevator_queue *old = q->elevator;
bool registered = old->registered;
- int err;
+ int i, err;
/*
* Turn on BYPASS and drain all requests w/ elevator private data.
@@ -942,7 +942,8 @@ static int elevator_switch(struct request_queue *q, struct elevator_type *new_e)
ioc_clear_queue(q);
spin_unlock_irq(q->queue_lock);
- blkg_destroy_all(q);
+ for (i = 0; i < BLKIO_NR_POLICIES; i++)
+ blkg_destroy_all(q, i, false);
/* allocate, init and register new elevator */
err = -ENOMEM;
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH 11/11] blkcg: unify blkg's for blkcg policies
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
` (9 preceding siblings ...)
2012-02-01 21:19 ` [PATCH 10/11] blkcg: let blkcg core manage per-queue blkg list and counter Tejun Heo
@ 2012-02-01 21:19 ` Tejun Heo
2012-02-02 0:37 ` [PATCH UPDATED " Tejun Heo
2012-02-02 19:29 ` [PATCHSET] blkcg: unify blkgs for different policies Vivek Goyal
11 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-01 21:19 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel, Tejun Heo
Currently, blkg is per cgroup-queue-policy combination. This is
unnatural and leads to various convolutions in partially used
duplicate fields in blkg, config / stat access, and general management
of blkgs.
This patch make blkg's per cgroup-queue and let them serve all
policies. blkgs are now created and destroyed by blkcg core proper.
This will allow further consolidation of common management logic into
blkcg core and API with better defined semantics and layering.
As a transitional step to untangle blkg management, elvswitch and
policy [de]registration, all blkgs except the root blkg are being shot
down during elvswitch and bypass. This patch adds blkg_root_update()
to update root blkg in place on policy change. This is hacky but
should be good enough as interim step until we get locking simplified
and switch over to in-place update for all blkgs.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
block/blk-cgroup.c | 215 +++++++++++++++++++++++++++++-------------------
block/blk-cgroup.h | 10 +--
block/blk-core.c | 3 +-
block/blk-sysfs.c | 4 +-
block/blk-throttle.c | 9 +--
block/cfq-iosched.c | 4 +-
block/elevator.c | 5 +-
include/linux/blkdev.h | 5 +-
8 files changed, 146 insertions(+), 109 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index a7f9363..ae988f0 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -461,16 +461,20 @@ EXPORT_SYMBOL_GPL(blkiocg_update_io_merged_stats);
*/
static void blkg_free(struct blkio_group *blkg)
{
- struct blkg_policy_data *pd;
+ int i;
if (!blkg)
return;
- pd = blkg->pd[blkg->plid];
- if (pd) {
- free_percpu(pd->stats_cpu);
- kfree(pd);
+ for (i = 0; i < BLKIO_NR_POLICIES; i++) {
+ struct blkg_policy_data *pd = blkg->pd[i];
+
+ if (pd) {
+ free_percpu(pd->stats_cpu);
+ kfree(pd);
+ }
}
+
kfree(blkg);
}
@@ -486,11 +490,10 @@ static void blkg_free(struct blkio_group *blkg)
* percpu stat breakage.
*/
static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
- struct request_queue *q,
- struct blkio_policy_type *pol)
+ struct request_queue *q)
{
struct blkio_group *blkg;
- struct blkg_policy_data *pd;
+ int i;
/* alloc and init base part */
blkg = kzalloc_node(sizeof(*blkg), GFP_ATOMIC, q->node);
@@ -499,34 +502,83 @@ static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
spin_lock_init(&blkg->stats_lock);
rcu_assign_pointer(blkg->q, q);
- INIT_LIST_HEAD(&blkg->q_node[0]);
- INIT_LIST_HEAD(&blkg->q_node[1]);
+ INIT_LIST_HEAD(&blkg->q_node);
blkg->blkcg = blkcg;
- blkg->plid = pol->plid;
blkg->refcnt = 1;
cgroup_path(blkcg->css.cgroup, blkg->path, sizeof(blkg->path));
- /* alloc per-policy data and attach it to blkg */
- pd = kzalloc_node(sizeof(*pd) + pol->pdata_size, GFP_ATOMIC,
- q->node);
- if (!pd) {
- blkg_free(blkg);
- return NULL;
+ for (i = 0; i < BLKIO_NR_POLICIES; i++) {
+ struct blkio_policy_type *pol = blkio_policy[i];
+ struct blkg_policy_data *pd;
+
+ if (!pol)
+ continue;
+
+ /* alloc per-policy data and attach it to blkg */
+ pd = kzalloc_node(sizeof(*pd) + pol->pdata_size, GFP_ATOMIC,
+ q->node);
+ if (!pd) {
+ blkg_free(blkg);
+ return NULL;
+ }
+
+ blkg->pd[i] = pd;
+ pd->blkg = blkg;
+
+ /* broken, read comment in the callsite */
+ pd->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
+ if (!pd->stats_cpu) {
+ blkg_free(blkg);
+ return NULL;
+ }
}
- blkg->pd[pol->plid] = pd;
- pd->blkg = blkg;
+ /* invoke per-policy init */
+ for (i = 0; i < BLKIO_NR_POLICIES; i++) {
+ struct blkio_policy_type *pol = blkio_policy[i];
+
+ if (pol)
+ pol->ops.blkio_init_group_fn(blkg);
+ }
+
+ return blkg;
+}
+
+/*
+ * XXX: This updates blkg policy data in-place for root blkg, which is
+ * necessary across policy registration as root blkgs aren't shot down.
+ * This hacky implementation is interim. Eventually, blkg shoot down will
+ * be replaced by proper in-place update.
+ */
+static struct blkio_group *blkg_root_update(struct blkio_group *blkg,
+ enum blkio_policy_id plid)
+{
+ struct request_queue *q = blkg->q;
+ struct blkio_policy_type *pol = blkio_policy[plid];
+ struct blkg_policy_data *pd;
- /* broken, read comment in the callsite */
+ kfree(blkg->pd[plid]);
+ blkg->pd[plid] = NULL;
+ pd = kzalloc(sizeof(*pd) + pol->pdata_size, GFP_ATOMIC);
+ if (!pd)
+ return NULL;
+
+ spin_unlock_irq(q->queue_lock);
+ rcu_read_unlock();
pd->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
+ rcu_read_lock();
+ spin_lock_irq(q->queue_lock);
+
if (!pd->stats_cpu) {
- blkg_free(blkg);
+ kfree(pd);
return NULL;
}
- /* invoke per-policy init */
+ blkg->pd[plid] = pd;
+ pd->blkg = blkg;
pol->ops.blkio_init_group_fn(blkg);
+
return blkg;
}
@@ -536,7 +588,6 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
bool for_root)
__releases(q->queue_lock) __acquires(q->queue_lock)
{
- struct blkio_policy_type *pol = blkio_policy[plid];
struct blkio_group *blkg, *new_blkg;
WARN_ON_ONCE(!rcu_read_lock_held());
@@ -551,9 +602,12 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
if (unlikely(blk_queue_bypass(q)) && !for_root)
return ERR_PTR(blk_queue_dead(q) ? -EINVAL : -EBUSY);
- blkg = blkg_lookup(blkcg, q, plid);
- if (blkg)
+ blkg = blkg_lookup(blkcg, q);
+ if (blkg) {
+ if (for_root)
+ return blkg_root_update(blkg, plid);
return blkg;
+ }
/* blkg holds a reference to blkcg */
if (!css_tryget(&blkcg->css))
@@ -571,7 +625,7 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
spin_unlock_irq(q->queue_lock);
rcu_read_unlock();
- new_blkg = blkg_alloc(blkcg, q, pol);
+ new_blkg = blkg_alloc(blkcg, q);
rcu_read_lock();
spin_lock_irq(q->queue_lock);
@@ -583,7 +637,7 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
}
/* did someone beat us to it? */
- blkg = blkg_lookup(blkcg, q, plid);
+ blkg = blkg_lookup(blkcg, q);
if (unlikely(blkg))
goto out;
@@ -598,8 +652,8 @@ struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
swap(blkg, new_blkg);
hlist_add_head_rcu(&blkg->blkcg_node, &blkcg->blkg_list);
- list_add(&blkg->q_node[plid], &q->blkg_list[plid]);
- q->nr_blkgs[plid]++;
+ list_add(&blkg->q_node, &q->blkg_list);
+ q->nr_blkgs++;
spin_unlock(&blkcg->lock);
out:
@@ -636,31 +690,30 @@ EXPORT_SYMBOL_GPL(blkiocg_del_blkio_group);
/* called under rcu_read_lock(). */
struct blkio_group *blkg_lookup(struct blkio_cgroup *blkcg,
- struct request_queue *q,
- enum blkio_policy_id plid)
+ struct request_queue *q)
{
struct blkio_group *blkg;
struct hlist_node *n;
hlist_for_each_entry_rcu(blkg, n, &blkcg->blkg_list, blkcg_node)
- if (blkg->q == q && blkg->plid == plid)
+ if (blkg->q == q)
return blkg;
return NULL;
}
EXPORT_SYMBOL_GPL(blkg_lookup);
-static void blkg_destroy(struct blkio_group *blkg, enum blkio_policy_id plid)
+static void blkg_destroy(struct blkio_group *blkg)
{
struct request_queue *q = blkg->q;
lockdep_assert_held(q->queue_lock);
/* Something wrong if we are trying to remove same group twice */
- WARN_ON_ONCE(list_empty(&blkg->q_node[plid]));
- list_del_init(&blkg->q_node[plid]);
+ WARN_ON_ONCE(list_empty(&blkg->q_node));
+ list_del_init(&blkg->q_node);
- WARN_ON_ONCE(q->nr_blkgs[plid] <= 0);
- q->nr_blkgs[plid]--;
+ WARN_ON_ONCE(q->nr_blkgs <= 0);
+ q->nr_blkgs--;
/*
* Put the reference taken at the time of creation so that when all
@@ -669,8 +722,7 @@ static void blkg_destroy(struct blkio_group *blkg, enum blkio_policy_id plid)
blkg_put(blkg);
}
-void blkg_destroy_all(struct request_queue *q, enum blkio_policy_id plid,
- bool destroy_root)
+void blkg_destroy_all(struct request_queue *q, bool destroy_root)
{
struct blkio_group *blkg, *n;
@@ -679,8 +731,7 @@ void blkg_destroy_all(struct request_queue *q, enum blkio_policy_id plid,
spin_lock_irq(q->queue_lock);
- list_for_each_entry_safe(blkg, n, &q->blkg_list[plid],
- q_node[plid]) {
+ list_for_each_entry_safe(blkg, n, &q->blkg_list, q_node) {
/* skip root? */
if (!destroy_root && blkg->blkcg == &blkio_root_cgroup)
continue;
@@ -691,7 +742,7 @@ void blkg_destroy_all(struct request_queue *q, enum blkio_policy_id plid,
* take care of destroying cfqg also.
*/
if (!blkiocg_del_blkio_group(blkg))
- blkg_destroy(blkg, plid);
+ blkg_destroy(blkg);
else
done = false;
}
@@ -776,43 +827,49 @@ blkiocg_reset_stats(struct cgroup *cgroup, struct cftype *cftype, u64 val)
#endif
blkcg = cgroup_to_blkio_cgroup(cgroup);
+ spin_lock(&blkio_list_lock);
spin_lock_irq(&blkcg->lock);
hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node) {
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkio_policy_type *pol;
+
+ list_for_each_entry(pol, &blkio_list, list) {
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
- spin_lock(&blkg->stats_lock);
- stats = &pd->stats;
+ spin_lock(&blkg->stats_lock);
+ stats = &pd->stats;
#ifdef CONFIG_DEBUG_BLK_CGROUP
- idling = blkio_blkg_idling(stats);
- waiting = blkio_blkg_waiting(stats);
- empty = blkio_blkg_empty(stats);
+ idling = blkio_blkg_idling(stats);
+ waiting = blkio_blkg_waiting(stats);
+ empty = blkio_blkg_empty(stats);
#endif
- for (i = 0; i < BLKIO_STAT_TOTAL; i++)
- queued[i] = stats->stat_arr[BLKIO_STAT_QUEUED][i];
- memset(stats, 0, sizeof(struct blkio_group_stats));
- for (i = 0; i < BLKIO_STAT_TOTAL; i++)
- stats->stat_arr[BLKIO_STAT_QUEUED][i] = queued[i];
+ for (i = 0; i < BLKIO_STAT_TOTAL; i++)
+ queued[i] = stats->stat_arr[BLKIO_STAT_QUEUED][i];
+ memset(stats, 0, sizeof(struct blkio_group_stats));
+ for (i = 0; i < BLKIO_STAT_TOTAL; i++)
+ stats->stat_arr[BLKIO_STAT_QUEUED][i] = queued[i];
#ifdef CONFIG_DEBUG_BLK_CGROUP
- if (idling) {
- blkio_mark_blkg_idling(stats);
- stats->start_idle_time = now;
- }
- if (waiting) {
- blkio_mark_blkg_waiting(stats);
- stats->start_group_wait_time = now;
- }
- if (empty) {
- blkio_mark_blkg_empty(stats);
- stats->start_empty_time = now;
- }
+ if (idling) {
+ blkio_mark_blkg_idling(stats);
+ stats->start_idle_time = now;
+ }
+ if (waiting) {
+ blkio_mark_blkg_waiting(stats);
+ stats->start_group_wait_time = now;
+ }
+ if (empty) {
+ blkio_mark_blkg_empty(stats);
+ stats->start_empty_time = now;
+ }
#endif
- spin_unlock(&blkg->stats_lock);
+ spin_unlock(&blkg->stats_lock);
- /* Reset Per cpu stats which don't take blkg->stats_lock */
- blkio_reset_stats_cpu(blkg, blkg->plid);
+ /* Reset Per cpu stats which don't take blkg->stats_lock */
+ blkio_reset_stats_cpu(blkg, pol->plid);
+ }
}
spin_unlock_irq(&blkcg->lock);
+ spin_unlock(&blkio_list_lock);
return 0;
}
@@ -1157,8 +1214,7 @@ static void blkio_read_conf(struct cftype *cft, struct blkio_cgroup *blkcg,
spin_lock_irq(&blkcg->lock);
hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node)
- if (BLKIOFILE_POLICY(cft->private) == blkg->plid)
- blkio_print_group_conf(cft, blkg, m);
+ blkio_print_group_conf(cft, blkg, m);
spin_unlock_irq(&blkcg->lock);
}
@@ -1213,8 +1269,6 @@ static int blkio_read_blkg_stats(struct blkio_cgroup *blkcg,
const char *dname = dev_name(blkg->q->backing_dev_info.dev);
int plid = BLKIOFILE_POLICY(cft->private);
- if (plid != blkg->plid)
- continue;
if (pcpu) {
cgroup_total += blkio_get_stat_cpu(blkg, plid,
cb, dname, type);
@@ -1324,9 +1378,9 @@ static int blkio_weight_write(struct blkio_cgroup *blkcg, int plid, u64 val)
blkcg->weight = (unsigned int)val;
hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node) {
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[plid];
- if (blkg->plid == plid && !pd->conf.weight)
+ if (!pd->conf.weight)
blkio_update_group_weight(blkg, plid, blkcg->weight);
}
@@ -1549,7 +1603,6 @@ static int blkiocg_pre_destroy(struct cgroup_subsys *subsys,
unsigned long flags;
struct blkio_group *blkg;
struct request_queue *q;
- struct blkio_policy_type *blkiop;
rcu_read_lock();
@@ -1575,11 +1628,7 @@ static int blkiocg_pre_destroy(struct cgroup_subsys *subsys,
*/
spin_lock(&blkio_list_lock);
spin_lock_irqsave(q->queue_lock, flags);
- list_for_each_entry(blkiop, &blkio_list, list) {
- if (blkiop->plid != blkg->plid)
- continue;
- blkg_destroy(blkg, blkiop->plid);
- }
+ blkg_destroy(blkg);
spin_unlock_irqrestore(q->queue_lock, flags);
spin_unlock(&blkio_list_lock);
} while (1);
@@ -1673,6 +1722,8 @@ void blkcg_exit_queue(struct request_queue *q)
list_del_init(&q->all_q_node);
mutex_unlock(&all_q_mutex);
+ blkg_destroy_all(q, true);
+
blk_throtl_exit(q);
}
@@ -1722,14 +1773,12 @@ static void blkcg_bypass_start(void)
__acquires(&all_q_mutex)
{
struct request_queue *q;
- int i;
mutex_lock(&all_q_mutex);
list_for_each_entry(q, &all_q_list, all_q_node) {
blk_queue_bypass_start(q);
- for (i = 0; i < BLKIO_NR_POLICIES; i++)
- blkg_destroy_all(q, i, false);
+ blkg_destroy_all(q, false);
}
}
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 83ce5fa..bd66936 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -178,13 +178,11 @@ struct blkg_policy_data {
struct blkio_group {
/* Pointer to the associated request_queue, RCU protected */
struct request_queue __rcu *q;
- struct list_head q_node[BLKIO_NR_POLICIES];
+ struct list_head q_node;
struct hlist_node blkcg_node;
struct blkio_cgroup *blkcg;
/* Store cgroup path */
char path[128];
- /* policy which owns this blk group */
- enum blkio_policy_id plid;
/* reference count */
int refcnt;
@@ -230,8 +228,7 @@ extern void blkcg_exit_queue(struct request_queue *q);
/* Blkio controller policy registration */
extern void blkio_policy_register(struct blkio_policy_type *);
extern void blkio_policy_unregister(struct blkio_policy_type *);
-extern void blkg_destroy_all(struct request_queue *q,
- enum blkio_policy_id plid, bool destroy_root);
+extern void blkg_destroy_all(struct request_queue *q, bool destroy_root);
/**
* blkg_to_pdata - get policy private data
@@ -382,8 +379,7 @@ extern struct blkio_cgroup *cgroup_to_blkio_cgroup(struct cgroup *cgroup);
extern struct blkio_cgroup *task_blkio_cgroup(struct task_struct *tsk);
extern int blkiocg_del_blkio_group(struct blkio_group *blkg);
extern struct blkio_group *blkg_lookup(struct blkio_cgroup *blkcg,
- struct request_queue *q,
- enum blkio_policy_id plid);
+ struct request_queue *q);
struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
struct request_queue *q,
enum blkio_policy_id plid,
diff --git a/block/blk-core.c b/block/blk-core.c
index 025ef60..6de6cb5 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -547,8 +547,7 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id)
INIT_LIST_HEAD(&q->timeout_list);
INIT_LIST_HEAD(&q->icq_list);
#if defined(CONFIG_BLK_CGROUP) || defined(CONFIG_BLK_CGROUP_MODULE)
- INIT_LIST_HEAD(&q->blkg_list[0]);
- INIT_LIST_HEAD(&q->blkg_list[1]);
+ INIT_LIST_HEAD(&q->blkg_list);
#endif
INIT_LIST_HEAD(&q->flush_queue[0]);
INIT_LIST_HEAD(&q->flush_queue[1]);
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index 00cdc98..aa41b47 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -480,6 +480,8 @@ static void blk_release_queue(struct kobject *kobj)
blk_sync_queue(q);
+ blkcg_exit_queue(q);
+
if (q->elevator) {
spin_lock_irq(q->queue_lock);
ioc_clear_queue(q);
@@ -487,8 +489,6 @@ static void blk_release_queue(struct kobject *kobj)
elevator_exit(q->elevator);
}
- blkcg_exit_queue(q);
-
if (rl->rq_pool)
mempool_destroy(rl->rq_pool);
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 1329412..e35ee7a 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -167,7 +167,7 @@ throtl_grp *throtl_lookup_tg(struct throtl_data *td, struct blkio_cgroup *blkcg)
if (blkcg == &blkio_root_cgroup)
return td->root_tg;
- return blkg_to_tg(blkg_lookup(blkcg, td->queue, BLKIO_POLICY_THROTL));
+ return blkg_to_tg(blkg_lookup(blkcg, td->queue));
}
static struct throtl_grp *throtl_lookup_create_tg(struct throtl_data *td,
@@ -704,8 +704,7 @@ static void throtl_process_limit_change(struct throtl_data *td)
throtl_log(td, "limits changed");
- list_for_each_entry_safe(blkg, n, &q->blkg_list[BLKIO_POLICY_THROTL],
- q_node[BLKIO_POLICY_THROTL]) {
+ list_for_each_entry_safe(blkg, n, &q->blkg_list, q_node) {
struct throtl_grp *tg = blkg_to_tg(blkg);
if (!tg->limits_changed)
@@ -1054,11 +1053,9 @@ void blk_throtl_exit(struct request_queue *q)
throtl_shutdown_wq(q);
- blkg_destroy_all(q, BLKIO_POLICY_THROTL, true);
-
/* If there are other groups */
spin_lock_irq(q->queue_lock);
- wait = q->nr_blkgs[BLKIO_POLICY_THROTL];
+ wait = q->nr_blkgs;
spin_unlock_irq(q->queue_lock);
/*
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 7093309..e3fa5d6 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -3468,14 +3468,12 @@ static void cfq_exit_queue(struct elevator_queue *e)
spin_unlock_irq(q->queue_lock);
- blkg_destroy_all(q, BLKIO_POLICY_PROP, true);
-
/*
* If there are groups which we could not unlink from blkcg list,
* wait for a rcu period for them to be freed.
*/
spin_lock_irq(q->queue_lock);
- wait = q->nr_blkgs[BLKIO_POLICY_PROP];
+ wait = q->nr_blkgs;
spin_unlock_irq(q->queue_lock);
cfq_shutdown_timer_wq(cfqd);
diff --git a/block/elevator.c b/block/elevator.c
index 4599615..49504cd 100644
--- a/block/elevator.c
+++ b/block/elevator.c
@@ -923,7 +923,7 @@ static int elevator_switch(struct request_queue *q, struct elevator_type *new_e)
{
struct elevator_queue *old = q->elevator;
bool registered = old->registered;
- int i, err;
+ int err;
/*
* Turn on BYPASS and drain all requests w/ elevator private data.
@@ -942,8 +942,7 @@ static int elevator_switch(struct request_queue *q, struct elevator_type *new_e)
ioc_clear_queue(q);
spin_unlock_irq(q->queue_lock);
- for (i = 0; i < BLKIO_NR_POLICIES; i++)
- blkg_destroy_all(q, i, false);
+ blkg_destroy_all(q, false);
/* allocate, init and register new elevator */
err = -ENOMEM;
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 5eb8a93..f4d40bc 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -363,9 +363,8 @@ struct request_queue {
struct list_head icq_list;
#if defined(CONFIG_BLK_CGROUP) || defined(CONFIG_BLK_CGROUP_MODULE)
- /* XXX: array size hardcoded to avoid include dependency (temporary) */
- struct list_head blkg_list[2];
- int nr_blkgs[2];
+ struct list_head blkg_list;
+ int nr_blkgs;
#endif
struct queue_limits limits;
--
1.7.7.3
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-01 21:19 ` [PATCH 11/11] blkcg: unify blkg's for blkcg policies Tejun Heo
@ 2012-02-02 0:37 ` Tejun Heo
2012-02-03 19:41 ` Vivek Goyal
` (2 more replies)
0 siblings, 3 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-02 0:37 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel
Currently, blkg is per cgroup-queue-policy combination. This is
unnatural and leads to various convolutions in partially used
duplicate fields in blkg, config / stat access, and general management
of blkgs.
This patch make blkg's per cgroup-queue and let them serve all
policies. blkgs are now created and destroyed by blkcg core proper.
This will allow further consolidation of common management logic into
blkcg core and API with better defined semantics and layering.
As a transitional step to untangle blkg management, elvswitch and
policy [de]registration, all blkgs except the root blkg are being shot
down during elvswitch and bypass. This patch adds blkg_root_update()
to update root blkg in place on policy change. This is hacky and racy
but should be good enough as interim step until we get locking
simplified and switch over to proper in-place update for all blkgs.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
I was thinking root blkgs should be updated on policy changes and
commit message and comments were written that way too but for some
reason I wrote code which updates on elvswitch, which leads to oops on
root blkg access after policy changes (ie. insmod cfq-iosched.ko).
Fixed. git branch updated accordingly.
Thanks.
block/blk-cgroup.c | 215 +++++++++++++++++++++++++++++--------------------
block/blk-cgroup.h | 10 --
block/blk-core.c | 3
block/blk-sysfs.c | 4
block/blk-throttle.c | 9 --
block/cfq-iosched.c | 4
block/elevator.c | 5 -
include/linux/blkdev.h | 5 -
8 files changed, 143 insertions(+), 112 deletions(-)
Index: work/block/blk-cgroup.c
===================================================================
--- work.orig/block/blk-cgroup.c
+++ work/block/blk-cgroup.c
@@ -461,16 +461,20 @@ EXPORT_SYMBOL_GPL(blkiocg_update_io_merg
*/
static void blkg_free(struct blkio_group *blkg)
{
- struct blkg_policy_data *pd;
+ int i;
if (!blkg)
return;
- pd = blkg->pd[blkg->plid];
- if (pd) {
- free_percpu(pd->stats_cpu);
- kfree(pd);
+ for (i = 0; i < BLKIO_NR_POLICIES; i++) {
+ struct blkg_policy_data *pd = blkg->pd[i];
+
+ if (pd) {
+ free_percpu(pd->stats_cpu);
+ kfree(pd);
+ }
}
+
kfree(blkg);
}
@@ -486,11 +490,10 @@ static void blkg_free(struct blkio_group
* percpu stat breakage.
*/
static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
- struct request_queue *q,
- struct blkio_policy_type *pol)
+ struct request_queue *q)
{
struct blkio_group *blkg;
- struct blkg_policy_data *pd;
+ int i;
/* alloc and init base part */
blkg = kzalloc_node(sizeof(*blkg), GFP_ATOMIC, q->node);
@@ -499,34 +502,45 @@ static struct blkio_group *blkg_alloc(st
spin_lock_init(&blkg->stats_lock);
rcu_assign_pointer(blkg->q, q);
- INIT_LIST_HEAD(&blkg->q_node[0]);
- INIT_LIST_HEAD(&blkg->q_node[1]);
+ INIT_LIST_HEAD(&blkg->q_node);
blkg->blkcg = blkcg;
- blkg->plid = pol->plid;
blkg->refcnt = 1;
cgroup_path(blkcg->css.cgroup, blkg->path, sizeof(blkg->path));
- /* alloc per-policy data and attach it to blkg */
- pd = kzalloc_node(sizeof(*pd) + pol->pdata_size, GFP_ATOMIC,
- q->node);
- if (!pd) {
- blkg_free(blkg);
- return NULL;
- }
+ for (i = 0; i < BLKIO_NR_POLICIES; i++) {
+ struct blkio_policy_type *pol = blkio_policy[i];
+ struct blkg_policy_data *pd;
+
+ if (!pol)
+ continue;
- blkg->pd[pol->plid] = pd;
- pd->blkg = blkg;
+ /* alloc per-policy data and attach it to blkg */
+ pd = kzalloc_node(sizeof(*pd) + pol->pdata_size, GFP_ATOMIC,
+ q->node);
+ if (!pd) {
+ blkg_free(blkg);
+ return NULL;
+ }
- /* broken, read comment in the callsite */
+ blkg->pd[i] = pd;
+ pd->blkg = blkg;
- pd->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
- if (!pd->stats_cpu) {
- blkg_free(blkg);
- return NULL;
+ /* broken, read comment in the callsite */
+ pd->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
+ if (!pd->stats_cpu) {
+ blkg_free(blkg);
+ return NULL;
+ }
}
/* invoke per-policy init */
- pol->ops.blkio_init_group_fn(blkg);
+ for (i = 0; i < BLKIO_NR_POLICIES; i++) {
+ struct blkio_policy_type *pol = blkio_policy[i];
+
+ if (pol)
+ pol->ops.blkio_init_group_fn(blkg);
+ }
+
return blkg;
}
@@ -536,7 +550,6 @@ struct blkio_group *blkg_lookup_create(s
bool for_root)
__releases(q->queue_lock) __acquires(q->queue_lock)
{
- struct blkio_policy_type *pol = blkio_policy[plid];
struct blkio_group *blkg, *new_blkg;
WARN_ON_ONCE(!rcu_read_lock_held());
@@ -551,7 +564,7 @@ struct blkio_group *blkg_lookup_create(s
if (unlikely(blk_queue_bypass(q)) && !for_root)
return ERR_PTR(blk_queue_dead(q) ? -EINVAL : -EBUSY);
- blkg = blkg_lookup(blkcg, q, plid);
+ blkg = blkg_lookup(blkcg, q);
if (blkg)
return blkg;
@@ -571,7 +584,7 @@ struct blkio_group *blkg_lookup_create(s
spin_unlock_irq(q->queue_lock);
rcu_read_unlock();
- new_blkg = blkg_alloc(blkcg, q, pol);
+ new_blkg = blkg_alloc(blkcg, q);
rcu_read_lock();
spin_lock_irq(q->queue_lock);
@@ -583,7 +596,7 @@ struct blkio_group *blkg_lookup_create(s
}
/* did someone beat us to it? */
- blkg = blkg_lookup(blkcg, q, plid);
+ blkg = blkg_lookup(blkcg, q);
if (unlikely(blkg))
goto out;
@@ -598,8 +611,8 @@ struct blkio_group *blkg_lookup_create(s
swap(blkg, new_blkg);
hlist_add_head_rcu(&blkg->blkcg_node, &blkcg->blkg_list);
- list_add(&blkg->q_node[plid], &q->blkg_list[plid]);
- q->nr_blkgs[plid]++;
+ list_add(&blkg->q_node, &q->blkg_list);
+ q->nr_blkgs++;
spin_unlock(&blkcg->lock);
out:
@@ -636,31 +649,30 @@ EXPORT_SYMBOL_GPL(blkiocg_del_blkio_grou
/* called under rcu_read_lock(). */
struct blkio_group *blkg_lookup(struct blkio_cgroup *blkcg,
- struct request_queue *q,
- enum blkio_policy_id plid)
+ struct request_queue *q)
{
struct blkio_group *blkg;
struct hlist_node *n;
hlist_for_each_entry_rcu(blkg, n, &blkcg->blkg_list, blkcg_node)
- if (blkg->q == q && blkg->plid == plid)
+ if (blkg->q == q)
return blkg;
return NULL;
}
EXPORT_SYMBOL_GPL(blkg_lookup);
-static void blkg_destroy(struct blkio_group *blkg, enum blkio_policy_id plid)
+static void blkg_destroy(struct blkio_group *blkg)
{
struct request_queue *q = blkg->q;
lockdep_assert_held(q->queue_lock);
/* Something wrong if we are trying to remove same group twice */
- WARN_ON_ONCE(list_empty(&blkg->q_node[plid]));
- list_del_init(&blkg->q_node[plid]);
+ WARN_ON_ONCE(list_empty(&blkg->q_node));
+ list_del_init(&blkg->q_node);
- WARN_ON_ONCE(q->nr_blkgs[plid] <= 0);
- q->nr_blkgs[plid]--;
+ WARN_ON_ONCE(q->nr_blkgs <= 0);
+ q->nr_blkgs--;
/*
* Put the reference taken at the time of creation so that when all
@@ -669,8 +681,7 @@ static void blkg_destroy(struct blkio_gr
blkg_put(blkg);
}
-void blkg_destroy_all(struct request_queue *q, enum blkio_policy_id plid,
- bool destroy_root)
+void blkg_destroy_all(struct request_queue *q, bool destroy_root)
{
struct blkio_group *blkg, *n;
@@ -679,8 +690,7 @@ void blkg_destroy_all(struct request_que
spin_lock_irq(q->queue_lock);
- list_for_each_entry_safe(blkg, n, &q->blkg_list[plid],
- q_node[plid]) {
+ list_for_each_entry_safe(blkg, n, &q->blkg_list, q_node) {
/* skip root? */
if (!destroy_root && blkg->blkcg == &blkio_root_cgroup)
continue;
@@ -691,7 +701,7 @@ void blkg_destroy_all(struct request_que
* take care of destroying cfqg also.
*/
if (!blkiocg_del_blkio_group(blkg))
- blkg_destroy(blkg, plid);
+ blkg_destroy(blkg);
else
done = false;
}
@@ -776,43 +786,49 @@ blkiocg_reset_stats(struct cgroup *cgrou
#endif
blkcg = cgroup_to_blkio_cgroup(cgroup);
+ spin_lock(&blkio_list_lock);
spin_lock_irq(&blkcg->lock);
hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node) {
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkio_policy_type *pol;
- spin_lock(&blkg->stats_lock);
- stats = &pd->stats;
+ list_for_each_entry(pol, &blkio_list, list) {
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
+
+ spin_lock(&blkg->stats_lock);
+ stats = &pd->stats;
#ifdef CONFIG_DEBUG_BLK_CGROUP
- idling = blkio_blkg_idling(stats);
- waiting = blkio_blkg_waiting(stats);
- empty = blkio_blkg_empty(stats);
+ idling = blkio_blkg_idling(stats);
+ waiting = blkio_blkg_waiting(stats);
+ empty = blkio_blkg_empty(stats);
#endif
- for (i = 0; i < BLKIO_STAT_TOTAL; i++)
- queued[i] = stats->stat_arr[BLKIO_STAT_QUEUED][i];
- memset(stats, 0, sizeof(struct blkio_group_stats));
- for (i = 0; i < BLKIO_STAT_TOTAL; i++)
- stats->stat_arr[BLKIO_STAT_QUEUED][i] = queued[i];
+ for (i = 0; i < BLKIO_STAT_TOTAL; i++)
+ queued[i] = stats->stat_arr[BLKIO_STAT_QUEUED][i];
+ memset(stats, 0, sizeof(struct blkio_group_stats));
+ for (i = 0; i < BLKIO_STAT_TOTAL; i++)
+ stats->stat_arr[BLKIO_STAT_QUEUED][i] = queued[i];
#ifdef CONFIG_DEBUG_BLK_CGROUP
- if (idling) {
- blkio_mark_blkg_idling(stats);
- stats->start_idle_time = now;
- }
- if (waiting) {
- blkio_mark_blkg_waiting(stats);
- stats->start_group_wait_time = now;
- }
- if (empty) {
- blkio_mark_blkg_empty(stats);
- stats->start_empty_time = now;
- }
+ if (idling) {
+ blkio_mark_blkg_idling(stats);
+ stats->start_idle_time = now;
+ }
+ if (waiting) {
+ blkio_mark_blkg_waiting(stats);
+ stats->start_group_wait_time = now;
+ }
+ if (empty) {
+ blkio_mark_blkg_empty(stats);
+ stats->start_empty_time = now;
+ }
#endif
- spin_unlock(&blkg->stats_lock);
+ spin_unlock(&blkg->stats_lock);
- /* Reset Per cpu stats which don't take blkg->stats_lock */
- blkio_reset_stats_cpu(blkg, blkg->plid);
+ /* Reset Per cpu stats which don't take blkg->stats_lock */
+ blkio_reset_stats_cpu(blkg, pol->plid);
+ }
}
spin_unlock_irq(&blkcg->lock);
+ spin_unlock(&blkio_list_lock);
return 0;
}
@@ -1157,8 +1173,7 @@ static void blkio_read_conf(struct cftyp
spin_lock_irq(&blkcg->lock);
hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node)
- if (BLKIOFILE_POLICY(cft->private) == blkg->plid)
- blkio_print_group_conf(cft, blkg, m);
+ blkio_print_group_conf(cft, blkg, m);
spin_unlock_irq(&blkcg->lock);
}
@@ -1213,8 +1228,6 @@ static int blkio_read_blkg_stats(struct
const char *dname = dev_name(blkg->q->backing_dev_info.dev);
int plid = BLKIOFILE_POLICY(cft->private);
- if (plid != blkg->plid)
- continue;
if (pcpu) {
cgroup_total += blkio_get_stat_cpu(blkg, plid,
cb, dname, type);
@@ -1324,9 +1337,9 @@ static int blkio_weight_write(struct blk
blkcg->weight = (unsigned int)val;
hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node) {
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[plid];
- if (blkg->plid == plid && !pd->conf.weight)
+ if (!pd->conf.weight)
blkio_update_group_weight(blkg, plid, blkcg->weight);
}
@@ -1549,7 +1562,6 @@ static int blkiocg_pre_destroy(struct cg
unsigned long flags;
struct blkio_group *blkg;
struct request_queue *q;
- struct blkio_policy_type *blkiop;
rcu_read_lock();
@@ -1575,11 +1587,7 @@ static int blkiocg_pre_destroy(struct cg
*/
spin_lock(&blkio_list_lock);
spin_lock_irqsave(q->queue_lock, flags);
- list_for_each_entry(blkiop, &blkio_list, list) {
- if (blkiop->plid != blkg->plid)
- continue;
- blkg_destroy(blkg, blkiop->plid);
- }
+ blkg_destroy(blkg);
spin_unlock_irqrestore(q->queue_lock, flags);
spin_unlock(&blkio_list_lock);
} while (1);
@@ -1673,6 +1681,8 @@ void blkcg_exit_queue(struct request_que
list_del_init(&q->all_q_node);
mutex_unlock(&all_q_mutex);
+ blkg_destroy_all(q, true);
+
blk_throtl_exit(q);
}
@@ -1722,14 +1732,12 @@ static void blkcg_bypass_start(void)
__acquires(&all_q_mutex)
{
struct request_queue *q;
- int i;
mutex_lock(&all_q_mutex);
list_for_each_entry(q, &all_q_list, all_q_node) {
blk_queue_bypass_start(q);
- for (i = 0; i < BLKIO_NR_POLICIES; i++)
- blkg_destroy_all(q, i, false);
+ blkg_destroy_all(q, false);
}
}
@@ -1744,6 +1752,39 @@ static void blkcg_bypass_end(void)
mutex_unlock(&all_q_mutex);
}
+/*
+ * XXX: This updates blkg policy data in-place for root blkg, which is
+ * necessary across policy registration as root blkgs aren't shot down.
+ * This broken and racy implementation is interim. Eventually, blkg shoot
+ * down will be replaced by proper in-place update.
+ */
+static void update_root_blkgs(enum blkio_policy_id plid)
+{
+ struct blkio_policy_type *pol = blkio_policy[plid];
+ struct request_queue *q;
+
+ list_for_each_entry(q, &all_q_list, all_q_node) {
+ struct blkio_group *blkg = blkg_lookup(&blkio_root_cgroup, q);
+ struct blkg_policy_data *pd;
+
+ kfree(blkg->pd[plid]);
+ blkg->pd[plid] = NULL;
+
+ if (!pol)
+ continue;
+
+ pd = kzalloc(sizeof(*pd) + pol->pdata_size, GFP_KERNEL);
+ WARN_ON_ONCE(!pd);
+
+ pd->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
+ WARN_ON_ONCE(!pd->stats_cpu);
+
+ blkg->pd[plid] = pd;
+ pd->blkg = blkg;
+ pol->ops.blkio_init_group_fn(blkg);
+ }
+}
+
void blkio_policy_register(struct blkio_policy_type *blkiop)
{
blkcg_bypass_start();
@@ -1754,6 +1795,7 @@ void blkio_policy_register(struct blkio_
list_add_tail(&blkiop->list, &blkio_list);
spin_unlock(&blkio_list_lock);
+ update_root_blkgs(blkiop->plid);
blkcg_bypass_end();
}
EXPORT_SYMBOL_GPL(blkio_policy_register);
@@ -1768,6 +1810,7 @@ void blkio_policy_unregister(struct blki
list_del_init(&blkiop->list);
spin_unlock(&blkio_list_lock);
+ update_root_blkgs(blkiop->plid);
blkcg_bypass_end();
}
EXPORT_SYMBOL_GPL(blkio_policy_unregister);
Index: work/block/blk-cgroup.h
===================================================================
--- work.orig/block/blk-cgroup.h
+++ work/block/blk-cgroup.h
@@ -178,13 +178,11 @@ struct blkg_policy_data {
struct blkio_group {
/* Pointer to the associated request_queue, RCU protected */
struct request_queue __rcu *q;
- struct list_head q_node[BLKIO_NR_POLICIES];
+ struct list_head q_node;
struct hlist_node blkcg_node;
struct blkio_cgroup *blkcg;
/* Store cgroup path */
char path[128];
- /* policy which owns this blk group */
- enum blkio_policy_id plid;
/* reference count */
int refcnt;
@@ -230,8 +228,7 @@ extern void blkcg_exit_queue(struct requ
/* Blkio controller policy registration */
extern void blkio_policy_register(struct blkio_policy_type *);
extern void blkio_policy_unregister(struct blkio_policy_type *);
-extern void blkg_destroy_all(struct request_queue *q,
- enum blkio_policy_id plid, bool destroy_root);
+extern void blkg_destroy_all(struct request_queue *q, bool destroy_root);
/**
* blkg_to_pdata - get policy private data
@@ -382,8 +379,7 @@ extern struct blkio_cgroup *cgroup_to_bl
extern struct blkio_cgroup *task_blkio_cgroup(struct task_struct *tsk);
extern int blkiocg_del_blkio_group(struct blkio_group *blkg);
extern struct blkio_group *blkg_lookup(struct blkio_cgroup *blkcg,
- struct request_queue *q,
- enum blkio_policy_id plid);
+ struct request_queue *q);
struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
struct request_queue *q,
enum blkio_policy_id plid,
Index: work/block/blk-throttle.c
===================================================================
--- work.orig/block/blk-throttle.c
+++ work/block/blk-throttle.c
@@ -167,7 +167,7 @@ throtl_grp *throtl_lookup_tg(struct thro
if (blkcg == &blkio_root_cgroup)
return td->root_tg;
- return blkg_to_tg(blkg_lookup(blkcg, td->queue, BLKIO_POLICY_THROTL));
+ return blkg_to_tg(blkg_lookup(blkcg, td->queue));
}
static struct throtl_grp *throtl_lookup_create_tg(struct throtl_data *td,
@@ -704,8 +704,7 @@ static void throtl_process_limit_change(
throtl_log(td, "limits changed");
- list_for_each_entry_safe(blkg, n, &q->blkg_list[BLKIO_POLICY_THROTL],
- q_node[BLKIO_POLICY_THROTL]) {
+ list_for_each_entry_safe(blkg, n, &q->blkg_list, q_node) {
struct throtl_grp *tg = blkg_to_tg(blkg);
if (!tg->limits_changed)
@@ -1054,11 +1053,9 @@ void blk_throtl_exit(struct request_queu
throtl_shutdown_wq(q);
- blkg_destroy_all(q, BLKIO_POLICY_THROTL, true);
-
/* If there are other groups */
spin_lock_irq(q->queue_lock);
- wait = q->nr_blkgs[BLKIO_POLICY_THROTL];
+ wait = q->nr_blkgs;
spin_unlock_irq(q->queue_lock);
/*
Index: work/block/cfq-iosched.c
===================================================================
--- work.orig/block/cfq-iosched.c
+++ work/block/cfq-iosched.c
@@ -3468,14 +3468,12 @@ static void cfq_exit_queue(struct elevat
spin_unlock_irq(q->queue_lock);
- blkg_destroy_all(q, BLKIO_POLICY_PROP, true);
-
/*
* If there are groups which we could not unlink from blkcg list,
* wait for a rcu period for them to be freed.
*/
spin_lock_irq(q->queue_lock);
- wait = q->nr_blkgs[BLKIO_POLICY_PROP];
+ wait = q->nr_blkgs;
spin_unlock_irq(q->queue_lock);
cfq_shutdown_timer_wq(cfqd);
Index: work/block/blk-core.c
===================================================================
--- work.orig/block/blk-core.c
+++ work/block/blk-core.c
@@ -547,8 +547,7 @@ struct request_queue *blk_alloc_queue_no
INIT_LIST_HEAD(&q->timeout_list);
INIT_LIST_HEAD(&q->icq_list);
#if defined(CONFIG_BLK_CGROUP) || defined(CONFIG_BLK_CGROUP_MODULE)
- INIT_LIST_HEAD(&q->blkg_list[0]);
- INIT_LIST_HEAD(&q->blkg_list[1]);
+ INIT_LIST_HEAD(&q->blkg_list);
#endif
INIT_LIST_HEAD(&q->flush_queue[0]);
INIT_LIST_HEAD(&q->flush_queue[1]);
Index: work/block/elevator.c
===================================================================
--- work.orig/block/elevator.c
+++ work/block/elevator.c
@@ -923,7 +923,7 @@ static int elevator_switch(struct reques
{
struct elevator_queue *old = q->elevator;
bool registered = old->registered;
- int i, err;
+ int err;
/*
* Turn on BYPASS and drain all requests w/ elevator private data.
@@ -942,8 +942,7 @@ static int elevator_switch(struct reques
ioc_clear_queue(q);
spin_unlock_irq(q->queue_lock);
- for (i = 0; i < BLKIO_NR_POLICIES; i++)
- blkg_destroy_all(q, i, false);
+ blkg_destroy_all(q, false);
/* allocate, init and register new elevator */
err = -ENOMEM;
Index: work/include/linux/blkdev.h
===================================================================
--- work.orig/include/linux/blkdev.h
+++ work/include/linux/blkdev.h
@@ -363,9 +363,8 @@ struct request_queue {
struct list_head icq_list;
#if defined(CONFIG_BLK_CGROUP) || defined(CONFIG_BLK_CGROUP_MODULE)
- /* XXX: array size hardcoded to avoid include dependency (temporary) */
- struct list_head blkg_list[2];
- int nr_blkgs[2];
+ struct list_head blkg_list;
+ int nr_blkgs;
#endif
struct queue_limits limits;
Index: work/block/blk-sysfs.c
===================================================================
--- work.orig/block/blk-sysfs.c
+++ work/block/blk-sysfs.c
@@ -480,6 +480,8 @@ static void blk_release_queue(struct kob
blk_sync_queue(q);
+ blkcg_exit_queue(q);
+
if (q->elevator) {
spin_lock_irq(q->queue_lock);
ioc_clear_queue(q);
@@ -487,8 +489,6 @@ static void blk_release_queue(struct kob
elevator_exit(q->elevator);
}
- blkcg_exit_queue(q);
-
if (rl->rq_pool)
mempool_destroy(rl->rq_pool);
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCHSET] blkcg: unify blkgs for different policies
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
` (10 preceding siblings ...)
2012-02-01 21:19 ` [PATCH 11/11] blkcg: unify blkg's for blkcg policies Tejun Heo
@ 2012-02-02 19:29 ` Vivek Goyal
2012-02-02 20:36 ` Tejun Heo
11 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-02 19:29 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Wed, Feb 01, 2012 at 01:19:05PM -0800, Tejun Heo wrote:
[..]
>
> * use unified stats updated under queue lock and drop percpu stats
> which should fix locking / context bug across percpu allocation.
Hi Tejun,
Does that mean that stat updation will happen under queue lock even if
there are no throttling rules? That will introduce extra queue lock on
fast path those who have throttling compiled in but are not using (common
case for distributions).
IMHO, we should keep the lockless per cpu stats and do the allocation in
worker thread. I was going through my messages and noticed that for
a workload queue lock contention had come down by 11% and in a separate
testing I could gain 4-5% with PCIe based flash drive for random reads
while I saturated the cpus.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 01/11] blkcg: let blkio_group point to blkio_cgroup directly
2012-02-01 21:19 ` [PATCH 01/11] blkcg: let blkio_group point to blkio_cgroup directly Tejun Heo
@ 2012-02-02 20:03 ` Vivek Goyal
2012-02-02 20:33 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-02 20:03 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Wed, Feb 01, 2012 at 01:19:06PM -0800, Tejun Heo wrote:
> Currently, blkg points to the associated blkcg via its css_id. This
> unnecessarily complicates dereferencing blkcg. Let blkg hold a
> reference to the associated blkcg and point directly to it and disable
> css_id on blkio_subsys.
>
> This change requires splitting blkiocg_destroy() into
> blkiocg_pre_destroy() and blkiocg_destroy() so that all blkg's can be
> destroyed and all the blkcg references held by them dropped during
> cgroup removal.
>
> Signed-off-by: Tejun Heo <tj@kernel.org>
> Cc: Vivek Goyal <vgoyal@redhat.com>
[..]
> +static void blkiocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)
> +{
> + struct blkio_cgroup *blkcg = cgroup_to_blkio_cgroup(cgroup);
> +
> if (blkcg != &blkio_root_cgroup)
> kfree(blkcg);
Hi Tejun,
What makes sure that all the blkg are gone and they have dropped their
reference to blkcg? IIUC, pre-destroy will just make sure to decouple
blkg from request queue as well as blkcg list and also drop joint cgroup
and request queue reference.
But there could well be some IO queued in the group which might have
its own reference and will be dropped later when IO completes. So at
the time of blkiocg_destroy() it is not guranteed that there are
no reference holders to blkcg?
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 02/11] block: relocate elevator initialized test from blk_cleanup_queue() to blk_drain_queue()
2012-02-01 21:19 ` [PATCH 02/11] block: relocate elevator initialized test from blk_cleanup_queue() to blk_drain_queue() Tejun Heo
@ 2012-02-02 20:20 ` Vivek Goyal
2012-02-02 20:35 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-02 20:20 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Wed, Feb 01, 2012 at 01:19:07PM -0800, Tejun Heo wrote:
> blk_cleanup_queue() may be called for a queue which doesn't have
> elevator initialized to skip invoking blk_drain_queue().
> blk_drain_queue() will be used for other purposes too and may be
> called for such half-initialized queues from other paths. Move
> q->elevator test into blk_drain_queue().
>
> Signed-off-by: Tejun Heo <tj@kernel.org>
> Cc: Vivek Goyal <vgoyal@redhat.com>
> ---
> block/blk-core.c | 16 +++++++++-------
> 1 files changed, 9 insertions(+), 7 deletions(-)
>
> diff --git a/block/blk-core.c b/block/blk-core.c
> index 1b73d06..ddda1cc 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -359,6 +359,13 @@ EXPORT_SYMBOL(blk_put_queue);
> */
> void blk_drain_queue(struct request_queue *q, bool drain_all)
> {
> + /*
> + * The caller might be trying to tear down @q before its elevator
> + * is initialized, in which case we don't want to call into it.
> + */
> + if (!q->elevator)
> + return;
> +
Shouldn't blk_throtl_drain() be called irrespective of the fact whether
elevator is initilialized or not? On bio based drivers, there will be
no elevator but bios can still be throttled.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 01/11] blkcg: let blkio_group point to blkio_cgroup directly
2012-02-02 20:03 ` Vivek Goyal
@ 2012-02-02 20:33 ` Tejun Heo
2012-02-02 20:55 ` Vivek Goyal
0 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-02 20:33 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
Hello,
On Thu, Feb 02, 2012 at 03:03:18PM -0500, Vivek Goyal wrote:
> > +static void blkiocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)
> > +{
> > + struct blkio_cgroup *blkcg = cgroup_to_blkio_cgroup(cgroup);
> > +
> > if (blkcg != &blkio_root_cgroup)
> > kfree(blkcg);
>
> What makes sure that all the blkg are gone and they have dropped their
> reference to blkcg? IIUC, pre-destroy will just make sure to decouple
> blkg from request queue as well as blkcg list and also drop joint cgroup
> and request queue reference.
>
> But there could well be some IO queued in the group which might have
> its own reference and will be dropped later when IO completes. So at
> the time of blkiocg_destroy() it is not guranteed that there are
> no reference holders to blkcg?
Yeah, and that would wait till all css refs held by blkgs are gone,
right?
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 02/11] block: relocate elevator initialized test from blk_cleanup_queue() to blk_drain_queue()
2012-02-02 20:20 ` Vivek Goyal
@ 2012-02-02 20:35 ` Tejun Heo
2012-02-02 20:37 ` Vivek Goyal
0 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-02 20:35 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
Hello,
On Thu, Feb 02, 2012 at 03:20:42PM -0500, Vivek Goyal wrote:
> > @@ -359,6 +359,13 @@ EXPORT_SYMBOL(blk_put_queue);
> > */
> > void blk_drain_queue(struct request_queue *q, bool drain_all)
> > {
> > + /*
> > + * The caller might be trying to tear down @q before its elevator
> > + * is initialized, in which case we don't want to call into it.
> > + */
> > + if (!q->elevator)
> > + return;
> > +
>
> Shouldn't blk_throtl_drain() be called irrespective of the fact whether
> elevator is initilialized or not? On bio based drivers, there will be
> no elevator but bios can still be throttled.
Hmmm... probably, but doesn't that mean the code is already broken for
bio based drivers using throtl?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCHSET] blkcg: unify blkgs for different policies
2012-02-02 19:29 ` [PATCHSET] blkcg: unify blkgs for different policies Vivek Goyal
@ 2012-02-02 20:36 ` Tejun Heo
2012-02-02 20:43 ` Vivek Goyal
0 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-02 20:36 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
Hello,
On Thu, Feb 02, 2012 at 02:29:58PM -0500, Vivek Goyal wrote:
> On Wed, Feb 01, 2012 at 01:19:05PM -0800, Tejun Heo wrote:
>
> [..]
> >
> > * use unified stats updated under queue lock and drop percpu stats
> > which should fix locking / context bug across percpu allocation.
>
> Does that mean that stat updation will happen under queue lock even if
> there are no throttling rules? That will introduce extra queue lock on
> fast path those who have throttling compiled in but are not using (common
> case for distributions).
No, I don't think extra locking would be necessary at all. We end up
grabbing queue_lock anyway. It's just the matter of where to update
the stats.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 02/11] block: relocate elevator initialized test from blk_cleanup_queue() to blk_drain_queue()
2012-02-02 20:35 ` Tejun Heo
@ 2012-02-02 20:37 ` Vivek Goyal
2012-02-02 20:38 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-02 20:37 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Thu, Feb 02, 2012 at 12:35:13PM -0800, Tejun Heo wrote:
> Hello,
>
> On Thu, Feb 02, 2012 at 03:20:42PM -0500, Vivek Goyal wrote:
> > > @@ -359,6 +359,13 @@ EXPORT_SYMBOL(blk_put_queue);
> > > */
> > > void blk_drain_queue(struct request_queue *q, bool drain_all)
> > > {
> > > + /*
> > > + * The caller might be trying to tear down @q before its elevator
> > > + * is initialized, in which case we don't want to call into it.
> > > + */
> > > + if (!q->elevator)
> > > + return;
> > > +
> >
> > Shouldn't blk_throtl_drain() be called irrespective of the fact whether
> > elevator is initilialized or not? On bio based drivers, there will be
> > no elevator but bios can still be throttled.
>
> Hmmm... probably, but doesn't that mean the code is already broken for
> bio based drivers using throtl?
Yes looks like it is already broken for bio based drivers. So it is a fix
independent of this patch/patch series.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 02/11] block: relocate elevator initialized test from blk_cleanup_queue() to blk_drain_queue()
2012-02-02 20:37 ` Vivek Goyal
@ 2012-02-02 20:38 ` Tejun Heo
0 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-02 20:38 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
On Thu, Feb 02, 2012 at 03:37:08PM -0500, Vivek Goyal wrote:
> > Hmmm... probably, but doesn't that mean the code is already broken for
> > bio based drivers using throtl?
>
> Yes looks like it is already broken for bio based drivers. So it is a fix
> independent of this patch/patch series.
Yeah, I'll write up a separate fix patch for mainline.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCHSET] blkcg: unify blkgs for different policies
2012-02-02 20:36 ` Tejun Heo
@ 2012-02-02 20:43 ` Vivek Goyal
2012-02-02 20:59 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-02 20:43 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Thu, Feb 02, 2012 at 12:36:09PM -0800, Tejun Heo wrote:
> Hello,
>
> On Thu, Feb 02, 2012 at 02:29:58PM -0500, Vivek Goyal wrote:
> > On Wed, Feb 01, 2012 at 01:19:05PM -0800, Tejun Heo wrote:
> >
> > [..]
> > >
> > > * use unified stats updated under queue lock and drop percpu stats
> > > which should fix locking / context bug across percpu allocation.
> >
> > Does that mean that stat updation will happen under queue lock even if
> > there are no throttling rules? That will introduce extra queue lock on
> > fast path those who have throttling compiled in but are not using (common
> > case for distributions).
>
> No, I don't think extra locking would be necessary at all. We end up
> grabbing queue_lock anyway. It's just the matter of where to update
> the stats.
What about bio based drivers? They might have their own internal locking
and not relying on queue lock. And conceptually, we first throttle the
bio, update the stats and then call into driver. So are you planning
to move blk_throtl_bio() call into request function of individual driver
to avoid taking queue lock twice.
In fact even for request based drivers, if bio is being merged onto
existing request on plug, we will not take any queue lock. So I am not
sure how would you avoid extra queue lock for this case.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 01/11] blkcg: let blkio_group point to blkio_cgroup directly
2012-02-02 20:33 ` Tejun Heo
@ 2012-02-02 20:55 ` Vivek Goyal
0 siblings, 0 replies; 42+ messages in thread
From: Vivek Goyal @ 2012-02-02 20:55 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Thu, Feb 02, 2012 at 12:33:52PM -0800, Tejun Heo wrote:
> Hello,
>
> On Thu, Feb 02, 2012 at 03:03:18PM -0500, Vivek Goyal wrote:
> > > +static void blkiocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)
> > > +{
> > > + struct blkio_cgroup *blkcg = cgroup_to_blkio_cgroup(cgroup);
> > > +
> > > if (blkcg != &blkio_root_cgroup)
> > > kfree(blkcg);
> >
> > What makes sure that all the blkg are gone and they have dropped their
> > reference to blkcg? IIUC, pre-destroy will just make sure to decouple
> > blkg from request queue as well as blkcg list and also drop joint cgroup
> > and request queue reference.
> >
> > But there could well be some IO queued in the group which might have
> > its own reference and will be dropped later when IO completes. So at
> > the time of blkiocg_destroy() it is not guranteed that there are
> > no reference holders to blkcg?
>
> Yeah, and that would wait till all css refs held by blkgs are gone,
> right?
Ok, I missed that. So ->destroy() is not called till all the css refs
are gone. So directory removal will wait for all the pending IO in the
group to finish.
I have some minor concerns here. A low prio group might not be able to
dispatch IO for quite some amount of time. Especially with CFQ, in presence of
sync IO, async IO can be starved for a very long time. So if some task
dumped bunch of low prio IO and exited and now we are removing the cgroup,
this task might have to wait for significant amount of time. (I am worried
about "hung task waited for 120 seconds kind of messages").
Also it is little unintutive to the user that why rmdir should be delayed.
Anyway, this is not a very strong concern. Just something to keep in mind.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCHSET] blkcg: unify blkgs for different policies
2012-02-02 20:43 ` Vivek Goyal
@ 2012-02-02 20:59 ` Tejun Heo
0 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-02 20:59 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
Hello,
On Thu, Feb 02, 2012 at 03:43:42PM -0500, Vivek Goyal wrote:
> What about bio based drivers? They might have their own internal locking
> and not relying on queue lock. And conceptually, we first throttle the
> bio, update the stats and then call into driver. So are you planning
> to move blk_throtl_bio() call into request function of individual driver
> to avoid taking queue lock twice.
>
> In fact even for request based drivers, if bio is being merged onto
> existing request on plug, we will not take any queue lock. So I am not
> sure how would you avoid extra queue lock for this case.
For request based drivers, it isn't an issue. If tracking info across
plug merging is necessary, just record the states in the plug
structure. The whole purpose of per-task plugging is exactly that -
buffering state before going through request_lock after all.
For ->make_request_fn() drivers, I'm not sure yet and will have to
look through different bio based drivers before deciding. At the
worst, we might require them to call stat update function under queue
lock or maybe provide fuzzier update or a way to opt out of per-blkg
stats. Let's see.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 06/11] blkcg: move refcnt to blkcg core
2012-02-01 21:19 ` [PATCH 06/11] blkcg: move refcnt to blkcg core Tejun Heo
@ 2012-02-02 22:07 ` Vivek Goyal
2012-02-02 22:11 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-02 22:07 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Wed, Feb 01, 2012 at 01:19:11PM -0800, Tejun Heo wrote:
> Currently, blkcg policy implementations manage blkg refcnt duplicating
> mostly identical code in both policies. This patch moves refcnt to
> blkg and let blkcg core handle refcnt and freeing of blkgs.
>
> * cfq blkgs now also get freed via RCU.
This can lead to situation where cfq root group (policy data) is still
around (yet to be freed after rcu perioed) but cfq has gone away
(cfq_exit_queue() followed by cfq_exit()). Does it matter? If some future
code is accessing cfqg under rcu, it can become a problem.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 06/11] blkcg: move refcnt to blkcg core
2012-02-02 22:07 ` Vivek Goyal
@ 2012-02-02 22:11 ` Tejun Heo
0 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-02 22:11 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
Hello,
On Thu, Feb 02, 2012 at 05:07:14PM -0500, Vivek Goyal wrote:
> On Wed, Feb 01, 2012 at 01:19:11PM -0800, Tejun Heo wrote:
> > Currently, blkcg policy implementations manage blkg refcnt duplicating
> > mostly identical code in both policies. This patch moves refcnt to
> > blkg and let blkcg core handle refcnt and freeing of blkgs.
> >
> > * cfq blkgs now also get freed via RCU.
>
> This can lead to situation where cfq root group (policy data) is still
> around (yet to be freed after rcu perioed) but cfq has gone away
> (cfq_exit_queue() followed by cfq_exit()). Does it matter? If some future
> code is accessing cfqg under rcu, it can become a problem.
If RCU is being used, the scope of access allowed under it should be
defined strictly. It's basically adding another layer of locking and
like any other locking, if the boundary isn't clearly defined, it's
gonna cause a problem. For blkcg, from the current usage, it's enough
to guarantee access to blkg proper including policy specific data and
under that scope, it doesn't matter whether request or cgroup goes
away.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 09/11] blkcg: move per-queue blkg list heads and counters to queue and blkg
2012-02-01 21:19 ` [PATCH 09/11] blkcg: move per-queue blkg list heads and counters to queue and blkg Tejun Heo
@ 2012-02-02 22:47 ` Vivek Goyal
2012-02-02 22:47 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-02 22:47 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Wed, Feb 01, 2012 at 01:19:14PM -0800, Tejun Heo wrote:
[..]
> diff --git a/block/blk-core.c b/block/blk-core.c
> index ab0a11b..025ef60 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -546,6 +546,10 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id)
> setup_timer(&q->timeout, blk_rq_timed_out_timer, (unsigned long) q);
> INIT_LIST_HEAD(&q->timeout_list);
> INIT_LIST_HEAD(&q->icq_list);
> +#if defined(CONFIG_BLK_CGROUP) || defined(CONFIG_BLK_CGROUP_MODULE)
BLK_CGROUP_MODULE not required as you got rid of this possiblity in the
first patch itself
>
> struct list_head icq_list;
> +#if defined(CONFIG_BLK_CGROUP) || defined(CONFIG_BLK_CGROUP_MODULE)
Ditto.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH 09/11] blkcg: move per-queue blkg list heads and counters to queue and blkg
2012-02-02 22:47 ` Vivek Goyal
@ 2012-02-02 22:47 ` Tejun Heo
0 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-02 22:47 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
Hello,
On Thu, Feb 2, 2012 at 2:47 PM, Vivek Goyal <vgoyal@redhat.com> wrote:
> On Wed, Feb 01, 2012 at 01:19:14PM -0800, Tejun Heo wrote:
>
> [..]
>> diff --git a/block/blk-core.c b/block/blk-core.c
>> index ab0a11b..025ef60 100644
>> --- a/block/blk-core.c
>> +++ b/block/blk-core.c
>> @@ -546,6 +546,10 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id)
>> setup_timer(&q->timeout, blk_rq_timed_out_timer, (unsigned long) q);
>> INIT_LIST_HEAD(&q->timeout_list);
>> INIT_LIST_HEAD(&q->icq_list);
>> +#if defined(CONFIG_BLK_CGROUP) || defined(CONFIG_BLK_CGROUP_MODULE)
>
> BLK_CGROUP_MODULE not required as you got rid of this possiblity in the
> first patch itself
>
>>
>> struct list_head icq_list;
>> +#if defined(CONFIG_BLK_CGROUP) || defined(CONFIG_BLK_CGROUP_MODULE)
>
> Ditto.
Ooh, right. Will remove them. Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-02 0:37 ` [PATCH UPDATED " Tejun Heo
@ 2012-02-03 19:41 ` Vivek Goyal
2012-02-03 20:59 ` Tejun Heo
2012-02-03 21:06 ` Vivek Goyal
2012-02-14 1:33 ` [PATCH UPDATED2 " Tejun Heo
2 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-03 19:41 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Wed, Feb 01, 2012 at 04:37:30PM -0800, Tejun Heo wrote:
> Currently, blkg is per cgroup-queue-policy combination. This is
> unnatural and leads to various convolutions in partially used
> duplicate fields in blkg, config / stat access, and general management
> of blkgs.
>
> This patch make blkg's per cgroup-queue and let them serve all
> policies. blkgs are now created and destroyed by blkcg core proper.
> This will allow further consolidation of common management logic into
> blkcg core and API with better defined semantics and layering.
>
> As a transitional step to untangle blkg management, elvswitch and
> policy [de]registration, all blkgs except the root blkg are being shot
> down during elvswitch and bypass. This patch adds blkg_root_update()
> to update root blkg in place on policy change. This is hacky and racy
> but should be good enough as interim step until we get locking
> simplified and switch over to proper in-place update for all blkgs.
- So we don't shoot down root group over elevator switch and policy
changes because we are not sure if we will be able to alloc new
group? It is not like elevator where we don't free the old one till
we have made sure that new one is allocated and initialized properly.
- I am assuming that we will change blkg_destroy_all() later to also
take policy as argument and only destroy policy data of respective
policy and not the whole group. (Well I guess we can destroy the whole
group if it was only policy on the group).
[..]
> static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
> - struct request_queue *q,
> - struct blkio_policy_type *pol)
> + struct request_queue *q)
Comment before this function still mentions "pol" as function argument.
[..]
> @@ -776,43 +786,49 @@ blkiocg_reset_stats(struct cgroup *cgrou
> #endif
>
> blkcg = cgroup_to_blkio_cgroup(cgroup);
> + spin_lock(&blkio_list_lock);
> spin_lock_irq(&blkcg->lock);
Isn't blkcg lock enough to protect against policy registration/deregistration.
A policy can not add/delete a group to cgroup list without blkcg list.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-03 19:41 ` Vivek Goyal
@ 2012-02-03 20:59 ` Tejun Heo
2012-02-03 21:44 ` Vivek Goyal
0 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-03 20:59 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
Hey, Vivek.
On Fri, Feb 03, 2012 at 02:41:05PM -0500, Vivek Goyal wrote:
> On Wed, Feb 01, 2012 at 04:37:30PM -0800, Tejun Heo wrote:
> > As a transitional step to untangle blkg management, elvswitch and
> > policy [de]registration, all blkgs except the root blkg are being shot
> > down during elvswitch and bypass. This patch adds blkg_root_update()
> > to update root blkg in place on policy change. This is hacky and racy
> > but should be good enough as interim step until we get locking
> > simplified and switch over to proper in-place update for all blkgs.
>
> - So we don't shoot down root group over elevator switch and policy
> changes because we are not sure if we will be able to alloc new
> group? It is not like elevator where we don't free the old one till
> we have made sure that new one is allocated and initialized properly.
No, because we policies cache root group and we don't have mechanism
to update them. I could have added that but root group management
should be moved to blkcg core anyway and in-place update will be
applied to all blkgs, so I just chose a dirty shortcut as an interim
step.
> - I am assuming that we will change blkg_destroy_all() later to also
> take policy as argument and only destroy policy data of respective
> policy and not the whole group. (Well I guess we can destroy the whole
> group if it was only policy on the group).
Yeap, that's what's scheduled.
> [..]
> > static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
> > - struct request_queue *q,
> > - struct blkio_policy_type *pol)
> > + struct request_queue *q)
>
> Comment before this function still mentions "pol" as function argument.
Will update.
> [..]
> > @@ -776,43 +786,49 @@ blkiocg_reset_stats(struct cgroup *cgrou
> > #endif
> >
> > blkcg = cgroup_to_blkio_cgroup(cgroup);
> > + spin_lock(&blkio_list_lock);
> > spin_lock_irq(&blkcg->lock);
>
> Isn't blkcg lock enough to protect against policy registration/deregistration.
> A policy can not add/delete a group to cgroup list without blkcg list.
But pol list can change regardless of that, no?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-02 0:37 ` [PATCH UPDATED " Tejun Heo
2012-02-03 19:41 ` Vivek Goyal
@ 2012-02-03 21:06 ` Vivek Goyal
2012-02-03 21:09 ` Tejun Heo
2012-02-14 1:33 ` [PATCH UPDATED2 " Tejun Heo
2 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-03 21:06 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Wed, Feb 01, 2012 at 04:37:30PM -0800, Tejun Heo wrote:
> Currently, blkg is per cgroup-queue-policy combination. This is
> unnatural and leads to various convolutions in partially used
> duplicate fields in blkg, config / stat access, and general management
> of blkgs.
>
> This patch make blkg's per cgroup-queue and let them serve all
> policies. blkgs are now created and destroyed by blkcg core proper.
> This will allow further consolidation of common management logic into
> blkcg core and API with better defined semantics and layering.
>
> As a transitional step to untangle blkg management, elvswitch and
> policy [de]registration, all blkgs except the root blkg are being shot
> down during elvswitch and bypass. This patch adds blkg_root_update()
> to update root blkg in place on policy change. This is hacky and racy
> but should be good enough as interim step until we get locking
> simplified and switch over to proper in-place update for all blkgs.
>
> Signed-off-by: Tejun Heo <tj@kernel.org>
> Cc: Vivek Goyal <vgoyal@redhat.com>
> ---
> I was thinking root blkgs should be updated on policy changes and
> commit message and comments were written that way too but for some
> reason I wrote code which updates on elvswitch, which leads to oops on
> root blkg access after policy changes (ie. insmod cfq-iosched.ko).
> Fixed. git branch updated accordingly.
Tejun,
Shouldn't we shoot down root group policy data upon elevator switch?
Otherwise once the elevator comes back, it will continue to use old
policy data (old stats, overwrite old weigts etc.).
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-03 21:06 ` Vivek Goyal
@ 2012-02-03 21:09 ` Tejun Heo
2012-02-03 21:10 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-03 21:09 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
Hello,
On Fri, Feb 03, 2012 at 04:06:56PM -0500, Vivek Goyal wrote:
> Shouldn't we shoot down root group policy data upon elevator switch?
> Otherwise once the elevator comes back, it will continue to use old
> policy data (old stats, overwrite old weigts etc.).
It's racy and hacky but we do shoot down policy specific data and
reinit them. The behavior should be fine although implementation is
seriously ugly for now.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-03 21:09 ` Tejun Heo
@ 2012-02-03 21:10 ` Tejun Heo
0 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-03 21:10 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
On Fri, Feb 03, 2012 at 01:09:22PM -0800, Tejun Heo wrote:
> Hello,
>
> On Fri, Feb 03, 2012 at 04:06:56PM -0500, Vivek Goyal wrote:
> > Shouldn't we shoot down root group policy data upon elevator switch?
> > Otherwise once the elevator comes back, it will continue to use old
> > policy data (old stats, overwrite old weigts etc.).
>
> It's racy and hacky but we do shoot down policy specific data and
> reinit them. The behavior should be fine although implementation is
> seriously ugly for now.
Oops, sorry, got confused again about elvswitch and pol registration.
Hmmm... yeah, we should probably be doing about the same thing on
elvswitch. I'll look into it.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-03 20:59 ` Tejun Heo
@ 2012-02-03 21:44 ` Vivek Goyal
2012-02-03 21:47 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-03 21:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Fri, Feb 03, 2012 at 12:59:10PM -0800, Tejun Heo wrote:
>
> > [..]
> > > @@ -776,43 +786,49 @@ blkiocg_reset_stats(struct cgroup *cgrou
> > > #endif
> > >
> > > blkcg = cgroup_to_blkio_cgroup(cgroup);
> > > + spin_lock(&blkio_list_lock);
> > > spin_lock_irq(&blkcg->lock);
> >
> > Isn't blkcg lock enough to protect against policy registration/deregistration.
> > A policy can not add/delete a group to cgroup list without blkcg list.
>
> But pol list can change regardless of that, no?
Ok, looks like now it is needed because blkcg lock will just gurantee that
blkg is around but blkg->pd[plid] can disappear if you are not holding
blkio_list lock (update_root_blkgs).
I am wondering if we should take blkcg->lock if blkg is on blkcg list and
is being modified in place. That way, once we are switching elevator,
we should be able to shoot down the policy data without taking blkio_list
lock.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-03 21:44 ` Vivek Goyal
@ 2012-02-03 21:47 ` Tejun Heo
2012-02-03 21:53 ` Vivek Goyal
0 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-03 21:47 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
On Fri, Feb 03, 2012 at 04:44:35PM -0500, Vivek Goyal wrote:
> Ok, looks like now it is needed because blkcg lock will just gurantee that
> blkg is around but blkg->pd[plid] can disappear if you are not holding
> blkio_list lock (update_root_blkgs).
>
> I am wondering if we should take blkcg->lock if blkg is on blkcg list and
> is being modified in place. That way, once we are switching elevator,
> we should be able to shoot down the policy data without taking blkio_list
> lock.
I think it gotta become something per-queue, not global, and if we
make it per-queue, it should be able to live inside queue_lock.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-03 21:47 ` Tejun Heo
@ 2012-02-03 21:53 ` Vivek Goyal
2012-02-03 22:14 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-03 21:53 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Fri, Feb 03, 2012 at 01:47:19PM -0800, Tejun Heo wrote:
> On Fri, Feb 03, 2012 at 04:44:35PM -0500, Vivek Goyal wrote:
> > Ok, looks like now it is needed because blkcg lock will just gurantee that
> > blkg is around but blkg->pd[plid] can disappear if you are not holding
> > blkio_list lock (update_root_blkgs).
> >
> > I am wondering if we should take blkcg->lock if blkg is on blkcg list and
> > is being modified in place. That way, once we are switching elevator,
> > we should be able to shoot down the policy data without taking blkio_list
> > lock.
>
> I think it gotta become something per-queue, not global, and if we
> make it per-queue, it should be able to live inside queue_lock.
Hmm... then blkiocg_reset_stats() will run into lock ordering issue. Can't
hold queue lock inside blkcg lock. I guess you will do some kind of
locking trick again as you did for io context logic.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-03 21:53 ` Vivek Goyal
@ 2012-02-03 22:14 ` Tejun Heo
2012-02-03 22:23 ` Vivek Goyal
0 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-03 22:14 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
On Fri, Feb 03, 2012 at 04:53:49PM -0500, Vivek Goyal wrote:
> Hmm... then blkiocg_reset_stats() will run into lock ordering issue. Can't
> hold queue lock inside blkcg lock. I guess you will do some kind of
> locking trick again as you did for io context logic.
Ummm... I don't know the details yet but decisions like making
policies per-queue are much higher level than locking details. There
are cases where implementation details become problematic enough and
high level decisions are affected but I don't really think that would
be the case here. It's not like reset stats is a hot path. In fact,
I don't even understand why the hell we have it. I guess it's another
blkcg braindamage.
If userland is interested in delta between certain periods, it's
supposed to read values at the start and end of the period and
subtract. Allowing reset might make sense if there's single bearded
admin looking at the raw numbers, but it breaks as soon as tools and
automation are involved. How would you decide who owns those
counters?
And we sadly can't even remove that at this point. Ugh................
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-03 22:14 ` Tejun Heo
@ 2012-02-03 22:23 ` Vivek Goyal
2012-02-03 22:28 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-03 22:23 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Fri, Feb 03, 2012 at 02:14:09PM -0800, Tejun Heo wrote:
> On Fri, Feb 03, 2012 at 04:53:49PM -0500, Vivek Goyal wrote:
> > Hmm... then blkiocg_reset_stats() will run into lock ordering issue. Can't
> > hold queue lock inside blkcg lock. I guess you will do some kind of
> > locking trick again as you did for io context logic.
>
> Ummm... I don't know the details yet but decisions like making
> policies per-queue are much higher level than locking details. There
> are cases where implementation details become problematic enough and
> high level decisions are affected but I don't really think that would
> be the case here. It's not like reset stats is a hot path. In fact,
> I don't even understand why the hell we have it. I guess it's another
> blkcg braindamage.
git commit of when it was added.
commit 84c124da9ff50bd71fab9c939ee5b7cd8bef2bd9
Author: Divyesh Shah <dpshah@google.com>
Date: Fri Apr 9 08:31:19 2010 +0200
..
o add a separate reset_stats interface
..
>
> If userland is interested in delta between certain periods, it's
> supposed to read values at the start and end of the period and
> subtract. Allowing reset might make sense if there's single bearded
> admin looking at the raw numbers, but it breaks as soon as tools and
> automation are involved. How would you decide who owns those
> counters?
>
Personally I consider it to be a debugging aid. User space applications
should do what you suggested and that is take delta between certain
periods.
Frankly speaking I have found it useful though. If I am debugging something,
I don't have to reboot the system to reset the stats and calculating the
difference from zero is easier for humans.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED 11/11] blkcg: unify blkg's for blkcg policies
2012-02-03 22:23 ` Vivek Goyal
@ 2012-02-03 22:28 ` Tejun Heo
0 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-03 22:28 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
On Fri, Feb 03, 2012 at 05:23:29PM -0500, Vivek Goyal wrote:
> Personally I consider it to be a debugging aid. User space applications
> should do what you suggested and that is take delta between certain
> periods.
>
> Frankly speaking I have found it useful though. If I am debugging something,
> I don't have to reboot the system to reset the stats and calculating the
> difference from zero is easier for humans.
There are many things different people may find useful but we
shouldn't add userland visible interface with marginal usefulness like
this one. Seriously, you can write up a ten line python script to
achieve what you want. It's just sad. :(
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH UPDATED2 11/11] blkcg: unify blkg's for blkcg policies
2012-02-02 0:37 ` [PATCH UPDATED " Tejun Heo
2012-02-03 19:41 ` Vivek Goyal
2012-02-03 21:06 ` Vivek Goyal
@ 2012-02-14 1:33 ` Tejun Heo
2012-02-15 17:02 ` Vivek Goyal
2 siblings, 1 reply; 42+ messages in thread
From: Tejun Heo @ 2012-02-14 1:33 UTC (permalink / raw)
To: axboe, vgoyal; +Cc: ctalbott, rni, linux-kernel
Currently, blkg is per cgroup-queue-policy combination. This is
unnatural and leads to various convolutions in partially used
duplicate fields in blkg, config / stat access, and general management
of blkgs.
This patch make blkg's per cgroup-queue and let them serve all
policies. blkgs are now created and destroyed by blkcg core proper.
This will allow further consolidation of common management logic into
blkcg core and API with better defined semantics and layering.
As a transitional step to untangle blkg management, elvswitch and
policy [de]registration, all blkgs except the root blkg are being shot
down during elvswitch and bypass. This patch adds blkg_root_update()
to update root blkg in place on policy change. This is hacky and racy
but should be good enough as interim step until we get locking
simplified and switch over to proper in-place update for all blkgs.
-v2: Root blkgs need to be updated on elvswitch too and blkg_alloc()
comment wasn't updated according to the function change. Fixed.
Both pointed out by Vivek.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
Modified to update root blkgs on elvswitch too.
Jens, there are a couple other changes but they're all trivial. I
didn't want to repost both patchsets for those changes. The git
branch has been updated. Ping me if you want the full series
reposted.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git blkcg-unified-blkg
I'd really like to get this into block/core and get some exposure in
linux-next. There still are a lot of transitional stuff but I feel
that there's now somewhat solid ground to build stuff atop. Vivek,
what do you think?
Thanks.
block/blk-cgroup.c | 227 +++++++++++++++++++++++++++++++--------------------
block/blk-cgroup.h | 11 --
block/blk-core.c | 3
block/blk-sysfs.c | 4
block/blk-throttle.c | 9 --
block/cfq-iosched.c | 4
block/elevator.c | 5 -
7 files changed, 151 insertions(+), 112 deletions(-)
Index: work/block/blk-cgroup.c
===================================================================
--- work.orig/block/blk-cgroup.c
+++ work/block/blk-cgroup.c
@@ -461,16 +461,20 @@ EXPORT_SYMBOL_GPL(blkiocg_update_io_merg
*/
static void blkg_free(struct blkio_group *blkg)
{
- struct blkg_policy_data *pd;
+ int i;
if (!blkg)
return;
- pd = blkg->pd[blkg->plid];
- if (pd) {
- free_percpu(pd->stats_cpu);
- kfree(pd);
+ for (i = 0; i < BLKIO_NR_POLICIES; i++) {
+ struct blkg_policy_data *pd = blkg->pd[i];
+
+ if (pd) {
+ free_percpu(pd->stats_cpu);
+ kfree(pd);
+ }
}
+
kfree(blkg);
}
@@ -478,19 +482,17 @@ static void blkg_free(struct blkio_group
* blkg_alloc - allocate a blkg
* @blkcg: block cgroup the new blkg is associated with
* @q: request_queue the new blkg is associated with
- * @pol: policy the new blkg is associated with
*
- * Allocate a new blkg assocating @blkcg and @q for @pol.
+ * Allocate a new blkg assocating @blkcg and @q.
*
* FIXME: Should be called with queue locked but currently isn't due to
* percpu stat breakage.
*/
static struct blkio_group *blkg_alloc(struct blkio_cgroup *blkcg,
- struct request_queue *q,
- struct blkio_policy_type *pol)
+ struct request_queue *q)
{
struct blkio_group *blkg;
- struct blkg_policy_data *pd;
+ int i;
/* alloc and init base part */
blkg = kzalloc_node(sizeof(*blkg), GFP_ATOMIC, q->node);
@@ -499,34 +501,45 @@ static struct blkio_group *blkg_alloc(st
spin_lock_init(&blkg->stats_lock);
rcu_assign_pointer(blkg->q, q);
- INIT_LIST_HEAD(&blkg->q_node[0]);
- INIT_LIST_HEAD(&blkg->q_node[1]);
+ INIT_LIST_HEAD(&blkg->q_node);
blkg->blkcg = blkcg;
- blkg->plid = pol->plid;
blkg->refcnt = 1;
cgroup_path(blkcg->css.cgroup, blkg->path, sizeof(blkg->path));
- /* alloc per-policy data and attach it to blkg */
- pd = kzalloc_node(sizeof(*pd) + pol->pdata_size, GFP_ATOMIC,
- q->node);
- if (!pd) {
- blkg_free(blkg);
- return NULL;
- }
+ for (i = 0; i < BLKIO_NR_POLICIES; i++) {
+ struct blkio_policy_type *pol = blkio_policy[i];
+ struct blkg_policy_data *pd;
- blkg->pd[pol->plid] = pd;
- pd->blkg = blkg;
+ if (!pol)
+ continue;
+
+ /* alloc per-policy data and attach it to blkg */
+ pd = kzalloc_node(sizeof(*pd) + pol->pdata_size, GFP_ATOMIC,
+ q->node);
+ if (!pd) {
+ blkg_free(blkg);
+ return NULL;
+ }
- /* broken, read comment in the callsite */
+ blkg->pd[i] = pd;
+ pd->blkg = blkg;
- pd->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
- if (!pd->stats_cpu) {
- blkg_free(blkg);
- return NULL;
+ /* broken, read comment in the callsite */
+ pd->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
+ if (!pd->stats_cpu) {
+ blkg_free(blkg);
+ return NULL;
+ }
}
/* invoke per-policy init */
- pol->ops.blkio_init_group_fn(blkg);
+ for (i = 0; i < BLKIO_NR_POLICIES; i++) {
+ struct blkio_policy_type *pol = blkio_policy[i];
+
+ if (pol)
+ pol->ops.blkio_init_group_fn(blkg);
+ }
+
return blkg;
}
@@ -536,7 +549,6 @@ struct blkio_group *blkg_lookup_create(s
bool for_root)
__releases(q->queue_lock) __acquires(q->queue_lock)
{
- struct blkio_policy_type *pol = blkio_policy[plid];
struct blkio_group *blkg, *new_blkg;
WARN_ON_ONCE(!rcu_read_lock_held());
@@ -551,7 +563,7 @@ struct blkio_group *blkg_lookup_create(s
if (unlikely(blk_queue_bypass(q)) && !for_root)
return ERR_PTR(blk_queue_dead(q) ? -EINVAL : -EBUSY);
- blkg = blkg_lookup(blkcg, q, plid);
+ blkg = blkg_lookup(blkcg, q);
if (blkg)
return blkg;
@@ -571,7 +583,7 @@ struct blkio_group *blkg_lookup_create(s
spin_unlock_irq(q->queue_lock);
rcu_read_unlock();
- new_blkg = blkg_alloc(blkcg, q, pol);
+ new_blkg = blkg_alloc(blkcg, q);
rcu_read_lock();
spin_lock_irq(q->queue_lock);
@@ -583,7 +595,7 @@ struct blkio_group *blkg_lookup_create(s
}
/* did someone beat us to it? */
- blkg = blkg_lookup(blkcg, q, plid);
+ blkg = blkg_lookup(blkcg, q);
if (unlikely(blkg))
goto out;
@@ -598,8 +610,8 @@ struct blkio_group *blkg_lookup_create(s
swap(blkg, new_blkg);
hlist_add_head_rcu(&blkg->blkcg_node, &blkcg->blkg_list);
- list_add(&blkg->q_node[plid], &q->blkg_list[plid]);
- q->nr_blkgs[plid]++;
+ list_add(&blkg->q_node, &q->blkg_list);
+ q->nr_blkgs++;
spin_unlock(&blkcg->lock);
out:
@@ -636,31 +648,30 @@ EXPORT_SYMBOL_GPL(blkiocg_del_blkio_grou
/* called under rcu_read_lock(). */
struct blkio_group *blkg_lookup(struct blkio_cgroup *blkcg,
- struct request_queue *q,
- enum blkio_policy_id plid)
+ struct request_queue *q)
{
struct blkio_group *blkg;
struct hlist_node *n;
hlist_for_each_entry_rcu(blkg, n, &blkcg->blkg_list, blkcg_node)
- if (blkg->q == q && blkg->plid == plid)
+ if (blkg->q == q)
return blkg;
return NULL;
}
EXPORT_SYMBOL_GPL(blkg_lookup);
-static void blkg_destroy(struct blkio_group *blkg, enum blkio_policy_id plid)
+static void blkg_destroy(struct blkio_group *blkg)
{
struct request_queue *q = blkg->q;
lockdep_assert_held(q->queue_lock);
/* Something wrong if we are trying to remove same group twice */
- WARN_ON_ONCE(list_empty(&blkg->q_node[plid]));
- list_del_init(&blkg->q_node[plid]);
+ WARN_ON_ONCE(list_empty(&blkg->q_node));
+ list_del_init(&blkg->q_node);
- WARN_ON_ONCE(q->nr_blkgs[plid] <= 0);
- q->nr_blkgs[plid]--;
+ WARN_ON_ONCE(q->nr_blkgs <= 0);
+ q->nr_blkgs--;
/*
* Put the reference taken at the time of creation so that when all
@@ -669,18 +680,49 @@ static void blkg_destroy(struct blkio_gr
blkg_put(blkg);
}
-void blkg_destroy_all(struct request_queue *q, enum blkio_policy_id plid,
- bool destroy_root)
+/*
+ * XXX: This updates blkg policy data in-place for root blkg, which is
+ * necessary across elevator switch and policy registration as root blkgs
+ * aren't shot down. This broken and racy implementation is temporary.
+ * Eventually, blkg shoot down will be replaced by proper in-place update.
+ */
+static void update_root_blkg(struct request_queue *q, enum blkio_policy_id plid)
+{
+ struct blkio_policy_type *pol = blkio_policy[plid];
+ struct blkio_group *blkg = blkg_lookup(&blkio_root_cgroup, q);
+ struct blkg_policy_data *pd;
+
+ if (!blkg)
+ return;
+
+ kfree(blkg->pd[plid]);
+ blkg->pd[plid] = NULL;
+
+ if (!pol)
+ return;
+
+ pd = kzalloc(sizeof(*pd) + pol->pdata_size, GFP_KERNEL);
+ WARN_ON_ONCE(!pd);
+
+ pd->stats_cpu = alloc_percpu(struct blkio_group_stats_cpu);
+ WARN_ON_ONCE(!pd->stats_cpu);
+
+ blkg->pd[plid] = pd;
+ pd->blkg = blkg;
+ pol->ops.blkio_init_group_fn(blkg);
+}
+
+void blkg_destroy_all(struct request_queue *q, bool destroy_root)
{
struct blkio_group *blkg, *n;
+ int i;
while (true) {
bool done = true;
spin_lock_irq(q->queue_lock);
- list_for_each_entry_safe(blkg, n, &q->blkg_list[plid],
- q_node[plid]) {
+ list_for_each_entry_safe(blkg, n, &q->blkg_list, q_node) {
/* skip root? */
if (!destroy_root && blkg->blkcg == &blkio_root_cgroup)
continue;
@@ -691,7 +733,7 @@ void blkg_destroy_all(struct request_que
* take care of destroying cfqg also.
*/
if (!blkiocg_del_blkio_group(blkg))
- blkg_destroy(blkg, plid);
+ blkg_destroy(blkg);
else
done = false;
}
@@ -710,6 +752,9 @@ void blkg_destroy_all(struct request_que
msleep(10); /* just some random duration I like */
}
+
+ for (i = 0; i < BLKIO_NR_POLICIES; i++)
+ update_root_blkg(q, i);
}
EXPORT_SYMBOL_GPL(blkg_destroy_all);
@@ -776,43 +821,49 @@ blkiocg_reset_stats(struct cgroup *cgrou
#endif
blkcg = cgroup_to_blkio_cgroup(cgroup);
+ spin_lock(&blkio_list_lock);
spin_lock_irq(&blkcg->lock);
hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node) {
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkio_policy_type *pol;
+
+ list_for_each_entry(pol, &blkio_list, list) {
+ struct blkg_policy_data *pd = blkg->pd[pol->plid];
- spin_lock(&blkg->stats_lock);
- stats = &pd->stats;
+ spin_lock(&blkg->stats_lock);
+ stats = &pd->stats;
#ifdef CONFIG_DEBUG_BLK_CGROUP
- idling = blkio_blkg_idling(stats);
- waiting = blkio_blkg_waiting(stats);
- empty = blkio_blkg_empty(stats);
+ idling = blkio_blkg_idling(stats);
+ waiting = blkio_blkg_waiting(stats);
+ empty = blkio_blkg_empty(stats);
#endif
- for (i = 0; i < BLKIO_STAT_TOTAL; i++)
- queued[i] = stats->stat_arr[BLKIO_STAT_QUEUED][i];
- memset(stats, 0, sizeof(struct blkio_group_stats));
- for (i = 0; i < BLKIO_STAT_TOTAL; i++)
- stats->stat_arr[BLKIO_STAT_QUEUED][i] = queued[i];
+ for (i = 0; i < BLKIO_STAT_TOTAL; i++)
+ queued[i] = stats->stat_arr[BLKIO_STAT_QUEUED][i];
+ memset(stats, 0, sizeof(struct blkio_group_stats));
+ for (i = 0; i < BLKIO_STAT_TOTAL; i++)
+ stats->stat_arr[BLKIO_STAT_QUEUED][i] = queued[i];
#ifdef CONFIG_DEBUG_BLK_CGROUP
- if (idling) {
- blkio_mark_blkg_idling(stats);
- stats->start_idle_time = now;
- }
- if (waiting) {
- blkio_mark_blkg_waiting(stats);
- stats->start_group_wait_time = now;
- }
- if (empty) {
- blkio_mark_blkg_empty(stats);
- stats->start_empty_time = now;
- }
+ if (idling) {
+ blkio_mark_blkg_idling(stats);
+ stats->start_idle_time = now;
+ }
+ if (waiting) {
+ blkio_mark_blkg_waiting(stats);
+ stats->start_group_wait_time = now;
+ }
+ if (empty) {
+ blkio_mark_blkg_empty(stats);
+ stats->start_empty_time = now;
+ }
#endif
- spin_unlock(&blkg->stats_lock);
+ spin_unlock(&blkg->stats_lock);
- /* Reset Per cpu stats which don't take blkg->stats_lock */
- blkio_reset_stats_cpu(blkg, blkg->plid);
+ /* Reset Per cpu stats which don't take blkg->stats_lock */
+ blkio_reset_stats_cpu(blkg, pol->plid);
+ }
}
spin_unlock_irq(&blkcg->lock);
+ spin_unlock(&blkio_list_lock);
return 0;
}
@@ -1157,8 +1208,7 @@ static void blkio_read_conf(struct cftyp
spin_lock_irq(&blkcg->lock);
hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node)
- if (BLKIOFILE_POLICY(cft->private) == blkg->plid)
- blkio_print_group_conf(cft, blkg, m);
+ blkio_print_group_conf(cft, blkg, m);
spin_unlock_irq(&blkcg->lock);
}
@@ -1213,8 +1263,6 @@ static int blkio_read_blkg_stats(struct
const char *dname = dev_name(blkg->q->backing_dev_info.dev);
int plid = BLKIOFILE_POLICY(cft->private);
- if (plid != blkg->plid)
- continue;
if (pcpu) {
cgroup_total += blkio_get_stat_cpu(blkg, plid,
cb, dname, type);
@@ -1324,9 +1372,9 @@ static int blkio_weight_write(struct blk
blkcg->weight = (unsigned int)val;
hlist_for_each_entry(blkg, n, &blkcg->blkg_list, blkcg_node) {
- struct blkg_policy_data *pd = blkg->pd[blkg->plid];
+ struct blkg_policy_data *pd = blkg->pd[plid];
- if (blkg->plid == plid && !pd->conf.weight)
+ if (!pd->conf.weight)
blkio_update_group_weight(blkg, plid, blkcg->weight);
}
@@ -1549,7 +1597,6 @@ static int blkiocg_pre_destroy(struct cg
unsigned long flags;
struct blkio_group *blkg;
struct request_queue *q;
- struct blkio_policy_type *blkiop;
rcu_read_lock();
@@ -1575,11 +1622,7 @@ static int blkiocg_pre_destroy(struct cg
*/
spin_lock(&blkio_list_lock);
spin_lock_irqsave(q->queue_lock, flags);
- list_for_each_entry(blkiop, &blkio_list, list) {
- if (blkiop->plid != blkg->plid)
- continue;
- blkg_destroy(blkg, blkiop->plid);
- }
+ blkg_destroy(blkg);
spin_unlock_irqrestore(q->queue_lock, flags);
spin_unlock(&blkio_list_lock);
} while (1);
@@ -1673,6 +1716,8 @@ void blkcg_exit_queue(struct request_que
list_del_init(&q->all_q_node);
mutex_unlock(&all_q_mutex);
+ blkg_destroy_all(q, true);
+
blk_throtl_exit(q);
}
@@ -1722,14 +1767,12 @@ static void blkcg_bypass_start(void)
__acquires(&all_q_mutex)
{
struct request_queue *q;
- int i;
mutex_lock(&all_q_mutex);
list_for_each_entry(q, &all_q_list, all_q_node) {
blk_queue_bypass_start(q);
- for (i = 0; i < BLKIO_NR_POLICIES; i++)
- blkg_destroy_all(q, i, false);
+ blkg_destroy_all(q, false);
}
}
@@ -1746,6 +1789,8 @@ static void blkcg_bypass_end(void)
void blkio_policy_register(struct blkio_policy_type *blkiop)
{
+ struct request_queue *q;
+
blkcg_bypass_start();
spin_lock(&blkio_list_lock);
@@ -1754,12 +1799,16 @@ void blkio_policy_register(struct blkio_
list_add_tail(&blkiop->list, &blkio_list);
spin_unlock(&blkio_list_lock);
+ list_for_each_entry(q, &all_q_list, all_q_node)
+ update_root_blkg(q, blkiop->plid);
blkcg_bypass_end();
}
EXPORT_SYMBOL_GPL(blkio_policy_register);
void blkio_policy_unregister(struct blkio_policy_type *blkiop)
{
+ struct request_queue *q;
+
blkcg_bypass_start();
spin_lock(&blkio_list_lock);
@@ -1768,6 +1817,8 @@ void blkio_policy_unregister(struct blki
list_del_init(&blkiop->list);
spin_unlock(&blkio_list_lock);
+ list_for_each_entry(q, &all_q_list, all_q_node)
+ update_root_blkg(q, blkiop->plid);
blkcg_bypass_end();
}
EXPORT_SYMBOL_GPL(blkio_policy_unregister);
Index: work/block/blk-cgroup.h
===================================================================
--- work.orig/block/blk-cgroup.h
+++ work/block/blk-cgroup.h
@@ -178,13 +178,11 @@ struct blkg_policy_data {
struct blkio_group {
/* Pointer to the associated request_queue, RCU protected */
struct request_queue __rcu *q;
- struct list_head q_node[BLKIO_NR_POLICIES];
+ struct list_head q_node;
struct hlist_node blkcg_node;
struct blkio_cgroup *blkcg;
/* Store cgroup path */
char path[128];
- /* policy which owns this blk group */
- enum blkio_policy_id plid;
/* reference count */
int refcnt;
@@ -230,8 +228,7 @@ extern void blkcg_exit_queue(struct requ
/* Blkio controller policy registration */
extern void blkio_policy_register(struct blkio_policy_type *);
extern void blkio_policy_unregister(struct blkio_policy_type *);
-extern void blkg_destroy_all(struct request_queue *q,
- enum blkio_policy_id plid, bool destroy_root);
+extern void blkg_destroy_all(struct request_queue *q, bool destroy_root);
/**
* blkg_to_pdata - get policy private data
@@ -313,7 +310,6 @@ static inline void blkcg_exit_queue(stru
static inline void blkio_policy_register(struct blkio_policy_type *blkiop) { }
static inline void blkio_policy_unregister(struct blkio_policy_type *blkiop) { }
static inline void blkg_destroy_all(struct request_queue *q,
- enum blkio_policy_id plid,
bool destory_root) { }
static inline void *blkg_to_pdata(struct blkio_group *blkg,
@@ -382,8 +378,7 @@ extern struct blkio_cgroup *cgroup_to_bl
extern struct blkio_cgroup *task_blkio_cgroup(struct task_struct *tsk);
extern int blkiocg_del_blkio_group(struct blkio_group *blkg);
extern struct blkio_group *blkg_lookup(struct blkio_cgroup *blkcg,
- struct request_queue *q,
- enum blkio_policy_id plid);
+ struct request_queue *q);
struct blkio_group *blkg_lookup_create(struct blkio_cgroup *blkcg,
struct request_queue *q,
enum blkio_policy_id plid,
Index: work/block/blk-throttle.c
===================================================================
--- work.orig/block/blk-throttle.c
+++ work/block/blk-throttle.c
@@ -167,7 +167,7 @@ throtl_grp *throtl_lookup_tg(struct thro
if (blkcg == &blkio_root_cgroup)
return td->root_tg;
- return blkg_to_tg(blkg_lookup(blkcg, td->queue, BLKIO_POLICY_THROTL));
+ return blkg_to_tg(blkg_lookup(blkcg, td->queue));
}
static struct throtl_grp *throtl_lookup_create_tg(struct throtl_data *td,
@@ -704,8 +704,7 @@ static void throtl_process_limit_change(
throtl_log(td, "limits changed");
- list_for_each_entry_safe(blkg, n, &q->blkg_list[BLKIO_POLICY_THROTL],
- q_node[BLKIO_POLICY_THROTL]) {
+ list_for_each_entry_safe(blkg, n, &q->blkg_list, q_node) {
struct throtl_grp *tg = blkg_to_tg(blkg);
if (!tg->limits_changed)
@@ -1054,11 +1053,9 @@ void blk_throtl_exit(struct request_queu
throtl_shutdown_wq(q);
- blkg_destroy_all(q, BLKIO_POLICY_THROTL, true);
-
/* If there are other groups */
spin_lock_irq(q->queue_lock);
- wait = q->nr_blkgs[BLKIO_POLICY_THROTL];
+ wait = q->nr_blkgs;
spin_unlock_irq(q->queue_lock);
/*
Index: work/block/cfq-iosched.c
===================================================================
--- work.orig/block/cfq-iosched.c
+++ work/block/cfq-iosched.c
@@ -3462,15 +3462,13 @@ static void cfq_exit_queue(struct elevat
spin_unlock_irq(q->queue_lock);
- blkg_destroy_all(q, BLKIO_POLICY_PROP, true);
-
#ifdef CONFIG_BLK_CGROUP
/*
* If there are groups which we could not unlink from blkcg list,
* wait for a rcu period for them to be freed.
*/
spin_lock_irq(q->queue_lock);
- wait = q->nr_blkgs[BLKIO_POLICY_PROP];
+ wait = q->nr_blkgs;
spin_unlock_irq(q->queue_lock);
#endif
cfq_shutdown_timer_wq(cfqd);
Index: work/block/blk-core.c
===================================================================
--- work.orig/block/blk-core.c
+++ work/block/blk-core.c
@@ -547,8 +547,7 @@ struct request_queue *blk_alloc_queue_no
INIT_LIST_HEAD(&q->timeout_list);
INIT_LIST_HEAD(&q->icq_list);
#ifdef CONFIG_BLK_CGROUP
- INIT_LIST_HEAD(&q->blkg_list[0]);
- INIT_LIST_HEAD(&q->blkg_list[1]);
+ INIT_LIST_HEAD(&q->blkg_list);
#endif
INIT_LIST_HEAD(&q->flush_queue[0]);
INIT_LIST_HEAD(&q->flush_queue[1]);
Index: work/block/elevator.c
===================================================================
--- work.orig/block/elevator.c
+++ work/block/elevator.c
@@ -876,7 +876,7 @@ static int elevator_switch(struct reques
{
struct elevator_queue *old = q->elevator;
bool registered = old->registered;
- int i, err;
+ int err;
/*
* Turn on BYPASS and drain all requests w/ elevator private data.
@@ -895,8 +895,7 @@ static int elevator_switch(struct reques
ioc_clear_queue(q);
spin_unlock_irq(q->queue_lock);
- for (i = 0; i < BLKIO_NR_POLICIES; i++)
- blkg_destroy_all(q, i, false);
+ blkg_destroy_all(q, false);
/* allocate, init and register new elevator */
err = -ENOMEM;
Index: work/block/blk-sysfs.c
===================================================================
--- work.orig/block/blk-sysfs.c
+++ work/block/blk-sysfs.c
@@ -480,6 +480,8 @@ static void blk_release_queue(struct kob
blk_sync_queue(q);
+ blkcg_exit_queue(q);
+
if (q->elevator) {
spin_lock_irq(q->queue_lock);
ioc_clear_queue(q);
@@ -487,8 +489,6 @@ static void blk_release_queue(struct kob
elevator_exit(q->elevator);
}
- blkcg_exit_queue(q);
-
if (rl->rq_pool)
mempool_destroy(rl->rq_pool);
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED2 11/11] blkcg: unify blkg's for blkcg policies
2012-02-14 1:33 ` [PATCH UPDATED2 " Tejun Heo
@ 2012-02-15 17:02 ` Vivek Goyal
2012-02-16 22:42 ` Tejun Heo
0 siblings, 1 reply; 42+ messages in thread
From: Vivek Goyal @ 2012-02-15 17:02 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, ctalbott, rni, linux-kernel
On Mon, Feb 13, 2012 at 05:33:47PM -0800, Tejun Heo wrote:
[..]
> Modified to update root blkgs on elvswitch too.
>
> Jens, there are a couple other changes but they're all trivial. I
> didn't want to repost both patchsets for those changes. The git
> branch has been updated. Ping me if you want the full series
> reposted.
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git blkcg-unified-blkg
>
> I'd really like to get this into block/core and get some exposure in
> linux-next. There still are a lot of transitional stuff but I feel
> that there's now somewhat solid ground to build stuff atop. Vivek,
> what do you think?
Hi Tejun,
As this patch series is getting longer (already 2 sets are out there), I
think it is reasonable to get some testing going in linux-next. I can think
of atleast few more things which you might be planning to push before
3.4 merge window opens.
- Cleanup in-place policy data update logic.
- Do not kill all the groups upon one policy switch. Keep the group around
and just kill the policy data of associated policy.
- Get rid if locking complexity and rcu around group removal. Replace with
double lock dancing stuff.
- Fix the per cpu alloc stat alloc mess. This is anyway needed
irrespective of cleanup.
Thanks
Vivek
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH UPDATED2 11/11] blkcg: unify blkg's for blkcg policies
2012-02-15 17:02 ` Vivek Goyal
@ 2012-02-16 22:42 ` Tejun Heo
0 siblings, 0 replies; 42+ messages in thread
From: Tejun Heo @ 2012-02-16 22:42 UTC (permalink / raw)
To: Vivek Goyal; +Cc: axboe, ctalbott, rni, linux-kernel
Hello,
On Wed, Feb 15, 2012 at 12:02:29PM -0500, Vivek Goyal wrote:
> - Cleanup in-place policy data update logic.
>
> - Do not kill all the groups upon one policy switch. Keep the group around
> and just kill the policy data of associated policy.
>
> - Get rid if locking complexity and rcu around group removal. Replace with
> double lock dancing stuff.
Yeap, just sent out this one and fix for stacking. I'll probably do
the in-place policy update thing next.
> - Fix the per cpu alloc stat alloc mess. This is anyway needed
> irrespective of cleanup.
I'll ping Andrew again on the mempool thing on more time.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 42+ messages in thread
end of thread, other threads:[~2012-02-16 22:43 UTC | newest]
Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-02-01 21:19 [PATCHSET] blkcg: unify blkgs for different policies Tejun Heo
2012-02-01 21:19 ` [PATCH 01/11] blkcg: let blkio_group point to blkio_cgroup directly Tejun Heo
2012-02-02 20:03 ` Vivek Goyal
2012-02-02 20:33 ` Tejun Heo
2012-02-02 20:55 ` Vivek Goyal
2012-02-01 21:19 ` [PATCH 02/11] block: relocate elevator initialized test from blk_cleanup_queue() to blk_drain_queue() Tejun Heo
2012-02-02 20:20 ` Vivek Goyal
2012-02-02 20:35 ` Tejun Heo
2012-02-02 20:37 ` Vivek Goyal
2012-02-02 20:38 ` Tejun Heo
2012-02-01 21:19 ` [PATCH 03/11] blkcg: add blkcg_{init|drain|exit}_queue() Tejun Heo
2012-02-01 21:19 ` [PATCH 04/11] blkcg: clear all request_queues on blkcg policy [un]registrations Tejun Heo
2012-02-01 21:19 ` [PATCH 05/11] blkcg: let blkcg core handle policy private data allocation Tejun Heo
2012-02-01 21:19 ` [PATCH 06/11] blkcg: move refcnt to blkcg core Tejun Heo
2012-02-02 22:07 ` Vivek Goyal
2012-02-02 22:11 ` Tejun Heo
2012-02-01 21:19 ` [PATCH 07/11] blkcg: make blkg->pd an array and move configuration and stats into it Tejun Heo
2012-02-01 21:19 ` [PATCH 08/11] blkcg: don't use blkg->plid in stat related functions Tejun Heo
2012-02-01 21:19 ` [PATCH 09/11] blkcg: move per-queue blkg list heads and counters to queue and blkg Tejun Heo
2012-02-02 22:47 ` Vivek Goyal
2012-02-02 22:47 ` Tejun Heo
2012-02-01 21:19 ` [PATCH 10/11] blkcg: let blkcg core manage per-queue blkg list and counter Tejun Heo
2012-02-01 21:19 ` [PATCH 11/11] blkcg: unify blkg's for blkcg policies Tejun Heo
2012-02-02 0:37 ` [PATCH UPDATED " Tejun Heo
2012-02-03 19:41 ` Vivek Goyal
2012-02-03 20:59 ` Tejun Heo
2012-02-03 21:44 ` Vivek Goyal
2012-02-03 21:47 ` Tejun Heo
2012-02-03 21:53 ` Vivek Goyal
2012-02-03 22:14 ` Tejun Heo
2012-02-03 22:23 ` Vivek Goyal
2012-02-03 22:28 ` Tejun Heo
2012-02-03 21:06 ` Vivek Goyal
2012-02-03 21:09 ` Tejun Heo
2012-02-03 21:10 ` Tejun Heo
2012-02-14 1:33 ` [PATCH UPDATED2 " Tejun Heo
2012-02-15 17:02 ` Vivek Goyal
2012-02-16 22:42 ` Tejun Heo
2012-02-02 19:29 ` [PATCHSET] blkcg: unify blkgs for different policies Vivek Goyal
2012-02-02 20:36 ` Tejun Heo
2012-02-02 20:43 ` Vivek Goyal
2012-02-02 20:59 ` Tejun Heo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).