From: Tejun Heo <tj@kernel.org>
To: Vivek Goyal <vgoyal@redhat.com>
Cc: axboe@kernel.dk, hughd@google.com, avi@redhat.com,
nate@cpanel.net, cl@linux-foundation.org,
linux-kernel@vger.kernel.org, dpshah@google.com,
ctalbott@google.com, rni@google.com,
Andrew Morton <akpm@linux-foundation.org>
Subject: Re: [PATCHSET] mempool, percpu, blkcg: fix percpu stat allocation and remove stats_lock
Date: Mon, 5 Mar 2012 14:13:21 -0800 [thread overview]
Message-ID: <20120305221321.GF1263@google.com> (raw)
In-Reply-To: <20120229173639.GB5930@redhat.com>
Hello, Vivek.
On Wed, Feb 29, 2012 at 12:36:39PM -0500, Vivek Goyal wrote:
> Index: tejun-misc/block/blk-cgroup.h
> ===================================================================
> --- tejun-misc.orig/block/blk-cgroup.h 2012-02-28 01:29:09.238256494 -0500
> +++ tejun-misc/block/blk-cgroup.h 2012-02-28 01:29:12.000000000 -0500
> @@ -180,6 +180,8 @@ struct blkio_group {
> struct request_queue *q;
> struct list_head q_node;
> struct hlist_node blkcg_node;
> + /* List of blkg waiting for per cpu stats memory to be allocated */
> + struct list_head pending_alloc_node;
Can we move this right on top of rcu_head? It's one of the coldest
entries. Also, long field names tend to be a bit painful. How about
just alloc_node?
> +static void blkio_stat_alloc_fn(struct work_struct *work)
> +{
> +
> + void *stat_ptr = NULL;
> + struct blkio_group *blkg, *n;
> + int i;
> +
> +alloc_stats:
> + spin_lock_irq(&pending_alloc_list_lock);
> + if (list_empty(&pending_alloc_list)) {
> + /* Nothing to do */
> + spin_unlock_irq(&pending_alloc_list_lock);
> + return;
> + }
> + spin_unlock_irq(&pending_alloc_list_lock);
> +
> + WARN_ON(stat_ptr != NULL);
> + stat_ptr = alloc_percpu(struct blkio_group_stats_cpu);
There will only one of this work item and if queued on nrt wq, only
one instance would be running. Why not just create static ps[NR_POLS]
array and fill it here.
> + /* Retry. Should there be an upper limit on number of retries */
> + if (stat_ptr == NULL)
> + goto alloc_stats;
> +
> + spin_lock_irq(&blkio_list_lock);
> + spin_lock(&pending_alloc_list_lock);
> +
> + list_for_each_entry_safe(blkg, n, &pending_alloc_list,
> + pending_alloc_node) {
> + for (i = 0; i < BLKIO_NR_POLICIES; i++) {
> + struct blkio_policy_type *pol = blkio_policy[i];
> + struct blkg_policy_data *pd;
> +
> + if (!pol)
> + continue;
> +
> + if (!blkg->pd[i])
> + continue;
> +
> + pd = blkg->pd[i];
> + if (pd->stats_cpu)
> + continue;
> +
> + pd->stats_cpu = stat_ptr;
> + stat_ptr = NULL;
> + break;
and install everything here at one go.
> + }
> +
> + if (i == BLKIO_NR_POLICIES - 1) {
> + /* We are done with this group */
> + list_del_init(&blkg->pending_alloc_node);
> + continue;
> + } else
> + /* Go allocate more memory */
> + break;
> + }
remove it from alloc list while holding alloc lock, unlock and go for
retrying or exit and don't worry about stats_cpu left in ps[] as we're
gonna be using that again later anyway.
> /* insert */
> spin_lock(&blkcg->lock);
> - swap(blkg, new_blkg);
> + spin_lock(&pending_alloc_list_lock);
Do we need this nested inside blkcg->lock? What's wrong with doing it
after release blkcg->lock?
> @@ -648,11 +701,16 @@ static void blkg_destroy(struct blkio_gr
> lockdep_assert_held(q->queue_lock);
> lockdep_assert_held(&blkcg->lock);
>
> + spin_lock(&pending_alloc_list_lock);
> +
> /* Something wrong if we are trying to remove same group twice */
> WARN_ON_ONCE(list_empty(&blkg->q_node));
> WARN_ON_ONCE(hlist_unhashed(&blkg->blkcg_node));
> list_del_init(&blkg->q_node);
> hlist_del_init_rcu(&blkg->blkcg_node);
> + list_del_init(&blkg->pending_alloc_node);
> +
> + spin_unlock(&pending_alloc_list_lock);
Why put the whole thing inside the alloc lock?
Thanks.
--
tejun
next prev parent reply other threads:[~2012-03-05 22:13 UTC|newest]
Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-02-23 22:30 [PATCHSET] mempool, percpu, blkcg: fix percpu stat allocation and remove stats_lock Tejun Heo
2012-02-23 22:30 ` [PATCH 1/8] mempool: factor out mempool_fill() Tejun Heo
2012-02-23 22:30 ` [PATCH 2/8] mempool: separate out __mempool_create() Tejun Heo
2012-02-23 22:30 ` [PATCH 3/8] mempool, percpu: implement percpu mempool Tejun Heo
2012-02-23 22:30 ` [PATCH 4/8] block: fix deadlock through percpu allocation in blk-cgroup Tejun Heo
2012-02-23 22:30 ` [PATCH 5/8] blkcg: don't use percpu for merged stats Tejun Heo
2012-02-23 22:30 ` [PATCH 6/8] blkcg: simplify stat reset Tejun Heo
2012-02-23 22:30 ` [PATCH 7/8] blkcg: restructure blkio_get_stat() Tejun Heo
2012-02-23 22:30 ` [PATCH 8/8] blkcg: remove blkio_group->stats_lock Tejun Heo
2012-02-23 22:43 ` [PATCHSET] mempool, percpu, blkcg: fix percpu stat allocation and remove stats_lock Andrew Morton
2012-02-23 23:01 ` Tejun Heo
2012-02-23 23:12 ` Tejun Heo
2012-02-23 23:22 ` Andrew Morton
2012-02-23 23:24 ` Tejun Heo
2012-02-24 14:20 ` Vivek Goyal
2012-02-25 21:44 ` Tejun Heo
2012-02-27 3:11 ` Vivek Goyal
2012-02-27 9:11 ` Tejun Heo
2012-02-27 19:43 ` Vivek Goyal
2012-02-29 17:36 ` Vivek Goyal
2012-03-05 22:13 ` Tejun Heo [this message]
2012-03-06 21:09 ` Vivek Goyal
2012-03-06 21:20 ` Andrew Morton
2012-03-06 21:34 ` Vivek Goyal
2012-03-06 21:55 ` Andrew Morton
2012-03-07 14:55 ` Vivek Goyal
2012-03-07 17:05 ` Tejun Heo
2012-03-07 19:13 ` Vivek Goyal
2012-03-07 19:22 ` Tejun Heo
2012-03-07 19:42 ` Vivek Goyal
2012-03-07 22:56 ` Tejun Heo
2012-03-07 23:08 ` Andrew Morton
2012-03-07 23:15 ` Tejun Heo
2012-03-07 23:05 ` Andrew Morton
2012-03-08 17:57 ` Vivek Goyal
2012-03-08 18:08 ` Tejun Heo
2012-03-08 18:11 ` Tejun Heo
2012-03-08 18:22 ` Vivek Goyal
2012-03-08 18:27 ` Tejun Heo
2012-03-15 16:48 ` Vivek Goyal
2012-03-15 16:59 ` Tejun Heo
2012-03-20 11:50 ` Jens Axboe
2012-03-08 20:16 ` Vivek Goyal
2012-03-08 20:33 ` Vivek Goyal
2012-03-08 20:35 ` Tejun Heo
2012-03-08 19:06 ` Andrew Morton
2012-02-25 3:44 ` Vivek Goyal
2012-02-25 21:46 ` Tejun Heo
2012-02-25 22:21 ` Tejun Heo
2012-02-27 14:25 ` Vivek Goyal
2012-02-27 14:40 ` Vivek Goyal
2012-03-05 17:45 ` Tejun Heo
2012-02-27 18:22 ` Vivek Goyal
2012-02-29 19:03 ` Vivek Goyal
2012-03-05 17:20 ` Tejun Heo
2012-03-05 18:03 ` Vivek Goyal
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120305221321.GF1263@google.com \
--to=tj@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=avi@redhat.com \
--cc=axboe@kernel.dk \
--cc=cl@linux-foundation.org \
--cc=ctalbott@google.com \
--cc=dpshah@google.com \
--cc=hughd@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=nate@cpanel.net \
--cc=rni@google.com \
--cc=vgoyal@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).