From: Gilad Ben-Yossef <gilad@benyossef.com> To: linux-kernel@vger.kernel.org Cc: Gilad Ben-Yossef <gilad@benyossef.com>, Chris Metcalf <cmetcalf@tilera.com>, Frederic Weisbecker <fweisbec@gmail.com>, Russell King <linux@arm.linux.org.uk>, linux-mm@kvack.org, Pekka Enberg <penberg@kernel.org>, Matt Mackall <mpm@selenic.com>, Sasha Levin <levinsasha928@gmail.com>, Rik van Riel <riel@redhat.com>, Andi Kleen <andi@firstfloor.org>, Mel Gorman <mel@csn.ul.ie>, Andrew Morton <akpm@linux-foundation.org>, Alexander Viro <viro@zeniv.linux.org.uk>, linux-fsdevel@vger.kernel.org, Avi Kivity <avi@redhat.com>, Michal Nazarewicz <mina86@mina86.com>, Kosaki Motohiro <kosaki.motohiro@gmail.com>, Milton Miller <miltonm@bga.com> Subject: [PATCH v8 5/8] slub: only IPI CPUs that have per cpu obj to flush Date: Sun, 5 Feb 2012 15:48:39 +0200 [thread overview] Message-ID: <1328449722-15959-4-git-send-email-gilad@benyossef.com> (raw) In-Reply-To: <1328448800-15794-1-git-send-email-gilad@benyossef.com> flush_all() is called for each kmem_cahce_destroy(). So every cache being destroyed dynamically ends up sending an IPI to each CPU in the system, regardless if the cache has ever been used there. For example, if you close the Infinband ipath driver char device file, the close file ops calls kmem_cache_destroy(). So running some infiniband config tool on one a single CPU dedicated to system tasks might interrupt the rest of the 127 CPUs dedicated to some CPU intensive or latency sensitive task. I suspect there is a good chance that every line in the output of "git grep kmem_cache_destroy linux/ | grep '\->'" has a similar scenario. This patch attempts to rectify this issue by sending an IPI to flush the per cpu objects back to the free lists only to CPUs that seem to have such objects. The check which CPU to IPI is racy but we don't care since asking a CPU without per cpu objects to flush does no damage and as far as I can tell the flush_all by itself is racy against allocs on remote CPUs anyway, so if you required the flush_all to be determinstic, you had to arrange for locking regardless. Without this patch the following artificial test case: $ cd /sys/kernel/slab $ for DIR in *; do cat $DIR/alloc_calls > /dev/null; done produces 166 IPIs on an cpuset isolated CPU. With it it produces none. The code path of memory allocation failure for CPUMASK_OFFSTACK=y config was tested using fault injection framework. Signed-off-by: Gilad Ben-Yossef <gilad@benyossef.com> Acked-by: Christoph Lameter <cl@linux.com> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> CC: Chris Metcalf <cmetcalf@tilera.com> CC: Frederic Weisbecker <fweisbec@gmail.com> CC: Russell King <linux@arm.linux.org.uk> CC: linux-mm@kvack.org CC: Pekka Enberg <penberg@kernel.org> CC: Matt Mackall <mpm@selenic.com> CC: Sasha Levin <levinsasha928@gmail.com> CC: Rik van Riel <riel@redhat.com> CC: Andi Kleen <andi@firstfloor.org> CC: Mel Gorman <mel@csn.ul.ie> CC: Andrew Morton <akpm@linux-foundation.org> CC: Alexander Viro <viro@zeniv.linux.org.uk> CC: linux-fsdevel@vger.kernel.org CC: Avi Kivity <avi@redhat.com> CC: Michal Nazarewicz <mina86@mina86.com> CC: Kosaki Motohiro <kosaki.motohiro@gmail.com> CC: Milton Miller <miltonm@bga.com> --- mm/slub.c | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 4907563..3d75f89 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2018,9 +2018,17 @@ static void flush_cpu_slab(void *d) __flush_cpu_slab(s, smp_processor_id()); } +static bool has_cpu_slab(int cpu, void *info) +{ + struct kmem_cache *s = info; + struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu); + + return !!(c->page); +} + static void flush_all(struct kmem_cache *s) { - on_each_cpu(flush_cpu_slab, s, 1); + on_each_cpu_cond(has_cpu_slab, flush_cpu_slab, s, 1, GFP_ATOMIC); } /* -- 1.7.0.4
WARNING: multiple messages have this Message-ID (diff)
From: Gilad Ben-Yossef <gilad@benyossef.com> To: linux-kernel@vger.kernel.org Cc: Gilad Ben-Yossef <gilad@benyossef.com>, Chris Metcalf <cmetcalf@tilera.com>, Frederic Weisbecker <fweisbec@gmail.com>, Russell King <linux@arm.linux.org.uk>, linux-mm@kvack.org, Pekka Enberg <penberg@kernel.org>, Matt Mackall <mpm@selenic.com>, Sasha Levin <levinsasha928@gmail.com>, Rik van Riel <riel@redhat.com>, Andi Kleen <andi@firstfloor.org>, Mel Gorman <mel@csn.ul.ie>, Andrew Morton <akpm@linux-foundation.org>, Alexander Viro <viro@zeniv.linux.org.uk>, linux-fsdevel@vger.kernel.org, Avi Kivity <avi@redhat.com>, Michal Nazarewicz <mina86@mina86.com>, Kosaki Motohiro <kosaki.motohiro@gmail.com>, Milton Miller <miltonm@bga.com> Subject: [PATCH v8 5/8] slub: only IPI CPUs that have per cpu obj to flush Date: Sun, 5 Feb 2012 15:48:39 +0200 [thread overview] Message-ID: <1328449722-15959-4-git-send-email-gilad@benyossef.com> (raw) In-Reply-To: <1328448800-15794-1-git-send-email-gilad@benyossef.com> flush_all() is called for each kmem_cahce_destroy(). So every cache being destroyed dynamically ends up sending an IPI to each CPU in the system, regardless if the cache has ever been used there. For example, if you close the Infinband ipath driver char device file, the close file ops calls kmem_cache_destroy(). So running some infiniband config tool on one a single CPU dedicated to system tasks might interrupt the rest of the 127 CPUs dedicated to some CPU intensive or latency sensitive task. I suspect there is a good chance that every line in the output of "git grep kmem_cache_destroy linux/ | grep '\->'" has a similar scenario. This patch attempts to rectify this issue by sending an IPI to flush the per cpu objects back to the free lists only to CPUs that seem to have such objects. The check which CPU to IPI is racy but we don't care since asking a CPU without per cpu objects to flush does no damage and as far as I can tell the flush_all by itself is racy against allocs on remote CPUs anyway, so if you required the flush_all to be determinstic, you had to arrange for locking regardless. Without this patch the following artificial test case: $ cd /sys/kernel/slab $ for DIR in *; do cat $DIR/alloc_calls > /dev/null; done produces 166 IPIs on an cpuset isolated CPU. With it it produces none. The code path of memory allocation failure for CPUMASK_OFFSTACK=y config was tested using fault injection framework. Signed-off-by: Gilad Ben-Yossef <gilad@benyossef.com> Acked-by: Christoph Lameter <cl@linux.com> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> CC: Chris Metcalf <cmetcalf@tilera.com> CC: Frederic Weisbecker <fweisbec@gmail.com> CC: Russell King <linux@arm.linux.org.uk> CC: linux-mm@kvack.org CC: Pekka Enberg <penberg@kernel.org> CC: Matt Mackall <mpm@selenic.com> CC: Sasha Levin <levinsasha928@gmail.com> CC: Rik van Riel <riel@redhat.com> CC: Andi Kleen <andi@firstfloor.org> CC: Mel Gorman <mel@csn.ul.ie> CC: Andrew Morton <akpm@linux-foundation.org> CC: Alexander Viro <viro@zeniv.linux.org.uk> CC: linux-fsdevel@vger.kernel.org CC: Avi Kivity <avi@redhat.com> CC: Michal Nazarewicz <mina86@mina86.com> CC: Kosaki Motohiro <kosaki.motohiro@gmail.com> CC: Milton Miller <miltonm@bga.com> --- mm/slub.c | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 4907563..3d75f89 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2018,9 +2018,17 @@ static void flush_cpu_slab(void *d) __flush_cpu_slab(s, smp_processor_id()); } +static bool has_cpu_slab(int cpu, void *info) +{ + struct kmem_cache *s = info; + struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu); + + return !!(c->page); +} + static void flush_all(struct kmem_cache *s) { - on_each_cpu(flush_cpu_slab, s, 1); + on_each_cpu_cond(has_cpu_slab, flush_cpu_slab, s, 1, GFP_ATOMIC); } /* -- 1.7.0.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2012-02-05 13:50 UTC|newest] Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top 2012-02-05 13:33 [PATCH v8 0/8] Reduce cross CPU IPI interference Gilad Ben-Yossef 2012-02-05 13:33 ` Gilad Ben-Yossef 2012-02-05 13:44 ` [PATCH v8 1/8] smp: introduce a generic on_each_cpu_mask function Gilad Ben-Yossef 2012-02-05 13:44 ` Gilad Ben-Yossef 2012-02-05 15:18 ` Srivatsa S. Bhat 2012-02-05 15:18 ` Srivatsa S. Bhat 2012-02-05 13:48 ` [PATCH v8 2/8] arm: move arm over to generic on_each_cpu_mask Gilad Ben-Yossef 2012-02-05 13:48 ` Gilad Ben-Yossef 2012-02-05 13:48 ` [PATCH v8 3/8] tile: move tile to use " Gilad Ben-Yossef 2012-02-05 13:48 ` Gilad Ben-Yossef 2012-02-05 13:48 ` [PATCH v8 4/8] smp: add func to IPI cpus based on parameter func Gilad Ben-Yossef 2012-02-05 13:48 ` Gilad Ben-Yossef 2012-02-05 15:36 ` Srivatsa S. Bhat 2012-02-05 15:36 ` Srivatsa S. Bhat 2012-02-05 15:46 ` Gilad Ben-Yossef 2012-02-05 15:46 ` Gilad Ben-Yossef 2012-02-05 16:00 ` Srivatsa S. Bhat 2012-02-05 16:00 ` Srivatsa S. Bhat 2012-02-05 16:03 ` Srivatsa S. Bhat 2012-02-05 16:03 ` Srivatsa S. Bhat 2012-02-08 9:30 ` Michal Nazarewicz 2012-02-08 9:30 ` Michal Nazarewicz 2012-02-09 0:03 ` Andrew Morton 2012-02-09 0:03 ` Andrew Morton 2012-02-09 8:08 ` Gilad Ben-Yossef 2012-02-09 8:08 ` Gilad Ben-Yossef 2012-02-09 8:13 ` Andrew Morton 2012-02-09 8:13 ` Andrew Morton 2012-02-09 9:53 ` Gilad Ben-Yossef 2012-02-09 9:53 ` Gilad Ben-Yossef 2012-02-05 13:48 ` Gilad Ben-Yossef [this message] 2012-02-05 13:48 ` [PATCH v8 5/8] slub: only IPI CPUs that have per cpu obj to flush Gilad Ben-Yossef 2012-02-05 13:48 ` [PATCH v8 6/8] fs: only send IPI to invalidate LRU BH when needed Gilad Ben-Yossef 2012-02-05 13:48 ` Gilad Ben-Yossef 2012-02-05 13:48 ` [PATCH v8 7/8] mm: only IPI CPUs to drain local pages if they exist Gilad Ben-Yossef 2012-02-05 13:48 ` Gilad Ben-Yossef 2012-02-08 9:33 ` Michal Nazarewicz 2012-02-08 9:33 ` Michal Nazarewicz 2012-02-09 8:09 ` Gilad Ben-Yossef 2012-02-09 8:09 ` Gilad Ben-Yossef 2012-02-05 13:48 ` [PATCH v8 8/8] mm: add vmstat counters for tracking PCP drains Gilad Ben-Yossef 2012-02-05 13:48 ` Gilad Ben-Yossef 2012-02-08 9:36 ` [PATCH v8 0/8] Reduce cross CPU IPI interference Michal Nazarewicz 2012-02-08 9:36 ` Michal Nazarewicz
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1328449722-15959-4-git-send-email-gilad@benyossef.com \ --to=gilad@benyossef.com \ --cc=akpm@linux-foundation.org \ --cc=andi@firstfloor.org \ --cc=avi@redhat.com \ --cc=cmetcalf@tilera.com \ --cc=fweisbec@gmail.com \ --cc=kosaki.motohiro@gmail.com \ --cc=levinsasha928@gmail.com \ --cc=linux-fsdevel@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux@arm.linux.org.uk \ --cc=mel@csn.ul.ie \ --cc=miltonm@bga.com \ --cc=mina86@mina86.com \ --cc=mpm@selenic.com \ --cc=penberg@kernel.org \ --cc=riel@redhat.com \ --cc=viro@zeniv.linux.org.uk \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.