From: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
To: Pekka Enberg <penberg@cs.helsinki.fi>
Cc: Christoph Lameter <cl@linux-foundation.org>,
Andi Kleen <andi@firstfloor.org>, Matthew Wilcox <matthew@wil.cx>,
Nick Piggin <nickpiggin@yahoo.com.au>,
Andrew Morton <akpm@linux-foundation.org>,
netdev@vger.kernel.org, sfr@canb.auug.org.au,
matthew.r.wilcox@intel.com, chinang.ma@intel.com,
linux-kernel@vger.kernel.org, sharad.c.tripathi@intel.com,
arjan@linux.intel.com, suresh.b.siddha@intel.com,
harita.chilukuri@intel.com, douglas.w.styner@intel.com,
peter.xihong.wang@intel.com, hubert.nueckel@intel.com,
chris.mason@oracle.com, srostedt@redhat.com,
linux-scsi@vger.kernel.org, andrew.vasquez@qlogic.com,
anirban.chakraborty@qlogic.com
Subject: Re: Mainline kernel OLTP performance update
Date: Fri, 23 Jan 2009 11:02:53 +0800 [thread overview]
Message-ID: <1232679773.11429.155.camel@ymzhang> (raw)
In-Reply-To: <1232617672.14549.25.camel@penberg-laptop>
On Thu, 2009-01-22 at 11:47 +0200, Pekka Enberg wrote:
> On Thu, 2009-01-22 at 17:28 +0800, Zhang, Yanmin wrote:
> > On Thu, 2009-01-22 at 11:15 +0200, Pekka Enberg wrote:
> > > On Thu, 2009-01-22 at 16:36 +0800, Zhang, Yanmin wrote:
> > > > On Wed, 2009-01-21 at 18:58 -0500, Christoph Lameter wrote:
> > > > > On Tue, 20 Jan 2009, Zhang, Yanmin wrote:
> > > > >
> > > > > > kmem_cache skbuff_head_cache's object size is just 256, so it shares the kmem_cache
> > > > > > with :0000256. Their order is 1 which means every slab consists of 2 physical pages.
> > > > >
> > > > > That order can be changed. Try specifying slub_max_order=0 on the kernel
> > > > > command line to force an order 0 alloc.
> > > > I tried slub_max_order=0 and there is no improvement on this UDP-U-4k issue.
> > > > Both get_page_from_freelist and __free_pages_ok's cpu time are still very high.
> > > >
> > > > I checked my instrumentation in kernel and found it's caused by large object allocation/free
> > > > whose size is more than PAGE_SIZE. Here its order is 1.
> > > >
> > > > The right free callchain is __kfree_skb => skb_release_all => skb_release_data.
> > > >
> > > > So this case isn't the issue that batch of allocation/free might erase partial page
> > > > functionality.
> > >
> > > So is this the kfree(skb->head) in skb_release_data() or the put_page()
> > > calls in the same function in a loop?
> > It's kfree(skb->head).
> >
> > >
> > > If it's the former, with big enough size passed to __alloc_skb(), the
> > > networking code might be taking a hit from the SLUB page allocator
> > > pass-through.
>
> Do we know what kind of size is being passed to __alloc_skb() in this
> case?
In function __alloc_skb, original parameter size=4155,
SKB_DATA_ALIGN(size)=4224, sizeof(struct skb_shared_info)=472, so
__kmalloc_track_caller's parameter size=4696.
> Maybe we want to do something like this.
>
> Pekka
>
> SLUB: revert page allocator pass-through
This patch amost fixes the netperf UDP-U-4k issue.
#slabinfo -AD
Name Objects Alloc Free %Fast
:0000256 1658 70350463 70348946 99 99
kmalloc-8192 31 70322309 70322293 99 99
:0000168 2592 143154 140684 93 28
:0004096 1456 91072 89644 99 96
:0000192 3402 63838 60491 89 11
:0000064 6177 49635 43743 98 77
So kmalloc-8192 appears. Without the patch, kmalloc-8192 hides.
kmalloc-8192's default order on my 8-core stoakley is 2.
1) If I start CPU_NUM clients and servers, SLUB's result is about 2% better than SLQB's;
2) If I start 1 clinet and 1 server, and bind them to different physical cpu, SLQB's result
is about 10% better than SLUB's.
I don't know why there is still 10% difference with item 2). Maybe cachemiss causes it?
>
> This is a revert of commit aadb4bc4a1f9108c1d0fbd121827c936c2ed4217 ("SLUB:
> direct pass through of page size or higher kmalloc requests").
> ---
>
> diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h
> index 2f5c16b..3bd3662 100644
> --- a/include/linux/slub_def.h
> +++ b/include/linux/slub_def.h
next prev parent reply other threads:[~2009-01-23 3:03 UTC|newest]
Thread overview: 122+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-01-13 21:10 Mainline kernel OLTP performance update Ma, Chinang
2009-01-13 22:44 ` Wilcox, Matthew R
2009-01-15 0:35 ` Andrew Morton
2009-01-15 1:21 ` Matthew Wilcox
2009-01-15 2:04 ` Andrew Morton
2009-01-15 2:27 ` Steven Rostedt
2009-01-15 7:11 ` Ma, Chinang
2009-01-19 18:04 ` Chris Mason
2009-01-19 18:37 ` Steven Rostedt
2009-01-19 18:55 ` Chris Mason
2009-01-19 19:07 ` Steven Rostedt
2009-01-19 23:40 ` Ingo Molnar
2009-01-15 2:39 ` Andi Kleen
2009-01-15 2:47 ` Matthew Wilcox
2009-01-15 3:36 ` Andi Kleen
2009-01-20 13:27 ` Jens Axboe
[not found] ` <588992150B702C48B3312184F1B810AD03A497632C@azsmsx501.amr.corp.intel.com>
2009-01-22 11:29 ` Jens Axboe
[not found] ` <588992150B702C48B3312184F1B810AD03A4F59632@azsmsx501.amr.corp.intel.com>
2009-01-27 8:28 ` Jens Axboe
2009-01-15 7:24 ` Nick Piggin
2009-01-15 9:46 ` Pekka Enberg
2009-01-15 13:52 ` Matthew Wilcox
2009-01-15 14:42 ` Pekka Enberg
2009-01-16 10:16 ` Pekka Enberg
2009-01-16 10:21 ` Nick Piggin
2009-01-16 10:31 ` Pekka Enberg
2009-01-16 10:42 ` Nick Piggin
2009-01-16 10:55 ` Pekka Enberg
2009-01-19 7:13 ` Nick Piggin
2009-01-19 8:05 ` Pekka Enberg
2009-01-19 8:33 ` Nick Piggin
2009-01-19 8:42 ` Nick Piggin
2009-01-19 8:47 ` Pekka Enberg
2009-01-19 8:57 ` Nick Piggin
2009-01-19 9:48 ` Pekka Enberg
2009-01-19 10:03 ` Nick Piggin
2009-01-16 20:59 ` Christoph Lameter
2009-01-16 0:27 ` Andrew Morton
2009-01-16 4:03 ` Nick Piggin
2009-01-16 4:12 ` Andrew Morton
2009-01-16 6:46 ` Nick Piggin
2009-01-16 6:55 ` Matthew Wilcox
2009-01-16 7:06 ` Nick Piggin
2009-01-16 7:53 ` Zhang, Yanmin
2009-01-16 10:20 ` Andi Kleen
2009-01-20 5:16 ` Zhang, Yanmin
2009-01-21 23:58 ` Christoph Lameter
2009-01-22 8:36 ` Zhang, Yanmin
2009-01-22 9:15 ` Pekka Enberg
2009-01-22 9:28 ` Zhang, Yanmin
2009-01-22 9:47 ` Pekka Enberg
2009-01-23 3:02 ` Zhang, Yanmin [this message]
2009-01-23 6:52 ` Pekka Enberg
2009-01-23 8:06 ` Pekka Enberg
2009-01-23 8:30 ` Zhang, Yanmin
2009-01-23 8:40 ` Pekka Enberg
2009-01-23 9:46 ` Pekka Enberg
2009-01-23 15:22 ` Christoph Lameter
2009-01-23 15:31 ` Pekka Enberg
2009-01-23 15:55 ` Christoph Lameter
2009-01-23 16:01 ` Pekka Enberg
2009-01-24 2:55 ` Zhang, Yanmin
2009-01-24 7:36 ` Pekka Enberg
2009-02-12 5:22 ` Zhang, Yanmin
2009-02-12 5:47 ` Zhang, Yanmin
2009-02-12 15:25 ` Christoph Lameter
2009-02-12 16:07 ` Pekka Enberg
2009-02-12 16:03 ` Pekka Enberg
2009-01-26 17:36 ` Christoph Lameter
2009-02-01 2:52 ` Zhang, Yanmin
2009-01-23 8:33 ` Nick Piggin
2009-01-23 9:02 ` Zhang, Yanmin
2009-01-23 18:40 ` care and feeding of netperf (Re: Mainline kernel OLTP performance update) Rick Jones
2009-01-23 18:51 ` Grant Grundler
2009-01-24 3:03 ` Zhang, Yanmin
2009-01-26 18:26 ` Rick Jones
2009-01-16 7:00 ` Mainline kernel OLTP performance update Andrew Morton
2009-01-16 7:25 ` Nick Piggin
2009-01-16 8:59 ` Nick Piggin
2009-01-16 18:11 ` Rick Jones
2009-01-19 7:43 ` Nick Piggin
2009-01-19 22:19 ` Rick Jones
2009-01-15 14:12 ` James Bottomley
2009-01-15 17:44 ` Andrew Morton
2009-01-15 18:00 ` Matthew Wilcox
2009-01-15 18:14 ` Steven Rostedt
2009-01-15 18:44 ` Gregory Haskins
2009-01-15 18:46 ` Wilcox, Matthew R
2009-01-15 19:44 ` Ma, Chinang
2009-01-16 18:14 ` Gregory Haskins
2009-01-16 19:09 ` Steven Rostedt
2009-01-20 12:45 ` Gregory Haskins
2009-01-15 19:28 ` Ma, Chinang
2009-01-15 16:48 ` Ma, Chinang
-- strict thread matches above, loose matches on Subject: below --
2010-01-25 18:26 Ma, Chinang
2009-05-04 15:54 Styner, Douglas W
2009-05-06 6:29 ` Anirban Chakraborty
2009-05-06 15:53 ` Wilcox, Matthew R
2009-05-06 18:05 ` Styner, Douglas W
2009-05-06 18:12 ` Wilcox, Matthew R
2009-05-06 18:24 ` Anirban Chakraborty
2009-05-06 19:25 ` Wilcox, Matthew R
2009-05-06 18:19 ` Styner, Douglas W
2009-04-28 17:22 Styner, Douglas W
2009-04-28 17:08 Styner, Douglas W
2009-04-29 7:29 ` Andrew Morton
2009-04-29 8:28 ` Andi Kleen
2009-04-29 16:00 ` Styner, Douglas W
2009-04-29 16:06 ` Wilcox, Matthew R
2009-04-29 16:19 ` Andi Kleen
2009-04-29 15:48 ` Styner, Douglas W
2009-04-29 16:07 ` Andrew Morton
2009-04-29 16:25 ` Peter Zijlstra
2009-04-29 17:46 ` Chris Mason
2009-04-29 18:06 ` Pallipadi, Venkatesh
2009-04-29 18:25 ` Styner, Douglas W
2009-04-29 17:52 ` Styner, Douglas W
2009-04-23 16:49 Styner, Douglas W
2009-04-27 7:02 ` Andi Kleen
2009-04-28 16:57 ` Chuck Ebbert
2009-04-28 17:15 ` James Bottomley
2009-04-28 17:17 ` Styner, Douglas W
2009-01-12 18:30 Ma, Chinang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1232679773.11429.155.camel@ymzhang \
--to=yanmin_zhang@linux.intel.com \
--cc=akpm@linux-foundation.org \
--cc=andi@firstfloor.org \
--cc=andrew.vasquez@qlogic.com \
--cc=anirban.chakraborty@qlogic.com \
--cc=arjan@linux.intel.com \
--cc=chinang.ma@intel.com \
--cc=chris.mason@oracle.com \
--cc=cl@linux-foundation.org \
--cc=douglas.w.styner@intel.com \
--cc=harita.chilukuri@intel.com \
--cc=hubert.nueckel@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=matthew.r.wilcox@intel.com \
--cc=matthew@wil.cx \
--cc=netdev@vger.kernel.org \
--cc=nickpiggin@yahoo.com.au \
--cc=penberg@cs.helsinki.fi \
--cc=peter.xihong.wang@intel.com \
--cc=sfr@canb.auug.org.au \
--cc=sharad.c.tripathi@intel.com \
--cc=srostedt@redhat.com \
--cc=suresh.b.siddha@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).