From: Bron Gondwana <brong@fastmail.fm>
To: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: robm@fastmail.fm, linux-kernel@vger.kernel.org,
Bron Gondwana <brong@fastmail.fm>, linux-mm <linux-mm@kvack.org>,
Christoph Lameter <cl@linux-foundation.org>,
Mel Gorman <mel@csn.ul.ie>
Subject: Re: Default zone_reclaim_mode = 1 on NUMA kernel is bad for file/email/web servers
Date: Thu, 30 Sep 2010 10:38:14 +0200 [thread overview]
Message-ID: <20100930083814.GA4618@brong.net> (raw)
In-Reply-To: <20100916184240.3BC9.A69D9226@jp.fujitsu.com>
On Thu, Sep 16, 2010 at 07:01:32PM +0900, KOSAKI Motohiro wrote:
> Cc to linux-mm and hpc guys. and intetionally full quote.
>
>
> > So over the last couple of weeks, I've noticed that our shiny new IMAP
> > servers (Dual Xeon E5520 + Intel S5520UR MB) with 48G of RAM haven't
> > been performing as well as expected, and there were some big oddities.
> > Namely two things stuck out:
> >
> > 1. There was free memory. There's 20T of data on these machines. The
> > kernel should have used lots of memory for caching, but for some
> > reason, it wasn't. cache ~ 2G, buffers ~ 25G, unused ~ 5G
> > 2. The machine has an SSD for very hot data. In total, there's about 16G
> > of data on the SSD. Almost all of that 16G of data should end up
> > being cached, so there should be little reading from the SSDs at all.
> > Instead we saw at peak times 2k+ blocks read/s from the SSDs. Again a
> > sign that caching wasn't working.
> >
> > After a bunch of googling, I found this thread.
> >
> > http://lkml.org/lkml/2009/5/12/586
> >
> > It appears that patch never went anywhere, and zone_reclaim_mode is
> > still defaulting to 1 on our pretty standard file/email/web server type
> > machine with a NUMA kernel.
> >
> > By changing it to 0, we saw an immediate massive change in caching
> > behaviour. Now cache ~ 27G, buffers ~ 7G and unused ~ 0.2G, and IO reads
> > from the SSD dropped to 100/s instead of 2000/s.
Apropos to all this, look what's showed up:
http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
More fun with NUMA. Though in the Mysql case I can see that there's no easy
answer because there really is one big process chewing most of the RAM.
The question in our case is: why isn't the kernel balancing the multiple
separate Cyrus instances across all the nodes? And why, as one of the
comments there says, isn't swap to NUMA being considered cheaper than
swap to disk! That's the real problem here - that Linux is considering
accessing remote RAM to be more expensive than accessing disk!
Bron.
WARNING: multiple messages have this Message-ID (diff)
From: Bron Gondwana <brong@fastmail.fm>
To: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: robm@fastmail.fm, linux-kernel@vger.kernel.org,
Bron Gondwana <brong@fastmail.fm>, linux-mm <linux-mm@kvack.org>,
Christoph Lameter <cl@linux-foundation.org>,
Mel Gorman <mel@csn.ul.ie>
Subject: Re: Default zone_reclaim_mode = 1 on NUMA kernel is bad for file/email/web servers
Date: Thu, 30 Sep 2010 10:38:14 +0200 [thread overview]
Message-ID: <20100930083814.GA4618@brong.net> (raw)
In-Reply-To: <20100916184240.3BC9.A69D9226@jp.fujitsu.com>
On Thu, Sep 16, 2010 at 07:01:32PM +0900, KOSAKI Motohiro wrote:
> Cc to linux-mm and hpc guys. and intetionally full quote.
>
>
> > So over the last couple of weeks, I've noticed that our shiny new IMAP
> > servers (Dual Xeon E5520 + Intel S5520UR MB) with 48G of RAM haven't
> > been performing as well as expected, and there were some big oddities.
> > Namely two things stuck out:
> >
> > 1. There was free memory. There's 20T of data on these machines. The
> > kernel should have used lots of memory for caching, but for some
> > reason, it wasn't. cache ~ 2G, buffers ~ 25G, unused ~ 5G
> > 2. The machine has an SSD for very hot data. In total, there's about 16G
> > of data on the SSD. Almost all of that 16G of data should end up
> > being cached, so there should be little reading from the SSDs at all.
> > Instead we saw at peak times 2k+ blocks read/s from the SSDs. Again a
> > sign that caching wasn't working.
> >
> > After a bunch of googling, I found this thread.
> >
> > http://lkml.org/lkml/2009/5/12/586
> >
> > It appears that patch never went anywhere, and zone_reclaim_mode is
> > still defaulting to 1 on our pretty standard file/email/web server type
> > machine with a NUMA kernel.
> >
> > By changing it to 0, we saw an immediate massive change in caching
> > behaviour. Now cache ~ 27G, buffers ~ 7G and unused ~ 0.2G, and IO reads
> > from the SSD dropped to 100/s instead of 2000/s.
Apropos to all this, look what's showed up:
http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
More fun with NUMA. Though in the Mysql case I can see that there's no easy
answer because there really is one big process chewing most of the RAM.
The question in our case is: why isn't the kernel balancing the multiple
separate Cyrus instances across all the nodes? And why, as one of the
comments there says, isn't swap to NUMA being considered cheaper than
swap to disk! That's the real problem here - that Linux is considering
accessing remote RAM to be more expensive than accessing disk!
Bron.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2010-09-30 8:38 UTC|newest]
Thread overview: 61+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-09-13 3:39 Default zone_reclaim_mode = 1 on NUMA kernel is bad for file/email/web servers Robert Mueller
2010-09-16 10:01 ` KOSAKI Motohiro
2010-09-16 10:01 ` KOSAKI Motohiro
2010-09-16 17:06 ` Christoph Lameter
2010-09-16 17:06 ` Christoph Lameter
2010-09-17 0:50 ` Robert Mueller
2010-09-17 0:50 ` Robert Mueller
2010-09-17 6:01 ` Shaohua Li
2010-09-17 6:01 ` Shaohua Li
2010-09-17 7:32 ` Robert Mueller
2010-09-17 7:32 ` Robert Mueller
2010-09-17 13:56 ` Christoph Lameter
2010-09-17 13:56 ` Christoph Lameter
2010-09-17 14:09 ` Bron Gondwana
2010-09-17 14:09 ` Bron Gondwana
2010-09-17 14:22 ` Christoph Lameter
2010-09-17 14:22 ` Christoph Lameter
2010-09-17 23:01 ` Bron Gondwana
2010-09-17 23:01 ` Bron Gondwana
2010-09-20 9:34 ` Mel Gorman
2010-09-20 9:34 ` Mel Gorman
2010-09-20 23:41 ` Default zone_reclaim_mode = 1 on NUMA kernel is bad forfile/email/web servers Rob Mueller
2010-09-20 23:41 ` Rob Mueller
2010-09-21 9:04 ` Mel Gorman
2010-09-21 9:04 ` Mel Gorman
2010-09-21 14:14 ` Christoph Lameter
2010-09-21 14:14 ` Christoph Lameter
2010-09-22 3:44 ` Rob Mueller
2010-09-22 3:44 ` Rob Mueller
2010-09-27 2:01 ` KOSAKI Motohiro
2010-09-27 2:01 ` KOSAKI Motohiro
2010-09-27 13:53 ` Christoph Lameter
2010-09-27 13:53 ` Christoph Lameter
2010-09-27 23:17 ` Robert Mueller
2010-09-27 23:17 ` Robert Mueller
2010-09-28 12:35 ` Christoph Lameter
2010-09-28 12:35 ` Christoph Lameter
2010-09-28 12:42 ` Bron Gondwana
2010-09-28 12:42 ` Bron Gondwana
2010-09-28 12:49 ` Christoph Lameter
2010-09-28 12:49 ` Christoph Lameter
2010-09-30 7:05 ` Andi Kleen
2010-09-30 7:05 ` Andi Kleen
2010-10-04 12:45 ` KOSAKI Motohiro
2010-10-04 12:45 ` KOSAKI Motohiro
2010-10-04 13:07 ` Christoph Lameter
2010-10-04 13:07 ` Christoph Lameter
2010-10-05 5:32 ` KOSAKI Motohiro
2010-10-05 5:32 ` KOSAKI Motohiro
2010-10-04 19:43 ` David Rientjes
2010-10-04 19:43 ` David Rientjes
2010-09-21 1:05 ` Default zone_reclaim_mode = 1 on NUMA kernel is bad for file/email/web servers KAMEZAWA Hiroyuki
2010-09-21 1:05 ` KAMEZAWA Hiroyuki
2010-09-27 2:04 ` KOSAKI Motohiro
2010-09-27 2:04 ` KOSAKI Motohiro
2010-09-27 2:06 ` KAMEZAWA Hiroyuki
2010-09-27 2:06 ` KAMEZAWA Hiroyuki
2010-09-23 11:44 ` Balbir Singh
2010-09-23 11:44 ` Balbir Singh
2010-09-30 8:38 ` Bron Gondwana [this message]
2010-09-30 8:38 ` Bron Gondwana
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100930083814.GA4618@brong.net \
--to=brong@fastmail.fm \
--cc=cl@linux-foundation.org \
--cc=kosaki.motohiro@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mel@csn.ul.ie \
--cc=robm@fastmail.fm \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.