From: Gavin Shan <gwshan@linux.vnet.ibm.com>
To: Anton Blanchard <anton@samba.org>
Cc: Balbir Singh <bsingharora@gmail.com>,
Gavin Shan <gwshan@linux.vnet.ibm.com>,
linuxppc-dev@lists.ozlabs.org
Subject: Re: [PATCH] powerpc/mm: Fix RECLAIM_DISTANCE
Date: Mon, 30 Jan 2017 15:38:23 +1100 [thread overview]
Message-ID: <20170130043823.GA30920@gwshan> (raw)
In-Reply-To: <20170130120240.5018f476@kryten>
On Mon, Jan 30, 2017 at 12:02:40PM +1100, Anton Blanchard wrote:
>> Anton suggested that NUMA distances in powerpc mattered and hurted
>> performance without this setting. We need to validate to see if this
>> is still true. A simple way to start would be benchmarking
>
>The original issue was that we never reclaimed local clean pagecache.
>
>I just tried all settings for /proc/sys/vm/zone_reclaim_mode and none
>of them caused me to reclaim local clean pagecache! We are very broken.
>
>I would think we have test cases for this, but here is a dumb one.
>First something to consume memory:
>
># cat alloc.c
>
>#include <stdlib.h>
>#include <unistd.h>
>#include <string.h>
>#include <assert.h>
>
>int main(int argc, char *argv[])
>{
> void *p;
>
> unsigned long size;
>
> size = strtoul(argv[1], NULL, 0);
>
> p = malloc(size);
> assert(p);
> memset(p, 0, size);
> printf("%p\n", p);
>
> sleep(3600);
>
> return 0;
>}
>
>Now create a file to consume pagecache. My nodes have 32GB each, so
>I create 16GB, enough to consume half of the node:
>
>dd if=/dev/zero of=/tmp/file bs=1G count=16
>
>Clear out our pagecache:
>
>sync
>echo 3 > /proc/sys/vm/drop_caches
>
>Bring it in on node 0:
>
>taskset -c 0 cat /tmp/file > /dev/null
>
>Consume 24GB of memory on node 0:
>
>taskset -c 0 ./alloc 25769803776
>
>In all zone reclaim modes, the pagecache never gets reclaimed:
>
># grep FilePages /sys/devices/system/node/node0/meminfo
>
>Node 0 FilePages: 16757376 kB
>
>And our alloc process shows lots of off node memory used:
>
>3ff9a4630000 default anon=393217 dirty=393217 N0=112474 N1=220490 N16=60253 kernelpagesize_kB=64
>
>Clearly nothing is working. Gavin, if your patch fixes this we should
>get it into stable too.
>
Anton, thanks for the detailed test case. I tried what you suggested
on the box that has only one node. The memory capacity is 16GB. So
the parameters I used are different from what you had. First of all,
I observed same behaviour that the pagecache can't be reclaimed when
allocating memory for heap. With the patch applied, the pagecache
can be dropped for page reclaim and more details are showed as below
Everything looks good. I'll put your testcase, its result and stable tag
to next revision.
root@palm8:/home/gavin# grep FilePages /sys/devices/system/node/node0/meminfo
Node 0 FilePages: 142400 kB
root@palm8:/home/gavin# sync
root@palm8:/home/gavin# echo 3 > /proc/sys/vm/drop_caches
root@palm8:/home/gavin# grep FilePages /sys/devices/system/node/node0/meminfo
Node 0 FilePages: 62848 kB
root@palm8:/home/gavin# du -sh file
8.1G file
root@palm8:/home/gavin# cat file > /dev/null
root@palm8:/home/gavin# grep FilePages /sys/devices/system/node/node0/meminfo
Node 0 FilePages: 8448000 kB
root@palm8:/home/gavin# ./alloc 17179869184
root@palm8:/home/gavin# grep FilePages /sys/devices/system/node/node0/meminfo
Node 0 FilePages: 387584 kB
Thanks,
Gavin
next prev parent reply other threads:[~2017-01-30 4:39 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-23 23:32 [PATCH] powerpc/mm: Fix RECLAIM_DISTANCE Gavin Shan
2017-01-25 3:57 ` Balbir Singh
2017-01-25 4:58 ` Gavin Shan
2017-01-27 12:49 ` Balbir Singh
2017-01-30 1:02 ` Anton Blanchard
2017-01-30 4:38 ` Gavin Shan [this message]
2017-01-30 21:11 ` Michael Ellerman
2017-01-31 5:01 ` Gavin Shan
2017-01-31 5:40 ` Gavin Shan
2017-02-07 23:40 ` Gavin Shan
2017-01-31 4:33 ` Gavin Shan
2017-01-31 4:58 ` Anton Blanchard
2017-01-31 5:30 ` Gavin Shan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170130043823.GA30920@gwshan \
--to=gwshan@linux.vnet.ibm.com \
--cc=anton@samba.org \
--cc=bsingharora@gmail.com \
--cc=linuxppc-dev@lists.ozlabs.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.