From: Andrea Arcangeli <aarcange@redhat.com>
To: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
Hillf Danton <dhillf@gmail.com>, Dan Smith <danms@us.ibm.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Andrew Morton <akpm@linux-foundation.org>,
Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@elte.hu>,
Paul Turner <pjt@google.com>,
Suresh Siddha <suresh.b.siddha@intel.com>,
Mike Galbraith <efault@gmx.de>,
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Lai Jiangshan <laijs@cn.fujitsu.com>,
Bharata B Rao <bharata.rao@gmail.com>,
Lee Schermerhorn <Lee.Schermerhorn@hp.com>,
Rik van Riel <riel@redhat.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>,
Christoph Lameter <cl@linux.com>
Subject: Re: [PATCH 35/35] autonuma: page_autonuma
Date: Tue, 29 May 2012 19:14:07 +0200 [thread overview]
Message-ID: <20120529171407.GH21339@redhat.com> (raw)
In-Reply-To: <1338309855.26856.130.camel@twins>
On Tue, May 29, 2012 at 06:44:15PM +0200, Peter Zijlstra wrote:
> On Fri, 2012-05-25 at 19:02 +0200, Andrea Arcangeli wrote:
> > Move the AutoNUMA per page information from the "struct page" to a
> > separate page_autonuma data structure allocated in the memsection
> > (with sparsemem) or in the pgdat (with flatmem).
> >
> > This is done to avoid growing the size of the "struct page" and the
> > page_autonuma data is only allocated if the kernel has been booted on
> > real NUMA hardware (or if noautonuma is passed as parameter to the
> > kernel).
> >
>
> Argh, please fold this change back into the series proper. If you want
> to keep it.. as it is its not really an improvement IMO, see below.
The whole objective of this patch is to avoid allocating the
page_autonuma structures when the kernel is booted on not NUMA
hardware.
It's not an improvement when booting the kernel on NUMA hardware
that's for sure.
I didn't merge it with the previous because this was the most
experimental recent change, so I wanted bisectability here. When
something goes wrong here, the kernel won't boot, so unless you use
kvm with gdbstub it's a little tricky to debug (indeed I debugged it
with gdbstub, there it's trivial).
> > +struct page_autonuma {
> > + /*
> > + * FIXME: move to pgdat section along with the memcg and allocate
> > + * at runtime only in presence of a numa system.
> > + */
> > + /*
> > + * To modify autonuma_last_nid lockless the architecture,
> > + * needs SMP atomic granularity < sizeof(long), not all archs
> > + * have that, notably some alpha. Archs without that requires
> > + * autonuma_last_nid to be a long.
> > + */
>
> Looking at arch/alpha/include/asm/xchg.h it looks to have that just
> fine, so maybe we simply don't support SMP on those early Alphas that
> had that weirdness.
I agree we should never risk that.
> This makes a shadow page frame of 32 bytes per page, or ~0.8% of memory.
> This isn't in fact an improvement.
>
> The suggestion done by Rik was to have something like a sqrt(nr_pages)
> (?) scaled array of such things containing the list_head and page
> pointer -- and leave the two nids in the regular page frame. Although I
> think you've got to fight the memcg people over that last word in struct
> page.
>
> That places a limit on the amount of pages that can be in migration
> concurrently, but also greatly reduces the memory overhead.
Yes, however for the last_nid I'd still need it for every page (and if
I allocate it dynamic I still first need to find a way to remove the
struct page pointer).
I thought to add a pointer in the memsection (or maybe to use a vmemmap
so that it won't even require a pointer in every memsection). I've to
check a few more things before I allow_the autonuma->page translation
without a page pointer, notably to verify the boot time allocations
points won't just allocate power of two blocks of memory (they
shouldn't but I didn't verify).
This is clearly a move in the right direction to avoid the memory
overhead when not booted on NUMA hardware, and I don't think there's
anything fundamental that prevents us remove the page pointer from the
page_autonuma structure, and to later experiment with a limited size
array of async migration structures.
next prev parent reply other threads:[~2012-05-29 17:15 UTC|newest]
Thread overview: 121+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-05-25 17:02 [PATCH 00/35] AutoNUMA alpha14 Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 01/35] mm: add unlikely to the mm allocation failure check Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 02/35] autonuma: make set_pmd_at always available Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 03/35] xen: document Xen is using an unused bit for the pagetables Andrea Arcangeli
2012-05-25 20:26 ` Konrad Rzeszutek Wilk
2012-05-26 15:59 ` Andrea Arcangeli
2012-05-29 14:10 ` Konrad Rzeszutek Wilk
2012-05-29 16:01 ` Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 04/35] autonuma: define _PAGE_NUMA_PTE and _PAGE_NUMA_PMD Andrea Arcangeli
2012-05-30 18:22 ` Konrad Rzeszutek Wilk
2012-05-30 18:34 ` Andrea Arcangeli
2012-05-30 20:01 ` Konrad Rzeszutek Wilk
2012-06-05 17:13 ` Andrea Arcangeli
2012-06-05 17:17 ` Konrad Rzeszutek Wilk
2012-06-05 17:40 ` Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 05/35] autonuma: x86 pte_numa() and pmd_numa() Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 06/35] autonuma: generic " Andrea Arcangeli
2012-05-30 20:23 ` Konrad Rzeszutek Wilk
2012-05-25 17:02 ` [PATCH 07/35] autonuma: teach gup_fast about pte_numa Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 08/35] autonuma: introduce kthread_bind_node() Andrea Arcangeli
2012-05-29 12:49 ` Peter Zijlstra
2012-05-29 16:11 ` Andrea Arcangeli
2012-05-29 17:04 ` Peter Zijlstra
2012-05-29 17:44 ` Andrea Arcangeli
2012-05-29 17:48 ` Peter Zijlstra
2012-05-29 18:15 ` Andrea Arcangeli
2012-05-30 20:26 ` Konrad Rzeszutek Wilk
2012-05-25 17:02 ` [PATCH 09/35] autonuma: mm_autonuma and sched_autonuma data structures Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 10/35] autonuma: define the autonuma flags Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 11/35] autonuma: core autonuma.h header Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 12/35] autonuma: CPU follow memory algorithm Andrea Arcangeli
2012-05-29 13:00 ` Peter Zijlstra
2012-05-29 13:54 ` Rik van Riel
2012-05-29 13:10 ` Peter Zijlstra
2012-06-22 17:36 ` Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 13/35] autonuma: add page structure fields Andrea Arcangeli
2012-05-29 13:16 ` Peter Zijlstra
2012-05-29 13:56 ` Rik van Riel
2012-05-29 14:54 ` Peter Zijlstra
2012-05-30 8:25 ` KOSAKI Motohiro
2012-05-30 9:06 ` Peter Zijlstra
2012-05-30 9:41 ` KOSAKI Motohiro
2012-05-30 9:55 ` Peter Zijlstra
2012-05-30 13:49 ` Andrea Arcangeli
2012-05-31 18:18 ` Peter Zijlstra
2012-06-05 14:51 ` Andrea Arcangeli
2012-06-19 18:06 ` Andrea Arcangeli
2012-05-29 16:38 ` Andrea Arcangeli
2012-05-29 16:46 ` Rik van Riel
2012-05-29 16:56 ` Peter Zijlstra
2012-05-29 18:35 ` Andrea Arcangeli
2012-05-29 17:38 ` Linus Torvalds
2012-05-29 18:09 ` Andrea Arcangeli
2012-05-29 20:42 ` Rik van Riel
2012-05-25 17:02 ` [PATCH 14/35] autonuma: knuma_migrated per NUMA node queues Andrea Arcangeli
2012-05-29 13:51 ` Peter Zijlstra
2012-05-30 0:14 ` Andrea Arcangeli
2012-05-30 18:19 ` Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 15/35] autonuma: init knuma_migrated queues Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 16/35] autonuma: autonuma_enter/exit Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 17/35] autonuma: call autonuma_setup_new_exec() Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 18/35] autonuma: alloc/free/init sched_autonuma Andrea Arcangeli
2012-05-30 20:55 ` Konrad Rzeszutek Wilk
2012-05-25 17:02 ` [PATCH 19/35] autonuma: alloc/free/init mm_autonuma Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 20/35] autonuma: avoid CFS select_task_rq_fair to return -1 Andrea Arcangeli
2012-05-29 14:02 ` Peter Zijlstra
2012-05-25 17:02 ` [PATCH 21/35] autonuma: teach CFS about autonuma affinity Andrea Arcangeli
2012-05-29 16:05 ` Peter Zijlstra
2012-05-25 17:02 ` [PATCH 22/35] autonuma: sched_set_autonuma_need_balance Andrea Arcangeli
2012-05-29 16:12 ` Peter Zijlstra
2012-05-29 17:33 ` Andrea Arcangeli
2012-05-29 17:43 ` Peter Zijlstra
2012-05-29 18:24 ` Andrea Arcangeli
2012-05-29 22:21 ` Peter Zijlstra
2012-05-25 17:02 ` [PATCH 23/35] autonuma: core Andrea Arcangeli
2012-05-29 11:45 ` Kirill A. Shutemov
2012-05-30 0:03 ` Andrea Arcangeli
2012-05-29 16:27 ` Peter Zijlstra
2012-05-25 17:02 ` [PATCH 24/35] autonuma: follow_page check for pte_numa/pmd_numa Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 25/35] autonuma: default mempolicy follow AutoNUMA Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 26/35] autonuma: call autonuma_split_huge_page() Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 27/35] autonuma: make khugepaged pte_numa aware Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 28/35] autonuma: retain page last_nid information in khugepaged Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 29/35] autonuma: numa hinting page faults entry points Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 30/35] autonuma: reset autonuma page data when pages are freed Andrea Arcangeli
2012-05-29 16:30 ` Peter Zijlstra
2012-05-29 16:49 ` Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 31/35] autonuma: initialize page structure fields Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 32/35] autonuma: link mm/autonuma.o and kernel/sched/numa.o Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 33/35] autonuma: add CONFIG_AUTONUMA and CONFIG_AUTONUMA_DEFAULT_ENABLED Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 34/35] autonuma: boost khugepaged scanning rate Andrea Arcangeli
2012-05-25 17:02 ` [PATCH 35/35] autonuma: page_autonuma Andrea Arcangeli
2012-05-29 16:44 ` Peter Zijlstra
2012-05-29 17:14 ` Andrea Arcangeli [this message]
2012-05-26 17:28 ` [PATCH 00/35] AutoNUMA alpha14 Rik van Riel
2012-05-26 20:42 ` Linus Torvalds
2012-05-29 15:53 ` Christoph Lameter
2012-05-29 16:08 ` Andrea Arcangeli
2012-05-30 14:46 ` Peter Zijlstra
2012-05-30 15:30 ` Ingo Molnar
2012-05-29 13:36 ` Kirill A. Shutemov
2012-05-29 15:43 ` Petr Holasek
2012-05-31 18:08 ` AutoNUMA15 Andrea Arcangeli
2012-06-07 2:30 ` AutoNUMA15 Zhouping Liu
2012-06-07 11:44 ` AutoNUMA15 Hillf Danton
2012-06-07 13:30 ` AutoNUMA15 Andrea Arcangeli
2012-06-07 14:08 ` AutoNUMA15 Zhouping Liu
2012-06-07 19:37 ` AutoNUMA15 Andrea Arcangeli
2012-06-08 6:09 ` AutoNUMA15 Zhouping Liu
2012-06-08 13:04 ` AutoNUMA15 Hillf Danton
2012-06-08 13:32 ` AutoNUMA15 Peter Zijlstra
2012-06-08 16:31 ` AutoNUMA15 Zhouping Liu
2012-06-08 13:43 ` AutoNUMA15 Chen
2012-06-21 7:29 ` AutoNUMA15 Alex Shi
2012-06-21 14:55 ` AutoNUMA15 Andrea Arcangeli
2012-06-26 7:52 ` AutoNUMA15 Alex Shi
2012-06-26 12:03 ` AutoNUMA15 Andrea Arcangeli
2012-07-12 2:36 ` AutoNUMA15 Alex Shi
2012-05-29 17:15 ` [PATCH 00/35] AutoNUMA alpha14 Andrea Arcangeli
2012-06-01 22:41 ` Mauricio Faria de Oliveira
2012-06-22 17:57 ` Andrea Arcangeli
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120529171407.GH21339@redhat.com \
--to=aarcange@redhat.com \
--cc=Lee.Schermerhorn@hp.com \
--cc=a.p.zijlstra@chello.nl \
--cc=akpm@linux-foundation.org \
--cc=bharata.rao@gmail.com \
--cc=cl@linux.com \
--cc=danms@us.ibm.com \
--cc=dhillf@gmail.com \
--cc=efault@gmx.de \
--cc=hannes@cmpxchg.org \
--cc=laijs@cn.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mingo@elte.hu \
--cc=paulmck@linux.vnet.ibm.com \
--cc=pjt@google.com \
--cc=riel@redhat.com \
--cc=suresh.b.siddha@intel.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=vatsa@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).