(no subject)

[Andrea Arcangeli]

Subject: Re: [PATCH 29/33] autonuma: page_autonuma
Reply-To: 
In-Reply-To: <0000013a2c223da2-632aa43e-21f8-4abd-a0ba-2e1b49881e3a-000000@email.amazonses.com>

Hi Christoph,

On Thu, Oct 04, 2012 at 02:16:14PM +0000, Christoph Lameter wrote:
> On Thu, 4 Oct 2012, Andrea Arcangeli wrote:
> 
> > Move the autonuma_last_nid from the "struct page" to a separate
> > page_autonuma data structure allocated in the memsection (with
> > sparsemem) or in the pgdat (with flatmem).
> 
> Note that there is a available word in struct page before the autonuma
> patches on x86_64 with CONFIG_HAVE_ALIGNED_STRUCT_PAGE.
> 
> In fact the page_autonuma fills up the structure to nicely fit in one 64
> byte cacheline.

Good point indeed.

So we could drop page_autonuma by creating a CONFIG_SLUB=y dependency
(AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it
also wouldn't be available on 32bit archs but the latter isn't a
problem).

I think it's a reasonable alternative to page_autonuma. Certainly it
looks more appealing than taking over 16 precious bits from
page->flags. There are still pros and cons. I'm neutral on it so more
comments would be welcome ;).

Andrea

PS. randomly moved some in Cc over to Bcc as I overflowed the max
header allowed on linux-kernel oops!

Re: your mail

[Christoph Lameter]

On Thu, 4 Oct 2012, Andrea Arcangeli wrote:

> So we could drop page_autonuma by creating a CONFIG_SLUB=y dependency
> (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it
> also wouldn't be available on 32bit archs but the latter isn't a
> problem).

Nope it should depend on page struct alignment. Other kernel subsystems
may be depeding on page struct alignment in the future (and some other
arches may already have that requirement)

Re: [PATCH 29/33] autonuma: page_autonuma

[Andrea Arcangeli]

Hi Christoph,

On Thu, Oct 04, 2012 at 06:17:37PM +0000, Christoph Lameter wrote:
> On Thu, 4 Oct 2012, Andrea Arcangeli wrote:
> 
> > So we could drop page_autonuma by creating a CONFIG_SLUB=y dependency
> > (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it
> > also wouldn't be available on 32bit archs but the latter isn't a
> > problem).
> 
> Nope it should depend on page struct alignment. Other kernel subsystems
> may be depeding on page struct alignment in the future (and some other
> arches may already have that requirement)

But currently only SLUB x86 64bit selects
CONFIG_HAVE_ALIGNED_STRUCT_PAGE:

arch/Kconfig:config HAVE_ALIGNED_STRUCT_PAGE
arch/x86/Kconfig:       select HAVE_ALIGNED_STRUCT_PAGE if SLUB && !M386
include/linux/mm_types.h:       defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
include/linux/mm_types.h:#ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
mm/slub.c:    defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
mm/slub.c:    defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
mm/slub.c:    defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)

So in practice a dependency on CONFIG_HAVE_ALIGNED_STRUCT_PAGE would
still mean the same: only available when SLUB enables it, and only on
x86 64bit (ppc64?).

If you mean CONFIG_AUTONUMA=y should select (not depend) on
CONFIG_HAVE_ALIGNED_STRUCT_PAGE, that would allow to enable it in all
.configs but it would have a worse cons: losing 8bytes per page
unconditionally (even when booting on non-NUMA hardware).

The current page_autonuma solution is substantially memory-cheaper
than selecting CONFIG_HAVE_ALIGNED_STRUCT_PAGE: it allocates 2bytes
per page at boot time but only if booting on real NUMA hardware
(without altering the page structure). So to me it looks still quite a
decent tradeoff.

Thanks,
Andrea

Re: [PATCH 29/33] autonuma: page_autonuma

[Christoph Lameter]

On Thu, 4 Oct 2012, Andrea Arcangeli wrote:

> If you mean CONFIG_AUTONUMA=y should select (not depend) on
> CONFIG_HAVE_ALIGNED_STRUCT_PAGE, that would allow to enable it in all
> .configs but it would have a worse cons: losing 8bytes per page
> unconditionally (even when booting on non-NUMA hardware).

I did not say anything like that. Still not convinced that autonuma is
worth doing and that it is beneficial given the complexity it adds to the
kernel. Just wanted to point out that there is a case to be made for
adding another word to the page struct.

Re: [PATCH 29/33] autonuma: page_autonuma

[Andrea Arcangeli]

Hi Christoph,

On Thu, Oct 04, 2012 at 07:11:51PM +0000, Christoph Lameter wrote:
> I did not say anything like that. Still not convinced that autonuma is
> worth doing and that it is beneficial given the complexity it adds to the
> kernel. Just wanted to point out that there is a case to be made for
> adding another word to the page struct.

You've seen the benchmarks, no other solution that exists today solves
all those cases and never showed a regression compared to
upstream. Running that much faster is very beneficial in my
view.

Expecting the admin of a 2 socket system to use hard bindings manually
is unrealistic, even for a 4 socket is unrealistic.

If you've 512 node system well then you can afford to setup everything
manually and boot with noautonuma, no argument about that.

About the complexity, well there's no simple solution to an hard
problem. The proof comes from the schednuma crowd that is currently
copying the AutoNUMA scheduler cpu-follow-memory design at full force
as we speak.

Thanks,
Andrea