Re: Is it possible to implement the per-node page cache for programs/libraries?

From: Barry Song <21cnbao@gmail.com>
To: Shijie Huang <shijie@amperemail.onmicrosoft.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
	viro@zeniv.linux.org.uk,
	Andrew Morton <akpm@linux-foundation.org>,
	linux-mm@kvack.org, Barry Song <song.bao.hua@hisilicon.com>,
	LKML <linux-kernel@vger.kernel.org>,
	Frank Wang <zwang@amperecomputing.com>
Subject: Re: Is it possible to implement the per-node page cache for programs/libraries?
Date: Wed, 1 Sep 2021 10:09:24 +0800	[thread overview]
Message-ID: <CAGsJ_4xtsaQL47D2D6PyW3vAr3+EwJqv-G-1V4-4GAYd2LnT3Q@mail.gmail.com> (raw)
In-Reply-To: <a2f423cf-9413-6bc8-e4d8-92374fc0449e@amperemail.onmicrosoft.com>

On Wed, Sep 1, 2021 at 11:09 AM Shijie Huang
<shijie@amperemail.onmicrosoft.com> wrote:
>
> Hi Everyone,
>
>      In the NUMA, we only have one page cache for each file. For the
> program/shared libraries, the
>
> remote-access delays longer then the  local-access.
>
> So, is it possible to implement the per-node page cache for
> programs/libraries?

as far as i know, this is an very interesting topic, we do have some
"solutions" on this.
MIPS kernel supports kernel TEXT replication:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/mips/sgi-ip27/Kconfig
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/mips/sgi-ip27/ip27-klnuma.c

config REPLICATE_KTEXT
bool "Kernel text replication support"
depends on SGI_IP27
select MAPPED_KERNEL
help
 Say Y here to enable replicating the kernel text across multiple
 nodes in a NUMA cluster.  This trades memory for speed.

for x86, RedHawk Linux（https://www.concurrent-rt.com/solutions/linux/）supports
kernel text replication.
here are some benchmark:
https://www.concurrent-rt.com/wp-content/uploads/2016/11/kernel-page-replication.pdf

For userspace, dplace from SGI can help replicate text:
https://www.spec.org/cpu2006/flags/SGI-platform.html

-r bl: specifies that text should be replicated on the NUMA node or
nodes where the process is running.
'b' indicates that binary (a.out) text should be replicated;
'l' indicates that library text should be replicated.

but all of the above except mips ktext replication are out of tree.

Please count me in if you have any solution and any pending patch.
I am interested in this topic.

>
>
>     We can do it like this:
>
>          1.) Add a new system call to control specific files to
> NUMA-aware, such as:
>
>                     set_numa_aware("/usr/lib/libc.so", enable);
>
>              After the system call, the page cache of libc.so has the
> flags "NUMA_ENABLED"
>
>
>          2.) When A new process tries to setup the MMU page table for
> libc.so, it will check
>
>               if NUMA_ENABLED is set. If it set, the kernel will give a
> page which is bind to the process's NUMA node.
>
>               By this way, we can eliminate the remote-access for
> programs/shared library.
>
>
> Is this proposal ok?  Or do you have a better idea?
>
>
> Thanks
>
> Huang Shijie

Thanks
barry