Re: [RFC 00/14] Dynamic Kernel Stacks

From: Pasha Tatashin <pasha.tatashin@soleen.com>
To: Matthew Wilcox <willy@infradead.org>
Cc: David Laight <David.Laight@aculab.com>,
	"H. Peter Anvin" <hpa@zytor.com>,
	 Kent Overstreet <kent.overstreet@linux.dev>,
	 "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-mm@kvack.org" <linux-mm@kvack.org>,
	 "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
	"x86@kernel.org" <x86@kernel.org>,  "bp@alien8.de" <bp@alien8.de>,
	"brauner@kernel.org" <brauner@kernel.org>,
	 "bristot@redhat.com" <bristot@redhat.com>,
	"bsegall@google.com" <bsegall@google.com>,
	 "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	 "dianders@chromium.org" <dianders@chromium.org>,
	 "dietmar.eggemann@arm.com" <dietmar.eggemann@arm.com>,
	 "eric.devolder@oracle.com" <eric.devolder@oracle.com>,
	"hca@linux.ibm.com" <hca@linux.ibm.com>,
	 "hch@infradead.org" <hch@infradead.org>,
	 "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
	"jgg@ziepe.ca" <jgg@ziepe.ca>,
	 "jpoimboe@kernel.org" <jpoimboe@kernel.org>,
	"jroedel@suse.de" <jroedel@suse.de>,
	 "juri.lelli@redhat.com" <juri.lelli@redhat.com>,
	"kinseyho@google.com" <kinseyho@google.com>,
	 "kirill.shutemov@linux.intel.com"
	<kirill.shutemov@linux.intel.com>,
	"lstoakes@gmail.com" <lstoakes@gmail.com>,
	 "luto@kernel.org" <luto@kernel.org>,
	"mgorman@suse.de" <mgorman@suse.de>,
	"mic@digikod.net" <mic@digikod.net>,
	 "michael.christie@oracle.com" <michael.christie@oracle.com>,
	"mingo@redhat.com" <mingo@redhat.com>,
	 "mjguzik@gmail.com" <mjguzik@gmail.com>,
	"mst@redhat.com" <mst@redhat.com>,
	 "npiggin@gmail.com" <npiggin@gmail.com>,
	"peterz@infradead.org" <peterz@infradead.org>,
	 "pmladek@suse.com" <pmladek@suse.com>,
	 "rick.p.edgecombe@intel.com" <rick.p.edgecombe@intel.com>,
	"rostedt@goodmis.org" <rostedt@goodmis.org>,
	 "surenb@google.com" <surenb@google.com>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	 "urezki@gmail.com" <urezki@gmail.com>,
	 "vincent.guittot@linaro.org" <vincent.guittot@linaro.org>,
	"vschneid@redhat.com" <vschneid@redhat.com>
Subject: Re: [RFC 00/14] Dynamic Kernel Stacks
Date: Mon, 18 Mar 2024 11:30:42 -0400	[thread overview]
Message-ID: <CA+CK2bDFUekHnh=XqLcLJS-VgTMrTRWPm4xgmXHWgFCd9JXYmw@mail.gmail.com> (raw)
In-Reply-To: <Zfhblk609XJn5clL@casper.infradead.org>

On Mon, Mar 18, 2024 at 11:19 AM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Mon, Mar 18, 2024 at 11:09:47AM -0400, Pasha Tatashin wrote:
> > The TLB load is going to be exactly the same as today, we already use
> > small pages for VMA mapped stacks. We won't need to have extra
> > flushing either, the mappings are in the kernel space, and once pages
> > are removed from the page table, no one is going to access that VA
> > space until that thread enters the kernel again. We will need to
> > invalidate the VA range only when the pages are mapped, and only on
> > the local cpu.
>
> No; we can pass pointers to our kernel stack to other threads.  The
> obvious one is a mutex; we put a mutex_waiter on our own stack and
> add its list_head to the mutex's waiter list.  I'm sure you can
> think of many other places we do this (eg wait queues, poll(), select(),
> etc).

Hm, it means that stack is sleeping in the kernel space, and has its
stack pages mapped and invalidated on the local CPU, but access from
the remote CPU to that stack pages would be problematic.

I think we still won't need IPI, but VA-range invalidation is actually
needed on unmaps, and should happen during context switch so every
time we go off-cpu. Therefore, what Brian/Andy have suggested makes
more sense instead of kernel/enter/exit paths.

Pasha