Re: [RFC 00/14] Dynamic Kernel Stacks

From: Kent Overstreet <kent.overstreet@linux.dev>
To: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>,
	linux-kernel@vger.kernel.org,  linux-mm@kvack.org,
	akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de,
	 brauner@kernel.org, bristot@redhat.com, bsegall@google.com,
	 dave.hansen@linux.intel.com, dianders@chromium.org,
	dietmar.eggemann@arm.com,  eric.devolder@oracle.com,
	hca@linux.ibm.com, hch@infradead.org,
	 jacob.jun.pan@linux.intel.com, jgg@ziepe.ca,
	jpoimboe@kernel.org, jroedel@suse.de,  juri.lelli@redhat.com,
	kinseyho@google.com, kirill.shutemov@linux.intel.com,
	 lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de,
	mic@digikod.net,  michael.christie@oracle.com, mingo@redhat.com,
	mjguzik@gmail.com, mst@redhat.com,  npiggin@gmail.com,
	peterz@infradead.org, pmladek@suse.com,
	 rick.p.edgecombe@intel.com, rostedt@goodmis.org,
	surenb@google.com, tglx@linutronix.de,  urezki@gmail.com,
	vincent.guittot@linaro.org, vschneid@redhat.com
Subject: Re: [RFC 00/14] Dynamic Kernel Stacks
Date: Thu, 14 Mar 2024 15:49:22 -0400	[thread overview]
Message-ID: <mvkjn34r3t3qz5qna72gbzp6ampgshemuvgqucjbxlckimh2fc@jqp63wy3v5ag> (raw)
In-Reply-To: <CA+CK2bDQu0jh55H=FQtA-5R-5Qy2=nqzcybEuN=RzVR75NBLRA@mail.gmail.com>

On Thu, Mar 14, 2024 at 03:34:03PM -0400, Pasha Tatashin wrote:
> On Thu, Mar 14, 2024 at 3:29 PM Kent Overstreet
> <kent.overstreet@linux.dev> wrote:
> >
> > On Thu, Mar 14, 2024 at 03:23:08PM -0400, Pasha Tatashin wrote:
> > > > >
> > > > > My point is that what matters is total memory use, not just memory used in
> > > > > the kernel. Amdahl's law.
> > > >
> > > > If userspace is running a few processes with many threads and the
> > > > userspace stacks are small, kernel stacks could end up dominating.
> > > >
> > > > I'd like to see some numbers though.
> > >
> > > The unused kernel stack pages occupy petabytes of memory across the fleet [1].
> >
> > Raw number doesn't mean much here (I know how many machines Google has,
> > of course it's going to be petabytes ;), percentage of system memory
> > would be better.
> >
> > What I'd _really_ like to see is raw output from memory allocation
> > profiling, so we can see how much memory is going to kernel stacks vs.
> > other kernel allocations.
> 
> I've heard there is memory profiling working that can help with that...

I heard you've tried it out, too :)

> While I do not have the data you are asking for, the other kernel
> allocations might be useful, but this particular project is targeted
> to help with reducing overhead where the memory is not used, or used
> in very extreme rare cases.

Well, do you think you could gather it? We shouldn't be blindly applying
performance optimizations; we need to know where to focus our efforts.

e.g. on my laptop I've currently got 356 processes for < 6M of kernel
stack out of 32G total ram, so clearly this isn't much use to me. If the
ratio is similar on your servers - nah, don't want it. I expect the
ratio is not similar and you are burning proportially more memory on
kernel stacks, but we still need to gather the data and do the math :)

> 
> > Number of kernel threads vs. number of user threads would also be good
> > to know - I've been seeing ps output lately where we've got a lot more
> > workqueue workers than we should, perhaps that's something that could be
> > addressed.
> 
> Yes, doing other optimizations make sense, reducing the total number
> kernel threads if possible might help as well. I will look into this
> as well to see how many user threads vs kernel threads we have.

Great, that will help too.