Re: [PATCH 2/2] s390/topology: add drawer scheduling domain level

From: Martin Schwidefsky <schwidefsky@de.ibm.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH 2/2] s390/topology: add drawer scheduling domain level
Date: Mon, 13 Jun 2016 16:37:40 +0200	[thread overview]
Message-ID: <20160613163740.0750f535@mschwide> (raw)
In-Reply-To: <20160613135302.GA30909@twins.programming.kicks-ass.net>

On Mon, 13 Jun 2016 15:53:02 +0200
Peter Zijlstra <peterz@infradead.org> wrote:

> On Mon, Jun 13, 2016 at 03:19:42PM +0200, Martin Schwidefsky wrote:
> > On Mon, 13 Jun 2016 15:06:47 +0200
> > Peter Zijlstra <peterz@infradead.org> wrote:
> > 
> > > On Mon, Jun 13, 2016 at 01:22:30PM +0200, Heiko Carstens wrote:
> > > > Yes, and actually we are all virt/LPAR always, so this is unfortunately not
> > > > very easy to do. And yes, I do agree that for the 1:1 case it most likely
> > > > would make sense, however we don't have any run-time guarantee to stay 1:1.
> > > 
> > > One option would be to make it a boot option; such that the
> > > administrator has to set it. At that point, if the admin creates
> > > multiple LPARs its on him.
> > 
> > Unfortunately not good enough. The LPAR code tries to optimize the layout
> > at the time a partition is activated. The landscape of already running
> > partitions can change at this point.
> 
> Would not the admin _know_ this? It would be him activating partitions
> after all, no?

This is all fine and good in a static environment where you can afford to
stop all partitions to do a reconfiguration. There you could get away with
a kernel option that enables "real" NUMA.

But as a general solution this fails. Consider this scenario: you have several
partitions already running with a workload that you do *not* want to interrupt
right now, think stock exchange. And now another partition urgently needs more
memory. To do this you have to shut it down, deactivate it, update the profile
with more memory, re-activate it and restart the OS. End result: memory
landscape could have changed.

> > To get around this you would have to activate *all* partitions first and
> > then start the operating systems in a second step.
> 
> Arguably, you only care about the single partition covering the entire
> machine case, so I don't see that being a problem.
> 
> Again, admin _knows_ this.

The single partitions case is boring, several large partitions to big for a
single node is the hard part.

> > And then there is concurrent repair which will move things around if a
> > piece of memory goes bad. This happens rarely though.
> 
> That would be magic disturbance indeed, nothing much to do about that.

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.