Re: [PATCH 1/2] ipc semaphores: reduce ipc_lock contention in semtimedop

From: Nick Piggin <npiggin@suse.de>
To: Chris Mason <chris.mason@oracle.com>,
	Manfred Spraul <manfred@colorfullife.com>,
	zach.brown@oracle.com, jens.axboe@oracle.com,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH 1/2] ipc semaphores: reduce ipc_lock contention in semtimedop
Date: Wed, 14 Apr 2010 05:25:51 +1000	[thread overview]
Message-ID: <20100413192551.GF5683@laptop> (raw)
In-Reply-To: <20100413190110.GR13327@think>

On Tue, Apr 13, 2010 at 03:01:10PM -0400, Chris Mason wrote:
> On Wed, Apr 14, 2010 at 04:57:56AM +1000, Nick Piggin wrote:
> > Yes, because it's not just a theoretical livelock, it can be basically
> > a certainty, given the right pattern of semops.
> > 
> > You could have two mostly-independent groups of processes, each taking
> > and releasing a different sem, which are always contended (eg. if it is
> > being used for a producer-consumer type situation, or even just mutual
> > exclusion with high contention).
> > 
> > Then you could have some overall management process for example which
> > tries to take both sems. It will never get it.
> 
> Ok, fair enough, I'll add the sequence number.
> 
> > 
> > 
> > > > I was looking at doing a sequence number to be able to sort these, but
> > > > it ended up getting over complex (and SAP was only using simple ops so
> > > > it didn't seem to need much better).
> > > > 
> > > > We want to be careful not to change semantics at all. And it gets
> > > > tricky quickly :( What about Zach's simpler wakeup API?
> > > 
> > > Yeah, that's why my patches include code to handle userland sending
> > > duplicate semids.
> > 
> > Duplicate semids? What do you mean?
> 
> Sorry, semnums...index into the array of semaphores.

OK, I wonder just how much it helps, and what.

> > >  Zach's simpler API is cooking too, but if I can get
> > > this done without insane complexity it helps with more than just the
> > > post/wait oracle workload.
> > 
> > Iam worried about complexity and slowing other cases, given that Oracle
> > DB seems willing to adapt to the (better suited) new API. So I'd be
> > interested to know what it helps outside Oracle.
> > 
> 
> Sure, I'd hope that your benchmark from last time around is faster now.

I didn't actually reproduce it here, I think it was a customer or
partner workload. But SAP only seemed to have one contended semnum in
its array, and it was being operated on with "simple" semops (so that's
about as far as the patches went).

I didn't notice anything that should make that go faster?

Yes, with such a workload, using semops is basically legacy and simple
mutexes should work better. So I'm not outright against improving sysv
sem performance for more complex cases where nothing else we have works
as well.