Re: SMP performance degradation with sysbench

[Nick Piggin <nickpiggin@yahoo.com.au>]

Andrea Arcangeli wrote:
> On Tue, Mar 13, 2007 at 09:37:54PM +1100, Nick Piggin wrote:
> 
>>Well it wasn't iowait time. From Anton's analysis, I would probably
>>say it was time waiting for either the glibc malloc mutex or MySQL
>>heap mutex.
> 
> 
> So it again makes little sense to me that this is idle time, unless
> some userland mutex has a usleep in the slow path which would be very
> wrong, in the worst case they should yield() (yield can still waste
> lots of cpu if two tasks in the slow paths calls it while the holder
> is not scheduled, but at least it wouldn't be idle time).

They'll be sleeping in futex_wait in the kernel, I think. One thread
will hold the critical mutex, some will be off doing their own thing,
but importantly there will be many sleeping for the mutex to become
available.

> Idle time is suspicious for a kernel issue in the scheduler or some
> userland inefficiency (the latter sounds more likely).

That is what I first suspected, because the dropoff appeared to happen
exactly after we saturated the CPU count: it seems like a scheduler
artifact.

However, I tested with a bigger system and actually the idle time
comes before we saturate all CPUs. Also, increasing the aggressiveness
of the load balancer did not drop idle time at all, so it is not a case
of some runqueues idle while others have many threads on them.


I guess googlemalloc (tcmalloc?) isn't suitable for a general purpose
glibc allocator. But I wonder if there are other improvements that glibc
can do here?

-- 
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com