Strange thread behaviour on 8-way x86 machine

Strange thread behaviour on 8-way x86 machine

[Sasha Pachev]

Hi,

I have observed a rather strange behaviour doing a multi-threaded CPU 
benchmark on an 8-way machine running 2.4.2 SMP kernel. Even when the 
priority is reniced to the highest possible value, I am still unable to reach 
more than 50% CPU utilization. My benchmark just creates a bunch of threads 
with pthread_create(), and then runs a simple integer computation in each 
thread. On a dual with 2.4.3 kernel, and a 4-way with 2.4.2 kernel, I am able 
to reach full CPU utilization. 

At first glance, it appears to be like some overzealous fairness problem in 
the scheduler that is exhibited only when you have more than 4 CPUs. Before I 
start scrutinizing the source trying to understand the inner workings of the 
scheduler, I would like to get some feedback from people that know something 
about the subject. Any ideas/suggestions would be appreciated.

-- 
MySQL Development Team
For technical support contracts, visit https://order.mysql.com/
   __  ___     ___ ____  __ 
  /  |/  /_ __/ __/ __ \/ /   Sasha Pachev <sasha@mysql.com>
 / /|_/ / // /\ \/ /_/ / /__  MySQL AB, http://www.mysql.com/
/_/  /_/\_, /___/\___\_\___/  Provo, Utah, USA
       <___/                  

Re: Strange thread behaviour on 8-way x86 machine

[Mike Kravetz]

On Tue, Jul 03, 2001 at 12:25:12PM -0600, Sasha Pachev wrote:
> Hi,
> 
> I have observed a rather strange behaviour doing a multi-threaded CPU 
> benchmark on an 8-way machine running 2.4.2 SMP kernel. Even when the 
> priority is reniced to the highest possible value, I am still unable to reach 
> more than 50% CPU utilization. My benchmark just creates a bunch of threads 
> with pthread_create(), and then runs a simple integer computation in each 
> thread. On a dual with 2.4.3 kernel, and a 4-way with 2.4.2 kernel, I am able 
> to reach full CPU utilization. 

I haven't had any problem fully utilizing 8 CPUs on 2.4.* kernels.  This
may seem obvious, but do you have more than 4 CPUs worth of work for the
system to do?  What is the runqueue length during this benchmark?

-- 
Mike Kravetz                                 mkravetz@sequent.com
IBM Linux Technology Center

Re: Strange thread behaviour on 8-way x86 machine

[Sasha Pachev]

On Tuesday 03 July 2001 12:51, Mike Kravetz wrote:
> On Tue, Jul 03, 2001 at 12:25:12PM -0600, Sasha Pachev wrote:
> > Hi,
> > 
> > I have observed a rather strange behaviour doing a multi-threaded CPU 
> > benchmark on an 8-way machine running 2.4.2 SMP kernel. Even when the 
> > priority is reniced to the highest possible value, I am still unable to 
reach 
> > more than 50% CPU utilization. My benchmark just creates a bunch of 
threads 
> > with pthread_create(), and then runs a simple integer computation in each 
> > thread. On a dual with 2.4.3 kernel, and a 4-way with 2.4.2 kernel, I am 
able 
> > to reach full CPU utilization. 
> 
> I haven't had any problem fully utilizing 8 CPUs on 2.4.* kernels.  This
> may seem obvious, but do you have more than 4 CPUs worth of work for the
> system to do?  What is the runqueue length during this benchmark?

Upon further investigation and testing, it turned out that the kernel was not 
at fault - the problem was high mutex contention, which caused frequent 
context switches, and the idle CPU was apparently from the scheduler waiting 
for the original CPU to become available too often.

On a side note, it would be nice if a process could communicate to the kernel 
that it would rather run on the first available CPU than wait for the perfect 
one to become available.

-- 
MySQL Development Team
For technical support contracts, visit https://order.mysql.com/
   __  ___     ___ ____  __ 
  /  |/  /_ __/ __/ __ \/ /   Sasha Pachev <sasha@mysql.com>
 / /|_/ / // /\ \/ /_/ / /__  MySQL AB, http://www.mysql.com/
/_/  /_/\_, /___/\___\_\___/  Provo, Utah, USA
       <___/                  

Re: Strange thread behaviour on 8-way x86 machine

[Rik van Riel]

On Fri, 6 Jul 2001, Sasha Pachev wrote:

> Upon further investigation and testing, it turned out that the kernel was not
> at fault - the problem was high mutex contention, which caused frequent
> context switches, and the idle CPU was apparently from the scheduler waiting
> for the original CPU to become available too often.
>
> On a side note, it would be nice if a process could communicate
> to the kernel that it would rather run on the first available
> CPU than wait for the perfect one to become available.

The kernel already does this.

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

Re: Strange thread behaviour on 8-way x86 machine

[Sasha Pachev]

On Friday 06 July 2001 13:24, Rik van Riel wrote:
> On Fri, 6 Jul 2001, Sasha Pachev wrote:
> 
> > Upon further investigation and testing, it turned out that the kernel was 
not
> > at fault - the problem was high mutex contention, which caused frequent
> > context switches, and the idle CPU was apparently from the scheduler 
waiting
> > for the original CPU to become available too often.
> >
> > On a side note, it would be nice if a process could communicate
> > to the kernel that it would rather run on the first available
> > CPU than wait for the perfect one to become available.
> 
> The kernel already does this.

Thanks for the info. Would you mind proving a one line pointer on how to tell 
this to the kernel?

-- 
MySQL Development Team
For technical support contracts, visit https://order.mysql.com/
   __  ___     ___ ____  __ 
  /  |/  /_ __/ __/ __ \/ /   Sasha Pachev <sasha@mysql.com>
 / /|_/ / // /\ \/ /_/ / /__  MySQL AB, http://www.mysql.com/
/_/  /_/\_, /___/\___\_\___/  Provo, Utah, USA
       <___/                  

Re: Strange thread behaviour on 8-way x86 machine

[Rik van Riel]

On Fri, 6 Jul 2001, Sasha Pachev wrote:
> On Friday 06 July 2001 13:24, Rik van Riel wrote:
> > On Fri, 6 Jul 2001, Sasha Pachev wrote:
> >
> > > Upon further investigation and testing, it turned out that the kernel was
> not
> > > at fault - the problem was high mutex contention, which caused frequent
> > > context switches, and the idle CPU was apparently from the scheduler
> waiting
> > > for the original CPU to become available too often.
> > >
> > > On a side note, it would be nice if a process could communicate
> > > to the kernel that it would rather run on the first available
> > > CPU than wait for the perfect one to become available.
> >
> > The kernel already does this.
>
> Thanks for the info. Would you mind proving a one line pointer
> on how to tell this to the kernel?

It always does this, by default.  AFAIK you cannot turn it off.

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/