Meaningless load?

Meaningless load?

[Simon Klinkert]

Hi folks,

I have a linux machine (2.6.32-220.7.1.el6.x86_64) with 500 processes 'working' on one nfs4 mountpoint. As I can see with ps and top, only one process is in the 'R' state and really working. The other 499 processes are in the 'D' state and probably waiting for this one process.

But now let's take a look at top one more time:

top - 14:33:57 up  4:27,  4 users,  load average: 501.16, 403.98, 205.52
Tasks: 766 total,   1 running, 765 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  2.1%sy,  0.0%ni, 94.9%id,  0.8%wa,  0.0%hi,  1.4%si,  0.0%st
Mem:  32872392k total,   768488k used, 32103904k free,    30608k buffers
Swap: 35094520k total,        0k used, 35094520k free,    97624k cached

Wow, a load of 501.16 and the CPUs are up to 97% idle.

I'm just wondering if the 'load' is really meaningful in this scenario. The machine is the whole time fully responsive and looks fine to me but maybe I didn't understand correctly what the load should mean. Is there any sensible interpretation of the load?

Thanks,
Simon

Re: Meaningless load?

[Peter Zijlstra]

On Wed, 2012-10-10 at 17:44 +0200, Simon Klinkert wrote:
> I'm just wondering if the 'load' is really meaningful in this
> scenario. The machine is the whole time fully responsive and looks
> fine to me but maybe I didn't understand correctly what the load
> should mean. Is there any sensible interpretation of the load?

I'll leave meaningful aside, but uninterruptible (D state) is part of
how the load thing is defined, so your 500 result is correct.

Re: Meaningless load?

[Simon Klinkert]

On 10.10.2012, at 18:22, Peter Zijlstra  wrote:

> On Wed, 2012-10-10 at 17:44 +0200, Simon Klinkert wrote:
>> I'm just wondering if the 'load' is really meaningful in this
>> scenario. The machine is the whole time fully responsive and looks
>> fine to me but maybe I didn't understand correctly what the load
>> should mean. Is there any sensible interpretation of the load?
> 
> I'll leave meaningful aside, but uninterruptible (D state) is part of
> how the load thing is defined, so your 500 result is correct.

Yes, the calculation of the load is correct but I still don't know how I should interpret the load…

On 11.10.2012, at 06:02, Mike Galbraith wrote:

> Makes perfect sense to me.  Work _is_ stack this high.  We don't and
> can't know whether the mountain is made of popcorn balls or boulders.

That's the point. Afaik the D state never represents 'work'. These processes are waiting for something.

Let's say we have 10,000 processes in the D state (and thus a load of ~10,000) doing nothing. What should the load tell me? The machine is under fire? There is nothing to do? There might be something to do but the machine doesn't know?

Simon

Re: Meaningless load?

[Mike Galbraith]

On Thu, 2012-10-11 at 09:19 +0200, Simon Klinkert wrote:
> On 10.10.2012, at 18:22, Peter Zijlstra wrote:
> 
> > On Wed, 2012-10-10 at 17:44 +0200, Simon Klinkert wrote:
> >> I'm just wondering if the 'load' is really meaningful in this
> >> scenario. The machine is the whole time fully responsive and looks
> >> fine to me but maybe I didn't understand correctly what the load
> >> should mean. Is there any sensible interpretation of the load?
> > 
> > I'll leave meaningful aside, but uninterruptible (D state) is part
> of
> > how the load thing is defined, so your 500 result is correct.
> 
> Yes, the calculation of the load is correct but I still don't know how
> I should interpret the load…
> 
> On 11.10.2012, at 06:02, Mike Galbraith wrote:
> 
> > Makes perfect sense to me. Work _is_ stack this high. We don't and
> > can't know whether the mountain is made of popcorn balls or
> boulders.
> 
> That's the point. Afaik the D state never represents 'work'. These
> processes are waiting for something.

Yeah, the whole pile is waiting, but they're not idle.  There are N
tasks pointed at CPUs.
> 
> Let's say we have 10,000 processes in the D state (and thus a load of
> ~10,000) doing nothing. What should the load tell me? The machine is
> under fire? There is nothing to do? There might be something to do but
> the machine doesn't know?

They are doing something, just not at the particular instant you see
them in D state.  D state pushing load through the roof tells you that
you have a bottleneck.  Whether the bottleneck is a bit of spinning rust
or insufficient NR_CPUS doesn't matter much, both are bottlenecks.

-Mike

Re: Meaningless load?

[Simon Klinkert]

On 11.10.2012, at 10:13, Mike Galbraith wrote:

>> On 11.10.2012, at 06:02, Mike Galbraith wrote:
>> 
>>> Makes perfect sense to me. Work _is_ stack this high. We don't and
>>> can't know whether the mountain is made of popcorn balls or
>> boulders.
>> 
>> That's the point. Afaik the D state never represents 'work'. These
>> processes are waiting for something.
> 
> Yeah, the whole pile is waiting, but they're not idle.  There are N
> tasks pointed at CPUs.
>> 
>> Let's say we have 10,000 processes in the D state (and thus a load of
>> ~10,000) doing nothing. What should the load tell me? The machine is
>> under fire? There is nothing to do? There might be something to do but
>> the machine doesn't know?
> 
> They are doing something, just not at the particular instant you see
> them in D state.  D state pushing load through the roof tells you that
> you have a bottleneck.  Whether the bottleneck is a bit of spinning rust
> or insufficient NR_CPUS doesn't matter much, both are bottlenecks.

Your explanation sounds correct to me but I think in my case, there are only 2-3 process 
waiting for spinning rust (or rather nfs) and the other processes are all in a heavy lock 
contention in the VFS layer. So a load of 10,000 is helpful to indicate that there is a (software)
bottleneck but if I want to see the 'real (work)load' on this machine it isn't really helpful to show 
a load of 10,000 instead of three or whatever. It's a question of interpretation.

Simon