Measuring tools - top and interrupts

Measuring tools - top and interrupts

[Danial Thom]

Running 2.6.17, it seems that top is reporting
100% idle with a network load of about 75K pps
(bridged) , which seems unlikely. Is it possible
that system load accounting is turned off by some
tunning knob?

Is there something that shows the current
interrupts/second in LINUX (such as systat in
'BSD)?

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Measuring tools - top and interrupts

[Randy.Dunlap]

On Thu, 22 Jun 2006 08:26:21 -0700 (PDT) Danial Thom wrote:

> Running 2.6.17, it seems that top is reporting
> 100% idle with a network load of about 75K pps
> (bridged) , which seems unlikely. Is it possible
> that system load accounting is turned off by some
> tunning knob?
> 
> Is there something that shows the current
> interrupts/second in LINUX (such as systat in
> 'BSD)?

You can use/modify http://www.xenotime.net/linux/scripts/sysalive.pl
for interrupts/second (it already does that).

---
~Randy

Re: Measuring tools - top and interrupts

[Bernd Eckenfels]

Randy.Dunlap <rdunlap@xenotime.net> wrote:
>> Is there something that shows the current
>> interrupts/second in LINUX (such as systat in
>> 'BSD)?
> 
> You can use/modify http://www.xenotime.net/linux/scripts/sysalive.pl
> for interrupts/second (it already does that).

vmstat shoes a "in" column, and mpstat shows multiple interrupt related
numbers.

Gruss
Bernd

Re: Measuring tools - top and interrupts

[Erik Mouw]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 658 bytes --]

On Thu, Jun 22, 2006 at 08:26:21AM -0700, Danial Thom wrote:
> Running 2.6.17, it seems that top is reporting
> 100% idle with a network load of about 75K pps
> (bridged) , which seems unlikely. Is it possible
> that system load accounting is turned off by some
> tunning knob?

75K packets/s isn't too hard for modern NICs, especially when using
NAPI.

> Is there something that shows the current
> interrupts/second in LINUX (such as systat in
> 'BSD)?

"vmstat 1£ has the number of interrupts in the "in" column.


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: Measuring tools - top and interrupts

[Danial Thom]

--- Erik Mouw <erik@harddisk-recovery.com> wrote:

> On Thu, Jun 22, 2006 at 08:26:21AM -0700,
> Danial Thom wrote:
> > Running 2.6.17, it seems that top is
> reporting
> > 100% idle with a network load of about 75K
> pps
> > (bridged) , which seems unlikely. Is it
> possible
> > that system load accounting is turned off by
> some
> > tunning knob?
> 
> 75K packets/s isn't too hard for modern NICs,
> especially when using
> NAPI.

Well thats just a ridiculous answer, so why
bother? 

You polling guys just crack me up. There isn't
much less work to be done with polling. The only
reason you THINK its less work is because the
measuring tools don't work properly. You still
have to process the same number of packets when
you poll, and you have polls instead of
interrupts. Since you can control the # of
interrupts with most cards, there is zero
advantage to polling, and more negatives.

And 75K pps may not be "much", but its still at
least 10% of what the system can handle, so it
should measure around a 10% load. 2.4 measures
about 12% load. So the only conclusion is that
load accounting is broken in 2.6.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Measuring tools - top and interrupts

[Erik Mouw]

On Thu, Jun 22, 2006 at 09:58:08AM -0700, Danial Thom wrote:
> --- Erik Mouw <erik@harddisk-recovery.com> wrote:
> > 75K packets/s isn't too hard for modern NICs,
> > especially when using
> > NAPI.
> 
> Well thats just a ridiculous answer, so why
> bother? 
> 
> You polling guys just crack me up. There isn't
> much less work to be done with polling. The only
> reason you THINK its less work is because the
> measuring tools don't work properly. You still
> have to process the same number of packets when
> you poll, and you have polls instead of
> interrupts. Since you can control the # of
> interrupts with most cards, there is zero
> advantage to polling, and more negatives.

There certainly is less work to be done with polling. Less IRQs means
less expensive context switches, which means a lower system load. See
Documentation/NAPI_HOWTO.txt for information and a link to the Linux
NAPI paper.

> And 75K pps may not be "much", but its still at
> least 10% of what the system can handle, so it
> should measure around a 10% load. 2.4 measures
> about 12% load. So the only conclusion is that
> load accounting is broken in 2.6.

Network traffic is usually IO bound, not CPU bound. The load figures
top shows tell something about the amount of work the CPU has to do,
not about how busy your PCI bus (or whatever bus the NIC lives on) is.

IIRC the networking layer in 2.6 differs quite a lot from 2.4, so the
load average figures can be quite misleading.


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: Measuring tools - top and interrupts

[Danial Thom]

--- Erik Mouw <erik@harddisk-recovery.com> wrote:

> On Thu, Jun 22, 2006 at 09:58:08AM -0700,
> Danial Thom wrote:
> > --- Erik Mouw <erik@harddisk-recovery.com>
> wrote:
> > > 75K packets/s isn't too hard for modern
> NICs,
> > > especially when using
> > > NAPI.
> > 
> > Well thats just a ridiculous answer, so why
> > bother? 
> > 
> > You polling guys just crack me up. There
> isn't
> > much less work to be done with polling. The
> only
> > reason you THINK its less work is because the
> > measuring tools don't work properly. You
> still
> > have to process the same number of packets
> when
> > you poll, and you have polls instead of
> > interrupts. Since you can control the # of
> > interrupts with most cards, there is zero
> > advantage to polling, and more negatives.
> 
> There certainly is less work to be done with
> polling. Less IRQs means
> less expensive context switches, which means a
> lower system load. See
> Documentation/NAPI_HOWTO.txt for information
> and a link to the Linux
> NAPI paper.
> 
> > And 75K pps may not be "much", but its still
> at
> > least 10% of what the system can handle, so
> it
> > should measure around a 10% load. 2.4
> measures
> > about 12% load. So the only conclusion is
> that
> > load accounting is broken in 2.6.
> 
> Network traffic is usually IO bound, not CPU
> bound. The load figures
> top shows tell something about the amount of
> work the CPU has to do,
> not about how busy your PCI bus (or whatever
> bus the NIC lives on) is.
> 
> IIRC the networking layer in 2.6 differs quite
> a lot from 2.4, so the
> load average figures can be quite misleading.

I'm sorry, but you're being an idiot if you think
that 16K interrupts per second and forwarding 75K
pps generate no cpu load. Its just that simple.
It also means that you've never profiled a kernel
because you don't understand where the loads are
generated. You've probably been on too many lists
with too many people who have no idea what
they're talking about.

The NAPI paper is complete rubbish because its
based on a principle which no longer exists: that
is the assumption that you'll get an interrupt
for every packet. Not only is that hardly ever
the case, modern controllers (such as Intel's
GigE controllers) can precisely mitigate the
interrupts in hardware. You *could* tune this on
the fly, but there is really no reason to,
because at low interrupt loads cpu overhead is
not a concern and lower latencies are preferred.
You'll never get an interrupt for consecutive
packets, and since packets usually arrive in
bursts, the mitigation model works even better
than if they arrived randomly.

With simple tuning the controller is more
efficient with hardware interrupts. None of the
arguments in the NAPI paper hold true when using
a modern GigE controller. 

Of course you folks could never really know that,
because the tools to measure things don't work,
and you all seem to be using systems that are so
kludged up that you don't even know what you're
measuring.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Measuring tools - top and interrupts

[Mike Galbraith]

On Thu, 2006-06-22 at 16:37 -0700, Danial Thom wrote:
> I'm sorry, but you're being an idiot if you think
> that 16K interrupts per second and forwarding 75K
> pps generate no cpu load. Its just that simple.
> It also means that you've never profiled a kernel
> because you don't understand where the loads are
> generated. You've probably been on too many lists
> with too many people who have no idea what
> they're talking about.

(what horrid manners)

Hm.  You may be right about the load average calculation being broken.

Below is a 100 second profile sample of my 3GHz P4 handling 15K
interrupts per second while receiving a flood ping.  My interpretation
is that tools should be showing ~10% cpu load rather than zero.  Am I'm
misinterpreting it?

 97574 total                                      0.0258
 89549 default_idle                             1017.6023
  1734 ioread16                                  36.8936
  1138 ioread8                                   24.7391
   974 rhine_start_tx                             1.3994
   534 __do_softirq                               3.8417
   331 handle_IRQ_event                           3.2772
   223 rhine_interrupt                            0.0739
   222 memset                                     7.9286
   194 nf_iterate                                 1.5520
   140 local_bh_enable                            1.0769
    99 __kmalloc                                  1.0532
    92 net_rx_action                              0.2000
    85 kfree                                      0.9884
    82 skb_release_data                           0.6406
    77 csum_partial_copy_generic                  0.3105
    73 ip_push_pending_frames                     0.0681
    71 __alloc_skb                                0.2898
    69 kmem_cache_free                            1.3529
    66 kmem_cache_alloc                           1.3750
    62 csum_partial                               0.2153
    61 rt_hash_code                               0.4959
    61 ip_append_data                             0.0253
    60 netif_receive_skb                          0.0516
    58 ip_rcv                                     0.0471
    58 ip_local_deliver                           0.0854
    58 eth_type_trans                             0.2489
    55 ip_output                                  0.0957
    52 icmp_reply                                 0.1187

Re: Measuring tools - top and interrupts

[Danial Thom]

--- Mike Galbraith <efault@gmx.de> wrote:

> On Thu, 2006-06-22 at 16:37 -0700, Danial Thom
> wrote:
> > I'm sorry, but you're being an idiot if you
> think
> > that 16K interrupts per second and forwarding
> 75K
> > pps generate no cpu load. Its just that
> simple.
> > It also means that you've never profiled a
> kernel
> > because you don't understand where the loads
> are
> > generated. You've probably been on too many
> lists
> > with too many people who have no idea what
> > they're talking about.
> 
> (what horrid manners)
> 
> Hm.  You may be right about the load average
> calculation being broken.
> 
> Below is a 100 second profile sample of my 3GHz
> P4 handling 15K
> interrupts per second while receiving a flood
> ping.  My interpretation
> is that tools should be showing ~10% cpu load
> rather than zero.  Am I'm
> misinterpreting it?
> 
>  97574 total                                   
>   0.0258
>  89549 default_idle                            
> 1017.6023
>   1734 ioread16                                
>  36.8936
>   1138 ioread8                                 
>  24.7391
>    974 rhine_start_tx                          
>   1.3994
>    534 __do_softirq                            
>   3.8417
>    331 handle_IRQ_event                        
>   3.2772
>    223 rhine_interrupt                         
>   0.0739
>    222 memset                                  
>   7.9286
>    194 nf_iterate                              
>   1.5520
>    140 local_bh_enable                         
>   1.0769
>     99 __kmalloc                               
>   1.0532
>     92 net_rx_action                           
>   0.2000
>     85 kfree                                   
>   0.9884
>     82 skb_release_data                        
>   0.6406
>     77 csum_partial_copy_generic               
>   0.3105
>     73 ip_push_pending_frames                  
>   0.0681
>     71 __alloc_skb                             
>   0.2898
>     69 kmem_cache_free                         
>   1.3529
>     66 kmem_cache_alloc                        
>   1.3750
>     62 csum_partial                            
>   0.2153
>     61 rt_hash_code                            
>   0.4959
>     61 ip_append_data                          
>   0.0253
>     60 netif_receive_skb                       
>   0.0516
>     58 ip_rcv                                  
>   0.0471
>     58 ip_local_deliver                        
>   0.0854
>     58 eth_type_trans                          
>   0.2489
>     55 ip_output                               
>   0.0957
>     52 icmp_reply                              
>   0.1187
> 
Thats a pretty crappy controller you have in with
that shiny P4...

I'm not sure that they want the tools to work.
They'll just call you a troll and go on
developing unnecessary things like NAPI because
they're still using controllers designed by DEC
(remember them?) back in the stone ages. 

Yet I regularly encounter people using cheap NICs
with expensive cpus on network-intensive
applications. But you'd think one or two people
would have a clue.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Measuring tools - top and interrupts

[Mike Galbraith]

On Fri, 2006-06-23 at 13:14 -0700, Danial Thom wrote:
>  
> Thats a pretty crappy controller you have in with
> that shiny P4...

Shrug.  It's a grocery store desktop box, and I'm quite happy with it,
but that has roughly zip to do with the price of tea in China.

> I'm not sure that they want the tools to work.
> They'll just call you a troll and go on
> developing unnecessary things like NAPI because
> they're still using controllers designed by DEC
> (remember them?) back in the stone ages. 
> 
> Yet I regularly encounter people using cheap NICs
> with expensive cpus on network-intensive
> applications. But you'd think one or two people
> would have a clue.

As does that.

	-Mike

Re: Measuring tools - top and interrupts

[Erik Mouw]

On Thu, Jun 22, 2006 at 04:37:44PM -0700, Danial Thom wrote:
> > On Thu, Jun 22, 2006 at 09:58:08AM -0700,
> > Danial Thom wrote:
> > > Well thats just a ridiculous answer, so why
> > > bother? 

[...]

> I'm sorry, but you're being an idiot if you think

So much for good manners. I accept such replies only from a few people
on this list, and you are not one of them.

*plonk*


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: Measuring tools - top and interrupts

[Alistair John Strachan]

On Friday 23 June 2006 00:37, Danial Thom wrote:
> Of course you folks could never really know that,
> because the tools to measure things don't work,
> and you all seem to be using systems that are so
> kludged up that you don't even know what you're
> measuring.

Next time attach a patch fixing, or at least bother to investigate the problem 
which you're so rudely flaming about. Maybe Linux is broken. Nobody said it 
wasn't. But you're closer to (and clearly care more about) the problem than 
99% of the people reading your email.

So why don't you help fix it, instead of laying into Erik and the netdev 
people (who incidentally aren't CC'ed)?

-- 
Cheers,
Alistair.

Third year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Measuring tools - top and interrupts

[Danial Thom]

--- Alistair John Strachan
<s0348365@sms.ed.ac.uk> wrote:

> On Friday 23 June 2006 00:37, Danial Thom
> wrote:
> > Of course you folks could never really know
> that,
> > because the tools to measure things don't
> work,
> > and you all seem to be using systems that are
> so
> > kludged up that you don't even know what
> you're
> > measuring.
> 
> Next time attach a patch fixing, or at least
> bother to investigate the problem 
> which you're so rudely flaming about. Maybe
> Linux is broken. Nobody said it 
> wasn't. But you're closer to (and clearly care
> more about) the problem than 
> 99% of the people reading your email.
> 
> So why don't you help fix it, instead of laying
> into Erik and the netdev 
> people (who incidentally aren't CC'ed)?

When you report a problem and some clown pipes in
with "its not a problem, linux is just so fast it
doesn't use any cpu!" you really don't have the
foundation for coming up with a solution or
helping to find one.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Measuring tools - top and interrupts

[sena seneviratne]

Hi Erik,

We have fixed this problem in the kernel 2.4.19 which has been installed in 
few selected computers of the "Middleware grid"
Sydney University.

Also load average has been calculated for each login user.

Thanks
Sena Seneviratne
Computer Engineering Lab
Sydney University




At 07:31 PM 6/22/2006 +0200, you wrote:
>On Thu, Jun 22, 2006 at 09:58:08AM -0700, Danial Thom wrote:
> > --- Erik Mouw <erik@harddisk-recovery.com> wrote:
> > > 75K packets/s isn't too hard for modern NICs,
> > > especially when using
> > > NAPI.
> >
> > Well thats just a ridiculous answer, so why
> > bother?
> >
> > You polling guys just crack me up. There isn't
> > much less work to be done with polling. The only
> > reason you THINK its less work is because the
> > measuring tools don't work properly. You still
> > have to process the same number of packets when
> > you poll, and you have polls instead of
> > interrupts. Since you can control the # of
> > interrupts with most cards, there is zero
> > advantage to polling, and more negatives.
>
>There certainly is less work to be done with polling. Less IRQs means
>less expensive context switches, which means a lower system load. See
>Documentation/NAPI_HOWTO.txt for information and a link to the Linux
>NAPI paper.
>
> > And 75K pps may not be "much", but its still at
> > least 10% of what the system can handle, so it
> > should measure around a 10% load. 2.4 measures
> > about 12% load. So the only conclusion is that
> > load accounting is broken in 2.6.
>
>Network traffic is usually IO bound, not CPU bound. The load figures
>top shows tell something about the amount of work the CPU has to do,
>not about how busy your PCI bus (or whatever bus the NIC lives on) is.
>
>IIRC the networking layer in 2.6 differs quite a lot from 2.4, so the
>load average figures can be quite misleading.
>
>
>Erik
>
>--
>+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
>| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Re: Measuring tools - top and interrupts

[Danial Thom]

> Network traffic is usually IO bound, not CPU
> bound. The load figures
> top shows tell something about the amount of
> work the CPU has to do,
> not about how busy your PCI bus (or whatever
> bus the NIC lives on) is.
> 
> IIRC the networking layer in 2.6 differs quite
> a lot from 2.4, so the
> load average figures can be quite misleading.
> 

For the record, *most* of the work are I/O calls
(ie reading and writing registers), which are not
in the "background". I/O calls become more and
more expensive as the bus becomes saturated as it
takes longer to get the bus to do the operation. 

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Measuring tools - top and interrupts

[Francois Romieu]

Danial Thom <danial_thom@yahoo.com> :
[bull removed]

You have been told twice in this thread to use vmstat.

Read again and shut up until you can provide a sensible bug report,
thank you.

-- 
Ueimor

Re: Measuring tools - top and interrupts

[Danial Thom]

--- Francois Romieu <romieu@fr.zoreil.com> wrote:

> Danial Thom <danial_thom@yahoo.com> :
> [bull removed]
> 
> You have been told twice in this thread to use
> vmstat.
> 
> Read again and shut up until you can provide a
> sensible bug report,
> thank you.

vmstat gives the same #s as top, and it doesn't
break down the interrupts, so its not the answer
to the question "is there something similar to
systat". Plus its clear that the guy who gave the
answer doesn't know what he's talking about,
since he's actually trying to explain away the
problem as being normal. 

Here's a sensible bug report. Bridging 75K pps.
vmstat output:

 0  0      0 425424   9984  40496    0    0     0
    0 16238    12  0  0 100  0
 0  0      0 425424   9984  40496    0    0     0
    0 16227    16  0  0 100  0
 0  0      0 425424   9984  40496    0    0     0
    0 16244     9  0  0 100  0

Over 16,000 interrupts per second and 100% idle. 

netstat:

eth2   1500   0       5      0      0      468886
     0      0      0 BMRU
eth3   1500   473201      0   5680      0       5
     0      0      0 BMRU

the system is perpetually 100% idle but its
dropping packets due to excessive backlog.

DT


> 
> -- 
> Ueimor
> -
> To unsubscribe from this list: send the line
> "unsubscribe linux-kernel" in
> the body of a message to
> majordomo@vger.kernel.org
> More majordomo info at 
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at 
> http://www.tux.org/lkml/
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Measuring tools - top and interrupts

[Francois Romieu]

Danial Thom <danial_thom@yahoo.com> :
[...]
> systat". Plus its clear that the guy who gave the
> answer doesn't know what he's talking about,
> since he's actually trying to explain away the
> problem as being normal. 

"75 kpps means 10% of the max load" was quite fun too.

[...]
> the system is perpetually 100% idle but its
> dropping packets due to excessive backlog.

No difference when you renice ksoftirqd to a strongly
negative value ?

-- 
Ueimor

Re: Measuring tools - top and interrupts

[Danial Thom]

--- Francois Romieu <romieu@fr.zoreil.com> wrote:

> Danial Thom <danial_thom@yahoo.com> :
> [...]
> > systat". Plus its clear that the guy who gave
> the
> > answer doesn't know what he's talking about,
> > since he's actually trying to explain away
> the
> > problem as being normal. 
> 
> "75 kpps means 10% of the max load" was quite
> fun too.
> 

What's *fun* about it? 2.4.24 shows a 12-13%
load. 

> [...]
> > the system is perpetually 100% idle but its
> > dropping packets due to excessive backlog.
> 
> No difference when you renice ksoftirqd to a
> strongly
> negative value ?

No, not really. Running the compiler causes
drops. re-nicing eliminates drops with no
userland activity.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Measuring tools - top and interrupts

[Francois Romieu]

Danial Thom <danial_thom@yahoo.com> :
[...]
> What's *fun* about it? 2.4.24 shows a 12-13%
> load. 

The assumption of linearity.

[...]
> > No difference when you renice ksoftirqd to a
> > strongly
> > negative value ?
> 
> No, not really. Running the compiler causes
> drops. re-nicing eliminates drops with no
> userland activity.

It may be interesting to know if the same drops
happen when you generate a disk-only load, ksoftirqd
still at the minimal value.

-- 
Ueimor

Re: Measuring tools - top and interrupts

[Björn Steinbrink]

On 2006.06.22 09:58:08 -0700, Danial Thom wrote:
> And 75K pps may not be "much", but its still at
> least 10% of what the system can handle, so it
> should measure around a 10% load. 2.4 measures
> about 12% load. So the only conclusion is that
> load accounting is broken in 2.6.

Are you by chance using procps < 3.1.12? The kernel reports absolute
values for cpu usage, the conversion to percentage is done by top/vmstat
itself. And those old versions don't know about the new fields that 2.6
kernels have in /proc/stat, thus they simply ignore the si and hi
values, producing quite misleading results...

Björn

PS: procps 3.1.12 was released in 2003, so if DEC was stone age and my
assumption about your tools holds, then your tools are like... medieval :)

Re: Measuring tools - top and interrupts

[sena seneviratne]

Björn,

Yes you have to change the procps to read the new /proc/stat

For example, after separating the load and disk IO measurements, also 
separating for per user at the kernel level, I had to introduce new code 
into procps/ to reflect those changes

Otherwise those tools (top, uptime etc) would not know the new formats etc.

Thanks
Sena
Sydney University


At 04:07 AM 6/24/2006 +0200, you wrote:
>On 2006.06.22 09:58:08 -0700, Danial Thom wrote:
> > And 75K pps may not be "much", but its still at
> > least 10% of what the system can handle, so it
> > should measure around a 10% load. 2.4 measures
> > about 12% load. So the only conclusion is that
> > load accounting is broken in 2.6.
>
>Are you by chance using procps < 3.1.12? The kernel reports absolute
>values for cpu usage, the conversion to percentage is done by top/vmstat
>itself. And those old versions don't know about the new fields that 2.6
>kernels have in /proc/stat, thus they simply ignore the si and hi
>values, producing quite misleading results...
>
>Björn
>
>PS: procps 3.1.12 was released in 2003, so if DEC was stone age and my
>assumption about your tools holds, then your tools are like... medieval :)
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Re: Measuring tools - top and interrupts

[Mike Galbraith]

On Thu, 2006-06-22 at 09:58 -0700, Danial Thom wrote:

> And 75K pps may not be "much", but its still at
> least 10% of what the system can handle, so it
> should measure around a 10% load. 2.4 measures
> about 12% load. So the only conclusion is that
> load accounting is broken in 2.6.

For UP, yes.  SMP kernel accounts irq processing time properly.

	-Mike

Re: Measuring tools - top and interrupts

[Mike Galbraith]

On Sat, 2006-06-24 at 07:59 +0200, Mike Galbraith wrote:
> On Thu, 2006-06-22 at 09:58 -0700, Danial Thom wrote:
> 
> > And 75K pps may not be "much", but its still at
> > least 10% of what the system can handle, so it
> > should measure around a 10% load. 2.4 measures
> > about 12% load. So the only conclusion is that
> > load accounting is broken in 2.6.
> 
> For UP, yes.  SMP kernel accounts irq processing time properly.

For my little box, the below cures it.

--- linux-2.6.17x/arch/i386/kernel/apic.c.org	2006-06-24 08:08:46.000000000 +0200
+++ linux-2.6.17x/arch/i386/kernel/apic.c	2006-06-24 08:09:16.000000000 +0200
@@ -1175,9 +1175,7 @@ EXPORT_SYMBOL(switch_ipi_to_APIC_timer);
 inline void smp_local_timer_interrupt(struct pt_regs * regs)
 {
 	profile_tick(CPU_PROFILING, regs);
-#ifdef CONFIG_SMP
 	update_process_times(user_mode_vm(regs));
-#endif
 
 	/*
 	 * We take the 'long' return path, and there every subsystem

Re: Measuring tools - top and interrupts

[Björn Steinbrink]

On 2006.06.24 08:26:23 +0200, Mike Galbraith wrote:
> On Sat, 2006-06-24 at 07:59 +0200, Mike Galbraith wrote:
> > On Thu, 2006-06-22 at 09:58 -0700, Danial Thom wrote:
> > 
> > > And 75K pps may not be "much", but its still at
> > > least 10% of what the system can handle, so it
> > > should measure around a 10% load. 2.4 measures
> > > about 12% load. So the only conclusion is that
> > > load accounting is broken in 2.6.
> > 
> > For UP, yes.  SMP kernel accounts irq processing time properly.

Do you actually see 100% idle? On both, UP and SMP, I see non-zero hi/si
values using "top". With IO-APIC enabled, I see only non-zero si values
for my tg3 NICs, and non-zero hi and si values for the nVidia NIC. With
IO-APIC disabled, I also see a non-zero hi value for the tg3 on UP,
guess that's normal... But I never see 100% idle while flooding the box
with pings.

> For my little box, the below cures it.
> 
> --- linux-2.6.17x/arch/i386/kernel/apic.c.org	2006-06-24 08:08:46.000000000 +0200
> +++ linux-2.6.17x/arch/i386/kernel/apic.c	2006-06-24 08:09:16.000000000 +0200
> @@ -1175,9 +1175,7 @@ EXPORT_SYMBOL(switch_ipi_to_APIC_timer);
>  inline void smp_local_timer_interrupt(struct pt_regs * regs)
>  {
>  	profile_tick(CPU_PROFILING, regs);
> -#ifdef CONFIG_SMP
>  	update_process_times(user_mode_vm(regs));
> -#endif
>  
>  	/*
>  	 * We take the 'long' return path, and there every subsystem

The non-SMP call to update_process_times() is in do_timer_interrupt_hook(),
so I guess the above is not the Right Thing to do. I don't even see how
you could reach smp_local_timer_interrupt() without going through
do_timer_interrupt_hook() first.

Björn

Re: Measuring tools - top and interrupts

[Mike Galbraith]

On Sat, 2006-06-24 at 11:21 +0200, Björn Steinbrink wrote:
> On 2006.06.24 08:26:23 +0200, Mike Galbraith wrote:
> > On Sat, 2006-06-24 at 07:59 +0200, Mike Galbraith wrote:
> > > On Thu, 2006-06-22 at 09:58 -0700, Danial Thom wrote:
> > > 
> > > > And 75K pps may not be "much", but its still at
> > > > least 10% of what the system can handle, so it
> > > > should measure around a 10% load. 2.4 measures
> > > > about 12% load. So the only conclusion is that
> > > > load accounting is broken in 2.6.
> > > 
> > > For UP, yes.  SMP kernel accounts irq processing time properly.
> 
> Do you actually see 100% idle?

Yes.

>  On both, UP and SMP, I see non-zero hi/si
> values using "top". With IO-APIC enabled, I see only non-zero si values
> for my tg3 NICs, and non-zero hi and si values for the nVidia NIC. With
> IO-APIC disabled, I also see a non-zero hi value for the tg3 on UP,
> guess that's normal... But I never see 100% idle while flooding the box
> with pings.

I have APIC and IO-APIC enabled.

> > For my little box, the below cures it.
> > 
> > --- linux-2.6.17x/arch/i386/kernel/apic.c.org	2006-06-24 08:08:46.000000000 +0200
> > +++ linux-2.6.17x/arch/i386/kernel/apic.c	2006-06-24 08:09:16.000000000 +0200
> > @@ -1175,9 +1175,7 @@ EXPORT_SYMBOL(switch_ipi_to_APIC_timer);
> >  inline void smp_local_timer_interrupt(struct pt_regs * regs)
> >  {
> >  	profile_tick(CPU_PROFILING, regs);
> > -#ifdef CONFIG_SMP
> >  	update_process_times(user_mode_vm(regs));
> > -#endif
> >  
> >  	/*
> >  	 * We take the 'long' return path, and there every subsystem
> 
> The non-SMP call to update_process_times() is in do_timer_interrupt_hook(),
> so I guess the above is not the Right Thing to do.

Ah, there it is.  That's what I was looking for.  I figured that doing
what I did had to be wrong, but tried it for grins anyway... was pretty
surprised when it worked (kinda).

	-Mike

Re: Measuring tools - top and interrupts

[Mike Galbraith]

On Sat, 2006-06-24 at 11:52 +0200, Mike Galbraith wrote:
> On Sat, 2006-06-24 at 11:21 +0200, Björn Steinbrink wrote:
> > 
> > The non-SMP call to update_process_times() is in do_timer_interrupt_hook(),
> > so I guess the above is not the Right Thing to do.
> 
> Ah, there it is.  That's what I was looking for.  I figured that doing
> what I did had to be wrong, but tried it for grins anyway... was pretty
> surprised when it worked (kinda).

Calling update_process_times() in do_timer_interrupt_hook() flat does
not work here.  Calling it in smp_local_timer_interrupt() works fine.  

Oh joy.

	-Mike

Re: Measuring tools - top and interrupts

[Björn Steinbrink]

On 2006.06.24 13:41:57 +0200, Mike Galbraith wrote:
> On Sat, 2006-06-24 at 11:52 +0200, Mike Galbraith wrote:
> > On Sat, 2006-06-24 at 11:21 +0200, Björn Steinbrink wrote:
> > > 
> > > The non-SMP call to update_process_times() is in do_timer_interrupt_hook(),
> > > so I guess the above is not the Right Thing to do.
> > 
> > Ah, there it is.  That's what I was looking for.  I figured that doing
> > what I did had to be wrong, but tried it for grins anyway... was pretty
> > surprised when it worked (kinda).
> 
> Calling update_process_times() in do_timer_interrupt_hook() flat does
> not work here.  Calling it in smp_local_timer_interrupt() works fine.  
> 
> Oh joy.

I can reproduce it now, seems to require CONFIG_4KSTACKS to fail. Can
you confirm that?

Björn

Re: Measuring tools - top and interrupts

[Mike Galbraith]

On Sat, 2006-06-24 at 17:40 +0200, Björn Steinbrink wrote:
> On 2006.06.24 13:41:57 +0200, Mike Galbraith wrote:
> > On Sat, 2006-06-24 at 11:52 +0200, Mike Galbraith wrote:
> > > On Sat, 2006-06-24 at 11:21 +0200, Björn Steinbrink wrote:
> > > > 
> > > > The non-SMP call to update_process_times() is in do_timer_interrupt_hook(),
> > > > so I guess the above is not the Right Thing to do.
> > > 
> > > Ah, there it is.  That's what I was looking for.  I figured that doing
> > > what I did had to be wrong, but tried it for grins anyway... was pretty
> > > surprised when it worked (kinda).
> > 
> > Calling update_process_times() in do_timer_interrupt_hook() flat does
> > not work here.  Calling it in smp_local_timer_interrupt() works fine.  
> > 
> > Oh joy.
> 
> I can reproduce it now, seems to require CONFIG_4KSTACKS to fail. Can
> you confirm that?

What a coincidence.  After trying a different compiler, and slogging
through a bunch of assembler trying to figure out how the heck this can
happen, I was just booting an 8k stack kernel (as a wild-ass guess;).

let's see.  Yeah, confirmed.

	-Mike

Re: Measuring tools - top and interrupts

[Björn Steinbrink]

On 2006.06.24 18:23:12 +0200, Mike Galbraith wrote:
> On Sat, 2006-06-24 at 17:40 +0200, Björn Steinbrink wrote:
> > On 2006.06.24 13:41:57 +0200, Mike Galbraith wrote:
> > > On Sat, 2006-06-24 at 11:52 +0200, Mike Galbraith wrote:
> > > > On Sat, 2006-06-24 at 11:21 +0200, Björn Steinbrink wrote:
> > > > > 
> > > > > The non-SMP call to update_process_times() is in do_timer_interrupt_hook(),
> > > > > so I guess the above is not the Right Thing to do.
> > > > 
> > > > Ah, there it is.  That's what I was looking for.  I figured that doing
> > > > what I did had to be wrong, but tried it for grins anyway... was pretty
> > > > surprised when it worked (kinda).
> > > 
> > > Calling update_process_times() in do_timer_interrupt_hook() flat does
> > > not work here.  Calling it in smp_local_timer_interrupt() works fine.  
> > > 
> > > Oh joy.
> > 
> > I can reproduce it now, seems to require CONFIG_4KSTACKS to fail. Can
> > you confirm that?
> 
> What a coincidence.  After trying a different compiler, and slogging
> through a bunch of assembler trying to figure out how the heck this can
> happen, I was just booting an 8k stack kernel (as a wild-ass guess;).
> 
> let's see.  Yeah, confirmed.

OK, it also depends on IO APIC being enabled and active, ie. noapic on
the kernel command line will fix it as well as disabling
CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it
happens though.
The only difference with IO APIC disabled seems to be that the irq
doesn't get ACKed before update_process_times() gets called.
And your "fix" makes it being called outside of the xtime_lock spinlock
and with a slightly different stack usage AFAICT.
But none of these should make a difference, right?

Björn

Re: Measuring tools - top and interrupts

[Mike Galbraith]

On Sat, 2006-06-24 at 21:25 +0200, Björn Steinbrink wrote:
> On 2006.06.24 18:23:12 +0200, Mike Galbraith wrote:
> > 
> > let's see.  Yeah, confirmed.
> 
> OK, it also depends on IO APIC being enabled and active, ie. noapic on
> the kernel command line will fix it as well as disabling
> CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it
> happens though.

Ditto.

> The only difference with IO APIC disabled seems to be that the irq
> doesn't get ACKed before update_process_times() gets called.
> And your "fix" makes it being called outside of the xtime_lock spinlock
> and with a slightly different stack usage AFAICT.

(it's still under the xtime lock)

> But none of these should make a difference, right?

Not that I can see, but then it's pretty dark down here.  Anybody got a
flashlight I can borrow? ;-)

	-Mike

Re: Measuring tools - top and interrupts

[Björn Steinbrink]

On 2006.06.25 07:06:33 +0200, Mike Galbraith wrote:
> On Sat, 2006-06-24 at 21:25 +0200, Björn Steinbrink wrote:
> > On 2006.06.24 18:23:12 +0200, Mike Galbraith wrote:
> > > 
> > > let's see.  Yeah, confirmed.
> > 
> > OK, it also depends on IO APIC being enabled and active, ie. noapic on
> > the kernel command line will fix it as well as disabling
> > CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it
> > happens though.
> 
> Ditto.
> 
> > The only difference with IO APIC disabled seems to be that the irq
> > doesn't get ACKed before update_process_times() gets called.
> > And your "fix" makes it being called outside of the xtime_lock spinlock
> > and with a slightly different stack usage AFAICT.
> 
> (it's still under the xtime lock)

No, with IO-APIC enabled, it's using the local APIC timer, thus
using_apic_timer is 1 and the path right after unlocking in
timer_interrupt() is taken towards update_process_times().

> > But none of these should make a difference, right?
> 
> Not that I can see, but then it's pretty dark down here.  Anybody got a
> flashlight I can borrow? ;-)

Guess I found one, not sure if it works correctly though ;)

Using 4K stacks, we have one stack for hard irqs and one for soft irqs,
both having their own threadinfo and thus preemptcount. Thus the call to
softirq_count() in account_system_time() will always return 0 when
called in hard irq context. Additionally preemptcount is always set to
HARDIRQ_OFFSET in hard irq context, so
hardirq_count() - hardirq_offset is 0 all the time as well.

But that doesn't fit the fact that I at least get hard irq accounting
when booting with noapic. And it also doesn't explain why your fix
works, fixing both, soft and hard irq accounting. Am I missing some
code path that calls smp_local_timer_interrupt? There's
smp_apic_timer_interrupt(), but that seems to be unused on i386.

Björn

[PATCH] i386: Fix softirq accounting with 4K stacks

[Björn Steinbrink]

On 2006.06.25 13:12:38 +0200, Björn Steinbrink wrote:
> On 2006.06.25 07:06:33 +0200, Mike Galbraith wrote:
> > On Sat, 2006-06-24 at 21:25 +0200, Björn Steinbrink wrote:
> > > OK, it also depends on IO APIC being enabled and active, ie. noapic on
> > > the kernel command line will fix it as well as disabling
> > > CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it
> > > happens though.
> > 
> > Ditto.
> > 
> > > The only difference with IO APIC disabled seems to be that the irq
> > > doesn't get ACKed before update_process_times() gets called.
> > > And your "fix" makes it being called outside of the xtime_lock spinlock
> > > and with a slightly different stack usage AFAICT.
> > 
> > (it's still under the xtime lock)
> 
> No, with IO-APIC enabled, it's using the local APIC timer, thus
> using_apic_timer is 1 and the path right after unlocking in
> timer_interrupt() is taken towards update_process_times().
> 
> > > But none of these should make a difference, right?
> > 
> > Not that I can see, but then it's pretty dark down here.  Anybody got a
> > flashlight I can borrow? ;-)
> 
> Guess I found one, not sure if it works correctly though ;)
> 
> Using 4K stacks, we have one stack for hard irqs and one for soft irqs,
> both having their own threadinfo and thus preemptcount. Thus the call to
> softirq_count() in account_system_time() will always return 0 when
> called in hard irq context. Additionally preemptcount is always set to
> HARDIRQ_OFFSET in hard irq context, so
> hardirq_count() - hardirq_offset is 0 all the time as well.
> 
> But that doesn't fit the fact that I at least get hard irq accounting
> when booting with noapic. And it also doesn't explain why your fix
> works, fixing both, soft and hard irq accounting. Am I missing some
> code path that calls smp_local_timer_interrupt? There's
> smp_apic_timer_interrupt(), but that seems to be unused on i386.

OK, the hardirq_count() thing is not HARDIRQ_OFFSET all the time. When
we interrupt a hard irq, we are already using the hard irq stack and
thus irq_enter() in do_IRQ() works as it should. The zero value for hi
is there with IO-APIC and 8K stacks as well, so that's either right or
an other bug.

Still no idea why your "fix" works, but the following patch also fixes
the problem and is at least a little more like the RightThing.


---

Copy the softirq bits in preempt_count from the current context into the
hardirq context when using 4K stacks to make the softirq_count macro work
correctly and thereby fix softirq cpu time accounting.

Signed-off-by: Björn Steinbrink <B.Steinbrink@gmx.de>

diff -Nurp a/arch/i386/kernel/irq.c b/arch/i386/kernel/irq.c
--- a/arch/i386/kernel/irq.c	2006-03-20 06:53:29.000000000 +0100
+++ b/arch/i386/kernel/irq.c	2006-06-25 15:49:52.000000000 +0200
@@ -95,6 +95,14 @@ fastcall unsigned int do_IRQ(struct pt_r
 		irqctx->tinfo.task = curctx->tinfo.task;
 		irqctx->tinfo.previous_esp = current_stack_pointer;
 
+		/*
+		 * Copy the softirq bits in preempt_count so that the
+		 * softirq checks work in the hardirq context.
+		 */
+		irqctx->tinfo.preempt_count =
+			irqctx->tinfo.preempt_count & ~SOFTIRQ_MASK |
+			curctx->tinfo.preempt_count & SOFTIRQ_MASK;
+
 		asm volatile(
 			"       xchgl   %%ebx,%%esp      \n"
 			"       call    __do_IRQ         \n"

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

[Arjan van de Ven]

On Sun, 2006-06-25 at 16:24 +0200, Björn Steinbrink wrote:
> Copy the softirq bits in preempt_count from the current context into the
> hardirq context when using 4K stacks to make the softirq_count macro work
> correctly and thereby fix softirq cpu time accounting.
> 
> Signed-off-by: Björn Steinbrink <B.Steinbrink@gmx.de>
> 
> diff -Nurp a/arch/i386/kernel/irq.c b/arch/i386/kernel/irq.c
> --- a/arch/i386/kernel/irq.c	2006-03-20 06:53:29.000000000 +0100
> +++ b/arch/i386/kernel/irq.c	2006-06-25 15:49:52.000000000 +0200
> @@ -95,6 +95,14 @@ fastcall unsigned int do_IRQ(struct pt_r
>  		irqctx->tinfo.task = curctx->tinfo.task;
>  		irqctx->tinfo.previous_esp = current_stack_pointer;
>  
> +		/*
> +		 * Copy the softirq bits in preempt_count so that the
> +		 * softirq checks work in the hardirq context.
> +		 */
> +		irqctx->tinfo.preempt_count =
> +			irqctx->tinfo.preempt_count & ~SOFTIRQ_MASK |
> +			curctx->tinfo.preempt_count & SOFTIRQ_MASK;
> +
>  		asm volatile(
>  			"       xchgl   %%ebx,%%esp      \n"
>  			"       call    __do_IRQ         \n"

Hi,

at first I got nervous about the asymmetry of this (eg why only do this
copying only on entry, and not a copy back on exit)... but then again
these bits shouldn't change so your patch is ok as is...
it's regrettable that we need to add this for the softirq accounting;
part of me wishes we would just count irq-vs-user time and be done with
it ;)


Acked-by: Arjan van de Ven <arjan@linux.intel.com>

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

[Mike Galbraith]

On Sun, 2006-06-25 at 16:24 +0200, Björn Steinbrink wrote:

> Still no idea why your "fix" works, but the following patch also fixes
> the problem and is at least a little more like the RightThing.

Yeah.  I don't know about you, but I fully intend to blatantly ignore
that 'why' ;-)

	-Mike

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

[Arjan van de Ven]

On Sun, 2006-06-25 at 19:44 +0200, Mike Galbraith wrote:
> On Sun, 2006-06-25 at 16:24 +0200, Björn Steinbrink wrote:
> 
> > Still no idea why your "fix" works, but the following patch also fixes
> > the problem and is at least a little more like the RightThing.
> 
> Yeah.  I don't know about you, but I fully intend to blatantly ignore
> that 'why' ;-)

the why is relatively easy ;)

since the "is a softirq executing" bit is on the stack, and each context
(user, soft and hard irq) has their own stack, it's not automatic that
the hardirq stack gets the softirq-executing flag... which your patch
fixes.

NMI's and apic irqs generally don't go via the normal irq path and thus
don't do a stack switch... so they don't lose the bit (for accounting
purposes)

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

[Björn Steinbrink]

On 2006.06.25 19:43:17 +0200, Arjan van de Ven wrote:
> On Sun, 2006-06-25 at 19:44 +0200, Mike Galbraith wrote:
> > On Sun, 2006-06-25 at 16:24 +0200, Björn Steinbrink wrote:
> > 
> > > Still no idea why your "fix" works, but the following patch also fixes
> > > the problem and is at least a little more like the RightThing.
> > 
> > Yeah.  I don't know about you, but I fully intend to blatantly ignore
> > that 'why' ;-)
> 
> the why is relatively easy ;)
> 
> since the "is a softirq executing" bit is on the stack, and each context
> (user, soft and hard irq) has their own stack, it's not automatic that
> the hardirq stack gets the softirq-executing flag... which your patch
> fixes.

That's mine, not Mike's. Mike's patch removed the #ifdef CONFIG_SMP
around update_process_times() in smp_local_timer_interrupt().

> NMI's and apic irqs generally don't go via the normal irq path and thus
> don't do a stack switch... so they don't lose the bit (for accounting
> purposes)

Hm, doesn't that mean that mean that hardirq accounting is still broken?
APIC irq comes in, increases hardirq count, then the timer irq fires,
switches the stack and looses the hardirq count that was incremented by
the APIC irq.

I just booted with both patches applied, mine and Mike's, and that
actually makes a difference in hardirq cpu time accounting. With my
patch only, hi is 0 in top while the box gets a ping flood. With both
patches, I get about 1% hi. Mike's patch causes update_process_times()
to be called twice on UP, but that alone shouldn't change the
percentages, right?
OTOH top shows "hi" as zero with 8K stacks as well unless Mike's patch
is applied, so the results with Mike's patch are bogus (if so, why?) or
hardirq accounting is broken in general.

Btw, which path do apic irqs go? I stumbled across the nmi stuff, but
didn't see anything special for the apic irqs.

Björn

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

[Mike Galbraith]

On Sun, 2006-06-25 at 20:42 +0200, Björn Steinbrink wrote:
> I just booted with both patches applied, mine and Mike's, and that
> actually makes a difference in hardirq cpu time accounting. With my
> patch only, hi is 0 in top while the box gets a ping flood. With both
> patches, I get about 1% hi. Mike's patch causes update_process_times()
> to be called twice on UP, but that alone shouldn't change the
> percentages, right?

Yes, you definitely need to comment out the other call if you test the
SMP path on UP+IO-APIC.

> OTOH top shows "hi" as zero with 8K stacks as well unless Mike's patch
> is applied, so the results with Mike's patch are bogus (if so, why?) or
> hardirq accounting is broken in general.

Something is certainly still b0rken.  I still get three different
answers to the question "what is my cpu usage" depending on
configuration.  With stock UP kernel with no IO-APIC, interrupt load is
all hi.  With your patch and IO-APIC, it's all si.  SMP shows a mix of
both.

I like the result of using the SMP path if you have an IO-APIC best,
though I haven't verified them against a profile for accuracy.  Taking a
peek at the profile confirms that it is indeed mixed, so anything
showing the load as being either hi or si has to be wrong.

	-Mike

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

[Mike Galbraith]

On Mon, 2006-06-26 at 04:23 +0200, Mike Galbraith wrote:

> Something is certainly still b0rken.  I still get three different
> answers to the question "what is my cpu usage" depending on
> configuration.  With stock UP kernel with no IO-APIC, interrupt load is
> all hi.  With your patch and IO-APIC, it's all si.  SMP shows a mix of
> both.

I didn't say that quite right.

The third case for my box is XT-PIC and your patch.  Previously, top
showed the 10% interrupt load of a flood ping as all hi.  Now it's both
hi and si when using XT-PIC, but it's no longer the 10% that agreed with
the profile, it's  10% hi, but with an added ~7% si.

	-Mike

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

[Björn Steinbrink]

On 2006.06.26 05:05:14 +0200, Mike Galbraith wrote:
> On Mon, 2006-06-26 at 04:23 +0200, Mike Galbraith wrote:
> 
> > Something is certainly still b0rken.  I still get three different
> > answers to the question "what is my cpu usage" depending on
> > configuration.  With stock UP kernel with no IO-APIC, interrupt load is
> > all hi.  With your patch and IO-APIC, it's all si.  SMP shows a mix of
> > both.
> 
> I didn't say that quite right.
> 
> The third case for my box is XT-PIC and your patch.  Previously, top
> showed the 10% interrupt load of a flood ping as all hi.  Now it's both
> hi and si when using XT-PIC, but it's no longer the 10% that agreed with
> the profile, it's  10% hi, but with an added ~7% si.

I see the following here, when a box is being ping flooded:

UP/SMP  stack-size  PIC-type  hi si
-----------------------------------
UP      8K          IO-APIC  0  16
UP      8K          XT-PIC   7  7
UP      4K          IO-APIC  0  16
UP      4K          XT-PIC   7  7

SMP     8K          IO-APIC  5  4 (forcedeth)
SMP     8K          IO-APIC  0  11 (tg3)

The UP system is a Thinkpad which only has a tg3 driven NIC. The SMP
system is x86_64, so there's no 4K stacks to test with, maybe I'll fetch
a i386 live CD to do some more valid tests on SMP.

The UP tests seem to show the IO-APIC hardirq are completely deferred
to be handled in a softirq, as the sum of "hi" and "si" with XT-PIC is
about equal to the "si" value with IO-APIC (the numbers are guessed
averages of the observed values, so the 2% difference is not too be
taken too serious).
The results for the forcedeth driven NIC do not agree though, and your
results differ from what I see as well, right? So I'm kinda lost again.

Björn

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

[Mike Galbraith]

On Mon, 2006-06-26 at 19:58 +0200, Björn Steinbrink wrote:

> The results for the forcedeth driven NIC do not agree though, and your
> results differ from what I see as well, right? So I'm kinda lost again.

I'm not.  Not any more.  After much fruitless rummaging, I did some more
profiling to verify the numbers.  It really does take that much more cpu
when booted with noapic, nearly 50%!  I've been chasing swamp gas.

Oh well.  I got a warm fuzzy wrt tools cpu usage numbers out of it.

Stick a fork in this bug, it's done ;-)

	-Mike

Re: Measuring tools - top and interrupts

[Danial Thom]

--- Björn Steinbrink <B.Steinbrink@gmx.de> wrote:

> On 2006.06.25 07:06:33 +0200, Mike Galbraith
> wrote:
> > On Sat, 2006-06-24 at 21:25 +0200, Björn
> Steinbrink wrote:
> > > On 2006.06.24 18:23:12 +0200, Mike
> Galbraith wrote:
> > > > 
> > > > let's see.  Yeah, confirmed.
> > > 
> > > OK, it also depends on IO APIC being
> enabled and active, ie. noapic on
> > > the kernel command line will fix it as well
> as disabling
> > > CONFIG_X86_IO_APIC. That doesn't help me at
> all to understand why it
> > > happens though.
> > 
> > Ditto.
> > 
> > > The only difference with IO APIC disabled
> seems to be that the irq
> > > doesn't get ACKed before
> update_process_times() gets called.
> > > And your "fix" makes it being called
> outside of the xtime_lock spinlock
> > > and with a slightly different stack usage
> AFAICT.
> > 
> > (it's still under the xtime lock)
> 
> No, with IO-APIC enabled, it's using the local
> APIC timer, thus
> using_apic_timer is 1 and the path right after
> unlocking in
> timer_interrupt() is taken towards
> update_process_times().
> 
> > > But none of these should make a difference,
> right?
> > 
> > Not that I can see, but then it's pretty dark
> down here.  Anybody got a
> > flashlight I can borrow? ;-)
> 
> Guess I found one, not sure if it works
> correctly though ;)
> 
> Using 4K stacks, we have one stack for hard
> irqs and one for soft irqs,
> both having their own threadinfo and thus
> preemptcount. Thus the call to
> softirq_count() in account_system_time() will
> always return 0 when
> called in hard irq context. Additionally
> preemptcount is always set to
> HARDIRQ_OFFSET in hard irq context, so
> hardirq_count() - hardirq_offset is 0 all the
> time as well.
> 
> But that doesn't fit the fact that I at least
> get hard irq accounting
> when booting with noapic. And it also doesn't
> explain why your fix
> works, fixing both, soft and hard irq
> accounting. Am I missing some
> code path that calls smp_local_timer_interrupt?
> There's
> smp_apic_timer_interrupt(), but that seems to
> be unused on i386.
> 
> Björn

I think the one thing we can surmise from this
thread is that you can't rely on kernel usage
statistics to be accurate, as its likely that
there are many, many cases that don't work
properly. It was always wrong in 2.4 as well.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Measuring tools - top and interrupts

[Mike Galbraith]

On Sun, 2006-06-25 at 13:45 -0700, Danial Thom wrote:

> I think the one thing we can surmise from this
> thread is that you can't rely on kernel usage
> statistics to be accurate, as its likely that
> there are many, many cases that don't work
> properly. It was always wrong in 2.4 as well.

Once identified, problems tend to get fixed.  This one will probably be
history soon.  You know the old saying though... "There are lies, there
are _damn_ lies, and then there are _statistics_".

Re: Measuring tools - top and interrupts

[Danial Thom]

--- Mike Galbraith <efault@gmx.de> wrote:

> On Sun, 2006-06-25 at 13:45 -0700, Danial Thom
> wrote:
> 
> > I think the one thing we can surmise from
> this
> > thread is that you can't rely on kernel usage
> > statistics to be accurate, as its likely that
> > there are many, many cases that don't work
> > properly. It was always wrong in 2.4 as well.
> 
> Once identified, problems tend to get fixed. 
> This one will probably be
> history soon.  You know the old saying
> though... "There are lies, there
> are _damn_ lies, and then there are
> _statistics_".

The usefulness of statistics is a function of the
breadth of understand of the person interpreting
them. 

But I don't think that applies here, because we
are not questioning the conclusions from the
statistics, only the accuracy of the gathering.
Its a completely different thing to say, for
example, that population growth is due to too
many babies when it might be immigration or a
reduction in the death rate, than it is to say
that the numbers gathered are simply wrong.

The truth is that you guys can't be bothered
unless the numbers are so far off that its an
embarrasment. Until someone who cares about it
(and knows  how to do it) puts their nose to the
grindstone and makes sure that everything is
being done correctly, they'll be largely useless.


DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Measuring tools - top and interrupts

[Mike Galbraith]

On Fri, 2006-06-30 at 06:41 -0700, Danial Thom wrote:
> 
> --- Mike Galbraith <efault@gmx.de> wrote:
> 
> > On Sun, 2006-06-25 at 13:45 -0700, Danial Thom
> > wrote:
> > 
> > > I think the one thing we can surmise from
> > this
> > > thread is that you can't rely on kernel usage
> > > statistics to be accurate, as its likely that
> > > there are many, many cases that don't work
> > > properly. It was always wrong in 2.4 as well.
> > 
> > Once identified, problems tend to get fixed. 
> > This one will probably be
> > history soon.  You know the old saying
> > though... "There are lies, there
> > are _damn_ lies, and then there are
> > _statistics_".
> 
> The usefulness of statistics is a function of the
> breadth of understand of the person interpreting
> them. 
> 
> But I don't think that applies here, because we
> are not questioning the conclusions from the
> statistics, only the accuracy of the gathering.
> Its a completely different thing to say, for
> example, that population growth is due to too
> many babies when it might be immigration or a
> reduction in the death rate, than it is to say
> that the numbers gathered are simply wrong.
> 
> The truth is that you guys can't be bothered
> unless the numbers are so far off that its an
> embarrasment.

Hey, let's play hide and go fuck yourself.  Tag, you're it.

*plonk*

	-Mike

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

[Chuck Ebbert]

In-Reply-To: <20060625184244.GA11921@atjola.homenet>

On Sun, 25 Jun 2006 20:42:44 +0200, Bjorn Steinbrink wrote:

> Btw, which path do apic irqs go? I stumbled across the nmi stuff, but
> didn't see anything special for the apic irqs.

arch/i386/kernel/entry.S has the macro BUILD_INTERRUPT(name, nr).

The code that uses this macro is in arch/i386/mach-*/entry_arch.h.

The macro prepends "smp_" to the name passed to the macro and the
generated asm code calls that function after saving registers, etc.

arch/i386/kernel/apic.c::apic_intr_init() calls set_intr_gate() to
point some of the interrupt gates at the correct functions.

-- 
Chuck
 "You can't read a newspaper if you can't read."  --George W. Bush

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

[Björn Steinbrink]

On 2006.06.26 18:54:00 -0400, Chuck Ebbert wrote:
> In-Reply-To: <20060625184244.GA11921@atjola.homenet>
> 
> On Sun, 25 Jun 2006 20:42:44 +0200, Bjorn Steinbrink wrote:
> 
> > Btw, which path do apic irqs go? I stumbled across the nmi stuff, but
> > didn't see anything special for the apic irqs.
> 
> arch/i386/kernel/entry.S has the macro BUILD_INTERRUPT(name, nr).
> 
> The code that uses this macro is in arch/i386/mach-*/entry_arch.h.
> 
> The macro prepends "smp_" to the name passed to the macro and the
> generated asm code calls that function after saving registers, etc.
> 
> arch/i386/kernel/apic.c::apic_intr_init() calls set_intr_gate() to
> point some of the interrupt gates at the correct functions.

Thanks, my view on the code was too cscope-centric and the grep runs
searched for the full function name.
That finally explains why calling update_process_times() in
smp_local_timer_interrupt() makes a difference on UP.

Björn